[meta] Improve tracing and available information

We've started to use and look into tracing a bit more and currently, it's pretty bare bones. There are a number of specific issues and ideas I have, and I'm creating this issue to write them down, discuss them and also bundle related issues together.

Note: we use this together with SigNoz, so I primarily have experience with that.

For now just listing some topics, will structure/group this more later.

Issues:

• Improve trace names: 📌 Request span name is too specific Needs review , [3525383]

Ideas to explore/discuss:

• Improve handling around the two spans and root span reporting, this might partially be a SigNoz specific issue. It only identifies drush requests as root spans, all web requests are reported with a missing root span. The two spans being created are also still a bit unclear to me and should maybe have different names. I don't currently see benefit with the information they provide
• Spans for views, in my experience, views can be among the slowest things especially if badly configured with many joins. Could include attributes for view id, display, arguments
• option to only report slow queries. Especially on cold caches, drupal can execute a huge amount of queries. That's not great, but not much can be done about that usually, so I'm wary of enabling query spans. However, if it could be combined with a configurable min-execution time, then that could be more useful
• Make more information available as attributes as events. Right now I think that could be more useful than spans, to help group related requests and ignore outliers. Examples that I can think of:
  • max memory usage
  • authenticated: yes/no. maybe roles. idea would be to easily ignore or focus on requests from editors working on the backend
  • cache status: basically the value of the x-drupal-cache and x-drupal-dynamic-cache headers. Allows to focus for example on dynamic page cache performance, uncacheable requests and so on
  • Certain database/cache metrics: Instead of reporting them as full spans, a more lightweight approach could be to just report an attribute with the amount of queries or cache lookups. Similar to how drupal core performance tests assert things