Performance issue when processing ui_patterns form slots

Problem/Motivation

This is a preliminary report, I'm hoping someone who knows the implementation can chime in.

I noticed a significant performance hit when rendering ui_patterns form slots.
Opening layout builder block settings forms that uses ui_patterns and slots can take a very long time (more than 60s in some scenarios).

I attached a PDF containing a request that takes 60s when opening the "configure" block form in layout builder on my project.
We can see more than 2M call counts in Drupal\Core\Plugin\Context\ContextDefinition.
Many seem to originate from uasort in \Drupal\ui_patterns\Element\ComponentFormBase::orderSources here (lines 439-444)

Steps to reproduce

This can be reproduced to some extent with a clean setup.
With demo_umami it doesn't seem as bad as on my project but it is still enough to grasp the issue. Maybe it needs more content types/fields to really become a problem.

- Install umami: drush si demo_umami
- Enable ui_patterns modules: drush en ui_patterns ui_patterns_blocks ui_patterns_field_formatters ui_patterns_layouts ui_patterns_library ui_patterns_views
- Go to /admin/structure/types/manage/article/display/full/layout and add a UI patterns block with slots. Inject fields in those slots with [Entity] ➜ [Field].
- Profile the request when clicking "configure" for that block

Proposed resolution

Commenting out the uasort calls in ComponentFormBase::orderSources does improve performance and at least makes the system usable. However, based on the call counts, the performance issue appears to go beyond that.

Remaining tasks

Provide better steps to reproduce so the performance issue is obvious.
Investigate the root cause and determine the appropriate solution.

User interface changes

N/A

API changes

?

Data model changes

?