Ability to configure multiple browsers

What we use to facilitate what you describe is https://www.drupal.org/project/layout_builder_restrictions →
It allows to set the allowed blocks per content type per region. Maybe not 100% what you describe.

Thanks for your feedback.

Today layout_builder_browser is not only providing the browser experience but it even defines which blocks to be displayed and group them by category. I understand that layout_builder_restrictions allows to select which layout or blocks are allowed but does not impact the contribution experience.

I need to check how the browser behaves today if some of the blocks configured are actually disabled / not allowed via layout_builder_restrictions. Ideally these blocks should not be listed in the browser. It would cover most of the use case I was describing. Only use case not covered is a block which should belong to category X in one content type layout builder UI and to category B for another content type. But it is probably an edge case which does not deserve to be taken in account directly by the module but should be possible using some alter hooks (which may already be possible).
If the browser is not impacted by layout_builder_restrictions, I think this should be the way to do to avoid rebuilding the restriction mechanism.

I will toggle entire categories by implementing https://git.drupalcode.org/project/layout_builder_browser/-/blob/8.x-1.x... to get "browsers" per group type.