Rendering a toolbar with a lot of links is slow

Problem/Motivation

admin_toolbar_tools adds a lot of links to the toolbar.
In our dev environment without cache almost half of the rendering time is spent on the toolbar.

We identified two things that are slow:

1. The access checks on each link. We used to have a workaround in admin_toolbar_links_access_filter_filter_non_accessible_links() that bypasses access checks for admin users but now admin_toolbar_links_access_filter is deprecated.
2. The rendering of the links. If in menu--toolbar.html.twig we replace {{ link(item.title, item.url) }} with <a href="{{ item.url }}">{{ item.title }}</a> it becomes much faster but it feels dirty.

Steps to reproduce

Disable cache entirely.
Measure the time it takes to generate a page with and without admin_toolbar_tools.

Proposed resolution

I'm not sure what can be done here, the module does not really do anything wrong, it's just the sheer amount of links that makes things slow.

Remaining tasks

User interface changes

API changes

Data model changes