Unify & simplify render & theme system: component-based rendering (enables pattern library, style guides, interface previews, client-side re-rendering)

I wonder where this fits now that we have SDC, and Experience Builder is in early development?

The Ideas project is being deprecated. This issue is moved to the Drupal project. Check that the selected component is correct. Also, add the relevant tags, especially any 'needs manager review' tags.

I wonder where this fits now that we have SDC, and Experience Builder is in early development?

We still need to work on this because:

• SDC is addressing only UI components, an important part of a modern Render API but not everything
• Experience Builder is a display builder, so an higher level tool. We need to address the lower levels

Nearly 5 years after my previous comment, here we are.

Current status

Let's not forget the Render API is great:

• Declarative: Easy to type. (de)Serializable if clean.
• Easy nesting: The Virtual DOM of Drupal. We are building a tree.
• Data bubbling: Declare locally, impact globally
• Asset libraries management: Our beloved libraries.yml
• Clever caching: Context, tags, keys…

❤️ Building display by assembling configurable plugins returning renderables.

However, the renderables themselves are an issue.

They are too many of them and they are of 3 kinds:

• 35 render elements in Core:
  

  $ grep -hoEr "#type' => '(\S+)'" core/ --exclude-dir tests | sort | uniq -c | sort -nr
  138 #type' => 'details'
    94 #type' => 'container'
    93 #type' => 'link'
    68 #type' => 'table'
    47 #type' => 'actions'
    36 #type' => 'inline_template'
    27 #type' => 'fieldset'
    23 #type' => 'html_tag'
    21 #type' => 'status_messages'
    14 #type' => 'pager'

• 99 theme hooks in Core:
  

  $ grep -hoEr "#theme' => '(\S+)'" core/ --exclude-dir tests | grep -v "__" | sort | uniq -c | sort -nr
  84 #theme' => 'item_list'
  21 #theme' => 'username'
  16 #theme' => 'image'
    7 #theme' => 'status_messages'
    7 #theme' => 'image_style'
    6 #theme' => 'table'
    6 #theme' => 'links'
    5 #theme' => 'update_version'
    5 #theme' => 'indentation'
    5 #theme' => 'file_upload_help'
    …
  

• 2 special ones:
  

     1016 '#markup'
         85 '#plain_text'
  

😱 135 renderables to learn/use? This is the issue.

What can we remove?

We have SDC for UI components, so:

• #type => component can replace a lot of hook_theme, the one describing UI components: breadcrumb, progress_bar, links...
• We can also replace the 34 render elements which are wrapper around hook_theme (because what they do, adding asset libraries, variable typing... is already done by SDC) we removed : table, status_messages, pager...

The hook_theme not suitable for for conversion to SDC are often annoying wrappers we can get rid of anyway : block, node, view, field…

The few remaining render elements can't be expressed as UI components:

• some must be kept because they represent other design systems artefacts (icons...) or low level bricks (HTML elements, placeholders...)
• some may (not found in my today look) be shortcut to Drupal callables returning renderables, it is better to explicitly call the callable (service, function, method..) instead

Deprecation of ThemeManager::render()

If all template file based renderable are managed by SDC and because SDC is not passing through the ThemeManager, we dont need the template loading feature of the ThemeManager and we get remove it to get roi of the old confusing stuff:

• Theme wrappers: Confusing and useless. You can always use an upper level instead.
• Templates suggestions: Not discoverable. Messy. Blur the business / view separation.
• Preprocess hooks: Risky. Unfriendly. Blur the business / view separation. Unpredictable order of execution.

Conclusion

We can have an equally useful Render API with less than 10 renderables (not counting Form API):

• Design systems artefacts:
  • #type => component: for UI components
  • #type => icon: for UI components
• Lower level bricks:
  • #type => inline_template
  • #type => html_tag : for HTML elements
  • #type => link
  • #markup
  • #plain_text
  • #lazy_builder

  Keeping the beloved universal render properties: #attached, #cache... and adding some related to design systems: #styles, #tokens, #mode...

  And now we have an purely design/UI based Render API!

  If we do that, step by step, deprecating slowly, not breaking anything, the display building tools from Core and Contrib will have a simpler implementation and will have an easier time introducing new features.

  To play safe, we can also introduce a new Render API alongside the existing, using @ instead of # for example:

  • #type => component, #component= "foo:bar" ⮕ @component => "foo:bar"
  • #type => icon ⮕ @icon => []
  • #type => inline_template, #template => "foo" ⮕ @template => "foo"
  • #type => html_tag, #tag => "foo" ⮕
    

    @element => "foo"</li>
        <li><code>#type =>link

    ⮕ @link

  • #markup ⮕ @markup</li>
  • #plain_text ⮕ @plain_text
  • #lazy_builder ⮕ @lazy_builder

I'm going to take the optimism path and assume that moving this from "ideas" to a real core issue is a promotion which increases its visibility and chances for success. 😅

I don't fully understand #type => component or how to entirely get rid of preprocess via SDCs yet (would be great to get separate issues for those opened maybe?), but everything else in #99 I'm 100% agreed with, and I would love to get rid of preprocess and as many (all?) theme hooks as possible.

Adding three concrete active issues that could use reviews/help.

There are also a lot of open child issues of #1804614: [meta] Consolidate similar twig templates and properly use theme suggestions in core → still.

how to entirely get rid of preprocess via SDCs yet

I'm more likely to implement a twig extension instead of a preprocess function nowadays. Maybe it would be enough to just improve the DX around twig extensions. E.g., attributes:

#[TwigFilter('without')]
public function withoutFilter($element, ...$args) {
// [...]
}

#[TwigFunction('url', options: ['is_safe_callback' => Foo::isUrlGenerationSafe])]
public function getUrl($name, $parameters = [], $options = []) {
// [...]
}

but everything else in #99 I'm 100% agreed

@catch in #106 ? ;)

I don't fully understand #type => component or how to entirely get rid of preprocess via SDCs yet

In UI Suite community (not only the contributors in drupal.org, but also the agencies doing projects I am interacting with), we are not using preprocess hooks anymore, for many years, except for forms. I don't really know why it is a less an issue for us. Maybe because we fully switched to a design system + display builder paradigm. It is worth investigating and writing that down.

Adding three concrete active issues that could use reviews/help.

Thanks you. I will have a look after Atlanta.

I'm more likely to implement a twig extension instead of a preprocess function nowadays.

Twig extensions have a high risk of being harmful:

• Written in PHP, they may hide some business logic specific to a specific project, or calls to CMS API. A SDC component is just a "dumb" piece of UI logic receiving already resolved data.
• Written in PHP, they may introduce PHP in a Drupal theme (which we cant to avoid, a theme must be a front dev friendly place) or a dependency to module.
• Instead of fixing the root cause (why this data is not OK to be injected in my template? We are moving the problematic mechanism elsewhere

In UI Suite community (not only the contributors in drupal.org, but also the agencies doing projects I am interacting with), we are not using preprocess hooks anymore, for many years, except for forms. I don't really know why it is a less an issue for us.

The projects might not be using them directly, but there are dozens/hundreds of preprocess hooks running on those projects via core and contrib modules. https://api.drupal.org/api/drupal/core%21modules%21node%21node.module/fu... is a monster just by itself. Removing just one variable is non-trivial, although at least we have deprecation support now.

The projects might not be using them directly, but there are dozens/hundreds of preprocess hooks running on those projects via core and contrib modules.

Of course.

Also, it may be the opportunity to challenge the various additions Drupal made to Twig last 10 years.

Today, in my very personal and humble opinion, I see only 5 additions which will stay relevant if we move to SDC, decoupling between UI logic and business logic, and design system philosophy:

• the PrintNode visitor to send all printed variable to the renderer service. That's the core of Drupal Twig rendering, it works very well and must be kept.
• Attribute object and the related create_attribute() function, but I hope this issue will allow us to get rid of it one of those days: ✨ HTML attributes as Twig mappings instead of PHP obejcts Active
• add_class() & set_attribute() filters, they make sense used with the SDC slots.
• the new icon() function for the Icon API. It follows Twig philosophy of using functions to generate printable data
• the translations helpers (t function, trans tag). Important feature for a CMS. Why not proposing this to upstream Twig?

So, other Drupal additions (clean_id(), attach_library(), link(), |add_suggestion()...) seems OK to be deprecated to me, once we have "cleaned" the Render API. I would happy to discuss this.

Coming over from 📌 Make Drupal core multi-frontend Active
@catch - Thank you for pointing me to this issue.

Moving it all over to components and solving challenges like getting rid of pre-processors would be an amazing and a big first step for moving multi-frontend. However, there is another related level of complexity in Drupal theming: The back and forth between Twig and Drupal rendering.

It's been a while since, a tried a twig-powered component approach with Drupal. pdureau, could you share you experience on this, I assume this is stilll how it works, i.e. when you build up a page with components? Like a node component rendering fields, and the fields "content" triggering the rendering of a field renderable again, would again render nodes and... - you get the idea. In my experience, that back and forth leads to very complex control flow, that becomes hard to debug when needed.

Instead, I'd love to have a a tree of components generated first, then in a second step have it rendered. When required for performance, it could have some explicit support for lazy-rendering/generating a sub-tree of components, but I think that should be explicit and used only rarely. If we could generate that kind of tree, it would be very hand to debug/see what the frontend is going to output by e.g. exposing that tree via API (optionally expanded). This would aid theming DX quite a bit imo.

Second, a rendered component tree served via API comes with the additional benefit of becoming renderable by different frontends. E.g., this is all what Lupus Decoupled does, make Drupal serve a tree of components via API, so a Vue/Nuxt frontend can pick it up.

Instead, I'd love to have a a tree of components generated first, then in a second step have it rendered.

This sounds like experience builder, and when experience builder is controlling the entire page, then it would probably apply to both the 'site chrome' and any entity content so effectively the entire page - although only when it's being used in this way.

it could have some explicit support for lazy-rendering/generating a sub-tree of components, but I think that should be explicit and used only rarely.

This is already explicit in the render API - you only get lazy rendering if you use #lazy_builder, and then placeholdering can be determined automatically but doesn't have to be. And you only get a render cache entry if you specify #cache keys and a #pre_render or lazy_builder callback.

However 'using it rarely' would go against a lot of current work in 🌱 [meta] Placeholder-driven performance improvements Active and 🌱 Adopt the Revolt event loop for async task orchestration Active as well as the existing render cache/dynamic page cache.

There are ways to provide the same async behaviour without actually using placeholders as long as you can iterate over items at the same level of the tree, for example see the implementation in 📌 Views entity rendering should use a lazy builder/placeholder Active which doesn't rely on #lazy_builder but does allow for async rendering. So we could reverse or partially reverse some of the placeholdering but keep the async, as long as the implementation allowed for a similar appropriate place to use revolt/an event loop.

But I don't see how to do render caching without #pre_render/#lazy_builder + #cache keys being used at least as much as it is now. When e.g. a views listing is not in the render cache, then you need to build up a full render array to render it, but when the views listing is cached, you only need the view/display ID - the whole point of render caching is to avoid having to run the listing query and load and render all the entities within it.