Unify & simplify render & theme system: component-based rendering (enables pattern library, style guides, interface previews, client-side re-rendering)

Problem/Motivation

Drupal's render & theme system are too complex to use. Let's improve this.

Goals

1. Improve the TX by un-WTF-ifying the theme and render system, which is currently a maze of:
   • hook_theme() (with variables vs render element), preprocess hooks, theme suggestions and many more related hooks — all tied together using the theme registry
   • #type (@RenderElement plugins) vs. #theme, which are kind of the same thing but not really: when to use which is unclear
   • callback buffet: #pre_render, #post_render, #lazy_builder
   • mysterious keys in enormous render arrays AKA render arrays of DOOM
   • #render_children, #theme_wrappers and friends determining where the render system morphs into the theme system and back again
   • different systems calling each other: understanding the entire flow is nearly impossible, and probably rivals the complexity of some simpler biological organisms
   • — @c4rl

   These have been known problems for years! First there was Form API, then Render API sprouted from that, and all the while there was the theme system, but starting in Drupal 7, the Render API and the theme system got deeply, deeply intertwined. Drupal 8 actually made it slightly better, but not enough.

2. Improve the TX for non-JS front end people ():
   • automatically generated pattern library (== all #type/@RenderElements — but without having to know those details)
   • automatically generated style guides for every theme (== pattern library with the theme's overrides/extensions applied)

   in other words: bring style guide-driven development to Drupal as a default rather than a labor-intensive, hacked-on after-thought (without the need to duplicate markup and thus keep them in sync).

3. Long-term: make it possible to reuse templates on the client-side ().
4. Retain compatibility with the existing Render & Theme systems. Allow for a gradual transition.
5. Support interface previews: ✨ Interface previews/skeleton screens through optional "preview" or "placeholder" templates Fixed

Requirements

To address all of those goals, I believe a component library can be the gateway to a solution. It can be, if the following requirements are met:

1. components are not deeply tied to Drupal, and in fact, can be developed independently of Drupal — this is how we can guarantee simplicity and ease of getting started: we actively prevent components from being tightly coupled to Drupal code
2. components have:
   1. markup: *.html.twig (Twig template — which may include some logic to process received variables, just like in Drupal 8)
   2. assets: CSS, JS files
   3. metadata: YAML file

   Nothing else.

3. modules and themes can specify components — modules can define patterns that any other module can use, themes can specify theme-specific components
4. components are defined in a simple directory structure:
   

   <extension> (module or theme)
    |- components
       |- <component name>
          |- <component name>.yml
          |- <component name>.twig
   

   Concrete example:

   core/modules/system      
    |- components           
       |- label             
          |- label.yml      
          |- label.html.twig
          |- label.css
   cores/themes/classy      
    |- components           
       |- label             
          |- label.yml      
          |- label.html.twig
          |- ajaxified-label.js
   

5. The YAML file specifies:
   1. the variables (inputs) of the component. For each variable:
      • type: only A) primitives such as string/integer/bool, B) arrays of primitives such as string[], C) other components: component or components to slot in other components (perhaps even component:<name> to only allow certain components ) — this enables 3 big wins:
        1. improved TX: type validation, to avoid weird bugs
        2. improved TX: no messy Doxygen/PHPDoc comments repeated in both templates and preprocess functions, and all overrides of either of those
        3. client-side re-rendering

        (Also: having type specifications in Twig templates instead is A) undesirable, B) quite likely impossible, C) quite likely impossible to parse without refactoring Twig, D) would pick up calculated variables.)

      • description
      • default value, if any
      • example value (to be used in pattern library & style guide)
      • preview value (to be used when the component's data is not available yet, because it's being used for an interface preview)
   2. documentation: purpose, when to use, how to use, accessibility, related links — in other words: information to show in the pattern library & style guides
   3. less important metadata: human-readable name, which other components this component includes, whether this component supports interface previews …
6. The Twig template (*.html.twig) performs all the necessary processing of the variables received, this ensures we don't depend on preprocess functions. This removes the need for front-end developers to dive into PHP.
7. components can be extended: add attributes, modify markup, and so on
8. components can be composed: combine multiple components to create a new component
9. To allow for a gradual transition, we cannot fully back away from render arrays nor the existing theme system. At best, we'll be able to remove render arrays, the current render system and the current theme system in Drupal 9.

Proposed resolution — or: how to transition

Long-form rationale

This part is less precise, but tells a hopefully helpful story of the two most important considerations: how we ended up with the current painful system (recommended reading: http://hackingdistributed.com/2016/04/05/how-software-gets-bloated/) and how we can at the same time start to make it possible to integrate with JS more easily.
And, how, funny enough, those actually need the same fundamental thing: simplicity.

JS: web apps vs web sites

"Apps" are the hot thing in software this decade. But building native apps for every platform is very expensive. So "web apps" are a thing: build once, run anywhere, deploy instantly.

Web apps must be written in JS (or at least compiled to JS), so, consequently, JavaScript is the hot language of this decade. As a result, we've seen enormous investments in JS: Node.js, browser engines' JS interpreters have become incredibly fast, and … JS-based frameworks. From jQuery UI to Angular to Ember to React to …

Everybody wants "an app". And so quite often what would have been a web site a few years ago now is a "web app". And since web apps are written in JS, this means Drupal is less likely to be chosen for those scenarios.

(I personally think the distinction between the two can be made by determining whether the business logic happens in JS on the client or on the server. Only if it's on the client, it is a "web app".)

But that doesn't mean there's no more need for "regular" web sites anymore. It doesn't make sense to build an app just to present hyperlinked information… because for that, we already have an app: the browser. We just need to feed it web sites: documents of structured content, with excellent accessibility & usability, beautiful layout & typography and most importantly: great information architecture.

(And indeed, there is a very, very blurry border between "web sites" and "web apps". I'm just trying to paint a picture of the two extremes, where the world of the web is still finding an equilibrium somewhere in between. And of course, in some cases, it makes sense to have parts on the client side and other parts on the server side.)

Drupal 8 is even better at building web sites than prior versions.

But Drupal is getting pressure to also support "web apps".

So we have a tension between wanting to improve Drupal for what it has traditionally done (improve front-end experience by improving its templates, its markup, removing preprocess functions, etc) and making it possible to build more app-like experiences with Drupal.
But Drupal is not written in JS. Perhaps at some point in the future, there will be tight integration. But we have no idea what that would look like, if it will happen.

Getting ready for the future, and facing the past

However, even with the current system, we have long-standing problems. Extremely frustrated reports go back to at least 2011. The theme system and render system are deeply intertwined. It's very confusing. The experience even to build just web sites (not web apps) is far from ideal. Drupal 7 introduced the render system, but let's not forget its origin: Form API. The render system was originally just for forms, but it's since been generalized to be used for all rendering.

• Drupal 6: Form API + theme system
• Drupal 7: Form API + theme system, and both depend on the render system

Drupal 8 has made big steps forward: Twig, removal of most preprocess functions, much cleaner templates, #theme is gone in favor of #type (but not at all completely…), no longer necessary to to sometimes call render() in templates …

Unfortunately, as soon as the render and theme system interact, it's still painful.

We started working on an experiment at Acquia, where we worked with the Angular & Ember.js teams to do a prototype of what a better commenting experience for Drupal would look like. A reference implementation in Drupal using the AJAX system ("the Drupal way") versus what they would do. They had to reuse our Twig templates. A big problem there was the fact that just about every template has "blobs of HTML": {{ content }}, which actually contained the majority of the interesting stuff. And those blobs of HTML are… yes, render arrays!

Of course, no JS is ever going to be able to render render arrays, because they're so deeply intertwined with PHP code. And it's impossible to automatically determine which Twig templates are associated with every subtree in a render array.

So this makes it effectively impossible to reuse our Twig templates in JS. Ideally, that would be possible, it'd make Drupal better prepared for the future. It's better to at least have that possibility than not to.

However, even today, and in fact, in years past, this very same problem has been a major frustration for themers: they could only go so far with achieving what they needed to achieve by creating/modifying templates and writing preprocess functions. Very often, they would need to dive deep into render arrays and implement lots of hooks.

Imagine if that weren't the case, and we'd have templates all the way down, rather than enormous blobs of the resulting HTML being defined in render arrays. Imagine if the resulting HTML was wholly based on templates. Templates all the way down. Power to the themers.

And imagine that rather than ill-defined variables, they'd actually have type hints. And validation. And examples. And actually usable documentation rather than incomplete (and duplicated) docblocks. Together, that would allow us to automatically generate a style guide/pattern library.

As a bonus, client-side (re-)rendering becomes possible.

References

A close-to-comprehensive list of the references I've used to write/build the above.

Remaining tasks

Lots.

User interface changes

None.

API changes

None, only additions.

Data model changes

None, only additions.