[META] Production-ready data storage

Proposed to @longwave to close #3468269: #3468269-23: `ComponentTreeStructure` data type: simplify the stored structure → .

Moved a significant chunk out of here and into 🌱 [META] Production-ready ComponentSource plugins Active .

There is a @todo in ComponentTreeItem linking to 🐛 Cannot delete a field which uses JSON type Active so I think that probably needs to be included here?

Opened 🐛 [PP-1] Use JSON schema type for sqlite and remove text workaround Active .

Additional things I think we need for a 1.0

* plugabble/flexible ComponentTreeLoader so that LB/PG can provide a translation layer for legacy data
* it would be good if default values were stored as an array instead of JSON for the sake of git diff on exported config - same reasons as ✨ Create a schema for "allowed_html" which provides a better config diff Needs work - the current json is going to be very painful for content template git conflicts

I think the last point is being tackled in 📌 Stop storing JSON blobs in XB config entities: improve DX Active .

Created

• 📌 [PP-2] Add paging to Component "Audit" operation content listing Postponed
• 📌 [PP-2] Support translations tracking on Component "Audit" operation content listing when using asymmetric translations Postponed
• 📌 [PP-2] Consider translations field usage when uninstalling the module providing a field Postponed

The third one might be a data integrity issue.

Track explicit input schema version: hash (the normalized) explicit input schema, assign version (sequential integer), store version: {hash, provider_version} mapping in Component config entity.

I don't think anything with automated sequential integers is going to work because we have to handle partial deployments of config between sites. Let's say you export a Section from your site that uses a component that is on version 2 of the schema. The SDCs and component config entities might already exist on the destination site, but if it was installed later than the site it was exported from, the same component might only be on version 1 of its schema.

📌 [later phase] Support matching `{type: array, …}` prop shapes Postponed is in.

🔄 📌 Calculate field and component dependencies on save and store them in an easy to retrieve format Active landed!

Added 📌 Stop storing JSON blobs in XB config entities: improve DX Active .

📌 Stop storing JSON blobs in XB config entities: improve DX Active is in 🥳

Added a link to 📌 Version component prop definitions for SDC and Code components Active to item 2.1. Wim and I basically came up with the same thing independently - nice!

WRT dependency/usage tracking: 📌 Calculate field and component dependencies on save and store them in an easy to retrieve format Active landed, and so did its follow-up 📌 [PP-1] Evaluate storing XB field type's "deps_*" columns in separate table Active . This cleaned up the technical debt around FieldTypeUninstallValidator and hence enabled me to close 3 stable blockers, see #3457504-56 for details.
This unblocked:

• short term: robustness-related functionality such as 🐛 Cannot delete JS components due to component depending on them Active
• long term: future functionality such as ✨ Usage info for code components with unpublished changes Postponed

Only remaining: 📌 [later phase] Support props defaults that have config dependencies Postponed . That is a stable blocker too, but won't require data storage changes. So: I'm assigning that subset of this meta a 🟢.

Explain relation to 📌 Allow XB to be used on all node types Active in the issue summary.

Added @larowlan's 📌 [PP-1] Consider not storing the ComponentTreeStructure data type as a JSON blob Postponed and 📌 [PP-1] Spike: Explore storing a hash lookup of component inputs Postponed .

Added 📌 Spike: Explore storing component inputs in separate columns (aka field union) Active .

Removed

debt: we know we'll need to support implicit inputs (e.g. block plugins can declare required + optional "contexts" that allow block plugins behave differently not only to component instance author's explicit wishes ("explicit inputs"), but also based on the component developer's logic ("implicit inputs")

because that doesn't affect the data storage.

Descoped #3523846 per @catch in #3523846-7: [PP-1] Spike: Explore storing component inputs in separate columns (aka field union) → . We may revisit that after 📌 [PP-1] Consider not storing the ComponentTreeStructure data type as a JSON blob Postponed . 👍

Added ✨ [PP-2] Allow users to name components for the specific context Postponed

Clarified 📌 Version component prop definitions for SDC and Code components Active

Landed 📌 [later phase] Support props defaults that have config dependencies Postponed .

Now going to overhaul this meta issue to clarify what's for beta vs stable …

📌 [PP-1] Consider not storing the ComponentTreeStructure data type as a JSON blob Postponed landed!

Next up: 📌 Version component prop definitions for SDC and Code components Active .

Good news! The current issue summary's

Blocked on the above first bringing clarity to the full set of needs first, before starting a potentially enormous refactor

is getting to that point of clarity thanks to @larowlan, @effulgentsia, @catch and I actively having started to push these forward in the past few weeks.

So: time for an overhaul of the issue summary!

Updated the issue summary to more clearly distinguish between:

1. beta blockers (i.e. necessary for 🌱 Milestone 1.0.0-beta1: Start creating non-throwaway sites Active )
2. stable blockers (i.e. necessary for 🌱 Milestone 1.0.0: Production Sites Postponed )
3. post-stable priorities (i.e. important, but after 🌱 Milestone 1.0.0: Production Sites Postponed )

… and within each of those, the areas within "data storage" they target. For beta, that's:

1. Dependencies/usage
2. Field type storage (content entities) schema
3. Configuration storage schema
4. Validation

1. Added 📌 [SPIKE] Prove that it's possible to apply block settings update paths to stored XB component trees Active as a beta blocker — well on its way 👍
2. Added 🐛 [PP-1] Use JSON schema type for sqlite and remove text workaround Active as a beta blocker, which is blocked on the 🐛 Cannot delete a field which uses JSON type Active core issue. There's a proposal
3. Moved #3495625 from stable blocker to post-release priority per #3495625-14: Remove ComponentTreeItemList::ROOT_UUID from hydration and client-to-server conversion →
4. Added 📌 [PP-1] Harden UUID validation in ComponentTreeStructureConstraintValidator Postponed as stable blocker — but already completed! 🥳
5. Added 📌 Spike: Explore adding configuration options to the tree item formatter to support alternate use-cases Active as stable blocker.

Posted "FYI, captured in the data storage meta" comments on:

• #3523842-16: Spike: Explore storing a hash lookup of the ComponentInputs field property: one hash per component instance →
• #3477428-53: [PP-1] Refactor (or decide not to) the XB field type to be multi-valued, to de-jsonify the tree, and to reference the field_union type of the prop values →
• #3440578-89: [PP-2] JSON-based data storage proposal for component-based page building →

Per @lauriii: add more features we'll eventually need to support.

And: update the status indicators for all of them.

Clarify JSON:API support, bump to 🟢 . Linked (not added!) 📌 ServerClientConversionTrait Active even though it actually belongs under 🌱 [META] Production-ready client-side data model + internal HTTP API Active , because it's how we'll add JSON:API support. This is 🟢 because after #3468272, the implementation path for it became crystal clear.

Added 📌 Ensure predictable config export order of config-defined component trees Active .

That doesn't mean I can remove though — that's still true.

Bumped from 🔴 to 🟡 based on
#3525746-6: Update the React client preview view → , and based on @penyaskito having proven in 📌 Personalization PoC Active that the single new ComponentSource approach works. That means zero data storage changes are needed!

Tempted to mark 🟢, but would prefer to be cautious instead of overly confident. Will only mark as such once more of it is working.

Triaged the issue queue component, as well as the @todos in XB's config schema.

Added many newly created issues — all of them stable blockers unless marked otherwise:

1. 📌 [11.2-only] Adopt `AtLeastOneOf` validation constraint for cardinality validation Postponed
2. ✨ Stop assuming default Field Widget settings suffice — add Field Widget settings support to `experience_builder.generated_field_explicit_input_ux: prop_field_definitions` Active
3. 📌 Tighten validation of `experience_builder.generated_field_explicit_input_ux: prop_field_definitions` Active
4. 📌 Tighten validation of `experience_builder.generated_field_explicit_input_ux: prop_field_definitions.[%key].expression` Active
5. 📌 [>=11.1.8] Remove `type: field.value.language` work-around once Postponed
6. 📌 [PP-1] Remove BetterConfigExistsContraint and move back to ConfigExistsContraint Active → post-stable priority
7. 📌 [later phase] [PP-1] Gracefully handle deleted regions in PageTemplate config entities" Postponed → post-stable priority
8. 📌 JavaScriptComponent config entities should have mutable machineNames Active
9. 📌 Allow XB to be used on all node types Active
10. ✨ [PP-1] Validate that content templates are attached to entity bundles that fulfill XB's requirements Postponed
11. ✨ Implement `::onDependencyRemoval()` for `Pattern` and `PageRegion` config entities: when a module providing a field type is uninstalled, replace it with the default StaticPropSource Active → post-stable priority
12. 📌 [PP-2] Add ComponentAuditabilityTest Active
13. 🐛 SdcPropKeysConstraintValidator::validate() should complain about extraneous keys too, not just missing keys Active

Having reviewed #3519352 and written #3519352-55: Content templates, part 3b: store exposed slot subtrees on individual entities → (specifically the part), I'm bumping from 🟡 to 🟢.

Per #37, I triaged all config management issues this morning.

Didn't find anything.

So I can move to 🥳 Note that #3526127-5: Ensure predictable config export order of config-defined component trees → might become a beta blocker.

did not have an indicator yet — the sole issue there is well on its way, so marked it 🟢.

#32 failed to update from 🟢 to ✅ for the cited reasons → .

The untranslatable_inputs sample diff for post-stable was incomplete.

Added 📌 Default content exports are invalid and hence are not correct after importing Active to stable blockers and 📌 Tighten validation of `parent_uuid` and `slot` on XB fields to match the strictness of config Active to beta blockers — thanks @justafish!

📌 Tighten validation of `parent_uuid` and `slot` on XB fields to match the strictness of config Active landed.

• ✨ [PP-2] Allow users to name components for the specific context Postponed was +1'd by @larowlan.
• 📌 Version component prop definitions for SDC and Code components Active massive progress by @larowlan and I both.

Landed beta blocker ✨ Content templates, part 3b: store exposed slot subtrees on individual entities Active ! 🥳

Added 📌 Spike: explore merits of one XB field per exposed slot in content templates Active as stable blocker.

✨ [PP-2] Allow users to name components for the specific context Postponed landed!

📌 Version component prop definitions for SDC and Code components Active landed!

But two beta-blocking bits were descoped from it:

1. Assigned to: larowlan 📌 [PP-1] Ensure deterministic version hashes Active — this one is much narrower in scope :)
2. Assigned to: larowlan 📌 Either rename `component_id` field property to `component`, or `version` to `component_version` Active — a bike shed that shouldn't block a ~3K LoC diff 😅

Bumping from 🟡 to 🟢 given #46.

Also bumping from 🟡 to 🟢 after discussing with @effulgentsia (he agreed), because A) we've discussed it in depth over the past ~2 weeks, B) @penyaskito has a working PoC of this at 📌 Personalization PoC Active , which he'll be updating tomorrow (since a few blockers have landed for him).

That means everything now is either 🟢 or ✅!

The two new beta blockers that spun out from #3523841 last night have already landed: 📌 Either rename `component_id` field property to `component`, or `version` to `component_version` Active + 📌 [PP-1] Ensure deterministic version hashes Active — thanks to @larowlan!

Added 📌 Add e2e tests that prove we can edit an old version of a component Active as a stable blocker (again: thanks Lee).

Finally: swapping the overall/catch-all 🌱 [META] Experience Builder Personalization Active for the more tightly scoped 📌 Personalization component source Active as proof for full confidence in this future feature:

1. feature: ability to store variants of component trees for use cases like personalization (i.e. different component subtree for Belgians & Brits vs everyone else), responsive design (i.e. breakpoint-specific overrides), and arbitrary future use cases

Added 🐛 Deterministic Component version hash should depend not only on source-specific settings, but also slots + explicit input schema Active as a beta blocker.

📌 Constraint slot names allowed by XB in its component tree Active landed.

Per discussion with @larowlan, moved 📌 Ensure predictable config export order of config-defined component trees Active from stable to beta blocking.

📌 Calculate field and component dependencies on save and store them in an easy to retrieve format Active and 📌 [PP-1] Evaluate storing XB field type's "deps_*" columns in separate table Active were done before 📌 [PP-1] Consider not storing the ComponentTreeStructure data type as a JSON blob Postponed and 📌 Version component prop definitions for SDC and Code components Active .

I think this means that a lot of (maybe all?) the data that was denormalized into a special database table can now be retrieved by querying the field tables (in combination with component config entity dependencies). Is there a still-open issue where this is being discussed? I wasn't able to find one.

Added 📌 Component instances in exposed `ContentTemplate` slots must use the exposed slot's machine name Active as stable blocker.

Added 📌 Revisit storage of dependencies in separate table now we have separate deltas per component Active for #51 @catch

🐛 Deterministic Component version hash should depend not only on source-specific settings, but also slots + explicit input schema Active and 📌 Ensure predictable config export order of config-defined component trees Active landed, bringing remaining back down to zero!

In , 📌 [SPIKE] Prove that it's possible to apply block settings update paths to stored XB component trees Active landed, and per #3520923-18: [PP-1] Use `json` schema type for SQLite and remove `text` workaround → , moved 🐛 [PP-1] Use JSON schema type for sqlite and remove text workaround Active to the bottom of that section.

Next (and IMHO last) most important issue: 📌 Revisit storage of dependencies in separate table now we have separate deltas per component Active .

Landed the stable-not-beta-blocking 🐛 SdcPropKeysConstraintValidator::validate() should complain about extraneous keys too, not just missing keys Active since @larowlan commented on it: #3525759-9: SdcPropKeysConstraintValidator::validate() should complain about extraneous keys too, not just missing keys → .

Re-checked the list of open issues tagged → and found one not listed here: 🐛 \Drupal\experience_builder\Plugin\DataType\ComponentInputs::getPropSources needs to take \Drupal\experience_builder\Plugin\ExperienceBuilder\ComponentSource\GeneratedFieldExplicitInputUxComponentSourceBase::rawInputValueToPropSourceArray into account Active .

Added 🐛 Component::onDependencyRemoval() should be an uninstall validator Active as a stable blocker.

(FYI: for beta blockers, the bulk of the attention is going to 📌 Revisit storage of dependencies in separate table now we have separate deltas per component Active .)

🐛 \Drupal\experience_builder\Plugin\DataType\ComponentInputs::getPropSources needs to take \Drupal\experience_builder\Plugin\ExperienceBuilder\ComponentSource\GeneratedFieldExplicitInputUxComponentSourceBase::rawInputValueToPropSourceArray into account Active is in, and the core blocker for 🐛 [PP-1] Use JSON schema type for sqlite and remove text workaround Active landed upstream! 🥳

Landed 🐛 Component::onDependencyRemoval() should be an uninstall validator Active (not here) which ended up (unexpectedly for me) blocking 📌 Revisit storage of dependencies in separate table now we have separate deltas per component Active .

That means the sole remaining data storage beta blocker is 🐛 [PP-1] Use JSON schema type for sqlite and remove text workaround Active , but the upstream core issue is in ( 🐛 Cannot delete a field which uses JSON type Active ), so now it’s a trivial update on our end once a Drupal core release ships — which should be trivial to do in the coming ~1.5 month.

New stable blockers:

1. 📌 [later phase] Make Component `audit` operation scalable Postponed (long known, just was missing from this meta)
2. 📌 Update `ComponentAudit::getConfigEntityDependenciesUsingComponent()` to support component versions Active (follow-up for 📌 Revisit storage of dependencies in separate table now we have separate deltas per component Active

New post-stable priority:

• 📌 [later phase] [PP-1] Prune unused `Component` config entity versions Active

wim leers → credited effulgentsia → .

wim leers → credited tedbow → .

As described at #3520923-29: [11.2.0-and-up] Use `json` schema type for SQLite and remove `text` workaround → , that can't happen until 11.2.0 is tagged. It's a trivial MR to land though :)

So: THIS IS DONE! 🥳🥳🥳 Special thanks to:

1. @larowlan, for the constant stream of reviews & MRs
2. @catch, for the constant feedback (from a core committer!)
3. @tedbow, for the translation test coverage + field type uninstall validator + component tree validation logic that was iterated upon but still lives on, these were all very valuable guardrails!
4. @penyaskito, for helping get the dependency calculation logic in place, which in all of the recent iterations by Lee and I had proved crucial!
5. @effulgentsia, for the original idea by and his chiming in when really necessary over the past few weeks 🙏

Signaling this is "done for now", i.e. for 🌱 Milestone 1.0.0-beta1: Start creating non-throwaway sites Active but NOT for 🌱 Milestone 1.0.0: Production Sites Postponed , by marking this .

Retroactively adding things that actually did happen for beta, but before this meta issue existed or was sufficiently groomed.

• 📌 XB Component config entity's `status` may only be `true` iff it meets all of XB's requirements Active
• 📌 Allow component source plugins that don't have a plugin_id setting Active

Ideally it would be good to do a spike on a BC loading layer for Paragraphs/Layout builder to triple check we've got a clean run at that in the future with the current data model

That would be lovely indeed. But I don't think that's a hard blocker for beta. Let's first make sure we actually achieve 🌱 Milestone 1.0.0-beta1: Start creating non-throwaway sites Active , and then let's do this. I don't see how expressing anything that can be built in Layout Builder/Paragraphs/… (see 🌱 [META] Support component types other than SDC Needs work ) could not be expressed in XB, because XB has more freedom/is much less prescriptive?

Also … didn't we discuss (and in fact you proposed it! 😄) to only convert existing data into the client-side data model, so you can load an existing component-tree-esque thing (Layout Builder, Paragraphs, whatever) into XB and then choose to publish or not?

IOW: require human judgment, because anything automated is unlikely to A) be perfect (pixel perfect match), B) forgiving (a human could judge that actually, it makes sense to use some other component because it is better or simpler or $reason).

An automated batch migration is likely to result in problems, whereas this allows for a nice gradual migration: they can both live side-by-side until every relevant entity has been switched over to XB.

Sorry, I wasn't clear, I'm not proposing any change to the discussed approach, just validating that the approach is viable. Will focus on 🌱 Milestone 1.0.0-beta1: Start creating non-throwaway sites Active

Added 🐛 Audit use of ::loadUnchanged Active which we need for sites running content moderation

🐛 [PP-1] Use JSON schema type for sqlite and remove text workaround Active landed as part of 🐛 PHPUnit Next Major tests failing Active 🥳

Discussed #3532712 with @larowlan, it's not beta-blocking — see #3532712-10: Audit use of ::loadUnchanged and any routes that might need to load the latest revision → .

Landed: 🐛 Component::onDependencyRemoval() should be an uninstall validator Active , 📌 [>=11.1.8] Remove `type: field.value.language` work-around once Postponed .

New: 🐛 ComponentTreeItem specifies wrong main property Active (and also landed 👍).

Validation stable blocker 📌 Tighten validation of `experience_builder.generated_field_explicit_input_ux: prop_field_definitions` Active is in! And a new one was found: 📌 Don't allow passing uncollapsed inputs if using default expression Active , 📌 Disallow component trees with `component_version: active` Active — the latter of which already landed too.

Added 📌 Improve the way to detect if the code component has drupalSettings Active (fixed!), ✨ [PP-1] Use AST to identify resources fetched by CodeComponents so they can be preloaded Postponed and 📌 Automatically transform the `drupal-11.2.2-with-xb-0.7.2-alpha1.filled` fixture when testing non-SQLite databases Active .

📌 Tighten validation of `experience_builder.generated_field_explicit_input_ux: prop_field_definitions.[%key].expression` Active landed.

ADR updated and now IMHO ready to be merged. Hoping to get +1s from @effulgentsia, @lauriii, @larowlan and @catch.

ADR looks good to me, thanks for taking the time to collate that

📌 Don't allow passing uncollapsed inputs if using default expression Active and 📌 Automatically transform the `drupal-11.2.2-with-xb-0.7.2-alpha1.filled` fixture when testing non-SQLite databases Active landed. 🥳

Thanks for confirming the ADR looks good to you, @larowlan — I'd like a +1 from at least @lauriii still, given he's the one who asked for the consequences to be more complete, and to also list negative consequences.

The changes look good and it gives a sense that we thought of both the pros and cons of these decisions 👍

Landed the stable-blocking subset of ✨ [PP-1] Validate that content templates are attached to entity bundles that fulfill XB's requirements Postponed already, hence moved the remainder to post-stable.

Thanks for the updates. ADRs are still pretty new to me so I'm not sure what the format/verbosity is supposed to be like overall, but this all looks correct now! The one thing I wondered about is if it needs more details about the trade-offs between flexible storage for inputs vs them not being queryable, but if that's more detail than intended seems fine to leave it implied.

Yep, it's supposed to be prescriptive of high-level architecture, not implementation details. Some of that is subjective. But historically, we've been very bad at actually finishing (i.e. merging/publishing) ADRs → , so @lauriii has urged us (rightfully so IMHO!) to err on the side of high-level. IOW: the details are for concrete implementation issues and detailed docs, the architectural choices and their rationales are for ADRs.

Very glad to have your blessing, so: merged! 😄🥳