Allow schema references in Single Directory Component prop schemas

This is blocked until SDC is an experimental module.

We could also allow referencing other components with a new custom scheme component:/. Any component could then reference prop definitions from other components:

# ...
props:
  type: object
  properties:
    bgColor:
      $ref: component:/my-theme:some-component#properties/colorProp
      title: Background Color
# ...

However, I don't think this is good UX. IMO components should not be artificially interlinked like this.

Adding a test only patch that shows the goal we want to accomplish.

I did start working on the retriever on the plane on my ride back from DrupalCon. Here is some progress, in case someone wants to pick it up.

Next we need to go into ComponentMetadata::parseSchemaInfo and instantiate a new SchemaStorage with the custom retriever to addSchema and then getSchema. This will resolve the references in the schema to save in the ComponentMetadata property.

However this will require the library justinrainbow/json-schema to be vetted as runtime drupal core dependency. Which is a big blocker.

Hello e0ipso,

On your "Single Directory Components in Drupal Core " article, there is this code snippet:

props:
  type: object
  properties:
    attributes:
      type: Drupal\Core\Template\Attribute
      title: Attributes

Source: https://www.lullabot.com/articles/getting-single-directory-components-dr...

How does it works?

However this will require the library justinrainbow/json-schema to be vetted as runtime drupal core dependency. Which is a big blocker.

Do you think using a PHP namespace as a JSON schema type can be a way of achieving this issue's goal ?

does it means SDC JSON schemas using PHP class as prop types are proper JSOn schema anymore?

We are using a superset of JSON-Schema. This was the best solution we could find to non-JSON data types, which are valid Twig variables.

is it relevant to use it outside validation?

This is the main place where it is used in core. In contrib, cl_editorial will likely fail to generate a form element for these classes.

Seems like this is SUPER important for DX, to avoid repeating the same definitions over and over again (and preventing subtle inconsistencies)? 🤓

Or am I misunderstanding this? How is it possible that we only have core/modules/sdc/src/metadata.schema.json and core/modules/sdc/src/metadata-full.schema.json today, because based on this issue I'd think there'd be many *.schema.json files? 🤔

Hi Wim,

We solved this in UI Patterns 2.x in this way.

We have added a new "prop type" plugin type which have a JSON schema annotation.

Examples:

 *   schema = {
 *     "type": "string",
 *     "pattern": "^#([A-Fa-f0-9]{6}|[A-Fa-f0-9]{3})$"
 *   }

https://git.drupalcode.org/project/ui_patterns/-/blob/2.0.x/src/Plugin/U...

 *   schema = {
 *     "type": "string",
 *     "format": "iri-reference"
 *   }

https://git.drupalcode.org/project/ui_patterns/-/blob/2.0.x/src/Plugin/U...

A developer can use JSON schema references targeting the prop type plugin ID:

props:
  type: object
  properties:
    bg:
      title: "Background"
      "$ref": "ui-patterns://color"
    src:
      title: "Media source"
      "$ref": "ui-patterns://url"

The URL are resolved by justinrainbow/json-schema thanks to a stream wrapper service https://git.drupalcode.org/project/ui_patterns/-/blob/2.0.x/src/SchemaMa...

With a bit of help to manage is recursive references: https://git.drupalcode.org/project/ui_patterns/-/blob/2.0.x/src/SchemaMa...

So, the "$ref" is replaced by the schema of the prop type plugin annotation.

This system is extensible by using new prop type plugins.

We would be happy to get rid of our implementation if Core is solving this issue too.

1. IMHO this is super important. Without somewhat precise metadata/schema/definitions of what the expected shapes and semantics for an SDC prop are … ill-fitting (shape: array instead of string, string instead of object …) or nonsensical (semantics: phone number instead of e-mail address, prose instead of URI …) will be passed, and the SDC will fail to render.

   So: bumping to .

2. I don't think this should be postponed on 📌 Promote justinrainbow/json-schema from dev-dependency to full dependency Needs work . In fact, it's the opposite: this is the issue that needs to be so clearly valuable that it'll convince core committers #3365985 is worth doing! Until we get to that point, the MR for this issue should just add that dependency … and then once this is RTBC, we'll postpone this issue on that issue, and land the dependency addition there first.
3. @pdureau in #17 has demonstrated a very "Drupal ecosystem-friendly" way to do this, and I'm curious about @e0ipso's thoughts on going either with his original proposal in the issue summary and implemented in #9, or with @pdureau's alternative implementation? 🤓 → marking !

The Needs Review Queue Bot → tested this issue.

While you are making the above changes, we recommend that you convert this patch to a merge request → . Merge requests are preferred over patches. Be sure to hide the old patch files as well. (Converting an issue to a merge request without other contributions to the issue will not receive credit.)

One goal that I want to keep in mind is that this needs to have a very low friction. That's why I initially thought about letting components reference props in other components. This had the benefit of not having to do anything to declare the referenceable schemas. Sadly, this is not viable. Mainly because the referenced component may be deleted, and then the schema would break. This stems from the fact that there is no dependency chain between components (which I believe it's a good thing).

So, if we don't want to introduce dependencies between components but we want to re-use schemas, what dependencies do we introduce? I think the easiest is to depend on modules / themes. They are the things we install and uninstall, and we do check dependencies on such operations.

#7 proposes a schema-definitions.json in a module, which can be referenced as module:/modulename/path-to-any-file#json-path-to-schema-definition. This may be too lose, and therefore give pause and be hard to document. Perhaps we want to enforce a specific name for the file (how about components/prop-types.json?), and then reference it component-schemas:/modulename#json-path-to-schema-definition.

One goal that I want to keep in mind is that this needs to have a very low friction.

+1

That's why I initially thought about letting components reference props in other components. This had the benefit of not having to do anything to declare the referenceable schemas. […] Sadly, this is not viable. Mainly because the referenced component may be deleted, and then the schema would break.

I think in #7 and #9, you were already heading in a different direction though, that would not be dependent on component existence? There it'd be individual extensions (modules/themes) that can declare additional JSON schema definitions. Since SDCs always ship as part of an extension (module or theme), it would be guaranteed that that JSON schema file will exist (thanks to containing module/theme's declared dependencies).

So +1 to:

#7 proposes a JSON file (with any name) in a module, which can be referenced as module:/modulename/path-to-any-file#json-path-to-schema-definition. This may be too lose, and therefore give pause and be hard to document.

Why would it be too loose? 🤔

Perhaps we want to enforce a specific name for the file (how about components/prop-types.json?), and then reference it component-schemas:/modulename#json-path-to-schema-definition.

👆 YES! 🤩 Heck, it could even be json-schema-definitions://<extension name>#$defs/<DEFNAME>, so for example json-schema-definitions://image_gallery#$defs/galleryTitleAndCaption and core could provide commonly used things like json-schema-definitions://core#$defs/image!

Tangentially related, but off-topic:

This stems from the fact that there is no dependency chain between components (which I believe it's a good thing).

🤔 I was wondering about this. Does that also mean it's impossible to compose SDCs using only SDCS? I.e. put SDC B into SDC A's slot, and declare that result as SDC C?

In the PoC I'm working on for Experience Builder, I was able to transform this SDC prop metadata:

    test-string-format-uri-image:
      title: 'String, format=uri, images only'
      type: string
      format: uri
      # @see \Drupal\image\Plugin\Field\FieldType\ImageItem::defaultFieldSettings()
      pattern: '\.(png|gif|jpg|jpeg|webp)$'

to

    test-string-format-uri-image:
      title: 'String, format=uri, images only'
      $ref: "json-schema-definitions://experience_builder.module/image-uri"

thanks to a new ./schema.json file in my module named experience_builder, as well as the more complex SDC prop metadata

    test-object-drupal-image:
      type: object
      required:
        - src
      properties:
        src:
          title: 'Image URL'
          type: string
          format: uri
          # @see \Drupal\image\Plugin\Field\FieldType\ImageItem::defaultFieldSettings()
          pattern: '\.(png|gif|jpg|jpeg|webp)$'
        alt:
          title: 'Alternative text'
          type: string
        width:
          title: 'Image width'
          type: integer
        height:
          title: 'Image height'
          type: integer

to

    test-object-drupal-image:
      $ref: "json-schema-definitions://experience_builder.module/image"

thanks to:

{
  "$defs": {
    "date-range": {
      "title": "date range",
      "type":  "object",
      "required": ["from", "to"],
      "properties": {
        "from": {
          "title": "Start date",
          "type": "string",
          "format": "date"
        },
        "to": {
          "title": "End date",
          "type": "string",
          "format": "date"
        }
      }
    },
    "image-uri": {
      "title": "Image URL",
      "type": "string",
      "format": "uri",
      "pattern": "\\.(png|gif|jpg|jpeg|webp)$"
    },
    "image": {
      "title": "image",
      "type":  "object",
      "required": ["src"],
      "properties": {
        "src": {
          "title": "Image URL",
          "$ref": "json-schema-definitions://experience_builder.module/image-uri"
        },
        "alt": {
          "title": "Alternative text",
          "type": "string"
        },
        "width": {
          "title": "Image width",
          "type": "integer"
        },
        "height": {
          "title": "Image height",
          "type": "integer"
        }
      }
    }
  }
}

plus a new stream wrapper service inspired by @pdureau's superb work in ui_patterns 2.0.x (see #17 — I'd never have found it otherwise!).

The SdcPropToFieldTypePropTest kernel test is still passing, meaning it's still finding Drupal field type props that match 1:1 shape-wise and semantically into exactly those SDC props. (Grep for the string ℹ︎␜entity:node:foo␝field_silly_image␞␟{src↝entity␜ℹ︎␜entity:file␝uri␞␟value, alt↠alt, width↠width, height↠height} — that's the string representation of the expression that defines the mapping from Drupal field type props into SDC props).

Code: https://git.drupalcode.org/project/experience_builder/-/commit/f784af412...

This is great progress! It shows that it can be done.

thanks to a new ./schema.json file in my module named experience_builder, [...]

I would advocate for a name that is more specific to components. I propose components/shared-schemas.json. This is because it is scoped to the components/ folder, and it conveys that you only need this if you want to share definitions.

plus a new stream wrapper service inspired by @pdureau's superb work in ui_patterns 2.0.x (see #17 — I'd never have found it otherwise!).

justinrainbow/json-schema has a the concept of UriRetriever, which accomplishes the same. The advantage is that by following their pattern we can resolve schemas recursively every time (without having to pluck out the reference and resolve it manually). This includes loading the schema for validation during development time.

I would advocate for a name that is more specific to components. I propose components/shared-schemas.json. This is because it is scoped to the components/ folder, and it conveys that you only need this if you want to share definitions.

I actually did that intentionally: the idea would be that other things in Drupal core or the general Drupal (contrib) ecosystem could use this same infrastructure. For SDC's purposes, we'd verify that there'd be a top-level $defs (with definitions underneath), but anything else in that file it'd just ignore.
… but that's probably a premature abstraction, so I'm fine with changing that — it was just a proposal 🤓 I liked the simplicity of /schema.json, but I don't feel strongly about it.

which accomplishes the same. The advantage is that by following their pattern we can resolve schemas recursively every time

Are you sure that actually works? https://github.com/justinrainbow/json-schema/issues/427 suggests it does not?

@lauriii in DM:

$ref: sdc-prop-type://experience_builder.module/image seems still a bit too verbose to remember by heart. Ideally it would be as simple as type: 'drupal-image'.

My response:
That’s not valid JSON Schema. Defining new types is not permitted. Although https://json-schema.org/understanding-json-schema/reference/schema#vocab... seems to be changing that … 🤞 Per https://github.com/json-schema-org/json-schema-vocabularies, there aren’t a whole lot of examples just yet.

@lauriii again:

I understand that, but we need to find a way to workaround that somehow because $ref: sdc-prop-type://experience_builder.module/image is not acceptable DX.

So I'm curious what you think, @e0ipso.

Still, trying to indulge Lauri here, by taking the above to its logical conclusion: :nerd_face:

The use of a dialect like I mentioned above would allow us to declare:

image:
  type: object
  format: drupal-image

instead of

image:
  $ref: sdc-prop-type://experience_builder.module/image

but with one significant downside: now you need a custom JSON schema implementation everywhere you want to run validation. See https://json-schema.org/understanding-json-schema/reference/schema#guide....

I understand that, but we need to find a way to workaround that somehow because $ref: sdc-prop-type://experience_builder.module/image is not acceptable DX.

What about $ref: module://experience_builder/image ?

This proposal with static declarations in shared-schemas.json is exciting.

But, while implementing it, let's not block the use of alternative and complementary reference systems, like the one we already use in UI Patterns 2: ui-patterns://color where "color" is a plugin ID.

They all can live together inside a single reference solver mechanism, because the implementation is the same until the stream wrappers. What happens after the stream wrappers can differ.

That's tricky because I disagree with the premise in #25.

I think type: 'drupal-image' is way more confusing. It only gives you a label. It doesn't give you any info about what a drupal-image is, what are its dependencies, or were to find more info about it. Everything is implicit with that.

On the other hand, using $ref: <uri>#<locator> you are:

• Using a standard and well documented feature of JSON-Schema.
• Using another standard (JSON Path, in this case) to locate a data sub-structure within a file.
• Declaring that this depends on the experience_builder module.
• Giving the user a clear place to look for the definition and investigate further.
• Using the same industry standard pattern already in use in other frameworks leveraging JSON-Schema, like OpenAPI definitions.

I agree with @pdureau that perhaps a different URI template would give less pause. $ref: module-components://experience_builder#definitions/image would be my preference, but I realize that's exactly what Lauri is trying to avoid.

This proposal with static declarations in shared-schemas.json is exciting.

+1!

Nothing to add to #27 — thanks for articulating it so clearly, @e0ipso 😊

Creating components is a common enough task that it should be really easy for someone who is a) not familiar with Drupal b) not a senior developer and hence c) does not know JSON-Schema. This is highly relevant for workflows where the user has a pre-existing design system that they want to integrate with Drupal. For this persona, type: 'drupal-image'would be the north star experience because they don't care whether we use JSON-Schema or not.

That said, using JSON-Schema as a starting point makes sense, but we should not let it define what is the developer experience we provide. We just don't want to require users to learn JSON-Schema in order to be successful. Because of this, we need to consider how we can simplify some of the more complex workflows instead of being dogmatic about implementing the specification.

What's being proposed in #26 seems to be making this more reasonable already. However, even that could be seen as a strange by someone who is not familiar with the syntax. This user might ask; what is difference between $ref and type, why do I now need these two now, and why does the value need to follow a strange pattern with ://.

Creating components is a common enough task that it should be really easy for someone who is a) not familiar with Drupal b) not a senior developer and hence c) does not know JSON-Schema.

I think that from the SDC perspective:

a) Writing a component does not require much knowledge about Drupal, by design (as you very well know from being a co-conspirator on SDC)
b) I don't think this is required for writing SDC, or referencing schemas.
c) I think this is a soft requirement. If you want to write components with schemas, you kinda need to know they way they are written (JSON Schema). Now, do you need to be an expert? Definitely not.

I think the discussion is not weather or not it's OK to elevate the barrier of entry to newcomers to Drupal, I think we all agree on keeping it the lowest possible.

I think the discussion is that @lauriii in #29 argues that the $ref thing is complicated, and might put people off. And I think that using made up types like drupal-image makes things more complicated and dis-empowers said newcomers.

In #27 I tried to argue why an open list of made up types is bad UX. How do we document the made up types that contrib modules provide? How do we make the eventual documentation page discoverable for front-end developers? How do we tell newcomers that there may be other types? (in addition to all those challenges, now we are back to our island instead of following an industry standard... I think I am repeating myself at this point, sorry).

@e0ipso are you saying that in your mind, the syntax proposed in #26 leads into a better DX for a user who is a) not familiar with Drupal b) not a senior developer c) and hence does not know JSON-Schema? Is it because it's able to better solve the problems you are raising in #30?

Yeah. I even have a slight preference with $ref: module-components://experience_builder#definitions/image* as mentioned in #27. Not because it is easier to write, or easier to remember, but because it's easier to copy&tweak and easier to document (specially when the component author is already writing optional JSON-Schema).

*My slight preference comes from the fact that it uses the $ref recommended syntax JSON Pointer. Which is documented for us, and the resolution mechanism is already implemented in the justinrainbow/json-schema library without additional stream wrappers.

I haven't read all the discussion here, but I noticed some discussion about the DX of JSON schema, or certain decisions we might want to make about the details of the JSON schema we want to use, so I want to quickly mention that https://github.com/vega/ts-json-schema-generator is a pretty great library that lets you convert TypeScript definitions to JSON schema. In other words, if we want to optimize for DX, we might want to consider a DX where people can write TypeScript definitions instead of writing JSON schema by hand, and then the JSON schema creation can be automated which would allow us to optimize the JSON schema to suit SDC's needs rather than optimizing for manually writing it.

While I am an advocate of TS (I even pushed an ADR at Lullabot to the effect), I do not think that we should prescribe it in order to write re-usable schemas. This is specially true since we are not writing JSX components in Drupal (yet).

In any case the import method is a bit similar. In one case we defer to the native module loader and a URI, and in the other we use a custom code loader and a URI.

A declarative format is better than an executable format for such definitions. No runtime needed, no performance issue, no security risk.

https://en.m.wikipedia.org/wiki/Rule_of_least_power

we are not writing JSX components in Drupal (yet).

Yet? Oh god 😳

Agreed that the source of truth should always be a JSON schema definition.

But I think @effulgentsia was merely stating that it's possible to provide a TypeScript-only DX that generates such a JSON schema definition. I doubt he's proposing to enforce it, nor make it the default.

The fact that it's possible is … nice, interesting and potentially powerful eventually :)

Ah! Thanks for the clarification @Wim Leers. I was taking it as a "there is no need to dwell too much in this now, since we'll be able so solve it with a build step".

To pile on that spirit that @effulgentsia brings on #33, I'll say that using industry standards will bring us more and more of these tools for free. Not only to generate the schemas, but also to read and leverage them. Staying true to the JSON-Schema standard will also allow us to auto-generate forms for the component props, synthetic examples, etc. Carving out (more) exceptions like drupal-image will make these tools fail without specific handling.