Define JSON Schema $refs for linking/embedding videos and linking documents

Created on 13 May 2025, about 1 month ago

Overview

XB already has

  1. json-schema-definitions://experience_builder.module/image, which allows rendering an image
  2. json-schema-definitions://experience_builder.module/image-uri, which allows linking an image

The 3 missing $refs to use in SDCs' schemas:

  1. linking a video
  2. rendering ("embedding") a video
  3. linking a document
  4. ๐Ÿ‘ˆ (most) browsers can only natively render PDFs, all other document types require specialized embedding widgets โ€” tackling that seems out of scope here?

Note that in core, the:

  • "document" media type allows txt rtf doc docx ppt pptx xls xlsx pdf odf odg odp ods odt fodt fods fodp fodg key numbers pages
  • "video" media type allows mp4
  • "remote video" media type allows YouTube + Vimeo

Proposed resolution

For the 3 missing concepts:

  1. Challenge: more often than not, you wouldn't link to a .mp4, .webm โ€ฆ URL, but you'd link to YouTube or Vimeo or โ€ฆ. How do we express that? ๐Ÿค” Seems like we'd need to draw inspiration from core's \Drupal\media\OEmbed\Resource, which has ::TYPE_LINK, ::TYPE_PHOTO, ::TYPE_RICH and ::TYPE_VIDEO.
  2. This pretty much requires an oembed URL, and then the oembed fetching to happen โ€ฆ inside the SDC itself? ๐Ÿค”
  3. Linking to a document seems simple enough:
        "document-uri": {
          "title": "Document URL",
          "type": "string",
          "format": "uri-reference",
          "pattern": "^(/|https?://)?.*\\.(txt|rtf|doc|docx|ppt|pptx|xls|xlsx|pdf|odf|odg|odp|ods|odt|fodt|fods|fodp|fodg|key|numbers|pages)(\\?.*)?(#.*)?$"
        },
    

User interface changes

TBD

โœจ Feature request
Status

Active

Version

0.0

Component

Shape matching

Created by

๐Ÿ‡ง๐Ÿ‡ชBelgium wim leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @wim leers
  • ๐Ÿ‡ง๐Ÿ‡ชBelgium wim leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ

    @lauriii, do you agree that does not make sense to support, and the 3 remaining missing $refs in the issue summary would address all needs?

  • ๐Ÿ‡ง๐Ÿ‡ชBelgium wim leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ
  • ๐Ÿ‡ง๐Ÿ‡ชBelgium wim leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ

    @lauriii, also see the issue summary: do you want only (oEmbed) remote video, or also local video URLs?

    Added more detail to the oEmbed video proposal.

  • ๐Ÿ‡ซ๐Ÿ‡ฎFinland lauriii Finland

    I think we should express the uploaded videos as follows in schema.json:

    {
      "video": {
        "title": "video",
        "type": "object",
        "required": ["src"],
        "properties": {
          "src": {
            "title": "Video URL",
            "type": "string",
            "format": "uri"
          },
          "poster": {
            "title": "Poster image URL",
            "$ref": "json-schema-definitions://experience_builder.module/image-uri"
          },
          "width": {
            "title": "Video width",
            "type": "integer"
          },
          "height": {
            "title": "Video height",
            "type": "integer"
          },
          "autoplay": {
            "title": "Autoplay",
            "type": "boolean"
          },
          "loop": {
            "title": "Loop",
            "type": "boolean"
          },
          "muted": {
            "title": "Muted",
            "type": "boolean"
          },
          "controls": {
            "title": "Show controls",
            "type": "boolean"
          }
        }
      }
    }
    

    For oEmbed, I'm not sure we need the max width / max height here. It seems something that the component itself should handle? We may need to expose the height / width of the resource, as well as title which would make this shape as follows in schema.json:

    {
      "oembed_video": {
        "title": "oembed_video",
        "type": "object",
        "required": ["url"],
        "properties": {
          "url": {
            "title": "oEmbed URL",
            "type": "string",
            "format": "uri",
            "contentMediaType": "application/oembed+json",
            "x-oembed-type": "video"
          },
          "width": {
            "title": "Video width",
            "type": "integer"
          },
          "height": {
            "title": "Video height",
            "type": "integer"
          },
          "title": {
            "title": "Video title",
            "type": "string"
          }
        }
      }
    }
    

    For documents, to display a link, we'd need a bit more than just the URI in the schema.json:

    {
      "document": {
        "title": "document",
        "type": "object",
        "required": ["src"],
        "properties": {
          "src": {
            "title": "Document URL",
            "$ref": "json-schema-definitions://experience_builder.module/document-uri"
          },
          "title": {
            "title": "Document title",
            "type": "string"
          },
          "description": {
            "title": "Document description",
            "type": "string"
          },
          "filename": {
            "title": "Filename",
            "type": "string"
          },
          "filesize": {
            "title": "File size",
            "type": "integer",
            "description": "File size in bytes"
          },
          "mimetype": {
            "title": "MIME type",
            "type": "string"
          }
        },
        "document-uri": {
          "title": "Document URL",
          "type": "string",
          "format": "uri-reference",
          "pattern": "^(/|https?://)?.*\\.(txt|rtf|doc|docx|ppt|pptx|xls|xlsx|pdf|odf|odg|odp|ods|odt|fodt|fods|fodp|fodg|key|numbers|pages)(\\?.*)?(#.*)?$"
        }
      },
    }
    
  • ๐Ÿ‡ง๐Ÿ‡ชBelgium wim leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ

    Walked @f.mazeikis through the Shape matching infrastructure. He's gonna get started on implementing the new json-schema-definitions://experience_builder.module/video object shape that @lauriii proposed in #5 ๐Ÿ‘

  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom f.mazeikis Brighton

    f.mazeikis โ†’ made their first commit to this issueโ€™s fork.

  • Pipeline finished with Failed
    11 days ago
    Total: 753s
    #521485
  • Pipeline finished with Running
    10 days ago
    #521714
  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom f.mazeikis Brighton

    @lauriii

    In #5 you've proposed shapes for video and oembed_video and I have some questions.

    1. For video you've proposed poster property - as @chirstian pointed out on the draft MR - the commonly used term (and already existing property in Drupal Media API) for poster would be thumbnail. This also applies to Youtube, vimeo and others. Is there a good reason we would not use thumbnail as terminology and the actual Drupal Media thumbnail property?
    2. Properties for height and width for video are not currently stored for video_file source, we could alter the source and add missing properties via hook_media_source_info_alter - is this what you would expect?
    3. For oembed_video described properties could be fetched from oEmbed resource provider (Youtube or Vimeo) - Media entity itself seems to fetch those at the runtime. Do you expect us to fetch the values and store/own them as prop inputs? Fetch the values and use them, but not store them? Do you want users to be able to provide their own title, width and height, that would override whatever oEmbed provider supplies?

    Depending on the answer to 3 it might look quite similar to 2 - just trying to figure out what the expectation is here, before I write a bunch of code.

  • ๐Ÿ‡ง๐Ÿ‡ชBelgium wim leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ

    #9:

    1. <video poster> is not semantically the same as \Drupal\media\Entity\Media::loadThumbnail().

      See https://developer.mozilla.org/en-US/docs/Web/API/HTMLVideoElement/poster and contrast with core/modules/image/config/install/image.style.thumbnail.yml.

      The media thumbnail is a (typically square) image used when browsing the media library. You cannot use this for <video poster: the aspect ratio would be wrong, and the quality is very likely far too low.

      Conclusion: without the ability to extract an image from a video (we definitely can't do that โ€” Drupal's media system doesn't even do this), we must make do without a poster. Which is fine, because as you can see: poster is optional!

      just do not populate poster by default ๐Ÿ‘

    2. Same thing here โ€” they're both optional. Since core's "Video" media type doesn't support it, I don't see how XB could support it.

      โš ๏ธ โš ๏ธ โš ๏ธ OTOH โ€” what you're dealing with here is that <video width> refers to the video display area, not the video's resolution.

      I bet that this is something @lauriii would like to see supported โ€” even if you could enter bad values (e.g. specifying width/height not respecting aspect ratio etc.), because it impacts the content authoring experience a lot.

      However, given the current reality of XB only allowing to use StaticPropSources to populate SDC/code component props, this technically requires a field type + widget that provides src (media library) + width/height (separate static inputs). This is what AdaptedPropSource was intended for, but XB's UI doesn't support this yet.

      we don't allow width and height to be entered either, because we cannot support it yet. Here too, that's possible, because both are optional ๐Ÿ‘

    3. Do you expect us to fetch the values and store/own them as prop inputs? Fetch the values and use them, but not store them? Do you want users to be able to provide their own title, width and height, that would override whatever oEmbed provider supplies?

      Here too, everything I wrote in response to #9.1 and #9.2 applies: width and height are not the ones you seem to think these are referring to (and would require AdaptedPropSource in the UI), and url is the only one that is required.

    tl;dr: forget about all optional key-value pairs, limit scope here to only the required ones, because they're the only ones that are feasible.

  • Pipeline finished with Failed
    7 days ago
    #523584
  • Pipeline finished with Failed
    7 days ago
    Total: 959s
    #523746
  • Pipeline finished with Failed
    7 days ago
    Total: 318s
    #523823
  • Pipeline finished with Failed
    7 days ago
    Total: 833s
    #523825
  • Pipeline finished with Failed
    6 days ago
    Total: 888s
    #525161
  • Pipeline finished with Failed
    5 days ago
    Total: 513s
    #525366
  • Pipeline finished with Failed
    5 days ago
    Total: 931s
    #525376
  • Pipeline finished with Failed
    5 days ago
    Total: 958s
    #525647
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States effulgentsia
  • Pipeline finished with Failed
    5 days ago
    Total: 925s
    #526014
  • Pipeline finished with Success
    5 days ago
    Total: 1055s
    #526027
  • Pipeline finished with Success
    5 days ago
    Total: 1004s
    #526097
  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom f.mazeikis Brighton

    Implemented feedback from #10 โœจ Define JSON Schema $refs for linking/embedding videos and linking documents Active , updated simple/video SDC component and added new simple/ombed_video SDC component. This was useful for testing via UI, but not quite sure if we want to keep it.
    Added comments to MR, moving to needs review.

  • Pipeline finished with Failed
    5 days ago
    Total: 1580s
    #526135
  • ๐Ÿ‡ง๐Ÿ‡ชBelgium wim leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ

    YAYAYAY!

    First local video from Drupal's Media Library rendered via a component:

    First remote video (about XB of course โ€” Lee's talk on YouTube!) rendered via a component:

    โ€ฆ unfortunately some kind of issue with YouTube refusing things, probably CSP-related, we can figure that out next ๐Ÿ‘

  • ๐Ÿ‡ง๐Ÿ‡ชBelgium wim leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ

    Found the culprit. oEmbed videos are passed as-is:

    <iframe src="https://www.youtube.com/watch?v=v747p7xEcgg" width="" height="" title="" data-xb-uuid="a9a2498e-e1a1-4464-b2ea-db72daf7ea60"></iframe>
    

    โ€ฆ which is incorrect.

    This should actually render something like

    <iframe src="https://example.com/media/oembed?url=https%3A//www.youtube.com/watch%3Fv%3Dv747p7xEcgg&max_width=0&max_height=0&hash=2kCbWsOQ7urG-UZLzC9KUBuT1B-7jnlX66TsoG_WL7g" width="" height="" title="" data-xb-uuid="a9a2498e-e1a1-4464-b2ea-db72daf7ea60"></iframe>
    

    Why? Because that's how core renders remote videos successfully. See for yourself by doing:

    1. Go to /admin/config/media/media-settings and enable stand-alone media URLs
    2. Create a video media entity for some YouTube video
    3. Navigate to /media/N
    4. Video appears!
    5. Put a breakpoint in OEmbedFormatter::viewElements(), observe the different steps, specifically how $url and $resource_url exist. This results in:
      array (
        '#type' => 'html_tag',
        '#tag' => 'iframe',
        '#attributes' => 
        array (
          'src' => 'http://core.test/media/oembed?url=https%3A//www.youtube.com/watch%3Fv%3Dv747p7xEcgg&max_width=0&max_height=0&hash=2kCbWsOQ7urG-UZLzC9KUBuT1B-7jnlX66TsoG_WL7g',
          'scrolling' => false,
          'width' => 200,
          'height' => 113,
          'class' => 
          array (
            0 => 'media-oembed-content',
          ),
          'loading' => 'lazy',
        ),
        '#attached' => 
        array (
          'library' => 
          array (
            0 => 'media/oembed.formatter',
          ),
        ),
      )

    I think this MR needs to perform that
    https://www.youtube.com/watch?v=v747p7xEcgg
    ๐Ÿ‘‡

    http://core.test/media/oembed?url=https%3A//www.youtube.com/watch%3Fv%3Dv747p7xEcgg&max_width=0&max_height=0&hash=2kCbWsOQ7urG-UZLzC9KUBuT1B-7jnlX66TsoG_WL7g
    

    transformation, by adding a new computed property on the field definitions for the oEmbed-using MediaType bundles of the Media entity type. That can be done using hook_entity_bundle_field_info_alter(). The MR already needs that for something else too: https://git.drupalcode.org/project/experience_builder/-/merge_requests/1...

    (The hash is coming from \Drupal\media\IFrameUrlHelper::getHash(), the rest from \Drupal\media\OEmbed\UrlResolver::getResourceUrl().)

    This would also result in \hook_oembed_resource_url_alter() etc firing (see the example implementation: it rewrites all YouTube URLs to the cookieless variant!)

  • ๐Ÿ‡ง๐Ÿ‡ชBelgium wim leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ
  • ๐Ÿ‡ง๐Ÿ‡ชBelgium wim leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ

    This needs work for:

    1. multiple remarks on the MR
    2. but most importantly because it's not yet resolving an author-provided URL to the corresponding oEmbed URL, which is why remote videos refuse to render

    Note: I'd be totally fine with splitting this across multiple MRs, so you could already land the "local video" MR immediately.

    Finally: this is not yet handling document links.

  • ๐Ÿ‡ง๐Ÿ‡ชBelgium wim leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ
  • Pipeline finished with Success
    4 days ago
    Total: 961s
    #526333
  • Pipeline finished with Failed
    4 days ago
    Total: 964s
    #526373
  • Pipeline finished with Failed
    4 days ago
    Total: 947s
    #526469
  • Pipeline finished with Canceled
    4 days ago
    Total: 353s
    #526821
  • Pipeline finished with Failed
    4 days ago
    Total: 717s
    #526824
  • Pipeline finished with Failed
    1 day ago
    #528630
Production build 0.71.5 2024