Extend recipe runner to create content provided in the recipe's /content folder

Created on 23 June 2022, about 2 years ago
Updated 23 April 2024, 5 months ago

Problem/Motivation

Recipes need to be able to ship content.

The good news is that there's already a pretty well-established way to ship content with an extension: the Default Content module โ†’ !

The bad news is that it's a contrib module. Core has no solution for default content.

Proposed resolution

Let's take a subset of Default Content's functionality and add it directly to the recipe system. We should create a new ContentImporter service which, given a recipe, can read the files in its content directory (if there is one) and import them as content entities.

Everything else Default Content does, we can leave out. That includes the Drush commands, all interfaces (those are part of Default Content's API, which we don't need for our purposes), the ability to export content (and normalize it for export), and the events that it dispatches.

Then, change the recipe runner to use the new ContentImporter to import the recipe's content. We'll want test coverage ensuring we can import content entities of various types (nodes, media items, taxonomy terms, shortcut sets, files), and their dependencies (for example, an image media will need to also import a file).

I don't think a recipe.yml should need to be explicit about its content. If there's a content directory, it gets imported; otherwise, it doesn't.

๐Ÿ“Œ Task
Status

Fixed

Version

10.3

Component

Code

Created by

๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom alexpott ๐Ÿ‡ช๐Ÿ‡บ๐ŸŒ

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • ๐Ÿ‡ง๐Ÿ‡ชBelgium Wim Leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ
  • ๐Ÿ‡ง๐Ÿ‡ชBelgium Wim Leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ

    This blocks #3418706: [PP-1] Populate the default shortcut set โ†’ , which is technically necessary for finishing the Recipe-ification of the Standard install profile in [#3417835.

  • ๐Ÿ‡ง๐Ÿ‡ชBelgium Wim Leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ

    Is there really nothing we can do here until โœจ Support default content entities Active lands? ๐Ÿค”

    Also, this is at least if not ?

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States phenaproxima Massachusetts

    I gave the default_content module a cursory look today and I'm wondering...what if we just, like, added the important parts of it to the recipe system directly?

    There are a number of things we don't strictly need from default_content, in order to make recipes import content. For example, the Drush commands are not necessary; neither is the ability to export content. The module also supplies an event-based API, which we could ditch for now. We could also rip out some deprecated material (like support for HAL+JSON-encoded content).

    Beyond that, it's really not that complicated of a module.

  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom alexpott ๐Ÿ‡ช๐Ÿ‡บ๐ŸŒ

    I think it is an acceptable stepping stone - I think a default content solution belongs in core. That can export and do the other things the module does. But I agree that this feels like a way we can go forward for now.

  • ๐Ÿ‡ง๐Ÿ‡ชBelgium Wim Leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ

    +1!

  • Status changed to Active 6 months ago
  • ๐Ÿ‡ง๐Ÿ‡ชBelgium Wim Leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ

    So I think that means โ€ฆ this? ๐Ÿ˜„

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States phenaproxima Massachusetts
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States phenaproxima Massachusetts

    Strategically speaking, I might add that getting this done will supercharge #3365143: [PP-1] Model core's Umami install profile as recipes โ†’ . Umami has shipped content from day one, and being able to bundle all that as a nice recipe (or set of recipes), instead of having to awkwardly install demo content via an otherwise superfluous module, is going to make recipes an even more attractive solution to automating site building tasks and creating great starting points.

    So...full steam ahead!

  • ๐Ÿ‡ง๐Ÿ‡ชBelgium Wim Leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ

    "supercharge" โ€” I like the sound of that! Full steam ahead indeed! ๐Ÿš‚

  • ๐Ÿ‡ฎ๐Ÿ‡ณIndia omkar.podey

    omkar.podey โ†’ made their first commit to this issueโ€™s fork.

  • Merge request !92Extend recipe runner to import content. โ†’ (Closed) created by omkar.podey
  • ๐Ÿ‡ฎ๐Ÿ‡ณIndia omkar.podey

    Even if we strip off all the normalise stuff we still need the denormalise for yaml content files right ?, this means a new class for denormalise as it's a lot of code .

  • Pipeline finished with Failed
    6 months ago
    Total: 173s
    #122999
  • Pipeline finished with Failed
    6 months ago
    Total: 234s
    #123000
  • ๐Ÿ‡ง๐Ÿ‡ชBelgium Wim Leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ

    #14: Correct, there's no way around that.

  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom alexpott ๐Ÿ‡ช๐Ÿ‡บ๐ŸŒ

    Can we make this MR against 11.x recipes - it'll need to land their first.

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States phenaproxima Massachusetts

    #14: IMHO we do not need the denormalization as a separate class. I know it's a lot of code, but in the interest of compactness and keeping API surface minimal, I think it's okay for the ContentImporter to be a single class that handles the denormalization as well.

    That said, if you really feel it would benefit from being more than one class, I don't feel super strongly.

  • Pipeline finished with Failed
    6 months ago
    Total: 200s
    #124783
  • Merge request !94Content importer 11.x โ†’ (Open) created by omkar.podey
  • Pipeline finished with Failed
    6 months ago
    Total: 194s
    #124792
  • Pipeline finished with Failed
    6 months ago
    Total: 222s
    #125044
  • ๐Ÿ‡ฎ๐Ÿ‡ณIndia omkar.podey

    Right now in the 11.x branch following the default_content module's convention which is to place the .yml file in the entity_type folder under the recipes folder works. You can try that by running php core/scripts/drupal recipe core/tests/fixtures/recipes/only_content.

    1. Needs cleaning next as multiple classes are added which could be combined.
    2. Needs implementation for handling media files.
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States thejimbirch Cape Cod, Massachusetts
  • Pipeline finished with Canceled
    6 months ago
    Total: 87s
    #125608
  • Pipeline finished with Failed
    6 months ago
    Total: 346s
    #125611
  • Pipeline finished with Failed
    6 months ago
    Total: 209s
    #126057
  • Pipeline finished with Canceled
    6 months ago
    Total: 52s
    #126322
  • Pipeline finished with Failed
    6 months ago
    Total: 167s
    #126323
  • Pipeline finished with Failed
    6 months ago
    Total: 199s
    #126366
  • Pipeline finished with Failed
    6 months ago
    Total: 182s
    #126395
  • Status changed to Needs review 6 months ago
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States phenaproxima Massachusetts

    Managed to get this sorted out (no pun intended - had to work with the Graph component to make it work), and the test coverage proves that multiple entities of different types are imported in dependency order when applying a recipe that ships content.

    I think this is ready for an initial review.

  • Pipeline finished with Canceled
    6 months ago
    Total: 129s
    #126418
  • Pipeline finished with Failed
    6 months ago
    Total: 207s
    #126421
  • Pipeline finished with Canceled
    6 months ago
    Total: 76s
    #126433
  • Pipeline finished with Failed
    6 months ago
    #126438
  • Pipeline finished with Failed
    6 months ago
    Total: 180s
    #126456
  • Pipeline finished with Failed
    6 months ago
    Total: 395s
    #126513
  • Pipeline finished with Failed
    6 months ago
    Total: 394s
    #126524
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States phenaproxima Massachusetts
  • Pipeline finished with Failed
    6 months ago
    Total: 504s
    #126580
  • Pipeline finished with Failed
    6 months ago
    Total: 371s
    #126596
  • Pipeline finished with Failed
    6 months ago
    #126607
  • Pipeline finished with Failed
    6 months ago
    Total: 365s
    #126768
  • Pipeline finished with Failed
    6 months ago
    Total: 428s
    #128989
  • ๐Ÿ‡ฎ๐Ÿ‡ณIndia narendraR Jaipur, India

    A follow-up documentation issue might be required to help understand how content can be created/imported using recipe system.

  • Pipeline finished with Failed
    6 months ago
    Total: 480s
    #129865
  • Pipeline finished with Failed
    6 months ago
    Total: 391s
    #129879
  • Pipeline finished with Success
    6 months ago
    Total: 404s
    #129886
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States phenaproxima Massachusetts
  • Pipeline finished with Success
    6 months ago
    Total: 394s
    #129939
  • ๐Ÿ‡ฎ๐Ÿ‡ณIndia narendraR Jaipur, India

    Thanks for the follow-up. One test scenario which is missing here is "Imported node with owned by user that does not exist".
    Rest LGTM.

  • Pipeline finished with Success
    5 months ago
    Total: 393s
    #131567
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States phenaproxima Massachusetts

    Good call on that test coverage; it found a bug!

    I think everything is fixed now.

  • Pipeline finished with Failed
    5 months ago
    Total: 276s
    #131691
  • Pipeline finished with Success
    5 months ago
    Total: 423s
    #131694
  • Status changed to RTBC 5 months ago
  • ๐Ÿ‡ฎ๐Ÿ‡ณIndia narendraR Jaipur, India

    Looks good to me, moving to RTBC.

  • Status changed to Needs work 5 months ago
  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom alexpott ๐Ÿ‡ช๐Ÿ‡บ๐ŸŒ

    Posted some comments on the MR. This is looking really nice. Just need to sort out the namespace. I suggest Drupal/Core/DefaultContent

    When you move it don't forget to update .recipes.gitlab-ci.yml with the new test and code locations.

  • Pipeline finished with Failed
    5 months ago
    Total: 474s
    #132588
  • Pipeline finished with Success
    5 months ago
    Total: 401s
    #132596
  • Pipeline finished with Canceled
    5 months ago
    Total: 13s
    #132780
  • Pipeline finished with Success
    5 months ago
    Total: 391s
    #132781
  • Pipeline finished with Success
    5 months ago
    Total: 393s
    #132785
  • Status changed to Needs review 5 months ago
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States phenaproxima Massachusetts

    I think I've addressed all feedback here, except for creating the follow-up.

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States phenaproxima Massachusetts

    Added a follow-up and linked it in the code.

  • Pipeline finished with Success
    5 months ago
    Total: 545s
    #132862
  • Status changed to Needs work 5 months ago
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States phenaproxima Massachusetts

    I just realized there's a huge blind spot in this MR: we're not testing importing translations!

    Definitely need to test that.

  • Pipeline finished with Success
    5 months ago
    Total: 447s
    #133255
  • Pipeline finished with Failed
    5 months ago
    Total: 239s
    #134455
  • Pipeline finished with Failed
    5 months ago
    Total: 463s
    #134481
  • Pipeline finished with Success
    5 months ago
    Total: 488s
    #134497
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States phenaproxima Massachusetts

    Okay, we're now testing translations. That said, there's still a bug here that I need to fix, then split off into its own blocking issue.

  • Pipeline finished with Failed
    5 months ago
    Total: 176s
    #134539
  • Pipeline finished with Failed
    5 months ago
    Total: 178s
    #134585
  • Pipeline finished with Success
    5 months ago
    Total: 424s
    #134660
  • Status changed to Needs review 5 months ago
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States phenaproxima Massachusetts
  • Status changed to Needs work 5 months ago
  • ๐Ÿ‡ง๐Ÿ‡ชBelgium Wim Leers Ghent ๐Ÿ‡ง๐Ÿ‡ช๐Ÿ‡ช๐Ÿ‡บ

    First round of review, but this is looking quite good already! ๐Ÿ˜„

    Some requests for test coverage (on top of the ones that @alexpott also asked for and are not yet added), some things that aren't clear, a bunch of nits.

    Looking forward to this one landing โ€ฆ but simultaneously somewhat worried that we're importing the same flaws/limitation as the Default Content module has. I have no real-world experience with it though, so I wholly defer to @alexpott making that judgment call. Because unless I'm mistaken, Alex does have quite a lot of experience with it?

  • Pipeline finished with Failed
    5 months ago
    Total: 395s
    #135372
  • Pipeline finished with Success
    5 months ago
    Total: 395s
    #135470
  • Status changed to Needs review 5 months ago
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States phenaproxima Massachusetts
  • Pipeline finished with Success
    5 months ago
    Total: 496s
    #138813
  • Pipeline finished with Success
    5 months ago
    Total: 492s
    #138869
  • Pipeline finished with Failed
    5 months ago
    Total: 532s
    #138997
  • Status changed to RTBC 5 months ago
  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom alexpott ๐Ÿ‡ช๐Ÿ‡บ๐ŸŒ

    I think this change is a great step into moving default content into core and providing recipes with the required functionality.

  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom alexpott ๐Ÿ‡ช๐Ÿ‡บ๐ŸŒ

    Committed and pushed b2dbc8f9a44 to 11.x and 838e7391d84 to 10.3.x. Thanks!

  • Status changed to Fixed 5 months ago
  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom alexpott ๐Ÿ‡ช๐Ÿ‡บ๐ŸŒ

    I think we should open a follow-up to add documentation. Especially around the expected flow of using the default_content module to create the content.

  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024