Allow "subrecipes" in the same git repo

Part of a recipe is composer.json for dependencies and other metadata.

Composer keeps it simple and supports only one composer.json per-project, at the root level of the repository.

Drupal’s support for sub-modules & sub-themes, each with their own dependencies, was one of the main reasons we created packages.drupal.org as a shim for supporting Drupalisms in Composer. We do not plan on doing anything special for recipes. Their metadata is distributed directly via Packagist.org as regular PHP projects, no special Drupal handling.

It's worth pointing out that there was a long discussion thread related to this in Drupal slack:
https://drupal.slack.com/archives/C072BF486FN/p1719955373571009?thread_t...

On the topic of "sub-things" being a Drupalism and therefore something we want to avoid repeating again, could git submodules be an approach that might be a better path forward here?

I would be very in favor of some sort of Git trickery here (subtree split?) to facilitate developing recipe suites in a monorepo setup. That's what I'm hoping Starshot will do, and I think it would be useful in other situations too.

Having recipes embedded in other recipes - similar to how modules can contain other modules - would be a hard no from me, as it contradicts the design of the recipe system and breaks the ability to disambiguate them. Let's not add more Drupalisms! :) But it sounds like we're on the same page about that, so I'm just repeating it here for the record.

I'd love to see an example of a recipe suite developed with a subtree split to Packagist. If that's the path Starshot follows (and I'm advocating for that), then I think that could serve as a good example and blaze a trail for other recipe authors.

I'm concerned that the discussion so far in this thread has largely been about the developer experience, which wasn't the main concern when I created this issue. I agree with the sentiment in #5 that if some git trickery could allow for a monorepo, as a recipe author myself I would consider that optimal.

That said regardless of whether or not we can or should provide an optimized recipe author experience, I would argue that we will have lost the mission for Starshot itself if site builders have to search and scout to find a collection of recipes that will match the functionality they could get from installing a single, popular plugin on a competing platform.

I would argue that we will have lost the mission for Starshot itself if site builders have to search and scour to find a collection of recipes that will match the functionality they could get from installing a single, popular plugin on a competing platform.

I'm not sure this is necessarily the case, although bear in mind I haven't actually used the recipes API yet, just reviewed various of it.

There seems like at least two possible ways to handle this:

1. Subtree split as mentioned above.

Let's say that recipe A provides the 'event' content type, and recipe B provides zoom API integration to add a meeting ID to events, then attach meeting recordings to a media video field on event content automatically when zoom makes them available.

Recipe B requires API key configuration etc. and is completely pointless for someone using the event content type for their local music scene so you'd never want it installed unless you're explicitly creating zoom meetings all the time.

If recipe A has a composer dependency on recipe B but otherwise no reference to it, then composer would install recipe B into the codebase, but that would just mean it's available on the filesystem. When you apply the event recipe A, nothing would happen, but project browser would identify recipe B and present it as available for installation - I think this is what's being suggested.

The main drawback I can see here is that if recipe A depends on recipe B, and recipe B depends on zoom_api module, then when a major version comes around, the modules that recipes B, C, D, E, F, G, H, I, J, and K depend on could mean a whole chain of dependencies in order to get recipe A ready for that major release.

2. A suggestions API based on applied recipes.

Let's say a site has just applied recipe A, project browser could then, via the d.o API, provide a list of recipes that depend on recipe A. This would mean not just the original recipe project, but any recipe on d..o could potentially be suggested based on its dependencies.

I have an (as yet mostly still in my head) idea about allowing projects on d.o to be subtree-split into different composer packages, with the eventual idea of deprecating submodule support in the extension system itself (to simplify extension discovery, autoloading etc.). e.g. a project with views/views.info and views_ui/views_ui.info could become two composer packages allowing the modules to be hosted separately and for us to discover modules in the vendor/ directory and autoload them from there. So if we added monorepo + subtree split support to d.o gitlab, that might be useful not only for recipes.

As part of the slack discussion linked above @thejimbirch mentioned composer suggest as an existing technical element that is relevant, but in practice not really used by anyone. In theory, a solution like the Project Browser could have support for composer suggest, and interactively present them as options to download and apply after the main recipe is applied. This could be path for making it easier to find and apply available extras for a recipe, but on its own it wouldn't allow for any distinction between "recommended" and "available" elements.

I haven't personally worked with git subtree splits, so I'm having trouble imagining how this would impact the process of managing all the recipes, particularly in what I believe is the intended future, where drupal.org projects are replaced by git.drupalcode.org projects. I would be concerned about any approach that adds more complexity to project maintainership.

Re: the scenarios presented in #7:

I agree with the point made in scenario 1, about the perils of composer requiring recipes and their associated modules if they weren't actually being used. Since composer has no concept of which modules are actually in use, this sounds like it will create unnecessary complications.

In scenario 2, the suggestions API sounds to me like the ecosystem association currently built into drupal.org. I'm not sure if this is planned to carry over to Gitlab at some point, but if it is, maybe that could be the underpinning for having additional recommendations?

That said regardless of whether or not we can or should provide an optimized recipe author experience, I would argue that we will have lost the mission for Starshot itself if site builders have to search and scour to find a collection of recipes that will match the functionality they could get from installing a single, popular plugin on a competing platform.

We could also say adding many recipes to one repo is jumping to a developer-focussed solution ahead of figuring out the user experience. Being able to find recipes designed to work together is something to consider in #3447063: [Meta] Plan for recipe findability on Drupal.org and Project Browser → .

Where the repositories are should be irrelevant to end-users. A monorepo would be a developer convenience for tightly-coupled recipes. Subtree splitting each recipe from the monorepo to each individual project makes the recipes functionally the exact same as each recipe in one repository.

In scenario 2, the suggestions API sounds to me like the ecosystem association currently built into drupal.org. I'm not sure if this is planned to carry over to Gitlab at some point, but if it is, maybe that could be the underpinning for having additional recommendations?

Projects remain on Drupal.org indefinitely, they are not moving to GitLab. This means we always have a place for metadata, categorization, user-focussed descriptions, etc for project browsing. This parallels other systems for browsing extensions - app stores, Wordpress plugins, browser extensions, and other equivalents to project browsing all have additional metadata and centralization, and don’t rely on just Git repositories.

we will have lost the mission for Starshot itself if site builders have to search and scour to find a collection of recipes that will match the functionality they could get from installing a single, popular plugin on a competing platform.

+1

adding many recipes to one repo is jumping to a developer-focussed solution ahead of figuring out the user experience

+1

Being able to find recipes designed to work together is something to consider in #3447063: [Meta] Plan for recipe findability on Drupal.org and Project Browser →

There's a very important distinction between child->parent and parent->child we shouldn't lose track of here.

Recipes need to be able to specify parent recipes they depend on, obviously enough.

But I think the essence of the proposal here is that a parent recipe should be able to define a curated set of child recipes, of both available and suggested types.

Doing so has a significant sitebuilder impact that goes beyond discoverability: it encourages recipes to be lightweight and modular.

I've read the issue and the Slack chat and I would like to summarize (I hope I got everything right).

We are touching several requeriments here:

• (1) Allow Recipes to declare suggested recipes (optional recipes that expand the recipe's functionality) (#0, #10)
• (2) Allow recipes to declare dependencies on other recipes (so they declare they expand another recipe's functionality), so they are 'available' when you apply the recipe they are dependent on (#10)
• (3) Allow PB to offer users related recipes based on suggested and available

From there, some problems/topics arise:

• (A) How Recipes declare related recipes in such way PB and composer get the info
• (B) How to manage recipes in terms of Composer packages to avoid issues like a snowball effect on recipe dependencies (recipe A suggests/recommends B, C, ..., and recipes M, N. O, etc, depend on A) so a site is not flooded with code from dependencies that are not used at all (#7)
• (C) How to manage the code of recipes in git terms: one repo per Composer package, a monorepo with subtree split with different composer packages ([#5, #6, #7]). This is less relevant as from a functional point of view as they behave the same (they are just different packages (#9))

The possible solutions (from #7):

• (S1) Rely on composer dependencies to install recipes and its dependencies and use that info in PB to offer related recipes (PB can guess the parent<->child relation of recipes, see also #10). This solution will hit the snowball effect mentioned above.
• (S2) Rely on D.org API that reports the relation of recipes and let PB display the related recipes. On installation, PB would composer require all the required recipes.

I would like to add my own thoughts:

• Relying on Composer to provide 1 and 2 is tricky because Composer doesn't cover this exact use case so it feels like we could develop a clever solution instead of an engineer solution. This will bite us back in the future
• Relying on data from Drupal.org feels better because we are proposing a funcionality on top of composer and php dependencies, on an upper level.

A practical proposal (but keep in mind that know PB just a user, not dev, I might be totally wrong):

• Recipes's yml file allows to declare suggested (and thus curated) optional recipes to apply
• Recipe's yml file allows to declare itself as suggested (and thus, not curated) optional recipes to apply given the parent recipe is being applied.
• With this info, given a recipe PB can display a list of suggested recipes (curated ones and not curated)
• When applying recipes, PB downloads all selected recipes and runs them. This way, only actual used dependencies are downloaded
• Each recipe is a Composer package
• In the future, depending on infra, recipes could be managed in a monorepo

In the future, depending on infra, recipes could be managed in a monorepo

This does not depend on infrastructure. An example of a monorepo being subtree split out is https://git.drupalcode.org/project/drupal_cms/-/blob/8e5cc51cdc876e8004f...