- Issue created by @hestenet
- Status changed to Fixed
over 1 year ago 5:15pm 26 May 2023 - 🇺🇸United States thejimbirch Cape Cod, Massachusetts
hestenet → credited thejimbirch → .
Automatically closed - issue fixed for 2 weeks with no activity.
eclipsegc
254 replies
eclipsegc
3 months ago
I’m a distribution maintainer and have been since Drupal 6. I have serious reservations about what I see, and I’d like some visibility into the decision making process and how we got here.
hestenet (he/him)
3 months ago
I think the best places to get the current details are probably:
https://www.drupal.org/about/core/strategic-initiatives-distributions-an... →
https://git.drupalcode.org/project/distributions_recipes/-/blob/1.0.x/do...
https://git.drupalcode.org/project/distributions_recipes/-/blob/1.0.x/do...
Drupal.org
Strategic Initiatives: Distributions and Recipes
[The Recipes Initiative] is a great example of making it easy for people to be part of Drupal and the open web." - Dries Buytaert, Driesnote, DrupalCon Prague The Problem Drupal is a highly flexible framework, but provides only basic configuration, requiring lots of adjustments for developing web applications. If site builders had the option to choose from predefined 'recipes' that assembled Drupal functionality for specific needs (e.g: blog site, news site, e-commerce site, etc) that would be a significant improvement to getting started with Drupal.
Jul 5th, 2022
GitLabGitLab
docs/recipe_roadmap.md · 1.0.x · project / distributions_recipes · GitLab
For more information about this repository, visit the project page at
https://www.drupal.org/project/distributions_recipes →
GitLabGitLab
docs/composer_project_template.md · 1.0.x · project / distributions_recipes · GitLab
For more information about this repository, visit the project page at
https://www.drupal.org/project/distributions_recipes →
hestenet (he/him)
3 months ago
And yes, a number of distro maintainers have attended these meetings. Commerce, Open Social, Varbase folks in particular have been fairly active, but I'm sure others.
eclipsegc
3 months ago
ok. Commerce I have long ancient connections to. I’ll maybe reach out there and talk more.
hestenet (he/him)
3 months ago
Would you like to articulate your concerns here or in subthreads?
eclipsegc
3 months ago
I’m happy to, though I have before
hestenet (he/him)
3 months ago
There's a bunch of history in the past meeting issues which is probably hard to parse through
eclipsegc
3 months ago
so perhaps I should find that thread instead
eclipsegc
3 months ago
I’m happy to do it again though
eclipsegc
3 months ago
perhaps I’ll do it better
hestenet (he/him)
3 months ago
That could work too - maybe re-link the prior thread as a new thread here again
eclipsegc
3 months ago
you want me to make a 5? or what are you thinking? (edited)
hestenet (he/him)
3 months ago
I guess we can keep using this one.
eclipsegc
3 months ago
ok
eclipsegc
3 months ago
so, here’s my basic concern
eclipsegc
3 months ago
Install profiles/distributions come in a few flavors.
eclipsegc
3 months ago
Here’s our product, deal with it
Here’s our product, and some stuff you might want to customize
eclipsegc
3 months ago
Drupal 8+ (I guess we’re calling this “Modern Drupal” now) is pretty good at 1.
eclipsegc
3 months ago
it’s absolutely terrible at 2
hestenet (he/him)
3 months ago
Truth
eclipsegc
3 months ago
when CMI was being initially developed, I pointed this out, a lot because the direction of CMI essentially precluded 2
eclipsegc
3 months ago
I was told not to worry, everything would work out
eclipsegc
3 months ago
it’s been about a decade, and the only reason I have any of the tools I want is because I built them… and personally, that’s a little irritating to lose that sort of fundamental capability (which D7 and earlier could all do while half-asleep) and also, I have a lot of D7 stuff to help bring forward into “Modern Drupal” as part of my regularly scheduled job.
hestenet (he/him)
3 months ago
I think customization wasn't super easy in 7 either for what it's worth, at least not when it came time to try and do a distro update.
hestenet (he/him)
3 months ago
(speaking from the COD d7 events example in particular)
eclipsegc
3 months ago
add to that the fact that I’ve been maintaining multiple distributions that fall into the “2” category since the D6 days
hestenet (he/him)
3 months ago
But that's neither here nor there, I would agree with the thrust of your conversation that the new effort should try to solve this problem, for sure.
eclipsegc
3 months ago
Well, there’s definitely effort involved, but at least it doesn’t try to delete all your config and uninstall the module.
eclipsegc
3 months ago
which is what Modern Drupal desires to do
eclipsegc
3 months ago
So (trying to be fair, please feel free to correct me if I’m wrong) the focus on “recipes” which seem to be
install this thing once
hope it’s what you need
never get updates for it because it’s not really code
yet another yaml thing (yayaml? lol, also I have serious reservations about more yaml, I know it’s in vogue, but I think it will prove to be a mistake)
You can maybe see why I’m worried about the direction?
eclipsegc
3 months ago
The initiative that purports to care about a space I have cared about for like… 14 years isn’t addressing any of the problems I have.
hestenet (he/him)
3 months ago
I think I follow what you mean.
So the approach with composer templates + config actions for recipes creates 'hey you've started off on the right foot - and you can update the individual components as they are updated - but we're losing the concept of updates for the holistic package' is that accurate?
eclipsegc
3 months ago
Maybe… I should perhaps outline my own solution a little bit and see if the juxtaposition of ideas makes it clearer?
eclipsegc
3 months ago
would that be welcome?
hestenet (he/him)
3 months ago
Sure.
hestenet (he/him)
3 months ago
I think I'm seeing a picture of a problem statement, but not quite the specific:
'I want to see A, B, and C problems specifically addressed'
eclipsegc
3 months ago
ok, so I’m working with legal to get this module released right now, I’d like to just point and say “something like this” but I have a few more hoops to jump through.
hestenet (he/him)
3 months ago
Fair enough
eclipsegc
3 months ago
but my code essentially does this:
eclipsegc
3 months ago
It allows install profiles to enumerate modules in the code base that are “optional”
Optional modules are evaluated during config import and they are kept installed even though the upstream core.extension says they’re not installed
Optional modules are removed from core.extension when config is exported.
Optional modules’ config is written to their module’s config directories (instead of the global sync dir) during config export. This allows developers to continue to improve their optional modules
Enabled optional modules config is imported during global sync operations
Optional modules (or maybe modules in general) often need to modify existing config (giving roles new perms is a great example, I think recipes has a similar use case) so they can include a config/partial directory, and any config of the same name is merged into profile config during module install/config import
Partials are unmerged during config export to keep upstream profile config clean.
:star-struck:
1
eclipsegc
3 months ago
there are other things, but this is a good starter list of the sorts of operations required to build and maintain complex install profiles that fall into category 2
hestenet (he/him)
3 months ago
Interesting. I'd be curious to get
@bircher
’s and
@alexpott
’s interpretation of where that overlaps and does not overlap with the kinds of capabilities that are being built so far.
hestenet (he/him)
3 months ago
But the primary 'problem space' that your code focuses on solving is 'optional modules and the optional config that goes with it' ?
eclipsegc
3 months ago
yes, because in my experience, that’s a pretty normal thing for install profiles to want, especially if you’re thinking of anything in the wix/squarespace kind of arena
:heavy_plus_sign:
1
eclipsegc
3 months ago
It’s a fun space, for sure, but it’s also super nuanced, and you need to have bi-directional config “transforms” where modules can modify active config but those same modifications, since they’re module specific, need to never leak into the profile’s config.
eclipsegc
3 months ago
@dww
glad you’re enjoying this lol
:laughing:
1
eclipsegc
3 months ago
And I’m not saying any of this is easy. It’s not really, and the code I have needs a ton of work to be:
usable for people who are not-me
core worthy
But the reality is that recipes look like a site builder tool, not an install profile maintainer tool.
eclipsegc
3 months ago
And regardless of whether or not that’s a fair statement, I know that none of this is addressing my real problems with “Modern Drupal” install profile development.
dww
3 months ago
Disclaimer: I've been away from this initiative for about 1/2 year, so I'm the wrong person to be "debating" any of this.
But the reality is that recipes look like a site builder tool, not an install profile maintainer tool.
I think that's mostly accurate. My understanding is that the idea is to try a whole different approach from how we've conceived of "distributions" in the past, and develop a new technology that solves some (but not all) of the problems based on all of our collective experience with the pain points of the previous approach(es).
dww
3 months ago
I believe the answer to "maintaining a distro sux!" from this initiative would be something like:
Yup, sure does. But that's because we've been trying to do too many different things with a single tool. Most of what people need from "distros" would be better served with composable recipes. Giant monolithic distros are a PITA for everyone from end users to distro maintainers. So lets not.
eclipsegc
3 months ago
ok
eclipsegc
3 months ago
so, are my criticisms unfair?
dww
3 months ago
Again, please don't take that as "the truth"... that's just my take.
eclipsegc
3 months ago
I’m open to that feedback
dww
3 months ago
Not necessarily "unfair" but perhaps "misplaced"? :sweat_smile: But really, I'm mostly a fly on this wall, and I defer to the folks who've actually been working on this to say more.
eclipsegc
3 months ago
ok
alexpott
3 months ago
@dww
you’re doing great - pretty much what I would say. I’ve also not had much time for the initiative recently.
@eclipsegc
feedback is great and criticism is fine too. Just to point out, I have driven a lot of the initial work on this initiative and I am a maintainer of a distribution.
:thankful:
1
:100:
1
eclipsegc
3 months ago
nods
alexpott
3 months ago
This initiative is aimed at site builders primarily. But it does hope in the long run to replace distributions.
eclipsegc
3 months ago
ok
dww
3 months ago
Yeah, I was gonna say something about
@eclipsegc
’s fear no distro maintainers have been involved until he came here to set us straight would be "unfair", although that's an unfair characterization of what
@eclipsegc
actually said. :wink:
eclipsegc
3 months ago
yeah, I definitely didn’t say that. There are a lot of people in Drupal who have tried their hand (at various levels of success) in building and maintaining an install profile, so it’d be silly to assume none were involved because anyone who’s been around long enough has at least done a bit of it.
:+1:
1
eclipsegc
3 months ago
:slightly_smiling_face:
alexpott
3 months ago
I think the problem of bi-directional config - or even tri-directional config (module author, distribution maintainer and site) - it really really hard. And all the solutions I’ve seen convince me that we should try as much as possible to avoid the problem altogether.
:100:
1
eclipsegc
3 months ago
So, last time I saw, recipes were essentially a set of yaml directives that Drupal sort of executes to produce an intended end result
eclipsegc
3 months ago
is that a fair characterization of their current state?
eclipsegc
3 months ago
almost a yaml rpc of sorts, except not remote.
alexpott
3 months ago
The reason I pointed out that I am a distribution maintainer was because
@eclipsegc
asked
Are there any distribution maintainers who have given feedback on the direction?
eclipsegc
3 months ago
yup, and I appreciate that reality and you outright saying it.
eclipsegc
3 months ago
so thank you for educating me on that
eclipsegc
3 months ago
it’s hard to ask such a question without sounding like an ass
:point_up:
1
:hugs:
1
eclipsegc
3 months ago
so I appreciate no one getting defensive about it, because that’s definitely not my intent
:hugs:
2
alexpott
3 months ago
Recipes are desperately trying not to have code. If they do then they are a distribution or a module.
eclipsegc
3 months ago
right
eclipsegc
3 months ago
I immediately thought of templates
eclipsegc
3 months ago
recipes want to do things like create new node bundles
alexpott
3 months ago
I have to take a child to bed. I will return to this thread tomorrow.
eclipsegc
3 months ago
cool
alexpott
3 months ago
Templating is definitely something we’re interested in. Getting user input while applying a recipe is discussed in the docs.
eclipsegc
3 months ago
do they provide templates for node bundles they create? I think the answer is “really don’t want to be in the business of code on disk”
eclipsegc
3 months ago
interesting
eclipsegc
3 months ago
ok, well little ones have to go to bed at the right time, otherwise everything sucks. :slightly_smiling_face:
eclipsegc
3 months ago
To sum up my basic position:
I’ve never seen anyone try to tackle the config transforms that I feel are necessary for the distributions I’ve characterized as type 2.
I fully acknowledge that could be hard to get right. I’m not sure that’s ever really stopped us in the past
The hardest bit of this nut to crack is probably when module dependencies get involved. I’ve not attempted to solve that in my own code yet, but it’s next on the chopping block because I’ll need it in the next sprint or two.
The lack of tracking files on disk and what config is user generated is a big part of the broader problem space too.
I worry about introducing a whole new paradigm. I’d be less worried if we tried transforms and came up with good reasons it won’t work.
eclipsegc
3 months ago
Thanks for listening. I appreciate it
hestenet (he/him)
3 months ago
Cheers - thanks for sharing those thoughts, and especially organizing them in a way that helps make the challenge clear.
bsnodgrass (he/him)
3 months ago
Wow, great thread, lots to think about… mostly commenting to track the thread
ccjjmartin
3 months ago
Just want to add that from my perspective this initiative does appear to remove the maintainers ability to do updates on sites. Please correct me if I have missed something.
In contract though it appears to actually be better at allowing customization of a site during initial setup.
In my personal opinion I tend to support the idea of allowing maintainers to update sites because it allows a single dev to make the change and everyone benefits. Rather than making 100,000 devs make the exact same change. I am generally the odd man out when this argument comes up though.
It seems like devs who build sites quickly will benefit but devs who offer long term support won't be very happy about this.
ccjjmartin
3 months ago
I am probably wrong though because every site I have supported that had a profile installed was a pain to update and I have removed a lot of profiles over the years to make site updates easier.
eclipsegc
3 months ago
I think this is a going point
@ccjjmartin
. I’m not sure if it’s completely fair because I don’t know all the details of the recipe proposal, but from my limited understanding, I can totally see where you’re coming from.
eclipsegc
3 months ago
I have an install profile I maintain right now, deployment is a breeze with it. It’s one of the things I really like about the config transforms I’m proposing. I only have about 30-ish sites on it now, but we’re growing super quickly, and everyone can have different(ish) setups within the confines of the optional module’s I’ve set up. Despite this, deployments tend to go off w/o any real issues. (edited)
ccjjmartin
3 months ago
Had to go back and skim the thread, I am not sure the "config transformations" was described in detail but I would guess it is to update config with some exact change.
dww
3 months ago
Disclaimers:
This might be a terrible idea. :sweat_smile:
This might have been proposed, discussed, ruled out before.
I haven't thought of "all" [sic] the implications of what I'm about to say...
A recipe is a set of instructions on how to take a site from state A to state B. If the maintainer of the recipe needs to "update" sites using it from state B to state C, what stops them from having an "update recipe" designed for that transformation?
Initial brainstorm of problems this would run into:
Versioning:
Do we have the notion of recipe versions at all yet?
Do we record the version of a recipe as it's applied?
Do we have a way to query a site to find out what version(s) of which recipe(s) have been "cooked"?
Would the "update" recipes work in the same namespace? Or would the main recipe always have to assume you're starting from state A, and subsequent releases either get you to state B or C depending on which version you use, and you'd need a separate foo_update namespace for the recipes to get you from B to C, etc?
(edited)
dww
3 months ago
e.g. I provide a recipe to add an "Album" content type to your site, which depends on the "artist" and "song" content types, etc. Time flies, Spotify is invented, and now I want to add a "Spotify link" field to my album content type. Can't I release an "update your albums to support Spotify" recipe?
eclipsegc
3 months ago
@dww
this is definitely in the realm of what I’ve seen discussed before. I THINK the sentiment towards this was generally negative when I saw the discussion, but that was some time ago and things obviously could have changed.
eclipsegc
3 months ago
it occurs to me, this looks a lot like a “friendlier” replacement for hook_update_N()
eclipsegc
3 months ago
@dww
to be clear, the notion of “update-ability” of recipes seemed like something I thought folks were generally against. Again, I might have misunderstood
thejimbirch
3 months ago
Using your recipe, what if I already installed a spotify field, and a bandcamp field?
thejimbirch
3 months ago
Could you update your recipe for new installs, and create a new recipe alteration to add a field if the content type exists?
dww
3 months ago
Yeah, exactly... it's a very messy can of worms. :sweat_smile: We don't necessarily have to solve it all. But I do wonder if there's anything we should bake into the initial version of this cake to at least make this can of worms easier to navigate in the future, especially re: versioning + remembering what we did to a site already. It does sorta sound like I'm proposing that we slowly re-implement hook_update_N() ...
dww
3 months ago
p.s. I'm not actually "proposing" this, just brainstorming... :joy:
eclipsegc
3 months ago
right
eclipsegc
3 months ago
and that’s kind of the head scratcher for me, or at least one of them. Recipes seem like the sort of thing we’d do with hook_update_N() today. And yes a “friendlier” version of that might be nice, but it does feel like we’d need to store some version number, along with an ever growing set of yaml to provide other upgrade instructions. And I think that was the argument against doing it in the first place.
eclipsegc
3 months ago
but again, sentiment might have changed, I don’t know
thejimbirch
3 months ago
There isn't a lot of momentum here, so I imagine sentiment is not set in stone.
thejimbirch
3 months ago
I came here after the Driesnote where the conversation centered around "Starter Kits". Which leads me to think that you start from it and don't get updates as you go. But I know you, and many other come from the distribution maintainership, so there are a lot of needs/wants to work out. (edited)
:100:
1
eclipsegc
3 months ago
yeah, and… if “distributions” wasn’t part of the initiative branding, I probably wouldn’t be talking much
eclipsegc
3 months ago
if this was just about trying to make things easier for onboarding new people to Drupal… ok cool. But distributions is allegedly part of the goal, and so I’m here.
thejimbirch
3 months ago
And we are glad.
dww
3 months ago
I can safely say one of the key points of this new technology is that it would allow you to compose multiple recipes in the same site. Right now, you either get a Commerce Kickstart site, or a COD site, or a Thunder site, or whatever, but you can't get a pre-configured store starting point bolted into your COD site.
eclipsegc
3 months ago
Yeah… that sounds good in theory
eclipsegc
3 months ago
I worry that in practice, it won’t be very effective
eclipsegc
3 months ago
mostly because each distro will probably provide its own entity bundles, and you’ll still have to figure out how to get it all applied properly
thejimbirch
3 months ago
Can you give an example of "applied properly"?
thejimbirch
3 months ago
If I have a Calendar recipe that provides and Event CT, a taxonomy vocabulary or two, and a calendar view.
eclipsegc
3 months ago
Sure
eclipsegc
3 months ago
I have a Calendar recipe that I want to be an event I can sell
eclipsegc
3 months ago
so I fire up my calendar recipe. It’s responsible for creating views and node bundle of event for me.
thejimbirch
3 months ago
lol, I was just thinking my example was too simple.
eclipsegc
3 months ago
I then try to apply Commerce Kickstart on top of it to sell events
eclipsegc
3 months ago
but of course, Commerce Sells products, not events
eclipsegc
3 months ago
and products are they’re own top level entity type with another type called variants that holds price variations
eclipsegc
3 months ago
so now I have 3 entity types (nodes, products and variants)
eclipsegc
3 months ago
Commerce isn’t selling my events, so I’ll need to add a product reference field to my event node bundle and ensure I create products to sell for each event first, or configure it with Inline Entity Form, or what have you
eclipsegc
3 months ago
point being… I didn’t necessarily save myself any real time, I still had to do the Site Builder tasks required to integrate the two things together, and we have to hope the recipes were simple enough that a bunch of other baked in assumptions didn’t come along (I used to work at Commerce Guys, I’ve worked on Commerce Kickstart, it does a LOT of configuring to get a Drupal site to be the way it wants)
eclipsegc
3 months ago
Granted… that was 9 years ago-ish now, but Distributions are fundamentally complicated.
thejimbirch
3 months ago
Makes sense. and looking back to D7 , I guess I should have named my recipe the Really Simple Calendar recipe. :stuck_out_tongue_winking_eye:
eclipsegc
3 months ago
But like… I’m sure the same is true for COD
eclipsegc
3 months ago
I bet it ships its own entity bundles
thejimbirch
3 months ago
I don't know what that is, Call of Duty?
:laughing:
1
eclipsegc
3 months ago
Conference Organizing Distribution I think
:+1:
1
eclipsegc
3 months ago
I prefix entity bundles in my distros with the distro name my_distro_news
eclipsegc
3 months ago
I don’t know what the future holds for a site, but I definitely never want users installing something that THINKS it knows what my bundles are because in reality… that’s more likely to break things than it is to magically give me new functionality that I actually want.
thejimbirch
3 months ago
I have done the same thing, just in modules, not in distos.
eclipsegc
3 months ago
Well, for the most part, distros are modules
eclipsegc
3 months ago
they’re just a special one off
thejimbirch
3 months ago
So how about this for a thought. Are recipes by themselves to simple to replace distributions?
thejimbirch
3 months ago
And could they be used as part of a distribution?
eclipsegc
3 months ago
I think that’s what Alex and others are trying to achieve
:+1:
1
thejimbirch
3 months ago
So, for a site builder, they could just use a simple calendar/blog recipe, but a distribution could use them also and more.
eclipsegc
3 months ago
I, for one, don’t really want a pile of yaml that describes all the nuances of how my distro is setup.
eclipsegc
3 months ago
it does very little to excite me about the process.
eclipsegc
3 months ago
and even if it’s just a distro-building tool and the end result is still me exporting my pile of config that represents all the changes that the recipes introduced… fundamentally (as discussed earlier in this same thread), that doesn’t give me better tools for maintaining my distribution.
eclipsegc
3 months ago
And it’s the maintenance that tends to be difficult for people.
thejimbirch
3 months ago
Thanks for that. I started a new topic to the meeting: https://drupal.slack.com/archives/C2THUBAVA/p1677626480498299
thejimbirch
:five: Who are the maintainers of the most used distributions and install profiles? And how can we get them here to conversate and contribute?
Posted in distributions-and-recipes | Feb 28th | View message
ccjjmartin
3 months ago
I believe the answer to this is yes:
https://drupal.slack.com/archives/C2THUBAVA/p1677626014674569?thread_ts=...
thejimbirch
So how about this for a thought. Are recipes by themselves to simple to replace distributions?
From a thread in distributions-and-recipes | Feb 28th | View reply
ccjjmartin
3 months ago
Prefixing bundle names seems pretty necessary to avoid name collisions.
dww
3 months ago
Prefixing bundle names seems pretty necessary to avoid name collisions.
And basically breaks the appeal of composability. :grimacing: Which is a feature and a bug. Which I believe is (one of) the Really Hard Problem(s):tm:
@eclipsegc
is worried about, and that we don't yet have a good answer to. I remember ~6 months ago we were talking about exactly this, and didn't have any leads on realistic solutions then. Doesn't sound like we've gotten any further...
Noel Rivas
3 months ago
Very interesting thread
leslieg
3 months ago
My simplistic understanding of recipes in relation to project browser was that I want to add functionality such as event management to my site and I shouldn’t have to install each individual module and I don’t know what the config needs are, a recipe would take care of this for me, I’m understanding the complexity of wanting multiple recipes that may conflict after reading this thread, and of keeping the recipe config updated,
:heavy_check_mark:
1
Gábor Hojtsy (he/him)
3 months ago
My understanding is also that recipes are meant to be a site builder productivity tool. Combining recipes may not necessarily give you ready to run tools, you may need to do a bit of wiring up, but you would get a head start as opposed to manually needing to pick out the modules and entity bundle structures, etc. I don’t think we’ll ever figure out a way to magically combine arbitrary sets of recipes and them being ready to go without a bit of site build to wire them together. So recipes would be simpler than distros, normally a recipe would not be “A site that has a blog and an event ticketing system with calendar and store”. Also my understanding is recipes would not aim to support updatability by design as they are meant to automate part of a manual site build process that people would do (which itself would not come with updates or future support either). Finally my understanding is distros could use recipes in the future and fill in the gaps with updates, etc. If they remain the tightly controlled distro like before, they could still have a controlled way to keep up to date.
Either way the primary or initial goal of the recipes initiative IMHO is to componentise the layer below distros for the benefit of the site builder that would not be satisfied with one particular distro but need a bit of functionality from here and there.
:heavy_check_mark:
4
ronaldtebrake
3 months ago
I’d like to start by saying we at Open Social also see some of the concerns you raised in here as well, and I’m aware this does not address them. But in the light of asking if there are any other distribution maintainers who have given feedback on the direction I’d also like to share what part excites us from this initiative.
As said before distributions are fundamentally hard to maintain, but they are also hard to grasp for end users, what they are, what they can or can’t do and how to work with them.
We noticed as a distribution we’re usually an entry point in to the ecosystem, depending on the experience and background that could be an entry point to Drupal, to Composer, to config, to all of that combined.
Catering the product to many different needs (some need e-learning integration, others want to have the ability to host virtual events) and so much more of those optional use cases;
On top of that building something thinking about the best experience for end-users we need more than just Drupal. SOLR, Redis, RabbitMQ, tools that help us build products that can also deliver the experience users need, however some users might not be able to host, maintain or work with that in general.
This flexibility for our end users has sometimes been in stark contrast with the rigidity of our open source Drupal distribution.
Recipes will give the site builders the ability to compose their version of whatever community needs they are catering to and not put the burden on us to create a one-size-fits-all-community-distribution
So that might not be much on a technical maintainership level I’m excited about it, rather on a this is our story, this is what it can and cannot do (which will also help maintaining our issue queue :’))
Unfortunately due to lack of resources we haven’t been able to fiddle around with recipes just yet and therefor haven’t gotten to a level of constructive feedback on any of the concerns raised.
bsnodgrass (he/him)
3 months ago
Thanks to everyone for this. There is a lot too unpack here and think about the direction of this initiative. Can I suggest we keep this thread going over the next week or so. There are still many people who have not weighted in with their thoughts.
bsnodgrass (he/him)
3 months ago
I would like to hear from
@kreynen
and I haven’t seen
@alexpott
come back on this topic yet.
eclipsegc
3 months ago
ok, so I could certainly see a world where (For example) there are distro-specific recipes that know what entity bundles exist in the distro and do modifications as necessary to them.
What I don’t really see is a world where recipes intended for vanilla Drupal sites interplay nicely without serious planning and alignment from collaborators ahead of time.
Also, I’ll point out that, per all my other objections, if a recipe is changing configuration of an install profile, we’ve still not introduced anything that’s going to preserve the recipe-introduced-config, instead CMI will attempt to bring the site in sync with the profile’s config, which will either:
Successfully remove the config introduced by the recipe
Fail to successfully update
If we want a world where recipes are generically composable, then we’ll need a UI for applying them that lets us select local target configs like bundles/view_modes/etc.
Either way, I still see a big burden in terms of distro maintenance and I think recipes are likely to make my “Scenario 2: Distros that have optional modules” style problem MORE common place, meaning we’ll REALLY want tools to help manage that problem space.
bircher
3 months ago
This thread is already over a hundred comments long and conflates several things.
But for the sake of discussion I will just add one more to it. Maybe for the next meeting we can split the different aspects into different topics and continue discussing what still needs to be elaborated on.
I think the first part of the discussion around the critisism of Drupal 8 and CMI and the feeling that Drupal 8 made things worse is based on a particular perspective. We discussed this in a previous lenghy thread on this channel and didn't come to an agrement.
My take is that it conflates the different use cases of Drupal7+features and erroneously assumes that the config import/export are the "modern" replacement for all of it. For what it is worth the blog post in which I tried to articulate the differences is nearing its sixth birthday: https://nuvole.org/blog/2017/apr/25/configuration-management-dimensions
With Nuvole, we gave a lot of Drupal 6&7 trainings on how to use the features module to deploy configuration of a site between environments, and one of the key take-aways was to build the site as if it was a distribution. We then also gave Drupal 8 trainings on the same subject where the take-away was to let go of the Drupal 7 mindset and enjoy the config export/import baked into core.
The biggest difference in Drupal 8+ is that there is a clear separation of what is content and what is config. This is the only thing that changed fundamentally and there have been contrib solutions to re-define the lines (config ignore makes config into content and serialisation/de-serialisation like default_content or content_sync make content more like config). But yes all these things and tools are for the "deploy config of the same site to other environments" use case Drupal 8+ caters too and there is absolutely nothing in core for the distribution use case.
And this is in my opinion where this initiative comes in!
I agree with what was said by dww, alexpott and gabor, and I think the characterisation by @elipsegc "almost a yaml rpc of sorts, except not remote." is also pretty good.
Recipes are a site builder tool, not meant to replace distributions, but meant to replace installation profiles. So they are a layer under the distribution, you could imagine a distribution recipe that you select at the beginning of a project and it would include all the sub-recipes and get you to the same state installing a current distribution gets you. Having "templates" or otherwise interactive recipes would be needed to replace the installer aspect of install profiles though so this is still a bit away. Update hooks would be in conventional modules instead of the install profile and the discovery of modules in the active install profiles folder would go away.
But as dww mused, recipes are "a set of instructions" and "what stops them from having an update recipe", and the reply by eclipsegc "this looks a lot like a “friendlier” replacement for hook_update_N()" are very important observations.
And I think this is getting us back to the point raised in previous meetings: One of the big outstanding problems with composability of recipes. We need to make sure to engineer the recipes in such a way that they can be applied again. This may be because a recipe is composed of recipes that share the same recpe in their compositions, but also so that an updated recipe can be run again at a later point. It may be up to the recipe author to decide what happens in the different scenarios, but I think this is where this mechanism will become interesting for distribution maintainers.
Because some things you will want to do in an update hook, you don't ask questions and the site has no option to opt out, this is great for structural changes that the module assumes, ie the original use case of update hooks. But maybe some of the other things could be by instructing users to re-run the recipe, or use the "update recipe". I think no matter how much we want to avoid this complication, very basic examples will fall on their face without it. And I think it requires us to be more explicit with what the recipe can expect and in what case it fails and in what case it is considered applied without doing anything. For example a recipe creating a role should not fail if the role exists already but with additional permissions, (but it could add the missing ones?). Or a recipe that adds a permission to a role should fail if the role doesn't exist but do nothing when the permission is already set etc.
Lastly I am looking forward to the new tool eclipsegc is working on! It sounds a lot like "config_split for config_distro" ie the problem space I am very familiar with and I am curious how he addressed some of the chalenges.
That said maybe
@eclipsegc
you should check out config_distro and config_sync the introduction blog post is also from almost already five years ago https://nuvole.org/blog/2018/apr/09/config-distro But I think it didn't get a lot of traction because distribution maintainers perfer more control rather than giving it to the users of their distribution. But it does do the transformations you like, but then doesn't interfere with the deployment, wich your solution does if you use the transformation event for the "normal" config import/export.
NuvoleNuvole
Configuration Management dimensions | Nuvole
Unfortunately none of us from Nuvole are attending DrupalCon Baltimore and can’t, therefore, attend the “hallway track” and join discussions in person. We have held presentations about advanced configuration management in Drupal 8 at all Drupal events we attended in the last year including the last DrupalCon in Dublin. So I’ll try to cover some concepts here that could help the discussion about how the configuration management can be improved.
https://nuvole.org/blog/2017/apr/25/configuration-management-dimensions
NuvoleNuvole
Config Distro | Nuvole
Config Distro allows distribution developers to say how the "distributions configuration" is supposed to be updated and it allows distribution-based site developers to treat the distribution maintainers as a developer on their team. (6 kB)
https://nuvole.org/blog/2018/apr/09/config-distro
bircher
3 months ago
@eclipsegc
you should really read the configuration management dimensions blog post, I think you fundamentally misunderstand how recipes and CMI interact
eclipsegc
3 months ago
@bircher
ok. I will put it on the short list of things to read.
bircher
3 months ago
thanks! and really looking forward to steal your solutions for making config_split better btw (edited)
bircher
3 months ago
i want to know how you get around sorting sequences (maybe for a different :thread: ) (edited)
eclipsegc
3 months ago
I will however say this “We then also gave Drupal 8 trainings on the same subject where the take-away was to let go of the Drupal 7 mindset and enjoy the config export/import baked into core.”
I’ve been stuck using core’s config export/import process. It is entirely insufficient for my needs and that’s the reality I’m trying to communicate about.
eclipsegc
3 months ago
make another thread for sorting sequences. I’d be interested to dig in a little
bircher
3 months ago
yes but you are building a distribution and not a project site... the config import export is not for you
eclipsegc
3 months ago
yes!
eclipsegc
3 months ago
and that’s precisely why I’d like to see this initiative tackle that problem
eclipsegc
3 months ago
also, I don’t fundamentally differentiate between “profile” and “distribution”. So understanding where you personally split those terms might be helpful in reading your responses
bircher
3 months ago
:smile: and that is precisely the plan. But taking one step back, like going from .inc files and features+ctools+strongarm to .yml
bircher
3 months ago
for me a profile is the folder in your file system with the code in it etc. the distribution is the idea that is manifested in the profile
bircher
3 months ago
very meta
eclipsegc
3 months ago
right, I don’t want profiles to go away. Just… fwiw
eclipsegc
3 months ago
:slightly_smiling_face:
bircher
3 months ago
profiles are just modules of which you can install only one of, and only at the very beginning of a project.
eclipsegc
3 months ago
yes, and they are a very handy and precise way of setting a default state for a site.
eclipsegc
3 months ago
minimal vs standard as a great example
bircher
3 months ago
yes but those could very well be recipes
:100:
1
dww
3 months ago
And install profiles have a lot of limitations and pain points.
:100:
1
eclipsegc
3 months ago
that too feels like a separate thread. Profiles, functionally, are much more capable than they were in previous Drupal versions.
dww
3 months ago
TL;DR: this initiative is trying to develop a new technology to supplement (and maybe completely replace) install profiles as the building blocks for distributions.
:this:
1
:100:
1
eclipsegc
3 months ago
for all my complaints about dealing with CMI in profiles, the profiles themselves are more like real modules these days
:100:
1
dww
3 months ago
Right, and we're saying: so just use an actual module, not this weird hybrid beast. :sweat_smile:
:this:
1
eclipsegc
3 months ago
at the risk of being wrong
eclipsegc
3 months ago
I don’t think that’s what you’re saying. And ACTUALLY saying that, might be part of what’s confusing me
eclipsegc
3 months ago
let me try, and let’s see if I’m full of crap
dww
3 months ago
We're all still evolving in our terminology and understanding here.
eclipsegc
3 months ago
what I think you’re really saying is something like this:
Profiles as they exist in Drupal core today are insufficient to the task of defining a “Distribution” without a lot of specialized knowledge and maybe some bailing wire and duct tape
Let’s remove them in favor of something which does the job of defining a Drupal site’s state better.
eclipsegc
3 months ago
is that the essence of what you want me to hear?
dww
3 months ago
Point 1: yes. Point 2: Not exactly. :joy: poor thread! :rolling_on_the_floor_laughing:
dww
3 months ago
I might try to summarize point 2 as:
let's add something new, which isn't itself a module, that can make it easier for (some) distro maintainers, but could also make life better for site builders not using a distro at all, to execute a set of instructions to achieve a discrete feature or piece of functionality.
dww
3 months ago
I think one of the key missing pieces in your view here is we are not only targeting distro maintainers.
dww
3 months ago
If that's all we were doing, you'd be right that this isn't sufficient. But that's not the only point of this effort
eclipsegc
3 months ago
Right, the counter point to that is that for me, I’ve seen the focus so completely OFF of distro maintainers that I’m concerned.
dww
3 months ago
I'm not sure that's a fair point. Again, most of the people who've actually done any real work in here are distro maintainers, and they're working to solve a (subset) of their problems with install profiles.
eclipsegc
3 months ago
I am only communicating about how I have read what people said. I’m completely aware it could be my bias
eclipsegc
3 months ago
but any time I see discussions about recipes, it tends to be focussed on site builders. Again that could be a bias I read into it, and I’m perfectly willing to accept that
eclipsegc
3 months ago
In any event, saying “We want a tool that can be used to define Drupal’s state” is a perfectly reasonable tool to want. And I will freely agree that profiles are a definition of starting (and evolving) state for a site.
dww
3 months ago
From a branding / marketing perspective, there are a lot more site builders than distro maintainers. :joy: it's a bigger audience to impress, woo, appeal to, etc. So don't be surprised when the BDFL is talking about it through that lens.
eclipsegc
3 months ago
it seems like we couple the idea of profile-specific modules to this to get any module-y features we want, and tada, profile replacement w/o a .profile
dww
3 months ago
Again, this isn't a tool to “define state”, it's a set of instructions to execute to try to achieve an outcome.
dww
3 months ago
The state is still defined in config
eclipsegc
3 months ago
well, that’s the current reality of Drupal, sure… but it’s a communication-miss if we want more people to come to the board and try to build that idea they have in Drupal vs improving some site they maintain.
eclipsegc
3 months ago
seems like we might be arguing about agreeing. Recipes want to be a tool that can enumerate instructions for getting from one state to another. That will likely involve a lot of config massaging, and maybe some other relevant data updates as well.
eclipsegc
3 months ago
or do I still not understand?
eclipsegc
3 months ago
I guess, given this basic understanding, I’d just point out that a PHP based interaction pattern would be nice to have. I really don’t want to maintain a bunch of yaml for these sorts of definitions when I’m doing it for a distribution.
eclipsegc
3 months ago
Migrate did this, and it was a big net-loss IMO
eclipsegc
3 months ago
Having yaml is fine, I just don’t want to be forced into using it exclusively
dww
3 months ago
Then have a “distro module” and do whatever you want.
eclipsegc
3 months ago
that’s roughly “status quo”
eclipsegc
3 months ago
right?
dww
3 months ago
But you don't need PHP to do a lot of what folks need to do, and there are big downsides to using PHP in those cases…
eclipsegc
3 months ago
there are big down sides to using EXCLUSIVELY yaml
:100:
1
eclipsegc
3 months ago
I’m just asking for both
eclipsegc
3 months ago
sorry, used the 4-letter j-word
dww
3 months ago
Right: this initiative is not deprecating modules. :joy:
eclipsegc
3 months ago
obviously, there’s some PHP that’s parsing the yaml, I’m asking that it be possible to use the PHP directly in a supported manner for the process of defining a distribution
:100:
1
dww
3 months ago
That, too, is still :100: going to be supported.
eclipsegc
3 months ago
because:
$service->doRecipeAction('my_action', ['my config'])
Allows me to cmd-click into doRecipeAction and figure out what feeds the list of possible actions, and git a full list whereas:
action: my_action
foo: 1
bar: 2
tells me nothing
alexpott
3 months ago
Also this initiative is providing a whole new way of interacting with config via plugins. There’s nothing to stop other projects coming along and using these plugins via PHP. And we might chose to use PHP as the templating config language (who knows). Our starting position is that recipes themselves will be no code but that doesn’t mean the things that recipes rely will be specific to recipes.
eclipsegc
3 months ago
sure, so I guess I’m specifically asking for something like:
web
profiles
my_profile
recipe.yaml
recipe.php
Where I could provide either option and we favor one if both are present or something. Just to give devs who want/need more the ability to perform PHP level instructions to help the distribution be built in the process. But mostly, I favor this for discoverability of what’s possible rather than having to read reams of documentation to learn what’s possible.
:thinking_face:
1
eclipsegc
3 months ago
I know that’s kind of hand-wavey
eclipsegc
3 months ago
again, I just don’t want to end up having a similar experience to how Migrate went where yes I had a big pile of PHP, but at least I could look at the methods being run and figure out what my options were vs yaml where, unless you can find the PHP that’s consuming it, you’re going to be SOL, and even then, depending on the complexity of your yaml, it could be very difficult to track it all down (in the absence of complete documentation that you’d need to read all of)
eclipsegc
3 months ago
short version, I don’t want distro-building to go the way of migration-building.
kreynen
3 months ago
@eclipsegc
this is all great… I’m slammed right now with other priorities, but we have been working towards building a D9/10 version of
https://www.drupal.org/project/express →
with Recipes
Drupal.org
Express
This project contains the contrib layer of the Express install profile developed by the University of Colorado Boulder for the Web Express service and used throughout the University of Colorado system. The Express install profile designed and supported by Developers to allow Site Owners to create great looking websites that meet a university's accessibility, security, and branding policies. Site Owner != Site Builders. Express users are not expected (or allowed) to create content types, fields, or views. While limiting, this approach makes it possible to maintain a large number of Express sites with a relatively small staff.
Sep 16th, 2015
eclipsegc
3 months ago
ok
kreynen
3 months ago
As I’ve mentioned before, our use case is now different from what we did in D7 because all of the Drupal using CU campuses are on Pantheon and we have Custom Upstreams as standard part of our infrastructure
eclipsegc
3 months ago
right
eclipsegc
3 months ago
your
https://www.drupal.org/project/profile_module_manager →
sounds VERY similar to what I’m trying to get released right now
Drupal.org
Profile Module Manager
Profile Module Manager is an alternative to managing modules within a profile's .info. It includes functionality similar to Apps and Feature Set, but is designed to manage "Module Bundles" on top of a core Install Profile. Using Profile Module Manager to start sites with a smaller core profile improves performance while also addressing these issues:
#2067229: Allow install profiles to declare base profiles for Drupal 7 →
🐛
Install profile is disabled for lots of different reasons and core doesn't allow for that
Fixed
Sep 10th, 2015
Warped
3 months ago
One problem with updates to recipes, modules, starterkits, or any other way to define Drupal objects is that the site builder can change a view, content type, entity, etc. What if there were actual templates, machine names that could only be defined and copied to other machine names, but were under total ownership of the recipe, module, starterkit maintainer. Probably revisionable. That way, they could revert to previous versions, apply diffs of changes, view changes, etc. I understand it would need a lot of new work, but this is something we're use to when programming. Maybe prefixing a machine name with a special character could define it as a template only. That way they couldn't modify it, and changes couldn't be wiped out just because there was an update applied. Demo content is another area where it would be nice to separate so installing a module wouldn't always create content that you always seem to need to remove. It would be nice to be able to create a project and turn on demo content, but have it off by default. Unfortunately there are so many features to Composer and modules that sometimes I feel we only use 10% of what is possible. Education is part of it. Good examples that show what is possible is another part.
eclipsegc
3 months ago
Drupal 7 and earlier had roughly this feature in the form of ctools exportables (fully disclosure, I’m a ctools maintainer). Exports could be overridden by the user, and while the module that owned the export might update it, it would never change the run time very a user had customized. If the user ever decided they had made a mistake or they wanted the new changes to the exportable (say something like a view) then they need only click the “Revert” button in the UI and then we’d throw away their override and go back to using the version on disk.
eclipsegc
3 months ago
We discussed this at length with CMI during its early stages. They opted to not do that at all, and don’t even track where config was initially imported from, so even if you wanted to, Modern Drupal instances need a lot of leg work to make this a thing. I’ve got a POC for this that I’m trying to clean up and give to some people to contribute.
eclipsegc
3 months ago
Content is a separate topic. But it does seem like a solvable thing too.
bircher
3 months ago
@eclipsegc
the module config being used to update the site is the domain of
https://www.drupal.org/project/config_sync →
I am sure you looked at that before your endeavour. I just wanted to mention it just in case.
Drupal.org
Configuration Synchronizer
Configuration Synchronizer provides methods for safely importing site configuration from updated modules, themes, or distributions. By taking a snapshot of configuration as installed and comparing the snapshot to the new module or theme as well as the current active configuration, Configuration Synchronizer safely merges in updates without overwriting customizations. Configuration Synchronizer is built on the Config Filter framework used by Configuration Split and several other modules. Instructions are for the 8.x-2.x branch, the active development branch.
Feb 11th, 2015
bsnodgrass (he/him)
3 months ago
@kreynen
re: we have been working towards building a D9/10 version of
https://www.drupal.org/project/express →
with Recipes - I hear you about being slammed, but are there some details about your experience with tooling available, you or another team member could share?
eclipsegc
3 months ago
@bircher
I didn’t, but I will now. Curious what you did.
bsnodgrass (he/him)
3 months ago
@bircher
I had
https://www.drupal.org/project/config_sync →
on my radar, but haven’t yet attempted to incorporate it into our process. Thanks for the nudge.
bircher
3 months ago
@eclipsegc
mostly not my code.. I just had some influence on the architecture.
@bsnodgrass (he/him)
it is rather opinionated as in: it tries its best to let users update config from all the modules
kreynen
3 months ago
@bsnodgrass (he/him)
happy to write up a summary of how we are approaching this since it ties into https://dev.to/kreynen/features-salesforce-and-drupal-have-in-common-pro.... In general, we are trying to approach CMS and CRM projects (as well as the tightly coupled CMS/CRM projects) with a similar approach. Recipes are similar to the pre/post package install scripts. I’m looking for better Salesforce documentation, but if you look at https://help.salesforce.com/s/articleView?id=sf.cg_concept_admin_install... and https://help.salesforce.com/s/articleView?id=sf.cg_concept_admin_install... it might give you some insight into how we are approaching this.
bsnodgrass (he/him)
3 months ago
@kreynen
That would be very helpful, in the meantime I will review these links!! Thanks
bsnodgrass (he/him)
2 months ago
@kreynen
Did you get a chance to write up any summary on your approach?
Also sent to the channel
bsnodgrass (he/him)
2 months ago
Lot’s on discussion on this thread… any thoughts on actionable items here?
eclipsegc
2 months ago
My primary concern was that recipes fundamentally don’t improve the distribution maintenance issue. They’re a tool for executing Drupal level actions (those could be config or even fringe into hook_update_N territory). That was and continues to be my concern. It is perhaps a branding issue mostly, but I’d be way more comfortable if there were material improvements to the distribution maintenance problem. I’ve outlined many of the problems I see (in this thread) which I think core should be addressing.
bsnodgrass (he/him)
2 months ago
Seems like a legitimate concern. Is that something that could be put into the Issue queue? with details of the improvements to the distribution maintenance problem/s?
bsnodgrass (he/him)
2 months ago
@bircher
any chance you could help summarize things in this thread into an issue? I don’t maintain any distributions so I don’t feel I can help much.
eclipsegc
2 months ago
I cannot really create public posts on d.o (employment agreement) so my hands are a little tied.
:+1::skin-tone-3:
1
bsnodgrass (he/him)
2 months ago
Ah!
bircher
2 months ago
I also don't maintain any distribution at the moment :laughing:
dww
2 months ago
:disappointed: "You're doing it all wrong. You're not taking any of my concerns into effect. But I can't write up my concerns in public due to an employment agreement." :exploding_head: WTActualF? :wink: I'm sorry, but if you can't publicly describe your concerns, we can't solve them. I'm not going to take my "free" time to try to read your mind based on your Slack comments and try to change course on this initiative accordingly. Can you not write up your concerns in general? You don't have to post NDA-protected code from your client projects. But if that's the stunning conclusion to this monster thread, I'm left a very :sad-panda: ...
eclipsegc
2 months ago
I’m happy to write it up. I just can’t post it to d.o
eclipsegc
2 months ago
Which tends to be the ask. But I’ve communicated pretty heavily what the problems are. So the interpretation feels a little “meh”.
eclipsegc
2 months ago
I don’t expect everyone to read 200+ comments either so. Will a Google doc suffice?
:+1::+1::skin-tone-3:
2
eclipsegc
2 months ago
I’m also actively working on my side to get code released that demonstrates solutions I think are promising. So again, not like I’m just being obstinate.
dww
2 months ago
Sure, I guess a gdoc is fine. Anything is better than a 200+ comment Slack thread and a "sorry, I can't post this on d.o". :grin:
eclipsegc
2 months ago
That was definitely NOT the conclusion of this thread though as I wrote a lot of detail in this thread about the actual problems.
eclipsegc
2 months ago
Sure
eclipsegc
2 months ago
I will see if I can make it happen before the next meeting.
:thankful:
3
:tada:
1
bsnodgrass (he/him)
2 months ago
@eclipsegc
That would be awesome!!!
Fixed
Meeting
hestenet → credited thejimbirch → .
Automatically closed - issue fixed for 2 weeks with no activity.