[META] What are the pain points of maintaining distributions, installation profiles, or other site starters?

The most difficult part of maintaining our distribution (Conversion Engine) is sifting through the configuration changes whenever we export configuration from a development site to add/update a feature of the distribution.

Our workflow is generally:
1. composer install
2. Create symlinks / scaffold Drupal dev site
3. Run the installer
4. Do the hacking, commit the codes, export the config to a specific config directory (we use $REPOROOT/active_config)
5. Run a robo script that first copies the active config directory to the profile config/install directory, and then strips the UUIDs and _core section of the config files. We use a .gitignore file, and some exclusion code in our robo script to exclude files we never want to distribute.

Things work really well, except that step five ends up constantly producing much larger diffs than we actually care about. We get keys in config files that are not even changed, but are in a different position, we get extra config keys that we have no idea why they changed or where they're coming from, hunting through the diff carefully to select just the items we intentionally changed is very doable, but it makes the process tedious and more error prone for devs less familiar with the install profile.

The next most difficult thing is probably the fact that we can't seem to install our distribution on hosting grade hardware, it needs to be done via Drush on a developers system due to the time it takes to install (around 6-7 minutes). We then just push the resulting database to the host. We used to have a problem with memory consumption in this case too, but judicious removal of modules that didn't _need_ to be installed on every new site spin up and some removal of config the distro doesn't need to control helped. For solutions, I can only think of a more configurable installer that does smaller sections of the site installation at a time so the requests are shorter, which I think would avoid timeouts / memory issues.

We also have a bit of difficulty with testing pre-release versions of the profile on client sites and coordinating feature testing on those sites for work in progress, then getting everything together to release, but I don't see how that can be helped at the Drupal level, thats an org issue.

Overall the experience feels good, additional tooling around config management and an installer than can handle bigger module/config loads on lower end hardware would be my two wish list items.

One more just came up during sprint retro: Demo content

We have a pretty solid system for reading YAML files and inserting them into our database during test runs, and I know that there is a general concept for demo content in Drupal, but the process is quite slow and manual to create demo content. I am sure many members of our product/management team would love to be able to scaffold out a bunch of entities and click an export button or something like it to dump all that content to fixture files that we could include with new site spin ups (similar to Umami).

We're going to probably end up re-purposing our test seeder code / fixtures to solve this problem, but worth thinking about a non-dev friendly way to maintain that kind of thing.

The largest one right now is being locked into a distribution/install profile. I don't want my site install to know anything about what was used to create it, and I want my root composer.json file to dictate the module versions and dependencies. Recipes should provide 'add on' or 'enhancements' of a particular flavor for you and be able to be removed once used.

When requiring a Recipe, its dependencies should be pulled into your main composer.json file so you can update / change them out at your discretion. Once the recipe is used in an installation, I can't see a reason to have to 'use it' again.

@dustinleblanc

For the config files, just use https://www.drupal.org/project/config_devel → . There is event to customize the config files for special use cases. Example to keep uuid in views, add enforced dependency to configs. https://git.drupalcode.org/project/openculturas/-/blob/1.3.x/profile/mod...

The bridge between sure you can customize thing and updates from the distro. We use atm https://www.drupal.org/project/update_helper → , so that updates works, when nothing was changes, but also does not break when something was changed.

New feature. Module or feature flag? Enable it by default or on demand? Or depends? :D

Hi
Thanks to @froboy for sending me link to this issue, I was looking at it almost a week before writing here.

Above my comment there are good points we all must consider converting to action items otherwise no sense to start a topic like this one.

I'm a System Architect of YMCA Website Services distribution ( formerly known as Open Y )
Our distribution is large, with more than 500 actively developed modules, half of those - contribs, the other half - created by our community of partners.

The major issue we are struggling with is a lack of leadership and responsibility.
Let me explain
We have experience developing and maintaining distribution since Drupal Core 8.0 rc... We now on Drupal 10 and this is only a start.

And the majority of the slowness we had from the Drupal.org community itself. ( Spoiler: I and my colleagues from other agencies had to adopt ~100 modules on Drupal.org as maintainers in order for the distribution to move faster and evolve accordingly to the YMCA community demand )

Major pain points:

- unhealthy dependency within the maintainers' community from tests - I believe this is due to the Core team having this as a requirement and everybody borrowing this as a best practice, but reality says - modules are usually blocked in the pace of development due to tests being coming before code is merged and released as alpha/beta/rc. As a suggestion - tests are great, but site builders' code accessibility should go before the tests are available.

- the bureaucracy of code ownership. Getting maintainer access rights is usually PITA, especially with trust - the majority of access is obtained after weeks of communication - just code push, no access to release management or GitLab admin options. My personal vision - if someone has resources to maintain - we should award these people as a first priority and support them as much as we can. Even for large modules I always see a lack of resources. We are locking our contributions to a few people in a community. We should become more open mined and trust the community more, especially new people. Even if some release is broken because of that - site maintainers are smart enough to hold with an upgrade if something is not clear and weigh in.

- we should stop the patch upload process and force everyone to start using Merge Requests. Enough for drupalism. It's one step - disable diff and patch extensions at drupal.org. This is a drupalism I'm personally shame about.

- we need the ability to assign issues to people that are not within the module's maintainer's list

- we need to improve the dashboard of issues across all projects one is the maintainer of - going through the list of issues per project is a controbuting factor people are abandoning Drupal.org in favor of other platforms, like GitHub

- Distributions or in the words of Drupal - profiles - are a great example of Drupal being integrated into large systems, in my personal opinion should be promoted as the second most important thing about Drupal after the Drupal core itself.

I have other, smaller things to discuss, but It's too much already to digest, so I'd rather stop.

Thank you

Thanks for all your input and helping shape the Recipes initiative's direction!

Automatically closed - issue fixed for 2 weeks with no activity.