🇺🇸United States @drumm

I now have a script to migrate sections of book hierarchy. My process will be:

• Complete deletions and anything else pending in the issue summary.
• Starting from deeper in the book hierarchy, like https://www.drupal.org/node/779080 → , do any final cleanups and migrate. An example final cleanup for a small section would be flattening child page content into a single parent page, which I have also automated.

I got together a quick script to migrate book hierarchies and have migrated:

• https://www.drupal.org/drupalorg/docs/archive-of-past-events →
• https://www.drupal.org/community/contributor-guide/reference-information... →

$page = node_load(33887);
$destination_guide = node_load(28580);
$test_mode = TRUE;
//$test_mode = FALSE;

if ($destination_guide->type !== 'guide') {
  print 'Not a guide!';
  return;
}

$queue = book_menu_subtree_data($page->book);
$nids = [];
$mlid_nid = [];
while ($item = array_shift($queue)) {
  if (empty($mlid_nid)) {
    $mlid_nid[$item['link']['plid']] = $destination_guide->nid;
  }
  $mlid_nid[$item['link']['mlid']] = $item['link']['nid'];
  $queue += $item['below'];
  $nids[$item['link']['nid']] = [
    'parent' => $mlid_nid[$item['link']['plid']],
    'type' => empty($item['below']) ? 'documentation' : 'guide',
  ];
}

foreach (node_load_multiple(array_keys($nids)) as $node) {
  print $node->title . ' (' . $node->nid . ') → ' . $nids[$node->nid]['parent'] . PHP_EOL;
  if ($node->type !== 'book') {
    print 'SKIPPING ' . $node->type . PHP_EOL;
    // TODO if this was to be a guide, bad things will happen!
    continue;
  }

  $node->type = $nids[$node->nid]['type'];
  $node->language = 'en';

  // Remove the page from book schema.
  if (!$test_mode) {
    book_node_delete($node);
  }
  $node->book = [];

  $node->menu = [
    'link_title' => $node->title,
    'menu_name' => 'menu-og-' . $nids[$node->nid]['parent'],
    'parent' => 'menu-og-' . $nids[$node->nid]['parent'] . ':0',
    'plid' => 0,
    'enabled' => 1,
  ];

  // Do not create alias during intermediate save.
  $node->path = ['pathauto' => FALSE];

  // Create a revision for the migration.
  $node->revision = TRUE;
  $node->log = t('Migrating to new documentation system.');

  $node_wrapper = entity_metadata_wrapper('node', $node);

  // Summary is a new field for documentation. Lets auto-fill it
  $node_wrapper->field_summary->set(truncate_utf8(strip_tags($node->body[LANGUAGE_NONE][0]['value']), 140, TRUE, FALSE));
  if (!$test_mode) {
    $node_wrapper->save();
  }

  // Use normal path aliasing.
  $node->path = ['pathauto' => TRUE];

  // Do not create a revision for the second save.
  $node->revision = FALSE;

  // Empty out summary attached to body field.
  $node->body[LANGUAGE_NONE][0]['summary'] = '';
  // Open comments.
  $node->comment = COMMENT_NODE_OPEN;

  // Set the group reference for the node. After the initial save so field
  // changes have kicked in.
  $node_wrapper->og_group_ref_documentation->set(intval($nids[$node->nid]['parent']));
  if ($node->type === 'guide') {
    $node_wrapper->{OG_GROUP_FIELD}->set(TRUE);
    $node->og_menu = TRUE;
    $node_wrapper->field_new_page_and_guide_review->set(1);
  }
  if (!$test_mode) {
    $node_wrapper->save();
  }
}

I don’t think I’ve seen the Views UI do permissions for fields, and “is maintainer of the project” would be a few hoops to jump through. Certainly would be doable with enough custom code, but I’d like to avoid that here. And hopefully non-maintainers might help here too.

https://www.drupal.org/list-changes/drupal/drafts → could get a filter for published and draft, and be renamed to something like “manage”

Bulk update complete!

Thanks, this update is starting to run

Add something like we have on s.d.o, which clearly mention not to commit anything and to read the policy. I think this is beneficial and the link is also very important. So I suggest to add:

Added in https://git.drupalcode.org/project/drupalorg/-/commit/00580c709adb36c0c7..., I made the language a little more direct.

On release creation, we could offer to link the release to drafted advisories, updating that advisory’s “Fixed in” field. That’s probably the way to go. If the advisory exists, that does have a link to the specific security issue.

Done in https://git.drupalcode.org/project/drupalorg/-/commit/0b2ff674daf7908716... & https://git.drupalcode.org/project/drupalorg/-/commit/47f37bb9d44f61f67f.... This also posts to the advisory’s issue when a Fixed in release is added to the advisory.

Remaining work here is now potentially adjusting the “Release window open” message and/or timing. Details are in comment #2 & #5.

This will update 4,681 issues, which lines up with https://www.drupal.org/project/issues/drupal?text=&status=Open&prioritie... → . The over-count is unpublished issues, which will also be updated.

The test issue for review is #6463: Add static caching for user_roles() → . I’ve confirmed it did have draft credit assigned, which was cleared.

This needs core maintainer approval before I start the bulk update.

Can we update the new message to include also the starting time (so to inform they, that they can start commiting from 16.00 UTC on Tuesdays)? If this message will be added on 8:30 UTC on Tuesdays, there is still a gap until the commit window for maintainers opens.

Adjusting what time of day the daily messages are posted is an option too. The current time is somewhat arbitrary. My thinking so far has been that its better to be a bit early, and its better to have a maintainer make the release a few hours early than them running late.

On release creation, we could offer to link the release to drafted advisories, updating that advisory’s “Fixed in” field. That’s probably the way to go. If the advisory exists, that does have a link to the specific security issue.

automate this somehow and add a comment to the gitlab issue, when a private release is created

That’s doable, but there isn’t anything linking the release to a specific security issue. So we’d have to comment on every open security issue for the project. That’s probably not a problem, although might be annoying if there are many open security issues for the project. In any case, implementation would have to be in packaging, so the many API requests to find and comment on the issues wouldn’t delay a web request.

We might want to go ahead and post a comment for every release, including non-security, so that we can spot active maintainers who might have missed the security issue, and other unexpected workflows.

With ✨ Add details about project & branches to security issues Active , along with posting the initial comment, the issue summary is updated to include a link to the project on Drupal.org, and a link to search for the string it inserts. I’ve updated the issue summaries for all open issues to match, so we have a good idea of how this works, for example: https://git.drupalcode.org/search?group_id=183118&scope=issues&search=%2...

If we want to add or change anything, this is implemented at https://git.drupalcode.org/project/drupalorg/-/blob/863c5fa0aedff993b59a...

I think we can call this fixed, leaving open at least for a couple days for any feedback on the wording. We’ll be mostly locked into what we choose now.

A couple of implementation details:

• The advanced search this links to has a lot more capability than https://git.drupalcode.org/groups/security/-/issues. The main issue search does not support quotes for exact matches
• All issue searches do not index the comments. Only the issue title & description are searched. (There is advanced search for comments, but that returns a listing of comments, without full context about the issue.)

This would be implemented as something very Drupal.org-specific, there isn’t a great way to make this sort of thing generic.

There are also other dependencies which a project might have. Instead of making a wall of dependencies which are tedious to think through, I think its best to just use composer, it will always have the most accurate view of the dependencies and your site.

7.x-3.x is good for now, but this might not be implemented until we're on 1.x, unless this is particularly urgent.

Since issues are moving to GitLab, we don’t have a local DB of issues, and whether they are open or closed. So we can’t make a simple View for this.

For draft change records, we have https://www.drupal.org/list-changes/drupal/drafts → which pulls in the issue status in the front-end, so closed issues can be spotted by scanning through the list. We could add the same column to https://www.drupal.org/list-changes/drupal →

Looks good to me, I'm thinking this can be merged any time and tag along with other deployments?

This is working well now: https://git.drupalcode.org/security/16-api-security/-/issues/1

📌 Replace “Recommended by the project’s maintainer” with stability indicator on project pages Active has some discussion:

I didn’t want the non-stable color to be too much of a warning. While we do hope production sites can run as many stable releases as possible, we also want maintainers to have some visibility for testing pre-releases.

📌 Manually test TUF-enabled Composer projects Active is where we’ve been getting some real-world testing

I think the memory usage looks a blocker: https://github.com/php-tuf/composer-integration/issues/127

Otherwise, we haven’t seen production issues in the last few weeks. When we do see them, they are hard to debug: https://github.com/php-tuf/composer-integration/issues/128

packages.drupal.org is TUF-enabled. For core, the drupal/* & php-tuf/* namespaces, there is packagist-signed.drupalcode.org to be added as another Composer repository. I’m guessing we might want a separate issue for adding the repository before packages.drupal.org

poker10 → credited drumm → .

I’m not sure if this should be a requirement or not, but I have heard of contributing websites that are intranets or otherwise not accessible from the public internet. That would make any localize.drupal.org → contributing site communication impossible.

In addition to a core improvement, we can definitely get the URL redirecting to an updated documentation page, since it is linked to in the Drupal admin UI

The project has been deleted.

Projects are generally not deleted from Drupal.org. This ensures that any code people are using remains available, and is no releases are replaced with a different codebase.

Since https://www.drupal.org/project/forumpaytest → was created in the last couple of days, and has no code, it can be deleted in this case.

We’ll handle any browsing requirements at 📌 [Meta] Plan for recipe Active . For now, we just need the data as soon as practically can be done.

(Off-topic - I took a quick survey of Drupal.org’s modern sites, the only reason we install ctools is pathauto, https://www.drupal.org/project/pathauto/releases/8.x-1.11 → it’s general popularity should decrease, but that will take years.)

Packagist.org only ships what’s in the repository for a project, see https://github.com/composer/packagist/issues/903

To get more in line with “standard” PHP overall, I expect we’ll want to do less packaging, not more. Even the current minimal packaging can cause annoyances 📌 Packaging info from .info.yml often creates conflicts when patching (ddo) Active

Done, https://packagist.org/packages/drupal/daisy_cms is now available

I did a quick query to see what the last 1,000 published forum posts use:

   1 Drupal 10.x,Drupal 9.x,Drupal 7.x
   1 Drupal 4.5.x or older
   1 Drupal 4.6.x
   1 Drupal 6.x
   1 Drupal 7.x,Drupal 8.x,Drupal 10.x
   2 Drupal 10.x,Drupal 9.x,Drupal 8.x
   2 Drupal 7.x,Drupal 9.x
   3 Drupal 10.x,Drupal 7.x
   3 Drupal 11.x,Drupal 10.x,Drupal 9.x
   3 Drupal 11.x,Drupal 10.x,Drupal 9.x,Drupal 8.x
   3 Drupal 8.x,Drupal 9.x,Drupal 10.x
   3 Drupal 9.x,Drupal 10.x
   4 Drupal 10.x,Drupal 9.x
  11 Drupal 11.x,Drupal 10.x
  18 Drupal 8.x
  45 Drupal 7.x
  74 Drupal 9.x
  84 NULL
 133 Drupal 11.x
 607 Drupal 10.x

All but 84 (8.4%) actually do select Drupal versions. So it is used more than I expected.

With Drupal’s semantic versioning, the Drupal version is indeed a lot less significant today.

The same taxonomy vocabulary is also used on case studies, so we’ll want to consider the impact there as well.

We can order the options in any way, so Drupal CMS could be at the top.

Merging older versions would require a bulk update of old forum posts & case studies to the updated term. This would effectively lose the original data. Not necessarily a bad thing, just noting that this would take some work.

Thinking about this another way - is the Drupal version field still useful on forum posts? Removing the field is an option. This would also effectively be data loss.

Going to leave this open for building the browsing UI on new.drupal.org. That will need to be backed by search API for full text search, so there might be additional steps to getting all the requirements ready.

The work in D7 is done, maintainers with a release that is a recipe can select categories when they edit their project. (We are getting away from “project types” whenever we can, projects change types between releases.)

We are building the recipe listing in new.drupal.org, so that part does not have to be built twice. The next step is ✨ Migrate new project_general fields for recipe browsing Active

drumm → created an issue.

Thanks for reporting, fixed

Those changes are now deployed. Please follow up to let me know if there was any change, or not.

I haven’t been able to reproduce this, but I did spot some problematic text wrapping which generally upset the layout here. Fixing that may help here.

Yep, they are release nodes, I was on autopilot with copy/paste. Runs okay with the changes applied locally for me.

Good catch, added those.

I think the only remaining work here and in child issues is for me to automate bulk moving the remaining 5,806 book pages.

Of course, if there is any remaining that is outdated or duplicitous, and can be deleted, please call that out and that can be done.

Thanks, that actually has deleted everything except the archive of past events. There’s some discussion of that at 📌 Migrate documentation into the new system Active , it would be good to keep that for posterity. I’ll handle that as an automated migration before closing out that issue. There is too much structure and too many pages to make manually migrating worth the time and effort, when it can be automated.

This was migrated

This was removed with 📌 Identify Migration archive page for delete or needs migration Active

Thanks for the work here! I’ve started going through the deletions, triple-checking and completing them.

I’ve deleted the remaining pages. Thanks!

I agree everything in this book can be deleted, and that’s now done.

I don’t think this is relevant now.

Projects have a documentation link field which can go anywhere, since things like documentation in GitLab pages are effectively off-site.

Documentation on Drupal.org has related content that can include projects, or any other content.

I don’t think so. It looks like a key part of the plan was to add a field to issues. That is not doable with GitLab, which does not have arbitrary fielded data.

@drumm super! I think the only description in the sidebar that needs an update is Ceremony/awards. How about "a special event to celebrate a major Drupal release or the accomplishments of our community members."

Thanks, done!

I’m thinking the same module categories would be good to have for consistency in browsing projects, whether there is a unified module & recipe search, or switching between the two modes.

Those look like projects that didn’t migrate for me locally for whatever reason, I didn’t pay attention to the console output there, but there was some.

So the plugin: migration_lookup section might not be right for the compound primary key in this table. We might not care since production should be in a better state.

This mostly works now, but this error appears 281 times

 [error]  All the id fields are required for a table migration. (/var/www/html/web/modules/contrib/migrate_plus/src/Plugin/migrate/destination/Table.php:180)
 [notice] Processed 64153 items (63872 created, 0 updated, 281 failed, 0 ignored) - done with 'drupalorg_migrate_project_release_supported_versions'

@pameeela - Is there any part of 📌 Decide on naming convention on recipes Active that could move forward with this? For example, a way for recipe maintainers to select “Site starter” or “Site feature”?

I see categories in the issue summary, does this mean they have the same https://www.drupal.org/docs/develop/managing-a-drupalorg-theme-module-or... → categories as modules?

The immediate next step will be adding a hidden field on general projects in D7 which rolls up the project’s releases’ composer types into a multivalued text field in anticipation of conditional field(s) for projects that have releases that are recipes. I initially thought this would be done only for search indexing, but we’ll likely want this data always available on general project nodes without rummaging through all releases.

We’re tentatively planning on taking a little extra time to build the actual browsing pages on new.drupal.org from the start, rather than building it twice.

Which web browser are you using?

✨ Provide file upload for forums Active is sufficient for tracking this issue, we do not want duplicates for the same site being upgraded.

retry-after: 10 is now in the response headers.

Deployed!

We recently enabled this new-to-us feature from a service provider, in response to API requests sent at an abusive rate. It looks like this is something we can customize to correct.

Our initial setting for the rate limit was somewhat arbitrary. Do you have a sense of how far below your previous crawl rate this kicked in at?

This is all deployed and I’ve updated the block on https://www.drupal.org/community/events →

Thanks!

This is now deployed

I’ve done the initial updates to the fields. I’ll plan to make the same changes to the “Events for all!” block on the right side of https://www.drupal.org/community/events → . Any updates for the copyediting there would be welcome.

drumm → made their first commit to this issue’s fork.

I’ll go ahead and deploy what I have so far in the issue summary, followups can always change the text later. In particular, https://www.drupal.org/docs/develop/managing-a-drupalorg-theme-module-or... → could be linked to when it has some updates to be more comprehensive and align with core’s current release cycles.

Crediting people who helped in 📌 False warnings about security updates Active

Updating for 📌 Remove “recommended” status for releases? Active

Remove note about feature branches which is no longer true.

drumm → created an issue.

- The recipe directory name—which is essentially what the recipe name comes down to—has only a loose connection with the project on d.org. I believe we should only send an application update if the recipe includes a composer.json file, so we can verify that it belongs to the drupal/ namespace as outlined in one of the conditions in the issue summary. This would filter out all custom recipes added at the project level (i.e., not available on d.org even if the recipe name matches a general project on d.org), as well as recipes under other vendor namespaces.

That sounds correct to me.

- This also means that all core recipes would never be tracked. Are we okay with that, or should we support core recipes too? If so, we’d need to collect version information from the drupal/core package.

Yes, we do want core recipe information, so that will need to be a special case.

Cleaning up the draft in the issue summary

The issue summary looks all good to me. Are there any open questions or likely revisions?

In general, removing options would be tough, since we will have to migrate the existing content to not use those values before we remove the options. Right now there are no removals requested in the issue summary.

We decided against using this module, since we remembered we have a way to embed blocks on pages built with paragraphs.

However, this module is popular and I’m sure a stable release would be generally appreciated.

Are there any stable blockers now? Blockers for an RC release?

The links are counting exactly what they link to. I think the request is to update the links to link to the issue listing filtered by only open issues, not all issues.

I agree this is not relevant since we publish advisories directly for the API composer audit uses

The issue summary mentions including the update status site key, which is used to de-dupe update status data per-site.

We will have to decide on a new algorithm for translating the recipe application numbers into popularity for ranking. Something along the lines of including the last N weeks, potentially with some decay function so the last week contributes more to the rank, and each earlier week contributes less. Until we have some real data, it isn’t worth much speculation about the specific implementation. The earlier we have data, the better.

Additional metadata that might help is welcome, as long as it doesn’t send identifying information about the site, or delay initial implementation in core. Adding method of installation to the query string with the request would be useful - installed via installer, project browser, drush, as a dependency, etc.

This is now deployed!

A flag isn’t needed, we already have data on what’s a recipe since ✨ Add composer release type field Active

Any listing will not have a chance to be good until ✨ Add tracking for when recipes from Drupal.org are applied Active moves forward. We can only sort by data we have, like alphabetical or last updated date.

drumm → created an issue.

This is now deployed

I’ve added a draft to #686918: Add help text to project release admin page to warn about marking branches unsupported and the impact on update status → for adding help text, please help review and improve it.

It looks like the conclusion of this issue was to add help text and documentation, so I think the other issue will cover making this improvement.

Making the title more straightforward 

Adding a draft to the issue summary for review and updating.

It looks like https://www.drupal.org/docs/develop/managing-a-drupalorg-theme-module-or... → might be a good page to link to for a detailed explanation, although that page could use an update for Drupal core’s current release schedule.

Updating for 📌 Remove “recommended” status for releases? Active

drumm → made their first commit to this issue’s fork.

No changes here, I think the behavior is likely different if you are logged in, which could help get it right in some cases.

I fully expect we’ll fix this as a side effect of upgrading to modern Drupal. Effort would be better spent in getting the community events themed with the new brand and landing page refreshed.

Now is a good time to do this since 📌 Remove “recommended” status for releases? Active did get deployed.

This is now all deployed.

Sorry for the abruptness of this change. This is something I’ve been mentioning in other issues, but should have had its own issue earlier. The timing is actually prompted by ✨ Support confidential issues in git.drupalcode.org for security team triage Active , it would be good for the new.drupal.org automations to post an automated comment including what versions of the project with a potential security issue are currently supported. So we need to migrate this data to the new site. Before migrating, we’re simplifying so there is less to migrate and less UI to port to modern Drupal when we get there.

This issue will remove the option to designate releases as “recommended” once it is complete. It will be the last thing to go, once everything that depends on it is removed. That will simplify the page and will be a good opportunity to follow up with #686918: Add help text to project release admin page to warn about marking branches unsupported and the impact on update status → in the near future.

People should see the higher version number, that’s a stable release, as the recommended version. “Recommended” was never communicated to Composer, or update status. So it was only ever shown on project page, many people might not have seen it for some time.

lifecycle: in .info.yml files may be a good way to communicate deprecation of old versions: https://www.drupal.org/docs/develop/creating-modules/let-drupal-know-abo... →

https://www.drupal.org/project/drupal → is now updated and drupalorg module no longer uses the project_release_supported_versions.recommended column.

The project is now a module and Composer install is now available.

The project was created as a distribution instead of module. I will convert it to a module.

Keeping this issue open for

The remaining use of the “recommended” filter is on https://www.drupal.org/project/drupal → . That will add the other two supported releases, currently 11.0.11 and 10.3.12.

I did go ahead and replace the mild grey for non-stable releases with the mildest grey. This should improve contrast, and reduce the highlighting of prereleases.

Whenever we do small design changes, we’ll be incorporating the new branding elements. Hopefully things don’t look too mismatched as they gradually change, but it's a lot more doable than attempting to get everything right all at once.

And yes, the Recommended option for maintainers will be removed once everything that depends on it is removed. This should happen in within the next week.

The remaining use of the “recommended” filter is on https://www.drupal.org/project/drupal → . That will add the other two supported releases, currently 11.0.11 and 10.3.12.

I didn’t want the non-stable color to be too much of a warning. While we do hope production sites can run as many stable releases as possible, we also want maintainers to have some visibility for testing pre-releases.

The text alternative is whether the release is alpha/beta/rc. I decided against additional labeling, since the releases are already very busy.

The initial deployment here is done, leaving open for https://www.drupal.org/project/drupal → which is still showing all “recommended” releases instead of supported.

Notes for deployment:

• cc views
• TRUNCATE cache_project_release_download_table;

This should be ready to deploy now.

The dev site build completed.

eowg-drupal.dev.devdrupal.org is building out and should be available in 30 minutes

Done, you should be able to ssh heatherwoz@wwwdev1.drupalsystems.org