We can continue putting redirects in by hand. Where should the two pages you linked go to?
We do keep records of deleted nodes, but now is not a good time to engineer new functionality as we’re migrating the site. That is something we could build once documentation is fully migrated, if the search for the previous page title is tested and shown to be useful.
This is all done, tested, and I’ve updated https://www.drupal.org/drupal-security-team/security-team-procedures/sec... →
Updating link for https://www.drupal.org/project/drupalorg/issues/3528485 📌 Move GitLab security issue triage project to hide issue counts Active
Yes, the labels filter is hiding in the left sidebar, at the bottom.
I did a quick pass through the orphaned book pages ( https://www.drupal.org/node/3522465 → ) and deleted a couple of pages for themes that didn’t have D7 or later versions.
The can be deleted but I don't have edit accees, the bitcahce project is unsupported.
And those have been deleted too, thanks!
What about the pages for modules that do not have a stable D7 release?
I think we have to keep those for now, there will be plenty of people with D7 sites that they need to figure out, and documentation has potential to help.
benjifisher → credited drumm → .
Even if this feature was present in GitLab, it is always best to keep patches locally
MR.patch URLs should never be used by composer patches. The upcoming 2.x version https://github.com/cweagans/composer-patches has some protections built-in, including https://github.com/cweagans/composer-patches/issues/347. The best way to help is to look into blockers for the 2.0.0 release of composer-patches, and help move those forward. Until then, you might consider setting up CI to flag improper composer-patches use in your projects.
Even if this were a GitLab feature, local copies of patches still eliminate the possibility of the remote URL becoming unavailable, changing, or causing other chaos. And you have a definite record of what your codebase was patched with. For example, we may need to clean up stale forks for closed issues in the future, that could affect the MRs from those forks. Indefinite MR patch hosting is not a service we can promise to provide.
Done and https://packagist.org/packages/drupal/settingsphp will be available in a few minutes.
The missing statuses are filled out now:
- Reviewed & tested by the community → Security status::reviewed & tested
- Ready for SA to be Published → Security status::ready to be published
- Postponed → Security status::postponed
- Closed (duplicate) → Closed::duplicate
- Closed (won't fix) → Closed::invalid, potentially might be other statuses
And what I mentioned in #5 is done by https://git.drupalcode.org/project/drupalorg/-/commit/109256299611a258d8... & https://git.drupalcode.org/security/triage/-/commit/cc6e081e6ab08d16f49b...
So I think everything we’ve thought of has been followed up on.
This was deployed and organization rank has recalculated.
Reported via Slack
This is now running in staging for verification.
Closed (duplicate) - We can link another issues in Gitlab, but there are only three options: relates to, blocks, is blocked by. If it will be sufficient to mark it as "relates to" and close it with a comment that it is a duplicate, then we probably do not need this label.
Closed (won't fix) - Not sure how to label such issues. Probably need to add this?
We should add both of these, I imagine they will be useful for retrospectives.
Reviewed & tested by the community - I think that we discussed to use milestones to schedule releases. So adding an issue to a milestone can be probably considered as RTBC?
I think we do want to add a label, that can have an automated response to tell everyone the next steps. That will be a bit better UI, labels are easier to remember and look for, and there is sometimes negotiation around which Wednesday a maintainer wants, or the team coordinating with something else. We could have the automation follow up if a ready issue is not assigned a milestone within a certain amount of time.
Ready for SA to be Published - We used this status when maintainers actually committed the fix and created release. Can this be automated somehow, so that we are notified about new private releases in the Gitlab issue? If so, then web probably do not need this?
📌 Update automated messages in Security Team's Gitlab Active did automate this, https://git.drupalcode.org/project/drupalorg/-/commit/47f37bb9 is the message added. It does not currently add a label. It was not working for the one_time_password issues since the advisories weren’t drafted starting from the GitLab issue, so there was no link from the advisory to the issue.
Ready for SA to be published also needs the advisory to be drafted. I think what I’ll add is:
- When a release is made, also add a label that the security release is done
- Add a “Ready to publish” label (trying to keep labels as concise as they can be)
- When an issue has both release done & advisory drafted, automatically add ready to publish. There isn’t a security team member has approved the SA draft state, but I think that’s fine to leave loose, as it will get a final review on security release Wednesday
We can add additional daily notifications leading up to the scheduled security release date, if either tag is missing. Those can be drafted in 📌 Update automated messages in Security Team's Gitlab Active or a separate followup
The create event permission should be restored now. There was a second role that I thought was redundant and outdated, but apparently it is the one that matters. And role syncing from www.drupal.org might not be working properly. So the site is indeed in bad condition and a liability to keep running.
https://new.drupal.org/browse/recipes is now available. ✨ Add tracking for when recipes from Drupal.org are applied Active must be taken on by someone if we want the page to be useful. Until that is done, this will be sorted alphabetically, and might not get much more prioritization.
This almost certainly is caused by OG menu module loading all menus for all documentation guides.
access to create new events on groups.drupal.org is no longer available.
This has been turned back on, not having email notifications for group members is a regression a few people contacted us about. And is not something we can practically implement on https://www.drupal.org/community/events/ → for now. We both don’t want to build new functionality on D7, where community events are now, and there isn’t the same group system for subscribing to a specific location/meetup.
benjifisher → credited drumm → .
Maybe instead of porting, we could just add the Event content type and functionality to d.o. main, then ?
That was done: https://www.drupal.org/community/events/ → and as of today, access to create new events on groups.drupal.org is no longer available.
We have also replaced Solr-powered search with Drupal core’s default search. We are migrating and upgrading Solr servers and can no longer support indexing groups since the compatibility issues are not worth trying to solve.
GitLab is a product and not customizable like Drupal is. This sort of UI change/layer isn’t something we could reasonably implement and maintain. This would be feature requests for https://gitlab.com/gitlab-org/gitlab/-/issues
Hover over a line of code inverts the line for easier readibility (also could be a preference)
This one might be more doable standalone issue for GitLab. There are already preferences around syntax highlighting and diff colors at https://git.drupalcode.org/-/profile/preferences. And there are already some hover effects on the line numbers for merge request diffs. I briefly searched for related issues and found https://gitlab.com/gitlab-org/gitlab/-/issues/300469
Somewhere on the top right corner you have a button to reveal comment threads: one button to show and unfurl the first and one button to unfurl all.
I briefly looked for an existing issue for this one too, I did not find one. There is a “Hide all comments” option in one of the three-dot menus, so it might be doable for GitLab to add “Show all comments” next to that.
As the Drupal community, there isn’t a lot more we can do to influence GitLab’s development other than opening good issues for their product. They do have good contributor onboarding, so its doable to contribute code to them, with the huge caveats that its a huge project and changes take care, and its mostly Ruby code.
There have been significant memory usage improvements which need review linked from https://github.com/php-tuf/composer-integration/issues/127. Those could help the runtime as well.
Most often I think I see the tag not being “on the branch.” The Git history can be completely arbitrary. Actually detecting this would require some walking through the commit graph since there is a valid edge case for some security releases:
- Development on the branch has moved on, there are many commits since the last tagged release. A security release is needed and the maintainer wants to make a release with minimal changes from the last tagged release.
- This effectively will be a branch made from the last tagged release or even a few commits after. (There doesn’t need to be a branch, tags don’t require branches.)
- So in this case we’d be looking at the history of the branch and tag, looking for a common ancestor that is “reasonable.” That will require some careful logic, every Git commit pair in a repository has a common ancestor, usually.
You can see the project is migrated at https://www.drupal.org/jsonapi/node/project_module?filter[field_project_... →
How project browser is setting up filters for querying might be filtering out the project for some reason.
This is all done now.
hestenet → credited drumm → .
Done!
I’ve started work on the next phase here - configuration updates to get general projects indexed.
The orphaned pages are now at https://www.drupal.org/node/3522465 →
I took a first pass through deleting placeholders, obvious D6 & earlier, and a couple terrible security ideas. 101 pages remain
This is now deployed so https://www.drupal.org/list-changes/drupal/drafts?field_change_record_st... → lists all published change records with their issue.
- Downloading drupal/core-project-message (11.x-dev 656efa0)
Failed to download drupal/core-project-message from dist: Maximum allowed download size reached. Content-length header indicates 10666 bytes. Allowed 10658 bytes
This is now fixed. I had put in logic that only signed each zip file once. The dev version zip archives have the Git commit ID in their name and change when a new commit is added. However, the target name drupal/core-project-message/11.9999999.9999999.9999999-dev isn’t changing and what is actually checked. The fix was https://gitlab.com/drupal-infrastructure/package-signing/packagist-signe... and https://gitlab.com/drupal-infrastructure/package-signing/packagist-signe...
benjifisher → credited drumm → .
The project is now a theme
We do not delete and recreate in this case. It can be changed to a theme.
- Downloading drupal/core-project-message (11.x-dev 656efa0)
Failed to download drupal/core-project-message from dist: Maximum allowed download size reached. Content-length header indicates 10666 bytes. Allowed 10658 bytesThis is a server-side issue. https://packagist-signed.drupalcode.org/metadata/bin_094-097.json says
"drupal/core-project-message/11.9999999.9999999.9999999-dev": {
"hashes": {
"sha256": "78163df67a5207b9b0ba0ede7ff2652396f5c66ee9d6ce5f137db24d4242bd6e"
},
"length": 10657
},But https://packagist-signed.drupalcode.org/dist/drupal/core-project-message... is a different file. It was last updated Feb 3, which is around when I was shoring things up, so it's hard to say if it was a systematic issue or debris from something that was fixed. I'll look at getting that re-signed.
TUF is signing a target named drupal/core-project-message/11.9999999.9999999.9999999-dev and presumably it has a previous version of that file. For packagist-signed, dev zipballs do change location with every commit, the commit hash is in the filename. I wonder if there’s some way to use this for better error messaging or working around issues.
Now that all books have been triaged in child issues, we are left with 123 orphaned book pages. There’s some gems like https://www.drupal.org/teapot → that should have a place to land, and others that can be deleted.
The comparison of modules is now at https://www.drupal.org/docs/7/extend/comparison-of-contributed-modules →
The remaining module documentation is now at https://www.drupal.org/docs/extending-drupal/contributed-modules/contrib... → where it can continue to be pruned, improved, moved, or left until the next major reorganization
Thanks everyone!
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Automated message: Contrib modules for building the site functionality guide deleted. Moving its content one level up in the documentation hierarchy.
Update post-migration 📌 Review and migrate "Site Building Guide" book Active
Distribution documentation is now at https://www.drupal.org/docs/getting-started/drupal-distributions/distrib... →
The pending deletions in the issue summary are now all done
The pages mentioned have already been deleted, and work is continuing in the parent issue, 📌 Review and migrate "Site Building Guide" book Active
Yes, I’ve merged as well and everything looks good
Yes, that’s easy to do.
This should be fixed now, but not closing until a couple merges go through without problems.
This is everything I have for this.
Unless I’m misremembering, this is not something we’ve ever done in the past for 5.x or 6.x. Moving forward with semantic versioning, it's not something that will come up again.
The bulk update process has no undo and requires a bit of attention to ensure it completes. It was never run frequently enough to justify spending the time to make it more self-sufficient. There are over 25 times more contrib 7.x issues than there were for core. So this would require at least a couple days of watching the process run.
Some contrib issues never were assigned to D8 releases and only existed in D7 branches. After EOL I went through my modules issue by issue and migrating several still relevant into the active release branch.
That is indeed the best practice. Every project’s maintainers are free to triage issues as they’d like, and we can’t generalize that closing all 7.x issues might be good for most projects.
I now have a script to migrate sections of book hierarchy. My process will be:
- Complete deletions and anything else pending in the issue summary.
- Starting from deeper in the book hierarchy, like https://www.drupal.org/node/779080 → , do any final cleanups and migrate. An example final cleanup for a small section would be flattening child page content into a single parent page, which I have also automated.
I got together a quick script to migrate book hierarchies and have migrated:
- https://www.drupal.org/drupalorg/docs/archive-of-past-events →
- https://www.drupal.org/community/contributor-guide/reference-information... →
$page = node_load(33887);
$destination_guide = node_load(28580);
$test_mode = TRUE;
//$test_mode = FALSE;
if ($destination_guide->type !== 'guide') {
print 'Not a guide!';
return;
}
$queue = book_menu_subtree_data($page->book);
$nids = [];
$mlid_nid = [];
while ($item = array_shift($queue)) {
if (empty($mlid_nid)) {
$mlid_nid[$item['link']['plid']] = $destination_guide->nid;
}
$mlid_nid[$item['link']['mlid']] = $item['link']['nid'];
$queue += $item['below'];
$nids[$item['link']['nid']] = [
'parent' => $mlid_nid[$item['link']['plid']],
'type' => empty($item['below']) ? 'documentation' : 'guide',
];
}
foreach (node_load_multiple(array_keys($nids)) as $node) {
print $node->title . ' (' . $node->nid . ') → ' . $nids[$node->nid]['parent'] . PHP_EOL;
if ($node->type !== 'book') {
print 'SKIPPING ' . $node->type . PHP_EOL;
// TODO if this was to be a guide, bad things will happen!
continue;
}
$node->type = $nids[$node->nid]['type'];
$node->language = 'en';
// Remove the page from book schema.
if (!$test_mode) {
book_node_delete($node);
}
$node->book = [];
$node->menu = [
'link_title' => $node->title,
'menu_name' => 'menu-og-' . $nids[$node->nid]['parent'],
'parent' => 'menu-og-' . $nids[$node->nid]['parent'] . ':0',
'plid' => 0,
'enabled' => 1,
];
// Do not create alias during intermediate save.
$node->path = ['pathauto' => FALSE];
// Create a revision for the migration.
$node->revision = TRUE;
$node->log = t('Migrating to new documentation system.');
$node_wrapper = entity_metadata_wrapper('node', $node);
// Summary is a new field for documentation. Lets auto-fill it
$node_wrapper->field_summary->set(truncate_utf8(strip_tags($node->body[LANGUAGE_NONE][0]['value']), 140, TRUE, FALSE));
if (!$test_mode) {
$node_wrapper->save();
}
// Use normal path aliasing.
$node->path = ['pathauto' => TRUE];
// Do not create a revision for the second save.
$node->revision = FALSE;
// Empty out summary attached to body field.
$node->body[LANGUAGE_NONE][0]['summary'] = '';
// Open comments.
$node->comment = COMMENT_NODE_OPEN;
// Set the group reference for the node. After the initial save so field
// changes have kicked in.
$node_wrapper->og_group_ref_documentation->set(intval($nids[$node->nid]['parent']));
if ($node->type === 'guide') {
$node_wrapper->{OG_GROUP_FIELD}->set(TRUE);
$node->og_menu = TRUE;
$node_wrapper->field_new_page_and_guide_review->set(1);
}
if (!$test_mode) {
$node_wrapper->save();
}
}I don’t think I’ve seen the Views UI do permissions for fields, and “is maintainer of the project” would be a few hoops to jump through. Certainly would be doable with enough custom code, but I’d like to avoid that here. And hopefully non-maintainers might help here too.
https://www.drupal.org/list-changes/drupal/drafts → could get a filter for published and draft, and be renamed to something like “manage”
Bulk update complete!
Thanks, this update is starting to run
Add something like we have on s.d.o, which clearly mention not to commit anything and to read the policy. I think this is beneficial and the link is also very important. So I suggest to add:
Added in https://git.drupalcode.org/project/drupalorg/-/commit/00580c709adb36c0c7..., I made the language a little more direct.
On release creation, we could offer to link the release to drafted advisories, updating that advisory’s “Fixed in” field. That’s probably the way to go. If the advisory exists, that does have a link to the specific security issue.
Done in https://git.drupalcode.org/project/drupalorg/-/commit/0b2ff674daf7908716... & https://git.drupalcode.org/project/drupalorg/-/commit/47f37bb9d44f61f67f.... This also posts to the advisory’s issue when a Fixed in release is added to the advisory.
Remaining work here is now potentially adjusting the “Release window open” message and/or timing. Details are in comment #2 & #5.
This will update 4,681 issues, which lines up with https://www.drupal.org/project/issues/drupal?text=&status=Open&prioritie... → . The over-count is unpublished issues, which will also be updated.
The test issue for review is #6463: Add static caching for user_roles() → . I’ve confirmed it did have draft credit assigned, which was cleared.
This needs core maintainer approval before I start the bulk update.
Can we update the new message to include also the starting time (so to inform they, that they can start commiting from 16.00 UTC on Tuesdays)? If this message will be added on 8:30 UTC on Tuesdays, there is still a gap until the commit window for maintainers opens.
Adjusting what time of day the daily messages are posted is an option too. The current time is somewhat arbitrary. My thinking so far has been that its better to be a bit early, and its better to have a maintainer make the release a few hours early than them running late.
On release creation, we could offer to link the release to drafted advisories, updating that advisory’s “Fixed in” field. That’s probably the way to go. If the advisory exists, that does have a link to the specific security issue.
automate this somehow and add a comment to the gitlab issue, when a private release is created
That’s doable, but there isn’t anything linking the release to a specific security issue. So we’d have to comment on every open security issue for the project. That’s probably not a problem, although might be annoying if there are many open security issues for the project. In any case, implementation would have to be in packaging, so the many API requests to find and comment on the issues wouldn’t delay a web request.
We might want to go ahead and post a comment for every release, including non-security, so that we can spot active maintainers who might have missed the security issue, and other unexpected workflows.
With ✨ Add details about project & branches to security issues Active , along with posting the initial comment, the issue summary is updated to include a link to the project on Drupal.org, and a link to search for the string it inserts. I’ve updated the issue summaries for all open issues to match, so we have a good idea of how this works, for example: https://git.drupalcode.org/search?group_id=183118&scope=issues&search=%2...
If we want to add or change anything, this is implemented at https://git.drupalcode.org/project/drupalorg/-/blob/863c5fa0aedff993b59a...
I think we can call this fixed, leaving open at least for a couple days for any feedback on the wording. We’ll be mostly locked into what we choose now.
A couple of implementation details:
- The advanced search this links to has a lot more capability than https://git.drupalcode.org/groups/security/-/issues. The main issue search does not support quotes for exact matches
- All issue searches do not index the comments. Only the issue title & description are searched. (There is advanced search for comments, but that returns a listing of comments, without full context about the issue.)
This would be implemented as something very Drupal.org-specific, there isn’t a great way to make this sort of thing generic.
There are also other dependencies which a project might have. Instead of making a wall of dependencies which are tedious to think through, I think its best to just use composer, it will always have the most accurate view of the dependencies and your site.
7.x-3.x is good for now, but this might not be implemented until we're on 1.x, unless this is particularly urgent.
Since issues are moving to GitLab, we don’t have a local DB of issues, and whether they are open or closed. So we can’t make a simple View for this.
For draft change records, we have https://www.drupal.org/list-changes/drupal/drafts → which pulls in the issue status in the front-end, so closed issues can be spotted by scanning through the list. We could add the same column to https://www.drupal.org/list-changes/drupal →
Looks good to me, I'm thinking this can be merged any time and tag along with other deployments?
This is working well now: https://git.drupalcode.org/security/16-api-security/-/issues/1
📌 Replace “Recommended by the project’s maintainer” with stability indicator on project pages Active has some discussion:
I didn’t want the non-stable color to be too much of a warning. While we do hope production sites can run as many stable releases as possible, we also want maintainers to have some visibility for testing pre-releases.
📌 Manually test TUF-enabled Composer projects Active is where we’ve been getting some real-world testing
I think the memory usage looks a blocker: https://github.com/php-tuf/composer-integration/issues/127
Otherwise, we haven’t seen production issues in the last few weeks. When we do see them, they are hard to debug: https://github.com/php-tuf/composer-integration/issues/128
packages.drupal.org is TUF-enabled. For core, the drupal/* & php-tuf/* namespaces, there is packagist-signed.drupalcode.org to be added as another Composer repository. I’m guessing we might want a separate issue for adding the repository before packages.drupal.org
I’m not sure if this should be a requirement or not, but I have heard of contributing websites that are intranets or otherwise not accessible from the public internet. That would make any localize.drupal.org → contributing site communication impossible.
In addition to a core improvement, we can definitely get the URL redirecting to an updated documentation page, since it is linked to in the Drupal admin UI