Please go ahead and make an initial commit of the module code. We normally do not do project type changes preemptively.
There was a request for updated data in Slack, it is attached.
More notes for myself for next time:
Get the current core components:
SET group_concat_max_len = 10000000000; SELECT group_concat(DISTINCT pcc.name SEPARATOR '\\|') FROM project_composer_component pcc INNER JOIN field_data_field_release_project fdf_rp ON fdf_rp.entity_id = pcc.release_nid AND fdf_rp.field_release_project_target_id = 3060\G
Filter the usage data:
grep '\sdrupal|.*\([the core components]\)$' < /var/log/updatestats/counts/[most recent].submodule_release_counts > drupal-component-usage-by-release-[today].txt
I guess tackling this challenge will also be relevant, even after moving issues to GitLab?
Yes, this will still be relevant, regardless of where issues are.
The best place I can think of putting this is maybe the “Administer releases” page, and/or something on each release page. Drupal.org’s DB does not have the default branch, so any specific messaging would need a front-end JS request to gather data, or have the very generic “Remember to match the default GitLab branch with the latest [or active?] dev release.”
As we migrate issues to GitLab, there will no longer be the opportunity to add button styling, or add arbitrary text to the issue UI. So we somewhat do want to make sure people are looking for the link without making everything bigger.
We will be able to post automated comments, like the “Now that this issue is closed…” comment you see on this issue, and we’ll also post an introductory comment. That will be styled like any other comment in GitLab, so we can start thinking about copyediting improvements that could help. Updating the automated comment as credit is updated would be a stretch goal.
🌱 [PP-1] Create components for a default design system Postponed is one more test update, including the version field update.
I’ll assume the copyediting has been reviewed enough and move forward with updating the 539 issues.
I’ve removed the block from node/*/edit pages with block visibility settings.
I finally decided on a wording and this is deployed. With 📌 Move issue credit listings to new.drupal.org Active split off into a separate issue, I think this is now fixed. Please reopen if I missed anything.
I opened 📌 Improve logging in locale_translation_download_source() Active to get more detail in the “Unable to download translation file…” log message
I opened
📌
Improve logging in locale_translation_download_source()
Active
to get more detail in the “Unable to download translation file…” log message. This issue looks like it is focusing on better handling than TypeError: fgets(). I think it makes sense to tackle both issues separately, to keep each issue more straightforward.
Unable to download translation file https://ftp.drupal.org/files/translations/all/restui/restui-8.x-1.22.fr.po
This error comes from locale_translation_download_source() and might be HTTP connectivity or issues with saving the file locally. Unfortunately, there is not enough detail in this log to know which issue it is. And if it is indeed an HTTP issue, it doesn’t have enough detail to start tracking it down.
I’d like to see the specific exception type, message, and HTTP status code & response if it is an HTTP error.
TypeError: fgets(): Argument #1 ($stream) must be of type resource, bool given in fgets()
is a side-effect of of the first error. 🐛 PoStreamReader::readLine() throws an error on module install Needs work has some work toward improving this error handling.
I haven’t been able to find any potential issues with serving these files for Drupal or Drush. Since the error may be with saving the file to the downloaded translations, I’d start with finding where it is writing and triple checking the permissions, making sure the user account running drush write access.
Something caused the migrated project maintainership data to be cleared. I’ve re-migrated the data, and the integrity checks look clean enough. So this is resolved, for now.
Leaving open since we need to do more log analysis & checking over things to figure out the root cause, make any necessary improvements, and get the integrity checking automated so we have good alerting for any future inconsistencies.
Thanks for the good bug report! This was also reported at 🐛 Issue credit controls doesn't follow permissions Active
We had another report of this that included a URL, so it was easy to track down. We do currently have a bad migration of project maintainers.
Are you logged into new.drupal.org? Its impossible to tell with the cropped image and no URL provided.
As part of simplifying Drupal.org and letting GitLab handle things it is good at, we no longer have commit listings on Drupal.org.
We removed our merging UI with ✨ New contribution records system Active
We removed our merging UI with ✨ New contribution records system Active
With ✨ New contribution records system Active , we no longer have per-comment attributions.
With ✨ New contribution records system Active , this no longer happens. There will just be the one-time notification.
Drupal.org no longer keeps commit-related information in its local DB, so this is not practical to implement. GitLab provides APIs on git.drupalcode.org. https://docs.gitlab.com/api/
This UI has moved with ✨ New contribution records system Active , and does explicitly mention that it is for copy/paste.
Hopefully this is no longer happening, especially with the new credit system. Of course, reopen with details if it does.
With ✨ New contribution records system Active , we no longer record attribution for each comment.
With #3039289: Make issue credit assignment comments visible → , crediting is no longer deeply integrated with updating issues.
This no longer happens, as part of wider changes to not suggest credit.
This part of the UI is now removed, we no longer track attribution per-comment after ✨ New contribution records system Active
The post-migration cleanups in D7 are now in place, removing access to credit-related fields. So the outdated data is no longer accessible, this is the only API. I think we can say this is fixed as there is documentation in place.
We’re no longer automatically suggesting credit, and this UI has moved with ✨ New contribution records system Active . Marking this fixed since I believe I did put a fix in place at the time.
With ✨ New contribution records system Active , we are no longer recording attribution per-comment, so this UI has been removed.
This has been deployed for awhile and the immediate followups & cleanups have been triaged & fixed. 🎉
Any future followups should be their own issues.
https://www.drupal.org/node/2373279/issue-credits → is paged, showing the first 50 issues, with 21 more pages. So everything is consistent as far as I can tell. This is all counting just the last 12 months, which is what currently matters for rankings and program eligibility.
Is there something I might be missing or an approximate number you expect to see?
Re 500 errors - right now the issue credit listings are especially fragile, since the D7 site makes HTTP requests to the new site as part of its own page rendering. We have lowered the timeout when making the Guzzle requests in D7, so that should limit the 500 errors, but if we missed something, or we’re making 4 requests that use most of their allowed 10s, that will 500. The setup is indeed fragile. Rather than sinking time into making this work better, we should get the Views moved: 📌 Move issue credit listings to new.drupal.org Active
In the cleanup at https://git.drupalcode.org/project/drupalorg/-/commit/54fdf4bd59067dbf47..., I brought back creating the hidden comments which should allow the names to transfer over. There’s one more followup at https://git.drupalcode.org/project/drupalorg/-/commit/ab02815e40efd30a56..., field_issue_credit should no longer be used anywhere.
This will get the names transferred over. Manually checking the boxes on the contribution record to actually grant the credit will be a necessary step. Leaving at needs review to double check this with the next advisory being published, and we might do a followup to automatically credit everyone for advisories.
This is actually the first time we’ll be sending email from new.drupal.org, and this has reminded us that we hadn’t set up outgoing email yet. That infrastructure setup is in progress.
I’m seeing 2 remaining edge cases:
- The “current” bit is sometimes mismatched between sites.
- I didn’t get job title in the diff, and I believe we will have issues there.
The short name for https://www.drupal.org/sandbox/eyalshalev/2767919 → has been reset, and you should be unblocked.
But how is possible for sandbox project to take a short name? For years I was convinced that only permanent/full projects are owning a a short name.
This should be true, but the enforcement is via a couple essentially multi-step forms implemented in D7 and earlier, so bugs have happened.
I like the more-lightweight format in #5. The spacing between lines could be reduced.
https://www.conventionalcommits.org/en/v1.0.0/ only specifies feat/fix or the ! appended. It looks like the list is up to each project to decide what makes sense for them. Is this set what makes sense for Drupal? How do we decide that? How do we decide when someone asks for another option.
I’ve packaged up the integrity checking as a shell script. Spot checking makes it look like a decent first pass.
The list of uids with a diff is:
sed -ne 's/^[+-][^\t]*\t\([0-9]*\)\t.*/\1/p' < user-organizations.diff | uniq
If that’s accurate, this currently affects 3,604 people.
And the initial version for the new site
SELECT ufd.name, ufd.uid, p_fon.field_organization_name_value AS organization_name, p_for.field_organization_reference_target_id AS organization_nid, p_fc.field_current_value AS current FROM users_field_data ufd LEFT JOIN user__field_user_organizations u_fuo ON u_fuo.entity_id = ufd.uid LEFT JOIN paragraph__field_organization_name p_fon ON p_fon.entity_id = u_fuo.field_user_organizations_target_id LEFT JOIN paragraph__field_organization_reference p_for ON p_for.entity_id = u_fuo.field_user_organizations_target_id LEFT JOIN paragraph__field_current p_fc ON p_fc.entity_id = u_fuo.field_user_organizations_target_id ORDER BY ufd.uid, organization_name;
For integrity checking, this is an initial query for a users at organizations manifest in D7. I has all users, so we get a username integrity check too. If we want to narrow the scope, some LEFT JOINs can be INNER
SELECT u.name, u.uid, fdf_on.field_organization_name_value AS organization_name, fdf_or.field_organization_reference_target_id AS organization_nid, fdf_c.field_current_value AS current FROM users u LEFT JOIN field_data_field_organizations fdf_o ON fdf_o.entity_id = u.uid LEFT JOIN field_data_field_organization_reference fdf_or ON fdf_or.entity_id = fdf_o.field_organizations_value LEFT JOIN field_data_field_current fdf_c ON fdf_c.entity_id = fdf_o.field_organizations_value LEFT JOIN field_data_field_organization_name fdf_on ON fdf_on.entity_id = fdf_o.field_organizations_value ORDER BY u.uid, organization_name;
This is now published. Likely there was some race condition which two versions of the node were saved at the same time.
I moved this to a fresh block implementation so we’re not relying on the old block being placed. This is deploying soon.
I recommend creating the 0.x-dev release for canvas before bulk updating issues. This will allow issues with the 0.x version to have that option available once moved.
https://www.drupal.org/project/canvas/issues/3439664 🌱 [later phase] Research Issue For Workspaces Extra Postponed is the test issue for final review before I start the bulk update. Note the version field would need to be updated on the next issue update. Assuming the version was 0.x-dev, adding the release will fix that.
Do we want to move closed issues too, or just open issues?
https://gitlab.com/gitlab-org/gitlab/-/issues/566636 did get at least one bugfix backport moving in GitLab. They did spot a typo in our configuration which was very likely to be the cause. I’ve corrected the typo and reconfigured, so I expect this to be resolved. Please reopen with details if this does happen again.
We’ve had some other reports that have potential to be the same issue. We need the line with the correlation_id from GIT_CURL_VERBOSE=1 git push to confirm it is the same issue, or track down something else. For example:
17:22:43.565250 http.c:662 <= Recv header: X-Gitlab-Meta: {"correlation_id":"01K3V6S15E9B305CFY6QCCRABF","version":"1"}
drumm → created an issue. See original summary → .
I couldn’t find a relevant existing issue, so I filed a new one. Since this is an exception internal to GitLab, we’ll want to wait for their fix or workaround.
That does lead to a good backtrace in GitLab
"exception.class" : "NoMethodError",
"exception.message" : "undefined method `include?' for nil:NilClass",in
"lib/gitlab/auth/ip_rate_limiter.rb:37:in `block in trusted_ip?'",
"lib/gitlab/auth/ip_rate_limiter.rb:37:in `any?'",
"lib/gitlab/auth/ip_rate_limiter.rb:37:in `trusted_ip?'",
"lib/gitlab/auth/ip_rate_limiter.rb:43:in `skip_rate_limit?'",
"lib/gitlab/auth/ip_rate_limiter.rb:31:in `banned?'",
"lib/gitlab/auth.rb:122:in `find_for_git_client'",So it seems there’s some edge case with involving GitLab’s rate limiting implementation
That block ID does have the user agent you found. I believe I have found a way to allowlist this, with a rate limit.
Ideally it should have a different name in core. That would avoid the possibility of issues.
We do have some functionality for this on git.drupalcode.org, a dependency on any core component is interpreted as a dependency on core. (Composer does not have a concept of components, sub-themes, sub-modules, those are Drupalisms.) So this may “just work,” although someone running a version of core without gin may need to explicitly require gin if they are starting by requiring something that depends on gin.
We have not wanted to get into the business of providing support for a container registry. While this might come “for free” with GitLab, there would be bandwidth costs and user support.
For our CI templates, we do use a 3rd party - GitLab.com: https://gitlab.com/drupal-infrastructure/drupalci/drupalci-environments
https://unix.stackexchange.com/questions/758893/ssh-connection-stop-at-d... has a variety of potential causes for hanging after SSH2_MSG_KEXINIT sent
Since this is happening as the SSH key exchange starts, the SSH auth log is likely to be the most useful on our end. I can search those by public key fingerprint, which is in your logs. I only see successes from a single IP:
Aug 28 16:26:44 gitlab1-aws sshd[2556359]: Accepted publickey for git from 47.xxx.xxx.xxx port 49461 ssh2: RSA SHA256:3xjfQwRbu3AxFq1N63QIkLmrYalgwWBb6iBw7np7Fss
That IP must be when you are not using zscalar. That’s consistent with your logs when using zscalar not getting to the phase where the public key is offered. That’s the first log entry for any given connection, so your connection may not be getting as far as something that might be logged on our end. https://config.zscaler.com/zscaler.net/hubs is too many IP ranges for me to be able to practically search for.
We aren’t doing anything to specifically block zscalar. Does zscaler support offer any help?
I’m unable to reproduce this issue. Without an original error message, I can not attempt to correlate this with any logs.
Does your workplace have any proxies or firewalls that might be affecting the HTTP Auth headers? Can you try from another network?
If the issue persists on other networks, please paste the full git command you are using, and full output? And prefix the git command like GIT_CURL_VERBOSE=1 git push …. Be sure to redact any credentials, but do leave the exact remote URI. More information is at https://docs.gitlab.com/topics/git/troubleshooting_git/
Bulk-move issues? (I think that's what this issue is about, but I've not been privy to any conversations, and this issue summary doesn't really say that — it might just as well mean that we're supposed to move 'em manually?)
That’s totally possible and what this issue is planning.
Bulk-move issue forks/MRs? We currently have 219 open MRs: https://git.drupalcode.org/project/experience_builder/-/merge_requests 😅
That’s not quite practical to do. Each issue that has code worked on after moving will want a fork of the new project. Assuming the whole commit history is moved, you can add both issue forks as remotes and merge/rebase code around. Theoretically we could automate making the equivalent forks and push branches around, but that would be a bit of effort to implement. And then when https://git.drupalcode.org/project/experience_builder/-/merge_requests/1483 lands, Git will likely be a wall of merge conflicts, so manual work, or a more-sophisticated automation will be needed.
Would be possible to rename canvas → canvas_old and experience_builder → canvas?
No. We do not rename projects. Existing releases and repositories are guaranteed to be indefinitely available, so we aren’t breaking people’s site builds. Renaming would be a lot more complexity, not at all a simplification.
benjifisher → credited drumm → .
That’s now done.
Their Drupal.org account is associated with https://groups.drupal.org/user/115
https://groups.drupal.org/user/4263 has only posted one comment as far as I can tell. I should be able to change that comment to be authored by their correct account and delete the duplicate account.
The redirects would be best implemented in .htaccess, so we’re modifying that either way. That allows local testing in development.
We do have the option of redirecting at the CDN, which does have slightly nicer syntax. It would still be ideal to keep this logic along with the site’s deployment.
The main thing I’ll want here is the copyediting for the comment that’s posted. Something along the lines of:
Drupal Canvas is the new name for Experience builder. See more details at [issue or other link]
Done!
The webform project page uses filtered HTML. As far as I know, it has never used full HTML.
The security risk would be that someone with full HTML access, or someone using their compromised account, posts a malicious <script> tag or other exploit, whether from social engineering, or without their knowledge as a step in a larger combined exploit.
Drupal core’s .htaccess will indeed have to be modified.
For the Drupal.org sites on Drupal 7, we had a restriction that only the root index.php could be routed to the PHP interpreter. It looks like the current For security reasons, deny access to other PHP files on public sites. section should do a good job of this. We should double check that it works, so that rogue PHP files are never run via an HTTP request.
Then we can look at adjusting the FilesMatch directive. Likely splitting it into a separate regex(es) that still deny \.php$ but allow something like api/[^/]*/search/.*\.php$
This relies on a webhook underneath. HTTP requests never have a 100% delivery rate, and in this case we have not over-engineered something to get closer to guaranteeing delivery. Another update to the merge request would have gotten a webhook that was delivered.
This is exactly what general projects were created for.
When releases of the project have a composer type, like drupal-recipe, we add functionality around that. We could do the same for other non-PHP project metadata if there is a need for it. Projects change their “type” between releases often enough that the “project type” concept breaks down.
We’ll need a way to locate the potential traffic in logs to begin investigation. Is there a way to identify zscalar traffic? Does it come from a specific IP or IP range? You can email help@drupal.org if you do not want to disclose IPs on this public issue.
Does GIT_SSH_COMMAND="ssh -vvv" git clone git@git.drupal.org:project/dkan_dataset_archiver.git offer any help on what part is hanging?
Since https://www.drupal.org/project/paragraphs_schedule → has code, it can not be deleted. I recommend updating the project page to mention that it is unsupported & obsolete.
Do you have a block ID from the response that Wallabag gets? Or can you confirm what user agent Wallabag uses? I need somewhere specific to start to see the nature of the blocking.
Submodules are tricky for Composer since there isn’t a concept of them in Composer, and the meta packages are indeed a hack.
Generating version ranges for when a submodule was part of a module would likely end up being a hard problem. The submodule might be removed and re-added later.
Unless there are extenuating circumstances, we leave account deletion for each person to do themselves.
On http://drupal.org/user-edit click “Update your username, email address, first and last name, or set up two-factor authentication,” then click “Delete account”
Tables, code blocks, and embedded images are all allowed in the standard filtered HTML text format.
Project pages and documentation should never be full HTML, since co-maintainers without access would not be able to edit their own project pages, and would remove the community editing of documentation.
Full HTML is used very sparingly since it allows creating security and privacy issues. It is rarely granted, since it is rarely necessary and comes with many downsides.
We’ll have to decide on redirect & delete OR keeping the pages for the history, with a notice at the top directing to the new version.
Redirects without deleting make the old pages mostly inaccessible, so they might as well be deleted. If we go this route, we need a list of URLs of pages to delete and their replacement URL.
If we do keep the old pages for their history, a notice could be added above the text like we have for Drupal 7 documentation, https://www.drupal.org/docs/7/install/before-installation → . And/or a specific message with the exact replacement page can be edited into each documentation page. If we go this route, we need the text for the message.
There isn’t a great way to control edits to documentation pages, community editing is generally a feature.
I’m somewhat active, for topics that relate to Drupal.org infrastructure.
I’ve added the category with the description
Provides AI-powered functionality, integrate with machine learning services, or enable AI-based features such as content generation, natural language processing, recommendation engines, computer vision, or artificially-intelligent automation.
Since these are now project categories, which recipes may use too.
Testing for Drupal core now pretty much fully within core itself. (If a custom container is needed, there is https://www.drupal.org/project/drupalci_environments → , although well-supported 3rd-party containers are preferred.) If this can land in a way that contributed projects could take advantage of, there is https://www.drupal.org/project/gitlab_templates →
(Updating to resolve Drupal.org issue indexing issue, please disregard)
Issues like https://gitlab.com/gitlab-org/gitlab/-/issues/377750 need to be resolved in GitLab first, since GitLab’s edit profile page has a mix of data that is canonically set from www.drupal.org, and should not get out of sync.
Since this has not been a priority for GitLab, if there is enough demand for this, we would have to implement this UI on www.drupal.org.
Until then, we err on the side of privacy for everyone.
A big concern I have with the credit system is maintainers needing to spend time doing crediting paperwork. The commit trailers are not used in any crediting on Drupal.org, they are only for people who read the commit history. I know core maintainers already take their crediting task seriously and spend time granting crediting as fairly as possible.
Even if we can do a decent job of automating authored/reviewed/reported/etc-by, are the extra choices for maintainers to add more detail just to a commit message worth their time? Maintainers who do want to add the extra detail can edit the commit message after copying it.
I think I generally like this idea. This has a good chance of being doable without additional API calls, and increased page load time, since we’re already loading commits to help maintainers make crediting decisions. If this does take more API calls, then we should look if we need to consider other options.
And this does let the contributor choose what they want per-issue, as long as they’re making a code contribution.
Clarifying the proposed resolution to
Use email address as included in the most recent commit, from any MR if there are multiple
The most recent commit has the best chance of being what the contributor wants, and can be amended with an easier force push. Most issues won’t have multiple MRs. For ones that do, I don’t think it practically matters too much which one wins; readable, efficient JS can be the priority over extra logic around prioritizing/choosing multiple MRs.
I also considered removing the “Credit all” checkbox entirely. We don’t have it in the old UI, and I’m not sure we want maintainers to be crediting everyone too often, it could give low-quality contributions more chances of receiving more credit than deserved. However, the current form structure has some scaffolding that’s helped by the API and “credit all” comes with it for free.
I requested this behavior originally - if the sort order changed as HTTP requests complete, that’d swap out what’s under the maintainer’s cursor, and lead to accidental crediting.
The other option would be to hide the entire table until all requests complete, or a timeout is reached. We decided it would be best to use the behavior that was implemented. We can always do more followups in separate issues if something better seems practical.