New contribution records system

Update: functionality-wise, everything is there. We're starting to review/improve and see if there is anything missing.

The D7 changes can be seen in the MR for this issue. The whole plan (D7 + D9 new system) is in the google doc linked in the issue description.

fjgarlin → changed the visibility of the branch 7.x-3.x to hidden.

fjgarlin → closed merge request !147

The new MR contains the code from the old one as it was 2 years behind and rebase didn't work straightaway.

I will need to deploy this again to a D7 site and test.

Updated deployment instructions.

Updated deployment commands.

Updated deployment instructions.

Updated deployment information, queues to set up, etc.

fjgarlin → changed the visibility of the branch 3327584-contribution-records-new-system to hidden.

fjgarlin → closed merge request !273

With such a complex MR, it was easier to redo the whole thing than rebasing. New MR: https://git.drupalcode.org/project/drupalorg/-/merge_requests/332

I will try to split the MR into two:
- One for pushing data to D10
- One for reading that data from D10

fjgarlin → closed merge request !332

First part: Send data to new.drupal.org:
- MR: https://git.drupalcode.org/project/drupalorg/-/merge_requests/337

- Settings on D7:
-- Set drupalorg_token to the same value as the one set in D10
-- Setup queue drupalorg_issue_events

- Settings on D10:
-- Allow 'contribution_records' in drupalorg_allowed_content_types
-- Allow /contribution-record* paths on Fastly
-- Setup environment variable so this line works: $config['drupalorg.settings']['token'] = getenv('DRUPALORG_TOKEN');
-- Setup queue contribution_records_import_queue_worker

Simplified deployment instructions.

Added a configurable way to throttle the requests to the new system, in case we need it.

The second MR https://git.drupalcode.org/project/drupalorg/-/merge_requests/337 is mostly ready, but it relies on MR336 being merged first together with the deployment steps for that part.

After that, and when this is merged, we should be able to switch back and forth between the "legacy" and "modern" credit system with the following commands:

Use modern system:

drush vset drupalorg_credit_system 'modern'
drush cc views

Switch back to legacy system:

drush vset drupalorg_credit_system 'legacy'
drush cc views

A third part, and once the new system is stable and in use, will be to clean up the code from the legacy system and remove the switch, but we will go one step at a time.

Updated the IS as we updated some of the token and headers names on both D7 and D10.

SA nodes can now be sent to the new site to create the corresponding Contribution Credit.

MR: https://git.drupalcode.org/project/drupalorg/-/merge_requests/350/diffs
Once deployed:
- drush drupalorg-contribution-records-sync --node-type=sa --raw-import=2
- drush queue-run drupalorg_issue_events (This might not be needed as it is the same queue as with the previous step)

fjgarlin → closed merge request !336

After the first full import run, we've had around ~1% missing on the new site. The reason for most of these (if not all) was that the issue didn't even have a comment, not even the automated comment #1 which is always linked to the author.

We will send the author with no attribution (as none can be calculated) when there are no comments on the issue. If the author makes a comment with attribution, then this last one will be taken instead.

MR to fix this situation: https://git.drupalcode.org/project/drupalorg/-/merge_requests/353/diffs

After that, we will need to re-run some of the above commands with selected NIDs.

Edge case for when there are no contributors: https://git.drupalcode.org/project/drupalorg/-/merge_requests/354/diffs

More edge cases:
- Issues with no title
- Issues with no comments and the users who created them no longer have an account

MR for that: https://git.drupalcode.org/project/drupalorg/-/merge_requests/355/diffs

fjgarlin → changed the visibility of the branch 3327584-empty-credits-count-fix to hidden.

fjgarlin → changed the visibility of the branch 3327584-send-contrib-records-to-d10 to hidden.

fjgarlin → changed the visibility of the branch 3327584-send-sa-to-d10 to hidden.

fjgarlin → changed the visibility of the branch 3327584-send-deleted-accounts-records to hidden.

fjgarlin → changed the visibility of the branch 3327584-always-send-author-as-credit to hidden.

fjgarlin → changed the visibility of the branch 3327584-always-include-orgs-in-api-request to hidden.

fjgarlin → closed merge request !361

The first phase is almost complete.

We have been running data integrity checks between the D7 system and the new system to make sure that all records and all attributions are set correctly.

The next phase means merging https://git.drupalcode.org/project/drupalorg/-/merge_requests/351/diffs. That will leave things as they are now but we will be able to switch between the current D7 system or the new system just by setting a flag. I've updated the issue description with the steps needed for this.

fjgarlin → changed the visibility of the branch 3327584-expose-author-uid-even-if-blocked to hidden.

The above merge is huge help for me going forwards. There will be more smaller MRs but won't need to do huge rebases and we won't need to re-review thousands of lines of code.

fjgarlin → changed the visibility of the branch 3327584-read-credits-new-system-part-2 to hidden.

fjgarlin → changed the visibility of the branch 3327584-read-credits-new-system-part-2 to active.

fjgarlin → changed the visibility of the branch 3327584-read-contrib-records-from-new-system to hidden.

fjgarlin → changed the visibility of the branch 3327584-read-credits-new-system-part-2 to hidden.

fjgarlin → changed the visibility of the branch 3327584-comment-update-queue-event to hidden.

MR367 merged.

The data integrity phase is now done. We have https://git.drupalcode.org/project/contribution_records/-/blob/1.0.x/scr... to compare D7 credits vs new system credits.

This is a HUGE achievement as there were more than 5 million records to migrate here. Over the last two weeks we were fixing edge cases in the code and data quality/consistency on D7.

We are now focusing on query performance so we can start querying the new system for reading credits.

fjgarlin → changed the visibility of the branch add-project-breadcrumb to hidden.

fjgarlin → changed the visibility of the branch 3327584-debug-org-rank to hidden.

We missed something really small in an if condition in the last MR. This one fixes it: https://git.drupalcode.org/project/drupalorg/-/merge_requests/370/diffs

fjgarlin → changed the visibility of the branch 3327584-small-fixes to hidden.

fjgarlin → changed the visibility of the branch 3327584-include-sa-param to hidden.