Problem/Motivation
Contributing to Drupal as a first timer can be difficult. There are several points of friction in account creation, and several 'Drupalisms' in the order of operations we require to start an issue and then begin your code contribution - even after the migration to GitLab for code hosting and the hybrid d.o issue/GitLab Merge Request model.
Project Goals
Significantly increase the amount of contribution to Drupal, particularly by opening up contribution opportunities to new audiences who may have found Drupal contribution intimidating or overwhelming in the past.
For a more detailed overview - review the video from DrupalCon: https://drive.google.com/file/d/1dqpx9CTYla9e3M1NpRIubbAs1ScbvnTg/view?u...
Strategy / Proposed Resolution
Reduce the friction of contribution to Drupal, particularly for brand-new contributors and so-called 'drive-by' contributors.
Increase our ability to capture those contributors and make it easier for them to continue investing in the Drupal project.
Proposed resolution: Targets
There are five key areas of friction in Drupal contribution today that we want to address with this initiative:
#1 - Discovery/Drupal contribution entry points
- There are multiple potential entry points where a potential contributor might find Drupal when they decide they want to contribute for the first time, including:
- Packagist
- GitHub Mirror
- Drupal GitLab Instance
- Drupal.org itself
- Maybe others
- Solution: In-place guidance on each of these third-party entry points, directing users to the correct place to begin their contribution journey - preferably *without* directing them to a complex Drupal-specific process.
- This might include 'welcome bots' set up to automatically redirect folks to the right places if they get off on the wrong foot.
- Scope: 🟢 Relatively straightforward - we can audit these entry points, see what sort of control we have over welcome messages/bots - and implement those measures, non-controversially.
#2 - Account management and authentication
- Most contributors already have a GitHub or GitLab account - and are used to being able to find a new project they want to contribute to and open a merge request within 5 minutes or less. Because Drupal requires a separate account sign-up process with a few verification steps for terms of service, git access, etc - we may have new users who 'abandon the contribution cart' because of the extra account management friction.
- Solution:
- Develop an Authentication/Single sign-on solution allowing new and existing contributors to use their existing identities to create an account and log in with a single click (or as close to it as we can achieve).
- Consolidate terms of service acceptance and confirmation steps as much as possible, to reduce extra steps that might block contribution.
- Scope: 🟡 Straightforward, but significant work to do. Even with bringing in a vendor, allowing third party login to create d.o accounts, as well as allowing D.O to act as an identity provider will require a rework of our current identity management, as well as a user experience review of the account creation, confirmation, terms acceptance, git access, and login processes.
- Status: 🟢 Must schedule migration. We have engaged Cloud-IAM as the hosted KeyCloak solution for our project, and worked out both the D7 and D10 integrations, as well as user migration scripts. We are in the process of performing test migrations, code reviews, and preparing to schedule the production migraiton.
#3 - Issue Migration (Code-first vs Issue-first workflow)
- In Drupal, all contributions must *start* with the creation of a Drupal.org issue. This is a pattern we deliberately enforced, wanting to ensure there was one canonical location where all communication about an issue took place and trying to encourage discussion of the best solution prior to code being written.
- However, this is not the common pattern in modern open source software development - most projects and most tooling providers they use (GitHub/GitLab) allow users to start with *either* an issue, or *directly by forking the code and opening a merge request*
- Problem: If we do use GitLab issues and merge requests instead of Drupal.org hosted issues we lose the structured metadata built into Drupal.org issues. We will likely have to replace these features with 'Scoped Labels' and other GitLab native issue management tools.
- Problem: GitLab has a much more limited permissions schema - Guest, Reporter, Developer, Maintainer, Owner - and we cannot customize the permission options available within each. We will have to think carefully about what permissions we grant, so that Drupal's more collaborative workflow can still take place, instead of certain activities being bottlenecked to only maintainers.
- Solution:
- Allow contributors to start either with issues or with merge requests, possibly including
- automatically generating an issue when merge requests are opened
- adding merge requests to existing issues (esp. issues already containing RTBC patches, or even multiple merge requests on issues with diverging proposed solutions)
- Implement scoped labels for meta data
- Investigate solutions to the permission/role model (perhaps: https://gitlab.com/groups/gitlab-org/-/epics/4168 )
- Scope: 🔴 Extremely large
- This is the element of the project that requires close collaboration with both Core and Contributed module maintainers to map existing contribution user stories to new workflows under the GitLab model.
- It also involves carefully determining the correct order of operations in enabling new features and triaging user feedback about the changes.
- This also requires the 'Contribution Recognition' in GitLab problem to be solved, one of the elements that is not in our direct control.'
- This is waiting for the GitLab team to prioritize the feature request: https://gitlab.com/gitlab-org/gitlab/-/issues/327138
- Mitigation: If GitLab rejects the feature request, or can't get it done in a reasonable timeframe, we will have to build a new drupal.org feature to create 'credit logs' that a bot can post to GitLab issues when complete, asking people to fill out to collect the credit info.
- Finally - this phase is the precursor for most of the work of phases 4 and 5, though some work can be done in parallel.
- Status: 🔴 Blocked - This is blocked on SSO deployment, item #2 above. However, we have solved the significant challenges posed above, in the following ways:
- 🟢 - We have a migration script that translates issue meta data into GitLab labels.
- 🟢 - We have a D10 credit content type awaiting deployment
- 🟢 - We have a migration script to populate issue credits into the new credit content type
- 🟢 - We have a bot for posting to GitLab issues with credit information and fork management
- 🟡 - We have sufficient permissions to work around some of GitLab's limitations with process, though we still hope they expand their permissions model.
🟢 COMPLETE
- DrupalCI's test results are posted directly to the issue queue, with a summary of passes and fails, and the ability to drill deeper into the tests to get console output and artifacts where required.
- DrupalCI also provides a matrix of test config
- Solution:
- As we move into GitLab issues we can convert DrupalCI to run using GitLab pipelines, which both give maintainers more direct control and should make it easier for contributors to see the status of their tests in progress.
- Caveat: Core tests still take a *really* long time, and that won't change just because we use DrupalCI as a test runner
- Solution: Core will need to work to optimize the speed of the tests being run by the CI system.
- Caveat: We currently centrally control the DrupalCI system, the available tests a project can choose to configure, and the resources spent on testing. Granting maintainers direct access to CI/Pipeline config for their projects could have a cost impact on the Drupal Association, or could result in projects not being properly tested for core compatibility.
- Scope: 🟡 Straightforward, but lots to do and test to avoid disruption
- Portions of this work can be completed prior to issue migration, for example- simply replacing the current Jenkins test orchestration layer with a GitLab pipeline layer
- We can also (and have already started to) allow maintainers control over GitLabCI and pipelines
- However, managing the test configuration options for testing Core, and for testing Contrib compatibility with Core will likely be very large tasks
- Finally, ensuring that the appropriate CI results messages appear, and/or that helpful bot messages are automatically posted, will require the issue migration portion of the project to be complete.
- Status: 🟢 Complete
#5 - Onboarding and automated guidance
- Drupal often offers a fairly overwhelming number of options in the user interface, and while we have exceptionally helpful materials in the form of the new Contributor Guide - that guidance doesn't appear *in context* as a user is attempting to make their contribution.
- This leaves contributors with unanswered questions:
- Where's the right place to start contributing?
- Did I follow the right process?
- When will I see the test results?
- Who will review my changes? And when?
- Solution: Use automation tools provided by GitLab to configure some more helpful onboarding bots - and to manage automatically assigned reviewers for certain portions of the Drupal codebase:
- Risk: As of today *any core contributor* is considered a sufficient reviewer to take an issue to 'RTBC' and only *then* is a core maintainer usually pulled into the process. Switching to 'assigned reviewers' is very likely to pull in core maintainers much earlier to the issue review process than they would normally.
- Mitigation: We could create a new class of 'core reviewers' - expanding the capacity of the maintainer.
- Scope: 🟡 Straightforward - if we can identify *who* owns and manages these welcome messages and workflows
- Once issues are migrated, we can grant access to either existing core maintainers, or new maintainers ('Welcome committee working group?') to configure these bot messages and automatic reviewer assignments.
- If it's determined that the *DA* should directly control and implement the welcome messages and workflows, we'll need the appropriate authority to do so in collab with core.
- Status: Blocked - 🔴 This is blocked on issue migration deployment, item #3 above. We can do preliminary work on the plan for the Drupal.org account registration and automated email portion, but that is item #1, this is for in-issue guidance.
Risks
Disrupting the core release cycle
If we do not carefully map out all of the user stories of the core release cycle, and have each element accounted for when we transition portions of the tooling to GitLab we risk disrupting the release cycle, and potentially even missing one of Drupal's 6-month release windows.
- Involve release managers heavily in defining the user stories and validating that the new GitLab based workflows accomplish their needs
- Choosing a deploy time shortly after a particular 6-month release, to give as much of the following 6 months as possible to troubleshoot and file off rough edges.
Disrupting the security release cycle
The Drupal Security team has a fairly bespoke process for taking in reported security issues, adding maintainers of the relevant projects to those issues, and working in an appropriately confidential way. Those tools are all Drupalisms based on existing Drupal.org tooling - and the ability to manage security reports and make security releases could be disrupted - potentially making a highly critical security release difficult to execute.
- Involve the security team heavily in defining the user stories and validating new workflows for the team
- As best as possible, try to choose a time to deploy changes with a *minor* security issue as the next release on the docket - so any issues can be ironed out not with a critical or greater vulnerability on the line.
Loss of Contribution Credit for issues
One of the most innovative and powerful tools the Drupal Association has built to foster contribution in the community is the Contribution Recognition system in the issue queues.
This attribution data is used not only to measure how much of Drupal development is volunteer vs. sponsored but also to determine which organizations contribute the most to the Drupal project.
This is key data in managing additional Drupal Association programs, such as the marketplace ranking system and the Drupal Certified Partner program. Without it, we lose our ability to reward 'makers' over 'takers' and thus reward good corporate citizenship in the Drupal.
In every case the canonical home of the contribution record will remain Drupal.org. We will parse the attribution data - import into Drupal.org user profiles, organizational profiles, and marketplace. This gives us the control we need over the weighting and measurement of that data, and allows project leadership to create the appropriate incentives for organizational contribution.
What will differ is whether that attribution data itself is stored directly in GitLab (if they add the feature described in Option 1 below) or if we have to create a separate integration to gather attribution data (see Option 2 below).
- Option 1 - we have pushed for GitLab itself to adopt a similar attribution system in their toolset. If this feature is added, we can continue to gather this attribution data and use it the way we do today - but furthermore, we can also begin to compare against other GitLab-hosted open source projects.
- Note: It might be possible for us to attempt to contribute such a feature upstream, but as of yet, we are still trying to get buy-in from the GitLab project managers - although we have managed to gather support from other projects and initiatives (LinuxFoudnation CHAOSS, CivCRM, and several GitLab enterprise customers).
However - we do not have Ruby development talent on staff - so directly contributing upstream would be a challenge (and would counteract any argument that we have 'saved' engineering time by moving to a 'vanilla' GitLab solution).
- Option 2 - As a backup - if GitLab cannot deliver the attribution system in a timely fashion - we could develop a 'contribution log' content type on Drupal.org, automatically create one for every GitLab issue/Merge Request - and use a bot to post a link to the credit node on each closed issue.
Loss of *desirable* drupalisms: easy cross-collaboration
In attempting to standardize as much as possible on 'vanilla' GitLab workflows, we are eliminating Drupalisms from our workflows, and some of those Drupalisms are actually superior in terms of the ability of a large community to collaborate on single issues, than what is offered by these alternate tools.
For example - right now:
- Anyone can open an issue on any project
- Anyone can comment on an issue for any project
- Anyone can create a patch or merge request on an issue for any project
- Anyone can 'review' that work, and update the issue status to RTBC
These are not a given in other toolsets. In both GitHub and GitLab - maintainers have to *grant* access to issues on their projects in order for these contributions to be made - and control is given to the original *reporter* of the merge request, rather than to the maintainers of the project.
Constraints on Scope
The accelerated move to additional GitLab features does *not* mean that we want to move absolutely everything to GitLab. We should understand the constraint on the scope.
Issues => GitLab
Per above, tactics 3-5 require that issues move from Drupal.org to GitLab issues, which will also bring with it new capabilities such as GitLabCI integration, Kanban boards, Scoped Labels, Bot integration, etc.
Project Pages: Discoverability => Drupal.org
The homepage for a given project should remain on Drupal.org, and indeed, everything related to the discoverability of projects really remains a core Drupal.org feature. Drupal.org home pages provide us additional metadata and descriptive information from maintainers that simply aren't available on GitLab project pages, and are required to support other initiatives, such as the Project Browser in Core initiative.
User Documentation (Drupal.org) vs Developer Documentation (GitLab)
Drupal.org has a robust documentation system that interlinks between projects and documentation sections that don't have to be tied to a particular project making it an ideal place for everything to do with end-user(or end-user/developer) documentation. That is, everything about how to install and use a particular project with Drupal should remain on Drupal.org.
At the same time, GitLab's documentation features - particularly the readme.md, wikis, and GitLab Pages, are ideally suited for developer documentation.
Release management => Drupal.org
This may need further exploration, but at least initially it seems prudent that packaging and release management continue to take place on Drupal.org.
At the very least, this will be required to support the Drupal.org Composer Façade as it currently exists, as well as to support the package signing mechanism being developed for the Automatic Updates initiative.
As a follow-up phase, we could revisit release management using GitLab.
It should be noted that there are a few areas where GitLab *should* begin taking over some elements of release management, such as for JavaScript libraries in the general project type, where the Decoupled Menus Initiative is already using GitLab pipelines to publish to NPM.
Remaining tasks
- Drupal Association completing contract for Authentication/SSO from third parties
- Auditing contribution workflow user stories from contrib, core, and security perspectives
- what else?
User interface changes
API changes
Data model changes