Exclude test files from release packages

Created on 15 July 2019, about 6 years ago
Updated 4 July 2025, about 1 month ago

Problem/Motivation

  1. Reduce disk space on local server. Automated tests greatly increase the size of a bare Drupal installation on disk. (As of 609912bced (8.8.x, July 15, 2019), they accounted for 43M of 94M (46%).) For many sites, especially simple ones, they provide no benefit to end users in production. The number and size of files can also be a problem for users of shared hosting (c.f. #2338671: Create a smaller Drupal core download, removing tests, for production sites. )
  2. Test files can contribute to security vulnerabilities. See comments #21/#24/.
  3. Including test files increases download times and the load on drupal.org.
  4. Having a smaller code base may improve Development experience.

It's common to apply patches even on live sites. Patches often contain changes to test code and would by default fail if it is missing (see #33. Therefore as a dependency of this issue, we need an enhancement to the patching mechanism to allow applying patches when test code is missing - see this issue.

Obviously some sites specifically want to include the tests, so there must be a mechanism for that as simply as possible. This issue should try not to make this scenario worse or use more resources than in the past. In particular note that if we force sites to 'prefer source' to get the test files then this is much worse for the load on Drupal.org servers, see #22.

We need to carefully consider what counts as test code. One commenter (#37) noted that they would be happy to lose the specific core module tests but they would need the testing framework. We could consider including the framework even when excluding tests, depending on what portion of the code size it accounts for and how many sites we judge are in this category.

The primary target of this issue is Drupal Core. However we should ensure that whatever we do is generic so that any contrib projects with lots of tests can copy the same approach.

Proposed resolution 1

Use the approach documented in Use gitattributes to keep your tests out of other people’s production to exclude all tests/ directories from release packages. Since the Drupal.org packaging system already uses git-archive to create releases , the change would involve nothing more than the addition of a few lines to our .gitattributes file and would take effect immediately. Note: This would have no effect on Git clones, so neither the testbot nor core contributors would be affected.

In this scenario, anyone who wants tests uses prefer-source. We need to test and document exactly how to do this targetting just one composer project. A downside is that downloading source is considerably slower and will adversely affect Drupal.org servers. If this is estimated to be a major problem then a possible mitigation would be to produce a second composer project something like drupal/core-dev that replaces drupal/core and includes test files. However this becomes a non-trivial change to Drupal.org infrastructure and there might be further side-effects e.g. the extra storage for two sets of releases.

Alternative resolution 2

Remove test files after download using the vendor hardening Composer plugin: #3086277: Add a composer plugin to clean up Drupal test directories . An advantage is that this doesn't force anyone to prefer-source in order to get the tests.

However there are various disadvantages

  • Does not solve goal 3 at all.
  • It requires sites to "opt in" by requiring drupal/core-vendor-hardening, which drupal/recommended-project currently does not do. We could fix that, but it wouldn't help sites that are already installed. It's somewhat counter-intuitive to use a tool named "vendor hardening" to harden core and we would need to fix #3104291: drupal-core-vendor-hardening should not assume vendor directory to allow it.
  • Any site that requires a single test would AFAICS need to disable vendor hardening globally.
  • Vendor hardening currently makes composer status pretty much unusable because every single deleted test file is displayed and any 'real' changes are lost amongst that.

Alternative resolution 3

Make all of the test infrastructure into a subtree split library (drupal/tests?) which can be optional. You'd require-dev it in your composer.json file, or remove it as you wished ( #2866082: [Plan] Roadmap for Simpletest ).

Remaining tasks

  • Decide which resolution to use.
  • Decide which directories to exclude - maybe keep the test infrastructure?
  • Fix any issues that are dependencies.
  • Carefully document and test the procedure for sites that do want to keep test code. Consider any possible adverse implications of this for load on Drupal.org infrastructure.
  • Consider possible unintended consequences.
  • Review/test (forthcoming) patch.
  • Write a change notice.
  • Decide when to commit/release the change.

User interface changes

If tests can be considered to belong to the user interface in any sense, the change is that they will disappear from tarballs.

Release notes snippet

TBD.

📌 Task
Status

Needs work

Version

11.0 🔥

Component

other

Created by

🇺🇸United States traviscarden

Live updates comments and jobs are added and updated live.
  • Needs framework manager review

    It is used to alert the framework manager core committer(s) that an issue significantly impacts (or has the potential to impact) multiple subsystems or represents a significant change or addition in architecture or public APIs, and their signoff is needed (see the governance policy draft for more information). If an issue significantly impacts only one subsystem, use Needs subsystem maintainer review instead, and make sure the issue component is set to the correct subsystem.

  • Needs release manager review

    It is used to alert the release manager core committer(s) that an issue significantly affects the overall technical debt or release timeline of Drupal, and their signoff is needed. See the governance policy draft for more information.

  • Needs change record

    A change record needs to be drafted before an issue is committed. Note: Change records used to be called change notifications.

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • 🇬🇧United Kingdom catch

    Proposal #1 seems good to me - it's completely reasonable to require people running tests to use prefer-source.

    Composer patches was brought up, but now that we're using MR diffs on d.o everywhere, it's not recommended to link to the online MR diff anyway for security reasons, instead people should commit a local copy of the patch (or use one uploaded to the issue in some cases). When preparing that patch, it's fine to delete any changes to test files. This also makes it more likely that the patch will apply longer if unrelated changes to test coverage are committed to the project too, sometimes I do it anyway for that reason, especially backporting something for a previous minor/major release.

  • 🇺🇸United States nicxvan
  • 🇬🇧United Kingdom joachim

    > Composer patches was brought up, but now that we're using MR diffs on d.o everywhere, it's not recommended to link to the online MR diff anyway for security reasons, instead people should commit a local copy of the patch (or use one uploaded to the issue in some cases). When preparing that patch, it's fine to delete any changes to test files.

    That's still going to be fairly fiddly to do.

    You're either manually poking around in a .patch file removing bits.

    Or you need the git skills to revert portions of a branch, and that's fiddly too -- something like:

    > git checkout 11.x -- */test/* [maybe? will wildcards work here?]

    and then commit the result to your own local branch. But then if the original MR is updated, you have to repeat that process all over again.

    I mean, sure, I often have local branches in my clone of core for backporting MRs to a stable version. But I'm not sure we can expect everybody to do that.

    The best bet would be a preprocessor to composer-patches, as has been suggested, but I don't know how close a 2.0 release is of that.

  • 🇳🇱Netherlands kingdutch

    I'm assuming a stance that if people want to use patches of unsupported (non-released) changes in their own project they can be expected to make their own patches. In that case, it's built in git functionality to to exclude files with git's powerful path matching syntax. That works in diff, log and format-patch to name a few of the commands that use the same underlying features.

    As an example for a recent Drupal core commit (using --stat to limit the output to what's relevant for the discussion):

    $ git diff --stat 422099995ecfe78f1e29248374fc05cd4a556c1d^!   
    
    core/modules/package_manager/src/ExecutableFinder.php                | 22 ++++++++++++++++++++--
    core/modules/package_manager/tests/src/Unit/ExecutableFinderTest.php | 38 ++++++++++++++++++++++++++++++++++++++
    2 files changed, 58 insertions(+), 2 deletions(-)
    
    $ git diff --stat 422099995ecfe78f1e29248374fc05cd4a556c1d^! -- ':!**/tests/**'
    core/modules/package_manager/src/ExecutableFinder.php | 22 ++++++++++++++++++++--
    1 file changed, 20 insertions(+), 2 deletions(-)
    

    Notice how the second diff doesn't include the test files and could be used to create a patch that would work.

  • 🇬🇧United Kingdom catch

    There's also the issue that including test changes in the diff increases the likelihood that the patch won't apply, either in the first place because there are conflicts with the older branch already, or if it's not committed anywhere yet, that test conflicts are introduced in minor versions even when the runtime code change itself would continue to apply. So there are benefits to removing test files already (I often do it if I'm doing a local backport of a core MR diff already partly for this reason).

  • 🇬🇧United Kingdom alexpott 🇪🇺🌍

    For me this is fantastic idea which we should definitely do. But given the disruption to patches I'd only do this in a major release so minor updates to sites that have patches applied don't break just because we've removed tests.

  • 🇬🇧United Kingdom catch

    I bumped this because of 📌 Package Manager actions intermittently blocked on A2 Hosting Active but also now think 📌 Permanently maintain a sandbox directory in package_manager Active is probably more urgent for package_manager/Drupal CMS. If the other issue works, then this issue would be a very nice to have on top (and for the other reasons it would be good), and starting in Drupal 12 seems reasonable, although if we somehow miss Drupal 12 then I'm not sure I'd want to wait until Drupal 13, could be Drupal 12.1 with a PSA since it's still early in the major release cycle.

  • First commit to issue fork.
Production build 0.71.5 2024