Contrib.social

Allow modules to provide default configuration for running tests

Open on Drupal.org →
Created on 16 September 2010, about 14 years ago
Updated 28 July 2024, 4 months ago

Problem

  • Tests that require other, optional modules cannot rely on any kind of "default" configuration for executing a test, and thus are forced to implement lots and complex code to create some.

Goal

  • Allow tests to rely on any kind of default configuration provided by other optional modules they depend on.

Details

  • #1541298: Remove Node module dependency from Testing profile → and many other issues run into the same problem; e.g., in that case:
    1. Lots of tests rely on Node module
    2. But Node module is no longer installed by default.
    3. Node module's default user role permissions for "access content" no longer exist by default.
    4. By merely installing Node module, users are not able to access or create content.
    5. Lots of test failures.
  • Modules need to be able to provide default configuration for tests.
  • The default configuration for tests may be entirely different to the regular default configuration of modules being setup via config and hook_install(). (not to be confused)
  • The testing configuration rather maps to our previous, unintended abuse of the Standard profile - whatever was defined there, all tests that did not care simply used it.
  • The problem space exponentially increases when considering contributed modules that were never able to rely on the Standard profile in the first place. Module Z that relies on module Y which relies on module X needs to create testing configuration for both Y and Z in order to test X. (both Y and Z are otherwise irrelevant for the test, just need to exist to make X work)
  • Any default testing configuration, regardless of how it is implemented, will be set in stone once after it has been committed, because it might be used by tests in other projects.
  • An ideal world solution would therefore account for testing configuration "versions" and/or variants. But we possibly want to defer that complexity to later.
  • Any default testing configuration is never supposed to serve as basis for testing exactly that actual configuration. It is only ever used for (other) tests that "just need something" to run. If the test result can vary on that other configuration, then DrupalTestCase::generatePermutations() has to be used to verify the expected behavior for all possible variations instead.
  • The ultimate goal is to reduce required dependencies of Drupal core to an absolute minimum, in order to (drastically) speed up the Testing profile. In a sense, the (utopian) goal is achieved when a functional/integration test completes as fast as a unit test (e.g., a test for System module that doesn't require any other dependencies than System module itself).
  • Pretty much all tests will be asking for default testing configuration for all other modules that they need to install.
  • In turn, it would make most sense to install the default testing configuration for all modules being installed for a test by default, except if a certain module was asked to be skipped. This would come closest to our current/previous abuse of the Standard profile - by merely asking for modules to install for your test, you also get the default config for each, unless explicitly skipped. The simplicity in this sounds very appealing.

Notes

  • Recent PSR-0 seem unrelated, since all of this affects functional/integration tests (and not unit tests), which fully rely on the module system in the first place, so an API hook based solution sounds most appropriate.
  • The new configuration system for D8 does not seem to help either, because the default testing configuration may only be conditional, depending on other modules being installed for a test. Additionally, it is unclear whether the new configuration system will support the required granularity in the first place (e.g., granting "access content" for two specific roles, as in above mentioned example).

Proposed solution #105

  1. Introduce hook_testing_install() to allow modules to provide default testing configuration.
  2. Make setUp() invoke hook_testing_install() for all modules being installed automatically.
  3. Allow the test case to skip the default testing configuration for certain modules (so it can be set up manually).

Our tests take longer and longer to run (~30 minutes on highly optimized testbots recently), so we really should make the new testing profile work properly.

Since my local box is quite slow, I've tried and tried and tried to use the testing profile all over again, but in almost most all cases, I was hitting annoying bugs caused by code that still relies on the standard profile.

This is just a start, the most obvious bug. We should try to convert a couple of core tests here to make sure that the profile really works like it should.

📌 Task
Status

Closed: outdated

Version

11.0 🔥

Component
Simpletest  →

Last updated about 2 months ago

Created by

🇩🇪Germany sun Karlsruhe

Live updates comments and jobs are added and updated live.
  • Performance

    It affects performance. It is often combined with the Needs profiling tag.

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Comment 4 months ago →
    🇳🇿New Zealand quietone

    Since this issue was opened there has been quite a bit of work and improvements in the testing environments. Perhaps the lack of discussion here for 9 years is a reflection of that. I think this is now outdated and closing as such. If that is incorrect, add a comment to explain.

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024