[plan] Discuss desired API and test suite following deprecation of SimpleTest

Problem/Motivation

This issue might turn into a plan with several [policy, no patch]-type sub issues.

In #2735005: Convert all Simpletest web tests to BrowserTestBase (or UnitTestBase/KernelTestBase) → there is some great work being done to deprecate SimpleTest in favor of a third-party standard testing framework, which will reduce our need to maintain duplicate code and also potentially make Drupal's test suite more learnable for developers familiar with (e.g.) Symfony's, and vice versa.

What I haven't seen yet though is a higher-level discussion of what the testing framework we want in the end is though, aside from "remove SimpleTest". So we have what we want to move away from, and specific strategies from moving away from it smoothly, but less clear objectives about what we want to move to. I'm deliberately keeping this as a separate discussion, in order to not derail simple conversions being done elsewhere.

Some of the questions we should consider follow.

API of the new test suite

In #2759879: Additional assertions for WebAssert and AssertLegacyTrait, part 2 → , when reviewing the patch, I noticed that we were implicitly making some decisions/compromises about the API we were converting to in the process of deprecating SimpleTest. For example, the deprecation of the legacy trait version of assertUniqueText():

+   * @deprecated Scheduled for removal in Drupal 9.0.0.
+   *   Use $this->getSession()->getPage()->getText() and substr_count() instead.

That is making a decision for later that we should not have assertUniqueText(). As @Lendude and @catch have pointed out, there are a few reasons we might want to remove a method:

• It's not used enough to justify the maintenance burden.
• It's redundant.
• Having too many methods can actually make it harder to pick the right one.

However, the competing concern is that having a well-named API of test methods improves readability, DX, and maintainability by not repeating boilerplate code over and over (which then in turn can have a bug introduced in any one place and we won't know).

So, we should have an architectural design discussion of what assertions we want to have or not, and use data from core and contrib test suites to justify keeping or removing each.

This also probably applies at other levels of the API, not just assertions, but it was the simple case I encountered. I also have concerns about (e.g.) the verbosity of our unit tests, and I want to make sure we don't end up in a place where our functional tests are equally difficult to follow.

In general, we should be striving for a test suite that is readable, maintainable, and easy for even novice developers to understand.

Structure of the test suite

As @catch pointed out in #2735005: Convert all Simpletest web tests to BrowserTestBase (or UnitTestBase/KernelTestBase) → :

There's a fair bit of duplication between the test coverage for modules like node/taxonomy and entity system tests. Similarly between tests in Views module vs. tests in views-data-providing modules. That's no fun to pick apart but would be good if we did.

So issues by theme, then sub-issues for those sounds good.

In general, I'd advise against blocking the conversion of any test on a redesign of it -- that leads to problematic scoping → that makes initiatives like these get stalled. However, as we do 1:1 conversions of tests, we should also discuss, separately, whether the tests even make sense and whether they should be the kind of tests that they are (kernel, unit, web test, etc.). Some high-level research would be great here, as would tracking observations folks make as they work on other tests.

Other stuff

There is probably more.

Proposed resolution

Discuss!

Remaining tasks

Sub-issues?

User interface changes

N/A

API changes

TBD

Data model changes

N/A