Add new 10.3.x database dump fixtures, without modules deprecated for removal in 11.x

We could potentially bundle 📌 [policy] Remove update hooks added until 10.3 from Drupal 11 RTBC in here too.

Updated issue link for update removals: 📌 [11.x] [PP-x] Remove updates added up until 10.3.0 from Drupal 11 Active .

Increasingly I think it's worth trying to combine these, so a single issue which removes deprecated modules from the database dumps, and also updates the database dumps to 10.3.0, this would mean only having to update the database fixtures once for Drupal 11, instead of multiple times as we did in Drupal 9.

The only problem is deciding what a 10.3.0 database looks like before it's released, which is going to be when we're not planning to commit any more updates to 10.3, but would commit them to 10.4 instead or a cut-off point where we say any updates added after that point will be left in 11.0.0 too.

Changing parent to the issue tracking removal of all the extensions.

Just trying to grasp what we're trying to do in here: The IS talks about the 9.4.4 fixture, #3 📌 Remove deprecated modules from update testing database dumps etc. Active tlaks about a 10.3.0 fixture.

Are we going to keep the 9.4.4 fixture around _and_ add a 10.3.0 one?
I would imagine we can loose the 9.4.4 one, since we don't support jumping from 9.x to 11.x?

@Spokje: AFAICT the issue summary is just describing the current, painful situation. I do not think the proposal is to keep a D9 DB dump.

Thanks @Wim Leers, I suspect as much, just wanted to make it clear (and yes, I also felt the pain of fixture-fiddling)

In 11.0.0 we would drop the 9.4.4 fixture, this requires having a 10.3.0 fixture (in reality it's going to probably have to be a 10.3.0 alpha fixture due to lack of time travel).

We could add the 10.3.0 fixture to 10.3.0 itself too though, since it will help with 11.x->10.x update path backports - they'd be able to use the same fixture for any dedicated test coverage.

Does it mean that 10.3 should ship both 9.4 and 103.alpha fixtures?

@andypost that's what we did with 9.5/10 and it's useful yes - it means any new updates in both 11.x and 10.x can be tested against the 10.3.0-alpha fixture, so makes backports easier.

So we know there's pain when changing fixtures, so we combining the changes so there's less pain.

Now what if a core module doesn't have the remove issue committed on time for 10.3.0?

We then have to go through the pain again when changing the fixture painfully made in 10.3.0 in 10.4.0?
Could we move the removal to 12.0.0, so there's less pain?

(Not sure about if there's already specific rules/documentation on this, just curious and pretty much done with the pain after the gazillion changes we did on fixtures during D9 => D10.)

Now what if a core module doesn't have its removal issue committed on time for the release of 10.3.0?

We then have to go through the pain again when changing the fixture painfully made in 10.3.0 in 10.4.0?
Could we move the removal to 12.0.0, so there's less pain?

I don't think we would deprecate any modules in 10.4.x for removal in 11.x, even if 11.0.0 gets pushed to December. So essentially anything deprecated in 10.3.x has to be removed in 11.x, but anything not deprecated in 10.3.x will probably have to be deprecated in 11.1.x or later for removal in 12.x yeah. There's not really any point deprecating a module in 10.4 for removal in 12.x because it doesn't reduce the amount of time that we have to support it in core for, so can't really see that happening either.

I think we only have 📌 [meta] Tasks to deprecate Statistics Active and 📌 Deprecate the Statistics module Postponed left from 🌱 Deprecate dependencies, libraries, modules, and themes that will be removed from Drupal 11 core Active (field_layout isn't going to happen this time, the javascript ones aren't modules and also aren't happening), so once those are done, that's a full slate of modules, and we then need to figure out the cut-off point for upgrade path removals and start working on this.

I came to conclusion to replace module with obsolete stub info.yml file working on 📌 Remove Action UI module Postponed
Removal of the stubs (help_topics and sdc ATM) could use separate issue after fuxtures update

Moreover in case of action module we have no hook to disable it to allow end-user to consider so it's one line removal from extension.list so could be done here to unblock removal

Statistics isn't quite deprecated yet, but I think it's close enough that we can remove it from the database dumps in parallel to the deprecation process now.

That leaves whether we're going to be committing any new and complicated upgrade paths to 10.3.x, if there's nothing about to be committed, I think we should probably go ahead here.

There are several steps, hence delaying this until it hopefully only has to be done once:

For each of 'base' and 'filled'

1. Load the database dump fixtures into the Drupal versions they were created from.
2. Uninstall all the to-be-removed core modules on that version.
3. Switch to the 10.3.x branch
4. Run updates.
5. Dump the database fixtures to new 10.3.x database dumps.
6. Create MRs for 11.x and 10.3.x to add the new dumps.

Once we have an MR with the new dumps, we can open an 11.x-only issue to remove all the pre-10.3.x updates hooks, and implement hook_update_last_removed()/hook_removed_post_updates(), and switch existing coverage to the new dumps (and remove coverage for all the updates we're removing). This will need to be stacked on top of the fixture additions until they're committed. Or it's possible to combine the whole lot in one issue but makes it a lot to review.

I have avoided making these dumps in the past and I would like to do them this time. I've got a basic script working to do the task. I just need to test the resulting database and have it run on all the source fixtures. So I am assigning this to myself.

Thank you @quietone!

I made a separate issue for the removal of hooks etc, 📌 Remove all the pre-10.3.0 updates hooks Fixed . I found it easier to keep that separate from the changes here to actually use the new dumps. There is a comment in that issue about some files that can probably be removed.

So, I merged that issue and updated tests and comments. As we know that will result in a large MR, and it is 124 files + 143980 − 8136.

One thing to mention is that when making the new dumps I found what I believe is an error in the 'filled' one. In that database node/1 is an Article content type and part of a book. However, the book configuration only allows content type of book to be in book. I resolved that by retaining node/1 and deleting the row from the book database table.

The issue split is good, we can commit the new dumps here first, then continue in the update removal issue, then after that module removals.

Would you mind posting your script to this issue for posterity?

Testing 7403 and there was a merge conflict around

deleted by us:   core/modules/views/tests/src/Kernel/ViewsConfigUpdaterTest.php

So resolved that. But still appears to have test failures now, so moving to NW for that as I don't want to step on anyone's toes.

Applying 7403 to 📌 Remove Tour module Postponed to see if that works though for a removal though

But the tests that are failing should they be deleted? Since the db dump has correct array keys

7404 is the MR to review here, 7403 is combined with 📌 Remove all the pre-10.3.0 updates hooks Fixed .

@catch can you rerun the javascript tests on 7404? I can't because of gitlab roles.

Okay so opened separate branch of just the DB dumps for 11.x 7426

The review missed that the steps in #14 are not complete. Also, in #22 the catch asked that the script I used was shared here.

On reviewing everything I found that I missed removing some updates over in 📌 Remove all the pre-10.3.0 updates hooks Fixed . Also, I forgot to change the script to not use drush, which currently doesn't install on 11.x. So, the script pauses and I use the UI to uninstall modules and run updates. With all that fixed these two issues are working as expected for 11.x

MR 7426 - This has the new dumps, changes needed for UpdatePathTestBaseTest to work with the new dumps, and the changes from 3439769 which does all the work to remove hooks.

Curious how it affected locale's tables, for tracker removal there's 2 strings

@andypost, I don't know. How do I find out? Is this different from other modules that have translations? What are the strings?

Specifically here all remains
https://git.drupalcode.org/project/drupal/-/blob/4cd72d0c4a9132bf2540e1f...

Hmm, I have not seen any work like that on the locale tables in the previous extension removal issues. AFAIK, the uninstall process does not alter the locale tables. I checked with a fresh install of Umami and that is true. I installed Umami, then install tracker and comment. After the table size for locale_location and locale_source increased. I then uninstalled those modules and the table size remained the same.

Maybe I am misunderstanding, I should be asleep instead of at the keyboard.

I think locale strings stay in the database after install, and also that's fine - the database is supposed to mimic what happens when you update real sites over time in the hope that this will pick up problems. e.g. if we ever added 'does this string still exist anywhere' auditing to locale module, it would have to deal with old strings. But also for real sites when they re-enable statistics, their translations will still be there.

Is there a reason to ship extra unused strings? it will slow down every upgrade test

Tests are passing. This adds dumps for 10.3.0 and updates the two tests that use \Drupal\FunctionalTests\Update\UpdatePathTestBase.

And this is the script I've been developing. It requires ddev and relies on my setup and commands.

@quietone would it be useful to commit the script too? Like you did for the sub tree split

It requires ddev and relies on my setup and commands.

No, I would not commit it because it will only work on my setup.

I can verify the following modules are removed from tests.

Actions
Actions UI
Book
Forum
Tour
Tracker

I see that the new database is added, and updated in code. Someone will still need to validate the references to the modules are removed from there also.

📌 Remove all the pre-10.3.0 updates hooks Fixed is actively using the new database dumps. The only other thing that could go wrong is if tests start failing when we remove deprecated modules, but I think we will only find out if there's a problem when we do that. Going to go ahead and mark this RTBC.

Actually there is a problem. I thought gitlab was being clever showing the uncompressed gzipped file, but it's actually not gzipped. Pushed to the 11.x MR actually gzipping the files, the dump script doesn't do this.

catch → changed the visibility of the branch 3439769-with-new-dumps to hidden.

catch → changed the visibility of the branch 3414563-remove-deprecated-modules to hidden.

catch → changed the visibility of the branch 3414563-11.x-just-db-dumps to hidden.

catch → changed the visibility of the branch 3414563-remove-deprecated-modules to active.

Tested this against 📌 Remove all the pre-10.3.0 updates hooks Fixed and got to a clean test run on there. Since all I did was rename the files to remove the .gz extension, then gzip them, back to RTBC.

Slightly more descriptive title.

I recreated the dumps locally and diff the output (fun) and did not see anything too surprising. There are some new updates for 10.3.x that change things but all that means is that either we need to recreate them or we need to leave the updates in.

Committed 9fdd5ef and pushed to 11.x. Thanks!
Committed 6e44daa and pushed to 10.3.x. Thanks!

I did not use the 10.3.x branch because the dumps there seem to be flawed and contain reference to an update on 10.3.x that has since been removed. I just manually backported the dumps (and only the dumps) to 10.3.x.

I think we should create a follow-up to ensure we have a test on 10.3.x using the new dumps just to make sure everything is working as expected.

quietone → closed merge request !7404

quietone → closed merge request !7403

Needs followup for last paragraph in #51

smustgrave → closed merge request !7426

it need one more follow-up for missed statistics settings

->values(array(
  'collection' => '',
  'name' => 'statistics.settings',
  'data' => 'a:2:{s:19:"count_content_views";i:0;s:15:"display_max_age";i:3600;}',
))

There are actually lots more remaining elements to remove, beyond statistics: there are still remainders from aggregator, book, color, forum, and statistics. Most of them are config data and translations.

Translations are supposed to stay, to represent sites that been upgraded from earlier days, the only reason to clean-up is to speed-up of upgrade tests

Yes, there is 'left over' data in the dumps. However, as catch explained in #36, these dumps are "supposed to mimic what happens when you update real sites over time in the hope that this will pick up problems." They are not created from a fresh install with data added. They are created by first installing the previous dumps on 9.4. The Issue Summary has more details about the process. The point is that these dumps are to reflect what really happens when an extension is uninstalled, not what we think should happen. So, the fact that data seems to be left behind is not something to be changed here. This is working as intended.

If you think the uninstall process should be removing more date then that should be discussed in a separate issue.

At least both #56 and #58 needs work for follow-up

regarding #58 - maybe statistics was not uninstalled properly? otherwise it's not clear why it's config is not removed

#58, #62. I discussed this again with catch and we both agree the dumps are correct. All the modules were uninstalled correctly. The 'statistics_settings', and another settings for the removed extensions are not in 'config', they are in 'config_snapshot'. That make sense so one can do a diff on configuration settings. Or, should a new config_snapshot should be made when an module is uninstalled. That would be a discussion for a separate issue.

Opened 🐛 config_snapshot can contain uninstalled and removed modules Active for the config_snapshot issue, going to move this back to fixed.

@catch. thanks for adding the followup.

Automatically closed - issue fixed for 2 weeks with no activity.