Update Hooks Can End Up Rolled Back with Schema Version Still Incremented

Problem/Motivation

We maintain an internal Drupal distribution. After releasing a new version of our distribution, we tried to upgrade/update a site that was running the older version of our distribution to the new one. During this upgrade, we ran schema updates through drush updatedb and encountered some bad data in our installation, so only the first 80% of our update hooks executed successfully before Drush bailed out on the failing update.

After correcting the offending record in the database and then running drush updatedb a second time, we found that only the update that failed was re-run. But, strangely, the site behaved as if the earlier update hooks had not run. Specifically, one of the hooks that had run successfully enabled a new module dependency (e.g. module A now depends on contrib module B, so the update hook enabled contrib module B), but we found that the dependency wasn't enabled on the site despite the hook having been run and Drupal reporting that module A had a schema version equal to the highest update hook in the install file.

It appears that Drupal ran the hooks, then rolled back the DB transaction/changes when the update at the end of the chain failed, but it still recorded the schema version changes in the database for the updates that ran successfully before the rollback.

Steps to reproduce

I'm not able to pinpoint exact repro steps because I could not get this failure to happen in the same sequence each time. But, the steps seem to be:

1. Create and install two modules (so that their schema versions start at 8000).
2. Modify the first module to add an update hook that enables a new dependency (make sure the new dependency is a module that's not already installed and enabled on the site).
3. Modify the second module to run a database query that results in an uncaught SQL exception or similar.
4. Run updates using Drush drush updatedb.
5. Ensure that the update hook for the second module failed, causing Drush to report that updates failed.
6. Check the site to determine if the new dependency of the first module is enabled.

Proposed resolution

I haven't delved into Core code to determine where this is happening, but it seems to me that either the changes from each update hook should be committed atomically rather than rolled-up into a single transaction; or, that the schema version updates in the database should happen in the same transaction as the update hooks themselves, to ensure that the schema version does not get incremented without the side-effects of the update taking effect.

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet