DatabaseCacheTagsChecksum is misbehaving because of the static cache

Problem/Motivation

We have following case in a post update hook when updating from 8.1 to 8.3 over "drush updb":

We create a field storage config for a node and save it. Afterwards in the same post update hook we create a field config for a specific node type for the new field storage and save it, which leads to the following exception and not continuing with the updates:

Failed: Drupal\Core\Field\FieldException: Attempt to create a field field_test that does not exist on entity type node.

What I've discovered is that an invalid entry for the CID "entity_field_storage_definitions:node:en" is being retrieved as valid, but the field storage definitions would have been rebuild, as the entry is not valid at all. As the entry is invalid and the data inside is obsolete the new storage definition is not present there, which is leading to the exception.

Debug information:
I've retrieved in the beginning of the post update the Cache Tags Invalidator Checksum Service and prior to creating the field storage definition it has the following (only the relevant ones) cache tags entries:
entity_field_info=39
entity_types=38

Here the fun starts as at this point this is not the state in the DB as there these cache tags have the following invalidation sums:
entity_field_info=40
entity_types=39

Meanwhile the important cache entry "entity_field_storage_definitions:node:en" in the cache_discovery table has the checksum of 79.
So based on the state in the DB the checksum for this entry is correctly estimated based on the cache tags invalidations in the DB.

Now we create the field storage config and save it and here the debug info again:
Cache Tags Invalidator Checksum Service:
entity_field_info - not present
entity_types=38

DB:
entity_field_info=41
entity_types=39
Checksum of the entry "entity_field_storage_definitions:node:en" - 79.

Now during the saving of the field config that cache entry for the node field storage definitions will be retrieved and checked if it is valid. The Cache Tags Invalidator Checksum Service has the false entry for the cache tag "entity_types=38" and is going to retrieve from the DB only the invalidations of the cache tag "entity_field_info", which value in the DB is 41.

So now the sum of both cache tags entity_types=38+entity_field_info=41 is 79 and matches the checksum of the cache entry for "entity_field_storage_definitions:node:en". However the correct sum of cache tags should have been entity_types=39+entity_field_info=41 = 80.

So what happens is that instead of considering the entry as invalid and rebuilding the field storage definitions for the node entity type we retrieve the obsolete storage definitions from the cache.

I would say that as it is running as part of drush updb all this happens in one single request, which looks pretty dangerous that there is suddenly such an inconsistency. As this might lead to all kind of troubles and also breaking the DB I am considering it as critical for now.

Proposed resolution

As such cases might happen and they might be worse during concurrent requests I think that we should remove the static cache of the Cache Tags Invalidator Checksum Service.

Remaining tasks

User interface changes

API changes

Data model changes