Calling EntityTypeManager::getDefinition during ConfigFactoryOverrideInterface::loadOverrides can lead to corrupt entity type handlers

Problem/Motivation

Late last year we started seeing a lot of errors popping up due to a missing moderation_state handler on node, media, and block_content entity types. It got so bad that Drupal uninstalled fields from the database, and all sorts of other issues started to occur. Route providers dropped off too leading to Route not found exceptions and our whole site became incredibly unstable.

Tracking this down, the errors started occurring on the 7th of December, which was when we deployed an upgrade to the Key module, going from 1.3 to 1.5. That introduced the KeyConfigOverrides class and functionality.

The problem here stems from KeyConfigOverrides::loadOverrides in that it calls EntityTypeManager::getDefinition and can be executed very early in the Drupal bootstrap process.

For us, this was during hook_module_implements_alter due to the fact that the Display Suite module calls:

\Drupal::config('ds.settings')->get('field_template')

Now if this is called when EntityTypeManager's definitions aren't cached, it leads to corrupt entity type handler caches as other modules don't get a chance to add their handlers in yet.

Here's some of the errors caused by this for searchability:

Symfony\Component\Routing\Exception\RouteNotFoundException: Route "entity.node.entity_hierarchy_reorder" does not exist. in Drupal\Core\Routing\RouteProvider->getRouteByName()

Warning: Creating default object from empty value in Drupal\workbench_moderation\EntityOperations->entityStorageLoad() (line 108 of /var/www/app/modules/contrib/workbench_moderation/src/EntityOperations.php)

Error: Call to a member function id() on null in Drupal\workbench_moderation\Form\EntityModerationForm->Drupal\workbench_moderation\Form\{closure}() (line 85 of /var/www/app/modules/contrib/workbench_moderation/src/Form/EntityModerationForm.php)

Notice: Trying to get property of non-object in Drupal\workbench_moderation\EntityOperations->entityStorageLoad() (line 107 of /var/www/app/modules/contrib/workbench_moderation/src/EntityOperations.php)

Notice: Undefined property: stdClass::$entity in Drupal\workbench_moderation\ModerationInformation->isLiveRevision() (line 212 of /var/www/app/modules/contrib/workbench_moderation/src/ModerationInformation.php)

And so on.....

Debugging into workbench_moderation_entity_base_field_info():
On Key 1.5 - I can see that the moderation_state handler does not exist after running \Drupal::entityTypeManager()->clearCachedDefinitions(); and loading a page.
On Key 1.3 - I can see that the handler is added.

Proposed resolution

???