Letting Drupal know about new/updated/deleted external entities

Here's my experience on a project we're working on.

We have an external system that notifies us (through a custom API endpoint) about changes on our external content. Once we receive a message (operation, type, ID), after some trial and error we ended up with the following code to invoke entity lifecycle hooks:

if ($action === 'delete') {
  $entity::preDelete($entity_storage, [$entity]);
  $this->moduleHandler->invokeAll($entity->getEntityTypeId() . '_predelete', [$entity]);
  $this->moduleHandler->invokeAll('entity_predelete', [$entity]);
  $entity::postDelete($entity_storage, [$entity]);
}
else {
  $entity->original = clone $entity;
  $entity->preSave($entity_storage);
  $this->moduleHandler->invokeAll($entity->getEntityTypeId() . '_presave', [$entity]);
  $this->moduleHandler->invokeAll('entity_presave', [$entity]);
  $entity->postSave($entity_storage);
}

$hook = $action === 'create' ? 'insert' : $action;
$this->moduleHandler->invokeAll($entity->getEntityTypeId() . '_' . $hook, [$entity]);
$this->moduleHandler->invokeAll('entity_' . $hook, [$entity]);

This has worked more or less OK for quite a while. It does have some gaps, however. Thinking about the entity translation-specific hooks, for example.

After some thinking, I think the most proper way is to be able to call $external_entity->save() or $external_entity->delete(). This comes very close (if not completely) to Drupal's default way of working with entities - but doing this could (if your external entities are not marked as read-only) send a write to the external API - which is undesired in this scenario (since we only want to call it to make Drupal do its thing on the new/updated/deleted entity).

I was wondering if it's a good idea to introduce something so that we can (temporarily) mark external entities on which save/delete operations are performed, to not push through to the external API. See patch attached. It's based on a similar functionality Drupal core (setSyncing and isSyncing).

With this patch, the code above could be changed to:

$entity->skipExternalStorageMutation();
$entity->save();
$entity->skipExternalStorageMutation(FALSE);

and

$entity->skipExternalStorageMutation();
$entity->delete();
$entity->skipExternalStorageMutation(FALSE);

Any thoughts? How is everyone else tackling this problem space?

I'm not there yet but I'll keep an eye on this issue. ;)

Can anyone shine some light on how/why external entities are bypassing those hooks?

Do we need some way to determine what to do in different scenarios? IE if an API call for data with an ID returns nothing, do we assume it's deleted, or something else?

if an API call for data with an ID returns nothing, do we assume it's deleted, or something else?

You can't "assume" things in my opinion, since you may have a service temporarily unavailable for instance, or a bug in a remote source. Maybe, it could be a setting on the storage client config, to tell how to behave...