Define entity types programmatically

This would be really useful, but AFAICT isn't possible yet.

Use cases include https://www.drupal.org/project/unomi_connect → , where we could define lots of different entities from Unomi as external entities.

The main problem is parts of the code that do this sort of thing:

    return $this->entityTypeManager
      ->getStorage('external_entity_type')
      ->load($this->getEntityTypeId());

Of course if the entity type is defined in code, then there is no config entity of type external_entity_type for it.

I think the fairly simple fix is to move more things into external_entities_entity_type_build() -- for example, the Query class uses this code to get the field mapper and the storage client. But external_entities_entity_type_build() could set both of these as properties of the entity type definition, which means that Query would only need to load the entity type definition from entity_type.manager.

Thanks. This is the code I actually used.

  $external_entity_storage = \Drupal::entityTypeManager()->getStorage('external_entity_type');
    /** @var \Drupal\external_entities\Entity\ExternalEntityType $external_entity_type */
    $external_entity_type = $external_entity_storage->load('test');
    if (!$external_entity_type) {
      $entity_definition = [
        'id' => 'test',
        'description' => $table_comment,
        'label' => sprintf('[%s]%s', $table_name, $table_comment),
        'label_plural' => $table_comment,
        'field_mapper_id' => 'simple',
        'storage_client_id' => 'xnttdb',
      ];
      $external_entity_type = $external_entity_storage->create($entity_definition);
      $external_entity_storage->save($external_entity_type);
    }

Ah, so you're creating an external_entity_type config entity on the fly.

What I'd like to do is have module code that can define an external entity, without the config entity being there at all.

Setting this to major, as it would require quite a few changes.

I've got this working in a custom module, but for just loading and viewing entities, I needed:

- custom query factory and query class
- custom entity storage handler

The custom storage needs to:
- return the query factory service
- override getFieldMapper(), getStorageClient(), doLoadMultiple(), setPersistentCache() -- basically, any code that looks for a config entity

The custom query factory needs to load the custom query class.

The custom query class needs to:
- override getStorageClient()
- override result() to create the field mapper plugin

What I'm thinking should be changed in this module is remove the use of the config entity outside of external_entities_entity_type_build(). Everything that's needed to be known about the external entity type should be set as properties in the entity type in external_entities_entity_type_build(). That should be enough to make it work with a programmatically defined external entity type, as that would put all the necessary properties in its entity type annotation.

I'm wondering whether as further steps we should also do either or both of the following -- but they would be architectural changes:

1. Define a custom entity type annotation, so programmatic entity types would be defined with an '@ExternalEntityType' annotation. That then means we can define a subclass of EntityType, and add useful methods on that for accessing things like the storage and field mapper.

2. Define a custom entity type group, i.e. 'external' instead of content/config. That's easy to do once we have the custom EntityType class.

3. For the storage and field mapper plugins, put the *classes* into the entity type definition. That way, programmatic entity types don't need to define a plugin, they can just define a class -- they don't need their storage and field mapper classes to be selectable in the UI. Referencing plugins by class instead of plugin ID feels a bit wrong, but they're updated on rebuild, so a change to a plugin (which requires a rebuild) would update in the entity type definition. One step further would be to make the plugins behave as entity handlers -- but then the plugin manager would need to be aware of how to instantiate them, as entity handlers don't get instantiated the same way as plugins.

@joachim, I read you twice, but I still got hard time to understand what you are trying to achieve! It may not be the way you explain it but it may just be because I don't understand the use case. ...and I'd like to understand that "why". :)

From my side, I have thought of a use case but I'm not sure it matches yours and I'm not sure I would need all what you explain to achieve mine. So, I would like to create a custom module that can defines a new external entity type that has more base fields than the default ones (currently id, uuid, title and language stuff added in v3 only at the moment while a patch is pending for v2 ✨ Make external entities language aware RTBC by the way). I'd like to let the user edit this xntt (I use "xntt" as a shortcut for eXternal eNTiTies which is longer to type :p) to "force" her/him map the base fields with whatever storage client and field mapper she/he wants so I could use that mapped entity in other parts of the module for some kind of processing. I'm working in the research in biology and we deal with "genes" for instance and I'd like my user to provide a "gene" entity from her/his data and then be able to process it through different kind of tools or pipelines. And I need that "gene" entity to have a certain set of properties like its source organism, its sequence, etc. Then I would be able to virtually work with any type of data source (REST service, custom databases, files,...) with any tool, even if that tool was not designed to use those source at first. So far, I think I could work with the current external entity type class and extend it. Maybe I'll figure out I can't when I'll try implementing things! :p

Is your use case similar?

@guignonv Basically, at the moment the module is written to put the ability to define external entity types in the hands of the site admin, by creating the entities in the admin UI.

I'd like it to be possible to allow third-party modules to also define external entity types.

So that, for example, you install Unomi Connect module, it installs External Entities as a dependency, and the Unomi external entity types are automatically defined when you enable them.

Ok, so I think we got a very similar goal: a custom module brings its own external entity types derived from external entities.

However, in your case, you don't seem to need to let site admin edit the provided "Unomi" external entity type(s) but correct me if I'm wrong. And maybe, you may not want its mapping or its storage client config to be visible as well (for admins of course)? You just need to "have" those entities available and see/use them.

In my case, I'd like my "gene" external entity type to be edited to set its mapping and maybe add extra field while I would programmatically add base fields (that can therefore can't be removed by the site admin) with some of them being compulsory. And that external entity type could not be removed.

So, when I will try to implement that, if I need to changes things on the external entities v3 side, I'll keep in mind that different level of edition should be allowed (ie. set "read-only" or not storage client config, field mapper config, add fields, manage field display,...). If it can't be currently managed from a custom module, maybe it will require some additions like events or hooks. I'll let you know if/when I make some progress.

@joachim, I tryed to create an external entity type programmatically and now, I understand better what you meant before (as it did not work the way I thought I would do it). I tried a couple of things without success yet. I am also wondering if it's better not to have a config entity for that.

I created an issue fork to test some code. If you think you could do something that works cleanly, feel free to assign this feature to yourself and to commit there, it would help! :)

To sum up the features I see/need:

• Programmatically create an external entity type that can not be removed through the UI (ie. only removed when the corresponding module is uninstalled)
• Be able to programmatically add base fields to that external entity type and make some of them compulsory
• Be able to programmatically choose if that external entity type:
  • config can be displayed
  • is editable (labels, desc., read-only, cache, annotation, etc.)
  • storage client can be changed, and if so, it should be possible to restrict the list of usable clients (eg. only REST client and derived)
  • storage client config can be edited
  • field mapper can be changed
  • field mapper config can be edited
  • fields can be added/removed (non-base fields)
  • display can be altered

But if one can create his own external entity type class that extends the base external entity type class, then, most of these features could be handled by altering the handlers (

 *   handlers = {
 *     "list_builder" = "Drupal\external_entities\ExternalEntityTypeListBuilder",
 *     "form" = {
 *       "add" = "Drupal\external_entities\Form\ExternalEntityTypeForm",
 *       "edit" = "Drupal\external_entities\Form\ExternalEntityTypeForm",
 *       "delete" = "Drupal\external_entities\Form\ExternalEntityTypeDeleteForm",
 *     }
 *   },

) but not all (for base fields, we may need to modify ExternalEntity::defaultBaseFieldDefinitions() to get additional fields from the type).

I wrote code to do this in https://www.drupal.org/project/unomi_connect → , in the MR for issue 📌 Use external_entities to declare Unomi types as Drupal entities Active . See https://git.drupalcode.org/issue/unomi_connect-3376616/-/blob/3376616-us... in particular.

Thank for the code link! :) I had a look and saw you've set custom handlers (view, list,...).

I'm investigating another approach than defining an xntt on the fly, or by defining new xntt type classes: I tried using a module "config/install/external_entities.external_entity_type..yml" definition. I've changed some stuff around to allow disabling edit/delete/plugin selection and it works quite well (I'm still struggling to disable field management).

However, I have not tried yet to override xntt handlers nor to add custom base fields... I'm hoping to find a way to do that through the same yaml file or a companion yaml.

If you have some opinions on this approach, feel free to share!

The downside of using a config/install file is that once it's installed, the site owns the configuration, and the contrib module that supplied the config can no longer make changes to it. Install config is good for starting points and templates, but not so good for my use case where the external entities for Unomi are the crucial machinery of the module.

@joachim: Point noted for the config. I don't understand why the module config can no longer be updated by the module but I believe I'll discover why one day as I've not played much with configs yet... :p If you have readings (doc, blogs,...) about that problem, feel free to share.

Regarding your proposal of getting ride of the external entity type (config entity) in the external_entities_entity_type_build(), I don't understand how you would solve some problems I see. For instance, if I use there a plugin manager to list plugins implementing external entities (content types) and defines those external entity types, then where are the field mapper and storage client configs stored? Are they hard-coded for each external entity content type? Maybe you though more than I did on that problem, with a wider knowledge on how things can be connected. I don't want to take too much of your time explaining things but I can't implement what you need if I don't fully understand it.

From my perspective, external entities need a UI to be configured with a place to store the config. That's the purpose of ExternalEntityType config entity class. That config can be retrieved by any external entity (content) instance through the method ExternalEntity::getExternalEntityType(). From there, it is possible to use storage clients and apply field mappers. The use case you present is the programmatic external content entities. In your precise use case, they don't need user configuration, which I understand. However, I would see a more generic use case (that would include yours as a specific sub-case) where one could provide a programmatic external entity type but would allow the user (admin) to customize it to some limited extends (customize the display or not, add custom fields or not, edit some parts of the storage client like an API key for instance, etc.). So I would not get ride of the ExternalEntityType config entity for that purpose.

So, I would allow the programmatic "customization" of an external entity type (just like I did on the branch I mentioned before) while keeping the regular way of working of external entities (ie. with user external entity type definitions). However, I think it could be possible to also add the management of programmatic external entity type that don't use a config entity but it would add a bit of complexity because we must keep the use of config entities for the other uses cases: we would have to manage the 2 cases in parallel. But I may have missed a more simple approach that would allow those uses cases all together. In other words, if you want a "config-entity-less" external entity, you should either provide some code (on this issue fork* for instance) to examine, or convince me it would really bring something useful compared to the other approach and I should investigate it from my side.

I'm sorry to ask you yo invest time in explanations but I don't want to invest time on my side for implementation without understanding why. :) But if you're fine with external entity type config entity that are just not editable by the user, then we're good to go and don't need to discuss further.

...and if other people want to join this discussion, please do!

*: regarding the issue fork, I am planning to merge my branch "3.0.x-config_xntt" on it once I got other parts of v3 implemented if that issue fork remains unchanged. I'm planning to provide 2 or 3 examples of programmatic xntt created either from config or on the fly or even maybe with annotations (still to investigate but for the config entity) with storage client config that would work anywhere (ie. using a file storage client for instance or a Drupal.org REST API).

FWIW, the approach External Entities is taking (external entity types as config) was inspired by https://www.drupal.org/project/eck → and (IMO) is certainly something that should be kept.

But I'm pretty sure @joachim is not asking to remove that part - rather support both scenario's (through config or through code).

I'm including that feature.

You can review the code but I'll add 1 or 2 more examples in the coming days. I also need to fix an issue when uninstalling the xntt_example_d7import example... but it is functional.

I've merged current solution to 3.0.x branch. I can still investigate the annotation approach but it's not a high priority as I don't think it could impact much current code source structure.

Sorry for not replying to this sooner -- I've been busy with other things and this fell off my radar.

> Regarding your proposal of getting ride of the external entity type (config entity) in the external_entities_entity_type_build(), I don't understand how you would solve some problems I see. For instance, if I use there a plugin manager to list plugins implementing external entities (content types) and defines those external entity types, then where are the field mapper and storage client configs stored?

My idea was that the config still work exactly as they currently do. But ALL the configuration from the config entities gets put into the entity type definition in external_entities_entity_type_build().

So from that point on, any code that needs to get something about the external entity type should look at the entity type definition. The config entity isn't used by another other code.

That means that a module can define an external entity type completely in code and the consuming code won't see the difference between a config-defined or a code-defined one.

Configs for plugins related to the entity type, such as the field mapper, can be put on the entity type definition, something like this:

@ContentEntityType(
... etc

"field_mapper_id": "foo"
"field_mapper_config": (
  // array of plugin config
)

)

I'm doing some work on this.

I think the way to handle this is to add our own entity handler type, 'external_entity_type'.

Instead of loading the external entity type config entity to find out things about the entity type, code should use the handler instead.

Config-defined types then get a standard ConfigEntityExternalEntityHandler, which just loads the config entity and hands over.

Code-defined types would need to define their own handler to return the right information.

OK, no worries. :)
I just want to make sure you noticed the change I made in "external_entities_entity_type_build()" (hook_entity_type_build) and in the ExternalEntity class:
I added "public static function getBaseDefinition()" that may do part of the trick you need. But I think it is not all that you have in mind...

Keep me tuned on the progress.

I've pushed a small proof of concept to branch external_entities-3374867.

It only converts the access control handler so far -- I wanted to see how it would work with a handler that currently expects to use the config entity.

The basic idea is that any code that wants to know something about an entity type's external functionality should ask the ExternalEntityHandler.

Config entity-based types use the ConfigEntityExternalEntityHandler, which hands over to the config entity. Code-based entity types should implement their own handler to return whatever makes sense -- they can either define things like readonly / annotatable in the entity type annotation data, or hardcode it in the handler, or something else.

(BTW I've experimented with a simpler and more self-contained way of mocking external entity remote data in the kernel test I added. I think it's clearer because there's less code and it's all in the same place, though I could be convinced to move the __invoke() to a separate class. The other advantage over the controller technique the existing tests use is that there is no actual HTTP request made, since the request is intercepted by the middleware. That means the test runs more quickly, because there isn't a whole other request that needs Drupal to be bootstrapped in a second process. See what you think :) )

I'll have a look but I'm already exited by the idea of having tests running faster without passing through real http requests! :D
BTW, I' ll move on other projects for the next days so I'll be less productive here in the coming days (weeks?). But I'll keep an eye on things and I still plan a beta for summer.

Looks good to me! :) I'll merge the handler branch soon.

@guignonv The work on that branch so far is just a proof-of-concept! I only converted one class; there's lots more that would need changing over.

OK. I'll hold on. Let me know when it is mature enough.
So far, the idea of a handler that supersedes the "type" sounds good to me.
Meanwhile, I'll work on ✨ Using multiple field mappers at once Active .

> So far, the idea of a handler that supersedes the "type" sounds good to me.

Great! I wanted to get your approval of the approach before doing the whole thing.

Hi @joachim, I'd like to release a beta version of v3 during summer. Regarding this issue, I am wondering if what you plan to do would include breaking changes which would therefore be a beta-blocker or if not.

If there should not be breaking changes (or it would be limited to very specific use cases), then it would not prevent me from releasing a beta and there's no rush. Otherwise, do you think you would have time before the end of summer to provide your changes (no pressure, just a question to plan something)? :)

@joachim, hold on if you have not already made a lot of changes. After much thought (and vacation ;p ), I'll change some stuff in the current design: I'll add an additional interface "ConfigurableExternalEntityTypeInterface" that will hold things related to field mappings and data storage (while stuff related to annotation will remain in ExternalEntityTypeInterface). Therefore, programmatically created external entity (types) will not require field mappers or storage clients.

I will externalize the mapping and storage management (and data aggregation) logic in a new object ExternalEntityManager (with an interface) that will use an external entity type to do its job. I'm here so far and have not finished the design yet but I'm thinking of a service to get the manager according to an external entity type. Maybe the manager class could be specified in the external entity type annotations... I'll think deep and see what make more sens, unless I got some suggestions from others!

I've committed the other major changes on field mapping and the way the ExternalEntityStorage class is used. ExternalEntityStorage is now used by Query and other processes needing to alter external entities. Therefore, it may help "customizing" external entities because one would only need to change the storage handler.

I've release beta 1 which provides multiple ways to define an external entity and external entity type programmatically. It is also possible to use a custom external entity storage. This version should be stable now. New features for creating external entities programmatically involving API changes could fit in a next version (ie. 3.1.x).

Change status back to "needs work" if you think you'll find time to work on it.