[META] Roadmap for v3.0

There should be v3 beta 1 by the end of this week.

First beta is delayed to next week due to still some instabilities. More tests are needed.

I've been heavily involved with the development of 2.x of this module (hence why I'm maintainer as well). Until a few months ago I wasn't aware a 3.x of this module was in the works. Is there an upgrade guide from 2.x to 3.x?

Yes there is one. The updates (in the install file) *should* be able to upgrade from v2 to v3. I say "should" because I've tested on my stuff and tried to make something generic but there are always things you miss when you make this kind of "heavy" migrations and I'm sure I missed some use cases... So, you should be able to upgrade your site to xntt v3. If you experience problems, create issues and I'll see how I can help. If you're talking about "code migration" (ie. custom plugins you made for instance), I am planing to write some doc about it. I have started something in the "DEVELOPMENT.md" file but it might not be so clear I admit. For code migration, I can provide help through issues if you need to know where did "such method" moved or how to replace "that piece of code" that was calling methods that are not there anymore.

I definitely need to write a doc on how to create new xntt plugins. That would help understanding the new design and also help migrating old code as well. There is a lot of work to do and this one is not "high priority" for me right now. I needed v3 to package it in a project beta by the end of the year with a couple of other stuff. Once I got that completed (hopefully before December...), I'll be able to allocate time to more documentation.

@guignonv
Thanks for the update. I have a project which makes heavily use of external entities (which is a reason I started the 2.x branch for). I'll free up some space to upgrade to v3 and report back on the things I had to change on my end.

My project is now in need for some extra features, such as ✨ Make external entities translatable Active and ✨ Allow to annotate without the use of inline entity form Active . I will be working on these the next few days (for v3). Could we get these on the roadmap for v3? They might require some (minor) breaking changes (not entirely sure yet) - so it would be nice to have them in before 3.x RC hits.

Good to see the collaboration here. I think this has huge potential for Drupal.

I am interested in making it easy to pull in external information from APIs, plus Drupal - Drupal interoperability.

Throw in a plugin for scraping content from html and we'll be cooking with plutonium.

I can help with documentation and testing things as they are released.

I upgraded my project (using several custom storage clients) from 2.x to 3.x. Here are some remarks:

• The update hooks transforming the config to be compatible with 3.x worked flawlessly!
• Most of the things I had to do is already documented in https://git.drupalcode.org/project/external_entities/-/blob/3.0.x/DEVELO... .
• In order to alter some filters, I had some custom hooks I invoked from my custom storage clients. I was able to replace these successfully with the TRANSLITERATE_DRUPAL_FILTERS event.
• I had some custom logic in my storage clients related to sorting parameters sent to the external source. I couldn't find a way to replace these, so I created ✨ Allow to alter sorts through event subscriber Needs review .
• I faced some other issues, for which I created several tickets. ( 🐛 ID is not assigned to newly saved external entities Needs review , 🐛 JsonPath::getMappedSourceFieldName() incompatible with various JSONPath notations Needs review , 🐛 Incompatibility with Monolog Needs review )

It went better than expected. Nice work!
I did see some stuff in code I have doubts about/maybe would do differently. If it's OK by you, and once I find the time, I'll create a separate issue to list these & have a place to discuss them.

I did see some stuff in code I have doubts about/maybe would do differently. If it's OK by you, and once I find the time, I'll create a separate issue to list these & have a place to discuss them.

Sure! Go ahead! Most of the re-design is mine and not perfect for sure. I'm open to discussion and changes if they make sense of course. Depending on the changes, some may need to be made in a 3.1 version but we'll see.

Hi! I'm glad to see this is still active after we talked about it 10 years ago 📌 Make Views use SqlEntityStorageInterface Needs work . Would you be able to provide the status of Views integration?

• 📌 Views integration Needs review looks done
• The docs → say it's not possible, but use the Search API
• There's a separate contrib module for it →

The views integration is done on v3 through a "companion" module (ie. sub-module provided with external entities that needs to be enabled) but needs improvements. So far, I use the view support of v3 on a production site and the only major problem I remember was column sorting not available on table display. But I don't have a good memory... I think there are also problems with multi-lingual support which is not complete in v3 yet (as you saw in other issues ;) ...there is still a PR that needs to be reworked/updated).

Regarding the documentation, it refers to v2 and it should be updated to tell that v3 does support views. In fact, it IS possible. I got inspired by the Search API module → . But still, keep in mind that it *may* not be efficient with REST services... while it can be relevant with other storage plugins (like external files or databases).

The separate module you saw is the one I did for v2, that I'm not really supporting it anymore (minor fixes) since I've integrated it to v3 (ie. supported in v3, no double work).

It appears I'm the only one who worked on that part and I would be grateful is I could get some help! I'm working on many things (and it's an understatement! :) ) and I have to prioritize my developments. Improving the views support is in my list but in the middle/bottom. {:-)

Thanks for the update! You've done an impressive amount of work here, and thanks for updating the Views doc page.

For my current project, we're starting to put together a proof of concept (PoC) for relying heavily on this module. So assuming that goes well, we should be able to help with stabilizing the issues that we're interested in, which includes:

• ✨ Make external entities translatable Needs work
• #3261988: Allow optional separate endpoint for query request on REST client plugin →
• ✨ Adding fields to add values in headers Needs review
• ✨ Support external entities related custom cache tag Needs review
• ✨ Authentication plugin system for use with the Rest external storage client Active
• 📌 Move automatic generation of path aliases to the pathauto submodule Active
• 🐛 Improve UX for deleting annotations/external entities Needs work
• ✨ what if entity is not found Needs review
• #3136194: Arguments in API endpoint path →
• ✨ Support Drupal's Entity Reference field type concept on external data source. Active
• 📌 Add basic test coverage Active

For now though, I think we should do a few things quickly; I can do these if you make me a maintainer:

• Deprecate the v2 Views module (which is no longer needed) in favour of the main module. (I'd also need maintainer access for this one.)
• Switch the release recommendation on the main project page from v2 to v3.
• Enabled automated testing in the project as I mentioned at 📌 Add basic test coverage Active .

Sounds good! Especially if I got some more help on v3! :-D
Regarding the module maintenance, I've asked Attiks → to get access and work on a v3. You should ask him to get access too.

Some of the issues you listed are already solved, some others aren't. When I started v3, it was because I needed many changes in v2 and the current maintainers could not handle those. So I got access to work on a separated v3 and left v2 to its current maintainers, to not break anything. That's why there are currently a v2 and a v3. Now v3 is quite mature, it might be time to discuss of the future of v2 with its current maintainers. Do they still need it or can they migrate to v3 while there are many breaking changes in the architecture...