Fetch from drupal.org API

Have you considered the list endpoint for fetching an individual item? Eg. https://www.drupal.org//api-d7/node.json?nid=2538674 →

Funny there are 2 possible calls to fetch a node :-(

@rp7 so is this now solved using your call? (my knowledge is fading)

No need for the double slashes //

https://www.drupal.org/api-d7/node.json?nid=2538674
https://www.drupal.org/api-d7/node.json?nid=3049317

out of curiosity and for troubleshooting I'd like to see a working example configuration for this ^^

I'm waist deep in external_entities right now, debugging is a bit challenging but I got a new exception that might help. Combination of patches involved. An example submodule with a working configuration would be an asset for sure.

I have not tested this with external entities, but the Drupal.org API uses Restfull webservices. the problem with the URL is the '.json' extension to the URL. External entities tries to fetch the detail page by adding the ID to the end of the endpoint URL. With the JSON extension this will fail.

If you leave out the extension and instead provide the JSON accept header, as per the documentation → you can have one URL for both the list and detail view. You can prove the header by using the API header fields.

I'm working on a 3.x plugin.

Thank you for that suggestion @pefferen. I was able to make a small plugin extending RestClient to make adjustments.

The listing and single item are working. Filters can be added to the parameters.

Hmm I got my wires crossed.. I had 2.x installed because I'm using other modules that depend on this version.

However I just tested the patch and it's working with 2.x.

Great! Just a note that should not impact your work on a storage client plugin but you should be aware of, just in case, there will be "soon" (July) a breaking change on the "field mapper plugin" part. Again, it's just to make sure you aware of that but I don't think it would impact your work since you're working on the field storage part. :)

I really appreciate that note.

I have been poking at this module for a while, looking forward to the day it's plug and play and I can quickly implement some features I have in mind.

This was quite a big breakthrough for me, I'm excited to keep experimenting with what I can do now. Going to look into using annotations with XNTT Manager and displaying XNTT fields on content pages.

Is there a better way to follow development than scanning the issue queue? Sounds like V3 is very close.

How about all the dependencies like XNTT Manager? Do you need help updating those?

We should continue this sub-topic under 📌 [META] Roadmap for v3.0 Active . ;)

I documented working config for 3.x using just the Rest client. Feel free to copy any of this to the official docs.

https://www.drupalarchitect.info/articles/configure-external-entities-3x...

TODO: Document how to use this in a search api view.

I have a search api view working, but it takes forever to index.

I installed the HTTP Client Log module, and I can see that it's doing a query for each item. This is not necessary because all the data is included in the listing results.

We need a way to tell it to use the data from the list instead of rerunning each item individually.

Also, the max count is not really usable for this API. We need to parse the url in the "last" key and get the page parameter. I put in 60000, which is more than the total number of modules and that seems to work, but can add extra pages to the default external entity list.

It takes quite a while to build the tracking info, 2+ hours to fetch 1040 pages of 50 items. Only including nid and title in the index. I think most of this time is just waiting for the api, which seems to take about 10 seconds per page.

After the search tracker is built, it runs the index and loads each external entity item. The data is already cached, so it should load it from there instead. How can I customize that?

Looks like XNTT Manager was able to set up a synchronization schedule, but that's not available for v3 yet.

I'm thinking we need more caching options to store the results per page, then on the entity store the page it was found on to look it up from the cache? Or build the single entity cache with the list results?

Will annotations help? Like when the entity is created, does it store the data for the item found in the list?

I looked into how Project Browser is pulling info, and found that it's using a new Drupal.org JSON API endpoint:

https://drupalorg.prod.cluster.drupalsystems.org/jsonapi

Used in: https://git.drupalcode.org/project/project_browser/-/blob/2.0.x/src/Plug...

The new official endoint is: https://www.drupal.org/jsonapi →

https://www.drupal.org/project/project_browser/issues/3462173 📌 Use the production URL for Drupal.org endpoint Fixed

It works with v3 beta 1 from my side. Please test and confirm or report problems here.

Excellent, I may find time to test this weekend. Thanks for all the updates.

I created a submodule, external_entities_drupalorg, which has config for the Project Browser JSONAPI endpoint for modules, and the drupal.org REST endpoint for issues.

This uses the Rest and JsonApi StorageClients, no custom plugins needed.

So far, there is just a body field, to demonstrate how to load data from the api.

Since the config is in the optional dir, the module can be uninstalled and you can modify the config as your own.

I used config_devel to keep the module's config up to date as I made changes. Recommended if anyone makes adjustments to this in the future.

Creating custom plugins could make the DX a little smoother by hardcoding things that are configurable in the UI, and exposing other options that are specific to these apis.

I am also looking into creating a custom plugin for Project Browser as a StorageClient. One thing I noticed it has is a total results count, which is very handy.

One thing to consider when trying to make listings or sync all items is that the Project Browser's module list endoint has a reasonable amount of items, but the issues endpoint could take a huge amount of time to sync, so probably isn't a good idea.

Looks like a good idea/approach (I had a quick look into the PR). Just a comment for next time: it's better to separate into 2 distinct PR the one for this issue and the one regarding the views ( 📌 Views integration Needs review ) to allow distinct changes and commits for the follow up of the code (unless issues are closely related of course).

I'll test when I got time and merge. But there's no hurry right?

I tested the example and adjusted a couple of things in the configs you made @mortona2k. Nice work, thank you for your help on that! I also found a couple of issues related to that example in External Entity storage and JSON:API client that were fixed at the same time.
Committed with Views integration by commit 2538706.
As usual, feel free to reopen the issue if needed.

Automatically closed - issue fixed for 2 weeks with no activity.