Add option to handle json wrapped in array

Straight reroll just to remove fuzz.

The last submitted patch, 6: views_rest_support-2988391-06.patch, failed testing. View results →
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

New patch fixes error.

Actually, this change seem to resolve the tracking lockup bug.

OOps, here's the new patch.

patch 11 is incorrect.

I was able to get the example working as mentioned in the issue summary, despite the tracking issue

This is working, I had to go through the steps a couple times closely, see the pdf linked in the issue summary. (and also the youtube video is helpful)

Confirming that patch #8 works.

With the beta1 version of External Entities I can configure an endpoint of a Drupal 10 exposing nodes as a Rest Views display without patching it.

I can set up the one as explained in the video with fields, configure A rest endpoint and map the fields via simple mapping.

I also tried configuring the rendering of the full entity with a Views Rest display (so all fields) then I need to map via the Xpath mapper since the JSON has more layers. Also in this case I get a working external entity overview and detail pages.

I think that the fix to loadMultiple() in the Rest storage client in #3368304 has made this possible.

Closing this issue, please reopen if I am wrong :)

I'll try to review this soon.

Ok reviewed a bit more, the rebuild tracking is timing out.

I set up a pager , the page variable is:
page
the offset variable is:
offset
the number of items per page is:
items_per_page

I set the view in drupal to 50 items per page
exposed the option to change it

enabled full pager

I'll review this again soon

📌 indexing external entities - tracking locks up Needs review

external data source is Drupal 8
consumer is Drupal 10 external_entities module.

@pefferen,
please update the documentation relating to "External Entities I can configure an endpoint of a Drupal 10 exposing nodes as a Rest Views display without patching it."

https://www.drupal.org/docs/8/modules/external-entities/views →

As far as I see it, external entities cannot use views directly, only by using Search API. The usage of Views as a rest endpoint goes beyond the scope of this module.

However, having documentation on how to set it up could be very useful.

As far as I see it, external entities cannot use views directly, only by using Search API. The usage of Views as a rest endpoint goes beyond the scope of this module.

Well, I have to disagree... :) I've written the xnttviews → plugin and it lets me use all my external entities with views. I am using mostly external database sources or files but I have it also working on external entities coming from a REST service. It's not a big module and it is inspired by what search API does. It currently supports filters, contextual filters, sorting and pagination (I even added a "Fuzzy search" filter for my personal needs which works fine if the client supports a 'FUZZY' operator). It has still issues for sure but they can be fixed. Last week I had an issue with pagination when it was set to "display all" but I fixed it. I also noticed a problem with table columns that are not sortable but I'm quite sure I can investigate and fix it as well...

I think it should be integrated into the External Entity module as part of its core once xnttviews is more mature (needs automated tests, code cleaning/refactoring).

If you give it a try, please use the dev version as I did not create a new beta release yet with the latest fixes. You definitely should give it a try.

@pefferen , I think you've misunderstood the use case.

I have another Drupal site, in my case, it's Drupal 8. I've used the above mentioned patch and on my remote Drupal 8 website I am exposing the nodes as json data from a Drupal 'view'. The Drupal 8 site does NOT have the external_entities module installed.

I have configured this external endpoint as an external entity mapping in my Drupal 10 website that has the external_entities module installed.

My question to you, what is your documentation for doing this without the above mentioned patch?

How should this be set up so that paging the external source works and so that I avoid the search_api tracking timeouts?

Shouldn't there be documentation be made available for this?

@joseph.olstad, I had a look to your patch which looks fine. However, I would like to propose or more generic approach than the one proposed here. The only use case it currently solves is the results in an array (at first level) but there are other similar uses cases.

What if the array of objects is stored in a more complex structure? Then, a new patch will be needed...

For instance, consider these 2 REST endpoints where data objects are stored in the sub arrays:
in "artObjects" for https://www.rijksmuseum.nl/api/en/collection?format=json&key=fpGQTuED and in "results.data" for https://www.musabase.org/brapi/v1/germplasm/?page=0&pageSize=10

Proposed modification: instead of a checkbox, provide a text field labeled "JONPath to dataset (use when objects are not returned in a simple JSON list but rather in a more complex structure)" and then grab the data using JSONPath as the library is included with External Entities. Then, for the first example, the path would be "$.artObjects.*" and for the second "$.result.data.*". It would handle many more use cases without too much complexity. Maybe an example could be added as field description (for instance: "Ex.: if the REST service returns objects wrapped in a structure {..., "data": [{object1},{object2},..]}, use the JSON path "$.data.*".").

Something like that (for the query() part):

use JsonPath\JsonObject;

...

// $results would contain the parsed JSON structure.
if (!empty($this->configuration['data_jsonpath'])) {
  $json_object = new JsonObject($results, TRUE);
  $results = $json_object->get($this->configuration['data_jsonpath']);
}
return $results;
...

Note: for the load() part, e complex result structure would be managed through field mapping (JSONPath field mapper for instance).

Now that 🐛 Storage client should return empty array instead of NULL Fixed is in, the patch needs a re-roll, though I'm really not sure about if this is the right way. But given the earlier code and now with the chance of an empty, is this the right way?

@leymannx, that looks like an accurate reroll. I'm not sure if I understand what the implications are with the @guignonv suggestions in comment #32 .

Unfortunately, my client ran out of budget to pay me to work on this so I have no time to dig hard back into this for now.

Regarding my #32 comment, you may ignore it here as we are dealing with v2. However, I did the change I proposed in v3 and v3 supports load() method returning NULL.

To clarify, it is up to the community to decide how this issue could be solved for v2 and my previous proposal might bring too much changes for a stable v2. I hope it's clearer. :)