Contrib.social

JSON API - Stop when data is empty

Open on Drupal.org →
Created on 22 July 2024, 4 months ago
Updated 13 September 2024, 2 months ago

Problem/Motivation

There is a setting for Estimated total number of entities, that will end the query when surpassed.

But if this number is too high, it will keep querying the next page even though results are empty.

If it's too low, the entity list won't show all items.

Steps to reproduce

Configure a JSON API endpoint for drupal.org modules.

Use this endpoint from project browser 2.x:
https://drupalorg.prod.cluster.drupalsystems.org/jsonapi/node/project_mo...

Proposed resolution

When the data key in the results are empty, stop the query.

Remaining tasks

Document how things work.

Seems like if I clear the cache and go to the entities list, it will re-fetch all items.
But on the first load, it will show all items on the first page (with estimated items at 10,000).
On refresh, it looks like it's doing a query by page count, and only showing the returned items?

Is the estimated count a hard limit? If there are more items, will it keep going?

I only need to sync all items the first time, then the individual sync is fast enough to update and cache on load.

Do annotations depend on all items being synced? Since I am referencing modules, I could just load them when referenced, but the autocomplete annotation field needs to contain all items for that to work. I am thinking of doing one full sync for just the title, machine name, and nid, then putting an annotation on that type to fetch the full data. Or would the multiple storage feature be better?

User interface changes

API changes

Data model changes

✨ Feature request
Status

Active

Version

3.0

Component

Code

Created by

🇺🇸United States mortona2k Seattle

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @mortona2k
  • Comment 4 months ago →
    🇫🇷France guignonv

    I want(ed) to implement a kind of "smart" entity count for endpoint missing a "total count". I also wanted to support endpoint using "previous/next" paging system rather than numbered pages. I didn't do it yet because I don't rate that as high priority tasks. If you look into the REST storage client, I put some @todos for that. I hope to have time to handle those by the end of this year but with no guarantees as I'm also involved in several other (non-Drupal) projects.

    I won't have time to investigate the issue you report but it might be a bug though, because it should stop if there's nothing. If you provide a fix, I can review the code and merge it however. :)

  • Comment 4 months ago →
    🇺🇸United States mortona2k Seattle

    It might be worth looking at how Migrate Plus does it: https://www.drupal.org/project/migrate_plus/issues/2640516 📌 Support paging through multiple requests Needs work

  • Comment 2 months ago →
    🇫🇷France guignonv

    I plan to implement something similar for beta 2, probably in November/December unless I got other priorities coming.

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024