Contrib.social

JSON pager "type: page" doesn't parse the last page

Open on Drupal.org →
Created on 19 November 2024, 11 months ago

Problem/Motivation

When using the pager for JSON data with type: page the last page doesn't get added to the next URLs.
See the code snippet below from data_parser/Json.php:

case 'page':
        if (NULL !== $selector_data && is_scalar($selector_data)) {
          // Just use 'page' as a default parameter key if not provided.
          $key = !empty($this->configuration['pager']['key']) ? $this->configuration['pager']['key'] : 'page';
          // Define the max page to generate.
          $max = $selector_data + 1;
          if (!empty($this->configuration['pager']['selector_max'])) {
            $max = $this->getSourceData($url, $this->configuration['pager']['selector_max']);
          }

          // Parse the url and replace the page param value and rebuild the url.
          $path = UrlHelper::parse($url);
          for ($page = $selector_data + 1; $page < $max; ++$page) {
            $path['query'][$key] = $page;
            $next_urls[] = Url::fromUri($path['path'], [
              'query' => $path['query'],
              'fragment' => $path['fragment'],
            ])->toString();
          }
        }
        break;

$selector_data is the current page number for the source while the selector_max parameter should define the maximum number of available pages.
The condition $page < $max in the for loop will stop the loop before reaching the max page.

Proposed resolution

Use $page <= $max to include the last page.

🐛 Bug report
Status

Active

Version

6.0

Component

Plugins

Created by

🇫🇮Finland pulakka

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Comment 11 months ago →
    🇺🇦Ukraine init90 Ukraine

    @pulakka Thanks! I can confirm that the problem exists.

    I use migration for recurring product imports from external API.

    How look API URLs process with the current codebase:

    Process data from https://API_URL/api/v1/products.json
Process data from https://API_URL/api/v1/products.json?page_no=2
Process data from https://API_URL/api/v1/products.json?page_no=3
...
Process data from https://API_URL/api/v1/products.json?page_no=23
Process data from https://API_URL/api/v1/products.json?page_no=24
Process data from https://API_URL/api/v1/products.json?page_no=25

    Number of imported products: 2527

    After the proposed correction:

    Process data from https://API_URL/api/v1/products.json
Process data from https://API_URL/api/v1/products.json?page_no=2
Process data from https://API_URL/api/v1/products.json?page_no=3
...
Process data from https://API_URL/api/v1/products.json?page_no=24
Process data from https://API_URL/api/v1/products.json?page_no=25
Process data from https://API_URL/api/v1/products.json?page_no=26

    Number of imported products: 2592

  • Merge request !116Fixed process of the last page by the 'page' pager type. → (Merged) created by init90
  • Comment 11 months ago →
    🇺🇦Ukraine init90 Ukraine
  • Pipeline finished with Failed
    11 months ago
    Total: 3683s
    #347251
  • Comment 11 months ago →
    🇫🇮Finland pulakka

    @init90 Thanks for the patch! Works well.

  • Status changed to Needs review 3 months ago
  • First commit to issue fork.
  • Pipeline finished with Skipped
    3 months ago
    #541353
  • Pipeline finished with Skipped
    3 months ago
    #541357
  • Comment 3 months ago →
    System Message
    • heddn → committed 92a41c4f on 6.0.x authored by init90 → 
      Issue #3488331 by init90, heddn, pulakka: JSON pager "type: page" doesn'...
  • Comment 3 months ago →
    heddn Nicaragua

    Thanks for the fix here.

  • Comment 3 months ago →
    heddn Nicaragua

    This causes tests to fail.

  • Comment 3 months ago →
    System Message
    • heddn → committed e1b6cd4b on 6.0.x 
      Revert "Issue #3488331 by init90, heddn, pulakka: JSON pager "type: page...
contrib.social Blog FAQ Discussions
Production build 0.71.5 2024