Optimize migrations, limit amount of API calls performed by the migrations

Created on 3 January 2024, about 1 year ago

Problem/Motivation

The module has a "watwat browser" which allows to pick which articles should be tracked by migrations ( located at /admin/content/watwat )
The articles that are chosen via that API browser, are stored in custom database table watwat_addtomigration, then everytime we run a migration those IDs are picked up to build migration URLs:
https://git.drupalcode.org/project/watwat/-/blob/2.0.x/src/Plugin/migrat...

The migration can be executed with provided PHP script: https://git.drupalcode.org/project/watwat/-/blob/2.0.x/scripts/WatwatMig...

The module builds 1 URL per article ID , so it can lead to a huge amount of API calls , slowing down the migrations in general.
The place where migration URLs are prepared can be seen here:
https://git.drupalcode.org/project/watwat/-/blob/2.0.x/src/Plugin/migrat...

WATWAT API can provide 10 articles per 1 API call:
https://www.watwat.be/api/v1/articles?id=2073,2070,144,148,245,310,329,3...
So this can limit amount of API calls performed by the migration.

Steps to reproduce

Debug the code in:
https://git.drupalcode.org/project/watwat/-/blob/2.0.x/src/Plugin/migrat...
When running migrations.

Proposed resolution

Rework the code to to build API URLs with 10 article IDs, instead of 1.
Quick dirty example:

$configuration['urls'] = [
  'https://www.watwat.be/api/v1/articles?id=2073,2070,144,148,245,310,329,330,331,332',
  'https://www.watwat.be/api/v1/articles?id=332,334,335,400,455,472,524,526,528,531',
  'https://www.watwat.be/api/v1/articles?id=1757,1765,1768,1811,1857,1893,2043,2081,2086'
];

Remaining tasks

Provide a MR.

User interface changes

N/A

API changes

N/A

Data model changes

N/A

Feature request
Status

Active

Version

2.0

Component

Code

Created by

🇵🇱Poland sandboxpl Poland 🇵🇱

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Production build 0.71.5 2024