Optimize migrations, prevent unnecessary calls for non-existing data

Created on 3 January 2024, 11 months ago

Problem/Motivation

The module has a "watwat browser" which allows to pick which articles should be tracked by migrations at /admin/content/watwat
The articles that are chosen via that API browser, are stored in custom database table watwat_addtomigration, then everytime we run a migration those IDs are picked up to build migration URLs:
https://git.drupalcode.org/project/watwat/-/blob/2.0.x/src/Plugin/migrat...

The module does not clean up the watwat_addtomigration table, so the site might stay in a situation where it iterates over IDs that do not exist in WATWAT API anymore.

Example of existing article in the API: https://www.watwat.be/api/v1/articles?id=2073
Example of non-existing article: https://www.watwat.be/api/v1/articles?id=2073123123

Steps to reproduce

This will be hard to reproduce on fresh install, since it's related to content changes in WATWAT API , but basically the steps are:
1) Install WAT WAT API browser module
2) Navigate to /admin/content/watwat and add couple of articles to the migration mapping
3) Wait X amount of time until some articles are removed from the WAT WAT API
4) Run and debug migrations - observe that migration is using outdated IDs, and is querying API endpoints that do not provide any data

Proposed resolution

Provide a drush command that can be used to clean up watwat_addtomigration table. Such a drush command can go over IDs stored in the database and compare them against the API , if the article does not exist in the API anymore it can be safely removed from the mapping.

Remaining tasks

Propose a MR

User interface changes

N/A

API changes

N/A

Data model changes

N/A

Feature request
Status

Active

Version

2.0

Component

Code

Created by

🇵🇱Poland sandboxpl Poland 🇵🇱

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Production build 0.71.5 2024