Move scraping logic to prepareRow()

Created on 6 February 2025, 18 days ago

Problem/Motivation

It appears that the scraping already occurs during the initialising process via either drush ms, drush mim or drush mr, whilst it should only scrape during the migrate import command.

Steps to reproduce

Run either of the drush status, migrate or rollback commands. All will scrape the URLs.

Proposed resolution

I believe we should shift the logic from initializeIterator() to prepareRow(), which would then only initiate the scraping at runtime of drush mim during the execution of the relevant row (URL) instead of scraping all URLs at once.

Remaining tasks

Review the change, create a patch/MR and test.

User interface changes

N/A

API changes

N/A

Data model changes

N/A

🐛 Bug report
Status

Active

Version

1.0

Component

Code

Created by

🇧🇪Belgium baikho Antwerp, BE

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

Production build 0.71.5 2024