Add support for migrate instrument (performance measurement)

Created on 16 January 2023, over 1 year ago
Updated 30 March 2023, about 1 year ago

Problem/Motivation

In Drupal 7 we could call the drush migrate-import command with a performance measuring instrument. It displays a table of time spent in each migration step.

This post offers my approach to how we could do it.

As the drush mim command has to be extended with --instrument argument, some simple changes are also in Migrate Tools module (see related issues section).

Example of the result table

command: drush mim blog --limit=5 --update --instrument

  • steps: A ordered number of measuring points within a single iteration (iteration means import of one record).
  • instrument: Instrument ID
  • count: How many times the instrument has been called within the migration run
  • time:total: Total time spent on a specific instrument
  • fraction: The same as Total time, but in percentage
  • min, max, avg: Min, Max, Average time spent on a specific instrument (mean)
  • range: A difference between min and max time
  • mid-range: A middle of range
  • offset: Average offset, a relative distance of avg (mean) time from mid-range. It reports how the values are distributed. If most values are above the mid-range, the offset is a positive number. It means there are a few extremely low values in the measurement.
  • status: A status of measuring instrument - still running or stopped

All time values are in milliseconds.

Steps to reproduce

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

✨ Feature request
Status

Needs work

Version

6.0

Component

Drush commands

Created by

πŸ‡¨πŸ‡ΏCzech Republic martin_klima

Live updates comments and jobs are added and updated live.
  • Needs tests

    The change is currently missing an automated test that fails when run with the original code, and succeeds when the bug has been fixed.

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Issue was unassigned.
  • Status changed to Needs review over 1 year ago
  • πŸ‡¨πŸ‡ΏCzech Republic martin_klima
  • πŸ‡³πŸ‡ΏNew Zealand quietone New Zealand

    Does anyone know the history about why this was not included?

    Migrate was added to core in #2125717: Migrate in core: patch #1 β†’ . There is one comment that mentions that instrument is not included #26 β†’ but there was no response.

  • πŸ‡³πŸ‡ΏNew Zealand quietone New Zealand

    I asked chx about the history in Slack. He said that at the time the goal was core migrations, which is not the same architecture as Migrate in D7. So, this feature wasn't in their minds at all.

  • πŸ‡ΊπŸ‡ΈUnited States mikelutz Michigan, USA

    I'm leaning towards pushing this to contrib. I realize it would be difficult to implement in a new contrib module, but migrage_tools has its own MigrateExecutable and drush commands so a lot of this could be done there. The only issue is they would have to override MigrateExecutable::import(), which they don't currently do, so in the future core changes there would also need to be implemented in tools.

    There's a few other issues here if we did do this in core, Wrapping SourcePluginBase::prepareRow() is useless, that's called from the actual source's prepareRow() where most of the heavy row preparation happens, but it wouldn't be counted here. You would want to wrap the actual $this->prepareRow() call in SourcePluginBase::next(). Wrapping the event calls could probably be done with two events with very high and very low weights instead of putting them in the executable. it needs tests, etc. While migrate_tools does have a set of druch commands, the core migration drush commands are in drush core, so if we did decide to do this here, we would want to add to the drush core commands, not the version in tools (well, both, probably) We will discuss again in the meeting tomorrow, but I'm leaning towards closing this, doing what we can in tools, and/or possibly adding additional hooks/events/something in core to allow this to be easier or more effective in contrib.

  • Status changed to Needs work about 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States benjifisher Boston area

    I am moving this issue from Drupal core to Migrate Tools. It will need a different implementation, so I am changing the status from NR to NW.

    We discussed this issue at two of the weekly migration meetings:

    Comment #9 summarizes the discussion. Let me repeat part of it: we will consider adding hooks or events to Drupal core in order to facilitate this feature, but first we would like to see how much of it can be implemented in a contrib module. We will also consider refactoring so that it is easier to override the core executable class.

Production build 0.69.0 2024