Allow batch_size to work with joinable map tables and highwater

Created on 25 July 2024, 5 months ago
Updated 28 August 2024, 4 months ago

Problem/Motivation

batch size is supposed to be an optimization option, but it breaks map joining and highwater. Given that all three are needed most on migrations with a large number of rows, explore what it might take to make them work better together.

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

Feature request
Status

Needs work

Version

11.0 🔥

Component
Migration 

Last updated 3 days ago

Created by

🇺🇸United States mikelutz Michigan, USA

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @mikelutz
  • 🇺🇸United States mikelutz Michigan, USA
  • 🇺🇸United States mikelutz Michigan, USA

    mikelutz changed the visibility of the branch 3463800-allow-batchsize-to to hidden.

  • Merge request !8928feat: initial test → (Open) created by mikelutz
  • Pipeline finished with Failed
    5 months ago
    Total: 166s
    #233947
  • Status changed to Needs work 4 months ago
  • 🇳🇮Nicaragua dinarcon

    Thanks for working on this. I tested the MR and I got a never ending migration, processing the same records over and over. I think that is caused because reading the highwater value to add the condition to the query only happens in the first run of the batch inside an if ($this->batch == 0) condition.

    The very first time that I run the migration, the highwater has not been set. So, the code from the MR keeps returning a query with LIMIT [BATCH_SIZE] OFFSET 0. Subsequent calls initializeIterator already have $this->batch greater than 0 so the code to add the highwater is never reached.

    If I force the migration to stop and run it again, the highwater condition is set, but it is never updated in subsequent calls initializeIterator because of the same reason.

    For this to work, I think the code that sets the highwater condition needs to be moved outside the if ($this->batch == 0) condition.

Production build 0.71.5 2024