Bulk Pattern Updates Fail on Large Databases

Created on 13 September 2023, about 1 year ago
Updated 16 October 2023, about 1 year ago

Problem/Motivation

The implementation of the bulk update process for pattern block content as of the Beta 7 release loads all layouts and related entities to iterate through them individually searching for blocks with available updates. To date this has worked ok, but on sites with large databases and many entities and layouts to load and iterate through, we're seeing out of memory errors. A revamped solution to run these updates through batch processing may be needed for these situations.

Proposed resolution

Reimplement bulk pattern content updates through a batched operation.

Remaining tasks

User interface changes

  • New Drush command and options for batch processing updates

API changes

New Batch Manager and Worker Services and Implementation Model

Two new batch manager services have been implemented to manage and execute batched operations. For each of these services, much of the implementation work has also been separated out into "batch worker" classes that get instantiated for the execution of each batching pass. This offers several benefits:

  • The primary work for each batch operation is much more easily tested as opposed to all static methods
  • Service dependencies may be injected into these workers
  • Implementing a base class for these types of workers allows for consolidation of a lot of redundant functionality
  • Additional batch operations could be implemented later by writing new workers

New Layout Discovery Batch Manager/Worker

This batch manager (LayoutDiscoveryBatchManager) and worker (LayoutDiscoveryBatchWorker) pair handle the loading of entities to identify layouts for processing. These discovered layouts are then loaded into a queue for subsequent processing via either batch process or typical execution of the Queue API. Unless explicitly disabled, a secondary batch operation using the LayoutUpdateBatchManager is queued for immediate execution to process all queued layouts.

New Layout Update Batch Manager/Worker

This batch manager (LayoutUpdateBatchManager) and worker (LayoutUpdateBatchWorker) pair handles processing of the items in the patternkit_layout_updates queue via a batch operation.

New Queue and QueueWorker: LayoutUpdateQueueWorker (patternkit_layout_updates)

  • This queue worker processes items expected to be queued by the new layout discovery batch service (LayoutDiscoveryBatchManager) and runs the update process for all patterns discovered within a layout designated in the queue item.
  • This queue may be run as part of a batch operation via LayoutUpdateBatchManager where the related batch worker (LayoutUpdateBatchWorker) will load and execute this queue worker plugin on each queue item.

New Drush Command: drush patternkit:batch-update

This new Drush command executes the bulk update process using the new batching services. It offers the same filter options as the existing drush patternkit:libUpdate command with the addition of the following:

  • --revisions: Set which revisions should be loaded for discovery.
  • --no-run-queue: If all discovered layouts should not be processed yet, this option allows for loading them into the queue and leaving them there for later processing.

Data model changes

None.

🐛 Bug report
Status

Fixed

Version

9.1

Component

Module Core

Created by

🇺🇸United States slucero Arkansas

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @slucero
  • 🇺🇸United States slucero Arkansas
  • 🇺🇸United States slucero Arkansas
  • 🇺🇸United States slucero Arkansas
  • 🇺🇸United States slucero Arkansas

    For testing, any existing pattern content could be altered to register as needing an update using the following command:

    drush sqlq "UPDATE pattern_revision AS revision SET revision.schema=NULL, revision.template=NULL, revision.hash=NULL WHERE revision.revision IN (SELECT revision FROM pattern_revision)" 
    

    After running this command, the new Drush update commands should find content to update during scans:

    drush patternkit:batch-update

  • Status changed to Needs review about 1 year ago
  • Open in Jenkins → Open on Drupal.org →
    Core: 9.5.x + Environment: PHP 7.4 & MySQL 5.7
    last update about 1 year ago
    359 pass
  • @slucero opened merge request.
  • 🇺🇸United States slucero Arkansas

    I've documented the follow-up task for adding an administrative UI for these batch operations in Add Administrative UI for Batch Updating Postponed .

  • Open in Jenkins → Open on Drupal.org →
    Core: 9.5.x + Environment: PHP 7.4 & MySQL 5.7
    last update about 1 year ago
    359 pass
  • 🇺🇸United States slucero Arkansas

    After reviewing this further, I've determined that the query in #5 won't actually cause the pattern blocks to register an update being needed since the hash will get recalculated from the same schema and template and see no change. Instead, the query below should properly mark all pattern revisions in a way to trigger update detection:

    drush sqlq "UPDATE pattern_revision AS revision SET revision.hash='ALTERED', revision.schema='', revision.template='', revision.version='OLD'"
    
  • Open in Jenkins → Open on Drupal.org →
    Core: 9.5.x + Environment: PHP 7.4 & MySQL 5.7
    last update about 1 year ago
    359 pass
  • Open in Jenkins → Open on Drupal.org →
    Core: 9.5.x + Environment: PHP 7.4 & MySQL 5.7
    last update about 1 year ago
    359 pass
  • Open in Jenkins → Open on Drupal.org →
    Core: 9.5.x + Environment: PHP 7.4 & MySQL 5.7
    last update about 1 year ago
    359 pass
  • Open in Jenkins → Open on Drupal.org →
    Core: 9.5.x + Environment: PHP 7.4 & MySQL 5.7
    last update about 1 year ago
    359 pass
  • Open in Jenkins → Open on Drupal.org →
    Core: 9.5.x + Environment: PHP 7.4 & MySQL 5.7
    last update about 1 year ago
    359 pass
  • Open in Jenkins → Open on Drupal.org →
    Core: 9.5.x + Environment: PHP 7.4 & MySQL 5.7
    last update about 1 year ago
    359 pass
  • Open in Jenkins → Open on Drupal.org →
    Core: 9.5.x + Environment: PHP 7.4 & MySQL 5.7
    last update about 1 year ago
    359 pass
  • Open in Jenkins → Open on Drupal.org →
    Core: 9.5.x + Environment: PHP 7.4 & MySQL 5.7
    last update about 1 year ago
    359 pass
  • 🇮🇳India minsharm India

    Retested the issue and result looks good to me. Validated below scenarios on D9 as well as on D10.

    Note : Before performing below scenarios, it is important to note that there are a total of 6 Patternkit blocks created on the website.

    Scenario 1: Validate Bulk Pattern Update for All Patterns

    In this scenario, the goal is to validate a bulk pattern update for all existing patterns. The following steps be followed:

    1) Run the following command to update existing pattern content that can be altered and register them for an update:

    drush sqlq "UPDATE pattern_revision AS revision SET revision.hash='ALTERED', revision.version='OLD'"

    2) After executing this command, the next step is to run a Drush update command that should find content to update during scans. Use the following command:

    drush patternkit:batch-update

    Results:

    1. The bulk of patterns was updated successfully without encountering any errors during the process.
    2. After the update, it was observed that no updates are available for any of the nodes. In the debugger output, "update_available" was found to be set to "false" for all the nodes.

    Scenario 2: Validate Bulk Pattern Update for All Patterns, Including Temporary (Draft) Versions of Layouts

    In this scenario, the objective is to validate a bulk pattern update for all patterns, including temporary (draft) versions of layouts. The steps for this scenario are as follows:

    1) Create Node 1 and Node 2.

    2) Run the following SQL command to update existing pattern content that can be altered and register them for an update:

    drush sqlq "UPDATE pattern_revision AS revision SET revision.hash='ALTERED', revision.version='OLD'"

    3) Clear the cache.

    4) Before running the batch command, navigate to the layout tab for Node 2. At this point, a temporary draft version of the layout is created.

    5) In the layout tab for Node 2, add a new block that will not be available for update. This is expected because it was added after the pattern alteration command.

    6) Run the batch-update command and clear the cache.

    Results:

    1. A total of 7 blocks have been updated, including one temporary layout block for Node 2. Please note that the block added in step 4 is not updated since it was added after the pattern alteration command.

    Check the JSON form for the Patternkit (PK) block for both nodes.

    Results for JSON Form:

    1. All patterns, including temporary (draft) ones, have been updated for all nodes and none of the patterns are now outdated.
    2. The behavior is now consistent and the temporary layout blocks are being updated to match what is displayed in the logged message in the terminal.
    3. I am not seeing the "You have unsaved changes" message for all the nodes after using the new batch-update command, as long as I don't change anything in the PK Block or open the Layout tab for any of the nodes before using the batch-update command. In the above scenario, the popup will be displayed solely for node 2 because I had opened the Layout tab for node 2 prior to executing the batch update command.

    Now, repeat the same process again to verify that the pattern added in step 4 above should be updated:

    1. Run the pattern alteration command.
    2. Run the batch update command.

    Results:

    1. A total of 9 blocks have been updated.
    • 6 were the blocks added initially.
    • 1 was the block added in step 4 above.
    • 2 were the blocks on temporary layouts for Node 2, as there are a total of 2 PK blocks present on Node 2's saved layout.
  • Status changed to RTBC about 1 year ago
  • 🇺🇸United States slucero Arkansas
  • Open in Jenkins → Open on Drupal.org →
    Core: 9.5.x + Environment: PHP 7.4 & MySQL 5.7
    last update about 1 year ago
    359 pass
    • slucero committed a03dd2fb on 9.1.x
      Issue #3387144 by slucero, minsharm: Bulk Pattern Updates Fail on Large...
  • Status changed to Fixed about 1 year ago
  • 🇺🇸United States slucero Arkansas

    Merged for inclusion in the Beta 8 release: 🌱 Patternkit Beta 8 Release Plan Active

  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024