On localize_server 3.0.x l10n_server_parser queue run is very slow

Created on 26 October 2023, about 1 year ago

Problem/Motivation

On DDEV when we try to run drush queue:run l10n_server_parser
parser seems very slow and takes hours to finish.

I tested it after installing localize server (main branch at commit 8df88fabdc5694751a5b9e97680c56a679a0466a
following the onboarding instructions

I see projects qetting parsed but each version can take several minutes.
eg.

[notice] Retrieving drupal-9.3.0-beta1.tar.gz for parsing.

took around 2min

It could be DDEV related as i didn't test another way (with a functioning ddev with enough resources and a good connexion).
My ddev runs with colima + mutagen

Steps to reproduce

install localize server ddev instance following https://gitlab.com/drupal-infrastructure/sites/localize.drupal.org
run
drush core:cron
drush l10n_server:scan
drush queue:run l10n_server_parser

πŸ› Bug report
Status

Active

Version

3.0

Component

Code

Created by

πŸ‡«πŸ‡·France ericdsd France

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @ericdsd
  • πŸ‡«πŸ‡·France ericdsd France
  • πŸ‡«πŸ‡·France ericdsd France

    Made some extra test on gitpod, initial parse from recreate.sh import ins approx. 5mins,
    but after that a regular
    drush queue:run l10n_server_parser
    that is not limited to a very shortlist of release takes forever.

    After 8 min running here is where we stand:

    gitpod@localizedrupalorg-web:/var/www/html$ drush queue:run l10n_server_parser
     [notice] Retrieving drupal-9.5.0-beta2.tar.gz for parsing.
     [warning] File /tmp/drupal-9.5.0-beta2.tar.gz already exists, deleting.
     [notice] drupal-9.5.0-beta2.tar.gz (7828 files, 10553 strings) scanned.
     [notice] Retrieving drupal-10.0.0-beta2.tar.gz for parsing.
     [notice] drupal-10.0.0-beta2.tar.gz (6783 files, 10075 strings) scanned.
     [notice] Retrieving drupal-9.3.22.tar.gz for parsing.
     [notice] drupal-9.3.22.tar.gz (7774 files, 10462 strings) scanned.
     [notice] Retrieving drupal-9.4.6.tar.gz for parsing.
    
  • πŸ‡©πŸ‡ͺGermany sanduhrs πŸ‡ͺπŸ‡Ί Heidelberg, Germany, Europe

    First time setup will take some time.
    recreate.sh currently downloads and parses 14 Drupal core releases.

    Just to get a feeling: there are currently ~170116 releases of Drupal core and contrib.
    If parsing each will take 2 min per release, it will take us ~230 days to recreate everything.
    But recreate.sh only parses Drupal core, which is much bigger than most of contrib.
    Nonetheless we are probably better off migrating things.

    Quickly looking at the releases since 2005, it's ~25 releases per day.
    With a peak of 146 published releases on 2020-06-03.

    We will have to support parsing ~150 releases per day on production :)

  • Status changed to Needs work about 1 year ago
  • πŸ‡«πŸ‡·France fmb PerpinyΓ , Catalonia, EU

    Thanks @sanduhrs. I understand we'd rather migrate l10n data. Now, regarding onboarding instructions, should we make this step optional, or rephrase it, for newcomers who might only work for a few hours during a sprint for instance?

  • πŸ‡«πŸ‡·France fmb PerpinyΓ , Catalonia, EU

    Make this step optional and improve documentation to explain how to import projects and strings.

Production build 0.71.5 2024