Make demo spin-up fast

Related Slack thread starts here: https://drupal.slack.com/archives/C072BF486FN/p1728065173571019?thread_t...

The mobile experience is currently using a preinstalled database without additional recipes (build job here)

the workflow change would be:

- always use the preinstall artifact
- after extraction, display list of recipes
- user can pick or skip, if picked then they are applied via php in browser code
- automated login
- redirect to site

I’d like to get @pameeela’s input here before we implement anything. This would be a significant lift; the time might be better spent making the installer faster in Wasm.

To be fair, what is proposed is a few hours work versus the effort to improve batch execution performance at install in Wasm or normal environments.

Adding this to the summary:
And if we are going to ship a preinstalled database, why not have the build script load the demo pages during build to pre-warm the cache for a snappy experience out of the box?

@phenaproxima #4, in the Slack thread, the significant motivator to me was this:

My question:

Will spinning up a Drupal site with that many modules take more than a minute no matter how fast we make the install process?

Mglaman's answer:

If there’s 100+ batch steps it’ll always take forever.

I ran an informal timing this morning. See screenshot for graphs.

• The current Demo experience from click to a usable site without adding a recipe: 320 seconds to usable site.
• Future, where we prebuild the database and move the recipe selector to be an optional step after we hit usable site: 25 seconds to usable site.
• Wordpress Playground: 11 seconds to usable site.

I think we should do both. We should speed up Drupal's install so that everyone else can benefit. And we should pre-build the demo baseline to get to a usable site in less than 30 seconds, and make the recipe installations and configuration changes optional after we have a usable site.

Moving to the trial, as I don't think we should make drastic changes to the normal installer

I totally agree that we need to speed up the trial. Given that the current "installer" isn't doing anything that is blocking install, it makes sense to me to preload everything we can. As already noted, applying recipes and whatever other config changes can be done after install, so let's do it.

Assigning to @mglaman to:

• Change the trial to download the preinstalled artifact
• Cease generating the not preinstalled artifact

Possibly add some cache priming to the preinstalled artifact by visiting some pages

This could be hard. Maybe not. I guess we could visit a page and then truncate the render cache. The problem is any rendered output has incorrect outgoing URLs.

Well, let's not worry about that, then. The installer is the really slow thing here, so bypassing it will get us quite far. If we want to do cache warming, we can do it in a separate issue if it's tricky.

📌 Add a cache prewarm API and use it to distribute cache rebuids after cache clears / during stampedes Needs work and https://github.com/drush-ops/drush/issues/5724 would allow low-level Drupal caches (plugin discovery et al) to be pre-warmed before anyone visits a page.

Got no where. Drush 13 now uses Drupal core's \Drupal\Core\Database\Connection::createConnectionOptionsFromUrl method which breaks SQLite file paths. It always appends the root path now, which makes it impossible to use relative paths. So after merging 📌 Remove drupal_cms_patches and move internal development patches into a separate file at the repository root Active all mobile trials broke.

I run into the SQLite file path issue too

I fixed it manually be setting database to core/db.sqlite instead of the auto-generate drush path

$databases['default']['default'] = array (
  'database' => 'core/db.sqlite',
  'prefix' => '',
  'driver' => 'sqlite',
  'namespace' => 'Drupal\\sqlite\\Driver\\Database\\sqlite',
  'autoload' => 'core/modules/sqlite/src/Driver/Database/sqlite/',
);

The problem is that we're trying to not provide a settings.php file. I guess we could post-process the file and clean up the absolute path.

Okey, I will try updating the build-trial command

REPLACING
composer config --merge --json 'extra.drupal-scaffold.file-mapping' '{"[web-root]/sites/default/default.settings.php": {"append": "./scaffold/default.settings.php"}}'
WITH
composer config --merge --json 'extra.drupal-scaffold.file-mapping' '{"[web-root]/sites/default/settings.php": "./scaffold/default.settings.php"}'

No we don't want that change, by post processing I mean a `sed` on the generated `settings.php`

Okey, I understand you don't want to break the default trial.zip, only the trial-installed.zip should have that change

https://git.drupalcode.org/issue/drupal_cms-3478858/-/jobs/3149845

⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯
FAIL tests/cgi-install.test.js > install-site.phpcode > installs the site from the artifact

Expected: "{"message":"Performing install task (14 \/ 14)","type":"install"}"
Received: "{"message":"Performing install task (13 \/ 13)","type":"install"}"

Should we just change the expected to 13 instead of 14, or are we missing something ?

The fact we're seeing that and not "database exists" error means something else isn't write. Because it's trying to install and the step is missing since the database configuration is already set. If the sqlite database was there, it'd error about it being installed already.

It should respond with

AssertionError: expected '{"message":"<ul>\n<li>To start over, …' to deeply equal '{"message":"Performing install task (…'

Expected: "{"message":"Performing install task (14 \/ 14)","type":"install"}"
Received: "{"message":"<ul>\n<li>To start over, you must empty your existing database and copy <em>default.settings.php<\/em> over <em>settings.php<\/em>.<\/li>\n<li>To upgrade an existing installation, proceed to the <a href=\"\/update.php\">update script<\/a>.<\/li>\n<li>View your <a href=\"http:\/\/:\">existing site<\/a>.<\/li>\n<\/ul>","type":"error"}"

Okey, I will continue looking

And we will also need to remove that part in this case, I think

Yeah it should be that error. But at the same time, I was going to keep the interactive and automated install stuff, so that we can go back to different installation options for testing. I was thinking of having the build-trial command do this:

 find . -depth -type d -name tests -exec rm -r -f "{}" ';'
 
 # Create a trial archive without a database
 composer archive --dir=../public/assets --file=trial-uninstalled --format=zip
 
 # Install the site so that the trial is immediately available.
 vendor/bin/drush site:install --yes
 
 # Create the trial archive.
 composer archive --dir=../public/assets --file=trial --format=zip

Don’t add the new artifact to the GitLab artifacts, and it can be used locally only

I agree 👍

The task that get skipped is

    'install_download_translation' => [
      'run' => $needs_download ? INSTALL_TASK_RUN_IF_REACHED : INSTALL_TASK_SKIP,
    ],

You were right about not seeing "database exists" error

the path in CI is different

  'database' => '/builds/issue/drupal_cms-3478858/trial/build/web/core/db.sqlite',

Because in CI we are not using DDEV

Update

I ended up adding trial-uninstalled.zip to artifacts entry so test job can download it, and run the installation process.

the two tests cgi-interactive-install.test.js, cgi-install.test.js were using trial.zip, and now that we changed it's name to trial-uninstalled.zip I updated both jobs

thanks so much boulaffasae! Pipeline is green, which means we're kept the existing tests in tact. It'd be great if we added a new test like `cgi-test` or something which used the install trial and verified the login script works

so it extracts and

await runPhpCode(php, rootPath + '/public/assets/login-admin.phpcode')
  const [, text] = await doRequest(phpCgi, '/cgi/drupal')
  assertOutput(cgiOut, 'GET /cgi/drupal 200')
  assertOutput(cgiErr, '')

  expect(text).toContain(`<title>Home | ${siteName}</title>`)

but without the install steps like the existing one

@mglaman done.

I added a new file trial/tests/cgi-preinstalled.test.js with tests from the old one (I removed the install-site.phpcode part and updated the site name)

Awesome, thanks so much. I think we could merge this MR but keep the page open to expose recipe selection from the UI.

Thoughts on next steps:

- we need an "apply recipes" script, like the install and login script, which reads the recipes from the install parameters
- take the new pre installed test and run that script, verify recipe added
- add to the UI a way to select the recipes, a hard coded list to start, which provides values added to install parameters
- add a new script which can find the available recipes and dynamically list them in the UI, I think this can be a follow up

I can help land this part unless you'd like to try, boulaffasae

Yes, I will be happy to continue working on this under your lead

Working on the 1st task

- "apply recipes" script, like the install and login script, which reads the recipes from the install parameters

If I understand correctly, we need to set-up a new file apply-recipes.phpcode, which should be called in the test phase.

The file should apply a recipe (drupal_cms_blog for exemple)

Currently, recipes can be applied using a PHP Command (when I go to Browse projects and select a recipe it show me the below)

cd /var/www/html/web
/usr/bin/php /var/www/html/web/core/scripts/drupal recipe /var/www/html/drupal_cms_blog

checking the source code of the drupal scripts, I was able to write the below

use Drupal\Core\Recipe\RecipeCommand;

$stdErr = fopen('php://stderr', 'w');

set_error_handler(function (...$args) use ($stdErr, &$errors) {
    fwrite($stdErr, print_r($args, 1));
});

$flavor = file_get_contents('/config/flavor.txt');
$docroot = '/persist/' . $flavor;

chdir($docroot . '/web');

$class_loader = require $docroot . '/vendor/autoload.php';

$command = new RecipeCommand($class_loader);

$command->run($input, $output);

Am I doing this right @mglaman ? Thank you

Yes let's start with these two tasks before moving to the UI

- we need an "apply recipes" script, like the install and login script, which reads the recipes from the install parameters
- take the new pre installed test and run that script, verify recipe added

The PHP script looks right. Good call on instantiating and calling RecipeCommand directly. I believe Symfony provides an ArrayInput class so we can easily pass the input parameters. We can pass a BufferOutput (named something like that) to catch the output and verify the outcome. Or to catch progress to pass up to the UI

I'm running into a small issue

Uncaught (in promise) TypeError: asyncifyStubs.posix_spawnp is not a function
    at _posix_spawnp (php-worker.mjs:9:312063)
    at php-worker.mjs.wasm:0x5dca76
    at php-worker.mjs.wasm:0x8b77e6
    at php-worker.mjs.wasm:0x6d8005
    at php-worker.mjs.wasm:0x39a346
    at php-worker.mjs.wasm:0xb248ab
    at ret.<computed> [as dynCall_vii] (php-worker.mjs:9:379933)
    at invoke_vii (php-worker.mjs:9:695383)
    at php-worker.mjs.wasm:0x726849
    at php-worker.mjs.wasm:0x2fa550
_posix_spawnp @ php-worker.mjs:9
$func4294 @ php-worker.mjs.wasm:0x5dca76
$func6680 @ php-worker.mjs.wasm:0x8b77e6
$execute_ex @ php-worker.mjs.wasm:0x6d8005
$zend_execute @ php-worker.mjs.wasm:0x39a346
$dynCall_vii @ php-worker.mjs.wasm:0xb248ab
ret.<computed> @ php-worker.mjs:9
invoke_vii @ php-worker.mjs:9
$zend_eval_stringl @ php-worker.mjs.wasm:0x726849
$zend_eval_string @ php-worker.mjs.wasm:0x2fa550
$dynCall_iiii @ php-worker.mjs.wasm:0xb2450a
ret.<computed> @ php-worker.mjs:9
invoke_iiii @ php-worker.mjs:9
$pib_run @ php-worker.mjs.wasm:0x2fa73e
ret.<computed> @ php-worker.mjs:9
Module._pib_run @ php-worker.mjs:9
ccall @ php-worker.mjs:9
_run @ PhpBase.mjs:186

This is the code I'm using

...
$command = new RecipeCommand($class_loader);

$input = new ArrayInput(['path' => $docroot . '/web/recipes/drupal_cms_blog']);
$output = new BufferedOutput();

$command->run($input, $output);

print json_encode([
  'message' => $output->fetch(),
  'type' => 'recipes',
], JSON_THROW_ON_ERROR) . PHP_EOL;

RecipeCommand will just prevent us from getting the progress and other informations (it use it's own output buffer - symfony)

Do you think executing batch directly in our .phpcode file would be much better ?

<?php

use Drupal\Core\Recipe\RecipeRunner;

$stdErr = fopen('php://stderr', 'w');

set_error_handler(function (...$args) use ($stdErr, &$errors) {
fwrite($stdErr, print_r($args, 1));
});

$flavor = file_get_contents('/config/flavor.txt');
$docroot = '/persist/' . $flavor;

chdir($docroot . '/web');

$class_loader = require $docroot . '/vendor/autoload.php';

use Drupal\Core\Recipe\Recipe;

$recipe = Recipe::createFromDirectory($docroot . '/web/recipes/drupal_cms_blog');

foreach (RecipeRunner::toBatchOperations($recipe) as $operation) {
$batch['operations'][] = $operation;
}

print json_encode([
'message' => $output->fetch(),
'type' => 'recipes',
], JSON_THROW_ON_ERROR) . PHP_EOL;

Using batch operations, I was able to install the blog recipe.

I see some small issues in console on json_decode level that I might need to fix before pushing the test

use Drupal\Core\DrupalKernel;
use Symfony\Component\HttpFoundation\Request;
use Drupal\Core\Recipe\Recipe;
use Drupal\Core\Recipe\RecipeRunner;

$stdErr = fopen('php://stderr', 'w');

set_error_handler(function (...$args) use ($stdErr, &$errors) {
    fwrite($stdErr, print_r($args, 1));
});

$flavor = file_get_contents('/config/flavor.txt');
$docroot = '/persist/' . $flavor;

chdir($docroot . '/web');

$class_loader = require $docroot . '/vendor/autoload.php';

$request = Request::create("/cgi/$flavor/", 'GET', [], [], [], [
  'HTTP_HOST' => $install_params['host'] ?? 'localhost',
  'REMOTE_ADDR' => '127.0.0.1',
  'SCRIPT_FILENAME' => "/persist/$flavor/web/index.php",
  'SCRIPT_NAME' => "/cgi/$flavor/index.php"
]);
$kernel = DrupalKernel::createFromRequest($request, $class_loader, 'prod');
$kernel->boot();
$kernel->loadLegacyIncludes();

$container = $kernel->getContainer();
$container->get('request_stack')->push($request);

$container->get('module_handler')->loadAll();

/** @var \Symfony\Component\HttpFoundation\Session\Session $session */
$session = $container->get('session');
$session->start();
$request->setSession($session);

$recipe = Recipe::createFromDirectory($docroot . '/web/recipes/drupal_cms_blog');

foreach (RecipeRunner::toBatchOperations($recipe) as $operation) {
  $batch['operations'][] = $operation;
}

foreach ($batch['operations'] as $operation) {
  [$function, $arguments] = $operation;

  // Execute each batch operation directly
  try {
      call_user_func_array($function, $arguments);
  } catch (Exception $e) {
    print json_encode([
        'message' => $e->getMessage(),
        'type' => 'error',
    ], JSON_THROW_ON_ERROR) . PHP_EOL;
  }
}

print json_encode([
  'message' => 'Beginning install tasks',
  'type' => 'recipes',
], JSON_THROW_ON_ERROR) . PHP_EOL;

I used the UI for testing locally, I think I'm only missing the validation code and I should have all tasks ready

Forgot to follow up on the issue

Do you think executing batch directly in our .phpcode file would be much better ?

I think so, because as you've done we can bubble progress messages to the UI

Hi, I added a new recipe.phpcode

https://git.drupalcode.org/project/drupal_cms/-/merge_requests/146/diffs#diff-content-bd7f5099028c3169a85a30babfd19ac150381f99

thanks so much @boulaffasae! assigning to myself, I'm going to fiddle around a little bit tomorrow

I do not have the bandwidth to work on this.

I will continue and keep you up to date

I updated the tests to include the new recipe.phpcode.

The recipe seems to be working fine, yet I have a small issue

+ <link rel="canonical" href="http://localhost:3000/cgi/drupal/">
+ <link rel="shortlink" href="http://localhost:3000/cgi/drupal/">
+ <meta name="referrer" content="unsafe-url">
+ <meta name="rights" content="Copyright &copy;2024 All rights reserved.">
+ <meta property="og:site_name" content="Drupal CMS">
+ <meta property="og:type" content="website">
+ <meta property="og:url" content="http://localhost:3000/cgi/drupal/node">
+ <meta name="twitter:card" content="summary_large_image">
+     <title>| Drupal CMS</title>

title is missing title.

  title: '[node:field_seo_title|node:title] | [site:name]'

This is the recipe configuration

Update the code to use drupal_cms_blog for now, I also refactored the message to use the default from RecipeRunner

It output the below for now, but we can fix that later

Installed <em class="placeholder">Better Exposed Filters</em> module.

DONE:
- we need an "apply recipes" script, like the install and login script
- take the new pre installed test and run that script

TODO:
- reads the recipes from the install parameters
- verify recipe added
- add to the UI a way to select the recipes, a hard coded list to start, which provides values added to install parameters
- add a new script which can find the available recipes and dynamically list them in the UI, I think this can be a follow up

DONE:
- reads the recipes from the install parameters
- verify recipe added

TODO:
- add to the UI a way to select the recipes, a hard coded list to start, which provides values added to install parameters
- add a new script which can find the available recipes and dynamically list them in the UI, I think this can be a follow up

DONE:
- add to the UI a way to select the recipes, a hard coded list to start, which provides values added to install parameters

TODO:
- add a new script which can find the available recipes and dynamically list them in the UI, I think this can be a follow up

I am sorry to be the bearer of bad news, but the WebAssembly trial will not happen (see 📌 Remove the WebAssembly trial Active ). It will be a hosted trial, so this issue should probably be closed.

Whether Web Assembly or not, I hope that the Drupal trial experience will load for people in 30 seconds or less.

darren oh → made their first commit to this issue’s fork.

darren oh → changed the visibility of the branch 0.x to hidden.

What exactly is happening here? I'm confused by the nature of these changes, and the reopening of this very old issue, which was associated with the defunct WebAssembly trial.

I don't mean to sound like a stick-in-the-mud here, but I don't think we should be working on the installer in this issue because it's hard to track. So I'm closing this again. If there's something you'd like to change in the installer, let's open a totally new issue to talk about it (with a fleshed-out issue summary) and figure out what is needed. (Fair warning that I'm generally resistant to installer changes; it is extremely opinionated and it took a long time to get it working the way it does.)

I want to thank everyone who contributed to this issue. I have been following it since it was opened because it had applicability beyond the WebAssembly trial experience. For those who want to continue working on fast spin-up, I have opened ✨ Add ability to skip starter recipe if it was preinstalled Active .

darren oh → closed merge request !348