While applying updates during cron, put the site into maintenance mode

The error is Unexpected changes were detected in composer.lock, which indicates that other Composer operations were performed since this Package Manager operation started. This can put the code base into an unreliable state and therefore is not allowed. from LockFileValidator. I tried to debug it further and found that it's actually failing on StatusCheckEvent in LockFileValidator because for some reason the stored active lock file hash is not same as current active lock file hash during StatusCheckEvent.

Thanks for the write-up in #5, that definitely helps. 👍

It's fascinating that only a single Build test is failing and every other test is passing, including the functional ones. That pretty much must mean that something is depending on/reacting to this line of your diff:

$state->set('system.maintenance_mode', TRUE);

Do tests also fail in the same way if you set any other state key-value pair? Or does it only happen for this one specific key-value pair?

P.S.: How did you debug it? Not in the Build test itself I suspect, but by executing cron on your local development environment?

I haven't tried to set any other state key-value pair. I will try that and see if it fails the same way or not.

I debugged it by looking at the page contents of the site by using file_put_contents() before $this->assertExpectedStageEventsFired(CronUpdateStage::class); in \Drupal\Tests\automatic_updates\Build\CoreUpdateTest::testCron and I see this error message on Drupal\package_manager\Event\StatusCheckEvent Unexpected changes were detected in composer.lock, which indicates that other Composer operations were performed since this Package Manager operation started. This can put the code base into an unreliable state and therefore is not allowed. and we can see it fails on StatusCheckEvent as I have attached the event name before the error message in the validator.

I discussed this with @tedbow in scrum and he suggested some ways to try to find the cause of it.

I see. Thanks for clarifying!

Good luck finding the root cause! 💪😄

• I commented $state->set('system.maintenance_mode', TRUE); and it passes. I also tried to change it set it to false i.e. $state->set('system.maintenance_mode', FALSE); and it also passes. So, it's something to do with enabling maintenance mode.
• I also tried to set a different state key-value pair and it doesn’t fail.
• As we can see in the diff that the only values which are different are 'content-hash' and 'version'. The 'version' for stored active lock file is '9.8.0' (installed version) where as for current active lock file it is '9.8.1'(version to update). So @tedbow and I thought maybe cron is running a update again while doing a update, but today I debugged it to see if that's the case but turn out it's not.

Searching for system.maintenance_mode reveals:

• \Drupal\automatic_updates\Controller\UpdateController::onFinish() interacts with it (sets it to FALSE), let's verify that that is not causing this
• \Drupal\automatic_updates\Form\UpdateReady::submitForm(): same thing

… but my #1 suspect is \Drupal\system\Access\CronAccessCheck::access() which has:

    elseif (\Drupal::state()->get('system.maintenance_mode')) {
      \Drupal::logger('cron')->notice('Cron could not run because the site is in maintenance mode.');
      return AccessResult::forbidden()->setCacheMaxAge(0);
    }

😳😵

This would mean that Drupal thinks Cron should never run in maintenance mode, but we are trying to install updates DURING CRON, and installing updates implies changing code, which requires maintenance mode.

This seems like a very interesting chicken-and-egg situation 🤔

@kunal.sachdev Can you please check (and hopefully refute rather than confirm) my #1 suspect? 🙏

I checked it and found that \Drupal\system\Access\CronAccessCheck::access() runs after we set maintenance_mode to TRUE in automatic_updates_cron().

Okay. So that rules out my theory in #10 (although it still represents a real problem in non-test scenarios). Good luck finding the root cause! 🧙🏽‍♂️

Ok, so tried to debug it even further and found that it might not be a problem of LockFileValidator because if I disable this validator then also it's failing the same way (i.e logging events till pre-apply and not going further) although I don't see the error message from LockFileValidator on the site, so I think it's something else that's stopping it from calling \Drupal\package_manager\StageBase::postApply.

I think I found why the postApply is not running. In Drupal\automatic_updates\CronUpdateStage::triggerPostApply it's failing while executing

    $url = Url::fromRoute('automatic_updates.cron.post_apply', [
      'stage_id' => $stage_id,
      'installed_version' => $start_version,
      'target_version' => $target_version,
      'key' => $this->state->get('system.cron_key'),
    ]);
    $url = $url->setAbsolute()->toString();

which is basically calling \Drupal\automatic_updates\CronUpdateStage::handlePostApply
because Site is under maintenance mode.This is also being logged see

    $curl = curl_init($url);
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
    $response = curl_exec($curl);
    $status = curl_getinfo($curl, CURLINFO_RESPONSE_CODE);
    if ($status !== 200) {
      $this->logger->error('Post-apply tasks failed with output: %status %response', [
        '%status' => $status,
        '%response' => $response,
      ]);
    }
    curl_close($curl);

and when I tried to see the $status and $response it's 503 and

Site under maintenance
Drupal is currently under maintenance. We should be back shortly. Thank you for your patience.

respectively.

We discussed this during yesterday's meeting. We realized we'll need to explicitly grant access to this route during maintenance mode, but also protect it to avoid anonymous users being able to hit it, so we need:

options:
    _maintenance_access: TRUE
    _csrf_token: 'TRUE'

in

automatic_updates.cron.post_apply:
  path: '/automatic-update/cron/post-apply/{stage_id}/{installed_version}/{target_version}/{key}'
  defaults:
    _controller: 'automatic_updates.cron_update_stage:handlePostApply'
  requirements:
    _access_system_cron: 'TRUE'

… and probably remove

  requirements:
    _access_system_cron: 'TRUE'

(which also means removing {key})

Given that I've now laid out all next steps, unassigning @phenaproxima, I think @kunal.sachdev can continue here 😊

I did all the steps in #17 📌 While applying updates during cron, put the site into maintenance mode Fixed , but it's still failing the same way.

I debugged it and checked and it's going in

if (!$this->maintenanceMode->exempt($this->account)) {
        // When the account is not exempt, other subscribers handle request.
        $this->eventDispatcher->dispatch($event, MaintenanceModeEvents::MAINTENANCE_MODE_REQUEST);
      }

in \Drupal\Core\EventSubscriber\MaintenanceModeSubscriber::onKernelRequestMaintenance and that's why going in \Drupal\Core\EventSubscriber\MaintenanceModeSubscriber::onMaintenanceModeRequest which is giving the error page where it says

Site under maintenance
Drupal is currently under maintenance. We should be back shortly. Thank you for your patience.

.

The build test is failing because the account does not have access to the site in maintenance mode see #19

1. This still needs tests.
2. @tedbow wrote → needs to be decided.
3. Posted review.

Since point 2 affects how my review would be addressed, we need a decision on point 2 first. Assigning to @tedbow for that.

re #23

2) we ended up needing this for 📌 Package Manager should keep an audit log of changes it applied to the active codebase Fixed , so these are now public

Needs work for my comment in the MR. Not sure what the correct solution is here though

We should #description property to the unattended_level form element we create in automatic_updates_form_update_settings_alter and add something like

When background updates updates are applied your site will briefly be put into maintenance mode.

Looks good!

Glad to see this done! 👏

Automatically closed - issue fixed for 2 weeks with no activity.