Allow failing over to database or other backends if Redis is down

Created on 23 December 2019, almost 5 years ago
Updated 3 May 2023, over 1 year ago

We are slowly migrating our sites away from memcached towards redis for application caching. What I really like about the memcache Drupal module is that the site keeps working, even though the memcached server is down. This means that memcached is not a single point of failure within the architecture. When it's down, cache is unavailable and the site will not perform as good. But it will be still available.

@Berdir: would you be open to enhanced functionality (maybe as a setting in the admin form) for your Redis module?
Please do let me know, so I can start and deliver a patch for you to review.

Feature request
Status

Needs review

Version

1.0

Component

Code

Created by

🇳🇱Netherlands basvanderheijden

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • 🇨🇦Canada joelpittet Vancouver

    Was trying this out with #20 and MR13 and ran into the following:

    The website encountered an unexpected error. Please try again later.
    Symfony\Component\DependencyInjection\Exception\ServiceCircularReferenceException: Circular reference detected for service "logger.factory", path: "module_handler -> cache.bootstrap -> logger.factory -> config.factory -> config.storage -> cache.config". in Drupal\Component\DependencyInjection\Container->get() (line 147 of core/lib/Drupal/Component/DependencyInjection/Container.php).
    
    Drupal\Component\DependencyInjection\Container->get('logger.factory') (Line: 683)
    Drupal::logger('redis') (Line: 330)
    watchdog_exception('redis', Object) (Line: 38)
    Drupal\redis\Client\PhpRedis->getClient('127.0.0.1', 6379, NULL, NULL, Array, ) (Line: 179)
    Drupal\redis\ClientFactory::getClient() (Line: 65)
    Drupal\redis\Cache\CacheBackendFactory->get('config') (Line: 83)
    Drupal\Core\Cache\ChainedFastBackendFactory->get('config') (Line: 83)
    Drupal\Core\Cache\CacheFactory->get('config')
    call_user_func_array(Array, Array) (Line: 255)
    Drupal\Component\DependencyInjection\Container->createService(Array, 'cache.config') (Line: 177)
    Drupal\Component\DependencyInjection\Container->get('cache.config', 1) (Line: 434)
    Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) (Line: 237)
    Drupal\Component\DependencyInjection\Container->createService(Array, 'config.storage') (Line: 177)
    Drupal\Component\DependencyInjection\Container->get('config.storage', 1) (Line: 434)
    Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) (Line: 237)
    Drupal\Component\DependencyInjection\Container->createService(Array, 'config.factory') (Line: 177)
    Drupal\Component\DependencyInjection\Container->get('config.factory', 1) (Line: 434)
    Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) (Line: 237)
    Drupal\Component\DependencyInjection\Container->createService(Array, 'logger.factory') (Line: 177)
    Drupal\Component\DependencyInjection\Container->get('logger.factory') (Line: 683)
    Drupal::logger('redis') (Line: 330)
    watchdog_exception('redis', Object) (Line: 38)
    Drupal\redis\Client\PhpRedis->getClient('127.0.0.1', 6379, NULL, NULL, Array, ) (Line: 179)
    Drupal\redis\ClientFactory::getClient() (Line: 65)
    Drupal\redis\Cache\CacheBackendFactory->get('bootstrap') (Line: 83)
    Drupal\Core\Cache\ChainedFastBackendFactory->get('bootstrap') (Line: 83)
    Drupal\Core\Cache\CacheFactory->get('bootstrap')
    call_user_func_array(Array, Array) (Line: 255)
    Drupal\Component\DependencyInjection\Container->createService(Array, 'cache.bootstrap') (Line: 177)
    Drupal\Component\DependencyInjection\Container->get('cache.bootstrap', 1) (Line: 434)
    Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) (Line: 237)
    Drupal\Component\DependencyInjection\Container->createService(Array, 'module_handler') (Line: 177)
    Drupal\Component\DependencyInjection\Container->get('module_handler') (Line: 602)
    Drupal\Core\DrupalKernel->preHandle(Object) (Line: 46)
    Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 106)
    Drupal\page_cache\StackMiddleware\PageCache->pass(Object, 1, 1) (Line: 85)
    Drupal\page_cache\StackMiddleware\PageCache->handle(Object, 1, 1) (Line: 48)
    Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 51)
    Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 23)
    Stack\StackedHttpKernel->handle(Object, 1, 1) (Line: 718)
    Drupal\Core\DrupalKernel->handle(Object) (Line: 19)
    

    My settings for this was with D9 + DDEV + Redis 6 provided by https://github.com/ddev/ddev-redis

    /**
     * Cache and Redis config.
     */
    $settings['redis.connection']['interface'] = 'PhpRedis';
    $settings['redis.connection']['host']      = '127.0.0.1';
    $settings['cache']['default'] = 'cache.backend.redis';
    $settings['redis.failover'] = TRUE;
    

    Also 👋 @star-szr

  • 🇨🇦Canada star-szr

    Hi @joelpittet! Thanks for the report and for testing.

    Do you have a line similar to the below in your settings file? It is pulled from the updated README. I'd be happy to dig into this further and I should be able to reproduce if the below setting does not fix things for you.

    Adjust path below accordingly, for example if your Redis module is at modules/contrib/redis then use that instead. Haven't dug into that aspect yet to see if there is a better way.
    $settings['container_yamls'][] = 'modules/redis/example.failover.services.yml';

  • Status changed to Needs work over 1 year ago
  • 🇺🇸United States daggerhart

    #20 patch works great for me against the module 1.5 and 1.6. The patch does not apply to 1.7 unfortunately.

    My config looks like this:

      $settings['container_yamls'][] = 'modules/contrib/redis/example.failover.services.yml';
      $settings['cache']['default'] = 'cache.backend.redis';
      $settings['redis.failover'] = TRUE;
      $settings['redis.connection']['interface'] = 'Predis';
      $settings['redis.connection']['host'] = 'redis_primary';
    

    I'll see if I can re-roll the patch, but may run out of time and have to stick w/ 1.6 for now.

  • Status changed to Needs review over 1 year ago
  • 🇺🇸United States daggerhart

    Reroll of #20 against 8.x-1.x branch. Only conflicts were in the README.

  • Status changed to RTBC over 1 year ago
  • 🇺🇸United States daggerhart

    I know it's not ideal for me to mark RTBC since I rolled the last patch, but I didn't really do that much so maybe it's okay. You decide.

    This is working well w/ module 1.7.

    My config:

      $settings['container_yamls'][] = 'modules/contrib/redis/example.failover.services.yml';
      $settings['cache']['default'] = 'cache.backend.redis';
      $settings['redis.failover'] = TRUE;
      $settings['redis.connection']['interface'] = 'Predis';
      $settings['redis.connection']['host'] = 'redis_primary';
    

    Tested:

    1. Visited the site and pages loaded fine.
      • No error messages in dblog.
      • No values in the `cache_render` db table.
    2. Shutdown my redis server.
    3. Visited the site and pages loaded fine.
      • (expected) Error messages in dblog says "Couldn't connect to redis, using failover" or similar.
      • (expected) Values were populated in the `cache_render` db table.
  • 🇺🇸United States RustedBucket

    In response to #29, I've applied patch in #28 and added the config entries. When shutting down the Redis server I continue to get:
    Error: Class "Redis" not found in Drupal\redis\Client\PhpRedis->getClient() (line 18 of /web/modules/contrib/redis/src/Client/PhpRedis.php) in the logs.

    The instantiation of the Redis class fails.
    In phpRedis.php
    $client = new \Redis();

    I think I have everything right.
    Copied modules/redis/example.failover.services.yml to /sites/default/redis.failover.services.yml

    settings.local.php

    $settings['container_yamls'][] = DRUPAL_ROOT . '/sites/default/redis.failover.services.yml';
    $settings['cache']['default'] = 'cache.backend.redis';
    $settings['redis.failover'] = TRUE;
    $settings['redis.connection']['interface'] = 'PhpRedis';
    $settings['redis.connection']['host'] = '127.0.0.1';
    

    Any thoughts?

  • 🇫🇷France pbonnefoi

    The #28 patch works great with a single Redis instance, but there one use case that is not covered yet.

    I have a Redis with Sentinel Master/Slave and therefore, my settings is having something like that :

    $settings['redis.connection']['host']      = ['XXX.XXX.X.X:XXXX', 'YYYY.YYY.Y.Y:XXXX];
    

    When one of the redis is down, caching is still working thanks to the patch here => https://www.drupal.org/project/redis/issues/3163436 🐛 Connection refuse in PhpRedis when sentinel/redis are down RTBC

    But when both are down, then the failover is not working. I'll try to find a solution and post it if so.

  • Status changed to Needs review 9 months ago
  • 🇷🇸Serbia milos33

    Hello.

    I have tried patch #28 on Redis 1.7, D10.1.8, PHP8.1 and it is not working. I have tried PhpRedis and Predis both. After applying patch I followed instructions from README, but when for example port is changed on Redis connection to simulate situation when Redis is down (Connection refused is error returned) I get error that container is not initialized. Same thing happens if Redis container is turned off in Docker. Drush outputs error that there is "circular reference detected for service LoggerChannelFactoryInterface". Looking with xdebug this error is coming from src/Client/PhpRedis.php, in try/catch when attempt to log exception via watchdog/logger is made on line 37.
    Any insight or suggestion is welcome.

    Thanks

  • 🇳🇱Netherlands Johan den Hollander

    Using the diff from the #13 MR with DDEV I got it to work. Applied the same settings to Lagoon settingsfile and this allows me to setup a new and empty environment.
    I also tried it by changing the redis port and that worked as well.

    use Drupal\Core\Installer\InstallerKernel;
    
    if (!InstallerKernel::installationAttempted() && extension_loaded('redis') && class_exists('Drupal\redis\ClientFactory')) {
      // Set Redis as the default backend for any cache bin not otherwise specified.
      $settings['cache']['default'] = 'cache.backend.redis';
      $settings['redis.connection']['host'] = 'redis';
      $settings['redis.connection']['port'] = 6379;
      $settings['redis.connection']['interface'] = 'PhpRedis';
    
      // Apply changes to the container configuration to better leverage Redis.
      // This includes using Redis for the lock and flood control systems, as well
      // as the cache tag checksum. Alternatively, copy the contents of that file
      // to your project-specific services.yml file, modify as appropriate, and
      // remove this line.
    //  $settings['container_yamls'][] = 'modules/contrib/redis/example.services.yml';
      $settings['container_yamls'][] = 'modules/redis/example.failover.services.yml';
      // Allow the services to work before the Redis module itself is enabled.
      $settings['container_yamls'][] = 'modules/contrib/redis/redis.services.yml';
    
      // Manually add the classloader path, this is required for the container cache bin definition below
      // and allows to use it without the redis module being enabled.
      $class_loader->addPsr4('Drupal\\redis\\', 'modules/contrib/redis/src');
    
      // Use redis for container cache.
      // The container cache is used to load the container definition itself, and
      // thus any configuration stored in the container itself is not available
      // yet. These lines force the container cache to use Redis rather than the
      // default SQL cache.
      $settings['bootstrap_container_definition'] = [
        'parameters' => [],
        'services' => [
          'redis.factory' => [
            'class' => 'Drupal\redis\ClientFactory',
          ],
          'cache.backend.redis' => [
            'class' => 'Drupal\redis\Cache\CacheBackendFactory',
            'arguments' => ['@redis.factory', '@cache_tags_provider.container', '@serialization.phpserialize'],
          ],
          'cache.container' => [
            'class' => '\Drupal\redis\Cache\PhpRedis',
            'factory' => ['@cache.backend.redis', 'get'],
            'arguments' => ['container'],
          ],
          'cache_tags_provider.container' => [
            'class' => 'Drupal\redis\Cache\RedisCacheTagsChecksum',
            'arguments' => ['@redis.factory'],
          ],
          'serialization.phpserialize' => [
            'class' => 'Drupal\Component\Serialization\PhpSerialize',
          ],
        ],
      ];
    }
    # Use failover services when Redis is not available.
    $settings['redis.failover'] = TRUE;
    
  • Status changed to Needs work 9 months ago
  • 🇬🇧United Kingdom vijaycs85 London, UK

    As mentioned in #25 and #33, the logger causing circular dependency (esp. with drush). I have added review comments in relavent lines in the MR #13

  • Status changed to Needs review 9 months ago
  • 🇬🇧United Kingdom vijaycs85 London, UK

    Addressed the logger issue.

  • 🇬🇧United Kingdom vijaycs85 London, UK

    re-rolling the patch as it doesn't apply cleanly on gitlab-ci.yml file and watchdog entry as it is causing circular dependency.

  • 🇮🇳India Ankush_03 Gurgaon, India 🇮🇳
  • 🇮🇳India Ankush_03 Gurgaon, India 🇮🇳

    As per #2 If we are not adding failover route then we need to handle exception.

  • 🇬🇧United Kingdom vijaycs85 London, UK

    rerolled without the changes in .gitlab-ci.yml as it is already in HEAD.

  • 🇧🇷Brazil jribeiro Campinas - São Paulo

    I got it working along with RedisCluster ( https://www.drupal.org/project/redis/issues/2900947 Implement initial RedisCluster client integration Needs review ) using AWS ElastiCache Redis SSO.

    The only problem I'm having now is that cache-clear doesn't remove the cached data from Database, so, if you have old cached data in the database, when Redis is down, it will load old content, as described in #37.

Production build 0.71.5 2024