Contrib.social

Memcache Lock issue when one of the memcache server is down

Open on Drupal.org →
Created on 18 December 2018, over 5 years ago
Updated 4 May 2023, about 1 year ago

While using PECL memcahed extension when one of the memcache servers is down, some times it leads to locking situation. So the entire site is unresponsive and returns 503.

Memcached settings

$conf['cache_backends'][] = 'sites/all/modules/contrib/memcache/memcache.inc';
$conf['cache_backends'][] = 'sites/all/modules/contrib/cache_consistent/cache_consistent.inc';
$conf['lock_inc'] = 'sites/all/modules/contrib/memcache/memcache-lock.inc';

$conf['cache_class_cache_form'] = 'DrupalDatabaseCache';
$conf['cache_default_class'] = 'ConsistentCache';
$conf['consistent_cache_default_class'] = 'MemCacheDrupal';
$conf['consistent_cache_default_safe'] = TRUE;
$conf['consistent_cache_buffer_mechanism'] = 'ConsistentCacheBuffer';
$conf['consistent_cache_default_strict'] = FALSE;
$conf['consistent_cache_strict_cache_bootstrap'] = TRUE;

$conf['memcache_stampede_protection'] = TRUE;
$conf['form_cache_expiration'] = 7200; //configuration for cache form expiration
$conf['memcache_stampede_protection_ignore'] = array(
// Ignore some cids in 'cache_bootstrap'.
'cache_bootstrap' => array(
'module_implements',
'variables',
'lookup_cache',
'schema:runtime:*',
'theme_registry:runtime:*',
'_drupal_file_scan_cache',
),
// Ignore all cids in the 'cache' bin starting with 'i18n:string:'
'cache' => array(
'i18n:string:*',
),
// Disable stampede protection for the entire 'cache_path' and 'cache_rules'
// bins.
'cache_path',
'cache_rules',
);
$conf['memcache_stampede_semaphore'] = 15;
$conf['memcache_stampede_wait_time'] = 1;
$conf['memcache_stampede_wait_limit'] = 2;

$conf['memcache_servers'] = array(
'x.x.x.x1:11211' => 'default',
'x.x.x.x2:11211' => 'default',
'x.x.x.x3:11211' => 'default',
'x.x.x.x4:11211' => 'default',
);
$conf['memcache_bins'] = array(
'cache' => 'default',
);
$conf['memcache_options'] = array(Memcached::OPT_CONNECT_TIMEOUT => 10,
Memcached::OPT_SERVER_FAILURE_LIMIT => 2,
Memcached::OPT_REMOVE_FAILED_SERVERS => true,
Memcached::OPT_RETRY_TIMEOUT => 30);

php-fpm.slow.log

18-Dec-2018 17:00:24] [pool www] pid 25991
script_filename = /var/www/html/current/www/index.php
[0x00007fe2da290fe8] usleep() /var/www/html/releases/20181213085237/www/sites/all/modules/contrib/memcache/memcache-lock-code.inc:122
[0x00007fe2da290e98] lock_wait() /var/www/html/releases/20181213085237/www/sites/all/modules/contrib/rules/rules.module:375
[0x00007fe2da290d28] rules_get_cache() /var/www/html/releases/20181213085237/www/sites/all/modules/contrib/rules/rules.module:1641
[0x00007fff92fe7120] rules_token_info() unknown:0
[0x00007fe2da290b80] call_user_func_array() /var/www/html/releases/20181213085237/www/includes/module.inc:965
[0x00007fe2da290a58] module_invoke_all() /var/www/html/releases/20181213085237/www/includes/token.inc:260
[0x00007fe2da2908c8] token_info() /var/www/html/releases/20181213085237/www/sites/all/modules/contrib/token/token.module:517
[0x00007fe2da290608] token_get_info() /var/www/html/releases/20181213085237/www/sites/all/modules/contrib/entity/entity_token.tokens.inc:176
[0x00007fff92fe8ae0] entity_token_tokens() unknown:0
[0x00007fe2da290420] call_user_func_array() /var/www/html/releases/20181213085237/www/includes/module.inc:965
[0x00007fe2da290288] module_invoke_all() /var/www/html/releases/20181213085237/www/includes/token.inc:178
[0x00007fe2da2900e8] token_generate() /var/www/html/releases/20181213085237/www/includes/token.inc:87
[0x00007fe2da28fef0] token_replace() /var/www/html/releases/20181213085237/www/sites/all/modules/contrib/file_entity/file_entity.file.inc:183
[0x00007fff92fea4a0] file_entity_file_load() unknown:0
[0x00007fe2da28fd50] call_user_func_array() /var/www/html/releases/20181213085237/www/includes/entity.inc:368
[0x00007fe2da28fb00] attachLoad() /var/www/html/releases/20181213085237/www/sites/all/modules/contrib/entitycache/includes/entitycache.entitycachecontrollerhelper.inc:150

🐛 Bug report
Status

Active

Version

1.6

Component

memcache-lock.inc

Created by

kpremcse

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Comment about 1 year ago →
    🇨🇦Canada fozboz

    I was experiencing the same issue. Try adding this to your settings.php.

    $settings['memcache']['options'] = [
  Memcached::OPT_DISTRIBUTION => Memcached::DISTRIBUTION_CONSISTENT
  Memcached::OPT_REMOVE_FAILED_SERVERS => true
];

    The README states that DISTRIBUTION_CONSISTENT is the default, but according to the PHP memcached manual the default is in fact DISTRIBUTION_MODULA.

    Additionally, you need OPT_REMOVE_FAILED_SERVERS enabled in order for keys to be routed to the other servers, otherwise you will experience a big slow down in your site while it tries and retries to get/set keys on the bad server.

contrib.social Blog FAQ Discussions
Production build 0.69.0 2024