Intermittent caching issue on Entities

This is interesting but unfortunately, Drupal 9 is no longer supported. Please validate the bug exists on a supported release and provide precise steps to reproduce this in the cases where it occurs.

It does occur on Drupal 10.x as well. We do not have production setup for Drupal 11.
It occurs only on production environment though.

Entities themselves don't and shouldn't have cache tags. Cache tags are for when multiple cache items depend on the entity, the entity cache item itself is refreshed when it's saved.

From the description, this sounds like it could be a duplicate of 🐛 Loading entity during save will cache old version Active .

Thank you @catch. @dinesh18 Can you confirm this is a duplicate?

thanks @catch and @cilefen, but the 2nd and 3rd points mentioned in the comment: Comment#6 https://www.drupal.org/project/drupal/issues/3474843 🐛 Loading entity during save will cache old version Active , seems to be promising. It's difficult for us to replicate on the local environments or even staging.

When only one user is updating the block content, not sure if that can cause a cache stampede?
Also, we have the memcache as the caching mechanism, so when we print the content using below drush command, timestamp can be seen updated but the data remains old
drush php-eval '$cache = \Drupal::service("cache.backend.memcache")->get("entity"); $r=$cache->get("values:block_content:25394); print_r($r);' .

On memcache you might need https://www.drupal.org/project/memcache/issues/2996615#new 🐛 Transaction support for cache (tags) invalidation Needs review - can you confirm whether you already have that applied?

<?php
/**
 * @file
 * Contains caching configuration.
 * Last change: 09/19/2019
 */

use Composer\Autoload\ClassLoader;

/**
 * Use memcache as cache backend.
 *
 * Autoload memcache classes and service container in case module is not
 * installed. Avoids the need to patch core and allows for overriding the
 * default backend when installing Drupal.
 *
 * @see https://www.drupal.org/node/2766509
 */
if (getenv('AH_SITE_ENVIRONMENT') &&
  array_key_exists('memcache', $settings) &&
  array_key_exists('servers', $settings['memcache']) &&
  !empty($settings['memcache']['servers'])
) {
  // Check for PHP Memcached libraries.
  $memcache_exists = class_exists('Memcache', FALSE);
  $memcached_exists = class_exists('Memcached', FALSE);
  $memcache_services_yml = DRUPAL_ROOT . '/modules/contrib/memcache/memcache.services.yml';
  $memcache_module_is_present = file_exists($memcache_services_yml);
  if ($memcache_module_is_present && ($memcache_exists || $memcached_exists)) {
    // Use Memcached extension if available.
    if ($memcached_exists) {
      $settings['memcache']['extension'] = 'Memcached';
    }
    if (class_exists(ClassLoader::class)) {
      $class_loader = new ClassLoader();
      $class_loader->addPsr4('Drupal\\memcache\\', DRUPAL_ROOT . '/modules/contrib/memcache/src');
      $class_loader->register();
      $settings['container_yamls'][] = $memcache_services_yml;

      // Acquia Default Settings for the memcache module
      // Default settings for the Memcache module.
      // Enable compression for PHP 7.
      $settings['memcache']['options'][Memcached::OPT_COMPRESSION] = TRUE;

      // Set key_prefix to avoid drush cr flushing all bins on multisite.
      $settings['memcache']['key_prefix'] = $conf['acquia_hosting_site_info']['db']['name'] . '_';
      
      // Decrease latency.
      $settings['memcache']['options'][Memcached::OPT_TCP_NODELAY] = TRUE;

      // Bootstrap cache.container with memcache rather than database.
      $settings['bootstrap_container_definition'] = [
        'parameters' => [],
        'services' => [
          'database' => [
            'class' => 'Drupal\Core\Database\Connection',
            'factory' => 'Drupal\Core\Database\Database::getConnection',
            'arguments' => ['default'],
          ],
          'settings' => [
            'class' => 'Drupal\Core\Site\Settings',
            'factory' => 'Drupal\Core\Site\Settings::getInstance',
          ],
          'memcache.settings' => [
            'class' => 'Drupal\memcache\MemcacheSettings',
            'arguments' => ['@settings'],
          ],
          'memcache.factory' => [
            'class' => 'Drupal\memcache\Driver\MemcacheDriverFactory',
            'arguments' => ['@memcache.settings'],
          ],
          'memcache.timestamp.invalidator.bin' => [
            'class' => 'Drupal\memcache\Invalidator\MemcacheTimestampInvalidator',
            'arguments' => ['@memcache.factory', 'memcache_bin_timestamps', 0.001],
          ],
          'memcache.backend.cache.container' => [
            'class' => 'Drupal\memcache\DrupalMemcacheInterface',
            'factory' => ['@memcache.factory', 'get'],
            'arguments' => ['container'],
          ],
          'cache_tags_provider.container' => [
            'class' => 'Drupal\Core\Cache\DatabaseCacheTagsChecksum',
            'arguments' => ['@database'],
          ],
          'cache.container' => [
            'class' => 'Drupal\memcache\MemcacheBackend',
            'arguments' => [
              'container',
              '@memcache.backend.cache.container',
              '@cache_tags_provider.container',
              '@memcache.timestamp.invalidator.bin',
              '@memcache.settings',
            ],
          ],
        ],
      ];

      // Use memcache for bootstrap, discovery, config instead of fast chained
      // backend to properly invalidate caches on multiple webs.
      // See https://www.drupal.org/node/2754947
      $settings['cache']['bins']['bootstrap'] = 'cache.backend.memcache';
      $settings['cache']['bins']['discovery'] = 'cache.backend.memcache';
      $settings['cache']['bins']['config'] = 'cache.backend.memcache';

      // Use memcache as the default bin.
      $settings['cache']['default'] = 'cache.backend.memcache';
    }
  }
}

I have this PHP file included via settings.php and later, we have override the below config in settings.php
$settings['cache']['default'] = 'cache.backend.memcache_transaction_aware';

Regarding #8 it's not about how many users are editing, but how much traffic the site gets overall, and previously you said you can't replicate this on dev or staging. Can you confirm whether it's still the case that this happens when editing on production with visitor traffic?

I created a custom block, although this block is not exposed to visual traffic. I try updating the content and intermittent I see the content is not getting updated. When I check the value in the memcache bins using below query :
drush php-eval '$cache = \Drupal::service("cache.backend.memcache")->get("entity"); $r=$cache->get("values:block_content:2456"); print_r($r);'
Please see screenshot where the created timestamp is updated but still the value remains same.

When the text updated to Interactive html block 54

drush php-eval '$cache = \Drupal::service("cache.backend.memcache")->get("entity"); $r=$cache->get("values:block_content:2456"); print_r($r);'
stdClass Object
(
    [cid] => values:block_content:2456
    [data] => Drupal\block_content\Entity\BlockContent Object
        (
            [entityTypeId:protected] => block_content
            [enforceIsNew:protected] =>
            [typedData:protected] =>
            [cacheContexts:protected] => Array
                (
                )

            [cacheTags:protected] => Array
                (
                )

            [cacheMaxAge:protected] => -1
            [_serviceIds:protected] => Array
                (
                )

            [_entityStorages:protected] => Array
                (
                )

            [values:protected] => Array
                (
                    [id] => Array
                        (
                            [x-default] => 2456
                        )

                    [revision_id] => Array
                        (
                            [x-default] => 7166
                        )

                    [type] => Array
                        (
                            [x-default] => interactive
                        )

                    [uuid] => Array
                        (
                            [x-default] => 154b3224-4e22-4bd4-9695-e7028fb2edba
                        )

                    [langcode] => Array
                        (
                            [x-default] => en
                        )

                    [revision_user] => Array
                        (
                            [x-default] =>
                        )

                    [revision_created] => Array
                        (
                            [x-default] => 1609746342
                        )

                    [revision_log] => Array
                        (
                            [x-default] =>
                        )

                    [revision_default] => Array
                        (
                            [x-default] => 1
                        )

                    [isDefaultRevision] => Array
                        (
                            [x-default] => 1
                        )

                    [status] => Array
                        (
                            [x-default] => 1
                        )

                    [info] => Array
                        (
                            [x-default] => Test Interactives
                        )

                    [changed] => Array
                        (
                            [x-default] => 1746524053
                        )

                    [default_langcode] => Array
                        (
                            [x-default] => 1
                        )

                    [revision_translation_affected] => Array
                        (
                            [x-default] => 1
                        )

                    [reusable] => Array
                        (
                            [x-default] => 1
                        )

                    [field_embed] => Array
                        (
                            [x-default] => Array
                                (
                                    [0] => Array
                                        (
                                            [value] => <p>Interactive html block 54</p>

                                            [format] => full_html
                                        )

                                )

                        )

                    [field_view_more] => Array
                        (
                            [x-default] => Array
                                (
                                    [0] => Array
                                        (
                                            [uri] => internal:/test
                                            [title] => Interactives
                                            [options] => Array
                                                (
                                                )

                                        )

                                )

                        )

                )

            [fields:protected] => Array
                (
                )

            [fieldDefinitions:protected] =>
            [languages:protected] =>
            [langcodeKey:protected] => langcode
            [defaultLangcodeKey:protected] => default_langcode
            [activeLangcode:protected] => x-default
            [defaultLangcode:protected] => en
            [translations:protected] => Array
                (
                    [x-default] => Array
                        (
                            [status] => 1
                        )

                )

            [translationInitialize:protected] =>
            [newRevision:protected] =>
            [isDefaultRevision:protected] => 1
            [entityKeys:protected] => Array
                (
                    [bundle] => interactive
                    [id] => 2456
                    [revision] => 7166
                    [uuid] => 154b3224-4e22-4bd4-9695-e7028fb2edba
                )

            [translatableEntityKeys:protected] => Array
                (
                    [label] => Array
                        (
                            [x-default] => Test Interactives
                        )

                    [langcode] => Array
                        (
                            [x-default] => en
                        )

                    [published] => Array
                        (
                            [x-default] => 1
                        )

                    [default_langcode] => Array
                        (
                            [x-default] => 1
                        )

                    [revision_translation_affected] => Array
                        (
                            [x-default] => 1
                        )

                )

            [validated:protected] =>
            [validationRequired:protected] =>
            [loadedRevisionId:protected] => 7166
            [revisionTranslationAffectedKey:protected] => revision_translation_affected
            [enforceRevisionTranslationAffected:protected] => Array
                (
                )

            [isSyncing:protected] =>
            [theme:protected] =>
            [accessDependency:protected] =>
        )

    [created] => 1746524053.907
    [expire] => -1
    [tags] => Array
        (
            [0] => block_content_values
            [1] => entity_field_info
            [2] => memcache:entity
        )

    [checksum] => 102
    [valid] => 1
)
 

Updated value to Interactive html block 55 but still shows the old content. Until cache is cleared, it shows the old content.

drush php-eval '$cache = \Drupal::service("cache.backend.memcache")->get("entity"); $r=$cache->get("values:block_content:2456"); print_r($r);'
stdClass Object
(
    [cid] => values:block_content:2456
    [data] => Drupal\block_content\Entity\BlockContent Object
        (
            [entityTypeId:protected] => block_content
            [enforceIsNew:protected] =>
            [typedData:protected] =>
            [cacheContexts:protected] => Array
                (
                )

            [cacheTags:protected] => Array
                (
                )

            [cacheMaxAge:protected] => -1
            [_serviceIds:protected] => Array
                (
                )

            [_entityStorages:protected] => Array
                (
                )

            [values:protected] => Array
                (
                    [id] => Array
                        (
                            [x-default] => 2456
                        )

                    [revision_id] => Array
                        (
                            [x-default] => 7166
                        )

                    [type] => Array
                        (
                            [x-default] => interactive
                        )

                    [uuid] => Array
                        (
                            [x-default] => 154b3224-4e22-4bd4-9695-e7028fb2edba
                        )

                    [langcode] => Array
                        (
                            [x-default] => en
                        )

                    [revision_user] => Array
                        (
                            [x-default] =>
                        )

                    [revision_created] => Array
                        (
                            [x-default] => 1609746342
                        )

                    [revision_log] => Array
                        (
                            [x-default] =>
                        )

                    [revision_default] => Array
                        (
                            [x-default] => 1
                        )

                    [isDefaultRevision] => Array
                        (
                            [x-default] => 1
                        )

                    [status] => Array
                        (
                            [x-default] => 1
                        )

                    [info] => Array
                        (
                            [x-default] => Test Interactives
                        )

                    [changed] => Array
                        (
                            [x-default] => 1746524053
                        )

                    [default_langcode] => Array
                        (
                            [x-default] => 1
                        )

                    [revision_translation_affected] => Array
                        (
                            [x-default] => 1
                        )

                    [reusable] => Array
                        (
                            [x-default] => 1
                        )

                    [field_embed] => Array
                        (
                            [x-default] => Array
                                (
                                    [0] => Array
                                        (
                                            [value] => <p>Interactive html block 54</p>

                                            [format] => full_html
                                        )

                                )

                        )

                    [field_view_more] => Array
                        (
                            [x-default] => Array
                                (
                                    [0] => Array
                                        (
                                            [uri] => internal:/test
                                            [title] => Interactives
                                            [options] => Array
                                                (
                                                )

                                        )

                                )

                        )

                )

            [fields:protected] => Array
                (
                )

            [fieldDefinitions:protected] =>
            [languages:protected] =>
            [langcodeKey:protected] => langcode
            [defaultLangcodeKey:protected] => default_langcode
            [activeLangcode:protected] => x-default
            [defaultLangcode:protected] => en
            [translations:protected] => Array
                (
                    [x-default] => Array
                        (
                            [status] => 1
                        )

                )

            [translationInitialize:protected] =>
            [newRevision:protected] =>
            [isDefaultRevision:protected] => 1
            [entityKeys:protected] => Array
                (
                    [bundle] => interactive
                    [id] => 2456
                    [revision] => 7166
                    [uuid] => 154b3224-4e22-4bd4-9695-e7028fb2edba
                )

            [translatableEntityKeys:protected] => Array
                (
                    [label] => Array
                        (
                            [x-default] => Test Interactives
                        )

                    [langcode] => Array
                        (
                            [x-default] => en
                        )

                    [published] => Array
                        (
                            [x-default] => 1
                        )

                    [default_langcode] => Array
                        (
                            [x-default] => 1
                        )

                    [revision_translation_affected] => Array
                        (
                            [x-default] => 1
                        )

                )

            [validated:protected] =>
            [validationRequired:protected] =>
            [loadedRevisionId:protected] => 7166
            [revisionTranslationAffectedKey:protected] => revision_translation_affected
            [enforceRevisionTranslationAffected:protected] => Array
                (
                )

            [isSyncing:protected] =>
            [theme:protected] =>
            [accessDependency:protected] =>
        )

    [created] => 1746524069.581
    [expire] => -1
    [tags] => Array
        (
            [0] => block_content_values
            [1] => entity_field_info
            [2] => memcache:entity
        )

    [checksum] => 102
    [valid] => 1
)

We able to replicate the issue on local by doing some load test and added the sleep method on the entity update.
To fix this, we saw that on the classTransactionAwareMemcacheBackend, we were missing the method

    /**
   * {@inheritdoc}
   */
  public function deleteMultiple(array $cids) {
    return $this->callOrBuffer(__FUNCTION__, func_get_args());
  }

after add this method, it fixed the issue.

We also observed, even changing the below bins to cache.backend.memcache_transaction_aware also fixes the issue, but not sure if it can have any impacts.
$settings['cache']['bins']['bootstrap'] = 'cache.backend.memcache';
$settings['cache']['bins']['discovery'] = 'cache.backend.memcache';
$settings['cache']['bins']['config'] = 'cache.backend.memcache';

Thank you @catch

Automatically closed - issue fixed for 2 weeks with no activity.