block_content block derivatives do not scale to thousands of block_content entities

Created on 30 January 2018, almost 7 years ago
Updated 24 January 2024, 10 months ago

Problem/Motivation

A request that immediately follows a cache rebuild or a memcached flush results in a major performance issue, as Drupal tries to cache all available blocks at once. We ran into an issue on a couple of applications where memcached didn't have enough memory allocated to withstand the onslaught of memcache_set. This resulted in multiple calls for all of the blocks to be set en masse in memcached - because it attempts to fulfill its initial request ("write ~2300 blocks to cache_discovery, please"), then runs out of room (hits the memory ceiling), then tries to complete the job by taking a second swipe at it. You can see this at work in this xhprof run:

MemcachePool_set is being called over 4600 times here - which is approximately 2X the number of custom blocks in the database:

mysql> SELECT COUNT(*) FROM block_content;
+----------+
| COUNT(*) |
+----------+
|     2294 |
+----------+
1 row in set (0.00 sec)

Proposed resolution

After discussing with @berdir and @larowlan the idea is to not load the derivates but have a single option for Block Content library and select the block via an entity_autocomplete element.

Remaining tasks

Verify approach
Post update hook for block config and layout builder
Update broken tests
Review

User interface changes

Todo

API changes

None

Data model changes

None

πŸ› Bug report
Status

Needs work

Version

11.0 πŸ”₯

Component
Block contentΒ  β†’

Last updated 4 days ago

Created by

πŸ‡ΊπŸ‡ΈUnited States solution33r

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • πŸ‡¬πŸ‡§United Kingdom catch

    The problem if course is how to get there from the current core implementation, because I honestly have no idea how to achieve that. The BC promise of drupal core makes such a change challenging. Best I can think of would be a setting that allows to switch between exposing them as derivatives vs. such an autocomplete field, then the deriative class would just not do anything if that setting is set and instead there might be a different, conditionally enabled block plugin that would work like that.

    Could we add a new always-on block plugin that allows this?

    Then we could add a config setting to disable the current derivatives. And then (possibly over a couple of major releases) deprecate all the old derivatives altogether. Seems like it would be hard to do a force-update from one blog plugin to another since classes etc. would change, but if we provide the new method and a killswitch, we can get rid of the old way over time. For example switch the config setting to off for new sites.

  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    That sounds like an option, the option to disable is for now more for those that have the problem with many blocks and could also do the work to switch their current block placements. Forcing everyone over is then for (much) later, if we care enough about it. an intermediate step could be to disable that by default on new sites.

  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    We might need to think about the process for creating new blocks, but it's also a chance for a less confusing process than we have now, the block selection could be similar to the media library where you can select an existing one or create one in a modal.

  • First commit to issue fork.
  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10

    We might need to think about the process for creating new blocks, but it's also a chance for a less confusing process than we have now, the block selection could be similar to the media library where you can select an existing one or create one in a modal.

    That's exactly what we've been discussing in our draft roadmap for block content, so we're on the same page

  • πŸ‡―πŸ‡΄Jordan abu-zakham

    Drupal core cache all reusable block in cache_entity, I think blocks should be cached on render instead of cache blocks on first run after cache rebuild, a workaround is to make blocks non-reusable

    update block_content_field_data set reusable = 0;
    
  • πŸ‡§πŸ‡¬Bulgaria yivanov

    @abu-zakham, making the blocks non-reusable will hide them from the block content overview page and they will not be available to add on Block layout page and in the Layout builder.

  • πŸ‡―πŸ‡΄Jordan abu-zakham

    @yivanov, yes you are right

  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Adding this to our 10.2 target. Since there’s a 10.1 patch is there a different approach we want to take?

  • Open on Drupal.org β†’
    Environment: PHP 8.2 & MySQL 8
    last update about 1 year ago
    Not currently mergeable.
  • @smustgrave opened merge request.
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Could use some feedback as I'm having issue with the ajax callback and when to load the blockContent.

  • last update about 1 year ago
    30,092 pass, 28 fail
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Another issue I found with using block_content:uuid in the settings is it messes with the redirects.

  • πŸ‡§πŸ‡¬Bulgaria yivanov

    Another issue I found with using block_content:uuid in the settings is it messes with the redirects.

    Could you elaborate more?

  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    > Sorry it's not clicking for me. How would a new plugin and leaving this one solve the issue?

    It won't. But it's impossible to just completely change the existing plugin and keep BC. See #54/55. We add a feature flag, new sites will use the new approach, existing sites will keep using the old plugin, they get a warning and need to convert all their stuff (which likely is going to be painful if they use layout builder). Maybe even with the feature flag, new placements will by default use the new approach, that we have to figure out.

    But that can came later, first we need to have a working implementation using the new plugin.

    As for the redirect, instead of preselecting block_content:UUID, you need to select block_content_reference or whatever we call it and have an additional reference parameter that you pass along that you then need to read again in config. So you need to read the current block in that order:

    1. form state
    2. query parameter
    3. configuration

    (2. and 3 should technically never be set at the same time, so it shouldn't really matter).

  • πŸ‡¬πŸ‡§United Kingdom catch
Production build 0.71.5 2024