Contrib.social

FieldResolver::resolveInternalIncludePath is very slow with complex and nested includes

Open on Drupal.org →
Created on 24 October 2023, about 1 year ago
Updated 26 October 2023, about 1 year ago

Problem/Motivation

We have a paragraphs based site with a content types, but about 30 paragraph types, several of which can contain nested paragraphs, media etc. We also use Paragraph Library, which adds to the nesting woes.

To ensure we retrieve all required data to render a node in one request, we use jsonapi_defaults and it's _default_includes. This is currently resulting in almost 3,000 "include paths".

As a result, requesting even a single node from JSON:API is taking 15 seconds or more. Using xhprof, I have an example request that took 33 seconds, 99% of which was in FieldResolveer::resolveInternalIncludePath (and it's recursions).

Proposed resolution

Obviously this site has some architectural considerations that we are working on. However, I'm think the position we've ended up in is probably not that unusual for a relatively complex paragraphs based editing experience. So putting that to one side...

Given that FieldResolveer::resolveInternalIncludePath works with a resource type (not a specific entity) and path, it seems to me that this would be a prime candidate for caching.

  • Statically: A single resource type / remaining path parts could be encountered multiple times, both adjacently and within nesting, and will produce the same result. Therefore there could be significant optimisations by statically caching this result to avoid recalculating the same data multiple times in a single request.
  • Persistently: As ResourceTypes change infrequently (i.e. entity/field changes), a resource type / path parts should be highly cacheable between requests, avoiding any need to calculate for almost all requests.

Remaining tasks

Hopefully this issue can turn into a task to add static & permanent caching to FieldResolveer::resolveInternalIncludePath!

User interface changes

None.

API changes

None.

Data model changes

Addition of caching.

Release notes snippet

Significant performance improvements in resolving JSON:API includes.

🐛 Bug report
Status

Needs work

Version

11.0 🔥

Component
JSON API  →

Last updated 9 days ago

Created by

🇬🇧United Kingdom andrewbelcher

Live updates comments and jobs are added and updated live.
  • Performance

    It affects performance. It is often combined with the Needs profiling tag.

Sign in to follow issues

Comments & Activities

  • Issue created by @andrewbelcher
  • Status changed to Needs work about 1 year ago
  • Comment about 1 year ago →
    🇬🇧United Kingdom andrewbelcher

    Here is a starter on an approach for the persistent cache. It caches based on the resource type and include path (parts, concatenated and hashed).

    Based on the performance testing below, there is definitely potential with warm caches, but a hit with cold caches.

    Some initial local performance test (with memcache as the default backend):

    Baseline: 325ms

    Patch as is:
    - With cold caches: 1,530ms
    - With warm caches: 137ms

    Using cache.jsonapi_resource_types (chained memory & default):
    - With cold caches: 1,100ms
    - With warm caches: 146ms

    As above but with the database as the default backend:

    Baseline: 340ms

    Patch as is:
    - With cold caches: 2,460ms
    - With warm caches: 175ms

    Using cache.jsonapi_resource_types (chained memory & default):
    - With cold caches: 2,550ms
    - With warm caches: 177ms

  • Open in Jenkins → Open on Drupal.org →
    Environment: PHP 8.2 & MySQL 8
    last update about 1 year ago
    Patch Failed to Apply
  • Comment about 1 year ago →
    🇮🇳India _utsavsharma

    Rerolled the patch for 11.x.

  • Open in Jenkins → Open on Drupal.org →
    Environment: PHP 8.2 & MySQL 8
    last update about 1 year ago
    30,438 pass
contrib.social Blog FAQ Discussions
Production build 0.71.5 2024