Explore ways to mark hook conversion status per module or file

Created on 28 November 2024, 21 days ago

Problem/Motivation

The bc layer for procedural hooks is slow.

Provide a way for modules to mark they have converted in info.yml

Provide a way for modules that have not fully converted to mark a file as having no procedural hooks after a certain point.

Steps to reproduce

Proposed resolution

Add flag in info yml
Add attribute called #[SkipHookScan]

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

โœจ Feature request
Status

Active

Version

11.0 ๐Ÿ”ฅ

Component

base system

Created by

๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @nicxvan
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan
  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom catch

    If we make this per-module, it will require the .info information - however, it will also mean we can skip scanning the directories and parsing the files at all.

    I'm not sure if we already check .info information in a container rebuild, if not, that might not be an option.

    Did also wonder about a tiny file next to the module.info.yml that indicates this - if it exists, conversion is done and stop there. Massive hack but again would be efficient.

  • Merge request !10402Add skip option to modules โ†’ (Open) created by nicxvan
  • Pipeline finished with Canceled
    19 days ago
    Total: 70s
    #354650
  • Pipeline finished with Canceled
    19 days ago
    Total: 69s
    #354821
  • Pipeline finished with Canceled
    19 days ago
    Total: 91s
    #354837
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan

    Ok, so the Attribute is much easier to implement, there are no tests yet, but it looks like it's working.

    I'm still a bit unsure why the skip_procedural isn't being picked up. On discovery and list it is in the array, but when scanProceduralHooks is called info isn't set yet, so it defaults to scan.

    It's also not set in moduleData, any advice would be appreciated.

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan

    Oh I figured it out, this definitely needs cleanup and optimizing.

  • Pipeline finished with Canceled
    19 days ago
    Total: 82s
    #354839
  • Pipeline finished with Canceled
    19 days ago
    Total: 71s
    #354849
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan

    Added some tests, let's see how broken things are.

  • Pipeline finished with Failed
    19 days ago
    Total: 279s
    #354854
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan

    I think marking core modules should be a follow up, but leaving a few now to see how tests react, I know for sure ordering is broken, and the new tests.

  • Pipeline finished with Canceled
    19 days ago
    Total: 826s
    #354857
  • Pipeline finished with Canceled
    19 days ago
    Total: 80s
    #354859
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan

    The only thing failing is the skip all, the attribute works as expected.

    When I debug and output the values for moduleInfo I see the test module 5 times. The first 4 times procedural is set to scan, the 5th it is set to skip, at this point though the hooks have already been scanned and cached.

  • Pipeline finished with Failed
    19 days ago
    Total: 613s
    #354866
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan

    It looks like the compiler pass happens before the info file is parsed.

    The attribute is working. I'm going to try the sibling file for now in a separate branch.

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan
  • Merge request !10405Resolve #3490355 "Sibling file approach" โ†’ (Open) created by nicxvan
  • Pipeline finished with Canceled
    19 days ago
    Total: 211s
    #354876
  • Pipeline finished with Failed
    19 days ago
    Total: 141s
    #354878
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan
  • Pipeline finished with Failed
    19 days ago
    Total: 776s
    #354883
  • Pipeline finished with Success
    19 days ago
    Total: 1251s
    #354893
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan
  • There are two open MRs. Can you clarify which one is for review and hide the other one?

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan

    Approach 2, the one called sibling file.
    I didn't hide the other one in case someone wanted to review that approach.

    Also for completeness @ghost suggested renaming covered modules to modulename.converted.module
    This would require updating extension discovery and would not handle .inc files from hook hook info.

    I have not provided a poc for @ghost's suggestion.

  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom catch

    The sibling file approach was my idea, but seeing it in the MR it doesn't look too bad. I wonder if 'hooks_converted.info.yml' would be more self documenting though?

    Also - if this is set, I think we could also optimise ::collectModuleHookImplementations to only scan inside src/Hook in the first place maybe? It could just completely ignore all other files then.

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan

    Do you mean moduleName.hooks_converted.yml?

    If so I can change that and look at optimizing that method.

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan

    Ok I refactored that bit, I think we have to leave:

    if ($extension === 'inc') {
      $parts = explode('.', $fileinfo->getFilename());
      if (count($parts) === 3 && $parts[0] === $module) {
        $this->groupIncludes[$parts[1]][] = $filename;
      }
    }
    

    Alone or risk breaking hook_hook_info

  • Pipeline finished with Canceled
    16 days ago
    Total: 181s
    #356450
  • Pipeline finished with Success
    16 days ago
    Total: 1047s
    #356460
  • I don't feel strongly about this, but I thought I'd mention it: what about using container parameters instead of the sibling file approach?

    So for my_module, in my_module.services.yml:

    parameters:
      my_module.hooks_converted: true
    

    In HookCollectorPass, the $container object would need to be passed down to the relevant method, but the logic would look something like
    $skip_procedural = $container->hasParameter(''$module.hooks_converted");

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan

    That is a great suggestion.

    Let me do it on yet another branch, just in case something weird happens early compile pass again.

    I'll close the sibling branch as long as this approach works.

  • Merge request !10423Resolve #3490355 "Container params" โ†’ (Closed) created by nicxvan
  • Pipeline finished with Failed
    16 days ago
    Total: 615s
    #356541
  • Pipeline finished with Failed
    16 days ago
    Total: 124s
    #356592
  • Pipeline finished with Failed
    16 days ago
    Total: 117s
    #356609
  • Pipeline finished with Success
    16 days ago
    Total: 930s
    #356648
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan

    nicxvan โ†’ changed the visibility of the branch 3490355-sibling-file-approach to hidden.

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan

    It's green, this has the added advantage of allowing you to mark a contrib module as converted if the maintainer has not converted it if you know there are no procedural hooks.

  • Pipeline finished with Canceled
    16 days ago
    Total: 723s
    #356667
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan

    nicxvan โ†’ changed the visibility of the branch 3490355-explore-ways-to to hidden.

  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom catch

    Looks great to me. We need a follow-up to add this to all core modules. Maybe one for all the ones that don't use hook_hook_info() / hook_module_implements_alter() and another issue for the ones that implement those two hooks.

    Once we've done those follow-up issues it would be useful to get a drush site:install comparison before/after to see how much this actually helped with Umami or Standard in terms of memory usage.

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan
  • Pipeline finished with Failed
    16 days ago
    Total: 1142s
    #356692
  • Pipeline finished with Success
    16 days ago
    Total: 835s
    #356715
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan
  • One small comment that isn't critical, but would be nice. Otherwise RTBC.

    • catch โ†’ committed 756c93fb on 11.x
      Issue #3490355 by nicxvan, godotislate, catch: Add procedural hook short...
    • catch โ†’ committed f03b175f on 11.1.x
      Issue #3490355 by nicxvan, godotislate, catch: Add procedural hook short...
  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom catch

    Service parameter is so much nicer than the extra file. We won't be able to tell the benefit from this until we convert core modules ๐Ÿ“Œ Mark core modules as converted to OOP hooks Active but feels like it should be a lot.

    Committed/pushed to 11.x and cherry-picked to 11.1.x, thanks!

  • ๐Ÿ‡ฏ๐Ÿ‡ตJapan ptmkenny

    As a module developer, I read over the change record and have a few questions:

    1. If a module does not have any files where procedural hooks can be defined (for example, the only php code is classes in the /src directory), is there any benefit to specifying the parameter module_name.hooks_converted: true?
    2. If a module has converted all hooks that can be converted to OOP but still has hooks that cannot be converted to OOP yet (for example, template_preprocess hooks), then module_name.hooks_converted: true cannot be specified and the attribute #[StopProceduralHookScan] needs to be used for such hooks, correct?

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan

    Correct, I think you have to be careful too about includes if you manually include them.

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan

    The patch here is required if you're going to use this.

    ๐Ÿ› Include .module files during HookCollection always even if not scanned Active

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States dww

    Wow, this moved fast. ๐Ÿ˜… nice work, yโ€™all!

    One complaint / concern. โ€œHooks_convertedโ€ is sort of vague and potentially confusing. Converted to what? Is it too late to propose something like only_object_hooks or something? The 3 of yโ€™all that worked on this issue are totally familiar with what hook conversion this is about, but I wonder about the other 99.99% of Drupal developers.

    Thanks again!
    -Derek

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan

    What about no_procedural_hooks

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States dww

    Right, something like that. Although I don't love configuration knobs of any sort that start with negatives. E.g. no_procedural_hooks: true is kinda weird. "Check this box to turn off the thing" is a UI anti-pattern. But setting has_procedural_hooks: false after conversion is weird, too, since if the parameter isn't defined, defaulting to TRUE is also maybe unexpected. I dunno, naming is hard. ๐Ÿ˜…

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States nicxvan

    It's complicated by the fact that we don't even check the value if it exists we skip procedural.

Production build 0.71.5 2024