Some theme hooks are not invoking (depends on templates order provided by filesystem)

I have just encountered this very issue this morning and couldn't figure out what the problem was intially.

My issue turned out to be the same as you have detailed in the original post:

If I have the following preprocess function: THEME_preprocess_field__paragraph__field_name()
and no template for field--paragraph--field-name
but do have a template for field--paragraph--field-name--bundle
the 'non bundle' (less specific) preprocess function will be available in the theme registry and apply to the more specific template.

If you do have a template as well for field--paragraph--field-name in addition to field--paragraph--field-name--bundle the theme registry then doesn't then have the 'non bundle' (more generic) preprocessor THEME_preprocess_field__paragraph__field_name() available.

The ksort in the patch does indeed fix the issues found and cause the order to be sorted and consistent.

Although @mdupont has a valid point here in that maybe we should be adding an additional key the options param of FileSystem::scanDirectory()...

Re-rolling patch for 10.1.x.

Wow, kudos to @chOdec for finding this bug.

This bug is a worst-case scenario for DX, because it introduces a random and non-deterministic behavior in template and hook processing.

Given the patch #8 is fixing the issue in twig_theme() I'm moving this back to theme queue.

The Needs Review Queue Bot → tested this issue.

While you are making the above changes, we recommend that you convert this patch to a merge request → . Merge requests are preferred over patches. Be sure to hide the old patch files as well. (Converting an issue to a merge request without other contributions to the issue will not receive credit.)

This needs tests
It would be helpful to get the issue summary updated to provide a scenario that doesn't use paragraphs hooks as the example so this is clearly a core issue. Even better - as part of creating the tests, make a test -only theme that reproduces the reported problem, yet works properly once the fix in #8 is present. If we have a test that fails without #8 and passes with it, I'm happy to commit.

It's probably worth checking a bit to see if the ksort is sufficient. In all cases does the specificity order match php sort order?

(and tbh even if it doesn't match 1:1, as long as it doesn't break anything, the ksort fix should happen and the edge cases figured out later).

I was able to reproduce the issue with Paragraphs, but as we cannot write test for paragraph I was trying to reproduce the same thing for Custom block but unfortunately I am unable to do so. can someone take a look at this?

I don't know if this bug can be reliably reproduced and test-cased because of

In code of this service is called readdir() function and the entries are returned in the order in which they are stored by the filesystem (according php documentation).

and

* Because order of templates (related theme hooks) depends on filesystem, it may vary on every environment.

Basically PHP's readdir() has a built-in random behavior.

Given the unintrusive nature of this patch, can we add this without tests?

This problem can be reproduced regardless of filesystem, and tests are important not just to ensure this does not cause problems, but to ensure future changes do not un-fix whatever is addressed here.

While it is true that the order of readdir() results returned by a single call can differ depending on the filesystem the \Drupal\Core\File\FileSystem::doScanDirectory
method calls readdir() recursively per directory. The recursive calls to

 do occur in a predictable order (parent before child) and returns an array with the subdirectory results appearing before the parent results 
 <code>return array_merge(array_merge(...$files_in_sub_dirs), $files_in_this_directory); 

If you have a template file you consistently want to appear before the others in the returned array, put it in a subdirectory of a directory including templates it should appear before.

In other words, this can be tested by adding a custom module or theme that provides templates with a directory structure where the template that should appear first is in a subdirectory of the directory with the templates it should appear after.

it may also be worth seeing if this is better to address in the logic of completeSuggestion since higher-priority theme suggestions do not necessarily correlate with alphabetical order.

Hi @bnjmnm, I tried the steps from #3270083-18: Some theme hooks are not invoking (depends on templates order provided by filesystem) →

In other words, this can be tested by adding a custom module or theme that provides templates with a directory structure where the template that should appear first is in a subdirectory of the directory with the templates it should appear after.

But it was loading

Main Directory Template

instead of subdirectory one.

if return array_merge(array_merge(...$files_in_sub_dirs), $files_in_this_directory); is reversed i.e return array_merge(, $files_in_this_directory, array_merge(...$files_in_sub_dirs)); , then it will load the sub directory one.

@yash.rode if there are two files with identical names then that is what would happen. Drupal doesn't support identically named templates in a theme so that doesn't have to be tested.

Prashant.c → made their first commit to this issue’s fork.

Raised a PR using the patch from https://www.drupal.org/project/drupal/issues/3270083#comment-15262719 🐛 Some theme hooks are not invoking (depends on templates order provided by filesystem) Needs work .

However, in this, we are sorting the template array in ascending order by key names.
Will this order will impact any rendering for example region templates applied after block, fields, and nodes?

However, in this, we are sorting the template array in ascending order by key names.
Will this order will impact any rendering for example region templates applied after block, fields, and nodes?

I think this is a reasonable concern and I was hoping the tests added would make this more evident. Sounds like you noticed the same thing and didn't need tests to see it 👍. The current solution clearly fixes it for some use cases, but alphabetical order is not necessarily the order of template priority. For example, if we take this logically ordered list of suggestions from a real site:

<!-- FILE NAME SUGGESTIONS:
   * field--node--body--article.html.twig
   * field--node--body.html.twig
   * field--node--article.html.twig
   * field--body.html.twig
   x field--text-with-summary.html.twig
   * field.html.twig
-->

Sorting it alphabetically changes it to an order we don't want

Array
(
    [0] => field--body.html.twig
    [1] => field--node--article.html.twig
    [2] => field--node--body--article.html.twig
    [3] => field--node--body.html.twig
    [4] => field--text-with-summary.html.twig
    [5] => field.html.twig
)

It is probably better if the solution focuses on the logic in \Drupal\Core\Theme\Registry::completeSuggestion, and make it so the logic is consistent regardless of the array order as neither filesystem order or alphabetical sorting reflect the suggestion priority.

But since this will need tests to land anyway, those can exist before looking into how to solve this in \Drupal\Core\Theme\Registry::completeSuggestion.