🇭🇺Hungary @mxr576

Should be fixed in https://www.drupal.org/project/llms_txt/releases/1.0.4 → . Please check and report back. Thank you

Please share any relevant settings of the Redirect module if you have it enabled
1. the Setting page contnet
2. If you have any URL redirects for any reasons for the /llms.txt path.

Please also update the issue title and especially the description with reproduction steps.

Thank you!

https://www.drupal.org/project/disable_route_normalizer → project page says:

This simple module is intended to be used on a multilingual site using redirect module when you need to have a language neutral content page without any redirection to a path prefix, which means there will not be any preferential language shown in the URL.

The functionality provided is available by default on clean Drupal installation but not when using redirect.

Okay, so you use Apache,we usually use Nginx. Do you also have Redirect module enabled? Or do you have an idea how to extend the current test coverage of the module with a failing test so it could be proved this change fixes a problem?

----------

Understanding _disable_route_normalizer in Drupal

_disable_route_normalizer is a special route/request attribute that tells Drupal's route normalizer (commonly exercised by the Redirect module's "Enforce clean and canonical URLs") to skip canonicalization and avoid issuing redirects for the current request or route, which is useful when custom path processing or non-canonical endpoints must not be altered or redirected.

What it does

• When set, it prevents the normalizer from rewriting or redirecting a path to its canonical form, so custom inbound path processors or language-specific paths can resolve without being forced to a different URL.
• It is often used to avoid unwanted 301 redirects triggered by the Redirect module when paths are intentionally "non-canonical," such as virtual paths or special endpoints.

How to use it on a route

• In a route definition, add the option/attribute to the route so requests hitting that route are ignored by the normalizer, for example in routing.yml via a route option that sets the attribute at runtime (various contrib issues and examples describe "Add _disable_route_normalizer to route" to stop normalization).
• Contrib modules and issue queues show adding this flag to routes like OpenID configuration endpoints or robots.txt analogs to prevent Redirect from interfering with expected behavior.

How to use it per request

• If setting per request (rather than per route), an event subscriber can set the request attribute before the Redirect module's kernel.request listener runs: set request->attributes['_disable_route_normalizer'] = true with a higher priority than the Redirect subscriber, so the normalizer sees the flag and skips redirecting.

When to reach for it

• Custom inbound-only path processors that map pretty or virtual paths to internal routes, where the Redirect module would otherwise bounce the user to a canonical URL and break the intended UX.
• Special endpoints (e.g., OAuth/OpenID discovery, metrics, or robots-like outputs) where any normalization or redirect would cause client failures or break protocol expectations, thus requiring the normalizer to be disabled for those routes.

Practical example

A module implements an inbound path processor to map /some-random-path to an internal node without changing outbound links. To avoid a forced redirect back to the canonical node path, an event subscriber sets _disable_route_normalizer on the request early, so the Redirect module respects the custom mapping and does not redirect the user away from the intended path.

Tested the change manually and also in the spin off issue where automated tests are finally running: https://git.drupalcode.org/project/ai_recipe_llm_optimized_content/-/pip...

/builds/project/ai_recipe_llm_optimized_content
$ # If the PROJECT_NAME variable has not been defined with an override value then get it by searching for *.info.yml # collapsed multi-line command
ls: cannot access '*.info.yml': No such file or directory

Just spotted this in the log, probably something to silence for a recipe build

mxr576 → created an issue.

mxr576 → created an issue.

mxr576 → created an issue.

We are waiting for upstream fixes here.

mxr576 → created an issue.

I think I found a related one: ✨ Expand Drupal\Core\Recipe\RecipeDiscovery to allow discovering available recipes, likely for use in Project Browser Active . If anybody has any insights to add there, please do.

Also let me know if you disagree with my current understanding on that every changes is necessary here, but testing on a recipes with dependencies on other recipes are still failing - and the current workaround is no go - because the limitations of the Recipe Installer in Drupal core.

There are two current use cases for this:

I believe there's a third use case that may be relevant: running automated tests on recipes with dependencies on other (contrib) recipes in GitLab CI environments. The main roadblock we encountered in ✨ Add recipes path handling Active is how Recipe Installer currently discovers recipes in the filesystem.

Question: Do you think this problem relates to this issue, or should it be reported separately? Or perhaps it's not a core issue at all?

I haven't tested this patch with my POC playground → alongside the GitLab changes from the other issue. From what I can tell, this issue only introduces a new API that isn't yet leveraged by the Recipe Installer ecosystem.

I have concerns about how QA environments are configured on GitLab (and also by DDEV Drupal Contrib for local development). In these setups, the component under test (module, theme, recipe, etc.) gets symlinked to a special location while residing in the site root. This setup pattern might create additional complications for recipe discovery and installation workflows.

Has anyone considered how this new API would interact with symlinked recipe scenarios common in CI/CD environments?

PS.: I assume DrupalCMS recipes never discovered this problem because they live in a monorepo and QA-d there, instead of in many-repo splits.

It would be nice if the core getRecipePath() actually did the correct thing, but it doesn't.

Exactly the reason why I am considering opening a ticket for the Recipe project, because I think the roadblock is there.

Thanks for the quick reply (DM on Slack) :)

I used the existing environment variable DRUPAL_PROJECT_FOLDER which is now set precisely to the required value for this. See this change to the d11-recipe branch test that is exactly what you could do.

Unfortunately, I do not think this is a valid solution because it couples the test to the execution environment.. The path discovery should only rely on what is inside the package under test for reference.

Recipes in Drupal core is special of course, but they also do dynamic discovery in tests: https://github.com/drupal/core/blob/11.2.5/modules/system/tests/src/Func...
Just as Drupal CMS and other recipes that has test coverage:
* https://git.drupalcode.org/project/drupal_cms_forms/-/blob/1.2.x/tests/s...
* https://git.drupalcode.org/search?search=%22applyRecipe%28%22&group_id=2... (yes, the list is not extensive)

Okay, so now after all this noise (sorry about that) I am probably at the point that I can report a valid problem that my test still fails because the llm_support dependency still cannot be installed by the recipe installer.

https://git.drupalcode.org/project/ai_recipe_llm_optimized_content/-/pip...

1) Drupal\Tests\ai_recipe_llm_optimized_content\Functional\RecipeApplicationTest::testApplicabilityAndIdempotency
Process exit code mismatch.
Expected: 0
Actual: 1
STDOUT:
STDERR:
In RecipeFileException.php line 56:
                                                                               
  Validation errors were found in /builds/project/ai_recipe_llm_optimized_con  
  tent/recipe.yml:                                                             
  - [recipes][0]: The llm_support recipe does not exist at /builds/project.    
                                                                               
recipe [-i|--input INPUT] [--] <path>
Failed asserting that 1 is identical to 0.
/builds/project/ai_recipe_llm_optimized_content/web/core/tests/Drupal/FunctionalTests/Core/Recipe/RecipeTestTrait.php:87
/builds/project/ai_recipe_llm_optimized_content/tests/src/Funtional/RecipeApplicationTest.php:49
FAILURES!

Thoughts? Error on my end?

RT complete FM...

https://git.drupalcode.org/project/gitlab_templates/-/blob/main/docs/inf...

ohh wait, even if my pipeline is on a fork, this shell script downloads the latest stable version of symlink-project.php? :O

get-file-via-curl.sh executing curl --retry 3 -OLf https://git.drupalcode.org/project/gitlab_templates/-/raw/default-ref/scripts/symlink_project.php

as far as I can tell the root cause of the failure that the recipe under testing was symlinked to the wrong folder, it is here instead of ../recipes.

https://git.drupalcode.org/project/ai_recipe_llm_optimized_content/-/job...

From pipeline log

Creating symlinks in /builds/project/ai_recipe_llm_optimized_content/web//ai_recipe_llm_optimized_content pointing back to files in /builds/project/ai_recipe_llm_optimized_content

Let me check if I can try the latest commit here in #3551179.

Composer jobs fail with

$ rm symlink_project.php
$ echo -e "\e[0Ksection_end:`date +%s`:symlink_output\r\e[0K"
$ if [[ -f composer.json.backup ]]; then # collapsed multi-line command
Restoring composer.json from backup
rm: cannot remove '/builds/project/ai_recipe_llm_optimized_content/recipes/ai_recipe_llm_optimized_content/composer.json': No such file or directory
Uploading artifacts for failed job 00:07
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: command terminated with exit code 1

https://git.drupalcode.org/project/ai_recipe_llm_optimized_content/-/job...

I think this is exactly the issue that I wanted to report and built a POC for this in 📌 POC: Testing on Gitlab with recipe dependencies is broken Active \o/

I am also glad to see my other recipe (llm_support) mentioned in this thread =], ai_recipe_llm_optimized_content adds a dependency on it and everything started to fall apart from that moment...

Let me check if I can try the latest commit here in #3551179.

mxr576 → created an issue.

Opened a follow up ✨ Ability to add markdown link field in a views view Active

mxr576 → created an issue.

Opened a follow up for resolving the main blocker in Markdownify #3551170: POC: Universal markdown conversion via .md URL suffix →

mxr576 → created an issue.

Opened 📌 Finalize repo initalization Active

I think this can be closed now. Feel free to re-open if you disagree.

mxr576 → created an issue.

Actually, this is a task for the module.

Moved all issues to Related issues.

Opened ✨ Add dependency on llm_support recipe Active .

mxr576 → created an issue.

So now that llm_support has main- and footer menu added → the way we agreed on Slack, I would suggest closing this issue and opening two follow ups for:
1. creating a proper readme/project page
2. adding a dependency on llm_support recipe.

mxr576 → created an issue.

Due to preparations for DrupalCon, MR#2 was merged.

Created a follow up for test coverage: #3550980: Test coverage for "Add field "Include in llms.txt" for all content types" →

mxr576 → created an issue.

Due to preparations for DrupalCon, MR#6 was merged.

Created a follow up for test coverage: #3550976: Test coverage for Add token for rendering a markdown version of a Views view →

mxr576 → created an issue.

mxr576 → created an issue.

Let's discuss on Drupal Slack if this feature should be part of LLM support recipe directly or not, currently no objection for that.

However, we either include non-marked menu links as they are today, or we may have a problem with always exposing markdown links, see #3548981-4: Output markdown menu links when Markdownify is enabled →

There are some unnecessary extra new lines, but I think we are getting there. Now the next task is probably a bit of a test coverage for this and the patch for llms_txt.

# Drush Site-Install



### Our work

Lorem ipsum dolor sit amet consectetur adipiscing elit quisque faucibus.

- [Circus of All](https://llm-support.local.pronovix.net/node/2.md)
- [Hotel Lorem](https://llm-support.local.pronovix.net/node/1.md)
 


### Resources

Sed ut perspiciatis unde omnis iste natus error sit voluptatem

- [Contact us](https://llm-support.local.pronovix.net/node/3.md)

Test coverage would be great, especially on cache invalidation.

Current state of MR#6 without changes on phpstan baseline so it could be applied on the llm_support recipe.

Also, the work is being done in ✨ Improve linking to md version of an entity Active could also help with directly generating markdown links for nodes.

Not alone, after that entity views data also has to be altered so the Markdown link would be available in Views field settings. \Drupal\views\EntityViewsData::addEntityLinks()

Maybe recent results in #3548981-4: Output markdown menu links when Markdownify is enabled → are just proves that we also do not and cannot serialize any kind of Views view to markdown. Links after the forced transformation may not work.

The special case we are building the ideal examole for transfermarion, maybe we should support those kinds of Views somehow...

Also, the work is being done in ✨ Improve linking to md version of an entity Active could also help with directly generating markdown links for nodes.

@a.dmitriiev shared the following code examples, thanks for this!

  $info['types']['llm_txt_views'] = [
    'name' => t('llms.txt: Views'),
    'description' => t('Render views in markdown for llms.txt'),
  ];
  $views = \Drupal::entityTypeManager()->getStorage('view')->loadMultiple();
  /** @var \Drupal\views\ViewEntityInterface $view */
  foreach ($views as $view) {
    foreach ($view->get('display') as $display) {
      $key = $view->id() . '=' . $display['id'];
      $info['tokens']['llm_txt_views'][$key] = [
        'name' => $view->label() . ' ' . $display['display_title'],
      ];
    }
  }

  if ($type === 'llm_txt_views') {
    foreach ($tokens as $name => $original) {
      [$view_id, $display_id] = explode('=', $name, 2);
      if (!empty($view_id) && !empty($display_id)) {
        $view = Views::getView($view_id);
        if (!empty($view) && $view->access($display_id)) {
          $view->setDisplay($display_id);
          $view->initHandlers();
          $preview = $view->preview($display_id);
          $html = \Drupal::service('renderer')->renderInIsolation($preview);
          if (\Drupal::moduleHandler()->moduleExists('markdownify')) {
            $base_url = \Drupal::service('router.request_context')->getCompleteBaseUrl();
            $markdown = \Drupal::service('markdownify.html_converter')->convert($html);
            $markdown = preg_replace('/('. preg_quote($base_url, '/') . '[a-zA-Z0-9-_\/]+)/', '${1}.md', $markdown);
            $replacements[$original] = $markdown;
          }
        }
      }
    }
  }

My spidey senses indicates that cacheability information is lost somewhere here, even if nothing in the call chain of $view->access($display_id) bubbles up cacheability information - why is that? probably there is a Core issue for this, nvm for now... - . So I think it would be better to run the complete rendering in a dedicated render context and bubbling up the collected cacheability information, just in case.

markdownify_views is surely needed for this feature. Second guessing a bit but should this feature be included in that module? What is your thought on this @iambatman?

I was also thinking about why views_embed_view() has not been leveraged in the original code, but then I realized that it is not important. The most important part probably doing the same what is done in \Drupal\markdownify_views\Controller\MarkdownifyViewPageController::handle(), or calling that code for consistently generating the markdown version of a Views view.

mxr576 → created an issue.

Thanks for the contribution, it looks like a simple change!

I would rebase from 1.x to be sure, but otherwise should be ready to be merged.

# Drush Site-Install

Main menu
- [All nodes](https://llms-txt.local.pronovix.net/node.md): All nodes on the site
- [Home](https://llms-txt.local.pronovix.net/.md)

User menu
- [My account](https://llms-txt.local.pronovix.net/user.md)
- [Log out](https://llms-txt.local.pronovix.net/user/logout?token=[redacted].md)

## Section 1
<p>Lorem ipsum...</p>

I think I see a couple of problem with appending .md suffix to all menu links.

For bravery and pushing the limits of this change, I have tried this with uid1.

1. https://llms-txt.local.pronovix.net/.md Makes no sense, but just the first proof that simple suffixing won't work.
2. https://llms-txt.local.pronovix.net/user.md HTTP 404
3. https://llms-txt.local.pronovix.net/user/logout?token=fzL0Ox4jS6qafdt6gz... Proves that simple suffixing does not work due to potential query parameters, it has to be figured out if and how the path can be modified. Probably we have to construct a new \Drupal\Core\Url object for that.
4. https://llms-txt.local.pronovix.net/node.md also HTTP 404 until markdownify_views is enabled, but this is just one proof for not all every internal url could be turned into a markdown url that actually resolves.

Pushed the progress so far

mxr576 → changed the visibility of the branch 3550681-add-field-include to active.

Moving back to the LLM support recipe

I have also evolved our original plain simple idea that was centered around a simple checkbox for including content in llms.txt to leveraging a including and grouping around taxonomy terms in llms.txt which could actually unleash the full potential of the CMS.

Please review and comment.

I just found out, that this recipe doesn't require the node module to be installed.

Not yet, it could be. As I wrote on Slack, I have started to work on this implementation in the weekend already. Everything is starting to click, other than that part of the original idea where the Views view is embeded to llms.txt with a token - unless we meant a custom token :)

Let's continue the discussion on Slack on this one.

Honestly, I still do not really know what kind of magic the _disable_route_normalizer is, but the related issue got fixed and the test coverage added to that confirms that this solution does not work, however the merged solution does. I am going to soon release a tagged release from the module, please check and verify.

Thanks for raising the issue and trying to solve it!

Fixed via https://git.drupalcode.org/project/llms_txt/-/commit/f05c61cae698025c557...

mxr576 → changed the visibility of the branch 3548950-llms.txt-on-default-path-only to hidden.

I have an alternative, working solution in the related issue. Please take a look at if you have time.

FTR, the component responsible for multilingual URL generation is `\Drupal\language\Plugin\LanguageNegotiation\LanguageNegotiationUrl`, but I haven't found an out-of-the-box way to exclude a route/path from this process. The only approach I've discovered involves implementing a `\Drupal\language\LanguageNegotiationMethodInterface` and bypassing the `/llms.txt` path there. However, that adds significantly more complexity than what I envision for this task.

Perhaps there's a simpler approach to achieve the expected behavior? I noticed the [Multilingual Exclude]( https://www.drupal.org/project/multilingual_exclude → ) module uses the _admin_path: true route option to achieve what we need.

https://git.drupalcode.org/project/multilingual_exclude/-/blob/1.0.x/src...

This could potentially offer a cleaner solution by modifying the route definition itself rather than intercepting the language negotiation process.

Isn't _disable_route_normalizer a Redirect module feature but the multilanguage problem exists even without that so it has to be addressed still? There may or may not be a Redirect module compatibility issue two which makes the suggested change relevant.

https://git.drupalcode.org/project/redirect/-/blob/8.x-1.12/src/EventSub...

Speaking of automated tests... test coverage would be also a must for having this change merged.

Thanks for the issue, this is actually a duplicate of another already existing issue so if this would not have a patch attached, I would have just closed it a duplicate.

(To be fair, the original issue was placed to the wrong project, so I have just moved it.)

I'll take a look at the fix. Could you please also create an MR from this so automated tests would be triggered? Patches are only used for backports nowadays.

Moving to the right project.

Just came to my mind that markdown url-s are provided by the Markdownify module, so maybe this belongs to Markdownify module?

View with included pages

Create an MR for llm_support recipe which adds the "Include in llms.txt checkbox" to all node types and also provides the "Included in llms.txt" Views view.

mxr576 → created an issue.

(see for example here)

I have also written a long reaction to that blog post and his Linkedin post defending and emphasizing the benefits of LLMs.txt, especially in context of CMS-es like Drupal.

https://www.linkedin.com/feed/update/urn:li:activity:7366036739120934915...

Hiding the patch for now to avoid confusion, especially since the MR diff should also apply on 2.2.x.

I assume this needs test, but first, we should know that if this improvement is acceptable in the scope of the Group module or not.

mxr576 → changed the visibility of the branch 3.3.x to hidden.

Korinna's patch also nicely applies on 3.3.x. I assume that should be the primary target for new features and 2.2.x is a target for potential backports.

mxr576 → made their first commit to this issue’s fork.

Enabled the "media" module and the error has disappeared, so the RC could be a missing dependency on media or an invalid assumption that it is enabled on a site, always.

I have just installed the module, visited the batch page, selected Node entity type and run into the same error that the description contains.

What else I can help with clarifying the RC?