Use markdown for high-level documentation

While the core issue is RTBC, there were no objections to moving in this direction. Therefore, I have made this issue so folks here are aware and perhaps can comment on next steps.

I see that the issue mentions "GitLab pages" as the agreed way forwards. A GitLab pages site is a separate site from api.drupal.org.
Is there any way that is expected for these two sites to communicate? Or is the goal to completely move away from api.drupal.org for code documentation?

@fjgarlin this is not really for code documentation as such, it's for high level documentation of subsystems.

The example I've been using (partly because I'm familiar with it, partly because it shows the issue), is the cache API.

We have:

https://www.drupal.org/docs/8/api/cache-api/cache-api →

And https://api.drupal.org/api/drupal/core%21core.api.php/group/cache/11.x

Both of these are prose, the information is split between the two, as cache API maintainer I know I can submit an MR for the api.drupal.org output (although I dread doing that due to the format), but I have no idea what sign-offs are required to update the handbook documentation, it looks like the section is owned by Wim Leers, I didn't know this until I checked, I'm not sure Wim even does.

The bigger problem is the two of these shouldn't exist, we should have one canonical place for this documentation without obvious duplication. The handbook could potentially have sections about other caching things like how to set up contrib modules (redis, varnish_purge) but at the moment the documentation is split.

We can compare this to Symfony's cache API documentation: https://symfony.com/doc/current/components/cache.html

or Laravel's https://laravel.com/docs/11.x/cache

Both of these are the first result on google and it's pretty clear that's where they want you to start reading. Additionally in the side nav it's clear how to get from there to other component documentation that's equivalent.

I'm not sure there is a fixed view on the other issue that gitlab pages should be used, that feels like more of a DA decision to me anyway.

Using GitLab Pages for core documentation is, I think, a core decision, not a DA.

Those Symfony and Laravel documentation pages are the equivalent of "GitLab pages", probably generated from MD files.
The api.drupal.org just parses code.

If we take the example of Laravel, these would be the matching sites in Drupal:
- https://laravel.com/docs/12.x => Site generated with GitLab pages from MD files in Drupal core (easy to achieve but maintained by core and generated via GitLab CI)
- https://api.laravel.com/docs/12.x/index.html => api.drupal.org (this is generated with Doctum, a tool I evaluated before porting the "api" module/site to modern Drupal)

Following on the "Cache" example, we'd have:
- Generated 100% by code: https://api.laravel.com/docs/12.x/Illuminate/Support/Facades/Cache.html (this is what api.drupal.org does)
- Generated 100% by MD files: https://laravel.com/docs/12.x/cache (source code at https://github.com/laravel/docs/blob/12.x/cache.md)

OK interesting. I think if the DA doesn't have any objection to gitlab pages, and also doesn't want to provide some kind of alternative hosting, then that is fine then!

For me this is less a replacement for api.drupal.org and more for certain parts of the handbook. We can then remove a lot of the long prose on api.drupal.org and link from there to the new docs for higher level concepts. So https://api.drupal.org/api/drupal/core%21core.api.php/group/cache/11.x would be much shorter, with a link, but still there with the overview of interfaces/classes etc.

I think if the DA doesn't have any objection to gitlab pages, and also doesn't want to provide some kind of alternative hosting, then that is fine then!

I'll tag @drumm to chip in just in case, but I don't think we have any objection. In fact, we support it by default in all contrib: https://project.pages.drupalcode.org/gitlab_templates/jobs/pages/

The hosting for that is done automatically in our GitLab instance.

I fully agree with that approach, one can be way more user friendly, one can be more technical. Both have a place and an audience and should be able to coexist (eg: the Laravel links and repos provided before).

Should you want quicker progress, you could start with a contrib project where you start putting all the MD files, structure, menu, etc and build the site from there and play with all the options available. Once fully validated and agreed, it can easily be moved to core under a "docs" folder if needed.

@fjgarlin, thank you.

GitLab pages will not be indexed in the Drupal.org site search, so there will be some loss of findability, but we probably will want to de-emphasize the site search anyway, as some important projects are on GitLab pages already.

The unique thing API module could do, and does for .txt files, is auto-link references to classes/functions/etc like any docblock comment. For example https://api.drupal.org/api/drupal/core%21INSTALL.txt/11.x has links to other files. That is not used much, so I’m sure serious use would find improvements to be made (URLs aren’t auto linked, but would be less-relevant for markdown) in addition to adding markdown parsing.

That said, I think it is all good to go ahead with GitLab pages. There should be some plan to completely replace and delete documentation like https://www.drupal.org/docs/8/api/cache-api/cache-api → , so we try to avoid the usual problem of good documentation initiatives accumulating into just more versions of more documentation to be disorganized.

> The unique thing API module could do, and does for .txt files, is auto-link references to classes/functions/etc like any docblock comment. For example https://api.drupal.org/api/drupal/core%21INSTALL.txt/11.x has links to other files. That is not used much,

?!

In the entity topic alone, I see 14 links to classes or interfaces.

I thought the issue about using markdown would be keeping API module responsible for outputting them. Moving to gitlab pages is scope creep.

It truly depends on what the goal is.

If we want the api module to parse MD files, it is possible (https://git.drupalcode.org/project/api/-/blob/2.x/src/Parser.php?ref_typ...), but that would just render the MD file, converting it to html, and into an api.drupal.org page (so similar to the https://api.drupal.org/api/drupal/core%21INSTALL.txt/11.x example provided above).

If the conversation is about a more polished documentation in general for core, that's when I refer back to comments #5 and #6 and the example of Laravel or Symfony (in this case they even have two repos, one for the actual code and one for the documentation).

So, it's really down to what the desired goal for core in here and what approach they will take.

The way that 🌱 Document high-level API concepts in an easier format Active is presented, it was about improving DX when writing documentation topics that live in core code.

I took that to mean that the ouput for readers should pretty much stay the same. So the entity API topic would be in an .md file instead of in entity.api.php, but we'd still have a page that looks like https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Entity%21... though at a different URL.

> (in this case they even have two repos, one for the actual code and one for the documentation).

We used to have that a long time ago -- pre-D5 maybe? -- and it wasn't good. The docs were brought into the same repo as core so that they get updated at the same time as the code changes.

So

@drumm in #10

there should be some plan to completely replace and delete documentation like https://www.drupal.org/docs/8/api/cache-api/cache-api → ,

And @joachim in #13:

The way that #3463660: Document high-level API concepts in an easier format is presented, it was about improving DX when writing documentation topics that live in core code.

Both of these are my main goals here - make prose API documentation easier to write (and probably in core), and then consolidate that high level API documentation in core so we can delete/redirect some of the handbook documentation.

I personally do not have a strong opinion between gitlab pages and api.drupal.org. api.drupal.org module seems more convenient for someone reading the documentation because it's not yet another place to look. Feels like there would be a higher chance of going down to one canonical API documentation place instead of up to three. However if we can put the docs in the same place in the same format then it would be possible to experiment a bit.

> api.drupal.org module seems more convenient for someone reading the documentation because it's not yet another place to look. Feels like there would be a higher chance of going down to one canonical API documentation place instead of up to three.

I agree with both those reasons, but for me they are why I am very strongly in favour of having the .md docs displayed on api.d.org.

We can work on that, see how things look and then take it from there to see if it would be enough.

Core already has some MD files (eg: readme) so there is no additional dependencies.

Changing priority mostly for issue queue sorting purposes.

WIP in the MR.

Rendering of https://git.drupalcode.org/project/examples/-/blob/4.0.x/CONTRIBUTING.md...

In #19, we can see how a parsing + rendering of an MD file would look like on the api site.

If this is the expectation, I can go ahead with this. The code is not complex. But I'd like validation of it before I continue.

    $docblock['documentation'] = $markdown_parser->convert($code);
    // Demote h1 to h2 as we already have a h1.

Wouldn't it be simpler to operate that transformation on the markdown source rather than the rendered HTML?

Not so sure, we would need to check multiple syntax options (https://www.markdownguide.org/basic-syntax/#alternate-syntax) and the code might be more complex in my opinion.

I'm merging this as it is. Once it is fully deployed to production and we start seeing MD files pages, we can create follow ups if needed.

It'll be deployed in the next round of deployments. I'll follow up with a link here once it's in production.

Markdown files are now parsed and displayed by api.drupal.org:
- https://api.drupal.org/api/drupal/core%21themes%21starterkit_theme%21REA...
- https://api.drupal.org/api/drupal/README.md/11.x

I created a follow- to allow html markup rendering from trusted projects (aka core): 📌 Allow html rendering of trusted projects in markdown files Active

But in the meantime, core can already test/experiment how MD files look like in the api.drupal.org site.

This looks really good to me, the main thing I'm not clear on is how we can create an index listing these files, and link from one to another in a way that will retain the version in the URL etc.

Something I think I suggested a long time ago elsewhere is that we move to having docs in a MODULE/api folder.

So docs topics would be .md files with paths MODULE/api/TOPIC.md, and the list of topics would be the list of those files. Or we have a master contents page that's created manually. But putting them in an api folder is still a good idea, I think.

Defo, agreeing on some conventions can make everything easiler.

The current MD files are all accessed the same way tho:
- https://api.drupal.org/api/drupal/core%21themes%21starterkit_theme%21REA...
-- https://api.drupal.org (host)
-- api (api project prefix)
-- drupal (project slug)
-- core%21themes%21starterkit_theme%21README.md (url-encoded path to the file)
-- 11.x (branch slug)

That goes for any file path.
We could, in a follow-up, not require the branch slug, check if the path is a file path, and in that case, add it automatically.

That way, all files would have https://api.drupal.org/api/drupal/PATH%21TO%21FILE.md for the latest/default branch, which I think it might be usable and easy enough to cross-link things.

It'd be great to agree on the above first, and then we can do ✨ Use default branch slug on file paths without it Active

That way, all files would have https://api.drupal.org/api/drupal/PATH%21TO%21FILE.md for the latest/default branch, which I think it might be usable and easy enough to cross-link things.

I was thinking how would this work if you're reading the .md file locally, but it would link to api.drupal.org (then presumably get redirected to the latest version) which seems fine to me anyway?

> I was thinking how would this work if you're reading the .md file locally, but it would link to api.drupal.org

Oh right, I see what you mean. Putting URLs to other files in the .md files is going to be pretty poor DX compared to now where you can just do '@link topic' or something similar.

If links contain the path of where the file is, we could scan the code and replace appropriately.

eg: this is a [link](/core/themes/starterkit_theme/README.md) that should work (on markdown)

Could be transformed to:
this is a [link](/api/drupal/core%21themes%21starterkit_theme%21README.md/11.x) that should work (on the api site)

Again, first agreement, then we can repurpose issue ✨ Use default branch slug on file paths without it Active to match what we need here.

#32 sounds good to me?

I think it could be improved to something more like the @topic links we do now in PHPdoc, but it's ok for now -- good enough that improving it shouldn't hold this up.

Great. I updated the issue description in ✨ Use default branch slug on file paths without it Active , so we can continue there.