[Meta, Plan] Move official Drupal docs into repository, host from static site CI artifact

Created on 15 April 2024, 7 months ago
Updated 14 May 2024, 6 months ago

Drupal is an amazing CMS/framework appropriate for building a wide range of ambitious, technically complex, interactive web applications.

Problem statement/motivation

Despite clear competitive advantages in the realms of technology, community, flexibility and more, the project suffers from significant marketing and DX debt of a ~20 year old project. This is particularly true in the realm of the official documentation.

At the outset, I want to state clearly that none of the opinions shared on this meta issue, and the proposals made, are not intended to diminish the many thousands of hours of contributed time spent on the docs to date. Rather, the tools we currently use to share our technical story are outdated and there are better options available to us. Contributors to Drupal's documentation deserve a better stage on which to shine.

There are initiatives underway to improve Drupal's position in the competitive landscape through outreach and marketing, however to my knowledge documentation is not officially part of these plans. I believe we can make a modest and realistic investment of time and effort resulting in a sea-change improvement in Drupal's documentation. A project with clear, accessible, up-to-date documentation is more appealing to developers and by extension, their employers who choose to use Drupal over many alternatives.

We fall short as a project on two main fronts: Outward-facing/end-user documentation experience, and contributor experience.

End-user documentation experience

A user searching for "Drupal documentation" on Google finds the first four results pointing at entirely different landing pages on Drupal.org: " Documentation → ," " Drupal Wiki → " (at /docs), " Develop → ," and " Drupal User Guide → ." These related, but rather distinct, entry points into Drupal present a disjointed experience. In addition, there are many cases where documentation refers to specific versions of Drupal, particularly Drupal 7. As D7 and D8+ are basically different software experiences, this adds to the confusion.

The quality of documentation on Drupal.org varies from detailed and technical to wrong and/or outdated, or not reflecting "best practice." Due to challenges in the contribution experience (see below) it is sometimes unclear that documentation can be updated, or user guide ownership means permissions are a hurdle to improving or correcting docs.

The limitations of the official docs have led some in the community to go their own way. A significant body of Drupal documentation exists in blog posts (e.g., on agency sites) and even comprehensive alternative docs like Selwyn Polit's Drupal Book. There will always be "docs in the wild" for any software, however my personal experience is that these docs are often superior or more trustworthy than the official documentation on drupal.org.

If you are an evaluator, Drupal's official documentation does not instill trust when compared to docs for Symfony or Laravel. Interestingly, the official WordPress docs are a bit neater but (IMO) very sparse and unhelpful.

Drupal's documentation is also not exclusively limited to the "official" docs. An important source of truth and updates to the software are contained in change records → . Core has a strong quality gate for CRs, however many users will find them difficult to discover via the documentation system and they are primarily linked to resolved issues. In fact, releases of core → link to commit messages only, and not change records, meaning end users do not have a "direct" path to viewing all change records applicable to a release (despite this information existing in the drupal.org database, at least by way of a freetext field.)

Contributor experience

Community contribution is at the heart of Drupal's success. We enjoy a real "first-mover advantage" in our community structure, especially our issue credit and organizational recognition system which has helped to incentivize and recognize open-source contributors.

Notably, contributions to documentation are severely under-counted in the contribution credit system → . In fact, it's not counted at all! Drupal.org user profiles do say "Over 5 edits to documentation," but that's both vague and unrelated to your issue credit "score," if that's important to you/your organization.

Drupal.org's documentation system is centered around the wiki, which is editable by registered users and does boast a "discuss" functionality similar to Wikipedia. While the wiki model works well for many knowledge bases, very few users contribute to the docs in this manner and there is no staging or change-proposal system in place.

Finally, and by way of segue into proposed solutions, documentation is decoupled from code. This means that updated documentation cannot be reliably enforced as a quality gate for core commits.

Proposed solutions

I propose that the official Drupal documentation move into the codebase, with documentation coexisting with code. With the move to GitLab CI, we have the ability to add custom jobs to the Drupal Core CI configuration, which could include generating a static site for all supported versions of the project. This site would be a CI artifact and would immediately update a docs site, e.g. at https://docs.drupal.org.

Now is the time for action on improving Drupal's documentation, both for end-users and contributors. Drupal.org is on D7 and much of its functionality is being pared down as a result of the GitLab migration. As the DA must plan the future of the D7 install, the docs system must be accounted for and migrated "somewhere."

Just like drupal.org's homegrown issue tracking system was created because it literally predates GitHub, GitLab and similar, the docs wiki has been eclipsed by more modern alternatives.

Benefits of such a move are significant. Contributors of documentation would get contribution credit for their work with no change to the underlying technology stack for assigning credit. The drupal.org technical footprint can shift docs away from the CMS and into the realm of cheap, repeatable CI jobs. Hosting costs are (potentially) reduced as a static site can be hosted on cheap object storage behind CDNs (cache invalidation considerations notwithstanding.) The docs can be more modern in style and functionality, since we can offload the technical foundation to open-source software like Docusaurus. And the docs generation system can even hook into things like automated docs generation from code (e.g. for API docs) as well as linting the documentation for dead external links.

A growing number of projects in the drupal.org ecosystem are already publishing their documentation with GitLab pages. We could wire up code to identify which projects have a pages integration and automatically link to them. External resources can be linked as well, creating a more interconnected web of docs, official and otherwise.

Non-technical or less-technical editors of documentation can take advantage of the web IDE in GitLab for proposing docs changes, and we can move to Markdown for formatting which is easy and increasingly familiar to end users (if Redditors can do it, so can we!)

Perhaps most importantly, core contribution can make docs updates and additions a mandatory quality gate for code contribution. If you create a new API or feature, it must be documented (if even just in Doxygen which is then parsed into something pretty). We could even convert change records to be in the codebase, and they would be packaged up at release time. (See Expo for an example of this.)

We have the tools, and even a large amount of existing documentation to seed the new-and-improved docs. Through at least July, my core contribution time is sponsored by the kind folks at Optasy, so I can commit to helping lead an initiative. While this is a "big" change for Drupal, I honestly don't think it's that "hard." The main challenge will be getting agreement on a project plan and coordinating with relevant stakeholders (core committers, the DA/drupal.org maintainers, existing docs owners) to build consensus. This community is full of people with a bias for action, and I'm optimistic that this could help with selling Drupal as a modern framework/CMS that is anything but old and crusty.

🌱 Plan
Status

Active

Version

11.0

Component
Documentation  →

Last updated 3 days ago

No maintainer
Created by

🇺🇸United States bradjones1 Digital Nomad Life

Live updates comments and jobs are added and updated live.
  • Documentation

    Primarily changes documentation, not code. For Drupal core issues, select the Documentation component instead of using this tag. In general, component selection is preferred over tag selection.

Sign in to follow issues

Comments & Activities

  • Issue created by @bradjones1
  • 🇺🇸United States bradjones1 Digital Nomad Life
  • 🇺🇸United States bradjones1 Digital Nomad Life
  • 🇬🇧United Kingdom catch

    This issue might need to be against Drupal.org (not sure which queue) rather than core, unless the proposal is to put the docs into core itself (I would assume it would be a separate gitlab project, or multiple projects, with its/their own maintainers though?).

    For an example of where the existing documentation is hard to maintain, see the running list of pages that need updating on #3398293: Consolidate local development environment documentation to recommend DDEV → , many of them are duplicates or nearly-duplicates of each other. I think that's an example of where there is too much documentation, and the 12+ pages really need to be pared down to one or two pages.

    Doing this would help solve some of the remaining issues with 📌 Template for Core release cycles diagram RTBC (except it would probably live in the docs repo instead of core then).

  • 🇺🇸United States bradjones1 Digital Nomad Life

    unless the proposal is to put the docs into core itself

    That is my proposal. One repo, with the docs being a CI artifact. I was updating the title to better reflect this while you were writing your comment! So I do think this is a core issue. (Though it certainly overlaps with drupal.org infrastructure that would be affected in the course of rolling this out.)

    Your question

    I would assume it would be a separate gitlab project, or multiple projects, with its/their own maintainers though?

    is an important one to answer early on, since it speaks to governance. My personal thought on this would be that perhaps there would be new core committers who focus mostly or only on docs, but that there need not be a separate repository or governance structure. Docs changes are "just as valuable" as code, so let's use this as an opportunity to get more people to contribute to core via the docs.

    That said, I think the real advantage here is that docs changes would be made in the same MRs/issues as code changes. If you're changing or adding an API or feature, the MR must also include updates to the docs. Similarly, I think we could re-work how change records are done where they are committed to the repo as well. We often see how CRs are forgotten, outdated, or not published because while they're linked on drupal.org, they are not intrinsic to the change being made. This will also help with the transition to GitLab since there is no GitLab equivalent of a free-standing change record. Basically, everything needed to make amazing release notes and updated docs would be contained in the commit.

  • 🇺🇸United States bradjones1 Digital Nomad Life

    If the existing docs were migrated to a repository, we'd have exactly the same problems that there were 12 pages, but it'd be 12 pages to delete in an MR.

    Migration would be an issue to figure out as part of this transition, for sure.

    You mention this as a "social" problem, and there is definitely a valid issue in deciding how to address differences in style or simply preference in the docs. The linked issue re: a recommended local development environment is a good example of how we're failed by the current system of not allowing for staged changes in the docs. In theory, you could have an issue like this on GitLab with competing MRs to solve the problem, but at the very least you can then see the various alternatives against one another instead of talking about it in circles. Or worse, pointing at various Google Docs or other fragments. I would be interested to hear what the existing core committers think of "sharing" a codebase with docs, but I do think the benefits outweigh the risks.

    As we have the tools to do far more continuous integration, I think a good outcome here would be that we have far _more_ commits to core, many of which are edits to the docs. Because docs changes don't come with the same BC and stability/maintainability considerations as code, there's less risk in "getting it wrong." The standard for updating the docs could simply be, is this a move in the right direction? Great, committed. Want to make another change to it next week? Great, let's do it.

  • 🇬🇧United Kingdom catch

    That said, I think the real advantage here is that docs changes would be made in the same MRs/issues as code changes. If you're changing or adding an API or feature, the MR must also include updates to the docs.

    This is demonstrably not the case with the documentation that core already ships with, look at the commit history for git log core/core.api.php

    For example, it links to generic documentation about SOAP and XML-RPC, neither of which are in core (xml-rpc used to be), but does not even mention JSON:API which has been in core for years now.

    Somewhere in the inclusion of JSON:API into core, we should have had an issue to include it in the high level web services docs, but the fact this never happened, there is probably not even an issue, is both a symptom of the disjointedness of the current documentation across different places, but also makes me extremely wary that the core issue queue would be the right place to put all this.

  • 🇺🇸United States bradjones1 Digital Nomad Life

    This is demonstrably not the case with the documentation that core already ships with, look at the commit history for git log core/core.api.php, it is... sparse.

    I appreciate this perspective, as it comes from a different place from where I sit. This might be a bit of a spicy take, but I haven't consulted api.drupal.org for any meaningful information about the framework since I regularly worked in D7. The interface is clunky, and yeah, as you mention, the content is severely out of date.

    It's also a great example of how disjointed our documentation is. So disjointed in fact, that I missed api.drupal.org entirely when talking about official docs. It is linked under "complete API reference" but for your average user that doesn't mean much to me. (E.g., in 2024 I think of "API" as an external API, and here we mean it to be the PHP API, which is not really common terminology.)

    So in your example, I think that file isn't updated because our documentation is so bad/outdated/tough to access and maintain that it's dead on the vine. As it stands, what's the point of updating a Doxygen block that is 3 clicks deep on a page on api.drupal.org that I'm not using, and isn't linked to any of the other "wiki" documentation on JSON:API?

    I do think we're on the same page, especially as you say

    ...both a symptom of the disjointedness of the current documentation across different places, but also makes me extremely wary that the core issue queue would be the right place to put all this.

    I do think we have an opportunity here with the impending changes necessary to drupal.org, as well as the DA's reboot of marketing efforts to sell Drupal, to make a big change for the positive. I don't so much care about what repository it ends up in (though I do still think there's a strong case to be made for it to be alongside core) as I do providing the project modern tooling for writing and releasing docs and making them reflect reality. On that we totally agree.

    As I think more about this topic, I think the better characterization of much of our documentation is "neglected," or even sometimes abandoned. There would certainly be some hard decisions in this process. Perhaps part of the migration could be a bold purge of anything that is old, clearly wrong, or confusing, and giving ourselves a solid, if smaller, starting point to move forward with something better.

  • 🇳🇿New Zealand quietone

    @bradjones1, thanks for commenting in the change record management issue, 📌 [meta] Improve change records Active

    This is an intriguing idea and may be best suitable for improving the quality of change records and their management. They are a small set of documentation and directly linked to an issue.

    A final note, I found the proposed resolution is the 3rd paragraph of that section and the rest is reasons supporting the idea. I think it would be easier to read and follow if the proposed resolution section was just the 3rd paragraph.

    Oh, I also agree that the core queue isn't the best place for this. This is something that could affect all project.

  • 🇺🇸United States bradjones1 Digital Nomad Life

    @quietone - thanks.

    Do I take your comment regarding change records to mean that you would only want to see CRs addressed with an "in-tree" approach? If so, do you think a broader change to docs is unnecessary? (Or that there's a necessary change, but this isn't it?)

    Updated the IS per your feedback. I agree it's a pretty broad/interconnected proposal, but also would like to continue to discuss the option of documentation living in the project repo (as it does for a growing number of contrib projects) until/unless someone empowered to pour cold water on that says it's not happening.

  • 🇳🇿New Zealand quietone

    @bradjones1, I read this MR shortly before an appointment and I got it wrong. On another read it is obvious that I skimmed this sentence

    I propose that the official Drupal documentation move into the codebase, with documentation coexisting with code.

    I disagree with adding the official Drupal documentation to the code base, including the change records. I do not see a gain in forcing people to download more bytes than they need to in order to run a Drupal site. And even if the official Drupal documentation was in a repository I don't think that in itself will improve the quality of the documentation as mentioned by catch in #4.

    I apologize for the confusion.

    Also, on further checking I think this should be in Documentation project → ,

  • 🇪🇸Spain fjgarlin

    I guess also it depends on the understanding of

    official Drupal documentation

    .
    For example, here are three ways to understand it and some thoughts on them.

    > Replacement for api.drupal.org
    If we understand it as replacement for api.drupal.org, there are reasons not to. I did some investigation in the day (https://github.com/fjgarlin/phpdocs_generators) when deciding whether to rebuild api.drupal.org in D10+ or use one of those tools.

    All those tools started to be really limited when the size of the project increased, due to massive JS files being generated.
    Those projects were thought as replacement for api.drupal.org as they did something similar: parse code.

    But due to the big limitations they had with big projects, we opted for rebuilding the site with D10+ (staging URL: https://api.stage.cluster.drupalsystems.org/).

    > Basic Drupal usage documentation
    If we want to generate a site from /docs markdown files, we could actually leverage GitLab pages. This is what we do to generate the documentation site for the GitLab templates in https://project.pages.drupalcode.org/gitlab_templates/, which is generated from the files in the /docs folder here https://git.drupalcode.org/project/gitlab_templates/-/tree/main/docs?ref...

    But I think it'd be key to limit the scope of what it’s really wanted. If we want to include an entry point Drupal-related documentation site, like https://selwynpolit.github.io/d9book/, that’d be easy. we could use GitLab pages and generate it easily under this URL https://project.pages.drupalcode.org/drupal (which is 404 now obviously). A few dozens or few hundreds of files might work, many hundreds or thousands of files might not.

    Then there is the challenge of keeping it up to date.

    > Drupal documentation + change records, user guides...
    I think this might be totally out of scope, otherwise it might create more issues than the ones solved. The problem here would be that we'd need to migrate everything or nothing. Otherwise we'd still need to write migrations for the ones remaining in drupal.org, then clean up the ones that were moved...

    Even if we'd do the above, we'd have the problem of the size mentioned before. There will be a point where the generated site will not be performant at all, for sure. It might be in 2 months or 2 years, but it will happen.

    Another problem would be the search, as we would not be able to connect the search on www.drupal.org (solr or elastic-search based) with a JS-based search of the generated site.

    --

    All in all, I just wanted to say that the scope needs to be clearly defined, because the solution will depend entirely on it.

  • 🇸🇰Slovakia poker10

    I agree that the documentation can be improved in various aspects, so thanks for creating this issue/proposal!

    I am not sure we can compare whole d.o. documentation with Symfony, or Laravel, as it seems to me these have pure technical documentation. Drupal has a lot of non-technical documentation pages (like this https://www.drupal.org/docs/develop/issues → , or other similar sections), and I think these pages benefits from a wiki-style documentation approach (editable by everyone and can contain information unrelated to the code). If we move such documentation to code, we will restrict the ability to make changes to technical users and it will also depend on a group of committers (so probably adding more work).

    Also there is a part of the documentation with information about contrib modules, which is also worth mentioning (for example https://www.drupal.org/docs/extending-drupal/contributed-modules/contrib... → ), because this would have to be outside the core repository, editable by contrib maintainers.

    Then, we have APIs documentation here: https://www.drupal.org/docs/develop/drupal-apis → , which can probably be more bind to the code itself, but it is only a part of the d.o. documentation. So yes, the scope of this is very important here.

  • 🇫🇷France andypost

    Btw the User guide is already a static files with broken 💔 regeneration, it' still looks the low hanging fruit to fix first as it multilingual and only predictable regeneration is a blocker for it (except leader)
    I think it's a first candidate to move to gitlab's pages, maybe over a time it can be replaced with help topics from core

  • 🇺🇸United States bradjones1 Digital Nomad Life

    Updated project to Documentation/Docs infrastructure component.

  • 🇺🇸United States bradjones1 Digital Nomad Life

    Thank you everyone for your thoughtful replies to my big hairy idea. A few responses/clarifications inline, as well as some thoughts on how to proceed.

    Re: #11 and #12 on adding content to the core repository:

    I do not see a gain in forcing people to download more bytes than they need to in order to run a Drupal site...Even if we'd do the above, we'd have the problem of the size mentioned before.

    This is a packaging question and is not really an issue, given the only supported method of installing Drupal is Composer. A similar discussion was had in #3426047-28: Bump PHPStan to level 9 and accept a large baseline → and documentation files can be excluded from the distributed package by using .gitattributes. Also given the majority of the source files for docs would be text and binary, cloning time would not be strongly affected. (Keeping in mind also when you clone the full repo you are getting every commit from the dawn of Drupal to today, and it's still super zippy.)

    Regarding api.drupal.org, it seems there is a path forward for it, at least when it comes to the technology stack. As mentioned in #7 and #12, though, there's the question of keeping things updated. I would say this is an issue no matter how the docs are managed and rendered. So on one hand this is a valid point - we need to find a way to incentivize people to keep the docs up to date with changes to code and process, but I don't think one can make the case that a decoupled docs system is any better at this than in-tree. That's why I suggest that changing our tooling could allow us to enforce up-to-date docs better, by making it a quality gate for committing to core.

    E.g. to catch's point on outdated docs with the API feature set - when JSON:API came into core, the docs would have come along with it in the MR, and at least in theory the docblocks that flow through to api.drupal.org would have been updated as well if that were our policy.

    That's a good segue into the issue of whether in-tree docs are "too technical" for contributors. Per #13:

    If we move such documentation to code, we will restrict the ability to make changes to technical users and it will also depend on a group of committers (so probably adding more work).

    I respectfully disagree with the idea that so-called "non-technical" users will not be able to edit documentation. As I mention above, markdown is very common throughout the non-geek web, and MD literacy is likely to be even higher among the Drupal community as these users are likely to encounter MD in their jobs (e.g., in ticketing systems.) Markdown will also be the method by which you style issues and comments when we make the GitLab conversion. Any open-source documentation build system we would use has integration for "Edit this page on GitLab" or similar. GitLab has a web IDE with real-time preview for markdown contributions, and it even walks you through creating an MR and commit with the UI. No repository cloning necessary.

    Re: the point about committer burden, we already have a few active core contributors who work on documentation initiatives. With the increasing use of GitLab for our repo management, we could potentially use code ownership and maybe commit hooks to "sandbox" docs from core's shipped code. More commits (and perhaps even a few more committers within a "docs fence") would IMO be a good thing.

    A further advantage over a repo/static site build is the ability to generate full-site previews of the change. This would require some CI integration with artifacts, but it's rather trivial to do. Wiki-style editing is basically an all-or-nothing make-the-change-or-don't paradigm. Yes, there are discuss pages, but let's be real - is anyone really using the discuss pages? And even if so, you have no way to stage and share your contribution for review by others.

    What's more, static docs generators natively support versioning and multilingual. The former is super important as Drupal now allows new features to be added in minors. The documentation can be further segmented into minor releases, meaning you will only see applicable docs for features available in your particular version. And as we are looking at a D10 LTS release, this is non-trivial: if you are on D10 LTS, your docs should reflect the state of the LTS vs. mainline.

    Re: the size of static-site docs, I don't think the comparison to the api.drupal.org size constraints are apt. A static docs site is just files on disk, and there is no real performance hit to having more files in my experience. Indexing codebases is a different problem space.

    That leads me to my last point, which is that we are kinda already doing in-tree docs anyway. @catch mentions 📌 Template for Core release cycles diagram RTBC , and core governance has Markdown in this repo. Not to mention the various .api.php files which drive api.drupal.org. And contrib is being explicitly encouraged to create and manage documentation via GitLab pages.

    I agree defining scope and child issues here would be paramount. For instance, while related to documentation overall, change records can be handled separately (even if they get rolled up into a docs static site) and affect other processes like release notes. That could be hashed out in the existing 📌 [meta] Improve change records Active .

    @drumm linked #2762837-67: Migrate documentation into the new system → in Slack regarding the reorganization/migration of certain content types on drupal.org. Looking through the linked pages, I personally am left with the impression that this would not be a useful expenditure of contributor time. The details certainly need to be hashed out, but my over-arching point is that managing our own documentation systems on drupal.org is not a responsible use of limited resources.

    IOW, I am submitting that what we're doing now for documentation tooling isn't really working for us. The GitLab initiative, impending sunset of D7 on drupal.org and a shifting competitive landscape in the past 20 years gives us an opportunity to make a great leap forward. If what we're doing now is preferable to some of the alternatives we're discussing, I'm open to that conclusion, but I personally don't see it. Many contributors to Drupal also contribute to other open-source projects, and versioned documentation (whether in-tree or a sister repo) is a very common pattern.

  • 🇳🇴Norway hansfn

    Just a quick background: I'm not a core developer or an active module maintainer. However, I have been on the documentation team (if that still exists) for a long time, and for some years now I have monitored all new doc guides / pages added (to our Wiki). I really, really welcome this effort, but I hardly know where to start talking about this ... I have added two random related issues / efforts.

    1. Should we setup some meetings to discuss this instead of writing page up and down? This topic is important enough to try some real-time discussion too.
    2. Versioned documentation is great. Why shouldn't we be able to do it when Symfony does it? (Symfony setup docs for 6.4, 7.0 and so on.)
    3. We need to clearly define the different docs we have and how we want to manage it. I don't think everything should be managed as part of code (but we still should versioning as much of it as we can).
    4. What about curation? I like that using GitLab automatically adds a review step. I think it will make our docs much better, but it of course raises the bar a little bit ... At the same time, there is no review on Wikipedia. But the Wikipedia policing is quite strict. I prefer up front policing ;-)
    5. We need a clear plan for module documentation too.
    6. And should we also support those who write big guides covering all between heaven → and earth → ...
    7. And so on ...
  • 🇺🇸United States ChristophWeber

    This is an interesting proposal and similar ideas have been kicked around among various people. I have submitted a BoF for DrupalCon to discuss the very topic, I encourage everyone to check the BoF schedule for "Improving the drupal.org Developer Experience" and join us there.

    Our own ideas are centered around tackling the contrib space first and rearchitect how documentation works there. Essentially move contrib docs into code (as proposed above) and therefore let everyone who can write MarkDown and submit a PR on GitLab be a contributor. Keeping contrib docs in code would also empower maintainers in a big way and encourage them to keep docs in sync with code. Contrib docs in code also opens the door to let the Doxygen mongrel we have scan code and drop API docs into the doc tree, plus employ other automated or AI tools to enrich the docs. Finally, we would have version support in the docs like our friends in Symfony-land have enjoyed for a decade.
    As a second stage we think it would be time to rethink and rearchitect core docs.

  • 🇺🇸United States ChristophWeber

    Regarding curation as @hansfn stated in #17:
    I believe Drupal core deserves professional attention to documentation, and perhaps the top N contrib projects too. (Actual N to be decided)
    For the rest of contrib the PR review is a fairly decent gate.

  • 🇬🇧United Kingdom catch

    So I think the layer that is very noticeably missing/hard to find in the current docs landscape is 'high level developer docs' and this is what are front and centre when you look at frameworks like Symfony.

    e.g.
    https://www.drupal.org/docs/drupal-apis/cache-api →

    The first link goes to https://www.drupal.org/docs/8/api/cache-api/cache-api →

    The first heading on that page is about cacheability metadata, but that duplicates https://www.drupal.org/docs/8/api/cache-api/cacheabledependencyinterface... → which is also about cacheability metadata and is linked from the first page lower down anyway! It shouldn't be the first thing you read about at all. There is hardly anything else there.

    Before the heading, there is a brief preamble, which links to https://api.drupal.org/api/drupal/core%21core.api.php/group/cache/8.2.x (yes, it links to the 8.2.x version).

    That page only covers the low level cache backend interface, not cache contexts and render caching, but it does at least have an introduction and more detail below.

    Compare this to https://symfony.com/doc/current/components/cache.html where you have a clear introduction, can read down, skip to sections etc.

    I am the cache API maintainer and a core committer, and I have no idea who is responsible for https://www.drupal.org/docs/drupal-apis/cache-api → or whether I would be able to just go in and make major changes without getting permission from someone with different roles on Drupal.org

    This is where supporting some markdown shipped with the code base would be nice, but I would see that as replacing the phpdoc 'topics' that we have in core already (that are hard to update/format), and only a few sections of the wiki, not all of the core documentation by any means.

  • 🇺🇸United States mherchel Gainesville, FL, US

    👋 Hi everyone. Its good to see this discussion happening here. We've been having similar discussions with the Drupal Association and various core committers and stakeholders.
    Some of my thoughts (that I'm fairly confident are shared across the group)

    • Great documentation is a real blocker for people getting started in Drupal.
    • Our competitors make this easy.
    • We agree that we need to have a file based documentation system that is easy to keep updated.
    • Currently there is no one empowered to make tough decisions within Drupal's documentation
    • The correct documentation is hard to find alongside the "noise" of old or invalid documentation

    The outline below clearly lays out the problems, goals, and proposed solutions and more.

    Problems:

    • People do not have a positive first experience when coming to use Drupal’s documentation. Why?
      • The information is ‘generally’ there, but hard to find
      • Information is very inconsistent quality and un-clear if it’s up-to-date or not
      • Outside of the user guide, there’s no “start here, go here next”
        • The user guide is very “build your first blog with Drupal”. After that it’s not clear “go here next”.
      • There’s a lot of good information on api.drupal.org but its hard to use
      • Google points a lot of people toward Drupal 7 documentation
    • Documentation is a pain to keep up to date. Why?
      • Use of HTML
      • Review process is weird
    • Invalid documentation sits around for years
      • No one is empowered to delete it
    • Documentation working group is inactive, probably needs new members who are currently engaged in Drupal community
      • It's hard to write world-class documentation by committee. Especially without good governance.
    • Updating the user guide (which is in Git) can be a pain in the butt for Neil (this according to Joe)
      • Additionally it’s difficult for anyone else to be able to make changes and preview those changes since it’s difficult to build the guide locally. It would be easier for translation teams to preview changes for example.

    Goals:

    • Lower burden of maintenance for documentation system
    • Make it easier to keep documentation updated
    • Separate docs based on Drupal major version
    • Tender loving care
    • Quick turnaround to documentation changes
    • Differentiate between “officially maintained” and “community maintained” docs in order to limit scope

    Solution being proposed:

    • Overhaul documentation to use build documentation to use an out-of-the-box open source industry-standard platform (e.g. ReadTheDocs)
    • Hire or contract a person to oversee the documentation system. This person would be part of the documentation working group, and would be empowered to make difficult decisions.
    • Split documentation between “official” and “community”. Hired person would have
  • 🇺🇸United States mherchel Gainesville, FL, US

    I feel where we're at a point where the highest levels of the DA and the core team are now aware of this and are willing to dedicate resources (money). I'm about to be in the process of putting together a rough proposal and budget to take to the board of directors to make this happen.

  • 🇺🇸United States bradjones1 Digital Nomad Life

    Thanks, Mike - this is an excellent summary and you sum up well an over-arching hurdle here - someone empowered with the proper level of high-level support to make tough decisions and move forward boldly. I really believe that a better experience from the drupal.org home page to meaningful documentation will be a sea change in usability and project respect in the competitive landscape.

    FYI your comment cut off after "Hired person would have..." I suspect due to a length constraint, mind pasting the rest of your comment so we can see the whole thing as intended?

  • 🇺🇸United States mherchel Gainesville, FL, US

    Done! Yeah, I was copying and pasting an outline from google docs (and then converting it into HTML to paste here), and I missed the last line.

  • 🇳🇴Norway hansfn

    Differentiate between “officially maintained” and “community maintained” docs in order to limit scope

    Yes, that was the starting point for curated / official guides → too. Using a better tool (e.g. ReadTheDocs) for the job helps, but the though problem is exactly how to differentiate and how to handle "community maintained". There is so much stuff I would like to delete, but people have invested so much time that it is hard to do without any guide lines. I think we need to discuss this properly.

    PS! We have struggled with this for a while → ...

  • 🇺🇸United States selwynpolit

    One of the struggles I had when creating my book - Drupal at your fingertips was to identify the audience. I've seen documentation that tries to be suitable for new users vs experienced users. That is a very tall order. We have the challenge of having so many different identities:

    • content editors
    • site builders
    • curious visitors
    • hobbyist developers & tinkerers
    • pro developers
    • front-end developers
    • back-end developers
    • full stack developers
    • business folks?
    • others?

    This may require that there be different versions of the docs. I tried to focus on pro developers but sprinkled in "the basics" sections in some spots. Not sure if that was the best way to go...

    If this is helpful to the initiative, I am using markdown and a theme called Vitepress and folks can submit PR's via github to update the book. Please feel free to check that out.

  • 🇺🇸United States drumm NY, US

    Drupal.org is on D7 and much of its functionality is being pared down as a result of the GitLab migration. As the DA must plan the future of the D7 install, the docs system must be accounted for and migrated "somewhere."

    I do not see a way around migrating a large portion of the documentation as-is on Drupal.org. There are 7,679 documentation pages in 1,972 guides. Unless 100% of the pages are put somewhere else, replaced with redirects, and deleted, we are still maintaining the current documentation system on Drupal.org. Migrating 1,000 nodes is the same as 10,000 nodes, so this has no potential to save cost or reduce technical debt for Drupal.org’s migration past Drupal 7.

    Where we do need to draw the line for technical debt is that Drupal.org actually has 2 other documentation systems leftover from previous efforts to “solve” documentation. #2762837: Migrate documentation into the new system → is needed to remove book nodes from Drupal.org, as these do not make sense to migrate on top of the current guides and pages.

    And as mentioned already, there is an existing official docs in Git, https://www.drupal.org/docs/official_docs → , which is not very active, https://git.drupalcode.org/project/user_guide/activity. I do not think this is a tooling problem. Git doesn’t make it easier to find motivated people with time and a shared vision of what needs to be documented for a well-defined audience. Git may make the workflow easier for some people, but the important part is good, organized writing and time to do it.

    Btw the User guide is already a static files with broken 💔 regeneration, it' still looks the low hanging fruit to fix first as it multilingual and only predictable regeneration is a blocker for it (except leader)

    🐛 Problem with encoding on Drupal.org user Guide for all chars except latin. Active was already fixed.

    So I think the layer that is very noticeably missing/hard to find in the current docs landscape is 'high level developer docs about low level APIs' and this is what are front and centre when you look at frameworks like Symfony.

    Getting more specific like this should help get to something achievable. I’m very skeptical of any proposal that claims it will solve Drupal documentation. Starting with one thing makes it possible to do that well. Something like high level developer documentation identifies an audience and a gap that can be filled.

    While the wiki model works well for many knowledge bases, very few users contribute to the docs in this manner and there is no staging or change-proposal system in place.

    The design of documentation guides on Drupal.org was for guide maintainers, and other people interested, to get notifications of edits. This allows people to be bold and make edits, and for others to see changes and review after the fact. If a maintainer is still active, they can catch any further improvements needed. If the maintainers are no longer active, the changes aren’t stuck in a change-proposal system.

    Any documentation plan needs to think long-term about who will maintain the documentation long-term. People don’t do the same thing forever, and that is a big part of what led to the current disjointedness.

    On GitLab pages

    With increasing use of GitLab pages, we’ll also have to think about the Drupal.org site search. If a lot of documentation is missing from www.drupal.org, and issues are no longer there either, we need to consider removing the site search or getting GitLab pages indexed. (Removing site search would not significantly reduce technical debt - we still need a solid, tuned, full-text search for project browsing.)

  • 🇺🇸United States itmaybejj

    It's worth noting, if you are triaging docs, that a decent percent of the existing docs are broken 🐛 Classes missing from Drupal 10 Fixed .

    I think it would help, regardless of what system we use, if there was a core documentation template focused on the person who is going to use but not understand the code. Write it for the new user. A lot of the existing function/class documentation is along the lines of "X: provides an interface to make an X."

    E.g.,

    1. Purpose of function
    2. Example use
    3. Links to extended how-tos

    If that was templated, it might also help keep track of out-of-date documentation, since somebody doing a code review would see the URL that would need to be updated with the change.

  • 🇺🇸United States selwynpolit

    I like the way the Drush documentation works in that the link to "latest" at https://www.drush.org/latest/ dynamically takes you to the latest version i.e. version 12 of the docs at https://www.drush.org/12.x/
    This could be effective for overview pages where we could set up redirects to send folks to the latest and greatest

Production build 0.71.5 2024