Gradually replace Drupal's AJAX system with HTMX

Created on 27 November 2023, 3 months ago
Updated 21 February 2024, 1 day ago

What is the problem to solve?

Drupal's AJAX system has served us well for over a decade. For over 15 years if you count its predecessors in CTools and AHAH. At the time of its creation, we had to create a custom system, because there weren't existing libraries with the functionality that we needed. It was fun to be at the bleeding edge of integrating AJAX into a server-rendered framework like Drupal.

But now HTMX exists. It's a well designed, well documented, well maintained library that's gaining a lot of traction, and I think it supports pretty much everything that Drupal needs from an AJAX system, including the ability for the response to trigger additional actions, which we can use to implement any of our current custom commands that don't have an already implemented corollary in HTMX. In the long run, I think we'd be well served by moving from our custom AJAX system to a library that an increasing number of developers are gaining familiarity with.

However, I don't think it would be easy to preserve Drupal's current AJAX API and change it to use HTMX under the hood. Instead, I think it would be better to convert usages of AJAX (for example, in Field UI, Layout Builder, Views UI, etc.) to use an HTMX-based API.

This would mean maintaining both Drupal's current AJAX API and a new HTMX API until contrib/custom modules have had sufficient time (possibly several major Drupal versions) to convert their ajax usages to htmx.

Who is this for?

Developers

Result: what will be the outcome?

(Describe the result: how will this improve things for the stated audience(s)?)

How can we know the desired result is achieved

(Usability test, a metric to track, feedback)

🌱 Plan
Status

Active

Component

Idea

Created by

πŸ‡ΊπŸ‡ΈUnited States effulgentsia

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @effulgentsia
  • πŸ‡ΊπŸ‡ΈUnited States cosmicdreams Minneapolis/St. Paul

    2 birds, 1 stone? How about the Off Canvas Dialog?

  • πŸ‡¬πŸ‡§United Kingdom catch

    There's already some work done on πŸ“Œ Add a native dialog element to deprecate the jQuery UI dialog Needs work on replacing dialog, although not moving especially quickly.

    On HTMX this is new to me, it seems like given it allows adding arbitrary extra things we could probably replicate everything. An annoying thing to replicate will be ajax_page_state and library handling, but that's perhaps a decent test case along with a relatively simple otherwise AJAX callback like replace.

  • I hadn't heard of HTMX, but after looking at the README and some examples, I think it's really cool. It would probably make writing code for AJAX forms a lot easier.

  • πŸ‡ΊπŸ‡ΈUnited States cosmicdreams Minneapolis/St. Paul

    The aspect of what makes HTMX interesting in OffCanvas dialog (and perhaps we should talk about the "toolbox" part of the layout builder experience) is that dynamic behaviors are all executed on the server. In Drupal, and other systems that were built in it's era, priority was given to server controlled logic. HTMX allows server driven system to have more dyanamic behaviors, as long as responses to dynamic requests are provided in HTML (instead of JSON).

    For Drupal to fulfill that requirement, we can provide a solution as simple as a Controller that returns a renderable (like every example of a custom block has ever showed) or something as advanced as BipPipe's placeholder handling.

    I'll take a shot at this idea this weekend and see what I can hack up.

  • πŸ‡ΊπŸ‡ΈUnited States cilefen

    Is the licensing compatible?

  • πŸ‡¬πŸ‡§United Kingdom catch

    It's BSD licensed which afaik is fine: https://github.com/bigskysoftware/htmx/blob/master/LICENSE

    Can't remember off the top of my head whether we've shipped BSD licensed js with core before.

  • I understand it's the holidays, but just chiming in that I'm very interested to see @cosmicdreams's demos of this.

    I've been experimenting with HTMX on a project, and it works great for giving the site a SPA-like feel. The problem I've run into is one typical of AJAX loaded content; That is, Javascript won't fire on subsequent page loads because the content was not a part of the DOM. HTMX's onLoad() function (https://htmx.org/docs/#3rd-party) is the remedy for this, and I can make it work if I write vanilla JS, but I do not know how to integrate it with Drupal behaviors.

    This works. It fires on every "page" loaded via HTMX:

    (function (Drupal, once, htmx, drupalSettings) {
    
      'use strict';
    
      htmx.onLoad(function(content) {
        var items = document.querySelectorAll('.js-listItem');
        console.log(items);
      });
    
    })(Drupal, once, htmx, drupalSettings);

    These do not:

    (function (Drupal, once, htmx, drupalSettings) {
    
      'use strict';
    
      Drupal.behaviors.testHTMX = {
        attach: (context, settings) => {
          var items = context.querySelector('.js-listItem');
          console.log(items);
        }
      };
    
    })(Drupal, once, htmx, drupalSettings);
    (function (Drupal, once, htmx, drupalSettings) {
    
      'use strict';
    
      Drupal.behaviors.testHTMX = {
        attach: (context, settings) => {
          htmx.onLoad(function(content) {
            var items = document.querySelectorAll('.js-listItem');
            console.log(items);
          });
        }
      };
    
    })(Drupal, once, htmx, drupalSettings);

    Let me know if there is any assistance I can provide in moving this forward. I'm no JS expert, but I am happy to assist where I can.

  • πŸ‡ΊπŸ‡ΈUnited States cilefen

    In the third example β€œcontent” is undefined. Did you men β€œcontext”?

    β€œContext” is sometimes the full DOM and sometimes part of the DOM, if there was an Ajax update. Usually you need to use it within the implementation.

  • You are correct. I did do it that way originally, but in my copy/pasting to get it into the post, I did not change it back to context.

  • πŸ‡ͺπŸ‡¨Ecuador jwilson3

    This works. It fires on every "page" loaded via HTMX:
    [...]
    These do not:
    [...]

    Behaviors are typically "reattached" to a new DOM context after an ajax operation runs, therefore, the reason the later examples do not work for you is because Drupal.behavior.testHTMX.attach is not called after an HTMX operation, only after a Drupal AJAX operation.

    Interoperability will be one problem to surmount if we have to maintain both systems as a hybrid solution possibly through several versions of Drupal (as mentioned in issue summary).

  • πŸ‡©πŸ‡°Denmark ressa Copenhagen

    HTMX looks incredible, a real game changer. ThePrimeagen just released Creator of HTMX Talks HTMX and also a HTMX & Go course (Preview).

    Theo's Truth About HTMX describes the concept well, and it would be fantastic with a simple demo of a HTMX-based Drupal.

  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10

    If I'm honest, I'm kind of wary/weary of replacing something that's worked for 17 years with 'the new shiny frontend thing'

    Can we get a use-case/list of pain points with the current setup to justify this change, which would be disruptive for many custom/contrib modules as well as core.

  • I agree with @larowlan. I don't know of any site that uses HTMX, as it's really new. Would this require a complete rewrite of large amounts of custom/contrib code?

    If HTMX sweeps the web and it becomes well-used across the web or as popular as jQuery, then go for it. Otherwise, it could become another sizeable barrier to getting started with Drupal or to upgrading to the next version.

  • πŸ‡«πŸ‡·France andypost

    Moreover jQ4 said they improved CSP and ajax requests https://blog.jquery.com/2024/02/06/jquery-4-0-0-beta/

  • πŸ‡ΊπŸ‡ΈUnited States FatherShawn New York

    However, I'm very intrigued by HTMX, which I only encountered because you opened this issue. I'm working my way through the documentation still but so far I'm very pleased with the way that it intentionally works with the hypermedia architecture of the web.

    I don't know enough to form an opinion yet, but I'm grateful for being asked to think about and explore a new idea. It may or may not be a replacement for jQuery powered AJAX but we also been discussing removing jQuery for some time.

  • πŸ‡¬πŸ‡§United Kingdom catch

    Can we get a use-case/list of pain points with the current setup

    It's pretty rare that I use the current AJAX system in depth on client projects, partly chance, partly because I'm happy if other people do that while I work on other stuff. The most recent time I did anything with it was in πŸ“Œ Use MessagesCommand in BigPipe Needs work .

    I will say from inheriting various sites/reviewing contrib over the years that I have seen some horrible custom js that is trying to re-implement things that should have been done in AJAX commands (like closing a dialog on success, for a recent example). And I don't personally find the AJAX system enjoyable or intuitive to work with, I think it can be hard for people who aren't already familiar with it to learn what should live where and how.

    For me a test of HTMX would be whether we're able to reduce some of the js in core modules that is interacting with the JavaScript js API - views_ui, field_ui, history, toolbar, editor, media_library, bigpipe, claro etc. (grep -rl core/drupal.ajax). Or if not, whether those examples become nicer.

    The things that ajax commands need to do is probably(?) not going to change that much, but if we're returning HTML from AJAX callbacks, or HTML + extras, they might just end up as new render elements instead of their own API, which would consolidate things a bit and be one less thing to learn.

  • πŸ‡«πŸ‡·France fgm Paris, France

    HTMX is about 4 years old, so not really new, but awareness seems to have exploded in the last 6 months.

    I happened to have considered creating a module for it, considering how it works. The basic idea would be to either implement new elements or alter existing elements when the module is active, with the matching properties as alternatives to the current AJAX ones. It would not prevent existing attributes from working, so could be created entirely in contrib initially, the main limitation being that a given context (dom subtree) should probably not combine both traditional Drupal Ajax and HTMX.

  • πŸ‡ΊπŸ‡ΈUnited States cilefen
  • πŸ‡¬πŸ‡§United Kingdom AaronMcHale Edinburgh, Scotland

    Can we get a use-case/list of pain points with the current setup

    I hit up against this issue back in 2018 and it's still a problem today, required a hacky workaround πŸ› Ajax is broken in forms built within an Ajax callback Active .

    Still a problem 6 years later. (Also was 2018 really 6 years ago?! Oh my...)

    I think that illustrates why we may benefit from replacing our custom AJAX solution with a solution well supported by the wider community.

  • πŸ‡©πŸ‡ͺGermany Anybody Porta Westfalica

    Thanks for all the feedback here. I think all makes sense. Introducing HTMX in Drupal is not a simple task and will require many resources.
    So starting in contrib, making concepts and keep an eye on the general adoption of HTMX is surely the right way to go. Especially we should have a look at DX improvements. I agree that the current AJAX system in Drupal works, but for me it always has been and still is kind of (wonderful) magic in many parts. I agree with @catch in #17 and saw many developers struggling with it. It's not really straight forward, but that doesn't mean it's not good!

    So I think we're at a point, where things work well, but it doesn't mean it couldn't get better. Especially for developers new to Drupal it might be a simplification and unification, similar to what happened with Symfony in Drupal 8 in contrast to the custom codebase in Drupal 7, that also worked great, but was a good choice to use a standard library to make the switch to Drupal easier.

    So there are many good reasons for and against HTMX and the decision has to be made wisely. Trying it out in contrib is a good next step and perhaps someone would like to research the general pro's and con's and feedback for HTMX?

  • πŸ‡ΊπŸ‡ΈUnited States cosmicdreams Minneapolis/St. Paul

    I agree with the contrib approach. HTMX needs to be able to demonstrate that in cases where it can impact pieces within core that it can be a "drop ship" replacement of those pieces.

    If I could have a magic wand and get everything I want, Core wouldn't use any 3rd party javascript libraries (and still be awesome). Some days it feels like that reality is achievable but others it seems foolish.

    That said, HTMX's value appears to be in interactivity. So when thinking about the intersection of core and HTMX the areas that have a lot of user interactivity, places like Views, Content Administration, and Recent log messages. Other parts that feel way too ambitious like Layout Manager and impacting ajax.js itself are things that may one day be on the table, but yea, whew sounds like a lot of dramatic change.

    And it should be said that HTMX isn't the "end of the line" / final form of the ideas that it advances. I'm concerned that if we would invest a ton of effort adopting this library in core that we would have to redo / undo some of that work in the near term.

    Going forward

    Where should we meet to collaborate on ideas?

  • πŸ‡«πŸ‡·France nod_ Lille

    Very much +1 to #17

  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10

    Interesting that in this PHP UK session he points out HTMX is something we've already had in Drupal https://youtu.be/Yb4mDfwOUXE

  • πŸ‡ΊπŸ‡ΈUnited States effulgentsia

    Yes, there's not much conceptually new in HTMX that we didn't have in Drupal 6's AHAH which then became Drupal 7's AJAX. But the HTMX community did a really nice job in creating a more refined API and documenting it well.

  • πŸ‡ΊπŸ‡ΈUnited States FatherShawn New York

    It seems to me that what is new in HTMX is expressing and documenting the actions in the HTML. By expanding the hypermedia controls using attributes and including all the HTTP verbs in those controls it completes HTTP/HTML to be a fully expressive hypermedia system.

    I've done some creative things in custom projects in the past with our AJAX system, but much of that was not evident by looking at the markup. I haven't built anything yet with HTMX, having just encountered it via this issue, but I'm really looking forward to taking it for a spin!

  • πŸ‡«πŸ‡·France fgm Paris, France

    For anyone curious about the Drupal parts in the talk above, the segment mentioning Drupal (and WP) starts at 21:00.

Production build https://api.contrib.social 0.61.6-2-g546bc20