Gradually replace Drupal's AJAX system with HTMX

Created on 27 November 2023, about 1 year ago

What is the problem to solve?

Drupal's AJAX system has served us well for over a decade. For over 15 years if you count its predecessors in CTools and AHAH. At the time of its creation, we had to create a custom system, because there weren't existing libraries with the functionality that we needed. It was fun to be at the bleeding edge of integrating AJAX into a server-rendered framework like Drupal.

But now HTMX exists. It's a well designed, well documented, well maintained library that's gaining a lot of traction, and I think it supports pretty much everything that Drupal needs from an AJAX system, including the ability for the response to trigger additional actions, which we can use to implement any of our current custom commands that don't have an already implemented corollary in HTMX. In the long run, I think we'd be well served by moving from our custom AJAX system to a library that an increasing number of developers are gaining familiarity with.

However, I don't think it would be easy to preserve Drupal's current AJAX API and change it to use HTMX under the hood. Instead, I think it would be better to convert usages of AJAX (for example, in Field UI, Layout Builder, Views UI, etc.) to use an HTMX-based API.

This would mean maintaining both Drupal's current AJAX API and a new HTMX API until contrib/custom modules have had sufficient time (possibly several major Drupal versions) to convert their ajax usages to htmx.

Who is this for?

Developers

Result: what will be the outcome?

(Describe the result: how will this improve things for the stated audience(s)?)

How can we know the desired result is achieved

(Usability test, a metric to track, feedback)

🌱 Plan
Status

Active

Version

11.0

Component
Ajax 

Last updated 25 minutes ago

Created by

🇺🇸United States effulgentsia

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @effulgentsia
  • 🇺🇸United States cosmicdreams Minneapolis/St. Paul

    2 birds, 1 stone? How about the Off Canvas Dialog?

  • 🇬🇧United Kingdom catch

    There's already some work done on 📌 Add a native dialog element to deprecate the jQuery UI dialog Needs work on replacing dialog, although not moving especially quickly.

    On HTMX this is new to me, it seems like given it allows adding arbitrary extra things we could probably replicate everything. An annoying thing to replicate will be ajax_page_state and library handling, but that's perhaps a decent test case along with a relatively simple otherwise AJAX callback like replace.

  • I hadn't heard of HTMX, but after looking at the README and some examples, I think it's really cool. It would probably make writing code for AJAX forms a lot easier.

  • 🇺🇸United States cosmicdreams Minneapolis/St. Paul

    The aspect of what makes HTMX interesting in OffCanvas dialog (and perhaps we should talk about the "toolbox" part of the layout builder experience) is that dynamic behaviors are all executed on the server. In Drupal, and other systems that were built in it's era, priority was given to server controlled logic. HTMX allows server driven system to have more dyanamic behaviors, as long as responses to dynamic requests are provided in HTML (instead of JSON).

    For Drupal to fulfill that requirement, we can provide a solution as simple as a Controller that returns a renderable (like every example of a custom block has ever showed) or something as advanced as BipPipe's placeholder handling.

    I'll take a shot at this idea this weekend and see what I can hack up.

  • Is the licensing compatible?

  • 🇬🇧United Kingdom catch

    It's BSD licensed which afaik is fine: https://github.com/bigskysoftware/htmx/blob/master/LICENSE

    Can't remember off the top of my head whether we've shipped BSD licensed js with core before.

  • I understand it's the holidays, but just chiming in that I'm very interested to see @cosmicdreams's demos of this.

    I've been experimenting with HTMX on a project, and it works great for giving the site a SPA-like feel. The problem I've run into is one typical of AJAX loaded content; That is, Javascript won't fire on subsequent page loads because the content was not a part of the DOM. HTMX's onLoad() function (https://htmx.org/docs/#3rd-party) is the remedy for this, and I can make it work if I write vanilla JS, but I do not know how to integrate it with Drupal behaviors.

    This works. It fires on every "page" loaded via HTMX:

    (function (Drupal, once, htmx, drupalSettings) {
    
      'use strict';
    
      htmx.onLoad(function(content) {
        var items = document.querySelectorAll('.js-listItem');
        console.log(items);
      });
    
    })(Drupal, once, htmx, drupalSettings);

    These do not:

    (function (Drupal, once, htmx, drupalSettings) {
    
      'use strict';
    
      Drupal.behaviors.testHTMX = {
        attach: (context, settings) => {
          var items = context.querySelector('.js-listItem');
          console.log(items);
        }
      };
    
    })(Drupal, once, htmx, drupalSettings);
    (function (Drupal, once, htmx, drupalSettings) {
    
      'use strict';
    
      Drupal.behaviors.testHTMX = {
        attach: (context, settings) => {
          htmx.onLoad(function(content) {
            var items = document.querySelectorAll('.js-listItem');
            console.log(items);
          });
        }
      };
    
    })(Drupal, once, htmx, drupalSettings);

    Let me know if there is any assistance I can provide in moving this forward. I'm no JS expert, but I am happy to assist where I can.

  • In the third example “content” is undefined. Did you men “context”?

    “Context” is sometimes the full DOM and sometimes part of the DOM, if there was an Ajax update. Usually you need to use it within the implementation.

  • You are correct. I did do it that way originally, but in my copy/pasting to get it into the post, I did not change it back to context.

  • 🇪🇨Ecuador jwilson3

    This works. It fires on every "page" loaded via HTMX:
    [...]
    These do not:
    [...]

    Behaviors are typically "reattached" to a new DOM context after an ajax operation runs, therefore, the reason the later examples do not work for you is because Drupal.behavior.testHTMX.attach is not called after an HTMX operation, only after a Drupal AJAX operation.

    Interoperability will be one problem to surmount if we have to maintain both systems as a hybrid solution possibly through several versions of Drupal (as mentioned in issue summary).

  • 🇩🇰Denmark ressa Copenhagen

    HTMX looks incredible, a real game changer. ThePrimeagen just released Creator of HTMX Talks HTMX and also a HTMX & Go course (Preview).

    Theo's Truth About HTMX describes the concept well, and it would be fantastic with a simple demo of a HTMX-based Drupal.

  • 🇦🇺Australia larowlan 🇦🇺🏝.au GMT+10

    If I'm honest, I'm kind of wary/weary of replacing something that's worked for 17 years with 'the new shiny frontend thing'

    Can we get a use-case/list of pain points with the current setup to justify this change, which would be disruptive for many custom/contrib modules as well as core.

  • I agree with @larowlan. I don't know of any site that uses HTMX, as it's really new. Would this require a complete rewrite of large amounts of custom/contrib code?

    If HTMX sweeps the web and it becomes well-used across the web or as popular as jQuery, then go for it. Otherwise, it could become another sizeable barrier to getting started with Drupal or to upgrading to the next version.

  • 🇫🇷France andypost

    Moreover jQ4 said they improved CSP and ajax requests https://blog.jquery.com/2024/02/06/jquery-4-0-0-beta/

  • 🇺🇸United States fathershawn New York

    However, I'm very intrigued by HTMX, which I only encountered because you opened this issue. I'm working my way through the documentation still but so far I'm very pleased with the way that it intentionally works with the hypermedia architecture of the web.

    I don't know enough to form an opinion yet, but I'm grateful for being asked to think about and explore a new idea. It may or may not be a replacement for jQuery powered AJAX but we also been discussing removing jQuery for some time.

  • 🇬🇧United Kingdom catch

    Can we get a use-case/list of pain points with the current setup

    It's pretty rare that I use the current AJAX system in depth on client projects, partly chance, partly because I'm happy if other people do that while I work on other stuff. The most recent time I did anything with it was in 📌 Use MessagesCommand in BigPipe to remove special casing of the messages placeholder Fixed .

    I will say from inheriting various sites/reviewing contrib over the years that I have seen some horrible custom js that is trying to re-implement things that should have been done in AJAX commands (like closing a dialog on success, for a recent example). And I don't personally find the AJAX system enjoyable or intuitive to work with, I think it can be hard for people who aren't already familiar with it to learn what should live where and how.

    For me a test of HTMX would be whether we're able to reduce some of the js in core modules that is interacting with the JavaScript js API - views_ui, field_ui, history, toolbar, editor, media_library, bigpipe, claro etc. (grep -rl core/drupal.ajax). Or if not, whether those examples become nicer.

    The things that ajax commands need to do is probably(?) not going to change that much, but if we're returning HTML from AJAX callbacks, or HTML + extras, they might just end up as new render elements instead of their own API, which would consolidate things a bit and be one less thing to learn.

  • 🇫🇷France fgm Paris, France

    HTMX is about 4 years old, so not really new, but awareness seems to have exploded in the last 6 months.

    I happened to have considered creating a module for it, considering how it works. The basic idea would be to either implement new elements or alter existing elements when the module is active, with the matching properties as alternatives to the current AJAX ones. It would not prevent existing attributes from working, so could be created entirely in contrib initially, the main limitation being that a given context (dom subtree) should probably not combine both traditional Drupal Ajax and HTMX.

  • 🇬🇧United Kingdom AaronMcHale Edinburgh, Scotland

    Can we get a use-case/list of pain points with the current setup

    I hit up against this issue back in 2018 and it's still a problem today, required a hacky workaround 🐛 Ajax is broken in forms built within an Ajax callback Active .

    Still a problem 6 years later. (Also was 2018 really 6 years ago?! Oh my...)

    I think that illustrates why we may benefit from replacing our custom AJAX solution with a solution well supported by the wider community.

  • 🇩🇪Germany Anybody Porta Westfalica

    Thanks for all the feedback here. I think all makes sense. Introducing HTMX in Drupal is not a simple task and will require many resources.
    So starting in contrib, making concepts and keep an eye on the general adoption of HTMX is surely the right way to go. Especially we should have a look at DX improvements. I agree that the current AJAX system in Drupal works, but for me it always has been and still is kind of (wonderful) magic in many parts. I agree with @catch in #17 and saw many developers struggling with it. It's not really straight forward, but that doesn't mean it's not good!

    So I think we're at a point, where things work well, but it doesn't mean it couldn't get better. Especially for developers new to Drupal it might be a simplification and unification, similar to what happened with Symfony in Drupal 8 in contrast to the custom codebase in Drupal 7, that also worked great, but was a good choice to use a standard library to make the switch to Drupal easier.

    So there are many good reasons for and against HTMX and the decision has to be made wisely. Trying it out in contrib is a good next step and perhaps someone would like to research the general pro's and con's and feedback for HTMX?

  • 🇺🇸United States cosmicdreams Minneapolis/St. Paul

    I agree with the contrib approach. HTMX needs to be able to demonstrate that in cases where it can impact pieces within core that it can be a "drop ship" replacement of those pieces.

    If I could have a magic wand and get everything I want, Core wouldn't use any 3rd party javascript libraries (and still be awesome). Some days it feels like that reality is achievable but others it seems foolish.

    That said, HTMX's value appears to be in interactivity. So when thinking about the intersection of core and HTMX the areas that have a lot of user interactivity, places like Views, Content Administration, and Recent log messages. Other parts that feel way too ambitious like Layout Manager and impacting ajax.js itself are things that may one day be on the table, but yea, whew sounds like a lot of dramatic change.

    And it should be said that HTMX isn't the "end of the line" / final form of the ideas that it advances. I'm concerned that if we would invest a ton of effort adopting this library in core that we would have to redo / undo some of that work in the near term.

    Going forward

    Where should we meet to collaborate on ideas?

  • 🇫🇷France nod_ Lille

    Very much +1 to #17

  • 🇦🇺Australia larowlan 🇦🇺🏝.au GMT+10

    Interesting that in this PHP UK session he points out HTMX is something we've already had in Drupal https://youtu.be/Yb4mDfwOUXE

  • 🇺🇸United States effulgentsia

    Yes, there's not much conceptually new in HTMX that we didn't have in Drupal 6's AHAH which then became Drupal 7's AJAX. But the HTMX community did a really nice job in creating a more refined API and documenting it well.

  • 🇺🇸United States fathershawn New York

    It seems to me that what is new in HTMX is expressing and documenting the actions in the HTML. By expanding the hypermedia controls using attributes and including all the HTTP verbs in those controls it completes HTTP/HTML to be a fully expressive hypermedia system.

    I've done some creative things in custom projects in the past with our AJAX system, but much of that was not evident by looking at the markup. I haven't built anything yet with HTMX, having just encountered it via this issue, but I'm really looking forward to taking it for a spin!

  • 🇫🇷France fgm Paris, France

    For anyone curious about the Drupal parts in the talk above, the segment mentioning Drupal (and WP) starts at 21:00.

  • 🇺🇸United States fathershawn New York

    I've stepped up to the HTMX module's request for a new maintainer and will be developing ideas for Drupal integration there. Collaborators encouraged 😀

  • 🇬🇧United Kingdom joachim

    One limitation of HTMX is that it's geared towards a simple swap: one element makes a request, and the server returns a single element which is replaced.

    Our AJAX API is more powerful than that: it allows any number of AJAX commands to be returned as a response.

    HTMX does have https://htmx.org/attributes/hx-swap-oob/, and an extension https://htmx.org/extensions/multi-swap/ but these don't seem as elegant as what we have.

  • 🇺🇸United States fathershawn New York

    Regular swaps in HTMX can replace or append with additional control on how and where that is executed.

    Out of band swaps add additional payload and they are not limited in number. Multiple out of band swaps could insert a variety of tags in the same request.

    When paired with javascript event listeners, those payloads can be processed. For example, using the out of band swap currently to add json in a script tag that describes additional CSS/JS assets needed by the inserted content, then loading those assets after the json is inserted.

    It is pushing me to think differently. It's like making stew with a new set of spices.

  • 🇺🇸United States rlnorthcutt Austin, TX

    I'm really excited to see this conversation, and I wanted to add a few things:

    Age of HTMX
    While HTMX itself was released in 2020, it is actually a rebuild of an earlier library called IntercoolerJs. So, the library itself is just about 4 years old, but the original version is 11 years old. This is not a "new shiny thing". It is actually an "old" shiny thing that has only recently been widely discovered.

    HTMX _is_ Ajax
    "The core of htmx is a set of attributes that allow you to issue AJAX requests directly from HTML" (from the docs). So, this is really just a more standardized and *easier* way to do Ajax. Think about how powerful it will be for developers to do much of their frontend Ajax work in Twig files. Pretty enticing.

    At the same time, the complexity of the current Ajax system (while powerful) means that it is not seeing the usage or improvements people expect in a modern framework. I can click a button to get Ajax filtering of a view (cool!), but I still get that same old little spinner in a potentially random place... and trying to improve or adjust that behavior is a non-starter for most backend devs. HTMX provides a cleaner and more standard way of dealing with the Ajax issue.

    Everything old is new again
    The reason that HTMX is getting so much attention these days is similar to the increased focus we have seen on static sites. Speed, performance, and the recognition that things have gotten a bit out of hand for for building content focused websites. In this case, we see that the push for React/Angular/Vue as a full stack framework has led to ever more complex systems to build, tree shake, compile, partially render, and hydrate HTML code. This means that many, many applications are way over-engineered. So, there is a recognition that simpler approaches are just as good and often better. Highly reactive apps need a large JS framework, but most apps only need basic interactivity.

    The core argument *for* HTMX goes like this - you make a call to a server to find data, run business logic, and then format it as JSON which is returned to the client. Then the client has to do more work to render that JSON at HTML and put it on the page. So, why not just return the HTML from the server directly? Its faster, its cache able, and it reduces the load (both time and JS assets) in the browser.

    HTMX is a perfect fit for Drupal
    Drupal is incredibly good at letting us generate HTML, and the templating system means we can generate it down to the field level. So, with very minor changes, we should be able to allow Drupal core to easily return HTML fragments of any entity template. Add in display modes and views, and suddenly you have the ability to create an entire HTMX backend with little to no code. Thats insanely powerful, and can give the project a shot in the arm. Instead of building a custom backend in python, laravel, or go... just use Drupal.

    Then, add in the incredibly powerful caching system, and now we are in completely new territory. Drupal's caching system is one of the "best kept secrets" of the project, and by adopting some minor enhancements (like HTML fragments), we can make it even more visible and accessible. I can see how an HTMX + Drupal approach could invigorate the project and bring it more attention and a whole new type of modern developer.

    HTMX is a perfect fit for Drupalistas
    The average Drupalista is either a backend dev (php) or a front end themer (twig, css, js). Most "modern" app builders using React/Vue/Svelte don't want to work with Drupal - they want to use their framework. We have seen how much resistance there is to using Drupal as a purely decoupled content API - most devs who want that will go for Contentful or Strapi. Drupal can act as a content API, but it compeltely ignores the rendering engine... which is a shame.

    HTMX is largely in line with this approach and is appealing to backend developers. It simplifies the front-end experience and gives us more
    "spa-like" functionality with much less effort.

    I think that this suggestion is about more than just making Ajax easier, adopting new tech, or even growing the community. I believe that this combination has the potential to reinvigorate the project by amplifying Drupal's core strengths and making the framework more relevant in the modern view.

  • 🇫🇷France pdureau Paris

    Thanks Ron for this detailed and exciting note.

  • 🇺🇸United States brianperry

    Want to highlight this from @rlnorthcutt's thoughtful post.

    > Drupal is incredibly good at letting us generate HTML, and the templating system means we can generate it down to the field level. So, with very minor changes, we should be able to allow Drupal core to easily return HTML fragments of any entity template. Add in display modes and views, and suddenly you have the ability to create an entire HTMX backend with little to no code. Thats insanely powerful, and can give the project a shot in the arm. Instead of building a custom backend in python, laravel, or go... just use Drupal.

    Had some similar comments in the HTMX issue queue 🌱 1.1 Goals for HTMX Active . A relatively small amount of functionality could make Drupal the ideal HTMX backend.

    Also worth noting, Astro recently released Page Partials as a feature targeted at this use case.

  • 🇩🇪Germany Anybody Porta Westfalica

    Thanks for the comments! Whao!

    Well, how to proceed?

    I think we all agree, this isn't a low hanging fruit, but might be a very tasty, juicy one high on the tree. So we should think about:

    • Raise Dries & Acquia's attention - I think he might be interested in the concept, and it might be very helpful to have his opinion and network
    • Discuss an initiative for this - it's far too large for single persons
    • Make a plan which Drupal version this might target. 12.x? 13.x?
    • Then we need a plan and proof of concept? ...
    • ?
  • 🇬🇧United Kingdom catch

    We don't target core major versions for things as such, if there was a working HTMX patch with a bc layer (however it might work), that could land in any Drupal core minor release. The only thing then is deciding that's specific to major versions, is deciding what version to deprecate the AJAX API for.

    The main people who need to be convinced here are Drupal core framework managers (tagging for that). I'm also going to move this to the core issue queue, the ideas queue somewhat works for new user-facing features but it doesn't really for developer-facing things since people just don't follow it enough.

    The hardest thing here (going by every other issue to do with the AJAX system recently) is going to be the backwards compatibility layer.

    For PHP, without having reviewed everything properly, it feels like we could make the current PHP AJAX API work with HTMX - i.e. #ajax on forms and AJAX commands don't specify the format that things are returned in, so the same inputs could be put into an HTMX response potentially, then we can change things with deprecations from there, but it wouldn't need every PHP implementation to be updated at the same time that way or maintaining two completely different ways to do things.

    On the other hand, adding bc layers for even one jQuery UI library that we're trying to replace (autocomplete, dialog) can take months of work and get stuck, and usage of the JavaScript API directly is less frequent than the PHP one, although e.g. Views uses it directly in core.

    So I wonder if we could have AJAX vs. HTMX modes which are mutually exclusive on a site. That would mean modules with their own JavaScript interacting with the AJAX API would need to re-implement the functionality so it works in HTMX mode and support both for a while, then sites could switch from one to the other over time, and we could eventually change the default for new sites. This would mean no 'bc layer' as such for the JavaScript AJAX API, it would just co-exist for a while and then be removed.

  • 🇺🇸United States fathershawn New York

    Then we need a plan and proof of concept? ...

    We are working on some proofs of concept in the HTMX contrib module Brian linked in #33. The first thing was to get assets from inserted content loading using the ajax page state approach we have now. The approach I took there may be a good pattern for some of the other components of the AJAX system in core:

    1. data is returned in JSON appended to the body
    2. An event listener processes the data and makes the change.
    3. The additional assets use case responds on HTMX load, but HTMX responses can return custom events so actions in our AJAX system beyond content replacement could be paired with an event and a listener.

    Since we often innovate and learn in contrib, collaborators warmly encouraged over there!

    That would mean modules with their own JavaScript interacting with the AJAX API would need to re-implement the functionality so it works in HTMX mode and support both for a while, then sites could switch from one to the other over time, and we could eventually change the default for new sites. This would mean no 'bc layer' as such for the JavaScript AJAX API, it would just co-exist for a while and then be removed.

    If we approached this in a similar manner to how we are dealing with the change from Annotations to Attributes for plugins, a module wouldn't have to support both but we would need to do so in core for a time. This would allow modules to change anything custom that they have done as needed without a bc layer.

  • 🇫🇷France nod_ Lille

    As one of the people tagged in #35 my response in #17 was pretty short, longer version: I'm all for it. I don't think the other want to keep the existing code either. Next step would be to open an issue with a PoC, (this issue is more for meta-talk, i'd rather have the discussion of implementation in it's own non-plan issue). Iterating in contrib is a good idea too, FYI I personally won't have time to be involved outside of core for the next few months.

    For the PoC what I have in mind:

    • One or two example of a replacement in core (at least)
    • A list of the various ajax commands we have in core and their feasibility (with a very rough complexity estimate easy/hard) with htmx
    • Test runs don't need to be green, although it' be nice if a couple of them were passing.
    • can we use this to handle ajax form submit? if yes, how complex does it look? would be great to drop our jquery formsubmit fork
    • A big thing here is to improve DX, so a couple of examples of how it could make it easier to introduce some ajax stuff in contrib/projects would be good.
    • rough dependency evaluation https://www.drupal.org/about/core/policies/core-dependency-policies/depe... (being a great meme poster is not enough :p)
    • BC strategy (support existing calls, or a new codepath altogether, etc.) I'm not too keen on two separate mods. I feel it'll make adoption much, much slower than spending time on the BC layer even if it takes a long time. (we can always put restrictions on what the BC layer can do) if we have 2 code paths, we'd need a contrib module to bridge the two and let contrib maintainers test with htmx easily without changing their code

    Feel free to open an issue already, you don't need to have all the above to get started and send code, focus on the fun parts for now :). It's just a list of the things we'll need at minimum on the way to a core commit, it'll probably be split up in several issues once it's mostly ready to make review/commit easier. I checked the code for the contrib module, some stuff could be simplified on the js side (you don't need to attach the eventlistener to the body in a behavior for example) all in all, looks promising.

    I'm expecting a lot of discussion in that issue so please have in mind it'll be a marathon, not a sprint. Ajax support in core is ancient and changing it might be challenging, but it's worth it and I'll be here to help along the way.

  • 🇺🇸United States fathershawn New York

    Thanks for the support and clear guidance @nod_ :)

    I opened 📌 [POC] Implementing some components of the Ajax system using HTMX Active

  • 🇺🇸United States fathershawn New York
  • 🇬🇧United Kingdom hugronaphor

    Glad to hear there is a consensus that existing approaches to handle ajax are outdated and not DX friendly.
    This is the exact reason why a while ago when I needed to build an SPA-like app, I needed something more mature and modern.

    I did look at that time at htmx but it felt to me that it's not the right solution to use especially that I have found out about Livewire. As a result I have ported it to Drupal as Wire.

    The main benefit is that it's architecture has been tested and iterated for years already by Laravel community and it can handle any task mentioned here including form submissions, reacting/emitting events, web-socket, ... in fact Symfony ecosystem got their own Livewire inspired Live Components

    To be clear, I'm not saying that I want my Wire in core instead of playing with htmx but I think we should look at the scope broader and consider something in between Wire and Symfony's Live components. Huge benefit on top of being able to create dynamic interfaces with ease, Laravel and Symfony developers would have a familiar tool in Drupal's core.

  • 🇫🇷France nod_ Lille

    Thanks! That's an interesting project. Haven't had time to dig into the code but happy to see alternative takes being explored.

    I'm personally for htmx because the philosophy is aligned with Drupal and what we try to do. And I like that a lot of things live in the markup instead of hidden away In a js object somewhere.

  • 🇦🇺Australia sime Melbourne

    My first encounter with htmx was watching theprimeagen review it through non-drupal lens. To me it felt like the DX is better than Drupal core and very much made me think about how comparitively hard Drupal ajax is.

  • 🇫🇷France fgm Paris, France

    Before doing a lot with that, there's an issue with HTMX which I would not know how to solve for now : HTMX basically makes the assumption of a stable <head> section across pages, except for the title: when elements are swapped by a hx-boost, possibly inheriting if from <body hx-boost="true">, all <head> content in responses is dropped except for the <title> element.

    However, Drupal pages have long been dependent on a strong relationship between <head> and <body> content, with different JS/CSS bundles from page to page, for example.

    Now, you can indeed push changes to <head> using multiple response top-level element with the hx-swap-oob attribute, but that needs an ID on each modified target, which needs to pre-exist on the source page, which seems like a lot of potential trouble.

    Even without using <body hx-boost="true">, partial replacement responses are often likely to require <head> changes.

    I think this is something which needs to be solved earlier than later for a PoC, to outline the patterns that can be used to work around that discrepancy between the HTML model and the Drupal model.

  • 🇬🇧United Kingdom catch

    However, Drupal pages have long been dependent on a strong relationship between and content, with different JS/CSS bundles from page to page, for example.

    So this is true, but for AJAX requests we already deliver assets separately outside the head markup and it's also the case for big pipe placeholders (which would be another thing to try a conversion of fairly early on since that relies entirely on the AJAX system with a small amount of custom js)..

    I think the existing concepts of ajax_page_state ought to work - i.e. we track the libraries that have already been sent, and when new libraries are sent, they go through the asset rendering system and are sent as a new aggregate - both script and stylesheet link tags are allowed in the body of the document so they can be sent with the HTML (or injected via js wherever). Drupal settings and HTML head links (favicon, feeds etc.) are a bit different though so definitely need to figure out those.

  • 🇫🇷France fgm Paris, France

    big pipe placeholders (which would be another thing to try a conversion of fairly early on

    Indeed, that reminds me of another potential difficulty, which is the use of the hx-trigger="load" or "revealed" or "intersect" relying on the default browsers events or the intersection API, which is likely to interfere with our use of behaviours to support multiple loads per context like document or other node e.g. with BigPipe.

  • 🇺🇸United States fathershawn New York

    Some more thoughts on the questions raised in #43 🌱 Gradually replace Drupal's AJAX system with HTMX Active , #44 🌱 Gradually replace Drupal's AJAX system with HTMX Active , and #45 🌱 Gradually replace Drupal's AJAX system with HTMX Active above:

    However, Drupal pages have long been dependent on a strong relationship between <head> and <body> content, with different JS/CSS bundles from page to page, for example.

    I think the existing concepts of ajax_page_state ought to work

    I think I do have this working with ajax_page_state in my contrib implemenation. HTMX includes a header, HX-Request with every request it makes. I have a response subscriber that only processes the response if this header was included on the request. On HTMX requests, the response is processed using logic adapted from our current AjaxResponseAttachmentsProcessor. The resulting asset data is attached to the response as JSON.

    Now, you can indeed push changes to <head> using multiple response top-level element with the hx-swap-oob attribute, but that needs an ID on each modified target, which needs to pre-exist on the source page, which seems like a lot of potential trouble.

    Perhaps HTMX has evolved it's capabilities as all my exposure is recent. The <script> tag containing the JSON asset data is wrapped with <div hx-swap-oob="beforeend:body"></div> as part of the response processing. The value on hx-swap-oob defines the swap strategy and target that is used for the inner HTML of the <div>. The result is a script tag appended to the end of the body. HTMX fires an event, htmx:oobAfterSwap, after it has processed and inserted our asset JSON. Currently I have a listener for this event that uses the loadjs library we already use to process and attach the assets.

    which is likely to interfere with our use of behaviours

    HTMX fires another event, htmx:load after each response has finished processing. I'm currently using an event listener on this event to call our Drupal.attachBehaviors(). All of this is verified in a test.

    I have less experience with the JS side of our core code and @nod_ said the JS here could be improved 🌱 Gradually replace Drupal's AJAX system with HTMX Active so please offer improvements either in 📌 [POC] Implementing some components of the Ajax system using HTMX Active or in the HTMX module issue queue .

    A related issue will be history support. If we enable back/forward navigation with the hx-push-url we probably need to make it so that the cacheability headers, among others, be updated for updated pages, because the result will often not match the source. Think, for example, of a search page filled with results: the results will at least carry cache tags for the results but the original page will not.

    If we did enable history support, what I think is important is that the url that is updated in the browser should be able to recreate the current display. Using the example of a search page, we commonly add query parameters to such urls containing the search terms. I don't believe that we are currently updating anything in the header on ajax responses, other than header JS included in the differential assets payload. If we find that we do want to add fully harmonizing the head section, we could explore adapting the HTMX head-support extension for our specific needs.

  • 🇦🇺Australia sime Melbourne
  • 🇺🇸United States kevinquillen

    What is the easiest or smallest POC to proof out here?

  • 🇺🇸United States fathershawn New York

    @kevinquillen At the request of @nod_ we are discussing that in 📌 [POC] Implementing some components of the Ajax system using HTMX Active

  • 🇺🇸United States fathershawn New York

    I've had an inquiry about the appropriate markup to use and have moved that to a child issue: 🌱 [policy, no patch] Choose a markup strategy for HTMX POC Fixed Please add your guidance there!

  • 🇫🇷France nod_ Lille

    done

  • 🇺🇸United States fathershawn New York
  • 🇨🇦Canada ambient.impact Toronto

    I just wanted to jump in and say that comment #31 ( #3404409-31: [Plan] Gradually replace Drupal's AJAX system with HTMX ) is a phenomenal summary and argument for why Drupal should fully embrace the HTMX approach and not compromise on that philosophy. 👏

    I die a little on the inside every time someone throws their hands up and wants to reimplement Drupal's front- and/or back-end using a purely front-end JavaScript framework, demoting Drupal to just a fancy JSON endpoint. We don't need to throw out the Cache API, Twig templates, all the security and reliability of rendering on the server, to fix the admitted UX problems associated with vanilla server rendered navigation.

    If it's page transitions and partial page updates you're after, use HTMX, Hotwire Turbo, or Unpoly. (See the RefreshLess module .) If it's that the server feels too slow, proper use of the Cache API can help a lot, but you can also implement a Service Worker to intercept and respond instantly while it sends requests to the server in the background. This is a solved engineering problem that works completely with server-rendered HTML.

  • 🇮🇹Italy kopeboy Milan

    Is using a Service worker with Drupal something a curious Site builder can do though?

  • 🇫🇷France dqd London | N.Y.C | Paris | Hamburg | Berlin

    #54: Regarding your second paragraph: I often phrased my worries (carefully and sensitively) on the "JS frameworks on top of Drupal" trending concept. I've never been convinced of it as a solution for every inactivity priorising project from day one when it started to take off so much. Too much. Especially not for random use cases. But it should not be demonized. It is a corner case concept. After my long-term experience with such concepts already long long time before (over 16 years), which actually brought me to Drupal 5/6 back in the days (as I filled an award-winning agency Flash site with content from Drupal) I was wondering how this boomerang came back and became such a new buzz with JS frameworks reinventing the wheel. It was an exiting Flash (actionscript) meets PHP project bringing me to Drupal, sure. And it was feeling like walking thru' a starship (game). But it showed actually also the reasons for my worries and limitations alerady and that this should be only done when Drupal is not the core of the project (the content administration part of the project only). It had already the same limitations like JS frameworks have on top of Drupal today. As the word "decoupling" perfectly describes. It decouples some of Drupals strength to be unreachable at frontend. JS frameworks can surely interact better with Drupal than Actionscript does back in the days. And again, there are very good reasons to use them for projects where the strengths of these JS frameworks have priority over all the features Drupal bring with core. But the base issue on the concept persist. But in the days of the starting JS framework hype there were only a few ears hearing me.

    Great to see an attempt to embrace HTMLX exactly for that gap/bridge in between, where Drupal is the core of the concept which just needs some more interactivity without loosing Drupals strengths. We hope to make some time available to contribute to this eco system soon. We are all in! +1

  • 🇺🇸United States fathershawn New York

    Your enthusiasm and engagement are a welcome gift @dqd! Our POC 📌 [POC] Implementing some components of the Ajax system using HTMX Active needs review. Your feedback there would really super.

  • Hi all, I'm Carson, the creator of htmx. Just wanted to say that I'm happy to chat w/anyone working on this. You can ping me at carson at bigsky dot software or jump on the htmx discord https://htmx.org/discord

  • 🇨🇦Canada ambient.impact Toronto

    @kopeboy: Yup, the PWA module provides a service worker sub-module. There are still a bunch of issues and things that can be improved, but it should work for basic offline stuff.

  • Hi there, my name is Carson Gross and I'm the creator of htmx. Just wanted to offer to help out/talk w/whoever is looking at htmx. You can email me (email off of https://bigsky.software) or jump on the discord, where i'm 1cg (https://htmx.org/discord).

  • 🇬🇧United Kingdom oily Greater London

    Is there not a danger that somewhere down the road of implementing HTMX it gets integrated into AJAX library or a subset of it? In the same way that AI searches are becoming integrated into search engines and becoming easier by the day to integrate into other software libraries/ components.

  • 🇩🇰Denmark ressa Copenhagen

    Welcome to Drupal Carson! I hope your involvement can help make this happen now.

    Since the interview with Primeagen I linked to in #12 🌱 Gradually replace Drupal's AJAX system with HTMX Active there was also So I Talked With Creator HTMX, in June 2024.

  • 🇬🇧United Kingdom oily Greater London

    Hi @1cg Just noticed #60. I assume you are liaising with other CMS open source projects? Any projects you can list that are taking HTMX on board?

  • 🇺🇸United States nicxvan

    @1cg welcome! I shared your offer on the Drupal Slack, I've been following this initiative for a bit I may see if I can gather some questions for you.

    @oily, I'm not sure I follow your concern in 61.

  • 🇬🇧United Kingdom oily Greater London

    #64 HTMX is heralded as the next AJAX? If so then in the Darwinian world of software development there are two possible outcomes.

    • HTMX ascends and AJAX descends
    • Before long HTMX replaces AJAX

    The other possible outcome is:

    • AJAX integrates HTMX, it cherry picks the best bits.

    There are more nuanced outcomes eg in the case of Javascript libraries many have been 'all the rage' for a year or two then disappeared into obscurity. But in this case it seems like HTMX has only one competitor: AJAX.

    The analogy with investment is useful because a project like Drupal.org has an infinite amount of time its largely volunteer workforce can devote to Drupal core development. That time is a scarce resource. Maybe we should wait for HTMX to achieve what I assume it intends to: to replace AJAX as the defacto standard for what AJAX does. In the long-run that will save many hours devoted to swapping out AJAX with HTMX.

  • 🇺🇸United States nicxvan

    So i do not see why the AJAX library cannot gradually cherry-pick from the HTMX library and develop in parallel with it. I would like to know why that is an impossible scenario

    I mean that is what this issue is, to begin to gradually replace with the best parts of htmx.

  • 🇬🇧United Kingdom oily Greater London

    #66 And the alternative is to keep using AJAX which will itself gradually absorb from HTMX. We just leave AJAX in place and leave AJAX to make all the cherry-picking decisions and do the heavy-lifting for us. Then any HTMX features will have passed through AJAX projects gates. If we gradually transition to HTMX then that would (in a sense) violate DRY or DRWTAD! : ) (Dont repeat what they are doing.)

  • 🇺🇸United States nicxvan

    In your analogy who is the project maintainer of ajax that would "absorb" features of htmx.

    Can you provide a link?

    Even if that were to exist evaluating new libraries is still valuable because they may meet the needs of a project better and iterate faster.

    There's a reason why pieces of home grown drupal bits have been getting off of the island.

    Sometimes there is a better way of doing things and adopting a library can help standardize and developers not familiar with drupal can pick it up quicker.

  • 🇬🇧United Kingdom oily Greater London

    #68 In my analogy there is no one maintainer. I assume it is an open source library like Drupal? So it is various committees who make the decisions. No one person?

    Even if that were to exist evaluating new libraries is still valuable because they may meet the needs of a project better and iterate faster.

    Absolutely. Your comments are valid. I am rejecting nothing. I am raising objections: questions that should be answered. If the vehicle analogy is not applicable then I would like to know why not.

    Sometimes there is a better way of doing things and adopting a library can help standardize and developers not familiar with drupal can pick it up quicker.

    Sometimes, indeed. I think we should have a clear roadmap of how the adoption of such a library or more like the removal of the AJAX library will benefit the project in the ways you describe. Can you give me any concrete examples?

  • 🇬🇧United Kingdom oily Greater London

    Another idea might be to keep as we generally do, a close eye on the Symfony project. How are they implementing HTMX? And with what velocity? Slow-burn, or swallowing it hook line and sinker.

  • 🇬🇧United Kingdom catch

    @oily

    I assume it is an open source library like Drupal? So it is various committees who make the decisions. No one person? ...should we wait for the JQuery project to implement HTMX?

    Not really. Drupal added jquery.form.js in #157752: AHAH FormsAPI Part 2, Extend to all elements. Kill redirectFormButton(). in 2007. At that point AJAX was called AHAH. It was not part of the jQuery or jQuery UI projects.

    There is no indication that jQuery is implementing any new features, they are focused on deprecating and removing features that are now available in JavaScript. So jQuery has never had a full featured AJAX library (in the Drupal sense) supported by the jQuery team and it's not going to add one. It's also not a 'standard set of functionality' in the sense that Drupal's implementation is unique.

    jquery.form.js was never fully ported to either jQuery 3 or jQuery 4, and the library hasn't had a commit for five years. https://github.com/jquery-form/form

    In 📌 [jQuery 4] jquery-form is unmaintained and not jQuery 4 compatible, fork it into core Fixed we had to fully fork the library into Drupal core for jQuery 4 compatibility. In a sign of how unmaintained it is and how few people are interested in maintaining core's AJAX JavaScript, some of that work was done by me, and I maybe touch JavaScript 2-3 times per year, because I was worried if it didn't get done, it would delay the entire Drupal 11 release or not allow us to update to jQuery 4. There is a five year old issue to try to modernize core's AJAX library to properly remove the jQuery Form dependency, but it is stuck: 📌 Remove jQuery Form dependency from misc/ajax.js Needs work .

    The PHP AJAX API is maintained as part of core, it is not exactly fast paced development, but people do sometimes work on it. However as far as this issue has got, we would be keeping all of the concepts of the Drupal's PHP AJAX API and just adapting them to work with HTMX instead. There would be some API changes but not huge conceptual changes to how things work now. Hopefully we can find ways to simplify/improve things but it wouldn't be a ground-up rewrite - precisely because the design of HTMX is not that far off how Drupal's AJAX system already works.

  • 🇺🇸United States fathershawn New York

    Hi @1cg! What a joy to see you arrive in this issue! I'm the most active maintainer for the contributed module and proof of concept 📌 [POC] Implementing some components of the Ajax system using HTMX Active that we are working on. I'll definitely be in touch today :)

    I can think of one thing right off, and that is for the HTMX project to add a security policy. If this proposal continues to gain traction, we would be adding a dependency and our core maintainers will be using our existing dependency evaluation process . Drupal recently worked with the Revolt event library 🌱 Adopt the Revolt event loop for async task orchestration Active on a similar policy. If it's helpful I'm happy to open a similar issue in your queue.

  • 🇬🇧United Kingdom oily Greater London

    #71 Read with interest. Thank you.

  • 🇫🇷France dqd London | N.Y.C | Paris | Hamburg | Berlin

    Another belated "Welcome to @1cg!" (creator of HTMX) from me and a Million thanks in advance for your invitation to help with being there for questions and for any support required in case of implementation. +1 (This should not be allowed to fall by the wayside during heated debates here)

    If you are interested in some thoughts, links and discussion happening in the moment at Slack Drupal #contribute, started by a question I raised over there (this is how I belately came here some days ago from @mstrelan who has kindly linked me here) feel free join us. Link: https://drupal.slack.com/archives/C1BMUQ9U6/p1737655260942599

  • 🇬🇧United Kingdom oily Greater London

    @dqd I prefer 'light sparring' no heat at all : )

  • 🇫🇷France dqd London | N.Y.C | Paris | Hamburg | Berlin

    Sorry, my fault (not my native tongue, I will del in my comment) ... :) And it was not meant in any critical way ;) Discussion is good. And important. Thanks for coming back on this.

  • Heya all, no worries I don't mind rough and tumble, I grew up on the forums in the 90s and 2000s and I've got a thick skin.

    Happy to help however I can as a non-drupal expert. Ultimately I think the idea of generalized hypermedia controls (https://dl.acm.org/doi/10.1145/3648188.3675127) is more important than htmx as a particular implementation, and there are a lot of different ways to achieve that including rolling your own. I'd be happy to assist with that if you'd like.

    htmx should be pretty stable going forward ( https://htmx.org/essays/future/)and thus shouldn't be a huge risk for a project adopting it. I do a walk through the library here: https://www.youtube.com/watch?v=javGxN-h9VQ

    There are some quirks to be aware of: https://htmx.org/quirks/

    I'm not currently talking w/any other CMS projects, although CraftCMS and ButterCMS are sponsors of htmx.

    @dqd I tried clicking on the slack link, but it said I need to have a drupal email. Probably a skill issue on my part :)

  • 🇬🇧United Kingdom oily Greater London

    Hi @1dcg, RE: #77 second paragraph. Can you please enlarge on what looks like the most immediate way you envision that Drupal can benefit from GHC? Im patenting that new? abbreviation GHC by the way :). You can use it but only with my permission : ).

  • 🇺🇸United States nicxvan

    No worries! You can join the slack here https://www.drupal.org/community/contributor-guide/reference-information...

    #contribute
    #core-development

    Are two good channels to join, feel free to ping me there, same username.

    Actually most of the people in this thread have the same usernames there!

  • @oily I think the best way to see what generalized hypermedia controls can do is to look through the htmx examples:

    https://htmx.org/examples

    The examples are all mocked out and you can see the request/response in the little request tracker thingie at the bottom to give you an idea of what the hypermedia exchanges look like.

  • 🇬🇧United Kingdom oily Greater London

    Thanks, I will check that. Just seen a youtube where you explain the concepts.

  • 🇩🇪Germany c-logemann Frankfurt/M, Germany

    Just after discovering HTMX a view months ago I saw a good descriptive German video on Youtube including a long critical part about software concepts and why HTMX is not good as a base of a complete frontend
    application. That's maybe true but that's not my usually need.

    I call myself a Drupal-Symfony-PHP Developer with a good toolbox on server level. So I first try to solve problems with Drupal and use other Server stuff if faster etc. But everytime I need some dynamic HTML parts in Drupal I always struggle with AJAX and JS at all and need the help of others. But I only needed under one hour to start with HTMX including using the module from zero to a functional solution for an internal project.
    From my perspective I cannot really help as much in this issue as I like because I'm not good at Javascript. I think I'm one of the ideal target Drupal developers of this change because using HTMX feels like a true Drupal concept for me. And which library is managing this behind the API doesn't matter for me as long as it stays lightweight. So maybe some other JS Frameworks came along and maybe also adopt this syntax. But the concept of HTMX stays and is named well in my opinion.

    So I think the critic about the name "HTMX" I cannot share. Extending HTML is the main idea as I understood in seconds. That concept of @1cg is what I'm very thankful of. So maybe it's helpful here to make a difference in the discussion about the concept and the library with the same name. So we can decide to implement the concept and than try to get the best technical implementation in core. I believe this is also the HTMX library but nothing I cannot really say with my lack of experience in JS frameworks etc.

  • 🇫🇷France dqd London | N.Y.C | Paris | Hamburg | Berlin

    Just wanted to inject shortly: I think implementing HTMX support into Drupal will not turn HTMX into

    a base of a complete frontend application

    here on Drupal's concept.

  • 🇺🇸United States fathershawn New York

    It was so encouraging to have all the discussion on this idea this week. The most helpful thing anyone can do to move forward is to review the approach in 📌 [POC] Implementing some components of the Ajax system using HTMX Active and suggest improvements or give a +1. It does not seem prudent to build out an implementation until we have consensus.

  • 🇫🇷France dqd London | N.Y.C | Paris | Hamburg | Berlin

    @faterhshawn: you asked me to review 📌 [POC] Implementing some components of the Ajax system using HTMX Active and I will come back to it soon when I am finally back on my desk.

    To prevent clutter in the Slack threads for those who want to chat about it (which makes sense sometimes before posting refined thoughts into the issue queues), I just created a Drupal Slack channel #htmx for everybody open to join. I try to collect all open threads of Slack in it, so:

    For @all following and contributing/discussing HTMX into core topics: feel free to join the Drupal Slack contrib channel #htmx

Production build 0.71.5 2024