Add support for ckeditor5-autoformat

Thanks for the work, IMHO this is must-have feature for CKEditor. However I'm setting status for "Needs work" because there's no way to disable the feature. One could argue there's no need to disable it, since Jira/Confluence doesn't allow one to disable auto-formatting either, but Drupal 10.0.0 shipped without it so we could choose one of the following:
1. Write change record and mention @jungle's sandbox module.
2. Create \Drupal\ckeditor5\Plugin\CKEditor5Plugin\AutoFormat plugin and include it in this issue.
3. Create follow-up issue and add AutoFormat plugin in that.

I think we can add a setting in the toolbar itself to toggle autoformat on and off.

This should affect the editor, not the selections of text like bold, italic, etc. Existing text won't be affected, when toggled off, from this point forward autoformat will be disabled, and vice versa.

Probably something like this

and the button is configurable like other buttons, site admin can choose to hide the button if they wish.

Via admittely-informal info gathering I'm now pretty convinced the "backspace to undo magic" pattern exists for most users even if they aren't fully aware of it. I even realized I've been doing this automatically without consciously acknowledging it. While I think a disable button would be ideal, I personally would allow a version without the button to get through provided there was a followup to continue debating whether or not to add it (and gauge the difficulty in adding it)

I would say a toolbar button to toggle it on/off is unnecessary, but fine, as long as the plugin can be active with no button.

There are no scenarios I've hit where I want this turned off. If anything, I wouldn't mind something to toggle it taking effect on paste, and not just on keyboard entries.

But if somebody wants to go to the work of adding a button, that seems like a reasonable compromise...

With regards to a toolbar toggle, I see potential UX/UI issues. After applying this patch and learning how this plugin works and subsequently looking at the source code, all we're talking about is a markdown user-interface that generates HTML source code. I wasted a lot of time to learn this today.

What if I want to paste my existing content already formatted in markdown, and keep it in markdown?

I'd like to see real markdown code support, as opposed to HTML source code, if for no other reason because I like to copy/paste/publish the README.md files I write for GitLab/Github modules on my own Drupal website. This functionality appears to soon be deprecated at the moment with no replacement options, potentially along with a confusing toggle button.

Sorry to be so glum, but this is my current assessment.

@Amstercad that's what the Markdown module is for. Unfortunately that doesn't seem close to having a D10 release... but I think that's the appropriate module for what you're saying you want. We still use that on a couple sites, for similar reasons...

Personally I would suggest that instead of an editor button to disable the autoformat behavior, we instead look at this as a configuration option in the text format. I think the simpler the better for the editor itself.

Good point @mandclu, I agree having an option in the text formats would be best!

I discussed this with @lauriii. He suggested doing some actual usability testing to gather data:

1. Present a user with a CKEditor 5 instance with ckeditor5-autoformat.
2. Ask them to recreate the following in CKEditor 5 with the explicit instruction to make it match exactly: every character, every formatting detail.
   

   This is a very interesting paragraph! Oh so interesting.

   But then it turns weird…

   Bob:

   _huh?_

   Alice:

   hah!

   Bob:

   # What is this?!

   Alice:

   * Wonderland!

UX-wise that would look like this with ckeditor5-autoformat:

👆This will test in multiple ways whether users can figure out the "backspace to undo" behavior themselves.

I'll test this with more and less technical people and report back!

That seems like a good approach for validating the hypotheses from #57! 👌 It might be interesting to add some emphasis or bold towards the end of the sample text to see what happens.

needs rebase with the target branch. I'm trying but keeping as needs work because isn't ready yet

Now testing this with:

This is a very interesting paragraph! Oh so interesting.

But then it turns weird…

Bob:

_huh?_

Alice:

hah!

Bob:

# What is this?!

Alice:

* Wonderland!

The end. Thanks!

per @lauriii's feedback in #64.

First usability test result: my girlfriend! 🤓 She's:

• not a programmer
• used to macOS conventions, and knows a few keyboard shortcuts
• comfortable with typing (but not a blind typist) and English
• a daily user of Microsoft Word, Google Docs, Outlook 365 and Gmail

Instructions I provided:

Observations:

1. 🔎 For the first 2 sentences she used CMD+I and CMD+B, she didn't click the toolbar buttons.
2. ✅ When she hit the _huh_? test case, she literally said "HUH" (🤣) and instinctively pressed backspace and moved on.
3. ✅ When she hit the # What is this?! test case, she didn't have that same instinct, she figured she'd fix it after finishing the typing of the words. She then realized she didn't know how to undo it, so she just selected the line, deleted it and started over. Upon entering # followed by , she noticed the # had disappeared and that the cursor had once again grown in size (which she hadn't noticed immediately the first time because she cannot type blindly). So: the same perceived weirdness occurred. She wanted to undo that, so she pressed backspace. The # became visible again and the cursor shrank to normal size. Another "HUH" followed by a 🤷‍♀️ later, she was able to continue typing the words and finished this part too.
4. ✅ When she hit the * Wonderland! test case, she had a similar fight. But this time she made the connection to the "list" toolbar item that had become enabled. She tried using that to get rid of the automatic formatting and revert back to the literal characters she had entered. That got her just the word without the * in front of it. So she typed * in front of it … only to notice this re-triggered the autoformatting. Then she pressed backspace again to undo that attempt, which actually yielded the result she was aiming for.
5. 🏁 The rest she then finished with ease.

Conclusion: test participant 1 with lots of computer interaction but not an "expert" user was able to figure this out without any issue. Her observation was . Only then did I explain her the concept of Markdown, and how the ability to type like that to generate formatted text is valuable. It made sense to her, although she would probably stick to CMD+B and friends. She is not familiar with this kind of symbol-based formatting and doesn't feel the need to start using this:

I will repeat the same test this weekend with a friend who's very used to Windows conventions. I'll provide a similar overview then.

#69 is very useful

Btw, thanks for rebasing, however GitLab still returning

Merge blocked: the source branch must be rebased onto the target branch

so should we keep as Needs work to track that?

No, that's just because since the last rebase, another 5 new commits have landed. GitLab is not very strong at providing helpful status messages 😅

Tests are passing. This is ready for review, so unassigning. As mentioned in #69, I will add at least one more usability test result this weekend. 👍

Yeah, now it appears as "mergeable", thanks

I'll show this on my CKEditor session at DrupalCon Pittsburgh 2023 (here's the invitation for anyone interested), and since the presentation will be intended for content editors I'm testing that as a "non-technical user" and on my tests it also looks good

On my first time using I wasn't aware of the concern raised at #54 about a way to disable that, but when I tried to rollback the transformation it was intuitive because of the "backspace-undo" approach, as cited at #57

Based on my experience, combined with Participant 1 at #69 I'm moving to RTBC ok? If anyone has more concerns and/or more participants with different opinions on UX, feel free to move back the issue status

If you paste in the #68 example does it preserve formatting or try to autoformat it? If it preserves, that's another way end users could workaround if they get completely stuck.

@catch, it does preserve the formatting, but it is a known issue in CKEditor5, not sure if it stay this way or not in the future.

from https://ckeditor.com/docs/ckeditor5/latest/features/autoformat.html#known-issues:

Pasting Markdown-formatted content does not automatically convert the pasted syntax markers into properly formatted content. GitHub issues: #2321, #2322

Thanks that is good to know, although I'm not sure what it means for here.

If you paste in the #68 example does it preserve formatting or try to autoformat it? If it preserves, that's another way end users could workaround if they get completely stuck.

Just tested this out. I can't upload a video because I'm on a flight (to Midcamp 🙌). I can confirm that when pasting the example from #68, CKEditor does not auto-format the pasted text.

The MR needs to be brought up to date. There's conflicts in both the package.json and yarn.lock. I'd do it but on a flight and time is running out.

Once this is brought up to date, it can likely be set back to RTBC

Rebased and updated @ckeditor/ckeditor5-autoformat to version ~37.1.0. 🍻

#73: Yes it is preserved:

Back to RTBC per #77.

@Wim Leers what do you think about #74?

I think that's not a big deal because autoformat is not a Markdown-to-HTML converter. It's a productivity tool.

I see this as a UX enhancement for power users to reduce the amount of finger/hand movement while creating content: pressing buttons typically requires to stop typing with one hand and instead use it to press on the mouse/trackpad/touch screen, or pressing CMD+B, CMD+I etc (although no shortcut exists for e.g. creating a <h2>), which at best requires repositioning hands on the keyboard and at worst requires looking at the keyboard.

(It not converting from Markdown to HTML upon pasting is also consistent with how Google Docs implements it — it'd be valuable in both places, but is IMHO no reason to stop shipping this.)

+1 on #82 great work and productivity improvement! Thank you all so much for your wonderful work every day!

Sorry what I mean is if they change the behaviour then the markdown-alike content in #3227354-68: Add support for ckeditor5-autoformat → will be treated as actual markdown and pasted incorrectly. This suggests it might be necessary to be able to disable autformatting at least on paste.

Ah, I see! You see the current behavior as a feature. Interesting point!

I was actually thinking that https://github.com/ckeditor/ckeditor5/issues/2322 should be configurable (through CKEditor 5's plugin-level configuration, and then Drupal can choose to always disable it or alternatively expose it as a setting).

But perhaps … it should just never happen? Or … better yet: upon pasting, it would tell the user Markdown syntax was detected, and ask if the user wants it to be converted.

(Clearly, these questions around the UX of pasting Markdown are why they have not yet implemented it 🤓)

Either way, https://github.com/ckeditor/ckeditor5/issues/2322 is very unlikely to happen any time soon. We can influence how it will be implemented (if it ever will), and will be able to adjust on our end to ensure it's the UX we believe is best for our users.

I've just copied relevant comments from this Drupal issue to that CKEditor 5 issue to ensure these concerns are taken into account. 👍

OK #85 makes sense to me and thanks for the comment on github.

The @ckeditor/ckeditor5-autoformat usage data looks really convincing. @ckeditor/ckeditor5-autoformat has 133 000 weekly downloads whereas @ckeditor/ckeditor5-essentials has 151 000 weekly downloads. I was questioning if the data I was seeing was accurate because the numbers were very high. However, I learned recently that it's because CKEditor 5 autoformat is part of the default build. Given this, I would be fine with us moving forward with this.

We could always add an option to disable this later. If someone needs to desperately get rid of this functionality on their site, they could do:

function hook_ckeditor5_plugin_info_alter(array &$plugin_definitions): void {
  unset($plugin_definitions['ckeditor5_autoformat']);
}

I read #69 not as "hey she figured out how to get out of it", but rather "see how she needed to poke around and do various trial and error to avoid getting this feature". In other words I see #69 as the editor being in the way of accomplishing what she wanted rather then being helpful. That said, the test scenario was intentionally covering the transformations and normal text would contain those elements less. Since me as a core product manager seem to be the only person remaining that is concerned that there is not an easy UI way to disable this, I think we may go ahead and see if we get user complaints.

I don't think #69 had the goal of providing additional evidence to back the importance of this feature. The goal of the interview was to try to address the concern that we had related to disabling this, which would be needed if this change stands in the way of users who are not familiar with this feature. I think the results from #69 are encouraging because at least that data point suggests that users have workarounds to to bypassing this feature in the case where it comes as a surprise. However, that's just one data point so we shouldn't make decisions solely based on that.

I think we are in agreement about the goal of #69. What I took away from #69 is that the workarounds are not evident for most of the cases, which in my view supported my concerns with this potentially causing frustration with a need to poke around to undo non-evident behaviour where the user may not realize its the same thing happening to them and trying to poke around to different places to resolve it.

Either way, once again I will not stop this from happening on those grounds since others don't seem to share that concern.

@lauriii in #87:

I was questioning if the data I was seeing was accurate because the numbers were very high. However, I learned recently that it's because CKEditor 5 autoformat is part of the default build. Given this, I would be fine with us moving forward with this.

To add to this, @Reinmar & @witeksocha told me in yesterday's CKEditor 5 meeting that:

1. the autoformat plugin is shipping by default in ALL of their predefined builds AND they are not getting complaints!
2. Comparing the autoformat package to others, we see by installs on npm, just 14% difference between it and a very fundamental plugin like basic-styles (bold, italic, etc.).
3. Despite the two points above, there are fairly few open issues for autoformat considering its adoption, and most of them are feature requests ("improvements" in their terminology).

Per this thread in #ckeditor5 Drupal Slack, the following core committers are +1:

• @catch
• @lauriii

I think/hope that the additional information/context I just provided also removes 90% of @Gábor Hojtsy's last hesitations, even if he's already indicated that he's okay with it.

Wim Leers → credited Reinmar → .

Wim Leers → credited witeksocha → .

There is some validity to the concern that some users might not understand the autoformatting and how to circumvent it.

However, I'm unsure if having an option to disable it in the UI will provide much benefit. I'm not sure if those who would like such a feature would have the intuition and exploration instincts to know to look for it much less find it. There's a certain percentage of users who don't have much of a capacity to "poke around", their difficulty with autoformatting would also mean difficulty thinking/figuring out how to disable it.

Yes #94 is my reasoning too, a button that says disable autoformatting is also likely to look disruptive, like it will remove all existing formatting to, so if someone is hesitant to experiment they also might be hesitant to click it. For me that means just enabling by default and hoping it doesn't trip people up (current patch) or having a site-wide/editor-wide setting for it (hopefully not necessary but we could maybe add down the line if it turns out to be).

#95++ — and all data we have indicates this will be unlikely to be necessary. Also: landing it now means that we'll have the entire beta + RC to hear back from people before 10.1.0 gets tagged at the end of June :)

I personally love using markdown so I understand how everybody in here is excited about this, but I'm a but concerned about it getting in the way of less experienced editors as @Gábor Hojtsy mentioned. I see devs or technical users as the target of this feature, while the main user of the WYSIWYG (content users - editor or author) will probably not be as tech-savvy. I recognize the easy learning curve of this, but how often will they use markdown (either in a Drupal site or any other tool they use)? Will they remember what they learned the next time? What #69 describes is exactly what I would expect from any non-technical user, but read as @Gábor Hojtsy said in #89.

But most important: how easy will it be for us to get the real users complains once this is in? And, do we have any data of real users asking for this?

I don't want to get in the way neither, but I'm a bit concerned that merging this enabled by default will impact negatively the usability. I'd personally suggest what @catch said in #94 or @mandclu in #64 (having a site-wide/config option/permissions/whatever way to enable/disable this). But instead of enabled by default I'd ship it disabled by default. Sorry all!

From the CKEditor team, I just wanted to let you know that this feature has been available for years, in all editors’ builds it has been set as the default. We’ve talked to hundreds of customers so far and haven't heard any issues regarding the usability.

It's also worth noting that auto-formatters for lists are available by default in Word or GDocs. Platforms like Notion or Confluence go further with full auto-format. These tools are quite widely used in various fields.

Although the syntax is similar to Markdown, IMHO, the argument that it targets technical users may be a bit exaggerated. I've seen authors creating content 10x faster than developers. At the end of the day, it's one of their main tools, and they happily advance their knowledge in it.

I think the autoformat feature would be a great addition to Drupal. It's a useful tool that can save time for users.

1. This is indeed targeted at delighting "power users", but that doesn't mean only technical users. There are plenty of professional content creators who spend hours per day crafting content in Drupal!
2. Nearly 100% of the time users will not run into this unless they're formatting incredibly obscure text.
3. ~133K out of ~205K or CKEditor 5 weekly installs have autoformat enabled, and this is not causing complaints.
4. More anecdotally, "backspace to undo magic" pattern exists for most users even if they aren't fully aware of it.

   — see #57 and confirmed with a non-technical person in #69.

5. Less anecdotally: Google Docs has the same exact "backspace to undo magic" pattern: see #52
6. In addition to "backspace to undo magic", there's also "Undo to undo magic": CTRL+Z/CMD+Z. That is supported by autoformat too. This is used in pretty much every rich text editor:
   • Microsoft Word: Change the capitalization or case of text
   • Google Docs
   • Apple Pages

   (In , start a new document, type test and observe how it auto-capitalizes to Test until you press CMD+Z!)

I disagree with shipping it as disabled by default. That means it de facto will almost never reach the users who would actually benefit.

If all major vendors ship both auto-correct and auto-capitalization enabled by default, why can't we ship with this, if it supports both of the widely used "undo magic" patterns (backspace and CTRL+Z/CMD+Z)?

EDIT: cross-posted this with @witeksocha.

While we've been focusing on the benefits of autoformat for power users, I believe that some of the autoformat features will also benefit regular users. For example, converting an asterisk to a list is a common pattern found in applications such as Wordpress Gutenberg, Microsoft Word, Apple Notes, and Google Docs. These applications enable this feature by default, so it makes sense for us to be consistent with that. However, some of these applications allow users to disable this functionality, while others do not.

There is a risk that this may cause friction with some users, as mentioned by product manager and usability maintainer feedback in #88 and #97. However, we haven't identified any common use cases where this would be occurring, and based on #69 it did not prevent them from getting the right text added. Based on #90 and #101, this feedback was not intended to block this issue, but to raise awareness on the potential risks of this issue.

Ideally this should not be an editor setting because it is essentially a user preference. One user might want this turned on, while another might not. Currently, we don't have a pre-existing mechanism for storing user preferences like this. However, if the risks raised here realize, we could provide an editor setting for this to allow people to turn off this feature. There's also a workaround in #87 how a contrib or custom module could disable the plugin.

Given all this, it seems fine for us to not provide a configuration option for disabling the autoformat plugin. We could introduce it as a new feature in the future if it turns out to be necessary.

Going forward and committing this. I believe committing this is a two-way door decision; there are multiple ways we could revert this decision depending on the outcome.

Committed bd5f9c0 and pushed to 10.1.x. Thanks!

lauriii → closed merge request !3083

Shouldn't there be an change record for this? I would think code snippet on https://www.drupal.org/project/drupal/issues/3227354#comment-15037129 ✨ Add support for ckeditor5-autoformat Fixed could be useful for sites where this feature should be disabled.

Automatically closed - issue fixed for 2 weeks with no activity.

Re-roll patch fro 9.5.x

Re-roll patch for Drupal 9.5.x to be working with https://www.drupal.org/project/drupal/issues/3362414 📌 Update CKEditor 5 to 38.0.1 Fixed

https://www.drupal.org/project/drupal/releases/10.1.0 → has been out with this functionality for 3 months tomorrow, zero complaints so far 🥳

#111 🎉!

I kind of agree with #107 that a big introduction of a feature like this could stand a change record of sorts, that tell folks how to disable it if they don't want it.

You're right! Wanna take a stab at that? 😊

Draft CR is here. Please review:

Change Record #3389141: CKEditor 5 now supports autoformatting with Markdown-like shortcodes →

On a separate note: @Wim Leers, in #1 you mention Autolink, which would be useful to everyone not just power users. I agree. Did a follow-up ever get created for that? My searches turn up nothing.

I asked for a review of the CR in Slack. @Wim Leers, responded with "This looks great! :slightly_smiling_face:"

Therefor, I have published the change record.

I know it's a little bit late but should we add to the CR that this only works while typing but not for copy paste yet? It is an issue still in progress on CKeditor's side: https://ckeditor.com/docs/ckeditor5/latest/features/pasting/paste-markdo... and I could confuse users testing this new feature with markdown text copy pasted from their Markdown notes and wonder why it won't work. Since other Markdown input like on Gitlab and Github and Google, Electron etc already works with copy/paste it is maybe worth mentioning it.

Possible example text which could be added on the bottom of the CR:

Below

Feature management

...

Limitations

Note: At the moment Markdown support works while typing only. The support for copy paste Markdown is still in progress. To follow progress read: https://ckeditor.com/docs/ckeditor5/latest/features/pasting/paste-markdo...