🇷🇸Serbia @vaish

Thanks for opening the MR, @ressa. I tweaked your text a bit. Please check it out.

You're welcome @vaish, and thank you for making an incredible module. It reduced the Bot strain a lot, and the server CPU usage went down from averaging ~90% to ~12%.

Thanks, @ressa. Glad to hear that.

Marking this issue as fixed.

You say "I wasn't sure if extra configuration was needed for Crawler Rate Limit..." but in the Step 3 it's already mentioned that:

Crawler Rate Limit requires only minimum Redis configuration (host and port). It's not necessary to configure cache, locking or any other Redis backend.

Maybe we could extend Step 3 so it reads:

Configure Redis module. Crawler Rate Limit requires only minimum Redis configuration (host and port). It's not necessary to configure cache, locking or any other Redis backend. If you already use Redis module, default Redis setup will work well with Crawler Rate Limit out of the box. For additional details consult Redis module's documentation.

Let me know what you think?

@ressa, thank for providing your feedback.

Thanks for pointing this out. Version 2 has been marked unsupported for a very long time. Instead, I created stable 3.1 release. I believe that should fix this issue.

Hey @marc.groth, I believe I know where is the confusion coming from. I usually add "open in new window" to external links so in my tests I always used external links and was focused on attributes added by Editor Advanced Link. If you check again, you will see that those attributes are indeed preserved after applying the patch from MR37. However, attributes that you are talking about are data-* attributes generated by Linkit module. Just now I verified that those get lost even with this patch applied.

My setup:
Drupal core 10.5.1
Editor Advanced link 2.3.1 with patch from MR 37 dated 9 Jul 2025
Linkit 7.0.5

In the branch 2.3.x this issue seems to be fixed by 📌 Refactor custom JS for CKEditor5 v45+ Active .

Thanks, @jannakha. MR !37 does fix this issue. As far as I can see, it also fixes the issue 🐛 Fix keyboard accessibility of advanced attribute fields Active .

vaish → created an issue.

Patch from #12 re-rolled to work with version 3.1.1.

Thanks, @thejimbirch. I appreciate your feedback. Are you proposing to use Drupal's Configuration API and make CRLs configuration importable/exportable? I would really love to make this module as easy to use as possible, however, my primary concern is performance. Considering that CRL needs to inspect (process) each and every request (being implemented as http middleware → ), my goal was to make it so that overhead of using the module is minimal. I believe that using the Configuration API would increase that overhead. Or did you have something else in mind? Let me know.

Thanks @ressa. I'm not sure if I would want to add anything else to the alert script. I see one-liners from my README as something I would want to use after I receive the alert email in order to analyze the logs and try to figure out the cause of the high load average. If you have any other ideas or need my help in adopting/tweaking the one-liners, please let me know.

Thanks for your feedback @ressa. Thanks for pointing out to the page about blocking bad bots and crawlers. Any suggestions what would you like to add to that script. Currently it only reports high load average which probably makes sense. Also, what do you think about mentioning this module on your page as one of the options for blocking bad bots and crawlers?

vaish → created an issue.

vaish → created an issue.

vaish → created an issue.

vaish → created an issue.

vaish → created an issue.

I have no metrics on how many requests it's blocked, and I'm not able to adjust the settings on the fly as everything is hard-coded in settings.php. It would be great to see these things.

You can analyze your server logs and look for requests with response code 429. Server logs, in general, are source of all the info you may need in order to configure the module optimally and monitor its effectiveness.

Everything is hard-coded in settings.php by design. Goal of this module is to have minimal effect on performance.

@alexh, sorry for the late reply. I was away due to personal reasons. I would like to try and help you resolve your issue if you are still interested.

I have personal experience with bots endlessly crawling faceted search result pages going through all possible combinations of facets. In my case, though, User Agents were commonly recognized as bots and handled by bot_traffic limits. However, only occasionally they would cross bot rate limits I set and only then they would be blocked.

If I understand correctly, in your case, User Agent strings are spoofing browsers and, on top of that, adding random strings to make them appear unique. Unique UA strings combined with multiple IP addresses make it very difficult for regular_traffic limit to ever be reached. That's where rate limiting by ASN should be able help.

I can think of several reasons for regular_traffic_asn limit not being reached:

• There is an issue with module configuration
• ASN database is outdated and it doesn't contain entries for IP addresses you are trying to rate limit.
• IP addresses you would like to rate limit are belonging to multiple different ASNs.

You also mentioned that these requests are bringing your server down. If that's the case, problem may be outside of Drupal and Crawler Rate Limit. When server receives more requests than it can handle, some requests will have to wait until they get processed. At some point server won't be able keep up and it will start returning 500 errors before Drupal and Crawler Rate Limit ever get a chance to handle the request.

Is there a log which shows if any limits by this module have been triggered?

Crawler Rate Limit does not perform any logging. That's by design. Goal of the module is to have minimal effect on performance. However, you can inspect your server logs and search for requests with response code 429.

Note that you can review status of the module on Drupal's Status Report page (/admin/reports/status). There you should be able to see if module is configured correctly and enabled.

I suggest you start by reviewing the Status Report page. Taking a screenshot of the Crawler Rate section and posting it here might also be a good idea.

Is there an option to further limit requests, e.g. disregard the User-Agent and only block by IP?

Rate limiting by only IP is not available.

Note that, at the moment, I'm working on a new feature which will allow you to block (as opposed to rate-limiting) requests by ASN. You will, of course, need to analyze and understand your website traffic in order to make sure that blocking a whole ASN won't block genuine visitor traffic. Again, your web server logs should give you all the info you may need for that.

vaish → made their first commit to this issue’s fork.

Thanks for the patch, @roman.haluska. Note that your patch came in at the same time I made some changes and removed the function you patched from the final version. See my previous comment for more details.

Thanks everyone. Please note that merged version contains some important changes compared to the initial MR.

1. Allowlist setting has been changed to an array and renamed:

    $settings['crawler_rate_limit.settings']['ip_address_allowlist'] = [];

2. Function updateClientIpXForwardedFor() has been removed. I believe that this functionality does not belong here. Figuring out real IP address should be handled for the whole Drupal install and not for a single module. Additionally, this particular implementation has been focused on Cloudflare and may or may not work with other proxies. Note that Symfony's getClientIp() method can read the client IP address from the "X-Forwarded-For" header when trusted proxies were set via "setTrustedProxies()". Drupal exposes this as a setting in the settings.php file. See:

• Using a load balancer or reverse proxy →
• New reverse proxy settings format introduced in Drupal 8.7.0 for Symfony 4 compatibility →
• Reverse Proxy Header →
• Trusted Reverse Proxy →
• CloudFlare →

Patch from #4 re-rolled for the 3.1.x version. Great work on the version 4 of the module!

poker10 → credited vaish → .

Thanks for the patch, @amanbtr72. Did you verify if your patch still works with Drupal 10.2?

vaish → created an issue.

vaish → created an issue.

vaish → created an issue.

vaish → made their first commit to this issue’s fork.

Merge request !11 created from the patch in #17. I've been using this patch in production for very long time.

@Blush I didn't include your changes from #19 because I didn't experience any issues with anonymous users not being able to use address autocomplete. If you still think there is a problem, please test the new release of the module which will be created soon and create new issue with steps how to reproduce it. Thanks.

vaish → changed the visibility of the branch 1.0.x to hidden.

@geoffreyr I decided to merge patch from #2 because I've been using it for a along time in production while I'm not familiar with the seemingly unrelated changes you added to MR !6. Could you please open up new issue for your changes where you would explain which problem you are trying to solve.

vaish → changed the visibility of the branch 3310478-webform-integration to hidden.

vaish → made their first commit to this issue’s fork.

vaish → created an issue.

@Sourav_Paul, I'm already working on this.

vaish → created an issue.

vaish → changed the visibility of the branch 2.0.x to hidden.

vaish → changed the visibility of the branch 3445097-warning-undefined-array to hidden.

Commit a1dedd2c was a mistake. It introduced a schema error and needs to be reverted.

This is expected behavior. Default value on the field edit page expects both country and phone number to be populated. If you only want to set default country, you should use form display setting.

vaish → made their first commit to this issue’s fork.

vaish → created an issue.

Thanks @chrisolof. Everything works great. I just made few minor tweaks.

Darell, there are two steps you need to complete before being able to use ASN-level rate limiting. You did download the ASN database already. What's left is to install PHP package geoip2/geoip2. That's what error message you got is about. You can install this package via composer, as usual.

composer require geoip2/geoip2

Please, let me know if you run into any other issues with this feature. I'm about to merge this MR but feel free to open a follow up issue if you find any bugs.

vaish → made their first commit to this issue’s fork.

vaish → created an issue.

Thanks for reporting the bug. I changed the default value from NULL to 'flag' to make it consistent with the other parts of the module. @nnkavya could you please confirm this MR still resolves your issue.

vaish → made their first commit to this issue’s fork.

Thank you.

vaish → made their first commit to this issue’s fork.

I resolved the merge conflict and addressed the nitpicks. Moving to needs review.

vaish → made their first commit to this issue’s fork.

I run into this issue after the upgrade to 4.2.6. Solution suggested in comment #10 worked for me. VBO field doesn't serve any purpose in the "data export" display.

I would rather try to avoid invoking any hooks in this module. In order for the module to be really useful, it should be able to process each request as early and as fast as possible. Invoking hook adds an overhead which is avoidable. For the same reason I decided not to use Drupal config and create regular configuration page.

Thanks for the feedback and let me know if you happen to decide to work on this.

This would be great improvement. Are you planning on creating a MR? Note that issue ✨ [Patch provided] Allow administrators to specify IP addresses and ranges which should never be rate limited Needs review provides a setting for ignoring IP addresses. We should make sure that the structure of the settings array and naming of the array keys remains consistent and intuitive after both settings have been added. E.g. both setting should use the same verb, either "ignore" or "allow", etc.

Thanks @generalredneck. It's great to have this added to the README.

Thanks for the screenshots. I looked into this and I can reproduce the issue.

If checkbox "My billing information is the same as my shipping information." is checked during the checkout, commerce will populate "Billing information" fields with the values taken from the "Shipping information" section. Value of the Phone field is not copied successfully and Phone field in the "Billing information" remains empty causing the validation error reported here. So, this error refers to the empty field in the "Billing information" section.

Copy of the values is initiated in the Drupal\commerce_shipping\ProfileFieldCopy::alterForm()

      // Copy over the current shipping field values, allowing widgets such as
      // TaxNumberDefaultWidget to rely on them. These values might change
      // during submit, so the profile is populated again in submitForm().
      $billing_profile->populateFromProfile($shipping_profile, $shipping_fields);

Drupal\profile\Entity::populateFromProfile() depends on the TypeData plugins to correctly set values for each of the fields. Both Phone number and Address fields implement Drupal\Core\TypedData\Plugin\DataType\Map plugin. Map::setValue() method is responsible for setting the values.

    // Update any existing property objects.
    foreach ($this->properties as $name => $property) {
      $value = $values[$name] ?? NULL;
      $property->setValue($value, FALSE);
      // Remove the value from $this->values to ensure it does not contain any
      // value for computed properties.
      unset($this->values[$name]);
    }

Problem is that in the case of the Phone Number $this->properties is empty (properties are not set) and therefore no values are set. Phone field defines the following properties: 'value', 'country', 'local_number', and 'extension' but for some reason those properties are not available in the setValues() method.

As a comparison, Address field defines its own properties ('address_line1', 'locality', 'country_code', etc) and those are all available in the setValue() method and values are correctly copied.

Unfortunately, I don't have time to look into this further right now. I hope somebody else can chime in.

Are you getting this error even after you populate the Phone field? Please clarify. Screenshot might be helpful here.

Thanks @orakili. MR looks good to me (both code and functionality) and if you are fine with the changes I made I think this can be moved to RTBC.

I run into this after installing content_entity_clone module on a site where admin_toolbar_tools is enabled and "Enable/Disable local tasks display" setting is turned on. Content entity clone uses hook_menu_local_tasks_alter() to create new local task but it doesn't define #active key. Fix is included in the MR for ✨ Admin content add link in operation Needs review . Unfortunately, I had trouble reproducing the issue. I'm getting it on a project with many contrib and custom modules but not on the clean Drupal install. Judging by the previous comments, issue is extremely rare but it may happen under very specific circumstances.

I was about to create new issue when I noticed that it was already fixed in this MR so I decided to add an additional related fix here instead of creating new issue. These are the issues I'm referring to:

1. Query string is missing when "Clone" local task is displayed in the admin toolbar. This requires that admin_toolbar_tools module is enabled and "Enable/Disable local tasks display" setting is turned on. This has been fixed in this MR by adding the third parameter $options to the Url::fromRoute() call.

  $options = [
    'query' => [
      'content_entity_clone' => $entity->id(),
    ],
  ];
  $url = Url::fromRoute($entity_creation_route_name, $entity_creation_route_parameters, $options);

2. I also run into the issue 🐛 Undefined array key "#active" in Drupal\admin_toolbar_tools\AdminToolbarToolsHelper->buildLocalTasksToolbar() Needs review . As per the docs local task is expected to have "#active" key but in this case key was not defined. Unfortunately, I had trouble reproducing this issue. I'm getting it on a project with many contrib and custom modules but not on the clean Drupal install. I still think it makes sense to define this key. Value will always be FALSE because it's impossible for "Clone" tab to ever be active.

3. I also fixed some minor typos.

Additionally, I noticed that this MR also fixes 🐛 Where is the link to create a clone from node? Closed: duplicate . Marking that other issue as duplicate.

Fix for this issue included in ✨ Admin content add link in operation Needs review .

vaish → made their first commit to this issue’s fork.

vaish → made their first commit to this issue’s fork.

Thanks @chrisolof.

Forgot to update credits.

Thank you, @zaryab_drupal.

vaish → made their first commit to this issue’s fork.

Thanks @mpaler. All these years we were using the patch from #7 which did the job for us. But recently, the module got some updates and the patch couldn't be applied anymore. So instead of re-rolling the patch that will never be merged I worked on the proper solution. Better late than never.

vaish → created an issue.