Currently there is no way to bypass the crawlers. For example if you have website which has facebook sso and according to the application we need to provide url which facebook crawls, enabling this shield module fails that crawling no matter if you add IP Address list or Domain
We can create textarea which will define crawler source and by pass it via adding one of the condition in middleware, so said crawler source will work as expected.
Needs work
1.0
Code
Not all content is available!
It's likely this issue predates Contrib.social: some issue and comment data are missing.
Hi @abhaypai,
It seems an interesting feature that you requested and provided a patch. Before including in the module, I think we should make it more generic and use the word "User agent" instead of crawler as it actually test the configured values against the user agent. We should probably display a warning message as the user agent is easily manipulable and so Shield can be by-passed using that method. Finally, we should add some tests.