Database of unwanted user agents

Created on 8 July 2023, over 1 year ago

Updated 26 July 2023, over 1 year ago

Problem/Motivation

This issue is to ensure that we collect agressive crawlers/bots as a user agent. This way we could create our own database that helps to block out unwanted user agents. Before using the crawlers, please be sure to test carefully that normal users are not being locked out.

User Agents

Primary observation/analysis of what application: Microsoft Office 365 crawler, primarily used by Outlook
Exact name of the user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36

Primary observation/analysis of what application: Microsoft Office 365 crawler, primarily used by Outlook
Exact name of the user agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b

📌 Task

Status

Active

Version

2.0

Component

Documentation

Created by

🇩🇪Germany zcht

Live updates comments and jobs are added and updated live.

Comments & Activities

Issue created by @zcht
Comment over 1 year ago →
🇩🇪Germany zcht
Comment over 1 year ago →
🇩🇪Germany zcht
Two new user agents, one from China and the other from the USA. Sometimes cause problems with login, the email links are scanned in advance and invalidated. You could check this and if it is the case, add it to the configuration to block it. It has not yet been possible to find out which companies the user agents belong to.

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, wie Gecko) Chrome/83.0.4103.97 Safari/537.36 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, wie Gecko) Chrome/75.0.3770.142 Safari/537.36

contrib.social Blog FAQ Discussions

Production build 0.71.5 2024