The Ultimate Bad Bot and Referer Blocker for Apache Web Server 2.4.x
This bot blocker list is designed to be an Apache include file and uses the apache BrowserMatchNoCase directive. This way the .conf file can be loaded once into memory by Apache and be available to all web sites that you operate. You simply need to use an Include statement (example below)
I personally find the BrowserMatchNoCase Directive to be more accurate than using SetEnvIfNoCase User-Agent because BrowserMatchNoCase is not case sensitive and from my tests is more accurate that SetEnvIfNoCase.
My method also results in a cleaner file to maintain that requires no complex regex other than the Name of the Bot. BrowserMatchNoCase will do the rest. You can use Regex if you like but it’s NOT needed and I proved it by testing with the Chrome extension User-Agent Switcher for Chrome.
- The user agent “Aboundex” is found without using “^Aboundex” … much simpler for anyone to maintain than other lists using Regex.
- Likewise, it is unnecessary to have “Download\ Demon” instead you now just have “Download Demon”.
- Additionally, if we have a rule, like below “Image Stripper” and a bot decides to change its User-Agent string to “NOT Image Stripper I Promise” he is picked up regardless and blocked immediately.
FEATURES OF THE APACHE ULTIMATE BAD BOT BLOCKER:
- Extensive Lists of Bad and Known Bad Bots and Scrapers (updated almost daily)
- Blocking of Spam Referrer Domains and Web Sites
- Blocking of SEO data collection companies like Semalt.com, Builtwith.com, WooRank.com and many others (updated regularly)
- Blocking of clickjacking Sites linked to Adware, Malware and Ransomware
- Blocking of Porn and Gambling Web Sites who use Lucrative Ways to Earn Money through Serving Ads by hopping off your domain names and websites.
- Blocking of Bad Domains and IP’s that you cannot even see in your Nginx Logs. Thanks to the Content Security Policy (CSP) on all my SSL sites I can see things trying to pull resources off my sites before they even get to Nginx and get blocked by the CSP.
- Anti-DDOS Filter and Rate Limiting of Aggressive Bots
- Alphabetically ordered for easier maintenance (Pull Requests Welcomed)
- Commented sections of certain important bots to be sure of before blocking
- Includes the IP range of Cyveillance who are known to ignore robots.txt rules and snoop around all over the Internet.
- Whitelisting of Google, Bing and Cloudflare IP Ranges
- Whitelisting of your own IP Ranges that you want to avoid blocking by mistake.
- Ability to add other IP ranges and IP blocks that you want to block out.
- If its out there and it’s bad it’s already in here and BLOCKED !!