Web Application Firewall (WAF) Comparison Project

by do son · October 22, 2023

Web Application Firewall (WAF) Comparison Project

This project repository contains testing datasets and tools to compare WAF efficacy in the two most important categories:

Security Coverage (True Positive Rate) – measures the WAF’s ability to correctly identify and block malicious requests is crucial in today’s threat landscape. It must preemptively block zero-day attacks as well as effectively tackle known attack techniques utilized by hackers
Precision (False Positive Rate) – measures the WAF’s ability to correctly allow legitimate requests. Any hindrance to these valid requests could lead to significant business disruption and an increased workload for administrators.

This project aims to measure the efficacy of each WAF against a variety of legitimate and malicious HTTP requests, taken from real-world scenarios.

The project is described in detail in this blog.

Methodology

Each WAF solution is tested against two data sets: legitimate and malicious. We then used a formula described below in detail to produce a single balanced score.

Legitimate Data Set

The Legitimate Requests Dataset is carefully designed to test WAF behaviors in real-world scenarios. To attain this, it includes 973,964 different HTTP requests from 185 real-web sites in 12 categories. Each dataset was recorded by browsing to real-world websites and conducting various operations in the site (for example, sign-up, selecting products and placing in a cart, etc) ensuring the presence of 100% legitimate requests.

The dataset can be found in the folder Data/Legitimate

Malicious Data Set

The Malicious Requests Dataset includes 73,924 malicious payloads from a broad spectrum of commonly experienced attack vectors:

SQL Injection-
Cross-Site Scripting (XSS)
XML External Entity (XXE)
Path Traversal
Command Execution
Log4Shell
Shellshock

The malicious payloads were sourced from the WAF Payload Collection GitHub page that was assembled by mgm security partners GmbH from Germany. This repository serves as a valuable resource, providing payloads specifically created for testing Web Application Firewall rules.

The dataset is available here

Tooling

To trigger the data sets through the different devices under test, we developed a simple test tool in Python. The test tool is designed to ingest data sets as input and send each request to the various WAFs being tested. It reads the data files from the data sets and uses the requests module in a multi-threaded manner to send the data to each WAF.

During the initial phase, the tool conducts a dual-layer health check for each WAF. This process first validates connectivity to each WAF, ensuring system communication. It then checks that each WAF is set to prevention mode, confirming its ability to actively block malicious requests.

The responses from each request sent by the test tool to the WAFs were systematically logged in a dedicated database for further analysis. The database we used is an AWS RDS instance running PostgreSQL (the database is not included in this repo). You can configure it to work with any SQL database of your preference by adjusting the settings in the config.py file.