Not Your Average Web Crawler: web crawler for bug hunting

Not Your Average Web Crawler

N.Y.A.W.C is a Python library that enables you to test your payload against all requests of a certain domain. It crawls all requests (e.g. GET, POST or PUT) in the specified scope and keeps track of the request and response data. During the crawling process, the callbacks enable you to insert your payload at specific places and test if they worked.

Crawling flow

1. You can define your startpoint (a request) and the crawling scope and then start the crawler.
2. The crawler repeatedly starts the first request in the queue until max threads is reached.
3. The crawler adds all requests found in the response to the end of the queue (except duplicates).
4. The crawler goes back to step #2 to spawn new requests repeatedly until max threads is reached.

Install

git clone https://github.com/tijme/not-your-average-web-crawler.git
pip install –upgrade nyawc
cd not-your-average-web-crawler.git
pip install -r requirements.txt
python -m unittest discover
python setup.py install

Usage

Copyright (c) 2017 Tijme Gommers