Dirscraper

Dirscraper is an OSINT scanning tool which assists penetration testers in identifying hidden, or previously unknown, directories on a domain or subdomain. This helps greatly in the recon stage of pentesting as it provides pentesters with a larger attack surface for the specific domain.

How does it work?

Dirscraper works by initially visiting the domain provided by the user. From there, it locates all relative script tags hosted on the website. After this, it reads the source code of all those javascript files and locates interesting subdomains and endpoints used in those javascript files. A lot of website developers will not make endpoints publically available but will still allow users to interact with them through javascript when appropriate. Sometimes it takes a rare corner case for this criteria to be met (and for a tool such as Burp Suite to pick up the request to the endpoint) and it becomes unpractical to manually locate these endpoints.

Install

git clone https://github.com/Cillian-Collins/dirscraper.git

pip install -r requirements.txt

Use

To run the program, you will need to open the directory containing the file with your terminal. From there, run the following command containing the URL of the site you wish to scan:

$ python dirscraper.py -u <URL>

When outputting to a file, you must select a filename (if it already exists, it will append results to the bottom, if it doesn’t exist it will create the new file). This flag is optional.

$ python dirscraper.py -u <URL> -o <FILE>

If you are scanning a website and do not wish to see the results displayed in the terminal, then you can set this flag. If you are not outputting to a file, then using this flag will make it impossible to see your results. This flag is optional.

python dirscraper.py -u <URL> -o <FILE> -s

Source: https://github.com/Cillian-Collins/