dirble v1.4.2 released: Fast directory scanning and scraping tool
dirble
Dirble is a website directory scanning tool for Windows and Linux. It’s designed to be fast to run and easy to use.
How it works
Directory Detection
Dirble detects files based on the response code sent by the server. The behavior can be loosely categorized by response code type.
- 200: the path exists and is valid
- 301, 302: redirection; report the code, size, and Location header
- 404: not found; by default, these responses are not reported
- All other response codes are reported in the Dirble format of + [url] (CODE:[code]|SIZE:[size])
A path is classified as a directory if a request to [url] (with no trailing slash) returns a 301 or 302 redirection to [url]/ (with a trailing slash). This gets reported with a D prefix and if recursion is enabled will be added to the scan queue. This method is not dependent on the redirection target existing or being accessible, so a separate request will be made to determine the response code and size of the directory.
Listable directories are detected by inspecting the content of url/: if it returns a 200 response code and the body contains either “parent directory”, “up to ” or “directory listing for” (case insensitive), then it is likely to be a listable directory. If –scrape-listable is enabled, URLs are parsed out of the listing (ignoring sorting links or out of scope links) and added to the scan queue if they have a trailing slash. Listable directories have an L prefix in the output.
Threading
The threading behavior of Dirble is based on the concepts of wordlists and jobs. A job is any task which can be run independently of other tasks, for example requesting a series of URLs. A wordlist is a list of words with a defined transformation, for example, the list {admin, config, shop} together with the transformation append “.php” forms a single wordlist instance.
To improve performance further, we introduce the concept of wordlist splitting. This is the process by which a single wordlist instance (i.e. words with a transformation) is broken up into multiple jobs, each responsible for a portion of the list. The number of interleaved portions that each wordlist is split into is defined by the –wordlist-split option (default 3).
Whenever a directory is detected (and recursion is enabled) new jobs are created for each split wordlist (with transformation) and added to a central job queue.
The maximum number of concurrent tasks is defined by the –max-threads parameter, and Dirble will start jobs as they are added to the queue, up to this limit. Whenever a job completes (i.e. a split wordlist is exhausted) Dirble will take the next job from the queue and start it.
Changelog v1.4.2
Changed
- Extension lists are now in an extensions directory
- Update dependencies
Download
Use
Copyright (C) 2019 nccgroup
Source: https://github.com/nccgroup/
