dcrawl: multi-threaded web crawler
dcrawl dcrawl is a simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names. How does it work? dcrawl takes one site URL as input and detects all <a href=…> links...
Tagged: dcrawl
