frontera v0.8.1: A scalable frontier for web crawlers
Frontera Overview Frontera is a web crawling framework consisting of crawl frontier, and distribution/scaling primitives, allowing to build a large-scale online web crawler. Frontera takes care of the logic and policies to follow during the...
