katana v1.0.4 releases: next-generation crawling and spidering framework

Katana

A next-generation crawling and spidering framework

Feature

• Fast And fully configurable web crawling
• Standard and Headless mode support
• JavaScript parsing / crawling
• Customizable automatic form filling
• Scope control – Preconfigured field / Regex
• Customizable output – Preconfigured fields
• INPUT – STDIN, URL and LIST
• OUTPUT – STDOUT, FILE, and JSON

Crawling Mode

Standard Mode

Standard crawling modality uses the standard go http library under the hood to handle HTTP requests/responses. This modality is much faster as it doesn’t have the browser overhead. Still, it analyzes HTTP responses body as is, without any javascript or DOM rendering, potentially missing post-dom-rendered endpoints or asynchronous endpoint calls that might happen in complex web applications depending, for example, on browser-specific events.

Headless Mode

Headless mode hooks internal headless calls to handle HTTP requests/responses directly within the browser context. This offers two advantages:

• The HTTP fingerprint (TLS and user agent) fully identify the client as a legitimate browser
• Better coverage since the endpoints are discovered analyzing the standard raw response, as in the previous modality, and also the browser-rendered one with javascript enabled.

Headless crawling is optional and can be enabled using -headless option.

Scope Control

Crawling can be endless if not scoped, as such katana comes with multiple support to define the crawl scope.

Changelog v1.0.4

🎉 New Features

• Added support for custom regex on field scope -fs option by @c3l3si4n in #571

katana -fs '(company-staging.io|company.com)' -u company.com

• Added option to disable redirects by @WigzyDev in #588

   -dr, -disable-redirects  disable following redirects (default false)

• Added -resume crawl option by @dogancanbakir in #538

   -resume string  resume crawl using resume.cfg

🐞 Bug Fixes

• Fixed issue with -match-regex option by @dogancanbakir in #565
• Fixed headless options parsing issue by @dogancanbakir in #564
• Fixed missing ExtractRelativeEndpoints() in scriptJSFileRegexParser() by @0x123456789 in #587

Other Changes

• Added build deps for go-tree-sitter -> tree-sitter by @dogancanbakir in #547
• Added readme syntax highlight by @apriil15 in #553

Install & Use

Copyright (c) 2022 ProjectDiscovery