URLextractor

Information gathering & website reconnaissance

Features:

• IP and hosting info like city and country (using FreegeoIP)
• DNS servers (using dig)
• ASN, Network range, ISP name (using RISwhois)
• Load balancer test
• Whois for abuse mail (using Spamcop)
• PAC (Proxy Auto Configuration) file
• Compares hashes to diff code
• robots.txt (recursively looking for hidden stuff)
• Source code (looking for passwords and users)
• External links (frames from other websites)
• Directory FUZZ (like Dirbuster and Wfuzz – using Dirbuster) directory list)
• URLvoid API – checks Google page rank, Alexa rank, and possible blacklists
• Provides useful links at other websites to correlate with IP/ASN
• Option to open ALL results in a browser at the end

Install

Requirement:

sudo apt-get install bc curl dnsutils libxml2-utils whois md5sha1sum lynx openssl -y

Download

git clone https://github.com/eschultze/URLextractor.git

Use

Configuration file:

CURL_TIMEOUT=15 #timeout in --connect-timeout

CURL_UA=Mozilla #user-agent (keep it simple)

INTERNAL=NO #YES OR NO (show internal network info)

URLVOID_KEY=your_API_key #using API from http://www.urlvoid.com/

FUZZ_LIMIT=10 #how many lines it will read from fuzz file

OPEN_TARGET_URLS=NO #open found URLs at the end of script

OPEN_EXTERNAL_LINKS=NO #open external links (frames) at the end of script

FIRST_TIME=YES #if first time check for dependecies

Run:

./extractor http://www.hackthissite.org/

Tips:

• Colorex: put colors to the output pip install colorex and use it like ./extractor http://www.hackthissite.org/ | colorex -g “INFO” -r “ALERT”
• Tldextract: is used by dnsenumeration function pip install tldextract

Copyright (c) 2016 eschultze

Source: https://github.com/eschultze/