Host Scanner: Active/passive network scanner and autonomous vulnerability assessment application

Host Scanner

The purpose of this project is to implement a network scanner with both active and passive data acquisition components, which can then autonomously identify services using the latest CPE dictionary from NIST and discover the vulnerabilities of those by querying the CVE database.

Features

• TCP Scanner
  • High-performance TCP scanner which initiates the three-way handshake (also called “TCP connect scan”) by multiplexing non-blocking sockets and grabbing their service banners.
• UDP Scanner
  • Makes use of a database of specifically crafted payloads mapped to port numbers in order to try and get an answer from the UDP services, if there are any listening.
• ICMP Pinger
  • Support for the use of ICMP Echo Request packets (also known as “standard ping”) in order to determine if a host is alive.
• ARP Pinger
  • Support for the use of ARP Who-Has Request packets in order to map online hosts on a local network. Useful when ICMP packets are filtered on a network.
• External Scanners
  • Ability to use external tools for all the active or passive scanning needs, instead of the built-in scanners:
    • Launch new scans with or process earlier XML outputs from Nmap.
• Online Sources
  • Support for passive reconnaissance by fetching data already available from relevant services intended for security researchers:
    • Shodan
    • Censys
    • Mr Looquer
• Service Identification
  • Autonomous Identification:
    • Latest CPE dictionary from NIST is used to map service banners to their CPE names.
  • Pattern-based Identification:
    • Database of regular expressions can be used as a redundancy to map service banners to their CPE names.
• Vulnerability Assessment
  • Based on the CVE database, the resolved CPE names (which also include version numbers) are matched against the affected software list of each CVE entry to discover service vulnerabilities.
• Package Lookup
  • Resolve CPE names to actual operating system packages and get a simple command to update only the vulnerable versions for:
    • Debian (oldstable to unstable), Ubuntu (all current and lts versions)
    • Red Hat (5-7), CentOS (5-7), Fedora (all current and rawhide)
• Vulnerability Validation
  • Use the package manager, security or bug tracker of the identified distribution to check whether a package is vulnerable, whether a vendor patch is available, and whether it is installed or not.
• Estimate System Upgrade Date
  • Using the changelog of the discovered packages with version numbers and security patch information, estimate the date range of the last system upgrade of the host.
• Reporting
  • Generate a L^AT_EX report of the scanned network, which includes open ports, identified services, discovered vulnerabilities and mitigation recommendations.
• Unit Tests
  • All features are covered by unit tests which are run on three platforms in order to ensure utmost stability and portability.
• Portability
  • Features are implemented (when a standardized API is not available) using raw sockets on Linux, WinPcap on Windows, and Berkeley Packet Filter on BSD / OS X.

Building

To compile and run the application, you must first install the dependencies, which can be done with:

• Debian/Ubuntu/Kali:

   apt install build-essential cmake libcurl4-openssl-dev libsqlite3-dev libboost-all-dev libz-dev
  

• RHEL/CentOS/Fedora:

   yum install gcc-c++ make cmake libcurl-devel sqlite-devel boost-devel-static zlib-devel
  

• FreeBSD:

   pkg install cmake curl sqlite3 boost-libs
  

• Mac OS X: (with Homebrew)

   xcode-select --install && brew install cmake curl sqlite boost
  

• Windows or any of the above platforms if problems arise with the vendor packages: (with Conan)

   conan install --build=missing
  

  When executed in the project root, downloads and/or builds all required dependencies for the project and generates a conanbuildinfo.cmake file, which when exists, will be used by CMakeLists.txt to configure include directories and link targets.

After the dependencies have been installed, you can check out the repository and compile it with the following commands:

git clone https://github.com/RoliSoft/Host-Scanner.git
cd Host-Scanner/build
cmake ..
make

If the compilation was successful, you can run it with the ./HostScanner command. Tests are also available, you may run them through make test or directly, by executing ./TestScanner.

Usage

usage: HostScanner [args] targets

arguments:



  -t [ --target ] arg     List of targets to scan:

                            Each can be a hostname, IP address, IP range or CIDR.

                            E.g. `192.168.1.1/24` is equivalent to `192.168.1.0-192.168.1.255`.



  -p [ --port ] arg       TCP ports to scan:

                            Can be a single port (80), a list (22,80) or a range (1-1024).

                            Range can be unbounded from either sides, simultaneously.

                            E.g. `1024-` will scan ports 1024-65535. `-` will scan all ports.

                            Specifying `top` or `t` will scan the top 100 most popular ports.



  -u [ --udp-port ] arg   UDP ports to scan:

                            Supports the same values as --port, with the difference that

                            specifying `top` will scan all of the ports with known payloads.



  -s [ --scanner ] arg    Scanner instance to use:

                            internal - Uses the built-in scanner. (active)

                            nmap     - Uses 3rd-party application Nmap. (active)

                            shodan   - Uses data from Shodan. (passive; requires API key)

                            censys   - Uses data from Censys. (passive; requires API key)

                            looquer  - Uses data from Mr Looquer. (passive; requires API key)

                            shosys   - Uses data from Shodan, Censys and Mr Looquer. (passive)



  --shodan-key arg        Specifies an API key for Shodan.

  --shodan-uri arg        Overrides the API endpoint used for Shodan. You may specify an URI

                          starting with file:// pointing to a directory containing previously

                          downloaded JSON responses.

                            Default: https://api.shodan.io/shodan



  --censys-key arg        Specifies an API key for Censys in the `uid:secret` format.

  --censys-uri arg        Overrides the API endpoint used for Censys. You may specify an URI

                          starting with file:// pointing to a directory containing previously

                          downloaded JSON responses.

                            Default: https://censys.io/api/v1



  --looquer-key arg       Specifies an API key for Mr Looquer.

  --looquer-uri arg       Overrides the API endpoint used for Mr Looquer. You may specify an URI

                          starting with file:// pointing to a directory containing previously

                          downloaded JSON responses.

                            Default: https://mrlooquer.com/api/v1



  -f [ --input-file ] arg Process an input file with the selected scanner.

                            E.g. the nmap scanner can parse XML reports.



  -d [ --delay ] arg      Delay between packets sent to the same host. Default is 3 for 100ms.

                          Possible values are 0..6, which have the same effect as nmap's -T:

                            0 - 5m, 1 - 15s, 2 - 400ms, 3 - 100ms, 4 - 10ms, 5 - 5ms, 6 - no delay



  -r [ --resolve ]        Resolves vulnerable CPE names to their actual package names depending on

                          the automatically detected operating system of the host.



  -w [ --validate ]       Validate all vulnerabilities with the package manager of the distribution.



  -e [ --estimate ]       Estimate date range of the last system upgrade based on the installed

                          package versions and security patches.



  -o [ --output-latex ] arg Exports the scan results into a LaTeX file, with all the available

                            information gathered during the scan.



  -x [ --passive ]        Globally disables active reconnaissance. Functionality using active

                          scanning will break, but ensures no accidental active scans will be

                          initiated, which might get construed as hostile.



  -l [ --logging ] arg    Logging level to use:

                            i, int - All messages.

                            d, dbg - All debug messages and up.

                            v, vrb - Enable verbosity, but don't overdo it.

                            m, msg - Print only regular messages. (default)

                            e, err - Print only error messages to stderr.



  -q [ --no-logo ]        Suppresses the ASCII logo.

  -v [ --version ]        Display version information.

  -h [ --help ]           Displays this message.

Examples

Scan a network for vulnerabilities on the top 100 TCP ports and known UDP ports using the internal scanners:

./HostScanner -p t -u t 192.168.1.0/24

Scan an IP address or netblock for vulnerabilities passively, with data from Shodan, Censys and Mr Looquer:

./HostScanner -x 178.62.192.0/18

Perform service identification and vulnerability analysis on an earlier XML output of nmap through nmap -oX report.xml …:

./HostScanner -s nmap -f report.xml

Get list of vulnerable packages and command to upgrade it on the host:

./HostScanner -r 192.168.1.66 192.168.1.71

The above will scan the TCP ports of the specified addresses, perform operating system and service detection followed by vulnerability analysis, and lookup the packages needed to be updated for the discovered CVEs to be mitigated:

[*] 192.168.1.66 is running cpe:/o:debian:debian_linux:8
[*] 192.168.1.71 is running cpe:/o:redhat:enterprise_linux:7
    ...
[*] 192.168.1.66 -> sudo apt-get install --only-upgrade apache2 php5 python2.7
[*] 192.168.1.71 -> sudo yum update httpd php python27-python

Copyright (c) 2016 RoliSoft <root@rolisoft.net>

Source: https://github.com/RoliSoft/