censys: query the Censys public scan database

Python code to query the Censys public scan database. This script is made around library censys-python (https://github.com/Censys/censys-python) and is inteded to make censys queries quick & easy from command-line.

• Requirements
• Usage
  • Note about looking for substrings
• Examples
  • Generic query IP or host (look for anything matching the string in Censys indexed data)
  • Count how much web servers have ‘SAP’ in their Server header
  • Get geo reparition of server with ‘ABAP’ in their Server header
  • Filter hosts by their HTML title
  • Retrieve the hosts that have SSL certificate with organization ‘Whatsapp’
  • Highlight which keys are matching string for a specific search query
  • Printing raw JSON record from database for a specific request
  • Listing example of fields we only want to dump in the records
  • Export to disk sites frontpage containing “Hacked by” in their title
  • Use tags provided by censys scanner to look at servers that need to be secured
• But what the heck, you want to hack the planet?!

Requirements

You need to create an account on https://censys.io and get your API key and secret at https://censys.io/account

Important note: your queries will be throttled. What is allowed is 0.2 tokens/second (60.0 per 5 minute bucket).

$ sudo pip install -r requirements.txt

Usage

$ censys_io.py --help

usage: censys_io.py [-h] [-m MATCH] [-f FILTER] [--count] [-r REPORT]

                    [-B REPORT_BUCKET] [-a ASN] [-c COUNTRY] [-o CERT_ORG]

                    [-i CERT_ISSUER] [-s CERT_HOST] [-S HTTP_SERVER]

                    [-t HTML_TITLE] [-b HTML_BODY] [-T TAGS] [--api_id API_ID]

                    [--api_secret API_SECRET] [-d] [-v] [-l LIMIT] [-H]

                    [--tsv]

                    [arguments [arguments ...]]



Censys query via command line



-- gelim



positional arguments:

  arguments             Censys query



optional arguments:

  -h, --help            show this help message and exit

  -m MATCH, --match MATCH

                        Highlight a string within an existing query result

  -f FILTER, --filter FILTER

                        Filter the JSON keys to display for each result (use value 'help' for interesting fields)

  --count               Print the count result and exit

  -r REPORT, --report REPORT

                        Stats on given field (use value 'help' for listing interesting fields)

  -B REPORT_BUCKET, --report_bucket REPORT_BUCKET

                        Bucket len in report mode (default: 50)

  -a ASN, --asn ASN     Filter with ASN (ex: 36040 for Google Inc.)

  -c COUNTRY, --country COUNTRY

                        Filter with country

  -o CERT_ORG, --cert-org CERT_ORG

                        Cert issued to org

  -i CERT_ISSUER, --cert-issuer CERT_ISSUER

                        Cert issued by org

  -s CERT_HOST, --cert-host CERT_HOST

                        hostname cert is issued to

  -S HTTP_SERVER, --http-server HTTP_SERVER

                        Server header

  -t HTML_TITLE, --html-title HTML_TITLE

                        Filter on html page title

  -b HTML_BODY, --html-body HTML_BODY

                        Filter on html body content

  -T TAGS, --tags TAGS  Filter on specific tags. E.g: -T tag1,tag2,... (use keyword 'list' to list usual tags

  --api_id API_ID       Censys API ID (optional if no env defined

  --api_secret API_SECRET

                        Censys API SECRET (optional if no env defined)

  -d, --debug           Debug informations

  -v, --verbose         Print raw JSON records

  -l LIMIT, --limit LIMIT

                        Limit to N results

  -H, --html            Renders html elements in a browser

  --tsv                 Export result of search in TSV format

For full details about the formatting rules for arguments see search syntax in page https://censys.io/ipv4/help?q=x%3Ax

For a quick and dirty test, you can build queries like:

• foo AND bar (will do a smart search by checking all keys with value foo and bar)
• path.to.key:foo
• key:foo (shortcut of previous, but will give strange results if there are collision with other keys)
• key:/regex/ (regexp support via operator ‘/’)
• key:"long string with spaces" (need to quote those strings)
• key:[200 TO 300] (int range queries)
• key:192.168.0.0/24 (IP range query)

Examples

Generic query IP or host (look for anything matching the string in Censys indexed data)

Let’s search for IP entries that contain the string “nmap” in one of their keys.

$ censy_io.py –limit 20 nmap
Number of results: 1002
5.196.225.134 Title: N/A SSL: dawidstachowiak.pl AS: OVH, (16276) Loc: FR / OS: N/A Tags: http, ssh, https
74.115.246.29 Title: BrainDump SSL: philmcclure.duckdns.org AS: ENERGIZE (19215) Loc: US / Pulaski OS: N/A Tags: http, ssh, https
104.237.156.37 Title: Starlight Networking Security Lab SSL: AS: LINODE-AP (63949) Loc: US / Absecon OS: N/A Tags: http, ssh
69.160.84.231 Title: N/A SSL: AS: FIBER (5048) Loc: US / Orem OS: CentOS Tags: http, ssh
45.79.82.183 Title: nweb.io SSL: nweb.io AS: LINODE-AP (63949) Loc: US / Absecon OS: N/A Tags: http, ssh, https
60.32.137.218 Title: Kyodo2.0 Digital-Lab News Map Project SSL: localhost.localdomain AS: OCN (4713) Loc: JP / Tokyo OS: Fedora Tags: dhe-export, rsa-export, http, https
104.237.129.231 Title: Ninja.Style SSL: AS: LINODE-AP (63949) Loc: US / Absecon OS: Ubuntu Tags: http, ssh
192.109.14.42 Title: PASA Pallas SSL: pasa.pallas.com AS: PALLAS-AS, (24861) Loc: DE / OS: N/A Tags: http, https
45.33.32.156 Title: Go ahead and ScanMe! SSL: AS: LINODE-AP (63949) Loc: US / Absecon OS: Ubuntu Tags: http, ssh
104.224.137.222 Title: SSL: AS: IT7NET (25820) Loc: US / Phoenix OS: CentOS Tags: http
119.81.35.59 Title: SL Labs SSL: AS: SOFTLAYER (36351) Loc: SG / Singapore OS: CentOS Tags: http
81.27.98.98 Title: Check for Web Servers and more SSL: AS: UK-NETCETERA (24851) Loc: GB / OS: Debian Tags: http
212.237.16.237 Title: Infosec Notes SSL: 2d8.ru AS: ARUBA-ASN, (31034) Loc: DK / OS: Ubuntu Tags: http, smtp, https
198.23.94.99 Title: SL Labs SSL: AS: SOFTLAYER (36351) Loc: US / San Jose OS: CentOS Tags: http
77.109.162.35 Title: Citrin Toolbox SSL: AS: INIT7, (13030) Loc: CH / OS: N/A Tags: http
121.42.165.133 Title: SSL: AS: CNNIC-ALIBABA-CN-NET-AP (37963) Loc: CN / Hangzhou OS: CentOS Tags: http, ssh
169.55.196.202 Title: SL Labs SSL: AS: SOFTLAYER (36351) Loc: US / OS: CentOS Tags: http
119.81.209.6 Title: SL Labs SSL: AS: SOFTLAYER (36351) Loc: SG / Singapore OS: CentOS Tags: http
216.59.36.36 Title: Wait, wha? SSL: AS: IMMEDION (15085) Loc: US / Greenville OS: N/A Tags: http
204.152.250.58 Title: My Blog SSL: AS: BCC-65-182-96-0-PHX (33055) Loc: US / Phoenix OS: N/A Tags: http