Machinae Security Intelligence Collector

Machinae is a tool for collecting intelligence from public sites/feeds about various security-related pieces of data: IP addresses, domain names, URLs, email addresses, file hashes and SSL fingerprints. It was inspired by Automater, another excellent tool for collecting information. The Machinae project was born from wishing to improve Automater in 4 areas:

1. Codebase – Bring Automater to python3 compatibility while making the code more pythonic
2. Configuration – Use a more human-readable configuration format (YAML)
3. Inputs – Support JSON parsing out-of-the-box without the need to write regular expressions, but still support regex scraping when needed
4. Outputs – Support additional output types, including JSON, while making extraneous output optional

Out-of-the-Box Data Sources

Machinae comes with out-of-the-box support for the following data sources:

• IPVoid
• URLVoid
• URL Unshortener (http://www.toolsvoid.com/unshorten-url)
• Malc0de
• SANS
• FreeGeoIP (freegeoip.io)
• Fortinet Category
• VirusTotal pDNS (via web scrape – commented out)
• VirusTotal pDNS (via JSON API)
• VirusTotal URL Report (via JSON API)
• VirusTotal File Report (via JSON API)
• Reputation Authority
• ThreatExpert
• VxVault
• ProjectHoneypot
• McAfee Threat Intelligence
• StopForumSpam
• Cymru MHR
• ICSI Certificate Notary
• TotalHash (disabled by default)
• DomainTools Parsed Whois (Requires API key)
• DomainTools Reverse Whois (Requires API key)
• DomainTools Reputation
• IP WHOIS (Using RIR REST interfaces)
• Hacked IP
• Metadefender Cloud (Requires API key)
• GreyNoise (Requires API key)
• IBM XForce (Required API key)

With additional data sources on the way.

Changelog v1.4.8

• MAC address observables now supported. MACVendors API enabled by default.

Installation

pip3 install machinae

Configuration File

Machinae supports a simple configuration merging system to allow you to make adjustments to the configuration without modifying the machinae.yml we provide you, making configuration updates a snap. This is done by finding a system-wide default configuration (default /etc/machinae.yml), merging into that a system-wide local configuration (/etc/machinae.local.yml) and finally a per-user local configuration (~/.machinae.yml). The system-wide configuration can also be located in the current working directory, can be set using the MACHINAE_CONFIG environment variable, or of course by using the-c or –config command line options. Configuration merging can be disabled by passing the –nomerge option, which will cause Machinae to only load the default system-wide configuration (or the one passed on the command line).

As an example of this, say you’d like to enable the Fortinet Category site, which is disabled by default. You could modify /etc/machinae.yml, but these changes would be overwritten by an update. Instead, you can put the following in either /etc/machinae.local.yml or ~/.machinae.yml:

fortinet_classify:

  default: true

Or, conversely, to disable a site, such as Virus Total pDNS:

vt_ip:

  default: false

vt_domain:

  default: false

Usage

Copyright (c) 2015 Hurricane Labs LLC

Source: https://github.com/HurricaneLabs/