machinae v1.4.7 releases: Machinae Security Intelligence Collector

Machinae Security Intelligence Collector

Machinae is a tool for collecting intelligence from public sites/feeds about various security-related pieces of data: IP addresses, domain names, URLs, email addresses, file hashes and SSL fingerprints. It was inspired by Automater, another excellent tool for collecting information. The Machinae project was born from wishing to improve Automater in 4 areas:

  1. Codebase – Bring Automater to python3 compatibility while making the code more pythonic
  2. Configuration – Use a more human-readable configuration format (YAML)
  3. Inputs – Support JSON parsing out-of-the-box without the need to write regular expressions, but still support regex scraping when needed
  4. Outputs – Support additional output types, including JSON, while making extraneous output optional

Out-of-the-Box Data Sources

Machinae comes with out-of-the-box support for the following data sources:

  • IPVoid
  • URLVoid
  • URL Unshortener (http://www.toolsvoid.com/unshorten-url)
  • Malc0de
  • SANS
  • FreeGeoIP (freegeoip.io)
  • Fortinet Category
  • VirusTotal pDNS (via web scrape – commented out)
  • VirusTotal pDNS (via JSON API)
  • VirusTotal URL Report (via JSON API)
  • VirusTotal File Report (via JSON API)
  • Reputation Authority
  • ThreatExpert
  • VxVault
  • ProjectHoneypot
  • McAfee Threat Intelligence
  • StopForumSpam
  • Cymru MHR
  • ICSI Certificate Notary
  • TotalHash (disabled by default)
  • DomainTools Parsed Whois (Requires API key)
  • DomainTools Reverse Whois (Requires API key)
  • DomainTools Reputation
  • IP WHOIS (Using RIR REST interfaces)

With additional data sources on the way.

Changelog v1.4.7

  • Resolves #25
  • JSON output now includes observable type.

Installation

pip3 install machinae

Configuration File

Machinae supports a simple configuration merging system to allow you to make adjustments to the configuration without modifying the machinae.yml we provide you, making configuration updates a snap. This is done by finding a system-wide default configuration (default /etc/machinae.yml), merging into that a system-wide local configuration (/etc/machinae.local.yml) and finally a per-user local configuration (~/.machinae.yml). The system-wide configuration can also be located in the current working directory, can be set using the MACHINAE_CONFIG environment variable, or of course by using the-c or –config command line options. Configuration merging can be disabled by passing the –nomerge option, which will cause Machinae to only load the default system-wide configuration (or the one passed on the command line).

As an example of this, say you’d like to enable the Fortinet Category site, which is disabled by default. You could modify /etc/machinae.yml, but these changes would be overwritten by an update. Instead, you can put the following in either /etc/machinae.local.yml or ~/.machinae.yml:

fortinet_classify:
  default: true

 

 

Or, conversely, to disable a site, such as Virus Total pDNS:

vt_ip:
  default: false
vt_domain:
  default: false

 

 

Usage

machinae

Copyright (c) 2015 Hurricane Labs LLC

Source: https://github.com/HurricaneLabs/

Share