intelmq v2.3 RC1 releases: collecting and processing security feeds
IntelMQ is a solution for IT security teams (CERTs, CSIRTs, abuse departments,…) for collecting and processing security feeds (such as log files) using a message queuing protocol. It’s a community-driven initiative called IHAP (Incident Handling Automation Project) which was conceptually designed by European CERTs/CSIRTs during several InfoSec events. Its main goal is to give to incident responders an easy way to collect & process threat intelligence thus improving the incident handling processes of CERTs.
IntelMQ’s design was influenced by AbuseHelper however it was re-written from scratch and aims at:
- Reducing the complexity of system administration
- Reducing the complexity of writing new bots for new data feeds
- Reducing the probability of events lost in all process with persistence functionality (even system crash)
- Use and improve the existing Data Harmonization Ontology
- Use JSON format for all messages
- Integration of the existing tools (AbuseHelper, CIF)
- Provide easy way to store data into Log Collectors like ElasticSearch, Splunk, databases (such as PostgreSQL)
- Provide easy way to create your own black-lists
- Provide easy communication with other systems via HTTP RESTFUL API
It follows the following basic meta-guidelines:
- Don’t break simplicity – KISS
- Keep it open source – forever
- Strive for perfection while keeping a deadline
- Reduce complexity/avoid feature bloat
- Embrace unit testing
- Code readability: test with inexperienced programmers
- Communicate clearly
lineparameter optional, as it is not needed for this method.
Bot.argparser: Added class method
argparse.ArgumentParser) for easy command line arguments parsing.
- Runtime configuration does not necessarily need a parameter entry for each block. Previously an at least empty block was required (PR#1604 by Filip Pokorný).
- Allow setting the pipeline host and the Redis cache host by environment variables for docker usage (PR#1669 by Sebastian Waldbauer).
- Add upgrade function for removal of HPHosts Hosts file feed and
PipelineError: Remove unused code to format exceptions.
create_request_session_from_bot: Changed bot argument to optional, uses defaults.conf as fallback, renamed to
create_request_session_from_botwill be removed in version 3.0.0.
RotatingFileHandlerfor allow log file rotation without external tools (PR#1637 by Vasek Bruzek).
IPAddresstype sanitation now accepts integer IP addresses and converts them to the string representation.
DateTime.parse_utc_isoformat: Add parameter
datetimeobject instead of string ISO format.
utc_isoformatformat, it pointed to a string and not a function, causing an exception when used.
DateTime.from_timestamp: Ensure that time zone information (
+00:00) is always present.
DateTime.__parsenow handles OverflowError exceptions from the dateutil library, happens for large numbers (e.g. telehpone numbers).
- Added upgrade function
- Add bot name to the resulting feed documentation (PR#1617 by Birger Schacht).
- Merged into
Copyright (C) 2014