intelmq v2.2.1 releases: collecting and processing security feeds
IntelMQ is a solution for IT security teams (CERTs, CSIRTs, abuse departments,…) for collecting and processing security feeds (such as log files) using a message queuing protocol. It’s a community-driven initiative called IHAP (Incident Handling Automation Project) which was conceptually designed by European CERTs/CSIRTs during several InfoSec events. Its main goal is to give to incident responders an easy way to collect & process threat intelligence thus improving the incident handling processes of CERTs.
IntelMQ’s design was influenced by AbuseHelper however it was re-written from scratch and aims at:
- Reducing the complexity of system administration
- Reducing the complexity of writing new bots for new data feeds
- Reducing the probability of events lost in all process with persistence functionality (even system crash)
- Use and improve the existing Data Harmonization Ontology
- Use JSON format for all messages
- Integration of the existing tools (AbuseHelper, CIF)
- Provide easy way to store data into Log Collectors like ElasticSearch, Splunk, databases (such as PostgreSQL)
- Provide easy way to create your own black-lists
- Provide easy communication with other systems via HTTP RESTFUL API
It follows the following basic meta-guidelines:
- Don’t break simplicity – KISS
- Keep it open source – forever
- Strive for perfection while keeping a deadline
- Reduce complexity/avoid feature bloat
- Embrace unit testing
- Code readability: test with inexperienced programmers
- Communicate clearly
- Add upgrade function for changed configuration of the feed “Abuse.ch URLHaus” (#1571, PR#1572 by Filip Pokorný).
- Add upgrade function for removal of HPHosts Hosts file feed and
- For IP Addresses, explicitly reject IPv6 addresses with scope ID (due to changed behavior in Python 3.9, #1550).
- Ignore line length (E501) in code-style checks altogether.
intelmq.bots.collectors.misp: Fix access to actual MISP object (PR#1548 by Tomas Bellus @tomas321)
intelmq.bots.collectors.stomp: Remove empty
intelmq.bots.parser.anubisnetworks.parser: Ignore “TestSinkholingLoss” events, these are not intended to be sent out at all.
intelmq.bots.parsers.generic.parser_csv: Allow values of type dictionary for parameter
intelmq.bots.parsers.hphosts: Removed, feed is unavailable (#1559).
intelmq.bots.parsers.cymru.parser_cap_program: Add support for comment “username” for “scanner” category.
intelmq.bots.parsers.malwareurl.parser: Check for valid FQDN and IP address in URL and IP address columns (PR#1585 by Marius Urkis).
intelmq.bots.experts.maxmind_geoip: On Python < 3.6, require maxminddb < 2, as that version does no longer support Python 3.5.
intelmq.bot.outputs.udp: Fix error handling on sending, had a bug itself.
- Update documentation of feed “Abuse.ch URLHaus” (#1571, PR#1572 by Filip Pokorný).
- Overhaul of all bots’ description fields (#1570).
- Overhaul pipeline configuration section and explain named queues better (#1577).
intelmq.bin.intelmq_gen_docs: Format parameters of types lists with double quotes around values to produce conform JSON, ready to copy and paste the value into the IntelMQ Manager’s bot parameter form.
debug: In JSON mode, use dictionaries instead of lists.
PATHto the paths shown.
$PATHenvironment variable if executable cannot be found.
malware_name_mapping: Change MISP Threat Actors URL to new URL (branch master -> main) in download script.
Copyright (C) 2014