logdata-anomaly-miner v2.5.1 releases: parses log data and allows to define analysis pipelines for anomaly detection
logdata-anomaly-miner
This tool parses log data and allows to define analysis pipelines for anomaly detection. It was designed to run the analysis with limited resources and the lowest possible permissions to make it suitable for production server use.
The logdata-anomaly-miner can be configured in two different formats: yaml and python. The preferred format is yaml and the default configuration file for it is /etc/aminer/config.yaml. The python-format can be configured in /etc/aminer/config.py and offers advanced possibilities to configure the logdata-anomaly-miner. However, this is only recommended for experts, as no errors are caught in the python configuration, which can make debugging very difficult. For both formats there are template configurations in /etc/aminer/template_config.yaml and /etc/aminer/template_config.py.
The basic structure of the logdata-anomaly-miner is illustrated in the following diagram:
Analysis Pipeline
The core component of the logdata-anomaly-miner is the “analysis pipeline”. It consists of the parts INPUT, ANALYSIS, and OUTPUT.
Changelog v2.5.1
Bugfixes:
- EFD: Fixed problem that appears with empty windows
- Fixed index out of range if matches are empty in JsonModelElement array.
- EFD: Fixed problem that appears with empty windows
- EFD: Enabled immediate detection without training, if both limits are set
- EFD: Fixed bug related to auto_include_flag
- Remove spaces in aminer logo
- ParserCounter: Fixed do_timer
- Fixed code to allow the usage of AtomFilterMatchAction in yaml configs
- Fixed JsonModelElement when json object is null
- Fix incorrect message of charset detector
- Fix match list handling for json objects
- Fix incorrect message of charset detector
Changes:
- Added nullable functionality to JsonModelElements
- Added include-directive to supervisord.conf
- ETD: Output warning when count first exceeds range
- EFD: Added option to output anomaly when the count first exceeds the range
- VTD: Added variable type ‘range’
- EFD: Added the function reset_counter
- EFD: Added option to set the lower and upper limit of the range interval
- Enhance EFD to consider multiple time windows
- VTD: Changed the value of parameter num_updates_until_var_reduction to track all variables from False to 0.
- PAD: Used the binom_test of the scipy package as test if the model should be reinitialized if too few anomalies occur than are expected
- Add ParsedLogAtom to aminer parser to ensure compatibility with lower versions
- Added script to add build-id to the version-string
- Support for installations from source in install-script
- Fixed and stadardize the persistence time of various detectors
- Refactoring
- Improve performance
- Improve output handling
- Improved testing
Install & Use
© Copyright 2021, Florian Skopik, Markus Wurzenberger, Max Landauer, Roman Fiedler, Wolfgang Hotwagner, Ernst Leierzopf, Georg Hoeld.
