loglizer

A Python toolkit for anomaly detection via log analysis

Loglizer is an open-source python tool for automatic log-based anomaly detection with machine learning techniques. In this project, six popular anomaly detection methods are implemented and evaluated on two public datasets. More details (e.g., experimental results, findings) can be found in our paper.

Anomaly detection plays an important role in the management of modern large-scale distributed systems. Logs, which record system runtime information, are widely used for anomaly detection. Traditionally, developers (or operators) often inspect the logs manually with keyword search and rule matching. The increasing scale and complexity of modern systems, however, make the volume of logs explode, which renders the infeasibility of manual inspection. To reduce manual effort, many anomaly detection methods based on automated log analysis are proposed.   In our paper, we provide a detailed review and evaluation of six state-of-the-art log-based anomaly detection methods, including three supervised methods and three unsupervised methods, and also release an open-source toolkit allowing ease of reuse. These methods have been evaluated on two publicly-available production log datasets.

Overview of the framework

1. Log collection: Logs are generated and collected by system and software during running, which includes distributed systems (e.g., Spark, Hadoop), standalone systems (e.g., Windows, Mac OS) and software (e.g., Zookeeper).
2. Log Parsing: Raw Logs contain too much runtime information (e.g., IP address, file name). This variable information is often removed after log parsing as they are useless for debugging. After parsing, raw logs become log events, which are an abstraction of raw logs. Details are given in our previous work: Logparser
3. Feature Extraction: Logs are grouped into log sequences via Task ID or time, and these log sequences are vectorized and weighted.
4. Anomaly Detection: Some machine learning models are trained and applied to detect anomalies.

The framework is illustrated as follows:

In our toolbox, we mainly focus on Feature Extraction and Anomaly Detection, while Log Collection and Log Parsing are out of the scope of this project. To be more specific, the input is the parsed log events, and the output is whether it is an anomaly for each log sequence.

Anomaly detection methods

• Supervised Anomaly Detection:
  1. Logistic Regression:
  Paper: Fingerprinting the Datacenter: Automated Classification of Performance Crises
  Affiliations: UC Berkeley, Cornell, Microsoft
  2. Decision Tree:
  Paper: Failure Diagnosis Using Decision Trees
  Affiliations: UC Berkeley, eBay
  3. SVM:
  Paper: Failure Prediction in IBM BlueGeneL Event Logs
  Affiliations: Rutgers University, IBM
• Unsupervised Anomaly Detection:
  1. Log Clustering:
  Paper: Log Clustering based Problem Identification for Online Service Systems
  Affiliations: Microsoft Research
  2. PCA:
  Paper: Large-Scale System Problems Detection by Mining Console Logs
  Affiliations: UC Berkeley
  3. Invariants Mining:
  Paper: Mining Invariants from Console Logs for System Problem Detection
  Affiliations: Microsoft Research

Download

Copyright (c) 2016 CUHK-CSE