SysFlow?

The SysFlow Telemetry Pipeline is a framework for monitoring cloud workloads and for creating performance and security analytics. The goal of this project is to build all the plumbing required for system telemetry so that users can focus on writing and sharing analytics on a scalable, common open-source platform. The backbone of the telemetry pipeline is a new data format called SysFlow, which lifts raw system event information into an abstraction that describes process behaviors, and their relationships with containers, files, and networks. This object-relational format is highly compact, yet it provides broad visibility into container clouds. We have also built several APIs that allow users to process SysFlow with their favorite toolkits. Learn more about SysFlow in the SysFlow specification document.

The SysFlow framework consists of the following sub-projects:

• sf-apis provides the SysFlow schema and programmatic APIs in go, python, and C++.
• sf-collector monitors and collects system call and event information from hosts and exports them in the SysFlow format using Apache Avro object serialization.
• sf-processor provides a performance-optimized policy engine for processing, enriching, filtering SysFlow events, generating alerts, and exporting the processed data to various targets.
• sf-exporter exports SysFlow traces to S3-compliant storage systems for archival purposes.
• sf-deployments contains deployment packages for SysFlow, including Docker, Helm, and OpenShift.
• sysflow is the documentation repository and issue tracker for the SysFlow framework.

SysFlow is an open specification for system event-level telemetry. The main goal of SysFlow is to create a standard and extensible data format for both security and performance analytics for compute workloads. An open standard will enable researchers and practitioners to more easily work on a common data format, and focus on analytics using open-source software.

The primary objective of SysFlow is to lift raw system call data into more semantic process behaviors which promote significant data reductions for longer-term forensic storage of data which is crucial for security analyzes. Through an object-relational model of entities, events, and flows, we enable SysFlow users to configure the desired granularity of data collection and filtering in order to facilitate most types of analysis in big data frameworks.

Install & Use

© Copyright 2021 IBM – Maintained by the SysFlow team Revision .1584c73a