streamalert v3.5 releases: Serverless, Realtime Data Analysis Framework

by do son · Published · Updated

streamalert

StreamAlert is a serverless, real-time data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.

Benefits

As partially outlined above, StreamAlert has some unique benefits:

  • Serverless — StreamAlert utilizes AWS Lambda, which means you don’t have to manage, patch or harden any new servers
  • Scalable — StreamAlert utilizes AWS Kinesis Streams, which will “scale from megabytes to terabytes per hour and from thousands to millions of PUT records per second”
  • Automated — StreamAlert utilizes Terraform, which means infrastructure and supporting services are represented as code and deployed via automation
  • Secure — StreamAlert uses secure transport (TLS), performs data analysis in a container/sandbox, segments data per your defined environments, and uses role-based access control (RBAC)
  • Open Source — Anyone can use or contribute to StreamAlert

High-level

  • Deployment is automated: simple, safe and repeatable for any AWS account
  • Easily scalable from megabytes to terabytes per day
  • Infrastructure maintenance is minimal, no devops expertise required
  • Infrastructure security is a default, no security expertise required
  • Supports data from different environments (ex: IT, PCI, Engineering)
  • Supports data from different environment types (ex: Cloud, Datacenter, Office)
  • Supports different types of data (ex: JSON, CSV, Key-Value, or Syslog)
  • Supports different use-cases like security, infrastructure, compliance and more

Architecture

StreamAlert utilizes the following services:

  • AWS Kinesis Streams — Datastream; AWS Lambda polls this stream (stream-based model)
  • AWS Kinesis Firehose — Loads streaming data into S3 long-term data storage
  • AWS Lambda (Python) — Data analysis and alerting
  • AWS SNS — Alert queue
  • AWS S3 — Optional datasources, long-term data storage, & long-term alert storage
  • AWS Cloudwatch — Infrastructure metrics
  • AWS KMS — Encryption and decryption of application secrets
  • AWS IAM — Role-based Access Control (RBAC)

Tutorial

Copyright 2017 Airbnb

Source: https://github.com/airbnb/

Tags: StreamAlert