streamalert v3.1 releases: Serverless, Realtime Data Analysis Framework
StreamAlert is a serverless, real-time data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.
As partially outlined above, StreamAlert has some unique benefits:
- Serverless — StreamAlert utilizes AWS Lambda, which means you don’t have to manage, patch or harden any new servers
- Scalable — StreamAlert utilizes AWS Kinesis Streams, which will “scale from megabytes to terabytes per hour and from thousands to millions of PUT records per second”
- Automated — StreamAlert utilizes Terraform, which means infrastructure and supporting services are represented as code and deployed via automation
- Secure — StreamAlert uses secure transport (TLS), performs data analysis in a container/sandbox, segments data per your defined environments, and uses role-based access control (RBAC)
- Open Source — Anyone can use or contribute to StreamAlert
- Deployment is automated: simple, safe and repeatable for any AWS account
- Easily scalable from megabytes to terabytes per day
- Infrastructure maintenance is minimal, no devops expertise required
- Infrastructure security is a default, no security expertise required
- Supports data from different environments (ex: IT, PCI, Engineering)
- Supports data from different environment types (ex: Cloud, Datacenter, Office)
- Supports different types of data (ex: JSON, CSV, Key-Value, or Syslog)
- Supports different use-cases like security, infrastructure, compliance and more
StreamAlert utilizes the following services:
- AWS Kinesis Streams — Datastream; AWS Lambda polls this stream (stream-based model)
- AWS Kinesis Firehose — Loads streaming data into S3 long-term data storage
- AWS Lambda (Python) — Data analysis and alerting
- AWS SNS — Alert queue
- AWS S3 — Optional datasources, long-term data storage, & long-term alert storage
- AWS Cloudwatch — Infrastructure metrics
- AWS KMS — Encryption and decryption of application secrets
- AWS IAM — Role-based Access Control (RBAC)
The concept of “stateful” alerting has always been a gap that StreamAlert has failed to bridge. We’ve introduced a feature we’ve dubbed Scheduled Queries as a way to help bridge that gap. Users can now write and deploy Athena queries that will run on a user-defined schedule. The results of these queries are then fed data back into StreamAlert’s Rules Engine for further processing and alerting. See the documentation for more information on getting up and running with Scheduled Queries.
See also: #1209
Support has been added for sending alerts to AWS Simple Email Service (SES). This enables sending richly formatted emails to recipients, as opposed to the previous method of using AWS SNS for sending only very simple emails. A huge thanks to @jack1902 for contributing this!
Support has also been added for sending alerts to Microsoft Teams. A huge thanks (again!) to @jack1902 for contributing this!
The Publishers testing implementation has been updated to support configuring tests for publishers directly within a test event file. For more information on how to add tests for Publishers, see the documentation.
See also: #1185
One of our biggest pain points in the StreamAlert ecosystem has been the speed of searches. This release adds support for Parquet as the storage format of data sent to S3 for historical data retention, and we’re already seeing vast improvements in comparison to JSON. In addition to this, Athena tables are also now created and managed via Terraform, removing the need for users to reason about them during deployment time.
See also: #1202
In addition to the updates to integration tests made as part of #1181, a larger update to the framework has migrated tests out of the
tests/integrationdirectory. Integration test files for rules should now live beside the rule being tested. The documentation for tests includes more details.
Thanks to @jack1902 for adding two new rules related to AWS Config!
Thanks to @chunyong-lin for open-sourcing a rule to alert on SSH login activity captured by osquery.
To view the complete list of all of the bugs fixed in v3.1.0, including many not mentioned above, see here.
To view the complete list of all changes included in v3.1.0, see here.