magpie: Open Raven’s Open Source Cloud Security Framework

by do son · Published August 11, 2021 · Updated August 11, 2021

Magpie

Open Raven’s Open Source Cloud Security Framework

What is Magpie?

Magpie is a free, open-source framework and a collection of community-developed plugins that can be used to build complete end-to-end security tools such as a CSPM or Cloud Security Posture Manager. The project was originally created and is maintained by Open Raven. We build commercial cloud-native data security tools and in doing so have learned a great deal about how to discover AWS assets and their security settings at scale.

We also heard that many people were frustrated with their existing security tools that couldn’t be extended and couldn’t work well with their other systems, so decided to create this Magpie framework and refactor and sync our core AWS commercial discovery code as the first plugin.

We plan to actively contribute additional modules to make Magpie a credible free open source alternative to commercial CSPM’s and welcome the community to join us in adding to the framework and building plugins.

Magpie also contains Open Raven’s DMAP technology, which allows users to enumerate and identify non-native services running on EC2 instances using a combination of port fingerprinting (think Nmap’s OS fingerprinting but on the application layer instead of the transport layer) and a little machine learning (decision trees).

Overview

Magpie Architecture

Magpie relies on plugins for all its integration capabilities. They are the core of the framework and key to integration with both cloud providers and downstream processing and storage.

Magpie is essentially a series of layers separated by FIFOs.

Depending on the configuration, these FIFOs are either 1) Java queues (in the default configuration) or 2) Kafka queues. Using Kafka queues allows Magpie to run in a distributed and highly scalable fashion where each layer may exist on separate compute instances.