karton v5.0 releases: Distributed malware processing framework
Distributed malware processing framework based on Python, Redis, and MinIO.
Karton is a robust framework for creating flexible and lightweight malware analysis backends. It can be used to connect malware* analysis systems into a robust pipeline with very little effort.
We’ve been in the automation business for a long time. We’re dealing with more and more threats, and we have to automate everything to keep up with incidents. Because of this, we often end up with many scripts stuck together with
duck duct tape and WD-40. These scripts are written by analysts in the heat of the moment, fragile and ugly – but they work, and produce intel that must be stored, processed further, sent to other systems, or shared with other organisations.
We needed a way to take our PoC scripts and easily insert them into our analysis pipeline. We also wanted to monitor their execution, centralise logging, improve robustness, reduce development inertia… For this exact purpose, we created Karton.
* while Karton was designed with malware analysis in mind, it works nicely in every microservice-oriented project.
Some Karton systems are universal and useful to everyone. We decided to share them with the community.
This repository. It contains the karton.system service – main service, responsible for dispatching tasks within the system. It also contains the karton.core module, that is used as a library by other systems.
A small Flask dashboard for task and queue management and monitoring.
The “router”. It recognises samples/files and produces various task types depending on the file format. Thanks to this, other systems may only listen for tasks with a specific format (for example, only zip archives).
Generic archive unpacker. Archives uploaded into the system will be extracted, and every file will be processed individually.
Malware extractor. It uses Yara rules and Python modules to extract static configuration from malware samples and analyses. It’s a fishing rod, not a fish – we don’t share the modules themselves. But it’s easy to write your own!
A very important part of the pipeline. Reporter submits all files, tags, comments and other intel produced during the analysis to MWDB. If you don’t use MWDB yet or just prefer other backends, it’s easy to write your own reporter.
Automatically runs Yara rules on all files in the pipeline, and tags samples appropriately. Rules not included ;).
Karton system that decodes files encoded with common methods, like hex, base64, etc. (You wouldn’t believe how common it is).
A small wrapper around AutoIt-Ripper that extracts embedded AutoIt scripts and resources from compiled AutoIt executables.
Automated black-box malware analysis system with DRAKVUF engine under the hood, which does not require an agent on guest OS.
This major release is focused mainly on fixing reliability and extensibility issues noticed during development of various Karton services. As always, we have tried not to make serious breaking changes, so 5.x.x Karton services are able to communicate with 4.x.x and you can do a rolling upgrade. If you want to perform an upgrade: it’s recommended to start with upgrading
karton-system to the latest version.
In this release we’ve also included few long-awaited features.
- Karton Core uses Boto3 S3 API client instead of Minio-Py.
karton.inishould be changed to new configuration scheme (
[minio]section is still supported but deprecated.
- [minio] + [s3] access_key = karton-test-access secret_key = karton-test-key - address = localhost:9000 + address = http://localhost:9000 bucket = karton - secure = 0
Configuses plain dict internally instead of
ConfigParserobject and provides slightly different interface for configuration item access. (#176)
Consumer.processmethod must accept task argument (currently processed task). Variant with no arguments is no longer supported. (dropped support as a part of #175)
- Enforced correct
Karton().main()will throw TypeError. (#182)
New features and improvements:
- Support for filter wildcards and exclusions (#179)
- Support for resources nested in lists and dicts (#186, thanks @mak for suggestions to the implementation #189)
- Better support for configuration/CLI arguments extensions (#176)
- LocalResource supports file-like object as contents source (#187)
- Support for
usernamearguments in Redis configuration (#184)
- Task timeout can be set to recover from hang during task processing (#175)
- Karton System GC timeouts are now configurable (#161, thanks @rakovskij-stanislav!)
- Added configurable socket timeout for Redis connection to improve handling of network issues (#184)
- Redis client name is no longer lost after reconnection, so count of Karton service replicas will be not affected after network issues (#173)
- Prevented double logging in case of
logging.basicConfigor construction of multiple Karton service objects (#172)
- Fixed resource handling in test utils (#180)
- Fixed SIGINT/SIGTERM handling (#181)
- Fixed method name typo in Karton System (#162, thanks @MaitreRenard!)
Copyright (c) 2020, CERT Polska
All rights reserved.