AIL framework v1.7 released: Analysis Information Leak framework

AIL framework – Framework for Analysis of Information Leaks

AIL is a modular framework to analyze potential information leaks from unstructured data sources like pastes from Pastebin or similar services or unstructured data streams. AIL framework is flexible and can be extended to support other functionalities to mine sensitive information.

Dashboard

Trending charts

Trending-WebTrending-Modules

Browsing

Browse-Pastes

Features

  • Modular architecture to handle streams of unstructured or structured information
  • Default support for external ZMQ feeds, such as provided by CIRCL or other providers
  • Multiple feed support
  • Each module can process and reprocess the information already processed by AIL
  • Detecting and extracting URLs including their geographical location (e.g. IP address location)
  • Extracting and validating potential leak of credit cards numbers, credentials, …
  • Extracting and validating email addresses leaked including DNS MX validation
  • Module for extracting Tor .onion addresses (to be further processed for analysis)
  • Keep tracks of duplicates (and diffing between each duplicate found)
  • Extracting and validating potential hostnames (e.g. to feed Passive DNS systems)
  • A full-text indexer module to index unstructured information
  • Statistics on modules and web
  • Real-time modules manager in terminal
  • Global sentiment analysis for each providers based on nltk vader module
  • Terms, Set of terms and Regex tracking and occurrence
  • Many more modules for extracting phone numbers, credentials and others
  • Alerting to MISP to share found leaks within a threat intelligence platform using MISP standard
  • Detect and decode Base64 and store files
  • Detect Amazon AWS and Google API keys
  • Detect Bitcoin address and Bitcoin private keys
  • Detect private keys and certificate
  • Tagging system with MISP Galaxy and MISP Taxonomies tags
  • UI paste submission
  • Create events on MISP and cases on The Hive
  • Automatic paste export at detection on MISP (events) and The Hive (alerts) on selected tags

Changelog v1.7

Changes

  • [correlation] clean files. [Terrtia]
  • [update v1.7] update thirdparty. [Terrtia]
  • [correlation] add cryptocurrency + refractor correlation. [Terrtia]
  • [Bitcoin] map cryptocurrency: bitcoin (DB pivot) [Terrtia]
  • [update v1.7] add update scripts. [Terrtia]
  • [pgpdump] reprocess tagged items + fix pgpdump. [Terrtia]
  • [Update] force update order. [Terrtia]
  • [PgpDump] fix graph + add new tags: pgp-signature pgp-public-key-block
    • avoid keys injection in pgp user_id. [Terrtia]
  • [decoded UI] add PgpDump UI + fix hashdecoded js. [Terrtia]
  • [decoded items] bootstrap 4 migration. [Terrtia]
  • [PgpDump] add PgpDump backend TODO: UI. [Terrtia]
  • [crawler] manual/auto crawler: always save screenshots. [Terrtia]
  • [crawler] manual/auto crawler: always save screenshots. [Terrtia]

Fix

  • [correlation] fix endpoint. [Terrtia]

Other

  • Update README.md. [Thirion Aurélien]
  • Merge branch ‘master’ of https://github.com/CIRCL/AIL-framework.
    [Terrtia]
  • Merge pull request #349 from kovacsbalu/fix-paste-encoding. [Thirion
    Aurélien]

    Fix #314

  • Use default encoding error from redis. [kovacsbalu]
  • Fix #314 Replace char on redis encoding error. Try to use local file
    on other error. [kovacsbalu]
  • Merge pull request #350 from kovacsbalu/fix-crawler-rotation. [Thirion
    Aurélien]

    fix: [crawler] rotation

  • Hopp, single quote 🙂 [kovacsbalu]
  • Fix crawler rotation. [kovacsbalu]

    Before this, crawler processed prioritized onions and after all starts prioritized regular.

Installation

Type these command lines for a fully automated installation and start AIL framework

git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
./installing_deps.sh
cd var/www/
./update_thirdparty.sh
cd ~/AIL-framework/
. ./AILENV/bin/activate
cd bin/
./LAUNCH.sh

 

 

 

Starting AIL web interface

To start the web interface, you first need to fetch the required Javascript/CSS files:

cd $AILENV
cd var/www/
bash update_thirdparty.sh

 

 

 

 

 

 

 

and then you can start the web interface python script:

cd $AILENV
cd var/www/
Flask_server.py

 

 

 

 

 

 

 

Eventually, you can browse the status of the AIL framework website at the following URL:

``http://localhost:7000/``

 

 

 

 

Copyright (C) 2014 Jules Debra
Copyright (C) 2014-2018 CIRCL – Computer Incident Response Center Luxembourg (c/o smile, security made in Lëtzebuerg, Groupement d’Intérêt Economique)
Copyright (c) 2014-2018 Raphaël Vinot
Copyright (c) 2014-2018 Alexandre Dulaunoy
Copyright (c) 2016-2018 Sami Mokaddem
Copyright (c) 2018 Thirion Aurélien

Source: https://github.com/CIRCL/

Share