leakScraper: process and visualize huge text files containing credentials

LeakScraper is an efficient set of tools to process and visualize huge text files containing credentials. These tools are designed to help pentesters/redteamers doing OSINT, credentials gathering and credentials stuffing attacks.

The different tools

LeakScraper is split into three parts :

• leakStandardizer: A tool to standardize leaks you got from some legit place on the internet. It takes in input a file containing credentials following some weird format, containing non-ascii characters or empty lines, lines containing invalid emails or no password. It will produce, with your help (using the regular expression), an easily greppable file using the following format : email:hash:plain (“plain” for “plain text password”).
• leakImporter: A tool to import a standardized leak file into a mongodb database. It will take care of extracting data from the file, putting it into a mysql comprehensive format, creating/managing indexes …
• leakScraper: A tool and an interface to excavate data from the database and display it nicely.

Postulates

• The covered usecase is the following: searching credentials belonging to a specific organization/company/structure. This is done by searching credentials associated with an email belonging to the organization in question. Eg: Searching credentials belonging to Microsoft is done by searching credentials associated with accounts registered with an email ending with “@microsoft.com”. It is the only usecase covered and it means a lot in terms of technical choices (database indexes and data representation in general).
• Leaks can weight several gigabytes. It means that each process (standardizing, imports and researchers) are using in-place algorithms in terms of memory. You can know beforehand how much memory these tools will use to process a specific file, and it will never exhaust your computer’s resources (unless you have a very old one).
• Processing huge files and working with a lot of data takes time. It’s important IMO to have visual/real-time feedback to know how much time processing/importing a file will take. It’s important to know if you just started a 7 hours long process or a 1,200 years long one.

Installation

• First things first: have a working mongodb server.
• git clone -b mongodb https://github.com/Acceis/leakScraper
  cd leakScraper
  sudo ./leakScraper/install.sh

Usage

Copyright (C) 2018 almandin 

Source: https://github.com/Acceis/