duplicut: Remove duplicates from MASSIVE wordlist

by do son · April 24, 2021

Modern password wordlist creation usually implies concatenating multiple data sources.

Ideally, most probable passwords should stand at the start of the wordlist, so most common passwords are cracked instantly.

With existing dedupe tools you are forced to choose if you prefer to preserve the order OR handle massive wordlists.

Unfortunately, wordlist creation requires both:

An uint64 is enough to index lines in hashmap, by packing size info within pointer’s extra bits:

If the whole file can’t fit in memory, it is split into virtual chunks, then each one is tested against the next chunks.

So complexity is equal to th triangle number:

git clone https://github.com/nil0x42/duplicut
cd duplicut/ && make

Features:
- Handle massive wordlists, even those whose size exceeds available RAM
- Filter lines by max length (-l option)
- Can remove lines containing non-printable ASCII chars (-p option)
- Press any key to show program status at runtime.
Implementation:
- Written in pure C code, designed to be fast
- Compressed hashmap items on 64-bit platforms
- Multithreading support
- [TODO]: Use huge memory pages to increase performance
Limitations:
- Any line longer than 255 chars is ignored
- Heavily tested on Linux x64, mostly untested on other platforms.