collectionparser/README.md

collectionparser

Command line tools to parse, deduplicate, and query multi-billion password collections (Collections 1-5, AntiPublic, Breach Compilation).

Responsible use. This project ships no data. Use it only on datasets you own or are authorized to process, for security research and defense. Do not commit datasets, hashes, or cracked credentials.

Processing produces a deduped a/b/c bucket tree that answers exact lookups in O(1) and wildcard queries with a parallel ripgrep scan:

$ collectionparser query --dataset /path/to/data --pattern [email protected]
[email protected]:toto123
$ collectionparser query --dataset /path/to/data --pattern '*@gmail.com'

Install

Requires Python 3.11+ and ripgrep (rg) for queries.

python -m venv venv && source venv/bin/activate
pip install -e .

Memory

Every command streams input and writes incrementally. Peak memory does not grow with file size: a single 100 GB+ file is parsed with a small constant footprint. sort shells out to the OS sort -u, which spills to disk.

• crack builds the hash index once and shares it with the workers by forking (copy-on-write) — one copy, not one per worker (verified: ~450 MB total for a 1M-hash index across 4 workers).
• query streams wildcard matches line by line from ripgrep, so a broad pattern over a huge dataset stays bounded.

Measured on real Collections 2-5 data (19.94 GB, three files): 160 MB peak RSS for the whole process tree, versus 15.4 GB for the original parser. Min RAM available stayed at 15.3 GB, swap unchanged.

Build the query dataset

build is the resumable, disk-bounded pipeline. It parses the source in batches, sorts+dedups each batch on a scratch disk, splits it into the bucket tree, then dedups each bucket. Peak disk is source + tree, so it fits where a global sort would not.

# --scratch is a fast disk for per-batch files, sort temp, and build state.
# --batch_gb caps each batch (peak disk = source + tree).
collectionparser build \
  --source /path/to/extracted \
  --dataset /path/to/dataset \
  --scratch /path/to/scratch \
  --batch_gb 120

Pause / resume: stop it any time (Ctrl-C or systemctl stop) and rerun the same command. Finished batches are recorded in build_state.json, and parsed files within the current batch in progress_<n>.txt; only unparsed files are re-parsed (resume is per file, not per line). The sort+split step of an interrupted batch is redone (duplicates are removed by the final dedup), so nothing is lost.

Progress: one machine-readable command.

collectionparser status --scratch /path/to/scratch            # one-line JSON
collectionparser status --scratch /path/to/scratch --pretty

State and stats live in --scratch: build_state.json, stats.md, and extract_failures.txt (archives that failed to extract, never silently skipped). sort temp is directed to --scratch via TMPDIR.

Archives (.rar/.zip/.7z/.tar.*/.gz/.bz2/.xz) are extracted automatically into --scratch for the duration of their batch and removed once parsed; the source is never modified. .zip/.tar.*/single-file .gz/.bz2/.xz use the Python standard library; .rar/.7z need 7z or unrar on PATH. Pass --no-extract to skip archives, or --no-dedup to skip the final per-bucket de-duplication.

Processing (manual)

collectionparser parse --path /path/to/extracted --success_file success.txt --failure_file failure.txt

rm -rf tmp && mkdir tmp  # sort needs room for temp files; /tmp is usually too small
cat success.txt | pv -cN cut | sort -T tmp -u -S 90% --parallel=12 | pv -cN cut > success_sorted.txt

collectionparser split --file success_sorted.txt --out data

Merging datasets

merge is destructive: back up the destination first.

cp -rf /path/to/collections1_data /path/to/big_dataset
collectionparser merge --src /path/to/collections2_5_data --dest /path/to/big_dataset

Query and export

# exact (O(1) bucket lookup) or wildcard (* and ?):
collectionparser query --dataset data --pattern 'john*'
collectionparser query --dataset data --pattern '*@gmail.com'

# JSON Lines for OpenSearch / bulk ingest:
collectionparser export --dataset data --out records.jsonl

Crack hashed passwords

parse/build keep hashed passwords (MD5/SHA1/SHA256) separate from plaintext ones instead of discarding them. build collects them in --scratch/hashes.txt; parse writes them to --hashes_file when given.

# Wordlist omitted -> rockyou is downloaded automatically; add the dataset's
# own plaintext passwords as extra candidates:
collectionparser crack \
  --hashes hashes.txt \
  --passwords-from dataset/ \
  --out cracked.txt
# Or bring your own wordlist(s):
collectionparser crack --hashes hashes.txt --wordlist words.txt --out cracked.txt

Pure standard library: hashlib digests with one process per CPU core, each attacking a strided slice of the wordlist, so memory stays bounded; candidate sources are merged through sort -u. Only unsalted MD5/SHA1/SHA256 are attackable — bcrypt is ignored. Feed cracked.txt back into the index with split/merge.

Measured performance

Single large-file test on Collections 2-5 (3 files, 19.94 GB, 3 workers):

metric	value
parse throughput	20.5 MB/s aggregate (6.8 MB/s per worker)
valid pairs	628,342,073
rejects	3,321,666
peak RSS	160 MB
sort -u	~125 MB/s
split (sorted input)	~43 MB/s

Split is ~12× slower on unsorted input (open-handle thrash); always sort first.

Commands

Run collectionparser --help or collectionparser <command> --help.

command	description
build	Batched, resumable: parse + dedup + split into the query tree.
status	Machine-readable build/dataset progress (JSON).
query	Exact or wildcard search over a dataset.
export	Export a dataset as JSON Lines.
crack	Dictionary-attack collected hashes with a wordlist.
parse	Parse an unstructured folder into success/failure/hashes files.
split	Convert a large sorted file into a query-friendly a/b/c folder.
sort	Sort and deduplicate every file in a query-friendly folder.
merge	Merge one query-friendly folder into another, then sort/dedup.

Credits

Derived from and inspired by philipperemy/3.7-billion-passwords-tools by Philippe Remy — the original parser and a/b/c index. This project rewrites it for memory safety, resumable batched builds, wildcard queries, hash collection/cracking and JSONL export.

Thanks, Philippe.

Development

pip install -e '.[dev]'
ruff format . && ruff check .   # style + lint
pytest                          # tests