collectionparser/README.md

collectionparser

Command line tools to parse, deduplicate, and query multi-billion password collections (Collections 1-5, AntiPublic, Breach Compilation).

Processing produces a deduped a/b/c bucket tree that answers exact lookups in O(1) and wildcard queries with a parallel ripgrep scan:

$ collectionparser query --dataset /path/to/data --pattern [email protected]
[email protected]:toto123
$ collectionparser query --dataset /path/to/data --pattern '*@gmail.com'

Install

Requires Python 3.11+ and ripgrep (rg) for queries.

python -m venv venv && source venv/bin/activate
pip install -e .

Memory

Every command streams input and writes incrementally. Peak memory does not grow with file size: a single 100 GB+ file is parsed with a small constant footprint. sort and clean shell out to the OS sort -u, which spills to disk.

Measured on real Collections 2-5 data (19.94 GB, three files): 160 MB peak RSS for the whole process tree, versus 15.4 GB for the original parser. Min RAM available stayed at 15.3 GB, swap unchanged.

Build the query dataset

build is the resumable, disk-bounded pipeline. It parses the source in batches, sorts+dedups each batch on a scratch disk, splits it into the bucket tree, then dedups each bucket. Peak disk is source + tree, so it fits where a global sort would not.

collectionparser build \
  --source /path/to/extracted \
  --dataset /path/to/dataset \
  --scratch /path/to/scratch \   # fast disk for batch files + sort temp
  --batch_gb 120

Pause / resume: stop it any time (Ctrl-C or systemctl stop) and rerun the same command. Finished batches are recorded in build_state.json, and parsed files within the current batch in progress_<n>.txt; only unparsed files are re-parsed. The sort+split step of an interrupted batch is redone (duplicates are removed by the final dedup), so nothing is lost. Stats are written to --scratch/stats.md.

Archives (.rar/.zip/.7z/.tar.*/.gz/.bz2/.xz) are extracted automatically into --scratch for the duration of their batch and removed once parsed; the source is never modified. .zip/.tar.*/single-file .gz/.bz2/.xz use the Python standard library; .rar/.7z need 7z or unrar on PATH (use --no-extract to skip archives entirely).

Processing (manual)

collectionparser parse --path /path/to/extracted --success_file success.txt --failure_file failure.txt

rm -rf tmp && mkdir tmp  # sort needs room for temp files; /tmp is usually too small
cat success.txt | pv -cN cut | sort -T tmp -u -S 90% --parallel=12 | pv -cN cut > success_sorted.txt

collectionparser split --file success_sorted.txt --out data

Merging datasets

merge is destructive: back up the destination first.

cp -rf /path/to/collections1_data /path/to/big_dataset
collectionparser merge --src /path/to/collections2_5_data --dest /path/to/big_dataset

Query and export

# exact (O(1) bucket lookup) or wildcard (* and ?):
collectionparser query --dataset data --pattern 'john*'
collectionparser query --dataset data --pattern '*@gmail.com'

# JSON Lines for OpenSearch / bulk ingest:
collectionparser export --dataset data --out records.jsonl

Measured performance

Single large-file test on Collections 2-5 (3 files, 19.94 GB, 3 workers):

metric	value
parse throughput	20.5 MB/s aggregate (6.8 MB/s per worker)
valid pairs	628,342,073
rejects	3,321,666
peak RSS	160 MB
sort -u	~125 MB/s
split (sorted input)	~43 MB/s

Split is ~12× slower on unsorted input (open-handle thrash); always sort first.

Commands

Run collectionparser --help or collectionparser <command> --help.

command	description
build	Batched, resumable: parse + dedup + split into the query tree.
query	Exact or wildcard search over a dataset.
export	Export a dataset as JSON Lines.
parse	Parse an unstructured folder into success/failure files.
split	Convert a large sorted file into a query-friendly a/b/c folder.
chunk	Split large TXT files into smaller ones.
sort	Sort and deduplicate every file in a query-friendly folder.
clean	Move misplaced records to the right bucket and sort.
merge	Merge one query-friendly folder into another.
test	Look up passwords for the emails in a file.
evaluate	Measure hit rate of one dataset against another.
filter-lines	Keep lines whose length is within a range.