$ collectionparser query --dataset /path/to/data --pattern [email protected]
[email protected]:toto123
$ collectionparser query --dataset /path/to/data --pattern '*@gmail.com'

Install

Requires Python 3.11+ and ripgrep (rg) for queries.

python -m venv venv && source venv/bin/activate
pip install -e .

Memory

Every command streams input and writes incrementally. Peak memory does not grow with file size: a single 100 GB+ file is parsed with a small constant footprint. sort and clean shell out to the OS sort -u, which spills to disk.

Measured on real Collections 2-5 data (19.94 GB, three files): 160 MB peak RSS for the whole process tree, versus 15.4 GB for the original parser. Min RAM available stayed at 15.3 GB, swap unchanged.

Build the query dataset

build is the resumable, disk-bounded pipeline. It parses the source in batches, sorts+dedups each batch on a scratch disk, splits it into the bucket tree, then dedups each bucket. Peak disk is source + tree, so it fits where a global sort would not.

# --scratch is a fast disk for per-batch files, sort temp, and build state.
# --batch_gb caps each batch (peak disk = source + tree).
collectionparser build \
  --source /path/to/extracted \
  --dataset /path/to/dataset \
  --scratch /path/to/scratch \
  --batch_gb 120

Pause / resume: stop it any time (Ctrl-C or systemctl stop) and rerun the same command. Finished batches are recorded in build_state.json, and parsed files within the current batch in progress_<n>.txt; only unparsed files are re-parsed. The sort+split step of an interrupted batch is redone (duplicates are removed by the final dedup), so nothing is lost.

State and stats live in --scratch: build_state.json, stats.md, and extract_failures.txt (archives that failed to extract, never silently skipped). sort temp is directed to --scratch via TMPDIR.

Archives (.rar/.zip/.7z/.tar.*/.gz/.bz2/.xz) are extracted automatically into --scratch for the duration of their batch and removed once parsed; the source is never modified. .zip/.tar.*/single-file .gz/.bz2/.xz use the Python standard library; .rar/.7z need 7z or unrar on PATH. Pass --no-extract to skip archives, or --no-dedup to skip the final per-bucket de-duplication.

Processing (manual)

collectionparser parse --path /path/to/extracted --success_file success.txt --failure_file failure.txt

rm -rf tmp && mkdir tmp  # sort needs room for temp files; /tmp is usually too small
cat success.txt | pv -cN cut | sort -T tmp -u -S 90% --parallel=12 | pv -cN cut > success_sorted.txt

collectionparser split --file success_sorted.txt --out data

Merging datasets

merge is destructive: back up the destination first.

cp -rf /path/to/collections1_data /path/to/big_dataset
collectionparser merge --src /path/to/collections2_5_data --dest /path/to/big_dataset

Query and export

# exact (O(1) bucket lookup) or wildcard (* and ?):
collectionparser query --dataset data --pattern 'john*'
collectionparser query --dataset data --pattern '*@gmail.com'

# JSON Lines for OpenSearch / bulk ingest:
collectionparser export --dataset data --out records.jsonl

Crack hashed passwords

parse/build keep hashed passwords (MD5/SHA1/SHA256) separate from plaintext ones instead of discarding them. build collects them in --scratch/hashes.txt; parse writes them to --hashes_file when given.

# Wordlist omitted -> rockyou is downloaded automatically; add the dataset's
# own plaintext passwords as extra candidates:
collectionparser crack \
  --hashes hashes.txt \
  --passwords-from dataset/ \
  --out cracked.txt
# Or bring your own wordlist(s):
collectionparser crack --hashes hashes.txt --wordlist words.txt --out cracked.txt

Pure standard library: hashlib digests with one process per CPU core, each attacking a strided slice of the wordlist, so memory stays bounded; candidate sources are merged through sort -u. Only unsalted MD5/SHA1/SHA256 are attackable — bcrypt is ignored. Feed cracked.txt back into the index with split/merge.

Measured performance

Single large-file test on Collections 2-5 (3 files, 19.94 GB, 3 workers):

metric	value
parse throughput	20.5 MB/s aggregate (6.8 MB/s per worker)
valid pairs	628,342,073
rejects	3,321,666
peak RSS	160 MB
`sort -u`	~125 MB/s
`split` (sorted input)	~43 MB/s

Split is ~12× slower on unsorted input (open-handle thrash); always sort first.

Commands

Run collectionparser --help or collectionparser <command> --help.

command	description
`build`	Batched, resumable: parse + dedup + split into the query tree.
`query`	Exact or wildcard search over a dataset.
`export`	Export a dataset as JSON Lines.
`crack`	Dictionary-attack collected hashes with a wordlist.
`parse`	Parse an unstructured folder into `success`/`failure`/`hashes` files.
`split`	Convert a large sorted file into a query-friendly `a/b/c` folder.
`chunk`	Split large TXT files into smaller ones.
`sort`	Sort and deduplicate every file in a query-friendly folder.
`clean`	Move misplaced records to the right bucket and sort.
`merge`	Merge one query-friendly folder into another.
`test`	Look up passwords for the emails in a file.
`evaluate`	Measure hit rate of one dataset against another.
`filter-lines`	Keep lines whose length is within a range.

Development

pip install -e '.[dev]'
ruff check .      # lint
pytest            # tests

README.md Unescape Escape

collectionparser