collectionparser/SKILL.md

name, description

name	description
collectionparser	Parse, index, merge, and query multi-billion-row email:password breach collections (Collections 1-5, AntiPublic, Breach Compilation). Use when a task involves huge credential dumps, building a query-friendly a/b/c index, cracking a list of emails, or merging/sorting such datasets. Memory-safe on 100GB+ files.

collectionparser

Turns a folder of raw credential dumps into an a/b/c file tree that can be queried in O(1), then looks up passwords for a list of emails.

Install (Python 3.11+):

python -m venv venv && source venv/bin/activate
pip install -e /path/to/collectionparser

External tools: sort and split (coreutils); rg (ripgrep) for query/test/evaluate; optional 7z/unrar to extract .rar/.7z archives.

How to run it for the user

1. Find the input: either a single raw/sorted text file, or a folder of files. Confirm the path — never guess and never write into the user's dataset before backing up.
2. Run the pipeline below, one step at a time, watching for the sanity checks.
3. Report the output paths and the row counts printed by each step.

Checking progress (when the user asks)

One machine-readable command, no log spelunking:

collectionparser status --scratch /path/to/scratch        # one-line JSON
collectionparser status --scratch /path/to/scratch --pretty

Fields: phase, complete, batches_done/batches_total, current_batch, current_files_done/current_files_total, input_gb, pairs, rejects, hashes, dataset_files, dataset_bytes. Report those. Only if status errors (e.g. no build_state.json) fall back to systemctl is-active collectionparser-build.service and the build log.

Pause / resume is per input file, not per line: files already merged are skipped, and at most the files still in flight when it stopped are re-parsed.

Pipeline (the intended order)

# 1. Raw dumps in a folder -> valid (email:password) and rejected lines.
collectionparser parse --path /path/to/raw --success_file success.txt --failure_file failure.txt

# 2. Sort + dedupe the whole success file. Keep tmp on a big disk.
rm -rf tmp && mkdir tmp
sort -T tmp -u -S 90% --parallel=12 success.txt > success_sorted.txt

# 3. Sorted unique file -> query-friendly a/b/c tree.
collectionparser split --file success_sorted.txt --out data

# 4. Optional repair pass (moves records to the right bucket, re-sorts).
collectionparser clean --path data

# 5. Query.
collectionparser test --file emails.txt --dataset data

To add another dataset to an existing index:

cp -rf data big_dataset                 # merge is destructive; work on a copy
collectionparser merge --src other_data --dest big_dataset

Commands

Command	Purpose	Key options
build	Batched, resumable parse + dedup + split into the query tree.	--source, --dataset, --scratch, --batch_gb, --extract/--no-extract
query	Exact or wildcard (*, ?) search over a dataset.	--dataset, --pattern
export	Export a dataset as JSON Lines (OpenSearch).	--dataset, --out
crack	Dictionary-attack collected MD5/SHA1/SHA256 hashes.	--hashes, --wordlist, --out, --workers
parse	Folder of raw files -> success/failure files. Streams.	--path, --success_file, --failure_file
split	One sorted file -> a/b/c tree.	--file, --out, --restart_from N
chunk	Split oversized TXT files.	--path, --size MB (default 50)
sort	sort -u every file in place.	--path
clean	Move misplaced records, then sort each file.	--path
merge	Merge --src into --dest (destructive). Runs clean after.	--src, --dest
test	Print passwords found for the emails in a file.	--file, --dataset
evaluate	Hit-rate of one index against another.	--old, --new
filter-lines	Keep lines within a length range.	--file, --out, --less_than, --more_than

Rules an agent must follow

• Prefer build for large sources. It parses in size-bounded batches so the disk never needs source + temp + tree at once. Stop it anytime and rerun the same command to resume: finished batches are in --scratch/build_state.json, parsed files in progress_<n>.txt. Stats land in --scratch/stats.md; any archive that fails extraction is written to --scratch/extract_failures.txt.
• Archives are auto-extracted (.zip/.tar.*/.gz via stdlib; .rar/.7z need 7z or unrar on PATH). The source is never modified. Use --no-extract to skip archive handling.
• split requires a sorted, unique input. Always run sort -u first; skip it and the index silently contains duplicates and unsorted buckets.
• Memory is already flat. Don't "optimize" by loading files in Python, and don't pre-chunk unless a single file is unwieldy — the tools stream.
• merge is destructive: it appends into --dest and deletes each --src file. Back up --dest before running.
• --dataset is a path to a query-friendly folder built by split.
• query/test/evaluate need rg on PATH. If rg is missing, queries return nothing; install ripgrep instead of working around it.
• query wildcards are anchored to the email field: */? never cross the : separator (*@gmail.com works, matching into the password does not).
• parse restarts safely. Completed inputs are recorded in progress.txt in the current directory; rerun the same command to resume. Don't delete it mid-run.
• parse/build write only plaintext passwords to success; hashed ones (MD5/SHA1/SHA256) go to a separate hashes file (--scratch/hashes.txt from build, --hashes_file from parse) so they can be attacked later: collectionparser crack --hashes hashes.txt --out cracked.txt (omit --wordlist to auto-download rockyou; add --passwords-from dataset/ to reuse known plaintext passwords). bcrypt is not attackable and is left untouched.

Verify the environment

collectionparser --help
# round trip on toy data (~1s)
printf '[email protected]:pw\[email protected]:pw\n' > /tmp/raw.txt
collectionparser filter-lines --file /tmp/raw.txt --out /tmp/o.txt --less_than 1 --more_than 50

If that works, the package and its entry point are installed correctly.