collectionparser/SKILL.md

---
name: collectionparser
description: Parse, index, merge, and query multi-billion-row email:password breach collections (Collections 1-5, AntiPublic, Breach Compilation). Use when a task involves huge credential dumps, building a query-friendly a/b/c index, cracking a list of emails, or merging/sorting such datasets. Memory-safe on 100GB+ files.
---

# collectionparser

Turns a folder of raw credential dumps into an `a/b/c` file tree that can be
queried in O(1), then looks up passwords for a list of emails.

Install (Python 3.11+):

```bash
python -m venv venv && source venv/bin/activate
pip install -e /path/to/collectionparser
```

External tools: `sort` and `split` (coreutils); `rg` (ripgrep) for
`query`/`test`/`evaluate`; optional `7z`/`unrar` to extract `.rar`/`.7z` archives.

## How to run it for the user

1. Find the input: either a single raw/sorted text file, or a folder of files.
   Confirm the path — never guess and never write into the user's dataset
   before backing up.
2. Run the pipeline below, one step at a time, watching for the sanity checks.
3. Report the output paths and the row counts printed by each step.

## Checking progress (when the user asks)

One machine-readable command, no log spelunking:

```bash
collectionparser status --scratch /path/to/scratch        # one-line JSON
collectionparser status --scratch /path/to/scratch --pretty
```

Fields: `phase`, `complete`, `batches_done`/`batches_total`, `current_batch`,
`current_files_done`/`current_files_total`, `input_gb`, `pairs`, `rejects`,
`hashes`, `dataset_files`, `dataset_bytes`. Report those. Only if `status`
errors (e.g. no `build_state.json`) fall back to
`systemctl is-active collectionparser-build.service` and the build log.

Pause / resume is **per input file**, not per line: files already merged are
skipped, and at most the files still in flight when it stopped are re-parsed.

## Pipeline (the intended order)

```bash
# 1. Raw dumps in a folder -> valid (email:password) and rejected lines.
collectionparser parse --path /path/to/raw --success_file success.txt --failure_file failure.txt

# 2. Sort + dedupe the whole success file. Keep tmp on a big disk.
rm -rf tmp && mkdir tmp
sort -T tmp -u -S 90% --parallel=12 success.txt > success_sorted.txt

# 3. Sorted unique file -> query-friendly a/b/c tree.
collectionparser split --file success_sorted.txt --out data

# 4. Optional repair pass (moves records to the right bucket, re-sorts).
collectionparser clean --path data

# 5. Query.
collectionparser test --file emails.txt --dataset data
```

To add another dataset to an existing index:

```bash
cp -rf data big_dataset                 # merge is destructive; work on a copy
collectionparser merge --src other_data --dest big_dataset
```

## Commands

| Command | Purpose | Key options |
| --- | --- | --- |
| `build` | Batched, resumable parse + dedup + split into the query tree. | `--source`, `--dataset`, `--scratch`, `--batch_gb`, `--extract/--no-extract` |
| `query` | Exact or wildcard (`*`, `?`) search over a dataset. | `--dataset`, `--pattern` |
| `export` | Export a dataset as JSON Lines (OpenSearch). | `--dataset`, `--out` |
| `crack` | Dictionary-attack collected MD5/SHA1/SHA256 hashes. | `--hashes`, `--wordlist`, `--out`, `--workers` |
| `parse` | Folder of raw files -> `success`/`failure` files. Streams. | `--path`, `--success_file`, `--failure_file` |
| `split` | One sorted file -> `a/b/c` tree. | `--file`, `--out`, `--restart_from N` |
| `chunk` | Split oversized TXT files. | `--path`, `--size MB` (default 50) |
| `sort` | `sort -u` every file in place. | `--path` |
| `clean` | Move misplaced records, then sort each file. | `--path` |
| `merge` | Merge `--src` into `--dest` (destructive). Runs `clean` after. | `--src`, `--dest` |
| `test` | Print passwords found for the emails in a file. | `--file`, `--dataset` |
| `evaluate` | Hit-rate of one index against another. | `--old`, `--new` |
| `filter-lines` | Keep lines within a length range. | `--file`, `--out`, `--less_than`, `--more_than` |

Global flag: `--debug` (verbose).

## Rules an agent must follow

- **Prefer `build` for large sources.** It parses in size-bounded batches so the
  disk never needs source + temp + tree at once. Stop it anytime and rerun the
  same command to resume: finished batches are in `--scratch/build_state.json`,
  parsed files in `progress_<n>.txt`. Stats land in `--scratch/stats.md`; any
  archive that fails extraction is written to `--scratch/extract_failures.txt`.
- **Archives are auto-extracted** (`.zip`/`.tar.*`/`.gz` via stdlib; `.rar`/`.7z`
  need `7z` or `unrar` on `PATH`). The source is never modified. Use
  `--no-extract` to skip archive handling.
- **`split` requires a sorted, unique input.** Always run `sort -u` first; skip
  it and the index silently contains duplicates and unsorted buckets.
- **Memory is already flat.** Don't "optimize" by loading files in Python, and
  don't pre-chunk unless a single file is unwieldy — the tools stream.
- **`merge` is destructive**: it appends into `--dest` and deletes each `--src`
  file. Back up `--dest` before running.
- **`--dataset` is a path** to a query-friendly folder built by `split`.
- **`query`/`test`/`evaluate` need `rg` on `PATH`.** If `rg` is missing, queries
  return nothing; install ripgrep instead of working around it.
- **`query` wildcards are anchored to the email field**: `*`/`?` never cross the
  `:` separator (`*@gmail.com` works, matching into the password does not).
- **`parse` restarts safely.** Completed inputs are recorded in `progress.txt`
  in the current directory; rerun the same command to resume. Don't delete it
  mid-run.
- `parse`/`build` write only plaintext passwords to `success`; hashed ones
  (MD5/SHA1/SHA256) go to a separate hashes file (`--scratch/hashes.txt` from
  `build`, `--hashes_file` from `parse`) so they can be attacked later:
  `collectionparser crack --hashes hashes.txt --out cracked.txt` (omit
  `--wordlist` to auto-download rockyou; add `--passwords-from dataset/` to reuse
  known plaintext passwords). bcrypt is not attackable and is left untouched.

## Verify the environment

```bash
collectionparser --help
# round trip on toy data (~1s)
printf '[email protected]:pw\[email protected]:pw\n' > /tmp/raw.txt
collectionparser filter-lines --file /tmp/raw.txt --out /tmp/o.txt --less_than 1 --more_than 50
```

If that works, the package and its entry point are installed correctly.