---
name: collectionparser
description: Parse, index, merge, and query multi-billion-row email:password breach collections (Collections 1-5, AntiPublic, Breach Compilation). Use when a task involves huge credential dumps, building a query-friendly a/b/c index, cracking a list of emails, or merging/sorting such datasets. Memory-safe on 100GB+ files.
---

# collectionparser

Turns a folder of raw credential dumps into an `a/b/c` file tree that can be
queried in O(1), then looks up passwords for a list of emails.

Install (Python 3.11+):

```bash
python -m venv venv && source venv/bin/activate
pip install -e /path/to/collectionparser
```

External tools it shells out to: `sort`, `split` (coreutils) and `rg`
(ripgrep, required only by `test`/`evaluate`).

## How to run it for the user

1. Find the input: either a single raw/sorted text file, or a folder of files.
   Confirm the path — never guess and never write into the user's dataset
   before backing up.
2. Run the pipeline below, one step at a time, watching for the sanity checks.
3. Report the output paths and the row counts printed by each step.

## Pipeline (the intended order)

```bash
# 1. Raw dumps in a folder -> valid (email:password) and rejected lines.
collectionparser parse --path /path/to/raw --success_file success.txt --failure_file failure.txt

# 2. Sort + dedupe the whole success file. Keep tmp on a big disk.
rm -rf tmp && mkdir tmp
sort -T tmp -u -S 90% --parallel=12 success.txt > success_sorted.txt

# 3. Sorted unique file -> query-friendly a/b/c tree.
collectionparser split --file success_sorted.txt --out data

# 4. Optional repair pass (moves records to the right bucket, re-sorts).
ulimit -n 60000
collectionparser clean --path data

# 5. Query.
collectionparser test --file emails.txt --dataset data
```

To add another dataset to an existing index:

```bash
cp -rf data big_dataset                 # merge is destructive; work on a copy
collectionparser merge --src other_data --dest big_dataset
```

## Commands

| Command | Purpose | Key options |
| --- | --- | --- |
| `build` | Batched, resumable parse + dedup + split into the query tree. | `--source`, `--dataset`, `--scratch`, `--batch_gb` |
| `query` | Exact or wildcard (`*`, `?`) search over a dataset. | `--dataset`, `--pattern` |
| `export` | Export a dataset as JSON Lines (OpenSearch). | `--dataset`, `--out` |
| `parse` | Folder of raw files -> `success`/`failure` files. Streams. | `--path`, `--success_file`, `--failure_file` |
| `split` | One sorted file -> `a/b/c` tree. | `--file`, `--out`, `--restart_from N` |
| `chunk` | Split oversized TXT files. | `--path`, `--size MB` (default 50) |
| `sort` | `sort -u` every file in place. | `--path` |
| `clean` | Move misplaced records, then sort each file. | `--path` |
| `merge` | Merge `--src` into `--dest` (destructive). Runs `clean` after. | `--src`, `--dest` |
| `test` | Print passwords found for the emails in a file. | `--file`, `--dataset` |
| `evaluate` | Hit-rate of one index against another. | `--old`, `--new` |
| `filter-lines` | Keep lines within a length range. | `--file`, `--out`, `--less_than`, `--more_than` |

Global flag: `--debug` (verbose; also bypasses the open-files check below).

## Rules an agent must follow

- **Prefer `build` for large sources.** It parses in size-bounded batches so the
  disk never needs source + temp + tree at once. Stop it anytime and rerun the
  same command to resume: finished batches are in `--scratch/build_state.json`,
  parsed files in `progress_<n>.txt`. Stats land in `--scratch/stats.md`.
- **Archives are auto-extracted** (`.zip`/`.tar.*`/`.gz` via stdlib; `.rar`/`.7z`
  need `7z` or `unrar` on `PATH`). The source is never modified. Use
  `--no-extract` to skip archive handling.
- **`split` requires a sorted, unique input.** Always run `sort -u` first; skip
  it and the index silently contains duplicates and unsorted buckets.
- **`split` and `clean` need `ulimit -n >= 60000`.** The CLI refuses to run
  otherwise. Raising the limit is the real fix; `--debug` only skips the check.
- **Memory is already flat.** Don't "optimize" by loading files in Python, and
  don't pre-chunk unless a single file is unwieldy — the tools stream.
- **`merge` is destructive**: it appends into `--dest` and deletes each `--src`
  file. Back up `--dest` before running.
- **`--dataset` is a path** to a query-friendly folder built by `split`.
- **`test`/`evaluate` need `rg` on `PATH`.** If `rg` is missing, queries return
  nothing; install ripgrep instead of working around it.
- **`parse` restarts safely.** Completed inputs are recorded in `progress.txt`
  in the current directory; rerun the same command to resume. Don't delete it
  mid-run.
- `parse` counts a `success` only for plaintext passwords. Hashes
  (MD5/SHA1/SHA256/bcrypt) are intentionally not written to `success.txt`.

## Verify the environment

```bash
collectionparser --help
# round trip on toy data (~1s)
printf 'a@x.com:pw\nb@y.com:pw\n' > /tmp/raw.txt
collectionparser filter-lines --file /tmp/raw.txt --out /tmp/o.txt --less_than 1 --more_than 50
```

If that works, the package and its entry point are installed correctly.