Rename project to collectionparser

SKILL.md

@@ -0,0 +1,111 @@
+---
+name: collectionparser
+description: Parse, index, merge, and query multi-billion-row email:password breach collections (Collections 1-5, AntiPublic, Breach Compilation). Use when a task involves huge credential dumps, building a query-friendly a/b/c index, cracking a list of emails, or merging/sorting such datasets. Memory-safe on 100GB+ files.
+---
+
+# collectionparser
+
+Turns a folder of raw credential dumps into an `a/b/c` file tree that can be
+queried in O(1), then looks up passwords for a list of emails.
+
+Install (Python 3.11+):
+
+```bash
+python -m venv venv && source venv/bin/activate
+pip install -e /path/to/collectionparser
+```
+
+External tools it shells out to: `sort`, `split` (coreutils) and `rg`
+(ripgrep, required only by `test`/`evaluate`).
+
+## How to run it for the user
+
+1. Find the input: either a single raw/sorted text file, or a folder of files.
+   Confirm the path — never guess and never write into the user's dataset
+   before backing up.
+2. Run the pipeline below, one step at a time, watching for the sanity checks.
+3. Report the output paths and the row counts printed by each step.
+
+## Pipeline (the intended order)
+
+```bash
+# 1. Raw dumps in a folder -> valid (email:password) and rejected lines.
+collectionparser parse --path /path/to/raw --success_file success.txt --failure_file failure.txt
+
+# 2. Sort + dedupe the whole success file. Keep tmp on a big disk.
+rm -rf tmp && mkdir tmp
+sort -T tmp -u -S 90% --parallel=12 success.txt > success_sorted.txt
+
+# 3. Sorted unique file -> query-friendly a/b/c tree.
+collectionparser split --file success_sorted.txt --out data
+
+# 4. Optional repair pass (moves records to the right bucket, re-sorts).
+ulimit -n 60000
+collectionparser clean --path data
+
+# 5. Query.
+collectionparser test --file emails.txt --dataset data
+```
+
+To add another dataset to an existing index:
+
+```bash
+cp -rf data big_dataset                 # merge is destructive; work on a copy
+collectionparser merge --src other_data --dest big_dataset
+```
+
+## Commands
+
+| Command | Purpose | Key options |
+| --- | --- | --- |
+| `build` | Batched, resumable parse + dedup + split into the query tree. | `--source`, `--dataset`, `--scratch`, `--batch_gb` |
+| `query` | Exact or wildcard (`*`, `?`) search over a dataset. | `--dataset`, `--pattern` |
+| `export` | Export a dataset as JSON Lines (OpenSearch). | `--dataset`, `--out` |
+| `parse` | Folder of raw files -> `success`/`failure` files. Streams. | `--path`, `--success_file`, `--failure_file` |
+| `split` | One sorted file -> `a/b/c` tree. | `--file`, `--out`, `--restart_from N` |
+| `chunk` | Split oversized TXT files. | `--path`, `--size MB` (default 50) |
+| `sort` | `sort -u` every file in place. | `--path` |
+| `clean` | Move misplaced records, then sort each file. | `--path` |
+| `merge` | Merge `--src` into `--dest` (destructive). Runs `clean` after. | `--src`, `--dest` |
+| `test` | Print passwords found for the emails in a file. | `--file`, `--dataset` |
+| `evaluate` | Hit-rate of one index against another. | `--old`, `--new` |
+| `filter-lines` | Keep lines within a length range. | `--file`, `--out`, `--less_than`, `--more_than` |
+
+Global flag: `--debug` (verbose; also bypasses the open-files check below).
+
+## Rules an agent must follow
+
+- **Prefer `build` for large sources.** It parses in size-bounded batches so the
+  disk never needs source + temp + tree at once. Stop it anytime and rerun the
+  same command to resume: finished batches are in `--scratch/build_state.json`,
+  parsed files in `progress_<n>.txt`. Stats land in `--scratch/stats.md`.
+- **Archives are auto-extracted** (`.zip`/`.tar.*`/`.gz` via stdlib; `.rar`/`.7z`
+  need `7z` or `unrar` on `PATH`). The source is never modified. Use
+  `--no-extract` to skip archive handling.
+- **`split` requires a sorted, unique input.** Always run `sort -u` first; skip
+  it and the index silently contains duplicates and unsorted buckets.
+- **`split` and `clean` need `ulimit -n >= 60000`.** The CLI refuses to run
+  otherwise. Raising the limit is the real fix; `--debug` only skips the check.
+- **Memory is already flat.** Don't "optimize" by loading files in Python, and
+  don't pre-chunk unless a single file is unwieldy — the tools stream.
+- **`merge` is destructive**: it appends into `--dest` and deletes each `--src`
+  file. Back up `--dest` before running.
+- **`--dataset` is a path** to a query-friendly folder built by `split`.
+- **`test`/`evaluate` need `rg` on `PATH`.** If `rg` is missing, queries return
+  nothing; install ripgrep instead of working around it.
+- **`parse` restarts safely.** Completed inputs are recorded in `progress.txt`
+  in the current directory; rerun the same command to resume. Don't delete it
+  mid-run.
+- `parse` counts a `success` only for plaintext passwords. Hashes
+  (MD5/SHA1/SHA256/bcrypt) are intentionally not written to `success.txt`.
+
+## Verify the environment
+
+```bash
+collectionparser --help
+# round trip on toy data (~1s)
+printf '[email protected]:pw\[email protected]:pw\n' > /tmp/raw.txt
+collectionparser filter-lines --file /tmp/raw.txt --out /tmp/o.txt --less_than 1 --more_than 50
+```
+
+If that works, the package and its entry point are installed correctly.