added pw cracking

SKILL.md

@@ -15,8 +15,8 @@ python -m venv venv && source venv/bin/activate
 pip install -e /path/to/collectionparser
 ```
 
-External tools it shells out to: `sort`, `split` (coreutils) and `rg`
-(ripgrep, required only by `test`/`evaluate`).
+External tools: `sort` and `split` (coreutils); `rg` (ripgrep) for
+`query`/`test`/`evaluate`; optional `7z`/`unrar` to extract `.rar`/`.7z` archives.
 
 ## How to run it for the user
 
@@ -40,7 +40,6 @@ sort -T tmp -u -S 90% --parallel=12 success.txt > success_sorted.txt
 collectionparser split --file success_sorted.txt --out data
 
 # 4. Optional repair pass (moves records to the right bucket, re-sorts).
-ulimit -n 60000
 collectionparser clean --path data
 
 # 5. Query.
@@ -58,9 +57,10 @@ collectionparser merge --src other_data --dest big_dataset
 
 | Command | Purpose | Key options |
 | --- | --- | --- |
-| `build` | Batched, resumable parse + dedup + split into the query tree. | `--source`, `--dataset`, `--scratch`, `--batch_gb` |
+| `build` | Batched, resumable parse + dedup + split into the query tree. | `--source`, `--dataset`, `--scratch`, `--batch_gb`, `--extract/--no-extract` |
 | `query` | Exact or wildcard (`*`, `?`) search over a dataset. | `--dataset`, `--pattern` |
 | `export` | Export a dataset as JSON Lines (OpenSearch). | `--dataset`, `--out` |
+| `crack` | Dictionary-attack collected MD5/SHA1/SHA256 hashes. | `--hashes`, `--wordlist`, `--out`, `--workers` |
 | `parse` | Folder of raw files -> `success`/`failure` files. Streams. | `--path`, `--success_file`, `--failure_file` |
 | `split` | One sorted file -> `a/b/c` tree. | `--file`, `--out`, `--restart_from N` |
 | `chunk` | Split oversized TXT files. | `--path`, `--size MB` (default 50) |
@@ -71,33 +71,38 @@ collectionparser merge --src other_data --dest big_dataset
 | `evaluate` | Hit-rate of one index against another. | `--old`, `--new` |
 | `filter-lines` | Keep lines within a length range. | `--file`, `--out`, `--less_than`, `--more_than` |
 
-Global flag: `--debug` (verbose; also bypasses the open-files check below).
+Global flag: `--debug` (verbose).
 
 ## Rules an agent must follow
 
 - **Prefer `build` for large sources.** It parses in size-bounded batches so the
   disk never needs source + temp + tree at once. Stop it anytime and rerun the
   same command to resume: finished batches are in `--scratch/build_state.json`,
-  parsed files in `progress_<n>.txt`. Stats land in `--scratch/stats.md`.
+  parsed files in `progress_<n>.txt`. Stats land in `--scratch/stats.md`; any
+  archive that fails extraction is written to `--scratch/extract_failures.txt`.
 - **Archives are auto-extracted** (`.zip`/`.tar.*`/`.gz` via stdlib; `.rar`/`.7z`
   need `7z` or `unrar` on `PATH`). The source is never modified. Use
   `--no-extract` to skip archive handling.
 - **`split` requires a sorted, unique input.** Always run `sort -u` first; skip
   it and the index silently contains duplicates and unsorted buckets.
-- **`split` and `clean` need `ulimit -n >= 60000`.** The CLI refuses to run
-  otherwise. Raising the limit is the real fix; `--debug` only skips the check.
 - **Memory is already flat.** Don't "optimize" by loading files in Python, and
   don't pre-chunk unless a single file is unwieldy — the tools stream.
 - **`merge` is destructive**: it appends into `--dest` and deletes each `--src`
   file. Back up `--dest` before running.
 - **`--dataset` is a path** to a query-friendly folder built by `split`.
-- **`test`/`evaluate` need `rg` on `PATH`.** If `rg` is missing, queries return
-  nothing; install ripgrep instead of working around it.
+- **`query`/`test`/`evaluate` need `rg` on `PATH`.** If `rg` is missing, queries
+  return nothing; install ripgrep instead of working around it.
+- **`query` wildcards are anchored to the email field**: `*`/`?` never cross the
+  `:` separator (`*@gmail.com` works, matching into the password does not).
 - **`parse` restarts safely.** Completed inputs are recorded in `progress.txt`
   in the current directory; rerun the same command to resume. Don't delete it
   mid-run.
-- `parse` counts a `success` only for plaintext passwords. Hashes
-  (MD5/SHA1/SHA256/bcrypt) are intentionally not written to `success.txt`.
+- `parse`/`build` write only plaintext passwords to `success`; hashed ones
+  (MD5/SHA1/SHA256) go to a separate hashes file (`--scratch/hashes.txt` from
+  `build`, `--hashes_file` from `parse`) so they can be attacked later:
+  `collectionparser crack --hashes hashes.txt --out cracked.txt` (omit
+  `--wordlist` to auto-download rockyou; add `--passwords-from dataset/` to reuse
+  known plaintext passwords). bcrypt is not attackable and is left untouched.
 
 ## Verify the environment