added pw cracking

README.md

@@ -39,10 +39,12 @@ dedups each bucket. Peak disk is source + tree, so it fits where a global sort
 would not.
 
 ```bash
+# --scratch is a fast disk for per-batch files, sort temp, and build state.
+# --batch_gb caps each batch (peak disk = source + tree).
 collectionparser build \
   --source /path/to/extracted \
   --dataset /path/to/dataset \
-  --scratch /path/to/scratch \   # fast disk for batch files + sort temp
+  --scratch /path/to/scratch \
   --batch_gb 120
 ```
 
@@ -50,14 +52,18 @@ collectionparser build \
 same command. Finished batches are recorded in `build_state.json`, and parsed
 files within the current batch in `progress_<n>.txt`; only unparsed files are
 re-parsed. The sort+split step of an interrupted batch is redone (duplicates are
-removed by the final dedup), so nothing is lost. Stats are written to
-`--scratch/stats.md`.
+removed by the final dedup), so nothing is lost.
+
+State and stats live in `--scratch`: `build_state.json`, `stats.md`, and
+`extract_failures.txt` (archives that failed to extract, never silently skipped).
+`sort` temp is directed to `--scratch` via `TMPDIR`.
 
 **Archives** (`.rar`/`.zip`/`.7z`/`.tar.*`/`.gz`/`.bz2`/`.xz`) are extracted
 automatically into `--scratch` for the duration of their batch and removed once
-parsed; the source is never modified. `.zip`/`.tar.*`/single-file `.gz`/`.bz2`/`.xz`
-use the Python standard library; `.rar`/`.7z` need `7z` or `unrar` on `PATH` (use
-`--no-extract` to skip archives entirely).
+parsed; the source is never modified. `.zip`/`.tar.*`/single-file
+`.gz`/`.bz2`/`.xz` use the Python standard library; `.rar`/`.7z` need `7z` or
+`unrar` on `PATH`. Pass `--no-extract` to skip archives, or `--no-dedup` to
+skip the final per-bucket de-duplication.
 
 ## Processing (manual)
 
@@ -90,6 +96,29 @@ collectionparser query --dataset data --pattern '*@gmail.com'
 collectionparser export --dataset data --out records.jsonl
 ```
 
+## Crack hashed passwords
+
+`parse`/`build` keep hashed passwords (MD5/SHA1/SHA256) separate from plaintext
+ones instead of discarding them. `build` collects them in `--scratch/hashes.txt`;
+`parse` writes them to `--hashes_file` when given.
+
+```bash
+# Wordlist omitted -> rockyou is downloaded automatically; add the dataset's
+# own plaintext passwords as extra candidates:
+collectionparser crack \
+  --hashes hashes.txt \
+  --passwords-from dataset/ \
+  --out cracked.txt
+# Or bring your own wordlist(s):
+collectionparser crack --hashes hashes.txt --wordlist words.txt --out cracked.txt
+```
+
+Pure standard library: `hashlib` digests with one process per CPU core, each
+attacking a strided slice of the wordlist, so memory stays bounded; candidate
+sources are merged through `sort -u`. Only unsalted MD5/SHA1/SHA256 are
+attackable — bcrypt is ignored. Feed `cracked.txt` back into the index with
+`split`/`merge`.
+
 ## Measured performance
 
 Single large-file test on Collections 2-5 (3 files, 19.94 GB, 3 workers):
@@ -114,7 +143,8 @@ Run `collectionparser --help` or `collectionparser <command> --help`.
 | `build` | Batched, resumable: parse + dedup + split into the query tree. |
 | `query` | Exact or wildcard search over a dataset. |
 | `export` | Export a dataset as JSON Lines. |
-| `parse` | Parse an unstructured folder into `success`/`failure` files. |
+| `crack` | Dictionary-attack collected hashes with a wordlist. |
+| `parse` | Parse an unstructured folder into `success`/`failure`/`hashes` files. |
 | `split` | Convert a large sorted file into a query-friendly `a/b/c` folder. |
 | `chunk` | Split large TXT files into smaller ones. |
 | `sort` | Sort and deduplicate every file in a query-friendly folder. |
@@ -123,3 +153,11 @@ Run `collectionparser --help` or `collectionparser <command> --help`.
 | `test` | Look up passwords for the emails in a file. |
 | `evaluate` | Measure hit rate of one dataset against another. |
 | `filter-lines` | Keep lines whose length is within a range. |
+
+## Development
+
+```bash
+pip install -e '.[dev]'
+ruff check .      # lint
+pytest            # tests
+```