mem safe & format

README.md

@@ -25,7 +25,13 @@ pip install -e .
 
 Every command streams input and writes incrementally. Peak memory does not grow
 with file size: a single 100 GB+ file is parsed with a small constant footprint.
-`sort` and `clean` shell out to the OS `sort -u`, which spills to disk.
+`sort` shells out to the OS `sort -u`, which spills to disk.
+
+- `crack` builds the hash index once and shares it with the workers by forking
+  (copy-on-write) — one copy, not one per worker (verified: ~450 MB total for a
+  1M-hash index across 4 workers).
+- `query` streams wildcard matches line by line from ripgrep, so a broad pattern
+  over a huge dataset stays bounded.
 
 Measured on real Collections 2-5 data (19.94 GB, three files): **160 MB peak RSS**
 for the whole process tree, versus **15.4 GB** for the original parser. Min RAM
@@ -155,13 +161,8 @@ Run `collectionparser --help` or `collectionparser <command> --help`.
 | `crack` | Dictionary-attack collected hashes with a wordlist. |
 | `parse` | Parse an unstructured folder into `success`/`failure`/`hashes` files. |
 | `split` | Convert a large sorted file into a query-friendly `a/b/c` folder. |
-| `chunk` | Split large TXT files into smaller ones. |
 | `sort` | Sort and deduplicate every file in a query-friendly folder. |
-| `clean` | Move misplaced records to the right bucket and sort. |
-| `merge` | Merge one query-friendly folder into another. |
-| `test` | Look up passwords for the emails in a file. |
-| `evaluate` | Measure hit rate of one dataset against another. |
-| `filter-lines` | Keep lines whose length is within a range. |
+| `merge` | Merge one query-friendly folder into another, then sort/dedup. |
 
 ## Development