Friendlier for LLMs

This commit is contained in:
StasonJatham committed 2026-09-19 14:33:52 +02:00
1 parent b98f53e956
commit d7b33d4d8a
5 files changed
+141 -34

No files matched your search

+11 -2
View File
@@ -51,8 +51,16 @@ collectionparser build \
**Pause / resume:** stop it any time (Ctrl-C or `systemctl stop`) and rerun the
same command. Finished batches are recorded in `build_state.json`, and parsed
files within the current batch in `progress_<n>.txt`; only unparsed files are
re-parsed. The sort+split step of an interrupted batch is redone (duplicates are
removed by the final dedup), so nothing is lost.
re-parsed (resume is **per file**, not per line). The sort+split step of an
interrupted batch is redone (duplicates are removed by the final dedup), so
nothing is lost.
**Progress:** one machine-readable command.
```bash
collectionparser status --scratch /path/to/scratch # one-line JSON
collectionparser status --scratch /path/to/scratch --pretty
```
State and stats live in `--scratch`: `build_state.json`, `stats.md`, and
`extract_failures.txt` (archives that failed to extract, never silently skipped).
@@ -141,6 +149,7 @@ Run `collectionparser --help` or `collectionparser <command> --help`.
| command | description |
| --- | --- |
| `build` | Batched, resumable: parse + dedup + split into the query tree. |
| `status` | Machine-readable build/dataset progress (JSON). |
| `query` | Exact or wildcard search over a dataset. |
| `export` | Export a dataset as JSON Lines. |
| `crack` | Dictionary-attack collected hashes with a wordlist. |