Friendlier for LLMs
This commit is contained in:
1 parent
b98f53e956
commit
d7b33d4d8a
5 files changed
+141
-34
No files matched your search
@@ -51,8 +51,16 @@ collectionparser build \
|
||||
**Pause / resume:** stop it any time (Ctrl-C or `systemctl stop`) and rerun the
|
||||
same command. Finished batches are recorded in `build_state.json`, and parsed
|
||||
files within the current batch in `progress_<n>.txt`; only unparsed files are
|
||||
re-parsed. The sort+split step of an interrupted batch is redone (duplicates are
|
||||
removed by the final dedup), so nothing is lost.
|
||||
re-parsed (resume is **per file**, not per line). The sort+split step of an
|
||||
interrupted batch is redone (duplicates are removed by the final dedup), so
|
||||
nothing is lost.
|
||||
|
||||
**Progress:** one machine-readable command.
|
||||
|
||||
```bash
|
||||
collectionparser status --scratch /path/to/scratch # one-line JSON
|
||||
collectionparser status --scratch /path/to/scratch --pretty
|
||||
```
|
||||
|
||||
State and stats live in `--scratch`: `build_state.json`, `stats.md`, and
|
||||
`extract_failures.txt` (archives that failed to extract, never silently skipped).
|
||||
@@ -141,6 +149,7 @@ Run `collectionparser --help` or `collectionparser <command> --help`.
|
||||
| command | description |
|
||||
| --- | --- |
|
||||
| `build` | Batched, resumable: parse + dedup + split into the query tree. |
|
||||
| `status` | Machine-readable build/dataset progress (JSON). |
|
||||
| `query` | Exact or wildcard search over a dataset. |
|
||||
| `export` | Export a dataset as JSON Lines. |
|
||||
| `crack` | Dictionary-attack collected hashes with a wordlist. |
|
||||
|
||||
Reference in new issue
Block a user