Friendlier for LLMs

This commit is contained in:
StasonJatham committed 2026-09-19 14:33:52 +02:00
1 parent b98f53e956
commit d7b33d4d8a
5 files changed
+141 -34

No files matched your search

+18
View File
@@ -26,6 +26,24 @@ External tools: `sort` and `split` (coreutils); `rg` (ripgrep) for
2. Run the pipeline below, one step at a time, watching for the sanity checks.
3. Report the output paths and the row counts printed by each step.
## Checking progress (when the user asks)
One machine-readable command, no log spelunking:
```bash
collectionparser status --scratch /path/to/scratch # one-line JSON
collectionparser status --scratch /path/to/scratch --pretty
```
Fields: `phase`, `complete`, `batches_done`/`batches_total`, `current_batch`,
`current_files_done`/`current_files_total`, `input_gb`, `pairs`, `rejects`,
`hashes`, `dataset_files`, `dataset_bytes`. Report those. Only if `status`
errors (e.g. no `build_state.json`) fall back to
`systemctl is-active collectionparser-build.service` and the build log.
Pause / resume is **per input file**, not per line: files already merged are
skipped, and at most the files still in flight when it stopped are re-parsed.
## Pipeline (the intended order)
```bash