Friendlier for LLMs
This commit is contained in:
1 parent
b98f53e956
commit
d7b33d4d8a
5 files changed
+141
-34
No files matched your search
@@ -26,6 +26,24 @@ External tools: `sort` and `split` (coreutils); `rg` (ripgrep) for
|
||||
2. Run the pipeline below, one step at a time, watching for the sanity checks.
|
||||
3. Report the output paths and the row counts printed by each step.
|
||||
|
||||
## Checking progress (when the user asks)
|
||||
|
||||
One machine-readable command, no log spelunking:
|
||||
|
||||
```bash
|
||||
collectionparser status --scratch /path/to/scratch # one-line JSON
|
||||
collectionparser status --scratch /path/to/scratch --pretty
|
||||
```
|
||||
|
||||
Fields: `phase`, `complete`, `batches_done`/`batches_total`, `current_batch`,
|
||||
`current_files_done`/`current_files_total`, `input_gb`, `pairs`, `rejects`,
|
||||
`hashes`, `dataset_files`, `dataset_bytes`. Report those. Only if `status`
|
||||
errors (e.g. no `build_state.json`) fall back to
|
||||
`systemctl is-active collectionparser-build.service` and the build log.
|
||||
|
||||
Pause / resume is **per input file**, not per line: files already merged are
|
||||
skipped, and at most the files still in flight when it stopped are re-parsed.
|
||||
|
||||
## Pipeline (the intended order)
|
||||
|
||||
```bash
|
||||
|
||||
Reference in new issue
Block a user