diff --git a/docs/EXTRACTION-ISOLATION.md b/docs/EXTRACTION-ISOLATION.md new file mode 100644 index 0000000..dc64ba8 --- /dev/null +++ b/docs/EXTRACTION-ISOLATION.md @@ -0,0 +1,58 @@ +# Document parser isolation + +Status: implementation started; current PDF/DOCX parsing still runs in the +indexing process. Existing byte/output limits do not establish parser memory, +CPU or crash isolation. This work does not delay the published Windows preview. + +## Required execution boundary + +Keep the shared Rust engine independent of the UI. The parent reads each +supported source once and fingerprints those exact bytes. PDF and DOCX workers +receive a format enum and raw bytes, never a filesystem path, index handle or +mutation capability. Existing bounded plain-text/code extraction retains its +exact source bytes and line positions. Filename indexing remains available when +a document parser fails. + +The protocol has a bounded, versioned header followed by length-delimited binary +input. Avoid base64 allocation overhead. Responses carry the input SHA-256 and +typed passages or explicit parser rejection. The parent independently validates +version, digest, count, aggregate text size and source locations. Serialized +JSON has its own output limit because escaping can expand extracted text. + +Run one process per document initially. Bound admission, stdin writes, stdout, +stderr and total elapsed time; a deadline must cover a blocked input writer as +well as reading the response. Kill and reap owned children on timeout, +cancellation, malformed output and failure. Missing workers or failed resource +setup must never silently fall back to unrestricted in-process parsing. +Parser errors retain a filename-only record with an extraction error; +cancellation remains distinct and rolls back reconciliation. + +Use Windows Job Objects for owned process termination and commitment limits. +Use supported Unix resource limits before parsing, but verify their actual +semantics per OS. A configured RLIMIT_AS alone does not prove a universal macOS +memory ceiling. Prefer existing bounded runtime patterns and established OS +bindings over inventing a new process manager. + +## Publication and rollout + +Current reconciliation holds a SQLite transaction across discovery, reads, +parsing and writes. The current source-stat check is immediately after reading, +before parsing. Add a post-worker source/revision check before inserting the +result; retain the original parser-input digest, not a substituted later hash. +Source files and SQLite are not one atomic transaction. Preserve semantic and +agent excerpt provenance checks and clear old passages on parser rejection. + +- [ ] Buffer API and typed protocol with independent limits and genuine fixtures. +- [ ] Headless Rust helper with no GUI dependencies or arbitrary path access. +- [ ] Parent process deadlines, cancellation, resource setup and kill/reap tests. +- [ ] Shared-engine document adapter, explicit failures and post-parser guards. +- [ ] Normal helper discovery and required release staging on all target systems. +- [ ] Relocated Windows/WSL packages and actual parser fixtures through the helper. +- [ ] Record startup/throughput/memory costs before considering persistent workers. +- [ ] Validate remaining macOS and representative corrupt/large-source behavior. + +Meaningful regressions include wrong version/hash, oversized header/output, +invalid locations, blocked stdin, worker crash/timeout, missing helper, +cancelled reconciliation, corrupt/encrypted documents and a source changing +during parsing. Shipping only protocol types or unit fixtures does not close +the isolated-extraction release gate. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 7db70d0..4d5b658 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -105,6 +105,7 @@ release. Keep their evidence and unchecked production gates intact. decode; broader codec/ABI coverage and desktop packaging remain open. - [ ] CPU OCR, image retrieval, audio transcription, video segment locations. - [ ] Isolated extraction workers with timeouts and resource ceilings. + See [implementation subgoals and provenance contract](EXTRACTION-ISOLATION.md). ## 5. Release gates - [x] Normal worker build/staging defaults to hardware-detecting auto with CPU