3.4 KiB
Document parser isolation
Status: implementation started; current PDF/DOCX parsing still runs in the indexing process. Existing byte/output limits do not establish parser memory, CPU or crash isolation. This work does not delay the published Windows preview.
Required execution boundary
Keep the shared Rust engine independent of the UI. The parent reads each supported source once and fingerprints those exact bytes. PDF and DOCX workers receive a format enum and raw bytes, never a filesystem path, index handle or mutation capability. Existing bounded plain-text/code extraction retains its exact source bytes and line positions. Filename indexing remains available when a document parser fails.
The protocol has a bounded, versioned header followed by length-delimited binary input. Avoid base64 allocation overhead. Responses carry the input SHA-256 and typed passages or explicit parser rejection. The parent independently validates version, digest, count, aggregate text size and source locations. Serialized JSON has its own output limit because escaping can expand extracted text.
Run one process per document initially. Bound admission, stdin writes, stdout, stderr and total elapsed time; a deadline must cover a blocked input writer as well as reading the response. Kill and reap owned children on timeout, cancellation, malformed output and failure. Missing workers or failed resource setup must never silently fall back to unrestricted in-process parsing. Parser errors retain a filename-only record with an extraction error; cancellation remains distinct and rolls back reconciliation.
Use Windows Job Objects for owned process termination and commitment limits. Use supported Unix resource limits before parsing, but verify their actual semantics per OS. A configured RLIMIT_AS alone does not prove a universal macOS memory ceiling. Prefer existing bounded runtime patterns and established OS bindings over inventing a new process manager.
Publication and rollout
Current reconciliation holds a SQLite transaction across discovery, reads, parsing and writes. The current source-stat check is immediately after reading, before parsing. Add a post-worker source/revision check before inserting the result; retain the original parser-input digest, not a substituted later hash. Source files and SQLite are not one atomic transaction. Preserve semantic and agent excerpt provenance checks and clear old passages on parser rejection.
- Buffer API and typed protocol with independent limits and genuine fixtures.
- Headless Rust helper with no GUI dependencies or arbitrary path access.
- Parent process deadlines, cancellation, resource setup and kill/reap tests.
- Shared-engine document adapter, explicit failures and post-parser guards.
- Normal helper discovery and required release staging on all target systems.
- Relocated Windows/WSL packages and actual parser fixtures through the helper.
- Record startup/throughput/memory costs before considering persistent workers.
- Validate remaining macOS and representative corrupt/large-source behavior.
Meaningful regressions include wrong version/hash, oversized header/output, invalid locations, blocked stdin, worker crash/timeout, missing helper, cancelled reconciliation, corrupt/encrypted documents and a source changing during parsing. Shipping only protocol types or unit fixtures does not close the isolated-extraction release gate.