58 lines
3.4 KiB
Markdown
58 lines
3.4 KiB
Markdown
|
# Document parser isolation
|
||
|
|
|
||
|
|
Status: implementation started; current PDF/DOCX parsing still runs in the
|
||
|
|
indexing process. Existing byte/output limits do not establish parser memory,
|
||
|
|
CPU or crash isolation. This work does not delay the published Windows preview.
|
||
|
|
|
||
|
|
## Required execution boundary
|
||
|
|
|
||
|
|
Keep the shared Rust engine independent of the UI. The parent reads each
|
||
|
|
supported source once and fingerprints those exact bytes. PDF and DOCX workers
|
||
|
|
receive a format enum and raw bytes, never a filesystem path, index handle or
|
||
|
|
mutation capability. Existing bounded plain-text/code extraction retains its
|
||
|
|
exact source bytes and line positions. Filename indexing remains available when
|
||
|
|
a document parser fails.
|
||
|
|
|
||
|
|
The protocol has a bounded, versioned header followed by length-delimited binary
|
||
|
|
input. Avoid base64 allocation overhead. Responses carry the input SHA-256 and
|
||
|
|
typed passages or explicit parser rejection. The parent independently validates
|
||
|
|
version, digest, count, aggregate text size and source locations. Serialized
|
||
|
|
JSON has its own output limit because escaping can expand extracted text.
|
||
|
|
|
||
|
|
Run one process per document initially. Bound admission, stdin writes, stdout,
|
||
|
|
stderr and total elapsed time; a deadline must cover a blocked input writer as
|
||
|
|
well as reading the response. Kill and reap owned children on timeout,
|
||
|
|
cancellation, malformed output and failure. Missing workers or failed resource
|
||
|
|
setup must never silently fall back to unrestricted in-process parsing.
|
||
|
|
Parser errors retain a filename-only record with an extraction error;
|
||
|
|
cancellation remains distinct and rolls back reconciliation.
|
||
|
|
|
||
|
|
Use Windows Job Objects for owned process termination and commitment limits.
|
||
|
|
Use supported Unix resource limits before parsing, but verify their actual
|
||
|
|
semantics per OS. A configured RLIMIT_AS alone does not prove a universal macOS
|
||
|
|
memory ceiling. Prefer existing bounded runtime patterns and established OS
|
||
|
|
bindings over inventing a new process manager.
|
||
|
|
|
||
|
|
## Publication and rollout
|
||
|
|
|
||
|
|
Current reconciliation holds a SQLite transaction across discovery, reads,
|
||
|
|
parsing and writes. The current source-stat check is immediately after reading,
|
||
|
|
before parsing. Add a post-worker source/revision check before inserting the
|
||
|
|
result; retain the original parser-input digest, not a substituted later hash.
|
||
|
|
Source files and SQLite are not one atomic transaction. Preserve semantic and
|
||
|
|
agent excerpt provenance checks and clear old passages on parser rejection.
|
||
|
|
|
||
|
|
- [ ] Buffer API and typed protocol with independent limits and genuine fixtures.
|
||
|
|
- [ ] Headless Rust helper with no GUI dependencies or arbitrary path access.
|
||
|
|
- [ ] Parent process deadlines, cancellation, resource setup and kill/reap tests.
|
||
|
|
- [ ] Shared-engine document adapter, explicit failures and post-parser guards.
|
||
|
|
- [ ] Normal helper discovery and required release staging on all target systems.
|
||
|
|
- [ ] Relocated Windows/WSL packages and actual parser fixtures through the helper.
|
||
|
|
- [ ] Record startup/throughput/memory costs before considering persistent workers.
|
||
|
|
- [ ] Validate remaining macOS and representative corrupt/large-source behavior.
|
||
|
|
|
||
|
|
Meaningful regressions include wrong version/hash, oversized header/output,
|
||
|
|
invalid locations, blocked stdin, worker crash/timeout, missing helper,
|
||
|
|
cancelled reconciliation, corrupt/encrypted documents and a source changing
|
||
|
|
during parsing. Shipping only protocol types or unit fixtures does not close
|
||
|
|
the isolated-extraction release gate.
|