searchi/docs/EXTRACTION-ISOLATION.md

# Document parser isolation

Status: implementation started; current PDF/DOCX parsing still runs in the
indexing process. Existing byte/output limits do not establish parser memory,
CPU or crash isolation. This work does not delay the published Windows preview.

## Required execution boundary

Keep the shared Rust engine independent of the UI. The parent reads each
supported source once and fingerprints those exact bytes. PDF and DOCX workers
receive a format enum and raw bytes, never a filesystem path, index handle or
mutation capability. Existing bounded plain-text/code extraction retains its
exact source bytes and line positions. Filename indexing remains available when
a document parser fails.

The protocol has a bounded, versioned header followed by length-delimited binary
input. Avoid base64 allocation overhead. Responses carry the input SHA-256 and
typed passages or explicit parser rejection. The parent independently validates
version, digest, count, aggregate text size and source locations. Serialized
JSON has its own output limit because escaping can expand extracted text.

Run one process per document initially. Bound admission, stdin writes, stdout,
stderr and total elapsed time; a deadline must cover a blocked input writer as
well as reading the response. Kill and reap owned children on timeout,
cancellation, malformed output and failure. Missing workers or failed resource
setup must never silently fall back to unrestricted in-process parsing.
Parser errors retain a filename-only record with an extraction error;
cancellation remains distinct and rolls back reconciliation.

Use Windows Job Objects for owned process termination and commitment limits.
Use supported Unix resource limits before parsing, but verify their actual
semantics per OS. A configured RLIMIT_AS alone does not prove a universal macOS
memory ceiling. Prefer existing bounded runtime patterns and established OS
bindings over inventing a new process manager.

## Publication and rollout

Current reconciliation holds a SQLite transaction across discovery, reads,
parsing and writes. The current source-stat check is immediately after reading,
before parsing. Add a post-worker source/revision check before inserting the
result; retain the original parser-input digest, not a substituted later hash.
Source files and SQLite are not one atomic transaction. Preserve semantic and
agent excerpt provenance checks and clear old passages on parser rejection.

- [x] Buffer API and typed protocol with independent limits and genuine fixtures.
- [ ] Headless Rust helper with no GUI dependencies or arbitrary path access.
- [ ] Parent process deadlines, cancellation, resource setup and kill/reap tests.
- [ ] Shared-engine document adapter, explicit failures and post-parser guards.
- [ ] Normal helper discovery and required release staging on all target systems.
- [ ] Relocated Windows/WSL packages and actual parser fixtures through the helper.
- [ ] Record startup/throughput/memory costs before considering persistent workers.
- [ ] Validate remaining macOS and representative corrupt/large-source behavior.

Meaningful regressions include wrong version/hash, oversized header/output,
invalid locations, blocked stdin, worker crash/timeout, missing helper,
cancelled reconciliation, corrupt/encrypted documents and a source changing
during parsing. Shipping only protocol types or unit fixtures does not close
the isolated-extraction release gate.

## Implemented first slice

`searchi-core::extraction_protocol` exposes path-free PDF/DOCX requests and
validated responses. Version one admits at most 32 MiB input, a 1-KiB JSON
header, 13 MiB serialized response and 2 MiB aggregate extracted text. Exact
payload length/SHA, response version/digest and document locations are checked;
parser rejection is explicit. Unknown nested source fields are rejected.

`extraction_transport` frames bounded JSON lengths as four-byte little-endian
integers followed by raw input bytes, with no base64 or filesystem paths.
It rejects oversized lengths before allocation and mismatched input before
writing. It is blocking I/O: the parent still must supply deadlines, cancellation,
EOF/process-status validation and owned-child cleanup.

All 50 core tests passed on Windows (18 unit and 32 integration), including
actual DOCX buffer extraction, corrupt PDF/DOCX rejection, raw binary byte
preservation, version/hash/location/size failures and truncated frames. Strict
all-target Clippy, Rust formatting and diff checks passed. Product extraction
defaults and code/text parsing are unchanged. No separate worker was launched;
process isolation and resource ceilings remain open.

## Worker and parent implementation contract

The next slice adds a separate `searchi-extract-worker` binary and a shared-core
parent adapter. Neither changes indexing defaults until integration and packaging
are complete. The adapter owns the child and three bounded I/O tasks; its total
deadline covers input writes, response/diagnostic reads and process exit. Cancel,
timeout, resource failure and malformed protocol remain distinct failures.

Windows uses a parent-owned Job Object with a 512-MiB process commitment limit
and kill-on-close. The parent assigns the job before writing input. The worker
reads only the four-byte frame prefix before checking current job membership and
limits, then admits the header/payload. Prefix arrival synchronizes job setup
without allowing document parsing first. It fails closed when protection is
missing. Unix applies address-space and CPU limits before input; actual macOS
semantics still require evidence. The proposed wall deadline is at most 30 s.

The worker handles one request and no path arguments. Extra arguments/input,
wrong framing or an unprotected Windows launch fail explicitly. Successful
parsing or structured parser rejection must produce one bounded, flushed frame;
the parent independently checks EOF and exit status. None of these limits is a
filesystem/network sandbox or proof that every malicious parser input is safe.
The helper does not add arbitrary file reads or writes to its application API.