searchi/docs/EXTRACTION-ISOLATION.md

Document parser isolation

Status: PDF/DOCX process isolation is included in Windows 0.1.3 and the updated Linux headless archive. Actual Windows/WSL parser checks passed. Native macOS and adversarial resource-exhaustion checks remain open.

Required execution boundary

Keep the shared Rust engine independent of the UI. The parent reads each supported source once and fingerprints those exact bytes. PDF and DOCX workers receive a format enum and raw bytes, never a filesystem path, index handle or mutation capability. Existing bounded plain-text/code extraction retains its exact source bytes and line positions. Filename indexing remains available when a document parser fails.

The protocol has a bounded, versioned header followed by length-delimited binary input. Avoid base64 allocation overhead. Responses carry the input SHA-256 and typed passages or explicit parser rejection. The parent independently validates version, digest, count, aggregate text size and source locations. Serialized JSON has its own output limit because escaping can expand extracted text.

Run one process per document initially. Bound admission, stdin writes, stdout, stderr and total elapsed time; a deadline must cover a blocked input writer as well as reading the response. Kill and reap owned children on timeout, cancellation, malformed output and failure. Missing workers or failed resource setup must never silently fall back to unrestricted in-process parsing. Parser errors retain a filename-only record with an extraction error; cancellation remains distinct and rolls back reconciliation.

Use Windows Job Objects for owned process termination and commitment limits. Use supported Unix resource limits before parsing, but verify their actual semantics per OS. A configured RLIMIT_AS alone does not prove a universal macOS memory ceiling. Prefer existing bounded runtime patterns and established OS bindings over inventing a new process manager.

Publication and rollout

Current reconciliation holds a SQLite transaction across discovery, reads, parsing and writes. Source checks run after reading and again after document parsing, before inserting results, preserving the original parser-input digest. Source files and SQLite are not one atomic transaction. Preserve semantic and agent excerpt provenance checks and clear old passages on parser rejection.

• Buffer API and typed protocol with independent limits and genuine fixtures.
• Headless Rust helper with no GUI dependencies or arbitrary path arguments (Windows and Ubuntu WSL execution tested; macOS resource semantics remain open).
• Parent process deadlines, cancellation, resource setup and kill/reap tests on Windows; actual Unix worker round trips also passed.
• Shared-engine document adapter, explicit failures and post-parser guards.
• Normal helper discovery and required release staging on all target systems.
• Relocated Windows/WSL packages and actual parser fixtures through the helper.
• Record startup/throughput/memory costs before considering persistent workers.
• Validate remaining macOS and representative corrupt/large-source behavior.

Meaningful regressions include wrong version/hash, oversized header/output, invalid locations, blocked stdin, worker crash/timeout, missing helper, cancelled reconciliation, corrupt/encrypted documents and a source changing during parsing. Shipping only protocol types or unit fixtures does not close the isolated-extraction release gate.

Implemented first slice

searchi-core::extraction_protocol exposes path-free PDF/DOCX requests and validated responses. Version one admits at most 32 MiB input, a 1-KiB JSON header, 13 MiB serialized response and 2 MiB aggregate extracted text. Exact payload length/SHA, response version/digest and document locations are checked; parser rejection is explicit. Unknown nested source fields are rejected.

extraction_transport frames bounded JSON lengths as four-byte little-endian integers followed by raw input bytes, with no base64 or filesystem paths. It rejects oversized lengths before allocation and mismatched input before writing. It is blocking I/O: the parent still must supply deadlines, cancellation, EOF/process-status validation and owned-child cleanup.

All 50 core tests passed on Windows (18 unit and 32 integration), including actual DOCX buffer extraction, corrupt PDF/DOCX rejection, raw binary byte preservation, version/hash/location/size failures and truncated frames. Strict all-target Clippy, Rust formatting and diff checks passed. Product extraction defaults and code/text parsing are unchanged. No separate worker was launched; process isolation and resource ceilings remain open.

Worker and parent implementation contract

The next slice adds a separate searchi-extract-worker binary and a shared-core parent adapter. Neither changes indexing defaults until integration and packaging are complete. The adapter owns the child and three bounded I/O tasks; its total deadline covers input writes, response/diagnostic reads and process exit. Cancel, timeout, resource failure and malformed protocol remain distinct failures.

Windows uses a parent-owned Job Object with a 512-MiB process commitment limit and kill-on-close. The parent assigns the job before writing input. The worker reads only the four-byte frame prefix before checking current job membership and limits, then admits the header/payload. Prefix arrival synchronizes job setup without allowing document parsing first. It fails closed when protection is missing. Unix applies address-space and CPU limits before input; actual macOS semantics still require evidence. The proposed wall deadline is at most 30 s.

The worker handles one request and no path arguments. Extra arguments/input, wrong framing or an unprotected Windows launch fail explicitly. Successful parsing or structured parser rejection must produce one bounded, flushed frame; the parent independently checks EOF and exit status. None of these limits is a filesystem/network sandbox or proof that every malicious parser input is safe. The helper does not add arbitrary file reads or writes to its application API.

Windows worker and process-owner evidence

The new workspace binary and searchi-core::extraction_process now work together. Three real worker subprocess tests cover DOCX parsing, corrupt PDF rejection, malformed/trailing input, wrong version/hash, excessive header, unknown arguments and missing Windows Job protection. Two further tests execute the actual helper through the shared parent adapter, checking paragraph text, source position, exact digest and parser-rejection/cancellation distinction.

Four controlled child-process cases exercise crash, oversized/trailing output, excess stderr, blocked input, missing response, timeout and cancellation. The stderr case has a valid response, so an unrelated missing-frame failure cannot mask the stderr budget. Per-test helper compiler activity is serialized to avoid contaminating two-second test deadlines; product deadlines were not relaxed. Owned temporary fixtures are cleaned after process/thread cleanup. Unix cleanup defers reaping until all I/O events, avoiding use of a reaped leader's process-group identifier. Native Unix execution still needs validation.

The final combined Windows CPU run passed 59 tests (54 core and 5 worker); the initially ignored actual-worker core test then passed explicitly, giving 60 executed checks. Strict all-target Clippy passed for both crates. No GPU or model inference was used. Windows Job configuration/verification is exercised, but adversarial memory-exhaustion behavior and native Unix limits remain open.

Shared indexing adapter

Current source uses the helper for PDF/DOCX by default. Headless clients discover it beside their executable; desktop indexing explicitly selects the bundle resource directory, including watcher reconciliation. An internal environment override supports validation. The in-process adapter is an explicit test API, never a failure fallback. Missing or failed helpers leave filenames searchable, record an extraction error, clear obsolete passages and retry on later rescans.

After parsing, bounded rereading checks the original byte digest and metadata before publication. Cancellation rolls back reconciliation. Six document tests passed, including genuine PDF extraction through the normal worker adapter. A controlled subprocess test also passed mutation with identical size/restored mtime during parsing and cancellation during parsing; both preserve the old index through rollback. Strict core Clippy and formatting passed. An earlier full-core run hit a two-second parent fixture timeout; that broader run is not claimed green by these focused results.

Required helper staging is implemented for Windows/Linux and included in their published packages. macOS staging remains open.

Desktop integration validation: 46 native tests passed on Windows with CPU-only settings. The desktop lockfile now records the shared process adapter dependencies. This checks compiled integration, not installed helper discovery or native UI interaction.

The optimized Windows helper built successfully and was copied into an owned validation directory outside target. The normal-adapter genuine PDF test passed against that relocated release helper (one test, 1.11 s). Its SHA-256 is 983156ddb96de1b4fb307564c1b882ffb75300e05f03f705ee8e2e668ff96ae1. This is helper relocation evidence, not a finished installer check.

Windows release staging

tools/stage_extraction_worker.py performs a locked optimized Cargo build, compares bounded core/worker/media build inputs before and after it, and stages only the helper and extraction-worker.json. Binary hash, size, architecture and source provenance are verified. Unrelated runtime resources are retained. Windows archive packaging requires the verified pair before output replacement; release desktop builds run the same verification before bundling.

The real Windows build/stage/verification passed. Twenty-two tooling tests passed normally and under Python -O, including missing, stale, tampered and architecture-mismatched helpers plus failed builds and preserved unrelated files. Windows and Linux headless packaging are now verified and published. Native macOS helper staging remains open.

Updated optimized desktop build passed with the required helper verification. The Windows archive built successfully; all 41 size/hash records matched its actual entries. The helper extracted from that archive passed genuine PDF indexing through the normal adapter (1.76 s). Archive SHA-256: 760259bbf07ed076857f0e47c0b536eb352e4d00973701bfe115d95f41989daf. This local archive check preceded publication of the verified Windows 0.1.3 installer.

Linux headless consistency

The Linux packager now requires a current verified helper/inventory in its runtime directory, checks x86-64 ELF/system dependency closure and executable archive mode. Nineteen package tests passed normally and with Python -O. Missing/stale/tampered helpers and unsupported dependencies preserve output.

An immutable source snapshot at 18ce700 executed four actual worker/parent integration tests under Ubuntu WSL, CPU only: real DOCX, corrupt PDF, exact source digest, cancellation distinction and malformed protocol rejection. Unix resource setup executed; adversarial exhaustion was not tested. The initial WSL nested-virtualization warning did not prevent the terminal passing Cargo run. The updated public Linux archive passed relocated PDF/DOCX indexing and genuine Google CPU search; see package evidence.

Shared workspace regression after publication

At source c692593, cargo test --workspace --locked -- --test-threads=1 completed successfully on Windows with SEARCHI_FORCE_CPU=1 and the built release extraction helper configured. 159 tests passed; 26 explicit model, benchmark and decoder tests were ignored. The previously intermittent parent fixture passed in this serial run without changing product deadlines. This does not replace separately executed genuine Google package checks or close platform/GPU/visual gates.

Small-document throughput observation

The published Linux CLI/helper indexed 100 synthetic documents on Ubuntu WSL (50 identical one-page PDFs, 50 identical one-paragraph DOCX files, 47950 total input bytes). CPU only, no semantic inference. Initial index: 0.893 seconds, 100 indexed, zero extraction failures. Unchanged byte-verifying rescan: 0.0124 seconds, 100 unchanged, zero failures. Both timings include CLI startup and SQLite work; they do not isolate parser startup or represent large real documents, Windows throughput, cold filesystem caches or steady-state search latency. No persistent parser/cache optimization is justified by this small observation. Raw ignored report: .searchi/qa/document-throughput-013.json, SHA-256 b86df0bf2ca0953317033b35b4642e6e0fadb93e50c499a7a28b1ae82512acd1.