searchi/docs/EXTRACTION-ISOLATION.md

# Document parser isolation

Status: PDF/DOCX process isolation is included in Windows 0.1.3 and the updated
Linux headless archive. Actual Windows/WSL parser checks passed. Native macOS
and adversarial resource-exhaustion checks remain open.

## Required execution boundary

Keep the shared Rust engine independent of the UI. The parent reads each
supported source once and fingerprints those exact bytes. PDF and DOCX workers
receive a format enum and raw bytes, never a filesystem path, index handle or
mutation capability. Existing bounded plain-text/code extraction retains its
exact source bytes and line positions. Filename indexing remains available when
a document parser fails.

The protocol has a bounded, versioned header followed by length-delimited binary
input. Avoid base64 allocation overhead. Responses carry the input SHA-256 and
typed passages or explicit parser rejection. The parent independently validates
version, digest, count, aggregate text size and source locations. Serialized
JSON has its own output limit because escaping can expand extracted text.

Run one process per document initially. Bound admission, stdin writes, stdout,
stderr and total elapsed time; a deadline must cover a blocked input writer as
well as reading the response. Kill and reap owned children on timeout,
cancellation, malformed output and failure. Missing workers or failed resource
setup must never silently fall back to unrestricted in-process parsing.
Parser errors retain a filename-only record with an extraction error;
cancellation remains distinct and rolls back reconciliation.

Use Windows Job Objects for owned process termination and commitment limits.
Use supported Unix resource limits before parsing, but verify their actual
semantics per OS. A configured RLIMIT_AS alone does not prove a universal macOS
memory ceiling. Prefer existing bounded runtime patterns and established OS
bindings over inventing a new process manager.

## Publication and rollout

Current reconciliation holds a SQLite transaction across discovery, reads,
parsing and writes. Source checks run after reading and again after document parsing, before
inserting results, preserving the original parser-input digest.
Source files and SQLite are not one atomic transaction. Preserve semantic and
agent excerpt provenance checks and clear old passages on parser rejection.

- [x] Buffer API and typed protocol with independent limits and genuine fixtures.
- [x] Headless Rust helper with no GUI dependencies or arbitrary path arguments
  (Windows and Ubuntu WSL execution tested; macOS resource semantics remain open).
- [x] Parent process deadlines, cancellation, resource setup and kill/reap tests
  on Windows; actual Unix worker round trips also passed.
- [x] Shared-engine document adapter, explicit failures and post-parser guards.
- [ ] Normal helper discovery and required release staging on all target systems.
- [x] Relocated Windows/WSL packages and actual parser fixtures through the helper.
- [ ] Record startup/throughput/memory costs before considering persistent workers.
- [ ] Validate remaining macOS and representative corrupt/large-source behavior.

Meaningful regressions include wrong version/hash, oversized header/output,
invalid locations, blocked stdin, worker crash/timeout, missing helper,
cancelled reconciliation, corrupt/encrypted documents and a source changing
during parsing. Shipping only protocol types or unit fixtures does not close
the isolated-extraction release gate.

## Implemented first slice

`searchi-core::extraction_protocol` exposes path-free PDF/DOCX requests and
validated responses. Version one admits at most 32 MiB input, a 1-KiB JSON
header, 13 MiB serialized response and 2 MiB aggregate extracted text. Exact
payload length/SHA, response version/digest and document locations are checked;
parser rejection is explicit. Unknown nested source fields are rejected.

`extraction_transport` frames bounded JSON lengths as four-byte little-endian
integers followed by raw input bytes, with no base64 or filesystem paths.
It rejects oversized lengths before allocation and mismatched input before
writing. It is blocking I/O: the parent still must supply deadlines, cancellation,
EOF/process-status validation and owned-child cleanup.

All 50 core tests passed on Windows (18 unit and 32 integration), including
actual DOCX buffer extraction, corrupt PDF/DOCX rejection, raw binary byte
preservation, version/hash/location/size failures and truncated frames. Strict
all-target Clippy, Rust formatting and diff checks passed. Product extraction
defaults and code/text parsing are unchanged. No separate worker was launched;
process isolation and resource ceilings remain open.

## Worker and parent implementation contract

The next slice adds a separate `searchi-extract-worker` binary and a shared-core
parent adapter. Neither changes indexing defaults until integration and packaging
are complete. The adapter owns the child and three bounded I/O tasks; its total
deadline covers input writes, response/diagnostic reads and process exit. Cancel,
timeout, resource failure and malformed protocol remain distinct failures.

Windows uses a parent-owned Job Object with a 512-MiB process commitment limit
and kill-on-close. The parent assigns the job before writing input. The worker
reads only the four-byte frame prefix before checking current job membership and
limits, then admits the header/payload. Prefix arrival synchronizes job setup
without allowing document parsing first. It fails closed when protection is
missing. Unix applies address-space and CPU limits before input; actual macOS
semantics still require evidence. The proposed wall deadline is at most 30 s.

The worker handles one request and no path arguments. Extra arguments/input,
wrong framing or an unprotected Windows launch fail explicitly. Successful
parsing or structured parser rejection must produce one bounded, flushed frame;
the parent independently checks EOF and exit status. None of these limits is a
filesystem/network sandbox or proof that every malicious parser input is safe.
The helper does not add arbitrary file reads or writes to its application API.

## Windows worker and process-owner evidence

The new workspace binary and `searchi-core::extraction_process` now work
together. Three real worker subprocess tests cover DOCX parsing, corrupt PDF
rejection, malformed/trailing input, wrong version/hash, excessive header,
unknown arguments and missing Windows Job protection. Two further tests execute
the actual helper through the shared parent adapter, checking paragraph text,
source position, exact digest and parser-rejection/cancellation distinction.

Four controlled child-process cases exercise crash, oversized/trailing output,
excess stderr, blocked input, missing response, timeout and cancellation.
The stderr case has a valid response, so an unrelated missing-frame failure
cannot mask the stderr budget. Per-test helper compiler activity is serialized
to avoid contaminating two-second test deadlines; product deadlines were not
relaxed. Owned temporary fixtures are cleaned after process/thread cleanup.
Unix cleanup defers reaping until all I/O events, avoiding use of a reaped
leader's process-group identifier. Native Unix execution still needs validation.

The final combined Windows CPU run passed 59 tests (54 core and 5 worker);
the initially ignored actual-worker core test then passed explicitly, giving
60 executed checks. Strict all-target Clippy passed for both crates. No GPU or
model inference was used. Windows Job configuration/verification is exercised,
but adversarial memory-exhaustion behavior and native Unix limits remain open.
## Shared indexing adapter

Current source uses the helper for PDF/DOCX by default. Headless clients discover
it beside their executable; desktop indexing explicitly selects the bundle
resource directory, including watcher reconciliation. An internal environment
override supports validation. The in-process adapter is an explicit test API,
never a failure fallback. Missing or failed helpers leave filenames searchable,
record an extraction error, clear obsolete passages and retry on later rescans.

After parsing, bounded rereading checks the original byte digest and metadata
before publication. Cancellation rolls back reconciliation. Six document tests
passed, including genuine PDF extraction through the normal worker adapter.
A controlled subprocess test also passed mutation with identical size/restored
mtime during parsing and cancellation during parsing; both preserve the old
index through rollback. Strict core Clippy and formatting passed. An earlier
full-core run hit a two-second parent fixture timeout; that broader run is not
claimed green by these focused results.

Required helper staging is implemented for Windows/Linux and included in their
published packages. macOS staging remains open.

Desktop integration validation: 46 native tests passed on Windows with CPU-only
settings. The desktop lockfile now records the shared process adapter dependencies.
This checks compiled integration, not installed helper discovery or native UI interaction.

The optimized Windows helper built successfully and was copied into an owned
validation directory outside `target`. The normal-adapter genuine PDF test
passed against that relocated release helper (one test, 1.11 s). Its SHA-256 is
`983156ddb96de1b4fb307564c1b882ffb75300e05f03f705ee8e2e668ff96ae1`.
This is helper relocation evidence, not a finished installer check.

## Windows release staging

`tools/stage_extraction_worker.py` performs a locked optimized Cargo build,
compares bounded core/worker/media build inputs before and after it, and stages
only the helper and `extraction-worker.json`. Binary hash, size, architecture
and source provenance are verified. Unrelated runtime resources are retained.
Windows archive packaging requires the verified pair before output replacement;
release desktop builds run the same verification before bundling.

The real Windows build/stage/verification passed. Twenty-two tooling tests
passed normally and under Python `-O`, including missing, stale, tampered and
architecture-mismatched helpers plus failed builds and preserved unrelated files.
Windows and Linux headless packaging are now verified and published. Native
macOS helper staging remains open.

Updated optimized desktop build passed with the required helper verification.
The Windows archive built successfully; all 41 size/hash records matched
its actual entries. The helper extracted from that archive passed genuine PDF
indexing through the normal adapter (1.76 s). Archive SHA-256:
`760259bbf07ed076857f0e47c0b536eb352e4d00973701bfe115d95f41989daf`.
This local archive check preceded publication of the verified Windows 0.1.3 installer.

## Linux headless consistency

The Linux packager now requires a current verified helper/inventory in its
runtime directory, checks x86-64 ELF/system dependency closure and executable
archive mode. Nineteen package tests passed normally and with Python `-O`.
Missing/stale/tampered helpers and unsupported dependencies preserve output.

An immutable source snapshot at `18ce700` executed four actual worker/parent
integration tests under Ubuntu WSL, CPU only: real DOCX, corrupt PDF, exact
source digest, cancellation distinction and malformed protocol rejection.
Unix resource setup executed; adversarial exhaustion was not tested. The initial
WSL nested-virtualization warning did not prevent the terminal passing Cargo run.
The updated public Linux archive passed relocated PDF/DOCX indexing and genuine
Google CPU search; see [package evidence](LINUX-HEADLESS.md).

## Shared workspace regression after publication

At source `c692593`, `cargo test --workspace --locked -- --test-threads=1`
completed successfully on Windows with `SEARCHI_FORCE_CPU=1` and the built
release extraction helper configured. 159 tests passed; 26 explicit model,
benchmark and decoder tests were ignored. The previously intermittent parent
fixture passed in this serial run without changing product deadlines. This
does not replace separately executed genuine Google package checks or close
platform/GPU/visual gates.

## Small-document throughput observation

The published Linux CLI/helper indexed 100 synthetic documents on Ubuntu WSL
(50 identical one-page PDFs, 50 identical one-paragraph DOCX files, 47950 total
input bytes). CPU only, no semantic inference. Initial index: 0.893 seconds,
100 indexed, zero extraction failures. Unchanged byte-verifying rescan: 0.0124
seconds, 100 unchanged, zero failures. Both timings include CLI startup and
SQLite work; they do not isolate parser startup or represent large real documents,
Windows throughput, cold filesystem caches or steady-state search latency.
No persistent parser/cache optimization is justified by this small observation.
Raw ignored report: `.searchi/qa/document-throughput-013.json`, SHA-256
`b86df0bf2ca0953317033b35b4642e6e0fadb93e50c499a7a28b1ae82512acd1`.

## macOS staging preparation

The helper staging tool now detects Darwin as macOS, recognizes native thin
64-bit little-endian Mach-O executables for Intel/Apple Silicon, and preserves
executable mode. Desktop release verification now covers macOS too. Header
layout and constants follow Apple's [loader header](https://github.com/apple-oss-distributions/xnu/blob/main/EXTERNAL_HEADERS/mach-o/loader.h)
and [CPU definitions](https://github.com/apple-oss-distributions/xnu/blob/main/osfmk/mach/machine.h).
Six staging tests passed normally and under optimized Python, including mocked
Darwin defaults and synthetic header rejection. The actual staged Windows
helper still verifies. This is preparation only: native macOS build, loader
closure, resource-limit behavior, signing and application execution remain open.
Universal/fat extraction-helper binaries are not supported by this staging tool.