first commit

apps/inspector/CAPABILITIES.md

@@ -0,0 +1,31 @@
+# Local inspector capabilities
+
+All inspection is local. No scan downloads models or calls cloud inference.
+
+- YARA: genuine pinned yara-python 4.5.4 in private Windows CPython 3.13 runtime. Centrally supplied sources have bounded size and count, includes disabled, imported modules disallowed, compile warnings treated as errors, and five-second match timeout. Rules compile from in-memory strings and scan bytes. Errors, unavailable engine, missing rules and oversize payloads block downloads. Compilation currently runs inside the inspector process; a native compiler hang is bounded by the browser's inspector request deadline but does not restart the inspector automatically.
+- Text: UTF-8 extraction and deterministic secret/literal/fingerprint classification.
+- PDF/OOXML: bounded extraction of text and embedded images; full rendered visual coverage is incomplete and outbound DLP fails closed.
+- Images: bundled RapidOCR 3.8.1 with CPU ONNXRuntime 1.24.4 and packaged PP-OCRv4 detection/recognition plus v2 classification weights. Explicit verified local model paths bypass all model downloads. Real Windows test renders a secret into an image and blocks it with network model downloading forbidden. Models primarily support Chinese/English; other languages and recognition accuracy require validation. OCR completion indicates all frames processed, not guaranteed recognition accuracy. Pillow parsing has dimension/frame limits. Optional `SAFETYNET_OCR_ENGINE=tesseract` uses administrator-installed trusted `TESSERACT_PATH`, optionally `TESSDATA_PREFIX` for local traineddata. Tesseract weights are not bundled. Missing OCR blocks DLP.
+- Audio/video: recognition of container types only. Transcription and full video inspection are unconfigured; DLP blocks rather than claiming safe coverage.
+- Semantic adapter: real Google EmbeddingGemma 2, Q8_0 GGUF export revision `bfcd298762cc34d0357ece5ebdd31791a3a374d8`, 768 dimensions. A verified native worker runs explicitly on CPU, reads bounded local NDJSON, and returns actual normalized vectors. Both GGUF weights and native runtime modules must pass size/SHA-256 checks before loading. Configure `EMBEDDING_MODEL_DIR` and `EMBEDDING_WORKER_PATH` to staged, protected local resources. No cloud inference, model download or Python ML framework is used. Text uses the same pinned `document-v1` profile for training and inspection; native token windows must cover exact UTF-8 bytes through the final byte, otherwise scanning fails closed. At most 64 vectors are allowed; incomplete or oversized training is refused. Native image embeddings process all bounded frames resized to at most672px with48px alignment. Native audio supports only mono16kHz PCM16 WAV up to30seconds; other media remains uninspectable. Actual model fixtures must run separately from protocol unit tests to establish operational readiness.
+- Reference registration: SHA-256 plus normalized eight-word shingle and paragraph hashes, coverage and registrationReady. Every shingle offset is inspected; paragraph normalization uses NFKC, lowercase, sharp-s to ss, final sigma to sigma and collapsed whitespace, shared with JavaScript policy enforcement. Up to 1000 hashes return per registration. Published `fragment-sha256` rules use this same canonicalization and detect excerpts after reformatting. No extracted plaintext returned or logged. Hashes remain sensitive reference metadata and require access control. The whole-file fingerprint is useful even for incomplete extraction, but registrationReady requires complete coverage.
+
+`POST /scan-download` accepts `{data:base64,policy}` and returns malware-only `{decision,ruleId,engine,sha256,bytes,matchedRules,coverage}`. A YARA allow is not a DLP allow. `/inspect` applies enabled YARA first and then DLP. `/reference-document` accepts `{data:base64}` and returns registration features.
+
+Dependencies and packaged model distributions are pinned with official PyPI wheel SHA-256 hashes and installed with `--require-hashes`. OCR model license review and a complete third-party software notice are still required before distribution. The Windows broker uses a Job Object limiting its process tree to four processes and 4 GiB, denying breakaway and killing descendants when the owner exits. Document parsing, YARA, OCR and inference now execute in a separate AppContainer per request. The worker verifies its actual AppContainer token and zero capabilities before importing parsers. Documents cross bounded anonymous pipes; source/runtime/model files receive package-SID read/execute access, never writable grants. Network and private-file probes, native child inheritance, deadlines, 16 MiB pipe integrity and orphan termination are exercised in `test_windows_appcontainer.py` and `artifacts/windows-appcontainer-proof.json`.
+
+The AppContainer can use its Windows-provided private package storage and system resources that Windows exposes to application packages. This is not a promise that no filesystem writes anywhere are possible. Runtime roots and their ancestors reject reparse points; external Python/parser/model overrides are rejected rather than granting access to other projects. An explicit installer/deployment-owner command provisions specific-SID ACLs; normal startup reads native DACLs without changing them. Missing or writable grants fail closed. Packaging provisions the completed copied resources. Protected Program Files operation under a separate standard user remains untested. No capability or loopback exemption is added. Broker startup allows 30 seconds for cold ACL verification; each parser request retains a 29-second worker deadline and 30-second browser request budget. Broker/worker/verifier bytecode writes are disabled so cold startup does not add unapproved files. Independent exploit/bypass assessment remains required.
+
+The isolated worker uses the bundled RapidOCR path. External Tesseract executables and their deployment environment are not forwarded into this boundary; a future separately validated sandbox configuration is required to support them.
+
+Training retains only returned fingerprints, fragment hashes and coverage metadata. The inspector does not retain source documents or extracted plaintext, and its HTTP handlers clear transient request references after processing. Python immutable byte/string allocations and native parser/OCR buffers cannot guarantee forensic memory erasure; OS paging and crash dumps need endpoint controls. Text and bundled RapidOCR image training use memory only. Optional Tesseract writes temporary decoded image frames and removes them on normal completion; secure deletion is not guaranteed, so use the default RapidOCR mode for memory-only training.
+
+Registration reports `extractionComplete` separately from `featuresComplete`, total/retained fragment counts and `fragmentHashesTruncated`. More than 1000 fragment hashes marks coverage incomplete and registrationReady false; a reduced document or separately approved whole-file fingerprint can then be used. A truncated feature set is never presented as complete protection.
+
+Sources: https://yara.readthedocs.io/en/stable/yarapython.html https://pypi.org/project/yara-python/4.5.4/ https://github.com/RapidAI/RapidOCR https://pypi.org/project/rapidocr/3.8.1/
+
+Google model sources: https://ai.google.dev/gemma/docs/embeddinggemma/model_card_2 https://ai.google.dev/gemma/docs/embeddinggemma/multimodal-embeddinggemma-with-sentence-transformers and https://huggingface.co/ggml-org/embeddinggemma-2-GGUF . Multimodal vectors support classification signals; unsupported parsing or incomplete extraction never becomes a DLP allow solely because a vector exists. Model similarity is probabilistic and requires customer-specific thresholds and false-negative testing. Native worker errors/timeouts are sanitized and kill the worker; startup readiness must attest CPU and no Vulkan module. Source strings and native memory are transient but cannot be securely erased by this Python adapter.
+
+Actual Windows CPU validation on 2026-10-12: four real-weight fixtures passed in 25.774 seconds, covering native image embeddings, native mono 16 kHz audio embeddings, multi-window document tail coverage, and a semantic-only policy that blocks a protected paraphrase while allowing unrelated public text. All emitted vectors have 768 finite normalized values. A separate three-document check measured protected-paraphrase cosine 0.861730 versus unrelated-text 0.606880; these are synthetic fixture measurements, not a general accuracy claim. The worker handshake pins the exact hardened build `b11461-no-logits-v1`; policy/runtime manifest field `b11461` identifies its upstream runtime release. Other suffixes are refused, CPU must be attested, and loaded Vulkan is refused.
+
+Repeat actual fixtures with staged resources using `SAFETYNET_TEST_NATIVE_SEMANTIC=1`, `EMBEDDING_MODEL_DIR`, and `EMBEDDING_WORKER_PATH`, then run the bundled Python `-m unittest discover -s apps/inspector -p test_semantic_real.py -v`. Routine protocol tests do not substitute for these opt-in real-model checks.