feat: add multimodal search and granular performance diagnostics
Core CI / core (ubuntu-latest) (push) Failing after 3m31s
Core CI / core (macos-latest) (push) Canceled after 0s
Core CI / core (windows-latest) (push) Canceled after 0s
Core CI / desktop (ubuntu-latest) (push) Successful in 5m39s
Core CI / desktop (macos-latest) (push) Canceled after 0s
Core CI / desktop (windows-latest) (push) Canceled after 0s

This commit is contained in:
karl committed 2026-10-12 19:16:23 +02:00
1 parent d3bb8c7618
commit b8400c4fbf
110 files changed
+79395 -250

No files matched your search

+84 -54
View File
@@ -1,64 +1,94 @@
# Validation evidence
Development host: Windows VM, x86_64 MSVC, Rust stable 1.99.0.
All local tests run on CPU; no GPU provider is configured.
Development host: Windows VM, x86_64 MSVC, stable Rust 1.99.0. All local
inference, FFmpeg decoding and WebView tests use CPU only. UI launches set
`WEBVIEW2_ADDITIONAL_BROWSER_ARGUMENTS='--disable-gpu --disable-accelerated-video-decode'`
and `SEARCHI_FORCE_CPU=1`.
## Current integration checkpoint (2026-10-12)
- `cargo test --workspace --locked`: 41 default tests passed after final core integration. Four asset-dependent
or explicit benchmark tests remain opt-in; a default pass does not run them.
- Native desktop tests: 7 passed with SEARCHI_FORCE_CPU=1 and WebView2 GPU disabled.
- Frontend: 7 async-state tests passed, Prettier passed, TypeScript/Vite build passed.
- Genuine Google CPU reference/window test passed separately: five DigUp reference
vectors had cosine similarity above 0.999; actual multilingual retrieval and
tokenizer tail-window coverage passed. MiniLM evidence below is historical.
- Repository description/topics were read back after update; the branded avatar
upload returned HTTP 204 and the repository API returned its avatar URL.
- New desktop styling is in progress. `design-qa.md` records the visual gate as
blocked because browser bootstrap and native screen capture are unavailable.
No visual match or complete UI interaction verification is claimed.
- Native desktop: **9/9 tests passed**, including structured trace/basic responses.
Post-generation cached query warmup is implemented but its interactive first-query gate remains open.
Frontend: **18/18 tests passed**, TypeScript/Vite production build and
formatting passed. Preview policy tests cover fractional timestamp seeking,
image labels, one active/latest pending request, stale replies and Escape.
- Semantic adapter: **8 default tests passed**; the explicit 48 MP resource test
remains opt-in. Strict targeted all-target Clippy and formatting passed.
Protocol bounds, invalid vectors, artifact size/checksum mismatch, old-worker
capability defaults and oversized JPEG rejection are covered.
- The CPU native helper rebuilt and staged successfully with genuine Google
MTMD image/audio support. Its handshake reports `media_supported:true`;
older text workers default false rather than claiming media readiness.
- Actual Google CPU media parity passed separately in **52.85 seconds**:
screenshot 280 cosine 0.99994194, photo 140: 0.99974583, frame 70: 0.99921928 and
speech: 0.99983318 against pinned MIT DigUp vectors. All exceed the 0.999 gate.
This verifies runtime parity, not broad retrieval quality.
- The release resource test encodes/decodes an actual 8000x6000 JPEG with EXIF
rotation and verifies region colors after scaled decoding/resizing. Decode
plus resize took 176–220 ms. A sampled whole-test peak was 146.86 MiB, including
source fixture generation; it is not decoder-only RSS or neural quality.
- The independent production media pipeline has run on 20 real assets/65 frozen
queries using actual Google inference, FFmpeg, metadata, persistence and HNSW.
It encoded 20/20 files with no failures in the recorded 30-second-window run.
Broad file recall is promising; precise speech timestamps, music confusion
and no-answer behavior remain failed quality gates. Preserve the earlier
direct harness failure separately. [Raw results and limitations](MEDIA-QUALITY.md).
- Actual document seed regression passed 24/24 semantic/hybrid queries. This is
a small synthetic regression, not a production relevance corpus.
- The explicit 100,000-vector release ANN diagnostic passed 98.5% recall@10,
HNSW p95 1.37 ms and warm service p95 6.40 ms with zero warm passage hashes.
Vectors are synthetic index benchmark data, never production embeddings.
- The fixed one-million-file first-input report records 40 inputs below 50 ms in
that run: [recorded benchmark](../crates/searchi-core/benchmarks/results/windows-cpu-full-candidate-first-final.json).
These measurements do not establish every query or concurrent-indexing goal.
Final UI integration: the branded compact launcher, local Inter/Fluent assets,
themes and motion implementation passed frontend build, formatting, and seven
state tests. Native launch succeeded; accessibility text showed real controls
and a warmed Google CPU worker. UI input and screenshots remain blocked by
Windows access/capture errors, as recorded in design-qa.md. Workspace formatting
and strict all-target Clippy passed. The fixed full-corpus first-input report
is in crates/searchi-core/benchmarks/results/windows-cpu-full-candidate-first-final.json;
all 40 measured filename inputs were below 50 ms in that run.
The integrated CPU workspace checkpoint passed **62 default tests** (8 assets, 6 CLI, 23 core, 4 media, 8 semantic, 13 service), with **10 explicit opt-in tests ignored**. Workspace and native all-target Clippy passed with warnings denied. The later semantic CPU-restart counter change passed workspace Clippy and the actual CPU query diagnostics test. Earlier aggregate counts are historical.
The active parallel development branches of work may add tests or change counts
after this checkpoint. Run the documented checks again after integration.
The updated pinned C++ helper compiled in an independent CPU build directory and
was staged for desktop resources without interrupting media evaluation. Actual
Google query diagnostics passed separately in 18.09 seconds including model
initialization and two queries: 13 tokens, tokenization 149 us, batch preparation
22 us, inference 112914 us, pooling/normalization 17 us. Rust worker wait/response
decode measured 113960 us. Debug/normal embedding cosine was 0.99999999. These
are one diagnostic sample under VM contention, not production latency evidence.
See [performance boundaries and raw overhead samples](PERFORMANCE.md).
## Historical foundation evidence
## Native UI and platform limits
Initial foundation checks (2026-10-12):
- `cargo test --workspace`: CLI parser/executable checks and engine retrieval,
lifecycle, exclusions, filters, migration and concurrent-reader checks passed.
- `cargo clippy --workspace --all-targets -- -D warnings`: passed.
- Rust formatting applied; final formatting verification follows integration.
- Desktop TypeScript/Vite build passed; npm audit reported zero vulnerabilities.
- Initial Windows desktop native check, embedded app build, strict Clippy and
two desktop tests passed (index gate recovery and guarded source opening).
- Desktop launched with WebView2 `--disable-gpu`. UI automation could not capture
the window: `GetCursorPos` access denied, then monitor capture error
`0x80070057`. Visual layout and end-to-end UI interactions remain unverified.
- Real Python CPU ONNX embedding proof completed on 8 documents / 6 queries:
5/6 top-1 accuracy, 6/6 recall@3. The strict top-1 quality gate is not met.
See `tools/semantic-proof/windows-cpu-result.json` for raw timings and rankings.
This tiny fixture is not evidence of large-corpus performance.
- Native Rust CPU embedding adapter compiled, passed its three unit tests and
the explicit local-model smoke test. The smoke reproduced the same 5/6 top-1
limitation. Default CI omits this asset-dependent test; it must be executed
separately with provisioned local artifacts.
- Integrated workspace validation: 19 default tests passed; 1 local-model test
is opt-in. Rust formatting and strict Clippy passed after lockfile refresh.
- PDF/DOCX fixture tests include matching source locations, corrupt replacement,
encrypted PDF behavior and compressed-input limits.
The branded launcher launched natively. Accessibility inspection showed real
controls and a warmed Google CPU worker. Native screenshots, input automation
and complete image/audio/video playback interactions remain blocked by Windows
access/capture errors recorded in [design QA](../design-qa.md). Passing pure
preview policy tests does not establish real WebView codec playback or visual
fidelity. Default-application opening is not exact timestamp navigation.
Not yet established: production latency targets, semantic inference, desktop
runtime behavior, Linux/macOS execution, installer signing or upgrades.
CI configuration is source configuration, not evidence that remote jobs ran.
Gitea workflows require matching Windows, Linux and macOS runners; runner
availability has not been established. GitHub workflows are also supplied for
future mirroring, but do not run merely because code was pushed to Gitea.
Windows x86-64 verified decoder provisioning/staging has executed. Manifest
entries for Linux and ARM64 are configuration, not evidence of native execution.
A vetted macOS decoder package remains missing. Linux/macOS worker/desktop
execution, production GPU parity/fallback, installer signing, upgrade/rollback,
licenses/source distribution, large libraries and calibrated absence handling
remain unexecuted release gates. FFmpeg subprocess memory is not sandbox-capped;
bounded outputs/timeouts are not an absolute RSS guarantee.
Gitea needs matching registered runners. GitHub workflows are supplied for
mirroring; they do not run merely because code is pushed to Gitea. The current
workflow checks do not automatically certify downloaded Google assets or the
separately built worker. CI configuration is not remote job evidence.
## Historical evidence
Initial engine/CLI checks covered retrieval, persistence, filesystem changes,
exclusions/filters, migration, concurrent readers, PDF pages, DOCX paragraphs,
corrupt replacement, encrypted PDF errors and bounded compressed input.
The first actual Google text adapter matched five pinned reference vectors
above 0.999, including multilingual queries and long-document window coverage.
The rejected MiniLM/ONNX experiment achieved 5/6 top-1 on its six-query fixture;
it is retained under the opt-in historical proof only and is not a production
fallback. [Historical experiment](../tools/semantic-proof/HISTORICAL.md).
Repository metadata/avatar updates were read back successfully. Historical
foundation/default test totals changed as implementation expanded and must not
be presented as the current aggregate.
Reproduction commands and exact interfaces are in [BUILDING.md](BUILDING.md),
[SEMANTIC.md](SEMANTIC.md), and [MEDIA-QUALITY.md](MEDIA-QUALITY.md).