perf: index filename candidates and avoid full-corpus retrieval scans

crates/searchi-core/benchmarks/README.md

@@ -0,0 +1,104 @@
+# CPU retrieval performance
+
+`examples/performance.rs` builds an explicitly synthetic SQLite corpus using the
+real core schema and normal FTS triggers. It measures warm core retrieval only;
+there is no filesystem scanning, document extraction, model inference, GUI
+rendering, or GPU provider in this benchmark.
+
+The default smoke fixture contains 10,000 file records and 1,000 searchable
+document/passages. The acceptance fixture contains 1,000,000 file records and
+100,000 searchable documents, one paragraph per document. The remaining records
+have filenames but no content. Names and documents are synthetic; they do not
+represent a realistic corpus distribution, so passing these measurements alone
+would not establish production performance.
+
+```powershell
+cargo run --release -p searchi-core --example performance -- --files 1000000 --documents 100000 --samples 20 --db target/benchmarks/full.db
+```
+
+Use a new database path when seeding. The harness refuses to overwrite an
+existing database. To compare engine versions on the same corpus, use `--reuse true`;
+this validates file/document counts and synthetic root ownership before opening
+and migrating the existing fixture. It records migration time separately and
+reports zero seed time. It seeds at most 5,000 records per transaction, checkpoints
+the WAL, then performs three warmups and 20 measured queries per case. Percentiles
+use the nearest-rank convention; with 20 samples, p95 is the 19th sorted sample.
+The first execution of each query is recorded separately before the remaining
+two warmups; this is first use of that query on the current engine, not a cold
+OS disk-cache test. Timing uses a monotonic clock. Results include committed database size and fixture
+seed time, which must not be described as document extraction throughput.
+
+Cases cover an exact filename near the end of the corpus, a broad filename
+substring, a unique content token, and a broad content token. All queries use the
+public search API with limit 20, including actual source-passage selection. Target
+p95 is below 50 ms for filenames and below 150 ms for keyword search. A result
+above either target is explicitly marked as a failure. Database file bytes exclude
+temporary WAL/SHM sidecars and process memory.
+
+## Reference environment
+
+- Date: 2026-10-12.
+- Windows VM: Windows 11 Pro, version 10.0.26300.
+- CPU name exposed by VM: AMD Ryzen 7 7800X3D 8-Core Processor.
+- VM allocation: 4 logical processors, 10,666,553,344 bytes RAM (about 9.93 GiB).
+- Rust release build, bundled SQLite, CPU only.
+- Toolchain: rustc 1.99.0 (b940084d7 2026-09-28), cargo 1.99.0.
+- Engine baseline: `a9b6592330d88745eadf2a748bb3a65f62830590`, schema 4.
+- Storage device and host scheduling contention are not characterized.
+
+Run results are stored in `results/`. Linux and macOS measurements are pending.
+No cross-platform acceptance claim can be inferred from the Windows VM results.
+
+## Baseline result
+
+The full acceptance fixture fails all four latency targets. This is a verified
+performance gap, not a production-ready retrieval result.
+
+| Query case | p50 | p95 | p95 target | Result |
+| --- | ---: | ---: | ---: | --- |
+| Exact filename | 404.76 ms | 633.82 ms | <50 ms | Failed |
+| Broad filename substring | 3,385.76 ms | 4,131.48 ms | <50 ms | Failed |
+| Unique content token | 311.63 ms | 355.78 ms | <150 ms | Failed |
+| Broad keyword | 2,186.24 ms | 2,746.91 ms | <150 ms | Failed |
+
+The 1M/100K synthetic fixture took 277.23 seconds to seed and its checkpointed
+database occupies 309,338,112 bytes (about 295 MiB). Smaller 10K/1K smoke results
+passed the same targets, showing why small-corpus tests must not be used as proof
+of scalability. Neither seed time measures document extraction throughput.
+
+The baseline implementation scanned file metadata for substring matching and used
+per-result passage lookups. Its measured behavior warranted a dedicated filename
+substring index, candidate-first FTS retrieval, and batched passage selection.
+The following section records the resulting implementation and measurements.
+
+## Optimized result, schema 7
+
+The unchanged full fixture was migrated and reused. All four **warm** target
+measurements pass in this Windows CPU run. This does not establish production
+acceptance for all workloads or operating systems.
+
+| Query case | Baseline p95 | Optimized p50 | Optimized p95 | First use | Warm target |
+| --- | ---: | ---: | ---: | ---: | --- |
+| Exact filename | 633.82 ms | 11.66 ms | 13.17 ms | 279.73 ms | Passed |
+| Broad filename substring | 4,131.48 ms | 0.39 ms | 1.68 ms | 3.34 ms | Passed |
+| Unique content token | 355.78 ms | 0.28 ms | 0.39 ms | 50.08 ms | Passed |
+| Broad keyword | 2,746.91 ms | 97.92 ms | 111.54 ms | 101.35 ms | Passed |
+
+The updated checkpointed database is 634,449,920 bytes (about 605 MiB). The
+improvement trades additional indexes and disk space for retrieval latency.
+The initial schema-4-to-5 backfill took 19.65 seconds on this fixture, as recorded
+in the intermediate report. The final report's 3.84 ms `migration_seconds` is an
+already-current-schema open, not a measurement of the initial index backfill.
+Indexed equality, trigram candidate retrieval, content-only FTS, and a single
+batched source lookup replaced full metadata joins and per-result FTS scans.
+Rarest-trigram vocabulary counts are cached in the reader engine. The first-use
+exact result exceeds the warm target and is explicitly retained as an outstanding
+latency issue; opening a fresh reader for every query discards this cache.
+
+`windows-cpu-full-optimized-v7.json` is the final measurement. Earlier schema 5/6
+reports are intermediate experiments and retain failed exact-query measurements.
+The final run did not overlap another benchmark process. Dedicated host isolation
+is still not established. Query distribution and filters are not represented by
+this four-case fixture; restrictive broad filters and sub-three-character searches
+remain potentially slow, while correctness tests ensure late matching files are
+not lost through arbitrary pre-filter caps.

crates/searchi-core/benchmarks/results/windows-cpu-full-optimized-v6.json

@@ -0,0 +1,47 @@
+{
+  "fixture": "synthetic SQLite metadata and text; no filesystem extraction or semantic inference",
+  "files": 1000000,
+  "documents": 100000,
+  "seed_seconds": 0.0,
+  "migration_seconds": 0.0055438,
+  "fixture_reused": true,
+  "database_bytes": 634449920,
+  "database_path": "target/benchmarks/full.db",
+  "architecture": "x86_64",
+  "os": "windows",
+  "cpu_only": true,
+  "queries": [
+    {
+      "query": "invoice_0999999.txt",
+      "samples": 20,
+      "p50_ms": 120.81269999999999,
+      "p95_ms": 137.9349,
+      "target_p95_ms": 50.0,
+      "target_met": false
+    },
+    {
+      "query": "invoice_",
+      "samples": 20,
+      "p50_ms": 0.5431,
+      "p95_ms": 0.7148,
+      "target_p95_ms": 50.0,
+      "target_met": true
+    },
+    {
+      "query": "marker0099999",
+      "samples": 20,
+      "p50_ms": 0.5455,
+      "p95_ms": 0.6604,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    },
+    {
+      "query": "cancellation",
+      "samples": 20,
+      "p50_ms": 98.23469999999999,
+      "p95_ms": 103.2545,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    }
+  ]
+}

crates/searchi-core/benchmarks/results/windows-cpu-full-optimized-v7.json

@@ -0,0 +1,51 @@
+{
+  "fixture": "synthetic SQLite metadata and text; no filesystem extraction or semantic inference",
+  "files": 1000000,
+  "documents": 100000,
+  "seed_seconds": 0.0,
+  "migration_seconds": 0.0038446,
+  "fixture_reused": true,
+  "database_bytes": 634449920,
+  "database_path": "target/benchmarks/full.db",
+  "architecture": "x86_64",
+  "os": "windows",
+  "cpu_only": true,
+  "queries": [
+    {
+      "query": "invoice_0999999.txt",
+      "samples": 20,
+      "p50_ms": 11.659600000000001,
+      "first_query_ms": 279.7262,
+      "p95_ms": 13.167399999999999,
+      "target_p95_ms": 50.0,
+      "target_met": true
+    },
+    {
+      "query": "invoice_",
+      "samples": 20,
+      "p50_ms": 0.3923,
+      "first_query_ms": 3.3355,
+      "p95_ms": 1.6800000000000002,
+      "target_p95_ms": 50.0,
+      "target_met": true
+    },
+    {
+      "query": "marker0099999",
+      "samples": 20,
+      "p50_ms": 0.2847,
+      "first_query_ms": 50.083000000000006,
+      "p95_ms": 0.3948,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    },
+    {
+      "query": "cancellation",
+      "samples": 20,
+      "p50_ms": 97.9169,
+      "first_query_ms": 101.3489,
+      "p95_ms": 111.5428,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    }
+  ]
+}

crates/searchi-core/benchmarks/results/windows-cpu-full-optimized.json

@@ -0,0 +1,47 @@
+{
+  "fixture": "synthetic SQLite metadata and text; no filesystem extraction or semantic inference",
+  "files": 1000000,
+  "documents": 100000,
+  "seed_seconds": 0.0,
+  "migration_seconds": 19.651645,
+  "fixture_reused": true,
+  "database_bytes": 633917440,
+  "database_path": "target/benchmarks/full.db",
+  "architecture": "x86_64",
+  "os": "windows",
+  "cpu_only": true,
+  "queries": [
+    {
+      "query": "invoice_0999999.txt",
+      "samples": 20,
+      "p50_ms": 115.7827,
+      "p95_ms": 127.39,
+      "target_p95_ms": 50.0,
+      "target_met": false
+    },
+    {
+      "query": "invoice_",
+      "samples": 20,
+      "p50_ms": 0.4833,
+      "p95_ms": 1.734,
+      "target_p95_ms": 50.0,
+      "target_met": true
+    },
+    {
+      "query": "marker0099999",
+      "samples": 20,
+      "p50_ms": 0.3963,
+      "p95_ms": 0.5996,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    },
+    {
+      "query": "cancellation",
+      "samples": 20,
+      "p50_ms": 105.1206,
+      "p95_ms": 112.6383,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    }
+  ]
+}

crates/searchi-core/benchmarks/results/windows-cpu-full.json

@@ -0,0 +1,45 @@
+{
+  "fixture": "synthetic SQLite metadata and text; no filesystem extraction or semantic inference",
+  "files": 1000000,
+  "documents": 100000,
+  "seed_seconds": 277.2314138,
+  "database_bytes": 309338112,
+  "database_path": "target/benchmarks/full.db",
+  "architecture": "x86_64",
+  "os": "windows",
+  "cpu_only": true,
+  "queries": [
+    {
+      "query": "invoice_0999999.txt",
+      "samples": 20,
+      "p50_ms": 404.75899999999996,
+      "p95_ms": 633.8166,
+      "target_p95_ms": 50.0,
+      "target_met": false
+    },
+    {
+      "query": "invoice_",
+      "samples": 20,
+      "p50_ms": 3385.7561,
+      "p95_ms": 4131.4819,
+      "target_p95_ms": 50.0,
+      "target_met": false
+    },
+    {
+      "query": "marker0099999",
+      "samples": 20,
+      "p50_ms": 311.6295,
+      "p95_ms": 355.7767,
+      "target_p95_ms": 150.0,
+      "target_met": false
+    },
+    {
+      "query": "cancellation",
+      "samples": 20,
+      "p50_ms": 2186.2423,
+      "p95_ms": 2746.9136,
+      "target_p95_ms": 150.0,
+      "target_met": false
+    }
+  ]
+}

crates/searchi-core/benchmarks/results/windows-cpu-small-optimized.json

@@ -0,0 +1,47 @@
+{
+  "fixture": "synthetic SQLite metadata and text; no filesystem extraction or semantic inference",
+  "files": 10000,
+  "documents": 1000,
+  "seed_seconds": 0.0,
+  "migration_seconds": 0.2795255,
+  "fixture_reused": true,
+  "database_bytes": 6320128,
+  "database_path": "target/benchmarks/small.db",
+  "architecture": "x86_64",
+  "os": "windows",
+  "cpu_only": true,
+  "queries": [
+    {
+      "query": "invoice_0009999.txt",
+      "samples": 20,
+      "p50_ms": 1.9414,
+      "p95_ms": 2.3285,
+      "target_p95_ms": 50.0,
+      "target_met": true
+    },
+    {
+      "query": "invoice_",
+      "samples": 20,
+      "p50_ms": 0.3281,
+      "p95_ms": 1.2427,
+      "target_p95_ms": 50.0,
+      "target_met": true
+    },
+    {
+      "query": "marker0000999",
+      "samples": 20,
+      "p50_ms": 0.2561,
+      "p95_ms": 0.40259999999999996,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    },
+    {
+      "query": "cancellation",
+      "samples": 20,
+      "p50_ms": 1.2418,
+      "p95_ms": 1.7393,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    }
+  ]
+}

crates/searchi-core/benchmarks/results/windows-cpu-small.json

@@ -0,0 +1,45 @@
+{
+  "fixture": "synthetic SQLite metadata and text; no filesystem extraction or semantic inference",
+  "files": 10000,
+  "documents": 1000,
+  "seed_seconds": 0.3428201,
+  "database_bytes": 3207168,
+  "database_path": "target/benchmarks/small.db",
+  "architecture": "x86_64",
+  "os": "windows",
+  "cpu_only": true,
+  "queries": [
+    {
+      "query": "invoice_0009999.txt",
+      "samples": 20,
+      "p50_ms": 3.0624,
+      "p95_ms": 3.8085,
+      "target_p95_ms": 50.0,
+      "target_met": true
+    },
+    {
+      "query": "invoice_",
+      "samples": 20,
+      "p50_ms": 20.8847,
+      "p95_ms": 28.4805,
+      "target_p95_ms": 50.0,
+      "target_met": true
+    },
+    {
+      "query": "marker0000999",
+      "samples": 20,
+      "p50_ms": 1.7667,
+      "p95_ms": 2.7321,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    },
+    {
+      "query": "cancellation",
+      "samples": 20,
+      "p50_ms": 12.2173,
+      "p95_ms": 16.5439,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    }
+  ]
+}