perf: index filename candidates and avoid full-corpus retrieval scans

2026-10-12 16:35:46 +02:00
1 parent a9b6592330
commit 55fa751571
@@ -4,7 +4,7 @@ The core has no desktop or inference dependencies. Its current retrieval engine
 combines literal filename matching with SQLite FTS5 BM25 keyword search. It does
 not implement semantic search.

-`SearchIndex::open` creates or migrates schema version 4. Current-schema opens do
+`SearchIndex::open` creates or migrates schema version 7. Current-schema opens do
 not acquire a write lock. WAL allows independent readers to use the committed
 index while reconciliation runs. All indexing changes are committed atomically;
 a traversal or plain-text file read error rolls back the entire reconciliation.
@@ -73,10 +73,39 @@ apply before limiting results. `search` retains the unfiltered API. Query text i
 escaped as literal FTS terms, not interpreted as operators. Results are capped at
 1,000 and queries at 4,096 bytes.

+Retrieval runs complete SQL-filtered lanes: indexed exact filename equality,
+literal filename substring, literal path substring, and content-only BM25.
+Multiword queries also have a tokenized name/path fallback, so `annual report`
+can find `annual_report.md`. Single-token filename identifiers are matched
+literally; the text tokenizer still handles punctuation inside document content.
+Exact filenames always precede other lanes. Ties use persistent file row IDs,
+and keyword ties use relevance then row ID. Selected rows are deduplicated.
+
+Filename/path substrings of at least three Unicode characters use an
+external-content trigram FTS index. The rarest known query trigram supplies a
+candidate superset; the whole literal substring is verified on the candidate.
+At most 256 unique query trigrams are inspected. This lookup budget does not cap
+candidate rows or hide matching results. All size/type/folder/date filters apply
+before each lane's SQL LIMIT. Shorter substrings retain complete metadata scans
+and can be slow. Broad queries under restrictive filters can also examine many
+matching candidates; no universal filtered-query latency claim is made.
+
+Trigram frequencies use the FTS vocabulary and a bounded 4,096-term cache on each
+`SearchIndex`. Initial frequency lookups can be expensive; the cache is cleared
+after reconciliation. Keep an independent reader engine alive across searches to
+benefit from warm behavior. These frequency estimates affect candidate selection
+cost, not correctness. A source query batches all finalists' indexed passage IDs
+and selects actual matching passages before choosing one source per file.
+
+Migration to the new indexes preserves file content and passage IDs and rebuilds
+both filename/path and passage indexes atomically. Large migrations can take time
+and increase database size; they must not be confused with normal current-schema
+opens. The measured Windows synthetic fixture grew from about 295 MiB to 605 MiB.
+
 ## Known limits

- Filename substring lookup uses `instr` and scans metadata. A dedicated substring
-  index and benchmarks are needed before claiming million-file latency targets.
+- Short substring queries and broad restrictive filters can still scan many
+  metadata candidates. Cold frequency-cache fills exceed the warm filename target.
 - Files changed without changing size or modification time require a future
  forced refresh/content-fingerprint option. Timestamp precision depends on the
  filesystem. Indexing does not yet run a filesystem watcher.
@@ -91,5 +120,5 @@ escaped as literal FTS terms, not interpreted as operators. Results are capped a
  Bounded worker isolation, persistent jobs, and crash-resumable scans remain
  production milestones.
 - OCR, media extraction, and real model inference belong to subsequent modules.
- Passage selection currently adds a bounded query per result; query batching and
-  passage ranking benchmarks remain necessary for large result lists.
+- Linux/macOS measurements, realistic mixed-document corpora, cold startup,
+  memory budgets, and indexing performance remain unverified.
@@ -0,0 +1,104 @@
+# CPU retrieval performance
+
+`examples/performance.rs` builds an explicitly synthetic SQLite corpus using the
+real core schema and normal FTS triggers. It measures warm core retrieval only;
+there is no filesystem scanning, document extraction, model inference, GUI
+rendering, or GPU provider in this benchmark.
+
+The default smoke fixture contains 10,000 file records and 1,000 searchable
+document/passages. The acceptance fixture contains 1,000,000 file records and
+100,000 searchable documents, one paragraph per document. The remaining records
+have filenames but no content. Names and documents are synthetic; they do not
+represent a realistic corpus distribution, so passing these measurements alone
+would not establish production performance.
+
+```powershell
+cargo run --release -p searchi-core --example performance -- --files 1000000 --documents 100000 --samples 20 --db target/benchmarks/full.db
+```
+
+Use a new database path when seeding. The harness refuses to overwrite an
+existing database. To compare engine versions on the same corpus, use `--reuse true`;
+this validates file/document counts and synthetic root ownership before opening
+and migrating the existing fixture. It records migration time separately and
+reports zero seed time. It seeds at most 5,000 records per transaction, checkpoints
+the WAL, then performs three warmups and 20 measured queries per case. Percentiles
+use the nearest-rank convention; with 20 samples, p95 is the 19th sorted sample.
+The first execution of each query is recorded separately before the remaining
+two warmups; this is first use of that query on the current engine, not a cold
+OS disk-cache test. Timing uses a monotonic clock. Results include committed database size and fixture
+seed time, which must not be described as document extraction throughput.
+
+Cases cover an exact filename near the end of the corpus, a broad filename
+substring, a unique content token, and a broad content token. All queries use the
+public search API with limit 20, including actual source-passage selection. Target
+p95 is below 50 ms for filenames and below 150 ms for keyword search. A result
+above either target is explicitly marked as a failure. Database file bytes exclude
+temporary WAL/SHM sidecars and process memory.
+
+## Reference environment
+
+- Date: 2026-10-12.
+- Windows VM: Windows 11 Pro, version 10.0.26300.
+- CPU name exposed by VM: AMD Ryzen 7 7800X3D 8-Core Processor.
+- VM allocation: 4 logical processors, 10,666,553,344 bytes RAM (about 9.93 GiB).
+- Rust release build, bundled SQLite, CPU only.
+- Toolchain: rustc 1.99.0 (b940084d7 2026-09-28), cargo 1.99.0.
+- Engine baseline: `a9b6592330d88745eadf2a748bb3a65f62830590`, schema 4.
+- Storage device and host scheduling contention are not characterized.
+
+Run results are stored in `results/`. Linux and macOS measurements are pending.
+No cross-platform acceptance claim can be inferred from the Windows VM results.
+
+## Baseline result
+
+The full acceptance fixture fails all four latency targets. This is a verified
+performance gap, not a production-ready retrieval result.
+
+| Query case | p50 | p95 | p95 target | Result |
+| --- | ---: | ---: | ---: | --- |
+| Exact filename | 404.76 ms | 633.82 ms | <50 ms | Failed |
+| Broad filename substring | 3,385.76 ms | 4,131.48 ms | <50 ms | Failed |
+| Unique content token | 311.63 ms | 355.78 ms | <150 ms | Failed |
+| Broad keyword | 2,186.24 ms | 2,746.91 ms | <150 ms | Failed |
+
+The 1M/100K synthetic fixture took 277.23 seconds to seed and its checkpointed
+database occupies 309,338,112 bytes (about 295 MiB). Smaller 10K/1K smoke results
+passed the same targets, showing why small-corpus tests must not be used as proof
+of scalability. Neither seed time measures document extraction throughput.
+
+The baseline implementation scanned file metadata for substring matching and used
+per-result passage lookups. Its measured behavior warranted a dedicated filename
+substring index, candidate-first FTS retrieval, and batched passage selection.
+The following section records the resulting implementation and measurements.
+
+## Optimized result, schema 7
+
+The unchanged full fixture was migrated and reused. All four **warm** target
+measurements pass in this Windows CPU run. This does not establish production
+acceptance for all workloads or operating systems.
+
+| Query case | Baseline p95 | Optimized p50 | Optimized p95 | First use | Warm target |
+| --- | ---: | ---: | ---: | ---: | --- |
+| Exact filename | 633.82 ms | 11.66 ms | 13.17 ms | 279.73 ms | Passed |
+| Broad filename substring | 4,131.48 ms | 0.39 ms | 1.68 ms | 3.34 ms | Passed |
+| Unique content token | 355.78 ms | 0.28 ms | 0.39 ms | 50.08 ms | Passed |
+| Broad keyword | 2,746.91 ms | 97.92 ms | 111.54 ms | 101.35 ms | Passed |
+
+The updated checkpointed database is 634,449,920 bytes (about 605 MiB). The
+improvement trades additional indexes and disk space for retrieval latency.
+The initial schema-4-to-5 backfill took 19.65 seconds on this fixture, as recorded
+in the intermediate report. The final report's 3.84 ms `migration_seconds` is an
+already-current-schema open, not a measurement of the initial index backfill.
+Indexed equality, trigram candidate retrieval, content-only FTS, and a single
+batched source lookup replaced full metadata joins and per-result FTS scans.
+Rarest-trigram vocabulary counts are cached in the reader engine. The first-use
+exact result exceeds the warm target and is explicitly retained as an outstanding
+latency issue; opening a fresh reader for every query discards this cache.
+
+`windows-cpu-full-optimized-v7.json` is the final measurement. Earlier schema 5/6
+reports are intermediate experiments and retain failed exact-query measurements.
+The final run did not overlap another benchmark process. Dedicated host isolation
+is still not established. Query distribution and filters are not represented by
+this four-case fixture; restrictive broad filters and sub-three-character searches
+remain potentially slow, while correctness tests ensure late matching files are
+not lost through arbitrary pre-filter caps.
@@ -0,0 +1,47 @@
+{
+  "fixture": "synthetic SQLite metadata and text; no filesystem extraction or semantic inference",
+  "files": 1000000,
+  "documents": 100000,
+  "seed_seconds": 0.0,
+  "migration_seconds": 0.0055438,
+  "fixture_reused": true,
+  "database_bytes": 634449920,
+  "database_path": "target/benchmarks/full.db",
+  "architecture": "x86_64",
+  "os": "windows",
+  "cpu_only": true,
+  "queries": [
+    {
+      "query": "invoice_0999999.txt",
+      "samples": 20,
+      "p50_ms": 120.81269999999999,
+      "p95_ms": 137.9349,
+      "target_p95_ms": 50.0,
+      "target_met": false
+    },
+    {
+      "query": "invoice_",
+      "samples": 20,
+      "p50_ms": 0.5431,
+      "p95_ms": 0.7148,
+      "target_p95_ms": 50.0,
+      "target_met": true
+    },
+    {
+      "query": "marker0099999",
+      "samples": 20,
+      "p50_ms": 0.5455,
+      "p95_ms": 0.6604,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    },
+    {
+      "query": "cancellation",
+      "samples": 20,
+      "p50_ms": 98.23469999999999,
+      "p95_ms": 103.2545,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    }
+  ]
+}
@@ -0,0 +1,51 @@
+{
+  "fixture": "synthetic SQLite metadata and text; no filesystem extraction or semantic inference",
+  "files": 1000000,
+  "documents": 100000,
+  "seed_seconds": 0.0,
+  "migration_seconds": 0.0038446,
+  "fixture_reused": true,
+  "database_bytes": 634449920,
+  "database_path": "target/benchmarks/full.db",
+  "architecture": "x86_64",
+  "os": "windows",
+  "cpu_only": true,
+  "queries": [
+    {
+      "query": "invoice_0999999.txt",
+      "samples": 20,
+      "p50_ms": 11.659600000000001,
+      "first_query_ms": 279.7262,
+      "p95_ms": 13.167399999999999,
+      "target_p95_ms": 50.0,
+      "target_met": true
+    },
+    {
+      "query": "invoice_",
+      "samples": 20,
+      "p50_ms": 0.3923,
+      "first_query_ms": 3.3355,
+      "p95_ms": 1.6800000000000002,
+      "target_p95_ms": 50.0,
+      "target_met": true
+    },
+    {
+      "query": "marker0099999",
+      "samples": 20,
+      "p50_ms": 0.2847,
+      "first_query_ms": 50.083000000000006,
+      "p95_ms": 0.3948,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    },
+    {
+      "query": "cancellation",
+      "samples": 20,
+      "p50_ms": 97.9169,
+      "first_query_ms": 101.3489,
+      "p95_ms": 111.5428,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    }
+  ]
+}
@@ -0,0 +1,47 @@
+{
+  "fixture": "synthetic SQLite metadata and text; no filesystem extraction or semantic inference",
+  "files": 1000000,
+  "documents": 100000,
+  "seed_seconds": 0.0,
+  "migration_seconds": 19.651645,
+  "fixture_reused": true,
+  "database_bytes": 633917440,
+  "database_path": "target/benchmarks/full.db",
+  "architecture": "x86_64",
+  "os": "windows",
+  "cpu_only": true,
+  "queries": [
+    {
+      "query": "invoice_0999999.txt",
+      "samples": 20,
+      "p50_ms": 115.7827,
+      "p95_ms": 127.39,
+      "target_p95_ms": 50.0,
+      "target_met": false
+    },
+    {
+      "query": "invoice_",
+      "samples": 20,
+      "p50_ms": 0.4833,
+      "p95_ms": 1.734,
+      "target_p95_ms": 50.0,
+      "target_met": true
+    },
+    {
+      "query": "marker0099999",
+      "samples": 20,
+      "p50_ms": 0.3963,
+      "p95_ms": 0.5996,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    },
+    {
+      "query": "cancellation",
+      "samples": 20,
+      "p50_ms": 105.1206,
+      "p95_ms": 112.6383,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    }
+  ]
+}
@@ -0,0 +1,45 @@
+{
+  "fixture": "synthetic SQLite metadata and text; no filesystem extraction or semantic inference",
+  "files": 1000000,
+  "documents": 100000,
+  "seed_seconds": 277.2314138,
+  "database_bytes": 309338112,
+  "database_path": "target/benchmarks/full.db",
+  "architecture": "x86_64",
+  "os": "windows",
+  "cpu_only": true,
+  "queries": [
+    {
+      "query": "invoice_0999999.txt",
+      "samples": 20,
+      "p50_ms": 404.75899999999996,
+      "p95_ms": 633.8166,
+      "target_p95_ms": 50.0,
+      "target_met": false
+    },
+    {
+      "query": "invoice_",
+      "samples": 20,
+      "p50_ms": 3385.7561,
+      "p95_ms": 4131.4819,
+      "target_p95_ms": 50.0,
+      "target_met": false
+    },
+    {
+      "query": "marker0099999",
+      "samples": 20,
+      "p50_ms": 311.6295,
+      "p95_ms": 355.7767,
+      "target_p95_ms": 150.0,
+      "target_met": false
+    },
+    {
+      "query": "cancellation",
+      "samples": 20,
+      "p50_ms": 2186.2423,
+      "p95_ms": 2746.9136,
+      "target_p95_ms": 150.0,
+      "target_met": false
+    }
+  ]
+}
@@ -0,0 +1,47 @@
+{
+  "fixture": "synthetic SQLite metadata and text; no filesystem extraction or semantic inference",
+  "files": 10000,
+  "documents": 1000,
+  "seed_seconds": 0.0,
+  "migration_seconds": 0.2795255,
+  "fixture_reused": true,
+  "database_bytes": 6320128,
+  "database_path": "target/benchmarks/small.db",
+  "architecture": "x86_64",
+  "os": "windows",
+  "cpu_only": true,
+  "queries": [
+    {
+      "query": "invoice_0009999.txt",
+      "samples": 20,
+      "p50_ms": 1.9414,
+      "p95_ms": 2.3285,
+      "target_p95_ms": 50.0,
+      "target_met": true
+    },
+    {
+      "query": "invoice_",
+      "samples": 20,
+      "p50_ms": 0.3281,
+      "p95_ms": 1.2427,
+      "target_p95_ms": 50.0,
+      "target_met": true
+    },
+    {
+      "query": "marker0000999",
+      "samples": 20,
+      "p50_ms": 0.2561,
+      "p95_ms": 0.40259999999999996,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    },
+    {
+      "query": "cancellation",
+      "samples": 20,
+      "p50_ms": 1.2418,
+      "p95_ms": 1.7393,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    }
+  ]
+}
@@ -0,0 +1,45 @@
+{
+  "fixture": "synthetic SQLite metadata and text; no filesystem extraction or semantic inference",
+  "files": 10000,
+  "documents": 1000,
+  "seed_seconds": 0.3428201,
+  "database_bytes": 3207168,
+  "database_path": "target/benchmarks/small.db",
+  "architecture": "x86_64",
+  "os": "windows",
+  "cpu_only": true,
+  "queries": [
+    {
+      "query": "invoice_0009999.txt",
+      "samples": 20,
+      "p50_ms": 3.0624,
+      "p95_ms": 3.8085,
+      "target_p95_ms": 50.0,
+      "target_met": true
+    },
+    {
+      "query": "invoice_",
+      "samples": 20,
+      "p50_ms": 20.8847,
+      "p95_ms": 28.4805,
+      "target_p95_ms": 50.0,
+      "target_met": true
+    },
+    {
+      "query": "marker0000999",
+      "samples": 20,
+      "p50_ms": 1.7667,
+      "p95_ms": 2.7321,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    },
+    {
+      "query": "cancellation",
+      "samples": 20,
+      "p50_ms": 12.2173,
+      "p95_ms": 16.5439,
+      "target_p95_ms": 150.0,
+      "target_met": true
+    }
+  ]
+}
@@ -0,0 +1,200 @@
+//! CPU-only synthetic retrieval benchmark. This does not measure filesystem
+//! scanning, extraction, embedding inference, or desktop rendering performance.
+use anyhow::{bail, Context, Result};
+use rusqlite::{params, Connection};
+use searchi_core::SearchIndex;
+use serde::Serialize;
+use std::{env, fs, path::PathBuf, time::Instant};
+
+#[derive(Serialize)]
+struct Measurements {
+    query: String,
+    samples: usize,
+    p50_ms: f64,
+    first_query_ms: f64,
+    p95_ms: f64,
+    target_p95_ms: f64,
+    target_met: bool,
+}
+
+#[derive(Serialize)]
+struct Report {
+    fixture: &'static str,
+    files: usize,
+    documents: usize,
+    seed_seconds: f64,
+    migration_seconds: f64,
+    fixture_reused: bool,
+    database_bytes: u64,
+    database_path: String,
+    architecture: &'static str,
+    os: &'static str,
+    cpu_only: bool,
+    queries: Vec<Measurements>,
+}
+
+fn main() -> Result<()> {
+    let arguments = env::args().skip(1).collect::<Vec<_>>();
+    let mut files = 10_000usize;
+    let mut documents = 1_000usize;
+    let mut samples = 20usize;
+    let mut database = PathBuf::from("target/benchmarks/synthetic.db");
+    let mut reuse = false;
+    for pair in arguments.chunks(2) {
+        if pair.len() != 2 {
+            bail!("usage: performance --files N --documents N --samples N --db NEW_PATH");
+        }
+        match pair[0].as_str() {
+            "--files" => files = pair[1].parse()?,
+            "--documents" => documents = pair[1].parse()?,
+            "--samples" => samples = pair[1].parse()?,
+            "--db" => database = PathBuf::from(&pair[1]),
+            "--reuse" => reuse = pair[1].parse()?,
+            option => bail!("unknown argument {option}"),
+        }
+    }
+    if files == 0
+        || files > 2_000_000
+        || documents == 0
+        || documents > files
+        || !(5..=1000).contains(&samples)
+    {
+        bail!("files must be1..2000000, documents1..files, samples5..1000");
+    }
+    if database.exists() && !reuse {
+        bail!(
+            "benchmark database must be a new path: {}",
+            database.display()
+        );
+    }
+    if reuse && !database.exists() {
+        bail!("reuse requires an existing synthetic fixture");
+    }
+    if let Some(parent) = database.parent() {
+        fs::create_dir_all(parent)?;
+    }
+    // Seed through the actual versioned schema and its normal indexing triggers.
+    if reuse {
+        let connection = Connection::open(&database)?;
+        let (actual_files, actual_documents, foreign_roots): (usize,usize,usize) = connection.query_row(
+            "SELECT count(*),count(NULLIF(content,'')),sum(root!='synthetic://benchmark') FROM files",[],
+            |row|Ok((row.get(0)?,row.get(1)?,row.get(2)?)))?;
+        if (actual_files, actual_documents, foreign_roots) != (files, documents, 0) {
+            bail!("existing fixture does not match configured synthetic corpus");
+        }
+    }
+    let migration_started = Instant::now();
+    let index = SearchIndex::open(&database)?;
+    drop(index);
+    let migration_seconds = migration_started.elapsed().as_secs_f64();
+    let start = Instant::now();
+    let mut connection = Connection::open(&database)?;
+    connection.execute_batch("PRAGMA synchronous=NORMAL;")?;
+    if !reuse {
+        for batch_start in (0..files).step_by(5000) {
+            let transaction = connection.transaction()?;
+            {
+                let mut statement = transaction.prepare(
+                    "INSERT INTO files(path,root,filename,content,size,mtime,extension)
+                VALUES (?1,'synthetic://benchmark',?2,?3,?4,0,'txt')",
+                )?;
+                let mut passage = transaction.prepare(
+                    "INSERT INTO passages(text,file_path,kind,number) VALUES (?1,?2,'paragraph',1)",
+                )?;
+                for number in batch_start..(batch_start + 5000).min(files) {
+                    let filename = format!("invoice_{number:07}.txt");
+                    let path = format!(
+                        "synthetic://benchmark/folder_{:04}/{filename}",
+                        number % 1000
+                    );
+                    let content = if number < documents {
+                        format!("Supplier termination cancellation clause budget analysis marker{number:07}")
+                    } else {
+                        String::new()
+                    };
+                    statement.execute(params![path, filename, content, content.len() as i64])?;
+                    if number < documents {
+                        passage.execute(params![content, path])?;
+                    }
+                }
+            }
+            transaction.commit()?;
+            if batch_start % 100_000 == 0 {
+                eprintln!(
+                    "seeded {} / {files} synthetic file records",
+                    (batch_start + 5000).min(files)
+                );
+            }
+        }
+    }
+    connection.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);")?;
+    drop(connection);
+    let seed_seconds = if reuse {
+        0.0
+    } else {
+        start.elapsed().as_secs_f64()
+    };
+    let database_bytes = fs::metadata(&database)?.len();
+    let index = SearchIndex::open(&database)?;
+    let exact = format!("invoice_{:07}.txt", files - 1);
+    let unique_content = format!("marker{:07}", documents - 1);
+    let cases = [
+        (exact, 50.0),
+        ("invoice_".into(), 50.0),
+        (unique_content, 150.0),
+        ("cancellation".into(), 150.0),
+    ];
+    let mut queries = Vec::new();
+    for (query, target) in cases {
+        let first_started = Instant::now();
+        index.search(&query, 20)?;
+        let first_query_ms = first_started.elapsed().as_secs_f64() * 1000.0;
+        for _ in 0..2 {
+            index.search(&query, 20)?;
+        }
+        let mut durations = Vec::with_capacity(samples);
+        for _ in 0..samples {
+            let started = Instant::now();
+            let results = index
+                .search(&query, 20)
+                .with_context(|| format!("benchmark query {query}"))?;
+            if results.is_empty() {
+                bail!("synthetic query {query} unexpectedly returned no results");
+            }
+            std::hint::black_box(results);
+            durations.push(started.elapsed().as_secs_f64() * 1000.0);
+        }
+        durations.sort_by(f64::total_cmp);
+        let percentile = |fraction: f64| {
+            durations[((durations.len() as f64 * fraction).ceil() as usize).saturating_sub(1)]
+        };
+        let p95_ms = percentile(0.95);
+        eprintln!("query {query:?}: p95 {p95_ms:.3} ms; target < {target:.0} ms");
+        queries.push(Measurements {
+            query,
+            samples,
+            p50_ms: percentile(0.5),
+            first_query_ms,
+            p95_ms,
+            target_p95_ms: target,
+            target_met: p95_ms < target,
+        });
+    }
+    let report = Report {
+        fixture:
+            "synthetic SQLite metadata and text; no filesystem extraction or semantic inference",
+        files,
+        documents,
+        seed_seconds,
+        migration_seconds,
+        fixture_reused: reuse,
+        database_bytes,
+        database_path: database.display().to_string(),
+        architecture: env::consts::ARCH,
+        os: env::consts::OS,
+        cpu_only: true,
+        queries,
+    };
+    println!("{}", serde_json::to_string_pretty(&report)?);
+    Ok(())
+}
@@ -5,16 +5,18 @@
 //! Worker isolation remains a production milestone.
 mod extraction;
 use anyhow::{bail, Context, Result};
-use rusqlite::{params, Connection, OptionalExtension};
+use rusqlite::{params, params_from_iter, types::Value, Connection, OptionalExtension};
 use serde::Serialize;
 use std::{
+    cell::RefCell,
+    collections::{BTreeSet, HashMap},
    fs,
    path::{Path, PathBuf},
    time::UNIX_EPOCH,
 };
 use walkdir::WalkDir;

-const SCHEMA_VERSION: i64 = 4;
+const SCHEMA_VERSION: i64 = 7;
 const MAX_RESULTS: usize = 1000;

 #[derive(Debug, Default, Serialize)]
@@ -92,6 +94,7 @@ pub struct IndexStats {
 pub struct SearchIndex {
    connection: Connection,
    database_path: Option<PathBuf>,
+    gram_frequencies: RefCell<HashMap<String, i64>>,
 }

 impl SearchIndex {
@@ -159,16 +162,66 @@ impl SearchIndex {
                CREATE TRIGGER IF NOT EXISTS files_passages_delete AFTER DELETE ON files BEGIN
                    DELETE FROM passages WHERE file_path=old.path; END;",
            )?;
-            if version != 4 {
+            if version > 0 && version < 4 {
                // Re-extract records from older schemas to populate genuine locations.
                initialization.execute("UPDATE files SET mtime=-1", [])?;
            }
-            initialization.execute_batch("PRAGMA user_version=4;")?;
+            if version < 5 {
+                initialization.execute_batch("CREATE TABLE passages_new(
+                    id INTEGER PRIMARY KEY, text TEXT NOT NULL, file_path TEXT NOT NULL,
+                    kind TEXT NOT NULL, number INTEGER NOT NULL);
+                    INSERT INTO passages_new SELECT rowid,text,file_path,kind,number FROM passages;
+                    DROP TRIGGER files_passages_delete;
+                    DROP TABLE passages;
+                    ALTER TABLE passages_new RENAME TO passages;
+                    CREATE TRIGGER files_passages_delete AFTER DELETE ON files BEGIN
+                        DELETE FROM passages WHERE file_path=old.path; END;
+                    CREATE INDEX passages_file ON passages(file_path,id);
+                    CREATE VIRTUAL TABLE passage_search USING fts5(text,content='passages',content_rowid='id',tokenize='unicode61');
+                    CREATE TRIGGER passages_insert AFTER INSERT ON passages BEGIN
+                        INSERT INTO passage_search(rowid,text) VALUES(new.id,new.text); END;
+                    CREATE TRIGGER passages_delete AFTER DELETE ON passages BEGIN
+                        INSERT INTO passage_search(passage_search,rowid,text) VALUES('delete',old.id,old.text); END;
+                    CREATE TRIGGER passages_update AFTER UPDATE ON passages BEGIN
+                        INSERT INTO passage_search(passage_search,rowid,text) VALUES('delete',old.id,old.text);
+                        INSERT INTO passage_search(rowid,text) VALUES(new.id,new.text); END;
+                    INSERT INTO passage_search(passage_search) VALUES('rebuild');
+                    CREATE INDEX files_filename ON files(lower(filename),id);
+                    CREATE VIRTUAL TABLE names USING fts5(filename,path,content='files',content_rowid='id',tokenize='trigram');
+                    CREATE TRIGGER files_names_insert AFTER INSERT ON files BEGIN
+                        INSERT INTO names(rowid,filename,path) VALUES(new.id,new.filename,new.path); END;
+                    CREATE TRIGGER files_names_delete AFTER DELETE ON files BEGIN
+                        INSERT INTO names(names,rowid,filename,path) VALUES('delete',old.id,old.filename,old.path); END;
+                    CREATE TRIGGER files_names_update AFTER UPDATE ON files BEGIN
+                        INSERT INTO names(names,rowid,filename,path) VALUES('delete',old.id,old.filename,old.path);
+                        INSERT INTO names(rowid,filename,path) VALUES(new.id,new.filename,new.path); END;
+                    INSERT INTO names(names) VALUES('rebuild');
+                    PRAGMA user_version=5;")?;
+            }
+            if version < 6 {
+                initialization.execute_batch("CREATE VIRTUAL TABLE content_search USING fts5(content,
+                    content='files',content_rowid='id',tokenize='unicode61');
+                    CREATE TRIGGER files_content_insert AFTER INSERT ON files WHEN new.content!='' BEGIN
+                        INSERT INTO content_search(rowid,content) VALUES(new.id,new.content); END;
+                    CREATE TRIGGER files_content_delete AFTER DELETE ON files WHEN old.content!='' BEGIN
+                        INSERT INTO content_search(content_search,rowid,content) VALUES('delete',old.id,old.content); END;
+                    CREATE TRIGGER files_content_update AFTER UPDATE ON files BEGIN
+                        INSERT INTO content_search(content_search,rowid,content)
+                            SELECT 'delete',old.id,old.content WHERE old.content!='';
+                        INSERT INTO content_search(rowid,content) SELECT new.id,new.content WHERE new.content!=''; END;
+                    INSERT INTO content_search(rowid,content) SELECT id,content FROM files WHERE content!='';
+                    PRAGMA user_version=6;")?;
+            }
+            initialization.execute_batch(
+                "CREATE VIRTUAL TABLE IF NOT EXISTS names_vocab USING fts5vocab(names,'row');
+                PRAGMA user_version=7;",
+            )?;
            initialization.commit()?;
        }
        Ok(Self {
            connection,
            database_path,
+            gram_frequencies: RefCell::new(HashMap::new()),
        })
    }

@@ -300,13 +353,14 @@ impl SearchIndex {
        )?;
        transaction.execute("DELETE FROM seen_paths", [])?;
        transaction.commit()?;
+        self.gram_frequencies.borrow_mut().clear();
        Ok(report)
    }

    /// Literal input is escaped before FTS parsing; user text cannot inject operators.
    /// Exact filenames precede partial filenames, which precede content matches.
-    /// Filename substring matching currently scans metadata; a dedicated substring
-    /// index is required before claiming million-file latency targets.
+    /// Substrings of at least three Unicode characters use a trigram index.
+    /// Shorter substrings retain complete matching using a slower metadata scan.
    pub fn search(&self, query: &str, limit: usize) -> Result<Vec<SearchResult>> {
        self.search_with_options(query, limit, &SearchOptions::default())
    }
@@ -370,65 +424,237 @@ impl SearchIndex {
            .map(|term| format!("\"{}\"", term.replace('"', "\"\"")))
            .collect::<Vec<_>>()
            .join(" AND ");
-        let mut statement = self.connection.prepare(
-            "WITH hits AS (
-            SELECT rowid AS id, bm25(search, 8.0, 2.0, 1.0) AS rank,
-                snippet(search, 2, '', '', ' … ', 24) AS excerpt
-            FROM search WHERE search MATCH ?1)
-            SELECT f.path, f.filename, COALESCE(h.excerpt, ''), COALESCE(h.rank, 0),
-                CASE WHEN lower(f.filename)=lower(?2) THEN 0
-                WHEN instr(lower(f.filename), lower(?2)) > 0 THEN 1 ELSE 2 END AS tier, f.extraction_error
-            FROM files f LEFT JOIN hits h ON h.id=f.id
-            WHERE (h.id IS NOT NULL OR instr(lower(f.filename), lower(?2)) > 0)
-            AND (?4 IS NULL OR substr(f.path,1,length(?4))=?4)
-            AND (?5 IS NULL OR f.extension=?5)
-            AND (?6 IS NULL OR f.size>=?6) AND (?7 IS NULL OR f.size<=?7)
-            AND (?8 IS NULL OR f.mtime/1000000>=?8) AND (?9 IS NULL OR f.mtime/1000000<=?9)
-            ORDER BY tier, h.rank, f.path LIMIT ?3",
-        )?;
-        let mut results = statement
-            .query_map(
-                params![
-                    fts,
-                    query,
-                    limit.min(MAX_RESULTS) as i64,
-                    folder_prefix,
-                    extension,
-                    min_size,
-                    max_size,
-                    options.modified_after_unix_ms,
-                    options.modified_before_unix_ms
-                ],
-                |row| {
-                    let tier: i64 = row.get(4)?;
-                    Ok(SearchResult {
-                        path: row.get(0)?,
-                        filename: row.get(1)?,
-                        snippet: row.get(2)?,
-                        score: -row.get::<_, f64>(3)?,
-                        match_kind: match tier {
-                            0 => "exact_filename",
-                            1 => "filename",
-                            _ => "content",
+        let limit = limit.min(MAX_RESULTS);
+        let filters = " AND (?3 IS NULL OR substr(f.path,1,length(?3))=?3)
+            AND (?4 IS NULL OR f.extension=?4)
+            AND (?5 IS NULL OR f.size>=?5) AND (?6 IS NULL OR f.size<=?6)
+            AND (?7 IS NULL OR f.mtime/1000000>=?7) AND (?8 IS NULL OR f.mtime/1000000<=?8)";
+        let filter_values = [
+            folder_prefix.map_or(Value::Null, Value::Text),
+            extension.map_or(Value::Null, Value::Text),
+            min_size.map_or(Value::Null, Value::Integer),
+            max_size.map_or(Value::Null, Value::Integer),
+            options
+                .modified_after_unix_ms
+                .map_or(Value::Null, Value::Integer),
+            options
+                .modified_before_unix_ms
+                .map_or(Value::Null, Value::Integer),
+        ];
+        let mut results = Vec::new();
+        let mut selected = Vec::<i64>::new();
+        // Each lane filters its entire candidate set before LIMIT. There is no
+        // hidden pre-filter candidate cap that can hide later matching files.
+        let trigram = query.chars().count() >= 3;
+        // A long FTS phrase may walk common leading trigram postings even when
+        // later trigrams are rare. Use the rarest known trigram as a candidate
+        // superset, then verify the whole literal substring. The 256-term lookup
+        // budget can only enlarge candidates; it never caps results. Frequency
+        // lookups are cached because FTS vocabulary counts have a first-use cost.
+        let characters = query.chars().collect::<Vec<_>>();
+        let grams = characters
+            .windows(3)
+            .map(|window| window.iter().collect::<String>())
+            .collect::<BTreeSet<_>>()
+            .into_iter()
+            .take(256)
+            .collect::<Vec<_>>();
+        let mut vocabulary = self
+            .connection
+            .prepare("SELECT doc FROM names_vocab WHERE term=?1")?;
+        let mut cache = self.gram_frequencies.borrow_mut();
+        if cache.len() + grams.len() > 4096 {
+            cache.clear();
        }
-                        .to_owned(),
+        let mut rare_first = grams
+            .into_iter()
+            .map(|gram| {
+                let term = gram.to_lowercase();
+                let frequency = if let Some(frequency) = cache.get(&term) {
+                    *frequency
+                } else {
+                    let frequency = vocabulary
+                        .query_row([&term], |row| row.get::<_, i64>(0))
+                        .optional()?
+                        .unwrap_or(0);
+                    cache.insert(term, frequency);
+                    frequency
+                };
+                Ok((frequency, gram))
+            })
+            .collect::<Result<Vec<_>>>()?;
+        rare_first.sort();
+        drop(cache);
+        let grams = rare_first
+            .into_iter()
+            .take(1)
+            .map(|(_, gram)| format!("\"{}\"", gram.replace('"', "\"\"")))
+            .collect::<Vec<_>>()
+            .join(" AND ");
+        let mut lanes = vec![
+            (
+                "exact_filename",
+                "files f".to_owned(),
+                "lower(f.filename)=lower(?1)".to_owned(),
+                query.to_owned(),
+                "f.id",
+            ),
+            (
+                "filename",
+                if trigram {
+                    "names CROSS JOIN files f ON f.id=names.rowid".into()
+                } else {
+                    "files f".into()
+                },
+                if trigram {
+                    "names MATCH ?1 AND instr(lower(f.filename),lower(?2))>0 AND lower(f.filename)!=lower(?2)".into()
+                } else {
+                    "instr(lower(f.filename),lower(?1))>0 AND lower(f.filename)!=lower(?2)".into()
+                },
+                if trigram {
+                    format!("filename:({grams})")
+                } else {
+                    query.into()
+                },
+                if trigram { "names.rowid" } else { "f.id" },
+            ),
+            (
+                "path",
+                if trigram {
+                    "names CROSS JOIN files f ON f.id=names.rowid".into()
+                } else {
+                    "files f".into()
+                },
+                if trigram {
+                    "names MATCH ?1 AND instr(lower(f.path),lower(?2))>0 AND instr(lower(f.filename),lower(?2))=0".into()
+                } else {
+                    "instr(lower(f.path),lower(?1))>0 AND instr(lower(f.filename),lower(?2))=0"
+                        .into()
+                },
+                if trigram {
+                    format!("path:({grams})")
+                } else {
+                    query.into()
+                },
+                if trigram { "names.rowid" } else { "f.id" },
+            ),
+            (
+                "content",
+                "content_search CROSS JOIN files f ON f.id=content_search.rowid".into(),
+                "content_search MATCH ?1".into(),
+                fts.clone(),
+                "rank,f.id",
+            ),
+        ];
+        // Single-token names/identifiers use literal substring matching. A
+        // multiword query can also match token-separated names such as annual_report.
+        if query.split_whitespace().count() > 1 {
+            lanes.push((
+                "name_terms",
+                "search CROSS JOIN files f ON f.id=search.rowid".into(),
+                "search MATCH ?1 AND rank MATCH 'bm25(8.0,2.0,1.0)'".into(),
+                fts.clone(),
+                "rank,f.id",
+            ));
+        }
+        for (kind, from, condition, expression, ordering) in lanes {
+            if results.len() == limit {
+                break;
+            }
+            let exclusions = if selected.is_empty() {
+                String::new()
+            } else {
+                format!(
+                    " AND f.id NOT IN ({})",
+                    selected
+                        .iter()
+                        .map(i64::to_string)
+                        .collect::<Vec<_>>()
+                        .join(",")
+                )
+            };
+            let rank = if matches!(kind, "content" | "name_terms") {
+                "rank"
+            } else {
+                "0.0"
+            };
+            let snippet = if kind == "content" {
+                "snippet(content_search,0,'','',' … ',24)"
+            } else if kind == "name_terms" {
+                "snippet(search,2,'','',' … ',24)"
+            } else {
+                "''"
+            };
+            let sql = format!(
+                "SELECT f.id,f.path,f.filename,{snippet},{rank},f.extraction_error FROM {from}
+                WHERE {condition} {filters} {exclusions} ORDER BY {ordering} LIMIT ?9"
+            );
+            let mut values = vec![Value::Text(expression), Value::Text(query.into())];
+            values.extend(filter_values.iter().cloned());
+            values.push(Value::Integer((limit - results.len()) as i64));
+            let mut statement = self.connection.prepare(&sql)?;
+            let matches = statement
+                .query_map(params_from_iter(values), |row| {
+                    Ok((
+                        row.get::<_, i64>(0)?,
+                        SearchResult {
+                            path: row.get(1)?,
+                            filename: row.get(2)?,
+                            snippet: row.get(3)?,
+                            score: -row.get::<_, f64>(4)?,
+                            match_kind: kind.into(),
                            source: None,
                            extraction_error: row.get(5)?,
-                    })
                        },
-            )?
+                    ))
+                })?
                .collect::<rusqlite::Result<Vec<_>>>()?;
-        let mut passages = self.connection.prepare("SELECT kind,number,snippet(passages,0,'','',' … ',24)
-            FROM passages WHERE passages MATCH ?1 AND file_path=?2 ORDER BY bm25(passages),rowid LIMIT 1")?;
+            for (id, result) in matches {
+                selected.push(id);
+                results.push(result);
+            }
+        }
+        if !results.is_empty() {
+            // Ownership has a B-tree index. The rowid IN constraint intersects
+            // the FTS postings with only the actual finalists' passage IDs.
+            let placeholders = (0..results.len())
+                .map(|_| "?")
+                .collect::<Vec<_>>()
+                .join(",");
+            let sql = format!("WITH matches AS MATERIALIZED (
+                SELECT p.file_path,p.kind,p.number,p.id,snippet(passage_search,0,'','',' … ',24) AS excerpt,
+                    bm25(passage_search) AS relevance
+                FROM passage_search JOIN passages p ON p.id=passage_search.rowid
+                WHERE passage_search MATCH ? AND passage_search.rowid IN (
+                    SELECT id FROM passages WHERE file_path IN ({placeholders})))
+                SELECT file_path,kind,number,excerpt FROM (
+                    SELECT *,row_number() OVER(PARTITION BY file_path ORDER BY relevance,id) AS ordinal FROM matches)
+                WHERE ordinal=1");
+            let values = std::iter::once(Value::Text(fts)).chain(
+                results
+                    .iter()
+                    .map(|result| Value::Text(result.path.clone())),
+            );
+            let mut statement = self.connection.prepare(&sql)?;
+            let sources = statement
+                .query_map(params_from_iter(values), |row| {
+                    Ok((
+                        row.get::<_, String>(0)?,
+                        (
+                            SourceLocation {
+                                kind: row.get(1)?,
+                                number: row.get(2)?,
+                            },
+                            row.get::<_, String>(3)?,
+                        ),
+                    ))
+                })?
+                .collect::<rusqlite::Result<HashMap<_, _>>>()?;
            for result in &mut results {
-            let passage: Option<(String, u32, String)> = passages
-                .query_row(params![fts, result.path], |row| {
-                    Ok((row.get(0)?, row.get(1)?, row.get(2)?))
-                })
-                .optional()?;
-            if let Some((kind, number, snippet)) = passage {
-                result.source = Some(SourceLocation { kind, number });
-                result.snippet = snippet;
+                if let Some((source, snippet)) = sources.get(&result.path) {
+                    result.source = Some(source.clone());
+                    result.snippet = snippet.clone();
+                }
            }
        }
        Ok(results)
@@ -0,0 +1,125 @@
+use anyhow::Result;
+use searchi_core::{SearchIndex, SearchOptions};
+use std::fs;
+
+#[test]
+fn filters_do_not_hide_late_matches_in_any_lane() -> Result<()> {
+    let root = tempfile::tempdir()?;
+    let rejected = root.path().join("early");
+    let wanted = root.path().join("late");
+    fs::create_dir(&rejected)?;
+    fs::create_dir(&wanted)?;
+    let db = root.path().join("index.db");
+    let mut index = SearchIndex::open(&db)?;
+    for number in 0..2000 {
+        fs::write(
+            rejected.join(format!("shared_{number:04}.txt")),
+            "shared retrieval term",
+        )?;
+    }
+    index.index_folder(&rejected)?;
+    fs::write(
+        wanted.join("shared_last.md"),
+        "shared retrieval term unique-evidence",
+    )?;
+    index.index_folder(&wanted)?;
+    let options = SearchOptions {
+        folder: Some(wanted.clone()),
+        extension: Some("md".into()),
+        ..Default::default()
+    };
+    assert_eq!(
+        index.search_with_options("shared", 1, &options)?[0].filename,
+        "shared_last.md"
+    );
+    assert_eq!(
+        index.search_with_options("retrieval", 1, &options)?[0].filename,
+        "shared_last.md"
+    );
+    assert_eq!(
+        index.search_with_options("ha", 1, &options)?[0].filename,
+        "shared_last.md"
+    );
+    let result = index.search_with_options("unique-evidence", 1, &options)?;
+    assert!(result[0].source.is_some());
+    assert!(result[0].snippet.contains("unique-evidence"));
+    Ok(())
+}
+
+#[test]
+fn exact_priority_literal_identifiers_and_index_lifecycle() -> Result<()> {
+    let root = tempfile::tempdir()?;
+    fs::create_dir(root.path().join("contracts"))?;
+    fs::write(
+        root.path().join("name_[%].txt"),
+        "identifier ABC-123 and term",
+    )?;
+    fs::write(root.path().join("name_[%].txt.backup"), "term")?;
+    fs::write(
+        root.path().join("contracts").join("other.md"),
+        "name identifier ABC-123",
+    )?;
+    fs::write(root.path().join("annual_report.md"), "unrelated text")?;
+    fs::write(root.path().join("abc_missing_xyz.txt"), "unrelated text")?;
+    let mut index = SearchIndex::open(":memory:")?;
+    index.index_folder(root.path())?;
+    let result = index.search("NAME_[%].TXT", 10)?;
+    assert_eq!(result[0].match_kind, "exact_filename");
+    assert_eq!(result[1].match_kind, "filename");
+    assert_eq!(index.search("contracts", 1)?[0].match_kind, "path");
+    assert_eq!(index.search("ABC-123", 10)?.len(), 2);
+    assert_eq!(
+        index.search("annual report", 1)?[0].filename,
+        "annual_report.md"
+    );
+    assert!(index.search("abc_xyz", 10)?.is_empty());
+    let first = index
+        .search("name", 10)?
+        .into_iter()
+        .map(|row| row.path)
+        .collect::<Vec<_>>();
+    let second = index
+        .search("name", 10)?
+        .into_iter()
+        .map(|row| row.path)
+        .collect::<Vec<_>>();
+    assert_eq!(first, second);
+    fs::rename(
+        root.path().join("name_[%].txt"),
+        root.path().join("replacement.txt"),
+    )?;
+    fs::remove_file(root.path().join("name_[%].txt.backup"))?;
+    index.index_folder(root.path())?;
+    assert!(index.search("name_[%].txt", 10)?.is_empty());
+    assert_eq!(index.search("placement", 1)?[0].filename, "replacement.txt");
+    Ok(())
+}
+
+#[test]
+fn v4_migration_backfills_names_and_preserves_passage_ids() -> Result<()> {
+    let root = tempfile::tempdir()?;
+    let db = root.path().join("old.db");
+    let connection = rusqlite::Connection::open(&db)?;
+    connection.execute_batch("CREATE TABLE files(id INTEGER PRIMARY KEY,path TEXT NOT NULL UNIQUE,
+        root TEXT NOT NULL,filename TEXT NOT NULL,content TEXT NOT NULL,size INTEGER NOT NULL,mtime INTEGER NOT NULL,
+        extension TEXT NOT NULL,extraction_error TEXT);
+        CREATE VIRTUAL TABLE search USING fts5(filename,path,content,content='files',content_rowid='id',tokenize='unicode61');
+        CREATE VIRTUAL TABLE passages USING fts5(text,file_path UNINDEXED,kind UNINDEXED,number UNINDEXED);
+        INSERT INTO files VALUES(41,'/fixture/report.pdf','/fixture','report.pdf','preserved evidence',18,123,'pdf',NULL);
+        INSERT INTO search(search) VALUES('rebuild');
+        INSERT INTO passages(rowid,text,file_path,kind,number) VALUES(88,'preserved evidence','/fixture/report.pdf','page',7);
+        PRAGMA user_version=4;")?;
+    drop(connection);
+    let index = SearchIndex::open(&db)?;
+    assert_eq!(index.search("port.p", 1)?[0].filename, "report.pdf");
+    assert_eq!(
+        index.search("evidence", 1)?[0]
+            .source
+            .as_ref()
+            .unwrap()
+            .number,
+        7
+    );
+    assert_eq!(index.passage_batch(0, 1)?[0].id, 88);
+    Ok(())
+}