searchi/docs/AUTOMATIC-INDEXING.md

# Automatic folder updates

The desktop now registers saved folders at startup and reconciles them in the
background. Native notifications use pinned `notify` 8.2.0. They are hints:
[notify documents](https://docs.rs/notify/8.2.0/notify/) missed events, network
filesystem limitations and platform-specific editor behavior. Periodic scans
remain necessary; a native watch failure keeps polling and reports the issue.

## Scheduling and bounds

- At most 64 roots and 128 retained paths per root; overflow becomes a full
  reconciliation rather than an unbounded event queue.
- A 750 ms quiet period coalesces bursts, with a five-second maximum delay.
- Startup reconciles saved folders; ordinary verification runs every five minutes.
- Busy manual indexing retains queued work. Failed scans retry with backoff from
  five seconds to one minute. Events arriving during a scan remain pending.
- Read/access-time events are ignored; native rescan flags still take priority.
- Missing roots keep existing records and retry attachment. Successful
  reattachment after a prior watch requests verification of missed media events.
- Callback panics become a retryable failure. Shutdown signals cancellation,
  wakes the owned thread and joins it without holding its state mutex.

Scans own separate SQLite connections. They do not hold the foreground search
mutex during traversal or extraction. A changed scan prepares its reader and
briefly swaps it into the desktop. UI events refresh counts and the active query,
reject results from before the change, and request normal semantic indexing.
Refresh timing starts at the change event rather than include idle time since
the original input. A bounded numeric snapshot retains the original input
timings separately. Intended selection survives temporary result absence;
explicit navigation changes that intent, and a new query or filter clears it.

## Targeted reconciliation

`SearchIndex::index_paths_with_options` handles bounded absolute normalized paths.
Files update individually; directory hints reconcile only that subtree. Missing
paths remove only records within their root and path boundary. Parent/child hints
are coalesced. Outside paths, traversal and symlink ancestors are rejected.
Read/traversal failure or cancellation rolls back the complete batch.

The cancellable folder and path variants check between entries and before commit.
An individual bounded parser input can finish before cancellation is observed.
Parser process isolation and bounded shutdown time still require further work.

Absolute exclusions complement directory-name exclusions. The desktop excludes
its data, model and worker directories from both scans and notifications.
Decoded video frames use owned scratch directories under application data,
so temporary frames cannot feed back into the index when a home folder is watched.

## Media freshness

Core schema 13 adds `files.change_revision`, initially zero for existing records.
Targeted media hints increment it even when file size and timestamp are unchanged,
and remove old media tags. Full native-rescan/overflow hints take the targeted
root path; ordinary periodic scans retain the metadata fast path for media.

Semantic schema 8 records the matching `file_change_revision` on media chunks.
An old media vector is ineligible after a hint until genuine byte-verified indexing
completes. Unchanged media can reuse completed vectors with the current revision;
changed bytes require new Google inference. Metadata probing captures the revision
before work and writes only if it still matches, preventing a stale probe from
restoring tags after a concurrent event. Generation activation remains guarded
by the original corpus revision and actual source SHA validation.

Periodic scans hash supported bounded document inputs, but do not hash every
large media file. A same-stat media overwrite with no observed native hint remains
a verification limitation; explicit semantic rebuilds verify actual media bytes.

## Executed validation and remaining gates

Windows native watcher tests observed real create/update/rename/delete events.
The final Windows workspace run passed 123 tests (19 explicitly ignored), and
the desktop suite passed 28 tests. Frontend build, 39 tests, formatting and strict
workspace/desktop Clippy passed. The real CPU FFmpeg decoder regression passed
with frame paths under the configured owned scratch parent.
Coordinator tests cover overflow, rescan, busy retention, concurrent events,
periodic scheduling, panic recovery and shutdown. Core tests cover same-stat
documents/media, subtree and boundary deletion, exclusions, migrations, rollback,
concurrent readers and cancellation after a write transaction starts.

The real Google mixed-media CPU regression passed in 83.70 seconds. It verified
that a same-stat image hint hides old vectors, changed image bytes cause one new
image encode, unchanged hinted image/audio sources reuse vectors, and current
revision tokens restore eligibility. This is correctness evidence on a tiny
fixture, not relevance quality or search/indexing throughput evidence.

Current-source native Linux checks also passed: 124 workspace tests, 28 desktop
Release tests including actual inotify create/update/rename/delete, and the
updated real Google mixed-media regression in 67.95 seconds. Native Linux
interaction evidence is in
[LINUX-DESKTOP-VALIDATION.md](LINUX-DESKTOP-VALIDATION.md). Actual watcher-driven
UI change/create/rename/delete passed on `657de3d`. The `8b993b2` native UI retest
confirmed independent refresh clocks, restoration of temporarily hidden
selection, and retention of explicit user choices. A 4.164-second final refresh
sample during background updates remains recorded; this is not a warm p95 pass.

macOS native watcher execution, root disappearance/reappearance on
real removable/network storage, Windows/macOS native UI refresh interaction, representative
large-corpus contention and power-loss behavior remain open. Full reconciliation
still holds a write transaction during extraction and can contend with semantic
writes; targeted batches reduce that work but do not establish a universal
latency bound. Resource admission currently covers neural inference, not all
filesystem, decoder, parser or ANN work.