searchi/docs/AUTOMATIC-INDEXING.md

Automatic folder updates

The desktop now registers saved folders at startup and reconciles them in the background. Native notifications use pinned notify 8.2.0. They are hints: notify documents missed events, network filesystem limitations and platform-specific editor behavior. Periodic scans remain necessary; a native watch failure keeps polling and reports the issue.

Scheduling and bounds

• At most 64 roots and 128 retained paths per root; overflow becomes a full reconciliation rather than an unbounded event queue.
• A 750 ms quiet period coalesces bursts, with a five-second maximum delay.
• Startup reconciles saved folders; ordinary verification runs every five minutes.
• Busy manual indexing retains queued work. Failed scans retry with backoff from five seconds to one minute. Events arriving during a scan remain pending.
• Read/access-time events are ignored; native rescan flags still take priority.
• Missing roots keep existing records and retry attachment. Successful reattachment after a prior watch requests verification of missed media events.
• Callback panics become a retryable failure. Shutdown signals cancellation, wakes the owned thread and joins it without holding its state mutex.

Scans own separate SQLite connections. They do not hold the foreground search mutex during traversal or extraction. A changed scan prepares its reader and briefly swaps it into the desktop. UI events refresh counts and the active query, reject results from before the change, and request normal semantic indexing. The original input/first-response timing survives an automatic query refresh.

Targeted reconciliation

SearchIndex::index_paths_with_options handles bounded absolute normalized paths. Files update individually; directory hints reconcile only that subtree. Missing paths remove only records within their root and path boundary. Parent/child hints are coalesced. Outside paths, traversal and symlink ancestors are rejected. Read/traversal failure or cancellation rolls back the complete batch.

The cancellable folder and path variants check between entries and before commit. An individual bounded parser input can finish before cancellation is observed. Parser process isolation and bounded shutdown time still require further work.

Absolute exclusions complement directory-name exclusions. The desktop excludes its data, model and worker directories from both scans and notifications. Decoded video frames use owned scratch directories under application data, so temporary frames cannot feed back into the index when a home folder is watched.

Media freshness

Core schema 13 adds files.change_revision, initially zero for existing records. Targeted media hints increment it even when file size and timestamp are unchanged, and remove old media tags. Full native-rescan/overflow hints take the targeted root path; ordinary periodic scans retain the metadata fast path for media.

Semantic schema 8 records the matching file_change_revision on media chunks. An old media vector is ineligible after a hint until genuine byte-verified indexing completes. Unchanged media can reuse completed vectors with the current revision; changed bytes require new Google inference. Metadata probing captures the revision before work and writes only if it still matches, preventing a stale probe from restoring tags after a concurrent event. Generation activation remains guarded by the original corpus revision and actual source SHA validation.

Periodic scans hash supported bounded document inputs, but do not hash every large media file. A same-stat media overwrite with no observed native hint remains a verification limitation; explicit semantic rebuilds verify actual media bytes.

Executed validation and remaining gates

Windows native watcher tests observed real create/update/rename/delete events. The final Windows workspace run passed 123 tests (19 explicitly ignored), and the desktop suite passed 28 tests. Frontend build, 39 tests, formatting and strict workspace/desktop Clippy passed. The real CPU FFmpeg decoder regression passed with frame paths under the configured owned scratch parent. Coordinator tests cover overflow, rescan, busy retention, concurrent events, periodic scheduling, panic recovery and shutdown. Core tests cover same-stat documents/media, subtree and boundary deletion, exclusions, migrations, rollback, concurrent readers and cancellation after a write transaction starts.

The real Google mixed-media CPU regression passed in 83.70 seconds. It verified that a same-stat image hint hides old vectors, changed image bytes cause one new image encode, unchanged hinted image/audio sources reuse vectors, and current revision tokens restore eligibility. This is correctness evidence on a tiny fixture, not relevance quality or search/indexing throughput evidence.

Linux and macOS native watcher execution, root disappearance/reappearance on real removable/network storage, native UI refresh interaction, representative large-corpus contention and power-loss behavior remain open. Full reconciliation still holds a write transaction during extraction and can contend with semantic writes; targeted batches reduce that work but do not establish a universal latency bound. Resource admission currently covers neural inference, not all filesystem, decoder, parser or ANN work.