90 lines
5.3 KiB
Markdown
90 lines
5.3 KiB
Markdown
# Automatic folder updates
|
|
|
|
The desktop now registers saved folders at startup and reconciles them in the
|
|
background. Native notifications use pinned `notify` 8.2.0. They are hints:
|
|
[notify documents](https://docs.rs/notify/8.2.0/notify/) missed events, network
|
|
filesystem limitations and platform-specific editor behavior. Periodic scans
|
|
remain necessary; a native watch failure keeps polling and reports the issue.
|
|
|
|
## Scheduling and bounds
|
|
|
|
- At most 64 roots and 128 retained paths per root; overflow becomes a full
|
|
reconciliation rather than an unbounded event queue.
|
|
- A 750 ms quiet period coalesces bursts, with a five-second maximum delay.
|
|
- Startup reconciles saved folders; ordinary verification runs every five minutes.
|
|
- Busy manual indexing retains queued work. Failed scans retry with backoff from
|
|
five seconds to one minute. Events arriving during a scan remain pending.
|
|
- Read/access-time events are ignored; native rescan flags still take priority.
|
|
- Missing roots keep existing records and retry attachment. Successful
|
|
reattachment after a prior watch requests verification of missed media events.
|
|
- Callback panics become a retryable failure. Shutdown signals cancellation,
|
|
wakes the owned thread and joins it without holding its state mutex.
|
|
|
|
Scans own separate SQLite connections. They do not hold the foreground search
|
|
mutex during traversal or extraction. A changed scan prepares its reader and
|
|
briefly swaps it into the desktop. UI events refresh counts and the active query,
|
|
reject results from before the change, and request normal semantic indexing.
|
|
The original input/first-response timing survives an automatic query refresh.
|
|
|
|
## Targeted reconciliation
|
|
|
|
`SearchIndex::index_paths_with_options` handles bounded absolute normalized paths.
|
|
Files update individually; directory hints reconcile only that subtree. Missing
|
|
paths remove only records within their root and path boundary. Parent/child hints
|
|
are coalesced. Outside paths, traversal and symlink ancestors are rejected.
|
|
Read/traversal failure or cancellation rolls back the complete batch.
|
|
|
|
The cancellable folder and path variants check between entries and before commit.
|
|
An individual bounded parser input can finish before cancellation is observed.
|
|
Parser process isolation and bounded shutdown time still require further work.
|
|
|
|
Absolute exclusions complement directory-name exclusions. The desktop excludes
|
|
its data, model and worker directories from both scans and notifications.
|
|
Decoded video frames use owned scratch directories under application data,
|
|
so temporary frames cannot feed back into the index when a home folder is watched.
|
|
|
|
## Media freshness
|
|
|
|
Core schema 13 adds `files.change_revision`, initially zero for existing records.
|
|
Targeted media hints increment it even when file size and timestamp are unchanged,
|
|
and remove old media tags. Full native-rescan/overflow hints take the targeted
|
|
root path; ordinary periodic scans retain the metadata fast path for media.
|
|
|
|
Semantic schema 8 records the matching `file_change_revision` on media chunks.
|
|
An old media vector is ineligible after a hint until genuine byte-verified indexing
|
|
completes. Unchanged media can reuse completed vectors with the current revision;
|
|
changed bytes require new Google inference. Metadata probing captures the revision
|
|
before work and writes only if it still matches, preventing a stale probe from
|
|
restoring tags after a concurrent event. Generation activation remains guarded
|
|
by the original corpus revision and actual source SHA validation.
|
|
|
|
Periodic scans hash supported bounded document inputs, but do not hash every
|
|
large media file. A same-stat media overwrite with no observed native hint remains
|
|
a verification limitation; explicit semantic rebuilds verify actual media bytes.
|
|
|
|
## Executed validation and remaining gates
|
|
|
|
Windows native watcher tests observed real create/update/rename/delete events.
|
|
The final Windows workspace run passed 123 tests (19 explicitly ignored), and
|
|
the desktop suite passed 28 tests. Frontend build, 39 tests, formatting and strict
|
|
workspace/desktop Clippy passed. The real CPU FFmpeg decoder regression passed
|
|
with frame paths under the configured owned scratch parent.
|
|
Coordinator tests cover overflow, rescan, busy retention, concurrent events,
|
|
periodic scheduling, panic recovery and shutdown. Core tests cover same-stat
|
|
documents/media, subtree and boundary deletion, exclusions, migrations, rollback,
|
|
concurrent readers and cancellation after a write transaction starts.
|
|
|
|
The real Google mixed-media CPU regression passed in 83.70 seconds. It verified
|
|
that a same-stat image hint hides old vectors, changed image bytes cause one new
|
|
image encode, unchanged hinted image/audio sources reuse vectors, and current
|
|
revision tokens restore eligibility. This is correctness evidence on a tiny
|
|
fixture, not relevance quality or search/indexing throughput evidence.
|
|
|
|
Linux and macOS native watcher execution, root disappearance/reappearance on
|
|
real removable/network storage, native UI refresh interaction, representative
|
|
large-corpus contention and power-loss behavior remain open. Full reconciliation
|
|
still holds a write transaction during extraction and can contend with semantic
|
|
writes; targeted batches reduce that work but do not establish a universal
|
|
latency bound. Resource admission currently covers neural inference, not all
|
|
filesystem, decoder, parser or ANN work.
|