safetynet/docs/development-observability.md

Development observability

Run npm run debug:collector and verify http://127.0.0.1:4318/health. Start the management service with OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318. Enable browser diagnostics through a normally published signed policy: enabled: true, mode: debug, endpoint above, and debug sampling of 1 for controlled reproduction. Administrators can disable diagnostics centrally; mandatory review still applies.

Reproduce the feature, inspect /api/summary for slow stages and inspect /api/trace/<traceId> for parent/child timings. Browser management requests carry W3C trace context into backend HTTP spans. AI inspection, provider requests, response reading/decoding and library organization have separate backend stages. Use the native tabs smoke fixture to verify real Electron exports and correlation.

The receiver listens only on numeric loopback, rejects foreign origins and hosts, and stores encrypted batches under ignored data/observability. AES-256-GCM keys are protected by Windows DPAPI. The summary contains bounded timing metadata; it is a local developer tool accessible to the same machine's users, not a production SIEM or an authenticated remote service. Recent trace lookup is held in memory and resets on restart. Each encrypted segment has a 50 MiB development limit. The collector retains an active segment and at most two generated retired segments, plus a separately pinned previous archive. Inspect activeBytes, legacyBytes, retiredBytes and rejectedWrites; investigate collection failures instead of assuming exports succeeded. See retention.

Ordinary traces never contain passwords, tokens, prompt bodies or document text. Security audit and restricted encrypted forensic evidence are separate pipelines. Debug mode now exports eight internal inspector stages: inspect, parse, OCR, OCR model initialization, YARA, macro extraction, semantic inference and semantic model initialization. The worker sends bounded in-memory timing records through its existing private pipe; it gains no network capability. Browser/backend import creates child spans with original timestamps under the active request trace. Invalid/off/stale metadata is discarded without changing the verdict. These named stages do not trace every function or native-model kernel. Add named stages when investigating further gaps, without weakening DLP or retrying protected actions.

The repository's AGENTS.md requires this workflow for subsequent development.

Coverage verification, 2026-10-13

The live collector reported 3,147 spans, 2,482 log records and 36 metric records, with zero rejected storage writes. Twenty focused diagnostics, OTLP export, inspector timing, collector validation/retention and failure-path tests passed. These counts are a process snapshot, not a completeness claim.

Agent RPC instrumentation now wraps each request in agent.request; decrypt, authorization, audit, dispatch, projection and encryption run beneath that parent. The browser diagnostic operation allowlist accepts the new name. A real protected RPC test verifies asynchronous parent/child nesting without recording arguments or response content. The combined transport/diagnostics suite passes 38 tests; A subsequent actual Electron run verifies exported parent/child phases linked to an attributable durable agent event; Windows packaged verification remains a separate gate.

The actual native agent audit proof links browser.agent to browser.management and server.request under trace f8e35fd442e718995d9274c646fb8e08. The native upload helper-failure proof links a recording browser.upload.fault span, its lease parent, two logs and the durable audit event under trace e6d26d4edf738a7b4f6f562ab2112f5b. Older traces may be in the protected pinned archive rather than the current in-memory index.

A sampled slow browser.parser scan lasted 1,049 ms but contained only the outer span in the live index. Do not infer which parser or model stage caused that duration from an outer span alone: reproduce with signed debug mode enabled and verify the expected inspector children before attributing a performance issue.

The current individual native proof links principal reader-a to trace 3a0c31cbf53d7808e39a91daf2aaa3d7. Its request took 81.019 ms, the two durable audit phases totaled 77.598 ms and adapter dispatch took 0.313 ms. This single local run points to audit persistence as the dominant stage; it is not a throughput benchmark. Preserve pre-execution durable audit when investigating latency. Do not optimize by removing an audit gate or retrying an action. The proof is in artifacts/individual-agent-native-proof.json. The final Windows package verifier checks current source and runs both shipped legacy and individual DPAPI/PQ clients successfully.

Latest interaction and session measurements

The actual protected interaction proof uses trace 32cbbb58088d193301c5f6ac530284af: 30 spans, verified parent correlation, capture 0.3353 ms, challenge 8.1735 ms, encryption 0.4718 ms, send 62.1817 ms and acknowledgement 0.4234 ms. The backend store stage took 54.2629 ms. Collector rejected writes were zero. These are one local reproduction's measurements, not performance guarantees.

After session-title classification was added, the actual native session inventory proof uses trace 64013e27cfc37bc3792983bc06149c87: request 85.675 ms, session adapter 1.0637 ms and durable audit phases 33.3975 and 47.2472 ms. This fixture did not enable the semantic model; it cannot establish semantic-model latency. The native proof confirms a confidential title is masked while preserving its opaque session ID.

A parallel fixture timed out before enrollment and therefore before signed diagnostics activation. Its isolated rerun and restart reached the shell in 794/806 ms. Safe local startup snapshots were added; the original timeout's cause is unresolved. Do not describe that failure as a traced DLP/model regression. The current Windows package proof and 515-test suite passed before the subsequent export-counter refinement. Ignored proof artifacts remain local.

After the refinement and policy child stages, the final suite passes 522 tests. Real signed policy/bookmark fixtures verify actual OTLP child spans and disabled transitions. Windows build, packaged-runtime verification and the native tabs proof pass. The native receiver attributes exports by actual service.name: browser exports stay at 18 across revocation while backend collection remains independent. The earlier aggregate assertion conflated the two services; its failed run is retained rather than presented as successful. Do not retroactively assign the original extra batch to a service without its original payload.

Foreground gesture development used actual traces to find a second attribution gap: a queue flusher started by one event delivered the next event under the first async context. withCorrelation now validates each durable record's own IDs and audit.deliver creates a recording delivery child under that source parent. Malformed IDs clear unrelated context, and current signed sampling/disable still applies. Real OTLP regression tests and three actual human-event traces prove distinct parents and complete evidence phase correlation. Trace b7783f42197d0cff83957b0ee96e8296 has 17 spans, capture 19.50 ms and send 13.20 ms; 62e09413b9a42f4a58bd7f99485176ac and 545ddac025113e50effed4e9a2ac87ee each also have 17 spans. Collector rejected writes were zero. These are local reproduction timings, not throughput guarantees.