225 lines
15 KiB
Markdown
225 lines
15 KiB
Markdown
# Development observability
|
|
|
|
Run `npm run debug:collector` and verify `http://127.0.0.1:4318/health`.
|
|
Start the management service with `OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318`.
|
|
Enable browser diagnostics through a normally published signed policy: `enabled: true`,
|
|
`mode: debug`, endpoint above, and debug sampling of 1 for controlled reproduction.
|
|
Administrators can disable diagnostics centrally; mandatory review still applies.
|
|
|
|
Reproduce the feature, inspect `/api/summary` for slow stages and inspect
|
|
`/api/trace/<traceId>` for parent/child timings. Browser management requests carry
|
|
W3C trace context into backend HTTP spans. AI inspection, provider requests,
|
|
response reading/decoding and library organization have separate backend stages.
|
|
Use the native tabs smoke fixture to verify real Electron exports and correlation.
|
|
|
|
The receiver listens only on numeric loopback, rejects foreign origins and hosts,
|
|
and stores encrypted batches under ignored `data/observability`. AES-256-GCM keys
|
|
are protected by Windows DPAPI. The summary contains bounded timing metadata;
|
|
it is a local developer tool accessible to the same machine's users, not a
|
|
production SIEM or an authenticated remote service. Recent trace lookup is held
|
|
in memory and resets on restart. Each encrypted segment has a 50 MiB development
|
|
limit. The collector retains an active segment and at most two generated retired
|
|
segments, plus a separately pinned previous archive. Inspect `activeBytes`,
|
|
`legacyBytes`, `retiredBytes` and `rejectedWrites`; investigate collection failures
|
|
instead of assuming exports succeeded. See [retention](development-collector-retention.md).
|
|
|
|
Ordinary traces never contain passwords, tokens, prompt bodies or document text.
|
|
Security audit and restricted encrypted forensic evidence are separate pipelines.
|
|
Debug mode now exports eight internal inspector stages: inspect, parse, OCR, OCR model initialization, YARA, macro extraction, semantic inference and semantic model initialization. The worker sends bounded in-memory timing records through its existing private pipe; it gains no network capability. Browser/backend import creates child spans with original timestamps under the active request trace. Invalid/off/stale metadata is discarded without changing the verdict. These named stages do not trace every function or native-model kernel. Add named stages when investigating further gaps, without weakening DLP or retrying protected actions.
|
|
|
|
The repository's AGENTS.md requires this workflow for subsequent development.
|
|
|
|
## Coverage verification, 2026-10-13
|
|
|
|
The live collector reported 3,147 spans, 2,482 log records and 36 metric records,
|
|
with zero rejected storage writes. Twenty focused diagnostics, OTLP export,
|
|
inspector timing, collector validation/retention and failure-path tests passed.
|
|
These counts are a process snapshot, not a completeness claim.
|
|
|
|
Agent RPC instrumentation now wraps each request in `agent.request`; decrypt,
|
|
authorization, audit, dispatch, projection and encryption run beneath that parent.
|
|
The browser diagnostic operation allowlist accepts the new name. A real protected
|
|
RPC test verifies asynchronous parent/child nesting without recording arguments
|
|
or response content. The combined transport/diagnostics suite passes 38 tests;
|
|
A subsequent actual Electron run verifies exported parent/child phases linked to an attributable durable agent event; Windows packaged verification remains a separate gate.
|
|
|
|
The actual native agent audit proof links `browser.agent` to
|
|
`browser.management` and `server.request` under trace
|
|
`f8e35fd442e718995d9274c646fb8e08`. The native upload helper-failure proof links a
|
|
recording `browser.upload.fault` span, its lease parent, two logs and the durable
|
|
audit event under trace `e6d26d4edf738a7b4f6f562ab2112f5b`. Older traces may be in
|
|
the protected pinned archive rather than the current in-memory index.
|
|
|
|
A sampled slow `browser.parser` scan lasted 1,049 ms but contained only the outer
|
|
span in the live index. Do not infer which parser or model stage caused that
|
|
duration from an outer span alone: reproduce with signed debug mode enabled and
|
|
verify the expected inspector children before attributing a performance issue.
|
|
|
|
The current individual native proof links principal `reader-a` to
|
|
trace `3a0c31cbf53d7808e39a91daf2aaa3d7`. Its request took 81.019 ms,
|
|
the two durable audit phases totaled 77.598 ms and adapter dispatch took
|
|
0.313 ms. This single local run points to audit persistence as the dominant
|
|
stage; it is not a throughput benchmark. Preserve pre-execution durable audit
|
|
when investigating latency. Do not optimize by removing an audit gate or retrying
|
|
an action. The proof is in `artifacts/individual-agent-native-proof.json`.
|
|
The final Windows package verifier checks current source and runs both shipped
|
|
legacy and individual DPAPI/PQ clients successfully.
|
|
|
|
## Latest interaction and session measurements
|
|
|
|
The actual protected interaction proof uses trace
|
|
`32cbbb58088d193301c5f6ac530284af`: 30 spans, verified parent correlation,
|
|
capture 0.3353 ms, challenge 8.1735 ms, encryption 0.4718 ms, send 62.1817 ms
|
|
and acknowledgement 0.4234 ms. The backend store stage took 54.2629 ms.
|
|
Collector rejected writes were zero. These are one local reproduction's
|
|
measurements, not performance guarantees.
|
|
|
|
After session-title classification was added, the actual native session inventory
|
|
proof uses trace `64013e27cfc37bc3792983bc06149c87`: request 85.675 ms,
|
|
session adapter 1.0637 ms and durable audit phases 33.3975 and 47.2472 ms.
|
|
This fixture did not enable the semantic model; it cannot establish semantic-model
|
|
latency. The native proof confirms a confidential title is masked while preserving
|
|
its opaque session ID.
|
|
|
|
A parallel fixture timed out before enrollment and therefore before signed
|
|
diagnostics activation. Its isolated rerun and restart reached the shell in
|
|
794/806 ms. Safe local startup snapshots were added; the original timeout's cause
|
|
is unresolved. Do not describe that failure as a traced DLP/model regression.
|
|
The current Windows package proof and 515-test suite passed before the subsequent
|
|
export-counter refinement. Ignored proof artifacts remain local.
|
|
|
|
After the refinement and policy child stages, the final suite passes 522 tests.
|
|
Real signed policy/bookmark fixtures verify actual OTLP child spans and disabled
|
|
transitions. Windows build, packaged-runtime verification and the native tabs
|
|
proof pass. The native receiver attributes exports by actual `service.name`:
|
|
browser exports stay at 18 across revocation while backend collection remains
|
|
independent. The earlier aggregate assertion conflated the two services; its
|
|
failed run is retained rather than presented as successful. Do not retroactively
|
|
assign the original extra batch to a service without its original payload.
|
|
|
|
Foreground gesture development used actual traces to find a second attribution
|
|
gap: a queue flusher started by one event delivered the next event under the first
|
|
async context. `withCorrelation` now validates each durable record's own IDs and
|
|
`audit.deliver` creates a recording delivery child under that source parent.
|
|
Malformed IDs clear unrelated context, and current signed sampling/disable still
|
|
applies. Real OTLP regression tests and three actual human-event traces prove
|
|
distinct parents and complete evidence phase correlation. Trace
|
|
`b7783f42197d0cff83957b0ee96e8296` has 17 spans, capture 19.50 ms and send 13.20 ms;
|
|
`62e09413b9a42f4a58bd7f99485176ac` and
|
|
`545ddac025113e50effed4e9a2ac87ee` each also have 17 spans. Collector rejected writes
|
|
were zero. These are local reproduction timings, not throughput guarantees.
|
|
|
|
On 2026-10-13, the running protected collector and the actual exported trace
|
|
`fd47bcfc8b259712101ad3c81c1d95c1` were inspected again. The request took
|
|
724.3133 ms, its first audit phase 682.6577 ms, and dispatch 2.0827 ms.
|
|
Its correlated audit delivery took 52.7383 ms. The audit phase alone does not
|
|
establish whether serialization, DPAPI, filesystem synchronization or scheduling
|
|
caused the delay. New synchronous persistence spans separate `audit.serialize`,
|
|
`audit.encrypt`, `audit.write`, `audit.fsync` and `audit.commit`, without changing
|
|
the synchronous persistence boundary or retrying a write. Only fixed operation
|
|
identifiers and timings enter diagnostic attributes; no audit payload or paths.
|
|
Previously collected traces cannot supply these newly added stages.
|
|
|
|
The subsequent actual Electron vault fixture passed with the added stages.
|
|
Trace `e98cac3e18c6bd7b0e0442f033a7fa2d` exports all five persistence
|
|
children under `agent.audit`: serialization 0.0544 ms, encryption 0.0501 ms,
|
|
write 7.2461 ms, fsync 2.3703 ms and close/rename 14.7423 ms. These timings
|
|
describe this new run; they do not resolve the previous 683 ms observation.
|
|
The five-stage parentage was checked against the live collector for both list
|
|
and fill requests, with zero rejected writes. Twelve existing diagnostics tests
|
|
and four dedicated real OTLP tests pass, including central disable, secret
|
|
redaction, synchronous return/throw preservation and no replay on instrumentation
|
|
failure. At that stage, session adapter exports were proven in a fixture;
|
|
native browser restoration required the subsequent integration proof below.
|
|
|
|
The actual two-process Windows session proof now exports save/list/restore stages
|
|
and their audit persistence children. A delayed HTTPS receiver also verifies
|
|
that manually switching away and back aborts restoration, preserves existing tabs
|
|
and never requests the second saved destination. Trace
|
|
`583a1d4dbc80fb2cbcb135b1622a4993` records the failed native stage at 1410.0137 ms
|
|
and owned-tab rollback at 0.7242 ms. The delay was intentional fixture behavior;
|
|
it is not a production performance result. Collector rejected writes were zero.
|
|
|
|
|
|
On 2026-10-13, the protected live receiver was queried for controlled-login trace
|
|
`aedaae84014f4d5e14c6d7b441d7e0f7`. Eight credential phases were verified to
|
|
have the same native request ancestor: claim, parse, match, residual, headers,
|
|
header-values, authorization and completion. The body gate took 216.4937 ms;
|
|
local inspection took 178.5054 ms, while its inspector span took 12.283203 ms
|
|
and YARA 11.866699 ms. Enterprise inspection took 4.8818 ms. The remaining
|
|
local-inspection interval cannot be attributed to parsing or YARA from these
|
|
spans alone; investigate IPC, scheduling and containment before claiming a cause.
|
|
The receiver reported zero rejected writes and uses Windows DPAPI plus AES-GCM
|
|
for local archive storage. These are actual local fixture measurements, not
|
|
production latency guarantees. Credential values and form descriptors are
|
|
excluded from debug attributes; security evidence uses its separate policy and
|
|
audit pipeline. Trace exports remain centrally disableable.
|
|
|
|
The follow-up review found credential substring HMAC checks outside the named
|
|
matching stage. They now use `credential.match`, including checks of additional
|
|
fields and unexpected destinations. A real OTLP receiver regression verifies
|
|
six internal stage parents, timestamps, secret exclusion and central disable.
|
|
The combined broker, OTLP, policy-transition and request/header deadline suite
|
|
passes 29 tests. The earlier native trace predates this final matching refinement;
|
|
its timings must not be presented as measurements of the refined code.
|
|
|
|
|
|
The subsequent real native split-header test failed although the broker suite was
|
|
green. A separate Electron 44.7 / Chromium 152 header-order probe established the
|
|
cause: `X-Credential-One` and `X-Credential-Two` arrived with `sec-ch-ua-mobile`
|
|
between them. Joining every header value did not reconstruct the synthetic
|
|
credential; joining the two fragments did. Evidence retains only header names,
|
|
indices and booleans (`artifacts/credential-header-order-proof.json`), never values.
|
|
This native fact motivates bounded two-fragment checks independent of header
|
|
iteration order. It does not prove coverage of arbitrary fragmentation/encoding.
|
|
|
|
|
|
After the bounded pair fix, the fresh actual Windows native run passes both
|
|
unregistered-credential escape cases. Trace
|
|
`6709b228cb8ce71a88acc7584a27c5a1` contains 101 spans; credential header checking
|
|
took 15.6353 ms (header-values child 14.4573 ms), authorization 0.9952 ms and
|
|
terminal audit completion 24.8035 ms. Actual request ancestry and original plus
|
|
alternate synthetic secret exclusion were verified. ICAP observed no credentials
|
|
in 54 scans; collector rejected writes remained zero. The full suite passes
|
|
668 tests. The failing predecessor remains separate evidence and is not counted
|
|
as a pass. These timings are local fixture observations, not general benchmarks.
|
|
|
|
|
|
The receiver-strengthened final native run also passes: exactly four approved
|
|
credential POSTs, zero public POSTs, zero replays and zero credential URL/header
|
|
leaks at the independent HTTPS receiver. It privately checks any two header
|
|
fragments rather than relying on all-values concatenation. Current trace
|
|
`050f7f44bc009f949c838f8a067649e4` has 101 spans with verified request ancestry;
|
|
collector rejected writes remain zero. Evidence: `artifacts/controlled-login-native-proof.json`
|
|
and `artifacts/controlled-login-receiver-proof-native.log`. Earlier native failures
|
|
and the prior successful proof remain distinct local evidence.
|
|
|
|
|
|
The screenshot increment used actual traces to separate capture, structural PNG
|
|
validation, OCR, enterprise inspection, audit, projection and final release.
|
|
The first native run (`2a2507264480c9ed8fdb585a5608d292`) spent 6953.918 ms in
|
|
local inspection, with OCR 6469.757 ms, capture 358.558 ms and AV 5.039 ms.
|
|
These local cold-run observations identify OCR as the largest measured child;
|
|
they do not prove the cause of all remaining IPC/containment overhead.
|
|
|
|
Independent review found unobserved open-shadow mutations and a late policy/tab
|
|
release gap after the helper returned. Shadow roots now have private observers
|
|
and pending-record draining; final encrypted reply requires a private actor-bound
|
|
policy/native-frame release guard checked synchronously beside reply. Latest
|
|
native trace `aad45c8a85f25619a233b78e132d7bad` verifies eight stages. The final
|
|
`visual.release` takes 0.2507 ms under the outer agent request; the seven helper
|
|
stages retain the inner screenshot action ancestry. Real tests reclassify the
|
|
page while retaining the screenshot grants, switch the foreground after helper
|
|
return and mutate/revert open-shadow content; all suppress image delivery.
|
|
|
|
The native fixture also proves real contained OCR of a confidential canvas,
|
|
mandatory ICAP PNG blocking and secret exclusion from ordinary ECS/OTLP output.
|
|
Five PNG AV scans include one expected block; collector rejected writes are zero.
|
|
Real OTLP regression tests prove responsible-stage ERROR status, redaction and
|
|
central disable. No pixels, encoded images, raw DOM attributes or extracted OCR
|
|
text are diagnostic attributes. The local receiver is protected and bounded,
|
|
not a production forensic archive. The admin fixture's earlier 429 failure came
|
|
from short synthetic cookies sharing the IP bucket; realistic-length synthetic
|
|
capabilities resolved it without changing production limits. Its subsequent
|
|
200 completed API requests all have correlated traces.
|