add read-only web UI and per-vendor readability prober

Web UI (internal/webui):
- dependency-free embedded HTML/JS served on the loopback listener
- /api/assets (provider/service/exposure/q filters) and /api/stats
- optional HTTP basic auth via UI_USER/UI_PASS

Prober (internal/probe):
- vendor-specific, unauthenticated, non-destructive checks:
  S3/R2/Spaces ListObjectsV2, GCS JSON object list, Azure ?comp=list,
  Firebase /.json?shallow=true, Supabase REST, and bare TCP connect for
  database endpoints (never reported readable, no auth attempted)
- exposure_detail records the factual reason (status code, region, tcp port)
- bounded random-sample queue instead of FIFO: one high-volume hostname
  template (mrap) no longer starves every other asset type
- schema: cloud_assets.exposure_detail + probed_at (idempotent ALTER)

Live: stable ~450MiB under 1GiB, no restarts, UI and probe states
(private/reachable/auth_required) flowing, 0 db errors.
This commit is contained in:
StasonJatham committed 2026-09-22 13:58:28 +02:00
1 parent 9710efec1a
commit 8fce55f8c9
13 files changed
+843 -72

No files matched your search

+7 -1
View File
@@ -60,7 +60,13 @@ func TestIntegration(t *testing.T) {
if err := s.WriteFindings(ctx, []pipeline.Finding{f}); err != nil {
t.Fatal(err)
}
if err := s.UpdateExposure(ctx, model.ProviderAWS.String(), model.ServiceS3.String(), f.Hostname, model.ExposureReachable); err != nil {
if err := s.UpdateExposure(ctx, model.ProviderAWS.String(), model.ServiceS3.String(), f.Hostname, model.ExposureReachable, "test"); err != nil {
t.Fatal(err)
}
if _, err := s.ListAssets(ctx, AssetFilter{Provider: "aws", Limit: 10}); err != nil {
t.Fatal(err)
}
if _, err := s.Stats(ctx); err != nil {
t.Fatal(err)
}
}