provider capability model, on-demand scan/backfill, more providers

Readability capabilities (exists/object_readable/listable) are now tracked
separately instead of a single exposure flag, matching how providers differ:
- Azure 'Blob' mode = object readable, not listable; 'Container' = listable
- Cloudflare R2 public URLs are never anonymously listable (root 500); R2 is
  object-read only and is no longer probed for listing
- S3/GCS: 200 list = listable; 403 + readable index.html/robots.txt =
  object readable only
New columns: cloud_assets.listable, cloud_assets.object_readable.

Azure: account-level container listing is never anonymous; discover containers
by probing a bounded common-name set (or the container from a URL path), then
list blobs.

Classifier: added the legacy AWS '.s3-<region>.amazonaws.com' bucket form.

On-demand scan: POST /api/scan {target} classifies a host or URL, follows CNAME
for custom domains, probes, and indexes readable resources; Azure container
comes from the URL path. UI has a Scan box.

Backfill: POST /api/reprobe re-probes already-discovered assets with the current
checks and indexes any readable ones, recovering findings from earlier probes.

DNS: CNAME-only enrichment (A/AAAA resolved only for matches), 64 workers,
larger queue.

Verified live against real public resources:
  veteran1.nyc3.digitaloceanspaces.com            listable, entries
  bucket-01.s3-eu-west-1.amazonaws.com            listable, entries
  tartugeo-podcast.storage.googleapis.com         listable, entries
  eroticfeeldev.blob.core.windows.net/images      listable, entries
This commit is contained in:
StasonJatham committed 2026-09-22 14:47:49 +02:00
1 parent a366905a43
commit 62a7d2eb2e
23 files changed
+936 -165

No files matched your search

+12 -12
View File
@@ -46,9 +46,9 @@ func TestProbeRedisReadable(t *testing.T) {
_, _ = c.Write([]byte("+PONG\r\n"))
})
p := New(Config{Workers: 1, Timeout: time.Second})
state, detail, ok := p.probeRedis(context.Background(), host, port)
if !ok || state != model.ExposurePublicRead {
t.Fatalf("got %s (%s) ok=%v, want public_read", state, detail, ok)
r, ok := p.probeRedis(context.Background(), host, port)
if !ok || r.Exposure != model.ExposurePublicRead || !r.ObjectReadable {
t.Fatalf("got %+v ok=%v, want public_read/object_readable", r, ok)
}
}
@@ -59,9 +59,9 @@ func TestProbeRedisAuthRequired(t *testing.T) {
_, _ = c.Write([]byte("-NOAUTH Authentication required.\r\n"))
})
p := New(Config{Workers: 1, Timeout: time.Second})
state, _, _ := p.probeRedis(context.Background(), host, port)
if state != model.ExposureAuthRequired {
t.Fatalf("got %s, want auth_required", state)
r, _ := p.probeRedis(context.Background(), host, port)
if r.Exposure != model.ExposureAuthRequired {
t.Fatalf("got %+v, want auth_required", r)
}
}
@@ -82,9 +82,9 @@ func TestProbePostgresTrust(t *testing.T) {
_, _ = c.Write([]byte{'R', 0, 0, 0, 8, 0, 0, 0, 0})
})
p := New(Config{Workers: 1, Timeout: time.Second})
state, detail, _ := p.probePostgres(context.Background(), host, port)
if state != model.ExposurePublicRead {
t.Fatalf("got %s (%s), want public_read", state, detail)
r, _ := p.probePostgres(context.Background(), host, port)
if r.Exposure != model.ExposurePublicRead || !r.ObjectReadable {
t.Fatalf("got %+v, want public_read/object_readable", r)
}
}
@@ -105,9 +105,9 @@ func TestProbePostgresPassword(t *testing.T) {
_, _ = c.Write([]byte{'R', 0, 0, 0, 8, 0, 0, 0, 5})
})
p := New(Config{Workers: 1, Timeout: time.Second})
state, _, _ := p.probePostgres(context.Background(), host, port)
if state != model.ExposureAuthRequired {
t.Fatalf("got %s, want auth_required", state)
r, _ := p.probePostgres(context.Background(), host, port)
if r.Exposure != model.ExposureAuthRequired {
t.Fatalf("got %+v, want auth_required", r)
}
}