provider capability model, on-demand scan/backfill, more providers

Readability capabilities (exists/object_readable/listable) are now tracked
separately instead of a single exposure flag, matching how providers differ:
- Azure 'Blob' mode = object readable, not listable; 'Container' = listable
- Cloudflare R2 public URLs are never anonymously listable (root 500); R2 is
  object-read only and is no longer probed for listing
- S3/GCS: 200 list = listable; 403 + readable index.html/robots.txt =
  object readable only
New columns: cloud_assets.listable, cloud_assets.object_readable.

Azure: account-level container listing is never anonymous; discover containers
by probing a bounded common-name set (or the container from a URL path), then
list blobs.

Classifier: added the legacy AWS '.s3-<region>.amazonaws.com' bucket form.

On-demand scan: POST /api/scan {target} classifies a host or URL, follows CNAME
for custom domains, probes, and indexes readable resources; Azure container
comes from the URL path. UI has a Scan box.

Backfill: POST /api/reprobe re-probes already-discovered assets with the current
checks and indexes any readable ones, recovering findings from earlier probes.

DNS: CNAME-only enrichment (A/AAAA resolved only for matches), 64 workers,
larger queue.

Verified live against real public resources:
  veteran1.nyc3.digitaloceanspaces.com            listable, entries
  bucket-01.s3-eu-west-1.amazonaws.com            listable, entries
  tartugeo-podcast.storage.googleapis.com         listable, entries
  eroticfeeldev.blob.core.windows.net/images      listable, entries
This commit is contained in:
StasonJatham committed 2026-09-22 14:47:49 +02:00
1 parent a366905a43
commit 62a7d2eb2e
23 files changed
+936 -165

No files matched your search

+24 -7
View File
@@ -80,6 +80,24 @@ func TestProbeFirebase(t *testing.T) {
}
}
func TestProbeAzureContainerDiscovery(t *testing.T) {
h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/images" && r.URL.Query().Get("comp") == "list" {
w.WriteHeader(http.StatusOK)
return
}
w.WriteHeader(http.StatusNotFound)
})
p := New(Config{Workers: 1, Timeout: time.Second})
host := serverHost(t, h)
state, detail := p.Probe(context.Background(), model.Candidate{
Hostname: host, Resource: "acct", Provider: model.ProviderAzure, Service: model.ServiceBlobStorage,
})
if state != model.ExposurePublicRead || !strings.Contains(detail, "images") {
t.Fatalf("got %s (%s), want public_read via container 'images'", state, detail)
}
}
func TestProbeDatabaseTCP(t *testing.T) {
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
@@ -97,17 +115,16 @@ func TestProbeDatabaseTCP(t *testing.T) {
}()
host, port, _ := net.SplitHostPort(ln.Addr().String())
p := New(Config{Workers: 1, Timeout: time.Second})
state, detail := p.checkTCP(context.Background(), host, mustAtoi(port))
if state != model.ExposureReachable {
t.Fatalf("got %s (%s), want reachable", state, detail)
r := p.checkTCP(context.Background(), host, mustAtoi(port))
if r.Exposure != model.ExposureReachable {
t.Fatalf("got %+v, want reachable", r)
}
if !strings.Contains(detail, "no authentication attempted") {
t.Fatalf("detail should be explicit about no auth: %q", detail)
if r.ObjectReadable || r.Listable {
t.Fatalf("tcp check must not claim readability: %+v", r)
}
// A closed port yields unknown, never readable.
state, _ = p.checkTCP(context.Background(), "127.0.0.1", 1)
if state == model.ExposurePublicRead {
if r := p.checkTCP(context.Background(), "127.0.0.1", 1); r.Exposure == model.ExposurePublicRead {
t.Fatal("closed port must never be public_read")
}
}