provider capability model, on-demand scan/backfill, more providers

Readability capabilities (exists/object_readable/listable) are now tracked
separately instead of a single exposure flag, matching how providers differ:
- Azure 'Blob' mode = object readable, not listable; 'Container' = listable
- Cloudflare R2 public URLs are never anonymously listable (root 500); R2 is
  object-read only and is no longer probed for listing
- S3/GCS: 200 list = listable; 403 + readable index.html/robots.txt =
  object readable only
New columns: cloud_assets.listable, cloud_assets.object_readable.

Azure: account-level container listing is never anonymous; discover containers
by probing a bounded common-name set (or the container from a URL path), then
list blobs.

Classifier: added the legacy AWS '.s3-<region>.amazonaws.com' bucket form.

On-demand scan: POST /api/scan {target} classifies a host or URL, follows CNAME
for custom domains, probes, and indexes readable resources; Azure container
comes from the URL path. UI has a Scan box.

Backfill: POST /api/reprobe re-probes already-discovered assets with the current
checks and indexes any readable ones, recovering findings from earlier probes.

DNS: CNAME-only enrichment (A/AAAA resolved only for matches), 64 workers,
larger queue.

Verified live against real public resources:
  veteran1.nyc3.digitaloceanspaces.com            listable, entries
  bucket-01.s3-eu-west-1.amazonaws.com            listable, entries
  tartugeo-podcast.storage.googleapis.com         listable, entries
  eroticfeeldev.blob.core.windows.net/images      listable, entries
This commit is contained in:
StasonJatham committed 2026-09-22 14:47:49 +02:00
1 parent a366905a43
commit 62a7d2eb2e
23 files changed
+936 -165

No files matched your search

+44 -2
View File
@@ -12,6 +12,7 @@ import (
"strconv"
"strings"
"tea.karlcom.de/karl/bucketeer/internal/scan"
"tea.karlcom.de/karl/bucketeer/internal/store"
)
@@ -29,9 +30,17 @@ type Store interface {
Stats(ctx context.Context) (store.Stats, error)
}
// Scanner probes and indexes a user-supplied host/URL on demand, and can
// re-probe already-discovered assets.
type Scanner interface {
Scan(ctx context.Context, target string) []scan.Result
Backfill(ctx context.Context, exposure string, limit int) (scan.BackfillReport, error)
}
type Config struct {
User string // optional HTTP basic auth username
Pass string // optional HTTP basic auth password
User string // optional HTTP basic auth username
Pass string // optional HTTP basic auth password
Scanner Scanner
}
// New returns the UI handler. If Config.User is set, all routes require basic auth.
@@ -84,6 +93,39 @@ func New(s Store, cfg Config) http.Handler {
}
writeJSON(w, map[string]any{"entries": entries, "total": total, "limit": limit, "offset": offset})
})
if cfg.Scanner != nil {
mux.HandleFunc("/api/scan", func(w http.ResponseWriter, r *http.Request) {
target := strings.TrimSpace(r.URL.Query().Get("target"))
if r.Method == http.MethodPost {
var body struct {
Target string `json:"target"`
}
_ = json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&body)
if body.Target != "" {
target = strings.TrimSpace(body.Target)
}
}
if target == "" || len(target) > 512 {
http.Error(w, "target required (<=512 chars)", http.StatusBadRequest)
return
}
writeJSON(w, cfg.Scanner.Scan(r.Context(), target))
})
mux.HandleFunc("/api/reprobe", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "POST only", http.StatusMethodNotAllowed)
return
}
q := r.URL.Query()
limit, _ := strconv.Atoi(q.Get("limit"))
rep, err := cfg.Scanner.Backfill(r.Context(), strings.TrimSpace(q.Get("exposure")), limit)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
writeJSON(w, rep)
})
}
mux.HandleFunc("/api/stats", func(w http.ResponseWriter, r *http.Request) {
st, err := s.Stats(r.Context())
if err != nil {