bucket discovery engine: CT domain -> candidate names -> listing index

CMS SANs are custom domains, not bucket names, so matching SANs finds almost
no buckets. This adds internal/discover: from each non-matching hostname it
derives candidate names (domain label + subdomain label + ~40 suffixes) plus a
built-in wordlist, and anonymously tests listing on S3, GCS and (optionally)
DigitalOcean Spaces. Every 200 is indexed (keys/sizes/dates only).

Bounded hard: global BUCKET_SCAN_QPS rate limit, 12h per-name dedup, random
sampled domain queue, MaxPerDomain cap. Never downloads contents.

Fix: GCS candidates now use the virtual-host form
(<bucket>.storage.googleapis.com) so buckets no longer collide on the
cloud_assets unique key; previously they all shared storage.googleapis.com and
overwrote each other.

Live: 30 open buckets in ~5 min (e.g. estimator-dev, image-prod,
update-media, mos-images S3; several GCS buckets), 11k+ objects indexed.
This commit is contained in:
StasonJatham committed 2026-09-22 15:53:31 +02:00
1 parent 816fa572df
commit a899362e18
9 files changed
+615 -32

No files matched your search

+7
View File
@@ -28,3 +28,10 @@ INDEX_MAX_OBJECTS=1000
# Optional HTTP basic auth for the web UI (/ and /api/*):
# UI_USER=admin
# UI_PASS=change-me-strong
# Bucket-name discovery (finds open buckets; CT SANs are domains, not buckets)
BUCKET_SCAN_ENABLED=true
BUCKET_SCAN_QPS=50
BUCKET_SCAN_WORKERS=16
BUCKET_SCAN_QUEUE=65536
BUCKET_SCAN_DO=false