bucket discovery engine: CT domain -> candidate names -> listing index

CMS SANs are custom domains, not bucket names, so matching SANs finds almost
no buckets. This adds internal/discover: from each non-matching hostname it
derives candidate names (domain label + subdomain label + ~40 suffixes) plus a
built-in wordlist, and anonymously tests listing on S3, GCS and (optionally)
DigitalOcean Spaces. Every 200 is indexed (keys/sizes/dates only).

Bounded hard: global BUCKET_SCAN_QPS rate limit, 12h per-name dedup, random
sampled domain queue, MaxPerDomain cap. Never downloads contents.

Fix: GCS candidates now use the virtual-host form
(<bucket>.storage.googleapis.com) so buckets no longer collide on the
cloud_assets unique key; previously they all shared storage.googleapis.com and
overwrote each other.

Live: 30 open buckets in ~5 min (e.g. estimator-dev, image-prod,
update-media, mos-images S3; several GCS buckets), 11k+ objects indexed.

README.md

@@ -127,6 +127,11 @@ Environment variables only; no config files, no runtime pattern files.
 | `PROBE_DEEP` | `true` | unauthenticated, read-only protocol reads for databases (Redis/Postgres/Mongo) |
 | `INDEX_ENABLED` / `INDEX_MAX_OBJECTS` | `true` / `1000` | index metadata of readable resources (keys/sizes only) |
 | `LOG_FORMAT` | `text` | `text` or `json` (JSON is best for log pipelines/LLM analysis) |
+| `BUCKET_SCAN_ENABLED` | `true` | discover open buckets by name enumeration |
+| `BUCKET_SCAN_QPS` | `50` | global cap on outbound discovery requests |
+| `BUCKET_SCAN_WORKERS` | `16` | discovery probe workers |
+| `BUCKET_SCAN_QUEUE` | `65536` | sampled domain backlog |
+| `BUCKET_SCAN_DO` | `false` | also test DigitalOcean Spaces regions |
 | `DEDUP_TTL` | `1h` | hostname/finding dedup window |
 | `METRICS_ADDR` | `:9090` | metrics/health/UI listener |
 | `UI_USER` / `UI_PASS` | *(unset)* | optional HTTP basic auth for the web UI |
@@ -305,6 +310,30 @@ go test ./internal/classify/ ./internal/pipeline/ -run '^$' -bench . -benchmem
 `BenchmarkClassifyMixed` and `TestHandleHostNoAllocs` fail unless the result is
 exactly 0 allocs/op, so the target is enforced by the test suite.
 
+### Bucket discovery
+
+Certificate SANs are custom domains, not bucket names, so matching SANs alone
+finds almost no buckets. Instead, every non-matching hostname feeds a discovery
+engine that:
+
+1. derives candidate names from the domain (`example` → `example`,
+   `example-assets`, `example-backups`, …, plus the subdomain label), also
+   testing a built-in wordlist of common names;
+2. tests anonymous listing per provider —
+   `https://<name>.s3.amazonaws.com/?list-type=2` (AWS),
+   `https://<name>.storage.googleapis.com` (GCS),
+   `https://<name>.<region>.digitaloceanspaces.com` (DO, optional);
+3. indexes every bucket that returns 200 (keys/sizes/dates only).
+
+It is aggressively bounded: a global `BUCKET_SCAN_QPS` limit, a 12h per-name
+dedup cache, and a random-sampled domain queue. Azure cannot be enumerated this
+way (account names are the namespace and containers need names), so Azure is
+handled from CT accounts plus a container wordlist, and R2 is never listable.
+
+This is what actually yields open buckets. Of the buckets found, S3 and GCS
+return full object listings; Azure needs a known container; R2 only serves
+objects you already know.
+
 ### DNS load balancing
 
 When `DNS_SERVERS` is unset, the pool is ~20 curated, reliable public resolvers