bucket discovery engine: CT domain -> candidate names -> listing index
CMS SANs are custom domains, not bucket names, so matching SANs finds almost no buckets. This adds internal/discover: from each non-matching hostname it derives candidate names (domain label + subdomain label + ~40 suffixes) plus a built-in wordlist, and anonymously tests listing on S3, GCS and (optionally) DigitalOcean Spaces. Every 200 is indexed (keys/sizes/dates only). Bounded hard: global BUCKET_SCAN_QPS rate limit, 12h per-name dedup, random sampled domain queue, MaxPerDomain cap. Never downloads contents. Fix: GCS candidates now use the virtual-host form (<bucket>.storage.googleapis.com) so buckets no longer collide on the cloud_assets unique key; previously they all shared storage.googleapis.com and overwrote each other. Live: 30 open buckets in ~5 min (e.g. estimator-dev, image-prod, update-media, mos-images S3; several GCS buckets), 11k+ objects indexed.
This commit is contained in:
1 parent
816fa572df
commit
a899362e18
9 files changed
+615
-32
No files matched your search
@@ -42,6 +42,10 @@ type Config struct {
|
||||
Enricher *dns.Enricher
|
||||
Write WriteFunc
|
||||
Metrics *metrics.Metrics
|
||||
|
||||
// Discover, when set, receives non-matching hostnames for bucket-name
|
||||
// enumeration (see internal/discover).
|
||||
Discover interface{ Submit(host string) }
|
||||
}
|
||||
|
||||
// Pipeline is the certificate-to-database data flow. It is safe to call
|
||||
@@ -176,6 +180,9 @@ func (p *Pipeline) HandleHost(host string) {
|
||||
}
|
||||
p.dedup.Set(host, true, p.cfg.DedupTTL)
|
||||
p.dnsQ.push(host)
|
||||
if p.cfg.Discover != nil {
|
||||
p.cfg.Discover.Submit(host)
|
||||
}
|
||||
}
|
||||
|
||||
func (p *Pipeline) drop() {
|
||||
|
||||
Reference in new issue
Block a user