parallel backfill, persist scanned exposure, random-sample DNS queue

- scanner.run now persists exposure_state/detail/capabilities for scanned
  assets (previously only the index was written).
- Backfill runs a bounded worker pool (16) so it verifies reachable assets
  without saturating the DB pool; DB_MAX_CONNS raised for the live test.
- CNAME enrichment queue is now a bounded random sample instead of a FIFO.
  The FIFO filled with the first N hostnames seen at startup and dropped the
  rest, so CNAME discovery only ever saw stale hostnames; random replacement
  gives representative coverage at bounded memory (same fix as the prober).

Verified live:
- all four known-public resources scan + index (DO, AWS s3-<region>, GCS,
  Azure container path), 38-1000 objects each.
- backfill of every CT-discovered 'reachable' asset (725) and 'auth_required'
  (12) completed with 0 listable: those are MRAP/non-bucket 400s and
  list-denied buckets, confirming the checks are actually running.
This commit is contained in:
StasonJatham committed 2026-09-22 15:09:34 +02:00
1 parent 62a7d2eb2e
commit 171c9b37eb
6 files changed
+196 -57

No files matched your search

+1 -1
View File
@@ -269,7 +269,7 @@ func TestDNSDedup(t *testing.T) {
p := New(Config{DNSQueueSize: 8, Metrics: m})
p.HandleHost("files.example.com")
p.HandleHost("files.example.com")
if got := len(p.dnsQ); got != 1 {
if got := p.dnsQ.len(); got != 1 {
t.Fatalf("dns queue len = %d, want 1", got)
}
}