parallel backfill, persist scanned exposure, random-sample DNS queue
- scanner.run now persists exposure_state/detail/capabilities for scanned assets (previously only the index was written). - Backfill runs a bounded worker pool (16) so it verifies reachable assets without saturating the DB pool; DB_MAX_CONNS raised for the live test. - CNAME enrichment queue is now a bounded random sample instead of a FIFO. The FIFO filled with the first N hostnames seen at startup and dropped the rest, so CNAME discovery only ever saw stale hostnames; random replacement gives representative coverage at bounded memory (same fix as the prober). Verified live: - all four known-public resources scan + index (DO, AWS s3-<region>, GCS, Azure container path), 38-1000 objects each. - backfill of every CT-discovered 'reachable' asset (725) and 'auth_required' (12) completed with 0 listable: those are MRAP/non-bucket 400s and list-denied buckets, confirming the checks are actually running.
This commit is contained in:
1 parent
62a7d2eb2e
commit
171c9b37eb
6 files changed
+196
-57
No files matched your search
+7
-1
@@ -135,7 +135,13 @@ func run() error {
|
||||
// Scanner prober: a separate, never-started Prober used synchronously by the
|
||||
// on-demand scan endpoint (Probe is stateless).
|
||||
scanProber := probe.New(probe.Config{Workers: 1, Timeout: cfg.ProbeTimeout, Deep: cfg.ProbeDeep, Metrics: m})
|
||||
scanner := &scan.Scanner{Prober: scanProber, Indexer: indexer, Enricher: enricher, Store: st}
|
||||
// Backfill does one DB write per target, so keep its concurrency near the
|
||||
// pool size to avoid starving the live pipeline and the UI.
|
||||
backfillWorkers := cfg.ProbeWorkers
|
||||
if backfillWorkers > 16 {
|
||||
backfillWorkers = 16
|
||||
}
|
||||
scanner := &scan.Scanner{Prober: scanProber, Indexer: indexer, Enricher: enricher, Store: st, BackfillWorkers: backfillWorkers}
|
||||
|
||||
var prober *probe.Prober
|
||||
if cfg.ActiveProbing {
|
||||
|
||||
Reference in new issue
Block a user