over-engineering audit: remove dead code, speculative config, duplicate queues

- delete embedded community resolver snapshot (6,760 lines) + EmbeddedServers +
  DNS_SERVERS=fresh; curated resolvers are the default. Simplify Warm (no
  sampling).
- one generic internal/sample.Queue[T] replaces three near-identical
  random-sample queues (probe, pipeline, discover).
- delete unreachable index.listOpenSearch (OpenSearch is never listable).
- delete classifyCloudFront (its match is always discarded, !ResourceSpecific).
- delete probe.Probe wrapper + Describe; tests use ProbeDetailed.
- delete store.Pool(), model.SourceDNSAlias, discover.QueueLen.
- drop dead config/no-op knobs: pipeline DirectWorkers, discover MaxPerDomain,
  discover S3/GCS/DO flags, BUCKET_SCAN_DO.
~-7,000 lines (mostly the resolver snapshot), 0 deps removed, tests green.
This commit is contained in:
StasonJatham committed 2026-09-22 17:12:30 +02:00
1 parent 81e53f8755
commit dc7bdae99d
22 files changed
+150 -7202

No files matched your search

+5 -22
View File
@@ -131,7 +131,6 @@ Environment variables only; no config files, no runtime pattern files.
| `BUCKET_SCAN_QPS` | `50` | global cap on outbound discovery requests |
| `BUCKET_SCAN_WORKERS` | `16` | discovery probe workers |
| `BUCKET_SCAN_QUEUE` | `65536` | sampled domain backlog |
| `BUCKET_SCAN_DO` | `false` | also test DigitalOcean Spaces regions |
| `DEDUP_TTL` | `1h` | hostname/finding dedup window |
| `METRICS_ADDR` | `:9090` | metrics/health/UI listener |
| `UI_USER` / `UI_PASS` | *(unset)* | optional HTTP basic auth for the web UI |
@@ -318,11 +317,10 @@ engine that:
1. takes the names that actually occur in the hostname — the subdomain label
and the registrable-domain label (`assets.example.com` → `assets`,
`example`). No names are fabricated, permuted, or drawn from a wordlist;
2. tests anonymous listing per provider —
`https://<name>.s3.amazonaws.com/?list-type=2` (AWS),
`https://<name>.storage.googleapis.com` (GCS),
`https://<name>.<region>.digitaloceanspaces.com` (DO, optional);
`example`). No names are fabricated or drawn from a wordlist;
2. tests anonymous listing —
`https://<name>.s3.amazonaws.com/?list-type=2` (AWS) and
`https://<name>.storage.googleapis.com` (GCS);
3. indexes every bucket that returns 200 (keys/sizes/dates only).
It is aggressively bounded: a global `BUCKET_SCAN_QPS` limit, a 12h per-name
@@ -343,22 +341,7 @@ healthy resolver and fails over to at most two others; each attempt is bounded
answers (NXDOMAIN) return immediately rather than being retried on every server,
so negative lookups do not multiply DNS traffic.
Set `DNS_SERVERS=fresh` to add a compiled-in snapshot of
[proabiral/Fresh-Resolvers](https://github.com/proabiral/Fresh-Resolvers)
(6,760 entries) on top of the curated set. It validates a random sample of 256 at
startup and cools down the rest on failure. Refresh the snapshot with:
```sh
curl -sL https://raw.githubusercontent.com/proabiral/Fresh-Resolvers/master/resolvers.txt \
-o internal/dns/resolvers.txt
```
> Probing or using thousands of community resolvers at once can saturate the
> container's own DNS and break unrelated lookups, which is why `fresh` is
> opt-in and sampled. These resolvers are also untrusted: they can observe and,
> in principle, forge DNS answers. That only affects enrichment (which endpoints
> are looked up), never the classifier. For sensitive deployments set
> `DNS_SERVERS` to your own resolvers, or `DNS_SERVERS=system`.
> For sensitive deployments set `DNS_SERVERS` to your own resolvers, or `DNS_SERVERS=system`.
> These community resolvers are untrusted: they can observe and, in principle,
> forge DNS answers. That only affects enrichment (which endpoints are looked