Web UI (internal/webui):
- dependency-free embedded HTML/JS served on the loopback listener
- /api/assets (provider/service/exposure/q filters) and /api/stats
- optional HTTP basic auth via UI_USER/UI_PASS
Prober (internal/probe):
- vendor-specific, unauthenticated, non-destructive checks:
S3/R2/Spaces ListObjectsV2, GCS JSON object list, Azure ?comp=list,
Firebase /.json?shallow=true, Supabase REST, and bare TCP connect for
database endpoints (never reported readable, no auth attempted)
- exposure_detail records the factual reason (status code, region, tcp port)
- bounded random-sample queue instead of FIFO: one high-volume hostname
template (mrap) no longer starves every other asset type
- schema: cloud_assets.exposure_detail + probed_at (idempotent ALTER)
Live: stable ~450MiB under 1GiB, no restarts, UI and probe states
(private/reachable/auth_required) flowing, 0 db errors.
- dns.TTLCache: stop preallocating every shard map to maxShard; this
allocated the full theoretical cache capacity up front (hundreds of MB
of empty buckets) and pushed the container over its memory limit,
causing an OOM restart loop under Docker's default 1g cap.
- pipeline: write direct provider matches immediately without a blocking
DNS enrichment step, so they can no longer be starved by the optional
CNAME queue. Add TestDirectNotStarvedByDNSBacklog.
- config: add DEDUP_SIZE, lower DNS_CACHE_SIZE/DEDUP_SIZE defaults to 256k.
- .env.example / README: document DEDUP_SIZE and cache sizing.
Live: stable at ~250-300 MiB with no restarts, direct queue draining to 0.
StasonJatham