MRAP aliases (<alias>.mrap.accesspoint.s3-global.amazonaws.com) were 91,560 of
91,561 S3 findings: AWS-generated, not customer buckets, never publicly
resolvable, and they monopolised pipeline capacity. CloudFront distributions are
CDN endpoints, not buckets or databases. Both are now excluded so real buckets
and databases get the capacity.
CloudFront/CDN endpoints return 200 for GET / but are not listable; the generic
HTTP check now says 'anonymous access succeeded' instead of 'listing allowed'.
Custom domains commonly CNAME to <dist>.cloudfront.net, so CDN-hosted static
sites were invisible to CNAME discovery. Classify them as AWS/WebApp and probe
https://<dist>.cloudfront.net/ for readability.
Random picks kept hitting unreachable public resolvers, costing a 2s timeout
on ~half of lookups and capping DNS throughput. Warm the default set at startup
(same health check used for the community list).
- scanner.run now persists exposure_state/detail/capabilities for scanned
assets (previously only the index was written).
- Backfill runs a bounded worker pool (16) so it verifies reachable assets
without saturating the DB pool; DB_MAX_CONNS raised for the live test.
- CNAME enrichment queue is now a bounded random sample instead of a FIFO.
The FIFO filled with the first N hostnames seen at startup and dropped the
rest, so CNAME discovery only ever saw stale hostnames; random replacement
gives representative coverage at bounded memory (same fix as the prober).
Verified live:
- all four known-public resources scan + index (DO, AWS s3-<region>, GCS,
Azure container path), 38-1000 objects each.
- backfill of every CT-discovered 'reachable' asset (725) and 'auth_required'
(12) completed with 0 listable: those are MRAP/non-bucket 400s and
list-denied buckets, confirming the checks are actually running.
Readability capabilities (exists/object_readable/listable) are now tracked
separately instead of a single exposure flag, matching how providers differ:
- Azure 'Blob' mode = object readable, not listable; 'Container' = listable
- Cloudflare R2 public URLs are never anonymously listable (root 500); R2 is
object-read only and is no longer probed for listing
- S3/GCS: 200 list = listable; 403 + readable index.html/robots.txt =
object readable only
New columns: cloud_assets.listable, cloud_assets.object_readable.
Azure: account-level container listing is never anonymous; discover containers
by probing a bounded common-name set (or the container from a URL path), then
list blobs.
Classifier: added the legacy AWS '.s3-<region>.amazonaws.com' bucket form.
On-demand scan: POST /api/scan {target} classifies a host or URL, follows CNAME
for custom domains, probes, and indexes readable resources; Azure container
comes from the URL path. UI has a Scan box.
Backfill: POST /api/reprobe re-probes already-discovered assets with the current
checks and indexes any readable ones, recovering findings from earlier probes.
DNS: CNAME-only enrichment (A/AAAA resolved only for matches), 64 workers,
larger queue.
Verified live against real public resources:
veteran1.nyc3.digitaloceanspaces.com listable, entries
bucket-01.s3-eu-west-1.amazonaws.com listable, entries
tartugeo-podcast.storage.googleapis.com listable, entries
eroticfeeldev.blob.core.windows.net/images listable, entries
Logging:
- structured slog with LOG_FORMAT=json|text and a stable key vocabulary
- periodic stats record with queue depths, db pool, goroutines, memory, dns/probe
- no per-certificate logging (hot path is counters only)
UI (Bucketeer branding, DESIGN.md):
- light+dark themes (OS default, persisted header toggle, no first-paint flash)
- official mascot: lossless WebP light/dark logos + favicon, embedded and served
with cache headers (1.2MB source PNGs -> ~27/22KB webp)
- Assets/Contents views, pagination (offset) and native virtual scrolling
(content-visibility) so large result sets do not blow up the DOM
- search with debounce/abort + URL state, exposure chips, sort, summary counts
Readability + indexing:
- internal/index listers for S3-compatible, GCS, Azure, Firebase, OpenSearch
(metadata only: keys/sizes/dates; never object contents)
- asset_entries table + store write/search; readable resources are indexed and
searchable in the UI
- deep, unauthenticated, read-only DB probes (PROBE_DEEP): Redis PING,
PostgreSQL StartupMessage, MongoDB OP_MSG listDatabases; OpenSearch/Neptune
HTTP reads
- Azure account-level listing fixed: 403 is reported reachable, not
auth_required, since anonymous access is per-container
DNS:
- curated reliable resolvers by default; DNS_SERVERS=fresh adds the compiled-in
Fresh-Resolvers snapshot with a bounded startup sample (warming all 6.7k at
once broke the container resolver)
- random healthy selection + failover + per-server cooldown
Other: prober Close fixes a shutdown hang; config/README updated.
Web UI (internal/webui):
- dependency-free embedded HTML/JS served on the loopback listener
- /api/assets (provider/service/exposure/q filters) and /api/stats
- optional HTTP basic auth via UI_USER/UI_PASS
Prober (internal/probe):
- vendor-specific, unauthenticated, non-destructive checks:
S3/R2/Spaces ListObjectsV2, GCS JSON object list, Azure ?comp=list,
Firebase /.json?shallow=true, Supabase REST, and bare TCP connect for
database endpoints (never reported readable, no auth attempted)
- exposure_detail records the factual reason (status code, region, tcp port)
- bounded random-sample queue instead of FIFO: one high-volume hostname
template (mrap) no longer starves every other asset type
- schema: cloud_assets.exposure_detail + probed_at (idempotent ALTER)
Live: stable ~450MiB under 1GiB, no restarts, UI and probe states
(private/reachable/auth_required) flowing, 0 db errors.
- dns.TTLCache: stop preallocating every shard map to maxShard; this
allocated the full theoretical cache capacity up front (hundreds of MB
of empty buckets) and pushed the container over its memory limit,
causing an OOM restart loop under Docker's default 1g cap.
- pipeline: write direct provider matches immediately without a blocking
DNS enrichment step, so they can no longer be starved by the optional
CNAME queue. Add TestDirectNotStarvedByDNSBacklog.
- config: add DEDUP_SIZE, lower DNS_CACHE_SIZE/DEDUP_SIZE defaults to 256k.
- .env.example / README: document DEDUP_SIZE and cache sizing.
Live: stable at ~250-300 MiB with no restarts, direct queue draining to 0.
StasonJatham