ResolveCNAME/ResolveIPs cached negative results as an empty Result but the
cache-hit path returned (empty, true), so enrichHost did chain[1:] on a
zero-length slice and the process panicked (exit 2). Negative cache hits now
return false, plus a defensive length guard in enrichHost and a regression test.
Readability capabilities (exists/object_readable/listable) are now tracked
separately instead of a single exposure flag, matching how providers differ:
- Azure 'Blob' mode = object readable, not listable; 'Container' = listable
- Cloudflare R2 public URLs are never anonymously listable (root 500); R2 is
object-read only and is no longer probed for listing
- S3/GCS: 200 list = listable; 403 + readable index.html/robots.txt =
object readable only
New columns: cloud_assets.listable, cloud_assets.object_readable.
Azure: account-level container listing is never anonymous; discover containers
by probing a bounded common-name set (or the container from a URL path), then
list blobs.
Classifier: added the legacy AWS '.s3-<region>.amazonaws.com' bucket form.
On-demand scan: POST /api/scan {target} classifies a host or URL, follows CNAME
for custom domains, probes, and indexes readable resources; Azure container
comes from the URL path. UI has a Scan box.
Backfill: POST /api/reprobe re-probes already-discovered assets with the current
checks and indexes any readable ones, recovering findings from earlier probes.
DNS: CNAME-only enrichment (A/AAAA resolved only for matches), 64 workers,
larger queue.
Verified live against real public resources:
veteran1.nyc3.digitaloceanspaces.com listable, entries
bucket-01.s3-eu-west-1.amazonaws.com listable, entries
tartugeo-podcast.storage.googleapis.com listable, entries
eroticfeeldev.blob.core.windows.net/images listable, entries
Logging:
- structured slog with LOG_FORMAT=json|text and a stable key vocabulary
- periodic stats record with queue depths, db pool, goroutines, memory, dns/probe
- no per-certificate logging (hot path is counters only)
UI (Bucketeer branding, DESIGN.md):
- light+dark themes (OS default, persisted header toggle, no first-paint flash)
- official mascot: lossless WebP light/dark logos + favicon, embedded and served
with cache headers (1.2MB source PNGs -> ~27/22KB webp)
- Assets/Contents views, pagination (offset) and native virtual scrolling
(content-visibility) so large result sets do not blow up the DOM
- search with debounce/abort + URL state, exposure chips, sort, summary counts
Readability + indexing:
- internal/index listers for S3-compatible, GCS, Azure, Firebase, OpenSearch
(metadata only: keys/sizes/dates; never object contents)
- asset_entries table + store write/search; readable resources are indexed and
searchable in the UI
- deep, unauthenticated, read-only DB probes (PROBE_DEEP): Redis PING,
PostgreSQL StartupMessage, MongoDB OP_MSG listDatabases; OpenSearch/Neptune
HTTP reads
- Azure account-level listing fixed: 403 is reported reachable, not
auth_required, since anonymous access is per-container
DNS:
- curated reliable resolvers by default; DNS_SERVERS=fresh adds the compiled-in
Fresh-Resolvers snapshot with a bounded startup sample (warming all 6.7k at
once broke the container resolver)
- random healthy selection + failover + per-server cooldown
Other: prober Close fixes a shutdown hang; config/README updated.
- dns.TTLCache: stop preallocating every shard map to maxShard; this
allocated the full theoretical cache capacity up front (hundreds of MB
of empty buckets) and pushed the container over its memory limit,
causing an OOM restart loop under Docker's default 1g cap.
- pipeline: write direct provider matches immediately without a blocking
DNS enrichment step, so they can no longer be starved by the optional
CNAME queue. Add TestDirectNotStarvedByDNSBacklog.
- config: add DEDUP_SIZE, lower DNS_CACHE_SIZE/DEDUP_SIZE defaults to 256k.
- .env.example / README: document DEDUP_SIZE and cache sizing.
Live: stable at ~250-300 MiB with no restarts, direct queue draining to 0.
StasonJatham