Readability capabilities (exists/object_readable/listable) are now tracked
separately instead of a single exposure flag, matching how providers differ:
- Azure 'Blob' mode = object readable, not listable; 'Container' = listable
- Cloudflare R2 public URLs are never anonymously listable (root 500); R2 is
object-read only and is no longer probed for listing
- S3/GCS: 200 list = listable; 403 + readable index.html/robots.txt =
object readable only
New columns: cloud_assets.listable, cloud_assets.object_readable.
Azure: account-level container listing is never anonymous; discover containers
by probing a bounded common-name set (or the container from a URL path), then
list blobs.
Classifier: added the legacy AWS '.s3-<region>.amazonaws.com' bucket form.
On-demand scan: POST /api/scan {target} classifies a host or URL, follows CNAME
for custom domains, probes, and indexes readable resources; Azure container
comes from the URL path. UI has a Scan box.
Backfill: POST /api/reprobe re-probes already-discovered assets with the current
checks and indexes any readable ones, recovering findings from earlier probes.
DNS: CNAME-only enrichment (A/AAAA resolved only for matches), 64 workers,
larger queue.
Verified live against real public resources:
veteran1.nyc3.digitaloceanspaces.com listable, entries
bucket-01.s3-eu-west-1.amazonaws.com listable, entries
tartugeo-podcast.storage.googleapis.com listable, entries
eroticfeeldev.blob.core.windows.net/images listable, entries
Logging:
- structured slog with LOG_FORMAT=json|text and a stable key vocabulary
- periodic stats record with queue depths, db pool, goroutines, memory, dns/probe
- no per-certificate logging (hot path is counters only)
UI (Bucketeer branding, DESIGN.md):
- light+dark themes (OS default, persisted header toggle, no first-paint flash)
- official mascot: lossless WebP light/dark logos + favicon, embedded and served
with cache headers (1.2MB source PNGs -> ~27/22KB webp)
- Assets/Contents views, pagination (offset) and native virtual scrolling
(content-visibility) so large result sets do not blow up the DOM
- search with debounce/abort + URL state, exposure chips, sort, summary counts
Readability + indexing:
- internal/index listers for S3-compatible, GCS, Azure, Firebase, OpenSearch
(metadata only: keys/sizes/dates; never object contents)
- asset_entries table + store write/search; readable resources are indexed and
searchable in the UI
- deep, unauthenticated, read-only DB probes (PROBE_DEEP): Redis PING,
PostgreSQL StartupMessage, MongoDB OP_MSG listDatabases; OpenSearch/Neptune
HTTP reads
- Azure account-level listing fixed: 403 is reported reachable, not
auth_required, since anonymous access is per-container
DNS:
- curated reliable resolvers by default; DNS_SERVERS=fresh adds the compiled-in
Fresh-Resolvers snapshot with a bounded startup sample (warming all 6.7k at
once broke the container resolver)
- random healthy selection + failover + per-server cooldown
Other: prober Close fixes a shutdown hang; config/README updated.
StasonJatham