- Store: skip identical CT checkpoint writes (CertSpotter calls StoreLogState
per batch across 64 logs; hashing the state avoids redundant writes).
- webui: cache /api/stats for 10s and raise API query timeouts; UI polls, so
repeated aggregate queries no longer queue behind checkpoint bursts.
- reword discovery detail (no 'name enumeration' for S3/GCS; only Azure uses a
container list).
CMS SANs are custom domains, not bucket names, so matching SANs finds almost
no buckets. This adds internal/discover: from each non-matching hostname it
derives candidate names (domain label + subdomain label + ~40 suffixes) plus a
built-in wordlist, and anonymously tests listing on S3, GCS and (optionally)
DigitalOcean Spaces. Every 200 is indexed (keys/sizes/dates only).
Bounded hard: global BUCKET_SCAN_QPS rate limit, 12h per-name dedup, random
sampled domain queue, MaxPerDomain cap. Never downloads contents.
Fix: GCS candidates now use the virtual-host form
(<bucket>.storage.googleapis.com) so buckets no longer collide on the
cloud_assets unique key; previously they all shared storage.googleapis.com and
overwrote each other.
Live: 30 open buckets in ~5 min (e.g. estimator-dev, image-prod,
update-media, mos-images S3; several GCS buckets), 11k+ objects indexed.
Logging:
- structured slog with LOG_FORMAT=json|text and a stable key vocabulary
- periodic stats record with queue depths, db pool, goroutines, memory, dns/probe
- no per-certificate logging (hot path is counters only)
UI (Bucketeer branding, DESIGN.md):
- light+dark themes (OS default, persisted header toggle, no first-paint flash)
- official mascot: lossless WebP light/dark logos + favicon, embedded and served
with cache headers (1.2MB source PNGs -> ~27/22KB webp)
- Assets/Contents views, pagination (offset) and native virtual scrolling
(content-visibility) so large result sets do not blow up the DOM
- search with debounce/abort + URL state, exposure chips, sort, summary counts
Readability + indexing:
- internal/index listers for S3-compatible, GCS, Azure, Firebase, OpenSearch
(metadata only: keys/sizes/dates; never object contents)
- asset_entries table + store write/search; readable resources are indexed and
searchable in the UI
- deep, unauthenticated, read-only DB probes (PROBE_DEEP): Redis PING,
PostgreSQL StartupMessage, MongoDB OP_MSG listDatabases; OpenSearch/Neptune
HTTP reads
- Azure account-level listing fixed: 403 is reported reachable, not
auth_required, since anonymous access is per-container
DNS:
- curated reliable resolvers by default; DNS_SERVERS=fresh adds the compiled-in
Fresh-Resolvers snapshot with a bounded startup sample (warming all 6.7k at
once broke the container resolver)
- random healthy selection + failover + per-server cooldown
Other: prober Close fixes a shutdown hang; config/README updated.
StasonJatham