Bucketeer

Certificate Transparency monitoring for cloud resources and open buckets. Bucketeer consumes CT logs, classifies every certificate identifier into a provider resource (storage bucket, database, data service), discovers and enumerates open buckets, and stores only useful findings — with a small web UI and a PostgreSQL-only state model. One stateless Go process + PostgreSQL.

---

What it does

• Consumes CT logs using CertSpotter internals in-process (no stdout, no JSON, no hooks, no subprocesses, no certificate files).
• Classifies certificate DNS identifiers with a zero-allocation classifier: AWS S3/RDS/Redshift/DocumentDB/Neptune/OpenSearch/MemoryDB/ElastiCache, Azure Blob/File/Queue/Table/SQL/Postgres/MySQL/Redis/Cosmos, Google Cloud Storage, Firebase, Cloudflare R2, DigitalOcean Spaces, Supabase and OCI.
• Enriches hostnames through DNS/CNAME with a bounded resolver pool.
• Discovers open buckets by deriving candidate names from real CT hostnames and testing anonymous listing on S3, GCS and (Azure) account + container.
• Verifies readability with unauthenticated, non-destructive checks and indexes the object keys of anything anonymously listable (metadata only, never contents).
• Persists findings in batches to PostgreSQL; CT resume state lives there too.

A detected endpoint does not mean the data is publicly accessible. Bucketeer reports facts (hostname seen in CT, resolved, answered, listed) separately from conclusions, and never attempts authentication.

Data flow

CT logs ──► CertSpotter internals (fetch/verify/parse)
              │  direct Go call: IdentifierSink.HandleIdentifiers
              ▼
        zero-allocation classifier
        ┌──────┴───────────────┐
   direct match          non-match
        │                     │
        │              DNS/CNAME enrichment (bounded pool)
        │                     │
        │              bucket-name discovery (S3/GCS/Azure)
        │                     │
        └──────────┬──────────┘
                   ▼
        dedup + bounded queues
                   ▼
        optional reachability + readability probe
                   ▼
        anonymous listing → metadata index
                   ▼
        batched PostgreSQL writer
                   ▼
             PostgreSQL

Quick start

cp .env.example .env      # set a strong POSTGRES_PASSWORD and matching DATABASE_URL
docker compose up -d

• UI + API: http://127.0.0.1:9090/
• Health: /healthz, readiness: /readyz, metrics: /metrics

The bucketeer container runs as non-root with a read-only root filesystem, no application volume, dropped Linux capabilities and CPU/memory/pids limits. The only persistent state is PostgreSQL.

Use docker compose stop / start (or up -d) to keep findings. docker compose down -v deletes the pgdata volume and every discovered asset with it.

Configuration

Everything is environment variables (no config files). Sensible defaults apply.

Variable	Default	Purpose
DATABASE_URL	(required)	PostgreSQL connection string
DB_MAX_CONNS	8	pgx pool size
DNS_SERVERS	(curated)	comma-separated resolvers, or system
DNS_WORKERS	64	DNS enrichment workers
DNS_TIMEOUT	5s	per-lookup timeout
DNS_CACHE_SIZE	262144	max cached DNS results
DNS_MAX_CNAME_DEPTH	8	CNAME hops
DNS_POSITIVE_TTL / DNS_NEGATIVE_TTL	10m / 1m	DNS cache TTLs
DEDUP_SIZE / DEDUP_TTL	262144 / 1h	dedup caches
DIRECT_QUEUE_SIZE / DNS_QUEUE_SIZE / DB_QUEUE_SIZE	8192 / 262144 / 8192	bounded queues
DB_BATCH_SIZE / DB_FLUSH_INTERVAL	500 / 2s	write batching
ACTIVE_PROBING	false	enable reachability/readability checks
PROBE_WORKERS / PROBE_TIMEOUT / PROBE_QUEUE_SIZE	8 / 5s / 65536	probe pool
PROBE_DEEP	true	unauthenticated read-only DB protocol checks
INDEX_ENABLED / INDEX_MAX_OBJECTS	true / 1000	index readable resources
BUCKET_SCAN_ENABLED / _QPS / _WORKERS / _QUEUE	true / 50 / 16 / 65536	bucket discovery
METRICS_ADDR	:9090	HTTP listener
UI_USER / UI_PASS	(unset)	optional HTTP basic auth for the UI/API
LOG_LEVEL / LOG_FORMAT	info / text	logging (json for pipelines)
STATS_INTERVAL	60s	aggregated stats interval
LOG_LIST_SOURCE	Google log list v3	CertSpotter log list
START_AT_END	false	skip existing CT backlog on first run
SHUTDOWN_TIMEOUT	20s	graceful shutdown budget
UI_BIND (compose)	127.0.0.1	host interface for the published port

HTTP API and UI

The web UI (Assets / Contents views, light and dark mode) and the API share one listener:

Call	Purpose
GET /api/stats	totals and breakdowns by provider/service/exposure
GET /api/assets?provider=&service=&exposure=&q=&sort=&limit=&offset=	asset list
GET /api/entries?q=&limit=&offset=	search indexed object keys
POST /api/scan {"target":"host-or-URL"}	classify, CNAME-follow, probe and index one target
POST /api/reprobe?exposure=&limit=	re-probe stored assets and index readable ones
GET /metrics /healthz /readyz	Prometheus / liveness / readiness

The listener has no auth by default; set UI_USER/UI_PASS and keep it bound to loopback (or a private network).

Reading the results

exposure_state distinguishes facts, and two capability flags keep them honest:

• public_read — an anonymous read succeeded. For buckets this is only meaningful together with listable.
• listable — the provider's anonymous list API returned object names; only then does Bucketeer store an index. S3/GCS/DO and Azure "Container" mode.
• object_readable — a known object was readable but listing is denied (Azure "Blob" mode, CDN distributions, R2).
• reachable — for databases, the port answered. No authentication is attempted, so a database is never reported as readable.

exposure_detail stores the factual reason (status code, region, container, dns: no public record, tcp/5432 open; no authentication attempted, …).

Bucket discovery

Certificate SANs are custom domains, not bucket names, so matching SANs alone finds almost no buckets. For every non-matching hostname Bucketeer takes the names that actually occur in it (the subdomain label and the registrable-domain label) and tests anonymous listing on S3 (<name>.s3.amazonaws.com) and GCS (<name>.storage.googleapis.com). Azure has two levels — the account maps to DNS and anonymous access is per container with no anonymous container enumeration — so existing accounts are probed against common container names. Every 200 is indexed. A global BUCKET_SCAN_QPS limit and a 12h per-name dedup keep it bounded; contents are never downloaded.

Database schema

• cloud_assets — one row per (provider, service, provider_hostname) with hostname, resource_identifier, region, resource_specific, public_semantic, exposure_state, exposure_detail, listable, object_readable, first_seen, last_seen, probed_at.
• asset_entries — the metadata index of readable resources (key/path, size, updated, content-type).
• observations — sources, CNAME chain and A/AAAA records.
• ct_checkpoints, ct_sths — CertSpotter resume state (no certificate contents).

Schema is applied idempotently at startup from embedded SQL in migrations/.

Performance

Optimized in order: I/O off the hot path, then serialization, then process boundaries, then allocations. The classifier uses no regexp/JSON/reflection/DNS/ HTTP/DB, lowercases only when needed, and parses labels by byte offset.

BenchmarkClassifyAWS-10            60 ns/op    0 B/op   0 allocs/op
BenchmarkClassifyAzure-10          45 ns/op    0 B/op   0 allocs/op
BenchmarkRejectNormalDomain-10     30 ns/op    0 B/op   0 allocs/op
BenchmarkHandleHostNonMatch-10    116 ns/op    0 B/op   0 allocs/op

Run them locally:

go test ./internal/classify/ ./internal/pipeline/ -run '^$' -bench . -benchmem

BenchmarkClassifyMixed and TestHandleHostNoAllocs fail unless the hot path is exactly 0 allocs/op, so the target is enforced by the test suite.

Testing

go test ./...          # unit + table-driven provider tests
go test -race ./...
go vet ./...

PostgreSQL integration tests run when DATABASE_URL is set.

Repository layout

cmd/bucketeer/            application entry point
internal/
  classify/               zero-allocation provider/resource classifier
  config/                 environment configuration
  discover/               bucket-name discovery from CT domains
  dns/                    resolver pool, TTL cache, CNAME following
  index/                  anonymous listing -> metadata index
  metrics/                Prometheus text endpoint
  model/                  shared enums and structs
  pipeline/               bounded queues, dedup, worker pools
  probe/                  reachability + readability checks
  sample/                 bounded random-sample queue
  scan/                   on-demand scan and backfill
  store/                  PostgreSQL schema, batch writer, CT state
  webui/                  embedded web UI
migrations/               embedded SQL schema
third_party/certspotter/  vendored upstream CT engine (minimal patch)
docs/                     DESIGN.md, SKILL.md
assets/                   brand assets

• docs/DESIGN.md — the UI design system.
• docs/SKILL.md — how an agent queries and audits a running instance and answers questions from the API and PostgreSQL.

Limitations

• CT rarely contains bucket hostnames directly; discovery is bounded and stochastic, not exhaustive. Open buckets accumulate over time.
• Azure containers need names (no anonymous enumeration); Cloudflare R2 is never listable.
• The database is the only state; there is no filesystem spool by design.

License

MIT. The vendored CertSpotter fork under third_party/ remains under its original Mozilla Public License v2.0 (third_party/certspotter/LICENSE).