ctcloud

ctcloud continuously consumes Certificate Transparency (CT) logs, finds cloud storage/database/data-service resources referenced by certificates, enriches candidates through DNS/CNAME resolution, and persists only useful findings to PostgreSQL.

It is a single stateless Go process plus PostgreSQL. CertSpotter's real CT fetching, validation and parsing code is embedded and called in-process; there is no stdout handoff, no JSON handoff, no hooks, no subprocesses and no certificate files.

A detected provider endpoint does not mean the underlying data is publicly accessible. ctcloud reports facts (hostname existed in CT, resolved, answered a request) separately from conclusions (exposure_state). Most findings are simply an endpoint that exists.

How it works

CT logs ──► CertSpotter internals (fetch/verify/parse)
              │  direct Go call: IdentifierSink.HandleIdentifiers
              ▼
        fast classifier  (0 allocs/op, no regexp/JSON/DNS)
        ┌──────┴───────┐
   direct match    non-match
        │              ▼
        │        DNS/CNAME enrichment
        │           (followed + classified)
        └──────┬───────┘
               ▼
        dedup + bounded queues
               ▼
     optional safe reachability
               ▼
        batched PostgreSQL writer
               ▼
           PostgreSQL

See ARCHITECTURE.md for per-stage rules.

Why CertSpotter is embedded (and patched)

Re-implementing CT log handling is a large, security-sensitive job. Upstream CertSpotter already does log-list handling, STH fetching/verification, Merkle tree maintenance, entry parsing and certificate reconstruction.

The only change is a two-hunk patch in third_party/certspotter that adds an IdentifierSink callback. After a certificate is parsed, the identifiers are handed to ctcloud with a direct Go function call and the watchlist/chain/ notification machinery is skipped:

if config.IdentifierSink != nil {
    config.IdentifierSink.HandleIdentifiers(ctx, identifiers)
    return nil
}

See third_party/certspotter/CTCLOUD_PATCH.md. This deliberately avoids stdout piping, JSON files, hook scripts, subprocesses and temporary certificate files, which are all slow, serializing and filesystem-bound. Upstream licensing/notices are preserved and the patch is isolated so it can be rebased.

Supported provider patterns

AWS — S3 (<bucket>.s3[.dualstack].<region>.amazonaws.com, <bucket>.s3-<region>.amazonaws.com legacy form, s3-website[-<region>]), S3 access points (s3-accesspoint[-fips], dualstack), multi-region access points (mrap.accesspoint.s3-global), S3 Express (s3express-*), RDS/Aurora/proxy (*.rds.amazonaws.com), DocumentDB, Redshift (provisioned + serverless), OpenSearch (search-* public / vpc-* VPC), Neptune, MemoryDB, ElastiCache. Generic API endpoints (s3.eu-central-1.amazonaws.com, dynamodb.*, redshift.<region>, rds.<region>, …) are rejected.

Azure — blob|dfs|file|queue|table|web.core.windows.net, *.database.windows.net, *.postgres|mysql.database.azure.com, *.documents.azure.com, *.{mongo,cassandra,gremlin}.cosmos.azure.com, *.redis.cache.windows.net, *.redis.azure.net, *.search.windows.net. Bare service endpoints are rejected.

Google Cloud / Firebase — <bucket>.storage.googleapis.com, <db>.firebaseio.com, <db>.<region>.firebasedatabase.app, <project>.appspot.com. storage.googleapis.com itself is rejected.

Cloudflare R2 — <account>.r2.cloudflarestorage.com; <account>.r2.dev is flagged as an explicitly public URL form.

DigitalOcean — <bucket>.<region>[.cdn].digitaloceanspaces.com, *.db.ondigitalocean.com. <region>.digitaloceanspaces.com is rejected.

Supabase — <ref>.supabase.co, db.<ref>.supabase.co (PostgreSQL), aws-<n>-<region>.pooler.supabase.com.

Oracle Cloud (OCI) — <namespace>.objectstorage|swiftobjectstorage.<region>.oci.customer-oci.com (resource-specific), and the generic objectstorage.<region>.oraclecloud.com forms (resource_specific = false; not stored as findings).

Provider-specific details live in internal/classify/.

Configuration

Environment variables only; no config files, no runtime pattern files.

Variable	Default	Purpose
DATABASE_URL	(required)	PostgreSQL connection string
DB_MAX_CONNS	8	pgx pool size
DNS_SERVERS	(unset → curated + community list)	comma-separated resolvers, or system for the OS resolver
DNS_WORKERS	16	DNS enrichment workers
DNS_TIMEOUT	5s	per-lookup timeout
DNS_CACHE_SIZE	262144	max DNS cache entries
DEDUP_SIZE	262144	max entries in each dedup cache
DNS_MAX_CNAME_DEPTH	8	max CNAME hops
DNS_POSITIVE_TTL / DNS_NEGATIVE_TTL	10m / 1m	DNS cache TTLs
DIRECT_QUEUE_SIZE	8192	direct-match queue capacity
DNS_QUEUE_SIZE	65536	DNS-enrichment queue capacity
DB_QUEUE_SIZE	8192	DB-write queue capacity
DB_BATCH_SIZE	500	findings per write batch
DB_FLUSH_INTERVAL	2s	max time before a batch is flushed
ACTIVE_PROBING	false	enable optional reachability checks
PROBE_WORKERS / PROBE_TIMEOUT	8 / 5s	probe worker pool
PROBE_QUEUE_SIZE	65536	probe sample size; when full, old entries are replaced at random (bounded memory)
PROBE_DEEP	true	unauthenticated, read-only protocol reads for databases (Redis/Postgres/Mongo)
INDEX_ENABLED / INDEX_MAX_OBJECTS	true / 1000	index metadata of readable resources (keys/sizes only)
LOG_FORMAT	text	text or json (JSON is best for log pipelines/LLM analysis)
BUCKET_SCAN_ENABLED	true	discover open buckets by name enumeration
BUCKET_SCAN_QPS	50	global cap on outbound discovery requests
BUCKET_SCAN_WORKERS	16	discovery probe workers
BUCKET_SCAN_QUEUE	65536	sampled domain backlog
DEDUP_TTL	1h	hostname/finding dedup window
METRICS_ADDR	:9090	metrics/health/UI listener
UI_USER / UI_PASS	(unset)	optional HTTP basic auth for the web UI
LOG_LEVEL	info	debug|info|warn|error
STATS_INTERVAL	60s	aggregated stats log interval
LOG_LIST_SOURCE	Google log list v3	certspotter log list URL
START_AT_END	false	skip existing backlog on first run
SHUTDOWN_TIMEOUT	20s	graceful shutdown budget

Running with Docker

cp .env.example .env      # set a strong POSTGRES_PASSWORD and matching DATABASE_URL
docker compose up --build

That starts postgres and the stateless ctcloud container. The application container runs as non-root, has a read-only root filesystem, drops all Linux capabilities, sets CPU/memory/pids limits and mounts no application volume. The only persistent state is PostgreSQL (CT checkpoints + cloud assets).

Endpoints on port 9090, bound to 127.0.0.1 only:

• GET / — web UI (see below)
• GET /api/assets, GET /api/stats — JSON for the UI
• GET /metrics, GET /healthz, GET /readyz

The listener has no authentication by default, so scrape it locally or from a sidecar and never expose it publicly. Set UI_USER/UI_PASS to require HTTP basic auth on the UI and its API.

Web UI

A small, dependency-free UI (vanilla JS, no build step, embedded in the binary) styled to the tokens and layout in DESIGN.md (neutral canvas, single 768px results column, flat rows, pill controls, no shadows).

• search box with 250ms debounce, stale-request cancellation and ?q= URL state
• exposure filter chips — use Public read to see open buckets, Reachable to see publicly reachable database endpoints — plus a sort control
• result rows show the provider hostname, probe detail, provider/service/region badges, resource identifier and relative last-seen/probed times
• summary line with total assets, publicly readable and reachable counts
• refreshes every 15s (paused when the tab is hidden), read-only
• light and dark mode: follows the OS by default, with a header toggle that is persisted in localStorage; the theme is applied before first paint to avoid a flash

Served from the same loopback port: http://127.0.0.1:9090/.

Brand assets

The UI ships the official Bucketeer mascot for both themes, optimized to lossless WebP and embedded in the binary (internal/webui/assets/):

Asset	Size	Use
logo-light.webp	~27 KB	black line art, light theme
logo-dark.webp	~22 KB	white line art, dark theme
favicon.png	~7 KB	browser tab

They are downscaled to 256px and encoded with cwebp -lossless -z 9, so they stay crisp on HiDPI displays while shrinking the ~1.2 MB source PNGs by ~96%. Regenerate after replacing the source PNGs at the repo root:

magick bucketeer-light.png -resize 256x -strip /tmp/l.png
magick bucketeer-dark.png  -resize 256x -strip /tmp/d.png
cwebp -quiet -lossless -z 9 -alpha_q 100 /tmp/l.png -o internal/webui/assets/logo-light.webp
cwebp -quiet -lossless -z 9 -alpha_q 100 /tmp/d.png -o internal/webui/assets/logo-dark.webp

Reachability and readability checks

ACTIVE_PROBING=false by default. When enabled, the prober issues one unauthenticated request per finding. It never sends credentials, downloads objects, mutates anything, runs database queries or attempts authentication.

Vendor	Check	public_read means
AWS S3, Cloudflare R2, DigitalOcean Spaces, OCI	anonymous ListObjectsV2 (?list-type=2&max-keys=1)	bucket policy grants s3:ListBucket to *
Google Cloud Storage	JSON API object list (/storage/v1/b/<bucket>/o?maxResults=1)	the bucket grants allUsers storage.objects.list
Azure Blob/File/Queue/Table	account-level ?comp=list	anonymous access is per container, so account-level 403 is reported reachable (not "not readable"). Container names cannot be derived from the account hostname.
Firebase Realtime DB	GET /.json?shallow=true	security rules allow anonymous read
Supabase	GET /rest/v1/	not testable without the public anon API key; reports auth_required
OpenSearch / Elasticsearch	GET /_cat/indices	security/access control disabled
Neptune	GET /status	a 2xx was returned
Cosmos / Appspot / Azure Search	root GET /	a 2xx was returned
Redis (ElastiCache, MemoryDB, Azure Redis)	unauthenticated PING	+PONG without AUTH
PostgreSQL (RDS/Aurora, Redshift, Azure, Supabase)	unauthenticated StartupMessage	AuthenticationOk (server trusts the connection)
MongoDB (DocumentDB, Cosmos Mongo)	OP_MSG listDatabases	command succeeds without auth
MySQL, Azure SQL, Cassandra (Cosmos)	TCP connect only	never — reported reachable

Every check sends no credentials: no Authorization header, no password, no token. "Anonymous" here means an unauthenticated request, exactly as an anonymous cloud principal would make.

Deep protocol probes (PROBE_DEEP, default on) only ever issue read-only requests or commands; nothing is written, and no authentication is attempted.

Probe outcomes are stored in exposure_state and a short factual exposure_detail string (e.g. s3 redirect region=eu-central-1, status=403 authentication required, tcp/5432 open; no authentication attempted). Database endpoints are never claimed to be readable: a raw public_read on a database is impossible by design.

Production notes

• Secrets — database credentials live in .env (gitignored); .env.example is the template. Inject the same variables from your orchestrator's secret store in production.
• Database TLS — the local compose stack uses sslmode=disable because the compose bridge network is private. For production, point DATABASE_URL at a TLS-enabled database, e.g. postgres://user:[email protected]:5432/ctcloud?sslmode=verify-full. pgx handles TLS natively; no extra flags are needed.
• Metrics — bound to loopback. There is no auth, no debug endpoints and no high-cardinality labels (never hostnames).
• Resource limits — mem_limit/cpus/pids_limit are set in compose so overload is bounded by the queue sizes rather than by the OOM killer.
• Authorization — leave ACTIVE_PROBING=false unless you are authorised to probe the discovered infrastructure.

Database schema

• cloud_assets — one row per (provider, service, provider_hostname), with hostname (original CT hostname), resource_identifier, region, resource_specific, public_semantic, exposure_state, exposure_detail, probed_at, first_seen, last_seen.
• observations — append-only observations linking an asset to a source_hostname, source_type (direct_ct/cname), CNAME chain and A/AAAA records.
• ct_checkpoints — per-log CT resume state (JSON-encoded certspotter LogState; Merkle positions, no certificates).
• ct_sths — downloaded, unverified signed tree heads needed for safe resume.

Schema is applied idempotently at startup from embedded SQL in migrations/. No certificate contents are ever stored.

Performance characteristics

Optimised in this order: I/O off the hot path, then serialization, then process boundaries, then allocations, then cheap suffix/label matching, then bounded pools, then DNS caching, dedup and batched writes.

The classifier:

• no regexp, JSON, reflection, DNS, HTTP, filesystem, DB or logging;
• no strings.Split, no fmt.Sprintf;
• lowercases only when an uppercase byte is present;
• parses labels by byte offsets, returning substrings (zero copy);
• fixed stack arrays, no unbounded maps or goroutines.

Benchmark results (Apple M4, Go 1.27)

BenchmarkClassifyAWS-10            60.03 ns/op    0 B/op   0 allocs/op
BenchmarkClassifyAzure-10          44.93 ns/op    0 B/op   0 allocs/op
BenchmarkRejectNormalDomain-10     30.28 ns/op    0 B/op   0 allocs/op
BenchmarkClassifyMixed-10         369.50 ns/op    0 B/op   0 allocs/op

The whole in-process sink path is allocation-free too:

BenchmarkHandleHostNonMatch-10      115.6 ns/op    0 B/op   0 allocs/op
BenchmarkHandleHostDirectMatch-10  155.7 ns/op    0 B/op   0 allocs/op

Run them yourself:

go test ./internal/classify/ ./internal/pipeline/ -run '^$' -bench . -benchmem

BenchmarkClassifyMixed and TestHandleHostNoAllocs fail unless the result is exactly 0 allocs/op, so the target is enforced by the test suite.

Bucket discovery

Certificate SANs are custom domains, not bucket names, so matching SANs alone finds almost no buckets. Instead, every non-matching hostname feeds a discovery engine that:

1. takes the names that actually occur in the hostname — the subdomain label and the registrable-domain label (assets.example.com → assets, example). No names are fabricated or drawn from a wordlist;
2. tests anonymous listing — https://<name>.s3.amazonaws.com/?list-type=2 (AWS) and https://<name>.storage.googleapis.com (GCS);
3. indexes every bucket that returns 200 (keys/sizes/dates only).

It is aggressively bounded: a global BUCKET_SCAN_QPS limit, a 12h per-name dedup cache, and a random-sampled domain queue. Azure cannot be enumerated this way (account names are the namespace and containers need names), so Azure is handled from CT accounts plus a container wordlist, and R2 is never listable.

This is what actually yields open buckets. Of the buckets found, S3 and GCS return full object listings; Azure needs a known container; R2 only serves objects you already know.

DNS load balancing

When DNS_SERVERS is unset, the pool is ~20 curated, reliable public resolvers (Cloudflare, Google, Quad9, OpenDNS, AdGuard, …). Each query picks a random healthy resolver and fails over to at most two others; each attempt is bounded (2s) and resolvers that fail repeatedly enter a 5-minute cooldown. Definitive answers (NXDOMAIN) return immediately rather than being retried on every server, so negative lookups do not multiply DNS traffic.

For sensitive deployments set DNS_SERVERS to your own resolvers, or DNS_SERVERS=system.

These community resolvers are untrusted: they can observe and, in principle, forge DNS answers. That only affects enrichment (which endpoints are looked up), never the classifier. For sensitive deployments set DNS_SERVERS to your own resolvers, or DNS_SERVERS=system.

Testing

go test ./...          # unit + table-driven provider tests
go test -race ./...    # race detector
go vet ./...

Integration tests for PostgreSQL run only when DATABASE_URL is set:

DATABASE_URL=postgres://ctcloud:ctcloud@localhost:5432/ctcloud go test ./internal/store/

Limitations

• net.Resolver.LookupCNAME returns the final canonical name, so intermediate CNAME hops are not observed (the final target is still classified). Switch to miekg/dns if the full chain is required.
• Optional reachability checks are off by default; when enabled they perform a single non-destructive HTTPS GET / only — no authentication, downloads, mutation or database queries.
• OCI generic oraclecloud.com endpoints are recognised but not stored, since the bucket name lives in the URL path rather than the hostname.
• Custom R2 (and other) domains are only discovered through DNS/CNAME enrichment, and only if the CT hostname itself appears in a certificate.