ctcloud

ctcloud continuously consumes Certificate Transparency (CT) logs, finds cloud storage/database/data-service resources referenced by certificates, enriches candidates through DNS/CNAME resolution, and persists only useful findings to PostgreSQL.

It is a single stateless Go process plus PostgreSQL. CertSpotter's real CT fetching, validation and parsing code is embedded and called in-process; there is no stdout handoff, no JSON handoff, no hooks, no subprocesses and no certificate files.

A detected provider endpoint does not mean the underlying data is publicly accessible. ctcloud reports facts (hostname existed in CT, resolved, answered a request) separately from conclusions (exposure_state). Most findings are simply an endpoint that exists.

How it works

CT logs ──► CertSpotter internals (fetch/verify/parse)
              │  direct Go call: IdentifierSink.HandleIdentifiers
              ▼
        fast classifier  (0 allocs/op, no regexp/JSON/DNS)
        ┌──────┴───────┐
   direct match    non-match
        │              ▼
        │        DNS/CNAME enrichment
        │           (followed + classified)
        └──────┬───────┘
               ▼
        dedup + bounded queues
               ▼
     optional safe reachability
               ▼
        batched PostgreSQL writer
               ▼
           PostgreSQL

See ARCHITECTURE.md for per-stage rules.

Why CertSpotter is embedded (and patched)

Re-implementing CT log handling is a large, security-sensitive job. Upstream CertSpotter already does log-list handling, STH fetching/verification, Merkle tree maintenance, entry parsing and certificate reconstruction.

The only change is a two-hunk patch in third_party/certspotter that adds an IdentifierSink callback. After a certificate is parsed, the identifiers are handed to ctcloud with a direct Go function call and the watchlist/chain/ notification machinery is skipped:

if config.IdentifierSink != nil {
    config.IdentifierSink.HandleIdentifiers(ctx, identifiers)
    return nil
}

See third_party/certspotter/CTCLOUD_PATCH.md. This deliberately avoids stdout piping, JSON files, hook scripts, subprocesses and temporary certificate files, which are all slow, serializing and filesystem-bound. Upstream licensing/notices are preserved and the patch is isolated so it can be rebased.

Supported provider patterns

AWS — S3 (<bucket>.s3[.dualstack].<region>.amazonaws.com, s3-website[-<region>]), S3 access points (s3-accesspoint[-fips], dualstack), multi-region access points (mrap.accesspoint.s3-global), S3 Express (s3express-*), RDS/Aurora/proxy (*.rds.amazonaws.com), DocumentDB, Redshift (provisioned + serverless), OpenSearch (search-* public / vpc-* VPC), Neptune, MemoryDB, ElastiCache. Generic API endpoints (s3.eu-central-1.amazonaws.com, dynamodb.*, redshift.<region>, rds.<region>, …) are rejected.

Azure — blob|dfs|file|queue|table|web.core.windows.net, *.database.windows.net, *.postgres|mysql.database.azure.com, *.documents.azure.com, *.{mongo,cassandra,gremlin}.cosmos.azure.com, *.redis.cache.windows.net, *.redis.azure.net, *.search.windows.net. Bare service endpoints are rejected.

Google Cloud / Firebase — <bucket>.storage.googleapis.com, <db>.firebaseio.com, <db>.<region>.firebasedatabase.app, <project>.appspot.com. storage.googleapis.com itself is rejected.

Cloudflare R2 — <account>.r2.cloudflarestorage.com; <account>.r2.dev is flagged as an explicitly public URL form.

DigitalOcean — <bucket>.<region>[.cdn].digitaloceanspaces.com, *.db.ondigitalocean.com. <region>.digitaloceanspaces.com is rejected.

Supabase — <ref>.supabase.co, db.<ref>.supabase.co (PostgreSQL), aws-<n>-<region>.pooler.supabase.com.

Oracle Cloud (OCI) — <namespace>.objectstorage|swiftobjectstorage.<region>.oci.customer-oci.com (resource-specific), and the generic objectstorage.<region>.oraclecloud.com forms (resource_specific = false; not stored as findings).

Provider-specific details live in internal/classify/.

Configuration

Environment variables only; no config files, no runtime pattern files.

Variable	Default	Purpose
DATABASE_URL	(required)	PostgreSQL connection string
DB_MAX_CONNS	8	pgx pool size
DNS_SERVERS	1.1.1.1:53,8.8.8.8:53,1.0.0.1:53	comma-separated resolvers (round-robin + failover); system for the host resolver
DNS_WORKERS	16	DNS enrichment workers
DNS_TIMEOUT	5s	per-lookup timeout
DNS_CACHE_SIZE	262144	max DNS cache entries
DEDUP_SIZE	262144	max entries in each dedup cache
DNS_MAX_CNAME_DEPTH	8	max CNAME hops
DNS_POSITIVE_TTL / DNS_NEGATIVE_TTL	10m / 1m	DNS cache TTLs
DIRECT_QUEUE_SIZE	8192	direct-match queue capacity
DNS_QUEUE_SIZE	65536	DNS-enrichment queue capacity
DB_QUEUE_SIZE	8192	DB-write queue capacity
DB_BATCH_SIZE	500	findings per write batch
DB_FLUSH_INTERVAL	2s	max time before a batch is flushed
ACTIVE_PROBING	false	enable optional reachability checks
PROBE_WORKERS / PROBE_TIMEOUT	8 / 5s	probe worker pool
PROBE_QUEUE_SIZE	65536	probe sample size; when full, old entries are replaced at random (bounded memory)
DEDUP_TTL	1h	hostname/finding dedup window
METRICS_ADDR	:9090	metrics/health/UI listener
UI_USER / UI_PASS	(unset)	optional HTTP basic auth for the web UI
LOG_LEVEL	info	debug|info|warn|error
STATS_INTERVAL	60s	aggregated stats log interval
LOG_LIST_SOURCE	Google log list v3	certspotter log list URL
START_AT_END	false	skip existing backlog on first run
SHUTDOWN_TIMEOUT	20s	graceful shutdown budget

Running with Docker

cp .env.example .env      # set a strong POSTGRES_PASSWORD and matching DATABASE_URL
docker compose up --build

That starts postgres and the stateless ctcloud container. The application container runs as non-root, has a read-only root filesystem, drops all Linux capabilities, sets CPU/memory/pids limits and mounts no application volume. The only persistent state is PostgreSQL (CT checkpoints + cloud assets).

Endpoints on port 9090, bound to 127.0.0.1 only:

• GET / — web UI (see below)
• GET /api/assets, GET /api/stats — JSON for the UI
• GET /metrics, GET /healthz, GET /readyz

The listener has no authentication by default, so scrape it locally or from a sidecar and never expose it publicly. Set UI_USER/UI_PASS to require HTTP basic auth on the UI and its API.

Web UI

A small, dependency-free UI lists discovered assets with provider/service, resource, region, exposure state and the probe detail, plus summary counts and provider/service/exposure filters. It is read-only, refreshes every 15s, and is served from the same loopback port (http://127.0.0.1:9090/).

Reachability and readability checks

ACTIVE_PROBING=false by default. When enabled, the prober issues one unauthenticated request per finding. It never sends credentials, downloads objects, mutates anything, runs database queries or attempts authentication.

Vendor	Check	public_read means
AWS S3, Cloudflare R2, DigitalOcean Spaces	anonymous ListObjectsV2 (?list-type=2&max-keys=1)	anonymous bucket listing succeeded
Google Cloud Storage	JSON API object list (/storage/v1/b/<bucket>/o?maxResults=1)	allUsers can list objects
Azure Blob/File/Queue/Table	anonymous ?comp=list	anonymous listing succeeded
Firebase Realtime DB	GET /.json?shallow=true	security rules allow anonymous read
Supabase	GET /rest/v1/	— (an API key is required, so this reports auth_required)
Cosmos, OpenSearch, Search, Appspot, OCI	root GET /	a 2xx was returned
RDS, Redshift, DocumentDB, Neptune, ElastiCache, MemoryDB, Azure SQL/Postgres/MySQL/Redis, Supabase Postgres	unauthenticated TCP connect only	never — reported as reachable (port open)

Probe outcomes are stored in exposure_state and a short factual exposure_detail string (e.g. s3 redirect region=eu-central-1, status=403 authentication required, tcp/5432 open; no authentication attempted). Database endpoints are never claimed to be readable: a raw public_read on a database is impossible by design.

Production notes

• Secrets — database credentials live in .env (gitignored); .env.example is the template. Inject the same variables from your orchestrator's secret store in production.
• Database TLS — the local compose stack uses sslmode=disable because the compose bridge network is private. For production, point DATABASE_URL at a TLS-enabled database, e.g. postgres://user:[email protected]:5432/ctcloud?sslmode=verify-full. pgx handles TLS natively; no extra flags are needed.
• Metrics — bound to loopback. There is no auth, no debug endpoints and no high-cardinality labels (never hostnames).
• Resource limits — mem_limit/cpus/pids_limit are set in compose so overload is bounded by the queue sizes rather than by the OOM killer.
• Authorization — leave ACTIVE_PROBING=false unless you are authorised to probe the discovered infrastructure.

Database schema

• cloud_assets — one row per (provider, service, provider_hostname), with hostname (original CT hostname), resource_identifier, region, resource_specific, public_semantic, exposure_state, exposure_detail, probed_at, first_seen, last_seen.
• observations — append-only observations linking an asset to a source_hostname, source_type (direct_ct/cname), CNAME chain and A/AAAA records.
• ct_checkpoints — per-log CT resume state (JSON-encoded certspotter LogState; Merkle positions, no certificates).
• ct_sths — downloaded, unverified signed tree heads needed for safe resume.

Schema is applied idempotently at startup from embedded SQL in migrations/. No certificate contents are ever stored.

Performance characteristics

Optimised in this order: I/O off the hot path, then serialization, then process boundaries, then allocations, then cheap suffix/label matching, then bounded pools, then DNS caching, dedup and batched writes.

The classifier:

• no regexp, JSON, reflection, DNS, HTTP, filesystem, DB or logging;
• no strings.Split, no fmt.Sprintf;
• lowercases only when an uppercase byte is present;
• parses labels by byte offsets, returning substrings (zero copy);
• fixed stack arrays, no unbounded maps or goroutines.

Benchmark results (Apple M4, Go 1.27)

BenchmarkClassifyAWS-10            60.03 ns/op    0 B/op   0 allocs/op
BenchmarkClassifyAzure-10          44.93 ns/op    0 B/op   0 allocs/op
BenchmarkRejectNormalDomain-10     30.28 ns/op    0 B/op   0 allocs/op
BenchmarkClassifyMixed-10         369.50 ns/op    0 B/op   0 allocs/op

The whole in-process sink path is allocation-free too:

BenchmarkHandleHostNonMatch-10      115.6 ns/op    0 B/op   0 allocs/op
BenchmarkHandleHostDirectMatch-10  155.7 ns/op    0 B/op   0 allocs/op

Run them yourself:

go test ./internal/classify/ ./internal/pipeline/ -run '^$' -bench . -benchmem

BenchmarkClassifyMixed and TestHandleHostNoAllocs fail unless the result is exactly 0 allocs/op, so the target is enforced by the test suite.

DNS load balancing

DNS lookups are round-robined across DNS_SERVERS. Each attempt is bounded (2s) and fails over to the next server on timeout/refused errors, so one dead resolver does not stall lookups. Definitive answers (NXDOMAIN) are returned immediately rather than retried on every server, so negative lookups do not multiply DNS traffic.

Testing

go test ./...          # unit + table-driven provider tests
go test -race ./...    # race detector
go vet ./...

Integration tests for PostgreSQL run only when DATABASE_URL is set:

DATABASE_URL=postgres://ctcloud:ctcloud@localhost:5432/ctcloud go test ./internal/store/

Limitations

• net.Resolver.LookupCNAME returns the final canonical name, so intermediate CNAME hops are not observed (the final target is still classified). Switch to miekg/dns if the full chain is required.
• Optional reachability checks are off by default; when enabled they perform a single non-destructive HTTPS GET / only — no authentication, downloads, mutation or database queries.
• OCI generic oraclecloud.com endpoints are recognised but not stored, since the bucket name lives in the URL path rather than the hostname.
• Custom R2 (and other) domains are only discovered through DNS/CNAME enrichment, and only if the CT hostname itself appears in a certificate.