# ctcloud `ctcloud` continuously consumes Certificate Transparency (CT) logs, finds cloud storage/database/data-service resources referenced by certificates, enriches candidates through DNS/CNAME resolution, and persists only useful findings to PostgreSQL. It is a **single stateless Go process plus PostgreSQL**. CertSpotter's real CT fetching, validation and parsing code is embedded and called **in-process**; there is no stdout handoff, no JSON handoff, no hooks, no subprocesses and no certificate files. > A detected provider endpoint **does not** mean the underlying data is publicly > accessible. `ctcloud` reports facts (hostname existed in CT, resolved, answered > a request) separately from conclusions (`exposure_state`). Most findings are > simply an endpoint that exists. ## How it works ```text CT logs ──► CertSpotter internals (fetch/verify/parse) │ direct Go call: IdentifierSink.HandleIdentifiers ▼ fast classifier (0 allocs/op, no regexp/JSON/DNS) ┌──────┴───────┐ direct match non-match │ ▼ │ DNS/CNAME enrichment │ (followed + classified) └──────┬───────┘ ▼ dedup + bounded queues ▼ optional safe reachability ▼ batched PostgreSQL writer ▼ PostgreSQL ``` See [ARCHITECTURE.md](ARCHITECTURE.md) for per-stage rules. ## Why CertSpotter is embedded (and patched) Re-implementing CT log handling is a large, security-sensitive job. Upstream CertSpotter already does log-list handling, STH fetching/verification, Merkle tree maintenance, entry parsing and certificate reconstruction. The only change is a two-hunk patch in `third_party/certspotter` that adds an `IdentifierSink` callback. After a certificate is parsed, the identifiers are handed to ctcloud with a direct Go function call and the watchlist/chain/ notification machinery is skipped: ```go if config.IdentifierSink != nil { config.IdentifierSink.HandleIdentifiers(ctx, identifiers) return nil } ``` See [`third_party/certspotter/CTCLOUD_PATCH.md`](third_party/certspotter/CTCLOUD_PATCH.md). This deliberately avoids stdout piping, JSON files, hook scripts, subprocesses and temporary certificate files, which are all slow, serializing and filesystem-bound. Upstream licensing/notices are preserved and the patch is isolated so it can be rebased. ## Supported provider patterns **AWS** — S3 (`.s3[.dualstack]..amazonaws.com`, `.s3-.amazonaws.com` legacy form, `s3-website[-]`), S3 access points (`s3-accesspoint[-fips]`, `dualstack`), multi-region access points (`mrap.accesspoint.s3-global`), S3 Express (`s3express-*`), RDS/Aurora/proxy (`*.rds.amazonaws.com`), DocumentDB, Redshift (provisioned + serverless), OpenSearch (`search-*` public / `vpc-*` VPC), Neptune, MemoryDB, ElastiCache. Generic API endpoints (`s3.eu-central-1.amazonaws.com`, `dynamodb.*`, `redshift.`, `rds.`, …) are rejected. **Azure** — `blob|dfs|file|queue|table|web.core.windows.net`, `*.database.windows.net`, `*.postgres|mysql.database.azure.com`, `*.documents.azure.com`, `*.{mongo,cassandra,gremlin}.cosmos.azure.com`, `*.redis.cache.windows.net`, `*.redis.azure.net`, `*.search.windows.net`. Bare service endpoints are rejected. **Google Cloud / Firebase** — `.storage.googleapis.com`, `.firebaseio.com`, `..firebasedatabase.app`, `.appspot.com`. `storage.googleapis.com` itself is rejected. **Cloudflare R2** — `.r2.cloudflarestorage.com`; `.r2.dev` is flagged as an explicitly public URL form. **DigitalOcean** — `.[.cdn].digitaloceanspaces.com`, `*.db.ondigitalocean.com`. `.digitaloceanspaces.com` is rejected. **Supabase** — `.supabase.co`, `db..supabase.co` (PostgreSQL), `aws--.pooler.supabase.com`. **Oracle Cloud (OCI)** — `.objectstorage|swiftobjectstorage..oci.customer-oci.com` (resource-specific), and the generic `objectstorage..oraclecloud.com` forms (`resource_specific = false`; not stored as findings). Provider-specific details live in `internal/classify/`. ## Configuration Environment variables only; no config files, no runtime pattern files. | Variable | Default | Purpose | | --- | --- | --- | | `DATABASE_URL` | *(required)* | PostgreSQL connection string | | `DB_MAX_CONNS` | `8` | pgx pool size | | `DNS_SERVERS` | *(unset → curated + community list)* | comma-separated resolvers, or `system` for the OS resolver | | `DNS_WORKERS` | `16` | DNS enrichment workers | | `DNS_TIMEOUT` | `5s` | per-lookup timeout | | `DNS_CACHE_SIZE` | `262144` | max DNS cache entries | | `DEDUP_SIZE` | `262144` | max entries in each dedup cache | | `DNS_MAX_CNAME_DEPTH` | `8` | max CNAME hops | | `DNS_POSITIVE_TTL` / `DNS_NEGATIVE_TTL` | `10m` / `1m` | DNS cache TTLs | | `DIRECT_QUEUE_SIZE` | `8192` | direct-match queue capacity | | `DNS_QUEUE_SIZE` | `65536` | DNS-enrichment queue capacity | | `DB_QUEUE_SIZE` | `8192` | DB-write queue capacity | | `DB_BATCH_SIZE` | `500` | findings per write batch | | `DB_FLUSH_INTERVAL` | `2s` | max time before a batch is flushed | | `ACTIVE_PROBING` | `false` | enable optional reachability checks | | `PROBE_WORKERS` / `PROBE_TIMEOUT` | `8` / `5s` | probe worker pool | | `PROBE_QUEUE_SIZE` | `65536` | probe sample size; when full, old entries are replaced at random (bounded memory) | | `PROBE_DEEP` | `true` | unauthenticated, read-only protocol reads for databases (Redis/Postgres/Mongo) | | `INDEX_ENABLED` / `INDEX_MAX_OBJECTS` | `true` / `1000` | index metadata of readable resources (keys/sizes only) | | `LOG_FORMAT` | `text` | `text` or `json` (JSON is best for log pipelines/LLM analysis) | | `DEDUP_TTL` | `1h` | hostname/finding dedup window | | `METRICS_ADDR` | `:9090` | metrics/health/UI listener | | `UI_USER` / `UI_PASS` | *(unset)* | optional HTTP basic auth for the web UI | | `LOG_LEVEL` | `info` | `debug\|info\|warn\|error` | | `STATS_INTERVAL` | `60s` | aggregated stats log interval | | `LOG_LIST_SOURCE` | Google log list v3 | certspotter log list URL | | `START_AT_END` | `false` | skip existing backlog on first run | | `SHUTDOWN_TIMEOUT` | `20s` | graceful shutdown budget | ## Running with Docker ```sh cp .env.example .env # set a strong POSTGRES_PASSWORD and matching DATABASE_URL docker compose up --build ``` That starts `postgres` and the stateless `ctcloud` container. The application container runs as non-root, has a read-only root filesystem, drops all Linux capabilities, sets CPU/memory/pids limits and mounts no application volume. The only persistent state is PostgreSQL (CT checkpoints + cloud assets). Endpoints on port `9090`, bound to `127.0.0.1` only: * `GET /` — web UI (see below) * `GET /api/assets`, `GET /api/stats` — JSON for the UI * `GET /metrics`, `GET /healthz`, `GET /readyz` The listener has no authentication by default, so scrape it locally or from a sidecar and never expose it publicly. Set `UI_USER`/`UI_PASS` to require HTTP basic auth on the UI and its API. ## Web UI A small, dependency-free UI (vanilla JS, no build step, embedded in the binary) styled to the tokens and layout in `DESIGN.md` (neutral canvas, single 768px results column, flat rows, pill controls, no shadows). * search box with 250ms debounce, stale-request cancellation and `?q=` URL state * exposure filter chips — use **Public read** to see open buckets, **Reachable** to see publicly reachable database endpoints — plus a sort control * result rows show the provider hostname, probe detail, provider/service/region badges, resource identifier and relative last-seen/probed times * summary line with total assets, publicly readable and reachable counts * refreshes every 15s (paused when the tab is hidden), read-only * light and dark mode: follows the OS by default, with a header toggle that is persisted in `localStorage`; the theme is applied before first paint to avoid a flash Served from the same loopback port: `http://127.0.0.1:9090/`. ### Brand assets The UI ships the official Bucketeer mascot for both themes, optimized to lossless WebP and embedded in the binary (`internal/webui/assets/`): | Asset | Size | Use | | --- | --- | --- | | `logo-light.webp` | ~27 KB | black line art, light theme | | `logo-dark.webp` | ~22 KB | white line art, dark theme | | `favicon.png` | ~7 KB | browser tab | They are downscaled to 256px and encoded with `cwebp -lossless -z 9`, so they stay crisp on HiDPI displays while shrinking the ~1.2 MB source PNGs by ~96%. Regenerate after replacing the source PNGs at the repo root: ```sh magick bucketeer-light.png -resize 256x -strip /tmp/l.png magick bucketeer-dark.png -resize 256x -strip /tmp/d.png cwebp -quiet -lossless -z 9 -alpha_q 100 /tmp/l.png -o internal/webui/assets/logo-light.webp cwebp -quiet -lossless -z 9 -alpha_q 100 /tmp/d.png -o internal/webui/assets/logo-dark.webp ``` ## Reachability and readability checks `ACTIVE_PROBING=false` by default. When enabled, the prober issues **one** unauthenticated request per finding. It never sends credentials, downloads objects, mutates anything, runs database queries or attempts authentication. | Vendor | Check | `public_read` means | | --- | --- | --- | | AWS S3, Cloudflare R2, DigitalOcean Spaces, OCI | anonymous `ListObjectsV2` (`?list-type=2&max-keys=1`) | bucket policy grants `s3:ListBucket` to `*` | | Google Cloud Storage | JSON API object list (`/storage/v1/b//o?maxResults=1`) | the bucket grants `allUsers` `storage.objects.list` | | Azure Blob/File/Queue/Table | account-level `?comp=list` | anonymous access is **per container**, so account-level 403 is reported `reachable` (not "not readable"). Container names cannot be derived from the account hostname. | | Firebase Realtime DB | `GET /.json?shallow=true` | security rules allow anonymous read | | Supabase | `GET /rest/v1/` | not testable without the public **anon API key**; reports `auth_required` | | OpenSearch / Elasticsearch | `GET /_cat/indices` | security/access control disabled | | Neptune | `GET /status` | a 2xx was returned | | Cosmos / Appspot / Azure Search | root `GET /` | a 2xx was returned | | Redis (ElastiCache, MemoryDB, Azure Redis) | unauthenticated `PING` | `+PONG` without `AUTH` | | PostgreSQL (RDS/Aurora, Redshift, Azure, Supabase) | unauthenticated StartupMessage | `AuthenticationOk` (server trusts the connection) | | MongoDB (DocumentDB, Cosmos Mongo) | OP_MSG `listDatabases` | command succeeds without auth | | MySQL, Azure SQL, Cassandra (Cosmos) | TCP connect only | *never* — reported `reachable` | Every check sends **no credentials**: no `Authorization` header, no password, no token. "Anonymous" here means an unauthenticated request, exactly as an anonymous cloud principal would make. Deep protocol probes (`PROBE_DEEP`, default on) only ever issue read-only requests or commands; nothing is written, and no authentication is attempted. Probe outcomes are stored in `exposure_state` and a short factual `exposure_detail` string (e.g. `s3 redirect region=eu-central-1`, `status=403 authentication required`, `tcp/5432 open; no authentication attempted`). Database endpoints are never claimed to be readable: a raw `public_read` on a database is impossible by design. ## Production notes * **Secrets** — database credentials live in `.env` (gitignored); `.env.example` is the template. Inject the same variables from your orchestrator's secret store in production. * **Database TLS** — the local compose stack uses `sslmode=disable` because the compose bridge network is private. For production, point `DATABASE_URL` at a TLS-enabled database, e.g. `postgres://user:pass@db.example.com:5432/ctcloud?sslmode=verify-full`. pgx handles TLS natively; no extra flags are needed. * **Metrics** — bound to loopback. There is no auth, no debug endpoints and no high-cardinality labels (never hostnames). * **Resource limits** — `mem_limit`/`cpus`/`pids_limit` are set in compose so overload is bounded by the queue sizes rather than by the OOM killer. * **Authorization** — leave `ACTIVE_PROBING=false` unless you are authorised to probe the discovered infrastructure. ## Database schema * `cloud_assets` — one row per `(provider, service, provider_hostname)`, with `hostname` (original CT hostname), `resource_identifier`, `region`, `resource_specific`, `public_semantic`, `exposure_state`, `exposure_detail`, `probed_at`, `first_seen`, `last_seen`. * `observations` — append-only observations linking an asset to a `source_hostname`, `source_type` (`direct_ct`/`cname`), CNAME chain and A/AAAA records. * `ct_checkpoints` — per-log CT resume state (JSON-encoded certspotter `LogState`; Merkle positions, no certificates). * `ct_sths` — downloaded, unverified signed tree heads needed for safe resume. Schema is applied idempotently at startup from embedded SQL in `migrations/`. No certificate contents are ever stored. ## Performance characteristics Optimised in this order: I/O off the hot path, then serialization, then process boundaries, then allocations, then cheap suffix/label matching, then bounded pools, then DNS caching, dedup and batched writes. The classifier: * no regexp, JSON, reflection, DNS, HTTP, filesystem, DB or logging; * no `strings.Split`, no `fmt.Sprintf`; * lowercases only when an uppercase byte is present; * parses labels by byte offsets, returning substrings (zero copy); * fixed stack arrays, no unbounded maps or goroutines. ### Benchmark results (Apple M4, Go 1.27) ```text BenchmarkClassifyAWS-10 60.03 ns/op 0 B/op 0 allocs/op BenchmarkClassifyAzure-10 44.93 ns/op 0 B/op 0 allocs/op BenchmarkRejectNormalDomain-10 30.28 ns/op 0 B/op 0 allocs/op BenchmarkClassifyMixed-10 369.50 ns/op 0 B/op 0 allocs/op ``` The whole in-process sink path is allocation-free too: ```text BenchmarkHandleHostNonMatch-10 115.6 ns/op 0 B/op 0 allocs/op BenchmarkHandleHostDirectMatch-10 155.7 ns/op 0 B/op 0 allocs/op ``` Run them yourself: ```sh go test ./internal/classify/ ./internal/pipeline/ -run '^$' -bench . -benchmem ``` `BenchmarkClassifyMixed` and `TestHandleHostNoAllocs` fail unless the result is exactly 0 allocs/op, so the target is enforced by the test suite. ### DNS load balancing When `DNS_SERVERS` is unset, the pool is ~20 curated, reliable public resolvers (Cloudflare, Google, Quad9, OpenDNS, AdGuard, …). Each query picks a random healthy resolver and fails over to at most two others; each attempt is bounded (2s) and resolvers that fail repeatedly enter a 5-minute cooldown. Definitive answers (NXDOMAIN) return immediately rather than being retried on every server, so negative lookups do not multiply DNS traffic. Set `DNS_SERVERS=fresh` to add a compiled-in snapshot of [proabiral/Fresh-Resolvers](https://github.com/proabiral/Fresh-Resolvers) (6,760 entries) on top of the curated set. It validates a random sample of 256 at startup and cools down the rest on failure. Refresh the snapshot with: ```sh curl -sL https://raw.githubusercontent.com/proabiral/Fresh-Resolvers/master/resolvers.txt \ -o internal/dns/resolvers.txt ``` > Probing or using thousands of community resolvers at once can saturate the > container's own DNS and break unrelated lookups, which is why `fresh` is > opt-in and sampled. These resolvers are also untrusted: they can observe and, > in principle, forge DNS answers. That only affects enrichment (which endpoints > are looked up), never the classifier. For sensitive deployments set > `DNS_SERVERS` to your own resolvers, or `DNS_SERVERS=system`. > These community resolvers are untrusted: they can observe and, in principle, > forge DNS answers. That only affects enrichment (which endpoints are looked > up), never the classifier. For sensitive deployments set `DNS_SERVERS` to your > own resolvers, or `DNS_SERVERS=system`. ## Testing ```sh go test ./... # unit + table-driven provider tests go test -race ./... # race detector go vet ./... ``` Integration tests for PostgreSQL run only when `DATABASE_URL` is set: ```sh DATABASE_URL=postgres://ctcloud:ctcloud@localhost:5432/ctcloud go test ./internal/store/ ``` ## Limitations * `net.Resolver.LookupCNAME` returns the final canonical name, so intermediate CNAME hops are not observed (the final target is still classified). Switch to `miekg/dns` if the full chain is required. * Optional reachability checks are off by default; when enabled they perform a single non-destructive HTTPS `GET /` only — no authentication, downloads, mutation or database queries. * OCI generic `oraclecloud.com` endpoints are recognised but not stored, since the bucket name lives in the URL path rather than the hostname. * Custom R2 (and other) domains are only discovered through DNS/CNAME enrichment, and only if the CT hostname itself appears in a certificate.