# ctcloud `ctcloud` continuously consumes Certificate Transparency (CT) logs, finds cloud storage/database/data-service resources referenced by certificates, enriches candidates through DNS/CNAME resolution, and persists only useful findings to PostgreSQL. It is a **single stateless Go process plus PostgreSQL**. CertSpotter's real CT fetching, validation and parsing code is embedded and called **in-process**; there is no stdout handoff, no JSON handoff, no hooks, no subprocesses and no certificate files. > A detected provider endpoint **does not** mean the underlying data is publicly > accessible. `ctcloud` reports facts (hostname existed in CT, resolved, answered > a request) separately from conclusions (`exposure_state`). Most findings are > simply an endpoint that exists. ## How it works ```text CT logs ──► CertSpotter internals (fetch/verify/parse) │ direct Go call: IdentifierSink.HandleIdentifiers ▼ fast classifier (0 allocs/op, no regexp/JSON/DNS) ┌──────┴───────┐ direct match non-match │ ▼ │ DNS/CNAME enrichment │ (followed + classified) └──────┬───────┘ ▼ dedup + bounded queues ▼ optional safe reachability ▼ batched PostgreSQL writer ▼ PostgreSQL ``` See [ARCHITECTURE.md](ARCHITECTURE.md) for per-stage rules. ## Why CertSpotter is embedded (and patched) Re-implementing CT log handling is a large, security-sensitive job. Upstream CertSpotter already does log-list handling, STH fetching/verification, Merkle tree maintenance, entry parsing and certificate reconstruction. The only change is a two-hunk patch in `third_party/certspotter` that adds an `IdentifierSink` callback. After a certificate is parsed, the identifiers are handed to ctcloud with a direct Go function call and the watchlist/chain/ notification machinery is skipped: ```go if config.IdentifierSink != nil { config.IdentifierSink.HandleIdentifiers(ctx, identifiers) return nil } ``` See [`third_party/certspotter/CTCLOUD_PATCH.md`](third_party/certspotter/CTCLOUD_PATCH.md). This deliberately avoids stdout piping, JSON files, hook scripts, subprocesses and temporary certificate files, which are all slow, serializing and filesystem-bound. Upstream licensing/notices are preserved and the patch is isolated so it can be rebased. ## Supported provider patterns **AWS** — S3 (`.s3[.dualstack]..amazonaws.com`, `.s3-.amazonaws.com` legacy form, `s3-website[-]`), S3 access points (`s3-accesspoint[-fips]`, `dualstack`), multi-region access points (`mrap.accesspoint.s3-global`), S3 Express (`s3express-*`), RDS/Aurora/proxy (`*.rds.amazonaws.com`), DocumentDB, Redshift (provisioned + serverless), OpenSearch (`search-*` public / `vpc-*` VPC), Neptune, MemoryDB, ElastiCache. Generic API endpoints (`s3.eu-central-1.amazonaws.com`, `dynamodb.*`, `redshift.`, `rds.`, …) are rejected. **Azure** — `blob|dfs|file|queue|table|web.core.windows.net`, `*.database.windows.net`, `*.postgres|mysql.database.azure.com`, `*.documents.azure.com`, `*.{mongo,cassandra,gremlin}.cosmos.azure.com`, `*.redis.cache.windows.net`, `*.redis.azure.net`, `*.search.windows.net`. Bare service endpoints are rejected. **Google Cloud / Firebase** — `.storage.googleapis.com`, `.firebaseio.com`, `..firebasedatabase.app`, `.appspot.com`. `storage.googleapis.com` itself is rejected. **Cloudflare R2** — `.r2.cloudflarestorage.com`; `.r2.dev` is flagged as an explicitly public URL form. **DigitalOcean** — `.[.cdn].digitaloceanspaces.com`, `*.db.ondigitalocean.com`. `.digitaloceanspaces.com` is rejected. **Supabase** — `.supabase.co`, `db..supabase.co` (PostgreSQL), `aws--.pooler.supabase.com`. **Oracle Cloud (OCI)** — `.objectstorage|swiftobjectstorage..oci.customer-oci.com` (resource-specific), and the generic `objectstorage..oraclecloud.com` forms (`resource_specific = false`; not stored as findings). Provider-specific details live in `internal/classify/`. ## Configuration Environment variables only; no config files, no runtime pattern files. | Variable | Default | Purpose | | --- | --- | --- | | `DATABASE_URL` | *(required)* | PostgreSQL connection string | | `DB_MAX_CONNS` | `8` | pgx pool size | | `DNS_SERVERS` | *(unset → curated + community list)* | comma-separated resolvers, or `system` for the OS resolver | | `DNS_WORKERS` | `16` | DNS enrichment workers | | `DNS_TIMEOUT` | `5s` | per-lookup timeout | | `DNS_CACHE_SIZE` | `262144` | max DNS cache entries | | `DEDUP_SIZE` | `262144` | max entries in each dedup cache | | `DNS_MAX_CNAME_DEPTH` | `8` | max CNAME hops | | `DNS_POSITIVE_TTL` / `DNS_NEGATIVE_TTL` | `10m` / `1m` | DNS cache TTLs | | `DIRECT_QUEUE_SIZE` | `8192` | direct-match queue capacity | | `DNS_QUEUE_SIZE` | `65536` | DNS-enrichment queue capacity | | `DB_QUEUE_SIZE` | `8192` | DB-write queue capacity | | `DB_BATCH_SIZE` | `500` | findings per write batch | | `DB_FLUSH_INTERVAL` | `2s` | max time before a batch is flushed | | `ACTIVE_PROBING` | `false` | enable optional reachability checks | | `PROBE_WORKERS` / `PROBE_TIMEOUT` | `8` / `5s` | probe worker pool | | `PROBE_QUEUE_SIZE` | `65536` | probe sample size; when full, old entries are replaced at random (bounded memory) | | `PROBE_DEEP` | `true` | unauthenticated, read-only protocol reads for databases (Redis/Postgres/Mongo) | | `INDEX_ENABLED` / `INDEX_MAX_OBJECTS` | `true` / `1000` | index metadata of readable resources (keys/sizes only) | | `LOG_FORMAT` | `text` | `text` or `json` (JSON is best for log pipelines/LLM analysis) | | `BUCKET_SCAN_ENABLED` | `true` | discover open buckets by name enumeration | | `BUCKET_SCAN_QPS` | `50` | global cap on outbound discovery requests | | `BUCKET_SCAN_WORKERS` | `16` | discovery probe workers | | `BUCKET_SCAN_QUEUE` | `65536` | sampled domain backlog | | `DEDUP_TTL` | `1h` | hostname/finding dedup window | | `METRICS_ADDR` | `:9090` | metrics/health/UI listener | | `UI_USER` / `UI_PASS` | *(unset)* | optional HTTP basic auth for the web UI | | `LOG_LEVEL` | `info` | `debug\|info\|warn\|error` | | `STATS_INTERVAL` | `60s` | aggregated stats log interval | | `LOG_LIST_SOURCE` | Google log list v3 | certspotter log list URL | | `START_AT_END` | `false` | skip existing backlog on first run | | `SHUTDOWN_TIMEOUT` | `20s` | graceful shutdown budget | ## Running with Docker ```sh cp .env.example .env # set a strong POSTGRES_PASSWORD and matching DATABASE_URL docker compose up --build ``` That starts `postgres` and the stateless `ctcloud` container. The application container runs as non-root, has a read-only root filesystem, drops all Linux capabilities, sets CPU/memory/pids limits and mounts no application volume. The only persistent state is PostgreSQL (CT checkpoints + cloud assets + index). > **Findings accumulate over time.** Use `docker compose stop` / `docker compose > start` (or plain `up -d`) to keep the `pgdata` volume. `docker compose down -v` > deletes the volume and every discovered asset with it. Endpoints on port `9090`, bound to `127.0.0.1` only: * `GET /` — web UI (see below) * `GET /api/assets`, `GET /api/stats` — JSON for the UI * `GET /metrics`, `GET /healthz`, `GET /readyz` The listener has no authentication by default, so scrape it locally or from a sidecar and never expose it publicly. Set `UI_USER`/`UI_PASS` to require HTTP basic auth on the UI and its API. ## Web UI A small, dependency-free UI (vanilla JS, no build step, embedded in the binary) styled to the tokens and layout in `DESIGN.md` (neutral canvas, single 768px results column, flat rows, pill controls, no shadows). * search box with 250ms debounce, stale-request cancellation and `?q=` URL state * exposure filter chips — use **Public read** to see open buckets, **Reachable** to see publicly reachable database endpoints — plus a sort control * result rows show the provider hostname, probe detail, provider/service/region badges, resource identifier and relative last-seen/probed times * summary line with total assets, publicly readable and reachable counts * refreshes every 15s (paused when the tab is hidden), read-only * light and dark mode: follows the OS by default, with a header toggle that is persisted in `localStorage`; the theme is applied before first paint to avoid a flash Served from the same loopback port: `http://127.0.0.1:9090/`. ### Brand assets The UI ships the official Bucketeer mascot for both themes, optimized to lossless WebP and embedded in the binary (`internal/webui/assets/`): | Asset | Size | Use | | --- | --- | --- | | `logo-light.webp` | ~27 KB | black line art, light theme | | `logo-dark.webp` | ~22 KB | white line art, dark theme | | `favicon.png` | ~7 KB | browser tab | They are downscaled to 256px and encoded with `cwebp -lossless -z 9`, so they stay crisp on HiDPI displays while shrinking the ~1.2 MB source PNGs by ~96%. Regenerate after replacing the source PNGs at the repo root: ```sh magick bucketeer-light.png -resize 256x -strip /tmp/l.png magick bucketeer-dark.png -resize 256x -strip /tmp/d.png cwebp -quiet -lossless -z 9 -alpha_q 100 /tmp/l.png -o internal/webui/assets/logo-light.webp cwebp -quiet -lossless -z 9 -alpha_q 100 /tmp/d.png -o internal/webui/assets/logo-dark.webp ``` ## Reachability and readability checks `ACTIVE_PROBING=false` by default. When enabled, the prober issues **one** unauthenticated request per finding. It never sends credentials, downloads objects, mutates anything, runs database queries or attempts authentication. | Vendor | Check | `public_read` means | | --- | --- | --- | | AWS S3, Cloudflare R2, DigitalOcean Spaces, OCI | anonymous `ListObjectsV2` (`?list-type=2&max-keys=1`) | bucket policy grants `s3:ListBucket` to `*` | | Google Cloud Storage | JSON API object list (`/storage/v1/b//o?maxResults=1`) | the bucket grants `allUsers` `storage.objects.list` | | Azure Blob/File/Queue/Table | account-level `?comp=list` | anonymous access is **per container**, so account-level 403 is reported `reachable` (not "not readable"). Container names cannot be derived from the account hostname. | | Firebase Realtime DB | `GET /.json?shallow=true` | security rules allow anonymous read | | Supabase | `GET /rest/v1/` | not testable without the public **anon API key**; reports `auth_required` | | OpenSearch / Elasticsearch | `GET /_cat/indices` | security/access control disabled | | Neptune | `GET /status` | a 2xx was returned | | Cosmos / Appspot / Azure Search | root `GET /` | a 2xx was returned | | Redis (ElastiCache, MemoryDB, Azure Redis) | unauthenticated `PING` | `+PONG` without `AUTH` | | PostgreSQL (RDS/Aurora, Redshift, Azure, Supabase) | unauthenticated StartupMessage | `AuthenticationOk` (server trusts the connection) | | MongoDB (DocumentDB, Cosmos Mongo) | OP_MSG `listDatabases` | command succeeds without auth | | MySQL, Azure SQL, Cassandra (Cosmos) | TCP connect only | *never* — reported `reachable` | Every check sends **no credentials**: no `Authorization` header, no password, no token. "Anonymous" here means an unauthenticated request, exactly as an anonymous cloud principal would make. Deep protocol probes (`PROBE_DEEP`, default on) only ever issue read-only requests or commands; nothing is written, and no authentication is attempted. Probe outcomes are stored in `exposure_state` and a short factual `exposure_detail` string (e.g. `s3 redirect region=eu-central-1`, `status=403 authentication required`, `tcp/5432 open; no authentication attempted`). Database endpoints are never claimed to be readable: a raw `public_read` on a database is impossible by design. ## Production notes * **Secrets** — database credentials live in `.env` (gitignored); `.env.example` is the template. Inject the same variables from your orchestrator's secret store in production. * **Database TLS** — the local compose stack uses `sslmode=disable` because the compose bridge network is private. For production, point `DATABASE_URL` at a TLS-enabled database, e.g. `postgres://user:pass@db.example.com:5432/ctcloud?sslmode=verify-full`. pgx handles TLS natively; no extra flags are needed. * **Metrics** — bound to loopback. There is no auth, no debug endpoints and no high-cardinality labels (never hostnames). * **Resource limits** — `mem_limit`/`cpus`/`pids_limit` are set in compose so overload is bounded by the queue sizes rather than by the OOM killer. * **Authorization** — leave `ACTIVE_PROBING=false` unless you are authorised to probe the discovered infrastructure. ## Database schema * `cloud_assets` — one row per `(provider, service, provider_hostname)`, with `hostname` (original CT hostname), `resource_identifier`, `region`, `resource_specific`, `public_semantic`, `exposure_state`, `exposure_detail`, `probed_at`, `first_seen`, `last_seen`. * `observations` — append-only observations linking an asset to a `source_hostname`, `source_type` (`direct_ct`/`cname`), CNAME chain and A/AAAA records. * `ct_checkpoints` — per-log CT resume state (JSON-encoded certspotter `LogState`; Merkle positions, no certificates). * `ct_sths` — downloaded, unverified signed tree heads needed for safe resume. Schema is applied idempotently at startup from embedded SQL in `migrations/`. No certificate contents are ever stored. ## Performance characteristics Optimised in this order: I/O off the hot path, then serialization, then process boundaries, then allocations, then cheap suffix/label matching, then bounded pools, then DNS caching, dedup and batched writes. The classifier: * no regexp, JSON, reflection, DNS, HTTP, filesystem, DB or logging; * no `strings.Split`, no `fmt.Sprintf`; * lowercases only when an uppercase byte is present; * parses labels by byte offsets, returning substrings (zero copy); * fixed stack arrays, no unbounded maps or goroutines. ### Benchmark results (Apple M4, Go 1.27) ```text BenchmarkClassifyAWS-10 60.03 ns/op 0 B/op 0 allocs/op BenchmarkClassifyAzure-10 44.93 ns/op 0 B/op 0 allocs/op BenchmarkRejectNormalDomain-10 30.28 ns/op 0 B/op 0 allocs/op BenchmarkClassifyMixed-10 369.50 ns/op 0 B/op 0 allocs/op ``` The whole in-process sink path is allocation-free too: ```text BenchmarkHandleHostNonMatch-10 115.6 ns/op 0 B/op 0 allocs/op BenchmarkHandleHostDirectMatch-10 155.7 ns/op 0 B/op 0 allocs/op ``` Run them yourself: ```sh go test ./internal/classify/ ./internal/pipeline/ -run '^$' -bench . -benchmem ``` `BenchmarkClassifyMixed` and `TestHandleHostNoAllocs` fail unless the result is exactly 0 allocs/op, so the target is enforced by the test suite. ### Bucket discovery Certificate SANs are custom domains, not bucket names, so matching SANs alone finds almost no buckets. Instead, every non-matching hostname feeds a discovery engine that: 1. takes the names that actually occur in the hostname — the subdomain label and the registrable-domain label (`assets.example.com` → `assets`, `example`). No names are fabricated or drawn from a wordlist; 2. tests anonymous listing — `https://.s3.amazonaws.com/?list-type=2` (AWS) and `https://.storage.googleapis.com` (GCS); 3. indexes every bucket that returns 200 (keys/sizes/dates only). It is aggressively bounded: a global `BUCKET_SCAN_QPS` limit, a 12h per-name dedup cache, and a random-sampled domain queue. Azure is handled separately because it has two levels: the account name maps to DNS (`.blob.core.windows.net`), and anonymous access is per container with no anonymous container enumeration — so for accounts that exist we probe a small set of common container names. R2 is never listable and is only probed for objects. This is what actually yields open buckets. Of the buckets found, S3 and GCS return full object listings; Azure needs a known container; R2 only serves objects you already know. ### DNS load balancing When `DNS_SERVERS` is unset, the pool is ~20 curated, reliable public resolvers (Cloudflare, Google, Quad9, OpenDNS, AdGuard, …). Each query picks a random healthy resolver and fails over to at most two others; each attempt is bounded (2s) and resolvers that fail repeatedly enter a 5-minute cooldown. Definitive answers (NXDOMAIN) return immediately rather than being retried on every server, so negative lookups do not multiply DNS traffic. > For sensitive deployments set `DNS_SERVERS` to your own resolvers, or `DNS_SERVERS=system`. > These community resolvers are untrusted: they can observe and, in principle, > forge DNS answers. That only affects enrichment (which endpoints are looked > up), never the classifier. For sensitive deployments set `DNS_SERVERS` to your > own resolvers, or `DNS_SERVERS=system`. ## Testing ```sh go test ./... # unit + table-driven provider tests go test -race ./... # race detector go vet ./... ``` Integration tests for PostgreSQL run only when `DATABASE_URL` is set: ```sh DATABASE_URL=postgres://ctcloud:ctcloud@localhost:5432/ctcloud go test ./internal/store/ ``` ## Limitations * `net.Resolver.LookupCNAME` returns the final canonical name, so intermediate CNAME hops are not observed (the final target is still classified). Switch to `miekg/dns` if the full chain is required. * Optional reachability checks are off by default; when enabled they perform a single non-destructive HTTPS `GET /` only — no authentication, downloads, mutation or database queries. * OCI generic `oraclecloud.com` endpoints are recognised but not stored, since the bucket name lives in the URL path rather than the hostname. * Custom R2 (and other) domains are only discovered through DNS/CNAME enrichment, and only if the CT hostname itself appears in a certificate.