ctcloud: CertSpotter-backed CT cloud-resource monitor

Single stateless Go process that consumes CT logs via embedded (patched)
CertSpotter internals, classifies cloud/database/storage DNS identifiers
in-process with a zero-allocation classifier, enriches via round-robin DNS
resolvers with CNAME following, and batch-writes useful findings to PostgreSQL.

- third_party/certspotter: 2-hunk IdentifierSink patch, no stdout/JSON/files
- internal/classify: AWS/Azure/GCP/Firebase/Cloudflare R2/DigitalOcean/Supabase/OCI
- internal/dns: sharded bounded TTL cache, multi-resolver load balancing + failover
- internal/pipeline: bounded queues, dedup, direct-priority load shedding
- internal/store: pgx batch writer, Postgres CT checkpoints (no cert contents)
- internal/probe: optional non-destructive reachability checks (off by default)
- Docker: distroless non-root, read-only rootfs, no app volume, resource limits
- tests enforce 0 allocs/op on the classifier and sink hot path

ARCHITECTURE.md

@@ -0,0 +1,92 @@
+# Architecture
+
+## Data flow
+
+```text
+Certificate Transparency Logs
+             │
+             ▼
+     CertSpotter internals            fetch / verify / parse (third_party)
+   (in-process, no stdout/JSON/files)
+             │
+             ▼
+       DNS identifiers
+             │
+             ▼
+        FAST CLASSIFIER              0 allocs/op, no regexp/JSON/DNS/DB
+        ┌────┴─────┐
+   direct cloud   normal
+    resource      hostname
+        │            │
+        │            ▼
+        │      DNS/CNAME enrichment   bounded workers + TTL cache
+        │            │
+        │            ▼
+        │      classify CNAME targets
+        │            │
+        └─────┬──────┘
+              ▼
+        dedup/enrichment             TTL caches; DB uniqueness is final
+              │
+              ▼
+    optional safe reachability       HTTPS GET / only, OFF by default
+              ▼
+        batch DB writer              bounded queue, pgx batches
+              ▼
+          PostgreSQL
+```
+
+## Stages and what is allowed
+
+| Stage | May do | Must not do |
+| --- | --- | --- |
+| CertSpotter fetch/verify/parse | network, crypto, in-memory DER | persist certs, write files |
+| Identifier callback | iterate `DNSNames`, call classifier | block, log per certificate, allocate needlessly |
+| Classifier | byte/string comparisons, label scan | regexp, JSON, reflection, DNS, HTTP, DB, files, per-id logging |
+| Direct queue | bound, non-blocking send | unbounded growth |
+| DNS enrichment | DNS via worker pool, TTL cache, CNAME follow | unbounded goroutines, persistence |
+| Dedup | memory-bounded TTL caches | unbounded maps |
+| Probe (optional) | DNS/TCP/TLS/HTTP HEAD/GET `/` | auth, downloads, mutation, DB or shell access |
+| DB writer | batched `pgx.Batch` writes | one synchronous row per event |
+| Checkpoints | JSON CT position in `ct_checkpoints`/`ct_sths` | certificate/issuer contents in DB |
+
+## Concurrency and backpressure
+
+* CertSpotter runs one monitor goroutine per CT log; it calls the identifier
+  sink concurrently.
+* The sink never blocks: it classifies inline and does a non-blocking send to
+  either the direct queue or the DNS queue. Full queues increment
+  `queue_drops_total` / `queue_full_total`.
+* Direct matches have a dedicated queue and worker pool, so speculative CNAME
+  enrichment can be shed first. DNS workers also skip work when the DB queue is
+  more than half full.
+* The DB writer batches by size or `DB_FLUSH_INTERVAL` and flushes on
+  shutdown.
+
+## Failure handling
+
+* DNS and CT network errors are retried by CertSpotter with bounded exponential
+  backoff; ctcloud counts them (`ct_errors_total`) without logging per event.
+* PostgreSQL writes retry a few times with backoff; persistent failure drops
+  items from the bounded DB queue rather than growing memory.
+* The process uses `context` throughout and on `SIGTERM` stops intake, drains
+  the pipeline, flushes the DB batch and exits within `SHUTDOWN_TIMEOUT`.
+
+## Metrics and health
+
+`GET /metrics` exposes counters (`ct_certificates_total`,
+`classifier_direct_matches_total`, `classifier_cname_matches_total`,
+`dns_*`, `db_*`, `queue_*`) and gauges (`queue_depth{queue="…"}`,
+`go_goroutines`, `process_system_bytes`). No hostnames or other high-cardinality
+labels are ever used.
+
+`GET /healthz` is a liveness check. `GET /readyz` verifies the CT monitor is
+initialised and PostgreSQL is reachable.
+
+## Why no files, pipes or subprocesses
+
+Everything between certificate parsing and classification stays in memory. The
+CertSpotter patch hands identifiers directly to Go code, removing serialization
+(JSON), IPC (stdout/pipe), process boundaries (subprocess/hooks) and filesystem
+I/O (certificate/state files). CT resume state is the only thing persisted, and
+it lives in PostgreSQL.