bucketeer/ARCHITECTURE.md

Architecture

Data flow

Certificate Transparency Logs
             │
             ▼
     CertSpotter internals            fetch / verify / parse (third_party)
   (in-process, no stdout/JSON/files)
             │
             ▼
       DNS identifiers
             │
             ▼
        FAST CLASSIFIER              0 allocs/op, no regexp/JSON/DNS/DB
        ┌────┴─────┐
   direct cloud   normal
    resource      hostname
        │            │
        │            ▼
        │      DNS/CNAME enrichment   bounded workers + TTL cache
        │            │
        │            ▼
        │      classify CNAME targets
        │            │
        └─────┬──────┘
              ▼
        dedup/enrichment             TTL caches; DB uniqueness is final
              │
              ▼
    optional safe reachability       HTTPS GET / only, OFF by default
              ▼
        batch DB writer              bounded queue, pgx batches
              ▼
          PostgreSQL

Stages and what is allowed

Stage	May do	Must not do
CertSpotter fetch/verify/parse	network, crypto, in-memory DER	persist certs, write files
Identifier callback	iterate DNSNames, call classifier	block, log per certificate, allocate needlessly
Classifier	byte/string comparisons, label scan	regexp, JSON, reflection, DNS, HTTP, DB, files, per-id logging
Direct queue	bound, non-blocking send	unbounded growth
DNS enrichment	DNS via worker pool, TTL cache, CNAME follow	unbounded goroutines, persistence
Dedup	memory-bounded TTL caches	unbounded maps
Probe (optional)	DNS/TCP/TLS/HTTP HEAD/GET /	auth, downloads, mutation, DB or shell access
DB writer	batched pgx.Batch writes	one synchronous row per event
Checkpoints	JSON CT position in ct_checkpoints/ct_sths	certificate/issuer contents in DB

Concurrency and backpressure

• CertSpotter runs one monitor goroutine per CT log; it calls the identifier sink concurrently.
• The sink never blocks: it classifies inline and does a non-blocking send to either the direct queue or the DNS queue. Full queues increment queue_drops_total / queue_full_total.
• Direct matches have a dedicated queue and worker pool, so speculative CNAME enrichment can be shed first. DNS workers also skip work when the DB queue is more than half full.
• The DB writer batches by size or DB_FLUSH_INTERVAL and flushes on shutdown.

Failure handling

• DNS and CT network errors are retried by CertSpotter with bounded exponential backoff; ctcloud counts them (ct_errors_total) without logging per event.
• PostgreSQL writes retry a few times with backoff; persistent failure drops items from the bounded DB queue rather than growing memory.
• The process uses context throughout and on SIGTERM stops intake, drains the pipeline, flushes the DB batch and exits within SHUTDOWN_TIMEOUT.

Metrics and health

GET /metrics exposes counters (ct_certificates_total, classifier_direct_matches_total, classifier_cname_matches_total, dns_*, db_*, queue_*) and gauges (queue_depth{queue="…"}, go_goroutines, process_system_bytes). No hostnames or other high-cardinality labels are ever used.

GET /healthz is a liveness check. GET /readyz verifies the CT monitor is initialised and PostgreSQL is reachable.

Why no files, pipes or subprocesses

Everything between certificate parsing and classification stays in memory. The CertSpotter patch hands identifiers directly to Go code, removing serialization (JSON), IPC (stdout/pipe), process boundaries (subprocess/hooks) and filesystem I/O (certificate/state files). CT resume state is the only thing persisted, and it lives in PostgreSQL.