bucketeer/ARCHITECTURE.md

# Architecture

## Data flow

```text
Certificate Transparency Logs
             │
             ▼
     CertSpotter internals            fetch / verify / parse (third_party)
   (in-process, no stdout/JSON/files)
             │
             ▼
       DNS identifiers
             │
             ▼
        FAST CLASSIFIER              0 allocs/op, no regexp/JSON/DNS/DB
        ┌────┴─────┐
   direct cloud   normal
    resource      hostname
        │            │
        │            ▼
        │      DNS/CNAME enrichment   bounded workers + TTL cache
        │            │
        │            ▼
        │      classify CNAME targets
        │            │
        └─────┬──────┘
              ▼
        dedup/enrichment             TTL caches; DB uniqueness is final
              │
              ▼
    optional safe reachability       HTTPS GET / only, OFF by default
              ▼
        batch DB writer              bounded queue, pgx batches
              ▼
          PostgreSQL
```

## Stages and what is allowed

| Stage | May do | Must not do |
| --- | --- | --- |
| CertSpotter fetch/verify/parse | network, crypto, in-memory DER | persist certs, write files |
| Identifier callback | iterate `DNSNames`, call classifier | block, log per certificate, allocate needlessly |
| Classifier | byte/string comparisons, label scan | regexp, JSON, reflection, DNS, HTTP, DB, files, per-id logging |
| Direct queue | bound, non-blocking send | unbounded growth |
| DNS enrichment | DNS via worker pool, TTL cache, CNAME follow | unbounded goroutines, persistence |
| Dedup | memory-bounded TTL caches | unbounded maps |
| Probe (optional) | one unauthenticated vendor read endpoint (S3/GCS/Azure/Firebase list) or a bare TCP connect for databases | credentials, object downloads, mutation, DB queries, auth attempts |
| DB writer | batched `pgx.Batch` writes | one synchronous row per event |
| Checkpoints | JSON CT position in `ct_checkpoints`/`ct_sths` | certificate/issuer contents in DB |

## Concurrency and backpressure

* CertSpotter runs one monitor goroutine per CT log; it calls the identifier
  sink concurrently.
* The sink never blocks: it classifies inline and does a non-blocking send to
  either the direct queue or the DNS queue. Full queues increment
  `queue_drops_total` / `queue_full_total`.
* Direct matches have a dedicated queue and worker pool, so speculative CNAME
  enrichment can be shed first. DNS workers also skip work when the DB queue is
  more than half full.
* The DB writer batches by size or `DB_FLUSH_INTERVAL` and flushes on
  shutdown.

## Failure handling

* DNS and CT network errors are retried by CertSpotter with bounded exponential
  backoff; ctcloud counts them (`ct_errors_total`) without logging per event.
* PostgreSQL writes retry a few times with backoff; persistent failure drops
  items from the bounded DB queue rather than growing memory.
* The process uses `context` throughout and on `SIGTERM` stops intake, drains
  the pipeline, flushes the DB batch and exits within `SHUTDOWN_TIMEOUT`.

## Metrics and health

`GET /metrics` exposes counters (`ct_certificates_total`,
`classifier_direct_matches_total`, `classifier_cname_matches_total`,
`dns_*`, `db_*`, `queue_*`) and gauges (`queue_depth{queue="…"}`,
`go_goroutines`, `process_system_bytes`). No hostnames or other high-cardinality
labels are ever used.

`GET /healthz` is a liveness check. `GET /readyz` verifies the CT monitor is
initialised and PostgreSQL is reachable.

## Why no files, pipes or subprocesses

Everything between certificate parsing and classification stays in memory. The
CertSpotter patch hands identifiers directly to Go code, removing serialization
(JSON), IPC (stdout/pipe), process boundaries (subprocess/hooks) and filesystem
I/O (certificate/state files). CT resume state is the only thing persisted, and
it lives in PostgreSQL.