StasonJatham
8fce55f8c9
Web UI (internal/webui): - dependency-free embedded HTML/JS served on the loopback listener - /api/assets (provider/service/exposure/q filters) and /api/stats - optional HTTP basic auth via UI_USER/UI_PASS Prober (internal/probe): - vendor-specific, unauthenticated, non-destructive checks: S3/R2/Spaces ListObjectsV2, GCS JSON object list, Azure ?comp=list, Firebase /.json?shallow=true, Supabase REST, and bare TCP connect for database endpoints (never reported readable, no auth attempted) - exposure_detail records the factual reason (status code, region, tcp port) - bounded random-sample queue instead of FIFO: one high-volume hostname template (mrap) no longer starves every other asset type - schema: cloud_assets.exposure_detail + probed_at (idempotent ALTER) Live: stable ~450MiB under 1GiB, no restarts, UI and probe states (private/reachable/auth_required) flowing, 0 db errors.
93 lines
3.9 KiB
Markdown
93 lines
3.9 KiB
Markdown
# Architecture
|
|
|
|
## Data flow
|
|
|
|
```text
|
|
Certificate Transparency Logs
|
|
│
|
|
▼
|
|
CertSpotter internals fetch / verify / parse (third_party)
|
|
(in-process, no stdout/JSON/files)
|
|
│
|
|
▼
|
|
DNS identifiers
|
|
│
|
|
▼
|
|
FAST CLASSIFIER 0 allocs/op, no regexp/JSON/DNS/DB
|
|
┌────┴─────┐
|
|
direct cloud normal
|
|
resource hostname
|
|
│ │
|
|
│ ▼
|
|
│ DNS/CNAME enrichment bounded workers + TTL cache
|
|
│ │
|
|
│ ▼
|
|
│ classify CNAME targets
|
|
│ │
|
|
└─────┬──────┘
|
|
▼
|
|
dedup/enrichment TTL caches; DB uniqueness is final
|
|
│
|
|
▼
|
|
optional safe reachability HTTPS GET / only, OFF by default
|
|
▼
|
|
batch DB writer bounded queue, pgx batches
|
|
▼
|
|
PostgreSQL
|
|
```
|
|
|
|
## Stages and what is allowed
|
|
|
|
| Stage | May do | Must not do |
|
|
| --- | --- | --- |
|
|
| CertSpotter fetch/verify/parse | network, crypto, in-memory DER | persist certs, write files |
|
|
| Identifier callback | iterate `DNSNames`, call classifier | block, log per certificate, allocate needlessly |
|
|
| Classifier | byte/string comparisons, label scan | regexp, JSON, reflection, DNS, HTTP, DB, files, per-id logging |
|
|
| Direct queue | bound, non-blocking send | unbounded growth |
|
|
| DNS enrichment | DNS via worker pool, TTL cache, CNAME follow | unbounded goroutines, persistence |
|
|
| Dedup | memory-bounded TTL caches | unbounded maps |
|
|
| Probe (optional) | one unauthenticated vendor read endpoint (S3/GCS/Azure/Firebase list) or a bare TCP connect for databases | credentials, object downloads, mutation, DB queries, auth attempts |
|
|
| DB writer | batched `pgx.Batch` writes | one synchronous row per event |
|
|
| Checkpoints | JSON CT position in `ct_checkpoints`/`ct_sths` | certificate/issuer contents in DB |
|
|
|
|
## Concurrency and backpressure
|
|
|
|
* CertSpotter runs one monitor goroutine per CT log; it calls the identifier
|
|
sink concurrently.
|
|
* The sink never blocks: it classifies inline and does a non-blocking send to
|
|
either the direct queue or the DNS queue. Full queues increment
|
|
`queue_drops_total` / `queue_full_total`.
|
|
* Direct matches have a dedicated queue and worker pool, so speculative CNAME
|
|
enrichment can be shed first. DNS workers also skip work when the DB queue is
|
|
more than half full.
|
|
* The DB writer batches by size or `DB_FLUSH_INTERVAL` and flushes on
|
|
shutdown.
|
|
|
|
## Failure handling
|
|
|
|
* DNS and CT network errors are retried by CertSpotter with bounded exponential
|
|
backoff; ctcloud counts them (`ct_errors_total`) without logging per event.
|
|
* PostgreSQL writes retry a few times with backoff; persistent failure drops
|
|
items from the bounded DB queue rather than growing memory.
|
|
* The process uses `context` throughout and on `SIGTERM` stops intake, drains
|
|
the pipeline, flushes the DB batch and exits within `SHUTDOWN_TIMEOUT`.
|
|
|
|
## Metrics and health
|
|
|
|
`GET /metrics` exposes counters (`ct_certificates_total`,
|
|
`classifier_direct_matches_total`, `classifier_cname_matches_total`,
|
|
`dns_*`, `db_*`, `queue_*`) and gauges (`queue_depth{queue="…"}`,
|
|
`go_goroutines`, `process_system_bytes`). No hostnames or other high-cardinality
|
|
labels are ever used.
|
|
|
|
`GET /healthz` is a liveness check. `GET /readyz` verifies the CT monitor is
|
|
initialised and PostgreSQL is reachable.
|
|
|
|
## Why no files, pipes or subprocesses
|
|
|
|
Everything between certificate parsing and classification stays in memory. The
|
|
CertSpotter patch hands identifiers directly to Go code, removing serialization
|
|
(JSON), IPC (stdout/pipe), process boundaries (subprocess/hooks) and filesystem
|
|
I/O (certificate/state files). CT resume state is the only thing persisted, and
|
|
it lives in PostgreSQL.
|