professionalize repo for public release
- rename ctcloud -> bucketeer everywhere (cmd/bucketeer, binary, compose service) - group brand assets under assets/ (light/dark logo + UI screenshot) - move long-form docs under docs/ (ARCHITECTURE, DESIGN, GOAL) and add docs/SKILL.md: how an agent queries/audits a running instance and answers user questions from the API and PostgreSQL - README: large logo (light/dark picture) and UI screenshot, repository layout - .gitignore: /bucketeer, .env, pgdata, build/test artifacts, OS junk - .dockerignore: keep brand/docs/markdown out of the build context
This commit is contained in:
1 parent
1ce5700cec
commit
fac79b6215
18 files changed
+202
-42
No files matched your search
@@ -0,0 +1,92 @@
|
||||
# Architecture
|
||||
|
||||
## Data flow
|
||||
|
||||
```text
|
||||
Certificate Transparency Logs
|
||||
│
|
||||
▼
|
||||
CertSpotter internals fetch / verify / parse (third_party)
|
||||
(in-process, no stdout/JSON/files)
|
||||
│
|
||||
▼
|
||||
DNS identifiers
|
||||
│
|
||||
▼
|
||||
FAST CLASSIFIER 0 allocs/op, no regexp/JSON/DNS/DB
|
||||
┌────┴─────┐
|
||||
direct cloud normal
|
||||
resource hostname
|
||||
│ │
|
||||
│ ▼
|
||||
│ DNS/CNAME enrichment bounded workers + TTL cache
|
||||
│ │
|
||||
│ ▼
|
||||
│ classify CNAME targets
|
||||
│ │
|
||||
└─────┬──────┘
|
||||
▼
|
||||
dedup/enrichment TTL caches; DB uniqueness is final
|
||||
│
|
||||
▼
|
||||
optional safe reachability HTTPS GET / only, OFF by default
|
||||
▼
|
||||
batch DB writer bounded queue, pgx batches
|
||||
▼
|
||||
PostgreSQL
|
||||
```
|
||||
|
||||
## Stages and what is allowed
|
||||
|
||||
| Stage | May do | Must not do |
|
||||
| --- | --- | --- |
|
||||
| CertSpotter fetch/verify/parse | network, crypto, in-memory DER | persist certs, write files |
|
||||
| Identifier callback | iterate `DNSNames`, call classifier | block, log per certificate, allocate needlessly |
|
||||
| Classifier | byte/string comparisons, label scan | regexp, JSON, reflection, DNS, HTTP, DB, files, per-id logging |
|
||||
| Direct queue | bound, non-blocking send | unbounded growth |
|
||||
| DNS enrichment | DNS via worker pool, TTL cache, CNAME follow | unbounded goroutines, persistence |
|
||||
| Dedup | memory-bounded TTL caches | unbounded maps |
|
||||
| Probe (optional) | one unauthenticated vendor read endpoint (S3/GCS/Azure/Firebase list) or a bare TCP connect for databases | credentials, object downloads, mutation, DB queries, auth attempts |
|
||||
| DB writer | batched `pgx.Batch` writes | one synchronous row per event |
|
||||
| Checkpoints | JSON CT position in `ct_checkpoints`/`ct_sths` | certificate/issuer contents in DB |
|
||||
|
||||
## Concurrency and backpressure
|
||||
|
||||
* CertSpotter runs one monitor goroutine per CT log; it calls the identifier
|
||||
sink concurrently.
|
||||
* The sink never blocks: it classifies inline and does a non-blocking send to
|
||||
either the direct queue or the DNS queue. Full queues increment
|
||||
`queue_drops_total` / `queue_full_total`.
|
||||
* Direct matches have a dedicated queue and worker pool, so speculative CNAME
|
||||
enrichment can be shed first. DNS workers also skip work when the DB queue is
|
||||
more than half full.
|
||||
* The DB writer batches by size or `DB_FLUSH_INTERVAL` and flushes on
|
||||
shutdown.
|
||||
|
||||
## Failure handling
|
||||
|
||||
* DNS and CT network errors are retried by CertSpotter with bounded exponential
|
||||
backoff; bucketeer counts them (`ct_errors_total`) without logging per event.
|
||||
* PostgreSQL writes retry a few times with backoff; persistent failure drops
|
||||
items from the bounded DB queue rather than growing memory.
|
||||
* The process uses `context` throughout and on `SIGTERM` stops intake, drains
|
||||
the pipeline, flushes the DB batch and exits within `SHUTDOWN_TIMEOUT`.
|
||||
|
||||
## Metrics and health
|
||||
|
||||
`GET /metrics` exposes counters (`ct_certificates_total`,
|
||||
`classifier_direct_matches_total`, `classifier_cname_matches_total`,
|
||||
`dns_*`, `db_*`, `queue_*`) and gauges (`queue_depth{queue="…"}`,
|
||||
`go_goroutines`, `process_system_bytes`). No hostnames or other high-cardinality
|
||||
labels are ever used.
|
||||
|
||||
`GET /healthz` is a liveness check. `GET /readyz` verifies the CT monitor is
|
||||
initialised and PostgreSQL is reachable.
|
||||
|
||||
## Why no files, pipes or subprocesses
|
||||
|
||||
Everything between certificate parsing and classification stays in memory. The
|
||||
CertSpotter patch hands identifiers directly to Go code, removing serialization
|
||||
(JSON), IPC (stdout/pipe), process boundaries (subprocess/hooks) and filesystem
|
||||
I/O (certificate/state files). CT resume state is the only thing persisted, and
|
||||
it lives in PostgreSQL.
|
||||
Reference in new issue
Block a user