ctcloud: CertSpotter-backed CT cloud-resource monitor
Single stateless Go process that consumes CT logs via embedded (patched) CertSpotter internals, classifies cloud/database/storage DNS identifiers in-process with a zero-allocation classifier, enriches via round-robin DNS resolvers with CNAME following, and batch-writes useful findings to PostgreSQL. - third_party/certspotter: 2-hunk IdentifierSink patch, no stdout/JSON/files - internal/classify: AWS/Azure/GCP/Firebase/Cloudflare R2/DigitalOcean/Supabase/OCI - internal/dns: sharded bounded TTL cache, multi-resolver load balancing + failover - internal/pipeline: bounded queues, dedup, direct-priority load shedding - internal/store: pgx batch writer, Postgres CT checkpoints (no cert contents) - internal/probe: optional non-destructive reachability checks (off by default) - Docker: distroless non-root, read-only rootfs, no app volume, resource limits - tests enforce 0 allocs/op on the classifier and sink hot path
This commit is contained in:
StasonJatham committed
1 parent
9e78a114ac
commit
e8f7d6bfdf
118 files changed
+16473
No files matched your search
@@ -0,0 +1,253 @@
|
||||
# ctcloud
|
||||
|
||||
`ctcloud` continuously consumes Certificate Transparency (CT) logs, finds
|
||||
cloud storage/database/data-service resources referenced by certificates,
|
||||
enriches candidates through DNS/CNAME resolution, and persists only useful
|
||||
findings to PostgreSQL.
|
||||
|
||||
It is a **single stateless Go process plus PostgreSQL**. CertSpotter's real CT
|
||||
fetching, validation and parsing code is embedded and called **in-process**;
|
||||
there is no stdout handoff, no JSON handoff, no hooks, no subprocesses and no
|
||||
certificate files.
|
||||
|
||||
> A detected provider endpoint **does not** mean the underlying data is publicly
|
||||
> accessible. `ctcloud` reports facts (hostname existed in CT, resolved, answered
|
||||
> a request) separately from conclusions (`exposure_state`). Most findings are
|
||||
> simply an endpoint that exists.
|
||||
|
||||
## How it works
|
||||
|
||||
```text
|
||||
CT logs ──► CertSpotter internals (fetch/verify/parse)
|
||||
│ direct Go call: IdentifierSink.HandleIdentifiers
|
||||
▼
|
||||
fast classifier (0 allocs/op, no regexp/JSON/DNS)
|
||||
┌──────┴───────┐
|
||||
direct match non-match
|
||||
│ ▼
|
||||
│ DNS/CNAME enrichment
|
||||
│ (followed + classified)
|
||||
└──────┬───────┘
|
||||
▼
|
||||
dedup + bounded queues
|
||||
▼
|
||||
optional safe reachability
|
||||
▼
|
||||
batched PostgreSQL writer
|
||||
▼
|
||||
PostgreSQL
|
||||
```
|
||||
|
||||
See [ARCHITECTURE.md](ARCHITECTURE.md) for per-stage rules.
|
||||
|
||||
## Why CertSpotter is embedded (and patched)
|
||||
|
||||
Re-implementing CT log handling is a large, security-sensitive job. Upstream
|
||||
CertSpotter already does log-list handling, STH fetching/verification, Merkle
|
||||
tree maintenance, entry parsing and certificate reconstruction.
|
||||
|
||||
The only change is a two-hunk patch in `third_party/certspotter` that adds an
|
||||
`IdentifierSink` callback. After a certificate is parsed, the identifiers are
|
||||
handed to ctcloud with a direct Go function call and the watchlist/chain/
|
||||
notification machinery is skipped:
|
||||
|
||||
```go
|
||||
if config.IdentifierSink != nil {
|
||||
config.IdentifierSink.HandleIdentifiers(ctx, identifiers)
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
See [`third_party/certspotter/CTCLOUD_PATCH.md`](third_party/certspotter/CTCLOUD_PATCH.md).
|
||||
This deliberately avoids stdout piping, JSON files, hook scripts, subprocesses
|
||||
and temporary certificate files, which are all slow, serializing and
|
||||
filesystem-bound. Upstream licensing/notices are preserved and the patch is
|
||||
isolated so it can be rebased.
|
||||
|
||||
## Supported provider patterns
|
||||
|
||||
**AWS** — S3 (`<bucket>.s3[.dualstack].<region>.amazonaws.com`,
|
||||
`s3-website[-<region>]`), S3 access points (`s3-accesspoint[-fips]`,
|
||||
`dualstack`), multi-region access points (`mrap.accesspoint.s3-global`), S3
|
||||
Express (`s3express-*`), RDS/Aurora/proxy (`*.rds.amazonaws.com`), DocumentDB,
|
||||
Redshift (provisioned + serverless), OpenSearch (`search-*` public / `vpc-*`
|
||||
VPC), Neptune, MemoryDB, ElastiCache. Generic API endpoints
|
||||
(`s3.eu-central-1.amazonaws.com`, `dynamodb.*`, `redshift.<region>`,
|
||||
`rds.<region>`, …) are rejected.
|
||||
|
||||
**Azure** — `blob|dfs|file|queue|table|web.core.windows.net`,
|
||||
`*.database.windows.net`, `*.postgres|mysql.database.azure.com`,
|
||||
`*.documents.azure.com`, `*.{mongo,cassandra,gremlin}.cosmos.azure.com`,
|
||||
`*.redis.cache.windows.net`, `*.redis.azure.net`, `*.search.windows.net`.
|
||||
Bare service endpoints are rejected.
|
||||
|
||||
**Google Cloud / Firebase** — `<bucket>.storage.googleapis.com`,
|
||||
`<db>.firebaseio.com`, `<db>.<region>.firebasedatabase.app`,
|
||||
`<project>.appspot.com`. `storage.googleapis.com` itself is rejected.
|
||||
|
||||
**Cloudflare R2** — `<account>.r2.cloudflarestorage.com`; `<account>.r2.dev`
|
||||
is flagged as an explicitly public URL form.
|
||||
|
||||
**DigitalOcean** — `<bucket>.<region>[.cdn].digitaloceanspaces.com`,
|
||||
`*.db.ondigitalocean.com`. `<region>.digitaloceanspaces.com` is rejected.
|
||||
|
||||
**Supabase** — `<ref>.supabase.co`, `db.<ref>.supabase.co` (PostgreSQL),
|
||||
`aws-<n>-<region>.pooler.supabase.com`.
|
||||
|
||||
**Oracle Cloud (OCI)** — `<namespace>.objectstorage|swiftobjectstorage.<region>.oci.customer-oci.com`
|
||||
(resource-specific), and the generic `objectstorage.<region>.oraclecloud.com`
|
||||
forms (`resource_specific = false`; not stored as findings).
|
||||
|
||||
Provider-specific details live in `internal/classify/`.
|
||||
|
||||
## Configuration
|
||||
|
||||
Environment variables only; no config files, no runtime pattern files.
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `DATABASE_URL` | *(required)* | PostgreSQL connection string |
|
||||
| `DB_MAX_CONNS` | `8` | pgx pool size |
|
||||
| `DNS_SERVERS` | `1.1.1.1:53,8.8.8.8:53,1.0.0.1:53` | comma-separated resolvers (round-robin + failover); `system` for the host resolver |
|
||||
| `DNS_WORKERS` | `16` | DNS enrichment workers |
|
||||
| `DNS_TIMEOUT` | `5s` | per-lookup timeout |
|
||||
| `DNS_CACHE_SIZE` | `1048576` | max DNS cache entries |
|
||||
| `DNS_MAX_CNAME_DEPTH` | `8` | max CNAME hops |
|
||||
| `DNS_POSITIVE_TTL` / `DNS_NEGATIVE_TTL` | `10m` / `1m` | DNS cache TTLs |
|
||||
| `DIRECT_QUEUE_SIZE` | `8192` | direct-match queue capacity |
|
||||
| `DNS_QUEUE_SIZE` | `65536` | DNS-enrichment queue capacity |
|
||||
| `DB_QUEUE_SIZE` | `8192` | DB-write queue capacity |
|
||||
| `DB_BATCH_SIZE` | `500` | findings per write batch |
|
||||
| `DB_FLUSH_INTERVAL` | `2s` | max time before a batch is flushed |
|
||||
| `ACTIVE_PROBING` | `false` | enable optional reachability checks |
|
||||
| `PROBE_WORKERS` / `PROBE_TIMEOUT` | `4` / `5s` | probe pool |
|
||||
| `DEDUP_TTL` | `1h` | hostname/finding dedup window |
|
||||
| `METRICS_ADDR` | `:9090` | metrics/health listener |
|
||||
| `LOG_LEVEL` | `info` | `debug\|info\|warn\|error` |
|
||||
| `STATS_INTERVAL` | `60s` | aggregated stats log interval |
|
||||
| `LOG_LIST_SOURCE` | Google log list v3 | certspotter log list URL |
|
||||
| `START_AT_END` | `false` | skip existing backlog on first run |
|
||||
| `SHUTDOWN_TIMEOUT` | `20s` | graceful shutdown budget |
|
||||
|
||||
## Running with Docker
|
||||
|
||||
```sh
|
||||
cp .env.example .env # set a strong POSTGRES_PASSWORD and matching DATABASE_URL
|
||||
docker compose up --build
|
||||
```
|
||||
|
||||
That starts `postgres` and the stateless `ctcloud` container. The application
|
||||
container runs as non-root, has a read-only root filesystem, drops all Linux
|
||||
capabilities, sets CPU/memory/pids limits and mounts no application volume. The
|
||||
only persistent state is PostgreSQL (CT checkpoints + cloud assets).
|
||||
|
||||
Endpoints: `GET /metrics`, `GET /healthz`, `GET /readyz` on port `9090`, bound
|
||||
to `127.0.0.1` only. The listener has no authentication, so scrape it locally or
|
||||
from a sidecar and never expose it publicly.
|
||||
|
||||
## Production notes
|
||||
|
||||
* **Secrets** — database credentials live in `.env` (gitignored); `.env.example`
|
||||
is the template. Inject the same variables from your orchestrator's secret
|
||||
store in production.
|
||||
* **Database TLS** — the local compose stack uses `sslmode=disable` because the
|
||||
compose bridge network is private. For production, point `DATABASE_URL` at a
|
||||
TLS-enabled database, e.g.
|
||||
`postgres://user:[email protected]:5432/ctcloud?sslmode=verify-full`.
|
||||
pgx handles TLS natively; no extra flags are needed.
|
||||
* **Metrics** — bound to loopback. There is no auth, no debug endpoints and no
|
||||
high-cardinality labels (never hostnames).
|
||||
* **Resource limits** — `mem_limit`/`cpus`/`pids_limit` are set in compose so
|
||||
overload is bounded by the queue sizes rather than by the OOM killer.
|
||||
* **Authorization** — leave `ACTIVE_PROBING=false` unless you are authorised to
|
||||
probe the discovered infrastructure.
|
||||
|
||||
## Database schema
|
||||
|
||||
* `cloud_assets` — one row per `(provider, service, provider_hostname)`, with
|
||||
`hostname` (original CT hostname), `resource_identifier`, `region`,
|
||||
`resource_specific`, `public_semantic`, `exposure_state`, `first_seen`,
|
||||
`last_seen`.
|
||||
* `observations` — append-only observations linking an asset to a
|
||||
`source_hostname`, `source_type` (`direct_ct`/`cname`), CNAME chain and
|
||||
A/AAAA records.
|
||||
* `ct_checkpoints` — per-log CT resume state (JSON-encoded certspotter
|
||||
`LogState`; Merkle positions, no certificates).
|
||||
* `ct_sths` — downloaded, unverified signed tree heads needed for safe resume.
|
||||
|
||||
Schema is applied idempotently at startup from embedded SQL in `migrations/`.
|
||||
No certificate contents are ever stored.
|
||||
|
||||
## Performance characteristics
|
||||
|
||||
Optimised in this order: I/O off the hot path, then serialization, then process
|
||||
boundaries, then allocations, then cheap suffix/label matching, then bounded
|
||||
pools, then DNS caching, dedup and batched writes.
|
||||
|
||||
The classifier:
|
||||
|
||||
* no regexp, JSON, reflection, DNS, HTTP, filesystem, DB or logging;
|
||||
* no `strings.Split`, no `fmt.Sprintf`;
|
||||
* lowercases only when an uppercase byte is present;
|
||||
* parses labels by byte offsets, returning substrings (zero copy);
|
||||
* fixed stack arrays, no unbounded maps or goroutines.
|
||||
|
||||
### Benchmark results (Apple M4, Go 1.27)
|
||||
|
||||
```text
|
||||
BenchmarkClassifyAWS-10 60.03 ns/op 0 B/op 0 allocs/op
|
||||
BenchmarkClassifyAzure-10 44.93 ns/op 0 B/op 0 allocs/op
|
||||
BenchmarkRejectNormalDomain-10 30.28 ns/op 0 B/op 0 allocs/op
|
||||
BenchmarkClassifyMixed-10 369.50 ns/op 0 B/op 0 allocs/op
|
||||
```
|
||||
|
||||
The whole in-process sink path is allocation-free too:
|
||||
|
||||
```text
|
||||
BenchmarkHandleHostNonMatch-10 115.6 ns/op 0 B/op 0 allocs/op
|
||||
BenchmarkHandleHostDirectMatch-10 155.7 ns/op 0 B/op 0 allocs/op
|
||||
```
|
||||
|
||||
Run them yourself:
|
||||
|
||||
```sh
|
||||
go test ./internal/classify/ ./internal/pipeline/ -run '^$' -bench . -benchmem
|
||||
```
|
||||
|
||||
`BenchmarkClassifyMixed` and `TestHandleHostNoAllocs` fail unless the result is
|
||||
exactly 0 allocs/op, so the target is enforced by the test suite.
|
||||
|
||||
### DNS load balancing
|
||||
|
||||
DNS lookups are round-robined across `DNS_SERVERS`. Each attempt is bounded
|
||||
(2s) and fails over to the next server on timeout/refused errors, so one dead
|
||||
resolver does not stall lookups. Definitive answers (NXDOMAIN) are returned
|
||||
immediately rather than retried on every server, so negative lookups do not
|
||||
multiply DNS traffic.
|
||||
|
||||
## Testing
|
||||
|
||||
```sh
|
||||
go test ./... # unit + table-driven provider tests
|
||||
go test -race ./... # race detector
|
||||
go vet ./...
|
||||
```
|
||||
|
||||
Integration tests for PostgreSQL run only when `DATABASE_URL` is set:
|
||||
|
||||
```sh
|
||||
DATABASE_URL=postgres://ctcloud:ctcloud@localhost:5432/ctcloud go test ./internal/store/
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
* `net.Resolver.LookupCNAME` returns the final canonical name, so intermediate
|
||||
CNAME hops are not observed (the final target is still classified). Switch to
|
||||
`miekg/dns` if the full chain is required.
|
||||
* Optional reachability checks are off by default; when enabled they perform a
|
||||
single non-destructive HTTPS `GET /` only — no authentication, downloads,
|
||||
mutation or database queries.
|
||||
* OCI generic `oraclecloud.com` endpoints are recognised but not stored, since
|
||||
the bucket name lives in the URL path rather than the hostname.
|
||||
* Custom R2 (and other) domains are only discovered through DNS/CNAME
|
||||
enrichment, and only if the CT hostname itself appears in a certificate.
|
||||
Reference in new issue
Block a user