StasonJatham
9710efec1a
- dns.TTLCache: stop preallocating every shard map to maxShard; this allocated the full theoretical cache capacity up front (hundreds of MB of empty buckets) and pushed the container over its memory limit, causing an OOM restart loop under Docker's default 1g cap. - pipeline: write direct provider matches immediately without a blocking DNS enrichment step, so they can no longer be starved by the optional CNAME queue. Add TestDirectNotStarvedByDNSBacklog. - config: add DEDUP_SIZE, lower DNS_CACHE_SIZE/DEDUP_SIZE defaults to 256k. - .env.example / README: document DEDUP_SIZE and cache sizing. Live: stable at ~250-300 MiB with no restarts, direct queue draining to 0.
255 lines
11 KiB
Markdown
255 lines
11 KiB
Markdown
# ctcloud
|
|
|
|
`ctcloud` continuously consumes Certificate Transparency (CT) logs, finds
|
|
cloud storage/database/data-service resources referenced by certificates,
|
|
enriches candidates through DNS/CNAME resolution, and persists only useful
|
|
findings to PostgreSQL.
|
|
|
|
It is a **single stateless Go process plus PostgreSQL**. CertSpotter's real CT
|
|
fetching, validation and parsing code is embedded and called **in-process**;
|
|
there is no stdout handoff, no JSON handoff, no hooks, no subprocesses and no
|
|
certificate files.
|
|
|
|
> A detected provider endpoint **does not** mean the underlying data is publicly
|
|
> accessible. `ctcloud` reports facts (hostname existed in CT, resolved, answered
|
|
> a request) separately from conclusions (`exposure_state`). Most findings are
|
|
> simply an endpoint that exists.
|
|
|
|
## How it works
|
|
|
|
```text
|
|
CT logs ──► CertSpotter internals (fetch/verify/parse)
|
|
│ direct Go call: IdentifierSink.HandleIdentifiers
|
|
▼
|
|
fast classifier (0 allocs/op, no regexp/JSON/DNS)
|
|
┌──────┴───────┐
|
|
direct match non-match
|
|
│ ▼
|
|
│ DNS/CNAME enrichment
|
|
│ (followed + classified)
|
|
└──────┬───────┘
|
|
▼
|
|
dedup + bounded queues
|
|
▼
|
|
optional safe reachability
|
|
▼
|
|
batched PostgreSQL writer
|
|
▼
|
|
PostgreSQL
|
|
```
|
|
|
|
See [ARCHITECTURE.md](ARCHITECTURE.md) for per-stage rules.
|
|
|
|
## Why CertSpotter is embedded (and patched)
|
|
|
|
Re-implementing CT log handling is a large, security-sensitive job. Upstream
|
|
CertSpotter already does log-list handling, STH fetching/verification, Merkle
|
|
tree maintenance, entry parsing and certificate reconstruction.
|
|
|
|
The only change is a two-hunk patch in `third_party/certspotter` that adds an
|
|
`IdentifierSink` callback. After a certificate is parsed, the identifiers are
|
|
handed to ctcloud with a direct Go function call and the watchlist/chain/
|
|
notification machinery is skipped:
|
|
|
|
```go
|
|
if config.IdentifierSink != nil {
|
|
config.IdentifierSink.HandleIdentifiers(ctx, identifiers)
|
|
return nil
|
|
}
|
|
```
|
|
|
|
See [`third_party/certspotter/CTCLOUD_PATCH.md`](third_party/certspotter/CTCLOUD_PATCH.md).
|
|
This deliberately avoids stdout piping, JSON files, hook scripts, subprocesses
|
|
and temporary certificate files, which are all slow, serializing and
|
|
filesystem-bound. Upstream licensing/notices are preserved and the patch is
|
|
isolated so it can be rebased.
|
|
|
|
## Supported provider patterns
|
|
|
|
**AWS** — S3 (`<bucket>.s3[.dualstack].<region>.amazonaws.com`,
|
|
`s3-website[-<region>]`), S3 access points (`s3-accesspoint[-fips]`,
|
|
`dualstack`), multi-region access points (`mrap.accesspoint.s3-global`), S3
|
|
Express (`s3express-*`), RDS/Aurora/proxy (`*.rds.amazonaws.com`), DocumentDB,
|
|
Redshift (provisioned + serverless), OpenSearch (`search-*` public / `vpc-*`
|
|
VPC), Neptune, MemoryDB, ElastiCache. Generic API endpoints
|
|
(`s3.eu-central-1.amazonaws.com`, `dynamodb.*`, `redshift.<region>`,
|
|
`rds.<region>`, …) are rejected.
|
|
|
|
**Azure** — `blob|dfs|file|queue|table|web.core.windows.net`,
|
|
`*.database.windows.net`, `*.postgres|mysql.database.azure.com`,
|
|
`*.documents.azure.com`, `*.{mongo,cassandra,gremlin}.cosmos.azure.com`,
|
|
`*.redis.cache.windows.net`, `*.redis.azure.net`, `*.search.windows.net`.
|
|
Bare service endpoints are rejected.
|
|
|
|
**Google Cloud / Firebase** — `<bucket>.storage.googleapis.com`,
|
|
`<db>.firebaseio.com`, `<db>.<region>.firebasedatabase.app`,
|
|
`<project>.appspot.com`. `storage.googleapis.com` itself is rejected.
|
|
|
|
**Cloudflare R2** — `<account>.r2.cloudflarestorage.com`; `<account>.r2.dev`
|
|
is flagged as an explicitly public URL form.
|
|
|
|
**DigitalOcean** — `<bucket>.<region>[.cdn].digitaloceanspaces.com`,
|
|
`*.db.ondigitalocean.com`. `<region>.digitaloceanspaces.com` is rejected.
|
|
|
|
**Supabase** — `<ref>.supabase.co`, `db.<ref>.supabase.co` (PostgreSQL),
|
|
`aws-<n>-<region>.pooler.supabase.com`.
|
|
|
|
**Oracle Cloud (OCI)** — `<namespace>.objectstorage|swiftobjectstorage.<region>.oci.customer-oci.com`
|
|
(resource-specific), and the generic `objectstorage.<region>.oraclecloud.com`
|
|
forms (`resource_specific = false`; not stored as findings).
|
|
|
|
Provider-specific details live in `internal/classify/`.
|
|
|
|
## Configuration
|
|
|
|
Environment variables only; no config files, no runtime pattern files.
|
|
|
|
| Variable | Default | Purpose |
|
|
| --- | --- | --- |
|
|
| `DATABASE_URL` | *(required)* | PostgreSQL connection string |
|
|
| `DB_MAX_CONNS` | `8` | pgx pool size |
|
|
| `DNS_SERVERS` | `1.1.1.1:53,8.8.8.8:53,1.0.0.1:53` | comma-separated resolvers (round-robin + failover); `system` for the host resolver |
|
|
| `DNS_WORKERS` | `16` | DNS enrichment workers |
|
|
| `DNS_TIMEOUT` | `5s` | per-lookup timeout |
|
|
| `DNS_CACHE_SIZE` | `262144` | max DNS cache entries |
|
|
| `DEDUP_SIZE` | `262144` | max entries in each dedup cache |
|
|
| `DNS_MAX_CNAME_DEPTH` | `8` | max CNAME hops |
|
|
| `DNS_POSITIVE_TTL` / `DNS_NEGATIVE_TTL` | `10m` / `1m` | DNS cache TTLs |
|
|
| `DIRECT_QUEUE_SIZE` | `8192` | direct-match queue capacity |
|
|
| `DNS_QUEUE_SIZE` | `65536` | DNS-enrichment queue capacity |
|
|
| `DB_QUEUE_SIZE` | `8192` | DB-write queue capacity |
|
|
| `DB_BATCH_SIZE` | `500` | findings per write batch |
|
|
| `DB_FLUSH_INTERVAL` | `2s` | max time before a batch is flushed |
|
|
| `ACTIVE_PROBING` | `false` | enable optional reachability checks |
|
|
| `PROBE_WORKERS` / `PROBE_TIMEOUT` | `4` / `5s` | probe pool |
|
|
| `DEDUP_TTL` | `1h` | hostname/finding dedup window |
|
|
| `METRICS_ADDR` | `:9090` | metrics/health listener |
|
|
| `LOG_LEVEL` | `info` | `debug\|info\|warn\|error` |
|
|
| `STATS_INTERVAL` | `60s` | aggregated stats log interval |
|
|
| `LOG_LIST_SOURCE` | Google log list v3 | certspotter log list URL |
|
|
| `START_AT_END` | `false` | skip existing backlog on first run |
|
|
| `SHUTDOWN_TIMEOUT` | `20s` | graceful shutdown budget |
|
|
|
|
## Running with Docker
|
|
|
|
```sh
|
|
cp .env.example .env # set a strong POSTGRES_PASSWORD and matching DATABASE_URL
|
|
docker compose up --build
|
|
```
|
|
|
|
That starts `postgres` and the stateless `ctcloud` container. The application
|
|
container runs as non-root, has a read-only root filesystem, drops all Linux
|
|
capabilities, sets CPU/memory/pids limits and mounts no application volume. The
|
|
only persistent state is PostgreSQL (CT checkpoints + cloud assets).
|
|
|
|
Endpoints: `GET /metrics`, `GET /healthz`, `GET /readyz` on port `9090`, bound
|
|
to `127.0.0.1` only. The listener has no authentication, so scrape it locally or
|
|
from a sidecar and never expose it publicly.
|
|
|
|
## Production notes
|
|
|
|
* **Secrets** — database credentials live in `.env` (gitignored); `.env.example`
|
|
is the template. Inject the same variables from your orchestrator's secret
|
|
store in production.
|
|
* **Database TLS** — the local compose stack uses `sslmode=disable` because the
|
|
compose bridge network is private. For production, point `DATABASE_URL` at a
|
|
TLS-enabled database, e.g.
|
|
`postgres://user:[email protected]:5432/ctcloud?sslmode=verify-full`.
|
|
pgx handles TLS natively; no extra flags are needed.
|
|
* **Metrics** — bound to loopback. There is no auth, no debug endpoints and no
|
|
high-cardinality labels (never hostnames).
|
|
* **Resource limits** — `mem_limit`/`cpus`/`pids_limit` are set in compose so
|
|
overload is bounded by the queue sizes rather than by the OOM killer.
|
|
* **Authorization** — leave `ACTIVE_PROBING=false` unless you are authorised to
|
|
probe the discovered infrastructure.
|
|
|
|
## Database schema
|
|
|
|
* `cloud_assets` — one row per `(provider, service, provider_hostname)`, with
|
|
`hostname` (original CT hostname), `resource_identifier`, `region`,
|
|
`resource_specific`, `public_semantic`, `exposure_state`, `first_seen`,
|
|
`last_seen`.
|
|
* `observations` — append-only observations linking an asset to a
|
|
`source_hostname`, `source_type` (`direct_ct`/`cname`), CNAME chain and
|
|
A/AAAA records.
|
|
* `ct_checkpoints` — per-log CT resume state (JSON-encoded certspotter
|
|
`LogState`; Merkle positions, no certificates).
|
|
* `ct_sths` — downloaded, unverified signed tree heads needed for safe resume.
|
|
|
|
Schema is applied idempotently at startup from embedded SQL in `migrations/`.
|
|
No certificate contents are ever stored.
|
|
|
|
## Performance characteristics
|
|
|
|
Optimised in this order: I/O off the hot path, then serialization, then process
|
|
boundaries, then allocations, then cheap suffix/label matching, then bounded
|
|
pools, then DNS caching, dedup and batched writes.
|
|
|
|
The classifier:
|
|
|
|
* no regexp, JSON, reflection, DNS, HTTP, filesystem, DB or logging;
|
|
* no `strings.Split`, no `fmt.Sprintf`;
|
|
* lowercases only when an uppercase byte is present;
|
|
* parses labels by byte offsets, returning substrings (zero copy);
|
|
* fixed stack arrays, no unbounded maps or goroutines.
|
|
|
|
### Benchmark results (Apple M4, Go 1.27)
|
|
|
|
```text
|
|
BenchmarkClassifyAWS-10 60.03 ns/op 0 B/op 0 allocs/op
|
|
BenchmarkClassifyAzure-10 44.93 ns/op 0 B/op 0 allocs/op
|
|
BenchmarkRejectNormalDomain-10 30.28 ns/op 0 B/op 0 allocs/op
|
|
BenchmarkClassifyMixed-10 369.50 ns/op 0 B/op 0 allocs/op
|
|
```
|
|
|
|
The whole in-process sink path is allocation-free too:
|
|
|
|
```text
|
|
BenchmarkHandleHostNonMatch-10 115.6 ns/op 0 B/op 0 allocs/op
|
|
BenchmarkHandleHostDirectMatch-10 155.7 ns/op 0 B/op 0 allocs/op
|
|
```
|
|
|
|
Run them yourself:
|
|
|
|
```sh
|
|
go test ./internal/classify/ ./internal/pipeline/ -run '^$' -bench . -benchmem
|
|
```
|
|
|
|
`BenchmarkClassifyMixed` and `TestHandleHostNoAllocs` fail unless the result is
|
|
exactly 0 allocs/op, so the target is enforced by the test suite.
|
|
|
|
### DNS load balancing
|
|
|
|
DNS lookups are round-robined across `DNS_SERVERS`. Each attempt is bounded
|
|
(2s) and fails over to the next server on timeout/refused errors, so one dead
|
|
resolver does not stall lookups. Definitive answers (NXDOMAIN) are returned
|
|
immediately rather than retried on every server, so negative lookups do not
|
|
multiply DNS traffic.
|
|
|
|
## Testing
|
|
|
|
```sh
|
|
go test ./... # unit + table-driven provider tests
|
|
go test -race ./... # race detector
|
|
go vet ./...
|
|
```
|
|
|
|
Integration tests for PostgreSQL run only when `DATABASE_URL` is set:
|
|
|
|
```sh
|
|
DATABASE_URL=postgres://ctcloud:ctcloud@localhost:5432/ctcloud go test ./internal/store/
|
|
```
|
|
|
|
## Limitations
|
|
|
|
* `net.Resolver.LookupCNAME` returns the final canonical name, so intermediate
|
|
CNAME hops are not observed (the final target is still classified). Switch to
|
|
`miekg/dns` if the full chain is required.
|
|
* Optional reachability checks are off by default; when enabled they perform a
|
|
single non-destructive HTTPS `GET /` only — no authentication, downloads,
|
|
mutation or database queries.
|
|
* OCI generic `oraclecloud.com` endpoints are recognised but not stored, since
|
|
the bucket name lives in the URL path rather than the hostname.
|
|
* Custom R2 (and other) domains are only discovered through DNS/CNAME
|
|
enrichment, and only if the CT hostname itself appears in a certificate.
|