diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..7f3284f --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 StasonJatham + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index 1e403f8..3c4fcc9 100644 --- a/README.md +++ b/README.md @@ -7,47 +7,202 @@ # Bucketeer -Bucketeer continuously consumes Certificate Transparency (CT) logs, finds -cloud storage/database/data-service resources referenced by certificates, -enriches candidates through DNS/CNAME resolution, discovers and indexes open -buckets, and persists only useful findings to PostgreSQL. +**Certificate Transparency monitoring for cloud resources and open buckets.** +Bucketeer consumes CT logs, classifies every certificate identifier into a +provider resource (storage bucket, database, data service), discovers and +enumerates open buckets, and stores only useful findings — with a small web UI +and a PostgreSQL-only state model. One stateless Go process + PostgreSQL. ![Bucketeer UI](assets/bucketeer-landing.jpg) -It is a **single stateless Go process plus PostgreSQL**. CertSpotter's real CT -fetching, validation and parsing code is embedded and called **in-process**; -there is no stdout handoff, no JSON handoff, no hooks, no subprocesses and no -certificate files. +--- -> A detected provider endpoint **does not** mean the underlying data is publicly -> accessible. `bucketeer` reports facts (hostname existed in CT, resolved, answered -> a request) separately from conclusions (`exposure_state`). Most findings are -> simply an endpoint that exists. +## What it does -## How it works +- Consumes CT logs using **CertSpotter internals in-process** (no stdout, no + JSON, no hooks, no subprocesses, no certificate files). +- Classifies certificate DNS identifiers with a **zero-allocation classifier**: + AWS S3/RDS/Redshift/DocumentDB/Neptune/OpenSearch/MemoryDB/ElastiCache, Azure + Blob/File/Queue/Table/SQL/Postgres/MySQL/Redis/Cosmos, Google Cloud Storage, + Firebase, Cloudflare R2, DigitalOcean Spaces, Supabase and OCI. +- Enriches hostnames through DNS/CNAME with a bounded resolver pool. +- **Discovers open buckets** by deriving candidate names from real CT hostnames + and testing anonymous listing on S3, GCS and (Azure) account + container. +- Verifies readability with unauthenticated, non-destructive checks and + **indexes the object keys** of anything anonymously listable (metadata only, + never contents). +- Persists findings in batches to PostgreSQL; CT resume state lives there too. + +> A detected endpoint does **not** mean the data is publicly accessible. +> Bucketeer reports facts (hostname seen in CT, resolved, answered, listed) +> separately from conclusions, and never attempts authentication. + +## Data flow ```text CT logs ──► CertSpotter internals (fetch/verify/parse) │ direct Go call: IdentifierSink.HandleIdentifiers ▼ - fast classifier (0 allocs/op, no regexp/JSON/DNS) - ┌──────┴───────┐ - direct match non-match - │ ▼ - │ DNS/CNAME enrichment - │ (followed + classified) - └──────┬───────┘ - ▼ + zero-allocation classifier + ┌──────┴───────────────┐ + direct match non-match + │ │ + │ DNS/CNAME enrichment (bounded pool) + │ │ + │ bucket-name discovery (S3/GCS/Azure) + │ │ + └──────────┬──────────┘ + ▼ dedup + bounded queues - ▼ - optional safe reachability - ▼ + ▼ + optional reachability + readability probe + ▼ + anonymous listing → metadata index + ▼ batched PostgreSQL writer - ▼ - PostgreSQL + ▼ + PostgreSQL ``` -See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for per-stage rules. +## Quick start + +```sh +cp .env.example .env # set a strong POSTGRES_PASSWORD and matching DATABASE_URL +docker compose up -d +``` + +- UI + API: `http://127.0.0.1:9090/` +- Health: `/healthz`, readiness: `/readyz`, metrics: `/metrics` + +The `bucketeer` container runs as non-root with a read-only root filesystem, no +application volume, dropped Linux capabilities and CPU/memory/pids limits. The +only persistent state is PostgreSQL. + +> Use `docker compose stop` / `start` (or `up -d`) to keep findings. +> `docker compose down -v` deletes the `pgdata` volume and every discovered +> asset with it. + +## Configuration + +Everything is environment variables (no config files). Sensible defaults apply. + +| Variable | Default | Purpose | +| --- | --- | --- | +| `DATABASE_URL` | *(required)* | PostgreSQL connection string | +| `DB_MAX_CONNS` | `8` | pgx pool size | +| `DNS_SERVERS` | *(curated)* | comma-separated resolvers, or `system` | +| `DNS_WORKERS` | `64` | DNS enrichment workers | +| `DNS_TIMEOUT` | `5s` | per-lookup timeout | +| `DNS_CACHE_SIZE` | `262144` | max cached DNS results | +| `DNS_MAX_CNAME_DEPTH` | `8` | CNAME hops | +| `DNS_POSITIVE_TTL` / `DNS_NEGATIVE_TTL` | `10m` / `1m` | DNS cache TTLs | +| `DEDUP_SIZE` / `DEDUP_TTL` | `262144` / `1h` | dedup caches | +| `DIRECT_QUEUE_SIZE` / `DNS_QUEUE_SIZE` / `DB_QUEUE_SIZE` | `8192` / `262144` / `8192` | bounded queues | +| `DB_BATCH_SIZE` / `DB_FLUSH_INTERVAL` | `500` / `2s` | write batching | +| `ACTIVE_PROBING` | `false` | enable reachability/readability checks | +| `PROBE_WORKERS` / `PROBE_TIMEOUT` / `PROBE_QUEUE_SIZE` | `8` / `5s` / `65536` | probe pool | +| `PROBE_DEEP` | `true` | unauthenticated read-only DB protocol checks | +| `INDEX_ENABLED` / `INDEX_MAX_OBJECTS` | `true` / `1000` | index readable resources | +| `BUCKET_SCAN_ENABLED` / `_QPS` / `_WORKERS` / `_QUEUE` | `true` / `50` / `16` / `65536` | bucket discovery | +| `METRICS_ADDR` | `:9090` | HTTP listener | +| `UI_USER` / `UI_PASS` | *(unset)* | optional HTTP basic auth for the UI/API | +| `LOG_LEVEL` / `LOG_FORMAT` | `info` / `text` | logging (`json` for pipelines) | +| `STATS_INTERVAL` | `60s` | aggregated stats interval | +| `LOG_LIST_SOURCE` | Google log list v3 | CertSpotter log list | +| `START_AT_END` | `false` | skip existing CT backlog on first run | +| `SHUTDOWN_TIMEOUT` | `20s` | graceful shutdown budget | +| `UI_BIND` (compose) | `127.0.0.1` | host interface for the published port | + +## HTTP API and UI + +The web UI (Assets / Contents views, light and dark mode) and the API share one +listener: + +| Call | Purpose | +| --- | --- | +| `GET /api/stats` | totals and breakdowns by provider/service/exposure | +| `GET /api/assets?provider=&service=&exposure=&q=&sort=&limit=&offset=` | asset list | +| `GET /api/entries?q=&limit=&offset=` | search indexed object keys | +| `POST /api/scan` `{"target":"host-or-URL"}` | classify, CNAME-follow, probe and index one target | +| `POST /api/reprobe?exposure=&limit=` | re-probe stored assets and index readable ones | +| `GET /metrics` `/healthz` `/readyz` | Prometheus / liveness / readiness | + +The listener has no auth by default; set `UI_USER`/`UI_PASS` and keep it bound to +loopback (or a private network). + +## Reading the results + +`exposure_state` distinguishes facts, and two capability flags keep them honest: + +- `public_read` — an anonymous read succeeded. For buckets this is only + meaningful together with `listable`. +- `listable` — the provider's anonymous list API returned object names; only + then does Bucketeer store an index. S3/GCS/DO and Azure "Container" mode. +- `object_readable` — a known object was readable but listing is denied (Azure + "Blob" mode, CDN distributions, R2). +- `reachable` — for databases, the port answered. **No authentication is + attempted**, so a database is never reported as readable. + +`exposure_detail` stores the factual reason (status code, region, container, +`dns: no public record`, `tcp/5432 open; no authentication attempted`, …). + +### Bucket discovery + +Certificate SANs are custom domains, not bucket names, so matching SANs alone +finds almost no buckets. For every non-matching hostname Bucketeer takes the +names that actually occur in it (the subdomain label and the registrable-domain +label) and tests anonymous listing on S3 (`.s3.amazonaws.com`) and GCS +(`.storage.googleapis.com`). Azure has two levels — the account maps to DNS +and anonymous access is per container with no anonymous container enumeration — +so existing accounts are probed against common container names. Every 200 is +indexed. A global `BUCKET_SCAN_QPS` limit and a 12h per-name dedup keep it +bounded; contents are never downloaded. + +## Database schema + +- `cloud_assets` — one row per `(provider, service, provider_hostname)` with + `hostname`, `resource_identifier`, `region`, `resource_specific`, + `public_semantic`, `exposure_state`, `exposure_detail`, `listable`, + `object_readable`, `first_seen`, `last_seen`, `probed_at`. +- `asset_entries` — the metadata index of readable resources (key/path, size, + updated, content-type). +- `observations` — sources, CNAME chain and A/AAAA records. +- `ct_checkpoints`, `ct_sths` — CertSpotter resume state (no certificate + contents). + +Schema is applied idempotently at startup from embedded SQL in `migrations/`. + +## Performance + +Optimized in order: I/O off the hot path, then serialization, then process +boundaries, then allocations. The classifier uses no regexp/JSON/reflection/DNS/ +HTTP/DB, lowercases only when needed, and parses labels by byte offset. + +```text +BenchmarkClassifyAWS-10 60 ns/op 0 B/op 0 allocs/op +BenchmarkClassifyAzure-10 45 ns/op 0 B/op 0 allocs/op +BenchmarkRejectNormalDomain-10 30 ns/op 0 B/op 0 allocs/op +BenchmarkHandleHostNonMatch-10 116 ns/op 0 B/op 0 allocs/op +``` + +Run them locally: + +```sh +go test ./internal/classify/ ./internal/pipeline/ -run '^$' -bench . -benchmem +``` + +`BenchmarkClassifyMixed` and `TestHandleHostNoAllocs` fail unless the hot path +is exactly 0 allocs/op, so the target is enforced by the test suite. + +## Testing + +```sh +go test ./... # unit + table-driven provider tests +go test -race ./... +go vet ./... +``` + +PostgreSQL integration tests run when `DATABASE_URL` is set. ## Repository layout @@ -69,353 +224,23 @@ internal/ webui/ embedded web UI migrations/ embedded SQL schema third_party/certspotter/ vendored upstream CT engine (minimal patch) -docs/ ARCHITECTURE.md, DESIGN.md, GOAL.md, SKILL.md +docs/ DESIGN.md, SKILL.md assets/ brand assets ``` -More: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) (data flow), -[docs/DESIGN.md](docs/DESIGN.md) (UI design system), -[docs/SKILL.md](docs/SKILL.md) (how an agent queries and audits a running instance). - -## Why CertSpotter is embedded (and patched) - -Re-implementing CT log handling is a large, security-sensitive job. Upstream -CertSpotter already does log-list handling, STH fetching/verification, Merkle -tree maintenance, entry parsing and certificate reconstruction. - -The only change is a two-hunk patch in `third_party/certspotter` that adds an -`IdentifierSink` callback. After a certificate is parsed, the identifiers are -handed to bucketeer with a direct Go function call and the watchlist/chain/ -notification machinery is skipped: - -```go -if config.IdentifierSink != nil { - config.IdentifierSink.HandleIdentifiers(ctx, identifiers) - return nil -} -``` - -See [`third_party/certspotter/CTCLOUD_PATCH.md`](third_party/certspotter/CTCLOUD_PATCH.md). -This deliberately avoids stdout piping, JSON files, hook scripts, subprocesses -and temporary certificate files, which are all slow, serializing and -filesystem-bound. Upstream licensing/notices are preserved and the patch is -isolated so it can be rebased. - -## Supported provider patterns - -**AWS** — S3 (`.s3[.dualstack]..amazonaws.com`, -`.s3-.amazonaws.com` legacy form, `s3-website[-]`), -S3 access points (`s3-accesspoint[-fips]`, -`dualstack`), multi-region access points (`mrap.accesspoint.s3-global`), S3 -Express (`s3express-*`), RDS/Aurora/proxy (`*.rds.amazonaws.com`), DocumentDB, -Redshift (provisioned + serverless), OpenSearch (`search-*` public / `vpc-*` -VPC), Neptune, MemoryDB, ElastiCache. Generic API endpoints -(`s3.eu-central-1.amazonaws.com`, `dynamodb.*`, `redshift.`, -`rds.`, …) are rejected. - -**Azure** — `blob|dfs|file|queue|table|web.core.windows.net`, -`*.database.windows.net`, `*.postgres|mysql.database.azure.com`, -`*.documents.azure.com`, `*.{mongo,cassandra,gremlin}.cosmos.azure.com`, -`*.redis.cache.windows.net`, `*.redis.azure.net`, `*.search.windows.net`. -Bare service endpoints are rejected. - -**Google Cloud / Firebase** — `.storage.googleapis.com`, -`.firebaseio.com`, `..firebasedatabase.app`, -`.appspot.com`. `storage.googleapis.com` itself is rejected. - -**Cloudflare R2** — `.r2.cloudflarestorage.com`; `.r2.dev` -is flagged as an explicitly public URL form. - -**DigitalOcean** — `.[.cdn].digitaloceanspaces.com`, -`*.db.ondigitalocean.com`. `.digitaloceanspaces.com` is rejected. - -**Supabase** — `.supabase.co`, `db..supabase.co` (PostgreSQL), -`aws--.pooler.supabase.com`. - -**Oracle Cloud (OCI)** — `.objectstorage|swiftobjectstorage..oci.customer-oci.com` -(resource-specific), and the generic `objectstorage..oraclecloud.com` -forms (`resource_specific = false`; not stored as findings). - -Provider-specific details live in `internal/classify/`. - -## Configuration - -Environment variables only; no config files, no runtime pattern files. - -| Variable | Default | Purpose | -| --- | --- | --- | -| `DATABASE_URL` | *(required)* | PostgreSQL connection string | -| `DB_MAX_CONNS` | `8` | pgx pool size | -| `DNS_SERVERS` | *(unset → curated + community list)* | comma-separated resolvers, or `system` for the OS resolver | -| `DNS_WORKERS` | `16` | DNS enrichment workers | -| `DNS_TIMEOUT` | `5s` | per-lookup timeout | -| `DNS_CACHE_SIZE` | `262144` | max DNS cache entries | -| `DEDUP_SIZE` | `262144` | max entries in each dedup cache | -| `DNS_MAX_CNAME_DEPTH` | `8` | max CNAME hops | -| `DNS_POSITIVE_TTL` / `DNS_NEGATIVE_TTL` | `10m` / `1m` | DNS cache TTLs | -| `DIRECT_QUEUE_SIZE` | `8192` | direct-match queue capacity | -| `DNS_QUEUE_SIZE` | `65536` | DNS-enrichment queue capacity | -| `DB_QUEUE_SIZE` | `8192` | DB-write queue capacity | -| `DB_BATCH_SIZE` | `500` | findings per write batch | -| `DB_FLUSH_INTERVAL` | `2s` | max time before a batch is flushed | -| `ACTIVE_PROBING` | `false` | enable optional reachability checks | -| `PROBE_WORKERS` / `PROBE_TIMEOUT` | `8` / `5s` | probe worker pool | -| `PROBE_QUEUE_SIZE` | `65536` | probe sample size; when full, old entries are replaced at random (bounded memory) | -| `PROBE_DEEP` | `true` | unauthenticated, read-only protocol reads for databases (Redis/Postgres/Mongo) | -| `INDEX_ENABLED` / `INDEX_MAX_OBJECTS` | `true` / `1000` | index metadata of readable resources (keys/sizes only) | -| `LOG_FORMAT` | `text` | `text` or `json` (JSON is best for log pipelines/LLM analysis) | -| `BUCKET_SCAN_ENABLED` | `true` | discover open buckets by name enumeration | -| `BUCKET_SCAN_QPS` | `50` | global cap on outbound discovery requests | -| `BUCKET_SCAN_WORKERS` | `16` | discovery probe workers | -| `BUCKET_SCAN_QUEUE` | `65536` | sampled domain backlog | -| `DEDUP_TTL` | `1h` | hostname/finding dedup window | -| `METRICS_ADDR` | `:9090` | metrics/health/UI listener | -| `UI_USER` / `UI_PASS` | *(unset)* | optional HTTP basic auth for the web UI | -| `LOG_LEVEL` | `info` | `debug\|info\|warn\|error` | -| `STATS_INTERVAL` | `60s` | aggregated stats log interval | -| `LOG_LIST_SOURCE` | Google log list v3 | certspotter log list URL | -| `START_AT_END` | `false` | skip existing backlog on first run | -| `SHUTDOWN_TIMEOUT` | `20s` | graceful shutdown budget | - -## Running with Docker - -```sh -cp .env.example .env # set a strong POSTGRES_PASSWORD and matching DATABASE_URL -docker compose up --build -``` - -That starts `postgres` and the stateless `bucketeer` container. The application -container runs as non-root, has a read-only root filesystem, drops all Linux -capabilities, sets CPU/memory/pids limits and mounts no application volume. The -only persistent state is PostgreSQL (CT checkpoints + cloud assets + index). - -> **Findings accumulate over time.** Use `docker compose stop` / `docker compose -> start` (or plain `up -d`) to keep the `pgdata` volume. `docker compose down -v` -> deletes the volume and every discovered asset with it. - -Endpoints on port `9090`, bound to `127.0.0.1` only: - -* `GET /` — web UI (see below) -* `GET /api/assets`, `GET /api/stats` — JSON for the UI -* `GET /metrics`, `GET /healthz`, `GET /readyz` - -The listener has no authentication by default, so scrape it locally or from a -sidecar and never expose it publicly. Set `UI_USER`/`UI_PASS` to require HTTP -basic auth on the UI and its API. - -## Web UI - -A small, dependency-free UI (vanilla JS, no build step, embedded in the binary) -styled to the tokens and layout in [docs/DESIGN.md](docs/DESIGN.md) (neutral canvas, single 768px -results column, flat rows, pill controls, no shadows). - -* search box with 250ms debounce, stale-request cancellation and `?q=` URL state -* exposure filter chips — use **Public read** to see open buckets, **Reachable** - to see publicly reachable database endpoints — plus a sort control -* result rows show the provider hostname, probe detail, provider/service/region - badges, resource identifier and relative last-seen/probed times -* summary line with total assets, publicly readable and reachable counts -* refreshes every 15s (paused when the tab is hidden), read-only -* light and dark mode: follows the OS by default, with a header toggle that is - persisted in `localStorage`; the theme is applied before first paint to avoid - a flash - -Served from the same loopback port: `http://127.0.0.1:9090/`. - -### Brand assets - -The UI ships the official Bucketeer mascot for both themes, optimized to -lossless WebP and embedded in the binary (`internal/webui/assets/`): - -| Asset | Size | Use | -| --- | --- | --- | -| `logo-light.webp` | ~27 KB | black line art, light theme | -| `logo-dark.webp` | ~22 KB | white line art, dark theme | -| `favicon.png` | ~7 KB | browser tab | - -They are downscaled to 256px and encoded with `cwebp -lossless -z 9`, so they -stay crisp on HiDPI displays while shrinking the ~1.2 MB source PNGs by ~96%. -Regenerate after replacing the source PNGs at the repo root: - -```sh -magick assets/bucketeer-light.png -resize 256x -strip /tmp/l.png -magick assets/bucketeer-dark.png -resize 256x -strip /tmp/d.png -cwebp -quiet -lossless -z 9 -alpha_q 100 /tmp/l.png -o internal/webui/assets/logo-light.webp -cwebp -quiet -lossless -z 9 -alpha_q 100 /tmp/d.png -o internal/webui/assets/logo-dark.webp -``` - -## Reachability and readability checks - -`ACTIVE_PROBING=false` by default. When enabled, the prober issues **one** -unauthenticated request per finding. It never sends credentials, downloads -objects, mutates anything, runs database queries or attempts authentication. - -| Vendor | Check | `public_read` means | -| --- | --- | --- | -| AWS S3, Cloudflare R2, DigitalOcean Spaces, OCI | anonymous `ListObjectsV2` (`?list-type=2&max-keys=1`) | bucket policy grants `s3:ListBucket` to `*` | -| Google Cloud Storage | JSON API object list (`/storage/v1/b//o?maxResults=1`) | the bucket grants `allUsers` `storage.objects.list` | -| Azure Blob/File/Queue/Table | account-level `?comp=list` | anonymous access is **per container**, so account-level 403 is reported `reachable` (not "not readable"). Container names cannot be derived from the account hostname. | -| Firebase Realtime DB | `GET /.json?shallow=true` | security rules allow anonymous read | -| Supabase | `GET /rest/v1/` | not testable without the public **anon API key**; reports `auth_required` | -| OpenSearch / Elasticsearch | `GET /_cat/indices` | security/access control disabled | -| Neptune | `GET /status` | a 2xx was returned | -| Cosmos / Appspot / Azure Search | root `GET /` | a 2xx was returned | -| Redis (ElastiCache, MemoryDB, Azure Redis) | unauthenticated `PING` | `+PONG` without `AUTH` | -| PostgreSQL (RDS/Aurora, Redshift, Azure, Supabase) | unauthenticated StartupMessage | `AuthenticationOk` (server trusts the connection) | -| MongoDB (DocumentDB, Cosmos Mongo) | OP_MSG `listDatabases` | command succeeds without auth | -| MySQL, Azure SQL, Cassandra (Cosmos) | TCP connect only | *never* — reported `reachable` | - -Every check sends **no credentials**: no `Authorization` header, no password, no -token. "Anonymous" here means an unauthenticated request, exactly as an anonymous -cloud principal would make. - -Deep protocol probes (`PROBE_DEEP`, default on) only ever issue read-only -requests or commands; nothing is written, and no authentication is attempted. - -Probe outcomes are stored in `exposure_state` and a short factual -`exposure_detail` string (e.g. `s3 redirect region=eu-central-1`, -`status=403 authentication required`, `tcp/5432 open; no authentication -attempted`). Database endpoints are never claimed to be readable: a raw -`public_read` on a database is impossible by design. - -## Production notes - -* **Secrets** — database credentials live in `.env` (gitignored); `.env.example` - is the template. Inject the same variables from your orchestrator's secret - store in production. -* **Database TLS** — the local compose stack uses `sslmode=disable` because the - compose bridge network is private. For production, point `DATABASE_URL` at a - TLS-enabled database, e.g. - `postgres://user:pass@db.example.com:5432/bucketeer?sslmode=verify-full`. - pgx handles TLS natively; no extra flags are needed. -* **Metrics** — bound to loopback. There is no auth, no debug endpoints and no - high-cardinality labels (never hostnames). -* **Resource limits** — `mem_limit`/`cpus`/`pids_limit` are set in compose so - overload is bounded by the queue sizes rather than by the OOM killer. -* **Authorization** — leave `ACTIVE_PROBING=false` unless you are authorised to - probe the discovered infrastructure. - -## Database schema - -* `cloud_assets` — one row per `(provider, service, provider_hostname)`, with - `hostname` (original CT hostname), `resource_identifier`, `region`, - `resource_specific`, `public_semantic`, `exposure_state`, `exposure_detail`, - `probed_at`, `first_seen`, `last_seen`. -* `observations` — append-only observations linking an asset to a - `source_hostname`, `source_type` (`direct_ct`/`cname`), CNAME chain and - A/AAAA records. -* `ct_checkpoints` — per-log CT resume state (JSON-encoded certspotter - `LogState`; Merkle positions, no certificates). -* `ct_sths` — downloaded, unverified signed tree heads needed for safe resume. - -Schema is applied idempotently at startup from embedded SQL in `migrations/`. -No certificate contents are ever stored. - -## Performance characteristics - -Optimised in this order: I/O off the hot path, then serialization, then process -boundaries, then allocations, then cheap suffix/label matching, then bounded -pools, then DNS caching, dedup and batched writes. - -The classifier: - -* no regexp, JSON, reflection, DNS, HTTP, filesystem, DB or logging; -* no `strings.Split`, no `fmt.Sprintf`; -* lowercases only when an uppercase byte is present; -* parses labels by byte offsets, returning substrings (zero copy); -* fixed stack arrays, no unbounded maps or goroutines. - -### Benchmark results (Apple M4, Go 1.27) - -```text -BenchmarkClassifyAWS-10 60.03 ns/op 0 B/op 0 allocs/op -BenchmarkClassifyAzure-10 44.93 ns/op 0 B/op 0 allocs/op -BenchmarkRejectNormalDomain-10 30.28 ns/op 0 B/op 0 allocs/op -BenchmarkClassifyMixed-10 369.50 ns/op 0 B/op 0 allocs/op -``` - -The whole in-process sink path is allocation-free too: - -```text -BenchmarkHandleHostNonMatch-10 115.6 ns/op 0 B/op 0 allocs/op -BenchmarkHandleHostDirectMatch-10 155.7 ns/op 0 B/op 0 allocs/op -``` - -Run them yourself: - -```sh -go test ./internal/classify/ ./internal/pipeline/ -run '^$' -bench . -benchmem -``` - -`BenchmarkClassifyMixed` and `TestHandleHostNoAllocs` fail unless the result is -exactly 0 allocs/op, so the target is enforced by the test suite. - -### Bucket discovery - -Certificate SANs are custom domains, not bucket names, so matching SANs alone -finds almost no buckets. Instead, every non-matching hostname feeds a discovery -engine that: - -1. takes the names that actually occur in the hostname — the subdomain label - and the registrable-domain label (`assets.example.com` → `assets`, - `example`). No names are fabricated or drawn from a wordlist; -2. tests anonymous listing — - `https://.s3.amazonaws.com/?list-type=2` (AWS) and - `https://.storage.googleapis.com` (GCS); -3. indexes every bucket that returns 200 (keys/sizes/dates only). - -It is aggressively bounded: a global `BUCKET_SCAN_QPS` limit, a 12h per-name -dedup cache, and a random-sampled domain queue. - -Azure is handled separately because it has two levels: the account name maps to -DNS (`.blob.core.windows.net`), and anonymous access is per container -with no anonymous container enumeration — so for accounts that exist we probe a -small set of common container names. R2 is never listable and is only probed for -objects. - -This is what actually yields open buckets. Of the buckets found, S3 and GCS -return full object listings; Azure needs a known container; R2 only serves -objects you already know. - -### DNS load balancing - -When `DNS_SERVERS` is unset, the pool is ~20 curated, reliable public resolvers -(Cloudflare, Google, Quad9, OpenDNS, AdGuard, …). Each query picks a random -healthy resolver and fails over to at most two others; each attempt is bounded -(2s) and resolvers that fail repeatedly enter a 5-minute cooldown. Definitive -answers (NXDOMAIN) return immediately rather than being retried on every server, -so negative lookups do not multiply DNS traffic. - -> For sensitive deployments set `DNS_SERVERS` to your own resolvers, or `DNS_SERVERS=system`. - -> These community resolvers are untrusted: they can observe and, in principle, -> forge DNS answers. That only affects enrichment (which endpoints are looked -> up), never the classifier. For sensitive deployments set `DNS_SERVERS` to your -> own resolvers, or `DNS_SERVERS=system`. - -## Testing - -```sh -go test ./... # unit + table-driven provider tests -go test -race ./... # race detector -go vet ./... -``` - -Integration tests for PostgreSQL run only when `DATABASE_URL` is set: - -```sh -DATABASE_URL=postgres://bucketeer:bucketeer@localhost:5432/bucketeer go test ./internal/store/ -``` +- [docs/DESIGN.md](docs/DESIGN.md) — the UI design system. +- [docs/SKILL.md](docs/SKILL.md) — how an agent queries and audits a running + instance and answers questions from the API and PostgreSQL. ## Limitations -* `net.Resolver.LookupCNAME` returns the final canonical name, so intermediate - CNAME hops are not observed (the final target is still classified). Switch to - `miekg/dns` if the full chain is required. -* Optional reachability checks are off by default; when enabled they perform a - single non-destructive HTTPS `GET /` only — no authentication, downloads, - mutation or database queries. -* OCI generic `oraclecloud.com` endpoints are recognised but not stored, since - the bucket name lives in the URL path rather than the hostname. -* Custom R2 (and other) domains are only discovered through DNS/CNAME - enrichment, and only if the CT hostname itself appears in a certificate. +- CT rarely contains bucket hostnames directly; discovery is bounded and + stochastic, not exhaustive. Open buckets accumulate over time. +- Azure containers need names (no anonymous enumeration); Cloudflare R2 is never + listable. +- The database is the only state; there is no filesystem spool by design. + +## License + +[MIT](LICENSE). The vendored CertSpotter fork under `third_party/` remains under +its original Mozilla Public License v2.0 (`third_party/certspotter/LICENSE`). diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md deleted file mode 100644 index e6d47b2..0000000 --- a/docs/ARCHITECTURE.md +++ /dev/null @@ -1,92 +0,0 @@ -# Architecture - -## Data flow - -```text -Certificate Transparency Logs - │ - ▼ - CertSpotter internals fetch / verify / parse (third_party) - (in-process, no stdout/JSON/files) - │ - ▼ - DNS identifiers - │ - ▼ - FAST CLASSIFIER 0 allocs/op, no regexp/JSON/DNS/DB - ┌────┴─────┐ - direct cloud normal - resource hostname - │ │ - │ ▼ - │ DNS/CNAME enrichment bounded workers + TTL cache - │ │ - │ ▼ - │ classify CNAME targets - │ │ - └─────┬──────┘ - ▼ - dedup/enrichment TTL caches; DB uniqueness is final - │ - ▼ - optional safe reachability HTTPS GET / only, OFF by default - ▼ - batch DB writer bounded queue, pgx batches - ▼ - PostgreSQL -``` - -## Stages and what is allowed - -| Stage | May do | Must not do | -| --- | --- | --- | -| CertSpotter fetch/verify/parse | network, crypto, in-memory DER | persist certs, write files | -| Identifier callback | iterate `DNSNames`, call classifier | block, log per certificate, allocate needlessly | -| Classifier | byte/string comparisons, label scan | regexp, JSON, reflection, DNS, HTTP, DB, files, per-id logging | -| Direct queue | bound, non-blocking send | unbounded growth | -| DNS enrichment | DNS via worker pool, TTL cache, CNAME follow | unbounded goroutines, persistence | -| Dedup | memory-bounded TTL caches | unbounded maps | -| Probe (optional) | one unauthenticated vendor read endpoint (S3/GCS/Azure/Firebase list) or a bare TCP connect for databases | credentials, object downloads, mutation, DB queries, auth attempts | -| DB writer | batched `pgx.Batch` writes | one synchronous row per event | -| Checkpoints | JSON CT position in `ct_checkpoints`/`ct_sths` | certificate/issuer contents in DB | - -## Concurrency and backpressure - -* CertSpotter runs one monitor goroutine per CT log; it calls the identifier - sink concurrently. -* The sink never blocks: it classifies inline and does a non-blocking send to - either the direct queue or the DNS queue. Full queues increment - `queue_drops_total` / `queue_full_total`. -* Direct matches have a dedicated queue and worker pool, so speculative CNAME - enrichment can be shed first. DNS workers also skip work when the DB queue is - more than half full. -* The DB writer batches by size or `DB_FLUSH_INTERVAL` and flushes on - shutdown. - -## Failure handling - -* DNS and CT network errors are retried by CertSpotter with bounded exponential - backoff; bucketeer counts them (`ct_errors_total`) without logging per event. -* PostgreSQL writes retry a few times with backoff; persistent failure drops - items from the bounded DB queue rather than growing memory. -* The process uses `context` throughout and on `SIGTERM` stops intake, drains - the pipeline, flushes the DB batch and exits within `SHUTDOWN_TIMEOUT`. - -## Metrics and health - -`GET /metrics` exposes counters (`ct_certificates_total`, -`classifier_direct_matches_total`, `classifier_cname_matches_total`, -`dns_*`, `db_*`, `queue_*`) and gauges (`queue_depth{queue="…"}`, -`go_goroutines`, `process_system_bytes`). No hostnames or other high-cardinality -labels are ever used. - -`GET /healthz` is a liveness check. `GET /readyz` verifies the CT monitor is -initialised and PostgreSQL is reachable. - -## Why no files, pipes or subprocesses - -Everything between certificate parsing and classification stays in memory. The -CertSpotter patch hands identifiers directly to Go code, removing serialization -(JSON), IPC (stdout/pipe), process boundaries (subprocess/hooks) and filesystem -I/O (certificate/state files). CT resume state is the only thing persisted, and -it lives in PostgreSQL. diff --git a/docs/GOAL.md b/docs/GOAL.md deleted file mode 100644 index 10a2e67..0000000 --- a/docs/GOAL.md +++ /dev/null @@ -1,1618 +0,0 @@ -# Goal - -Build a production-ready, extremely resource-efficient Certificate Transparency monitoring service in **Go**, running entirely in **Docker**, using **CertSpotter's existing CT ingestion/parsing implementation**. - -The purpose is to continuously consume CT logs, identify cloud storage/database/data-service resources, enrich candidates via DNS/CNAME resolution, optionally perform minimal safe reachability checks, and persist only useful enriched findings into PostgreSQL. - -The primary design objective is: - -> **Maximum throughput and minimum CPU, memory, allocation, disk, serialization, and IPC overhead.** - -Do not build another generic CertStream consumer. Reuse CertSpotter's CT downloading, validation, parsing, and log-monitoring logic. - ---- - -# Critical architectural requirement - -DO NOT use CertSpotter like this: - -```text -certspotter - → stdout - → JSON - → pipe - → parser -``` - -DO NOT use: - -```text -CertSpotter - → JSON_FILENAME - → filesystem - → hook - → subprocess - → parser -``` - -Instead modify/integrate CertSpotter so parsed certificate DNS identifiers are handed directly to our Go code **inside the same process**. - -Target architecture: - -```text -Certificate Transparency Logs - │ - ▼ - CertSpotter internals - fetch / verify / parse - │ - ▼ - DNS identifiers - │ - ▼ - FAST CLASSIFIER - │ - ┌─────┴─────┐ - │ │ - direct cloud normal - resource hostname - │ │ - │ ▼ - │ DNS/CNAME enrichment - │ │ - │ ▼ - │ classify CNAME targets - │ │ - └─────┬─────┘ - ▼ - dedup/enrichment - │ - ▼ - optional safe reachability - │ - ▼ - batch DB writer - │ - ▼ - PostgreSQL -``` - -Everything between certificate parsing and candidate classification must remain in memory. - ---- - -# CertSpotter integration - -Use the upstream CertSpotter Go code as the CT engine. - -DO NOT reimplement CT log handling unless absolutely required. - -Inspect the current CertSpotter source before modifying anything. - -Create the **smallest possible maintainable patch** that exposes parsed certificate identifiers to our code. - -Preferred abstraction: - -```go -type IdentifierSink interface { - HandleIdentifier(ctx context.Context, identifier Identifier) -} -``` - -or an equally efficient callback. - -The goal is: - -```text -CertSpotter parsed certificate - ↓ -direct Go function call - ↓ -classifier -``` - -There must be: - -```text -NO stdout handoff -NO JSON handoff -NO hook scripts -NO subprocesses -NO certificate files -NO JSON files -NO temporary certificate files -``` - -Certificate DER/metadata may exist temporarily in memory while CertSpotter processes it, but must never be persisted. - -Do not use CertSpotter's normal certificate/output persistence functionality. - -Preserve upstream licensing/notices and keep our CertSpotter changes isolated so upstream changes can later be rebased. - ---- - -# Runtime filesystem policy - -The application must not persist certificates, SAN lists, JSON events, queues, enrichment payloads, or findings to the filesystem. - -No filesystem-based spool. - -No file-based queue. - -No external configuration files. - -No pattern YAML at runtime. - -No file uploads or file ingestion API. - -Configuration comes from environment variables. - -Provider patterns should be compiled into the binary. - -The Docker application container should require **no persistent volume**. - -Run with a read-only root filesystem. - ---- - -# CT checkpoint persistence - -We still need restart-safe CT monitoring. - -Do NOT persist CertSpotter checkpoint/state files locally. - -Store only the minimum required CT progress/checkpoint metadata in PostgreSQL. - -Example conceptual table: - -```text -ct_checkpoints - -log_id -tree_size -position / index -updated_at -``` - -Inspect CertSpotter's real state model and preserve whatever fields are actually required. - -If CertSpotter currently assumes filesystem-backed state, introduce a minimal storage abstraction: - -```go -type StateStore interface { - Load(...) - Save(...) -} -``` - -Implement: - -```text -PostgresStateStore -``` - -Do not store certificate contents in it. - ---- - -# Hot path requirements - -The certificate identifier classifier is the most performance-sensitive part. - -It must have: - -```text -NO regexp -NO JSON -NO reflection -NO interface{} maps -NO database access -NO DNS -NO HTTP -NO filesystem -NO subprocess -NO logging per identifier -NO unnecessary heap allocations -NO string splitting into []string -NO fmt.Sprintf -``` - -Target: - -```text -CertSpotter DNS identifier - ↓ -normalize - ↓ -cheap provider suffix rejection - ↓ -manual label/suffix classifier - ↓ -small struct - ↓ -bounded channel/ring -``` - -Use: - -```go -strings.HasSuffix -strings.HasPrefix -strings.Index / LastIndexByte -direct byte/string comparisons -manual backwards DNS-label scanning -small uint8 enums -``` - -Do not use regex. - -Do not use Hyperscan. - -Do not use Aho-Corasick unless benchmarking proves it faster for this actual tiny pattern set. - -For this number of signatures, direct suffix/label matching should be faster and simpler. - ---- - -# Allocation requirements - -The classifier itself should ideally achieve: - -```text -0 allocs/op -``` - -after receiving the hostname string. - -Avoid lowercasing every hostname if unnecessary. - -Fast-path ASCII lowercase names directly. - -Only normalize when uppercase characters are actually present. - -Avoid: - -```go -strings.Split() -``` - -because it allocates. - -Parse DNS labels by indexes. - ---- - -# Core data structures - -Use compact enums. - -Example: - -```go -type Provider uint8 - -const ( - ProviderUnknown Provider = iota - ProviderAWS - ProviderAzure - ProviderGCP - ProviderCloudflare - ProviderDigitalOcean - ProviderSupabase - ProviderOracle -) - -type Service uint8 - -const ( - ServiceUnknown Service = iota - - ServiceObjectStorage - ServiceS3 - ServiceBlobStorage - ServiceDatabase - ServicePostgres - ServiceMySQL - ServiceRDS - ServiceDocumentDB - ServiceRedshift - ServiceOpenSearch - ServiceNeptune - ServiceRedis - ServiceMemoryDB - ServiceCosmosDB - ServiceFirebaseDB -) -``` - -A hot-path match should look roughly like: - -```go -type Candidate struct { - Hostname string - Provider Provider - Service Service - Source Source -} -``` - -Keep hot-path structs small. - -Do not attach large certificate structures to downstream jobs. - ---- - -# Source tracking - -We need to distinguish: - -```text -DIRECT_CT -CNAME -DNS_ALIAS -``` - -Example: - -```text -assets.company.com - CNAME -company-assets.s3.eu-central-1.amazonaws.com -``` - -should produce an enriched record containing both: - -```text -original_hostname = assets.company.com -provider_hostname = company-assets.s3.eu-central-1.amazonaws.com -source = CNAME -provider = AWS -service = S3 -``` - ---- - -# Provider classification - -Implement dedicated provider classifiers. - -## AWS - -Recognize resource-specific forms including: - -```text -# S3 -.s3.amazonaws.com -.s3..amazonaws.com -.s3.dualstack..amazonaws.com -.s3-website..amazonaws.com -.s3-website-.amazonaws.com - -# Access Points --.s3-accesspoint..amazonaws.com -*.s3-accesspoint.dualstack..amazonaws.com -*.s3-accesspoint-fips..amazonaws.com - -# Multi-region -*.mrap.accesspoint.s3-global.amazonaws.com - -# S3 Express -*.s3express-..amazonaws.com - -# RDS / Aurora / RDS Proxy -*.rds.amazonaws.com - -# DocumentDB -*.docdb.amazonaws.com - -# Redshift -*.redshift.amazonaws.com -*.redshift-serverless.amazonaws.com - -# OpenSearch -search-*.es.amazonaws.com -vpc-*.es.amazonaws.com - -# Neptune -*.neptune.amazonaws.com - -# MemoryDB -*.memorydb..amazonaws.com - -# ElastiCache -*.cache.amazonaws.com -``` - -Further classify AWS subtypes where cheap. - -Examples: - -```text -cluster-* → Aurora/cluster -cluster-ro-* → reader -proxy-* → RDS proxy - -search-* → OpenSearch public endpoint form -vpc-* → OpenSearch VPC endpoint form -``` - -Do not treat generic AWS service API endpoints as customer resources. - -Examples to reject: - -```text -s3.eu-central-1.amazonaws.com -dynamodb.eu-central-1.amazonaws.com -redshift.eu-central-1.amazonaws.com -``` - ---- - -# Azure - -Recognize: - -```text -*.blob.core.windows.net -*.dfs.core.windows.net -*.file.core.windows.net -*.queue.core.windows.net -*.table.core.windows.net -*.web.core.windows.net - -*.database.windows.net -*.postgres.database.azure.com -*.mysql.database.azure.com - -*.documents.azure.com -*.mongo.cosmos.azure.com -*.cassandra.cosmos.azure.com -*.gremlin.cosmos.azure.com - -*.redis.cache.windows.net -*.redis.azure.net - -*.search.windows.net -``` - -Require a customer/resource-specific prefix. - -Do not classify bare provider service endpoints as resource findings. - ---- - -# Google Cloud / Firebase - -Recognize: - -```text -.storage.googleapis.com - -.firebaseio.com - -..firebasedatabase.app - -*.appspot.com -``` - -Do not classify: - -```text -storage.googleapis.com -``` - -itself as a resource. - ---- - -# Cloudflare - -Recognize: - -```text -.r2.cloudflarestorage.com -*.r2.dev -``` - -Classify: - -```text -r2.cloudflarestorage.com → Cloudflare R2 API/resource endpoint -r2.dev → public R2 development/public endpoint -``` - -Remember that arbitrary custom R2 domains will only be discovered through DNS/CNAME enrichment. - ---- - -# DigitalOcean - -Recognize: - -```text -..digitaloceanspaces.com -..cdn.digitaloceanspaces.com - -*.db.ondigitalocean.com -``` - -Do not treat: - -```text -.digitaloceanspaces.com -``` - -as a customer bucket. - ---- - -# Supabase - -Recognize: - -```text -.supabase.co -db..supabase.co -aws--.pooler.supabase.com -``` - -Classify direct PostgreSQL separately where possible: - -```text -db..supabase.co -``` - ---- - -# Oracle Cloud / OCI - -Recognize: - -```text -.objectstorage..oci.customer-oci.com -.swiftobjectstorage..oci.customer-oci.com -``` - -Also recognize generic OCI storage service endpoints but mark them: - -```text -resource_specific = false -``` - -Examples: - -```text -objectstorage..oraclecloud.com -swiftobjectstorage..oraclecloud.com -``` - -These should normally not become high-value findings because the bucket name exists in the path rather than the hostname. - ---- - -# Classification quality - -Every matcher must determine: - -```go -type Match struct { - Provider Provider - Service Service - ResourceSpecific bool - PublicSemantic PublicSemantic -} -``` - -`PublicSemantic` should distinguish cases where hostname format itself carries useful public/private information. - -Example: - -```text -AWS search-*.es.amazonaws.com - → public network endpoint form - -AWS vpc-*.es.amazonaws.com - → VPC endpoint form - -Cloudflare *.r2.dev - → explicitly public URL form -``` - -Do NOT equate: - -```text -resource exists -``` - -with: - -```text -data is publicly accessible -``` - ---- - -# Direct-match path - -Every SAN/DNS identifier received from CertSpotter should first be passed through the provider classifier. - -Direct provider matches should be extremely cheap: - -```text -SAN - ↓ -provider suffix rejection - ↓ -specific classifier - ↓ -Candidate -``` - -Direct matches go immediately into enrichment. - ---- - -# CNAME discovery - -Provider-native names alone are insufficient because certificates frequently contain custom hostnames: - -```text -files.example.com - ↓ CNAME -foo.blob.core.windows.net -``` - -or: - -```text -db.example.com - ↓ CNAME -prod.abc.eu-central-1.rds.amazonaws.com -``` - -Therefore also support DNS enrichment of CT hostnames. - -Pipeline: - -```text -CT hostname - ↓ -dedupe/cache - ↓ -DNS - ↓ -follow CNAME chain - ↓ -run classifier against EVERY CNAME target - ↓ -candidate -``` - -Follow CNAME chains with: - -```text -maximum depth: configurable -default: 8 -``` - -Detect loops. - ---- - -# DNS resolver - -DNS enrichment must be optimized heavily. - -Use a fixed worker pool. - -Do not spawn unbounded goroutines. - -Use either Go's resolver or `miekg/dns`, whichever benchmarks better for our workload. - -Requirements: - -```text -connection reuse where applicable -bounded concurrency -context cancellation -timeouts -positive TTL cache -negative TTL cache -CNAME cache -duplicate suppression -``` - -The DNS cache is memory-only. - -No persistence. - -Use a memory-bounded implementation. - -It must have a configurable maximum number of entries. - -If implementing a custom sharded TTL cache is simpler/faster than bringing in a heavy dependency, prefer the custom implementation. - ---- - -# Backpressure - -Do not allow enrichment to stall CertSpotter indefinitely. - -Use bounded queues. - -Separate: - -```text -direct-match queue - -DNS-enrichment queue - -DB-write queue -``` - -Expose counters for: - -```text -queue depth -queue full events -dropped optional DNS enrichments -direct matches -CNAME matches -DNS errors -DB errors -``` - -Direct provider matches should receive higher priority than speculative CNAME enrichment. - -If load shedding becomes necessary, drop optional CNAME enrichment before dropping direct cloud-resource matches. - ---- - -# Deduplication - -Certificates and CT logs produce duplicates. - -Perform cheap in-memory dedup before enrichment. - -Key approximately: - -```text -normalized hostname -``` - -After classification: - -```text -provider + service + provider_hostname -``` - -Use memory-bounded TTL caches. - -Do not keep an unbounded map. - -Database uniqueness is the final deduplication layer. - ---- - -# Enrichment model - -For useful candidates collect, when cheap: - -```text -original CT hostname -provider hostname -provider -service -region if extractable -resource identifier if extractable - -first seen -last seen - -CNAME chain - -A records -AAAA records - -public DNS resolution boolean - -source: - direct_ct - cname - -resource_specific - -public_semantic -``` - -Do not store the full certificate unless there is a concrete future requirement. - -Optional useful certificate metadata: - -```text -certificate fingerprint -not_before -not_after -issuer CN -CT log ID -``` - -Only include these if CertSpotter already has them available without additional expensive parsing/copying. - ---- - -# Safe reachability checks - -Make active network checks optional: - -```text -ACTIVE_PROBING=false -``` - -by default. - -When enabled, only perform minimal non-destructive checks on infrastructure the operator is authorized to assess. - -Allowed checks: - -```text -DNS resolution -TCP connect -TLS handshake -HTTP HEAD -HTTP GET / -``` - -No authentication attempts. - -No password guessing. - -No object downloading. - -No mutation. - -No PUT/POST/DELETE. - -No database queries. - -No exploitation. - -No brute force. - -For HTTP-based storage endpoints, use response status, headers, and provider error codes to classify: - -```text -reachable -not_found -auth_required -public_read -unknown -``` - -Do not claim something is "exposed" merely because its hostname exists. - ---- - -# Exposure state - -Use an explicit enum: - -```go -type ExposureState uint8 - -const ( - ExposureUnknown ExposureState = iota - ExposurePublicDNS - ExposureReachable - ExposureAuthRequired - ExposurePublicRead - ExposurePrivate - ExposureNotFound -) -``` - -Keep factual observations separate from conclusions. - ---- - -# PostgreSQL - -Use PostgreSQL with: - -```text -pgx/v5 -pgxpool -``` - -Do not write one row per network event synchronously. - -Use an asynchronous bounded DB writer with batching. - -Use `COPY FROM` or efficient pgx batches where appropriate. - -Suggested schema: - -```text -cloud_assets ------------- -id -provider -service -hostname -provider_hostname -resource_identifier -region -resource_specific -public_semantic -first_seen -last_seen -exposure_state - -UNIQUE(provider, service, provider_hostname) -``` - -and: - -```text -observations ------------- -asset_id -observed_at -source_hostname -source_type -cname_chain -ipv4 -ipv6 -cert_fingerprint -ct_log_id -``` - -Exact schema may be improved if benchmarking/query requirements justify it. - -Do not store observations that provide no useful information. - -Avoid unnecessary JSONB if typed PostgreSQL arrays/columns are more efficient. - ---- - -# Checkpoints - -Separate CT processing state from cloud-asset data. - -Example: - -```text -ct_checkpoints --------------- -log_id PRIMARY KEY -state -updated_at -``` - -Persist only enough to resume monitoring reliably. - -Batch/debounce checkpoint writes rather than writing on every certificate if CertSpotter semantics permit. - -Correctness must be preserved. - ---- - -# Logging - -Logging must never be on the certificate hot path. - -Do not log every certificate. - -Do not log every SAN. - -Do not log every rejected hostname. - -Log: - -```text -startup -shutdown -fatal errors -persistent DB errors -periodic aggregated statistics -``` - -Default periodic stats interval: - -```text -60 seconds -``` - -Use structured logging but keep it lightweight. - ---- - -# Metrics - -Expose lightweight Prometheus-compatible metrics on an optional HTTP listener. - -At minimum: - -```text -ct_certificates_total -ct_identifiers_total - -classifier_direct_matches_total -classifier_cname_matches_total - -dns_queries_total -dns_cache_hits_total -dns_errors_total - -candidates_total - -db_rows_total -db_errors_total - -queue_depth -queue_drops_total - -process memory -goroutines -``` - -Do not add expensive high-cardinality labels. - -Never label metrics by hostname. - ---- - -# Health - -Provide: - -```text -GET /healthz -GET /readyz -``` - -`readyz` should verify: - -```text -CT monitor initialized -database reachable -critical worker pipelines running -``` - ---- - -# Configuration - -Environment variables only. - -Suggested configuration: - -```text -DATABASE_URL= - -DNS_WORKERS= -DNS_TIMEOUT= -DNS_CACHE_SIZE= -DNS_MAX_CNAME_DEPTH= - -DIRECT_QUEUE_SIZE= -DNS_QUEUE_SIZE= -DB_QUEUE_SIZE= - -DB_BATCH_SIZE= -DB_FLUSH_INTERVAL= - -ACTIVE_PROBING=false -PROBE_WORKERS= -PROBE_TIMEOUT= - -METRICS_ADDR=:9090 -LOG_LEVEL=info -STATS_INTERVAL=60s -``` - -Provide sensible defaults. - -Validate configuration on startup. - ---- - -# Docker - -Create a production Docker setup. - -Use a multi-stage Dockerfile. - -Build a static/minimal Go binary where possible. - -Final image should be minimal, for example distroless or equivalent. - -Run as non-root. - -Requirements: - -```text -read-only root filesystem -no application volume -no CertSpotter volume -no certificate storage -no writable application directory -``` - -Use: - -```text -tmpfs -``` - -only if absolutely unavoidable; preferably the application should not require it. - -Provide: - -```text -Dockerfile -docker-compose.yml -``` - -Docker Compose should contain: - -```text -bucketeer -postgres -``` - -The application itself must remain stateless apart from PostgreSQL. - ---- - -# Code organization - -Keep the project simple. - -Suggested layout: - -```text -cmd/bucketeer/ - main.go - -internal/ - certspotter/ - classify/ - classify.go - aws.go - azure.go - gcp.go - cloudflare.go - digitalocean.go - supabase.go - oracle.go - - dns/ - probe/ - pipeline/ - store/ - metrics/ - config/ - model/ - -migrations/ -Dockerfile -docker-compose.yml -README.md -ARCHITECTURE.md -``` - -If CertSpotter is vendored/forked, keep upstream changes clearly separated. - -Do not introduce microservices. - -This should be **one Go application process** plus PostgreSQL. - -No Kafka. - -No Redis. - -No RabbitMQ. - -No external queue. - -No Elasticsearch. - -No unnecessary infrastructure. - ---- - -# Performance strategy - -Optimize in this order: - -1. eliminate I/O from hot path -2. eliminate serialization -3. eliminate process boundaries -4. eliminate allocations -5. cheap provider rejection -6. cheap suffix/label matching -7. bounded worker pools -8. DNS caching -9. deduplication before enrichment -10. batched DB writes - -Do not prematurely implement complex SIMD/native libraries unless benchmarks demonstrate an actual bottleneck. - ---- - -# Benchmarks - -Add benchmarks for classification. - -Example: - -```go -func BenchmarkClassifyAWS(b *testing.B) -func BenchmarkClassifyAzure(b *testing.B) -func BenchmarkClassifyMixed(b *testing.B) -func BenchmarkRejectNormalDomain(b *testing.B) -``` - -Benchmark realistic input: - -```text -www.example.com -foo.bar.example.org -bucket.s3.eu-central-1.amazonaws.com -foo.blob.core.windows.net -db.example.supabase.co -project.firebaseio.com -search-example-abc.eu-west-1.es.amazonaws.com -``` - -Important metric: - -```text -allocs/op -``` - -Classifier target: - -```text -0 allocs/op -``` - -Do not make CI depend on an absolute nanosecond threshold because runner performance varies. - -Report benchmark numbers in README. - ---- - -# Tests - -Implement table-driven tests covering every provider format. - -Test: - -```text -positive resource matches -generic provider endpoints rejected -uppercase handling -trailing dot -wildcards -invalid hostnames -very long hostnames -CNAME chains -CNAME loops -duplicate events -IPv4/IPv6 -context cancellation -database retries -queue saturation -``` - -Particularly ensure these DO NOT produce resource findings: - -```text -storage.googleapis.com -s3.eu-central-1.amazonaws.com -redshift.eu-central-1.amazonaws.com -fra1.digitaloceanspaces.com -``` - -while: - -```text -bucket.storage.googleapis.com -bucket.s3.eu-central-1.amazonaws.com -cluster.id.eu-central-1.redshift.amazonaws.com -bucket.fra1.digitaloceanspaces.com -``` - -do. - ---- - -# Wildcards - -Handle certificate wildcards carefully. - -Example: - -```text -*.example.com -``` - -must not become a literal DNS query. - -Do not interpret provider-wide wildcard certificates such as: - -```text -*.blob.core.windows.net -``` - -as discovered customer resources. - -Track wildcard certificates separately or ignore them for resource-specific classification unless a concrete hostname can be derived from another SAN. - ---- - -# Graceful shutdown - -Use contexts throughout. - -On SIGTERM: - -```text -stop accepting new enrichment -finish bounded in-flight enrichment -flush DB batch -save CT checkpoint -close DB -exit -``` - -Shutdown must have a maximum timeout. - -Docker should be able to stop the container cleanly. - ---- - -# Failure handling - -The program must survive transient: - -```text -DNS failures -network timeouts -CT connection failures -PostgreSQL connection resets -``` - -with bounded exponential backoff and jitter. - -Never retry in a tight loop. - -If PostgreSQL remains unavailable and queues approach capacity, prefer controlled backpressure/load shedding instead of unbounded memory growth. - ---- - -# Security - -Run as non-root. - -No shell needed in final container. - -No exposed debug endpoints. - -Set HTTP server timeouts. - -Validate all hostname lengths. - -Treat all CT data as untrusted input. - -Never execute certificate-controlled values. - -Never construct shell commands. - -Use parameterized SQL only. - ---- - -# README - -Document: - -```text -what the application does -architecture -why CertSpotter is embedded/patched -why stdout/hooks/files are deliberately avoided -supported provider patterns -configuration -Docker startup -database schema -performance characteristics -benchmark results -limitations -``` - -Clearly explain: - -```text -A detected provider endpoint does NOT automatically mean -the underlying data is publicly accessible. -``` - ---- - -# ARCHITECTURE.md - -Include a concise data-flow diagram: - -```text -CT - ↓ -CertSpotter - ↓ -Identifier callback - ↓ -Direct classifier ───────────────┐ - ↓ non-match │ -DNS/CNAME │ - ↓ │ -Provider classifier │ - └───────────────┬───────────────┘ - ↓ - dedupe - ↓ - enrichment - ↓ - optional probing - ↓ - DB batch - ↓ - PostgreSQL -``` - -Explain which operations are allowed in each stage. - ---- - -# Implementation priorities - -Implement in this order: - -## Phase 1 - -```text -CertSpotter integration -in-memory identifier callback -provider classifier -classification tests -benchmarks -``` - -Do not continue until the classifier has zero unnecessary allocations. - -## Phase 2 - -```text -bounded pipeline -dedup -DNS resolver -CNAME following -DNS cache -``` - -## Phase 3 - -```text -PostgreSQL schema -batch writer -Postgres CT checkpoint state -``` - -## Phase 4 - -```text -optional safe probing -metrics -health endpoints -``` - -## Phase 5 - -```text -Docker hardening -benchmarks -documentation -integration tests -``` - ---- - -# Important implementation rule - -Do not blindly follow this design if inspection of the current CertSpotter source reveals a more direct or efficient integration point. - -The goal is not to satisfy a predetermined class hierarchy. - -The goal is: - -```text -reuse CertSpotter -+ -remove output/storage overhead -+ -receive parsed DNS identifiers directly -+ -classify with essentially negligible CPU cost -+ -only enrich potentially useful data -+ -persist final useful results efficiently -``` - -If CertSpotter already exposes an internal abstraction that accomplishes this cleanly, use it rather than creating another layer. - ---- - -# Things you must NOT do - -Do not: - -```text -write certificates to disk -write JSON events to disk -read JSON event files -call CertSpotter through exec.Command -parse CertSpotter stdout -use hooks -fork a process per certificate -use regex in the classifier -use shell pipelines -store every CT certificate -store every SAN -create unbounded goroutines -create unbounded queues -create unbounded maps -write each result individually to PostgreSQL -introduce Kafka/Redis just for queues -build multiple microservices -``` - ---- - -# Definition of done - -The project is complete when: - -1. A single Dockerized Go process continuously consumes CT logs using CertSpotter internals. -2. CertSpotter never writes certificate/result files. -3. Parsed DNS identifiers are transferred directly into our classifier via Go calls. -4. The classifier covers AWS, Azure, GCP/Firebase, Cloudflare R2, DigitalOcean, Supabase and OCI patterns above. -5. Direct provider resources are detected immediately. -6. CNAME chains are asynchronously enriched and classified. -7. Duplicate processing is bounded and minimized. -8. Only useful resource findings reach PostgreSQL. -9. CT resume/checkpoint state lives in PostgreSQL rather than local files. -10. Classifier benchmarks demonstrate zero unnecessary allocations. -11. Resource usage remains bounded under overload. -12. Docker requires no application persistent volume. -13. `go test ./...` passes. -14. `go test -race ./...` passes. -15. `go vet ./...` passes. -16. Docker build succeeds. -17. The README explains exactly how to run and benchmark it. - -Before declaring completion, inspect the resulting program under load and remove any unnecessary: - -```text -allocations -copies -serialization -logging -locking -filesystem access -goroutine creation -database round trips -``` - -Favor simple, measurable performance over architectural complexity. diff --git a/third_party/certspotter/CTCLOUD_PATCH.md b/third_party/certspotter/BUCKETEER_PATCH.md similarity index 92% rename from third_party/certspotter/CTCLOUD_PATCH.md rename to third_party/certspotter/BUCKETEER_PATCH.md index a5c4059..ac2df2f 100644 --- a/third_party/certspotter/CTCLOUD_PATCH.md +++ b/third_party/certspotter/BUCKETEER_PATCH.md @@ -1,4 +1,4 @@ -# ctcloud patch to upstream CertSpotter +# Bucketeer patch to upstream CertSpotter This directory is a copy of upstream CertSpotter (https://github.com/SSLMate/certspotter), pulled in as a local module and @@ -39,7 +39,7 @@ if config.IdentifierSink != nil { ## Why -The sink gives ctcloud parsed DNS identifiers via a direct in-process Go call: +The sink gives Bucketeer parsed DNS identifiers via a direct in-process Go call: ```text CT log entry -> CertSpotter parse -> IdentifierSink.HandleIdentifiers -> classifier diff --git a/third_party/certspotter/monitor/config.go b/third_party/certspotter/monitor/config.go index 9d2fb31..335a9bd 100644 --- a/third_party/certspotter/monitor/config.go +++ b/third_party/certspotter/monitor/config.go @@ -17,8 +17,8 @@ import ( ) // IdentifierSink receives the DNS identifiers parsed from every monitored -// certificate. This is a CTCLOUD PATCH to upstream certspotter; see -// CTCLOUD_PATCH.md. When non-nil it is invoked on the monitoring hot path +// certificate. This is a BUCKETEER PATCH to upstream certspotter; see +// BUCKETEER_PATCH.md. When non-nil it is invoked on the monitoring hot path // instead of the watchlist/notification machinery. type IdentifierSink interface { HandleIdentifiers(ctx context.Context, identifiers *certspotter.Identifiers) @@ -32,7 +32,7 @@ type Config struct { Verbose bool HealthCheckInterval time.Duration - // IdentifierSink is a CTCLOUD PATCH (see CTCLOUD_PATCH.md). When set, + // IdentifierSink is a BUCKETEER PATCH (see BUCKETEER_PATCH.md). When set, // parsed identifiers are handed directly to it in-process and the normal // watchlist/chain/notification path is skipped. IdentifierSink IdentifierSink diff --git a/third_party/certspotter/monitor/process.go b/third_party/certspotter/monitor/process.go index 958a686..fd4852c 100644 --- a/third_party/certspotter/monitor/process.go +++ b/third_party/certspotter/monitor/process.go @@ -101,8 +101,8 @@ func processCertificate(ctx context.Context, config *Config, entry *LogEntry, ce if err != nil { return processMalformedLogEntry(ctx, config, entry, err) } - // CTCLOUD PATCH: hand identifiers directly to the in-process sink and skip - // the watchlist/chain/notification path. See CTCLOUD_PATCH.md. + // BUCKETEER PATCH: hand identifiers directly to the in-process sink and skip + // the watchlist/chain/notification path. See BUCKETEER_PATCH.md. if config.IdentifierSink != nil { config.IdentifierSink.HandleIdentifiers(ctx, identifiers) return nil