bucketeer/README.md

# ctcloud

`ctcloud` continuously consumes Certificate Transparency (CT) logs, finds
cloud storage/database/data-service resources referenced by certificates,
enriches candidates through DNS/CNAME resolution, and persists only useful
findings to PostgreSQL.

It is a **single stateless Go process plus PostgreSQL**. CertSpotter's real CT
fetching, validation and parsing code is embedded and called **in-process**;
there is no stdout handoff, no JSON handoff, no hooks, no subprocesses and no
certificate files.

> A detected provider endpoint **does not** mean the underlying data is publicly
> accessible. `ctcloud` reports facts (hostname existed in CT, resolved, answered
> a request) separately from conclusions (`exposure_state`). Most findings are
> simply an endpoint that exists.

## How it works

```text
CT logs ──► CertSpotter internals (fetch/verify/parse)
              │  direct Go call: IdentifierSink.HandleIdentifiers
              ▼
        fast classifier  (0 allocs/op, no regexp/JSON/DNS)
        ┌──────┴───────┐
   direct match    non-match
        │              ▼
        │        DNS/CNAME enrichment
        │           (followed + classified)
        └──────┬───────┘
               ▼
        dedup + bounded queues
               ▼
     optional safe reachability
               ▼
        batched PostgreSQL writer
               ▼
           PostgreSQL
```

See [ARCHITECTURE.md](ARCHITECTURE.md) for per-stage rules.

## Why CertSpotter is embedded (and patched)

Re-implementing CT log handling is a large, security-sensitive job. Upstream
CertSpotter already does log-list handling, STH fetching/verification, Merkle
tree maintenance, entry parsing and certificate reconstruction.

The only change is a two-hunk patch in `third_party/certspotter` that adds an
`IdentifierSink` callback. After a certificate is parsed, the identifiers are
handed to ctcloud with a direct Go function call and the watchlist/chain/
notification machinery is skipped:

```go
if config.IdentifierSink != nil {
    config.IdentifierSink.HandleIdentifiers(ctx, identifiers)
    return nil
}
```

See [`third_party/certspotter/CTCLOUD_PATCH.md`](third_party/certspotter/CTCLOUD_PATCH.md).
This deliberately avoids stdout piping, JSON files, hook scripts, subprocesses
and temporary certificate files, which are all slow, serializing and
filesystem-bound. Upstream licensing/notices are preserved and the patch is
isolated so it can be rebased.

## Supported provider patterns

**AWS** — S3 (`<bucket>.s3[.dualstack].<region>.amazonaws.com`,
`s3-website[-<region>]`), S3 access points (`s3-accesspoint[-fips]`,
`dualstack`), multi-region access points (`mrap.accesspoint.s3-global`), S3
Express (`s3express-*`), RDS/Aurora/proxy (`*.rds.amazonaws.com`), DocumentDB,
Redshift (provisioned + serverless), OpenSearch (`search-*` public / `vpc-*`
VPC), Neptune, MemoryDB, ElastiCache. Generic API endpoints
(`s3.eu-central-1.amazonaws.com`, `dynamodb.*`, `redshift.<region>`,
`rds.<region>`, …) are rejected.

**Azure** — `blob|dfs|file|queue|table|web.core.windows.net`,
`*.database.windows.net`, `*.postgres|mysql.database.azure.com`,
`*.documents.azure.com`, `*.{mongo,cassandra,gremlin}.cosmos.azure.com`,
`*.redis.cache.windows.net`, `*.redis.azure.net`, `*.search.windows.net`.
Bare service endpoints are rejected.

**Google Cloud / Firebase** — `<bucket>.storage.googleapis.com`,
`<db>.firebaseio.com`, `<db>.<region>.firebasedatabase.app`,
`<project>.appspot.com`. `storage.googleapis.com` itself is rejected.

**Cloudflare R2** — `<account>.r2.cloudflarestorage.com`; `<account>.r2.dev`
is flagged as an explicitly public URL form.

**DigitalOcean** — `<bucket>.<region>[.cdn].digitaloceanspaces.com`,
`*.db.ondigitalocean.com`. `<region>.digitaloceanspaces.com` is rejected.

**Supabase** — `<ref>.supabase.co`, `db.<ref>.supabase.co` (PostgreSQL),
`aws-<n>-<region>.pooler.supabase.com`.

**Oracle Cloud (OCI)** — `<namespace>.objectstorage|swiftobjectstorage.<region>.oci.customer-oci.com`
(resource-specific), and the generic `objectstorage.<region>.oraclecloud.com`
forms (`resource_specific = false`; not stored as findings).

Provider-specific details live in `internal/classify/`.

## Configuration

Environment variables only; no config files, no runtime pattern files.

| Variable | Default | Purpose |
| --- | --- | --- |
| `DATABASE_URL` | *(required)* | PostgreSQL connection string |
| `DB_MAX_CONNS` | `8` | pgx pool size |
| `DNS_SERVERS` | `1.1.1.1:53,8.8.8.8:53,1.0.0.1:53` | comma-separated resolvers (round-robin + failover); `system` for the host resolver |
| `DNS_WORKERS` | `16` | DNS enrichment workers |
| `DNS_TIMEOUT` | `5s` | per-lookup timeout |
| `DNS_CACHE_SIZE` | `262144` | max DNS cache entries |
| `DEDUP_SIZE` | `262144` | max entries in each dedup cache |
| `DNS_MAX_CNAME_DEPTH` | `8` | max CNAME hops |
| `DNS_POSITIVE_TTL` / `DNS_NEGATIVE_TTL` | `10m` / `1m` | DNS cache TTLs |
| `DIRECT_QUEUE_SIZE` | `8192` | direct-match queue capacity |
| `DNS_QUEUE_SIZE` | `65536` | DNS-enrichment queue capacity |
| `DB_QUEUE_SIZE` | `8192` | DB-write queue capacity |
| `DB_BATCH_SIZE` | `500` | findings per write batch |
| `DB_FLUSH_INTERVAL` | `2s` | max time before a batch is flushed |
| `ACTIVE_PROBING` | `false` | enable optional reachability checks |
| `PROBE_WORKERS` / `PROBE_TIMEOUT` | `8` / `5s` | probe worker pool |
| `PROBE_QUEUE_SIZE` | `65536` | probe sample size; when full, old entries are replaced at random (bounded memory) |
| `DEDUP_TTL` | `1h` | hostname/finding dedup window |
| `METRICS_ADDR` | `:9090` | metrics/health/UI listener |
| `UI_USER` / `UI_PASS` | *(unset)* | optional HTTP basic auth for the web UI |
| `LOG_LEVEL` | `info` | `debug\|info\|warn\|error` |
| `STATS_INTERVAL` | `60s` | aggregated stats log interval |
| `LOG_LIST_SOURCE` | Google log list v3 | certspotter log list URL |
| `START_AT_END` | `false` | skip existing backlog on first run |
| `SHUTDOWN_TIMEOUT` | `20s` | graceful shutdown budget |

## Running with Docker

```sh
cp .env.example .env      # set a strong POSTGRES_PASSWORD and matching DATABASE_URL
docker compose up --build
```

That starts `postgres` and the stateless `ctcloud` container. The application
container runs as non-root, has a read-only root filesystem, drops all Linux
capabilities, sets CPU/memory/pids limits and mounts no application volume. The
only persistent state is PostgreSQL (CT checkpoints + cloud assets).

Endpoints on port `9090`, bound to `127.0.0.1` only:

* `GET /` — web UI (see below)
* `GET /api/assets`, `GET /api/stats` — JSON for the UI
* `GET /metrics`, `GET /healthz`, `GET /readyz`

The listener has no authentication by default, so scrape it locally or from a
sidecar and never expose it publicly. Set `UI_USER`/`UI_PASS` to require HTTP
basic auth on the UI and its API.

## Web UI

A small, dependency-free UI lists discovered assets with provider/service,
resource, region, exposure state and the probe detail, plus summary counts and
provider/service/exposure filters. It is read-only, refreshes every 15s, and is
served from the same loopback port (`http://127.0.0.1:9090/`).

## Reachability and readability checks

`ACTIVE_PROBING=false` by default. When enabled, the prober issues **one**
unauthenticated request per finding. It never sends credentials, downloads
objects, mutates anything, runs database queries or attempts authentication.

| Vendor | Check | `public_read` means |
| --- | --- | --- |
| AWS S3, Cloudflare R2, DigitalOcean Spaces | anonymous `ListObjectsV2` (`?list-type=2&max-keys=1`) | anonymous bucket listing succeeded |
| Google Cloud Storage | JSON API object list (`/storage/v1/b/<bucket>/o?maxResults=1`) | `allUsers` can list objects |
| Azure Blob/File/Queue/Table | anonymous `?comp=list` | anonymous listing succeeded |
| Firebase Realtime DB | `GET /.json?shallow=true` | security rules allow anonymous read |
| Supabase | `GET /rest/v1/` | — (an API key is required, so this reports `auth_required`) |
| Cosmos, OpenSearch, Search, Appspot, OCI | root `GET /` | a 2xx was returned |
| RDS, Redshift, DocumentDB, Neptune, ElastiCache, MemoryDB, Azure SQL/Postgres/MySQL/Redis, Supabase Postgres | unauthenticated TCP connect only | *never* — reported as `reachable` (port open) |

Probe outcomes are stored in `exposure_state` and a short factual
`exposure_detail` string (e.g. `s3 redirect region=eu-central-1`,
`status=403 authentication required`, `tcp/5432 open; no authentication
attempted`). Database endpoints are never claimed to be readable: a raw
`public_read` on a database is impossible by design.

## Production notes

* **Secrets** — database credentials live in `.env` (gitignored); `.env.example`
  is the template. Inject the same variables from your orchestrator's secret
  store in production.
* **Database TLS** — the local compose stack uses `sslmode=disable` because the
  compose bridge network is private. For production, point `DATABASE_URL` at a
  TLS-enabled database, e.g.
  `postgres://user:[email protected]:5432/ctcloud?sslmode=verify-full`.
  pgx handles TLS natively; no extra flags are needed.
* **Metrics** — bound to loopback. There is no auth, no debug endpoints and no
  high-cardinality labels (never hostnames).
* **Resource limits** — `mem_limit`/`cpus`/`pids_limit` are set in compose so
  overload is bounded by the queue sizes rather than by the OOM killer.
* **Authorization** — leave `ACTIVE_PROBING=false` unless you are authorised to
  probe the discovered infrastructure.

## Database schema

* `cloud_assets` — one row per `(provider, service, provider_hostname)`, with
  `hostname` (original CT hostname), `resource_identifier`, `region`,
  `resource_specific`, `public_semantic`, `exposure_state`, `exposure_detail`,
  `probed_at`, `first_seen`, `last_seen`.
* `observations` — append-only observations linking an asset to a
  `source_hostname`, `source_type` (`direct_ct`/`cname`), CNAME chain and
  A/AAAA records.
* `ct_checkpoints` — per-log CT resume state (JSON-encoded certspotter
  `LogState`; Merkle positions, no certificates).
* `ct_sths` — downloaded, unverified signed tree heads needed for safe resume.

Schema is applied idempotently at startup from embedded SQL in `migrations/`.
No certificate contents are ever stored.

## Performance characteristics

Optimised in this order: I/O off the hot path, then serialization, then process
boundaries, then allocations, then cheap suffix/label matching, then bounded
pools, then DNS caching, dedup and batched writes.

The classifier:

* no regexp, JSON, reflection, DNS, HTTP, filesystem, DB or logging;
* no `strings.Split`, no `fmt.Sprintf`;
* lowercases only when an uppercase byte is present;
* parses labels by byte offsets, returning substrings (zero copy);
* fixed stack arrays, no unbounded maps or goroutines.

### Benchmark results (Apple M4, Go 1.27)

```text
BenchmarkClassifyAWS-10            60.03 ns/op    0 B/op   0 allocs/op
BenchmarkClassifyAzure-10          44.93 ns/op    0 B/op   0 allocs/op
BenchmarkRejectNormalDomain-10     30.28 ns/op    0 B/op   0 allocs/op
BenchmarkClassifyMixed-10         369.50 ns/op    0 B/op   0 allocs/op
```

The whole in-process sink path is allocation-free too:

```text
BenchmarkHandleHostNonMatch-10      115.6 ns/op    0 B/op   0 allocs/op
BenchmarkHandleHostDirectMatch-10  155.7 ns/op    0 B/op   0 allocs/op
```

Run them yourself:

```sh
go test ./internal/classify/ ./internal/pipeline/ -run '^$' -bench . -benchmem
```

`BenchmarkClassifyMixed` and `TestHandleHostNoAllocs` fail unless the result is
exactly 0 allocs/op, so the target is enforced by the test suite.

### DNS load balancing

DNS lookups are round-robined across `DNS_SERVERS`. Each attempt is bounded
(2s) and fails over to the next server on timeout/refused errors, so one dead
resolver does not stall lookups. Definitive answers (NXDOMAIN) are returned
immediately rather than retried on every server, so negative lookups do not
multiply DNS traffic.

## Testing

```sh
go test ./...          # unit + table-driven provider tests
go test -race ./...    # race detector
go vet ./...
```

Integration tests for PostgreSQL run only when `DATABASE_URL` is set:

```sh
DATABASE_URL=postgres://ctcloud:ctcloud@localhost:5432/ctcloud go test ./internal/store/
```

## Limitations

* `net.Resolver.LookupCNAME` returns the final canonical name, so intermediate
  CNAME hops are not observed (the final target is still classified). Switch to
  `miekg/dns` if the full chain is required.
* Optional reachability checks are off by default; when enabled they perform a
  single non-destructive HTTPS `GET /` only — no authentication, downloads,
  mutation or database queries.
* OCI generic `oraclecloud.com` endpoints are recognised but not stored, since
  the bucket name lives in the URL path rather than the hostname.
* Custom R2 (and other) domains are only discovered through DNS/CNAME
  enrichment, and only if the CT hostname itself appears in a certificate.