bucketeer/README.md

# ctcloud

`ctcloud` continuously consumes Certificate Transparency (CT) logs, finds
cloud storage/database/data-service resources referenced by certificates,
enriches candidates through DNS/CNAME resolution, and persists only useful
findings to PostgreSQL.

It is a **single stateless Go process plus PostgreSQL**. CertSpotter's real CT
fetching, validation and parsing code is embedded and called **in-process**;
there is no stdout handoff, no JSON handoff, no hooks, no subprocesses and no
certificate files.

> A detected provider endpoint **does not** mean the underlying data is publicly
> accessible. `ctcloud` reports facts (hostname existed in CT, resolved, answered
> a request) separately from conclusions (`exposure_state`). Most findings are
> simply an endpoint that exists.

## How it works

```text
CT logs ──► CertSpotter internals (fetch/verify/parse)
              │  direct Go call: IdentifierSink.HandleIdentifiers
              ▼
        fast classifier  (0 allocs/op, no regexp/JSON/DNS)
        ┌──────┴───────┐
   direct match    non-match
        │              ▼
        │        DNS/CNAME enrichment
        │           (followed + classified)
        └──────┬───────┘
               ▼
        dedup + bounded queues
               ▼
     optional safe reachability
               ▼
        batched PostgreSQL writer
               ▼
           PostgreSQL
```

See [ARCHITECTURE.md](ARCHITECTURE.md) for per-stage rules.

## Why CertSpotter is embedded (and patched)

Re-implementing CT log handling is a large, security-sensitive job. Upstream
CertSpotter already does log-list handling, STH fetching/verification, Merkle
tree maintenance, entry parsing and certificate reconstruction.

The only change is a two-hunk patch in `third_party/certspotter` that adds an
`IdentifierSink` callback. After a certificate is parsed, the identifiers are
handed to ctcloud with a direct Go function call and the watchlist/chain/
notification machinery is skipped:

```go
if config.IdentifierSink != nil {
    config.IdentifierSink.HandleIdentifiers(ctx, identifiers)
    return nil
}
```

See [`third_party/certspotter/CTCLOUD_PATCH.md`](third_party/certspotter/CTCLOUD_PATCH.md).
This deliberately avoids stdout piping, JSON files, hook scripts, subprocesses
and temporary certificate files, which are all slow, serializing and
filesystem-bound. Upstream licensing/notices are preserved and the patch is
isolated so it can be rebased.

## Supported provider patterns

**AWS** — S3 (`<bucket>.s3[.dualstack].<region>.amazonaws.com`,
`<bucket>.s3-<region>.amazonaws.com` legacy form, `s3-website[-<region>]`),
S3 access points (`s3-accesspoint[-fips]`,
`dualstack`), multi-region access points (`mrap.accesspoint.s3-global`), S3
Express (`s3express-*`), RDS/Aurora/proxy (`*.rds.amazonaws.com`), DocumentDB,
Redshift (provisioned + serverless), OpenSearch (`search-*` public / `vpc-*`
VPC), Neptune, MemoryDB, ElastiCache. Generic API endpoints
(`s3.eu-central-1.amazonaws.com`, `dynamodb.*`, `redshift.<region>`,
`rds.<region>`, …) are rejected.

**Azure** — `blob|dfs|file|queue|table|web.core.windows.net`,
`*.database.windows.net`, `*.postgres|mysql.database.azure.com`,
`*.documents.azure.com`, `*.{mongo,cassandra,gremlin}.cosmos.azure.com`,
`*.redis.cache.windows.net`, `*.redis.azure.net`, `*.search.windows.net`.
Bare service endpoints are rejected.

**Google Cloud / Firebase** — `<bucket>.storage.googleapis.com`,
`<db>.firebaseio.com`, `<db>.<region>.firebasedatabase.app`,
`<project>.appspot.com`. `storage.googleapis.com` itself is rejected.

**Cloudflare R2** — `<account>.r2.cloudflarestorage.com`; `<account>.r2.dev`
is flagged as an explicitly public URL form.

**DigitalOcean** — `<bucket>.<region>[.cdn].digitaloceanspaces.com`,
`*.db.ondigitalocean.com`. `<region>.digitaloceanspaces.com` is rejected.

**Supabase** — `<ref>.supabase.co`, `db.<ref>.supabase.co` (PostgreSQL),
`aws-<n>-<region>.pooler.supabase.com`.

**Oracle Cloud (OCI)** — `<namespace>.objectstorage|swiftobjectstorage.<region>.oci.customer-oci.com`
(resource-specific), and the generic `objectstorage.<region>.oraclecloud.com`
forms (`resource_specific = false`; not stored as findings).

Provider-specific details live in `internal/classify/`.

## Configuration

Environment variables only; no config files, no runtime pattern files.

| Variable | Default | Purpose |
| --- | --- | --- |
| `DATABASE_URL` | *(required)* | PostgreSQL connection string |
| `DB_MAX_CONNS` | `8` | pgx pool size |
| `DNS_SERVERS` | *(unset → curated + community list)* | comma-separated resolvers, or `system` for the OS resolver |
| `DNS_WORKERS` | `16` | DNS enrichment workers |
| `DNS_TIMEOUT` | `5s` | per-lookup timeout |
| `DNS_CACHE_SIZE` | `262144` | max DNS cache entries |
| `DEDUP_SIZE` | `262144` | max entries in each dedup cache |
| `DNS_MAX_CNAME_DEPTH` | `8` | max CNAME hops |
| `DNS_POSITIVE_TTL` / `DNS_NEGATIVE_TTL` | `10m` / `1m` | DNS cache TTLs |
| `DIRECT_QUEUE_SIZE` | `8192` | direct-match queue capacity |
| `DNS_QUEUE_SIZE` | `65536` | DNS-enrichment queue capacity |
| `DB_QUEUE_SIZE` | `8192` | DB-write queue capacity |
| `DB_BATCH_SIZE` | `500` | findings per write batch |
| `DB_FLUSH_INTERVAL` | `2s` | max time before a batch is flushed |
| `ACTIVE_PROBING` | `false` | enable optional reachability checks |
| `PROBE_WORKERS` / `PROBE_TIMEOUT` | `8` / `5s` | probe worker pool |
| `PROBE_QUEUE_SIZE` | `65536` | probe sample size; when full, old entries are replaced at random (bounded memory) |
| `PROBE_DEEP` | `true` | unauthenticated, read-only protocol reads for databases (Redis/Postgres/Mongo) |
| `INDEX_ENABLED` / `INDEX_MAX_OBJECTS` | `true` / `1000` | index metadata of readable resources (keys/sizes only) |
| `LOG_FORMAT` | `text` | `text` or `json` (JSON is best for log pipelines/LLM analysis) |
| `BUCKET_SCAN_ENABLED` | `true` | discover open buckets by name enumeration |
| `BUCKET_SCAN_QPS` | `50` | global cap on outbound discovery requests |
| `BUCKET_SCAN_WORKERS` | `16` | discovery probe workers |
| `BUCKET_SCAN_QUEUE` | `65536` | sampled domain backlog |
| `DEDUP_TTL` | `1h` | hostname/finding dedup window |
| `METRICS_ADDR` | `:9090` | metrics/health/UI listener |
| `UI_USER` / `UI_PASS` | *(unset)* | optional HTTP basic auth for the web UI |
| `LOG_LEVEL` | `info` | `debug\|info\|warn\|error` |
| `STATS_INTERVAL` | `60s` | aggregated stats log interval |
| `LOG_LIST_SOURCE` | Google log list v3 | certspotter log list URL |
| `START_AT_END` | `false` | skip existing backlog on first run |
| `SHUTDOWN_TIMEOUT` | `20s` | graceful shutdown budget |

## Running with Docker

```sh
cp .env.example .env      # set a strong POSTGRES_PASSWORD and matching DATABASE_URL
docker compose up --build
```

That starts `postgres` and the stateless `ctcloud` container. The application
container runs as non-root, has a read-only root filesystem, drops all Linux
capabilities, sets CPU/memory/pids limits and mounts no application volume. The
only persistent state is PostgreSQL (CT checkpoints + cloud assets + index).

> **Findings accumulate over time.** Use `docker compose stop` / `docker compose
> start` (or plain `up -d`) to keep the `pgdata` volume. `docker compose down -v`
> deletes the volume and every discovered asset with it.

Endpoints on port `9090`, bound to `127.0.0.1` only:

* `GET /` — web UI (see below)
* `GET /api/assets`, `GET /api/stats` — JSON for the UI
* `GET /metrics`, `GET /healthz`, `GET /readyz`

The listener has no authentication by default, so scrape it locally or from a
sidecar and never expose it publicly. Set `UI_USER`/`UI_PASS` to require HTTP
basic auth on the UI and its API.

## Web UI

A small, dependency-free UI (vanilla JS, no build step, embedded in the binary)
styled to the tokens and layout in `DESIGN.md` (neutral canvas, single 768px
results column, flat rows, pill controls, no shadows).

* search box with 250ms debounce, stale-request cancellation and `?q=` URL state
* exposure filter chips — use **Public read** to see open buckets, **Reachable**
  to see publicly reachable database endpoints — plus a sort control
* result rows show the provider hostname, probe detail, provider/service/region
  badges, resource identifier and relative last-seen/probed times
* summary line with total assets, publicly readable and reachable counts
* refreshes every 15s (paused when the tab is hidden), read-only
* light and dark mode: follows the OS by default, with a header toggle that is
  persisted in `localStorage`; the theme is applied before first paint to avoid
  a flash

Served from the same loopback port: `http://127.0.0.1:9090/`.

### Brand assets

The UI ships the official Bucketeer mascot for both themes, optimized to
lossless WebP and embedded in the binary (`internal/webui/assets/`):

| Asset | Size | Use |
| --- | --- | --- |
| `logo-light.webp` | ~27 KB | black line art, light theme |
| `logo-dark.webp` | ~22 KB | white line art, dark theme |
| `favicon.png` | ~7 KB | browser tab |

They are downscaled to 256px and encoded with `cwebp -lossless -z 9`, so they
stay crisp on HiDPI displays while shrinking the ~1.2 MB source PNGs by ~96%.
Regenerate after replacing the source PNGs at the repo root:

```sh
magick bucketeer-light.png -resize 256x -strip /tmp/l.png
magick bucketeer-dark.png  -resize 256x -strip /tmp/d.png
cwebp -quiet -lossless -z 9 -alpha_q 100 /tmp/l.png -o internal/webui/assets/logo-light.webp
cwebp -quiet -lossless -z 9 -alpha_q 100 /tmp/d.png -o internal/webui/assets/logo-dark.webp
```

## Reachability and readability checks

`ACTIVE_PROBING=false` by default. When enabled, the prober issues **one**
unauthenticated request per finding. It never sends credentials, downloads
objects, mutates anything, runs database queries or attempts authentication.

| Vendor | Check | `public_read` means |
| --- | --- | --- |
| AWS S3, Cloudflare R2, DigitalOcean Spaces, OCI | anonymous `ListObjectsV2` (`?list-type=2&max-keys=1`) | bucket policy grants `s3:ListBucket` to `*` |
| Google Cloud Storage | JSON API object list (`/storage/v1/b/<bucket>/o?maxResults=1`) | the bucket grants `allUsers` `storage.objects.list` |
| Azure Blob/File/Queue/Table | account-level `?comp=list` | anonymous access is **per container**, so account-level 403 is reported `reachable` (not "not readable"). Container names cannot be derived from the account hostname. |
| Firebase Realtime DB | `GET /.json?shallow=true` | security rules allow anonymous read |
| Supabase | `GET /rest/v1/` | not testable without the public **anon API key**; reports `auth_required` |
| OpenSearch / Elasticsearch | `GET /_cat/indices` | security/access control disabled |
| Neptune | `GET /status` | a 2xx was returned |
| Cosmos / Appspot / Azure Search | root `GET /` | a 2xx was returned |
| Redis (ElastiCache, MemoryDB, Azure Redis) | unauthenticated `PING` | `+PONG` without `AUTH` |
| PostgreSQL (RDS/Aurora, Redshift, Azure, Supabase) | unauthenticated StartupMessage | `AuthenticationOk` (server trusts the connection) |
| MongoDB (DocumentDB, Cosmos Mongo) | OP_MSG `listDatabases` | command succeeds without auth |
| MySQL, Azure SQL, Cassandra (Cosmos) | TCP connect only | *never* — reported `reachable` |

Every check sends **no credentials**: no `Authorization` header, no password, no
token. "Anonymous" here means an unauthenticated request, exactly as an anonymous
cloud principal would make.

Deep protocol probes (`PROBE_DEEP`, default on) only ever issue read-only
requests or commands; nothing is written, and no authentication is attempted.

Probe outcomes are stored in `exposure_state` and a short factual
`exposure_detail` string (e.g. `s3 redirect region=eu-central-1`,
`status=403 authentication required`, `tcp/5432 open; no authentication
attempted`). Database endpoints are never claimed to be readable: a raw
`public_read` on a database is impossible by design.

## Production notes

* **Secrets** — database credentials live in `.env` (gitignored); `.env.example`
  is the template. Inject the same variables from your orchestrator's secret
  store in production.
* **Database TLS** — the local compose stack uses `sslmode=disable` because the
  compose bridge network is private. For production, point `DATABASE_URL` at a
  TLS-enabled database, e.g.
  `postgres://user:[email protected]:5432/ctcloud?sslmode=verify-full`.
  pgx handles TLS natively; no extra flags are needed.
* **Metrics** — bound to loopback. There is no auth, no debug endpoints and no
  high-cardinality labels (never hostnames).
* **Resource limits** — `mem_limit`/`cpus`/`pids_limit` are set in compose so
  overload is bounded by the queue sizes rather than by the OOM killer.
* **Authorization** — leave `ACTIVE_PROBING=false` unless you are authorised to
  probe the discovered infrastructure.

## Database schema

* `cloud_assets` — one row per `(provider, service, provider_hostname)`, with
  `hostname` (original CT hostname), `resource_identifier`, `region`,
  `resource_specific`, `public_semantic`, `exposure_state`, `exposure_detail`,
  `probed_at`, `first_seen`, `last_seen`.
* `observations` — append-only observations linking an asset to a
  `source_hostname`, `source_type` (`direct_ct`/`cname`), CNAME chain and
  A/AAAA records.
* `ct_checkpoints` — per-log CT resume state (JSON-encoded certspotter
  `LogState`; Merkle positions, no certificates).
* `ct_sths` — downloaded, unverified signed tree heads needed for safe resume.

Schema is applied idempotently at startup from embedded SQL in `migrations/`.
No certificate contents are ever stored.

## Performance characteristics

Optimised in this order: I/O off the hot path, then serialization, then process
boundaries, then allocations, then cheap suffix/label matching, then bounded
pools, then DNS caching, dedup and batched writes.

The classifier:

* no regexp, JSON, reflection, DNS, HTTP, filesystem, DB or logging;
* no `strings.Split`, no `fmt.Sprintf`;
* lowercases only when an uppercase byte is present;
* parses labels by byte offsets, returning substrings (zero copy);
* fixed stack arrays, no unbounded maps or goroutines.

### Benchmark results (Apple M4, Go 1.27)

```text
BenchmarkClassifyAWS-10            60.03 ns/op    0 B/op   0 allocs/op
BenchmarkClassifyAzure-10          44.93 ns/op    0 B/op   0 allocs/op
BenchmarkRejectNormalDomain-10     30.28 ns/op    0 B/op   0 allocs/op
BenchmarkClassifyMixed-10         369.50 ns/op    0 B/op   0 allocs/op
```

The whole in-process sink path is allocation-free too:

```text
BenchmarkHandleHostNonMatch-10      115.6 ns/op    0 B/op   0 allocs/op
BenchmarkHandleHostDirectMatch-10  155.7 ns/op    0 B/op   0 allocs/op
```

Run them yourself:

```sh
go test ./internal/classify/ ./internal/pipeline/ -run '^$' -bench . -benchmem
```

`BenchmarkClassifyMixed` and `TestHandleHostNoAllocs` fail unless the result is
exactly 0 allocs/op, so the target is enforced by the test suite.

### Bucket discovery

Certificate SANs are custom domains, not bucket names, so matching SANs alone
finds almost no buckets. Instead, every non-matching hostname feeds a discovery
engine that:

1. takes the names that actually occur in the hostname — the subdomain label
   and the registrable-domain label (`assets.example.com` → `assets`,
   `example`). No names are fabricated or drawn from a wordlist;
2. tests anonymous listing —
   `https://<name>.s3.amazonaws.com/?list-type=2` (AWS) and
   `https://<name>.storage.googleapis.com` (GCS);
3. indexes every bucket that returns 200 (keys/sizes/dates only).

It is aggressively bounded: a global `BUCKET_SCAN_QPS` limit, a 12h per-name
dedup cache, and a random-sampled domain queue.

Azure is handled separately because it has two levels: the account name maps to
DNS (`<account>.blob.core.windows.net`), and anonymous access is per container
with no anonymous container enumeration — so for accounts that exist we probe a
small set of common container names. R2 is never listable and is only probed for
objects.

This is what actually yields open buckets. Of the buckets found, S3 and GCS
return full object listings; Azure needs a known container; R2 only serves
objects you already know.

### DNS load balancing

When `DNS_SERVERS` is unset, the pool is ~20 curated, reliable public resolvers
(Cloudflare, Google, Quad9, OpenDNS, AdGuard, …). Each query picks a random
healthy resolver and fails over to at most two others; each attempt is bounded
(2s) and resolvers that fail repeatedly enter a 5-minute cooldown. Definitive
answers (NXDOMAIN) return immediately rather than being retried on every server,
so negative lookups do not multiply DNS traffic.

> For sensitive deployments set `DNS_SERVERS` to your own resolvers, or `DNS_SERVERS=system`.

> These community resolvers are untrusted: they can observe and, in principle,
> forge DNS answers. That only affects enrichment (which endpoints are looked
> up), never the classifier. For sensitive deployments set `DNS_SERVERS` to your
> own resolvers, or `DNS_SERVERS=system`.

## Testing

```sh
go test ./...          # unit + table-driven provider tests
go test -race ./...    # race detector
go vet ./...
```

Integration tests for PostgreSQL run only when `DATABASE_URL` is set:

```sh
DATABASE_URL=postgres://ctcloud:ctcloud@localhost:5432/ctcloud go test ./internal/store/
```

## Limitations

* `net.Resolver.LookupCNAME` returns the final canonical name, so intermediate
  CNAME hops are not observed (the final target is still classified). Switch to
  `miekg/dns` if the full chain is required.
* Optional reachability checks are off by default; when enabled they perform a
  single non-destructive HTTPS `GET /` only — no authentication, downloads,
  mutation or database queries.
* OCI generic `oraclecloud.com` endpoints are recognised but not stored, since
  the bucket name lives in the URL path rather than the hostname.
* Custom R2 (and other) domains are only discovered through DNS/CNAME
  enrichment, and only if the CT hostname itself appears in a certificate.