bucketeer/README.md

# ctcloud

`ctcloud` continuously consumes Certificate Transparency (CT) logs, finds
cloud storage/database/data-service resources referenced by certificates,
enriches candidates through DNS/CNAME resolution, and persists only useful
findings to PostgreSQL.

It is a **single stateless Go process plus PostgreSQL**. CertSpotter's real CT
fetching, validation and parsing code is embedded and called **in-process**;
there is no stdout handoff, no JSON handoff, no hooks, no subprocesses and no
certificate files.

> A detected provider endpoint **does not** mean the underlying data is publicly
> accessible. `ctcloud` reports facts (hostname existed in CT, resolved, answered
> a request) separately from conclusions (`exposure_state`). Most findings are
> simply an endpoint that exists.

## How it works

```text
CT logs ──► CertSpotter internals (fetch/verify/parse)
              │  direct Go call: IdentifierSink.HandleIdentifiers
              ▼
        fast classifier  (0 allocs/op, no regexp/JSON/DNS)
        ┌──────┴───────┐
   direct match    non-match
        │              ▼
        │        DNS/CNAME enrichment
        │           (followed + classified)
        └──────┬───────┘
               ▼
        dedup + bounded queues
               ▼
     optional safe reachability
               ▼
        batched PostgreSQL writer
               ▼
           PostgreSQL
```

See [ARCHITECTURE.md](ARCHITECTURE.md) for per-stage rules.

## Why CertSpotter is embedded (and patched)

Re-implementing CT log handling is a large, security-sensitive job. Upstream
CertSpotter already does log-list handling, STH fetching/verification, Merkle
tree maintenance, entry parsing and certificate reconstruction.

The only change is a two-hunk patch in `third_party/certspotter` that adds an
`IdentifierSink` callback. After a certificate is parsed, the identifiers are
handed to ctcloud with a direct Go function call and the watchlist/chain/
notification machinery is skipped:

```go
if config.IdentifierSink != nil {
    config.IdentifierSink.HandleIdentifiers(ctx, identifiers)
    return nil
}
```

See [`third_party/certspotter/CTCLOUD_PATCH.md`](third_party/certspotter/CTCLOUD_PATCH.md).
This deliberately avoids stdout piping, JSON files, hook scripts, subprocesses
and temporary certificate files, which are all slow, serializing and
filesystem-bound. Upstream licensing/notices are preserved and the patch is
isolated so it can be rebased.

## Supported provider patterns

**AWS** — S3 (`<bucket>.s3[.dualstack].<region>.amazonaws.com`,
`<bucket>.s3-<region>.amazonaws.com` legacy form, `s3-website[-<region>]`),
S3 access points (`s3-accesspoint[-fips]`,
`dualstack`), multi-region access points (`mrap.accesspoint.s3-global`), S3
Express (`s3express-*`), RDS/Aurora/proxy (`*.rds.amazonaws.com`), DocumentDB,
Redshift (provisioned + serverless), OpenSearch (`search-*` public / `vpc-*`
VPC), Neptune, MemoryDB, ElastiCache. Generic API endpoints
(`s3.eu-central-1.amazonaws.com`, `dynamodb.*`, `redshift.<region>`,
`rds.<region>`, …) are rejected.

**Azure** — `blob|dfs|file|queue|table|web.core.windows.net`,
`*.database.windows.net`, `*.postgres|mysql.database.azure.com`,
`*.documents.azure.com`, `*.{mongo,cassandra,gremlin}.cosmos.azure.com`,
`*.redis.cache.windows.net`, `*.redis.azure.net`, `*.search.windows.net`.
Bare service endpoints are rejected.

**Google Cloud / Firebase** — `<bucket>.storage.googleapis.com`,
`<db>.firebaseio.com`, `<db>.<region>.firebasedatabase.app`,
`<project>.appspot.com`. `storage.googleapis.com` itself is rejected.

**Cloudflare R2** — `<account>.r2.cloudflarestorage.com`; `<account>.r2.dev`
is flagged as an explicitly public URL form.

**DigitalOcean** — `<bucket>.<region>[.cdn].digitaloceanspaces.com`,
`*.db.ondigitalocean.com`. `<region>.digitaloceanspaces.com` is rejected.

**Supabase** — `<ref>.supabase.co`, `db.<ref>.supabase.co` (PostgreSQL),
`aws-<n>-<region>.pooler.supabase.com`.

**Oracle Cloud (OCI)** — `<namespace>.objectstorage|swiftobjectstorage.<region>.oci.customer-oci.com`
(resource-specific), and the generic `objectstorage.<region>.oraclecloud.com`
forms (`resource_specific = false`; not stored as findings).

Provider-specific details live in `internal/classify/`.

## Configuration

Environment variables only; no config files, no runtime pattern files.

| Variable | Default | Purpose |
| --- | --- | --- |
| `DATABASE_URL` | *(required)* | PostgreSQL connection string |
| `DB_MAX_CONNS` | `8` | pgx pool size |
| `DNS_SERVERS` | *(unset → curated + community list)* | comma-separated resolvers, or `system` for the OS resolver |
| `DNS_WORKERS` | `16` | DNS enrichment workers |
| `DNS_TIMEOUT` | `5s` | per-lookup timeout |
| `DNS_CACHE_SIZE` | `262144` | max DNS cache entries |
| `DEDUP_SIZE` | `262144` | max entries in each dedup cache |
| `DNS_MAX_CNAME_DEPTH` | `8` | max CNAME hops |
| `DNS_POSITIVE_TTL` / `DNS_NEGATIVE_TTL` | `10m` / `1m` | DNS cache TTLs |
| `DIRECT_QUEUE_SIZE` | `8192` | direct-match queue capacity |
| `DNS_QUEUE_SIZE` | `65536` | DNS-enrichment queue capacity |
| `DB_QUEUE_SIZE` | `8192` | DB-write queue capacity |
| `DB_BATCH_SIZE` | `500` | findings per write batch |
| `DB_FLUSH_INTERVAL` | `2s` | max time before a batch is flushed |
| `ACTIVE_PROBING` | `false` | enable optional reachability checks |
| `PROBE_WORKERS` / `PROBE_TIMEOUT` | `8` / `5s` | probe worker pool |
| `PROBE_QUEUE_SIZE` | `65536` | probe sample size; when full, old entries are replaced at random (bounded memory) |
| `PROBE_DEEP` | `true` | unauthenticated, read-only protocol reads for databases (Redis/Postgres/Mongo) |
| `INDEX_ENABLED` / `INDEX_MAX_OBJECTS` | `true` / `1000` | index metadata of readable resources (keys/sizes only) |
| `LOG_FORMAT` | `text` | `text` or `json` (JSON is best for log pipelines/LLM analysis) |
| `BUCKET_SCAN_ENABLED` | `true` | discover open buckets by name enumeration |
| `BUCKET_SCAN_QPS` | `50` | global cap on outbound discovery requests |
| `BUCKET_SCAN_WORKERS` | `16` | discovery probe workers |
| `BUCKET_SCAN_QUEUE` | `65536` | sampled domain backlog |
| `BUCKET_SCAN_DO` | `false` | also test DigitalOcean Spaces regions |
| `DEDUP_TTL` | `1h` | hostname/finding dedup window |
| `METRICS_ADDR` | `:9090` | metrics/health/UI listener |
| `UI_USER` / `UI_PASS` | *(unset)* | optional HTTP basic auth for the web UI |
| `LOG_LEVEL` | `info` | `debug\|info\|warn\|error` |
| `STATS_INTERVAL` | `60s` | aggregated stats log interval |
| `LOG_LIST_SOURCE` | Google log list v3 | certspotter log list URL |
| `START_AT_END` | `false` | skip existing backlog on first run |
| `SHUTDOWN_TIMEOUT` | `20s` | graceful shutdown budget |

## Running with Docker

```sh
cp .env.example .env      # set a strong POSTGRES_PASSWORD and matching DATABASE_URL
docker compose up --build
```

That starts `postgres` and the stateless `ctcloud` container. The application
container runs as non-root, has a read-only root filesystem, drops all Linux
capabilities, sets CPU/memory/pids limits and mounts no application volume. The
only persistent state is PostgreSQL (CT checkpoints + cloud assets).

Endpoints on port `9090`, bound to `127.0.0.1` only:

* `GET /` — web UI (see below)
* `GET /api/assets`, `GET /api/stats` — JSON for the UI
* `GET /metrics`, `GET /healthz`, `GET /readyz`

The listener has no authentication by default, so scrape it locally or from a
sidecar and never expose it publicly. Set `UI_USER`/`UI_PASS` to require HTTP
basic auth on the UI and its API.

## Web UI

A small, dependency-free UI (vanilla JS, no build step, embedded in the binary)
styled to the tokens and layout in `DESIGN.md` (neutral canvas, single 768px
results column, flat rows, pill controls, no shadows).

* search box with 250ms debounce, stale-request cancellation and `?q=` URL state
* exposure filter chips — use **Public read** to see open buckets, **Reachable**
  to see publicly reachable database endpoints — plus a sort control
* result rows show the provider hostname, probe detail, provider/service/region
  badges, resource identifier and relative last-seen/probed times
* summary line with total assets, publicly readable and reachable counts
* refreshes every 15s (paused when the tab is hidden), read-only
* light and dark mode: follows the OS by default, with a header toggle that is
  persisted in `localStorage`; the theme is applied before first paint to avoid
  a flash

Served from the same loopback port: `http://127.0.0.1:9090/`.

### Brand assets

The UI ships the official Bucketeer mascot for both themes, optimized to
lossless WebP and embedded in the binary (`internal/webui/assets/`):

| Asset | Size | Use |
| --- | --- | --- |
| `logo-light.webp` | ~27 KB | black line art, light theme |
| `logo-dark.webp` | ~22 KB | white line art, dark theme |
| `favicon.png` | ~7 KB | browser tab |

They are downscaled to 256px and encoded with `cwebp -lossless -z 9`, so they
stay crisp on HiDPI displays while shrinking the ~1.2 MB source PNGs by ~96%.
Regenerate after replacing the source PNGs at the repo root:

```sh
magick bucketeer-light.png -resize 256x -strip /tmp/l.png
magick bucketeer-dark.png  -resize 256x -strip /tmp/d.png
cwebp -quiet -lossless -z 9 -alpha_q 100 /tmp/l.png -o internal/webui/assets/logo-light.webp
cwebp -quiet -lossless -z 9 -alpha_q 100 /tmp/d.png -o internal/webui/assets/logo-dark.webp
```

## Reachability and readability checks

`ACTIVE_PROBING=false` by default. When enabled, the prober issues **one**
unauthenticated request per finding. It never sends credentials, downloads
objects, mutates anything, runs database queries or attempts authentication.

| Vendor | Check | `public_read` means |
| --- | --- | --- |
| AWS S3, Cloudflare R2, DigitalOcean Spaces, OCI | anonymous `ListObjectsV2` (`?list-type=2&max-keys=1`) | bucket policy grants `s3:ListBucket` to `*` |
| Google Cloud Storage | JSON API object list (`/storage/v1/b/<bucket>/o?maxResults=1`) | the bucket grants `allUsers` `storage.objects.list` |
| Azure Blob/File/Queue/Table | account-level `?comp=list` | anonymous access is **per container**, so account-level 403 is reported `reachable` (not "not readable"). Container names cannot be derived from the account hostname. |
| Firebase Realtime DB | `GET /.json?shallow=true` | security rules allow anonymous read |
| Supabase | `GET /rest/v1/` | not testable without the public **anon API key**; reports `auth_required` |
| OpenSearch / Elasticsearch | `GET /_cat/indices` | security/access control disabled |
| Neptune | `GET /status` | a 2xx was returned |
| Cosmos / Appspot / Azure Search | root `GET /` | a 2xx was returned |
| Redis (ElastiCache, MemoryDB, Azure Redis) | unauthenticated `PING` | `+PONG` without `AUTH` |
| PostgreSQL (RDS/Aurora, Redshift, Azure, Supabase) | unauthenticated StartupMessage | `AuthenticationOk` (server trusts the connection) |
| MongoDB (DocumentDB, Cosmos Mongo) | OP_MSG `listDatabases` | command succeeds without auth |
| MySQL, Azure SQL, Cassandra (Cosmos) | TCP connect only | *never* — reported `reachable` |

Every check sends **no credentials**: no `Authorization` header, no password, no
token. "Anonymous" here means an unauthenticated request, exactly as an anonymous
cloud principal would make.

Deep protocol probes (`PROBE_DEEP`, default on) only ever issue read-only
requests or commands; nothing is written, and no authentication is attempted.

Probe outcomes are stored in `exposure_state` and a short factual
`exposure_detail` string (e.g. `s3 redirect region=eu-central-1`,
`status=403 authentication required`, `tcp/5432 open; no authentication
attempted`). Database endpoints are never claimed to be readable: a raw
`public_read` on a database is impossible by design.

## Production notes

* **Secrets** — database credentials live in `.env` (gitignored); `.env.example`
  is the template. Inject the same variables from your orchestrator's secret
  store in production.
* **Database TLS** — the local compose stack uses `sslmode=disable` because the
  compose bridge network is private. For production, point `DATABASE_URL` at a
  TLS-enabled database, e.g.
  `postgres://user:[email protected]:5432/ctcloud?sslmode=verify-full`.
  pgx handles TLS natively; no extra flags are needed.
* **Metrics** — bound to loopback. There is no auth, no debug endpoints and no
  high-cardinality labels (never hostnames).
* **Resource limits** — `mem_limit`/`cpus`/`pids_limit` are set in compose so
  overload is bounded by the queue sizes rather than by the OOM killer.
* **Authorization** — leave `ACTIVE_PROBING=false` unless you are authorised to
  probe the discovered infrastructure.

## Database schema

* `cloud_assets` — one row per `(provider, service, provider_hostname)`, with
  `hostname` (original CT hostname), `resource_identifier`, `region`,
  `resource_specific`, `public_semantic`, `exposure_state`, `exposure_detail`,
  `probed_at`, `first_seen`, `last_seen`.
* `observations` — append-only observations linking an asset to a
  `source_hostname`, `source_type` (`direct_ct`/`cname`), CNAME chain and
  A/AAAA records.
* `ct_checkpoints` — per-log CT resume state (JSON-encoded certspotter
  `LogState`; Merkle positions, no certificates).
* `ct_sths` — downloaded, unverified signed tree heads needed for safe resume.

Schema is applied idempotently at startup from embedded SQL in `migrations/`.
No certificate contents are ever stored.

## Performance characteristics

Optimised in this order: I/O off the hot path, then serialization, then process
boundaries, then allocations, then cheap suffix/label matching, then bounded
pools, then DNS caching, dedup and batched writes.

The classifier:

* no regexp, JSON, reflection, DNS, HTTP, filesystem, DB or logging;
* no `strings.Split`, no `fmt.Sprintf`;
* lowercases only when an uppercase byte is present;
* parses labels by byte offsets, returning substrings (zero copy);
* fixed stack arrays, no unbounded maps or goroutines.

### Benchmark results (Apple M4, Go 1.27)

```text
BenchmarkClassifyAWS-10            60.03 ns/op    0 B/op   0 allocs/op
BenchmarkClassifyAzure-10          44.93 ns/op    0 B/op   0 allocs/op
BenchmarkRejectNormalDomain-10     30.28 ns/op    0 B/op   0 allocs/op
BenchmarkClassifyMixed-10         369.50 ns/op    0 B/op   0 allocs/op
```

The whole in-process sink path is allocation-free too:

```text
BenchmarkHandleHostNonMatch-10      115.6 ns/op    0 B/op   0 allocs/op
BenchmarkHandleHostDirectMatch-10  155.7 ns/op    0 B/op   0 allocs/op
```

Run them yourself:

```sh
go test ./internal/classify/ ./internal/pipeline/ -run '^$' -bench . -benchmem
```

`BenchmarkClassifyMixed` and `TestHandleHostNoAllocs` fail unless the result is
exactly 0 allocs/op, so the target is enforced by the test suite.

### Bucket discovery

Certificate SANs are custom domains, not bucket names, so matching SANs alone
finds almost no buckets. Instead, every non-matching hostname feeds a discovery
engine that:

1. derives candidate names from the domain (`example` → `example`,
   `example-assets`, `example-backups`, …, plus the subdomain label), also
   testing a built-in wordlist of common names;
2. tests anonymous listing per provider —
   `https://<name>.s3.amazonaws.com/?list-type=2` (AWS),
   `https://<name>.storage.googleapis.com` (GCS),
   `https://<name>.<region>.digitaloceanspaces.com` (DO, optional);
3. indexes every bucket that returns 200 (keys/sizes/dates only).

It is aggressively bounded: a global `BUCKET_SCAN_QPS` limit, a 12h per-name
dedup cache, and a random-sampled domain queue. Azure cannot be enumerated this
way (account names are the namespace and containers need names), so Azure is
handled from CT accounts plus a container wordlist, and R2 is never listable.

This is what actually yields open buckets. Of the buckets found, S3 and GCS
return full object listings; Azure needs a known container; R2 only serves
objects you already know.

### DNS load balancing

When `DNS_SERVERS` is unset, the pool is ~20 curated, reliable public resolvers
(Cloudflare, Google, Quad9, OpenDNS, AdGuard, …). Each query picks a random
healthy resolver and fails over to at most two others; each attempt is bounded
(2s) and resolvers that fail repeatedly enter a 5-minute cooldown. Definitive
answers (NXDOMAIN) return immediately rather than being retried on every server,
so negative lookups do not multiply DNS traffic.

Set `DNS_SERVERS=fresh` to add a compiled-in snapshot of
[proabiral/Fresh-Resolvers](https://github.com/proabiral/Fresh-Resolvers)
(6,760 entries) on top of the curated set. It validates a random sample of 256 at
startup and cools down the rest on failure. Refresh the snapshot with:

```sh
curl -sL https://raw.githubusercontent.com/proabiral/Fresh-Resolvers/master/resolvers.txt \
  -o internal/dns/resolvers.txt
```

> Probing or using thousands of community resolvers at once can saturate the
> container's own DNS and break unrelated lookups, which is why `fresh` is
> opt-in and sampled. These resolvers are also untrusted: they can observe and,
> in principle, forge DNS answers. That only affects enrichment (which endpoints
> are looked up), never the classifier. For sensitive deployments set
> `DNS_SERVERS` to your own resolvers, or `DNS_SERVERS=system`.

> These community resolvers are untrusted: they can observe and, in principle,
> forge DNS answers. That only affects enrichment (which endpoints are looked
> up), never the classifier. For sensitive deployments set `DNS_SERVERS` to your
> own resolvers, or `DNS_SERVERS=system`.

## Testing

```sh
go test ./...          # unit + table-driven provider tests
go test -race ./...    # race detector
go vet ./...
```

Integration tests for PostgreSQL run only when `DATABASE_URL` is set:

```sh
DATABASE_URL=postgres://ctcloud:ctcloud@localhost:5432/ctcloud go test ./internal/store/
```

## Limitations

* `net.Resolver.LookupCNAME` returns the final canonical name, so intermediate
  CNAME hops are not observed (the final target is still classified). Switch to
  `miekg/dns` if the full chain is required.
* Optional reachability checks are off by default; when enabled they perform a
  single non-destructive HTTPS `GET /` only — no authentication, downloads,
  mutation or database queries.
* OCI generic `oraclecloud.com` endpoints are recognised but not stored, since
  the bucket name lives in the URL path rather than the hostname.
* Custom R2 (and other) domains are only discovered through DNS/CNAME
  enrichment, and only if the CT hostname itself appears in a certificate.