bucketeer/README.md

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="assets/bucketeer-dark.png">
    <img src="assets/bucketeer-light.png" alt="Bucketeer" width="220">
  </picture>
</p>

# Bucketeer

**Certificate Transparency monitoring for cloud resources and open buckets.**
Bucketeer consumes CT logs, classifies every certificate identifier into a
provider resource (storage bucket, database, data service), discovers and
enumerates open buckets, and stores only useful findings — with a small web UI
and a PostgreSQL-only state model. One stateless Go process + PostgreSQL.

![Bucketeer UI](assets/bucketeer-landing.jpg)

---

## What it does

- Consumes CT logs using **CertSpotter internals in-process** (no stdout, no
  JSON, no hooks, no subprocesses, no certificate files).
- Classifies certificate DNS identifiers with a **zero-allocation classifier**:
  AWS S3/RDS/Redshift/DocumentDB/Neptune/OpenSearch/MemoryDB/ElastiCache, Azure
  Blob/File/Queue/Table/SQL/Postgres/MySQL/Redis/Cosmos, Google Cloud Storage,
  Firebase, Cloudflare R2, DigitalOcean Spaces, Supabase and OCI.
- Enriches hostnames through DNS/CNAME with a bounded resolver pool.
- **Discovers open buckets** by deriving candidate names from real CT hostnames
  and testing anonymous listing on S3, GCS and (Azure) account + container.
- Verifies readability with unauthenticated, non-destructive checks and
  **indexes the object keys** of anything anonymously listable (metadata only,
  never contents).
- Persists findings in batches to PostgreSQL; CT resume state lives there too.

> A detected endpoint does **not** mean the data is publicly accessible.
> Bucketeer reports facts (hostname seen in CT, resolved, answered, listed)
> separately from conclusions, and never attempts authentication.

## Data flow

```text
CT logs ──► CertSpotter internals (fetch/verify/parse)
              │  direct Go call: IdentifierSink.HandleIdentifiers
              ▼
        zero-allocation classifier
        ┌──────┴───────────────┐
   direct match          non-match
        │                     │
        │              DNS/CNAME enrichment (bounded pool)
        │                     │
        │              bucket-name discovery (S3/GCS/Azure)
        │                     │
        └──────────┬──────────┘
                   ▼
        dedup + bounded queues
                   ▼
        optional reachability + readability probe
                   ▼
        anonymous listing → metadata index
                   ▼
        batched PostgreSQL writer
                   ▼
             PostgreSQL
```

## Quick start

```sh
cp .env.example .env      # set a strong POSTGRES_PASSWORD and matching DATABASE_URL
docker compose up -d
```

- UI + API: `http://127.0.0.1:9090/`
- Health: `/healthz`, readiness: `/readyz`, metrics: `/metrics`

The `bucketeer` container runs as non-root with a read-only root filesystem, no
application volume, dropped Linux capabilities and CPU/memory/pids limits. The
only persistent state is PostgreSQL.

> Use `docker compose stop` / `start` (or `up -d`) to keep findings.
> `docker compose down -v` deletes the `pgdata` volume and every discovered
> asset with it.

## Configuration

Everything is environment variables (no config files). Sensible defaults apply.

| Variable | Default | Purpose |
| --- | --- | --- |
| `DATABASE_URL` | *(required)* | PostgreSQL connection string |
| `DB_MAX_CONNS` | `8` | pgx pool size |
| `DNS_SERVERS` | *(curated)* | comma-separated resolvers, or `system` |
| `DNS_WORKERS` | `64` | DNS enrichment workers |
| `DNS_TIMEOUT` | `5s` | per-lookup timeout |
| `DNS_CACHE_SIZE` | `262144` | max cached DNS results |
| `DNS_MAX_CNAME_DEPTH` | `8` | CNAME hops |
| `DNS_POSITIVE_TTL` / `DNS_NEGATIVE_TTL` | `10m` / `1m` | DNS cache TTLs |
| `DEDUP_SIZE` / `DEDUP_TTL` | `262144` / `1h` | dedup caches |
| `DIRECT_QUEUE_SIZE` / `DNS_QUEUE_SIZE` / `DB_QUEUE_SIZE` | `8192` / `262144` / `8192` | bounded queues |
| `DB_BATCH_SIZE` / `DB_FLUSH_INTERVAL` | `500` / `2s` | write batching |
| `ACTIVE_PROBING` | `false` | enable reachability/readability checks |
| `PROBE_WORKERS` / `PROBE_TIMEOUT` / `PROBE_QUEUE_SIZE` | `8` / `5s` / `65536` | probe pool |
| `PROBE_DEEP` | `true` | unauthenticated read-only DB protocol checks |
| `INDEX_ENABLED` / `INDEX_MAX_OBJECTS` | `true` / `1000` | index readable resources |
| `BUCKET_SCAN_ENABLED` / `_QPS` / `_WORKERS` / `_QUEUE` | `true` / `50` / `16` / `65536` | bucket discovery |
| `METRICS_ADDR` | `:9090` | HTTP listener |
| `UI_USER` / `UI_PASS` | *(unset)* | optional HTTP basic auth for the UI/API |
| `LOG_LEVEL` / `LOG_FORMAT` | `info` / `text` | logging (`json` for pipelines) |
| `STATS_INTERVAL` | `60s` | aggregated stats interval |
| `LOG_LIST_SOURCE` | Google log list v3 | CertSpotter log list |
| `START_AT_END` | `false` | skip existing CT backlog on first run |
| `SHUTDOWN_TIMEOUT` | `20s` | graceful shutdown budget |
| `UI_BIND` (compose) | `127.0.0.1` | host interface for the published port |

## HTTP API and UI

The web UI (Assets / Contents views, light and dark mode) and the API share one
listener:

| Call | Purpose |
| --- | --- |
| `GET /api/stats` | totals and breakdowns by provider/service/exposure |
| `GET /api/assets?provider=&service=&exposure=&q=&sort=&limit=&offset=` | asset list |
| `GET /api/entries?q=&limit=&offset=` | search indexed object keys |
| `POST /api/scan` `{"target":"host-or-URL"}` | classify, CNAME-follow, probe and index one target |
| `POST /api/reprobe?exposure=&limit=` | re-probe stored assets and index readable ones |
| `GET /metrics` `/healthz` `/readyz` | Prometheus / liveness / readiness |

The listener has no auth by default; set `UI_USER`/`UI_PASS` and keep it bound to
loopback (or a private network).

## Reading the results

`exposure_state` distinguishes facts, and two capability flags keep them honest:

- `public_read` — an anonymous read succeeded. For buckets this is only
  meaningful together with `listable`.
- `listable` — the provider's anonymous list API returned object names; only
  then does Bucketeer store an index. S3/GCS/DO and Azure "Container" mode.
- `object_readable` — a known object was readable but listing is denied (Azure
  "Blob" mode, CDN distributions, R2).
- `reachable` — for databases, the port answered. **No authentication is
  attempted**, so a database is never reported as readable.

`exposure_detail` stores the factual reason (status code, region, container,
`dns: no public record`, `tcp/5432 open; no authentication attempted`, …).

### Bucket discovery

Certificate SANs are custom domains, not bucket names, so matching SANs alone
finds almost no buckets. For every non-matching hostname Bucketeer takes the
names that actually occur in it (the subdomain label and the registrable-domain
label) and tests anonymous listing on S3 (`<name>.s3.amazonaws.com`) and GCS
(`<name>.storage.googleapis.com`). Azure has two levels — the account maps to DNS
and anonymous access is per container with no anonymous container enumeration —
so existing accounts are probed against common container names. Every 200 is
indexed. A global `BUCKET_SCAN_QPS` limit and a 12h per-name dedup keep it
bounded; contents are never downloaded.

## Database schema

- `cloud_assets` — one row per `(provider, service, provider_hostname)` with
  `hostname`, `resource_identifier`, `region`, `resource_specific`,
  `public_semantic`, `exposure_state`, `exposure_detail`, `listable`,
  `object_readable`, `first_seen`, `last_seen`, `probed_at`.
- `asset_entries` — the metadata index of readable resources (key/path, size,
  updated, content-type).
- `observations` — sources, CNAME chain and A/AAAA records.
- `ct_checkpoints`, `ct_sths` — CertSpotter resume state (no certificate
  contents).

Schema is applied idempotently at startup from embedded SQL in `migrations/`.

## Performance

Optimized in order: I/O off the hot path, then serialization, then process
boundaries, then allocations. The classifier uses no regexp/JSON/reflection/DNS/
HTTP/DB, lowercases only when needed, and parses labels by byte offset.

```text
BenchmarkClassifyAWS-10            60 ns/op    0 B/op   0 allocs/op
BenchmarkClassifyAzure-10          45 ns/op    0 B/op   0 allocs/op
BenchmarkRejectNormalDomain-10     30 ns/op    0 B/op   0 allocs/op
BenchmarkHandleHostNonMatch-10    116 ns/op    0 B/op   0 allocs/op
```

Run them locally:

```sh
go test ./internal/classify/ ./internal/pipeline/ -run '^$' -bench . -benchmem
```

`BenchmarkClassifyMixed` and `TestHandleHostNoAllocs` fail unless the hot path
is exactly 0 allocs/op, so the target is enforced by the test suite.

## Testing

```sh
go test ./...          # unit + table-driven provider tests
go test -race ./...
go vet ./...
```

PostgreSQL integration tests run when `DATABASE_URL` is set.

## Repository layout

```text
cmd/bucketeer/            application entry point
internal/
  classify/               zero-allocation provider/resource classifier
  config/                 environment configuration
  discover/               bucket-name discovery from CT domains
  dns/                    resolver pool, TTL cache, CNAME following
  index/                  anonymous listing -> metadata index
  metrics/                Prometheus text endpoint
  model/                  shared enums and structs
  pipeline/               bounded queues, dedup, worker pools
  probe/                  reachability + readability checks
  sample/                 bounded random-sample queue
  scan/                   on-demand scan and backfill
  store/                  PostgreSQL schema, batch writer, CT state
  webui/                  embedded web UI
migrations/               embedded SQL schema
third_party/certspotter/  vendored upstream CT engine (minimal patch)
docs/                     DESIGN.md, SKILL.md
assets/                   brand assets
```

- [docs/DESIGN.md](docs/DESIGN.md) — the UI design system.
- [docs/SKILL.md](docs/SKILL.md) — how an agent queries and audits a running
  instance and answers questions from the API and PostgreSQL.

## Limitations

- CT rarely contains bucket hostnames directly; discovery is bounded and
  stochastic, not exhaustive. Open buckets accumulate over time.
- Azure containers need names (no anonymous enumeration); Cloudflare R2 is never
  listable.
- The database is the only state; there is no filesystem spool by design.

## License

[MIT](LICENSE). The vendored CertSpotter fork under `third_party/` remains under
its original Mozilla Public License v2.0 (`third_party/certspotter/LICENSE`).