rewrite README, add MIT license, rename vendored patch doc
- self-contained README (removed references to deleted ARCHITECTURE/GOAL docs) - add LICENSE (MIT) - rename third_party/certspotter/CTCLOUD_PATCH.md -> BUCKETEER_PATCH.md and update the in-code references
This commit is contained in:
StasonJatham committed
1 parent
fac79b6215
commit
47e033c95c
7 files changed
+224
-2088
No files matched your search
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 StasonJatham
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -7,47 +7,202 @@
|
||||
|
||||
# Bucketeer
|
||||
|
||||
Bucketeer continuously consumes Certificate Transparency (CT) logs, finds
|
||||
cloud storage/database/data-service resources referenced by certificates,
|
||||
enriches candidates through DNS/CNAME resolution, discovers and indexes open
|
||||
buckets, and persists only useful findings to PostgreSQL.
|
||||
**Certificate Transparency monitoring for cloud resources and open buckets.**
|
||||
Bucketeer consumes CT logs, classifies every certificate identifier into a
|
||||
provider resource (storage bucket, database, data service), discovers and
|
||||
enumerates open buckets, and stores only useful findings — with a small web UI
|
||||
and a PostgreSQL-only state model. One stateless Go process + PostgreSQL.
|
||||
|
||||
|
||||
|
||||
It is a **single stateless Go process plus PostgreSQL**. CertSpotter's real CT
|
||||
fetching, validation and parsing code is embedded and called **in-process**;
|
||||
there is no stdout handoff, no JSON handoff, no hooks, no subprocesses and no
|
||||
certificate files.
|
||||
---
|
||||
|
||||
> A detected provider endpoint **does not** mean the underlying data is publicly
|
||||
> accessible. `bucketeer` reports facts (hostname existed in CT, resolved, answered
|
||||
> a request) separately from conclusions (`exposure_state`). Most findings are
|
||||
> simply an endpoint that exists.
|
||||
## What it does
|
||||
|
||||
## How it works
|
||||
- Consumes CT logs using **CertSpotter internals in-process** (no stdout, no
|
||||
JSON, no hooks, no subprocesses, no certificate files).
|
||||
- Classifies certificate DNS identifiers with a **zero-allocation classifier**:
|
||||
AWS S3/RDS/Redshift/DocumentDB/Neptune/OpenSearch/MemoryDB/ElastiCache, Azure
|
||||
Blob/File/Queue/Table/SQL/Postgres/MySQL/Redis/Cosmos, Google Cloud Storage,
|
||||
Firebase, Cloudflare R2, DigitalOcean Spaces, Supabase and OCI.
|
||||
- Enriches hostnames through DNS/CNAME with a bounded resolver pool.
|
||||
- **Discovers open buckets** by deriving candidate names from real CT hostnames
|
||||
and testing anonymous listing on S3, GCS and (Azure) account + container.
|
||||
- Verifies readability with unauthenticated, non-destructive checks and
|
||||
**indexes the object keys** of anything anonymously listable (metadata only,
|
||||
never contents).
|
||||
- Persists findings in batches to PostgreSQL; CT resume state lives there too.
|
||||
|
||||
> A detected endpoint does **not** mean the data is publicly accessible.
|
||||
> Bucketeer reports facts (hostname seen in CT, resolved, answered, listed)
|
||||
> separately from conclusions, and never attempts authentication.
|
||||
|
||||
## Data flow
|
||||
|
||||
```text
|
||||
CT logs ──► CertSpotter internals (fetch/verify/parse)
|
||||
│ direct Go call: IdentifierSink.HandleIdentifiers
|
||||
▼
|
||||
fast classifier (0 allocs/op, no regexp/JSON/DNS)
|
||||
┌──────┴───────┐
|
||||
direct match non-match
|
||||
│ ▼
|
||||
│ DNS/CNAME enrichment
|
||||
│ (followed + classified)
|
||||
└──────┬───────┘
|
||||
▼
|
||||
zero-allocation classifier
|
||||
┌──────┴───────────────┐
|
||||
direct match non-match
|
||||
│ │
|
||||
│ DNS/CNAME enrichment (bounded pool)
|
||||
│ │
|
||||
│ bucket-name discovery (S3/GCS/Azure)
|
||||
│ │
|
||||
└──────────┬──────────┘
|
||||
▼
|
||||
dedup + bounded queues
|
||||
▼
|
||||
optional safe reachability
|
||||
▼
|
||||
▼
|
||||
optional reachability + readability probe
|
||||
▼
|
||||
anonymous listing → metadata index
|
||||
▼
|
||||
batched PostgreSQL writer
|
||||
▼
|
||||
PostgreSQL
|
||||
▼
|
||||
PostgreSQL
|
||||
```
|
||||
|
||||
See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for per-stage rules.
|
||||
## Quick start
|
||||
|
||||
```sh
|
||||
cp .env.example .env # set a strong POSTGRES_PASSWORD and matching DATABASE_URL
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
- UI + API: `http://127.0.0.1:9090/`
|
||||
- Health: `/healthz`, readiness: `/readyz`, metrics: `/metrics`
|
||||
|
||||
The `bucketeer` container runs as non-root with a read-only root filesystem, no
|
||||
application volume, dropped Linux capabilities and CPU/memory/pids limits. The
|
||||
only persistent state is PostgreSQL.
|
||||
|
||||
> Use `docker compose stop` / `start` (or `up -d`) to keep findings.
|
||||
> `docker compose down -v` deletes the `pgdata` volume and every discovered
|
||||
> asset with it.
|
||||
|
||||
## Configuration
|
||||
|
||||
Everything is environment variables (no config files). Sensible defaults apply.
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `DATABASE_URL` | *(required)* | PostgreSQL connection string |
|
||||
| `DB_MAX_CONNS` | `8` | pgx pool size |
|
||||
| `DNS_SERVERS` | *(curated)* | comma-separated resolvers, or `system` |
|
||||
| `DNS_WORKERS` | `64` | DNS enrichment workers |
|
||||
| `DNS_TIMEOUT` | `5s` | per-lookup timeout |
|
||||
| `DNS_CACHE_SIZE` | `262144` | max cached DNS results |
|
||||
| `DNS_MAX_CNAME_DEPTH` | `8` | CNAME hops |
|
||||
| `DNS_POSITIVE_TTL` / `DNS_NEGATIVE_TTL` | `10m` / `1m` | DNS cache TTLs |
|
||||
| `DEDUP_SIZE` / `DEDUP_TTL` | `262144` / `1h` | dedup caches |
|
||||
| `DIRECT_QUEUE_SIZE` / `DNS_QUEUE_SIZE` / `DB_QUEUE_SIZE` | `8192` / `262144` / `8192` | bounded queues |
|
||||
| `DB_BATCH_SIZE` / `DB_FLUSH_INTERVAL` | `500` / `2s` | write batching |
|
||||
| `ACTIVE_PROBING` | `false` | enable reachability/readability checks |
|
||||
| `PROBE_WORKERS` / `PROBE_TIMEOUT` / `PROBE_QUEUE_SIZE` | `8` / `5s` / `65536` | probe pool |
|
||||
| `PROBE_DEEP` | `true` | unauthenticated read-only DB protocol checks |
|
||||
| `INDEX_ENABLED` / `INDEX_MAX_OBJECTS` | `true` / `1000` | index readable resources |
|
||||
| `BUCKET_SCAN_ENABLED` / `_QPS` / `_WORKERS` / `_QUEUE` | `true` / `50` / `16` / `65536` | bucket discovery |
|
||||
| `METRICS_ADDR` | `:9090` | HTTP listener |
|
||||
| `UI_USER` / `UI_PASS` | *(unset)* | optional HTTP basic auth for the UI/API |
|
||||
| `LOG_LEVEL` / `LOG_FORMAT` | `info` / `text` | logging (`json` for pipelines) |
|
||||
| `STATS_INTERVAL` | `60s` | aggregated stats interval |
|
||||
| `LOG_LIST_SOURCE` | Google log list v3 | CertSpotter log list |
|
||||
| `START_AT_END` | `false` | skip existing CT backlog on first run |
|
||||
| `SHUTDOWN_TIMEOUT` | `20s` | graceful shutdown budget |
|
||||
| `UI_BIND` (compose) | `127.0.0.1` | host interface for the published port |
|
||||
|
||||
## HTTP API and UI
|
||||
|
||||
The web UI (Assets / Contents views, light and dark mode) and the API share one
|
||||
listener:
|
||||
|
||||
| Call | Purpose |
|
||||
| --- | --- |
|
||||
| `GET /api/stats` | totals and breakdowns by provider/service/exposure |
|
||||
| `GET /api/assets?provider=&service=&exposure=&q=&sort=&limit=&offset=` | asset list |
|
||||
| `GET /api/entries?q=&limit=&offset=` | search indexed object keys |
|
||||
| `POST /api/scan` `{"target":"host-or-URL"}` | classify, CNAME-follow, probe and index one target |
|
||||
| `POST /api/reprobe?exposure=&limit=` | re-probe stored assets and index readable ones |
|
||||
| `GET /metrics` `/healthz` `/readyz` | Prometheus / liveness / readiness |
|
||||
|
||||
The listener has no auth by default; set `UI_USER`/`UI_PASS` and keep it bound to
|
||||
loopback (or a private network).
|
||||
|
||||
## Reading the results
|
||||
|
||||
`exposure_state` distinguishes facts, and two capability flags keep them honest:
|
||||
|
||||
- `public_read` — an anonymous read succeeded. For buckets this is only
|
||||
meaningful together with `listable`.
|
||||
- `listable` — the provider's anonymous list API returned object names; only
|
||||
then does Bucketeer store an index. S3/GCS/DO and Azure "Container" mode.
|
||||
- `object_readable` — a known object was readable but listing is denied (Azure
|
||||
"Blob" mode, CDN distributions, R2).
|
||||
- `reachable` — for databases, the port answered. **No authentication is
|
||||
attempted**, so a database is never reported as readable.
|
||||
|
||||
`exposure_detail` stores the factual reason (status code, region, container,
|
||||
`dns: no public record`, `tcp/5432 open; no authentication attempted`, …).
|
||||
|
||||
### Bucket discovery
|
||||
|
||||
Certificate SANs are custom domains, not bucket names, so matching SANs alone
|
||||
finds almost no buckets. For every non-matching hostname Bucketeer takes the
|
||||
names that actually occur in it (the subdomain label and the registrable-domain
|
||||
label) and tests anonymous listing on S3 (`<name>.s3.amazonaws.com`) and GCS
|
||||
(`<name>.storage.googleapis.com`). Azure has two levels — the account maps to DNS
|
||||
and anonymous access is per container with no anonymous container enumeration —
|
||||
so existing accounts are probed against common container names. Every 200 is
|
||||
indexed. A global `BUCKET_SCAN_QPS` limit and a 12h per-name dedup keep it
|
||||
bounded; contents are never downloaded.
|
||||
|
||||
## Database schema
|
||||
|
||||
- `cloud_assets` — one row per `(provider, service, provider_hostname)` with
|
||||
`hostname`, `resource_identifier`, `region`, `resource_specific`,
|
||||
`public_semantic`, `exposure_state`, `exposure_detail`, `listable`,
|
||||
`object_readable`, `first_seen`, `last_seen`, `probed_at`.
|
||||
- `asset_entries` — the metadata index of readable resources (key/path, size,
|
||||
updated, content-type).
|
||||
- `observations` — sources, CNAME chain and A/AAAA records.
|
||||
- `ct_checkpoints`, `ct_sths` — CertSpotter resume state (no certificate
|
||||
contents).
|
||||
|
||||
Schema is applied idempotently at startup from embedded SQL in `migrations/`.
|
||||
|
||||
## Performance
|
||||
|
||||
Optimized in order: I/O off the hot path, then serialization, then process
|
||||
boundaries, then allocations. The classifier uses no regexp/JSON/reflection/DNS/
|
||||
HTTP/DB, lowercases only when needed, and parses labels by byte offset.
|
||||
|
||||
```text
|
||||
BenchmarkClassifyAWS-10 60 ns/op 0 B/op 0 allocs/op
|
||||
BenchmarkClassifyAzure-10 45 ns/op 0 B/op 0 allocs/op
|
||||
BenchmarkRejectNormalDomain-10 30 ns/op 0 B/op 0 allocs/op
|
||||
BenchmarkHandleHostNonMatch-10 116 ns/op 0 B/op 0 allocs/op
|
||||
```
|
||||
|
||||
Run them locally:
|
||||
|
||||
```sh
|
||||
go test ./internal/classify/ ./internal/pipeline/ -run '^$' -bench . -benchmem
|
||||
```
|
||||
|
||||
`BenchmarkClassifyMixed` and `TestHandleHostNoAllocs` fail unless the hot path
|
||||
is exactly 0 allocs/op, so the target is enforced by the test suite.
|
||||
|
||||
## Testing
|
||||
|
||||
```sh
|
||||
go test ./... # unit + table-driven provider tests
|
||||
go test -race ./...
|
||||
go vet ./...
|
||||
```
|
||||
|
||||
PostgreSQL integration tests run when `DATABASE_URL` is set.
|
||||
|
||||
## Repository layout
|
||||
|
||||
@@ -69,353 +224,23 @@ internal/
|
||||
webui/ embedded web UI
|
||||
migrations/ embedded SQL schema
|
||||
third_party/certspotter/ vendored upstream CT engine (minimal patch)
|
||||
docs/ ARCHITECTURE.md, DESIGN.md, GOAL.md, SKILL.md
|
||||
docs/ DESIGN.md, SKILL.md
|
||||
assets/ brand assets
|
||||
```
|
||||
|
||||
More: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) (data flow),
|
||||
[docs/DESIGN.md](docs/DESIGN.md) (UI design system),
|
||||
[docs/SKILL.md](docs/SKILL.md) (how an agent queries and audits a running instance).
|
||||
|
||||
## Why CertSpotter is embedded (and patched)
|
||||
|
||||
Re-implementing CT log handling is a large, security-sensitive job. Upstream
|
||||
CertSpotter already does log-list handling, STH fetching/verification, Merkle
|
||||
tree maintenance, entry parsing and certificate reconstruction.
|
||||
|
||||
The only change is a two-hunk patch in `third_party/certspotter` that adds an
|
||||
`IdentifierSink` callback. After a certificate is parsed, the identifiers are
|
||||
handed to bucketeer with a direct Go function call and the watchlist/chain/
|
||||
notification machinery is skipped:
|
||||
|
||||
```go
|
||||
if config.IdentifierSink != nil {
|
||||
config.IdentifierSink.HandleIdentifiers(ctx, identifiers)
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
See [`third_party/certspotter/CTCLOUD_PATCH.md`](third_party/certspotter/CTCLOUD_PATCH.md).
|
||||
This deliberately avoids stdout piping, JSON files, hook scripts, subprocesses
|
||||
and temporary certificate files, which are all slow, serializing and
|
||||
filesystem-bound. Upstream licensing/notices are preserved and the patch is
|
||||
isolated so it can be rebased.
|
||||
|
||||
## Supported provider patterns
|
||||
|
||||
**AWS** — S3 (`<bucket>.s3[.dualstack].<region>.amazonaws.com`,
|
||||
`<bucket>.s3-<region>.amazonaws.com` legacy form, `s3-website[-<region>]`),
|
||||
S3 access points (`s3-accesspoint[-fips]`,
|
||||
`dualstack`), multi-region access points (`mrap.accesspoint.s3-global`), S3
|
||||
Express (`s3express-*`), RDS/Aurora/proxy (`*.rds.amazonaws.com`), DocumentDB,
|
||||
Redshift (provisioned + serverless), OpenSearch (`search-*` public / `vpc-*`
|
||||
VPC), Neptune, MemoryDB, ElastiCache. Generic API endpoints
|
||||
(`s3.eu-central-1.amazonaws.com`, `dynamodb.*`, `redshift.<region>`,
|
||||
`rds.<region>`, …) are rejected.
|
||||
|
||||
**Azure** — `blob|dfs|file|queue|table|web.core.windows.net`,
|
||||
`*.database.windows.net`, `*.postgres|mysql.database.azure.com`,
|
||||
`*.documents.azure.com`, `*.{mongo,cassandra,gremlin}.cosmos.azure.com`,
|
||||
`*.redis.cache.windows.net`, `*.redis.azure.net`, `*.search.windows.net`.
|
||||
Bare service endpoints are rejected.
|
||||
|
||||
**Google Cloud / Firebase** — `<bucket>.storage.googleapis.com`,
|
||||
`<db>.firebaseio.com`, `<db>.<region>.firebasedatabase.app`,
|
||||
`<project>.appspot.com`. `storage.googleapis.com` itself is rejected.
|
||||
|
||||
**Cloudflare R2** — `<account>.r2.cloudflarestorage.com`; `<account>.r2.dev`
|
||||
is flagged as an explicitly public URL form.
|
||||
|
||||
**DigitalOcean** — `<bucket>.<region>[.cdn].digitaloceanspaces.com`,
|
||||
`*.db.ondigitalocean.com`. `<region>.digitaloceanspaces.com` is rejected.
|
||||
|
||||
**Supabase** — `<ref>.supabase.co`, `db.<ref>.supabase.co` (PostgreSQL),
|
||||
`aws-<n>-<region>.pooler.supabase.com`.
|
||||
|
||||
**Oracle Cloud (OCI)** — `<namespace>.objectstorage|swiftobjectstorage.<region>.oci.customer-oci.com`
|
||||
(resource-specific), and the generic `objectstorage.<region>.oraclecloud.com`
|
||||
forms (`resource_specific = false`; not stored as findings).
|
||||
|
||||
Provider-specific details live in `internal/classify/`.
|
||||
|
||||
## Configuration
|
||||
|
||||
Environment variables only; no config files, no runtime pattern files.
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `DATABASE_URL` | *(required)* | PostgreSQL connection string |
|
||||
| `DB_MAX_CONNS` | `8` | pgx pool size |
|
||||
| `DNS_SERVERS` | *(unset → curated + community list)* | comma-separated resolvers, or `system` for the OS resolver |
|
||||
| `DNS_WORKERS` | `16` | DNS enrichment workers |
|
||||
| `DNS_TIMEOUT` | `5s` | per-lookup timeout |
|
||||
| `DNS_CACHE_SIZE` | `262144` | max DNS cache entries |
|
||||
| `DEDUP_SIZE` | `262144` | max entries in each dedup cache |
|
||||
| `DNS_MAX_CNAME_DEPTH` | `8` | max CNAME hops |
|
||||
| `DNS_POSITIVE_TTL` / `DNS_NEGATIVE_TTL` | `10m` / `1m` | DNS cache TTLs |
|
||||
| `DIRECT_QUEUE_SIZE` | `8192` | direct-match queue capacity |
|
||||
| `DNS_QUEUE_SIZE` | `65536` | DNS-enrichment queue capacity |
|
||||
| `DB_QUEUE_SIZE` | `8192` | DB-write queue capacity |
|
||||
| `DB_BATCH_SIZE` | `500` | findings per write batch |
|
||||
| `DB_FLUSH_INTERVAL` | `2s` | max time before a batch is flushed |
|
||||
| `ACTIVE_PROBING` | `false` | enable optional reachability checks |
|
||||
| `PROBE_WORKERS` / `PROBE_TIMEOUT` | `8` / `5s` | probe worker pool |
|
||||
| `PROBE_QUEUE_SIZE` | `65536` | probe sample size; when full, old entries are replaced at random (bounded memory) |
|
||||
| `PROBE_DEEP` | `true` | unauthenticated, read-only protocol reads for databases (Redis/Postgres/Mongo) |
|
||||
| `INDEX_ENABLED` / `INDEX_MAX_OBJECTS` | `true` / `1000` | index metadata of readable resources (keys/sizes only) |
|
||||
| `LOG_FORMAT` | `text` | `text` or `json` (JSON is best for log pipelines/LLM analysis) |
|
||||
| `BUCKET_SCAN_ENABLED` | `true` | discover open buckets by name enumeration |
|
||||
| `BUCKET_SCAN_QPS` | `50` | global cap on outbound discovery requests |
|
||||
| `BUCKET_SCAN_WORKERS` | `16` | discovery probe workers |
|
||||
| `BUCKET_SCAN_QUEUE` | `65536` | sampled domain backlog |
|
||||
| `DEDUP_TTL` | `1h` | hostname/finding dedup window |
|
||||
| `METRICS_ADDR` | `:9090` | metrics/health/UI listener |
|
||||
| `UI_USER` / `UI_PASS` | *(unset)* | optional HTTP basic auth for the web UI |
|
||||
| `LOG_LEVEL` | `info` | `debug\|info\|warn\|error` |
|
||||
| `STATS_INTERVAL` | `60s` | aggregated stats log interval |
|
||||
| `LOG_LIST_SOURCE` | Google log list v3 | certspotter log list URL |
|
||||
| `START_AT_END` | `false` | skip existing backlog on first run |
|
||||
| `SHUTDOWN_TIMEOUT` | `20s` | graceful shutdown budget |
|
||||
|
||||
## Running with Docker
|
||||
|
||||
```sh
|
||||
cp .env.example .env # set a strong POSTGRES_PASSWORD and matching DATABASE_URL
|
||||
docker compose up --build
|
||||
```
|
||||
|
||||
That starts `postgres` and the stateless `bucketeer` container. The application
|
||||
container runs as non-root, has a read-only root filesystem, drops all Linux
|
||||
capabilities, sets CPU/memory/pids limits and mounts no application volume. The
|
||||
only persistent state is PostgreSQL (CT checkpoints + cloud assets + index).
|
||||
|
||||
> **Findings accumulate over time.** Use `docker compose stop` / `docker compose
|
||||
> start` (or plain `up -d`) to keep the `pgdata` volume. `docker compose down -v`
|
||||
> deletes the volume and every discovered asset with it.
|
||||
|
||||
Endpoints on port `9090`, bound to `127.0.0.1` only:
|
||||
|
||||
* `GET /` — web UI (see below)
|
||||
* `GET /api/assets`, `GET /api/stats` — JSON for the UI
|
||||
* `GET /metrics`, `GET /healthz`, `GET /readyz`
|
||||
|
||||
The listener has no authentication by default, so scrape it locally or from a
|
||||
sidecar and never expose it publicly. Set `UI_USER`/`UI_PASS` to require HTTP
|
||||
basic auth on the UI and its API.
|
||||
|
||||
## Web UI
|
||||
|
||||
A small, dependency-free UI (vanilla JS, no build step, embedded in the binary)
|
||||
styled to the tokens and layout in [docs/DESIGN.md](docs/DESIGN.md) (neutral canvas, single 768px
|
||||
results column, flat rows, pill controls, no shadows).
|
||||
|
||||
* search box with 250ms debounce, stale-request cancellation and `?q=` URL state
|
||||
* exposure filter chips — use **Public read** to see open buckets, **Reachable**
|
||||
to see publicly reachable database endpoints — plus a sort control
|
||||
* result rows show the provider hostname, probe detail, provider/service/region
|
||||
badges, resource identifier and relative last-seen/probed times
|
||||
* summary line with total assets, publicly readable and reachable counts
|
||||
* refreshes every 15s (paused when the tab is hidden), read-only
|
||||
* light and dark mode: follows the OS by default, with a header toggle that is
|
||||
persisted in `localStorage`; the theme is applied before first paint to avoid
|
||||
a flash
|
||||
|
||||
Served from the same loopback port: `http://127.0.0.1:9090/`.
|
||||
|
||||
### Brand assets
|
||||
|
||||
The UI ships the official Bucketeer mascot for both themes, optimized to
|
||||
lossless WebP and embedded in the binary (`internal/webui/assets/`):
|
||||
|
||||
| Asset | Size | Use |
|
||||
| --- | --- | --- |
|
||||
| `logo-light.webp` | ~27 KB | black line art, light theme |
|
||||
| `logo-dark.webp` | ~22 KB | white line art, dark theme |
|
||||
| `favicon.png` | ~7 KB | browser tab |
|
||||
|
||||
They are downscaled to 256px and encoded with `cwebp -lossless -z 9`, so they
|
||||
stay crisp on HiDPI displays while shrinking the ~1.2 MB source PNGs by ~96%.
|
||||
Regenerate after replacing the source PNGs at the repo root:
|
||||
|
||||
```sh
|
||||
magick assets/bucketeer-light.png -resize 256x -strip /tmp/l.png
|
||||
magick assets/bucketeer-dark.png -resize 256x -strip /tmp/d.png
|
||||
cwebp -quiet -lossless -z 9 -alpha_q 100 /tmp/l.png -o internal/webui/assets/logo-light.webp
|
||||
cwebp -quiet -lossless -z 9 -alpha_q 100 /tmp/d.png -o internal/webui/assets/logo-dark.webp
|
||||
```
|
||||
|
||||
## Reachability and readability checks
|
||||
|
||||
`ACTIVE_PROBING=false` by default. When enabled, the prober issues **one**
|
||||
unauthenticated request per finding. It never sends credentials, downloads
|
||||
objects, mutates anything, runs database queries or attempts authentication.
|
||||
|
||||
| Vendor | Check | `public_read` means |
|
||||
| --- | --- | --- |
|
||||
| AWS S3, Cloudflare R2, DigitalOcean Spaces, OCI | anonymous `ListObjectsV2` (`?list-type=2&max-keys=1`) | bucket policy grants `s3:ListBucket` to `*` |
|
||||
| Google Cloud Storage | JSON API object list (`/storage/v1/b/<bucket>/o?maxResults=1`) | the bucket grants `allUsers` `storage.objects.list` |
|
||||
| Azure Blob/File/Queue/Table | account-level `?comp=list` | anonymous access is **per container**, so account-level 403 is reported `reachable` (not "not readable"). Container names cannot be derived from the account hostname. |
|
||||
| Firebase Realtime DB | `GET /.json?shallow=true` | security rules allow anonymous read |
|
||||
| Supabase | `GET /rest/v1/` | not testable without the public **anon API key**; reports `auth_required` |
|
||||
| OpenSearch / Elasticsearch | `GET /_cat/indices` | security/access control disabled |
|
||||
| Neptune | `GET /status` | a 2xx was returned |
|
||||
| Cosmos / Appspot / Azure Search | root `GET /` | a 2xx was returned |
|
||||
| Redis (ElastiCache, MemoryDB, Azure Redis) | unauthenticated `PING` | `+PONG` without `AUTH` |
|
||||
| PostgreSQL (RDS/Aurora, Redshift, Azure, Supabase) | unauthenticated StartupMessage | `AuthenticationOk` (server trusts the connection) |
|
||||
| MongoDB (DocumentDB, Cosmos Mongo) | OP_MSG `listDatabases` | command succeeds without auth |
|
||||
| MySQL, Azure SQL, Cassandra (Cosmos) | TCP connect only | *never* — reported `reachable` |
|
||||
|
||||
Every check sends **no credentials**: no `Authorization` header, no password, no
|
||||
token. "Anonymous" here means an unauthenticated request, exactly as an anonymous
|
||||
cloud principal would make.
|
||||
|
||||
Deep protocol probes (`PROBE_DEEP`, default on) only ever issue read-only
|
||||
requests or commands; nothing is written, and no authentication is attempted.
|
||||
|
||||
Probe outcomes are stored in `exposure_state` and a short factual
|
||||
`exposure_detail` string (e.g. `s3 redirect region=eu-central-1`,
|
||||
`status=403 authentication required`, `tcp/5432 open; no authentication
|
||||
attempted`). Database endpoints are never claimed to be readable: a raw
|
||||
`public_read` on a database is impossible by design.
|
||||
|
||||
## Production notes
|
||||
|
||||
* **Secrets** — database credentials live in `.env` (gitignored); `.env.example`
|
||||
is the template. Inject the same variables from your orchestrator's secret
|
||||
store in production.
|
||||
* **Database TLS** — the local compose stack uses `sslmode=disable` because the
|
||||
compose bridge network is private. For production, point `DATABASE_URL` at a
|
||||
TLS-enabled database, e.g.
|
||||
`postgres://user:[email protected]:5432/bucketeer?sslmode=verify-full`.
|
||||
pgx handles TLS natively; no extra flags are needed.
|
||||
* **Metrics** — bound to loopback. There is no auth, no debug endpoints and no
|
||||
high-cardinality labels (never hostnames).
|
||||
* **Resource limits** — `mem_limit`/`cpus`/`pids_limit` are set in compose so
|
||||
overload is bounded by the queue sizes rather than by the OOM killer.
|
||||
* **Authorization** — leave `ACTIVE_PROBING=false` unless you are authorised to
|
||||
probe the discovered infrastructure.
|
||||
|
||||
## Database schema
|
||||
|
||||
* `cloud_assets` — one row per `(provider, service, provider_hostname)`, with
|
||||
`hostname` (original CT hostname), `resource_identifier`, `region`,
|
||||
`resource_specific`, `public_semantic`, `exposure_state`, `exposure_detail`,
|
||||
`probed_at`, `first_seen`, `last_seen`.
|
||||
* `observations` — append-only observations linking an asset to a
|
||||
`source_hostname`, `source_type` (`direct_ct`/`cname`), CNAME chain and
|
||||
A/AAAA records.
|
||||
* `ct_checkpoints` — per-log CT resume state (JSON-encoded certspotter
|
||||
`LogState`; Merkle positions, no certificates).
|
||||
* `ct_sths` — downloaded, unverified signed tree heads needed for safe resume.
|
||||
|
||||
Schema is applied idempotently at startup from embedded SQL in `migrations/`.
|
||||
No certificate contents are ever stored.
|
||||
|
||||
## Performance characteristics
|
||||
|
||||
Optimised in this order: I/O off the hot path, then serialization, then process
|
||||
boundaries, then allocations, then cheap suffix/label matching, then bounded
|
||||
pools, then DNS caching, dedup and batched writes.
|
||||
|
||||
The classifier:
|
||||
|
||||
* no regexp, JSON, reflection, DNS, HTTP, filesystem, DB or logging;
|
||||
* no `strings.Split`, no `fmt.Sprintf`;
|
||||
* lowercases only when an uppercase byte is present;
|
||||
* parses labels by byte offsets, returning substrings (zero copy);
|
||||
* fixed stack arrays, no unbounded maps or goroutines.
|
||||
|
||||
### Benchmark results (Apple M4, Go 1.27)
|
||||
|
||||
```text
|
||||
BenchmarkClassifyAWS-10 60.03 ns/op 0 B/op 0 allocs/op
|
||||
BenchmarkClassifyAzure-10 44.93 ns/op 0 B/op 0 allocs/op
|
||||
BenchmarkRejectNormalDomain-10 30.28 ns/op 0 B/op 0 allocs/op
|
||||
BenchmarkClassifyMixed-10 369.50 ns/op 0 B/op 0 allocs/op
|
||||
```
|
||||
|
||||
The whole in-process sink path is allocation-free too:
|
||||
|
||||
```text
|
||||
BenchmarkHandleHostNonMatch-10 115.6 ns/op 0 B/op 0 allocs/op
|
||||
BenchmarkHandleHostDirectMatch-10 155.7 ns/op 0 B/op 0 allocs/op
|
||||
```
|
||||
|
||||
Run them yourself:
|
||||
|
||||
```sh
|
||||
go test ./internal/classify/ ./internal/pipeline/ -run '^$' -bench . -benchmem
|
||||
```
|
||||
|
||||
`BenchmarkClassifyMixed` and `TestHandleHostNoAllocs` fail unless the result is
|
||||
exactly 0 allocs/op, so the target is enforced by the test suite.
|
||||
|
||||
### Bucket discovery
|
||||
|
||||
Certificate SANs are custom domains, not bucket names, so matching SANs alone
|
||||
finds almost no buckets. Instead, every non-matching hostname feeds a discovery
|
||||
engine that:
|
||||
|
||||
1. takes the names that actually occur in the hostname — the subdomain label
|
||||
and the registrable-domain label (`assets.example.com` → `assets`,
|
||||
`example`). No names are fabricated or drawn from a wordlist;
|
||||
2. tests anonymous listing —
|
||||
`https://<name>.s3.amazonaws.com/?list-type=2` (AWS) and
|
||||
`https://<name>.storage.googleapis.com` (GCS);
|
||||
3. indexes every bucket that returns 200 (keys/sizes/dates only).
|
||||
|
||||
It is aggressively bounded: a global `BUCKET_SCAN_QPS` limit, a 12h per-name
|
||||
dedup cache, and a random-sampled domain queue.
|
||||
|
||||
Azure is handled separately because it has two levels: the account name maps to
|
||||
DNS (`<account>.blob.core.windows.net`), and anonymous access is per container
|
||||
with no anonymous container enumeration — so for accounts that exist we probe a
|
||||
small set of common container names. R2 is never listable and is only probed for
|
||||
objects.
|
||||
|
||||
This is what actually yields open buckets. Of the buckets found, S3 and GCS
|
||||
return full object listings; Azure needs a known container; R2 only serves
|
||||
objects you already know.
|
||||
|
||||
### DNS load balancing
|
||||
|
||||
When `DNS_SERVERS` is unset, the pool is ~20 curated, reliable public resolvers
|
||||
(Cloudflare, Google, Quad9, OpenDNS, AdGuard, …). Each query picks a random
|
||||
healthy resolver and fails over to at most two others; each attempt is bounded
|
||||
(2s) and resolvers that fail repeatedly enter a 5-minute cooldown. Definitive
|
||||
answers (NXDOMAIN) return immediately rather than being retried on every server,
|
||||
so negative lookups do not multiply DNS traffic.
|
||||
|
||||
> For sensitive deployments set `DNS_SERVERS` to your own resolvers, or `DNS_SERVERS=system`.
|
||||
|
||||
> These community resolvers are untrusted: they can observe and, in principle,
|
||||
> forge DNS answers. That only affects enrichment (which endpoints are looked
|
||||
> up), never the classifier. For sensitive deployments set `DNS_SERVERS` to your
|
||||
> own resolvers, or `DNS_SERVERS=system`.
|
||||
|
||||
## Testing
|
||||
|
||||
```sh
|
||||
go test ./... # unit + table-driven provider tests
|
||||
go test -race ./... # race detector
|
||||
go vet ./...
|
||||
```
|
||||
|
||||
Integration tests for PostgreSQL run only when `DATABASE_URL` is set:
|
||||
|
||||
```sh
|
||||
DATABASE_URL=postgres://bucketeer:bucketeer@localhost:5432/bucketeer go test ./internal/store/
|
||||
```
|
||||
- [docs/DESIGN.md](docs/DESIGN.md) — the UI design system.
|
||||
- [docs/SKILL.md](docs/SKILL.md) — how an agent queries and audits a running
|
||||
instance and answers questions from the API and PostgreSQL.
|
||||
|
||||
## Limitations
|
||||
|
||||
* `net.Resolver.LookupCNAME` returns the final canonical name, so intermediate
|
||||
CNAME hops are not observed (the final target is still classified). Switch to
|
||||
`miekg/dns` if the full chain is required.
|
||||
* Optional reachability checks are off by default; when enabled they perform a
|
||||
single non-destructive HTTPS `GET /` only — no authentication, downloads,
|
||||
mutation or database queries.
|
||||
* OCI generic `oraclecloud.com` endpoints are recognised but not stored, since
|
||||
the bucket name lives in the URL path rather than the hostname.
|
||||
* Custom R2 (and other) domains are only discovered through DNS/CNAME
|
||||
enrichment, and only if the CT hostname itself appears in a certificate.
|
||||
- CT rarely contains bucket hostnames directly; discovery is bounded and
|
||||
stochastic, not exhaustive. Open buckets accumulate over time.
|
||||
- Azure containers need names (no anonymous enumeration); Cloudflare R2 is never
|
||||
listable.
|
||||
- The database is the only state; there is no filesystem spool by design.
|
||||
|
||||
## License
|
||||
|
||||
[MIT](LICENSE). The vendored CertSpotter fork under `third_party/` remains under
|
||||
its original Mozilla Public License v2.0 (`third_party/certspotter/LICENSE`).
|
||||
@@ -1,92 +0,0 @@
|
||||
# Architecture
|
||||
|
||||
## Data flow
|
||||
|
||||
```text
|
||||
Certificate Transparency Logs
|
||||
│
|
||||
▼
|
||||
CertSpotter internals fetch / verify / parse (third_party)
|
||||
(in-process, no stdout/JSON/files)
|
||||
│
|
||||
▼
|
||||
DNS identifiers
|
||||
│
|
||||
▼
|
||||
FAST CLASSIFIER 0 allocs/op, no regexp/JSON/DNS/DB
|
||||
┌────┴─────┐
|
||||
direct cloud normal
|
||||
resource hostname
|
||||
│ │
|
||||
│ ▼
|
||||
│ DNS/CNAME enrichment bounded workers + TTL cache
|
||||
│ │
|
||||
│ ▼
|
||||
│ classify CNAME targets
|
||||
│ │
|
||||
└─────┬──────┘
|
||||
▼
|
||||
dedup/enrichment TTL caches; DB uniqueness is final
|
||||
│
|
||||
▼
|
||||
optional safe reachability HTTPS GET / only, OFF by default
|
||||
▼
|
||||
batch DB writer bounded queue, pgx batches
|
||||
▼
|
||||
PostgreSQL
|
||||
```
|
||||
|
||||
## Stages and what is allowed
|
||||
|
||||
| Stage | May do | Must not do |
|
||||
| --- | --- | --- |
|
||||
| CertSpotter fetch/verify/parse | network, crypto, in-memory DER | persist certs, write files |
|
||||
| Identifier callback | iterate `DNSNames`, call classifier | block, log per certificate, allocate needlessly |
|
||||
| Classifier | byte/string comparisons, label scan | regexp, JSON, reflection, DNS, HTTP, DB, files, per-id logging |
|
||||
| Direct queue | bound, non-blocking send | unbounded growth |
|
||||
| DNS enrichment | DNS via worker pool, TTL cache, CNAME follow | unbounded goroutines, persistence |
|
||||
| Dedup | memory-bounded TTL caches | unbounded maps |
|
||||
| Probe (optional) | one unauthenticated vendor read endpoint (S3/GCS/Azure/Firebase list) or a bare TCP connect for databases | credentials, object downloads, mutation, DB queries, auth attempts |
|
||||
| DB writer | batched `pgx.Batch` writes | one synchronous row per event |
|
||||
| Checkpoints | JSON CT position in `ct_checkpoints`/`ct_sths` | certificate/issuer contents in DB |
|
||||
|
||||
## Concurrency and backpressure
|
||||
|
||||
* CertSpotter runs one monitor goroutine per CT log; it calls the identifier
|
||||
sink concurrently.
|
||||
* The sink never blocks: it classifies inline and does a non-blocking send to
|
||||
either the direct queue or the DNS queue. Full queues increment
|
||||
`queue_drops_total` / `queue_full_total`.
|
||||
* Direct matches have a dedicated queue and worker pool, so speculative CNAME
|
||||
enrichment can be shed first. DNS workers also skip work when the DB queue is
|
||||
more than half full.
|
||||
* The DB writer batches by size or `DB_FLUSH_INTERVAL` and flushes on
|
||||
shutdown.
|
||||
|
||||
## Failure handling
|
||||
|
||||
* DNS and CT network errors are retried by CertSpotter with bounded exponential
|
||||
backoff; bucketeer counts them (`ct_errors_total`) without logging per event.
|
||||
* PostgreSQL writes retry a few times with backoff; persistent failure drops
|
||||
items from the bounded DB queue rather than growing memory.
|
||||
* The process uses `context` throughout and on `SIGTERM` stops intake, drains
|
||||
the pipeline, flushes the DB batch and exits within `SHUTDOWN_TIMEOUT`.
|
||||
|
||||
## Metrics and health
|
||||
|
||||
`GET /metrics` exposes counters (`ct_certificates_total`,
|
||||
`classifier_direct_matches_total`, `classifier_cname_matches_total`,
|
||||
`dns_*`, `db_*`, `queue_*`) and gauges (`queue_depth{queue="…"}`,
|
||||
`go_goroutines`, `process_system_bytes`). No hostnames or other high-cardinality
|
||||
labels are ever used.
|
||||
|
||||
`GET /healthz` is a liveness check. `GET /readyz` verifies the CT monitor is
|
||||
initialised and PostgreSQL is reachable.
|
||||
|
||||
## Why no files, pipes or subprocesses
|
||||
|
||||
Everything between certificate parsing and classification stays in memory. The
|
||||
CertSpotter patch hands identifiers directly to Go code, removing serialization
|
||||
(JSON), IPC (stdout/pipe), process boundaries (subprocess/hooks) and filesystem
|
||||
I/O (certificate/state files). CT resume state is the only thing persisted, and
|
||||
it lives in PostgreSQL.
|
||||
-1618
File diff suppressed because it is too large.
Load diff
+2
-2
@@ -1,4 +1,4 @@
|
||||
# ctcloud patch to upstream CertSpotter
|
||||
# Bucketeer patch to upstream CertSpotter
|
||||
|
||||
This directory is a copy of upstream CertSpotter
|
||||
(https://github.com/SSLMate/certspotter), pulled in as a local module and
|
||||
@@ -39,7 +39,7 @@ if config.IdentifierSink != nil {
|
||||
|
||||
## Why
|
||||
|
||||
The sink gives ctcloud parsed DNS identifiers via a direct in-process Go call:
|
||||
The sink gives Bucketeer parsed DNS identifiers via a direct in-process Go call:
|
||||
|
||||
```text
|
||||
CT log entry -> CertSpotter parse -> IdentifierSink.HandleIdentifiers -> classifier
|
||||
+3
-3
@@ -17,8 +17,8 @@ import (
|
||||
)
|
||||
|
||||
// IdentifierSink receives the DNS identifiers parsed from every monitored
|
||||
// certificate. This is a CTCLOUD PATCH to upstream certspotter; see
|
||||
// CTCLOUD_PATCH.md. When non-nil it is invoked on the monitoring hot path
|
||||
// certificate. This is a BUCKETEER PATCH to upstream certspotter; see
|
||||
// BUCKETEER_PATCH.md. When non-nil it is invoked on the monitoring hot path
|
||||
// instead of the watchlist/notification machinery.
|
||||
type IdentifierSink interface {
|
||||
HandleIdentifiers(ctx context.Context, identifiers *certspotter.Identifiers)
|
||||
@@ -32,7 +32,7 @@ type Config struct {
|
||||
Verbose bool
|
||||
HealthCheckInterval time.Duration
|
||||
|
||||
// IdentifierSink is a CTCLOUD PATCH (see CTCLOUD_PATCH.md). When set,
|
||||
// IdentifierSink is a BUCKETEER PATCH (see BUCKETEER_PATCH.md). When set,
|
||||
// parsed identifiers are handed directly to it in-process and the normal
|
||||
// watchlist/chain/notification path is skipped.
|
||||
IdentifierSink IdentifierSink
|
||||
|
||||
+2
-2
@@ -101,8 +101,8 @@ func processCertificate(ctx context.Context, config *Config, entry *LogEntry, ce
|
||||
if err != nil {
|
||||
return processMalformedLogEntry(ctx, config, entry, err)
|
||||
}
|
||||
// CTCLOUD PATCH: hand identifiers directly to the in-process sink and skip
|
||||
// the watchlist/chain/notification path. See CTCLOUD_PATCH.md.
|
||||
// BUCKETEER PATCH: hand identifiers directly to the in-process sink and skip
|
||||
// the watchlist/chain/notification path. See BUCKETEER_PATCH.md.
|
||||
if config.IdentifierSink != nil {
|
||||
config.IdentifierSink.HandleIdentifiers(ctx, identifiers)
|
||||
return nil
|
||||
|
||||
Reference in new issue
Block a user