bucketeer/docs/SKILL.md

---
name: bucketeer-ops
description: Query and audit a running Bucketeer instance (CT cloud-resource monitor) over its HTTP API and PostgreSQL, and answer user questions about discovered assets, open buckets, indexed objects and databases.
---

# Operating and auditing Bucketeer

Bucketeer is a single Go process + PostgreSQL that consumes CT logs, classifies
cloud resources, discovers open buckets, and indexes their contents. This skill is
for an agent that needs to **answer questions from the data** without reading the
source.

## Access

Assume a local deployment (`docker compose`):

- API base: `http://127.0.0.1:9090` (override with `METRICS_ADDR` / `UI_BIND`).
- If `UI_USER`/`UI_PASS` are set, send HTTP basic auth on every request.
- PostgreSQL: `docker compose exec -T postgres psql -U bucketeer -d bucketeer`
  (or use `$DATABASE_URL` directly).

Never write to the database or restart the stack; Bucketeer is the only writer.
`docker compose down -v` **deletes all findings** — never use it here.

## Vocabulary (answer carefully)

- `exposure_state`: `unknown | public_dns | reachable | auth_required |
  public_read | private | not_found`.
- `public_read` = an anonymous read succeeded. That is a **fact**, not a claim
  that all data is sensitive or safe.
- `listable` = the provider's list API returned object names; only then is there
  an index. `object_readable` = a known object was readable but listing is denied
  (common for Azure "Blob" mode and CDN).
- `reachable` (databases) = the port answered; **no authentication was
  attempted**. A database is never reported readable.
- Sources: `direct_ct`, `cname`, `manual`, `discovery`.

## HTTP API (read)

| Call | Answers |
| --- | --- |
| `GET /api/stats` | total assets, publicly readable, reachable, indexed items, breakdowns by provider/service/exposure |
| `GET /api/assets?exposure=public_read&listable=true&limit=100` | open, enumerable buckets |
| `GET /api/assets?provider=aws&service=s3&q=<term>&sort=exposure&limit=100&offset=0` | filtered/search asset list (envelope `{assets,total,limit,offset}`) |
| `GET /api/entries?q=<term>&limit=100&offset=0` | search indexed object keys/paths (envelope `{entries,total,...}`) |
| `GET /metrics` | Prometheus counters (ct_, dns_, probe_, bucket_candidates_, bucket_open_, db_, queue_) |
| `GET /healthz`, `GET /readyz` | liveness / DB + monitor readiness |

## HTTP API (actions — do only when the user asks)

- `POST /api/scan` with JSON `{"target":"<host-or-URL>"}` — classify, follow
  CNAME, probe and index a specific target. Azure container comes from the URL
  path. Returns per-target `{provider,service,exposure,listable,entries,sample}`.
- `POST /api/reprobe?exposure=reachable&limit=10000` — re-probe already-stored
  assets with current checks and index any that are listable (async, 202).

## Database (read-only) and the questions it answers

Schema: `cloud_assets` (one row per provider/service/hostname), `asset_entries`
(indexed keys), `observations` (sources, CNAME chain, IPs), `ct_checkpoints` and
`ct_sths` (CT resume state; no contents).

Open buckets with object counts:
```sql
SELECT a.provider, a.service, a.provider_hostname,
       count(e.id) AS objects
FROM cloud_assets a
LEFT JOIN asset_entries e ON e.asset_id = a.id
WHERE a.exposure_state = 'public_read' AND a.listable
GROUP BY 1,2,3 ORDER BY objects DESC;
```

What is inside a bucket:
```sql
SELECT e.kind, e.name, e.size, e.updated
FROM asset_entries e JOIN cloud_assets a ON a.id = e.asset_id
WHERE a.provider_hostname = 'example.s3.amazonaws.com'
ORDER BY e.name LIMIT 200;
```

Search indexed paths across everything:
```sql
SELECT a.provider, a.provider_hostname, e.name, e.size
FROM asset_entries e JOIN cloud_assets a ON a.id = e.asset_id
WHERE e.name ILIKE '%backup%' ORDER BY e.size DESC LIMIT 100;
```

Databases that answered (reachable != readable):
```sql
SELECT provider, service, provider_hostname, exposure_state, exposure_detail
FROM cloud_assets
WHERE service IN ('rds','postgres','mysql','redis','memorydb','elasticache','documentdb','neptune','cosmosdb')
ORDER BY last_seen DESC LIMIT 100;
```

Object-readable but not listable (e.g. Azure "Blob" mode, CDN):
```sql
SELECT provider, provider_hostname, exposure_detail FROM cloud_assets
WHERE object_readable AND NOT listable LIMIT 100;
```

## Answering rules

1. Always filter on `listable` before claiming a bucket can be enumerated.
2. Never equate `reachable` with readable; never claim data is "exposed" from a
   hostname alone.
3. If a counter is zero, say the discovery window found none yet — discovery is
   continuous and stochastic, not a guarantee. Suggest `/api/reprobe` (re-check
   stored assets) or `/api/scan` (test a specific target).
4. Quote the stored `exposure_detail` as the factual reason.