StasonJatham
fac79b6215
- rename ctcloud -> bucketeer everywhere (cmd/bucketeer, binary, compose service) - group brand assets under assets/ (light/dark logo + UI screenshot) - move long-form docs under docs/ (ARCHITECTURE, DESIGN, GOAL) and add docs/SKILL.md: how an agent queries/audits a running instance and answers user questions from the API and PostgreSQL - README: large logo (light/dark picture) and UI screenshot, repository layout - .gitignore: /bucketeer, .env, pgdata, build/test artifacts, OS junk - .dockerignore: keep brand/docs/markdown out of the build context
111 lines
4.7 KiB
Markdown
111 lines
4.7 KiB
Markdown
---
|
|
name: bucketeer-ops
|
|
description: Query and audit a running Bucketeer instance (CT cloud-resource monitor) over its HTTP API and PostgreSQL, and answer user questions about discovered assets, open buckets, indexed objects and databases.
|
|
---
|
|
|
|
# Operating and auditing Bucketeer
|
|
|
|
Bucketeer is a single Go process + PostgreSQL that consumes CT logs, classifies
|
|
cloud resources, discovers open buckets, and indexes their contents. This skill is
|
|
for an agent that needs to **answer questions from the data** without reading the
|
|
source.
|
|
|
|
## Access
|
|
|
|
Assume a local deployment (`docker compose`):
|
|
|
|
- API base: `http://127.0.0.1:9090` (override with `METRICS_ADDR` / `UI_BIND`).
|
|
- If `UI_USER`/`UI_PASS` are set, send HTTP basic auth on every request.
|
|
- PostgreSQL: `docker compose exec -T postgres psql -U bucketeer -d bucketeer`
|
|
(or use `$DATABASE_URL` directly).
|
|
|
|
Never write to the database or restart the stack; Bucketeer is the only writer.
|
|
`docker compose down -v` **deletes all findings** — never use it here.
|
|
|
|
## Vocabulary (answer carefully)
|
|
|
|
- `exposure_state`: `unknown | public_dns | reachable | auth_required |
|
|
public_read | private | not_found`.
|
|
- `public_read` = an anonymous read succeeded. That is a **fact**, not a claim
|
|
that all data is sensitive or safe.
|
|
- `listable` = the provider's list API returned object names; only then is there
|
|
an index. `object_readable` = a known object was readable but listing is denied
|
|
(common for Azure "Blob" mode and CDN).
|
|
- `reachable` (databases) = the port answered; **no authentication was
|
|
attempted**. A database is never reported readable.
|
|
- Sources: `direct_ct`, `cname`, `manual`, `discovery`.
|
|
|
|
## HTTP API (read)
|
|
|
|
| Call | Answers |
|
|
| --- | --- |
|
|
| `GET /api/stats` | total assets, publicly readable, reachable, indexed items, breakdowns by provider/service/exposure |
|
|
| `GET /api/assets?exposure=public_read&listable=true&limit=100` | open, enumerable buckets |
|
|
| `GET /api/assets?provider=aws&service=s3&q=<term>&sort=exposure&limit=100&offset=0` | filtered/search asset list (envelope `{assets,total,limit,offset}`) |
|
|
| `GET /api/entries?q=<term>&limit=100&offset=0` | search indexed object keys/paths (envelope `{entries,total,...}`) |
|
|
| `GET /metrics` | Prometheus counters (ct_, dns_, probe_, bucket_candidates_, bucket_open_, db_, queue_) |
|
|
| `GET /healthz`, `GET /readyz` | liveness / DB + monitor readiness |
|
|
|
|
## HTTP API (actions — do only when the user asks)
|
|
|
|
- `POST /api/scan` with JSON `{"target":"<host-or-URL>"}` — classify, follow
|
|
CNAME, probe and index a specific target. Azure container comes from the URL
|
|
path. Returns per-target `{provider,service,exposure,listable,entries,sample}`.
|
|
- `POST /api/reprobe?exposure=reachable&limit=10000` — re-probe already-stored
|
|
assets with current checks and index any that are listable (async, 202).
|
|
|
|
## Database (read-only) and the questions it answers
|
|
|
|
Schema: `cloud_assets` (one row per provider/service/hostname), `asset_entries`
|
|
(indexed keys), `observations` (sources, CNAME chain, IPs), `ct_checkpoints` and
|
|
`ct_sths` (CT resume state; no contents).
|
|
|
|
Open buckets with object counts:
|
|
```sql
|
|
SELECT a.provider, a.service, a.provider_hostname,
|
|
count(e.id) AS objects
|
|
FROM cloud_assets a
|
|
LEFT JOIN asset_entries e ON e.asset_id = a.id
|
|
WHERE a.exposure_state = 'public_read' AND a.listable
|
|
GROUP BY 1,2,3 ORDER BY objects DESC;
|
|
```
|
|
|
|
What is inside a bucket:
|
|
```sql
|
|
SELECT e.kind, e.name, e.size, e.updated
|
|
FROM asset_entries e JOIN cloud_assets a ON a.id = e.asset_id
|
|
WHERE a.provider_hostname = 'example.s3.amazonaws.com'
|
|
ORDER BY e.name LIMIT 200;
|
|
```
|
|
|
|
Search indexed paths across everything:
|
|
```sql
|
|
SELECT a.provider, a.provider_hostname, e.name, e.size
|
|
FROM asset_entries e JOIN cloud_assets a ON a.id = e.asset_id
|
|
WHERE e.name ILIKE '%backup%' ORDER BY e.size DESC LIMIT 100;
|
|
```
|
|
|
|
Databases that answered (reachable != readable):
|
|
```sql
|
|
SELECT provider, service, provider_hostname, exposure_state, exposure_detail
|
|
FROM cloud_assets
|
|
WHERE service IN ('rds','postgres','mysql','redis','memorydb','elasticache','documentdb','neptune','cosmosdb')
|
|
ORDER BY last_seen DESC LIMIT 100;
|
|
```
|
|
|
|
Object-readable but not listable (e.g. Azure "Blob" mode, CDN):
|
|
```sql
|
|
SELECT provider, provider_hostname, exposure_detail FROM cloud_assets
|
|
WHERE object_readable AND NOT listable LIMIT 100;
|
|
```
|
|
|
|
## Answering rules
|
|
|
|
1. Always filter on `listable` before claiming a bucket can be enumerated.
|
|
2. Never equate `reachable` with readable; never claim data is "exposed" from a
|
|
hostname alone.
|
|
3. If a counter is zero, say the discovery window found none yet — discovery is
|
|
continuous and stochastic, not a guarantee. Suggest `/api/reprobe` (re-check
|
|
stored assets) or `/api/scan` (test a specific target).
|
|
4. Quote the stored `exposure_detail` as the factual reason.
|