professionalize repo for public release
- rename ctcloud -> bucketeer everywhere (cmd/bucketeer, binary, compose service) - group brand assets under assets/ (light/dark logo + UI screenshot) - move long-form docs under docs/ (ARCHITECTURE, DESIGN, GOAL) and add docs/SKILL.md: how an agent queries/audits a running instance and answers user questions from the API and PostgreSQL - README: large logo (light/dark picture) and UI screenshot, repository layout - .gitignore: /bucketeer, .env, pgdata, build/test artifacts, OS junk - .dockerignore: keep brand/docs/markdown out of the build context
This commit is contained in:
1 parent
1ce5700cec
commit
fac79b6215
18 files changed
+202
-42
No files matched your search
+110
@@ -0,0 +1,110 @@
|
||||
---
|
||||
name: bucketeer-ops
|
||||
description: Query and audit a running Bucketeer instance (CT cloud-resource monitor) over its HTTP API and PostgreSQL, and answer user questions about discovered assets, open buckets, indexed objects and databases.
|
||||
---
|
||||
|
||||
# Operating and auditing Bucketeer
|
||||
|
||||
Bucketeer is a single Go process + PostgreSQL that consumes CT logs, classifies
|
||||
cloud resources, discovers open buckets, and indexes their contents. This skill is
|
||||
for an agent that needs to **answer questions from the data** without reading the
|
||||
source.
|
||||
|
||||
## Access
|
||||
|
||||
Assume a local deployment (`docker compose`):
|
||||
|
||||
- API base: `http://127.0.0.1:9090` (override with `METRICS_ADDR` / `UI_BIND`).
|
||||
- If `UI_USER`/`UI_PASS` are set, send HTTP basic auth on every request.
|
||||
- PostgreSQL: `docker compose exec -T postgres psql -U bucketeer -d bucketeer`
|
||||
(or use `$DATABASE_URL` directly).
|
||||
|
||||
Never write to the database or restart the stack; Bucketeer is the only writer.
|
||||
`docker compose down -v` **deletes all findings** — never use it here.
|
||||
|
||||
## Vocabulary (answer carefully)
|
||||
|
||||
- `exposure_state`: `unknown | public_dns | reachable | auth_required |
|
||||
public_read | private | not_found`.
|
||||
- `public_read` = an anonymous read succeeded. That is a **fact**, not a claim
|
||||
that all data is sensitive or safe.
|
||||
- `listable` = the provider's list API returned object names; only then is there
|
||||
an index. `object_readable` = a known object was readable but listing is denied
|
||||
(common for Azure "Blob" mode and CDN).
|
||||
- `reachable` (databases) = the port answered; **no authentication was
|
||||
attempted**. A database is never reported readable.
|
||||
- Sources: `direct_ct`, `cname`, `manual`, `discovery`.
|
||||
|
||||
## HTTP API (read)
|
||||
|
||||
| Call | Answers |
|
||||
| --- | --- |
|
||||
| `GET /api/stats` | total assets, publicly readable, reachable, indexed items, breakdowns by provider/service/exposure |
|
||||
| `GET /api/assets?exposure=public_read&listable=true&limit=100` | open, enumerable buckets |
|
||||
| `GET /api/assets?provider=aws&service=s3&q=<term>&sort=exposure&limit=100&offset=0` | filtered/search asset list (envelope `{assets,total,limit,offset}`) |
|
||||
| `GET /api/entries?q=<term>&limit=100&offset=0` | search indexed object keys/paths (envelope `{entries,total,...}`) |
|
||||
| `GET /metrics` | Prometheus counters (ct_, dns_, probe_, bucket_candidates_, bucket_open_, db_, queue_) |
|
||||
| `GET /healthz`, `GET /readyz` | liveness / DB + monitor readiness |
|
||||
|
||||
## HTTP API (actions — do only when the user asks)
|
||||
|
||||
- `POST /api/scan` with JSON `{"target":"<host-or-URL>"}` — classify, follow
|
||||
CNAME, probe and index a specific target. Azure container comes from the URL
|
||||
path. Returns per-target `{provider,service,exposure,listable,entries,sample}`.
|
||||
- `POST /api/reprobe?exposure=reachable&limit=10000` — re-probe already-stored
|
||||
assets with current checks and index any that are listable (async, 202).
|
||||
|
||||
## Database (read-only) and the questions it answers
|
||||
|
||||
Schema: `cloud_assets` (one row per provider/service/hostname), `asset_entries`
|
||||
(indexed keys), `observations` (sources, CNAME chain, IPs), `ct_checkpoints` and
|
||||
`ct_sths` (CT resume state; no contents).
|
||||
|
||||
Open buckets with object counts:
|
||||
```sql
|
||||
SELECT a.provider, a.service, a.provider_hostname,
|
||||
count(e.id) AS objects
|
||||
FROM cloud_assets a
|
||||
LEFT JOIN asset_entries e ON e.asset_id = a.id
|
||||
WHERE a.exposure_state = 'public_read' AND a.listable
|
||||
GROUP BY 1,2,3 ORDER BY objects DESC;
|
||||
```
|
||||
|
||||
What is inside a bucket:
|
||||
```sql
|
||||
SELECT e.kind, e.name, e.size, e.updated
|
||||
FROM asset_entries e JOIN cloud_assets a ON a.id = e.asset_id
|
||||
WHERE a.provider_hostname = 'example.s3.amazonaws.com'
|
||||
ORDER BY e.name LIMIT 200;
|
||||
```
|
||||
|
||||
Search indexed paths across everything:
|
||||
```sql
|
||||
SELECT a.provider, a.provider_hostname, e.name, e.size
|
||||
FROM asset_entries e JOIN cloud_assets a ON a.id = e.asset_id
|
||||
WHERE e.name ILIKE '%backup%' ORDER BY e.size DESC LIMIT 100;
|
||||
```
|
||||
|
||||
Databases that answered (reachable != readable):
|
||||
```sql
|
||||
SELECT provider, service, provider_hostname, exposure_state, exposure_detail
|
||||
FROM cloud_assets
|
||||
WHERE service IN ('rds','postgres','mysql','redis','memorydb','elasticache','documentdb','neptune','cosmosdb')
|
||||
ORDER BY last_seen DESC LIMIT 100;
|
||||
```
|
||||
|
||||
Object-readable but not listable (e.g. Azure "Blob" mode, CDN):
|
||||
```sql
|
||||
SELECT provider, provider_hostname, exposure_detail FROM cloud_assets
|
||||
WHERE object_readable AND NOT listable LIMIT 100;
|
||||
```
|
||||
|
||||
## Answering rules
|
||||
|
||||
1. Always filter on `listable` before claiming a bucket can be enumerated.
|
||||
2. Never equate `reachable` with readable; never claim data is "exposed" from a
|
||||
hostname alone.
|
||||
3. If a counter is zero, say the discovery window found none yet — discovery is
|
||||
continuous and stochastic, not a guarantee. Suggest `/api/reprobe` (re-check
|
||||
stored assets) or `/api/scan` (test a specific target).
|
||||
4. Quote the stored `exposure_detail` as the factual reason.
|
||||
Reference in new issue
Block a user