bucketeer/docs/SKILL.md

name, description

name	description
bucketeer-ops	Query and audit a running Bucketeer instance (CT cloud-resource monitor) over its HTTP API and PostgreSQL, and answer user questions about discovered assets, open buckets, indexed objects and databases.

Operating and auditing Bucketeer

Bucketeer is a single Go process + PostgreSQL that consumes CT logs, classifies cloud resources, discovers open buckets, and indexes their contents. This skill is for an agent that needs to answer questions from the data without reading the source.

Access

Assume a local deployment (docker compose):

• API base: http://127.0.0.1:9090 (override with METRICS_ADDR / UI_BIND).
• If UI_USER/UI_PASS are set, send HTTP basic auth on every request.
• PostgreSQL: docker compose exec -T postgres psql -U bucketeer -d bucketeer (or use $DATABASE_URL directly).

Never write to the database or restart the stack; Bucketeer is the only writer. docker compose down -v deletes all findings — never use it here.

Vocabulary (answer carefully)

• exposure_state: unknown | public_dns | reachable | auth_required | public_read | private | not_found.
• public_read = an anonymous read succeeded. That is a fact, not a claim that all data is sensitive or safe.
• listable = the provider's list API returned object names; only then is there an index. object_readable = a known object was readable but listing is denied (common for Azure "Blob" mode and CDN).
• reachable (databases) = the port answered; no authentication was attempted. A database is never reported readable.
• Sources: direct_ct, cname, manual, discovery.

HTTP API (read)

Call	Answers
GET /api/stats	total assets, publicly readable, reachable, indexed items, breakdowns by provider/service/exposure
GET /api/assets?exposure=public_read&listable=true&limit=100	open, enumerable buckets
GET /api/assets?provider=aws&service=s3&q=<term>&sort=exposure&limit=100&offset=0	filtered/search asset list (envelope {assets,total,limit,offset})
GET /api/entries?q=<term>&limit=100&offset=0	search indexed object keys/paths (envelope {entries,total,...})
GET /metrics	Prometheus counters (ct_, dns_, probe_, bucket_candidates_, bucket_open_, db_, queue_)
GET /healthz, GET /readyz	liveness / DB + monitor readiness

HTTP API (actions — do only when the user asks)

• POST /api/scan with JSON {"target":"<host-or-URL>"} — classify, follow CNAME, probe and index a specific target. Azure container comes from the URL path. Returns per-target {provider,service,exposure,listable,entries,sample}.
• POST /api/reprobe?exposure=reachable&limit=10000 — re-probe already-stored assets with current checks and index any that are listable (async, 202).

Database (read-only) and the questions it answers

Schema: cloud_assets (one row per provider/service/hostname), asset_entries (indexed keys), observations (sources, CNAME chain, IPs), ct_checkpoints and ct_sths (CT resume state; no contents).

Open buckets with object counts:

SELECT a.provider, a.service, a.provider_hostname,
       count(e.id) AS objects
FROM cloud_assets a
LEFT JOIN asset_entries e ON e.asset_id = a.id
WHERE a.exposure_state = 'public_read' AND a.listable
GROUP BY 1,2,3 ORDER BY objects DESC;

What is inside a bucket:

SELECT e.kind, e.name, e.size, e.updated
FROM asset_entries e JOIN cloud_assets a ON a.id = e.asset_id
WHERE a.provider_hostname = 'example.s3.amazonaws.com'
ORDER BY e.name LIMIT 200;

Search indexed paths across everything:

SELECT a.provider, a.provider_hostname, e.name, e.size
FROM asset_entries e JOIN cloud_assets a ON a.id = e.asset_id
WHERE e.name ILIKE '%backup%' ORDER BY e.size DESC LIMIT 100;

Databases that answered (reachable != readable):

SELECT provider, service, provider_hostname, exposure_state, exposure_detail
FROM cloud_assets
WHERE service IN ('rds','postgres','mysql','redis','memorydb','elasticache','documentdb','neptune','cosmosdb')
ORDER BY last_seen DESC LIMIT 100;

Object-readable but not listable (e.g. Azure "Blob" mode, CDN):

SELECT provider, provider_hostname, exposure_detail FROM cloud_assets
WHERE object_readable AND NOT listable LIMIT 100;

Answering rules

1. Always filter on listable before claiming a bucket can be enumerated.
2. Never equate reachable with readable; never claim data is "exposed" from a hostname alone.
3. If a counter is zero, say the discovery window found none yet — discovery is continuous and stochastic, not a guarantee. Suggest /api/reprobe (re-check stored assets) or /api/scan (test a specific target).
4. Quote the stored exposure_detail as the factual reason.