professionalize repo for public release
- rename ctcloud -> bucketeer everywhere (cmd/bucketeer, binary, compose service) - group brand assets under assets/ (light/dark logo + UI screenshot) - move long-form docs under docs/ (ARCHITECTURE, DESIGN, GOAL) and add docs/SKILL.md: how an agent queries/audits a running instance and answers user questions from the API and PostgreSQL - README: large logo (light/dark picture) and UI screenshot, repository layout - .gitignore: /bucketeer, .env, pgdata, build/test artifacts, OS junk - .dockerignore: keep brand/docs/markdown out of the build context
This commit is contained in:
StasonJatham committed
1 parent
1ce5700cec
commit
fac79b6215
18 files changed
+202
-42
No files matched your search
@@ -1,9 +1,18 @@
|
||||
# ctcloud
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="assets/bucketeer-dark.png">
|
||||
<img src="assets/bucketeer-light.png" alt="Bucketeer" width="220">
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
`ctcloud` continuously consumes Certificate Transparency (CT) logs, finds
|
||||
# Bucketeer
|
||||
|
||||
Bucketeer continuously consumes Certificate Transparency (CT) logs, finds
|
||||
cloud storage/database/data-service resources referenced by certificates,
|
||||
enriches candidates through DNS/CNAME resolution, and persists only useful
|
||||
findings to PostgreSQL.
|
||||
enriches candidates through DNS/CNAME resolution, discovers and indexes open
|
||||
buckets, and persists only useful findings to PostgreSQL.
|
||||
|
||||
|
||||
|
||||
It is a **single stateless Go process plus PostgreSQL**. CertSpotter's real CT
|
||||
fetching, validation and parsing code is embedded and called **in-process**;
|
||||
@@ -11,7 +20,7 @@ there is no stdout handoff, no JSON handoff, no hooks, no subprocesses and no
|
||||
certificate files.
|
||||
|
||||
> A detected provider endpoint **does not** mean the underlying data is publicly
|
||||
> accessible. `ctcloud` reports facts (hostname existed in CT, resolved, answered
|
||||
> accessible. `bucketeer` reports facts (hostname existed in CT, resolved, answered
|
||||
> a request) separately from conclusions (`exposure_state`). Most findings are
|
||||
> simply an endpoint that exists.
|
||||
|
||||
@@ -38,7 +47,35 @@ CT logs ──► CertSpotter internals (fetch/verify/parse)
|
||||
PostgreSQL
|
||||
```
|
||||
|
||||
See [ARCHITECTURE.md](ARCHITECTURE.md) for per-stage rules.
|
||||
See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for per-stage rules.
|
||||
|
||||
## Repository layout
|
||||
|
||||
```text
|
||||
cmd/bucketeer/ application entry point
|
||||
internal/
|
||||
classify/ zero-allocation provider/resource classifier
|
||||
config/ environment configuration
|
||||
discover/ bucket-name discovery from CT domains
|
||||
dns/ resolver pool, TTL cache, CNAME following
|
||||
index/ anonymous listing -> metadata index
|
||||
metrics/ Prometheus text endpoint
|
||||
model/ shared enums and structs
|
||||
pipeline/ bounded queues, dedup, worker pools
|
||||
probe/ reachability + readability checks
|
||||
sample/ bounded random-sample queue
|
||||
scan/ on-demand scan and backfill
|
||||
store/ PostgreSQL schema, batch writer, CT state
|
||||
webui/ embedded web UI
|
||||
migrations/ embedded SQL schema
|
||||
third_party/certspotter/ vendored upstream CT engine (minimal patch)
|
||||
docs/ ARCHITECTURE.md, DESIGN.md, GOAL.md, SKILL.md
|
||||
assets/ brand assets
|
||||
```
|
||||
|
||||
More: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) (data flow),
|
||||
[docs/DESIGN.md](docs/DESIGN.md) (UI design system),
|
||||
[docs/SKILL.md](docs/SKILL.md) (how an agent queries and audits a running instance).
|
||||
|
||||
## Why CertSpotter is embedded (and patched)
|
||||
|
||||
@@ -48,7 +85,7 @@ tree maintenance, entry parsing and certificate reconstruction.
|
||||
|
||||
The only change is a two-hunk patch in `third_party/certspotter` that adds an
|
||||
`IdentifierSink` callback. After a certificate is parsed, the identifiers are
|
||||
handed to ctcloud with a direct Go function call and the watchlist/chain/
|
||||
handed to bucketeer with a direct Go function call and the watchlist/chain/
|
||||
notification machinery is skipped:
|
||||
|
||||
```go
|
||||
@@ -147,7 +184,7 @@ cp .env.example .env # set a strong POSTGRES_PASSWORD and matching DATABASE
|
||||
docker compose up --build
|
||||
```
|
||||
|
||||
That starts `postgres` and the stateless `ctcloud` container. The application
|
||||
That starts `postgres` and the stateless `bucketeer` container. The application
|
||||
container runs as non-root, has a read-only root filesystem, drops all Linux
|
||||
capabilities, sets CPU/memory/pids limits and mounts no application volume. The
|
||||
only persistent state is PostgreSQL (CT checkpoints + cloud assets + index).
|
||||
@@ -169,7 +206,7 @@ basic auth on the UI and its API.
|
||||
## Web UI
|
||||
|
||||
A small, dependency-free UI (vanilla JS, no build step, embedded in the binary)
|
||||
styled to the tokens and layout in `DESIGN.md` (neutral canvas, single 768px
|
||||
styled to the tokens and layout in [docs/DESIGN.md](docs/DESIGN.md) (neutral canvas, single 768px
|
||||
results column, flat rows, pill controls, no shadows).
|
||||
|
||||
* search box with 250ms debounce, stale-request cancellation and `?q=` URL state
|
||||
@@ -201,8 +238,8 @@ stay crisp on HiDPI displays while shrinking the ~1.2 MB source PNGs by ~96%.
|
||||
Regenerate after replacing the source PNGs at the repo root:
|
||||
|
||||
```sh
|
||||
magick bucketeer-light.png -resize 256x -strip /tmp/l.png
|
||||
magick bucketeer-dark.png -resize 256x -strip /tmp/d.png
|
||||
magick assets/bucketeer-light.png -resize 256x -strip /tmp/l.png
|
||||
magick assets/bucketeer-dark.png -resize 256x -strip /tmp/d.png
|
||||
cwebp -quiet -lossless -z 9 -alpha_q 100 /tmp/l.png -o internal/webui/assets/logo-light.webp
|
||||
cwebp -quiet -lossless -z 9 -alpha_q 100 /tmp/d.png -o internal/webui/assets/logo-dark.webp
|
||||
```
|
||||
@@ -249,7 +286,7 @@ attempted`). Database endpoints are never claimed to be readable: a raw
|
||||
* **Database TLS** — the local compose stack uses `sslmode=disable` because the
|
||||
compose bridge network is private. For production, point `DATABASE_URL` at a
|
||||
TLS-enabled database, e.g.
|
||||
`postgres://user:[email protected]:5432/ctcloud?sslmode=verify-full`.
|
||||
`postgres://user:[email protected]:5432/bucketeer?sslmode=verify-full`.
|
||||
pgx handles TLS natively; no extra flags are needed.
|
||||
* **Metrics** — bound to loopback. There is no auth, no debug endpoints and no
|
||||
high-cardinality labels (never hostnames).
|
||||
@@ -367,7 +404,7 @@ go vet ./...
|
||||
Integration tests for PostgreSQL run only when `DATABASE_URL` is set:
|
||||
|
||||
```sh
|
||||
DATABASE_URL=postgres://ctcloud:ctcloud@localhost:5432/ctcloud go test ./internal/store/
|
||||
DATABASE_URL=postgres://bucketeer:bucketeer@localhost:5432/bucketeer go test ./internal/store/
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
Reference in new issue
Block a user