karl/bucketeer
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Bucketeer UI (dark mode, pagination, virtual scroll), readable+indexed resources, LLM-friendly logs

Browse Source 
Logging:
- structured slog with LOG_FORMAT=json|text and a stable key vocabulary
- periodic stats record with queue depths, db pool, goroutines, memory, dns/probe
- no per-certificate logging (hot path is counters only)

UI (Bucketeer branding, DESIGN.md):
- light+dark themes (OS default, persisted header toggle, no first-paint flash)
- official mascot: lossless WebP light/dark logos + favicon, embedded and served
  with cache headers (1.2MB source PNGs -> ~27/22KB webp)
- Assets/Contents views, pagination (offset) and native virtual scrolling
  (content-visibility) so large result sets do not blow up the DOM
- search with debounce/abort + URL state, exposure chips, sort, summary counts

Readability + indexing:
- internal/index listers for S3-compatible, GCS, Azure, Firebase, OpenSearch
  (metadata only: keys/sizes/dates; never object contents)
- asset_entries table + store write/search; readable resources are indexed and
  searchable in the UI
- deep, unauthenticated, read-only DB probes (PROBE_DEEP): Redis PING,
  PostgreSQL StartupMessage, MongoDB OP_MSG listDatabases; OpenSearch/Neptune
  HTTP reads
- Azure account-level listing fixed: 403 is reported reachable, not
  auth_required, since anonymous access is per-container

DNS:
- curated reliable resolvers by default; DNS_SERVERS=fresh adds the compiled-in
  Fresh-Resolvers snapshot with a bounded startup sample (warming all 6.7k at
  once broke the container resolver)
- random healthy selection + failover + per-server cooldown

Other: prober Close fixes a shutdown hang; config/README updated.
This commit is contained in:
StasonJatham committed 2026-09-22 14:33:03 +02:00
1 parent 8fce55f8c9
commit a366905a43
28 files changed
+9744 -241
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+88 -16
View File
@@ -108,7 +108,7 @@ Environment variables only; no config files, no runtime pattern files.
| --- | --- | --- |
| `DATABASE_URL` | *(required)* | PostgreSQL connection string |
| `DB_MAX_CONNS` | `8` | pgx pool size |
| `DNS_SERVERS` | `1.1.1.1:53,8.8.8.8:53,1.0.0.1:53` | comma-separated resolvers (round-robin + failover); `system` for the host resolver |
| `DNS_SERVERS` | *(unset → curated + community list)* | comma-separated resolvers, or `system` for the OS resolver |
| `DNS_WORKERS` | `16` | DNS enrichment workers |
| `DNS_TIMEOUT` | `5s` | per-lookup timeout |
| `DNS_CACHE_SIZE` | `262144` | max DNS cache entries |
@@ -123,6 +123,9 @@ Environment variables only; no config files, no runtime pattern files.
| `ACTIVE_PROBING` | `false` | enable optional reachability checks |
| `PROBE_WORKERS` / `PROBE_TIMEOUT` | `8` / `5s` | probe worker pool |
| `PROBE_QUEUE_SIZE` | `65536` | probe sample size; when full, old entries are replaced at random (bounded memory) |
| `PROBE_DEEP` | `true` | unauthenticated, read-only protocol reads for databases (Redis/Postgres/Mongo) |
| `INDEX_ENABLED` / `INDEX_MAX_OBJECTS` | `true` / `1000` | index metadata of readable resources (keys/sizes only) |
| `LOG_FORMAT` | `text` | `text` or `json` (JSON is best for log pipelines/LLM analysis) |
| `DEDUP_TTL` | `1h` | hostname/finding dedup window |
| `METRICS_ADDR` | `:9090` | metrics/health/UI listener |
| `UI_USER` / `UI_PASS` | *(unset)* | optional HTTP basic auth for the web UI |
@@ -156,10 +159,44 @@ basic auth on the UI and its API.
## Web UI
A small, dependency-free UI lists discovered assets with provider/service,
resource, region, exposure state and the probe detail, plus summary counts and
provider/service/exposure filters. It is read-only, refreshes every 15s, and is
served from the same loopback port (`http://127.0.0.1:9090/`).
A small, dependency-free UI (vanilla JS, no build step, embedded in the binary)
styled to the tokens and layout in `DESIGN.md` (neutral canvas, single 768px
results column, flat rows, pill controls, no shadows).
* search box with 250ms debounce, stale-request cancellation and `?q=` URL state
* exposure filter chips — use **Public read** to see open buckets, **Reachable**
to see publicly reachable database endpoints — plus a sort control
* result rows show the provider hostname, probe detail, provider/service/region
badges, resource identifier and relative last-seen/probed times
* summary line with total assets, publicly readable and reachable counts
* refreshes every 15s (paused when the tab is hidden), read-only
* light and dark mode: follows the OS by default, with a header toggle that is
persisted in `localStorage`; the theme is applied before first paint to avoid
a flash
Served from the same loopback port: `http://127.0.0.1:9090/`.
### Brand assets
The UI ships the official Bucketeer mascot for both themes, optimized to
lossless WebP and embedded in the binary (`internal/webui/assets/`):
| Asset | Size | Use |
| --- | --- | --- |
| `logo-light.webp` | ~27 KB | black line art, light theme |
| `logo-dark.webp` | ~22 KB | white line art, dark theme |
| `favicon.png` | ~7 KB | browser tab |
They are downscaled to 256px and encoded with `cwebp -lossless -z 9`, so they
stay crisp on HiDPI displays while shrinking the ~1.2 MB source PNGs by ~96%.
Regenerate after replacing the source PNGs at the repo root:
```sh
magick bucketeer-light.png -resize 256x -strip /tmp/l.png
magick bucketeer-dark.png -resize 256x -strip /tmp/d.png
cwebp -quiet -lossless -z 9 -alpha_q 100 /tmp/l.png -o internal/webui/assets/logo-light.webp
cwebp -quiet -lossless -z 9 -alpha_q 100 /tmp/d.png -o internal/webui/assets/logo-dark.webp
```
## Reachability and readability checks
@@ -169,13 +206,25 @@ objects, mutates anything, runs database queries or attempts authentication.
| Vendor | Check | `public_read` means |
| --- | --- | --- |
| AWS S3, Cloudflare R2, DigitalOcean Spaces | anonymous `ListObjectsV2` (`?list-type=2&max-keys=1`) | anonymous bucket listing succeeded |
| Google Cloud Storage | JSON API object list (`/storage/v1/b/<bucket>/o?maxResults=1`) | `allUsers` can list objects |
| Azure Blob/File/Queue/Table | anonymous `?comp=list` | anonymous listing succeeded |
| AWS S3, Cloudflare R2, DigitalOcean Spaces, OCI | anonymous `ListObjectsV2` (`?list-type=2&max-keys=1`) | bucket policy grants `s3:ListBucket` to `*` |
| Google Cloud Storage | JSON API object list (`/storage/v1/b/<bucket>/o?maxResults=1`) | the bucket grants `allUsers` `storage.objects.list` |
| Azure Blob/File/Queue/Table | account-level `?comp=list` | anonymous access is **per container**, so account-level 403 is reported `reachable` (not "not readable"). Container names cannot be derived from the account hostname. |
| Firebase Realtime DB | `GET /.json?shallow=true` | security rules allow anonymous read |
| Supabase | `GET /rest/v1/` | — (an API key is required, so this reports `auth_required`) |
| Cosmos, OpenSearch, Search, Appspot, OCI | root `GET /` | a 2xx was returned |
| RDS, Redshift, DocumentDB, Neptune, ElastiCache, MemoryDB, Azure SQL/Postgres/MySQL/Redis, Supabase Postgres | unauthenticated TCP connect only | *never* — reported as `reachable` (port open) |
| Supabase | `GET /rest/v1/` | not testable without the public **anon API key**; reports `auth_required` |
| OpenSearch / Elasticsearch | `GET /_cat/indices` | security/access control disabled |
| Neptune | `GET /status` | a 2xx was returned |
| Cosmos / Appspot / Azure Search | root `GET /` | a 2xx was returned |
| Redis (ElastiCache, MemoryDB, Azure Redis) | unauthenticated `PING` | `+PONG` without `AUTH` |
| PostgreSQL (RDS/Aurora, Redshift, Azure, Supabase) | unauthenticated StartupMessage | `AuthenticationOk` (server trusts the connection) |
| MongoDB (DocumentDB, Cosmos Mongo) | OP_MSG `listDatabases` | command succeeds without auth |
| MySQL, Azure SQL, Cassandra (Cosmos) | TCP connect only | *never* — reported `reachable` |
Every check sends **no credentials**: no `Authorization` header, no password, no
token. "Anonymous" here means an unauthenticated request, exactly as an anonymous
cloud principal would make.
Deep protocol probes (`PROBE_DEEP`, default on) only ever issue read-only
requests or commands; nothing is written, and no authentication is attempted.
Probe outcomes are stored in `exposure_state` and a short factual
`exposure_detail` string (e.g. `s3 redirect region=eu-central-1`,
@@ -257,11 +306,34 @@ exactly 0 allocs/op, so the target is enforced by the test suite.
### DNS load balancing
DNS lookups are round-robined across `DNS_SERVERS`. Each attempt is bounded
(2s) and fails over to the next server on timeout/refused errors, so one dead
resolver does not stall lookups. Definitive answers (NXDOMAIN) are returned
immediately rather than retried on every server, so negative lookups do not
multiply DNS traffic.
When `DNS_SERVERS` is unset, the pool is ~20 curated, reliable public resolvers
(Cloudflare, Google, Quad9, OpenDNS, AdGuard, …). Each query picks a random
healthy resolver and fails over to at most two others; each attempt is bounded
(2s) and resolvers that fail repeatedly enter a 5-minute cooldown. Definitive
answers (NXDOMAIN) return immediately rather than being retried on every server,
so negative lookups do not multiply DNS traffic.
Set `DNS_SERVERS=fresh` to add a compiled-in snapshot of
[proabiral/Fresh-Resolvers](https://github.com/proabiral/Fresh-Resolvers)
(6,760 entries) on top of the curated set. It validates a random sample of 256 at
startup and cools down the rest on failure. Refresh the snapshot with:
```sh
curl -sL https://raw.githubusercontent.com/proabiral/Fresh-Resolvers/master/resolvers.txt \
-o internal/dns/resolvers.txt
```
> Probing or using thousands of community resolvers at once can saturate the
> container's own DNS and break unrelated lookups, which is why `fresh` is
> opt-in and sampled. These resolvers are also untrusted: they can observe and,
> in principle, forge DNS answers. That only affects enrichment (which endpoints
> are looked up), never the classifier. For sensitive deployments set
> `DNS_SERVERS` to your own resolvers, or `DNS_SERVERS=system`.
> These community resolvers are untrusted: they can observe and, in principle,
> forge DNS answers. That only affects enrichment (which endpoints are looked
> up), never the classifier. For sensitive deployments set `DNS_SERVERS` to your
> own resolvers, or `DNS_SERVERS=system`.
## Testing