add read-only web UI and per-vendor readability prober

Web UI (internal/webui):
- dependency-free embedded HTML/JS served on the loopback listener
- /api/assets (provider/service/exposure/q filters) and /api/stats
- optional HTTP basic auth via UI_USER/UI_PASS

Prober (internal/probe):
- vendor-specific, unauthenticated, non-destructive checks:
  S3/R2/Spaces ListObjectsV2, GCS JSON object list, Azure ?comp=list,
  Firebase /.json?shallow=true, Supabase REST, and bare TCP connect for
  database endpoints (never reported readable, no auth attempted)
- exposure_detail records the factual reason (status code, region, tcp port)
- bounded random-sample queue instead of FIFO: one high-volume hostname
  template (mrap) no longer starves every other asset type
- schema: cloud_assets.exposure_detail + probed_at (idempotent ALTER)

Live: stable ~450MiB under 1GiB, no restarts, UI and probe states
(private/reachable/auth_required) flowing, 0 db errors.

README.md

@@ -121,9 +121,11 @@ Environment variables only; no config files, no runtime pattern files.
 | `DB_BATCH_SIZE` | `500` | findings per write batch |
 | `DB_FLUSH_INTERVAL` | `2s` | max time before a batch is flushed |
 | `ACTIVE_PROBING` | `false` | enable optional reachability checks |
-| `PROBE_WORKERS` / `PROBE_TIMEOUT` | `4` / `5s` | probe pool |
+| `PROBE_WORKERS` / `PROBE_TIMEOUT` | `8` / `5s` | probe worker pool |
+| `PROBE_QUEUE_SIZE` | `65536` | probe sample size; when full, old entries are replaced at random (bounded memory) |
 | `DEDUP_TTL` | `1h` | hostname/finding dedup window |
-| `METRICS_ADDR` | `:9090` | metrics/health listener |
+| `METRICS_ADDR` | `:9090` | metrics/health/UI listener |
+| `UI_USER` / `UI_PASS` | *(unset)* | optional HTTP basic auth for the web UI |
 | `LOG_LEVEL` | `info` | `debug\|info\|warn\|error` |
 | `STATS_INTERVAL` | `60s` | aggregated stats log interval |
 | `LOG_LIST_SOURCE` | Google log list v3 | certspotter log list URL |
@@ -142,9 +144,44 @@ container runs as non-root, has a read-only root filesystem, drops all Linux
 capabilities, sets CPU/memory/pids limits and mounts no application volume. The
 only persistent state is PostgreSQL (CT checkpoints + cloud assets).
 
-Endpoints: `GET /metrics`, `GET /healthz`, `GET /readyz` on port `9090`, bound
-to `127.0.0.1` only. The listener has no authentication, so scrape it locally or
-from a sidecar and never expose it publicly.
+Endpoints on port `9090`, bound to `127.0.0.1` only:
+
+* `GET /` — web UI (see below)
+* `GET /api/assets`, `GET /api/stats` — JSON for the UI
+* `GET /metrics`, `GET /healthz`, `GET /readyz`
+
+The listener has no authentication by default, so scrape it locally or from a
+sidecar and never expose it publicly. Set `UI_USER`/`UI_PASS` to require HTTP
+basic auth on the UI and its API.
+
+## Web UI
+
+A small, dependency-free UI lists discovered assets with provider/service,
+resource, region, exposure state and the probe detail, plus summary counts and
+provider/service/exposure filters. It is read-only, refreshes every 15s, and is
+served from the same loopback port (`http://127.0.0.1:9090/`).
+
+## Reachability and readability checks
+
+`ACTIVE_PROBING=false` by default. When enabled, the prober issues **one**
+unauthenticated request per finding. It never sends credentials, downloads
+objects, mutates anything, runs database queries or attempts authentication.
+
+| Vendor | Check | `public_read` means |
+| --- | --- | --- |
+| AWS S3, Cloudflare R2, DigitalOcean Spaces | anonymous `ListObjectsV2` (`?list-type=2&max-keys=1`) | anonymous bucket listing succeeded |
+| Google Cloud Storage | JSON API object list (`/storage/v1/b/<bucket>/o?maxResults=1`) | `allUsers` can list objects |
+| Azure Blob/File/Queue/Table | anonymous `?comp=list` | anonymous listing succeeded |
+| Firebase Realtime DB | `GET /.json?shallow=true` | security rules allow anonymous read |
+| Supabase | `GET /rest/v1/` | — (an API key is required, so this reports `auth_required`) |
+| Cosmos, OpenSearch, Search, Appspot, OCI | root `GET /` | a 2xx was returned |
+| RDS, Redshift, DocumentDB, Neptune, ElastiCache, MemoryDB, Azure SQL/Postgres/MySQL/Redis, Supabase Postgres | unauthenticated TCP connect only | *never* — reported as `reachable` (port open) |
+
+Probe outcomes are stored in `exposure_state` and a short factual
+`exposure_detail` string (e.g. `s3 redirect region=eu-central-1`,
+`status=403 authentication required`, `tcp/5432 open; no authentication
+attempted`). Database endpoints are never claimed to be readable: a raw
+`public_read` on a database is impossible by design.
 
 ## Production notes
 
@@ -167,8 +204,8 @@ from a sidecar and never expose it publicly.
 
 * `cloud_assets` — one row per `(provider, service, provider_hostname)`, with
   `hostname` (original CT hostname), `resource_identifier`, `region`,
-  `resource_specific`, `public_semantic`, `exposure_state`, `first_seen`,
-  `last_seen`.
+  `resource_specific`, `public_semantic`, `exposure_state`, `exposure_detail`,
+  `probed_at`, `first_seen`, `last_seen`.
 * `observations` — append-only observations linking an asset to a
   `source_hostname`, `source_type` (`direct_ct`/`cname`), CNAME chain and
   A/AAAA records.