add read-only web UI and per-vendor readability prober
Web UI (internal/webui): - dependency-free embedded HTML/JS served on the loopback listener - /api/assets (provider/service/exposure/q filters) and /api/stats - optional HTTP basic auth via UI_USER/UI_PASS Prober (internal/probe): - vendor-specific, unauthenticated, non-destructive checks: S3/R2/Spaces ListObjectsV2, GCS JSON object list, Azure ?comp=list, Firebase /.json?shallow=true, Supabase REST, and bare TCP connect for database endpoints (never reported readable, no auth attempted) - exposure_detail records the factual reason (status code, region, tcp port) - bounded random-sample queue instead of FIFO: one high-volume hostname template (mrap) no longer starves every other asset type - schema: cloud_assets.exposure_detail + probed_at (idempotent ALTER) Live: stable ~450MiB under 1GiB, no restarts, UI and probe states (private/reachable/auth_required) flowing, 0 db errors.
This commit is contained in:
1 parent
9710efec1a
commit
8fce55f8c9
13 files changed
+843
-72
No files matched your search
+1
-1
@@ -46,7 +46,7 @@ Certificate Transparency Logs
|
||||
| Direct queue | bound, non-blocking send | unbounded growth |
|
||||
| DNS enrichment | DNS via worker pool, TTL cache, CNAME follow | unbounded goroutines, persistence |
|
||||
| Dedup | memory-bounded TTL caches | unbounded maps |
|
||||
| Probe (optional) | DNS/TCP/TLS/HTTP HEAD/GET `/` | auth, downloads, mutation, DB or shell access |
|
||||
| Probe (optional) | one unauthenticated vendor read endpoint (S3/GCS/Azure/Firebase list) or a bare TCP connect for databases | credentials, object downloads, mutation, DB queries, auth attempts |
|
||||
| DB writer | batched `pgx.Batch` writes | one synchronous row per event |
|
||||
| Checkpoints | JSON CT position in `ct_checkpoints`/`ct_sths` | certificate/issuer contents in DB |
|
||||
|
||||
|
||||
Reference in new issue
Block a user