add read-only web UI and per-vendor readability prober
Web UI (internal/webui): - dependency-free embedded HTML/JS served on the loopback listener - /api/assets (provider/service/exposure/q filters) and /api/stats - optional HTTP basic auth via UI_USER/UI_PASS Prober (internal/probe): - vendor-specific, unauthenticated, non-destructive checks: S3/R2/Spaces ListObjectsV2, GCS JSON object list, Azure ?comp=list, Firebase /.json?shallow=true, Supabase REST, and bare TCP connect for database endpoints (never reported readable, no auth attempted) - exposure_detail records the factual reason (status code, region, tcp port) - bounded random-sample queue instead of FIFO: one high-volume hostname template (mrap) no longer starves every other asset type - schema: cloud_assets.exposure_detail + probed_at (idempotent ALTER) Live: stable ~450MiB under 1GiB, no restarts, UI and probe states (private/reachable/auth_required) flowing, 0 db errors.
This commit is contained in:
1 parent
9710efec1a
commit
8fce55f8c9
13 files changed
+843
-72
No files matched your search
@@ -31,11 +31,14 @@ type Config struct {
|
||||
DBBatchSize int
|
||||
DBFlushInterval time.Duration
|
||||
|
||||
ActiveProbing bool
|
||||
ProbeWorkers int
|
||||
ProbeTimeout time.Duration
|
||||
ActiveProbing bool
|
||||
ProbeWorkers int
|
||||
ProbeTimeout time.Duration
|
||||
ProbeQueueSize int
|
||||
|
||||
MetricsAddr string
|
||||
UIUser string
|
||||
UIPass string
|
||||
LogLevel string
|
||||
StatsInterval time.Duration
|
||||
|
||||
@@ -70,9 +73,12 @@ func Load() (*Config, error) {
|
||||
DBBatchSize: envInt("DB_BATCH_SIZE", 500),
|
||||
DBFlushInterval: envDuration("DB_FLUSH_INTERVAL", 2*time.Second),
|
||||
ActiveProbing: envBool("ACTIVE_PROBING", false),
|
||||
ProbeWorkers: envInt("PROBE_WORKERS", 4),
|
||||
ProbeWorkers: envInt("PROBE_WORKERS", 8),
|
||||
ProbeTimeout: envDuration("PROBE_TIMEOUT", 5*time.Second),
|
||||
ProbeQueueSize: envInt("PROBE_QUEUE_SIZE", 1<<16),
|
||||
MetricsAddr: envString("METRICS_ADDR", ":9090"),
|
||||
UIUser: envString("UI_USER", ""),
|
||||
UIPass: envString("UI_PASS", ""),
|
||||
LogLevel: envString("LOG_LEVEL", "info"),
|
||||
StatsInterval: envDuration("STATS_INTERVAL", 60*time.Second),
|
||||
LogListSource: envString("LOG_LIST_SOURCE", DefaultLogListSource),
|
||||
|
||||
Reference in new issue
Block a user