StasonJatham
a899362e18
CMS SANs are custom domains, not bucket names, so matching SANs finds almost no buckets. This adds internal/discover: from each non-matching hostname it derives candidate names (domain label + subdomain label + ~40 suffixes) plus a built-in wordlist, and anonymously tests listing on S3, GCS and (optionally) DigitalOcean Spaces. Every 200 is indexed (keys/sizes/dates only). Bounded hard: global BUCKET_SCAN_QPS rate limit, 12h per-name dedup, random sampled domain queue, MaxPerDomain cap. Never downloads contents. Fix: GCS candidates now use the virtual-host form (<bucket>.storage.googleapis.com) so buckets no longer collide on the cloud_assets unique key; previously they all shared storage.googleapis.com and overwrote each other. Live: 30 open buckets in ~5 min (e.g. estimator-dev, image-prod, update-media, mos-images S3; several GCS buckets), 11k+ objects indexed.
38 lines
1.3 KiB
Bash
38 lines
1.3 KiB
Bash
# Copy to .env and edit. .env is gitignored.
|
|
|
|
POSTGRES_USER=ctcloud
|
|
POSTGRES_PASSWORD=change-me-strong
|
|
POSTGRES_DB=ctcloud
|
|
|
|
# Local compose (private bridge network, no TLS):
|
|
DATABASE_URL=postgres://ctcloud:change-me-strong@postgres:5432/ctcloud?sslmode=disable
|
|
|
|
# Production: use a TLS-enabled database instead, e.g.
|
|
# DATABASE_URL=postgres://user:[email protected]:5432/ctcloud?sslmode=verify-full
|
|
|
|
# DNS_SERVERS unset = ~20 curated public resolvers. Options:
|
|
# DNS_SERVERS=fresh curated + compiled-in Fresh-Resolvers (sampled)
|
|
# DNS_SERVERS=system use the OS resolver
|
|
# DNS_SERVERS=1.1.1.1,8.8.8.8 your own comma-separated list
|
|
# DNS_SERVERS=system
|
|
DNS_WORKERS=16
|
|
DNS_CACHE_SIZE=262144
|
|
DEDUP_SIZE=262144
|
|
LOG_LEVEL=info
|
|
LOG_FORMAT=text # json for log pipelines / LLM analysis
|
|
ACTIVE_PROBING=false
|
|
PROBE_DEEP=true # unauthenticated read-only DB protocol checks
|
|
INDEX_ENABLED=true # index metadata of readable resources
|
|
INDEX_MAX_OBJECTS=1000
|
|
|
|
# Optional HTTP basic auth for the web UI (/ and /api/*):
|
|
# UI_USER=admin
|
|
# UI_PASS=change-me-strong
|
|
|
|
# Bucket-name discovery (finds open buckets; CT SANs are domains, not buckets)
|
|
BUCKET_SCAN_ENABLED=true
|
|
BUCKET_SCAN_QPS=50
|
|
BUCKET_SCAN_WORKERS=16
|
|
BUCKET_SCAN_QUEUE=65536
|
|
BUCKET_SCAN_DO=false
|