bucketeer

3 Commits 1 Branch 0 Tags

3 Commits

Author	SHA1	Message	Date
StasonJatham	9710efec1a	fix memory blowup and keep direct matches from being dropped - dns.TTLCache: stop preallocating every shard map to maxShard; this allocated the full theoretical cache capacity up front (hundreds of MB of empty buckets) and pushed the container over its memory limit, causing an OOM restart loop under Docker's default 1g cap. - pipeline: write direct provider matches immediately without a blocking DNS enrichment step, so they can no longer be starved by the optional CNAME queue. Add TestDirectNotStarvedByDNSBacklog. - config: add DEDUP_SIZE, lower DNS_CACHE_SIZE/DEDUP_SIZE defaults to 256k. - .env.example / README: document DEDUP_SIZE and cache sizing. Live: stable at ~250-300 MiB with no restarts, direct queue draining to 0.	2026-09-22 13:33:05 +02:00
StasonJatham	e8f7d6bfdf	ctcloud: CertSpotter-backed CT cloud-resource monitor Single stateless Go process that consumes CT logs via embedded (patched) CertSpotter internals, classifies cloud/database/storage DNS identifiers in-process with a zero-allocation classifier, enriches via round-robin DNS resolvers with CNAME following, and batch-writes useful findings to PostgreSQL. - third_party/certspotter: 2-hunk IdentifierSink patch, no stdout/JSON/files - internal/classify: AWS/Azure/GCP/Firebase/Cloudflare R2/DigitalOcean/Supabase/OCI - internal/dns: sharded bounded TTL cache, multi-resolver load balancing + failover - internal/pipeline: bounded queues, dedup, direct-priority load shedding - internal/store: pgx batch writer, Postgres CT checkpoints (no cert contents) - internal/probe: optional non-destructive reachability checks (off by default) - Docker: distroless non-root, read-only rootfs, no app volume, resource limits - tests enforce 0 allocs/op on the classifier and sink hot path	2026-09-22 13:13:10 +02:00
StasonJatham	9e78a114ac	first commit	2026-09-22 12:26:54 +02:00

Author

SHA1

Message

Date

StasonJatham

9710efec1a

fix memory blowup and keep direct matches from being dropped

- dns.TTLCache: stop preallocating every shard map to maxShard; this
  allocated the full theoretical cache capacity up front (hundreds of MB
  of empty buckets) and pushed the container over its memory limit,
  causing an OOM restart loop under Docker's default 1g cap.
- pipeline: write direct provider matches immediately without a blocking
  DNS enrichment step, so they can no longer be starved by the optional
  CNAME queue. Add TestDirectNotStarvedByDNSBacklog.
- config: add DEDUP_SIZE, lower DNS_CACHE_SIZE/DEDUP_SIZE defaults to 256k.
- .env.example / README: document DEDUP_SIZE and cache sizing.

Live: stable at ~250-300 MiB with no restarts, direct queue draining to 0.

2026-09-22 13:33:05 +02:00

StasonJatham

e8f7d6bfdf

ctcloud: CertSpotter-backed CT cloud-resource monitor

Single stateless Go process that consumes CT logs via embedded (patched)
CertSpotter internals, classifies cloud/database/storage DNS identifiers
in-process with a zero-allocation classifier, enriches via round-robin DNS
resolvers with CNAME following, and batch-writes useful findings to PostgreSQL.

- third_party/certspotter: 2-hunk IdentifierSink patch, no stdout/JSON/files
- internal/classify: AWS/Azure/GCP/Firebase/Cloudflare R2/DigitalOcean/Supabase/OCI
- internal/dns: sharded bounded TTL cache, multi-resolver load balancing + failover
- internal/pipeline: bounded queues, dedup, direct-priority load shedding
- internal/store: pgx batch writer, Postgres CT checkpoints (no cert contents)
- internal/probe: optional non-destructive reachability checks (off by default)
- Docker: distroless non-root, read-only rootfs, no app volume, resource limits
- tests enforce 0 allocs/op on the classifier and sink hot path

2026-09-22 13:13:10 +02:00

StasonJatham

9e78a114ac

first commit

2026-09-22 12:26:54 +02:00