# Goal Build a production-ready, extremely resource-efficient Certificate Transparency monitoring service in **Go**, running entirely in **Docker**, using **CertSpotter's existing CT ingestion/parsing implementation**. The purpose is to continuously consume CT logs, identify cloud storage/database/data-service resources, enrich candidates via DNS/CNAME resolution, optionally perform minimal safe reachability checks, and persist only useful enriched findings into PostgreSQL. The primary design objective is: > **Maximum throughput and minimum CPU, memory, allocation, disk, serialization, and IPC overhead.** Do not build another generic CertStream consumer. Reuse CertSpotter's CT downloading, validation, parsing, and log-monitoring logic. --- # Critical architectural requirement DO NOT use CertSpotter like this: ```text certspotter → stdout → JSON → pipe → parser ``` DO NOT use: ```text CertSpotter → JSON_FILENAME → filesystem → hook → subprocess → parser ``` Instead modify/integrate CertSpotter so parsed certificate DNS identifiers are handed directly to our Go code **inside the same process**. Target architecture: ```text Certificate Transparency Logs │ ▼ CertSpotter internals fetch / verify / parse │ ▼ DNS identifiers │ ▼ FAST CLASSIFIER │ ┌─────┴─────┐ │ │ direct cloud normal resource hostname │ │ │ ▼ │ DNS/CNAME enrichment │ │ │ ▼ │ classify CNAME targets │ │ └─────┬─────┘ ▼ dedup/enrichment │ ▼ optional safe reachability │ ▼ batch DB writer │ ▼ PostgreSQL ``` Everything between certificate parsing and candidate classification must remain in memory. --- # CertSpotter integration Use the upstream CertSpotter Go code as the CT engine. DO NOT reimplement CT log handling unless absolutely required. Inspect the current CertSpotter source before modifying anything. Create the **smallest possible maintainable patch** that exposes parsed certificate identifiers to our code. Preferred abstraction: ```go type IdentifierSink interface { HandleIdentifier(ctx context.Context, identifier Identifier) } ``` or an equally efficient callback. The goal is: ```text CertSpotter parsed certificate ↓ direct Go function call ↓ classifier ``` There must be: ```text NO stdout handoff NO JSON handoff NO hook scripts NO subprocesses NO certificate files NO JSON files NO temporary certificate files ``` Certificate DER/metadata may exist temporarily in memory while CertSpotter processes it, but must never be persisted. Do not use CertSpotter's normal certificate/output persistence functionality. Preserve upstream licensing/notices and keep our CertSpotter changes isolated so upstream changes can later be rebased. --- # Runtime filesystem policy The application must not persist certificates, SAN lists, JSON events, queues, enrichment payloads, or findings to the filesystem. No filesystem-based spool. No file-based queue. No external configuration files. No pattern YAML at runtime. No file uploads or file ingestion API. Configuration comes from environment variables. Provider patterns should be compiled into the binary. The Docker application container should require **no persistent volume**. Run with a read-only root filesystem. --- # CT checkpoint persistence We still need restart-safe CT monitoring. Do NOT persist CertSpotter checkpoint/state files locally. Store only the minimum required CT progress/checkpoint metadata in PostgreSQL. Example conceptual table: ```text ct_checkpoints log_id tree_size position / index updated_at ``` Inspect CertSpotter's real state model and preserve whatever fields are actually required. If CertSpotter currently assumes filesystem-backed state, introduce a minimal storage abstraction: ```go type StateStore interface { Load(...) Save(...) } ``` Implement: ```text PostgresStateStore ``` Do not store certificate contents in it. --- # Hot path requirements The certificate identifier classifier is the most performance-sensitive part. It must have: ```text NO regexp NO JSON NO reflection NO interface{} maps NO database access NO DNS NO HTTP NO filesystem NO subprocess NO logging per identifier NO unnecessary heap allocations NO string splitting into []string NO fmt.Sprintf ``` Target: ```text CertSpotter DNS identifier ↓ normalize ↓ cheap provider suffix rejection ↓ manual label/suffix classifier ↓ small struct ↓ bounded channel/ring ``` Use: ```go strings.HasSuffix strings.HasPrefix strings.Index / LastIndexByte direct byte/string comparisons manual backwards DNS-label scanning small uint8 enums ``` Do not use regex. Do not use Hyperscan. Do not use Aho-Corasick unless benchmarking proves it faster for this actual tiny pattern set. For this number of signatures, direct suffix/label matching should be faster and simpler. --- # Allocation requirements The classifier itself should ideally achieve: ```text 0 allocs/op ``` after receiving the hostname string. Avoid lowercasing every hostname if unnecessary. Fast-path ASCII lowercase names directly. Only normalize when uppercase characters are actually present. Avoid: ```go strings.Split() ``` because it allocates. Parse DNS labels by indexes. --- # Core data structures Use compact enums. Example: ```go type Provider uint8 const ( ProviderUnknown Provider = iota ProviderAWS ProviderAzure ProviderGCP ProviderCloudflare ProviderDigitalOcean ProviderSupabase ProviderOracle ) type Service uint8 const ( ServiceUnknown Service = iota ServiceObjectStorage ServiceS3 ServiceBlobStorage ServiceDatabase ServicePostgres ServiceMySQL ServiceRDS ServiceDocumentDB ServiceRedshift ServiceOpenSearch ServiceNeptune ServiceRedis ServiceMemoryDB ServiceCosmosDB ServiceFirebaseDB ) ``` A hot-path match should look roughly like: ```go type Candidate struct { Hostname string Provider Provider Service Service Source Source } ``` Keep hot-path structs small. Do not attach large certificate structures to downstream jobs. --- # Source tracking We need to distinguish: ```text DIRECT_CT CNAME DNS_ALIAS ``` Example: ```text assets.company.com CNAME company-assets.s3.eu-central-1.amazonaws.com ``` should produce an enriched record containing both: ```text original_hostname = assets.company.com provider_hostname = company-assets.s3.eu-central-1.amazonaws.com source = CNAME provider = AWS service = S3 ``` --- # Provider classification Implement dedicated provider classifiers. ## AWS Recognize resource-specific forms including: ```text # S3 .s3.amazonaws.com .s3..amazonaws.com .s3.dualstack..amazonaws.com .s3-website..amazonaws.com .s3-website-.amazonaws.com # Access Points -.s3-accesspoint..amazonaws.com *.s3-accesspoint.dualstack..amazonaws.com *.s3-accesspoint-fips..amazonaws.com # Multi-region *.mrap.accesspoint.s3-global.amazonaws.com # S3 Express *.s3express-..amazonaws.com # RDS / Aurora / RDS Proxy *.rds.amazonaws.com # DocumentDB *.docdb.amazonaws.com # Redshift *.redshift.amazonaws.com *.redshift-serverless.amazonaws.com # OpenSearch search-*.es.amazonaws.com vpc-*.es.amazonaws.com # Neptune *.neptune.amazonaws.com # MemoryDB *.memorydb..amazonaws.com # ElastiCache *.cache.amazonaws.com ``` Further classify AWS subtypes where cheap. Examples: ```text cluster-* → Aurora/cluster cluster-ro-* → reader proxy-* → RDS proxy search-* → OpenSearch public endpoint form vpc-* → OpenSearch VPC endpoint form ``` Do not treat generic AWS service API endpoints as customer resources. Examples to reject: ```text s3.eu-central-1.amazonaws.com dynamodb.eu-central-1.amazonaws.com redshift.eu-central-1.amazonaws.com ``` --- # Azure Recognize: ```text *.blob.core.windows.net *.dfs.core.windows.net *.file.core.windows.net *.queue.core.windows.net *.table.core.windows.net *.web.core.windows.net *.database.windows.net *.postgres.database.azure.com *.mysql.database.azure.com *.documents.azure.com *.mongo.cosmos.azure.com *.cassandra.cosmos.azure.com *.gremlin.cosmos.azure.com *.redis.cache.windows.net *.redis.azure.net *.search.windows.net ``` Require a customer/resource-specific prefix. Do not classify bare provider service endpoints as resource findings. --- # Google Cloud / Firebase Recognize: ```text .storage.googleapis.com .firebaseio.com ..firebasedatabase.app *.appspot.com ``` Do not classify: ```text storage.googleapis.com ``` itself as a resource. --- # Cloudflare Recognize: ```text .r2.cloudflarestorage.com *.r2.dev ``` Classify: ```text r2.cloudflarestorage.com → Cloudflare R2 API/resource endpoint r2.dev → public R2 development/public endpoint ``` Remember that arbitrary custom R2 domains will only be discovered through DNS/CNAME enrichment. --- # DigitalOcean Recognize: ```text ..digitaloceanspaces.com ..cdn.digitaloceanspaces.com *.db.ondigitalocean.com ``` Do not treat: ```text .digitaloceanspaces.com ``` as a customer bucket. --- # Supabase Recognize: ```text .supabase.co db..supabase.co aws--.pooler.supabase.com ``` Classify direct PostgreSQL separately where possible: ```text db..supabase.co ``` --- # Oracle Cloud / OCI Recognize: ```text .objectstorage..oci.customer-oci.com .swiftobjectstorage..oci.customer-oci.com ``` Also recognize generic OCI storage service endpoints but mark them: ```text resource_specific = false ``` Examples: ```text objectstorage..oraclecloud.com swiftobjectstorage..oraclecloud.com ``` These should normally not become high-value findings because the bucket name exists in the path rather than the hostname. --- # Classification quality Every matcher must determine: ```go type Match struct { Provider Provider Service Service ResourceSpecific bool PublicSemantic PublicSemantic } ``` `PublicSemantic` should distinguish cases where hostname format itself carries useful public/private information. Example: ```text AWS search-*.es.amazonaws.com → public network endpoint form AWS vpc-*.es.amazonaws.com → VPC endpoint form Cloudflare *.r2.dev → explicitly public URL form ``` Do NOT equate: ```text resource exists ``` with: ```text data is publicly accessible ``` --- # Direct-match path Every SAN/DNS identifier received from CertSpotter should first be passed through the provider classifier. Direct provider matches should be extremely cheap: ```text SAN ↓ provider suffix rejection ↓ specific classifier ↓ Candidate ``` Direct matches go immediately into enrichment. --- # CNAME discovery Provider-native names alone are insufficient because certificates frequently contain custom hostnames: ```text files.example.com ↓ CNAME foo.blob.core.windows.net ``` or: ```text db.example.com ↓ CNAME prod.abc.eu-central-1.rds.amazonaws.com ``` Therefore also support DNS enrichment of CT hostnames. Pipeline: ```text CT hostname ↓ dedupe/cache ↓ DNS ↓ follow CNAME chain ↓ run classifier against EVERY CNAME target ↓ candidate ``` Follow CNAME chains with: ```text maximum depth: configurable default: 8 ``` Detect loops. --- # DNS resolver DNS enrichment must be optimized heavily. Use a fixed worker pool. Do not spawn unbounded goroutines. Use either Go's resolver or `miekg/dns`, whichever benchmarks better for our workload. Requirements: ```text connection reuse where applicable bounded concurrency context cancellation timeouts positive TTL cache negative TTL cache CNAME cache duplicate suppression ``` The DNS cache is memory-only. No persistence. Use a memory-bounded implementation. It must have a configurable maximum number of entries. If implementing a custom sharded TTL cache is simpler/faster than bringing in a heavy dependency, prefer the custom implementation. --- # Backpressure Do not allow enrichment to stall CertSpotter indefinitely. Use bounded queues. Separate: ```text direct-match queue DNS-enrichment queue DB-write queue ``` Expose counters for: ```text queue depth queue full events dropped optional DNS enrichments direct matches CNAME matches DNS errors DB errors ``` Direct provider matches should receive higher priority than speculative CNAME enrichment. If load shedding becomes necessary, drop optional CNAME enrichment before dropping direct cloud-resource matches. --- # Deduplication Certificates and CT logs produce duplicates. Perform cheap in-memory dedup before enrichment. Key approximately: ```text normalized hostname ``` After classification: ```text provider + service + provider_hostname ``` Use memory-bounded TTL caches. Do not keep an unbounded map. Database uniqueness is the final deduplication layer. --- # Enrichment model For useful candidates collect, when cheap: ```text original CT hostname provider hostname provider service region if extractable resource identifier if extractable first seen last seen CNAME chain A records AAAA records public DNS resolution boolean source: direct_ct cname resource_specific public_semantic ``` Do not store the full certificate unless there is a concrete future requirement. Optional useful certificate metadata: ```text certificate fingerprint not_before not_after issuer CN CT log ID ``` Only include these if CertSpotter already has them available without additional expensive parsing/copying. --- # Safe reachability checks Make active network checks optional: ```text ACTIVE_PROBING=false ``` by default. When enabled, only perform minimal non-destructive checks on infrastructure the operator is authorized to assess. Allowed checks: ```text DNS resolution TCP connect TLS handshake HTTP HEAD HTTP GET / ``` No authentication attempts. No password guessing. No object downloading. No mutation. No PUT/POST/DELETE. No database queries. No exploitation. No brute force. For HTTP-based storage endpoints, use response status, headers, and provider error codes to classify: ```text reachable not_found auth_required public_read unknown ``` Do not claim something is "exposed" merely because its hostname exists. --- # Exposure state Use an explicit enum: ```go type ExposureState uint8 const ( ExposureUnknown ExposureState = iota ExposurePublicDNS ExposureReachable ExposureAuthRequired ExposurePublicRead ExposurePrivate ExposureNotFound ) ``` Keep factual observations separate from conclusions. --- # PostgreSQL Use PostgreSQL with: ```text pgx/v5 pgxpool ``` Do not write one row per network event synchronously. Use an asynchronous bounded DB writer with batching. Use `COPY FROM` or efficient pgx batches where appropriate. Suggested schema: ```text cloud_assets ------------ id provider service hostname provider_hostname resource_identifier region resource_specific public_semantic first_seen last_seen exposure_state UNIQUE(provider, service, provider_hostname) ``` and: ```text observations ------------ asset_id observed_at source_hostname source_type cname_chain ipv4 ipv6 cert_fingerprint ct_log_id ``` Exact schema may be improved if benchmarking/query requirements justify it. Do not store observations that provide no useful information. Avoid unnecessary JSONB if typed PostgreSQL arrays/columns are more efficient. --- # Checkpoints Separate CT processing state from cloud-asset data. Example: ```text ct_checkpoints -------------- log_id PRIMARY KEY state updated_at ``` Persist only enough to resume monitoring reliably. Batch/debounce checkpoint writes rather than writing on every certificate if CertSpotter semantics permit. Correctness must be preserved. --- # Logging Logging must never be on the certificate hot path. Do not log every certificate. Do not log every SAN. Do not log every rejected hostname. Log: ```text startup shutdown fatal errors persistent DB errors periodic aggregated statistics ``` Default periodic stats interval: ```text 60 seconds ``` Use structured logging but keep it lightweight. --- # Metrics Expose lightweight Prometheus-compatible metrics on an optional HTTP listener. At minimum: ```text ct_certificates_total ct_identifiers_total classifier_direct_matches_total classifier_cname_matches_total dns_queries_total dns_cache_hits_total dns_errors_total candidates_total db_rows_total db_errors_total queue_depth queue_drops_total process memory goroutines ``` Do not add expensive high-cardinality labels. Never label metrics by hostname. --- # Health Provide: ```text GET /healthz GET /readyz ``` `readyz` should verify: ```text CT monitor initialized database reachable critical worker pipelines running ``` --- # Configuration Environment variables only. Suggested configuration: ```text DATABASE_URL= DNS_WORKERS= DNS_TIMEOUT= DNS_CACHE_SIZE= DNS_MAX_CNAME_DEPTH= DIRECT_QUEUE_SIZE= DNS_QUEUE_SIZE= DB_QUEUE_SIZE= DB_BATCH_SIZE= DB_FLUSH_INTERVAL= ACTIVE_PROBING=false PROBE_WORKERS= PROBE_TIMEOUT= METRICS_ADDR=:9090 LOG_LEVEL=info STATS_INTERVAL=60s ``` Provide sensible defaults. Validate configuration on startup. --- # Docker Create a production Docker setup. Use a multi-stage Dockerfile. Build a static/minimal Go binary where possible. Final image should be minimal, for example distroless or equivalent. Run as non-root. Requirements: ```text read-only root filesystem no application volume no CertSpotter volume no certificate storage no writable application directory ``` Use: ```text tmpfs ``` only if absolutely unavoidable; preferably the application should not require it. Provide: ```text Dockerfile docker-compose.yml ``` Docker Compose should contain: ```text ctcloud postgres ``` The application itself must remain stateless apart from PostgreSQL. --- # Code organization Keep the project simple. Suggested layout: ```text cmd/ctcloud/ main.go internal/ certspotter/ classify/ classify.go aws.go azure.go gcp.go cloudflare.go digitalocean.go supabase.go oracle.go dns/ probe/ pipeline/ store/ metrics/ config/ model/ migrations/ Dockerfile docker-compose.yml README.md ARCHITECTURE.md ``` If CertSpotter is vendored/forked, keep upstream changes clearly separated. Do not introduce microservices. This should be **one Go application process** plus PostgreSQL. No Kafka. No Redis. No RabbitMQ. No external queue. No Elasticsearch. No unnecessary infrastructure. --- # Performance strategy Optimize in this order: 1. eliminate I/O from hot path 2. eliminate serialization 3. eliminate process boundaries 4. eliminate allocations 5. cheap provider rejection 6. cheap suffix/label matching 7. bounded worker pools 8. DNS caching 9. deduplication before enrichment 10. batched DB writes Do not prematurely implement complex SIMD/native libraries unless benchmarks demonstrate an actual bottleneck. --- # Benchmarks Add benchmarks for classification. Example: ```go func BenchmarkClassifyAWS(b *testing.B) func BenchmarkClassifyAzure(b *testing.B) func BenchmarkClassifyMixed(b *testing.B) func BenchmarkRejectNormalDomain(b *testing.B) ``` Benchmark realistic input: ```text www.example.com foo.bar.example.org bucket.s3.eu-central-1.amazonaws.com foo.blob.core.windows.net db.example.supabase.co project.firebaseio.com search-example-abc.eu-west-1.es.amazonaws.com ``` Important metric: ```text allocs/op ``` Classifier target: ```text 0 allocs/op ``` Do not make CI depend on an absolute nanosecond threshold because runner performance varies. Report benchmark numbers in README. --- # Tests Implement table-driven tests covering every provider format. Test: ```text positive resource matches generic provider endpoints rejected uppercase handling trailing dot wildcards invalid hostnames very long hostnames CNAME chains CNAME loops duplicate events IPv4/IPv6 context cancellation database retries queue saturation ``` Particularly ensure these DO NOT produce resource findings: ```text storage.googleapis.com s3.eu-central-1.amazonaws.com redshift.eu-central-1.amazonaws.com fra1.digitaloceanspaces.com ``` while: ```text bucket.storage.googleapis.com bucket.s3.eu-central-1.amazonaws.com cluster.id.eu-central-1.redshift.amazonaws.com bucket.fra1.digitaloceanspaces.com ``` do. --- # Wildcards Handle certificate wildcards carefully. Example: ```text *.example.com ``` must not become a literal DNS query. Do not interpret provider-wide wildcard certificates such as: ```text *.blob.core.windows.net ``` as discovered customer resources. Track wildcard certificates separately or ignore them for resource-specific classification unless a concrete hostname can be derived from another SAN. --- # Graceful shutdown Use contexts throughout. On SIGTERM: ```text stop accepting new enrichment finish bounded in-flight enrichment flush DB batch save CT checkpoint close DB exit ``` Shutdown must have a maximum timeout. Docker should be able to stop the container cleanly. --- # Failure handling The program must survive transient: ```text DNS failures network timeouts CT connection failures PostgreSQL connection resets ``` with bounded exponential backoff and jitter. Never retry in a tight loop. If PostgreSQL remains unavailable and queues approach capacity, prefer controlled backpressure/load shedding instead of unbounded memory growth. --- # Security Run as non-root. No shell needed in final container. No exposed debug endpoints. Set HTTP server timeouts. Validate all hostname lengths. Treat all CT data as untrusted input. Never execute certificate-controlled values. Never construct shell commands. Use parameterized SQL only. --- # README Document: ```text what the application does architecture why CertSpotter is embedded/patched why stdout/hooks/files are deliberately avoided supported provider patterns configuration Docker startup database schema performance characteristics benchmark results limitations ``` Clearly explain: ```text A detected provider endpoint does NOT automatically mean the underlying data is publicly accessible. ``` --- # ARCHITECTURE.md Include a concise data-flow diagram: ```text CT ↓ CertSpotter ↓ Identifier callback ↓ Direct classifier ───────────────┐ ↓ non-match │ DNS/CNAME │ ↓ │ Provider classifier │ └───────────────┬───────────────┘ ↓ dedupe ↓ enrichment ↓ optional probing ↓ DB batch ↓ PostgreSQL ``` Explain which operations are allowed in each stage. --- # Implementation priorities Implement in this order: ## Phase 1 ```text CertSpotter integration in-memory identifier callback provider classifier classification tests benchmarks ``` Do not continue until the classifier has zero unnecessary allocations. ## Phase 2 ```text bounded pipeline dedup DNS resolver CNAME following DNS cache ``` ## Phase 3 ```text PostgreSQL schema batch writer Postgres CT checkpoint state ``` ## Phase 4 ```text optional safe probing metrics health endpoints ``` ## Phase 5 ```text Docker hardening benchmarks documentation integration tests ``` --- # Important implementation rule Do not blindly follow this design if inspection of the current CertSpotter source reveals a more direct or efficient integration point. The goal is not to satisfy a predetermined class hierarchy. The goal is: ```text reuse CertSpotter + remove output/storage overhead + receive parsed DNS identifiers directly + classify with essentially negligible CPU cost + only enrich potentially useful data + persist final useful results efficiently ``` If CertSpotter already exposes an internal abstraction that accomplishes this cleanly, use it rather than creating another layer. --- # Things you must NOT do Do not: ```text write certificates to disk write JSON events to disk read JSON event files call CertSpotter through exec.Command parse CertSpotter stdout use hooks fork a process per certificate use regex in the classifier use shell pipelines store every CT certificate store every SAN create unbounded goroutines create unbounded queues create unbounded maps write each result individually to PostgreSQL introduce Kafka/Redis just for queues build multiple microservices ``` --- # Definition of done The project is complete when: 1. A single Dockerized Go process continuously consumes CT logs using CertSpotter internals. 2. CertSpotter never writes certificate/result files. 3. Parsed DNS identifiers are transferred directly into our classifier via Go calls. 4. The classifier covers AWS, Azure, GCP/Firebase, Cloudflare R2, DigitalOcean, Supabase and OCI patterns above. 5. Direct provider resources are detected immediately. 6. CNAME chains are asynchronously enriched and classified. 7. Duplicate processing is bounded and minimized. 8. Only useful resource findings reach PostgreSQL. 9. CT resume/checkpoint state lives in PostgreSQL rather than local files. 10. Classifier benchmarks demonstrate zero unnecessary allocations. 11. Resource usage remains bounded under overload. 12. Docker requires no application persistent volume. 13. `go test ./...` passes. 14. `go test -race ./...` passes. 15. `go vet ./...` passes. 16. Docker build succeeds. 17. The README explains exactly how to run and benchmark it. Before declaring completion, inspect the resulting program under load and remove any unnecessary: ```text allocations copies serialization logging locking filesystem access goroutine creation database round trips ``` Favor simple, measurable performance over architectural complexity.