bucketeer/GOAL.md

Goal

Build a production-ready, extremely resource-efficient Certificate Transparency monitoring service in Go, running entirely in Docker, using CertSpotter's existing CT ingestion/parsing implementation.

The purpose is to continuously consume CT logs, identify cloud storage/database/data-service resources, enrich candidates via DNS/CNAME resolution, optionally perform minimal safe reachability checks, and persist only useful enriched findings into PostgreSQL.

The primary design objective is:

Maximum throughput and minimum CPU, memory, allocation, disk, serialization, and IPC overhead.

Do not build another generic CertStream consumer. Reuse CertSpotter's CT downloading, validation, parsing, and log-monitoring logic.

---

Critical architectural requirement

DO NOT use CertSpotter like this:

certspotter
 → stdout
 → JSON
 → pipe
 → parser

DO NOT use:

CertSpotter
 → JSON_FILENAME
 → filesystem
 → hook
 → subprocess
 → parser

Instead modify/integrate CertSpotter so parsed certificate DNS identifiers are handed directly to our Go code inside the same process.

Target architecture:

Certificate Transparency Logs
             │
             ▼
     CertSpotter internals
   fetch / verify / parse
             │
             ▼
       DNS identifiers
             │
             ▼
       FAST CLASSIFIER
             │
       ┌─────┴─────┐
       │           │
 direct cloud     normal
 resource         hostname
       │           │
       │           ▼
       │      DNS/CNAME enrichment
       │           │
       │           ▼
       │     classify CNAME targets
       │           │
       └─────┬─────┘
             ▼
        dedup/enrichment
             │
             ▼
   optional safe reachability
             │
             ▼
       batch DB writer
             │
             ▼
         PostgreSQL

Everything between certificate parsing and candidate classification must remain in memory.

---

CertSpotter integration

Use the upstream CertSpotter Go code as the CT engine.

DO NOT reimplement CT log handling unless absolutely required.

Inspect the current CertSpotter source before modifying anything.

Create the smallest possible maintainable patch that exposes parsed certificate identifiers to our code.

Preferred abstraction:

type IdentifierSink interface {
    HandleIdentifier(ctx context.Context, identifier Identifier)
}

or an equally efficient callback.

The goal is:

CertSpotter parsed certificate
       ↓
direct Go function call
       ↓
classifier

There must be:

NO stdout handoff
NO JSON handoff
NO hook scripts
NO subprocesses
NO certificate files
NO JSON files
NO temporary certificate files

Certificate DER/metadata may exist temporarily in memory while CertSpotter processes it, but must never be persisted.

Do not use CertSpotter's normal certificate/output persistence functionality.

Preserve upstream licensing/notices and keep our CertSpotter changes isolated so upstream changes can later be rebased.

---

Runtime filesystem policy

The application must not persist certificates, SAN lists, JSON events, queues, enrichment payloads, or findings to the filesystem.

No filesystem-based spool.

No file-based queue.

No external configuration files.

No pattern YAML at runtime.

No file uploads or file ingestion API.

Configuration comes from environment variables.

Provider patterns should be compiled into the binary.

The Docker application container should require no persistent volume.

Run with a read-only root filesystem.

---

CT checkpoint persistence

We still need restart-safe CT monitoring.

Do NOT persist CertSpotter checkpoint/state files locally.

Store only the minimum required CT progress/checkpoint metadata in PostgreSQL.

Example conceptual table:

ct_checkpoints

log_id
tree_size
position / index
updated_at

Inspect CertSpotter's real state model and preserve whatever fields are actually required.

If CertSpotter currently assumes filesystem-backed state, introduce a minimal storage abstraction:

type StateStore interface {
    Load(...)
    Save(...)
}

Implement:

PostgresStateStore

Do not store certificate contents in it.

---

Hot path requirements

The certificate identifier classifier is the most performance-sensitive part.

It must have:

NO regexp
NO JSON
NO reflection
NO interface{} maps
NO database access
NO DNS
NO HTTP
NO filesystem
NO subprocess
NO logging per identifier
NO unnecessary heap allocations
NO string splitting into []string
NO fmt.Sprintf

Target:

CertSpotter DNS identifier
        ↓
normalize
        ↓
cheap provider suffix rejection
        ↓
manual label/suffix classifier
        ↓
small struct
        ↓
bounded channel/ring

Use:

strings.HasSuffix
strings.HasPrefix
strings.Index / LastIndexByte
direct byte/string comparisons
manual backwards DNS-label scanning
small uint8 enums

Do not use regex.

Do not use Hyperscan.

Do not use Aho-Corasick unless benchmarking proves it faster for this actual tiny pattern set.

For this number of signatures, direct suffix/label matching should be faster and simpler.

---

Allocation requirements

The classifier itself should ideally achieve:

0 allocs/op

after receiving the hostname string.

Avoid lowercasing every hostname if unnecessary.

Fast-path ASCII lowercase names directly.

Only normalize when uppercase characters are actually present.

Avoid:

strings.Split()

because it allocates.

Parse DNS labels by indexes.

---

Core data structures

Use compact enums.

Example:

type Provider uint8

const (
    ProviderUnknown Provider = iota
    ProviderAWS
    ProviderAzure
    ProviderGCP
    ProviderCloudflare
    ProviderDigitalOcean
    ProviderSupabase
    ProviderOracle
)

type Service uint8

const (
    ServiceUnknown Service = iota

    ServiceObjectStorage
    ServiceS3
    ServiceBlobStorage
    ServiceDatabase
    ServicePostgres
    ServiceMySQL
    ServiceRDS
    ServiceDocumentDB
    ServiceRedshift
    ServiceOpenSearch
    ServiceNeptune
    ServiceRedis
    ServiceMemoryDB
    ServiceCosmosDB
    ServiceFirebaseDB
)

A hot-path match should look roughly like:

type Candidate struct {
    Hostname string
    Provider Provider
    Service  Service
    Source   Source
}

Keep hot-path structs small.

Do not attach large certificate structures to downstream jobs.

---

Source tracking

We need to distinguish:

DIRECT_CT
CNAME
DNS_ALIAS

Example:

assets.company.com
    CNAME
company-assets.s3.eu-central-1.amazonaws.com

should produce an enriched record containing both:

original_hostname = assets.company.com
provider_hostname = company-assets.s3.eu-central-1.amazonaws.com
source            = CNAME
provider          = AWS
service           = S3

---

Provider classification

Implement dedicated provider classifiers.

AWS

Recognize resource-specific forms including:

# S3
<bucket>.s3.amazonaws.com
<bucket>.s3.<region>.amazonaws.com
<bucket>.s3.dualstack.<region>.amazonaws.com
<bucket>.s3-website.<region>.amazonaws.com
<bucket>.s3-website-<region>.amazonaws.com

# Access Points
<access-point>-<account>.s3-accesspoint.<region>.amazonaws.com
*.s3-accesspoint.dualstack.<region>.amazonaws.com
*.s3-accesspoint-fips.<region>.amazonaws.com

# Multi-region
*.mrap.accesspoint.s3-global.amazonaws.com

# S3 Express
*.s3express-<zone>.<region>.amazonaws.com

# RDS / Aurora / RDS Proxy
*.rds.amazonaws.com

# DocumentDB
*.docdb.amazonaws.com

# Redshift
*.redshift.amazonaws.com
*.redshift-serverless.amazonaws.com

# OpenSearch
search-*.es.amazonaws.com
vpc-*.es.amazonaws.com

# Neptune
*.neptune.amazonaws.com

# MemoryDB
*.memorydb.<region>.amazonaws.com

# ElastiCache
*.cache.amazonaws.com

Further classify AWS subtypes where cheap.

Examples:

cluster-*       → Aurora/cluster
cluster-ro-*    → reader
proxy-*         → RDS proxy

search-*        → OpenSearch public endpoint form
vpc-*           → OpenSearch VPC endpoint form

Do not treat generic AWS service API endpoints as customer resources.

Examples to reject:

s3.eu-central-1.amazonaws.com
dynamodb.eu-central-1.amazonaws.com
redshift.eu-central-1.amazonaws.com

---

Azure

Recognize:

*.blob.core.windows.net
*.dfs.core.windows.net
*.file.core.windows.net
*.queue.core.windows.net
*.table.core.windows.net
*.web.core.windows.net

*.database.windows.net
*.postgres.database.azure.com
*.mysql.database.azure.com

*.documents.azure.com
*.mongo.cosmos.azure.com
*.cassandra.cosmos.azure.com
*.gremlin.cosmos.azure.com

*.redis.cache.windows.net
*.redis.azure.net

*.search.windows.net

Require a customer/resource-specific prefix.

Do not classify bare provider service endpoints as resource findings.

---

Google Cloud / Firebase

Recognize:

<bucket>.storage.googleapis.com

<database>.firebaseio.com

<database>.<region>.firebasedatabase.app

*.appspot.com

Do not classify:

storage.googleapis.com

itself as a resource.

---

Cloudflare

Recognize:

<account-id>.r2.cloudflarestorage.com
*.r2.dev

Classify:

r2.cloudflarestorage.com → Cloudflare R2 API/resource endpoint
r2.dev                  → public R2 development/public endpoint

Remember that arbitrary custom R2 domains will only be discovered through DNS/CNAME enrichment.

---

DigitalOcean

Recognize:

<bucket>.<region>.digitaloceanspaces.com
<bucket>.<region>.cdn.digitaloceanspaces.com

*.db.ondigitalocean.com

Do not treat:

<region>.digitaloceanspaces.com

as a customer bucket.

---

Supabase

Recognize:

<project-ref>.supabase.co
db.<project-ref>.supabase.co
aws-<index>-<region>.pooler.supabase.com

Classify direct PostgreSQL separately where possible:

db.<project-ref>.supabase.co

---

Oracle Cloud / OCI

Recognize:

<namespace>.objectstorage.<region>.oci.customer-oci.com
<namespace>.swiftobjectstorage.<region>.oci.customer-oci.com

Also recognize generic OCI storage service endpoints but mark them:

resource_specific = false

Examples:

objectstorage.<region>.oraclecloud.com
swiftobjectstorage.<region>.oraclecloud.com

These should normally not become high-value findings because the bucket name exists in the path rather than the hostname.

---

Classification quality

Every matcher must determine:

type Match struct {
    Provider         Provider
    Service          Service
    ResourceSpecific bool
    PublicSemantic   PublicSemantic
}

PublicSemantic should distinguish cases where hostname format itself carries useful public/private information.

Example:

AWS search-*.es.amazonaws.com
    → public network endpoint form

AWS vpc-*.es.amazonaws.com
    → VPC endpoint form

Cloudflare *.r2.dev
    → explicitly public URL form

Do NOT equate:

resource exists

with:

data is publicly accessible

---

Direct-match path

Every SAN/DNS identifier received from CertSpotter should first be passed through the provider classifier.

Direct provider matches should be extremely cheap:

SAN
 ↓
provider suffix rejection
 ↓
specific classifier
 ↓
Candidate

Direct matches go immediately into enrichment.

---

CNAME discovery

Provider-native names alone are insufficient because certificates frequently contain custom hostnames:

files.example.com
     ↓ CNAME
foo.blob.core.windows.net

or:

db.example.com
     ↓ CNAME
prod.abc.eu-central-1.rds.amazonaws.com

Therefore also support DNS enrichment of CT hostnames.

Pipeline:

CT hostname
   ↓
dedupe/cache
   ↓
DNS
   ↓
follow CNAME chain
   ↓
run classifier against EVERY CNAME target
   ↓
candidate

Follow CNAME chains with:

maximum depth: configurable
default: 8

Detect loops.

---

DNS resolver

DNS enrichment must be optimized heavily.

Use a fixed worker pool.

Do not spawn unbounded goroutines.

Use either Go's resolver or miekg/dns, whichever benchmarks better for our workload.

Requirements:

connection reuse where applicable
bounded concurrency
context cancellation
timeouts
positive TTL cache
negative TTL cache
CNAME cache
duplicate suppression

The DNS cache is memory-only.

No persistence.

Use a memory-bounded implementation.

It must have a configurable maximum number of entries.

If implementing a custom sharded TTL cache is simpler/faster than bringing in a heavy dependency, prefer the custom implementation.

---

Backpressure

Do not allow enrichment to stall CertSpotter indefinitely.

Use bounded queues.

Separate:

direct-match queue

DNS-enrichment queue

DB-write queue

Expose counters for:

queue depth
queue full events
dropped optional DNS enrichments
direct matches
CNAME matches
DNS errors
DB errors

Direct provider matches should receive higher priority than speculative CNAME enrichment.

If load shedding becomes necessary, drop optional CNAME enrichment before dropping direct cloud-resource matches.

---

Deduplication

Certificates and CT logs produce duplicates.

Perform cheap in-memory dedup before enrichment.

Key approximately:

normalized hostname

After classification:

provider + service + provider_hostname

Use memory-bounded TTL caches.

Do not keep an unbounded map.

Database uniqueness is the final deduplication layer.

---

Enrichment model

For useful candidates collect, when cheap:

original CT hostname
provider hostname
provider
service
region if extractable
resource identifier if extractable

first seen
last seen

CNAME chain

A records
AAAA records

public DNS resolution boolean

source:
  direct_ct
  cname

resource_specific

public_semantic

Do not store the full certificate unless there is a concrete future requirement.

Optional useful certificate metadata:

certificate fingerprint
not_before
not_after
issuer CN
CT log ID

Only include these if CertSpotter already has them available without additional expensive parsing/copying.

---

Safe reachability checks

Make active network checks optional:

ACTIVE_PROBING=false

by default.

When enabled, only perform minimal non-destructive checks on infrastructure the operator is authorized to assess.

Allowed checks:

DNS resolution
TCP connect
TLS handshake
HTTP HEAD
HTTP GET /

No authentication attempts.

No password guessing.

No object downloading.

No mutation.

No PUT/POST/DELETE.

No database queries.

No exploitation.

No brute force.

For HTTP-based storage endpoints, use response status, headers, and provider error codes to classify:

reachable
not_found
auth_required
public_read
unknown

Do not claim something is "exposed" merely because its hostname exists.

---

Exposure state

Use an explicit enum:

type ExposureState uint8

const (
    ExposureUnknown ExposureState = iota
    ExposurePublicDNS
    ExposureReachable
    ExposureAuthRequired
    ExposurePublicRead
    ExposurePrivate
    ExposureNotFound
)

Keep factual observations separate from conclusions.

---

PostgreSQL

Use PostgreSQL with:

pgx/v5
pgxpool

Do not write one row per network event synchronously.

Use an asynchronous bounded DB writer with batching.

Use COPY FROM or efficient pgx batches where appropriate.

Suggested schema:

cloud_assets
------------
id
provider
service
hostname
provider_hostname
resource_identifier
region
resource_specific
public_semantic
first_seen
last_seen
exposure_state

UNIQUE(provider, service, provider_hostname)

and:

observations
------------
asset_id
observed_at
source_hostname
source_type
cname_chain
ipv4
ipv6
cert_fingerprint
ct_log_id

Exact schema may be improved if benchmarking/query requirements justify it.

Do not store observations that provide no useful information.

Avoid unnecessary JSONB if typed PostgreSQL arrays/columns are more efficient.

---

Checkpoints

Separate CT processing state from cloud-asset data.

Example:

ct_checkpoints
--------------
log_id PRIMARY KEY
state
updated_at

Persist only enough to resume monitoring reliably.

Batch/debounce checkpoint writes rather than writing on every certificate if CertSpotter semantics permit.

Correctness must be preserved.

---

Logging

Logging must never be on the certificate hot path.

Do not log every certificate.

Do not log every SAN.

Do not log every rejected hostname.

Log:

startup
shutdown
fatal errors
persistent DB errors
periodic aggregated statistics

Default periodic stats interval:

60 seconds

Use structured logging but keep it lightweight.

---

Metrics

Expose lightweight Prometheus-compatible metrics on an optional HTTP listener.

At minimum:

ct_certificates_total
ct_identifiers_total

classifier_direct_matches_total
classifier_cname_matches_total

dns_queries_total
dns_cache_hits_total
dns_errors_total

candidates_total

db_rows_total
db_errors_total

queue_depth
queue_drops_total

process memory
goroutines

Do not add expensive high-cardinality labels.

Never label metrics by hostname.

---

Health

Provide:

GET /healthz
GET /readyz

readyz should verify:

CT monitor initialized
database reachable
critical worker pipelines running

---

Configuration

Environment variables only.

Suggested configuration:

DATABASE_URL=

DNS_WORKERS=
DNS_TIMEOUT=
DNS_CACHE_SIZE=
DNS_MAX_CNAME_DEPTH=

DIRECT_QUEUE_SIZE=
DNS_QUEUE_SIZE=
DB_QUEUE_SIZE=

DB_BATCH_SIZE=
DB_FLUSH_INTERVAL=

ACTIVE_PROBING=false
PROBE_WORKERS=
PROBE_TIMEOUT=

METRICS_ADDR=:9090
LOG_LEVEL=info
STATS_INTERVAL=60s

Provide sensible defaults.

Validate configuration on startup.

---

Docker

Create a production Docker setup.

Use a multi-stage Dockerfile.

Build a static/minimal Go binary where possible.

Final image should be minimal, for example distroless or equivalent.

Run as non-root.

Requirements:

read-only root filesystem
no application volume
no CertSpotter volume
no certificate storage
no writable application directory

Use:

tmpfs

only if absolutely unavoidable; preferably the application should not require it.

Provide:

Dockerfile
docker-compose.yml

Docker Compose should contain:

ctcloud
postgres

The application itself must remain stateless apart from PostgreSQL.

---

Code organization

Keep the project simple.

Suggested layout:

cmd/ctcloud/
    main.go

internal/
    certspotter/
    classify/
        classify.go
        aws.go
        azure.go
        gcp.go
        cloudflare.go
        digitalocean.go
        supabase.go
        oracle.go

    dns/
    probe/
    pipeline/
    store/
    metrics/
    config/
    model/

migrations/
Dockerfile
docker-compose.yml
README.md
ARCHITECTURE.md

If CertSpotter is vendored/forked, keep upstream changes clearly separated.

Do not introduce microservices.

This should be one Go application process plus PostgreSQL.

No Kafka.

No Redis.

No RabbitMQ.

No external queue.

No Elasticsearch.

No unnecessary infrastructure.

---

Performance strategy

Optimize in this order:

1. eliminate I/O from hot path
2. eliminate serialization
3. eliminate process boundaries
4. eliminate allocations
5. cheap provider rejection
6. cheap suffix/label matching
7. bounded worker pools
8. DNS caching
9. deduplication before enrichment
10. batched DB writes

Do not prematurely implement complex SIMD/native libraries unless benchmarks demonstrate an actual bottleneck.

---

Benchmarks

Add benchmarks for classification.

Example:

func BenchmarkClassifyAWS(b *testing.B)
func BenchmarkClassifyAzure(b *testing.B)
func BenchmarkClassifyMixed(b *testing.B)
func BenchmarkRejectNormalDomain(b *testing.B)

Benchmark realistic input:

www.example.com
foo.bar.example.org
bucket.s3.eu-central-1.amazonaws.com
foo.blob.core.windows.net
db.example.supabase.co
project.firebaseio.com
search-example-abc.eu-west-1.es.amazonaws.com

Important metric:

allocs/op

Classifier target:

0 allocs/op

Do not make CI depend on an absolute nanosecond threshold because runner performance varies.

Report benchmark numbers in README.

---

Tests

Implement table-driven tests covering every provider format.

Test:

positive resource matches
generic provider endpoints rejected
uppercase handling
trailing dot
wildcards
invalid hostnames
very long hostnames
CNAME chains
CNAME loops
duplicate events
IPv4/IPv6
context cancellation
database retries
queue saturation

Particularly ensure these DO NOT produce resource findings:

storage.googleapis.com
s3.eu-central-1.amazonaws.com
redshift.eu-central-1.amazonaws.com
fra1.digitaloceanspaces.com

while:

bucket.storage.googleapis.com
bucket.s3.eu-central-1.amazonaws.com
cluster.id.eu-central-1.redshift.amazonaws.com
bucket.fra1.digitaloceanspaces.com

do.

---

Wildcards

Handle certificate wildcards carefully.

Example:

*.example.com

must not become a literal DNS query.

Do not interpret provider-wide wildcard certificates such as:

*.blob.core.windows.net

as discovered customer resources.

Track wildcard certificates separately or ignore them for resource-specific classification unless a concrete hostname can be derived from another SAN.

---

Graceful shutdown

Use contexts throughout.

On SIGTERM:

stop accepting new enrichment
finish bounded in-flight enrichment
flush DB batch
save CT checkpoint
close DB
exit

Shutdown must have a maximum timeout.

Docker should be able to stop the container cleanly.

---

Failure handling

The program must survive transient:

DNS failures
network timeouts
CT connection failures
PostgreSQL connection resets

with bounded exponential backoff and jitter.

Never retry in a tight loop.

If PostgreSQL remains unavailable and queues approach capacity, prefer controlled backpressure/load shedding instead of unbounded memory growth.

---

Security

Run as non-root.

No shell needed in final container.

No exposed debug endpoints.

Set HTTP server timeouts.

Validate all hostname lengths.

Treat all CT data as untrusted input.

Never execute certificate-controlled values.

Never construct shell commands.

Use parameterized SQL only.

---

README

Document:

what the application does
architecture
why CertSpotter is embedded/patched
why stdout/hooks/files are deliberately avoided
supported provider patterns
configuration
Docker startup
database schema
performance characteristics
benchmark results
limitations

Clearly explain:

A detected provider endpoint does NOT automatically mean
the underlying data is publicly accessible.

---

ARCHITECTURE.md

Include a concise data-flow diagram:

CT
 ↓
CertSpotter
 ↓
Identifier callback
 ↓
Direct classifier ───────────────┐
 ↓ non-match                     │
DNS/CNAME                        │
 ↓                               │
Provider classifier              │
 └───────────────┬───────────────┘
                 ↓
              dedupe
                 ↓
             enrichment
                 ↓
          optional probing
                 ↓
             DB batch
                 ↓
            PostgreSQL

Explain which operations are allowed in each stage.

---

Implementation priorities

Implement in this order:

Phase 1

CertSpotter integration
in-memory identifier callback
provider classifier
classification tests
benchmarks

Do not continue until the classifier has zero unnecessary allocations.

Phase 2

bounded pipeline
dedup
DNS resolver
CNAME following
DNS cache

Phase 3

PostgreSQL schema
batch writer
Postgres CT checkpoint state

Phase 4

optional safe probing
metrics
health endpoints

Phase 5

Docker hardening
benchmarks
documentation
integration tests

---

Important implementation rule

Do not blindly follow this design if inspection of the current CertSpotter source reveals a more direct or efficient integration point.

The goal is not to satisfy a predetermined class hierarchy.

The goal is:

reuse CertSpotter
+
remove output/storage overhead
+
receive parsed DNS identifiers directly
+
classify with essentially negligible CPU cost
+
only enrich potentially useful data
+
persist final useful results efficiently

If CertSpotter already exposes an internal abstraction that accomplishes this cleanly, use it rather than creating another layer.

---

Things you must NOT do

Do not:

write certificates to disk
write JSON events to disk
read JSON event files
call CertSpotter through exec.Command
parse CertSpotter stdout
use hooks
fork a process per certificate
use regex in the classifier
use shell pipelines
store every CT certificate
store every SAN
create unbounded goroutines
create unbounded queues
create unbounded maps
write each result individually to PostgreSQL
introduce Kafka/Redis just for queues
build multiple microservices

---

Definition of done

The project is complete when:

1. A single Dockerized Go process continuously consumes CT logs using CertSpotter internals.
2. CertSpotter never writes certificate/result files.
3. Parsed DNS identifiers are transferred directly into our classifier via Go calls.
4. The classifier covers AWS, Azure, GCP/Firebase, Cloudflare R2, DigitalOcean, Supabase and OCI patterns above.
5. Direct provider resources are detected immediately.
6. CNAME chains are asynchronously enriched and classified.
7. Duplicate processing is bounded and minimized.
8. Only useful resource findings reach PostgreSQL.
9. CT resume/checkpoint state lives in PostgreSQL rather than local files.
10. Classifier benchmarks demonstrate zero unnecessary allocations.
11. Resource usage remains bounded under overload.
12. Docker requires no application persistent volume.
13. go test ./... passes.
14. go test -race ./... passes.
15. go vet ./... passes.
16. Docker build succeeds.
17. The README explains exactly how to run and benchmark it.

Before declaring completion, inspect the resulting program under load and remove any unnecessary:

allocations
copies
serialization
logging
locking
filesystem access
goroutine creation
database round trips

Favor simple, measurable performance over architectural complexity.