provider capability model, on-demand scan/backfill, more providers
Readability capabilities (exists/object_readable/listable) are now tracked
separately instead of a single exposure flag, matching how providers differ:
- Azure 'Blob' mode = object readable, not listable; 'Container' = listable
- Cloudflare R2 public URLs are never anonymously listable (root 500); R2 is
object-read only and is no longer probed for listing
- S3/GCS: 200 list = listable; 403 + readable index.html/robots.txt =
object readable only
New columns: cloud_assets.listable, cloud_assets.object_readable.
Azure: account-level container listing is never anonymous; discover containers
by probing a bounded common-name set (or the container from a URL path), then
list blobs.
Classifier: added the legacy AWS '.s3-<region>.amazonaws.com' bucket form.
On-demand scan: POST /api/scan {target} classifies a host or URL, follows CNAME
for custom domains, probes, and indexes readable resources; Azure container
comes from the URL path. UI has a Scan box.
Backfill: POST /api/reprobe re-probes already-discovered assets with the current
checks and indexes any readable ones, recovering findings from earlier probes.
DNS: CNAME-only enrichment (A/AAAA resolved only for matches), 64 workers,
larger queue.
Verified live against real public resources:
veteran1.nyc3.digitaloceanspaces.com listable, entries
bucket-01.s3-eu-west-1.amazonaws.com listable, entries
tartugeo-podcast.storage.googleapis.com listable, entries
eroticfeeldev.blob.core.windows.net/images listable, entries
This commit is contained in:
1 parent
a366905a43
commit
62a7d2eb2e
23 files changed
+936
-165
No files matched your search
@@ -255,11 +255,13 @@ func (p *Pipeline) dnsWorker(ctx context.Context) {
|
||||
}
|
||||
|
||||
func (p *Pipeline) enrichHost(ctx context.Context, host string) {
|
||||
res, ok := p.cfg.Enricher.Resolve(ctx, host)
|
||||
// CNAME-only during enrichment: this is the high-volume path and A/AAAA is
|
||||
// not needed to classify the target. IPs are resolved only for matches.
|
||||
chain, ok := p.cfg.Enricher.ResolveCNAME(ctx, host)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
for _, target := range res.Chain[1:] {
|
||||
for _, target := range chain[1:] {
|
||||
m, matched := classify.Classify(target)
|
||||
if !matched || !m.ResourceSpecific {
|
||||
continue
|
||||
@@ -273,9 +275,11 @@ func (p *Pipeline) enrichHost(ctx context.Context, host string) {
|
||||
}
|
||||
f := Finding{
|
||||
Candidate: candidateFrom(target, host, m, model.SourceCNAME),
|
||||
CNAMEChain: res.Chain,
|
||||
CNAMEChain: chain,
|
||||
}
|
||||
if ips, ok := p.cfg.Enricher.ResolveIPs(ctx, target); ok {
|
||||
appendIPs(&f, ips)
|
||||
}
|
||||
appendIPs(&f, res.IPs)
|
||||
p.pushDB(ctx, f)
|
||||
}
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user