provider capability model, on-demand scan/backfill, more providers
Readability capabilities (exists/object_readable/listable) are now tracked
separately instead of a single exposure flag, matching how providers differ:
- Azure 'Blob' mode = object readable, not listable; 'Container' = listable
- Cloudflare R2 public URLs are never anonymously listable (root 500); R2 is
object-read only and is no longer probed for listing
- S3/GCS: 200 list = listable; 403 + readable index.html/robots.txt =
object readable only
New columns: cloud_assets.listable, cloud_assets.object_readable.
Azure: account-level container listing is never anonymous; discover containers
by probing a bounded common-name set (or the container from a URL path), then
list blobs.
Classifier: added the legacy AWS '.s3-<region>.amazonaws.com' bucket form.
On-demand scan: POST /api/scan {target} classifies a host or URL, follows CNAME
for custom domains, probes, and indexes readable resources; Azure container
comes from the URL path. UI has a Scan box.
Backfill: POST /api/reprobe re-probes already-discovered assets with the current
checks and indexes any readable ones, recovering findings from earlier probes.
DNS: CNAME-only enrichment (A/AAAA resolved only for matches), 64 workers,
larger queue.
Verified live against real public resources:
veteran1.nyc3.digitaloceanspaces.com listable, entries
bucket-01.s3-eu-west-1.amazonaws.com listable, entries
tartugeo-podcast.storage.googleapis.com listable, entries
eroticfeeldev.blob.core.windows.net/images listable, entries
This commit is contained in:
1 parent
a366905a43
commit
62a7d2eb2e
23 files changed
+936
-165
No files matched your search
@@ -125,6 +125,7 @@ const (
|
||||
SourceDirectCT
|
||||
SourceCNAME
|
||||
SourceDNSAlias
|
||||
SourceManual
|
||||
)
|
||||
|
||||
func (s Source) String() string {
|
||||
@@ -135,6 +136,8 @@ func (s Source) String() string {
|
||||
return "cname"
|
||||
case SourceDNSAlias:
|
||||
return "dns_alias"
|
||||
case SourceManual:
|
||||
return "manual"
|
||||
default:
|
||||
return "unknown"
|
||||
}
|
||||
@@ -196,6 +199,38 @@ func (e ExposureState) String() string {
|
||||
}
|
||||
}
|
||||
|
||||
// ParseProvider maps a stored provider string back to its enum.
|
||||
func ParseProvider(s string) Provider {
|
||||
switch s {
|
||||
case "aws":
|
||||
return ProviderAWS
|
||||
case "azure":
|
||||
return ProviderAzure
|
||||
case "gcp":
|
||||
return ProviderGCP
|
||||
case "cloudflare":
|
||||
return ProviderCloudflare
|
||||
case "digitalocean":
|
||||
return ProviderDigitalOcean
|
||||
case "supabase":
|
||||
return ProviderSupabase
|
||||
case "oracle":
|
||||
return ProviderOracle
|
||||
default:
|
||||
return ProviderUnknown
|
||||
}
|
||||
}
|
||||
|
||||
// ParseService maps a stored service string back to its enum.
|
||||
func ParseService(s string) Service {
|
||||
for svc := ServiceUnknown + 1; svc <= ServiceWebApp; svc++ {
|
||||
if svc.String() == s {
|
||||
return svc
|
||||
}
|
||||
}
|
||||
return ServiceUnknown
|
||||
}
|
||||
|
||||
// Match is the compact result of classifying a single hostname. The string
|
||||
// fields are substrings of the input hostname (zero-copy).
|
||||
type Match struct {
|
||||
@@ -207,6 +242,15 @@ type Match struct {
|
||||
Resource string
|
||||
}
|
||||
|
||||
// ProbeTarget is a persisted asset selected for re-probing/backfill.
|
||||
type ProbeTarget struct {
|
||||
Provider string
|
||||
Service string
|
||||
Hostname string
|
||||
Resource string
|
||||
Region string
|
||||
}
|
||||
|
||||
// Entry is one indexed item of a readable resource (object key, container,
|
||||
// prefix, database index, root key). It is metadata only — never the content.
|
||||
type Entry struct {
|
||||
@@ -224,6 +268,7 @@ type Candidate struct {
|
||||
Original string // original CT hostname (equal to Hostname for direct matches)
|
||||
Region string
|
||||
Resource string
|
||||
Container string // optional storage container/share hint (Azure)
|
||||
Provider Provider
|
||||
Service Service
|
||||
Source Source
|
||||
|
||||
Reference in new issue
Block a user