discover Azure Blob containers by account name
Azure has two levels: the account maps to DNS (<account>.blob.core.windows.net) and anonymous access is per container with no anonymous container enumeration. Discovery now tests incoming labels as Azure account names (DNS existence) and, for accounts that exist, probes a small set of common containers, indexing any that list anonymously. CONTENT still never downloaded. Also document that docker compose down -v wipes all discovered assets; use stop/start to keep the pgdata volume. Live: connectsystem.blob.core.windows.net indexed (33 objects).
This commit is contained in:
1 parent
dc7bdae99d
commit
32035d0b9d
3 files changed
+111
-4
No files matched your search
@@ -150,7 +150,11 @@ docker compose up --build
|
||||
That starts `postgres` and the stateless `ctcloud` container. The application
|
||||
container runs as non-root, has a read-only root filesystem, drops all Linux
|
||||
capabilities, sets CPU/memory/pids limits and mounts no application volume. The
|
||||
only persistent state is PostgreSQL (CT checkpoints + cloud assets).
|
||||
only persistent state is PostgreSQL (CT checkpoints + cloud assets + index).
|
||||
|
||||
> **Findings accumulate over time.** Use `docker compose stop` / `docker compose
|
||||
> start` (or plain `up -d`) to keep the `pgdata` volume. `docker compose down -v`
|
||||
> deletes the volume and every discovered asset with it.
|
||||
|
||||
Endpoints on port `9090`, bound to `127.0.0.1` only:
|
||||
|
||||
@@ -324,9 +328,13 @@ engine that:
|
||||
3. indexes every bucket that returns 200 (keys/sizes/dates only).
|
||||
|
||||
It is aggressively bounded: a global `BUCKET_SCAN_QPS` limit, a 12h per-name
|
||||
dedup cache, and a random-sampled domain queue. Azure cannot be enumerated this
|
||||
way (account names are the namespace and containers need names), so Azure is
|
||||
handled from CT accounts plus a container wordlist, and R2 is never listable.
|
||||
dedup cache, and a random-sampled domain queue.
|
||||
|
||||
Azure is handled separately because it has two levels: the account name maps to
|
||||
DNS (`<account>.blob.core.windows.net`), and anonymous access is per container
|
||||
with no anonymous container enumeration — so for accounts that exist we probe a
|
||||
small set of common container names. R2 is never listable and is only probed for
|
||||
objects.
|
||||
|
||||
This is what actually yields open buckets. Of the buckets found, S3 and GCS
|
||||
return full object listings; Azure needs a known container; R2 only serves
|
||||
|
||||
Reference in new issue
Block a user