164 lines
5.9 KiB
Markdown
164 lines
5.9 KiB
Markdown
# collectionparser
|
||
|
||
Command line tools to parse, deduplicate, and query multi-billion password
|
||
collections (Collections 1-5, AntiPublic, Breach Compilation).
|
||
|
||
Processing produces a deduped `a/b/c` bucket tree that answers exact lookups in
|
||
O(1) and wildcard queries with a parallel ripgrep scan:
|
||
|
||
```
|
||
$ collectionparser query --dataset /path/to/data --pattern [email protected]
|
||
[email protected]:toto123
|
||
$ collectionparser query --dataset /path/to/data --pattern '*@gmail.com'
|
||
```
|
||
|
||
## Install
|
||
|
||
Requires Python 3.11+ and [`ripgrep`](https://github.com/BurntSushi/ripgrep) (`rg`) for queries.
|
||
|
||
```bash
|
||
python -m venv venv && source venv/bin/activate
|
||
pip install -e .
|
||
```
|
||
|
||
## Memory
|
||
|
||
Every command streams input and writes incrementally. Peak memory does not grow
|
||
with file size: a single 100 GB+ file is parsed with a small constant footprint.
|
||
`sort` and `clean` shell out to the OS `sort -u`, which spills to disk.
|
||
|
||
Measured on real Collections 2-5 data (19.94 GB, three files): **160 MB peak RSS**
|
||
for the whole process tree, versus **15.4 GB** for the original parser. Min RAM
|
||
available stayed at 15.3 GB, swap unchanged.
|
||
|
||
## Build the query dataset
|
||
|
||
`build` is the resumable, disk-bounded pipeline. It parses the source in batches,
|
||
sorts+dedups each batch on a scratch disk, splits it into the bucket tree, then
|
||
dedups each bucket. Peak disk is source + tree, so it fits where a global sort
|
||
would not.
|
||
|
||
```bash
|
||
# --scratch is a fast disk for per-batch files, sort temp, and build state.
|
||
# --batch_gb caps each batch (peak disk = source + tree).
|
||
collectionparser build \
|
||
--source /path/to/extracted \
|
||
--dataset /path/to/dataset \
|
||
--scratch /path/to/scratch \
|
||
--batch_gb 120
|
||
```
|
||
|
||
**Pause / resume:** stop it any time (Ctrl-C or `systemctl stop`) and rerun the
|
||
same command. Finished batches are recorded in `build_state.json`, and parsed
|
||
files within the current batch in `progress_<n>.txt`; only unparsed files are
|
||
re-parsed. The sort+split step of an interrupted batch is redone (duplicates are
|
||
removed by the final dedup), so nothing is lost.
|
||
|
||
State and stats live in `--scratch`: `build_state.json`, `stats.md`, and
|
||
`extract_failures.txt` (archives that failed to extract, never silently skipped).
|
||
`sort` temp is directed to `--scratch` via `TMPDIR`.
|
||
|
||
**Archives** (`.rar`/`.zip`/`.7z`/`.tar.*`/`.gz`/`.bz2`/`.xz`) are extracted
|
||
automatically into `--scratch` for the duration of their batch and removed once
|
||
parsed; the source is never modified. `.zip`/`.tar.*`/single-file
|
||
`.gz`/`.bz2`/`.xz` use the Python standard library; `.rar`/`.7z` need `7z` or
|
||
`unrar` on `PATH`. Pass `--no-extract` to skip archives, or `--no-dedup` to
|
||
skip the final per-bucket de-duplication.
|
||
|
||
## Processing (manual)
|
||
|
||
```bash
|
||
collectionparser parse --path /path/to/extracted --success_file success.txt --failure_file failure.txt
|
||
|
||
rm -rf tmp && mkdir tmp # sort needs room for temp files; /tmp is usually too small
|
||
cat success.txt | pv -cN cut | sort -T tmp -u -S 90% --parallel=12 | pv -cN cut > success_sorted.txt
|
||
|
||
collectionparser split --file success_sorted.txt --out data
|
||
```
|
||
|
||
## Merging datasets
|
||
|
||
`merge` is destructive: back up the destination first.
|
||
|
||
```bash
|
||
cp -rf /path/to/collections1_data /path/to/big_dataset
|
||
collectionparser merge --src /path/to/collections2_5_data --dest /path/to/big_dataset
|
||
```
|
||
|
||
## Query and export
|
||
|
||
```bash
|
||
# exact (O(1) bucket lookup) or wildcard (* and ?):
|
||
collectionparser query --dataset data --pattern 'john*'
|
||
collectionparser query --dataset data --pattern '*@gmail.com'
|
||
|
||
# JSON Lines for OpenSearch / bulk ingest:
|
||
collectionparser export --dataset data --out records.jsonl
|
||
```
|
||
|
||
## Crack hashed passwords
|
||
|
||
`parse`/`build` keep hashed passwords (MD5/SHA1/SHA256) separate from plaintext
|
||
ones instead of discarding them. `build` collects them in `--scratch/hashes.txt`;
|
||
`parse` writes them to `--hashes_file` when given.
|
||
|
||
```bash
|
||
# Wordlist omitted -> rockyou is downloaded automatically; add the dataset's
|
||
# own plaintext passwords as extra candidates:
|
||
collectionparser crack \
|
||
--hashes hashes.txt \
|
||
--passwords-from dataset/ \
|
||
--out cracked.txt
|
||
# Or bring your own wordlist(s):
|
||
collectionparser crack --hashes hashes.txt --wordlist words.txt --out cracked.txt
|
||
```
|
||
|
||
Pure standard library: `hashlib` digests with one process per CPU core, each
|
||
attacking a strided slice of the wordlist, so memory stays bounded; candidate
|
||
sources are merged through `sort -u`. Only unsalted MD5/SHA1/SHA256 are
|
||
attackable — bcrypt is ignored. Feed `cracked.txt` back into the index with
|
||
`split`/`merge`.
|
||
|
||
## Measured performance
|
||
|
||
Single large-file test on Collections 2-5 (3 files, 19.94 GB, 3 workers):
|
||
|
||
| metric | value |
|
||
| --- | --- |
|
||
| parse throughput | 20.5 MB/s aggregate (6.8 MB/s per worker) |
|
||
| valid pairs | 628,342,073 |
|
||
| rejects | 3,321,666 |
|
||
| peak RSS | 160 MB |
|
||
| `sort -u` | ~125 MB/s |
|
||
| `split` (sorted input) | ~43 MB/s |
|
||
|
||
Split is ~12× slower on unsorted input (open-handle thrash); always sort first.
|
||
|
||
## Commands
|
||
|
||
Run `collectionparser --help` or `collectionparser <command> --help`.
|
||
|
||
| command | description |
|
||
| --- | --- |
|
||
| `build` | Batched, resumable: parse + dedup + split into the query tree. |
|
||
| `query` | Exact or wildcard search over a dataset. |
|
||
| `export` | Export a dataset as JSON Lines. |
|
||
| `crack` | Dictionary-attack collected hashes with a wordlist. |
|
||
| `parse` | Parse an unstructured folder into `success`/`failure`/`hashes` files. |
|
||
| `split` | Convert a large sorted file into a query-friendly `a/b/c` folder. |
|
||
| `chunk` | Split large TXT files into smaller ones. |
|
||
| `sort` | Sort and deduplicate every file in a query-friendly folder. |
|
||
| `clean` | Move misplaced records to the right bucket and sort. |
|
||
| `merge` | Merge one query-friendly folder into another. |
|
||
| `test` | Look up passwords for the emails in a file. |
|
||
| `evaluate` | Measure hit rate of one dataset against another. |
|
||
| `filter-lines` | Keep lines whose length is within a range. |
|
||
|
||
## Development
|
||
|
||
```bash
|
||
pip install -e '.[dev]'
|
||
ruff check . # lint
|
||
pytest # tests
|
||
```
|