collectionparser/README.md

![collectionparser](/collectionparser_logo.png)

# collectionparser

Command line tools to parse, deduplicate, and query multi-billion password
collections (Collections 1-5, AntiPublic, Breach Compilation).

> **Responsible use.** This project ships **no data**. Use it only on datasets
> you own or are authorized to process, for security research and defense.
> Do not commit datasets, hashes, or cracked credentials.

Processing produces a deduped `a/b/c` bucket tree that answers exact lookups in
O(1) and wildcard queries with a parallel ripgrep scan:

```
$ collectionparser query --dataset /path/to/data --pattern [email protected]
[email protected]:toto123
$ collectionparser query --dataset /path/to/data --pattern '*@gmail.com'
```

## Install

Requires Python 3.11+ and [`ripgrep`](https://github.com/BurntSushi/ripgrep) (`rg`) for queries.

```bash
python -m venv venv && source venv/bin/activate
pip install -e .
```

## Memory

Every command streams input and writes incrementally. Peak memory does not grow
with file size: a single 100 GB+ file is parsed with a small constant footprint.
`sort` shells out to the OS `sort -u`, which spills to disk.

- `crack` builds the hash index once and shares it with the workers by forking
  (copy-on-write) — one copy, not one per worker (verified: ~450 MB total for a
  1M-hash index across 4 workers).
- `query` streams wildcard matches line by line from ripgrep, so a broad pattern
  over a huge dataset stays bounded.

Measured on real Collections 2-5 data (19.94 GB, three files): **160 MB peak RSS**
for the whole process tree, versus **15.4 GB** for the original parser. Min RAM
available stayed at 15.3 GB, swap unchanged.

## Build the query dataset

`build` is the resumable, disk-bounded pipeline. It parses the source in batches,
sorts+dedups each batch on a scratch disk, splits it into the bucket tree, then
dedups each bucket. Peak disk is source + tree, so it fits where a global sort
would not.

```bash
# --scratch is a fast disk for per-batch files, sort temp, and build state.
# --batch_gb caps each batch (peak disk = source + tree).
collectionparser build \
  --source /path/to/extracted \
  --dataset /path/to/dataset \
  --scratch /path/to/scratch \
  --batch_gb 120
```

**Pause / resume:** stop it any time (Ctrl-C or `systemctl stop`) and rerun the
same command. Finished batches are recorded in `build_state.json`, and parsed
files within the current batch in `progress_<n>.txt`; only unparsed files are
re-parsed (resume is **per file**, not per line). The sort+split step of an
interrupted batch is redone (duplicates are removed by the final dedup), so
nothing is lost.

**Progress:** one machine-readable command.

```bash
collectionparser status --scratch /path/to/scratch            # one-line JSON
collectionparser status --scratch /path/to/scratch --pretty
```

State and stats live in `--scratch`: `build_state.json`, `stats.md`, and
`extract_failures.txt` (archives that failed to extract, never silently skipped).
`sort` temp is directed to `--scratch` via `TMPDIR`.

**Archives** (`.rar`/`.zip`/`.7z`/`.tar.*`/`.gz`/`.bz2`/`.xz`) are extracted
automatically into `--scratch` for the duration of their batch and removed once
parsed; the source is never modified. `.zip`/`.tar.*`/single-file
`.gz`/`.bz2`/`.xz` use the Python standard library; `.rar`/`.7z` need `7z` or
`unrar` on `PATH`. Pass `--no-extract` to skip archives, or `--no-dedup` to
skip the final per-bucket de-duplication.

## Processing (manual)

```bash
collectionparser parse --path /path/to/extracted --success_file success.txt --failure_file failure.txt

rm -rf tmp && mkdir tmp  # sort needs room for temp files; /tmp is usually too small
cat success.txt | pv -cN cut | sort -T tmp -u -S 90% --parallel=12 | pv -cN cut > success_sorted.txt

collectionparser split --file success_sorted.txt --out data
```

## Merging datasets

`merge` is destructive: back up the destination first.

```bash
cp -rf /path/to/collections1_data /path/to/big_dataset
collectionparser merge --src /path/to/collections2_5_data --dest /path/to/big_dataset
```

## Query and export

```bash
# exact (O(1) bucket lookup) or wildcard (* and ?):
collectionparser query --dataset data --pattern 'john*'
collectionparser query --dataset data --pattern '*@gmail.com'

# JSON Lines for OpenSearch / bulk ingest:
collectionparser export --dataset data --out records.jsonl
```

## Crack hashed passwords

`parse`/`build` keep hashed passwords (MD5/SHA1/SHA256) separate from plaintext
ones instead of discarding them. `build` collects them in `--scratch/hashes.txt`;
`parse` writes them to `--hashes_file` when given.

```bash
# Wordlist omitted -> rockyou is downloaded automatically; add the dataset's
# own plaintext passwords as extra candidates:
collectionparser crack \
  --hashes hashes.txt \
  --passwords-from dataset/ \
  --out cracked.txt
# Or bring your own wordlist(s):
collectionparser crack --hashes hashes.txt --wordlist words.txt --out cracked.txt
```

Pure standard library: `hashlib` digests with one process per CPU core, each
attacking a strided slice of the wordlist, so memory stays bounded; candidate
sources are merged through `sort -u`. Only unsalted MD5/SHA1/SHA256 are
attackable — bcrypt is ignored. Feed `cracked.txt` back into the index with
`split`/`merge`.

The **target index is held in RAM** (roughly 250 bytes per hash), so a
multi-hundred-million-hash file will exhaust memory. Pass `--max-hashes N`
(e.g. `--max-hashes 8000000`) to stream the input in bounded chunks at the cost
of rescanning the wordlist per chunk.

## Measured performance

Single large-file test on Collections 2-5 (3 files, 19.94 GB, 3 workers):

| metric | value |
| --- | --- |
| parse throughput | 20.5 MB/s aggregate (6.8 MB/s per worker) |
| valid pairs | 628,342,073 |
| rejects | 3,321,666 |
| peak RSS | 160 MB |
| `sort -u` | ~125 MB/s |
| `split` (sorted input) | ~43 MB/s |

Split is ~12× slower on unsorted input (open-handle thrash); always sort first.

## Commands

Run `collectionparser --help` or `collectionparser <command> --help`.

| command | description |
| --- | --- |
| `build` | Batched, resumable: parse + dedup + split into the query tree. |
| `status` | Machine-readable build/dataset progress (JSON). |
| `query` | Exact or wildcard search over a dataset. |
| `export` | Export a dataset as JSON Lines. |
| `crack` | Dictionary-attack collected hashes with a wordlist. |
| `parse` | Parse an unstructured folder into `success`/`failure`/`hashes` files. |
| `split` | Convert a large sorted file into a query-friendly `a/b/c` folder. |
| `sort` | Sort and deduplicate every file in a query-friendly folder. |
| `merge` | Merge one query-friendly folder into another, then sort/dedup. |

## Credits

Derived from and inspired by
[philipperemy/3.7-billion-passwords-tools](https://github.com/philipperemy/3.7-billion-passwords-tools)
by [Philippe Remy](https://github.com/philipperemy) — the original parser and
`a/b/c` index. This project rewrites it for memory safety, resumable batched
builds, wildcard queries, hash collection/cracking and JSONL export.

Thanks, Philippe.

## Development

```bash
pip install -e '.[dev]'
ruff format . && ruff check .   # style + lint
pytest                          # tests
```