collectionparser/README.md

# collectionparser

Command line tools to parse, deduplicate, and query multi-billion password
collections (Collections 1-5, AntiPublic, Breach Compilation).

Processing produces a deduped `a/b/c` bucket tree that answers exact lookups in
O(1) and wildcard queries with a parallel ripgrep scan:

```
$ collectionparser query --dataset /path/to/data --pattern [email protected]
[email protected]:toto123
$ collectionparser query --dataset /path/to/data --pattern '*@gmail.com'
```

## Install

Requires Python 3.11+ and [`ripgrep`](https://github.com/BurntSushi/ripgrep) (`rg`) for queries.

```bash
python -m venv venv && source venv/bin/activate
pip install -e .
```

## Memory

Every command streams input and writes incrementally. Peak memory does not grow
with file size: a single 100 GB+ file is parsed with a small constant footprint.
`sort` and `clean` shell out to the OS `sort -u`, which spills to disk.

Measured on real Collections 2-5 data (19.94 GB, three files): **160 MB peak RSS**
for the whole process tree, versus **15.4 GB** for the original parser. Min RAM
available stayed at 15.3 GB, swap unchanged.

## Build the query dataset

`build` is the resumable, disk-bounded pipeline. It parses the source in batches,
sorts+dedups each batch on a scratch disk, splits it into the bucket tree, then
dedups each bucket. Peak disk is source + tree, so it fits where a global sort
would not.

```bash
collectionparser build \
  --source /path/to/extracted \
  --dataset /path/to/dataset \
  --scratch /path/to/scratch \   # fast disk for batch files + sort temp
  --batch_gb 120
```

**Pause / resume:** stop it any time (Ctrl-C or `systemctl stop`) and rerun the
same command. Finished batches are recorded in `build_state.json`, and parsed
files within the current batch in `progress_<n>.txt`; only unparsed files are
re-parsed. The sort+split step of an interrupted batch is redone (duplicates are
removed by the final dedup), so nothing is lost. Stats are written to
`--scratch/stats.md`.

**Archives** (`.rar`/`.zip`/`.7z`/`.tar.*`/`.gz`/`.bz2`/`.xz`) are extracted
automatically into `--scratch` for the duration of their batch and removed once
parsed; the source is never modified. `.zip`/`.tar.*`/single-file `.gz`/`.bz2`/`.xz`
use the Python standard library; `.rar`/`.7z` need `7z` or `unrar` on `PATH` (use
`--no-extract` to skip archives entirely).

## Processing (manual)

```bash
collectionparser parse --path /path/to/extracted --success_file success.txt --failure_file failure.txt

rm -rf tmp && mkdir tmp  # sort needs room for temp files; /tmp is usually too small
cat success.txt | pv -cN cut | sort -T tmp -u -S 90% --parallel=12 | pv -cN cut > success_sorted.txt

collectionparser split --file success_sorted.txt --out data
```

## Merging datasets

`merge` is destructive: back up the destination first.

```bash
cp -rf /path/to/collections1_data /path/to/big_dataset
collectionparser merge --src /path/to/collections2_5_data --dest /path/to/big_dataset
```

## Query and export

```bash
# exact (O(1) bucket lookup) or wildcard (* and ?):
collectionparser query --dataset data --pattern 'john*'
collectionparser query --dataset data --pattern '*@gmail.com'

# JSON Lines for OpenSearch / bulk ingest:
collectionparser export --dataset data --out records.jsonl
```

## Measured performance

Single large-file test on Collections 2-5 (3 files, 19.94 GB, 3 workers):

| metric | value |
| --- | --- |
| parse throughput | 20.5 MB/s aggregate (6.8 MB/s per worker) |
| valid pairs | 628,342,073 |
| rejects | 3,321,666 |
| peak RSS | 160 MB |
| `sort -u` | ~125 MB/s |
| `split` (sorted input) | ~43 MB/s |

Split is ~12× slower on unsorted input (open-handle thrash); always sort first.

## Commands

Run `collectionparser --help` or `collectionparser <command> --help`.

| command | description |
| --- | --- |
| `build` | Batched, resumable: parse + dedup + split into the query tree. |
| `query` | Exact or wildcard search over a dataset. |
| `export` | Export a dataset as JSON Lines. |
| `parse` | Parse an unstructured folder into `success`/`failure` files. |
| `split` | Convert a large sorted file into a query-friendly `a/b/c` folder. |
| `chunk` | Split large TXT files into smaller ones. |
| `sort` | Sort and deduplicate every file in a query-friendly folder. |
| `clean` | Move misplaced records to the right bucket and sort. |
| `merge` | Merge one query-friendly folder into another. |
| `test` | Look up passwords for the emails in a file. |
| `evaluate` | Measure hit rate of one dataset against another. |
| `filter-lines` | Keep lines whose length is within a range. |