126 lines
4.5 KiB
Markdown
126 lines
4.5 KiB
Markdown
# collectionparser
|
||
|
||
Command line tools to parse, deduplicate, and query multi-billion password
|
||
collections (Collections 1-5, AntiPublic, Breach Compilation).
|
||
|
||
Processing produces a deduped `a/b/c` bucket tree that answers exact lookups in
|
||
O(1) and wildcard queries with a parallel ripgrep scan:
|
||
|
||
```
|
||
$ collectionparser query --dataset /path/to/data --pattern [email protected]
|
||
[email protected]:toto123
|
||
$ collectionparser query --dataset /path/to/data --pattern '*@gmail.com'
|
||
```
|
||
|
||
## Install
|
||
|
||
Requires Python 3.11+ and [`ripgrep`](https://github.com/BurntSushi/ripgrep) (`rg`) for queries.
|
||
|
||
```bash
|
||
python -m venv venv && source venv/bin/activate
|
||
pip install -e .
|
||
```
|
||
|
||
## Memory
|
||
|
||
Every command streams input and writes incrementally. Peak memory does not grow
|
||
with file size: a single 100 GB+ file is parsed with a small constant footprint.
|
||
`sort` and `clean` shell out to the OS `sort -u`, which spills to disk.
|
||
|
||
Measured on real Collections 2-5 data (19.94 GB, three files): **160 MB peak RSS**
|
||
for the whole process tree, versus **15.4 GB** for the original parser. Min RAM
|
||
available stayed at 15.3 GB, swap unchanged.
|
||
|
||
## Build the query dataset
|
||
|
||
`build` is the resumable, disk-bounded pipeline. It parses the source in batches,
|
||
sorts+dedups each batch on a scratch disk, splits it into the bucket tree, then
|
||
dedups each bucket. Peak disk is source + tree, so it fits where a global sort
|
||
would not.
|
||
|
||
```bash
|
||
collectionparser build \
|
||
--source /path/to/extracted \
|
||
--dataset /path/to/dataset \
|
||
--scratch /path/to/scratch \ # fast disk for batch files + sort temp
|
||
--batch_gb 120
|
||
```
|
||
|
||
**Pause / resume:** stop it any time (Ctrl-C or `systemctl stop`) and rerun the
|
||
same command. Finished batches are recorded in `build_state.json`, and parsed
|
||
files within the current batch in `progress_<n>.txt`; only unparsed files are
|
||
re-parsed. The sort+split step of an interrupted batch is redone (duplicates are
|
||
removed by the final dedup), so nothing is lost. Stats are written to
|
||
`--scratch/stats.md`.
|
||
|
||
**Archives** (`.rar`/`.zip`/`.7z`/`.tar.*`/`.gz`/`.bz2`/`.xz`) are extracted
|
||
automatically into `--scratch` for the duration of their batch and removed once
|
||
parsed; the source is never modified. `.zip`/`.tar.*`/single-file `.gz`/`.bz2`/`.xz`
|
||
use the Python standard library; `.rar`/`.7z` need `7z` or `unrar` on `PATH` (use
|
||
`--no-extract` to skip archives entirely).
|
||
|
||
## Processing (manual)
|
||
|
||
```bash
|
||
collectionparser parse --path /path/to/extracted --success_file success.txt --failure_file failure.txt
|
||
|
||
rm -rf tmp && mkdir tmp # sort needs room for temp files; /tmp is usually too small
|
||
cat success.txt | pv -cN cut | sort -T tmp -u -S 90% --parallel=12 | pv -cN cut > success_sorted.txt
|
||
|
||
collectionparser split --file success_sorted.txt --out data
|
||
```
|
||
|
||
## Merging datasets
|
||
|
||
`merge` is destructive: back up the destination first.
|
||
|
||
```bash
|
||
cp -rf /path/to/collections1_data /path/to/big_dataset
|
||
collectionparser merge --src /path/to/collections2_5_data --dest /path/to/big_dataset
|
||
```
|
||
|
||
## Query and export
|
||
|
||
```bash
|
||
# exact (O(1) bucket lookup) or wildcard (* and ?):
|
||
collectionparser query --dataset data --pattern 'john*'
|
||
collectionparser query --dataset data --pattern '*@gmail.com'
|
||
|
||
# JSON Lines for OpenSearch / bulk ingest:
|
||
collectionparser export --dataset data --out records.jsonl
|
||
```
|
||
|
||
## Measured performance
|
||
|
||
Single large-file test on Collections 2-5 (3 files, 19.94 GB, 3 workers):
|
||
|
||
| metric | value |
|
||
| --- | --- |
|
||
| parse throughput | 20.5 MB/s aggregate (6.8 MB/s per worker) |
|
||
| valid pairs | 628,342,073 |
|
||
| rejects | 3,321,666 |
|
||
| peak RSS | 160 MB |
|
||
| `sort -u` | ~125 MB/s |
|
||
| `split` (sorted input) | ~43 MB/s |
|
||
|
||
Split is ~12× slower on unsorted input (open-handle thrash); always sort first.
|
||
|
||
## Commands
|
||
|
||
Run `collectionparser --help` or `collectionparser <command> --help`.
|
||
|
||
| command | description |
|
||
| --- | --- |
|
||
| `build` | Batched, resumable: parse + dedup + split into the query tree. |
|
||
| `query` | Exact or wildcard search over a dataset. |
|
||
| `export` | Export a dataset as JSON Lines. |
|
||
| `parse` | Parse an unstructured folder into `success`/`failure` files. |
|
||
| `split` | Convert a large sorted file into a query-friendly `a/b/c` folder. |
|
||
| `chunk` | Split large TXT files into smaller ones. |
|
||
| `sort` | Sort and deduplicate every file in a query-friendly folder. |
|
||
| `clean` | Move misplaced records to the right bucket and sort. |
|
||
| `merge` | Merge one query-friendly folder into another. |
|
||
| `test` | Look up passwords for the emails in a file. |
|
||
| `evaluate` | Measure hit rate of one dataset against another. |
|
||
| `filter-lines` | Keep lines whose length is within a range. |
|