Rename project to collectionparser
This commit is contained in:
StasonJatham committed
1 parent
997df90c21
commit
3205ab2311
26 files changed
+1285
-1420
No files matched your search
@@ -1,177 +1,125 @@
|
||||
# Collections 1, Collections 2-5 and AntiPublic Parser
|
||||
# collectionparser
|
||||
|
||||
Command line tools to manipulate the data from those multi-billion passwords collections.
|
||||
Command line tools to parse, deduplicate, and query multi-billion password
|
||||
collections (Collections 1-5, AntiPublic, Breach Compilation).
|
||||
|
||||
The full processing will take a couple of days and will generate a file structure that can be queried almost in o(1).
|
||||
Processing produces a deduped `a/b/c` bucket tree that answers exact lookups in
|
||||
O(1) and wildcard queries with a parallel ripgrep scan:
|
||||
|
||||
```
|
||||
$ <query> john.applessed@apple.com
|
||||
john.applessed@apple.com:toto123
|
||||
$ collectionparser query --dataset /path/to/data --pattern john.smith@gmail.com
|
||||
john.smith@gmail.com:toto123
|
||||
$ collectionparser query --dataset /path/to/data --pattern '*@gmail.com'
|
||||
```
|
||||
|
||||
The total number of unique records in the final dataset (Collection 1 to 5 + AntiPublic + Breach Compilation) is around 3.72 billions (_3,372,591,561_ to be precise).
|
||||
## Install
|
||||
|
||||
|
||||
### Set Up
|
||||
|
||||
Create a virtual environment and install the package.
|
||||
|
||||
```
|
||||
virtualenv -p python3 venv
|
||||
source venv/bin/activate
|
||||
make install
|
||||
```
|
||||
|
||||
## Extraction
|
||||
|
||||
https://superuser.com/questions/1308374/how-to-extract-all-tar-gz-files-present-in-multiple-folders-at-a-time-to-another
|
||||
Requires Python 3.11+ and [`ripgrep`](https://github.com/BurntSushi/ripgrep) (`rg`) for queries.
|
||||
|
||||
```bash
|
||||
find "$1" -name '*.tar.gz' -execdir tar -xzvf '{}' -C extracted \;
|
||||
find . -name "*.rar" -exec unrar x -o+ {} \;
|
||||
python -m venv venv && source venv/bin/activate
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
## Processing
|
||||
## Memory
|
||||
|
||||
Processing the Collection 1 is much faster than the Collections 2-5. The estimates for Collections 2-5 are reported below.
|
||||
Every command streams input and writes incrementally. Peak memory does not grow
|
||||
with file size: a single 100 GB+ file is parsed with a small constant footprint.
|
||||
`sort` and `clean` shell out to the OS `sort -u`, which spills to disk.
|
||||
|
||||
The parsing took around 20 hours on my server (CPU i7-8700K, 32GB of memory). I didn't have a large enough SSD to store all the temporary computations so everything
|
||||
was done on a standard HDD. A faster disk will surely make the processing faster.
|
||||
Measured on real Collections 2-5 data (19.94 GB, three files): **160 MB peak RSS**
|
||||
for the whole process tree, versus **15.4 GB** for the original parser. Min RAM
|
||||
available stayed at 15.3 GB, swap unchanged.
|
||||
|
||||
The sorting/removing duplicates step took 15 hours in total.
|
||||
## Build the query dataset
|
||||
|
||||
The splitting into the smaller files (this file struct makes every query almost instantaneous) took a couple of hours at most.
|
||||
|
||||
In total, expect around 2 days to process the Collections 2-5.
|
||||
`build` is the resumable, disk-bounded pipeline. It parses the source in batches,
|
||||
sorts+dedups each batch on a scratch disk, splits it into the bucket tree, then
|
||||
dedups each bucket. Peak disk is source + tree, so it fits where a global sort
|
||||
would not.
|
||||
|
||||
```bash
|
||||
breach parse --path /path/to/extracted --success_file success.txt --failure_file failure.txt --cython_acceleration
|
||||
rm -rf tmp && mkdir tmp # you need like 750GB in tmp/. By default /tmp/ is not enough for this!
|
||||
collectionparser build \
|
||||
--source /path/to/extracted \
|
||||
--dataset /path/to/dataset \
|
||||
--scratch /path/to/scratch \ # fast disk for batch files + sort temp
|
||||
--batch_gb 120
|
||||
```
|
||||
|
||||
**Pause / resume:** stop it any time (Ctrl-C or `systemctl stop`) and rerun the
|
||||
same command. Finished batches are recorded in `build_state.json`, and parsed
|
||||
files within the current batch in `progress_<n>.txt`; only unparsed files are
|
||||
re-parsed. The sort+split step of an interrupted batch is redone (duplicates are
|
||||
removed by the final dedup), so nothing is lost. Stats are written to
|
||||
`--scratch/stats.md`.
|
||||
|
||||
**Archives** (`.rar`/`.zip`/`.7z`/`.tar.*`/`.gz`/`.bz2`/`.xz`) are extracted
|
||||
automatically into `--scratch` for the duration of their batch and removed once
|
||||
parsed; the source is never modified. `.zip`/`.tar.*`/single-file `.gz`/`.bz2`/`.xz`
|
||||
use the Python standard library; `.rar`/`.7z` need `7z` or `unrar` on `PATH` (use
|
||||
`--no-extract` to skip archives entirely).
|
||||
|
||||
## Processing (manual)
|
||||
|
||||
```bash
|
||||
collectionparser parse --path /path/to/extracted --success_file success.txt --failure_file failure.txt
|
||||
|
||||
rm -rf tmp && mkdir tmp # sort needs room for temp files; /tmp is usually too small
|
||||
cat success.txt | pv -cN cut | sort -T tmp -u -S 90% --parallel=12 | pv -cN cut > success_sorted.txt
|
||||
breach split --file success_sorted.txt --out data
|
||||
|
||||
collectionparser split --file success_sorted.txt --out data
|
||||
```
|
||||
|
||||
## Converting the format of BreachCompilation to the new format
|
||||
## Merging datasets
|
||||
|
||||
The dataset is available here: https://github.com/philipperemy/tensorflow-1.4-billion-password-analysis
|
||||
|
||||
It's easy to convert the large `BreachCompilation` dataset to this format by running those commands.
|
||||
|
||||
Expect those commands to take some time (less than a day).
|
||||
`merge` is destructive: back up the destination first.
|
||||
|
||||
```bash
|
||||
find /path/to/BreachCompilation/ -type f -exec cat {} + > breach_compilation.txt
|
||||
rm -rf tmp && mkdir tmp # By default /tmp/ is not enough for this!
|
||||
cat breach_compilation.txt | pv -cN cut | sort -T tmp -u -S 90% --parallel=12 | pv -cN cut > breach_compilation_sorted.txt
|
||||
breach split --file breach_compilation_sorted.txt --out data_breach_compilation_sorted
|
||||
```
|
||||
|
||||
From there a simple `breach merge` will be enough to merge it to the Collections 1 & 2 to 5.
|
||||
|
||||
## Merging all the datasets together
|
||||
|
||||
Run the Collection 1 and Collections 2-5 through the processing step described above.
|
||||
|
||||
You will have two directories: `/path/to/collections1_data` and `/path/to/collections2_5_data`.
|
||||
|
||||
Additionally, if you have the other dataset BreachCompilation, you will have another directory `/path/to/data_breach_compilation_sorted`, generated by the step above.
|
||||
|
||||
The merge is destructive so it's better to create a copy of the output first and then merge each one into the output.
|
||||
|
||||
```
|
||||
cp -rf /path/to/collections1_data /path/to/big_dataset
|
||||
breach merge --src /path/to/collections2_5_data --dest /path/to/big_dataset
|
||||
breach merge --src /path/to/data_breach_compilation_sorted --dest /path/to/big_dataset
|
||||
collectionparser merge --src /path/to/collections2_5_data --dest /path/to/big_dataset
|
||||
```
|
||||
|
||||
## Usage
|
||||
## Query and export
|
||||
|
||||
The manual of the command line tool can be fetched by running `breach dumphelp`.
|
||||
```bash
|
||||
# exact (O(1) bucket lookup) or wildcard (* and ?):
|
||||
collectionparser query --dataset data --pattern 'john*'
|
||||
collectionparser query --dataset data --pattern '*@gmail.com'
|
||||
|
||||
# JSON Lines for OpenSearch / bulk ingest:
|
||||
collectionparser export --dataset data --out records.jsonl
|
||||
```
|
||||
Usage: cli [OPTIONS] COMMAND [ARGS]...
|
||||
|
||||
Options:
|
||||
--debug / --no-debug
|
||||
--help Show this message and exit.
|
||||
## Measured performance
|
||||
|
||||
Commands:
|
||||
chunk chunk large TXT files into smaller files.
|
||||
clean Cleans a query friendly folder PATH. Move incorrect records and
|
||||
sort the files.
|
||||
dumphelp
|
||||
evaluate Evaluates some metrics such as precision/recall (e.g. is OLD into
|
||||
NEW).
|
||||
merge Merges dataset SRC into dataset DEST.
|
||||
parse Parses an unstructured folder PATH of many files and generates two
|
||||
files: SUCCESS_FILE and FAILURE_FILE. All valid email:password
|
||||
will go to SUCCESS_FILE.
|
||||
sort Sorts a query friendly folder PATH. Target is itself.
|
||||
split Converts a large FILE to a query friendly folder OUT (e.g. a/b/c).
|
||||
Use RESTART_FROM to resume from the i-th line.
|
||||
test Infers passwords of a list of emails defined in FILE with a query
|
||||
friendly folder DATASET.
|
||||
Single large-file test on Collections 2-5 (3 files, 19.94 GB, 3 workers):
|
||||
|
||||
Usage: cli dumphelp [OPTIONS]
|
||||
| metric | value |
|
||||
| --- | --- |
|
||||
| parse throughput | 20.5 MB/s aggregate (6.8 MB/s per worker) |
|
||||
| valid pairs | 628,342,073 |
|
||||
| rejects | 3,321,666 |
|
||||
| peak RSS | 160 MB |
|
||||
| `sort -u` | ~125 MB/s |
|
||||
| `split` (sorted input) | ~43 MB/s |
|
||||
|
||||
Options:
|
||||
--help Show this message and exit.
|
||||
Split is ~12× slower on unsorted input (open-handle thrash); always sort first.
|
||||
|
||||
Usage: cli split [OPTIONS]
|
||||
## Commands
|
||||
|
||||
Options:
|
||||
--file FILE [required]
|
||||
--out DIRECTORY [required]
|
||||
--restart_from INTEGER [default: 0]
|
||||
--help Show this message and exit.
|
||||
Run `collectionparser --help` or `collectionparser <command> --help`.
|
||||
|
||||
Usage: cli chunk [OPTIONS]
|
||||
|
||||
Options:
|
||||
--path DIRECTORY [required]
|
||||
--size INTEGER [default: 50]
|
||||
--help Show this message and exit.
|
||||
|
||||
Usage: cli sort [OPTIONS]
|
||||
|
||||
Options:
|
||||
--path DIRECTORY [required]
|
||||
--help Show this message and exit.
|
||||
|
||||
Usage: cli clean [OPTIONS]
|
||||
|
||||
Options:
|
||||
--path DIRECTORY [required]
|
||||
--help Show this message and exit.
|
||||
|
||||
Usage: cli test [OPTIONS]
|
||||
|
||||
Options:
|
||||
--file FILE [required]
|
||||
--dataset [breach_compilation|collections_1|collections_2_5|all]
|
||||
[required]
|
||||
--help Show this message and exit.
|
||||
|
||||
Usage: cli parse [OPTIONS]
|
||||
|
||||
Options:
|
||||
--path DIRECTORY [required]
|
||||
--success_file FILE [required]
|
||||
--failure_file FILE [required]
|
||||
--cython_acceleration / --no-cython_acceleration
|
||||
--help Show this message and exit.
|
||||
|
||||
Usage: cli merge [OPTIONS]
|
||||
|
||||
Options:
|
||||
--src DIRECTORY [required]
|
||||
--dest DIRECTORY [required]
|
||||
--help Show this message and exit.
|
||||
|
||||
Usage: cli evaluate [OPTIONS]
|
||||
|
||||
Options:
|
||||
--old DIRECTORY [required]
|
||||
--new DIRECTORY [required]
|
||||
--help Show this message and exit.
|
||||
```
|
||||
| command | description |
|
||||
| --- | --- |
|
||||
| `build` | Batched, resumable: parse + dedup + split into the query tree. |
|
||||
| `query` | Exact or wildcard search over a dataset. |
|
||||
| `export` | Export a dataset as JSON Lines. |
|
||||
| `parse` | Parse an unstructured folder into `success`/`failure` files. |
|
||||
| `split` | Convert a large sorted file into a query-friendly `a/b/c` folder. |
|
||||
| `chunk` | Split large TXT files into smaller ones. |
|
||||
| `sort` | Sort and deduplicate every file in a query-friendly folder. |
|
||||
| `clean` | Move misplaced records to the right bucket and sort. |
|
||||
| `merge` | Merge one query-friendly folder into another. |
|
||||
| `test` | Look up passwords for the emails in a file. |
|
||||
| `evaluate` | Measure hit rate of one dataset against another. |
|
||||
| `filter-lines` | Keep lines whose length is within a range. |
|
||||
Reference in new issue
Block a user