karl/collectionparser
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

mem safe & format

Browse Source
This commit is contained in:
StasonJatham committed 2026-09-19 14:48:59 +02:00
1 parent 2cc8e1d788
commit 8f92e64ffd
13 files changed
+813 -656
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

SKILL.md
+9 -15
View File
@@ -57,11 +57,8 @@ sort -T tmp -u -S 90% --parallel=12 success.txt > success_sorted.txt
# 3. Sorted unique file -> query-friendly a/b/c tree.
collectionparser split --file success_sorted.txt --out data
# 4. Optional repair pass (moves records to the right bucket, re-sorts).
collectionparser clean --path data
# 5. Query.
collectionparser test --file emails.txt --dataset data
# 4. Query.
collectionparser query --dataset data --pattern '[email protected]'
```
To add another dataset to an existing index:
@@ -79,15 +76,11 @@ collectionparser merge --src other_data --dest big_dataset
| `query` | Exact or wildcard (`*`, `?`) search over a dataset. | `--dataset`, `--pattern` |
| `export` | Export a dataset as JSON Lines (OpenSearch). | `--dataset`, `--out` |
| `crack` | Dictionary-attack collected MD5/SHA1/SHA256 hashes. | `--hashes`, `--wordlist`, `--out`, `--workers` |
| `parse` | Folder of raw files -> `success`/`failure` files. Streams. | `--path`, `--success_file`, `--failure_file` |
| `parse` | Folder of raw files -> `success`/`failure`/`hashes` files. Streams. | `--path`, `--success_file`, `--failure_file`, `--hashes_file` |
| `split` | One sorted file -> `a/b/c` tree. | `--file`, `--out`, `--restart_from N` |
| `chunk` | Split oversized TXT files. | `--path`, `--size MB` (default 50) |
| `sort` | `sort -u` every file in place. | `--path` |
| `clean` | Move misplaced records, then sort each file. | `--path` |
| `merge` | Merge `--src` into `--dest` (destructive). Runs `clean` after. | `--src`, `--dest` |
| `test` | Print passwords found for the emails in a file. | `--file`, `--dataset` |
| `evaluate` | Hit-rate of one index against another. | `--old`, `--new` |
| `filter-lines` | Keep lines within a length range. | `--file`, `--out`, `--less_than`, `--more_than` |
| `merge` | Merge `--src` into `--dest` (destructive), then sort/dedup. | `--src`, `--dest` |
| `status` | JSON build/dataset progress. | `--scratch`, `--pretty` |
## Rules an agent must follow
@@ -106,8 +99,8 @@ collectionparser merge --src other_data --dest big_dataset
- **`merge` is destructive**: it appends into `--dest` and deletes each `--src`
file. Back up `--dest` before running.
- **`--dataset` is a path** to a query-friendly folder built by `split`.
- **`query`/`test`/`evaluate` need `rg` on `PATH`.** If `rg` is missing, queries
return nothing; install ripgrep instead of working around it.
- **`query` needs `rg` on `PATH`.** If `rg` is missing, queries return nothing;
install ripgrep instead of working around it.
- **`query` wildcards are anchored to the email field**: `*`/`?` never cross the
`:` separator (`*@gmail.com` works, matching into the password does not).
- **`parse` restarts safely.** Completed inputs are recorded in `progress.txt`
@@ -126,7 +119,8 @@ collectionparser merge --src other_data --dest big_dataset
collectionparser --help
# round trip on toy data (~1s)
printf '[email protected]:pw\[email protected]:pw\n' > /tmp/raw.txt
collectionparser filter-lines --file /tmp/raw.txt --out /tmp/o.txt --less_than 1 --more_than 50
collectionparser split --file /tmp/raw.txt --out /tmp/tree
test -f /tmp/tree/a/symbols && echo OK
```
If that works, the package and its entry point are installed correctly.