diff --git a/README.md b/README.md index 403e410..eda3354 100644 --- a/README.md +++ b/README.md @@ -129,7 +129,7 @@ repository. Examples under [examples/](examples/) use synthetic data. - Exit `2`: an error or incomplete processing; inspect `SUMMARY.txt` and `details/summary.json`. If the scan failed before presentation, inspect `summary.json` at the result root instead. -- Missing years/timezones are not guessed. See [README.txt](README.txt) for time configuration. +- Missing years/timezones are not guessed. See time configuration below. - Binary appliance logs may need an external decoder. Decode failures can also come from ordinary text. - Damaged inputs and regex timeouts are reported. Archive path traversal, unsafe links and exhausted quotas stop preparation. - Processing uses one worker, incremental reads and disk-backed SQLite. There is no hard process-wide RAM/CPU quota or automatic resume. @@ -138,6 +138,20 @@ repository. Examples under [examples/](examples/) use synthetic data. Tests were executed on macOS/ARM64. Windows and Linux instructions are provided, but those platforms have not yet been validated in the recorded test environment. +## Time configuration + +Use known device settings when logs omit their year or timezone: + +```sh +netscaler-triage scan --input ADC-A=./logs --year 2026 --timezone ADC-A=UTC +``` + +The appliance name, year and timezone above are examples. Explicit timestamps +are retained; ambiguous daylight-saving times remain unresolved. Use +`--time-config` for per-device settings and per-file year overrides; see +[examples/time.json](examples/time.json). The effective assumptions are recorded +in `details/configuration.json`. + ## Help and development ```sh @@ -154,7 +168,7 @@ uv run ruff format --check src tests typing_stubs uv build ``` -See [README.txt](README.txt) for the detailed plain-text manual and +See [RULES.txt](RULES.txt) for rule-pack semantics and [VALIDATION.txt](VALIDATION.txt) for test scope and known coverage limits. ## License diff --git a/README.txt b/README.txt deleted file mode 100644 index ca4d432..0000000 --- a/README.txt +++ /dev/null @@ -1,181 +0,0 @@ -NETSCALER TRIAGE - -Offline log triage for NetScaler ADC and Gateway. -Give it a log folder or archive. It prepares the input, searches for suspicious -patterns and public indicators, explains matching rules and decoded command text, -finds update/version records, and links related events on the same appliance. -Original files are not modified. Payloads are never executed or fetched. - -INSTALL - -Requires Python 3.14 or newer. From the unpacked source folder, on macOS/Linux: - python3 -m venv .venv - source .venv/bin/activate - -Or on Windows PowerShell: - py -m venv .venv - .venv\Scripts\Activate.ps1 - -Then install the local project: - python -m pip install . - -A built wheel can instead be installed with python -m pip install PATH_TO_WHEEL. -For a locked developer installation, use uv sync --locked with a suitable -installed Python interpreter. The lockfile records exact dependency versions. -Installation may need internet access; analysis itself runs offline. - -START HERE - - netscaler-triage logs.7z - -Or use an unpacked folder: - netscaler-triage ./logs - -Or run inside the folder containing your logs: - netscaler-triage - -ZIP, TAR, TAR.GZ/TGZ, GZIP and 7z are detected automatically, including nested -archives. Unsupported formats are reported. Several inputs can be combined: - netscaler-triage ./appliance-a ./appliance-b another-export.zip - -The result folder is created automatically and printed when the run finishes. ---output PATH chooses a new result folder. Existing results are never overwritten. -Appliance groups are inferred from folder structure; review the preparation -manifest when a combined export has an unusual layout. - -For an encrypted archive, request a hidden password prompt: - netscaler-triage logs.7z --password - ---password VALUE also works, but may expose the password in shell history or -process listings. The same password is used for nested archives. 7z, ZipCrypto -and AES-ZIP are supported. Passwords are not saved in results. - -WHAT TO READ FIRST - -SUMMARY.txt is the readable overview: status, counts, limitations and next steps. -FINDINGS.csv contains only rule/indicator matches, explanations, source references -and indicator statuses. A match is not proof of execution or compromise. -IOCS.csv contains raw search candidates and defanged display values, not a blocklist. - -Technical exports are grouped under details/: findings.jsonl (all selected events, -including context/errors), summary.json, updates.jsonl, correlations.jsonl, -coverage.json, input_manifest.json, configuration.json, indicator_hits.csv, -file_hash_hits.json, preparation_manifest.json and evidence.sqlite. -Prepared input copies remain in inputs/. Preparation-manifest paths are relative -to the result root. Decoding details retain originals, transformations and errors. - -FINDINGS.csv prefixes spreadsheet-sensitive text with an apostrophe; exact values -remain in JSON. Authorized-test/context/reference hits are explicitly labelled. -Direct engine.scan calls and failed/interrupted CLI scans retain flat exports. - -ADD YOUR OWN RULES OR IOCS - -Bundled public references are loaded automatically. They are a dated, attributed -collection, not a claim to cover every known attack. No private case addresses, -appliance identities or environment-specific assumptions are included in defaults. -References are not downloaded during a scan. - - netscaler-triage ./logs --config examples/rules.yml - netscaler-triage ./logs --ioc-file indicators.txt - netscaler-triage ./logs --pattern "(?i)your_search_expression" - -Repeat --config or --ioc-file to combine packs. An explicit --config directory -loads supported packs recursively in sorted relative-path order; no private or -customer directory is loaded unless you select it. - - netscaler-triage ./logs --config ./my-rules - Sigma YAML is the primary rule -format, parsed with pySigma and evaluated locally with the supported subset in -RULES.txt. Unsupported features are rejected explicitly. Defaults include 47 -Sigma rules and 246 deduplicated public reference indicators. -THREAT_INTELLIGENCE.txt indexes all 52 sources, uses and limitations; -config/threat_intel.json is the complete machine-readable inventory. - -JSON, CSV and one reference per line in TXT remain supported. TXT infers clear -IP addresses, SHA-256 hashes, HTTP(S) URLs and absolute Unix paths; other text -stays literal. Explicit CSV/JSON types are unchanged. An exported IOCS.csv -can be loaded again. Examples contain synthetic values, not real intelligence. -Keep local case rules under private/, which is gitignored and excluded from -release builds; load them explicitly with --config private/your_rules.yml. -Private packs are never discovered or enabled automatically. -Example files are included in the source release. -Rule packs are data files. Python plugin code is not loaded from them. - -TIME AND DEVICE IDENTITY - -Explicit timestamps are retained. A missing year or timezone is not guessed. -For known time settings and explicit appliance names on prepared folders: - netscaler-triage scan --input ADC-A=./logs --year 2026 --timezone ADC-A=UTC - -Use your actual device settings; UTC and the year above are examples only. -Ambiguous daylight-saving times remain unresolved. Time assumptions and per-file -year overrides can be supplied using --time-config; see examples/time.json. - -LIMITS AND INCOMPLETE RESULTS - -Exit 0 means the configured processing completed, not that the appliance is clean. -Exit 2 means an error or incomplete processing; inspect SUMMARY.txt and -details/summary.json. If the scan failed before presentation, inspect -summary.json at the result root instead. -Invalid decoding candidates may be ordinary text, not malicious payloads. -Unknown text formats remain searchable, even if no timestamp is parsed. -Proprietary binary logs require an appropriate external decoder. - -Unreadable files, damaged archives and regex timeouts are recorded while other -valid inputs continue. Path traversal, unsafe links and exhausted archive quotas -stop preparation. A controlled interruption preserves the processed database -where possible. There is no automatic resume after interruption. - -A single worker reads lines incrementally. SQLite stores events and large sorts -on disk; small decoding caches are bounded. One CPU core can be busy. This is not -a hard process-wide RAM or CPU quota: archive metadata, large rule lists and file -inventories need additional memory. Allow disk space for preparation and results. - -Defaults: 2 GiB preparation budget, 100,000 archive entries, nesting depth 3, -1 MiB per line and 512 MiB expanded per log file. Limits never mean a silent -successful scan. Advanced options can raise them for legitimate larger inputs. - -HELP AND OPTIONAL COMMANDS - - netscaler-triage --help - netscaler-triage scan --help-all - netscaler-triage extract logs.7z prepared - netscaler-triage decode payload.txt - netscaler-triage indicators text.txt - netscaler-triage inventory ./logs - -Short help shows common options. Advanced help includes resource limits, time -assumptions and --no-default-patterns, which disables built-in rules and bundled -public references; explicitly supplied packs still apply. - -DEVELOPMENT AND RELEASE - - uv run python -W error -m unittest discover -s tests - uv run mypy - uv run ruff check src tests typing_stubs - uv run ruff format --check src tests typing_stubs - uv build - -Synthetic tests cover parsing, archive safety, decoding, matching, correlations -and packaging hygiene. VALIDATION.txt records what was actually tested. Runtime -environments and analysis outputs are not release files. Source distributions -use an explicit allowlist to avoid bundling unrelated documents or case settings. - -LICENSE - -Project code is licensed under the MIT License; see LICENSE.txt. -Third-party dependencies and referenced external materials retain their own -licenses. The project does not relicense original external publications. - -VALIDATE PACKS BEFORE A SCAN - - netscaler-triage rules --config ./my-rules - -This offline command loads and compiles the bundled defaults and selected packs, -then prints the Sigma rule, regex rule and explicit active indicator counts. It -does not prepare or scan logs, contact external services or create artifacts. -Use --no-default-patterns to validate only the selected files; --json gives a -machine-readable result. Invalid Sigma, unsupported rules and conflicting rule -identifiers fail validation. Syntax validation does not prove detection quality. -Directory discovery skips README.txt, LICENSE.txt and CHANGELOG.txt documents -(case-insensitive); an explicitly named TXT file still loads as references using TXT type inference. diff --git a/pyproject.toml b/pyproject.toml index 8b6ae25..773ef6a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -27,7 +27,7 @@ netscaler-triage = "netscaler_triage.cli:main" packages = ["src/netscaler_triage"] [tool.hatch.build.targets.sdist] -include = ["src/", "tests/", "typing_stubs/", "examples/", "config/threat_intel.json", "README.md", "README.txt", "RULES.txt", "VALIDATION.txt", "THREAT_INTELLIGENCE.txt", "LICENSE.txt", "uv.lock", ".python-version", ".gitignore", "pyproject.toml"] +include = ["src/", "tests/", "typing_stubs/", "examples/", "config/threat_intel.json", "README.md", "RULES.txt", "VALIDATION.txt", "THREAT_INTELLIGENCE.txt", "LICENSE.txt", "uv.lock", ".python-version", ".gitignore", "pyproject.toml"] [dependency-groups] dev = [ diff --git a/tests/test_release_hygiene.py b/tests/test_release_hygiene.py index d05b140..c3ec797 100644 --- a/tests/test_release_hygiene.py +++ b/tests/test_release_hygiene.py @@ -40,7 +40,6 @@ class ReleaseHygieneTests(unittest.TestCase): "tests/", "typing_stubs/", "examples/", - "README.txt", "README.md", "RULES.txt", "VALIDATION.txt",