Simplify scan results with readable summary and matched-event CSV

README.md

@@ -57,21 +57,35 @@ directories; review the preparation manifest for unusual export layouts.
 
 ## Read the results
 
+Start with these three files:
+
+```text
+triage-results-.../
+  SUMMARY.txt       Start here
+  FINDINGS.csv      Review matched events
+  IOCS.csv          Search candidates for your SIEM
+  details/         Full evidence and machine-readable exports
+  inputs/          Prepared input copies, when preparation was needed
+```
+
 | File | Purpose |
 | --- | --- |
-| `summary.json` | Processing status, counts and incomplete coverage |
+| `SUMMARY.txt` | Readable overview: coverage, counts, limitations and what to open next |
-| `findings.jsonl` | Events, source lines, matching rules and decoding explanations |
+| `FINDINGS.csv` | Only rule/indicator matches, with explanations and original source references |
-| `IOCS.csv` | Exact SIEM search values and separate defanged display values |
+| `IOCS.csv` | SIEM search candidates; not an automatic blocklist |
-| `updates.jsonl` | Installation and software-version evidence |
-| `correlations.jsonl` | Related events on the same appliance; temporal candidates, not causal proof |
-| `coverage.json` | Per-file coverage and errors |
-| `input_manifest.json` | Original relative paths and SHA-256 checksums |
-| `preparation_manifest.json` | Archive preparation and appliance grouping |
 
-The output also includes an SQLite event index, effective configuration and
+All technical exports are under `details/`: full events in `findings.jsonl`,
-indicator/hash hit files. JSONL contains one JSON object per line. URL/Base64
+URL/Base64 decoding, updates, correlations, checksums, coverage and the SQLite
-decoding retains original values, transformation steps, checksums and errors.
+index. Prepared input copies, if needed, remain in `inputs/`.
-An installer command alone does not establish successful patch completion.
+
+`details/findings.jsonl` includes context and processing errors as well as matches;
+use `FINDINGS.csv` to browse matched events. A match is not proof of execution.
+The CSV keeps authorized-test and reference statuses visible. Spreadsheet-sensitive
+cells are prefixed with an apostrophe; exact original values remain in JSON.
+
+This layout applies to CLI scans. Direct Python calls to `engine.scan` retain
+flat machine-readable exports. Failed or interrupted scans can also retain the
+flat layout; inspect their `summary.json` before using partial results.
 
 ## Add indicators and Sigma rules
 
@@ -112,7 +126,9 @@ repository. Examples under [examples/](examples/) use synthetic data.
 ## Limits and interpretation
 
 - Exit `0`: configured processing completed; **not** a clean-system verdict.
-- Exit `2`: an error or incomplete processing; inspect `summary.json`.
+- Exit `2`: an error or incomplete processing; inspect `SUMMARY.txt` and
+  `details/summary.json`. If the scan failed before presentation, inspect
+  `summary.json` at the result root instead.
 - Missing years/timezones are not guessed. See [README.txt](README.txt) for time configuration.
 - Binary appliance logs may need an external decoder. Decode failures can also come from ordinary text.
 - Damaged inputs and regex timeouts are reported. Archive path traversal, unsafe links and exhausted quotas stop preparation.

README.txt

@@ -52,32 +52,21 @@ and AES-ZIP are supported. Passwords are not saved in results.
 
 WHAT TO READ FIRST
 
-summary.json tells you what was processed and whether anything was incomplete.
+SUMMARY.txt is the readable overview: status, counts, limitations and next steps.
-findings.jsonl contains selected events, original source/line references, matching
+FINDINGS.csv contains only rule/indicator matches, explanations, source references
-rule details, extracted indicators and safe decoding steps with explanations.
+and indicator statuses. A match is not proof of execution or compromise.
-Each line is a separate JSON object. Explanations describe what command text
+IOCS.csv contains raw search candidates and defanged display values, not a blocklist.
-could do; they do not establish that it ran successfully.
 
-IOCS.csv contains exact values for SIEM searches and a separate defanged display
+Technical exports are grouped under details/: findings.jsonl (all selected events,
-value. Observed values are candidates, not an automatic blocklist.
+including context/errors), summary.json, updates.jsonl, correlations.jsonl,
-updates.jsonl contains update/version evidence. An installer command is not by
+coverage.json, input_manifest.json, configuration.json, indicator_hits.csv,
-itself proof of a completed patch. correlations.jsonl contains temporal candidates
+file_hash_hits.json, preparation_manifest.json and evidence.sqlite.
-between log files on the same appliance, not proof of causation.
+Prepared input copies remain in inputs/. Preparation-manifest paths are relative
+to the result root. Decoding details retain originals, transformations and errors.
 
-Other outputs support verification:
+FINDINGS.csv prefixes spreadsheet-sensitive text with an apostrophe; exact values
-    evidence.sqlite             On-disk event index
+remain in JSON. Authorized-test/context/reference hits are explicitly labelled.
-    coverage.json               Per-file coverage and read/matching errors
+Direct engine.scan calls and failed/interrupted CLI scans retain flat exports.
-    input_manifest.json         Original relative paths and SHA-256 checksums
-    configuration.json          Effective rules, references and time assumptions
-    indicator_hits.csv          Matches against configured reference indicators
-    file_hash_hits.json         Reference matches against original file SHA-256
-    preparation_manifest.json   Preparation actions, groups and exclusions
-    inputs/                     Prepared inputs when automatic preparation runs
-
-URL/Base64 transformations retain the original value, transformation, parent
-step, decoded result or error, and checksum. Binary decoded content is displayed
-as hex and is not executed. Finding rule_details retain matching rule descriptions
-and source metadata; custom packs can provide their own explanations.
 
 ADD YOUR OWN RULES OR IOCS
 
@@ -125,7 +114,9 @@ year overrides can be supplied using --time-config; see examples/time.json.
 LIMITS AND INCOMPLETE RESULTS
 
 Exit 0 means the configured processing completed, not that the appliance is clean.
-Exit 2 means an error or incomplete processing; inspect summary.json.
+Exit 2 means an error or incomplete processing; inspect SUMMARY.txt and
+details/summary.json. If the scan failed before presentation, inspect
+summary.json at the result root instead.
 Invalid decoding candidates may be ordinary text, not malicious payloads.
 Unknown text formats remain searchable, even if no timestamp is parsed.
 Proprietary binary logs require an appropriate external decoder.

VALIDATION.txt

@@ -5,9 +5,9 @@ validation environment. Synthetic path checks do not replace target-OS tests.
 
 CODE AUDIT
 
-134 tests passed with Python warnings treated as errors in 36.475 seconds.
+136 tests passed with Python warnings treated as errors in 37.031 seconds.
-Strict mypy passed for 35 source files. Ruff lint and format checks passed for
+Strict mypy passed for 37 source files. Ruff lint and format checks passed for
-36 files. The enabled
+38 files. The enabled
 lint families are E, F, I, UP, B, SIM, RUF and PERF; this is not a claim that every
 optional rule from every analyzer is enabled.
 
@@ -131,3 +131,12 @@ A subsequent Unicode literal-lookup guard has synthetic regression coverage,
 but the entire private fixture set was not rerun after that guard. These checks
 validate known examples and important negative cases, not zero false positives
 or complete detection of all future attacks. Private evidence is not shipped.
+
+CLI RESULT PRESENTATION
+
+CLI scans produce SUMMARY.txt, matched-event FINDINGS.csv and IOCS.csv at the
+result root. Complete machine-readable evidence is retained under details/;
+prepared inputs remain under inputs/. Tests cover separation of context from
+matches, preservation of JSON evidence, authorized-test labels, spreadsheet
+formula protection and no-matches wording. Direct engine.scan callers retain
+the flat layout. A synthetic CLI smoke check confirmed the new directory layout.

src/netscaler_triage/cli.py

@@ -24,6 +24,7 @@ from .engine import dump, scan, validate_output
 from .indicators import extract_indicators
 from .ingest import extract_archive, inventory
 from .preparation import PreparationEntry, prepare_inputs
+from .presentation import present_results
 
 
 class Parser(argparse.ArgumentParser):
@@ -55,7 +56,8 @@ def display_result(result: object, json_output: bool = False) -> None:
             table.add_row(key.replace("_", " ").title(), str(value))
         console.print(table)
         console.print(
-            "Use --json for all fields. Review summary.json for coverage and warnings.", style="dim"
+            "Use --json for all fields. Start with SUMMARY.txt; evidence is under details/.",
+            style="dim",
         )
     else:
         console.print_json(data=result, ensure_ascii=False)
@@ -371,12 +373,16 @@ def main(argv: list[str] | None = None) -> int:
                 Console(stderr=True).print(
                     Text(
                         f"Preparation incomplete: {len(preparation_errors)} input error(s). "
-                        "Readable inputs were analyzed; review preparation_manifest.json.",
+                        "Readable inputs were analyzed; review details/preparation_manifest.json.",
                         style="yellow",
                     )
                 )
+            present_results(args.output, result)
             display_result(result, args.json)
             Console(stderr=True).print(Text(f"Results saved to: {os.path.relpath(args.output)}"))
+            Console(stderr=True).print(
+                "Start with SUMMARY.txt, then FINDINGS.csv. Evidence: details/."
+            )
             return 0 if result["status"] == "complete" else 2
         if args.command == "extract":
             if args.destination is None:

src/netscaler_triage/presentation.py

@@ -0,0 +1,116 @@
+"""A small reader-facing result set with full evidence retained under details/."""
+
+import csv
+import json
+from collections import Counter
+from pathlib import Path
+from typing import Any
+
+from .indicators import defang
+
+
+def cell(value: object) -> str:
+    """Prevent spreadsheet formula interpretation of untrusted log text."""
+    text = str(value if value is not None else "")
+    return "'" + text if text.lstrip().startswith(("=", "+", "-", "@")) else text
+
+
+def present_results(output: Path, summary: dict[str, Any]) -> None:
+    counts: Counter[str] = Counter()
+    with (
+        (output / "findings.jsonl").open(encoding="utf-8") as events,
+        (output / "FINDINGS.csv").open("x", encoding="utf-8", newline="") as target,
+    ):
+        writer = csv.writer(target)
+        writer.writerow(
+            [
+                "event_id",
+                "appliance",
+                "timestamp",
+                "source",
+                "line",
+                "rules",
+                "explanation",
+                "indicator_status",
+                "matched_indicators",
+                "matching_incomplete",
+                "evidence",
+            ]
+        )
+        for line in events:
+            event = json.loads(line)
+            rules = event.get("rules", [])
+            hits = event.get("indicator_matches", [])
+            if not rules and not hits:
+                continue
+            counts["matches"] += 1
+            statuses = sorted({str(hit.get("status", "unknown")) for hit in hits})
+            if "authorized_test" in statuses:
+                counts["authorized_test"] += 1
+            explanation = "; ".join(
+                str(rule.get("description", rule.get("title", "")))
+                for rule in event.get("rule_details", [])
+            )
+            writer.writerow(
+                cell(value)
+                for value in [
+                    event.get("id"),
+                    event.get("appliance"),
+                    event.get("timestamp"),
+                    event.get("source"),
+                    event.get("line"),
+                    "; ".join(rules),
+                    explanation,
+                    "; ".join(statuses),
+                    "; ".join(
+                        defang(str(hit.get("value", "")), str(hit.get("type", ""))) for hit in hits
+                    ),
+                    bool(event.get("analysis_errors")),
+                    f"details/findings.jsonl: event id {event.get('id')}",
+                ]
+            )
+    stats = summary.get("counts", {})
+    lines = [
+        "NETSCALER TRIAGE — START HERE",
+        "",
+        f"Processing status: {summary.get('status', 'unknown')}",
+        f"Records read: {stats.get('records', 0)}",
+        f"Events matching a rule or indicator: {counts['matches']}",
+        f"Matching events including an authorized-test indicator: {counts['authorized_test']}",
+        f"Update/version records: {stats.get('update_records', 0)}",
+        "",
+        "WHAT TO OPEN",
+        "",
+        "1. FINDINGS.csv — rule/indicator matches with explanations and original references.",
+        "   One row per matching event. Sort by appliance, rules and indicator_status.",
+        "2. IOCS.csv — search candidates for SIEM; not an automatic blocklist.",
+        "3. details/ — full JSON events, decoding, update records, correlations and checksums.",
+        "   details/findings.jsonl also retains context and processing-error events.",
+        "",
+        "HOW TO INTERPRET THIS",
+        "",
+        "A match is a lead, not proof that a command ran or that the appliance was compromised.",
+        "authorized_test/context_only/reference_only must not be treated as confirmed attackers.",
+        "An authorized-test indicator does not authorize every action in that event.",
+        "No matches does not prove the system is clean.",
+        "",
+        "COVERAGE",
+        "",
+        f"Read errors: {stats.get('read_errors', 0)}",
+        f"Matching errors: {stats.get('matching_errors', 0)}",
+        f"Decoding failures: {stats.get('decoding_errors', 0)}",
+        f"Inventory errors: {stats.get('inventory_errors', 0)}",
+        f"Preparation errors: {stats.get('preparation_errors', 0)}",
+        f"Original integrity: {summary.get('integrity', 'unknown')}",
+        "partial means some processing was incomplete; inspect details/summary.json",
+        "and details/coverage.json.",
+        "A decoding failure can come from ordinary text; it is not itself an attack.",
+        "Prepared input copies, if present, are under inputs/. Preparation paths are relative",
+        "to this result directory, including those in details/preparation_manifest.json.",
+    ]
+    (output / "SUMMARY.txt").write_text("\n".join(lines) + "\n", encoding="utf-8")
+    details = output / "details"
+    details.mkdir()
+    for path in sorted(output.iterdir()):
+        if path.is_file() and path.name not in {"SUMMARY.txt", "FINDINGS.csv", "IOCS.csv"}:
+            path.rename(details / path.name)

tests/test_cli.py

@@ -89,7 +89,9 @@ class CliTests(unittest.TestCase):
             self.assertEqual(len(destinations), 2)
             self.assertEqual((logs / "sample.log").read_text(), original)
             for destination in destinations:
-                config = json.loads((destination / "configuration.json").read_text())
+                config = json.loads((destination / "details/configuration.json").read_text())
+                self.assertTrue((destination / "SUMMARY.txt").is_file())
+                self.assertTrue((destination / "FINDINGS.csv").is_file())
                 self.assertIsNone(config["year"])
                 self.assertEqual(config["timezones"], {})
 
@@ -105,7 +107,7 @@ class CliTests(unittest.TestCase):
                 self.assertEqual(
                     main(["scan", "--input", f"ADC={logs}", "--output", str(root / "out")]), 0
                 )
-            config = json.loads((root / "out/configuration.json").read_text())
+            config = json.loads((root / "out/details/configuration.json").read_text())
             self.assertEqual(config["appliances"], ["ADC"])
 
 

tests/test_presentation.py

@@ -0,0 +1,49 @@
+import csv
+import json
+import tempfile
+import unittest
+from pathlib import Path
+
+from netscaler_triage.presentation import present_results
+
+
+class PresentationTests(unittest.TestCase):
+    def test_matches_are_separate_from_context_and_evidence_is_preserved(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            output = Path(temporary)
+            events = [
+                {"id": 1, "rules": [], "categories": ["update"]},
+                {"id": 2, "rules": [], "analysis_errors": ["timeout"]},
+                {
+                    "id": 3,
+                    "rules": ["probe"],
+                    "source": "=unsafe",
+                    "rule_details": [{"description": "A command probe, not execution."}],
+                    "indicator_matches": [{"status": "authorized_test"}],
+                },
+            ]
+            original = "".join(json.dumps(event) + "\n" for event in events)
+            (output / "findings.jsonl").write_text(original)
+            (output / "summary.json").write_text('{"status":"partial"}')
+            (output / "IOCS.csv").write_text("type,value\n")
+            (output / "inputs").mkdir()
+            present_results(output, {"status": "partial", "counts": {"records": 3}})
+            with (output / "FINDINGS.csv").open() as handle:
+                rows = list(csv.DictReader(handle))
+            self.assertEqual(len(rows), 1)
+            self.assertEqual(rows[0]["event_id"], "3")
+            self.assertEqual(rows[0]["source"], "'=unsafe")
+            self.assertEqual(rows[0]["indicator_status"], "authorized_test")
+            self.assertEqual((output / "details/findings.jsonl").read_text(), original)
+            self.assertIn("Processing status: partial", (output / "SUMMARY.txt").read_text())
+            self.assertEqual(
+                {path.name for path in output.iterdir()},
+                {"SUMMARY.txt", "FINDINGS.csv", "IOCS.csv", "details", "inputs"},
+            )
+
+    def test_no_matches_is_not_a_clean_verdict(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            output = Path(temporary)
+            (output / "findings.jsonl").write_text("")
+            present_results(output, {"status": "complete"})
+            self.assertIn("No matches does not prove", (output / "SUMMARY.txt").read_text())

tests/test_sigma_integration.py

@@ -66,7 +66,7 @@ class SigmaIntegrationTests(unittest.TestCase):
                     ]
                 )
             self.assertEqual(code, 0)
-            rows = findings(output)
+            rows = findings(output / "details")
             matches = [row for row in rows if row.get("rules")]
             self.assertEqual(len(matches), 1)
             self.assertEqual(matches[0]["line"], 1)