Initial release: offline NetScaler triage with Sigma rules and MIT license

RULES.txt

@@ -0,0 +1,247 @@
+RULE PACKS AND IOC LISTS
+
+A pack is loaded with --config; repeat the option to combine packs. Matching is
+offline. YAML files are genuine Sigma detection rules parsed by official pySigma.
+The local evaluator supports the explicit subset below, not full Sigma semantics.
+JSON, CSV and TXT remain supported as described in LEGACY JSON PACKS below.
+
+Bundled data/public_sigma.yml provides 47 rules: 41 behavior rules and six IOC
+groups containing 246 attributed public references. These are not local findings.
+Use --no-default-patterns to disable bundled defaults. Private packs are opt-in;
+keep them in the ignored private/ directory and load them explicitly:
+
+netscaler-triage ./logs --config examples/rules.yml
+netscaler-triage ./logs --config private/rules.yml --config private/more-rules.yml
+
+SUPPORTED SIGMA SUBSET
+
+examples/rules.yml is a synthetic example with a UUID and provenance. Separate
+rules with ---. IDs, or titles when IDs are absent, must be unique within a pack.
+Use stable unique IDs across packs. Rules supply detections, not executable code.
+
+Required logsource product: netscaler. Optional service must equal the normalized
+event kind: http_access, http_error, syslog, shell_log or unknown. Omit service to
+include all kinds. Other products, categories, custom logsource attributes and
+custom taxonomies are rejected. A descriptive definition adds no filtering logic.
+
+Supported detection semantics:
+    Keyword lists: substring matching against raw; list entries use OR.
+    Field maps: fields use AND; values in a list normally use OR.
+    Strings: case-insensitive; * and ? are Sigma wildcards with standard literal
+    wildcard escaping. Field strings match the complete field value.
+    Modifiers: contains, startswith, endswith, re and all, including combinations
+    accepted by pySigma. all requires every listed value to match.
+    Regex: search semantics; case-sensitive unless inline flags such as (?i)
+    change it. The configured regex timeout applies.
+    Null: matches a missing or null field. Numeric/Boolean comparisons preserve
+    value types; integer 200 and string "200" differ.
+    Conditions: and, or, not, parentheses, 1 of and all of selector patterns or
+    them. A list of conditions is evaluated with OR.
+
+Unsupported rules fail at loading, without silent reinterpretation. Rejected
+constructs include aggregation, Sigma correlation rules, filters, collection
+actions, timeframe, backend pipelines, placeholders and modifiers such as cidr,
+exists, base64, fieldref or cased. This evaluator does not generate SIEM queries.
+The separate event correlation stage does not implement Sigma correlation rules.
+
+NORMALIZED FIELDS AND DECODED VIEWS
+
+Parsed fields are flattened, then top-level event metadata is overlaid. Names are
+case-insensitive. Common fields, available only when present in the input, are:
+    raw, kind, appliance, source, line, timestamp
+    source_ip, uri, user_agent, method, protocol, status, bytes, referer
+    session_id, transaction_id, request_id, context, return_code, command_status
+    sha256
+
+source is the relative log filename. src/client_ip alias source_ip; _raw/message
+alias raw. HTTP status is an integer. sha256 is a list of hashes of successfully
+transformed payload bytes; it is not evidence of execution. List-valued string
+fields match if any item matches. Other field names can be matched when supplied
+by a caller, but are not automatically extracted. Decoded SHA-1 and MD5 fields
+are not calculated by the engine. Missing fields are not invented.
+
+Sigma is evaluated once per event against its original view and available
+successfully decoded text views. Decoded views replace raw only, never uri,
+user_agent, source_ip or other structured fields. A rule can therefore combine
+decoded content with the original request context and explicit exclusions.
+
+rule_details preserves ID/name/title/description/references/level/falsepositives,
+source and basis. raw identifies the original event view, including structured
+fields. For decoded:N, sigma_decoded_views[N] identifies the corresponding entry
+in the event's decoding list. Structured-only matches are not attributed to
+unrelated decoded views. A matching view is not evidence that a payload ran.
+
+Optional custom indicators metadata contains type/value/status plus provenance.
+It is retained only as provenance, not as unconditional IOC checks. To activate
+an independent indicator, provide an explicit CSV, TXT or JSON indicator pack.
+This prevents catalog values from bypassing a Sigma condition. Catalog arrays
+are excluded from each hit's custom_attributes to avoid repeated large exports.
+Metadata does not replace the actual Sigma detection/condition definition.
+
+LIMITS AND COVERAGE
+
+Per YAML file: 4 MiB UTF-8 input, at most 256 documents, bounded token count,
+32 nesting levels and 10,000 nodes per compiled condition. Anchors, aliases,
+custom YAML tags and duplicate mapping keys are rejected. Unsupported rules
+cause a startup error. Per event: at most 64 decoded views and 1,048,576 characters
+per evaluated string field or decoded view, in addition to ingestion byte limits.
+Referenced list fields are limited to 10,000 elements. Unreferenced metadata is
+not evaluated. Sigma IDs must be unique across all selected packs.
+Timeouts and SigmaEvaluationLimit preserve the affected event, record incomplete
+matching, mark coverage partial and allow later records to be processed. They are
+neither negative findings nor proof of attacks.
+
+Custom metadata repeated per hit is limited to 16 KiB serialized JSON. Each rule
+may provide at most 10,000 catalog rows, each limited to 128 KiB; type/value strings
+are limited to 65,536 characters. Pack-size limits also apply.
+
+Sigma references:
+https://sigmahq.io/sigma-specification/specification/sigma-rules-specification.html
+https://sigmahq-pysigma.readthedocs.io/en/latest/reference/collection.html
+
+LEGACY JSON PACKS
+
+A UTF-8 JSON pack may contain patterns and indicators. Bundled defaults still
+apply unless explicitly disabled. The following address is documentation-only.
+
+Minimal example:
+
+{
+  "patterns": [
+    {
+      "id": "custom_download_command",
+      "pattern": "(?i)\\b(?:curl|wget)\\s+https?://",
+      "description": "A download command appears in the log. Verify its arguments, origin and whether it executed."
+    }
+  ],
+  "indicators": [
+    {"type": "ipv4", "value": "192.0.2.42", "status": "reference_only"}
+  ]
+}
+
+The address is a documentation example, not an attack indicator.
+A complete synthetic pack is available in examples/rules.json.
+
+PATTERNS
+
+patterns is a list of objects with pattern (a regex) and an optional stable id.
+description explains why a match may matter without claiming an attack succeeded.
+source/source_id/sources can identify provenance. Use distinct IDs across packs.
+Matched rule metadata is retained in rule_details, including matches after safe
+decoding. Keep rules specific: broad matches on ordinary IPs or login messages
+can create large, low-value result sets. Inline flags such as (?i) control case.
+Regex timeouts mean incomplete checks, not negative results.
+
+INDICATORS
+
+indicators is a list with type, value and optional status and source.
+Common types: ipv4, ipv6, domain, url, path, sha256 and literal. Use original values,
+not defanged display strings. IP boundaries are checked. SHA-256 values are checked
+against original file bytes and successfully decoded bytes, as well as log text.
+Other hash-shaped text can be extracted; file hashing computes SHA-256 only.
+
+Status records context; it does not suppress the source event:
+    reference_only       Published or supplied reference, not a local verdict
+    authorized_test      An explicitly identified permitted test
+    context_only         Environment/context value, not automatically hostile
+
+Document sources in a top-level sources list with id, title and url. Preserve
+dates, attribution and uncertainty. A reference match is a lead; establish its
+local role from the actual log fields.
+
+OTHER INPUT FORMATS
+
+TXT: one reference per line; blank lines and lines starting with # are ignored.
+Unambiguous IPv4/IPv6 addresses, 64-digit hexadecimal SHA-256 hashes, HTTP(S)
+URLs and whitespace-free absolute Unix paths receive their matching type. IP
+boundaries then prevent 192.0.2.1 from matching 192.0.2.10. Other text, including
+multiword commands, stays literal. CSV/JSON explicit types remain unchanged.
+CSV: type,value,status, or indicator_type,indicator,status. The tool's IOCS.csv
+output can be used directly. A legacy ip column is also accepted.
+Legacy rule CSV: ID,Muster, where Muster contains the regex expression.
+JSON: the pack object above, or a list of pattern objects.
+
+TIME CONFIGURATION
+
+--time-config accepts JSON with year, timezones and file_years. timezones maps
+explicit appliance labels to IANA timezones. file_years maps APPLIANCE/relative/log
+to a year when a specific yearless file needs an override. Example:
+
+{
+  "year": 2026,
+  "timezones": {"ADC-A": "UTC"},
+  "file_years": {"ADC-A/messages.previous": 2025}
+}
+
+These values are examples, not inferred settings. Explicit timestamps take
+precedence. Choose assumptions based on the actual evidence.
+
+PUBLIC INTELLIGENCE PROVENANCE
+
+The bundled threat_intel.json is the canonical inventory. Indicators are unique
+by type and normalized value, with stable IDs. source_ids/source_urls enumerate
+all reporting sources; observations retain each source's role, confidence, dates,
+original status and any MISP direction/to_ids conflicts. The source field can
+contain multiple source IDs after deduplication. Reference-only matching does
+not turn shared infrastructure, scanner markers or historical IOCs into confirmed
+attackers. context_only and observation_category retain those distinctions.
+
+Sources list retrieval attempts, requested URLs, access limitations and linked
+indicator/rule IDs. Excluded observables and external whole-file YARA libraries
+remain inventoried with reasons, without claiming equivalent Sigma coverage.
+THREAT_INTELLIGENCE.txt provides the readable index.
+
+Percent decoding, Base64 (including padding-free candidates) and nested layers
+feed both IOC and Sigma matching. Raw records remain unchanged. The candidate
+gate uses encoding evidence; arbitrary obfuscation is not exhaustively decoded.
+A plain indicator match, a decoded byte hash and a source-file hash are different
+evidence types. Do not interpret a textual pathname as proof that a file exists.
+
+ADDING RULES WITHOUT CHANGING CODE
+
+Create a directory such as my-rules and put Sigma YAML, JSON, CSV or TXT packs
+inside it. Load it explicitly:
+    netscaler-triage ./logs --config ./my-rules
+Repeat --config for additional files or directories. Directory discovery is
+recursive and sorted by relative filename; repeated identical files load once.
+Hidden entries, __pycache__ and node_modules directories are ignored. Symlinks
+are rejected. Unsupported file extensions inside a directory are ignored.
+An empty directory or a malformed supported pack produces a named error.
+Discovery is bounded to 64 files, 4096 directories and 32 nested levels. Each
+Sigma file is limited to 4 MiB; other formats to 16 MiB; the total is 32 MiB.
+No customer or private pack is activated automatically.
+
+For exact IP or path references, create my-rules/local.csv:
+    type,value,status,source,description
+    ipv4,192.0.2.9,authorized_test,local,Approved external test
+    path,/example/suspicious-file,reference_only,local,Investigate file context
+
+The addresses and paths above are synthetic examples. Extra CSV columns such as
+source, description, confidence and evidence are preserved verbatim. This also
+applies to legacy ID,Muster regex CSV files. Use JSON when metadata requires
+structured lists or nested objects. Keep reference values in their original
+form rather than defanged display form; matching does not establish compromise.
+
+VALIDATE PACKS BEFORE A SCAN
+
+    netscaler-triage rules --config ./my-rules
+
+This offline command loads and compiles the bundled defaults and selected packs,
+then prints the Sigma rule, regex rule and explicit active indicator counts. It
+does not prepare or scan logs, contact external services or create artifacts.
+Use --no-default-patterns to validate only the selected files; --json gives a
+machine-readable result. Invalid Sigma, unsupported rules and conflicting rule
+identifiers fail validation. Syntax validation does not prove detection quality.
+Directory discovery skips README.txt, LICENSE.txt and CHANGELOG.txt documents
+(case-insensitive); an explicitly named TXT file still loads as references using TXT type inference.
+
+TYPED INDICATOR MATCHING
+
+IP addresses and hashes require complete token boundaries. Domain indicators
+also match subdomains, but not concatenated names or unrelated suffixes. Paths
+are case-sensitive; a trailing slash denotes a directory prefix. URL schemes
+and authorities are case-insensitive, while paths and queries are exact.
+Typed indicators prevent incidental substrings from becoming matches. Use an
+explicit regex rule when different matching semantics are needed. A reference
+match or an authentication failure is not automatically malicious; preserve
+status and provenance and evaluate the surrounding event.