[high] Re-apply #99: report collection errors and broken filters in sigma check (reverted with #105)

[elhoim]

BLUF

• What: re-applies [high] Report collection errors and broken filters in sigma check #99 ("Report collection errors and broken filters in sigma check"). It was merged, but the revert cdd5436 that came in with [high] Write finalized, postprocessed queries with --output-dir #105 (merge 8effb4c) silently removed it from main.
• Problem it fixes (unchanged): sigma check --fail-on-error passes (exit 0, "Found 0 errors") on files that sigma convert rejects - unknown collection action: and invalid filters are never reported.
• Change: a clean cherry-pick of the original commit 0d0231f onto current main; [high] Write finalized, postprocessed queries with --output-dir #105 did not touch check.py, so there was no conflict.
• Tests: [high] Report collection errors and broken filters in sigma check #99's tests are back; 3 of them fail on current main and pass here. Full suite with the poetry.lock pins (pySigma 1.4.0, click 8.4.2, pyparsing 3.3.2, Python 3.12): 125 passed, 1 skipped.

Priority: high

What happened

main no longer contains #99, although it was merged:

• [high] Report collection errors and broken filters in sigma check #99 (47712d0) and [high] Exit non-zero and keep converting when a rule fails with --output-dir #98 (c8348d7) were merged on 2026-09-27 before [high] Write finalized, postprocessed queries with --output-dir #105.
• [high] Write finalized, postprocessed queries with --output-dir #105 then had conflicts with main. The resolution 86d3041 ("Merge github/main: resolve conflicts in convert.py and test_convert.py") brought [high] Exit non-zero and keep converting when a rule fails with --output-dir #98 and [high] Report collection errors and broken filters in sigma check #99 into the [high] Write finalized, postprocessed queries with --output-dir #105 branch, but it fails tests/test_convert.py::test_convert_output_dir_basic (reproduced locally).
• It was reverted six minutes later by cdd5436 ("Revert "Merge github/main: ...""), and [high] Write finalized, postprocessed queries with --output-dir #105 was merged with that revert (8effb4c). The revert removed the whole merged main side from the branch, so merging [high] Write finalized, postprocessed queries with --output-dir #105 took [high] Exit non-zero and keep converting when a rule fails with --output-dir #98's and [high] Report collection errors and broken filters in sigma check #99's changes back out of main: 8effb4c deletes the 22 lines [high] Report collection errors and broken filters in sigma check #99 added to sigma/cli/check.py and its 51 lines of tests, and restores the old --output-dir error handling.

Probably nobody intended to revert #98 and #99; they were collateral of undoing a broken conflict resolution.

Original description (#99)

BLUF

• Problem: sigma check only reads the errors attached to rules in SigmaCollection.rules. It ignores errors that belong to no rule, such as an unknown action: or an invalid filter (filters are stored in SigmaCollection.filters).
• Impact: sigma check --fail-on-error --fail-on-issues rules/ passes (exit 0, "Found 0 errors") on a file that sigma convert -t <backend> rules/ rejects with "Errors found in Sigma rules". A CI gate built on sigma check lets these files through.
• Fix: load_and_check_rules() now also reports the entries of SigmaCollection.errors that don't belong to a loaded rule. They count as rule errors, so --fail-on-error applies, and they are added to the --junitxml report. Per-rule errors, which pySigma also copies into SigmaCollection.errors, are still counted once.
• Tests: four new CliRunner tests: unknown action, invalid filter, JUnit output, and no double counting. The first three fail on main and pass with the fix.

Priority: high

Details

Root cause: sigma/cli/check.py:130-175. load_and_check_rules() loops over rule_collection.rules and reads rule.errors. It never reads rule_collection.errors, but sigma convert does, via check_rule_errors() (sigma/cli/rules.py:39-49). The two commands therefore disagree on:

• Collection-level errors. For example action: repaet raises SigmaCollectionError. No rule is created, so the error lives only in SigmaCollection.errors.
• Filter errors. A SigmaFilter is kept in SigmaCollection.filters, not in .rules, so its errors (for example an invalid id) are never seen.

pySigma also puts every per-rule error into SigmaCollection.errors (collection.py errors.extend(parsed_rule.errors)). The fix therefore skips errors that are already attached to a rule in .rules, compared by object identity. The count for tests/files/invalid stays at 6.

Before:

$ printf 'action: repaet\ntitle: typo in action keyword\n' > rules/typo.yml
$ sigma check --fail-on-error --fail-on-issues rules/
Found 0 errors, 0 condition errors and 0 issues.
$ echo $?
0

After:

=== Sigma Rule Errors ===
Unknown Sigma collection action 'repaet' in rule 1 in rules/typo.yml

=== Summary ===
Found 1 errors, 0 condition errors and 0 issues.
...
Check failure
$ echo $?
1

Testing

• New tests in tests/test_check.py:
  • test_check_unknown_collection_action (fails on main)
  • test_check_invalid_filter (fails on main)
  • test_check_collection_error_junitxml (fails on main)
  • test_check_rule_errors_not_double_counted (regression guard)
• Full suite, run the way CI runs it (Python 3.12, dependencies pinned to poetry.lock: pySigma 1.4.0, click 8.4.2, pyparsing 3.3.2): pytest --cov=sigma → 122 passed, 1 skipped.

🤖 Generated with Claude Code