[medium] Detect output-file collisions with --output-dir and count files honestly

[elhoim]

BLUF

• Problem: with --output-dir, results that map to the same output file overwrite each other silently. "Wrote N file(s)" still counts every write. Results are also grouped by rule.id or rule.title, so two id-less rules that share a title get merged.
• Impact: sigma convert -t <backend> -od out/ rules/ loses queries without any error:
  • a multi-condition rule under the default {stem}.txt keeps only its last query and reports "Wrote 3 file(s)" for 1 file
  • linux/proc.yml and windows/proc.yml both end up as one proc.txt
• Fix: results are grouped per rule object.
  • Results from the same source file that share an output file are joined into it, the same way single-file output joins them.
  • Results from different source files (or binary results) that collide are reported and not overwritten, and the command exits 1.
  • "Wrote N" counts distinct files.
• Tests: three new CliRunner tests, one per case. All fail on main and pass with the fix. The existing --output-dir tests pass unchanged.

Priority: medium

Details

Root cause: sigma/cli/convert.py

• 173-187: results are grouped by rule.id or rule.title. Two different rules without id that share a title land in one group. They are written as one "multi-query rule" under the first rule's file name.
• 213-249: each result is written with output_path.write_bytes() without checking whether this run already wrote that path. files_written is increased on every call.
• 231-236: a rule with several queries renders {index}. The default template {stem}.txt has no {index}, so all queries render to the same name and only the last one survives.

The new write step works in two passes:

1. Compute the output path of every result.
2. Write each distinct path once:
   • All results for the path come from the same source file (a multi-condition rule, or several rules in one YAML file, both without {index}): they are joined into one file. Strings are joined with a blank line and dicts as JSON lines, as in the single-output mode. This keeps the existing behaviour of tests/files/multiple_rules/ and of the correlation test, and nothing is lost.
   • Results come from different source files, or the result is binary and cannot be joined: the first result is written, the others are reported as errors, and the command exits 1. The message suggests {path} / {index}.

Paths are compared with os.path.normcase(os.path.abspath(...)), so collisions that differ only in case are also caught on Windows.

Before (linux/proc.yml and windows/proc.yml):

$ sigma convert -t text_query_test -od out rules/
Wrote 2 file(s) to out
$ ls out
proc.txt

After:

Error: Output file 'out/proc.txt' is already used for rule 'Linux proc' (source: rules/linux/proc.yml). Not written: rule 'Windows proc' (source: rules/windows/proc.yml)
Wrote 1 file(s) to out
Error: 1 result(s) not written because their output file names collide. Use {path} and/or {index} in --output-filename-template to get distinct file names.

This PR is independent of the other --output-dir PR, which makes conversion errors fail the command. Both touch write_separate_files(). Whichever lands second, I'll rebase it.

Testing

• New tests in tests/test_convert.py, all failing on main:
  • test_convert_output_dir_multiple_queries_without_index
  • test_convert_output_dir_same_stem_different_sources (also checks that {path}/{stem}.txt writes both)
  • test_convert_output_dir_rules_without_id_same_title
• Full suite, run the way CI runs it (Python 3.12, dependencies pinned to poetry.lock: pySigma 1.4.0, click 8.4.2, pyparsing 3.3.2): pytest --cov=sigma → 121 passed, 1 skipped.

🤖 Generated with Claude Code