[high] Write finalized, postprocessed queries with --output-dir

[elhoim]

BLUF

• Problem: --output-dir writes the queries before they are finalized. The chosen --format, every query postprocessing item of the processing pipelines (embed, template, nest, ...) and the backend's per-query finalizer are skipped. Base rules of correlations that are not output (generate: false) are written too.
• Impact: the same rules give different queries with -o and -od. For example, a pipeline that embeds index=prod (...) scopes the query with -o, but the file written with -od holds the bare query that runs over all indexes. With tests/files/sigma_correlation_rules.yml, -o gives 3 queries and -od writes 6.
• Fix: convert the collection without the callback, then write the finalized conversion result (rule.get_conversion_result()) of each rule returned by rule_collection.get_output_rules().
• Tests: three new CliRunner tests (output format, pipeline postprocessing, non-output base rules). All three fail on main and pass with the fix. The full suite passes (121 passed, 1 skipped).

Priority: high

Details

Root cause: sigma/cli/convert.py:151-195 (regression from d0dfcdf, "Support correlation rules with --output-dir using callback mechanism").

write_separate_files() only keeps what the callback of backend.convert() receives. In pySigma, Backend.convert_rule() calls that callback right after finish_query() (sigma/conversion/base.py:274-275). Everything below happens after it:

• finalize_query() at lines 281-290, which runs finalize_query_<format> and the pipeline's postprocess_query()
• the if rule._output check at line 297

convert_correlation_rule() has the same order (callback at 744-745, finalize_query at 750-758). pySigma documents the callback as "called for each condition conversion", so it is meant to see intermediate results. This is how the CLI uses the API, not a pySigma bug.

After backend.convert() returns, pySigma has stored the finalized queries on each rule (rule.set_conversion_result(finalized_queries)). The fix reads those, and only for rules in get_output_rules(), which is the same set convert() returns queries for. Rules that were not converted (conversion failed) have no result and are skipped. get_output_rules(), get_conversion_result() and SigmaConversionError all exist in pySigma 1.3.0, the lower bound of the dependency pin.

Behavior change: base rules referenced by a correlation rule are no longer written unless they have generate: true. This matches -o. The comment in test_convert_output_dir_with_correlation_rules said files were expected "for base rules and correlation rules". I updated it.

Out of scope: the collection-level finalize_output_<format> and the pipeline finalizers (pipeline.finalize()) work on the whole rule set, not on a single query. They are still not applied per file. For example, -f savedsearches -od now writes the savedsearches stanza of each rule, but not the [default] header that the collection-level finalizer adds.

Overlap with open PRs:

• [high] Exit non-zero and keep converting when a rule fails with --output-dir #98 replaces the backend.convert(...) call with a per-rule convert_rule() loop that still uses the callback.
• [medium] Detect output-file collisions with --output-dir and count files honestly #100 changes the grouping and the write loop in the same function.

Both will conflict textually with this change, but they compose. With #98, the loop can read rule.get_conversion_result() after each convert_rule() instead of using the callback. Whichever PR lands second needs a mechanical rebase. The except Exception that swallows conversion errors is left as is here, because #98 handles it.

Testing

• tests/test_convert.py::test_convert_output_dir_applies_output_format: -f test -od writes [ ... ], the same as -f test to stdout.
• tests/test_convert.py::test_convert_output_dir_applies_pipeline_postprocessing: a pipeline with an embed postprocessing item wraps the written query.
• tests/test_convert.py::test_convert_output_dir_skips_non_output_correlation_base_rules: "Wrote 3 file(s)" instead of 6.
• All three fail on main and pass with the fix.
• The full suite passes locally with the CI command pytest --cov=sigma --cov-report term --cov-report xml:cov.xml -vv on Python 3.12: 121 passed, 1 skipped.

🤖 Generated with Claude Code