executions fail-running reports success without committing

[lewisjared]

ref executions fail-running prints Successfully marked N execution(s) as failed and flagged their execution groups as dirty but leaves the database unchanged when the session already has a transaction open.

The write is wrapped in:

with session.begin_nested() if session.in_transaction() else session.begin():

begin_nested() opens a SAVEPOINT. Leaving the block releases the savepoint, it does not commit the enclosing transaction, so the changes are discarded when the session closes. The success message is printed unconditionally after the block, so there is no signal that nothing happened.

Reproduced against v0.16.2 on Postgres. Counting the affected rows before and after, twice in a row:

3336            # executions with successful IS NULL for this provider
Successfully marked 3336 execution(s) as failed and flagged their execution groups as dirty.
3336            # unchanged

This matters because fail-running is the documented escape hatch for exactly this situation. ExecutionGroup.should_run returns False while the last execution has successful IS NULL, so an execution abandoned by an OOM-killed worker blocks its group from ever being resolved again. fail_stale_in_progress_executions only reaps executions older than stale_after_seconds (6 hours by default), so anything more recent has no working recovery path and the operator is told the recovery succeeded.

A plain session.commit() after the loop, or committing the outer transaction when one was already open, would fix it.