Skip to content

Report a failing Ember suite instead of exiting the process - #660

Merged
tricknotes merged 1 commit into
mainfrom
fix-ember-test-exit
Sep 4, 2026
Merged

Report a failing Ember suite instead of exiting the process#660
tricknotes merged 1 commit into
mainfrom
fix-ember-test-exit

Conversation

@tricknotes

@tricknotes tricknotes commented Sep 4, 2026

Copy link
Copy Markdown
Owner

Summary

Shell#test ran ember test through Runner#run!, which calls exit when the command fails. App#test therefore could never return the false its success? call promises: any consumer running the suite from Ruby was terminated mid-run.

The gem's own RSpec suite is such a consumer — when the EmberCli::App#test example's ember test run fails (e.g. no Chrome available), the whole RSpec process exits at that point, reporting however many examples had run so far with 0 failures. Observed as example counts varying between 2 and 103 across runs of the same revision.

Changes:

  • Shell#test runs the suite with Runner#run and returns the status, so App#test reports the outcome to its caller
  • EmberCli.test! raises EmberCli::TestFailureError (naming the failing application) when a suite fails, so library consumers can rescue the failure in Ruby instead of having their process terminated; the exception still exits rake ember:test nonzero
  • Runner#run! keeps its exiting behavior for compile and install, whose rake tasks rely on it (its spec documents the SystemExit)
  • Add specs for Shell#test (both outcomes) and EmberCli.test! (runs every suite; raises on failure), and a CHANGELOG entry

Verification

  • spec/lib now runs to completion deterministically: the same example count on every run (previously the count varied per seed), with the one genuine failure on this machine being the Chrome-dependent App#test example — reported as a failure instead of aborting the suite
  • On CI, where Chrome is available, that example passes as before

@tricknotes
tricknotes force-pushed the fix-ember-test-exit branch 2 times, most recently from 43826fa to 1187595 Compare September 4, 2026 08:54
`Shell#test` ran `ember test` through `Runner#run!`,
which exits the calling process when the command fails.
`App#test` therefore could never return the `false` its `success?` call promises:
any consumer running the suite from Ruby — RSpec included — was terminated mid-run,
with an exit and a partial summary instead of a test failure.

Run the suite with `Runner#run` and return the status,
so `App#test` reports the outcome to its caller.
`EmberCli.test!` now raises `EmberCli::TestFailureError` when a suite fails,
so library consumers can rescue the failure in Ruby,
while the exception still exits `rake ember:test` nonzero.

`Runner#run!` keeps its exiting behavior for `compile` and `install`,
whose rake tasks rely on it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tricknotes
tricknotes merged commit fb52b91 into main Sep 4, 2026
12 checks passed
@tricknotes
tricknotes deleted the fix-ember-test-exit branch September 4, 2026 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants