Test verbosity in output

[stickies-v]

With 316 test cases as of #14, the runner output is getting pretty verbose, and finding which tests failed increasingly hard. Perhaps by default only failed tests should be printed now?

Similarly, running make should maybe not show all the individual successful tests that are automatically run as part of the build process?

[stringintech]

Yes, the output was getting really noisy... Just pushed cf0e26f as the last commit for #14. Please let me know if this needs any further tweaks.

[stickies-v]

Perhaps worth doing in a separate PR? Have some minor thoughts, but now #14 is doing json schema, primitive type tests, and verbosity which are all orthogonal changes I think?

To avoid things looking stuck, printing the current test running on the latest line / updating it in-place would be quite nice UX (but not really important). Also, summarizing failed tests (just the name for greppability) right above the TOTAL SUMMARY would be useful, at all verbosity levels. Perhaps grouped by suite?

[stringintech]

That makes sense, and I think you’re right that these are cleaner as separate PRs. I’ll drop the last commit from this PR and address this issue in a follow-up.
I sometimes batch orthogonal changes together mainly to reduce back-and-forth around testing new releases, and to make better use of reviewer time. Perhaps I shouldn’t :)

[stringintech]

To avoid things looking stuck, printing the current test running on the latest line / updating it in-place would be quite nice UX (but not really important)

I also thought about that. I agree it would be nice UX, but I considered it a bit premature for now since the runner seems to be fast enough that there is not much waiting during execution.

Also, summarizing failed tests (just the name for greppability) right above the TOTAL SUMMARY

Do the following outputs look reasonable?

./runner --handler ./conformance-handler

=== Block Operations (block.json) ===
Covers block parsing, invalid-input rejection, hash and serialization getters, header access, transaction access by index, copy equality, and object cleanup using the mainnet genesis block
Total: 22, Passed: 2, Failed: 1, Skipped: 19
  ✗ block#3 (Creates a block from the full mainnet genesis block bytes)
      Invalid response: expected success with no error, but got error

=== Chain Operations (chain.json) ===
Sets up blocks, checks chain state, and verifies that the chain tip changes as expected after a reorg scenario
Total: 33, Passed: 5, Failed: 1, Skipped: 27
  ✗ chain#6 (Creates block 1 from raw bytes)
      Invalid response: expected success with no error, but got error


============================================================
FAILED TESTS
============================================================
block.json block#3
chain.json chain#6

============================================================
TOTAL SUMMARY
============================================================
Total Tests: 311
Passed:      263
Failed:      2
Skipped:     46
============================================================

./runner --handler ./conformance-handler -v

=== Block Operations (block.json) ===
Covers block parsing, invalid-input rejection, hash and serialization getters, header access, transaction access by index, copy equality, and object cleanup using the mainnet genesis block
Total: 22, Passed: 2, Failed: 1, Skipped: 19
  ✗ block#3 (Creates a block from the full mainnet genesis block bytes)
      Invalid response: expected success with no error, but got error

      Request chain
      ────────────────────────────────────────
{"id":"block#3","method":"btck_block_create","params":{"raw_block":"0100000000000000000000000000000000000000000000000000000000000000000000003ba3edfd7a7b12b27ac72c3e67768f617fc81bc3888a51323a9fb8aa4b1e5e4a29ab5f49ffff001d1dac2b7c0101000000010000000000000000000000000000000000000000000000000000000000000000ffffffff4d04ffff001d0104455468652054696d65732030332f4a616e2f32303039204368616e63656c6c6f72206f6e206272696e6b206f66207365636f6e64206261696c6f757420666f722062616e6b73ffffffff0100f2052a01000000434104678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5fac00000000"},"ref":"$block"}

      Received response
      ────────────────────────────────────────
      {"result":null,"error":{}}

      Expected response
      ────────────────────────────────────────
      {"result":{"ref":"$block"}}



=== Block Hash Operations (block_hash.json) ===
Covers block hash object creation from valid and invalid bytes, byte serialization, equality checks, copying, and cleanup
Total: 10, Passed: 10, Failed: 0, Skipped: 0

=== Block Header Operations (block_header.json) ===
Covers block header parsing, invalid-input rejection, hash comparison and serialization, field getters, copy equality, and object cleanup using the mainnet genesis block header
Total: 18, Passed: 18, Failed: 0, Skipped: 0

=== Chain Operations (chain.json) ===
Sets up blocks, checks chain state, and verifies that the chain tip changes as expected after a reorg scenario
Total: 33, Passed: 5, Failed: 1, Skipped: 27
  ✗ chain#6 (Creates block 1 from raw bytes)
      Invalid response: expected success with no error, but got error

      Request chain
      ────────────────────────────────────────
{"id":"chain#6","method":"btck_block_create","params":{"raw_block":"0000002006226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f35b99ed4e2e165de2ad77f1bba48049358c9bb740445f3c83ebdb3e83aa5bca8dbe5494dffff7f200000000001020000000001010000000000000000000000000000000000000000000000000000000000000000ffffffff025100feffffff0200f2052a010000001976a9142b4569203694fc997e13f2c0a1383b9e16c77a0d88ac0000000000000000266a24aa21a9ede2f61c3f71d1defd3fa999dfa36953755c690689799962b48bebd836974e8cf90120000000000000000000000000000000000000000000000000000000000000000000000000"},"ref":"$block_1"}

      Received response
      ────────────────────────────────────────
      {"result":null,"error":{}}

      Expected response
      ────────────────────────────────────────
      {"result":{"ref":"$block_1"}}



=== Script Pubkey Operations (script_pubkey.json) ===
Covers script pubkey object creation from raw bytes, byte serialization, copying, and cleanup
Total: 9, Passed: 9, Failed: 0, Skipped: 0

=== Script Verification - CLTV (script_verify_cltv.json) ===
Verifies a P2SH output with OP_CHECKLOCKTIMEVERIFY locked to block 100. Transaction with locktime=100 passes with P2SH + CHECKLOCKTIMEVERIFY and all pre-taproot flags; transaction with locktime=50 fails when CHECKLOCKTIMEVERIFY is enforced but passes with P2SH only.
Total: 14, Passed: 14, Failed: 0, Skipped: 0

=== Script Verification - CSV (script_verify_csv.json) ===
Verifies a P2SH output with OP_CHECKSEQUENCEVERIFY locked to sequence 10. Transaction with sequence=10 passes with P2SH + CHECKSEQUENCEVERIFY and all pre-taproot flags; transaction with sequence=5 fails when CHECKSEQUENCEVERIFY is enforced but passes with P2SH only.
Total: 14, Passed: 14, Failed: 0, Skipped: 0

=== Script Verification Error Cases (script_verify_errors.json) ===
Test cases where the verification operation fails to determine validity of the script due to bad user input
Total: 8, Passed: 8, Failed: 0, Skipped: 0

=== Script Verification - P2PKH (script_verify_p2pkh.json) ===
Verifies a mainnet P2PKH output against three transaction variants: valid sig (passes with no flags and all pre-taproot flags), corrupted sig (always fails), non-DER sig (passes without DERSIG, fails with it). Mainnet tx aca326a724eda9a461c10a876534ecd5ae7b27f10f26c3862fb996f80ea2d45d.
Total: 19, Passed: 19, Failed: 0, Skipped: 0

=== Script Verification - P2SH Multisig (script_verify_p2sh_multisig.json) ===
Verifies a mainnet P2SH 2-of-3 multisig output against three variants: valid sigs (passes with P2SH and all pre-taproot flags), corrupted sig (fails with P2SH, passes without it), non-null dummy element (passes with P2SH only, fails when NULLDUMMY is also set). Mainnet tx 3cd7f78499632d6f672d8a9412ae756b29c41342954c97846e0d153c7753a37e.
Total: 20, Passed: 20, Failed: 0, Skipped: 0

=== Script Verification - P2SH-P2WPKH (script_verify_p2sh_p2wpkh.json) ===
Verifies a mainnet P2SH-P2WPKH output against two variants: valid witness sig (passes with P2SH + WITNESS and all pre-taproot flags), corrupted witness sig (fails with WITNESS enforced, passes with P2SH only). Mainnet tx 07dea5918a500d7476b1d116d80507a66bc2167681b2e6ca7dd99dbc6d95c31d.
Total: 14, Passed: 14, Failed: 0, Skipped: 0

=== Script Verification - P2SH-P2WSH (script_verify_p2sh_p2wsh.json) ===
Verifies a mainnet P2SH-P2WSH (2-of-3 multisig) output against two variants: valid witness sigs (passes with P2SH + WITNESS and all pre-taproot flags), corrupted witness sig (fails with WITNESS enforced, passes with P2SH only). Mainnet tx 017be55761bf5a3920c73778810a6be4c3315dc6efa4f31b590bc3bc1da9d75f.
Total: 14, Passed: 14, Failed: 0, Skipped: 0

=== Script Verification - P2TR Key-Path (script_verify_p2tr_keypath.json) ===
Verifies a mainnet P2TR key-path spend. Requires one spent output for precomputed transaction data. Valid Schnorr sig passes with P2SH + WITNESS + TAPROOT and all flags; corrupted Schnorr sig fails when TAPROOT is enforced, passes with P2SH + WITNESS only. Mainnet tx 33e794d097969002ee05d336686fc03c9e15a597c1b9827669460fac98799036.
Total: 16, Passed: 16, Failed: 0, Skipped: 0

=== Script Verification - P2TR Script-Path (script_verify_p2tr_scriptpath.json) ===
Verifies a mainnet P2TR script-path spend at input index 1 of a two-input transaction. Requires two spent outputs for precomputed transaction data. Valid script-path witness passes with P2SH + WITNESS + TAPROOT and all flags; corrupted sig fails when TAPROOT is enforced, passes with P2SH + WITNESS only. Mainnet tx 1ba232a8bf936cf24155292c9a4330298278f572bacc78455eb68e3552197c30.
Total: 20, Passed: 20, Failed: 0, Skipped: 0

=== Script Verification - P2WPKH (script_verify_p2wpkh.json) ===
Verifies a mainnet native P2WPKH output with two amount values: correct amount (5003 sat) passes with P2SH + WITNESS and all pre-taproot flags; wrong amount (5002 sat) fails the witness commitment check when WITNESS is enforced, passes with P2SH only. Mainnet tx 00000000102d4e899ec7cc3656d91ab83aa8e95807dabb90fbe16a1a9e70b6ab.
Total: 10, Passed: 10, Failed: 0, Skipped: 0

=== Script Verification - P2WSH (script_verify_p2wsh.json) ===
Verifies a mainnet native P2WSH output at input index 1 of a two-input transaction. Valid HTLC-style witness script passes with P2SH + WITNESS and all pre-taproot flags; corrupted witness sig fails with WITNESS enforced, passes with P2SH only. Mainnet tx 12fc05be6778b06e77191e8fb18fee632b2d92efa0b6830e1cf63e28723a8b8f.
Total: 14, Passed: 14, Failed: 0, Skipped: 0

=== Transaction Operations (transaction.json) ===
Covers transaction parsing, invalid-input rejection, txid and serialization getters, input and output access by index, copying, and object cleanup
Total: 19, Passed: 19, Failed: 0, Skipped: 0

=== Transaction Input Operations (transaction_input.json) ===
Covers transaction input object access by index, out point getters, input copying, out point copying, and cleanup
Total: 15, Passed: 15, Failed: 0, Skipped: 0

=== Transaction Output Operations (transaction_output.json) ===
Covers transaction output object creation from a script pubkey and amount, amount and script getters, copying, and cleanup
Total: 10, Passed: 10, Failed: 0, Skipped: 0

=== Txid Operations (txid.json) ===
Covers txid objects returned from transactions, byte serialization, equality checks, copying, and cleanup
Total: 12, Passed: 12, Failed: 0, Skipped: 0


============================================================
FAILED TESTS
============================================================
block.json block#3
chain.json chain#6

============================================================
TOTAL SUMMARY
============================================================
Total Tests: 311
Passed:      263
Failed:      2
Skipped:     46
============================================================

[stickies-v]

Do the following outputs look reasonable?

Yeah I think that's great! Perhaps add the description too, that helps to spot patterns without much clutter?

============================================================
FAILED TESTS
============================================================
block.json block#3 (Creates a block from the full mainnet genesis block bytes)
chain.json chain#6 (Creates block 1 from raw bytes)

[stringintech]

Thanks a lot for the suggestions @stickies-v. Opened #17 to address them.