Conversation
| ); | ||
| if !execution.success { | ||
| let setup_hint = if phase == "baseline" { | ||
| " A fresh parallel worktree has no existing build directory; use \ |
There was a problem hiding this comment.
Every baseline failure returns
this error
*** No errors detected
Traceback (most recent call last):
File "/tmp/bcore-mutation-workers-KSJzCG/worker-0/./build/test/functional/test_runner.py", line 962, in <module>
main()
File "/tmp/bcore-mutation-workers-KSJzCG/worker-0/./build/test/functional/test_runner.py", line 464, in main
os.makedirs(tmpdir)
File "/home/ubuntu/.pyenv/versions/3.10.14/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/tmp/test_runner_₿_🏃_20260817_212053'
Error: InvalidInput("baseline command failed in worker 0. A fresh parallel worktree has no existing build directory; use --setup-command if the test command does not configure it.")
analysis command had --setup-command, the error message is misleading
bcore-mutation analyze --sqlite mutation.db --run-id=2 --parallel 2 --file-path="src/net_processing.cpp" --setup-command 'cmake -B build && cmake --build build -j4 --target bitcoind test_bitcoin' -c 'cmake --build build -j4 --target bitcoind test_bitcoin && ./build/bin/test_bitcoin --run_test=denialofservice_tests,peerman_tests,net_tests,txdownload_tests && ./build/test/functional/test_runner.py --failfast p2p_handshake.py p2p_leak.py p2p_sendtxrcncl.py p2p_tx_download.py p2p_blocksonly.py p2p_addrv2_relay.py p2p_node_network_limited.py'
the actual error was from test_runner , got the error after the analysis had run for about 50 minutes
|
Not sure if i am testing this right But i generated mutants for verack message and then started analysis with this command Observation both workers were visibly running I could see them in the process list and the build directories growing. Then, roughly 45 minutes in,output stopped and never resumed. From the terminal I had no way to tell whether the build was still progressing, the baseline was running, a mutant was executing normally, or something had deadlocked. All four states look identical: no output. While looking at /proc from a second shell the two in-flight mutants had:
I think it makes sense to me now that i would experience more timeouts I stopped the run at ~100 minutes with zero mutant results recorded. Maybe we can add a periodic [worker N] mutant 53, 12m elapsed i am still testing using verack I’ll see how long this it takes probably more than 4 hours |
|
This time, I only had 2 mutants. |
Parallel workers get an isolated Git worktree, but their commands inherited the parent process environment, so every worker shared one system temporary directory and one functional test port range. Bitcoin Core's test_runner.py names its scratch directory after a timestamp with second granularity and creates it without exist_ok. Workers start together and do identical work, so they reach that line within the same second and all but one fail with FileExistsError. Functional test ports are derived from the test index rather than the process, so concurrent workers running the same test list would then bind the same ports. Give each worker a private temporary directory (TMPDIR, TMP, TEMP) and its own TEST_RUNNER_PORT_MIN. A runner spans three port ranges, so only three workers fit below the maximum port number; higher --parallel values now warn instead of silently sharing ports. Baseline failures also suggested --setup-command even when one was provided, hiding the real error. The hint is now conditional, and the message quotes the tail of the failing command's output.
|
Thanks, @naiyoma. I pushed two commits to fix some things you noticed. One of the issues is that parallel workers get an isolated Git worktree, but their commands inherited the parent process environment, so every worker shared one system temporary directory and one functional test port range. Also, changed it as commands now stream their output. |
Any reason to use 1200 seconds as timeout? |
I was initially thinking that the default -t 300 was causing the 40 minutes of silence that I experienced, so I tried increasing it to see if it would improve anything. but, there was no observable difference. |
nice, will test again |
|
I’ve retested it with verack, exactly as I did here. #54 (comment), Got a timeout i think the default timeout is too low trying now with 10800 |
|
Retested with this mutant #54 (comment), which is shorter. I got the same output, and it was quicker this time 30 minutes. STDERR: KILLED ✅ MUTATION SCORE: 100.00% (3 killed / 3 total) |
|
@brunoerg My concern so far is that longer mutants seem to always time out, and I think this is because of the default. |
|
I compared per-mutant analysis time between the normal (non-parallel) run and 68 mutants come from version and verack messages My takeaway from this is that I could merge batches of mutants, and running them in parallel would save time
|
|
still looking into this timeout #54 (comment), the issue might have been on my end, i didn't get this today |
Next, I’ll combine 4 messages. This will be around 100 mutants or more, and then I’ll compare the times. I expect parallel to have a better outcome. |
|
The more mutants, the more time parallel saves. |
A worker printed nothing until its command finished, so a 45 minute build, a running baseline, a working mutant and a stalled command all looked the same from the terminal. Commands now stream their output instead of buffering it, and report every 60 seconds how long they have been running along with their most recent line of output. Streaming also fixes a stall. Waiting for a buffered capture waits for the output pipes to close, not just for the shell to exit, so any background process that outlives the command kept the worker blocked with no CPU use and nothing printed until the timeout expired. The pipes are now drained with a grace period, after which the lingering processes are terminated. Commands run in their own process group, and a timeout terminates the whole group rather than only the shell. A leftover build or bitcoind node would otherwise keep holding ports and datadirs, and turn one timeout into timeouts for every mutant that followed. The private process group also means the terminal no longer delivers Ctrl+C to the workers' builds and tests, so cancellation has to reach them through the tool. Every command of a run holds a cancellation signal; on Ctrl+C the workers terminate their process trees with the same grace period as a timeout, the orchestrator drains their results so interrupted mutants are recorded as errors rather than left running, and only then are the worktrees removed. Aborting the worker tasks instead would only have killed the shell and orphaned everything it started.
Perfect! |
|
I generated 127 mutants for 9 messages covering VERSION, VERACK, SENDCMPCT, WTXIDRELAY, SENDADDRV2, FEATURE, SENDTXRCNCL, ADDR/ADDRV2, and INV. I analyzed them using I then compared these results with the individual runs I did previously, where I analyzed each message separately. I remember it taking around 6–7 hours to get to the INV message, whereas the parallel run took ~2 hours, which is much faster. If you look at the table below, you’ll see that the individual runs had more surviving mutants. When I compared the two, there were two things that affected the results. One is that I was running on different head commits on
|
Great time results. |
2 participants
No description provided.