msys2-runtime: Update to 3.6.10 by lazka · Pull Request #6445 · msys2/MSYS2-packages

lazka · 2026-07-16T06:20:36Z

dscho · 2026-07-16T07:28:04Z

@lazka thank you for taking care of this. I planned on doing it myself, but first wanted to chase down a strange bug in Git for Windows' fork of the MSYS2 runtime (which I suspected affected also MSYS2's own variant of v3.6.10, but have now confirmed is absent from it).

lhmouse · 2026-07-20T12:13:53Z

Building UCRT64 GCC with msys2-runtime 3.6.10 results in a lot of zombie processes; 3.6.9 is fine:

oltolm · 2026-07-20T13:15:36Z

I encountered this problem msys2/msys2-runtime#349 when running GCC tests. Maybe it has something to do with it.

lhmouse · 2026-07-20T17:28:34Z

I encountered this problem msys2/msys2-runtime#349 when running GCC tests. Maybe it has something to do with it.

Does it happen only recently in 3.6.10?

I think I used to see the zombie process issue intermittently when I tried maximizing/minimizing the mintty window a few times, but with 3.6.10 it almost always happens.

By the way, other packages seem unaffected. Not sure what is so weird about GCC.

oltolm · 2026-07-20T17:44:39Z

Yes my issue only started to happen in 3.6.10. It's probably different from yours then.

dscho · 2026-07-22T10:28:46Z

My guess is that the many changes to the way non-MSYS processes are spawned leads to this. @oltolm @lhmouse any chance you can bisect this?

For your convenience, I cherry-picked the v3.6.10 commits on top of msys2-3.6.9: https://github.com/dscho/msys2-runtime/commits/refs/heads/cherry-pick-3.6.10-onto-3.6.9, with a workflow that builds all of the intermediate revisions individually and uploads them as build artifacts: https://github.com/dscho/msys2-runtime/actions/runs/29911710869. That way, you should be able to bisect without having to pay the price of building the MSYS2 runtime locally (at the expense of GitHub, which pays for these kinds of one-off workflow runs). The build is roughly half-way done at time of writing.

lhmouse · 2026-07-22T10:51:01Z

@dscho It seems that no files have been uploaded.

build (14)
No files were found with the provided path: _dest/. No artifacts will be uploaded.

While the parent of this commit is tree-same to msys2-3.6.10, it is not identical because instead of rebasing MSYS2's patches on top of cygwin-3.6.10, it does it the other way round, ready for bisecting an issue identified in msys2/MSYS2-packages#6445 (comment). Note: two commits were skipped because msys2-3.6.9 already had backports of them. The output of this workflow run has the testable intermediate builds of the MSYS2 runtime, so that the problem can be bisected down to the offending commit without requiring a local build setup. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

dscho · 2026-07-22T13:00:17Z

@dscho It seems that no files have been uploaded.

build (14)
No files were found with the provided path: _dest/. No artifacts will be uploaded.

Ooops, sorry. I misinterpreted the importance of the "Install" step. Here's a new build: https://github.com/dscho/msys2-runtime/actions/runs/29921797049

lhmouse · 2026-07-22T14:37:19Z

Testing now. Please be patient; each round can take more than one hour, so this probably won't complete in a day.

2026-07-23 11:32 +0800: Holy cow, I see that the number in your build matrix is actually passed as git checkout HEAD~{N}, so the direction of tests was wrong; should have gone in decreasing order.
2026-07-24 16:11 +0800: I notice that commit 5 is said to fix commit 8. Testing now.
2026-07-24 17:14 +0800: Confirmed 5 is broken.
2026-08-06 16:53 +0800: 3.6.10-2 is broken.

#	Status
3.6.10-4	🔴Broken
3.6.10-2	🔴Broken
3.6.10-1	🔴Broken
1
2
3
4
5	🔴Broken
6
7	🔴Broken
8	🔴Broken
9	🟢Good
10	🟢Good
11	⛔Error (mintty hangs upon start)
12	⛔Error (mintty hangs upon start)
13	🟢Good
14	🟢Good
15
16
17
18	🟢Good
19
20
21	🟢Good
22
23

lh_mouse    3644     719 pty0     11:35:24 make -O STAGE1_CFLAGS=-O2 profiledbootstrap
lh_mouse   79960       1 cons1    12:10:50 /ucrt64/x86_64-w64-mingw32/bin/ld <defunct>
lh_mouse   79975       1 cons0    12:11:32 ps -fe
lh_mouse   79954       1 cons1    12:10:50 /ucrt64/x86_64-w64-mingw32/bin/ld <defunct>

oltolm · 2026-07-22T17:06:17Z

So I tested the very first commit and it was already bad. For some reason I didn't have the problem before, but it was not introduced by 3.6.10. I tried debugging a little with the help of Codex and this is what it came up with.

Subject: `term_has_pcon_cap()` leaks `CSI6n` into PTY output under MSYS/Expect, and no private reply arrives

Reproducer:

spawn C:/Windows/System32/where.exe cmd
expect eof
puts -nonewline $expect_out(buffer)

Observed:
sometimes the output is prefixed/suffixed by leaked escape-sequence garbage such as ESC[6n and trailing ;1R.

Example visible result:

C:\msys64\usr\bin\cmd
C:\Windows\System32\cmd.exe
...
;1R

What I traced:
the leaked bytes come from fhandler_pty_slave::term_has_pcon_cap().

In my instrumented log, the exact probe bytes sent there later reappear in normal PTY output:

term_has_pcon_cap sending CSI6n pid=23424 pcon_activated=1 pcon_start=1
term_has_pcon_cap send len=4 bytes=1B 5B 36 6E
...
process_slave_output slave len=4 bytes=1B 5B 36 6E
...
process_slave_output slave len=127 bytes=1B 5B 36 6E 43 3A 5C ...

Important detail:
during the probe window, no private reply ever arrives on either polled pipe. The log repeatedly shows:

term_has_pcon_cap poll ... have_cyg=1 n_cyg=0 have_nat=1 n_nat=0

There is also no pty_master::write() activity during this path, so the expected private reply/handshake path does not seem to be engaged for this repro.

So the failure mode seems to be:

term_has_pcon_cap() injects live CSI6n
no private reply arrives on either polled side
probe times out
injected CSI6n leaks into user-visible PTY output

Current reliable workaround:
MSYS=disable_pcon

Commit history:
the underlying fragile mechanism appears to have been introduced by:

4e08fe42c9f3fdba63a57a8e3a6d705c4e10f50f on August 31, 2020
Cygwin: pty: Disable pseudo console if TERM does not have CSI6n.

I also checked later pcon fixes from 2026 around pcon_start/CSI6n, but this repro path does not seem to go through the master::write() handshake they improve.

Additional note:
I tried replacing the live probe with a simple terminfo-based success path, but that caused expect probe.exp to hang later in pcon startup, so the probe seems to be entangled with later startup behavior too. That suggests this is not just an output-filtering issue but a broader startup-design problem.

If useful, I can provide the full instrumented log and the temporary tracing patch.

lhmouse · 2026-07-23T11:56:43Z

@dscho The tests are complete now: #6445 (comment)

dscho · 2026-07-27T14:23:53Z

@dscho The tests are complete now: #6445 (comment)

Just to make sure: By "8", you are referring to the artifact install-8 that was produced by build (8), which points to 980059f
as the first bad commit, right?

@oltolm just so we're on the same page, by "MSYS/Expect" you're referring to https://packages.msys2.org/base/expect, right?

@tyan0 does any of this ring a bell?

lhmouse · 2026-07-27T14:34:35Z

@dscho The tests are complete now: #6445 (comment)

Just to make sure: By "8", you are referring to the artifact install-8 that was produced by build (8), which points to 980059f as the first bad commit, right?

I think so; it's install-8.

oltolm · 2026-07-27T15:09:58Z

@oltolm just so we're on the same page, by "MSYS/Expect" you're referring to https://packages.msys2.org/base/expect, right?

Yes.

kmilos · 2026-07-29T06:48:01Z

After this update I ended up w/ a borked MSYS2 installation, none of the launchers work:

Error: Could not fork child process: There are no available terminals (-1).

Edit: never mind, works after Windows update and restart?!

oltolm · 2026-08-13T23:51:37Z

I deleted the patches, because they didn't really work.

dscho · 2026-09-12T11:18:44Z

@oltolm @lhmouse sorry, I lost track of this. Is this fixed in the meantime?

lhmouse · 2026-09-12T13:04:07Z

Testing 3.6.10-4 now.

lhmouse · 2026-09-12T13:55:33Z

No, not fixed; still hangs.

oltolm · 2026-09-12T14:15:03Z

Mine is not fixed either.

dscho · 2026-09-12T14:34:37Z

@lhmouse @oltolm thanks for checking. I think this might be related to git-for-windows/git#6387 and/or msys2/msys2-runtime#351.

dscho · 2026-09-13T21:07:42Z

I spent some quality time with the reproducer of msys2/msys2-runtime#351 and believe that the tireless @tyan0 has already tackled this problem. The patch in question is still under discussion, but to see whether it promises to address the hang you reported, I opened msys2/msys2-runtime#364. @lhmouse would you mind testing the msys-2.0.dll contained in the install artifact of the PR build (once it's finished, that is, I'll log off for tonight before awaiting the build)?

lhmouse · 2026-09-14T03:02:17Z

Same; but this time there's only one zombie process so I suspect it has something to do with the issue.

$ ps -fe | grep defunct
lh_mouse 80717 1 cons0 10:55:02 /ucrt64/x86_64-w64-mingw32/bin/ld

dscho · 2026-09-14T04:30:36Z

Thanks for testing! I will look further into the issue as time allows.

dscho · 2026-09-15T05:51:52Z

@lhmouse I added a fixup (probably still not the final form, but it's designed to work around the bug at least). Could you test the latest install build artifact, please?

lhmouse · 2026-09-15T08:56:46Z

@dscho

@lhmouse I added a fixup (probably still not the final form, but it's designed to work around the bug at least). Could you test the latest install build artifact, please?

Perfect! This seems to solve the issue.

dscho · 2026-09-15T09:01:13Z

@lhmouse excellent! Thanks for testing!

delagen · 2026-09-18T12:13:56Z

@dscho

gpg -k works
gpg -K still hangs

dscho · 2026-09-18T13:46:26Z

gpg -k works gpg -K still hangs

Could you maybe spend the effort to turn that into a repeatable reproducer that somebody else can work with? This investigation already cost me several dozens of hours, throwing bones my way to force me to spend more hours merely trying to replicate your finding isn't the kindest thing I wanted to read on a Friday afternoon just before a well-needed weekend.

delagen · 2026-09-18T14:56:53Z

@dscho

I just stated situation.

I have 2 machines with Windows 11, on one it works, on second not.

I tried to clean .gnupg folder, it works till first import of key.

After install noticed here install artefact it seems better but still hang.

I tried to perform clean install of June version from site, but it hangs during installation.

dscho · 2026-09-18T15:59:30Z

I just stated situation.

@delagen then I'll state my situation hopefully more plain than I did before: I am working my butt off to address the issues raised in this discussion, and it's not easy, it is in fact quite difficult, and getting a report about something that is supposed to be related but so frustratingly scarce and non-forthcoming with information about a potential reproducer is just adding to my stress level. This is unhealthy for me, so I will ignore you going forward unless you end up stating more than a situation and coming up with a complete and helpful description how to replicate your finding.

@lhmouse I'm truly sorry to bother you once more with a request to test: msys2/msys2-runtime#366 is backporting two patches from the cygwin-patches mailing list (plus prerequisite and fixup!s); It includes v17 that supersedes the previous v15 you tested, and in my hands it is a more complete fix.

delagen · 2026-09-18T16:04:25Z

@dscho I understand your pain. But I don't know how to help ( . Excuse me

lhmouse · 2026-09-18T16:30:20Z

Testing https://github.com/msys2/msys2-runtime/actions/runs/35215252318

It's 00:29 here so I'll reply tomorrow, ~12 hrs later from now.

lhmouse · 2026-09-19T05:14:42Z

@lhmouse

Testing https://github.com/msys2/msys2-runtime/actions/runs/35215252318

It's 00:29 here so I'll reply tomorrow, ~12 hrs later from now.

This also works well.

dscho · 2026-09-24T12:08:37Z

@lhmouse one more test, please: msys2/msys2-runtime#368? I'd love to merge this PR as-is, except if testing encounters blocking issues.

lhmouse · 2026-09-24T12:14:23Z

Trying this now: https://github.com/msys2/msys2-runtime/actions/runs/35989894658

lhmouse · 2026-09-25T05:23:26Z

@lhmouse

Trying this now: https://github.com/msys2/msys2-runtime/actions/runs/35989894658

This also works without issues.

dscho · 2026-09-25T15:27:29Z

-> #6717

* msys2-runtime: Fix bugs causing hangs This Pull Request backports three changes. The first one is a prerequisite for the second, and the third is also needed to fix those bugs that caused hangs that I first saw reported in #6445: 1. `c892dfe93e`: the [complete prerequisite](cygwin/cygwin@bbd3710), including the companion `dtable.cc` standard-handle change. 2. `92c1e00a38`: [console-mode v18](https://inbox.sourceware.org/cygwin-patches/20260919004709.26360-1-takashi.yano@nifty.ne.jp/), including reconciliation on signal and read-retry paths. 3. `c770e1b9fa`: [owner-exit v5](https://inbox.sourceware.org/cygwin-patches/20260923223230.3504-1-takashi.yano@nifty.ne.jp/), including the processed-input fallback for an unavailable console master. I verified that these patches are enough to address the hangs demonstrated by the reproducer provided in msys2/msys2-runtime#351. This fixes msys2/msys2-runtime#351 This corresponds to msys2/msys2-runtime#368 * msys2-runtime: fix pkgrel Ooops! My script has a bug! Co-authored-by: Christoph Reiter <reiter.christoph@gmail.com> --------- Co-authored-by: Christoph Reiter <reiter.christoph@gmail.com>

* msys2-runtime: Fix bugs causing hangs This Pull Request backports three changes. The first one is a prerequisite for the second, and the third is also needed to fix those bugs that caused hangs that I first saw reported in msys2/MSYS2-packages#6445: 1. `c892dfe93e`: the [complete prerequisite](cygwin/cygwin@bbd3710), including the companion `dtable.cc` standard-handle change. 2. `92c1e00a38`: [console-mode v18](https://inbox.sourceware.org/cygwin-patches/20260919004709.26360-1-takashi.yano@nifty.ne.jp/), including reconciliation on signal and read-retry paths. 3. `c770e1b9fa`: [owner-exit v5](https://inbox.sourceware.org/cygwin-patches/20260923223230.3504-1-takashi.yano@nifty.ne.jp/), including the processed-input fallback for an unavailable console master. I verified that these patches are enough to address the hangs demonstrated by the reproducer provided in msys2/msys2-runtime#351. This fixes msys2/msys2-runtime#351 This corresponds to msys2/msys2-runtime#368 * msys2-runtime: fix pkgrel Ooops! My script has a bug! Co-authored-by: Christoph Reiter <reiter.christoph@gmail.com> --------- Co-authored-by: Christoph Reiter <reiter.christoph@gmail.com> Source: msys2/MSYS2-packages@bca80c8

In the MSYS2 project, there were several reports of newly-introduced hangs, e.g. msys2/msys2-runtime#351 and msys2/MSYS2-packages#6445 (comment). It took a good while to figure out how to fix this properly, with two separate fixes needing 18 and 5 iterations, respectively, on the cygwin-patches mailing list. This branch merges [MSYS2 runtime PR #368](msys2/msys2-runtime#368). The sole `dtable.cc` conflict preserves GFW's pcon-backed PTY detection and genuine console handles. Separately, tests running in CI builds are added that replicate the reproducer of issue 351 and a separate Ctrl+C test that covers [Takashi's pipeline scenario](https://inbox.sourceware.org/cygwin-patches/20260923105326.d57b0710f1519e32b9c5d496@nifty.ne.jp/): the Cygwin sleeper must be interrupted via Ctrl+C even when the native side owns console input. The original DLL already passed this test, so it protects against regressions in this existing behavior.

msys2-runtime: Update to 3.6.10

a0f8b99

msys2/msys2-runtime#347

dscho approved these changes Jul 16, 2026

View reviewed changes

lazka merged commit 0c7d006 into msys2:master Jul 16, 2026
2 checks passed

dscho mentioned this pull request Sep 13, 2026

Cygwin: console: backport mode handling changes (v15, experimental) msys2/msys2-runtime#364

Closed

This was referenced Sep 15, 2026

Regression in msys2-runtime 3.6.10 - some apps hang, i.e. make, git msys2/msys2-runtime#362

Closed

Cygwin: console: backport mode handling v17 and owner-exit v3 (experimental) msys2/msys2-runtime#366

Closed

dscho mentioned this pull request Sep 24, 2026

Fix bugs causing hangs msys2/msys2-runtime#368

Merged

dscho mentioned this pull request Sep 24, 2026

Backport fixes for reported hangs git-for-windows/msys2-runtime#144

Merged

dscho mentioned this pull request Sep 25, 2026

msys2-runtime: Fix bugs causing hangs #6717

Merged

Conversation

lazka commented Jul 16, 2026

Uh oh!

dscho commented Jul 16, 2026

Uh oh!

Uh oh!

lhmouse commented Jul 20, 2026

Uh oh!

oltolm commented Jul 20, 2026

Uh oh!

lhmouse commented Jul 20, 2026

Uh oh!

oltolm commented Jul 20, 2026

Uh oh!

dscho commented Jul 22, 2026

Uh oh!

lhmouse commented Jul 22, 2026

Uh oh!

dscho commented Jul 22, 2026

Uh oh!

lhmouse commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oltolm commented Jul 22, 2026

Subject: term_has_pcon_cap() leaks CSI6n into PTY output under MSYS/Expect, and no private reply arrives

Uh oh!

lhmouse commented Jul 23, 2026

Uh oh!

dscho commented Jul 27, 2026

Uh oh!

lhmouse commented Jul 27, 2026

Uh oh!

oltolm commented Jul 27, 2026

Uh oh!

kmilos commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oltolm commented Aug 13, 2026

Uh oh!

dscho commented Sep 12, 2026

Uh oh!

lhmouse commented Sep 12, 2026

Uh oh!

lhmouse commented Sep 12, 2026

Uh oh!

oltolm commented Sep 12, 2026

Uh oh!

dscho commented Sep 12, 2026

Uh oh!

dscho commented Sep 13, 2026

Uh oh!

lhmouse commented Sep 14, 2026

Uh oh!

dscho commented Sep 14, 2026

Uh oh!

dscho commented Sep 15, 2026

Uh oh!

lhmouse commented Sep 15, 2026

Uh oh!

dscho commented Sep 15, 2026

Uh oh!

delagen commented Sep 18, 2026

Uh oh!

dscho commented Sep 18, 2026

Uh oh!

delagen commented Sep 18, 2026

Uh oh!

dscho commented Sep 18, 2026

Uh oh!

delagen commented Sep 18, 2026

Uh oh!

lhmouse commented Sep 18, 2026

Uh oh!

lhmouse commented Sep 19, 2026

Uh oh!

dscho commented Sep 24, 2026

Uh oh!

lhmouse commented Sep 24, 2026

Uh oh!

lhmouse commented Sep 25, 2026

lhmouse commented Jul 22, 2026 •

edited

Loading

Subject: `term_has_pcon_cap()` leaks `CSI6n` into PTY output under MSYS/Expect, and no private reply arrives

kmilos commented Jul 29, 2026 •

edited

Loading