github-actions: add Windows CI for Cygwin testsuite by aswin-mcw · Pull Request #344 · msys2/msys2-runtime

aswin-mcw · 2026-06-15T18:28:32Z

Summary

Run Cygwin's testsuite against the MSYS2 runtime in CI on Windows, and add the
fixes and (justified) XFAIL annotations needed to bring the suite up cleanly on
x86_64. The testsuite is wired in as an additional job on the existing
build.yaml workflow (reusing its build output), following the commit
structure suggested in review.

Implementation

The Cygwin testsuite runs as a new cygwin-tests job in
.github/workflows/build.yaml instead of a standalone workflow, so it reuses
the existing build job rather than duplicating the build steps:

The build job additionally compiles the test programs (make check_programs)
and uploads the built worktree as an artifact.
The cygwin-tests job downloads that worktree, starts cygserver, and runs
make check with MSYS=winsymlinks:nativestrict.
Test logs (*.log / *.trs) are uploaded for debugging/regression analysis.

The freshly built msys-2.0.dll is staged under testinst/usr/bin by the
runtime's own install rules, so the tests exercise the just-built runtime.

Commit structure

This follows the structure requested in review: the individual fixes are
fixup! commits targeting the correct topic commits (so they squash into the
right place on the next rebase onto a newer Cygwin), line endings are enforced
via .gitattributes, and the CI change is a single commit on build.yaml. The
only standalone additions are the two XFAIL commits (each with a full rationale)
and small CI-plumbing fixes.

Fixes included

UNC-style path fix — when cwd or a PATH entry is /, path construction
produced //<cmd>, which Windows/MSYS2 interprets as a UNC network path,
breaking command resolution from the install root. Fixed by treating / as an
empty prefix. Fixes fcntl07, fcntl07b, cancel11.
busybox setup fix — the syscall test needs sh/sleep/ls from busybox,
but the previous setup pointed at a non-existent path
(/usr/libexec/busybox/bin). Fixed to copy from /usr/bin, staged under the
/usr/bin layout, with busybox added to the CI packages.
Test DLL linkage — test DLLs link against libmsys-2.0.
cygserver startup race — cygserver is started asynchronously; the SysV
IPC tests (semtest/shmtest/msgtest) would race its startup and fail
intermittently. The job now waits for cygserver to be up before running the
suite.

Tests marked XFAIL (with rationale)

Permission tests requiring ACL-backed mounts — access01, access05,
chmod01, stat02, umask03, and the EACCES case of symlink03. MSYS2 mounts
with noacl by default, so POSIX permission bits are not backed by Windows ACLs
and chmod()/umask() cannot enforce them; Cygwin mounts with ACLs by default,
which is why these pass there. Enabling acl was tried and is not viable: it
makes this group pass but regresses stat01/fstat02/fstat04, doesn't fix
umask03, and runs a non-default configuration — so it only trades one set of
failures for another. (The full reasoning is in the commit message.)

Symlink tests needing non-native semantics — symlink01, symlink05. With
MSYS=winsymlinks:nativestrict most symlink tests now pass (lstat02,
readlink01/02, symlink02/04). These two require creating a symlink to a
non-existing ("dangling") target, which native NTFS symlinks cannot represent,
so they fail in setup under nativestrict.

Known failure (not masked)

mingw/cygload — left as a real FAIL rather than XFAIL'd. It is a native
(non-MSYS2) program that LoadLibrary()s the runtime DLL, and it fails with
error 126 because the absolute DLL path passed by cygrun.sh is mangled by
MSYS2's automatic argument path conversion (D:/a/... becomes the invalid
D;A:\...). This is a test-harness path-passing bug, not a runtime defect;
a clean fix is still being worked out.

Current results (x86_64)

TOTAL: 282 | PASS: 269 | XFAIL: 12 | FAIL: 1 (mingw/cygload) | SKIP/XPASS/ERROR: 0

Up from ~151 passing. This establishes an automated baseline for running
Cygwin's testsuite against the MSYS2 runtime on x64 and a foundation for
tracking regressions.

dscho

Thank you for working on this! I'm still not quite certain that those failures should be expected. Ideally we would really try hard to let them all succeed. I vaguely remember running the entire test suite at some point and running into tremendous problems before running cygserver, but I lost my notes on that experiment. I guess I'll try to re-run this on my side ;-)

dscho · 2026-07-29T17:07:53Z

+    - name: Build
+      shell: msys2 {0}
+      run: |
+        (cd winsup && ./autogen.sh)
+        mkdir build && cd build
+        ../configure \
+          --prefix=/usr \
+          --disable-doc \
+          --disable-dumper
+        make -j$(nproc) MINGW_CXXFLAGS="-Wno-error=unused-but-set-variable"


That merely duplicates what the build workflow already does. Can't we simply add running the Cygwin test suite as an additional job to the build.yaml definition?

That merely duplicates what the build workflow already does. Can't we simply add running the Cygwin test suite as an additional job to the build.yaml definition?

Agreed — done. I removed the standalone cygwin.yml and added the testsuite as a cygwin-tests job in build.yaml that reuses the existing build job's output (it uploads the built worktree as an artifact; the test job downloads it and runs make check) instead of rebuilding. No duplication now.

dscho · 2026-07-29T17:09:57Z

        int experrno;
    } Test_cases[] = {
        { Fname, F_OK,  "F_OK", 0 },
-        { Fname, X_OK,  "X_OK", 0 },


I'm not quite certain why this should fail with MSYS2 when it succeeds with Cygwin. After all, Cygwin has to contend with the same ACL vs POSIX permissions incompatibility.

I'm not saying that this patch is wrong, what I'm saying is that the commit message needs to provide a good rationale for this. I like to point people to the guidance in https://github.blog/2022-06-30-write-better-commits-build-better-projects/ to improve commit messages, in particular with a strong focus on this part:

What you’re doing Why you’re doing it

High-level (strategic) Intent (what does this accomplish?) Context (why does the code do what it does now?)

Low-level (tactical) Implementation (what did you do to accomplish your goal?) Justification (why is this change being made?)

You're right that "MSYS2 doesn't replicate chmod semantics" isn't a real rationale. The actual cause is MSYS2's default noacl mount: permission bits aren't backed by Windows ACLs, so chmod()/umask() can't enforce them (the mode comes from the DOS read-only attribute only). Cygwin mounts with ACLs by default, which is why the same tests pass there.

I verified this empirically instead of assuming: mounting the test tree with acl does make access01/05, chmod01, stat02 pass — but it regresses stat01/fstat02/fstat04 (they then read ACL-derived owner/mode values the tests don't expect), doesn't fix umask03/symlink03, and runs a config MSYS2 doesn't use by default. So acl only trades one set of failures for another. (An earlier run looked like acl broke the cygserver IPC tests — that was actually the cygserver startup race, not ACLs.)

This is now captured in the commit message (Cygwin: testsuite: XFAIL permission tests that fail under MSYS2's noacl default), following the intent/context/implementation/justification structure you linked.

dscho · 2026-07-29T17:11:49Z

+	winsup.api/ltp/lstat02 \
+	winsup.api/ltp/readlink01 \
+	winsup.api/ltp/readlink02 \
+	winsup.api/ltp/symlink01 \
+	winsup.api/ltp/symlink03 \
+	winsup.api/ltp/symlink04 \
+	winsup.api/ltp/symlink05 \


Would these tests not succeed with MSYS=winsymlinks:nativestrict?

"
Good call — with MSYS=winsymlinks:nativestrict (which the job now sets) most of them pass, so I dropped the blanket symlink XFAILs: lstat02, readlink01/02, symlink02/04 now pass. Three remain, for specific reasons:

symlink05 — creates a symlink to a non-existing (dangling) target and expects success; a native NTFS symlink must record the target type at creation, so nativestrict can't create a dangling link.
symlink01 — its setup creates symlinks to both real and non-existing ((null)) targets and verifies them with lstat(); the dangling ones can't be created under nativestrict, so it aborts in setup ("Test Case Declaration Error"). Its ELOOP / over-long-pathname case actually passes.
symlink03 — its failing case expects EACCES when the target directory isn't writable, which is the same noacl permission-enforcement gap as the chmod group.
The two link modes winsymlinks:lnk/:sys would let these pass but regress the many tests relying on native symlink semantics, so nativestrict stays the default. Both are spelled out in the commit messages.

dscho · 2026-07-29T17:14:08Z

-	cd ${builddir}/testinst/bin && cp /usr/libexec/busybox/bin/busybox.exe sh.exe
-	cd ${builddir}/testinst/bin && cp /usr/libexec/busybox/bin/busybox.exe sleep.exe
-	cd ${builddir}/testinst/bin && cp /usr/libexec/busybox/bin/busybox.exe ls.exe
+	$(MKDIR_P) ${builddir}/usr/bin
+	cd ${builddir}/usr/bin && cp /usr/bin/busybox.exe sh.exe
+	cd ${builddir}/usr/bin && cp /usr/bin/busybox.exe sleep.exe
+	cd ${builddir}/usr/bin && cp /usr/bin/busybox.exe ls.exe


I agree with this change, but it should be marked as fixup! - Move root to /usr. - Change sorting mount points. - By default mount without ACLs. - Can read /etc/fstab with short mount point format. so that it gets squashed into the correct commit upon the next rebase of the MSYS2 runtime to the next Cygwin version.

Done — this is now a fixup! - Move root to /usr. … commit so it squashes into that topic commit on the next rebase onto the newer Cygwin version.

dscho · 2026-07-29T18:22:11Z

I vaguely remember running the entire test suite at some point and running into tremendous problems before running cygserver, but I lost my notes on that experiment.

I found them again! #135 (comment)

cygwin: testsuite: link test DLLs against libmsys-2.0

…lt mount without ACLs. - Can read /etc/fstab with short mount point format.

…lt mount without ACLs. - Can read /etc/fstab with short mount point format. Cygwin: testsuite: install the test runtime under testinst/usr/bin Mirror the real MSYS2 layout so init_installation_root() resolves to testinst. MSYS2 strips three levels from the DLL path (<root>/usr/bin/msys-2.0.dll -> <root>); with the DLL and helpers under testinst/bin the extra strip overshot the root to ${builddir}, so /tmp resolved to ${builddir}/tmp instead of testinst/tmp. Put the test DLL and the sh/sleep/ls helpers under testinst/usr/bin and point runtime_root there; / -> testinst, /bin and /usr/bin -> testinst/usr/bin, /tmp stays testinst/tmp.

Bash is _really_ unhappy with Carriage Returns: ../../.././winsup/testsuite/cygrun.sh: 5: : not found ../../.././winsup/testsuite/cygrun.sh: 8: : not found ../../.././winsup/testsuite/cygrun.sh: 10: : not found ../../.././winsup/testsuite/cygrun.sh: 18: Syntax error: end of file unexpected (expecting "then") Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

dscho · 2026-07-30T02:08:03Z

For the record: https://github.com/msys2/msys2-runtime/compare/dscho:also-run-cygwin-testsuite-in-ci demonstrates the commit structure that I would strongly prefer.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Download the worktree artifact into the workspace root instead of a directory named wt.tgz, so that 'tar -xzf wt.tgz' unpacks the tarball rather than failing on a directory of the same name.

Start cygserver via the always-present Windows PowerShell 5.1 instead of pwsh, which is not on the restricted PATH used by this step. Without it cygserver never started and the SysV IPC tests (semtest, shmtest, msgtest) failed spuriously.

…cl default MSYS2 mounts its filesystems with the "noacl" option by default (see the "- Move root to /usr. ... - By default mount without ACLs." patch), so POSIX permission bits are not backed by Windows ACLs and chmod()/umask() cannot enforce them; the mode is derived from the DOS read-only attribute only. Cygwin, in contrast, mounts with ACLs enabled by default, which is why the following LTP tests pass under Cygwin but fail under the MSYS2 runtime: - access01, access05 - access() must reflect the mode set by chmod() - chmod01 - chmod() must change the permission bits - stat02 - st_mode must reflect chmod() - umask03 - umask() must mask the mode of created files - symlink03 - expects EACCES when the target directory is not writable These are expected platform differences, not runtime defects. Enabling ACLs is not a viable alternative, as confirmed in CI: mounting the test tree with the "acl" flag makes access01, access05, chmod01 and stat02 pass, but it does NOT fix umask03 or symlink03, and it regresses the stat/fstat metadata tests (stat01, fstat02, fstat04), which then read ACL-derived ownership/mode values the tests do not expect. In other words, "acl" only trades one set of failures for another, while running a configuration MSYS2 does not use by default. (An earlier experiment also appeared to break the cygserver-backed System V IPC tests -- semtest, shmtest, msgtest -- under "acl". That turned out to be an unrelated cygserver startup race, not an ACL effect; it is fixed separately by waiting for cygserver before running the suite, and those tests pass under both noacl and acl.) Since noacl is the deliberate MSYS2 default and enabling ACLs does not make the suite pass, mark these tests XFAIL so the suite documents the expected difference from Cygwin instead of masking a regression.

…emantics The Cygwin testsuite runs with MSYS=winsymlinks:nativestrict so that symbolic links are created as genuine native NTFS symlinks; this lets the bulk of the symlink/readlink/lstat tests (lstat02, readlink01/02, symlink02/04) pass without Cygwin's ".lnk" or magic-file fallbacks. A native NTFS symlink must record at creation time whether its target is a file or a directory, so it cannot be created for a non-existing ("dangling") target. Two tests depend on exactly that and therefore fail: - symlink05 - creates a symlink to a non-existing object name and expects the call to succeed; nativestrict cannot create such a link. - symlink01 - a combined SGI symlink-behaviour test whose setup creates symlinks pointing at both real and non-existing ("(null)") targets and verifies them with lstat(). The dangling entries cannot be created under nativestrict, so the setup aborts with "Test Case Declaration Error" / lstat failures (BROK). (Its ELOOP / over-long-pathname case itself passes.) Both fail for a fundamental Win32 reason under nativestrict, not a runtime defect. The non-native link modes (winsymlinks:lnk / :sys) would let them pass but would regress the many tests that rely on native symlink semantics, so nativestrict remains the correct default. Mark these two tests XFAIL.

Wait for cygserver to be running before invoking "make check". cygserver is started in the background, and the test harness immediately launches the suite with -j8, so the System V IPC tests (semtest, shmtest, msgtest) could run before cygserver had finished initializing and fail intermittently. Poll for the cygserver process (bounded to 60s) and add a short settle delay before starting the tests so their outcome no longer depends on winning a startup race.

aswin-mcw · 2026-07-30T21:53:36Z

Thank you for working on this! I'm still not quite certain that those failures should be expected. Ideally we would really try hard to let them all succeed. I vaguely remember running the entire test suite at some point and running into tremendous problems before running cygserver, but I lost my notes on that experiment. I guess I'll try to re-run this on my side ;-)

Thanks! I took "try hard to let them succeed" seriously and reworked this substantially. With cygserver started before make check and MSYS=winsymlinks:nativestrict, the suite now runs at 269 PASS / 12 XFAIL / 1 FAIL (up from ~151 passing).

You were right that cygserver is essential — the SysV IPC tests (semtest/shmtest/msgtest) fail without it. I also hit a variant of your "tremendous problems": cygserver starts asynchronously and the IPC tests were racing its startup and failing intermittently, so the job now waits for cygserver to be up before running the suite.

Rather than blanket-XFAIL the rest, I investigated each one and only kept XFAILs that are genuine platform limitations, each justified in its commit message (see the per-thread replies). The one honest exception is mingw/cygload, which I've deliberately left as a real FAIL rather than mask it — it's a test-harness path-mangling bug (the DLL path is corrupted to D;A:\… before LoadLibrary), not a runtime defect; I'm still working on a clean fix.

aswin-mcw · 2026-07-30T21:58:13Z

I vaguely remember running the entire test suite at some point and running into tremendous problems before running cygserver, but I lost my notes on that experiment.

I found them again! #135 (comment)

Thanks for digging those up — that matches exactly what we saw. The cygwin-tests job starts cygserver before make check (and now waits for it to come up first); without it, semtest/shmtest/msgtest fail just as your notes describe.

aswin-mcw · 2026-07-30T21:59:30Z

For the record: https://github.com/msys2/msys2-runtime/compare/dscho:also-run-cygwin-testsuite-in-ci demonstrates the commit structure that I would strongly prefer.

Thanks for the reference branch — I've restructured to match it: the fixes are fixup! commits targeting the right topic commits, line endings are enforced via .gitattributes rather than ad-hoc, and the testsuite runs as a job in build.yaml. The only additions on top are the two XFAIL commits (with rationale) and small CI-plumbing fixes for cygserver startup and the worktree artifact.

chandru-mcw force-pushed the github-actions-cygwin-testsuite branch from a440be4 to 130a3e9 Compare June 25, 2026 11:18

aswin-mcw mentioned this pull request Jun 25, 2026

Verify Msys2 build on Windows #341

Open

aswin-mcw force-pushed the github-actions-cygwin-testsuite branch from d805eae to a05ca60 Compare July 29, 2026 10:31

aswin-mcw mentioned this pull request Jul 29, 2026

Cygwin: align testsuite tmp directory with runtime path resolution #343

Open

dscho reviewed Jul 29, 2026

View reviewed changes

aswin-mcw and others added 5 commits July 30, 2026 03:58

fixup! Rename dll from cygwin to msys

950854f

cygwin: testsuite: link test DLLs against libmsys-2.0

fixup! - Move root to /usr. - Change sorting mount points. - By defau…

126fb1d

…lt mount without ACLs. - Can read /etc/fstab with short mount point format.

fixup! - Move root to /usr. - Change sorting mount points. - By defau…

feae351

…lt mount without ACLs. - Can read /etc/fstab with short mount point format.

dscho and others added 6 commits July 30, 2026 16:09

ci: run Cygwin's testsuite on the MSYS2 runtime

edfa4c7

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

fixup! ci: run Cygwin's testsuite on the MSYS2 runtime

1e1df1d

Download the worktree artifact into the workspace root instead of a directory named wt.tgz, so that 'tar -xzf wt.tgz' unpacks the tarball rather than failing on a directory of the same name.

aswin-mcw force-pushed the github-actions-cygwin-testsuite branch 3 times, most recently from a05ca60 to 960c58c Compare July 30, 2026 21:47

aswin-mcw changed the base branch from msys2-3.6.9 to msys2-3.6.10 July 30, 2026 21:48

	What you’re doing	Why you’re doing it
High-level (strategic)	Intent (what does this accomplish?)	Context (why does the code do what it does now?)
Low-level (tactical)	Implementation (what did you do to accomplish your goal?)	Justification (why is this change being made?)

Conversation

aswin-mcw commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Implementation

Commit structure

Fixes included

Tests marked XFAIL (with rationale)

Known failure (not masked)

Current results (x86_64)

Uh oh!

dscho left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dscho commented Jul 29, 2026

Uh oh!

dscho commented Jul 30, 2026

Uh oh!

aswin-mcw commented Jul 30, 2026

Uh oh!

aswin-mcw commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aswin-mcw commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aswin-mcw commented Jun 15, 2026 •

edited

Loading

aswin-mcw commented Jul 30, 2026 •

edited

Loading