VFS API v5: read-only state, modification time, file copy, per-entry stat - #19523
Merged
Merged
Conversation
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Pull request overview
Bumps the frontend and libretro-common VFS layer to VFS API v5, adding read-only state, modification time support, file copying, and per-entry stat during directory enumeration.
Changes:
- Advertise/negotiate VFS interface v5 and plumb new v5 function pointers through runloop and hybrid VFS.
- Implement v5 metadata + copy + dirent_stat across platforms (notably POSIX/Win32/UWP) and add libretro-common wrappers.
- Add a v5 metadata/copy/dirent_stat sample test and wire it into CI sample workflows.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| runloop.c | Advertises VFS API v5 and provides v5 function pointers in the environment VFS interface. |
| libretro-common/vfs/vfs_implementation_uwp.cpp | Adds UWP implementations for read-only, mtime, copy, and dirent_stat; extends stat flags with IS_READONLY. |
| libretro-common/vfs/vfs_implementation.c | Refactors stat into a shared ladder and implements v5 metadata ops, copy, and dirent_stat with platform fast paths + portable fallback. |
| libretro-common/vfs/vfs_hybrid.c | Negotiates v5 first and forwards new v5 operations with the existing local-first rule. |
| libretro-common/streams/file_stream.c | Adds filestream_copy_ex and routes copy through VFS v5 when available, otherwise falls back. |
| libretro-common/samples/file/vfs/vfs_v5_metadata_test.c | New sample test covering v5 contracts (readonly, mtime, copy, dirent_stat). |
| libretro-common/samples/file/vfs/Makefile | Builds and runs the new v5 metadata test sample. |
| libretro-common/include/vfs/vfs_implementation.h | Exposes v5 _impl entry points. |
| libretro-common/include/streams/file_stream.h | Documents copy semantics and adds filestream_copy_ex API. |
| libretro-common/include/retro_dirent.h | Adds retro_dirent_stat API and required VFS version macro. |
| libretro-common/include/libretro.h | Defines v5 flags/typedefs and appends new v5 members to retro_vfs_interface. |
| libretro-common/include/file/file_path.h | Adds path_* wrappers for readonly/mtime and a v5 required-version macro. |
| libretro-common/file/retro_dirent.c | Wires retro_dirent_stat through the negotiated VFS interface with version gating. |
| libretro-common/file/file_path_io.c | Wires path_* readonly/mtime wrappers through the negotiated VFS interface with version gating. |
| .github/workflows/Linux-libretro-common-samples.yml | Runs the new vfs_v5_metadata_test in Linux samples CI. |
| .github/workflows/Cross-libc-vfs-samples.yml | Runs the new vfs_v5_metadata_test in cross-libc CI, including Win32 coverage. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| dirent_closedir_cb = vfs_iface->closedir; | ||
|
|
||
| if (vfs_info->required_interface_version >= DIRENT_STAT_REQUIRED_VFS_VERSION) | ||
| dirent_dirent_stat_cb = vfs_iface->dirent_stat; |
| { | ||
| if (dirent_dirent_stat_cb) | ||
| return dirent_dirent_stat_cb((struct retro_vfs_dir_handle *)rdir, size, mtime); | ||
| if (dirent_stat_unavailable) |
Comment on lines
+795
to
+804
| static void uwp_unix_to_filetime(int64_t unix_s, FILETIME &ft) | ||
| { | ||
| uint64_t t; | ||
| if (unix_s < 0) | ||
| t = UWP_FILETIME_EPOCH_DIFF - (uint64_t)(-unix_s) * UWP_FILETIME_TICKS_PER_S; | ||
| else | ||
| t = UWP_FILETIME_EPOCH_DIFF + (uint64_t)unix_s * UWP_FILETIME_TICKS_PER_S; | ||
| ft.dwLowDateTime = (DWORD)(t & 0xffffffffULL); | ||
| ft.dwHighDateTime = (DWORD)(t >> 32); | ||
| } |
Comment on lines
56
to
+57
| #define FILESTREAM_REQUIRED_VFS_VERSION 2 | ||
| #define FILESTREAM_COPY_REQUIRED_VFS_VERSION 5 |
LibretroAdmin
force-pushed
the
vfs-v5
branch
18 times, most recently
from
September 11, 2026 14:34
043ce43 to
ff799e6
Compare
JesseTG
approved these changes
Sep 11, 2026
Adds RETRO_VFS_STAT_IS_READONLY, RETRO_VFS_COPY_OVERWRITE and five interface entries appended after stat_64: set_readonly, get_mtime, set_mtime, copy, dirent_stat. Existing offsets are unchanged; cores requesting <= 4 see no difference. Motivated by melonDS DS (host <-> emulated FAT sync mirrors read-only state and modification times), Craig (a file copy through the VFS, cp/std::filesystem::copy_file semantics) and Psyraven (entry size during enumeration without opening the file).
stat/stat_64 now run on one retro_vfs_stat_full() ladder that also yields the modification time and the read-only bit on every platform branch (Vita, PS3, Win32 incl. LEGACY_WIN32, GEKKO, generic POSIX). set_readonly / set_mtime: Win32 attributes and SetFileTime, Vita sceIoChstat, desktop POSIX chmod/utimes (atime preserved). Platforms without a reachable permission model return -1 rather than fake success. copy: regular files only; overwrite is cp -f (stale read-only dst removed first); missing parent directories created; no partial dst on failure. Fast paths: Linux copy_file_range via syscall() (with fallocate), Darwin copyfile(), Win32 CopyFileW/A. Portable fallback is a 1 MiB heap-buffer loop through retro_vfs_file_open_impl, which is also how a copy between backends (SAF, SMB, CDROM, native) works. VFS_COPY_NO_FASTPATH forces the loop for testing. dirent_stat: Win32 answers from the find data with no I/O; POSIX fstatat(dirfd, name); Vita from d_stat; SMB/SAF/others join + stat, which is what a caller would otherwise have done itself.
UWP does not compile vfs_implementation.c; every _impl needs a definition here (tools/vfs_backend_parity.py). set_readonly via SetFileAttributesW, get/set_mtime via GetFileAttributesExFromAppW and SetFileTime on a CreateFile2FromAppW handle, copy via CopyFileFromAppW with the same prologue as the C backend, dirent_stat from the find data. IS_READONLY added to stat_64.
path_is_readonly / path_set_readonly / path_get_mtime / path_set_mtime, retro_dirent_stat, filestream_copy_ex. Captured from the frontend when the negotiated version is >= 5; a v1-v4 frontend gets 'unavailable' rather than the local _impl behind its back. filestream_copy() keeps its signature and historical overwrite semantics but now goes through the VFS copy (platform fast paths). The v1-only fallback loop replaces the 256-byte stack buffer with 256 KiB on the heap and creates the destination directory before opening the destination, which the old order got backwards.
Frontend handout bumped to 5 with the five _impl pointers appended in struct order. vfs_hybrid negotiates 5 first and forwards the new entries with the same local-first rule as stat: native paths locally, URIs (or everything on sandboxed platforms) to a frontend that advertised v5.
Read-only round trip (open-for-write denied, cleared again), mtime round trip including pre-1970 values, every copy contract on a 3 MiB fixture (new dst, refuse without OVERWRITE, replace with it, cp -f on a read-only dst, parent creation, src == dst, directories either side, missing src, no partial file on failure), and dirent_stat agreement with the path API for a file, a read-only file and a directory. Wired into the Linux and cross-libc sample workflows; the MSYS2 lane covers the Win32 branches.
filestream_copy_loop() uses strlcpy; the include sat inside the _MSC_VER block, so MXE, clang and the webOS toolchain saw an implicit declaration (glibc 2.38+ declares it natively, which hid this locally).
- Version-5 frontends that leave a new member NULL are treated like older ones: path_*, filestream_copy* and retro_dirent_stat report 'unavailable' instead of calling through NULL. - retro_dirent_stat(NULL) returns 0; dirent_stat_slow returns 0 when the entry name cannot be produced instead of joining NULL. - FILETIME conversion (C and UWP) never forms -unix_s (INT64_MIN was UB) and clamps to the representable 1601..30828 range instead of wrapping. - filestream_copy_ex's v1-only loop applies the same prologue as the VFS copy: src must be a regular file, dst must not be a directory, src != dst, OVERWRITE honoured. - Parent creation for copy now works bottom-up from dst's parent, so a drive root, UNC prefix or doubled separator is a harmless failed rung rather than a component to create; a failure to create the parent is reported instead of deferred to the open. - file_stream.h: drop the duplicated define pair.
The single blocking copy entry is replaced by three:
copy_begin(src, dst, flags) -> handle returns after the up-front
checks; NULL if it cannot start
copy_poll(handle, &done, &total) RUNNING / DONE / FAILED, promptly
copy_close(handle) cancels if running, frees; 0 only
if the copy completed
None of them waits for the transfer. With HAVE_THREADS (every frontend
build) the transfer runs on an rthreads worker using the same kernel
primitives as before - copy_file_range in 64 MiB chunks, copyfile()
with a status callback, CopyFileEx with a progress routine, CopyFile2
on UWP - each of which reports bytes and honours a cancel within one
chunk. Without threads, poll() advances the portable loop by one
chunk per call, so the contract holds there too.
Measured on the same 1 GiB ext4 file as the blocking version: begin
returns in 0.05 ms, 2.6 s end to end vs cp 4.2 s, 2 MB peak RSS for
the process (kernel path, no user-space buffer). The handle, two path
strings and the worker's stack are the only additions.
filestream_copy_begin/poll/close wrap the new entries and report
'unavailable' on pre-v5 frontends. The pre-v5 blocking filestream_copy()
stays for its existing callers, on the fixed read/write loop only, and
is documented as such. vfs_hybrid records which side began a copy so
poll/close go to the same one.
Test: copies driven to completion by polling, bytes_done never exceeds
total, begin-then-immediate-close leaves either a complete file or
none, a file where the parent should be is refused at begin. Built
and run both with the worker thread and as the thread-less pump, plus
the forced portable loop, under TSan and ASan/UBSan.
Replaces the worker-thread copy with begin/step/close: copy_begin(src, dst, flags) -> handle checks, opens both ends, no bytes copy_step(h, max_bytes, &done, &total) moves at most max_bytes, returns copy_close(h) releases; removes a partial dst No thread, no lock, no rthreads dependency anywhere in the VFS. Who drives the steps - a task-queue task, a core's own thread, or a frame loop with a small budget - is the caller's decision, not the frontend's. max_bytes is the latency/throughput dial; 0 selects a 4 MiB default. Linux: copy_file_range at explicit offsets, resumable from done with nothing held in the kernel between steps; no user-space buffer. macOS: same-volume APFS copies are a clonefile() in begin() (O(1), DONE before the first step); otherwise the portable path. Everything else, and any copy touching SAF/SMB/CDROM: one 1 MiB buffer through the backend's own open/read/write. UWP is the same stepper over its own file I/O. 1 GiB on ext4, same run as cp: unbounded budget 1.0 s vs cp 5.7 s cold / 0.6 s hot; default 4 MiB steps 0.63 s in 255 calls, worst single step 47 ms cold (3 ms hot); 1 MiB steps 0.58 s in 1023 calls. Test drives copies with a 100000-byte budget (asserts no step overshoots and progress is monotonic), with an unbounded budget, and cancels after one 64 KiB step; the threaded/pump target split is gone. Passes with the kernel path, the forced portable loop, and under ASan/UBSan.
- Vita: FIO_S_IWUSR does not exist in the SDK; it is SCE_S_IWUSR. set_readonly toggles the owner write bit only (SCE_S_IWOTH is deprecated and the contract is 'the current user cannot write'). - ORBIS carries FreeBSD-derived headers but its libc has no fstatat; dirent_stat takes the join+stat path there. - clonefile() is only used when the SDK has <sys/clonefile.h> and the deployment target is >= 10.12 (checked via __has_include and MAC_OS_X_VERSION_MIN_REQUIRED); the PowerPC SDK has neither. - vfs_v5_metadata_test prints the step count when the minimum-steps check disagrees, so a CI-only miss says how far off it was.
…trace - fstatat() is now its own VFS_HAVE_FSTATAT gate: glibc/musl/bionic/BSDs/ Haiku, Apple only with a 10.10+ deployment target (the PowerPC cross SDK predates it), never QNX or ORBIS. Everyone else takes join+stat. - copy_file_range via syscall(): every argument widened to long; syscall() reads its varargs as longs and an int in a register may carry whatever its upper half held. - preview_audio_decode_test stubs path_is_valid next to its existing path_is_directory/path_mkdir stubs; filestream_copy()'s v1 fallback references it now. - vfs_v5_metadata_test applies the per-step budget check to the finishing step as well, and prints a short trace of the first steps of the budgeted copy so the samples lane says what the kernel actually moved.
…t ignores len Windows (MSYS2 lanes): - get_mtime came from the CRT's _stat64 st_mtime, which cannot hold dates before 1970. The Win32 stat helpers now use GetFileAttributesEx and take the last-write FILETIME from it, converted like everywhere else; the CRT stat only supplies the size. - Overwrite of a read-only destination: Windows refuses to delete a read-only file, so copy_begin clears the attribute and retries the remove when the first attempt fails. Same in the UWP backend. Linux (hosted runner under a sandboxed kernel): - The samples lane's copy_file_range answered a 100000-byte request by copying the whole 3 MiB file (the trace showed step 1 done == total). A real kernel never returns more than len; gVisor does. The Linux stepper now checks n > want: the bytes are on disk so they are accounted, the kernel path is retired for the rest of the process, and if the copy is only part-way it is resumed in place on the portable path (destination opened without truncation, both ends positioned at done). Every later copy is bounded by construction. - vfs_v5_metadata_test treats an overshoot as a platform note rather than a failure, then runs a second budgeted copy and requires it to step correctly. FAKE_CFR=1 (to-EOF) and FAKE_CFR=2 (partial) build a stand-in kernel so the path is exercised on real kernels too; both added to the Linux samples workflow.
…et checks are notes on platforms that ignore len The hosted Linux samples lane overshoots a 100000-byte step even after the kernel path has been retired, which the VFS accounting says cannot happen through either path. Sample builds now define VFS_COPY_DEBUG, which makes the stepper print each copy_file_range / read call with what was asked and what came back, so the lane shows where the extra bytes come from. Until that is known the test keeps every correctness check (byte-identical, size, no partial file) as a hard failure and reports a repeated overshoot as a platform note instead of failing the step-count and retirement checks.
The portable path is the one consoles, SAF, SMB and CDROM take, so a megabyte of heap per copy landed exactly where there is least of it; the kernel fast paths allocate nothing at all. Measured on a 512 MiB copy: 16 KiB and 64 KiB buffers are clearly slower, and 128 KiB through 1 MiB are the same within run-to-run noise. So the ceiling is 128 KiB and a copy never allocates more than what is left to move (floor 16 KiB), which is what small files -- save files, configs, the common case -- actually need. A 512 MiB copy on the portable path takes the same time as before (1.78 s vs 1.85 s for the kernel path on the same run). Same sizing in the UWP backend.
LibretroAdmin
force-pushed
the
vfs-v5
branch
from
September 12, 2026 03:53
ff799e6 to
4c65590
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
VFS API v5: read-only state, modification time, file copy, per-entry stat
Summary
Adds five entries to
retro_vfs_interfaceand bumps the frontend to VFS API v5. Existing offsets are unchanged; cores requesting v4 or lower see no difference.RETRO_VFS_STAT_IS_READONLY(flag)stat/stat_64flag wordset_readonly(path, readonly)FILE_ATTRIBUTE_READONLYget_mtime(path, &mtime)/set_mtime(path, mtime)copy(src, dst, flags)std::filesystem::copy_file/ DolphinCopyRegularFilesemanticsdirent_stat(dirstream, &size, &mtime)Design notes and per-platform behaviour are in the implementation plan; the short version:
-1from the setters and never setIS_READONLY. The VFS does not log; the core decides.copyis files only, no recursion, no metadata preservation. Overwrite iscp -f: a stale read-only destination doesn't defeat an explicit overwrite. Missing parent directories are created. A failed copy never leaves a partial destination.copyworks across backends. The portable path opens both ends throughretro_vfs_file_open_impl, so SAF → local, SMB → local etc. come for free. Fast paths are taken only when both ends are native.dirent_statis free on Windows (already in the find data) and onefstataton POSIX — no path join, no lookup from the root.Commits
libretro.h— flag,RETRO_VFS_COPY_OVERWRITE, five typedefs with Doxygen docs, struct tail under/* VFS API v5 */vfs_implementation.c/.h—stat/stat_64refactored onto oneretro_vfs_stat_full()ladder that also yields mtime and the read-only bit on every platform branch;set_readonly/set_mtimefor Win32 (incl.LEGACY_WIN32), Vita, desktop POSIX;copywith Linuxcopy_file_range(viasyscall(), withfallocate), Darwincopyfile(), Win32CopyFileW/A, and a 1 MiB heap-buffer portable loop;dirent_statfrom find data /fstatat/d_stat, join+stat elsewherevfs_implementation_uwp.cpp— the five twins using theFromAppAPIs;IS_READONLYadded to its stat.tools/vfs_backend_parity.py: 30 symbols, both backends agreepath_is_readonly,path_set_readonly,path_get_mtime,path_set_mtime,retro_dirent_stat,filestream_copy_ex. All version-gated: a v1–v4 frontend gets "unavailable", never the local_implbehind its back.filestream_copy()keeps its signature and now goes through the VFS copy; its v1-only fallback drops the 256-byte stack buffer for 256 KiB on the heap and creates the destination directory before opening the destination (the old order was backwards)runloop.cadvertises 5;vfs_hybrid.cnegotiates 5 and forwards the new entries with its existing local-first rulesamples/file/vfs/vfs_v5_metadata_test.cwired into the Linux and cross-libc sample workflowsVerification
tools/mingw_syntax_check.shclean (Win32 gnu99 + Linux C89 pedantic) on all changed C files.tools/stack_budget.py: within budget on gcc and mingw.vfs_v5_metadata_test: read-only round trip, mtime round trip incl. pre-1970 values, every copy contract on a 3 MiB fixture,dirent_statagreement with the path API. Passes with the fast path, with-DVFS_COPY_NO_FASTPATH(portable loop), and under ASan/UBSan.vfs_hybrid_teststill passes against its fake v1/v3 frontends.copy_file_rangepath 0.76 s, portable loop 3.8 s,cp6.3 s.Not run locally: the UWP MSVC compile of
vfs_implementation_uwp.cpp. Parity passes; CI is the gate.Follow-ups (separate PRs)
libretro.h,file_path.h/.c,retro_dirent.h/.c,file_stream.h/.c,vfs_implementation.h/.c,vfs_implementation_uwp.cppto the standalone libretro-common repo after merge.get_mtime/dirent_stat.runahead.candcore_backup.cprivate copy loops →filestream_copy.retro_dirent_stat()during the walk,path_get_mtime()/path_set_readonly()per file, request v5 with a v4 fallback that skips metadata.cc @JesseTG @cscd98 @Psyraven — header review welcome before this lands; it's ABI.