VFSI: Vectorized Filesystem Interfaces

VFSI is a Rust filesystem client built around vector operations. It grew out of my PhD research on vNFS and related work on filesystem interfaces. It batches independent file operations into a small number of protocol requests, letting applications benefit from NFSv4 compounds and SMB2/3 compound requests without mounting a filesystem or rewriting their data paths in C.

The original vNFS research proposed porting applications to faster, vectorized filesystem interfaces in 2017. Making such ports widespread was difficult then. With today's agentic coding tools, we can make them practical.

The repository provides:

• portable vector operations through Vfsi and singular helpers through VfsiExt;
• NFSv4.1, NFSv4.2, optional SMB2/3, and local backends;
• the nfs4fs NFS fsspec package, the low-level Python vsmb client, and the vsmbfs fsspec adapter for SMB;
• the libntirpc-sys Rust bindings for libntirpc;
• a versioned C ABI with a checked-in header;
• integration tests against NFS-Ganesha, a patched NFSv4.2 COPY server, and Samba dialects from SMB 2.1 through SMB 3.1.1.

Development, issues, and releases live in vmingchen/vnfs. vfsi/vfsi is its promoted public mirror. The version manifest records published packages and the four pinned application ports; see the repository policy.

Why vectorize filesystem operations?

Traditional filesystem APIs issue operations one path at a time. On a remote filesystem, latency can dominate when an application touches many small files. VFSI accepts arrays of paths, ranges, or rename/copy pairs and lets the backend combine them into protocol-native compounds. One fs.cat([...]) or fs.pipe({...}) can therefore complete in a bounded number of network round trips instead of scaling linearly with the file count.

The frozen du benchmark compares kernel and VFSI traversal on one NFS tree, with output parity, measured RPC counts, and native plus controlled-delay results.

Python quick start

nfs4fs is currently Linux-only and supports CPython 3.9 and newer.

python -m pip install nfs4fs

The installed package registers itself with fsspec; no side-effect import is required:

import fsspec

fs = fsspec.filesystem(
    "nfs4", host="nfs.example", root="exports/data", auth="auth_sys"
)
fs.pipe({"/a.txt": b"hello", "/b.txt": b"world"})
print(fs.cat(["/a.txt", "/b.txt"]))

# The server can also be supplied by the URL.
with fsspec.open(
    "nfs4://nfs.example/exports/data/a.txt", "rb", auth="auth_sys"
) as file:
    print(file.read())

For SMB, install vsmb for the low-level vector API or vsmbfs for fsspec:

python -m pip install vsmbfs

import fsspec

fs = fsspec.filesystem("vsmbfs", host="samba.example", share="data")
print(fs.ls("/"))

See the nfs4fs guide and vsmbfs guide for protocol-specific configuration.

Rust operation naming

Frontend vector methods use v + operation, such as Vfsi::vstatfs. Singular helpers in VfsiExt omit the prefix, such as statfs, and delegate to the vector operation. Shared backend methods add _impl, such as vstatfs_impl. Backend-specific helpers, when needed, use _nfs, _local, or _smb, such as vstatfs_nfs. Scalar backend operations omit v, for example open_impl. Modifiers precede the suffix, as in vread_into_impl.

Backend implementers import vfsi_sync::backend::{HandleBackend, VectorBackend}. HandleBackend defines handle I/O, metadata, and lifecycle hooks; VectorBackend: HandleBackend adds vector engines and workflow overrides. Both are object-safe execution contracts. Shared defaults live in vfsi_sync::backend::helpers. Applications use Vfsi for vector operations and VfsiExt for singular helpers and composed workflows. The old FileSystem and Backend names and sfsi/vfsi namespace aliases are removed. Lifecycle/configuration methods keep descriptive names, and protocol internals keep their wire vocabulary. VfsiExt also provides bulk composed workflows.

Vector methods on Vfsi return VecResult<T>: an outer rejection means no input was dispatched; otherwise each input has its own success or OpError. Results retain input order without requiring sequential execution. An item error records completion certainty and any acknowledged I/O progress. VfsiExt provides strict conveniences such as getattrs_all and listdirs_all; these select one error after execution and discard other outcomes. Use vector results directly for selective recovery, without assuming a successful prefix or automatically replaying failed mutations.

This is a breaking Rust migration. VecFs/VecFsExt are removed, backend typed opens use vopen_impl with per-input outcomes, and all writes use the borrowed vwrite_impl boundary. Owned preparation storage uses WriteOp::borrowed() without copying payloads. See the vector-result migration guide for ownership, failure certainty, compatibility policies and the remaining verification gates. Typed attribute updates use vsetattrs_impl; raw attribute masks use vsetattrs_raw_impl. C/Python operation names are unchanged. See the backend guide for current execution contracts and the remaining compatibility aliases.

Filesystem capacity and limits

Vfsi::vstatfs queries paths and open handles in input order. The singular VfsiExt::statfs helper delegates to that vector operation:

use vnfs::{files::Target, Result, Vfsi, VfsiExt};

fn show_capacity(fs: &impl Vfsi) -> Result<()> {
    let capacities = fs.vstatfs(&["/data", "/archive"])?;
    let file = fs.open("/data/example")?;
    let capacity = fs.statfs(Target::File(&file))?;
    println!(
        "{} namespaces; available bytes: {:?}",
        capacities.len(),
        capacity.available_bytes()
    );
    fs.close_files(vec![file])?;
    Ok(())
}

FilesystemStats reports total, free, and caller-available byte and file counts, plus supported limits and flags, through getters such as available_bytes(). Unknown fields are None, including limits the backend cannot determine. These are observations, not reservations or the client's ResourceLimits policy. Paths follow final symlinks; handles query their retained objects even after rename.

The Linux local backend uses anchored descriptors with fstatvfs and fpathconf. Byte counts use f_frsize. file_size_bits exposes POSIX _PC_FILESIZEBITS; it does not assert an exact maximum file size. NFS batches GETATTR for SPACE_, FILES_, MAXNAME, MAXLINK, and MAXFILESIZE and respects the server's returned attribute bitmap. NFS does not report block sizes or mount flags; an explicit read-only client mount reports read_only = Some(true). Auto dispatches each batch through its retained route. Backends without this operation return an unsupported error for nonempty requests.

Copy options

The shared extent-copy backend entry point is vcopy_impl(pairs, CopyOptions). CopyOptions::new() follows final source symlinks and allows server-side copy with a client-copy fallback. Use .with_follow_source_symlinks(false) to recreate source links with their original text; offsets and length do not apply to links. Data-copy destination symlinks keep their existing following behavior. Local and NFS support preserving source links; SMB returns an unsupported error for that option. Existing whole-file frontend and binding calls retain the default following behavior. The former backend copyv and lcopyv methods are consolidated into vcopy_impl; vcopy_data_impl (formerly dupv) is the explicit client-copy engine.

Project organization

The platform is organized as a modular monorepo: shared contracts, synchronous interfaces, protocol backends, bindings, and adapters are separate workspace packages while the published vnfs crate provides the NFS-facing application API. Modified third-party applications live in independent VFSI organization repositories and are registered at pinned revisions rather than vendored or added as submodules.

See VFSI project organization for the source tree, repository ownership, API facets, port naming, maturity, compatibility, and mirror-promotion policies. The documentation index separates current guides from dated benchmark and release records.

Rust

Use vnfs for the NFS-focused application crate, vfsi-core and vfsi-sync for backend-neutral interfaces, and the protocol crates vfsi-nfs, vfsi-smb, vfsi-posix, and vfsi-uring when selecting backends directly. vfsi-local supplies shared local machinery, not a default executor. This keeps protocol dependencies out of packages that do not use them.

NFS connections negotiate NFSv4.2 and fall back to NFSv4.1. Rust clients use AUTH_SYS by default and can opt into Kerberos RPCSEC_GSS (krb5 or krb5i) with the rpcsec-gss feature. Server-side COPY is capability-aware and falls back to client-side I/O when unavailable. The separate vfsi-smb backend negotiates SMB 2.0.2 through SMB 3.1.1, observes server credit and I/O limits, reconnects transport sessions, and uses server-side copy when the server advertises it.

Project status

VFSI and its Python packages are beta software. The core behavior is covered by Rust, Python, upstream fsspec contract, C ABI, NFS, and Samba integration tests. AUTH_SYS remains the default; Rust and Python clients can opt into Kerberos RPCSEC_GSS (krb5 or krb5i). The private-realm integration test covers authentication, renewable-ticket expiry, and reconnect. The API is synchronous, and Linux is the only native-package platform. Read-only NFS operations recover automatically from transport and session failures with bounded reconnect/reopen attempts; mutations are not replayed when a lost reply makes their outcome ambiguous. See the vnfs production notes and failure semantics.

Issues and focused pull requests are welcome. Start with CONTRIBUTING.md; security reports follow SECURITY.md.

Background

NFSv4 compounds are defined by the NFS standards, but conventional POSIX-style APIs rarely expose them to applications. VFSI follows the idea explored in the FAST '17 paper vNFS: Maximizing NFS Performance with Compounds and Vectorized I/O. The original research client was implemented in C; this repository provides a Rust implementation and protocol-neutral bindings.

License

Licensed under either Apache License 2.0 or MIT, at your option.