From 2dec0b6832e1c528bbe4d5cd7d6d55996b8dac06 Mon Sep 17 00:00:00 2001 From: theBlanketBaron Date: Sat, 14 Nov 2026 13:23:03 -0500 Subject: [PATCH] RAL-192: lock holders found by open descriptor when the locking pid is gone /proc/locks names the process that took a lock; dsm-heavy's bash takes the slot and execs systemd-run --scope, and under systemd 261 the job doesn't keep that pid, so every dsm-heavy holder read as a pid that had exited, and stalls' stopped and possibly-hung checks never prompted for one (issues log 1.86). When the kernel's holder is gone or doesn't have the file open, the holders are the topmost processes with an open descriptor on it, by device and inode through /proc//fd; the row says found_by and keeps kernel_holders; none found is a blind spot naming how many processes couldn't be read. Co-Authored-By: Claude Opus 5.5 --- docs/brief.md | 3 +- docs/roadmap.md | 10 ++++ docs/stalls.md | 21 +++++++++ src/in_flight.rs | 7 +++ src/lock_holder.rs | 113 +++++++++++++++++++++++++++++++++++++++++++++ tests/service.rs | 96 ++++++++++++++++++++++++++++++++++++++ 6 files changed, 249 insertions(+), 1 deletion(-) diff --git a/docs/brief.md b/docs/brief.md index e22676d..07acd1e 100644 --- a/docs/brief.md +++ b/docs/brief.md @@ -39,7 +39,8 @@ rally's projects, with every list bounded and every note cut short (well under 2 10. **In flight** (RAL-191): the jobs a closed window would end (issues log 1.73, when every session's background jobs ended with the shell windows and nothing said what had been running). For each project's `stalls.locks` in the projects list beside the pairing, every held lock: its holder's pid, state, command line and working - directory, from `/proc` (observed), and the owner its lock file names (declared, as `dsm-heavy` writes it). + directory, from `/proc` (observed), and the owner its lock file names (declared, as `dsm-heavy` writes it). A holder whose + taker has exited is found by its open descriptor, as stalls finds it (RAL-192). Each fair pool's queue comes too, with holders and waiters by owner (RAL-188's reader, see docs/stalls.md). Free slots are counted, not listed, and a `summary` says "3 locks held, 1 job waiting for a slot". It's read on this machine, by the CLI, wherever the service is, since that's where the jobs run. Read-only. What it diff --git a/docs/roadmap.md b/docs/roadmap.md index 04a0031..33e31af 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1495,3 +1495,13 @@ principles named were durable, accurate and reactive. Found in rally 0x7 by plus and each fair pool's queue by owner (RAL-188), with a one-line summary; an unreadable projects list, no project naming a lock, an unreadable lock or an unseen queue is named in `blind`, never shown as nothing running; with unit tests and a CLI test with real `flock` holders (docs/brief.md). +- [ ] **RAL-192: Lock holders found by open descriptor when the locking pid is gone.** From RAL-191's first live + run (2026-11-14, issues log 1.86): every dsm-heavy slot's holder by `/proc/locks` was a pid that had exited + (dsm-heavy's bash, gone once it execs `systemd-run --scope`), while its job held the lock by the inherited + descriptor, so stalls' stopped and possibly-hung checks had never prompted for a dsm-heavy job. Done when: a + holder `/proc/locks` names that has exited or doesn't have the file open is replaced by the topmost processes + (whose parent doesn't hold it too) with an open descriptor on the file, matched by device and inode through + `/proc//fd`, the row saying `found_by` and keeping `kernel_holders`; no such process found is a blind + spot naming how many processes' open files couldn't be read, never "no holder"; the brief's `in_flight` shows + the same; with a test that the stopped and possibly-hung checks prompt for a holder found this way + (docs/stalls.md). diff --git a/docs/stalls.md b/docs/stalls.md index 1ff28cb..5b29a45 100644 --- a/docs/stalls.md +++ b/docs/stalls.md @@ -106,6 +106,27 @@ waiting on something legitimate, such as the network, looks the same. Rally stop been agreed with the operator, so `--hung-after` has no default, and without it nothing is sampled. CPU time and start times are the kernel's (`/proc//stat`, observed). Linux only. +## A holder that took the lock and exited (RAL-192) + +From the watchman's first live run of RAL-191 on 2026-11-14 (issues log 1.86): every `dsm-heavy` slot's holder, +by `/proc/locks`, was a process that no longer existed. `/proc/locks` names the process that took a lock. For +`dsm-heavy` that's its own bash, which takes the slot and execs `systemd-run --scope`, and under systemd 261 the +job doesn't keep that pid. The lock stays held, by the open file description the job's processes inherited. So +the stopped-holder and possibly-hung rows above walked a process that didn't exist and never prompted, for any +`dsm-heavy` job. + +When a holder `/proc/locks` names has exited, or doesn't have the file open, the holders are found by their open +descriptors instead. Every process's `/proc//fd` is read for a link naming the lock file, whose target has +the lock file's device and inode. Of those processes, the topmost are taken: the ones whose parent doesn't hold +the file too, so a job and its children are one holder. The lock's row says `found_by` and keeps the kernel's +pid in `kernel_holders`. The stopped and possibly-hung checks then walk the real job's tree. + +If no process can be found with the file open, the lock can't be read, and that's a blind spot, never "no +holder". The reason names the kernel's pid, and how many processes' open files couldn't be read (another +user's, or a sandboxed one's). Under `--prompts-only` that fails the run (RAL-174). When holders are found while +some processes' files couldn't be read, `unreadable_processes` says how many. Kernel threads open no files and +aren't counted. Only processes this user can see are read. + ## A fair slot pool's queue (RAL-188) From the DiscoSim rally's issues log, 1.84: one team's 25 `dsm-heavy` jobs took most of the heavy slots, and diff --git a/src/in_flight.rs b/src/in_flight.rs index ddfa8f1..13cb1e3 100644 --- a/src/in_flight.rs +++ b/src/in_flight.rs @@ -48,6 +48,13 @@ pub fn lock_row(lock: &str, seen: &Seen, owner: Option<&str>) -> Value { "lock": lock, "held": true, "holders": holders, "kind": "observed", "owner": owner, "owner_kind": "declared: the first word of the lock file", }); + if seen.by_descriptor { + row["found_by"] = json!(crate::lock_holder::BY_DESCRIPTOR); + row["kernel_holders"] = json!(seen.kernel_holders); + if seen.unreadable > 0 { + row["unreadable_processes"] = json!(seen.unreadable); + } + } if seen.matched == Matched::InodeOnly { row["matched_by"] = json!( "inode only: no line had the file's device, so a file elsewhere with the same inode \ diff --git a/src/lock_holder.rs b/src/lock_holder.rs index b376355..5728f82 100644 --- a/src/lock_holder.rs +++ b/src/lock_holder.rs @@ -169,6 +169,56 @@ pub struct Seen { pub cmdlines: BTreeMap, /// How long the machine had been up when this was read, in milliseconds (RAL-163). pub uptime_ms: u64, + /// The holders `/proc/locks` names, when they aren't the ones above (RAL-192): the process + /// that took the lock, which may have exited while its children kept the file open, and the + /// lock with it, as `dsm-heavy`'s bash does once it execs `systemd-run --scope`. + pub kernel_holders: Vec, + /// Whether the holders were found by their open descriptors on the file (RAL-192). + pub by_descriptor: bool, + /// How many processes' open files couldn't be read while looking for them (RAL-192). + pub unreadable: usize, +} + +/// The processes with an open descriptor on the file with this device and inode, from each +/// process's `/proc//fd` (RAL-192), with how many processes' descriptors couldn't be read. +/// A descriptor is matched by its link naming `path`, then by the device and inode of what it +/// opens, so a file renamed or replaced since isn't taken for this one. +#[cfg(target_os = "linux")] +fn open_on(path: &Path, dev: u64, ino: u64, pids: impl Iterator) -> (Vec, usize) { + use std::os::unix::fs::MetadataExt; + let mut found = Vec::new(); + let mut unreadable = 0; + for pid in pids { + let Ok(fds) = std::fs::read_dir(format!("/proc/{pid}/fd")) else { + // Gone since the listing, or another user's: only the second is a blind spot. + if Path::new(&format!("/proc/{pid}")).exists() { + unreadable += 1; + } + continue; + }; + let holds = fds.flatten().any(|fd| { + std::fs::read_link(fd.path()).is_ok_and(|target| target == path) + && std::fs::metadata(fd.path()).is_ok_and(|m| m.dev() == dev && m.ino() == ino) + }); + if holds { + found.push(pid); + } + } + (found, unreadable) +} + +/// Of the processes holding a file open, those whose parent doesn't hold it too: the top of each +/// tree, so a job and its children are one holder. +fn topmost(holding: &[u32], procs: &BTreeMap) -> Vec { + holding + .iter() + .copied() + .filter(|pid| { + procs + .get(pid) + .is_none_or(|proc| !holding.contains(&proc.ppid)) + }) + .collect() } /// Below this share of one CPU over the sample, a lock holder's whole process tree is possibly @@ -365,6 +415,11 @@ pub fn judge(lock: &str, seen: &Seen, roster: &BTreeMap) -> Opti })) } +/// How a holder found by its open descriptor is labelled (RAL-192). +pub const BY_DESCRIPTOR: &str = "open descriptor: the process /proc/locks names (kernel_holders) \ + had exited or didn't have the file open, so the holders are the topmost processes with an \ + open descriptor on it, by device and inode"; + /// Reads one lock's holders, waiters and the processes on this machine, from `/proc`. #[cfg(target_os = "linux")] pub fn read(lock: &Path) -> Result { @@ -388,6 +443,7 @@ pub fn read(lock: &Path) -> Result { if seen.holders.is_empty() { return Ok(seen); } + let canonical = std::fs::canonicalize(lock).unwrap_or_else(|_| lock.to_path_buf()); let entries = std::fs::read_dir("/proc").map_err(|e| format!("can't list /proc: {e}"))?; for entry in entries.flatten() { let Some(pid) = entry @@ -404,6 +460,39 @@ pub fn read(lock: &Path) -> Result { seen.procs.insert(pid, proc); } } + // The kernel names the process that took the lock. If it's gone, or doesn't have the file + // open, the lock is held by whatever inherited its descriptor: find those (RAL-192). + let (kernel_open, _) = open_on(&canonical, dev, meta.ino(), seen.holders.iter().copied()); + if seen.holders.iter().any(|pid| !kernel_open.contains(pid)) { + // Kernel threads (kthreadd, pid 2, and its children) open no files and show no fd list. + let user_space = seen + .procs + .values() + .filter(|proc| proc.pid != 2 && proc.ppid != 2) + .map(|proc| proc.pid); + let (holding, unreadable) = open_on(&canonical, dev, meta.ino(), user_space); + seen.unreadable = unreadable; + if holding.is_empty() { + let gone: Vec = seen.holders.iter().map(u32::to_string).collect(); + return Err(if unreadable > 0 { + format!( + "its holder by /proc/locks ({}) doesn't have it open, and the open files of \ + {unreadable} process{} couldn't be read, so what holds it can't be told", + gone.join(", "), + if unreadable == 1 { "" } else { "es" } + ) + } else { + format!( + "its holder by /proc/locks ({}) doesn't have it open, and no process this user \ + can see does", + gone.join(", ") + ) + }); + } + seen.kernel_holders = std::mem::take(&mut seen.holders); + seen.holders = topmost(&holding, &seen.procs); + seen.by_descriptor = true; + } let wanted: Vec = seen .holders .iter() @@ -501,6 +590,13 @@ pub fn check_with( "waiters": seen.waiters.len(), "kind": "observed", "note": "Open file description locks (pid -1) are skipped.", }); + if seen.by_descriptor { + row["found_by"] = json!(BY_DESCRIPTOR); + row["kernel_holders"] = json!(seen.kernel_holders); + if seen.unreadable > 0 { + row["unreadable_processes"] = json!(seen.unreadable); + } + } match seen.matched { Matched::Nothing => { row["holder"] = json!("no holder found in /proc/locks"); @@ -559,6 +655,23 @@ mod tests { /// RAL-163: a holder's tree that used almost no CPU between two reads, and has run long /// enough, is possibly hung, named by its busiest process's command and working directory, /// with the team guessed from a nickname in that directory. + /// RAL-192: of the processes holding a file open, only those whose parent doesn't hold it. + #[test] + fn a_job_and_its_children_holding_a_lock_are_one_holder() { + let procs: BTreeMap = [ + proc(10, 1, 'S', "timeout"), + proc(11, 10, 'R', "godot"), + proc(12, 11, 'S', "godot"), + proc(20, 1, 'S', "bash"), + ] + .into_iter() + .map(|p| (p.pid, p)) + .collect(); + assert_eq!(topmost(&[10, 11, 12, 20], &procs), [10, 20]); + // A parent that doesn't hold it doesn't count; one gone since the listing is a top. + assert_eq!(topmost(&[11, 12, 99], &procs), [11, 99]); + } + #[test] fn a_holder_doing_almost_nothing_is_possibly_hung() { let at = |cpu: u64| { diff --git a/tests/service.rs b/tests/service.rs index b20f467..b88fa6d 100644 --- a/tests/service.rs +++ b/tests/service.rs @@ -4069,6 +4069,102 @@ fn a_lock_holder_doing_almost_nothing_is_flagged_as_possibly_hung() { assert_eq!(row["kind"], "observed"); } +/// RAL-192: a lock taken by a shell that started its job and exited, as dsm-heavy's bash does once +/// it execs `systemd-run --scope`, is held by the child that inherited its descriptor, while +/// /proc/locks still names the shell. The holder is found by its open descriptor, the kernel's pid +/// kept beside it, so stalls' stopped and possibly-hung checks prompt for the real job. +#[cfg(target_os = "linux")] +#[test] +fn a_lock_whose_taker_exited_is_held_by_the_child_with_its_descriptor() { + let service = start(); + let pairing = service.add("a1", false); + let config = service.dir.path().join("client/config.json"); + assert!(pair(&config, &pairing.code()).status.success()); + let slots = service.dir.path().join("slots"); + std::fs::create_dir_all(&slots).unwrap(); + let lock = slots.join("heavy.1.lock"); + let child_pid = slots.join("child.pid"); + // The shell takes the lock, starts a job that inherits the descriptor, says the job's pid and + // exits; the job keeps the lock. + let taker = Command::new("sh") + .args([ + "-c", + "exec 9>>\"$0\"; flock 9; sleep 60 /dev/null 2>&1 & echo $! > \"$1\"; exit 0", + ]) + .arg(&lock) + .arg(&child_pid) + .output() + .unwrap(); + assert!(taker.status.success()); + let job: u32 = std::fs::read_to_string(&child_pid) + .unwrap() + .trim() + .parse() + .unwrap(); + let pattern = slots.join("heavy.*.lock").display().to_string(); + let stalls = |extra: &[&str]| { + let out = rally() + .arg("--config") + .arg(&config) + .env("RALLY_ROSTER_FROM", "no-such-shim-for-tests") + .args(["stalls", "--lock", &pattern]) + .args(extra) + .output() + .unwrap(); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + serde_json::from_slice::(&out.stdout).unwrap() + }; + // Idle, and running past --hung-after: possibly hung, named by the job's own pid. + std::thread::sleep(Duration::from_millis(1200)); + let out = stalls(&["--hung-after", "1s", "--cpu-sample", "1s"]); + let row = &out["locks"][0]; + assert_eq!(row["holders"], json!([job]), "{out}"); + assert_eq!(row["found_by"], rally::lock_holder::BY_DESCRIPTOR); + let kernel = row["kernel_holders"].as_array().unwrap(); + assert_eq!(kernel.len(), 1, "{out}"); + assert_ne!( + kernel[0], + json!(job), + "the kernel names the shell that took it: {out}" + ); + let hung = out["prompts"] + .as_array() + .unwrap() + .iter() + .find(|p| p["row"] == "possibly_hung_lock_holder") + .unwrap_or_else(|| panic!("no possibly_hung_lock_holder: {out}")); + assert_eq!(hung["processes"][0]["holder"], json!(job)); + assert!( + hung["processes"][0]["summary"] + .as_str() + .unwrap() + .starts_with("possibly hung: sleep 60, "), + "{hung}" + ); + // Stopped: the stopped-holder prompt names it. + let stop = Command::new("kill") + .args(["-STOP", &job.to_string()]) + .status() + .unwrap(); + assert!(stop.success()); + let out = stalls(&[]); + let stopped = out["prompts"] + .as_array() + .unwrap() + .iter() + .find(|p| p["row"] == "stopped_lock_holder") + .unwrap_or_else(|| panic!("no stopped_lock_holder: {out}")); + assert_eq!(stopped["processes"][0]["pid"], json!(job), "{stopped}"); + assert_eq!(stopped["processes"][0]["state"], "T"); + let _ = Command::new("kill") + .args(["-KILL", &job.to_string()]) + .status(); +} + /// RAL-188: a fair slot pool's waiters hold files of their own in a waiting directory, not the /// slots, so stalls reads that directory beside the pattern: each held slot's owner, each waiter /// by owner and how long, and a prompt when one has waited past --queue-wait-after while another