diff --git a/docs/brief.md b/docs/brief.md index e22676d..07acd1e 100644 --- a/docs/brief.md +++ b/docs/brief.md @@ -39,7 +39,8 @@ rally's projects, with every list bounded and every note cut short (well under 2 10. **In flight** (RAL-191): the jobs a closed window would end (issues log 1.73, when every session's background jobs ended with the shell windows and nothing said what had been running). For each project's `stalls.locks` in the projects list beside the pairing, every held lock: its holder's pid, state, command line and working - directory, from `/proc` (observed), and the owner its lock file names (declared, as `dsm-heavy` writes it). + directory, from `/proc` (observed), and the owner its lock file names (declared, as `dsm-heavy` writes it). A holder whose + taker has exited is found by its open descriptor, as stalls finds it (RAL-192). Each fair pool's queue comes too, with holders and waiters by owner (RAL-188's reader, see docs/stalls.md). Free slots are counted, not listed, and a `summary` says "3 locks held, 1 job waiting for a slot". It's read on this machine, by the CLI, wherever the service is, since that's where the jobs run. Read-only. What it diff --git a/docs/roadmap.md b/docs/roadmap.md index 04a0031..33e31af 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1495,3 +1495,13 @@ principles named were durable, accurate and reactive. Found in rally 0x7 by plus and each fair pool's queue by owner (RAL-188), with a one-line summary; an unreadable projects list, no project naming a lock, an unreadable lock or an unseen queue is named in `blind`, never shown as nothing running; with unit tests and a CLI test with real `flock` holders (docs/brief.md). +- [ ] **RAL-192: Lock holders found by open descriptor when the locking pid is gone.** From RAL-191's first live + run (2026-11-14, issues log 1.86): every dsm-heavy slot's holder by `/proc/locks` was a pid that had exited + (dsm-heavy's bash, gone once it execs `systemd-run --scope`), while its job held the lock by the inherited + descriptor, so stalls' stopped and possibly-hung checks had never prompted for a dsm-heavy job. Done when: a + holder `/proc/locks` names that has exited or doesn't have the file open is replaced by the topmost processes + (whose parent doesn't hold it too) with an open descriptor on the file, matched by device and inode through + `/proc//fd`, the row saying `found_by` and keeping `kernel_holders`; no such process found is a blind + spot naming how many processes' open files couldn't be read, never "no holder"; the brief's `in_flight` shows + the same; with a test that the stopped and possibly-hung checks prompt for a holder found this way + (docs/stalls.md). diff --git a/docs/stalls.md b/docs/stalls.md index 1ff28cb..5b29a45 100644 --- a/docs/stalls.md +++ b/docs/stalls.md @@ -106,6 +106,27 @@ waiting on something legitimate, such as the network, looks the same. Rally stop been agreed with the operator, so `--hung-after` has no default, and without it nothing is sampled. CPU time and start times are the kernel's (`/proc//stat`, observed). Linux only. +## A holder that took the lock and exited (RAL-192) + +From the watchman's first live run of RAL-191 on 2026-11-14 (issues log 1.86): every `dsm-heavy` slot's holder, +by `/proc/locks`, was a process that no longer existed. `/proc/locks` names the process that took a lock. For +`dsm-heavy` that's its own bash, which takes the slot and execs `systemd-run --scope`, and under systemd 261 the +job doesn't keep that pid. The lock stays held, by the open file description the job's processes inherited. So +the stopped-holder and possibly-hung rows above walked a process that didn't exist and never prompted, for any +`dsm-heavy` job. + +When a holder `/proc/locks` names has exited, or doesn't have the file open, the holders are found by their open +descriptors instead. Every process's `/proc//fd` is read for a link naming the lock file, whose target has +the lock file's device and inode. Of those processes, the topmost are taken: the ones whose parent doesn't hold +the file too, so a job and its children are one holder. The lock's row says `found_by` and keeps the kernel's +pid in `kernel_holders`. The stopped and possibly-hung checks then walk the real job's tree. + +If no process can be found with the file open, the lock can't be read, and that's a blind spot, never "no +holder". The reason names the kernel's pid, and how many processes' open files couldn't be read (another +user's, or a sandboxed one's). Under `--prompts-only` that fails the run (RAL-174). When holders are found while +some processes' files couldn't be read, `unreadable_processes` says how many. Kernel threads open no files and +aren't counted. Only processes this user can see are read. + ## A fair slot pool's queue (RAL-188) From the DiscoSim rally's issues log, 1.84: one team's 25 `dsm-heavy` jobs took most of the heavy slots, and diff --git a/src/in_flight.rs b/src/in_flight.rs index ddfa8f1..13cb1e3 100644 --- a/src/in_flight.rs +++ b/src/in_flight.rs @@ -48,6 +48,13 @@ pub fn lock_row(lock: &str, seen: &Seen, owner: Option<&str>) -> Value { "lock": lock, "held": true, "holders": holders, "kind": "observed", "owner": owner, "owner_kind": "declared: the first word of the lock file", }); + if seen.by_descriptor { + row["found_by"] = json!(crate::lock_holder::BY_DESCRIPTOR); + row["kernel_holders"] = json!(seen.kernel_holders); + if seen.unreadable > 0 { + row["unreadable_processes"] = json!(seen.unreadable); + } + } if seen.matched == Matched::InodeOnly { row["matched_by"] = json!( "inode only: no line had the file's device, so a file elsewhere with the same inode \ diff --git a/src/lock_holder.rs b/src/lock_holder.rs index b376355..5728f82 100644 --- a/src/lock_holder.rs +++ b/src/lock_holder.rs @@ -169,6 +169,56 @@ pub struct Seen { pub cmdlines: BTreeMap, /// How long the machine had been up when this was read, in milliseconds (RAL-163). pub uptime_ms: u64, + /// The holders `/proc/locks` names, when they aren't the ones above (RAL-192): the process + /// that took the lock, which may have exited while its children kept the file open, and the + /// lock with it, as `dsm-heavy`'s bash does once it execs `systemd-run --scope`. + pub kernel_holders: Vec, + /// Whether the holders were found by their open descriptors on the file (RAL-192). + pub by_descriptor: bool, + /// How many processes' open files couldn't be read while looking for them (RAL-192). + pub unreadable: usize, +} + +/// The processes with an open descriptor on the file with this device and inode, from each +/// process's `/proc//fd` (RAL-192), with how many processes' descriptors couldn't be read. +/// A descriptor is matched by its link naming `path`, then by the device and inode of what it +/// opens, so a file renamed or replaced since isn't taken for this one. +#[cfg(target_os = "linux")] +fn open_on(path: &Path, dev: u64, ino: u64, pids: impl Iterator) -> (Vec, usize) { + use std::os::unix::fs::MetadataExt; + let mut found = Vec::new(); + let mut unreadable = 0; + for pid in pids { + let Ok(fds) = std::fs::read_dir(format!("/proc/{pid}/fd")) else { + // Gone since the listing, or another user's: only the second is a blind spot. + if Path::new(&format!("/proc/{pid}")).exists() { + unreadable += 1; + } + continue; + }; + let holds = fds.flatten().any(|fd| { + std::fs::read_link(fd.path()).is_ok_and(|target| target == path) + && std::fs::metadata(fd.path()).is_ok_and(|m| m.dev() == dev && m.ino() == ino) + }); + if holds { + found.push(pid); + } + } + (found, unreadable) +} + +/// Of the processes holding a file open, those whose parent doesn't hold it too: the top of each +/// tree, so a job and its children are one holder. +fn topmost(holding: &[u32], procs: &BTreeMap) -> Vec { + holding + .iter() + .copied() + .filter(|pid| { + procs + .get(pid) + .is_none_or(|proc| !holding.contains(&proc.ppid)) + }) + .collect() } /// Below this share of one CPU over the sample, a lock holder's whole process tree is possibly @@ -365,6 +415,11 @@ pub fn judge(lock: &str, seen: &Seen, roster: &BTreeMap) -> Opti })) } +/// How a holder found by its open descriptor is labelled (RAL-192). +pub const BY_DESCRIPTOR: &str = "open descriptor: the process /proc/locks names (kernel_holders) \ + had exited or didn't have the file open, so the holders are the topmost processes with an \ + open descriptor on it, by device and inode"; + /// Reads one lock's holders, waiters and the processes on this machine, from `/proc`. #[cfg(target_os = "linux")] pub fn read(lock: &Path) -> Result { @@ -388,6 +443,7 @@ pub fn read(lock: &Path) -> Result { if seen.holders.is_empty() { return Ok(seen); } + let canonical = std::fs::canonicalize(lock).unwrap_or_else(|_| lock.to_path_buf()); let entries = std::fs::read_dir("/proc").map_err(|e| format!("can't list /proc: {e}"))?; for entry in entries.flatten() { let Some(pid) = entry @@ -404,6 +460,39 @@ pub fn read(lock: &Path) -> Result { seen.procs.insert(pid, proc); } } + // The kernel names the process that took the lock. If it's gone, or doesn't have the file + // open, the lock is held by whatever inherited its descriptor: find those (RAL-192). + let (kernel_open, _) = open_on(&canonical, dev, meta.ino(), seen.holders.iter().copied()); + if seen.holders.iter().any(|pid| !kernel_open.contains(pid)) { + // Kernel threads (kthreadd, pid 2, and its children) open no files and show no fd list. + let user_space = seen + .procs + .values() + .filter(|proc| proc.pid != 2 && proc.ppid != 2) + .map(|proc| proc.pid); + let (holding, unreadable) = open_on(&canonical, dev, meta.ino(), user_space); + seen.unreadable = unreadable; + if holding.is_empty() { + let gone: Vec = seen.holders.iter().map(u32::to_string).collect(); + return Err(if unreadable > 0 { + format!( + "its holder by /proc/locks ({}) doesn't have it open, and the open files of \ + {unreadable} process{} couldn't be read, so what holds it can't be told", + gone.join(", "), + if unreadable == 1 { "" } else { "es" } + ) + } else { + format!( + "its holder by /proc/locks ({}) doesn't have it open, and no process this user \ + can see does", + gone.join(", ") + ) + }); + } + seen.kernel_holders = std::mem::take(&mut seen.holders); + seen.holders = topmost(&holding, &seen.procs); + seen.by_descriptor = true; + } let wanted: Vec = seen .holders .iter() @@ -501,6 +590,13 @@ pub fn check_with( "waiters": seen.waiters.len(), "kind": "observed", "note": "Open file description locks (pid -1) are skipped.", }); + if seen.by_descriptor { + row["found_by"] = json!(BY_DESCRIPTOR); + row["kernel_holders"] = json!(seen.kernel_holders); + if seen.unreadable > 0 { + row["unreadable_processes"] = json!(seen.unreadable); + } + } match seen.matched { Matched::Nothing => { row["holder"] = json!("no holder found in /proc/locks"); @@ -559,6 +655,23 @@ mod tests { /// RAL-163: a holder's tree that used almost no CPU between two reads, and has run long /// enough, is possibly hung, named by its busiest process's command and working directory, /// with the team guessed from a nickname in that directory. + /// RAL-192: of the processes holding a file open, only those whose parent doesn't hold it. + #[test] + fn a_job_and_its_children_holding_a_lock_are_one_holder() { + let procs: BTreeMap = [ + proc(10, 1, 'S', "timeout"), + proc(11, 10, 'R', "godot"), + proc(12, 11, 'S', "godot"), + proc(20, 1, 'S', "bash"), + ] + .into_iter() + .map(|p| (p.pid, p)) + .collect(); + assert_eq!(topmost(&[10, 11, 12, 20], &procs), [10, 20]); + // A parent that doesn't hold it doesn't count; one gone since the listing is a top. + assert_eq!(topmost(&[11, 12, 99], &procs), [11, 99]); + } + #[test] fn a_holder_doing_almost_nothing_is_possibly_hung() { let at = |cpu: u64| { diff --git a/tests/service.rs b/tests/service.rs index b20f467..b88fa6d 100644 --- a/tests/service.rs +++ b/tests/service.rs @@ -4069,6 +4069,102 @@ fn a_lock_holder_doing_almost_nothing_is_flagged_as_possibly_hung() { assert_eq!(row["kind"], "observed"); } +/// RAL-192: a lock taken by a shell that started its job and exited, as dsm-heavy's bash does once +/// it execs `systemd-run --scope`, is held by the child that inherited its descriptor, while +/// /proc/locks still names the shell. The holder is found by its open descriptor, the kernel's pid +/// kept beside it, so stalls' stopped and possibly-hung checks prompt for the real job. +#[cfg(target_os = "linux")] +#[test] +fn a_lock_whose_taker_exited_is_held_by_the_child_with_its_descriptor() { + let service = start(); + let pairing = service.add("a1", false); + let config = service.dir.path().join("client/config.json"); + assert!(pair(&config, &pairing.code()).status.success()); + let slots = service.dir.path().join("slots"); + std::fs::create_dir_all(&slots).unwrap(); + let lock = slots.join("heavy.1.lock"); + let child_pid = slots.join("child.pid"); + // The shell takes the lock, starts a job that inherits the descriptor, says the job's pid and + // exits; the job keeps the lock. + let taker = Command::new("sh") + .args([ + "-c", + "exec 9>>\"$0\"; flock 9; sleep 60 /dev/null 2>&1 & echo $! > \"$1\"; exit 0", + ]) + .arg(&lock) + .arg(&child_pid) + .output() + .unwrap(); + assert!(taker.status.success()); + let job: u32 = std::fs::read_to_string(&child_pid) + .unwrap() + .trim() + .parse() + .unwrap(); + let pattern = slots.join("heavy.*.lock").display().to_string(); + let stalls = |extra: &[&str]| { + let out = rally() + .arg("--config") + .arg(&config) + .env("RALLY_ROSTER_FROM", "no-such-shim-for-tests") + .args(["stalls", "--lock", &pattern]) + .args(extra) + .output() + .unwrap(); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + serde_json::from_slice::(&out.stdout).unwrap() + }; + // Idle, and running past --hung-after: possibly hung, named by the job's own pid. + std::thread::sleep(Duration::from_millis(1200)); + let out = stalls(&["--hung-after", "1s", "--cpu-sample", "1s"]); + let row = &out["locks"][0]; + assert_eq!(row["holders"], json!([job]), "{out}"); + assert_eq!(row["found_by"], rally::lock_holder::BY_DESCRIPTOR); + let kernel = row["kernel_holders"].as_array().unwrap(); + assert_eq!(kernel.len(), 1, "{out}"); + assert_ne!( + kernel[0], + json!(job), + "the kernel names the shell that took it: {out}" + ); + let hung = out["prompts"] + .as_array() + .unwrap() + .iter() + .find(|p| p["row"] == "possibly_hung_lock_holder") + .unwrap_or_else(|| panic!("no possibly_hung_lock_holder: {out}")); + assert_eq!(hung["processes"][0]["holder"], json!(job)); + assert!( + hung["processes"][0]["summary"] + .as_str() + .unwrap() + .starts_with("possibly hung: sleep 60, "), + "{hung}" + ); + // Stopped: the stopped-holder prompt names it. + let stop = Command::new("kill") + .args(["-STOP", &job.to_string()]) + .status() + .unwrap(); + assert!(stop.success()); + let out = stalls(&[]); + let stopped = out["prompts"] + .as_array() + .unwrap() + .iter() + .find(|p| p["row"] == "stopped_lock_holder") + .unwrap_or_else(|| panic!("no stopped_lock_holder: {out}")); + assert_eq!(stopped["processes"][0]["pid"], json!(job), "{stopped}"); + assert_eq!(stopped["processes"][0]["state"], "T"); + let _ = Command::new("kill") + .args(["-KILL", &job.to_string()]) + .status(); +} + /// RAL-188: a fair slot pool's waiters hold files of their own in a waiting directory, not the /// slots, so stalls reads that directory beside the pattern: each held slot's owner, each waiter /// by owner and how long, and a prompt when one has waited past --queue-wait-after while another