Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/brief.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,8 @@ rally's projects, with every list bounded and every note cut short (well under 2
10. **In flight** (RAL-191): the jobs a closed window would end (issues log 1.73, when every session's background
jobs ended with the shell windows and nothing said what had been running). For each project's `stalls.locks`
in the projects list beside the pairing, every held lock: its holder's pid, state, command line and working
directory, from `/proc` (observed), and the owner its lock file names (declared, as `dsm-heavy` writes it).
directory, from `/proc` (observed), and the owner its lock file names (declared, as `dsm-heavy` writes it). A holder whose
taker has exited is found by its open descriptor, as stalls finds it (RAL-192).
Each fair pool's queue comes too, with holders and waiters by owner (RAL-188's reader, see docs/stalls.md).
Free slots are counted, not listed, and a `summary` says "3 locks held, 1 job waiting for a slot". It's read
on this machine, by the CLI, wherever the service is, since that's where the jobs run. Read-only. What it
Expand Down
10 changes: 10 additions & 0 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -1495,3 +1495,13 @@ principles named were durable, accurate and reactive. Found in rally 0x7 by plus
and each fair pool's queue by owner (RAL-188), with a one-line summary; an unreadable projects list, no
project naming a lock, an unreadable lock or an unseen queue is named in `blind`, never shown as nothing
running; with unit tests and a CLI test with real `flock` holders (docs/brief.md).
- [ ] **RAL-192: Lock holders found by open descriptor when the locking pid is gone.** From RAL-191's first live
run (2026-11-14, issues log 1.86): every dsm-heavy slot's holder by `/proc/locks` was a pid that had exited
(dsm-heavy's bash, gone once it execs `systemd-run --scope`), while its job held the lock by the inherited
descriptor, so stalls' stopped and possibly-hung checks had never prompted for a dsm-heavy job. Done when: a
holder `/proc/locks` names that has exited or doesn't have the file open is replaced by the topmost processes
(whose parent doesn't hold it too) with an open descriptor on the file, matched by device and inode through
`/proc/<pid>/fd`, the row saying `found_by` and keeping `kernel_holders`; no such process found is a blind
spot naming how many processes' open files couldn't be read, never "no holder"; the brief's `in_flight` shows
the same; with a test that the stopped and possibly-hung checks prompt for a holder found this way
(docs/stalls.md).
21 changes: 21 additions & 0 deletions docs/stalls.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,27 @@ waiting on something legitimate, such as the network, looks the same. Rally stop
been agreed with the operator, so `--hung-after` has no default, and without it nothing is sampled. CPU time and
start times are the kernel's (`/proc/<pid>/stat`, observed). Linux only.

## A holder that took the lock and exited (RAL-192)

From the watchman's first live run of RAL-191 on 2026-11-14 (issues log 1.86): every `dsm-heavy` slot's holder,
by `/proc/locks`, was a process that no longer existed. `/proc/locks` names the process that took a lock. For
`dsm-heavy` that's its own bash, which takes the slot and execs `systemd-run --scope`, and under systemd 261 the
job doesn't keep that pid. The lock stays held, by the open file description the job's processes inherited. So
the stopped-holder and possibly-hung rows above walked a process that didn't exist and never prompted, for any
`dsm-heavy` job.

When a holder `/proc/locks` names has exited, or doesn't have the file open, the holders are found by their open
descriptors instead. Every process's `/proc/<pid>/fd` is read for a link naming the lock file, whose target has
the lock file's device and inode. Of those processes, the topmost are taken: the ones whose parent doesn't hold
the file too, so a job and its children are one holder. The lock's row says `found_by` and keeps the kernel's
pid in `kernel_holders`. The stopped and possibly-hung checks then walk the real job's tree.

If no process can be found with the file open, the lock can't be read, and that's a blind spot, never "no
holder". The reason names the kernel's pid, and how many processes' open files couldn't be read (another
user's, or a sandboxed one's). Under `--prompts-only` that fails the run (RAL-174). When holders are found while
some processes' files couldn't be read, `unreadable_processes` says how many. Kernel threads open no files and
aren't counted. Only processes this user can see are read.

## A fair slot pool's queue (RAL-188)

From the DiscoSim rally's issues log, 1.84: one team's 25 `dsm-heavy` jobs took most of the heavy slots, and
Expand Down
7 changes: 7 additions & 0 deletions src/in_flight.rs
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,13 @@ pub fn lock_row(lock: &str, seen: &Seen, owner: Option<&str>) -> Value {
"lock": lock, "held": true, "holders": holders, "kind": "observed",
"owner": owner, "owner_kind": "declared: the first word of the lock file",
});
if seen.by_descriptor {
row["found_by"] = json!(crate::lock_holder::BY_DESCRIPTOR);
row["kernel_holders"] = json!(seen.kernel_holders);
if seen.unreadable > 0 {
row["unreadable_processes"] = json!(seen.unreadable);
}
}
if seen.matched == Matched::InodeOnly {
row["matched_by"] = json!(
"inode only: no line had the file's device, so a file elsewhere with the same inode \
Expand Down
113 changes: 113 additions & 0 deletions src/lock_holder.rs
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,56 @@ pub struct Seen {
pub cmdlines: BTreeMap<u32, String>,
/// How long the machine had been up when this was read, in milliseconds (RAL-163).
pub uptime_ms: u64,
/// The holders `/proc/locks` names, when they aren't the ones above (RAL-192): the process
/// that took the lock, which may have exited while its children kept the file open, and the
/// lock with it, as `dsm-heavy`'s bash does once it execs `systemd-run --scope`.
pub kernel_holders: Vec<u32>,
/// Whether the holders were found by their open descriptors on the file (RAL-192).
pub by_descriptor: bool,
/// How many processes' open files couldn't be read while looking for them (RAL-192).
pub unreadable: usize,
}

/// The processes with an open descriptor on the file with this device and inode, from each
/// process's `/proc/<pid>/fd` (RAL-192), with how many processes' descriptors couldn't be read.
/// A descriptor is matched by its link naming `path`, then by the device and inode of what it
/// opens, so a file renamed or replaced since isn't taken for this one.
#[cfg(target_os = "linux")]
fn open_on(path: &Path, dev: u64, ino: u64, pids: impl Iterator<Item = u32>) -> (Vec<u32>, usize) {
use std::os::unix::fs::MetadataExt;
let mut found = Vec::new();
let mut unreadable = 0;
for pid in pids {
let Ok(fds) = std::fs::read_dir(format!("/proc/{pid}/fd")) else {
// Gone since the listing, or another user's: only the second is a blind spot.
if Path::new(&format!("/proc/{pid}")).exists() {
unreadable += 1;
}
continue;
};
let holds = fds.flatten().any(|fd| {
std::fs::read_link(fd.path()).is_ok_and(|target| target == path)
&& std::fs::metadata(fd.path()).is_ok_and(|m| m.dev() == dev && m.ino() == ino)
});
if holds {
found.push(pid);
}
}
(found, unreadable)
}

/// Of the processes holding a file open, those whose parent doesn't hold it too: the top of each
/// tree, so a job and its children are one holder.
fn topmost(holding: &[u32], procs: &BTreeMap<u32, Proc>) -> Vec<u32> {
holding
.iter()
.copied()
.filter(|pid| {
procs
.get(pid)
.is_none_or(|proc| !holding.contains(&proc.ppid))
})
.collect()
}

/// Below this share of one CPU over the sample, a lock holder's whole process tree is possibly
Expand Down Expand Up @@ -365,6 +415,11 @@ pub fn judge(lock: &str, seen: &Seen, roster: &BTreeMap<String, String>) -> Opti
}))
}

/// How a holder found by its open descriptor is labelled (RAL-192).
pub const BY_DESCRIPTOR: &str = "open descriptor: the process /proc/locks names (kernel_holders) \
had exited or didn't have the file open, so the holders are the topmost processes with an \
open descriptor on it, by device and inode";

/// Reads one lock's holders, waiters and the processes on this machine, from `/proc`.
#[cfg(target_os = "linux")]
pub fn read(lock: &Path) -> Result<Seen, String> {
Expand All @@ -388,6 +443,7 @@ pub fn read(lock: &Path) -> Result<Seen, String> {
if seen.holders.is_empty() {
return Ok(seen);
}
let canonical = std::fs::canonicalize(lock).unwrap_or_else(|_| lock.to_path_buf());
let entries = std::fs::read_dir("/proc").map_err(|e| format!("can't list /proc: {e}"))?;
for entry in entries.flatten() {
let Some(pid) = entry
Expand All @@ -404,6 +460,39 @@ pub fn read(lock: &Path) -> Result<Seen, String> {
seen.procs.insert(pid, proc);
}
}
// The kernel names the process that took the lock. If it's gone, or doesn't have the file
// open, the lock is held by whatever inherited its descriptor: find those (RAL-192).
let (kernel_open, _) = open_on(&canonical, dev, meta.ino(), seen.holders.iter().copied());
if seen.holders.iter().any(|pid| !kernel_open.contains(pid)) {
// Kernel threads (kthreadd, pid 2, and its children) open no files and show no fd list.
let user_space = seen
.procs
.values()
.filter(|proc| proc.pid != 2 && proc.ppid != 2)
.map(|proc| proc.pid);
let (holding, unreadable) = open_on(&canonical, dev, meta.ino(), user_space);
seen.unreadable = unreadable;
if holding.is_empty() {
let gone: Vec<String> = seen.holders.iter().map(u32::to_string).collect();
return Err(if unreadable > 0 {
format!(
"its holder by /proc/locks ({}) doesn't have it open, and the open files of \
{unreadable} process{} couldn't be read, so what holds it can't be told",
gone.join(", "),
if unreadable == 1 { "" } else { "es" }
)
} else {
format!(
"its holder by /proc/locks ({}) doesn't have it open, and no process this user \
can see does",
gone.join(", ")
)
});
}
seen.kernel_holders = std::mem::take(&mut seen.holders);
seen.holders = topmost(&holding, &seen.procs);
seen.by_descriptor = true;
}
let wanted: Vec<u32> = seen
.holders
.iter()
Expand Down Expand Up @@ -501,6 +590,13 @@ pub fn check_with(
"waiters": seen.waiters.len(), "kind": "observed",
"note": "Open file description locks (pid -1) are skipped.",
});
if seen.by_descriptor {
row["found_by"] = json!(BY_DESCRIPTOR);
row["kernel_holders"] = json!(seen.kernel_holders);
if seen.unreadable > 0 {
row["unreadable_processes"] = json!(seen.unreadable);
}
}
match seen.matched {
Matched::Nothing => {
row["holder"] = json!("no holder found in /proc/locks");
Expand Down Expand Up @@ -559,6 +655,23 @@ mod tests {
/// RAL-163: a holder's tree that used almost no CPU between two reads, and has run long
/// enough, is possibly hung, named by its busiest process's command and working directory,
/// with the team guessed from a nickname in that directory.
/// RAL-192: of the processes holding a file open, only those whose parent doesn't hold it.
#[test]
fn a_job_and_its_children_holding_a_lock_are_one_holder() {
let procs: BTreeMap<u32, Proc> = [
proc(10, 1, 'S', "timeout"),
proc(11, 10, 'R', "godot"),
proc(12, 11, 'S', "godot"),
proc(20, 1, 'S', "bash"),
]
.into_iter()
.map(|p| (p.pid, p))
.collect();
assert_eq!(topmost(&[10, 11, 12, 20], &procs), [10, 20]);
// A parent that doesn't hold it doesn't count; one gone since the listing is a top.
assert_eq!(topmost(&[11, 12, 99], &procs), [11, 99]);
}

#[test]
fn a_holder_doing_almost_nothing_is_possibly_hung() {
let at = |cpu: u64| {
Expand Down
96 changes: 96 additions & 0 deletions tests/service.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4069,6 +4069,102 @@ fn a_lock_holder_doing_almost_nothing_is_flagged_as_possibly_hung() {
assert_eq!(row["kind"], "observed");
}

/// RAL-192: a lock taken by a shell that started its job and exited, as dsm-heavy's bash does once
/// it execs `systemd-run --scope`, is held by the child that inherited its descriptor, while
/// /proc/locks still names the shell. The holder is found by its open descriptor, the kernel's pid
/// kept beside it, so stalls' stopped and possibly-hung checks prompt for the real job.
#[cfg(target_os = "linux")]
#[test]
fn a_lock_whose_taker_exited_is_held_by_the_child_with_its_descriptor() {
let service = start();
let pairing = service.add("a1", false);
let config = service.dir.path().join("client/config.json");
assert!(pair(&config, &pairing.code()).status.success());
let slots = service.dir.path().join("slots");
std::fs::create_dir_all(&slots).unwrap();
let lock = slots.join("heavy.1.lock");
let child_pid = slots.join("child.pid");
// The shell takes the lock, starts a job that inherits the descriptor, says the job's pid and
// exits; the job keeps the lock.
let taker = Command::new("sh")
.args([
"-c",
"exec 9>>\"$0\"; flock 9; sleep 60 </dev/null >/dev/null 2>&1 & echo $! > \"$1\"; exit 0",
])
.arg(&lock)
.arg(&child_pid)
.output()
.unwrap();
assert!(taker.status.success());
let job: u32 = std::fs::read_to_string(&child_pid)
.unwrap()
.trim()
.parse()
.unwrap();
let pattern = slots.join("heavy.*.lock").display().to_string();
let stalls = |extra: &[&str]| {
let out = rally()
.arg("--config")
.arg(&config)
.env("RALLY_ROSTER_FROM", "no-such-shim-for-tests")
.args(["stalls", "--lock", &pattern])
.args(extra)
.output()
.unwrap();
assert!(
out.status.success(),
"{}",
String::from_utf8_lossy(&out.stderr)
);
serde_json::from_slice::<Value>(&out.stdout).unwrap()
};
// Idle, and running past --hung-after: possibly hung, named by the job's own pid.
std::thread::sleep(Duration::from_millis(1200));
let out = stalls(&["--hung-after", "1s", "--cpu-sample", "1s"]);
let row = &out["locks"][0];
assert_eq!(row["holders"], json!([job]), "{out}");
assert_eq!(row["found_by"], rally::lock_holder::BY_DESCRIPTOR);
let kernel = row["kernel_holders"].as_array().unwrap();
assert_eq!(kernel.len(), 1, "{out}");
assert_ne!(
kernel[0],
json!(job),
"the kernel names the shell that took it: {out}"
);
let hung = out["prompts"]
.as_array()
.unwrap()
.iter()
.find(|p| p["row"] == "possibly_hung_lock_holder")
.unwrap_or_else(|| panic!("no possibly_hung_lock_holder: {out}"));
assert_eq!(hung["processes"][0]["holder"], json!(job));
assert!(
hung["processes"][0]["summary"]
.as_str()
.unwrap()
.starts_with("possibly hung: sleep 60, "),
"{hung}"
);
// Stopped: the stopped-holder prompt names it.
let stop = Command::new("kill")
.args(["-STOP", &job.to_string()])
.status()
.unwrap();
assert!(stop.success());
let out = stalls(&[]);
let stopped = out["prompts"]
.as_array()
.unwrap()
.iter()
.find(|p| p["row"] == "stopped_lock_holder")
.unwrap_or_else(|| panic!("no stopped_lock_holder: {out}"));
assert_eq!(stopped["processes"][0]["pid"], json!(job), "{stopped}");
assert_eq!(stopped["processes"][0]["state"], "T");
let _ = Command::new("kill")
.args(["-KILL", &job.to_string()])
.status();
}

/// RAL-188: a fair slot pool's waiters hold files of their own in a waiting directory, not the
/// slots, so stalls reads that directory beside the pattern: each held slot's owner, each waiter
/// by owner and how long, and a prompt when one has waited past --queue-wait-after while another
Expand Down
Loading