Status-Server liveness, multi-server comparison, --bind source interface

README.md

@@ -57,6 +57,7 @@ Skipped this step? The probe notices: on a first-run timeout it prints this exac
 
 | Check | What it proves |
 |---|---|
+| **Status-Server** | An [RFC 5997](https://www.rfc-editor.org/rfc/rfc5997) liveness ping that runs first and **consumes no authentication attempt** — nothing shows up in the server's auth log. PASS if the server answers; a neutral INFO (never a failure) if it doesn't, since many servers leave it off. See [Liveness & multi-server](#liveness--comparing-servers). |
 | **Reachability** | The server answers on UDP/1812 — and how fast. A timeout means unreachable, not listening, **or the probe isn't whitelisted / the secret is wrong** (servers silently drop unverifiable requests). |
 | **Shared secret** | Cryptographically verifies the server's reply signature. A pass *proves* the secret matches — no more guessing whether "everyone's getting rejected" is a secret problem or something else. |
 | **BlastRADIUS posture** | Observes whether the server signs its replies with a **Message-Authenticator** — the mitigation for the RADIUS/UDP reply-forgery flaw [CVE-2024-3596](https://blastradius.fail) ("BlastRADIUS"). PASS if it does; WARN, with config pointers, if it accepts the probe's (signed) request but replies unsigned. Observation only — see below. |
@@ -193,7 +194,7 @@ $ authhound-probe radsec test --server radius.corp.com \
 
 | Flag | Purpose |
 |---|---|
-| `--server HOST[:port]` | RADIUS server (default port 1812). **Required.** |
+| `--server HOST[:port]` | RADIUS server (default port 1812). **Required.** Comma-separate several to compare them — see [Comparing servers](#liveness--comparing-servers). |
 | `--secret SECRET` | Shared secret (**required**, but prefer `AUTHHOUND_SECRET` / `--secret-file` / `--secret-stdin` — see [below](#where-credentials-come-from)). |
 | `--secret-file FILE` | Read the shared secret from a file (must not be world-readable on unix). |
 | `--secret-stdin` | Read the shared secret from standard input (one line). |
@@ -211,6 +212,7 @@ $ authhound-probe radsec test --server radius.corp.com \
 | `--server-name NAME` | Expected server-certificate name (TLS SNI). |
 | `--nas-id NAME` | NAS-Identifier to send (default `authhound-probe`). |
 | `--timeout DURATION` | Per-request timeout (default `5s`). |
+| `--bind IP[:port]` | Source IP to send from, for pinning the outgoing interface on a multi-homed host — see [Binding a source interface](#binding-a-source-interface---bind). |
 | `--json` | Machine-readable output for scripts / RMM ([schema](docs/json-schema.md)). |
 | `--strict` | Exit non-zero on **warnings** too (e.g. a soon-to-expire cert), for scheduled monitoring. |
 | `--no-color` | Force plain output. Colour is auto-detected otherwise — see [Colour](#colour-and-windows-terminals). |
@@ -328,6 +330,72 @@ With `--json`, each iteration's results plus per-check aggregate statistics
 (success counts, timeouts, latency min/median/p95/max) appear in an additive
 `repeat` block — see the [schema](docs/json-schema.md).
 
+### Liveness & comparing servers
+
+**Status-Server** runs first: an [RFC 5997](https://www.rfc-editor.org/rfc/rfc5997)
+liveness ping that a server answers *without* logging an authentication attempt.
+It's the polite way to ask "are you alive?" — nothing shows up in the auth log.
+Many servers leave it off, so silence here is a neutral **INFO**, never a
+failure; the reachability check below is the authoritative one. To turn it on in
+FreeRADIUS, set `status_server = yes` in `radiusd.conf`.
+
+**Comparing servers.** Almost every site runs a primary and a secondary RADIUS
+server, and clients reach them through DNS round-robin or a shared VIP. When one
+of the pair is quietly broken — down, unregistered, or drifted out of config —
+roughly half of authentications fail *depending on which server the client
+happened to hit*. That is the single most common hidden cause of "it works
+sometimes" tickets, and a single-server test can't see it. Comma-separate the
+servers and the probe tests each, then prints a comparison:
+
+```console
+$ export AUTHHOUND_SECRET='shared-secret'
+$ authhound-probe radius test --server radius1.corp.com,radius2.corp.com --pap alice
+
+=== Server 1/2: radius1.corp.com:1812 ===
+... per-check results ...
+Verdict: 4 passed, 0 failed, 0 warnings, 5 skipped
+
+=== Server 2/2: radius2.corp.com:1812 ===
+... per-check results ...
+Verdict: 0 passed, 1 failed, 0 warnings, 8 skipped
+
+Comparison across servers:
+  radius1.corp.com:1812 is responding, but radius2.corp.com:1812 is NOT. If
+  clients reach these servers via DNS round-robin or a shared VIP, roughly 50%
+  of authentications would fail intermittently depending on which server they
+  land on — the classic 'it works sometimes' ticket. Take the unresponsive
+  server(s) out of rotation or bring them back.
+```
+
+The verdict also covers the subtler case where both servers answer but
+**disagree** — one accepts a login the other rejects, or assigns a different
+VLAN — which points at config or replication drift between them.
+
+The exit code follows the combined result: a FAIL on *any* server fails the run
+(and under `--strict`, a WARN does too). Each server is still bounded by the same
+hard-coded rate ceiling; comparing servers never raises the load on any one of
+them. `--count` and multiple `--server` are mutually exclusive — chase
+intermittency on one server, compare across servers separately. With `--json`,
+per-server blocks and the verdict appear in additive `servers` / `comparison`
+fields — see the [schema](docs/json-schema.md#servers--comparison-multiple---server).
+
+### Binding a source interface (`--bind`)
+
+Jump boxes and monitoring servers are usually multi-homed. `--bind IP[:port]`
+pins the source address the probe sends from, so RADIUS leaves the interface you
+intend (and reaches a server whose firewall only permits that address):
+
+```console
+$ authhound-probe radius test --server radius.corp.com --bind 10.20.0.5
+```
+
+The source must be a local IP literal on this host (not a hostname). This is also
+the address the server sees, so it's the one to register as a RADIUS client — and
+the [Step 0 registration snippet](#step-0--register-the-probe-on-your-server-one-time)
+the probe prints on a timeout automatically reflects the bound IP, so you can
+paste it as-is. (If NAT sits between this host and the server, the server still
+sees the post-NAT address — register that instead.)
+
 ### Where credentials come from
 
 This tool is meant to run on shared jump boxes, so it never *requires* a secret or

cmd/authhound-probe/leak_test.go

@@ -32,6 +32,7 @@ func TestNoCredentialLeak(t *testing.T) {
 		NASIdentifier: "authhound-probe",
 	}
 	checks := []check.Check{
+		check.StatusServer{},
 		check.Reachability{},
 		check.SharedSecret{},
 		check.PAP{User: "alice", Pass: sentinelPass},

cmd/authhound-probe/main.go

@@ -13,8 +13,10 @@ package main
 
 import (
 	"context"
+	"errors"
 	"flag"
 	"fmt"
+	"net"
 	"os"
 	"os/signal"
 	"runtime/debug"
@@ -173,7 +175,7 @@ func cmdRadsecTest(args []string) int {
 
 func cmdRadiusTest(args []string) int {
 	fs := flag.NewFlagSet("radius test", flag.ExitOnError)
-	server := fs.String("server", "", "RADIUS server host or host:port (default port 1812)")
+	server := fs.String("server", "", "RADIUS server host or host:port (default port 1812); comma-separate several to compare them (e.g. primary,secondary)")
 	secret := fs.String("secret", "", "shared secret (leaks into shell history/ps — prefer AUTHHOUND_SECRET, --secret-file, or the prompt)")
 	secretFile := fs.String("secret-file", "", "read the shared secret from this file (must not be world-readable on unix)")
 	secretStdin := fs.Bool("secret-stdin", false, "read the shared secret from standard input (one line)")
@@ -193,6 +195,7 @@ func cmdRadiusTest(args []string) int {
 	count := fs.Int("count", 1, "run the checks N times (2..50) and report aggregate statistics — for chasing intermittent failures")
 	interval := fs.Duration("interval", 2*time.Second, "pause between --count iterations (a hard-coded safety floor applies)")
 	timeout := fs.Duration("timeout", 5*time.Second, "per-request timeout")
+	bind := fs.String("bind", "", "source IP[:port] to send from, for pinning the outgoing interface on a multi-homed host")
 	jsonOut := fs.Bool("json", false, "emit results as JSON instead of text")
 	noColor := fs.Bool("no-color", false, "disable ANSI colour")
 	strict := fs.Bool("strict", false, "exit non-zero on warnings too (for scheduled monitoring)")
@@ -211,9 +214,20 @@ func cmdRadiusTest(args []string) int {
 	}
 	provided := map[string]bool{}
 	fs.Visit(func(f *flag.Flag) { provided[f.Name] = true })
-	addr := *server
-	if !strings.Contains(addr, ":") {
-		addr += ":1812"
+
+	servers, err := parseServers(*server)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "error:", err)
+		return 2
+	}
+
+	// --bind pins the outgoing interface on a multi-homed host. The resolved
+	// source address flows into every socket the run opens, so the detected
+	// source IP (and thus the registration hint) reflects the bind.
+	localAddr, err := resolveBindAddr(*bind)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "error:", err)
+		return 2
 	}
 
 	portType, ok := nasPortTypes[*nasPortType]
@@ -233,6 +247,13 @@ func cmdRadiusTest(args []string) int {
 		fmt.Fprintln(os.Stderr, "error: --interval only makes sense with --count")
 		return 2
 	}
+	// --count multiplies requests against ONE server; comparing several servers
+	// is a different job. Keeping them separate keeps the output legible and the
+	// per-server load obviously bounded. Test one server at a time with --count.
+	if len(servers) > 1 && *count != 1 {
+		fmt.Fprintln(os.Stderr, "error: --count tests a single server; drop it to compare multiple --server entries, or test one server at a time")
+		return 2
+	}
 
 	prompter := credential.Default()
 	secretValue, err := prompter.Resolve(credential.Spec{
@@ -250,12 +271,14 @@ func cmdRadiusTest(args []string) int {
 		return 2
 	}
 
+	// Address is set per server below; everything else is shared across a
+	// multi-server comparison.
 	target := check.Target{
-		Address:       addr,
 		Secret:        secretValue,
 		Timeout:       *timeout,
 		NASIdentifier: *nasID,
 		NASPortType:   portType,
+		LocalAddr:     localAddr,
 	}
 
 	papUser, papPass, err := resolveCreds(prompter, *pap, "--pap", *passwordFile)
@@ -291,30 +314,38 @@ func cmdRadiusTest(args []string) int {
 	}
 	target.Expect = expect
 
+	// Status-Server runs first: an RFC 5997 liveness ping that doesn't consume an
+	// auth attempt. The rest follow in dependency order (reachability/secret before
+	// the auth methods that rely on them).
+	checks := []check.Check{
+		check.StatusServer{},
+		check.Reachability{},
+		check.SharedSecret{},
+		check.BlastRADIUS{},
+		check.PAP{User: papUser, Pass: papPass},
+		check.PEAPMSCHAPv2{User: peapUser, Pass: peapPass, ServerName: *serverName},
+		check.EAPTTLS{User: ttlsUser, Pass: ttlsPass, ServerName: *serverName},
+		check.EAPTLS{CertFile: *clientCert, KeyFile: *clientKey, ServerName: *serverName},
+		check.ServerCert{ServerName: *serverName},
+		check.MTUProbe{Enabled: *mtu},
+	}
+
+	if len(servers) > 1 {
+		return runMultiServer(target, checks, servers, *jsonOut, *noColor, *strict)
+	}
+
+	target.Address = servers[0]
+	plan := check.Plan{Target: target, Checks: checks}
+
 	// Sink: JSON for scripting, text for humans.
 	var sink resultSink
 	if *jsonOut {
 		sink = report.NewJSONSink(os.Stdout)
 	} else {
-		fmt.Printf("Testing RADIUS server %s (as NAS %q)\n\n", addr, *nasID)
+		fmt.Printf("Testing RADIUS server %s (as NAS %q)\n\n", servers[0], *nasID)
 		sink = report.NewTextSink(os.Stdout, report.UseColor(os.Stdout, *noColor))
 	}
 
-	plan := check.Plan{
-		Target: target,
-		Checks: []check.Check{
-			check.Reachability{},
-			check.SharedSecret{},
-			check.BlastRADIUS{},
-			check.PAP{User: papUser, Pass: papPass},
-			check.PEAPMSCHAPv2{User: peapUser, Pass: peapPass, ServerName: *serverName},
-			check.EAPTTLS{User: ttlsUser, Pass: ttlsPass, ServerName: *serverName},
-			check.EAPTLS{CertFile: *clientCert, KeyFile: *clientKey, ServerName: *serverName},
-			check.ServerCert{ServerName: *serverName},
-			check.MTUProbe{Enabled: *mtu},
-		},
-	}
-
 	if *count != 1 {
 		return runRepeat(plan, sink, *count, *interval, *strict, *jsonOut)
 	}
@@ -326,6 +357,119 @@ func cmdRadiusTest(args []string) int {
 	return exitCode(sink, *strict)
 }
 
+// parseServers splits the --server value on commas into normalized host:port
+// entries (default port 1812), dropping empties (a trailing comma) and
+// duplicates while preserving order. At least one server must remain.
+func parseServers(raw string) ([]string, error) {
+	seen := map[string]bool{}
+	var out []string
+	for _, part := range strings.Split(raw, ",") {
+		s := strings.TrimSpace(part)
+		if s == "" {
+			continue
+		}
+		if !strings.Contains(s, ":") {
+			s += ":1812"
+		}
+		if seen[s] {
+			continue
+		}
+		seen[s] = true
+		out = append(out, s)
+	}
+	if len(out) == 0 {
+		return nil, errors.New("--server is required")
+	}
+	return out, nil
+}
+
+// resolveBindAddr turns a --bind IP[:port] value into the source address sockets
+// bind to. Empty means "let the OS choose". The source must be an IP literal (a
+// specific local address on this host), never a hostname — binding is about
+// which interface leaves, not name resolution. Port defaults to 0 (ephemeral).
+func resolveBindAddr(s string) (*net.UDPAddr, error) {
+	if s == "" {
+		return nil, nil
+	}
+	host, port, err := net.SplitHostPort(s)
+	if err != nil {
+		// Most commonly there's no port (a bare source IP); retry with port 0.
+		host, port = s, "0"
+	}
+	if net.ParseIP(host) == nil {
+		return nil, fmt.Errorf("--bind %q: source must be a local IP address, not a hostname", s)
+	}
+	addr, err := net.ResolveUDPAddr("udp", net.JoinHostPort(host, port))
+	if err != nil {
+		return nil, fmt.Errorf("--bind %q is not a valid IP[:port]: %w", s, err)
+	}
+	return addr, nil
+}
+
+// runMultiServer runs the full check plan against each server in turn, prints
+// each server's block, then a comparison verdict — split-brain between RADIUS
+// servers being a classic hidden cause of "intermittent" auth tickets. Each
+// server is still bounded by the runner's rate ceiling; comparing servers never
+// raises the load on any one of them. The exit code follows the combined result
+// (a FAIL on any server fails the run; under --strict, a WARN does too).
+func runMultiServer(base check.Target, checks []check.Check, servers []string, jsonOut, noColor, strict bool) int {
+	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
+	defer stop()
+
+	var jsink *report.JSONSink
+	if jsonOut {
+		jsink = report.NewJSONSink(os.Stdout)
+	}
+
+	var runs []check.ServerRun
+	for i, srv := range servers {
+		if ctx.Err() != nil {
+			fmt.Fprintf(os.Stderr, "interrupted — tested %d of %d servers\n", i, len(servers))
+			break
+		}
+		target := base
+		target.Address = srv
+		plan := check.Plan{Target: target, Checks: checks}
+
+		var results []check.Result
+		if jsonOut {
+			runner := check.Runner{Sink: jsink} // accumulates combined results + summary
+			results = runner.Run(ctx, plan)
+		} else {
+			fmt.Printf("=== Server %d/%d: %s ===\n\n", i+1, len(servers), srv)
+			ts := report.NewTextSink(os.Stdout, report.UseColor(os.Stdout, noColor))
+			runner := check.Runner{Sink: ts}
+			results = runner.Run(ctx, plan)
+			_ = ts.Close()
+			fmt.Println()
+		}
+		runs = append(runs, check.ServerRun{Server: srv, Results: results})
+	}
+
+	cmp := check.CompareServers(runs)
+	if jsonOut {
+		jsink.SetServers(runs, cmp)
+		_ = jsink.Close()
+	} else {
+		fmt.Print(report.ComparisonBlock(cmp))
+	}
+	return multiExitCode(runs, strict)
+}
+
+// multiExitCode maps the combined multi-server results to the process exit code,
+// independent of the sink: 1 if any server had a FAIL, or — under --strict — a
+// WARN; otherwise 0.
+func multiExitCode(runs []check.ServerRun, strict bool) int {
+	for _, r := range runs {
+		for _, res := range r.Results {
+			if res.Status == check.StatusFail || (strict && res.Status == check.StatusWarn) {
+				return 1
+			}
+		}
+	}
+	return 0
+}
+
 // runRepeat drives --count: N sequential iterations, then the aggregate
 // verdicts through the normal sink, so text/JSON rendering and the exit-code
 // contract are identical to a single run. Ctrl-C mid-loop aggregates the