Typed disassembly: operands, flow, and the unwind region

CHANGELOG.md

@@ -8,6 +8,107 @@ All notable changes to this project are documented here. The format follows
 
 ### Added
 
+- **Disassembly carries its operands as values, decoded from the encoding.** `Instruction` gains
+  `mnemonic`, `operands` and `flow` beside the `text` it already had, so a caller asking what an
+  instruction *compares against* or *branches to* reads a field instead of re-parsing a rendering
+  downstream. They come from decoding `bytes` — the engine's own read of the instruction, so no
+  extra round trip — with `iced-x86`, and the rendering stays verbatim in `text` because it is what
+  a listing prints.
+  **Decoding, rather than reading the rendering, is the whole design and was arrived at the hard
+  way.** The first implementation parsed the third column, and a symbol's own punctuation kept
+  taking operands apart: a comma inside `std::map<int,int>`, a parenthesis inside `operator()`, a
+  bracket inside `operator[]` — three review rounds, three characters, each severing a direct call
+  edge that a walk then dropped as indirect. The mnemonic table had the same shape, growing an
+  entry a round for `int` by vector, `xbegin`, `xabort` and `hlt`, because a hand-written list of
+  what transfers control is never finished. An encoding has neither ambiguity, and a decoder's
+  flow control is complete by construction. Symbols leave the picture entirely: a destination is an
+  address, and naming it is `symbol_for`'s job.
+  `Operand` is `Register`, `Immediate`, `Memory`, `Target` or `Other`. `Flow` carries every
+  destination as an `Option`, because a direct transfer encodes a displacement and an indirect one
+  encodes a register, and a caller treating `None` as "no edge" stays sound. `Unknown` and
+  `Unreadable` are separate, and the line between them is whether there are bytes: a `???`
+  rendering has none and stops a walk, while an instruction set this does not decode — or an
+  encoding newer than the pinned decoder — has an instruction there and falls through. A walk that
+  fell through the first would step through *bytes*, one address at a time, to its own cap; one
+  that stopped at the second would discard the rest of a routine over a version skew.
+  A memory operand claims a static `address` only where the instruction alone determines one. A
+  segment override does not: `gs:[188h]` is the KPCR, its linear address is the segment base plus
+  the displacement, and that base is a runtime fact. A RIP-relative operand keeps the displacement
+  it **encodes** rather than the decoder's normalised target, which would otherwise report
+  `[rip+0xffa]` at `0x1000` as a displacement of `0x2000` — a second copy of `address` where the
+  addressing expression should be. A displacement is signed at the width its *address registers*
+  are: the decoder keeps a 32-bit effective address in 32 bits, so `[ebp-8]` arrives as
+  `0xfffffff8` and a straight widening cast reported the commonest local-variable reference there
+  is as 4,294,967,288. An absolute reference with no register stays unsigned, a 32-bit
+  `[0xfffff000]` being a high address rather than a negative offset.
+  A near branch's destination lands in the address space its instruction came from. Decoding 32-bit
+  code computes a 32-bit target, so an instruction whose own address carries a high half — a narrow
+  effective machine over a wide address, which `.effmach x86` produces — would otherwise name a
+  destination in a different address space from itself, which a module-bounds check rejects and a
+  reader follows to the wrong place. The high half is inherited from the instruction rather than
+  sign-extended, taking the address form from the caller's own value instead of assuming the
+  engine's. 64-bit decoding is left alone deliberately: a `rel32` reaches ±2 GB and so may cross a
+  4 GB boundary, where inheriting would drag a correct target back four gigabytes. An **absolute**
+  memory address is canonicalised the same way and for the same reason, absolute globals and
+  import slots being ordinary in x86 kernel code.
+  Inherited rather than sign-extended, and that is measured rather than preferred: against a
+  32-bit target, `? 80002000` evaluates to `80002000` and `.formats` prints `Hex: 80002000` —
+  eight digits, unextended — so sign-extending would invent `ffffffff80002000` for an address the
+  engine calls `80002000`, and on a 32-bit kernel would do so for nearly every address. The cost
+  is one straddling case, a low instruction reaching a high absolute, which keeps the low half and
+  is pinned by a test naming the measurement.
+  A displacement's signed width is the **effective address width**, and neither of the two simpler
+  readings of that survives: the *index register's* width breaks a VSIB gather, whose index is an
+  `xmm`/`ymm`/`zmm`, so the extension becomes a no-op and a negative displacement comes back as
+  four billion; the *encoded field's* width breaks EVEX, whose `disp8` is compressed, so the
+  decoder returns it already scaled by the tuple while the field is still one byte and extending
+  from eight bits turns a real `+128` into `-128`. The address registers give the width where
+  there are any, an address-size override being exactly what makes them narrow, and the encoded
+  field gives it for a pure VSIB form, where compression cannot arise because it needs a base. All
+  three readings are pinned, so the two wrong ones fail a test rather than being re-proposed.
+  An `EIP`-relative operand — a 64-bit instruction under an address-size override — has its
+  displacement computed at 32 bits, the decoder wrapping the target to that width while the
+  instruction's own next address stays 64, which otherwise puts the delta four gigabytes out.
+  Reading is gated on `InstructionSet`: x86 and x64 are decoded, and anything else — ARM64 today —
+  reports its mnemonic, no operands and `Flow::Unknown`.
+  Measured against a whole real dispatch routine rather than composed lines
+  (`examples/typed_disassembly.rs`): 376 instructions of `mountmgr!MountMgrDeviceControl` on a
+  26100 image, **zero** unrecognised operands, zero unknown flows, and its eleven control-code
+  compares recovered as values — identical before and after the decoder replaced the parser.
+- `DebugEngine::decode_range` decodes a span from **one** memory read instead of one engine call
+  per instruction, which is what a bounded traversal over a hundred functions needs. Its
+  instructions carry no `text`, nothing having rendered them; a caller needing a rendering for the
+  few it displays asks `disassemble` for those. The two paths are compared against each other in
+  the example over a real function's first region: 23 instructions, 23 compared, 0 disagreements.
+- `DebugEngine::effective_processor_type` reports the processor the engine is **rendering** in, as
+  against the physical one `processor_type` already answered. The two diverge wherever one machine
+  runs another's code — a WOW64 process, x64 emulated on ARM64, any target after `.effmach` — and
+  it is the effective one that discriminates a *rendering*, so `instruction_set` reads it. Measured
+  by forcing the divergence: `.effmach x86` on an x64 kernel dump moves the effective type to
+  `0x14c` while the physical stays `0x8664`, and the reading follows it rather than decoding an x64
+  unwind record against x86 output. Anything reading the target's **structures** still wants the
+  physical type, a pointer's width being a fact about the machine rather than about a rendering,
+  so the pool and heap walkers are unchanged.
+- `DebugEngine::function_extent` returns the unwind **region** containing an address, from the
+  image's `.pdata`, rebased. A region is **not** a function, and using it as one loses code: MSVC
+  splits a function across several entries, and this answers `0x14750..0x147a3` for
+  `mountmgr!MountMgrDeviceControl` — 83 bytes, which `.fnent` confirms — while that routine's
+  compare chain lives past `0x147dd`. Bounding a walk with it recovered zero control codes where
+  following the flow recovered twelve. The x64 entry is three `u32` RVAs and not the 64-bit
+  addresses the API's name suggests; reading them as `u64` reports "no entry" for a function that
+  plainly has one.
+  It answers a three-state `FunctionExtent` rather than an `Option`, because the two non-answers
+  are different facts. `NoEntry` is reported for the one measured failure that means it —
+  `E_NOINTERFACE`, which a real dbgeng 10.x gives for address zero, for a header page, and for
+  every x86 address, 32-bit Windows having no unwind table — and any other failure is returned as
+  an error rather than read as a leaf. `Unsupported` covers every instruction set but x64:
+  ARM64's record is two words whose second is packed unwind data, measured as `needed = 8` with
+  `[0x0025df60, 0x0005f218]` for `nt!KeBugCheckEx` on an ARM64 kernel dump, and read as an end
+  address that is a bogus region which — for any function below the `.xdata` RVA — contains the
+  address asked about and passes every sanity check. The engine's own `needed` is checked against
+  the x64 shape rather than assumed.
+- `DebugEngine::symbol_for` is the public half of the existing symbol lookup: the `module!Symbol`
+  an address resolves to and how far past it, or `None` for a driver with no PDB.
 - **Exception events are readable as values.** `DebugEngine::last_event` returns a `DebugEvent` —
   kind, engine process and thread, and, when the event carried one, an `ExceptionRecord` with the
   code, flags, faulting address and parameters. That is `.exr -1` typed, and it is the user-mode

Cargo.toml

@@ -29,6 +29,16 @@ exclude = [
 crate-type = ["rlib", "cdylib"]
 
 [dependencies]
+# The x86/x64 instruction decoder behind `Instruction`'s typed fields. The engine renders
+# disassembly as text and offers no structured form, and recovering operands from that rendering
+# means deciding what a character means from the fact that it is present -- which does not survive
+# real symbols: `std::map<int,int>` puts a comma inside one, `operator()` a parenthesis,
+# `operator[]` a bracket, and each of those severed a direct call edge in turn. Decoding the
+# encoding the engine already hands back has none of those ambiguities and needs no extra round
+# trip. Default features off: `decoder` and `instr_info` are the whole of what is used -- no
+# formatter, because the engine's own rendering is what this crate promises, and no encoder. It
+# brings one transitive dependency, `lazy_static`, for the tables `instr_info` reads.
+iced-x86 = { version = "1.21", default-features = false, features = ["std", "decoder", "instr_info"] }
 hex = "0.4.3"
 thiserror = "2.0.18"
 windows-core = "0.62.2"

examples/function_entry_probe.rs

@@ -0,0 +1,81 @@
+//! Measurement for `DebugEngine::function_extent`: what the engine fills in, and what it says
+//! when there is nothing to fill.
+//!
+//! Two questions the API's documentation does not answer, and both decide code:
+//!
+//! - **Which failure means "no entry"?** `GetFunctionEntryByOffset` returns `Result<()>`, and a
+//!   leaf function, a data address and a broken engine are all `Err`. Mapping every one of them to
+//!   `None` makes a failed query indistinguishable from a function that has no unwind record.
+//! - **What is the entry's layout off x64?** The x64 record is three `u32` RVAs. ARM64's is two
+//!   words whose second is packed unwind data or an `.xdata` RVA — so reading it as an end address
+//!   is a bogus extent rather than an error, which is the worst shape a wrong answer can take.
+//!
+//! ```text
+//! cargo run --example function_entry_probe -- <dump> <addr-or-symbol>... [--exepath <path>]
+//! ```
+
+use dbgscope::dbgeng::{DebugEngine, FunctionExtent};
+
+fn main() {
+    let mut args = std::env::args().skip(1);
+    let Some(dump) = args.next() else {
+        eprintln!("usage: function_entry_probe <dump> <addr-or-symbol>... [--exepath <path>]");
+        std::process::exit(2);
+    };
+    let mut wanted = Vec::new();
+    let mut image_path = None;
+    let mut effmach = None;
+    while let Some(arg) = args.next() {
+        match arg.as_str() {
+            "--exepath" => image_path = args.next(),
+            // `.effmach` is the one way to make the physical and effective processor types
+            // disagree on a fixture that is not a WOW64 or emulated target.
+            "--effmach" => effmach = args.next(),
+            _ => wanted.push(arg),
+        }
+    }
+
+    let e = DebugEngine::new();
+    e.open_dump(&dump).expect("opening the dump failed");
+    e.wait_for_event(30_000).expect("the dump did not load");
+    if let Some(path) = &image_path {
+        e.execute_command(&format!(".exepath+ {path}"))
+            .expect("setting the image search path failed");
+        e.reload_symbols("/f").expect("reloading failed");
+    }
+
+    if let Some(machine) = &effmach {
+        e.execute_command(&format!(".effmach {machine}"))
+            .expect("setting the effective machine failed");
+    }
+
+    // Physical against effective: `Disassemble` renders with the second, so the second is what
+    // discriminates the reading. They diverge wherever one machine runs another's code.
+    println!(
+        "processor: physical {:?} effective {:?} -> {:?}\n",
+        e.processor_type().map(|m| format!("{m:#x}")),
+        e.effective_processor_type().map(|m| format!("{m:#x}")),
+        e.instruction_set()
+    );
+    for name in &wanted {
+        let address = match name.strip_prefix("0x") {
+            Some(hex) => u64::from_str_radix(hex, 16).expect("a hexadecimal address"),
+            None => match e.symbol_offset(name) {
+                Ok(address) => address,
+                Err(error) => {
+                    println!("{name}: did not resolve ({error})\n");
+                    continue;
+                }
+            },
+        };
+        println!("{name} = {address:#x}");
+        match e.function_extent(address) {
+            Ok(FunctionExtent::Region { begin, end }) => {
+                println!("  region {begin:#x}..{end:#x} ({} bytes)\n", end - begin)
+            }
+            Ok(FunctionExtent::NoEntry) => println!("  no entry\n"),
+            Ok(FunctionExtent::Unsupported(set)) => println!("  not decoded for {set:?}\n"),
+            Err(error) => println!("  error: {error}\n"),
+        }
+    }
+}