Skip to content

Add AI skills: analyze-debug-files, find-related-github-issues, trace-code-introduction - #2584

Draft
David Paulson (dpaulson45) wants to merge 5 commits into
mainfrom
dpaul-AISkill
Draft

Add AI skills: analyze-debug-files, find-related-github-issues, trace-code-introduction#2584
David Paulson (dpaulson45) wants to merge 5 commits into
mainfrom
dpaul-AISkill

Conversation

@dpaulson45

Copy link
Copy Markdown
Member

Summary

Adds three new Copilot AI skills for CSS-Exchange debug-log triage, plus a supporting extension to the existing find-release-tag-for-script-version skill.

New skills

  • analyze-debug-files — end-to-end pipeline that ingests a CSS-Exchange debug log, identifies the source script + release-tag baseline, walks the dependency graph via .build/Build.ps1 output, and writes a root-cause report (DebugAnalysis-<timestamp>.md) into the caller-supplied directory.
  • find-related-github-issues — searches microsoft/CSS-Exchange issues for prior reports of a given exception. Invocable directly during triage or as a Step 7 sub-invocation of analyze-debug-files.
  • trace-code-introduction — resolves a specific source line + range at a pinned SHA to the commit that introduced it (via git log -L and git blame). Used by analyze-debug-files to attribute unhandled exceptions to their originating change.

Extension

  • find-release-tag-for-script-version now emits a ConfirmedCommitSha field alongside ConfirmedTag. analyze-debug-files needs the 40-character SHA to build a scratch worktree deterministically and to key its per-SHA dependency cache; a tag alone is ambiguous once tags are re-pointed.

Notable design decisions

  • Per-SHA dependency cache at %LOCALAPPDATA%\CSS-Exchange\dependency-cache\<sha>\. On cache hit, analyze-debug-files skips the ~107 s .build/Build.ps1 invocation (measured ~74× speedup). The cache is worktree-independent — XML keys are normalized from absolute worktree paths to repo-relative paths so Steps 6-8 can read source with git show <sha>:<path> regardless of which run produced the cache.
  • Report fidelity guardrails — Step 8's template enforces strict source-slice rules (# L<n> trailing annotations, no ellipses, consecutive-line check) with a post-render regex assertion so cited source cannot be silently truncated or paraphrased.
  • Trust model — personal-machine, no ownership preflight; the caller's supplied output directory is trusted, and the report is written directly there.
  • Input validation — all three new skills validate gh CLI arguments, git pathspecs, and shell-metacharacter surfaces before invoking subprocesses (see Get-SafeQueryPhrase, allow-list regexes in Trace-CodeIntroduction.ps1).

Testing

  • Pre-commit hooks (PSScriptAnalyzer + formatter) pass on all 4 commits.
  • .build\SpellCheck.ps1 — clean (0 issues, 810 files).
  • Manual end-to-end validated against a real HealthChecker debug log (PS 4.0 / Server 2012 R2 / -ForceLegacy scenario). Cache miss + cache hit both verified; normalized XML keys resolve via git show <sha>:<path>.

Commits

  1. Extend find-release-tag-for-script-version with ConfirmedCommitSha + cspell dictionary additions.
  2. Add find-related-github-issues skill.
  3. Add trace-code-introduction skill.
  4. Add analyze-debug-files skill.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Adds a Copilot skill that maps a CSS-Exchange script name + version stamp
(YY.MM.DD.HHMM) back to the earliest GitHub release tag that shipped that
build, by walking releases in ascending tag-date order and matching the
File + Version pair in each release's ScriptVersions.csv.

Includes:
- Find-ReleaseTagForScriptVersion.ps1 helper with structured result
  (Script, Version, ConfirmedTag, SHA256Hash, Status, WindowExhausted,
  EarlierGaps, Tried[]) and status enum distinguishing match-earliest,
  match-possibly-not-earliest, not-found-complete, not-found-inconclusive,
  and not-found-no-candidates.
- SKILL.md documenting the workflow, status semantics, per-candidate
  Tried statuses, and WorkFolder rejection rules.

Security posture:
- Repository pinned to github.com/<owner>/<repo> so an inherited GH_HOST
  cannot redirect requests or leak an ambient enterprise token.
- WorkFolder rejects UNC/extended-UNC/provider-qualified paths,
  non-FileSystem PSDrives, network/CD-ROM/unknown drive types,
  SUBST/raw-DOS-device aliases (via QueryDosDevice), and paths whose
  volume root or any existing ancestor is a filesystem reparse point.
- Downloaded CSVs go through strict header, per-row, and per-field
  validation; File comparison is ordinal, and Version/SHA256 formats
  are regex-checked before any value is surfaced.
- All caller-visible strings from downloaded content are size-capped
  and stripped of control characters.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds a Get-CommitShaForTag helper that resolves the matched release tag
to its target commit SHA via the GitHub API, and surfaces it on the
result object as ConfirmedCommitSha alongside ConfirmedTag. The resolver
pins to github.com and validates the SHA shape before returning; any
failure yields $null rather than throwing so the primary tag match is
never blocked. Downstream skills (analyze-debug-files) require a
40-character SHA to build a scratch worktree deterministically and to
key the per-SHA dependency cache; a tag alone is ambiguous once tags
are re-pointed.

Also adds five cspell dictionary entries (metacharacters, misattributed,
misrouting, triaging, worktree) used across the new skills.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New Copilot skill that, given an exception message and script name,
searches microsoft/CSS-Exchange issues for prior reports of the same
failure. Invoked directly when triaging a fresh exception, or as a
Step 7 sub-invocation of analyze-debug-files after each unhandled
finding.

Provides:
- SKILL.md contract: input shape, output shape, scoring criteria,
  hard cap on results returned to the caller.
- Find-RelatedGitHubIssues.ps1: sanitizes user-controlled exception
  content against gh CLI argument injection (Get-SafeQueryPhrase
  strips quotes, search-syntax metacharacters, backticks, newlines),
  runs 3-4 randomized query variants, and deduplicates by issue
  number before returning ranked candidates.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New Copilot skill that traces a specific source line back to the commit
that introduced it. Used by analyze-debug-files Step 7 to determine
whether an unhandled exception was authored inside the failing script's
own history (own-repo introduction) or inherited from a Shared/ file.

Provides:
- SKILL.md contract: input (repo-relative path + line range + SHA),
  output shape, caveat that git log -L reports the whole enclosing
  function/block so sibling statements added later can be
  misattributed as the introducing commit.
- Trace-CodeIntroduction.ps1: validates repository and path against
  a strict allow-list (no shell metacharacters, no traversal, no
  absolute or UNC paths), invokes git log -L and git blame against
  the pinned worktree, and returns a structured record with commit
  SHA, author, date, and message.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New Copilot skill that analyzes CSS-Exchange debug log files, identifies
the source script + release-tag baseline, and produces a root-cause
report for unhandled exceptions.

Pipeline:
- Step 1a: streaming inventory (Get-DebugFileMetadata.ps1) — detects
  CSS-Exchange output shape; hard-stops if the file is not a
  recognized script log.
- Step 2-4: identifies the script + version and confirms the release
  tag + commit SHA via find-release-tag-for-script-version.
- Step 5: obtains the dependency graph. Loads from a per-SHA cache at
  $env:LOCALAPPDATA\CSS-Exchange\dependency-cache\<sha>\ when
  available; otherwise materializes a scratch worktree, runs
  .build/Build.ps1, and populates the cache. Cache hit avoids the
  ~107-second Build.ps1 run (measured 74x speedup on the primed
  path). XML keys are normalized from absolute worktree paths to
  repo-relative form so Steps 6-8 can read source with
  git show <sha>:<path> regardless of which branch produced the XML.
- Step 6-7: per-finding source reads and BFS across the dependency
  graph, with optional sub-invocations of trace-code-introduction and
  find-related-github-issues.
- Step 8: renders DebugAnalysis-<timestamp>.md into the caller-supplied
  directory. Includes STRICT source-slice fidelity rules and a
  post-render assertion (regex-enforced format, forbidden ellipses,
  consecutive-line check) so cited source cannot be silently truncated
  or paraphrased.

Trust model: personal machine, no ownership preflight, report written
directly to the caller-supplied directory.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dpaulson45

Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical and moderate findings remain.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds three Copilot skills for CSS-Exchange debug-log triage, GitHub issue correlation, and code-introduction tracing, plus confirmed commit-SHA support for release lookup.

Changes:

  • Adds debug-log metadata parsing, dependency caching, and report generation.
  • Adds related GitHub issue search and source-history tracing.
  • Extends release resolution with ConfirmedCommitSha and updates spelling data.
File summaries
File Description
.github/skills/trace-code-introduction/Trace-CodeIntroduction.ps1 Traces source ranges to introducing commits.
.github/skills/trace-code-introduction/SKILL.md Documents code-introduction tracing.
.github/skills/find-release-tag-for-script-version/SKILL.md Documents confirmed commit-SHA output.
.github/skills/find-release-tag-for-script-version/Find-ReleaseTagForScriptVersion.ps1 Resolves release tags and commit SHAs.
.github/skills/find-related-github-issues/SKILL.md Documents related-issue lookup.
.github/skills/find-related-github-issues/Find-RelatedGitHubIssues.ps1 Searches and classifies related issues.
.github/skills/analyze-debug-files/SKILL.md Defines the end-to-end analysis workflow.
.github/skills/analyze-debug-files/Get-DebugFileMetadata.ps1 Parses and sanitizes debug logs.
.build/cspell-words.txt Adds skill-specific dictionary terms.
Review details

Suppressed comments (12)

.github/skills/analyze-debug-files/Get-DebugFileMetadata.ps1:555

  • The sanitizer removes C0 controls and DEL, but it leaves the C1 range U+0080–U+009F. Those characters can be returned in snippets and violate the report's explicit no-stray-C0/C1 assertion. Extend the character class to cover \x7F-\x9F.
    '[\x00-\x08\x0B\x0C\x0E-\x1F\x7F]',

.github/skills/analyze-debug-files/Get-DebugFileMetadata.ps1:672

  • The regex-timeout fallback uses code -ge 0x20 as its printable test, which also accepts C1 controls U+0080–U+009F. A timeout therefore bypasses the normal sanitizer's intended output guarantee and can still place those controls in the report. Use the same printable range as the normal path here.
            if (($code -ge 0x20 -and $code -ne 0x7F) -or $code -eq 0x09) {

.github/skills/analyze-debug-files/Get-DebugFileMetadata.ps1:992

  • HealthChecker's Get-ErrorsThatOccurred also emits ----Errors that occurred that was not handled remotely---- after the normal unhandled footer (see Diagnostics/HealthChecker/Helpers/Get-ErrorsThatOccurred.ps1:37-40). This state machine only recognizes the handled/unhandled headers, so a run with hidden remote errors is still marked SummaryComplete with UnhandledCount=0, causing the analyzer to report a clean run while silently dropping those errors. Add a remote-section signal/count or downgrade completion when that header is present.
                } elseif ($Script:UnhandledSummaryHeaderRegex.IsMatch($line)) {
                    $summaryState = 'unhandled'
                    $summaryUnhandledCount = 0
                    $unhandledHeaderLine = $lineNumber
                    $unhandledHeaderCount++

.github/skills/analyze-debug-files/Get-DebugFileMetadata.ps1:417

  • The logger formats [System.DateTime]::Now with the machine's culture (Shared/LoggerFunctions.ps1:60-63), so the common en-US output includes an AM/PM suffix. This regex and the AcceptedTimestampFormats below reject that form, leaving $ts null on every timestamped line; version candidates, completion signals, summary events, and body-evidence correlation then become unavailable for ordinary HealthChecker logs. Parse the logger's culture-dependent format (or add the 12-hour forms) consistently to the framing regex and accepted formats.
    '\A\s*\[(?<ts>[0-9]{1,2}/[0-9]{1,2}/[0-9]{4}\s+[0-9]{1,2}:[0-9]{2}:[0-9]{2}(?:\.[0-9]+)?)\]',

.github/skills/analyze-debug-files/Get-DebugFileMetadata.ps1:490

  • This explanation cites Write-Host, but Get-ErrorsThatOccurred.ps1 emits both summary headers with Write-Verbose. The headers are untimestamped because the message starts with CRLF and the logger prefixes the timestamp before those newlines; documenting the actual mechanism is important because this comment justifies the anchored header regex.
# cannot forge an authoritative summary. HealthChecker emits these as
# untimestamped lines (the writers in Get-ErrorsThatOccurred.ps1 use
# `Write-Host`, not `Write-Verbose`), so anchor to line start/end.

.github/skills/analyze-debug-files/SKILL.md:576

  • The suffix loop does not actually require a unique match: an absolute key can end with multiple repository paths, and the first hashtable key wins. For example, a key ending in .../Shared/OutputOverrides/Write-Error.ps1 can also match a shorter nested path if one exists, causing the dependency graph to resolve to the wrong source file. Collect suffix matches and accept only the unique/longest path (or reject ambiguity).
            if ($winKey.EndsWith($k, [System.StringComparison]::OrdinalIgnoreCase)) {
                return $Index[$k]

.github/skills/analyze-debug-files/SKILL.md:479

  • A cache miss can be caused by an existing corrupt or incomplete $cacheDir (for example, invalid metadata). This branch deletes the newly built temp directory whenever the final directory exists, leaving the bad entry in place, so every subsequent run rebuilds and remains BuildOnly. Distinguish a valid concurrent winner from a stale invalid entry and replace or quarantine the latter.
                if (Test-Path -LiteralPath $cacheDir -PathType Container) {
                    Remove-Item -LiteralPath $tempDir -Recurse -Force -ErrorAction SilentlyContinue
                } else {
                    Move-Item -LiteralPath $tempDir -Destination $cacheDir
                    $materializationSource = 'BuildAndCached'

.github/skills/analyze-debug-files/SKILL.md:1041

  • find-related-github-issues defines PartialLookup as usable retained results with incomplete coverage, not as an unavailable lookup. Treating every non-Ok status as unavailable would hide those matches from the report and contradict the sibling skill's contract; render retained results plus an incompleteness note for PartialLookup, reserving "unavailable" for the failure statuses.
**Failure modes.** Both skills return a `Status` field. When
`Status -ne 'Ok'`, render a single-line "lookup unavailable" note in
the corresponding subsection and continue with the report. A failing
provenance lookup MUST NOT abort the report.

.github/skills/analyze-debug-files/SKILL.md:437

  • This invocation is still subject to the caller's $PSNativeCommandUseErrorActionPreference. When that preference is $true and Build.ps1 exits nonzero—the condition this step explicitly says may be cosmetic—PowerShell throws before the XML existence checks, so the documented BuildOnly/cache flow aborts instead of using the generated XML. Disable native-error promotion only around this invocation and restore the caller's value in finally, or invoke the process through an API that captures the exit code without throwing.
            & pwsh -NoProfile -File (Join-Path $worktreeRoot '.build\Build.ps1')

.github/skills/find-related-github-issues/Find-RelatedGitHubIssues.ps1:88

  • Mandatory does not reject whitespace-only input. In that case normalization produces no query, the final status falls through to Ok with empty results, and callers can incorrectly report that no related issue exists instead of an invalid/failed lookup. Reject an empty or whitespace-only top-level exception before building the query list.
    [string]$TopLevelException,

.github/skills/find-related-github-issues/Find-RelatedGitHubIssues.ps1:128

  • These replacements create literal placeholder tokens, but the subsequent exact GitHub query is built from the normalized phrase. For Failure at C:\a\b, the query becomes "Failure at path" after delimiter stripping; GitHub does not treat path as a wildcard, so an issue saying Failure at C:\other\d is never returned and normalization cannot provide the advertised stable matching. Build search phrases that omit volatile spans or use unquoted invariant terms, while keeping normalization for local classification.
    $s = [regex]::Replace($s, '[A-Za-z]:\\[^\s"'']+', '<path>')
    # Strip UNC paths.
    $s = [regex]::Replace($s, '\\\\[^\s"'']+', '<unc>')
    # Strip randomized temp module names like tmpEXO_3fzjpepe.o0p
    $s = [regex]::Replace($s, 'tmpEXO_[A-Za-z0-9]+(?:\.[A-Za-z0-9]+)?', 'tmpEXO_<random>')

.github/skills/find-release-tag-for-script-version/Find-ReleaseTagForScriptVersion.ps1:358

  • For a valid empty gh release list response ([]), ConvertFrom-Json can produce $null (especially on Windows PowerShell 5.1). @($parsedReleases) then creates an array containing one null element, so the loop below throws Unexpected release entry shape from gh instead of reaching the not-found-no-candidates result. Preserve $null as an empty array before iterating.
    $releases = @($parsedReleases)
  • Files reviewed: 9/9 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +512 to +516
$Script:ErrorIndexRegex = [regex]::new(
# Iter-23 (RD-branch-10): require a strict timestamp shape
# (matching $Script:TimestampRegex) inside the brackets. See
# SummaryFooterRegex above for rationale.
'\A\s*\[[0-9]{1,2}/[0-9]{1,2}/[0-9]{4}\s+[0-9]{1,2}:[0-9]{2}:[0-9]{2}(?:\.[0-9]+)?\]\s*:\s*Error\s+Index\s*[:=]',
try {
# FileShare.ReadWrite so a still-running writer (rare for
# post-run debug artifacts but supported) does not lock us out.
$stream = [System.IO.File]::Open($FileInfo.FullName, [System.IO.FileMode]::Open, [System.IO.FileAccess]::Read, [System.IO.FileShare]::ReadWrite)
if ($LASTEXITCODE -ne 0 -or [string]::IsNullOrWhiteSpace($originUrl)) {
throw "Could not read local git origin URL."
}
if ($originUrl -notmatch '(?i)github\.com[:/]+microsoft/CSS-Exchange(\.git)?$') {
if ($blameOk) {
# Porcelain first line of each entry is `<sha> <origLine> <finalLine> [<groupSize>]`.
foreach ($ln in $blameOut) {
if ($ln -is [string] -and $ln -match '\A([0-9a-fA-F]{40})\s+\d+\s+\d+') {
@tweekerz
Rob Whaley (tweekerz) marked this pull request as ready for review September 11, 2026 01:03
@tweekerz
Rob Whaley (tweekerz) requested a review from a team as a code owner September 11, 2026 01:03
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@tweekerz
Rob Whaley (tweekerz) marked this pull request as draft September 11, 2026 01:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants