Skip to content

doc: document CodeQL alert triage and re-raised alerts - #1716

Merged
Pedro Henrique Penna (ppenna) merged 1 commit into
devfrom
agents/issue-1655-codeql-reraised-alerts
Nov 17, 2026
Merged

Pedro Henrique Penna (ppenna) merged 1 commit into
devfrom
agents/issue-1655-codeql-reraised-alerts

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Part of #1655

What does this pull request change?

It documents how this repository's CodeQL alerts are triaged, and what to do
when a dismissed alert comes back under a new number. Start with the new
Code scanning section of doc/ci.md, after Issue link, which:

  • says how CodeQL runs here (default setup for Actions, Python, and Rust, with
    the default suite, on pull requests to dev, pushes to it, and weekly),
    which checks it adds (Analyze (...) and CodeQL), and that
    Required status check does not wait for them;
  • records the convention that the reviews of ci: enable CodeQL code scanning for Actions, Python, and Rust #1655's alerts followed: review
    in an area: security issue, or privately under SECURITY.md for a
    vulnerability, then fix the code, or dismiss the alert with the reason that
    fits and a comment that cites the issue. It also says that codeql[...] and
    lgtm[...] comments suppress nothing here, and that renaming code only to
    escape a query is no answer;
  • explains how GitHub matches a result to an alert (its rule, its file's path,
    and a hash of the first 100 characters, other than spaces and tabs, from the
    start of the flagged line), which changes therefore re-raise a dismissed
    alert as a new one, and how to handle them: dismiss them again in the pull
    request that raises them, with the original reason and issue, once the
    original review still holds.

A bullet under Pull Requests And CI in .github/copilot-instructions.md
points the agents, which make most renames, to it.

Left out, and still open in #1655: the choice between default setup and an
advanced workflow (draft #1675 proposes one, and would then update this
section's first paragraph), the Rust coverage limits, how scan results
interact with the merge queue and the required checks, and an owner for new
alerts. Nothing here automates a re-dismissal or suppresses an alert, so a
re-raised alert still needs someone to dismiss it.

Why is this change needed?

A dismissal does not stop GitHub from raising the same false positive again.
GitHub matches each result of an analysis to an alert by its rule, its file's
path, and its primaryLocationLineHash fingerprint, not by its line number.
So a result whose file is renamed, moved, or split, or whose flagged code is
edited, gets a new, open alert, while the dismissed one matches nothing. This
has already happened: #1671 renamed nvx-rs/ to nvx/, and its own analysis
replaced alerts 23, 26, and 28 to 32 with alerts 33 to 39, at the same rules
and lines, which kept their numbers on dev once it merged. Those alerts were
still open; a dismissal would not have carried over either. Nothing
documented this, or how to triage an alert at all.

Rejected alternatives, from the review of alert 27 in #1667:

  • Inline # codeql[RULE] comments. Default setup ignores them. Making them
    work needs an advanced workflow that adds CodeQL's AlertSuppression.ql
    query and runs advanced-security/dismiss-alerts on dev pushes with
    security-events: write. That action dismisses as "won't fix" with the
    comment "Suppressed via SARIF", and anyone could silence an alert by adding
    a comment. Worth revisiting only if ci: enable CodeQL code scanning for Actions, Python, and Rust #1655 moves to an advanced workflow.
  • advanced-security/filter-sarif, filtering by path and rule. It drops
    the results before upload, which leaves no alert and no record: the silent
    suppression that ci: enable CodeQL code scanning for Actions, Python, and Rust #1655 rules out.
  • query-filters or paths-ignore. Too broad, since they disable a rule
    for production code too, or every rule for the test files, and they also
    need an advanced workflow.
  • Renaming code to escape a query's name heuristics. That hides a result
    instead of recording its review.

How was this change validated?

Checks and results, on Windows:

Command Result
python scripts/local_gates.py --base origin/dev --run --keep-going test-changes, markdown-links, and test-layout passed. nvx-cli ran 2,302 tests, skipping 91, with 6 errors, each a subprocess.TimeoutExpired after the 10 seconds that two MicrovmTests give Git's sh on this loaded host: 5 subtests of test_host_loopback_allow_script_checks_publication_and_traffic and [tcp-missed] of test_private_network_ingress_script_checks_policy_and_traffic. Both tests passed twice when rerun alone, and the [tcp-missed] timeout also occurs at origin/dev (742ef7c4) on this host
The steps of nvx-cli after the one that failed The seven nvx.py ... --help checks passed. scripts/test_adversarial.py ran 84 tests with 1 failure, test_process_timeout_bounds_blocked_stdin_writer measuring 1,875 ms against its 1,500 ms bound, which passed 3 times when rerun alone
python scripts/check_markdown_links.py, after the last edit Passed
python -m unittest scripts/test_tool_versions.py scripts/test_markdown_links.py -v 118 tests OK, 3 skipped (they need a POSIX shell); test_consumers_and_guides_hard_code_no_pin covers both edited files
python scripts/check_test_changes.py --base origin/dev 0 changes to existing tests

None of the tests that failed locally reads the changed files, and CI ran
them all on this head: run 44112601088
passed, with NVX CLI / Linux, NVX CLI / Windows,
Lint, typing, and formatting, and Required status check, and the
CodeQL check found no new alerts.

The facts that the section states, checked against dev at 659493256316:

Fact Evidence
Default setup, languages, suite, and schedule GET /repos/microsoft/nvx/code-scanning/default-setup: configured; actions, python, rust; default; weekly
Checks, and what is required #1651 and #1671 have Analyze (actions), Analyze (python), Analyze (rust), and CodeQL check runs, the last linking to the branch's open alerts; dev's rules require only Required status check
No suppression query The SARIF of analyses 2186479668, 2186487685, and 2186499772 gives every result a primaryLocationLineHash and none a suppressions entry
The fingerprint A reimplementation of the CodeQL Action's fingerprint reproduces alert 27's a4d54c21248b79dd:1 from scripts/test_nvx_tools.py; inserting lines above it or re-indenting it keeps the hash, and editing line 18765 or 18766 changes it
A rename re-raises alerts Alerts 23, 26, and 28 to 32 under nvx-rs/ are fixed; alerts 33 to 39 under nvx/ were raised on refs/pull/1671/head before #1671 merged, and are the same alerts on dev
The dismissal convention Alerts 1, 2, 16 to 18, 27, and 33 to 39 were dismissed as false positive or used in tests, with a comment that cites #1658, #1659, #1660, #1661, #1662, or #1667

Backend, host OS, host type, and guest: none, since the change is
documentation only.

Left to CI: nothing more; it ran the Linux and Windows CLI suites and the
quality job, which passed.

Checklist

  • Tests cover the new or fixed behavior, in the layout that
    python3 scripts/check_test_layout.py checks.
  • python3 scripts/check_test_changes.py --base origin/dev lists no change
    to an existing test, or a Changes to existing tests section below
    explains each one.
  • The documentation that the change affects is updated, including
    .github/copilot-instructions.md and the skills when a convention or
    workflow changes.
  • A change to the public API of nvx updates nvx-rs/public-api/ with
    python3 scripts/public_api.py --update.
  • Snapshots, protocols, and on-disk formats stay compatible, or the
    description says how the change versions or migrates them.

A dismissed CodeQL alert does not stay dismissed when its code moves to
another file or the code at its location changes: GitHub matches each
result to an alert by its rule, the path of its file, and its
primaryLocationLineHash fingerprint, so such a result gets a new, open
alert. Renaming nvx-rs/ to nvx/ in #1671 replaced alerts 23, 26, and
28 to 32 with alerts 33 to 39 this way, at the same rules and lines;
the pull request's own analysis raised them before it merged, and they
kept their numbers on dev. Nothing documented how to triage CodeQL
alerts, or that this happens.

Add a Code scanning section to doc/ci.md. It says how the default
setup runs and which checks it adds, and records the convention that
the reviews of #1655's alerts followed: an area: security issue, then
a fix, or a dismissal with its reason and a comment that cites the
issue. It explains why codeql[...] comments and evasive renames are no
answer, and how to handle a re-raised alert: dismiss it again in the
pull request that raises it, with the original reason and issue, once
the original review still holds. A bullet in the Copilot instructions
points the agents that make most renames to it.

Evidence: dev's analyses at 6594932 (2186479668, 2186487685, and
2186499772) give every result a primaryLocationLineHash and none a
suppression. A reimplementation of the CodeQL Action's fingerprint
reproduces alert 27's a4d54c21248b79dd:1 from scripts/test_nvx_tools.py
at that revision: inserting lines above it or re-indenting it keeps the
hash, and editing line 18765 or 18766 changes it. The default-setup API
reports the actions, python, and rust languages, the default suite, and
a weekly schedule, and dev's rules require only Required status check.

No test reads this prose; check_markdown_links.py checks its links.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 17, 2026 15:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the documentation updates covering CodeQL alert triage and re-raised alerts.

No open issues 🎉

What changed in this PR

This documentation-only PR advances #1655 by documenting NVX’s CodeQL alert triage process and how to handle dismissed alerts that reappear after code changes.

Changes:

  • Documents the current CodeQL setup, analysis checks, and relationship to required CI checks.
  • Defines evidence-backed triage and dismissal guidance, including private reporting for vulnerabilities.
  • Explains alert matching and directs contributors and agents to review re-raised alerts before merging.
File Description
doc/​ci.md Adds Code scanning guidance covering setup, triage, dismissals, and re-raised alerts.
.github/​copilot-instructions.md Adds agent guidance for handling re-raised alerts and links to the CI documentation.

Review effort: Balanced

@ppenna
Pedro Henrique Penna (ppenna) added this pull request to the merge queue Nov 17, 2026
Merged via the queue into dev with commit ad711be Nov 17, 2026
50 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the agents/issue-1655-codeql-reraised-alerts branch November 17, 2026 16:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants