Skip to content

fix: keep committed external skills through APM Materialize - #240

Merged
DevSecNinja merged 5 commits into
mainfrom
devsecninja-fix-materialize-security-audit
Nov 15, 2026
Merged

DevSecNinja merged 5 commits into
mainfrom
devsecninja-fix-materialize-security-audit

Conversation

@DevSecNinja

@DevSecNinja DevSecNinja commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Closes #237

Why #233 deletes the skill

It's not line endings or a Windows-generated lock. The committed security-audit files are LF and their SHA-256 hashes match apm.lock.yaml exactly.

apm-cli 0.35.0 trusts a skill file's overwrite baseline only from per-user deployment receipts (~/.apm/deployment-receipts/ plus .apm-local/installation). It never trusts the committed lock (DeploymentMaterializationSession.baseline). A fresh runner has no receipts. For a root-level skill package, _integrate_native_skill has no "identical, adopt it" shortcut, unlike agents, which are "adopted". So every committed file is skipped with no trustworthy local deployment baseline. Nothing gets deployed for the package, so the cleanup phase deletes all 21 lock-tracked paths as stale.

Upstream tracks this as microsoft/apm#3372. v0.35.1 fails closed instead of deleting. Adopting unchanged committed skills (microsoft/apm#3564, microsoft/apm#3638) is only on main, not released yet.

Fix

  • APM Materialize: a new scripts/apm-external-skills.py prepare step removes the external skill folders recorded in apm.lock.yaml before apm install. APM then redeploys them from the pinned commit, and a missing target needs no receipt. Git still holds the committed copy, so the materialize PR diff still shows any upstream change. The review gate is unchanged, and apm-overrides.json is still re-applied after the install.

  • Guard: apm-external-skills.py check runs after the install and fails the run, rather than opening a PR, in any of these cases:

    • a locked dependency deploys nothing;
    • a deployed file is missing, or a skill folder has no SKILL.md;
    • a deployed path is git-ignored (a new or renamed skill folder without its .gitignore negation);
    • a dependency lost a path at the commit the baseline lock already pins (HEAD in Materialize; the PR base in PR CI via check --base);
    • a committed .agents/skills/ file isn't claimed by the lock.

    tests/test_apm_external_skills.py runs the same check against the committed tree on PRs (Markdown Validation) and in lefthook. On chore: materialize APM primitives #233's head it fails with cloudflare/security-audit-skill/skills/security-audit deploys no files. prepare refuses to delete through a symlinked .agents or .agents/skills.

  • validate-coverage-ledger.cjs and validate-findings.cjs regain their upstream 100755 mode, which the Windows commit in feat: add cloudflare security-audit skill #226 dropped. Otherwise every Linux materialize run would show a mode change.

  • docs/apm.md explains the workaround and when to drop it.

Testing

Run in separate testing contexts on WSL Debian (aarch64):

  • Repro with real apm-cli 0.35.0 and an empty HOME, on main (7e4ff4b): apm install --update --only apm exits 0 but deletes all 20 skill files (Cleaned 21 stale deployed files).
  • Same run with this branch's prepare step, then the overrides and check: the skill is untouched and check exits 0. The only lock change is generated_at, plus the stale require-push-approval.sh hash from feat(hooks): let git-push-approval allow feature-branch pushes #230, which is legitimate drift. A second run is deterministic.
  • test_apm_external_skills.py (11 tests), test_apm_overrides.py, validate-prompts.sh and check-index.sh all pass. Manual Linux scenarios also fail as expected: a renamed skill, a partial file loss, a symlinked parent. Every signed commit was made with all lefthook hooks passing on Linux.

This PR doesn't touch apm.yml, so merging it won't trigger APM Materialize. Dispatch it manually, or wait for the next release bump. It should then open a PR with only that lock refresh.

apm-cli 0.35 trusts a skill file's overwrite baseline only from per-user
deployment receipts, so on a fresh runner it skips the committed
security-audit skill and then deletes it as stale (microsoft/apm#3372).
APM Materialize now removes the lock-recorded external skill folders before
`apm install`, so they are redeployed from the pinned commit, and fails the
run if a locked dependency loses its files. A PR-time test runs the same
check against the committed lock. The two upstream-executable validator
scripts regain their exec bit, which the Windows commit in #226 dropped.

Closes #237

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@DevSecNinja DevSecNinja added the bug Something isn't working label Nov 15, 2026
@DevSecNinja
DevSecNinja requested a balanced review from Copilot November 15, 2026 11:38
@coderabbitai

coderabbitai Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The change adds a script that prepares locked external skill folders for APM deployment and checks deployed files against the lockfile and tracked skill files. The materialization workflow runs these commands around installation. CI, pre-commit, and documentation also cover the script.

Changes: External Skill Materialization

Layer / File(s) Summary
Lock parsing and preparation
scripts/apm-external-skills.py, tests/test_apm_external_skills.py
The script parses locked dependencies and deployed files, then removes existing locked skill directories before deployment. Tests cover lock parsing and preparation behavior.
Deployment checks
scripts/apm-external-skills.py, tests/test_apm_external_skills.py
The script reports dependencies without deployed files, missing locked files, and tracked skill files not claimed by the lock. The CLI returns status codes for reported problems and errors. Tests cover check results and command execution.
Materialization and validation integration
.github/workflows/apm-materialize.yml, .github/workflows/markdown-validation.yml, .lefthook.toml, docs/apm.md
The materialization workflow runs prepare before APM installation and check afterward. CI and pre-commit run the tests. The documentation describes the behavior and commands.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant APMMaterializeWorkflow
  participant ExternalSkillsScript
  participant APM
  APMMaterializeWorkflow->>ExternalSkillsScript: Run prepare
  APMMaterializeWorkflow->>APM: Run install
  APMMaterializeWorkflow->>ExternalSkillsScript: Run check
Loading
Merge Risk: Minimal · up to 61033

This change keeps committed external skills from being deleted by APM Materialize on fresh runners, and it adds a check that catches dropped deployments. No concrete merge-blocking issue remains.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the fix and its subject: committed external skills being kept through APM Materialize. The changeset adds prepare and check commands in scripts/apm-external-skills.py, wires them into …
Linked Issues check Passed Issue #237 has one active direct link and no active parent context. Its coding requirements are:

1. Remove lock-recorded external skill folders before apm install, so APM redeploys them. Implemente…
Out of Scope Changes check Passed All changes map to #237 or to support for it:

- .github/workflows/apm-materialize.yml: prepare and check steps. In scope.
- scripts/apm-external-skills.py and `tests/test_apm_external_skills.…
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Full details: Title check

Explanation

The title names the fix and its subject: committed external skills being kept through APM Materialize. The changeset adds prepare and check commands in scripts/apm-external-skills.py, wires them into the APM Materialize workflow, and documents the workaround. The title matches the main change and uses the 'fix:' convention. It is concise and specific enough for a teammate scanning history.

Full details: Linked Issues check

Explanation

Issue #237 has one active direct link and no active parent context. Its coding requirements are:

  1. Remove lock-recorded external skill folders before apm install, so APM redeploys them. Implemented by prepare(root, deps) in scripts/apm-external-skills.py. The APM Materialize workflow runs prepare before apm install when apm.yml exists. The script refuses to remove symlinks or non-directories.
  2. Keep the committed copy as the review baseline. The prepare step only removes working-tree folders. The committed files stay in Git, so the materialize PR diff still shows upstream changes.
  3. Add a guard that fails materialize and PR CI when a locked dependency deploys nothing, a deployed file is missing, or a committed .agents/skills/ file is not claimed by the lock. Implemented by check(root, deps, tracked), run after install in the workflow. tests/test_apm_external_skills.py runs the same check against the committed tree in .github/workflows/markdown-validation.yml and in .lefthook.toml.

The author reports a real-CLI reproduction with apm-cli 0.35.0 and an empty HOME. The author reports that the prepare/install/overrides/check sequence preserves the skill and passes check. This is author-reported evidence, not verified in this review. The request to close #233 unmerged is administrative and does not apply.

Full details: Out of Scope Changes check

Explanation

All changes map to #237 or to support for it:

  • .github/workflows/apm-materialize.yml: prepare and check steps. In scope.
  • scripts/apm-external-skills.py and tests/test_apm_external_skills.py: implementation and tests of the prepare/check logic. In scope.
  • .github/workflows/markdown-validation.yml and .lefthook.toml: run the check on PRs and in pre-commit hooks. These run the guard required by #237. In scope.
  • docs/apm.md: documents the workaround and when to remove it. Supporting documentation. In scope.
  • validate-coverage-ledger.cjs and validate-findings.cjs mode restored to 100755: not a coding requirement of #237. The PR description says the Windows commit in #226 dropped these modes, and that without the restore every Linux materialize run shows a mode change. This is a small, stated reason tied to materialize runs. Not a demonstrated unrelated change.

The mode restore is not visible in the supplied summary. Its effect was checked only through the PR description. No out-of-scope change was demonstrated.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 662d1d59-4beb-4d27-8d5d-d99a0ae03798
Commits

Reviewing files that changed from the base of the PR and between 7e4ff4b and 6103394.

Files selected for processing: 8

  • .agents/skills/security-audit/validate-coverage-ledger.cjs
  • .agents/skills/security-audit/validate-findings.cjs
  • .github/workflows/apm-materialize.yml
  • .github/workflows/markdown-validation.yml
  • .lefthook.toml
  • docs/apm.md
  • scripts/apm-external-skills.py
  • tests/test_apm_external_skills.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical issues allow deletion outside the checkout through symlinked parent directories and let an empty deployed skill pass the preservation check without a claimed, regular SKILL.md.

2 open issues
What changed in this PR

Addresses #237 by working around APM’s fresh-checkout behavior that deletes committed external skills, preserving the toolkit’s reviewable dependency snapshots.

Changes:

  • Adds lock-driven preparation before installation and deployment checks afterward.
  • Integrates regression tests into CI and pre-commit hooks, and documents the workaround.
  • Restores upstream executable modes for the two vendored validators.
File Description
tests/​test_apm_external_skills.py Adds parsing, preparation, deployment, and committed-tree checks.
scripts/​apm-external-skills.py Adds external-skill cleanup and post-install validation commands.
docs/​apm.md Explains the workaround, fresh-clone usage, and removal criteria.
.lefthook.toml Runs external-skill tests before commits.
.github/​workflows/​markdown-validation.yml Adds relevant triggers and external-skill tests.
.github/​workflows/​apm-materialize.yml Prepares skills before installation and validates them before opening a PR.
.agents/​skills/​security-audit/​validate-findings.cjs Restores executable mode without changing validator logic.
.agents/​skills/​security-audit/​validate-coverage-ledger.cjs Restores executable mode without changing validator logic.

Review effort: Balanced

Comment thread scripts/apm-external-skills.py Resolved
Comment thread scripts/apm-external-skills.py Resolved
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

SkillSpector report: ✅ Passed

SkillSpector 2.13.4 static scan (no LLM, no network) of 21 AI primitive targets at 7231f1b.
HIGH/CRITICAL findings block unless the owner accepted them in .github/skillspector-accepted.json. Findings are heuristic, not confirmed vulnerabilities.

Severity Reported Accepted Blocking
CRITICAL 0 0 0
HIGH 25 25 0
MEDIUM 53 - -
LOW 3 - -

Top findings

Severity Rule Location Finding Status
HIGH MP3 .agents/skills/security-audit/AI-AND-LLM.md:29 Memory Manipulation: delete memory Accepted: External cloudflare/security-audit-skill (vendored via apm.yml): audit guidance asking who may create, update and delete agent memory across tenants; it does no...
HIGH PE3 .agents/skills/security-audit/ATTACK-CLASSES.md:118 Credential Access: credentials.json Accepted: External cloudflare/security-audit-skill (vendored via apm.yml): checklist item asking whether credentials.json or .env files are committed to the audited repo;...
HIGH PE3 .agents/skills/security-audit/DESKTOP-MOBILE-AND-LOCAL-IPC.md:15 Credential Access: keychain Accepted: External cloudflare/security-audit-skill (vendored via apm.yml): names keychain ACLs as an OS control to account for when judging a trust boundary; it does not ...
HIGH AE1 .agents/skills/security-audit/SKILL.md:175 Incomplete referenced artifact analysis: validate-coverage-ledger.cjs (partial) Accepted: External cloudflare/security-audit-skill (vendored via apm.yml): derived from the accepted static_parse_limit on the bundled validator, which only reads its JSO...
HIGH AE1 .agents/skills/security-audit/SKILL.md:175 Incomplete referenced artifact analysis: validate-findings.cjs (partial) Accepted: External cloudflare/security-audit-skill (vendored via apm.yml): derived from the accepted static_parse_limit on the bundled validator, which only reads its JSO...
HIGH PE3 .agents/skills/security-audit/validate-coverage-ledger.test.cjs:495 Credential Access: /etc/passwd Accepted: External cloudflare/security-audit-skill (vendored via apm.yml): test fixture asserting the ledger validator rejects /etc/passwd as an unsafe absolute path; the...
HIGH PE3 .agents/skills/security-audit/validate-findings.test.cjs:320 Credential Access: /etc/passwd Accepted: External cloudflare/security-audit-skill (vendored via apm.yml): test fixture (unsafe source paths) asserting the findings validator rejects /etc/passwd; the fi...
HIGH PE3 .agents/skills/security-audit/validate-findings.test.cjs:606 Credential Access: /etc/passwd Accepted: External cloudflare/security-audit-skill (vendored via apm.yml): test fixture (unsafe fingerprint paths) asserting the findings validator rejects /etc/passwd; t...
HIGH AE1 .apm/skills/obsidian/SKILL.md:217 Incomplete referenced artifact analysis: references/FUNCTIONS_REFERENCE.md (partial) Accepted: Derived from the accepted static_parse_limit on FUNCTIONS_REFERENCE.md: a Markdown link to bundled reference docs, not a risky artifact.
HIGH AR2 .apm/skills/renovate-triage-and-merge/references/testing.md:38 Anti-Refusal Statement: no warning Accepted: Test expectation that happy-path results emit no warning; not an instruction to suppress refusals.

Showing 10 of 81 findings, blocking first; see Code Scanning for all.

Analysis limitations (34)
Target Location Reason Status
.apm/skills/commit-and-release .apm/skills/commit-and-release/SKILL.md:171 reference_missing Caveat, non-blocking
.apm/skills/commit-and-release .apm/skills/commit-and-release/SKILL.md:173 reference_missing Caveat, non-blocking
.apm/skills/commit-and-release .apm/skills/commit-and-release/SKILL.md:181 reference_missing Caveat, non-blocking
.apm/skills/meeting-transcript-creation .apm/skills/meeting-transcript-creation/SKILL.md:17 reference_missing Caveat, non-blocking
.apm/skills/morning-briefing .apm/skills/morning-briefing/SKILL.md:49 reference_missing Caveat, non-blocking
.apm/skills/morning-briefing .apm/skills/morning-briefing/SKILL.md:58 reference_missing Caveat, non-blocking
.apm/skills/obsidian .apm/skills/obsidian/SKILL.md:32 reference_missing Accepted: The path-like reference is generic `[text](url)` link syntax in prose, not a bundled file.
.apm/skills/obsidian .apm/skills/obsidian/references/FUNCTIONS_REFERENCE.md:69 static_parse_limit Accepted: Obsidian Bases formula examples in a reference doc; the parser cannot bound them as commands but checked all input.
.apm/skills/renovate-triage-and-merge .apm/skills/renovate-triage-and-merge/SKILL.md:198 reference_missing Caveat, non-blocking
.apm/skills/renovate-triage-and-merge .apm/skills/renovate-triage-and-merge/SKILL.md:204 reference_missing Caveat, non-blocking

Showing 10 of 34 limitations; see the run summary for all.

Full results: Code Scanning alerts · workflow run and full reports

`prepare` now refuses to remove a skill folder when `.agents`,
`.agents/skills` or the folder itself is a symlink, so it can't delete
outside the checkout. `check` now requires every deployed skill folder to
keep a claimed SKILL.md, so a skill emptied down to its directory fails.

Refs #237

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The check accepts Git-ignored replacement skill files, allowing publication to delete the old folder and update the lock without including replacement content.

1 open issue
2 resolved since last review

Review effort: Balanced

Comment thread scripts/apm-external-skills.py Resolved
A renamed or newly added external skill folder lacks its .gitignore
negation, so the materialize PR would carry the lock entry but not the
files. `check` now asks `git check-ignore` about every lock-deployed path
and names the ignored folder once.

Refs #237

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The check can miss a lost report-schema.json when its tracking entry is also removed, allowing publication of an incomplete skill and preventing the findings validator from loading its schema.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Detect partially missing files when a dependency commit is unchanged

scripts/​apm-external-skills.py:126

If installation drops report-schema.json from both disk and deployed_files while retaining the folder and SKILL.md, this existence condition skips its tracked path and check succeeds. Materialize can then publish an incomplete skill even at an unchanged resolved commit, leaving the findings validator unable to load its schema. Retain and compare the pre-install inventory when a dependency's resolved commit is unchanged, and add a regression case for this partial-loss scenario.

1 resolved since last review

Review effort: Balanced

If `apm install` drops one file from both disk and `deployed_files`
while the dependency's commit stays the same, the earlier checks passed.
`check` now compares the lock with the one at HEAD and fails on any
deployed path lost at an unchanged resolved commit.

Refs #237

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The committed-tree check can miss deletion of a companion file and its lock entry because PR CI compares against HEAD rather than the PR base revision, leaving a validation gap that needs a baseline fix and regression test.

1 open issue

Review effort: Balanced

Comment thread scripts/apm-external-skills.py Resolved
In PR CI, HEAD already holds the PR's lock, so a PR could drop a locked
companion file and its lock entry together. `check --base REV` compares
with REV's lock instead, and Markdown Validation runs it against the PR
base. An unreadable explicit base fails rather than passing.

Refs #237

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the external-skill preservation changes.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: APM Materialize deletes the committed security-audit skill

2 participants