Invalidate plugins if they've crashed the app

[saintmatthieu]

Resolves: audacity/audacity#11604

It can be that a once validated plugin later crashes the application, like MuseHub-distributed plugins did in the recent past, when their license expired.

With this PR, this can happen only once per offending plugin (rather than every time): next time, at startup, it'll be marked as invalid and won't be available in the app anymore.

MSS will benefit from this PR without further work.
Audacity will need to use the lock guard in its plugin-loading code - a small change, and not one needed for successful compilation.

• I signed the CLA as username:
• The title of the PR describes the problem it addresses.
• Each commit's message describes its purpose and effects, and references the issue it resolves. If changes are extensive, there is a sequence of easily reviewable commits.
• The code in the PR follows the coding rules.
• I understand all aspects of the code I'm contributing and I'm able to explain it if requested.
• The code compiles and runs on my machine, preferably after each commit individually. I have manually tested and verified that my changes fulfil their intended purpose.
• No prior attempts to resolve this problem exist, or if they do, I listed them in my PR description and described how I avoided repeating past mistakes.
• There are no unnecessary changes.
• I created a unit test or vtest to verify the changes I made (if applicable).

Build configuration

audacity: audacity/audacity/master
audacity platforms: linux_x64
musescore: musescore/MuseScore/main
musescore platforms: linux_x64

[saintmatthieu]

/build

[github-actions]

Build dispatched: https://github.com/musescore/muse_framework/actions/runs/31399126365

[coderabbitai]

📝 Walkthrough

Walkthrough

The audio-plugin module adds a load-guard interface and sentinel-file implementation. VST module creation now records load start and completion. Initialization detects dangling loads from prior interrupted operations and marks matching plugins with Err::PluginCrashedOnLoad. The module registers the load-guard implementation and skips crash marking during audio-plugin registration. Tests cover sentinel lifecycle, dangling-load recovery, plugin state updates, and empty recovery state.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	The framework implements crash tracking and invalidation, but Audacity plugin-loading integration required by issue #11604 is not included.	Add or explicitly track the Audacity plugin-loading integration that calls the load guard before and after plugin loading.
Docstring Coverage	⚠️ Warning	Docstring coverage is 2.94% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the main change: invalidating plugins after they crash the application.
Description check	✅ Passed	The description includes the issue, motivation, implementation summary, completed checklist, and build configuration.
Out of Scope Changes check	✅ Passed	The changes remain focused on persistent audio-plugin load tracking, crash invalidation, VST integration, and related tests.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@framework/audioplugins/audiopluginsmodule.cpp`:
- Around line 78-84: Update AudioPluginsContext::onInit to handle the failure
result from markCrashedPluginsAsBroken(). Propagate that failure through
initialization, or prevent subsequent normal plugin loading when recovery does
not succeed, ensuring a crashed plugin cannot be loaded again in the same run.

In `@framework/audioplugins/iaudiopluginsloadguard.h`:
- Around line 49-50: Update the IAudioPluginsLoadGuard::endLoad method to return
Ret instead of void, and propagate the sentinel cleanup result from its
implementations. Ensure callers check and handle cleanup failure—retrying or
resolving it before recovery consumes the record—so a successfully loaded plugin
is not later marked crashed by markCrashedPluginsAsBroken().

In `@framework/audioplugins/internal/audiopluginsloadguard.h`:
- Around line 37-41: Update
framework/audioplugins/internal/audiopluginsloadguard.h lines 37-41 so endLoad
reports cleanup failure; update its implementation in
framework/audioplugins/internal/audiopluginsloadguard.cpp lines 55-65 to
propagate the sentinel removal result and retry or durably record completion;
add coverage in framework/audioplugins/tests/audiopluginsloadguardtest.cpp lines
70-92 for failed removal and verify recovery does not classify a completed load
as PluginCrashedOnLoad.

In `@framework/audioplugins/tests/registeraudiopluginsscenariotest.cpp`:
- Around line 682-683: Require both recovery tests to verify the load-guard
query by replacing the ON_CALL stubs for m_loadGuard->danglingLoads() at
framework/audioplugins/tests/registeraudiopluginsscenariotest.cpp:682-683 and
:707-708 with EXPECT_CALL(...).WillOnce(...) using the existing crashed plugin
ID result.

In `@framework/vst/internal/vstmodulesrepository.cpp`:
- Around line 79-81: Check the return value of
loadGuard()->beginLoad(resourceId) and immediately return the failure before
invoking createModule(). Preserve the existing endLoad(resourceId) cleanup for
successful module creation.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: b067d4d8-f081-4628-bbb5-7aca56f941c2

📥 Commits

Reviewing files that changed from the base of the PR and between 476d928 and 4221255.

📒 Files selected for processing (14)

• framework/audioplugins/CMakeLists.txt
• framework/audioplugins/audiopluginserrors.h
• framework/audioplugins/audiopluginsmodule.cpp
• framework/audioplugins/iaudiopluginsloadguard.h
• framework/audioplugins/internal/audiopluginsloadguard.cpp
• framework/audioplugins/internal/audiopluginsloadguard.h
• framework/audioplugins/internal/registeraudiopluginsscenario.cpp
• framework/audioplugins/internal/registeraudiopluginsscenario.h
• framework/audioplugins/tests/CMakeLists.txt
• framework/audioplugins/tests/audiopluginsloadguardtest.cpp
• framework/audioplugins/tests/mocks/audiopluginsloadguardmock.h
• framework/audioplugins/tests/registeraudiopluginsscenariotest.cpp
• framework/vst/internal/vstmodulesrepository.cpp
• framework/vst/internal/vstmodulesrepository.h

[kryksyh]

/build

[github-actions]

Build dispatched: https://github.com/musescore/muse_framework/actions/runs/31406600268

[luapmartin]

looks nice, have you checked for disk access rights etc. on all platforms for writing and reading that sentinel file?

[saintmatthieu]

It's an interesting question, @luapmartin, hadn't thought about that.
I think that if the user denies the permission to access the file system, no regression occurs. For the mechanism to work, access must be granted on the first run (before the app crashes) and on the follow-up run (to read the sentinels, modify the plugin registry, and clean up). AFAICS the worst that can happen is
: the user grants access in the first run, a plugin crashes the app, the user updates the binary, the user denies access, and there's a dangling sentinel.

[luapmartin]

ok just add to QA to do a test run with denied access I guess then

[kryksyh]

ok just add to QA to do a test run with denied access I guess then

The sentinel file is saved in the application data directory, it is always writable to the app.