feat(tools): Run tests #95

Activity

1. CalvinAllen commented on Jul 8, 2026

   

   CalvinAllen

   on Jul 8, 2026

   Contributor

   More actions

   @SG-FVSP Some of these seem possible, but I'm not able to find VS SDK APIs for a number of them. I'm not sure how GH Copilot is doing it (they probably have internal access to other features the rest of us don't)

   What I'm seeing right now is that I can add -

   1. Run All Tests
   2. Debug All Tests
   3. Run Test by Class/Method
   4. Debug Test by Class/Method

   The rest don't have APIs I can tap into
2. CalvinAllen

   self-assigned this

   on Aug 5, 2026
3. CalvinAllen commented on Sep 3, 2026

   

   CalvinAllen

   on Sep 3, 2026

   Contributor

   More actions

   @SG-FVSP Following up on my earlier comment — I went back and dug through the actual VS assemblies (VS 2022 17.14 and VS 2026 18.0) rather than just hunting the docs, and I was wrong about a few of these. Revising:

   ✅ What we can do

   1. Run All Tests / Debug All Tests
   TestExplorer.RunAllTests / TestExplorer.DebugAllTests.

   2. Run / Debug by class or method
   TestExplorer.RunAllTestsInContext / TestExplorer.DebugAllTestsInContext. These are caret-context based, so the tool drives the caret to the target class/method first, then invokes.

   3. Wait for a run to actually finish, and report when it does
   This one I'd missed. IOperationState in Microsoft.VisualStudio.TestWindow.Interfaces.dll raises state changes off the TestOperationStates enum — TestExecutionStarting, TestExecutionStarted, TestExecutionFinished, DiscoveryFinished, TestExecutionCanceling, etc. That means the run/debug tools can await real completion instead of firing and returning immediately, which is what makes everything below actually useful.

   4. Counts of passing / failed / skipped / un-run
   I flat out missed this one — it's public API, and it's exactly what you asked for:

   namespace Microsoft.VisualStudio.TestWindow.Extensibility;
   
   public interface ITestExplorerStatsService
   {
       TestExplorerStats TestExplorerStats { get; }
       bool IsTestExplorerStatsServiceRunning { get; }
       event EventHandler<TestExplorerStats> TestExplorerStatsChanged;
   }
   
   public struct TestExplorerStats
   {
       public int PassedTestCount { get; }
       public int FailedTestCount { get; }
       public int SkippedTestCount { get; }
       public int NotRunTestCount { get; }
   }

   It's MEF-exported by TestExplorerStatsService, so IComponentModel.GetService<ITestExplorerStatsService>() gets you there. Public surface is identical between VS 2022 and VS 2026.

   5. Kick off the coverage analyzer
   Test.AnalyzeCodeCoverageForAllTests, plus Test.CodeCoverageResults to surface the window. Caveat in the "can't" section below.

   6. Read coverage numbers
   Two routes, both workable:

   • VS 2026 only: IReportService.GetReportAsync() returns CoverageRunData → CoverageFileData { Path, LinesCovered, LinesNotCovered, LinesPartiallyCovered, LineCoverageData[] }, with per-line Covered / NotCovered / PartiallyCovered status.
   • Portable: shell out to dotnet-coverage / Microsoft.CodeCoverage.Console to convert .coverage → Cobertura and parse. Works on both VS versions and actually gives proper module/class/method rollups.

   7. Execute terminal commands
   ITerminalService is a real brokered service (Microsoft.VisualStudio.Terminal.TerminalService) — CreateTerminalAsync, AttachTerminalStreamAsync (bidirectional stream), GetTerminalGuidsAsync, ShowAsync, CloseAsync. Identical across both VS versions. Caveat below.

   ❌ What we can't do (and why)

   1. Fetch the list of specific failed / un-run tests by name

   This is the one that's genuinely walled off, and I can now say exactly why. The per-test detail lives in TestResultDetails, TestNodeDetailsSummary and TestWindowViewModel — all internal to Microsoft.VisualStudio.TestWindow.Core / .Internal. TestWindow.Core.dll carries:

   InternalsVisibleTo("Microsoft.VisualStudio.TestWindow.Copilot.Internal,
                       PublicKey=002400000480000094000000060200000024000052534131...")
   

   and Microsoft.VisualStudio.TestWindow.Copilot.Internal.dll references those internal types 50+ times.

   So — to answer my own earlier question about how Copilot does it: it's InternalsVisibleTo gated on Microsoft's strong-name key. Not something a third-party extension can obtain. Reflection would technically work in-process, but it's unsupported and would break silently on any VS update, so I'm not shipping that.

   The public API gives us aggregate counts and nothing more granular. If you need actual test names + outcomes, the practical path is running via vstest.console.exe / dotnet test with a TRX logger and parsing the result — but that's a separate run from Test Explorer's own state, not a read of it.

   2. Class / module-level coverage tree straight from VS

   VS 2026's coverage contracts only expose per-file and per-line data. There's no class or module rollup in the API — you'd have to aggregate spans yourself and use Roslyn to map them back to symbols. The dotnet-coverage → Cobertura route gives you that hierarchy for free, so that's the better answer if you want the expanded tree.

   3. Coverage at all on VS 2022 Community / Professional

   Code coverage was Enterprise-only through VS 2022. It became available in all editions in VS 2026 — I confirmed this on disk (CommonExtensions\Microsoft\AnalyzeCodeCoverage\ exists in VS 2026 Community, not in VS 2022 Community). So any coverage tool has to degrade gracefully rather than assume it's there.

   4. Terminal command execution with reliable exit codes / output parsing

   AttachTerminalStreamAsync hands back a raw PTY stream — ANSI escape codes, no command boundaries, no exit codes. Fine for "run this and let the user watch it in VS," not fine for "run OpenCover and tell me whether it succeeded." For the latter I'd rather use a plain process-execution tool that can actually return an exit code and clean stdout/stderr.

   ---

   Net: this moves from 4-of-8 to roughly 6.5-of-8. I'll split the workable pieces into separate issues — test run/debug with real completion tracking, stats, terminal, and coverage. Thanks for pushing on this one, the stats API in particular was a good catch to go back for.
4. CalvinAllen commented on Sep 3, 2026

   

   CalvinAllen

   on Sep 3, 2026

   Contributor

   More actions

   Split the workable pieces out into their own issues:

   • feat(tools): add test run and debug tools with completion tracking #101 — feat(tools): add test run and debug tools with completion tracking
   • feat(tools): add test statistics tool #102 — feat(tools): add test statistics tool
   • feat(tools): add visual studio terminal tools #103 — feat(tools): add visual studio terminal tools
   • feat(tools): add code coverage tools #104 — feat(tools): add code coverage tools

   Keeping this issue open as the umbrella until those land.

   The one item with no path forward is the list of individual failed / un-run test names — that stays blocked on InternalsVisibleTo and isn't tracked separately, since there's nothing actionable to do about it.
5. CalvinAllen

   added a commit that references this issue on Sep 3, 2026

   feat(tools): add Visual Studio terminal tools (#106)

   ...

   91fb380
6. CalvinAllen commented on Sep 3, 2026

   

   CalvinAllen

   on Sep 3, 2026

   Contributor

   More actions

   Correction to my assessment above, found while implementing #104 (PR #107).

   I said the module/class coverage tree was out of reach:

   2. Class / module-level coverage tree straight from VS — VS 2026's coverage contracts only expose per-file and per-line data. There's no class or module rollup in the API.

   That was true of Microsoft.CodeCoverage.VisualStudio.Contracts (IReportService), which is what I looked at. I missed a better assembly.

   Microsoft.CodeCoverage.IO.dll — at Common7\IDE\CommonExtensions\Microsoft\TestWindow\VsTest, present in VS 2022 and VS 2026, every edition — exposes:

   CoverageFileUtility.ReadCoverageFile(path) -> CoverageData
   CoverageData.Modules -> ModuleWrapper { Name, Path, TargetFramework, Functions }
   Function : CoverageStatistics { Name, NamespaceName, TypeName }
   CoverageStatistics { LinesCovered, LinesPartiallyCovered, LinesNotCovered,
                        BlocksCovered, BlocksNotCovered }

   Function carries TypeName and NamespaceName, so grouping by declaring type gives the module → class → method tree. ModuleWrapper and Function both derive from CoverageStatistics, so the counts are pre-computed.

   @SG-FVSP — this means your request

   -Fetch specific Module\class Coverage percentage or the fully expanded coverage tree details

   is supported, on both VS 2022 and VS 2026, in any edition. It shipped as coverage_report in #107. One caveat: running coverage is still edition-gated (Enterprise only through VS 2022, all editions from VS 2026) — but reading an existing .coverage file works everywhere.

   Updated status of this issue

   Request	Status
   Run All Tests	✅ test_run_all (#105)
   Run individual test / class / tree node	✅ test_run (#105)
   Run tests in debug mode	✅ test_debug_all, test_debug (#105)
   Fetch count of passing/failed/skipped/un-run	✅ test_stats (#105)
   Run test coverage analyzer	✅ coverage_analyze (#107)
   Fetch module/class coverage % or full tree	✅ coverage_report (#107)
   Execute VS terminal commands	✅ terminal_run (#106)
   Fetch list of remaining failed or un-run tests	❌ still blocked

   Seven of eight. The one genuine hard stop remains the names of individual failed or un-run tests — that data is internal to Microsoft.VisualStudio.TestWindow.Core and gated behind InternalsVisibleTo on Microsoft's strong-name key, as detailed earlier in this thread. test_stats gives the counts; for names, running via vstest.console.exe / dotnet test with a TRX logger and parsing the result remains the workaround.

   Leaving this issue open until the four PRs have been exercised against a live Visual Studio.
7. CalvinAllen

   closed this as completedon Sep 9, 2026