Unblock Integration Data Updater and parallelize API regeneration

.agents/skills/update-integrations/SKILL.md

@@ -161,6 +161,9 @@ The script performs the following for every package listed in `aspire-integratio
    Provisioning SDK overlays restore with the same-version `Aspire.Hosting`
    reference to model their AppHost context; other packages retain standalone
    per-package restore graphs. Dependency errors remain fatal.
+   Restores run in parallel, up to 8 at a time by default (`-Parallelism` changes
+   the limit; `-Sequential` restores one package at a time). A failed `dotnet restore`
+   is attempted up to three times before the package counts as failed.
 4. Selects the best-matching target framework folder (prefers `net10.0`, then `net9.0`, etc.).
 5. Uses Roslyn to analyze the assembly and extract all public types, members, XML docs, and attributes.
 6. Writes a `{Package}.{Version}.json` file to `src/frontend/src/data/pkgs/`.
@@ -222,6 +225,7 @@ The script reports a summary of successes, failures, and skipped packages. Commo
 - NuGet API resolution failure
 
 Packages that fail should be investigated individually — they may need a different framework target or may not ship a public API surface.
+Because restores are retried, a restore failure in the summary usually has a persistent cause, such as a pinned version that's no longer published on the package's feed.
 
 If an official release feed is required but not yet publicly reachable, rerun with one of the release feed override environment variables set.
 
@@ -257,6 +261,26 @@ combines their literals; each package's API JSON retains its exact enum surface.
 Use the package-specific SDK to validate behavior that depends on those differences.
 The generator rejects nonzero SDK dump exits and error diagnostics even when a JSON
 file was produced. An incomplete dump isn't a successful API-generation result.
+The one exception is a known scanner defect: a method inherited by several proxy
+types is exported once per derived type under the base type's capability ID, and the
+scanner reports each collision as a `Duplicate capability` error. When a dump exits
+nonzero after writing its JSON, the script passes `--tolerate-known-scanner-diagnostics`
+to the transformer, which accepts only that diagnostic shape, and only when the dump's
+`HandleTypes` show that each defining type is, or derives from, the type that owns the
+capability ID. Any other error diagnostic, or a nonzero exit without one, still fails the
+package. The script lists tolerated diagnostics in its summary, and
+`update-integration-data.ps1` raises a CI warning and lists the affected packages in the
+pull request body. Remove the tolerance
+once the shipped Aspire CLI includes [microsoft/aspire#20443](https://github.com/microsoft/aspire/pull/20443).
+
+Packages are generated in waves: `Aspire.Hosting` first, then packages that don't need
+supporting scan context, then packages that do. Packages in a wave run concurrently,
+up to 8 at a time by default; set `ASPIRE_TS_API_PARALLELISM` to change the limit (`1`
+processes one package at a time). `-AspireRepoPath` runs always process one package at
+a time because they build shared repo projects. Each `aspire` CLI call is limited to 10
+minutes and each helper tool call to 5 minutes. A dump or export that fails without
+producing a result, for example after a timeout or a transient restore failure, is
+attempted up to three times.
 
 When a TypeScript module is regenerated for a new package version, stale `ts-modules` JSON files for older versions of that same package are deleted automatically. Regenerated modules with no functions or types are omitted from `src/frontend/src/data/ts-modules/`.
 

src/frontend/scripts/update-integration-data.ps1

@@ -368,6 +368,8 @@ $pkgSummary = ''
 $pkgSkippedPackages = ''
 $tsApiSummary = ''
 $tsSkippedPackages = ''
+$tsToleratedPackages = ''
+$tsToleratedDiagnostics = [System.Collections.Generic.List[string]]::new()
 $twoslashSummary = ''
 $semanticSummary = ''
 
@@ -478,6 +480,34 @@ if ($versionsChanged -and -not $SkipRegen) {
     }
     else { '' }
 
+    # The generator tolerates only the known duplicate inherited-capability scanner
+    # defect (fixed by microsoft/aspire#20443) and lists each tolerated diagnostic
+    # under this line as "  [Package] message". Surface them loudly for review.
+    $inToleratedSection = $false
+    foreach ($line in $tsLog -split "`r?`n") {
+        if ($line -match '^\s*Packages with tolerated ATS diagnostics:\s*(?<Packages>.+?)\s*$') {
+            $tsToleratedPackages = $Matches.Packages
+            $inToleratedSection = $true
+        }
+        elseif ($inToleratedSection -and $line -match '^\s+\[(?<Package>[^\]]+)\]\s+(?<Message>.+?)\s*$') {
+            $tsToleratedDiagnostics.Add("$($Matches.Package): $($Matches.Message)")
+        }
+        else {
+            $inToleratedSection = $false
+        }
+    }
+    if ($tsToleratedPackages) {
+        $toleratedWarning = "TypeScript modules were generated despite known ATS scanner diagnostics for: $tsToleratedPackages. " +
+            "Their APIs may be incomplete until the Aspire CLI includes microsoft/aspire#20443; review them before merging."
+        Write-Warning $toleratedWarning
+        foreach ($diagnostic in $tsToleratedDiagnostics) {
+            Write-Warning "  $diagnostic"
+        }
+        if ($IsCI) {
+            Write-Host "::warning title=Tolerated ATS scanner diagnostics::$toleratedWarning"
+        }
+    }
+
     $twoslashSummary = 'succeeded'
 
     # 3c. Cross-artifact semantic validation. This runs only after every
@@ -607,6 +637,27 @@ if ($regenRan) {
     if ($tsSkippedPackages) {
         [void]$sb.AppendLine("- TypeScript packages skipped because they export no ATS functions: ``$tsSkippedPackages``")
     }
+    if ($tsToleratedPackages) {
+        [void]$sb.AppendLine("")
+        [void]$sb.AppendLine("### :warning: Tolerated ATS scanner diagnostics")
+        [void]$sb.AppendLine("")
+        [void]$sb.AppendLine("The Aspire CLI reported known duplicate inherited-capability diagnostics for the packages below. Their TypeScript modules were generated anyway, so the APIs named in these diagnostics may be incomplete until the CLI includes the scanner fix from microsoft/aspire#20443. **Review these modules before merging.**")
+        [void]$sb.AppendLine("")
+        foreach ($packageName in $tsToleratedPackages -split ',\s*') {
+            [void]$sb.AppendLine("- ``$packageName``")
+        }
+        if ($tsToleratedDiagnostics.Count -gt 0) {
+            [void]$sb.AppendLine("")
+            [void]$sb.AppendLine("<details>")
+            [void]$sb.AppendLine("<summary>Tolerated diagnostics</summary>")
+            [void]$sb.AppendLine("")
+            foreach ($diagnostic in $tsToleratedDiagnostics) {
+                [void]$sb.AppendLine("- ``$diagnostic``")
+            }
+            [void]$sb.AppendLine("")
+            [void]$sb.AppendLine("</details>")
+        }
+    }
 }
 else {
     [void]$sb.AppendLine("_No integration package versions changed in this run — API reference regeneration was skipped._")
@@ -628,6 +679,9 @@ if ($iconWarnings) {
 if ($regenRan) {
     [void]$sb.AppendLine("- [ ] New/removed ``pkgs/`` and ``ts-modules/`` files match the version changes")
     [void]$sb.AppendLine("- [ ] ``src/frontend/src/data/twoslash/aspire.d.ts`` is included in the diff")
+    if ($tsToleratedPackages) {
+        [void]$sb.AppendLine("- [ ] TypeScript modules generated with tolerated ATS scanner diagnostics were reviewed")
+    }
 }
 $prBody = $sb.ToString()
 

src/tools/AtsJsonGenerator/GenerateCommand.cs

@@ -49,6 +49,11 @@ internal static class GenerateCommand
         Description = "Paths to core and supporting-context docs-site JSON files. Capabilities and types already present in these packages are excluded from the output.",
     };
 
+    private static readonly Option<bool> s_tolerateKnownScannerDiagnosticsOption = new("--tolerate-known-scanner-diagnostics")
+    {
+        Description = "Accept error diagnostics caused by known ATS scanner defects and report each one. Any other error diagnostic still fails.",
+    };
+
     public static RootCommand GetCommand()
     {
         var command = new RootCommand("Transforms 'aspire sdk dump --format json' output into docs-site JSON.")
@@ -60,6 +65,7 @@ public static RootCommand GetCommand()
             s_sourceRepoOption,
             s_sourceCommitOption,
             s_baseOption,
+            s_tolerateKnownScannerDiagnosticsOption,
         };
 
         command.SetAction(static parseResult =>
@@ -71,8 +77,10 @@ public static RootCommand GetCommand()
             var sourceRepo = parseResult.GetValue(s_sourceRepoOption);
             var sourceCommit = parseResult.GetValue(s_sourceCommitOption);
             var basePaths = parseResult.GetValue(s_baseOption);
+            var tolerateKnownScannerDiagnostics = parseResult.GetValue(s_tolerateKnownScannerDiagnosticsOption);
 
-            return TransformFile(input, output, packageName, version, sourceRepo, sourceCommit, basePaths);
+            return TransformFile(
+                input, output, packageName, version, sourceRepo, sourceCommit, basePaths, tolerateKnownScannerDiagnostics);
         });
 
         return command;
@@ -85,7 +93,8 @@ internal static int TransformFile(
         string? version,
         string? sourceRepo,
         string? sourceCommit,
-        string[]? basePaths = null)
+        string[]? basePaths = null,
+        bool tolerateKnownScannerDiagnostics = false)
     {
         if (!File.Exists(inputPath))
         {
@@ -104,7 +113,8 @@ internal static int TransformFile(
             return 1;
         }
 
-        var result = AtsTransformer.Transform(dump, packageName, version, sourceRepo, sourceCommit);
+        var result = AtsTransformer.Transform(
+            dump, packageName, version, sourceRepo, sourceCommit, tolerateKnownScannerDiagnostics);
 
         // Supporting packages contribute scan targets, not APIs owned by this package.
         foreach (var basePath in basePaths ?? [])
@@ -159,6 +169,15 @@ internal static int TransformFile(
         var wroteFile = StableFileWriter.WriteIfChanged(outputPath, JsonSerializer.Serialize(result, options));
 
         Console.WriteLine($"{(wroteFile ? "Generated" : "Unchanged")}: {outputPath} ({result.Functions.Count} functions, {result.HandleTypes.Count} handles, {result.DtoTypes.Count} DTOs, {result.EnumTypes.Count} enums)");
+
+        if (tolerateKnownScannerDiagnostics)
+        {
+            foreach (var message in KnownScannerDiagnostics.GetToleratedErrors(dump))
+            {
+                Console.WriteLine($"{KnownScannerDiagnostics.ToleratedOutputPrefix} {message}");
+            }
+        }
+
         return 0;
     }
 }

src/tools/AtsJsonGenerator/Helpers/AtsTransformer.cs

@@ -10,16 +10,20 @@ internal static class AtsTransformer
     /// <summary>
     /// Transform the deserialized dump output into a <see cref="TsPackageModel"/>.
     /// </summary>
+    /// <param name="tolerateKnownScannerDiagnostics">
+    /// Accept error diagnostics that <see cref="KnownScannerDiagnostics"/> attributes to known scanner defects.
+    /// Any other error diagnostic still fails the transform.
+    /// </param>
     public static TsPackageModel Transform(
         AtsDumpRoot dump,
         string packageName,
         string? version = null,
         string? sourceRepository = null,
-        string? sourceCommit = null)
+        string? sourceCommit = null,
+        bool tolerateKnownScannerDiagnostics = false)
     {
-        var errors = dump.Diagnostics
-            .Where(diagnostic => diagnostic.Severity.Equals("Error", StringComparison.OrdinalIgnoreCase))
-            .Select(diagnostic => diagnostic.Message)
+        var errors = GetErrorDiagnostics(dump)
+            .Where(message => !tolerateKnownScannerDiagnostics || !KnownScannerDiagnostics.IsInheritedDuplicateCapability(message, dump))
             .ToArray();
         if (errors.Length > 0)
         {
@@ -92,6 +96,11 @@ public static TsPackageModel Transform(
         };
     }
 
+    internal static IEnumerable<string> GetErrorDiagnostics(AtsDumpRoot dump) =>
+        dump.Diagnostics
+            .Where(diagnostic => diagnostic.Severity.Equals("Error", StringComparison.OrdinalIgnoreCase))
+            .Select(diagnostic => diagnostic.Message);
+
     private static string? NormalizeSourceRepository(string? sourceRepository)
     {
         if (string.IsNullOrWhiteSpace(sourceRepository))

src/tools/AtsJsonGenerator/Helpers/KnownScannerDiagnostics.cs

@@ -0,0 +1,82 @@
+using System.Text.RegularExpressions;
+
+namespace AtsJsonGenerator.Helpers;
+
+/// <summary>
+/// Recognizes ATS scanner error diagnostics caused by known scanner defects.
+/// </summary>
+/// <remarks>
+/// The Aspire CLI scanner exports a method inherited by several proxy types once per
+/// derived type, all under the base type's capability ID. The scanner keeps the first
+/// definition and reports each collision as an error. Remove this tolerance once the
+/// shipped Aspire CLI includes the fix (https://github.com/microsoft/aspire/pull/20443).
+/// </remarks>
+internal static partial class KnownScannerDiagnostics
+{
+    /// <summary>
+    /// Prefix of the line written for each tolerated diagnostic. generate-ts-api-json.ps1 parses this prefix.
+    /// </summary>
+    public const string ToleratedOutputPrefix = "Tolerated known ATS scanner diagnostic:";
+
+    /// <summary>
+    /// Returns whether <paramref name="message"/> reports the inherited duplicate capability defect.
+    /// </summary>
+    /// <remarks>
+    /// Both definitions must be handle types in <paramref name="dump"/> that are, or derive from, the
+    /// type that owns the capability ID. Unrelated types that share a capability ID are a genuine collision.
+    /// </remarks>
+    public static bool IsInheritedDuplicateCapability(string message, AtsDumpRoot dump)
+    {
+        var match = InheritedDuplicateCapabilityPattern().Match(message);
+        if (!match.Success)
+        {
+            return false;
+        }
+
+        var capabilityNamespace = match.Groups["namespace"].Value;
+        var capabilityTypeName = $"{capabilityNamespace}.{match.Groups["type"].Value}";
+        var first = SplitQualifiedMember(match.Groups["first"].Value);
+        var second = SplitQualifiedMember(match.Groups["second"].Value);
+
+        return first.Namespace.Equals(capabilityNamespace, StringComparison.Ordinal)
+            && second.Namespace.Equals(capabilityNamespace, StringComparison.Ordinal)
+            && !first.TypeName.Equals(second.TypeName, StringComparison.Ordinal)
+            && first.Member.Equals(second.Member, StringComparison.Ordinal)
+            && first.Member.Equals(match.Groups["method"].Value, StringComparison.OrdinalIgnoreCase)
+            && IsSameOrDerivedHandleType(dump, first.TypeName, capabilityTypeName)
+            && IsSameOrDerivedHandleType(dump, second.TypeName, capabilityTypeName);
+    }
+
+    /// <summary>
+    /// Returns the error diagnostics in <paramref name="dump"/> that match a known scanner defect.
+    /// </summary>
+    public static IReadOnlyList<string> GetToleratedErrors(AtsDumpRoot dump) =>
+        AtsTransformer.GetErrorDiagnostics(dump)
+            .Where(message => IsInheritedDuplicateCapability(message, dump))
+            .ToArray();
+
+    private static bool IsSameOrDerivedHandleType(AtsDumpRoot dump, string typeName, string baseTypeName)
+    {
+        var handleType = dump.HandleTypes.FirstOrDefault(handle =>
+            AtsTransformer.StripAssemblyPrefix(handle.AtsTypeId).Equals(typeName, StringComparison.Ordinal));
+
+        return handleType is not null
+            && (typeName.Equals(baseTypeName, StringComparison.Ordinal)
+                || handleType.BaseTypeHierarchy.Any(baseType =>
+                    AtsTransformer.StripAssemblyPrefix(baseType.TypeId).Equals(baseTypeName, StringComparison.Ordinal)));
+    }
+
+    private static (string Namespace, string TypeName, string Member) SplitQualifiedMember(string qualifiedMember)
+    {
+        // The pattern guarantees at least three non-empty dot-separated segments.
+        var memberSeparator = qualifiedMember.LastIndexOf('.');
+        var typeName = qualifiedMember[..memberSeparator];
+
+        return (typeName[..typeName.LastIndexOf('.')], typeName, qualifiedMember[(memberSeparator + 1)..]);
+    }
+
+    [GeneratedRegex(
+        @"^Duplicate capability '(?<namespace>\w+(?:\.\w+)*)/(?<type>\w+)\.(?<method>\w+)': defined at '(?<first>\w+(?:\.\w+){2,})' and '(?<second>\w+(?:\.\w+){2,})'\. Remove \[AspireExport\] from one of them or use different capability IDs\.\z",
+        RegexOptions.CultureInvariant)]
+    private static partial Regex InheritedDuplicateCapabilityPattern();
+}

src/tools/AtsJsonGenerator/README.md

@@ -8,6 +8,12 @@ The Aspire CLI can emit a JSON description of all TypeScript-accessible capabili
 
 The companion `generate-ts-api-json.ps1` script reads the generated C# package JSON files in `src/frontend/src/data/pkgs/` and regenerates TypeScript API JSON using the same `Aspire.Hosting*` and `CommunityToolkit.Aspire.Hosting*` package/version sets. Modules with no generated functions or types are omitted so empty packages don't appear in the TypeScript API reference.
 
+### Script behavior
+
+- **Concurrency:** `Aspire.Hosting` is generated first, then packages that don't need supporting scan context, then packages that do. Packages in each wave run concurrently, up to 8 at a time by default. Set `ASPIRE_TS_API_PARALLELISM` to change the limit; `1` processes one package at a time. `-AspireRepoPath` runs always process one package at a time because dumps build shared repo projects.
+- **Timeouts and retries:** Each `aspire` CLI call is limited to 10 minutes, and each transformer or enum-supplement call to 5 minutes. A dump or export that fails without producing a result is attempted up to three times.
+- **Error diagnostics:** A dump that reports error diagnostics fails the package, with one exception. When the CLI writes the dump and exits nonzero, the script passes `--tolerate-known-scanner-diagnostics` to the transformer. That option accepts only the known inherited duplicate-capability scanner defect, where a method inherited by several proxy types is reported as `Duplicate capability '…': defined at 'Namespace.TypeA.Method' and 'Namespace.TypeB.Method'`. The transformer also checks the dump's `HandleTypes`: each defining type must be the type that owns the capability ID or derive from it, so unrelated types that share a capability ID aren't tolerated. Any other error diagnostic still fails the package. The transformer prints each tolerated diagnostic, and the script summary lists them under `Packages with tolerated ATS diagnostics:` for `update-integration-data.ps1` to surface in its pull request. Remove the tolerance once the shipped Aspire CLI includes [microsoft/aspire#20443](https://github.com/microsoft/aspire/pull/20443).
+
 ## Usage
 
 ### Single file