Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ packages = registries.BulkFetchPackagesWithConcurrency(ctx, purls, nil, 5)
| RubyGems | `pkg:gem/rails@7.1.0` |
| Terraform | `pkg:terraform/hashicorp/consul/aws@0.11.0` |
| GitHub Actions | `pkg:githubactions/actions/checkout@v4` |
| VS Code extension | `pkg:vscode-extension/redhat/vscode-yaml@1.15.0` |
| Alpine | `pkg:apk/alpine/curl@8.22.0-r0?arch=x86_64&repository_url=https://dl-cdn.alpinelinux.org/alpine/v3.23/main/x86_64` |

## Direct Registry Usage
Expand Down Expand Up @@ -197,12 +198,15 @@ import _ "github.com/git-pkgs/registries/all"
| Swift | `swift` | URL required |
| Alpine | `apk` | URL required |
| GitHub Actions | `githubactions` | https://api.github.com |
| VS Code extension | `vscode-extension` | https://open-vsx.org |
| APM | None | URL required |

The Alpine URL is one repository architecture directory, such as `https://dl-cdn.alpinelinux.org/alpine/v3.23/main/x86_64`, or the `APKINDEX.tar.gz` inside it. The client doesn't pick a release, component or architecture from a mirror root. It reads v2 `APKINDEX.tar.gz` indexes. When a directory serves a v3 `Packages.adb`, the client returns an error rather than reading the v2 index.

GitHub Actions are git repositories, so `FetchPackage` reads `/repos/{owner}/{repo}` and `FetchVersions` lists tags from the GitHub REST API. Unauthenticated requests are limited to 60 per hour. A subdirectory action such as `pkg:githubactions/github/codeql-action/init@v3` resolves to the `github/codeql-action` repository. `URLs().Download` returns the `/archive/{ref}.tar.gz` source tarball for any tag, branch or commit SHA. GitHub doesn't guarantee these tarballs are byte-stable, so versions carry no `Integrity`. Dependencies and maintainers return an error wrapping `errors.ErrUnsupported`. For GitHub Enterprise Server, pass the `/api/v3` API root as the base URL.

VS Code extensions are read from [Open VSX](https://open-vsx.org), or from a self-hosted Open VSX server given as the base URL. The purl spec makes the Visual Studio Marketplace the default for `vscode-extension`, but the client has no Marketplace support, so a PURL without `repository_url` also resolves to Open VSX. `URLs().PURL` always adds the registry as `repository_url`. Names are `publisher/name` or VS Code's `publisher.name`. An extension can publish a separate build for each target platform, such as `linux-x64` or `darwin-arm64`. Each version from `FetchVersions` and `FetchVersion` carries one artifact per build, with `platform`, `signature` (the `.sigzip`) and `sha256` (the URL of the checksum file) in its metadata. `URLs().Download` returns the universal build, which a version built only for specific platforms doesn't have; the `platform` qualifier, as in `pkg:vscode-extension/muhammad-sammy/csharp@2.160.9?platform=linux-x64`, makes `fetch.Resolver` pick the build for that platform, or the universal one when there is none. `FetchDependencies` lists both the extensions a version depends on and the ones it bundles as an extension pack. `FetchMaintainers` returns the user who published the latest version.

### Swift registries

Swift lookups require an explicit registry URL and a registry identity:
Expand Down Expand Up @@ -566,7 +570,7 @@ info, err := resolver.Resolve(ctx, "cargo", "serde", "1.0.0")
// info.Integrity may be populated for metadata-resolved URLs
```

When a version has several files, `ResolveWithOptions` picks one by PURL qualifiers. `arch`, `distro`, `subdir` and `build` are matched against artifact metadata, and an artifact that doesn't record a qualifier isn't ruled out by it. `repository_url` and `channel` are ignored: register a registry for that URL instead.
When a version has several files, `ResolveWithOptions` picks one by PURL qualifiers. `arch`, `distro`, `subdir`, `build` and `platform` are matched against artifact metadata, and an artifact that doesn't record a qualifier isn't ruled out by it. An artifact whose `platform` is `universal` matches any requested platform, after one built for that platform. When a registry lists no artifacts for the version, or doesn't list the version, the URL from `URLs().Download` is returned. `repository_url` and `channel` are ignored: register a registry for that URL instead.

```go
info, err := resolver.ResolveWithOptions(ctx, "conda", "zlib", "1.3.1", fetch.ResolveOptions{
Expand Down
3 changes: 2 additions & 1 deletion all/all.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
//
// // Now all ecosystems are available
// ecosystems := registries.SupportedEcosystems()
// // ["apk", "apm", "brew", "cargo", "clojars", "cocoapods", "composer", "conda", "cpan", "cran", "deno", "dub", "elm", "gem", "githubactions", "golang", "hackage", "haxelib", "helm", "hex", "julia", "luarocks", "maven", "nimble", "npm", "nuget", "pub", "pypi", "swift", "terraform"]
// // ["apk", "apm", "brew", "cargo", "clojars", "cocoapods", "composer", "conda", "cpan", "cran", "deno", "dub", "elm", "gem", "githubactions", "golang", "hackage", "haxelib", "helm", "hex", "julia", "luarocks", "maven", "nimble", "npm", "nuget", "pub", "pypi", "swift", "terraform", "vscode-extension"]
package all

import (
Expand Down Expand Up @@ -43,4 +43,5 @@ import (
_ "github.com/git-pkgs/registries/internal/rubygems"
_ "github.com/git-pkgs/registries/internal/swift"
_ "github.com/git-pkgs/registries/internal/terraform"
_ "github.com/git-pkgs/registries/internal/vscodeextension"
)
59 changes: 51 additions & 8 deletions fetch/resolver.go
Original file line number Diff line number Diff line change
Expand Up @@ -113,14 +113,22 @@ type ResolveOptions struct {
Integrity string
// Qualifiers are PURL qualifiers. Those in selectionQualifiers pick among
// the artifacts a registry lists for a version; an artifact that does not
// record a qualifier is not ruled out by it. repository_url and channel
// record a qualifier is not ruled out by it. An artifact whose platform is
// universal serves any platform, after one built for the platform asked
// for. repository_url and channel
// choose the registry, which is up to the caller, and are ignored here.
Qualifiers map[string]string
}

// selectionQualifiers are the PURL qualifiers matched against artifact
// metadata of the same name.
var selectionQualifiers = []string{"arch", "distro", "subdir", "build"}
var selectionQualifiers = []string{"arch", "distro", "subdir", "build", qualifierPlatform}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

const (
qualifierPlatform = "platform"
// universalPlatform is the platform of a build that runs on every one.
universalPlatform = "universal"
)

func (o ResolveOptions) hasSelectionQualifier() bool {
for _, key := range selectionQualifiers {
Expand Down Expand Up @@ -166,7 +174,16 @@ func (r *Resolver) ResolveWithOptions(
}

// For ecosystems like PyPI, we need to fetch metadata to get the URL
return r.resolveFromMetadata(ctx, reg, name, version, options)
info, err := r.resolveFromMetadata(ctx, reg, name, version, options)
if (errors.Is(err, ErrNoDownloadURL) || errors.Is(err, ErrNotFound)) && options.hasSelectionQualifier() {
// The registry lists no artifacts to pick among, or lists the
// version under another number as RubyGems does for a platform
// build, so the built URL is the only one there is.
if url := reg.URLs().Download(name, version); url != "" {
return matchSingleArtifact(&ArtifactInfo{URL: url, Filename: filenameFromURL(url)}, options, nil)
}
}
return info, err
}

// resolveWithoutRegistry handles ecosystems with predictable URLs
Expand Down Expand Up @@ -274,6 +291,27 @@ func (r *Resolver) resolveFromMetadata(
}

func (r *Resolver) selectArtifact(ctx context.Context, candidates []registries.Artifact, options ResolveOptions) (*ArtifactInfo, error) {
// A build for the requested platform wins over a universal one, so
// universal builds are only let in on a second pass.
universal := []bool{false}
if options.Qualifiers[qualifierPlatform] != "" {
universal = []bool{false, true}
}
for _, allowUniversal := range universal {
info, err := r.selectArtifactPass(ctx, candidates, options, allowUniversal)
if err != nil || info != nil {
return info, err
}
}
return nil, ErrNoMatchingArtifact
}

func (r *Resolver) selectArtifactPass(
ctx context.Context,
candidates []registries.Artifact,
options ResolveOptions,
allowUniversal bool,
) (*ArtifactInfo, error) {
for _, candidate := range candidates {
filename := candidate.Filename
if filename == "" {
Expand All @@ -285,7 +323,7 @@ func (r *Resolver) selectArtifact(ctx context.Context, candidates []registries.A
if options.Integrity != "" && !integrityMatches(candidate.Integrity, options.Integrity) {
continue
}
if !qualifiersMatch(candidate.Metadata, options.Qualifiers) {
if !qualifiersMatch(candidate.Metadata, options.Qualifiers, allowUniversal) {
continue
}
if err := r.checkMetadataURL(ctx, candidate.URL); err != nil {
Expand All @@ -298,18 +336,23 @@ func (r *Resolver) selectArtifact(ctx context.Context, candidates []registries.A
Size: candidate.Size,
}, nil
}
return nil, ErrNoMatchingArtifact
return nil, nil
}

func qualifiersMatch(metadata map[string]any, qualifiers map[string]string) bool {
func qualifiersMatch(metadata map[string]any, qualifiers map[string]string, allowUniversal bool) bool {
for _, key := range selectionQualifiers {
want := qualifiers[key]
if want == "" {
continue
}
if got, ok := metadata[key].(string); ok && got != "" && got != want {
return false
got, ok := metadata[key].(string)
if !ok || got == "" || got == want {
continue
}
if key == qualifierPlatform && got == universalPlatform && allowUniversal {
continue
}
return false
}
return true
}
Expand Down
149 changes: 149 additions & 0 deletions fetch/vscode_resolver_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
package fetch

import (
"context"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"strings"
"testing"

"github.com/git-pkgs/registries"
"github.com/git-pkgs/registries/client"
_ "github.com/git-pkgs/registries/internal/vscodeextension"
)

func openVSXBuild(version, platform string) string {
file := "acme.tool-" + version
dir := version
if platform != "universal" {
file += "@" + platform
dir = platform + "/" + version
}
return fmt.Sprintf(`{"namespace": "acme", "name": "tool", "version": %q, "targetPlatform": %q,
"timestamp": "2026-01-02T03:04:05Z",
"files": {"download": "https://open-vsx.org/api/acme/tool/%s/file/%s.vsix"}}`, version, platform, dir, file)
}

// The platform qualifier picks a build among those a version lists. A
// universal build serves any platform, and a build for the platform wins
// over it.
func TestResolveOpenVSXPlatform(t *testing.T) {
builds := []string{
openVSXBuild("1.0.0", "linux-x64"),
openVSXBuild("1.0.0", "darwin-arm64"),
openVSXBuild("2.0.0", "universal"),
openVSXBuild("3.0.0", "universal"),
openVSXBuild("3.0.0", "linux-x64"),
}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/api/v2/-/query" || r.URL.Query().Get("offset") != "0" {
_, _ = w.Write([]byte(`{"offset": 0, "totalSize": 0, "extensions": []}`))
return
}
_, _ = fmt.Fprintf(w, `{"offset": 0, "totalSize": %d, "extensions": [%s]}`, len(builds), strings.Join(builds, ","))
}))
defer server.Close()

reg, err := registries.New("vscode-extension", server.URL, registries.DefaultClient())
if err != nil {
t.Fatal(err)
}
r := NewResolver()
r.RegisterRegistry(reg)

tests := []struct {
name string
version string
platform string
want string
}{
{"platform-only version", "1.0.0", "linux-x64", "acme.tool-1.0.0@linux-x64.vsix"},
{"platform-only version, second build", "1.0.0", "darwin-arm64", "acme.tool-1.0.0@darwin-arm64.vsix"},
{"universal-only version", "2.0.0", "linux-x64", "acme.tool-2.0.0.vsix"},
{"both, platform build wins", "3.0.0", "linux-x64", "acme.tool-3.0.0@linux-x64.vsix"},
{"both, other platform gets universal", "3.0.0", "darwin-arm64", "acme.tool-3.0.0.vsix"},
{"both, universal asked for", "3.0.0", "universal", "acme.tool-3.0.0.vsix"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
info, err := r.ResolveWithOptions(context.Background(), "vscode-extension", "acme/tool", tt.version,
ResolveOptions{Qualifiers: map[string]string{"platform": tt.platform}})
if err != nil {
t.Fatalf("Resolve failed: %v", err)
}
if info.Filename != tt.want {
t.Errorf("Filename = %q, want %q", info.Filename, tt.want)
}
})
}

_, err = r.ResolveWithOptions(context.Background(), "vscode-extension", "acme/tool", "1.0.0",
ResolveOptions{Qualifiers: map[string]string{"platform": "win32-x64"}})
if !errors.Is(err, ErrNoMatchingArtifact) {
t.Errorf("platform without a build: err = %v, want ErrNoMatchingArtifact", err)
}
}

type urlOnlyRegistry struct{ fakeRegistry }

func (f *urlOnlyRegistry) URLs() client.URLBuilder { //nolint:ireturn // satisfying Registry interface
return &client.BaseURLs{DownloadFn: func(name, version string) string {
return "https://example.com/downloads/" + name + "-" + version + ".gem"
}}
}

// A registry that lists no artifacts, such as RubyGems, has nothing to match
// a selection qualifier against, so its built URL is still the answer. That
// also holds where it lists the version under another number: RubyGems
// lists a platform build as 1.0.0-java.
func TestResolveSelectionQualifierWithoutArtifactsUsesTheBuiltURL(t *testing.T) {
for name, listed := range map[string]string{"listed": "1.0.0", "listed with its platform": "1.0.0-java"} {
t.Run(name, func(t *testing.T) {
r := NewResolver()
r.RegisterRegistry(&urlOnlyRegistry{fakeRegistry{versions: []registries.Version{{Number: listed}}}})

info, err := r.ResolveWithOptions(context.Background(), "fake", "pkg", "1.0.0",
ResolveOptions{Qualifiers: map[string]string{"platform": "java"}})
if err != nil {
t.Fatalf("Resolve failed: %v", err)
}
if want := "https://example.com/downloads/pkg-1.0.0.gem"; info.URL != want {
t.Errorf("URL = %q, want %q", info.URL, want)
}
})
}
}

// A version that lists artifacts and has none for the qualifier is an error,
// not the built URL.
func TestResolveSelectionQualifierWithoutMatchDoesNotUseTheBuiltURL(t *testing.T) {
r := NewResolver()
r.RegisterRegistry(&urlOnlyRegistry{fakeRegistry{versions: []registries.Version{{
Number: "1.0.0",
Artifacts: []registries.Artifact{{URL: "https://example.com/pkg-1.0.0-linux.bin", Metadata: map[string]any{"platform": "linux-x64"}}},
}}}})
_, err := r.ResolveWithOptions(context.Background(), "fake", "pkg", "1.0.0",
ResolveOptions{Qualifiers: map[string]string{"platform": "win32-x64"}})
if !errors.Is(err, ErrNoMatchingArtifact) {
t.Errorf("err = %v, want ErrNoMatchingArtifact", err)
}
}

// Only the platform qualifier has a value that stands for every other.
func TestResolveUniversalOnlyStandsForPlatforms(t *testing.T) {
r := NewResolver()
r.RegisterRegistry(&fakeRegistry{versions: []registries.Version{{
Number: "1.0.0",
Artifacts: []registries.Artifact{
{URL: "https://example.com/pkg-1.0.0.bin", Metadata: map[string]any{"arch": "universal", "platform": "linux-x64"}},
},
}}})

_, err := r.ResolveWithOptions(context.Background(), "fake", "pkg", "1.0.0",
ResolveOptions{Qualifiers: map[string]string{"arch": "amd64", "platform": "linux-x64"}})
if !errors.Is(err, ErrNoMatchingArtifact) {
t.Errorf("err = %v, want ErrNoMatchingArtifact", err)
}
}
Loading
Loading