Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -348,6 +348,7 @@ Each upstream used by a built-in package route can be set in YAML or JSON under
| `upstream.oci_default` | `PROXY_UPSTREAM_OCI_DEFAULT` | `https://registry-1.docker.io` |
| `upstream.debian` | `PROXY_UPSTREAM_DEBIAN` | `http://deb.debian.org/debian` |
| `upstream.rpm` | `PROXY_UPSTREAM_RPM` | `https://dl.fedoraproject.org/pub/fedora/linux` |
| `upstream.rpm_repository_paths` | - | `["updates/{release}/Everything/{arch}", "releases/{release}/Everything/{arch}/os"]` |
| `upstream.homebrew_api` | `PROXY_UPSTREAM_HOMEBREW_API` | `https://formulae.brew.sh/api` |
| `upstream.homebrew_artifact` | `PROXY_UPSTREAM_HOMEBREW_ARTIFACT` | `https://ghcr.io` |
| `upstream.apk_snapshots` | `PROXY_UPSTREAM_APK_SNAPSHOTS` | disabled; see [Alpine APK snapshots](#alpine-apk-snapshots) |
Expand Down Expand Up @@ -2006,6 +2007,8 @@ A CRAN version is mirrored as its source package from `upstream.cran`, for examp

An Alpine package is mirrored from the repository in `upstream.apk` named as the namespace, `alpine` by default, as syft writes it in image SBOMs: `pkg:apk/alpine/busybox@1.36.1-r29?arch=x86_64&distro=alpine-3.20.3`. The `distro` qualifier gives the release (`v3.20`) and `arch` the architecture; packages without either are refused. Edge isn't mirrored: a `distro` with a pre-release suffix, such as `alpine-3.21.0_alpha20240807`, is refused, but an edge system that reports a plain release number can't be told apart and is mirrored from that release. The package is looked up in the `main` and then the `community` `APKINDEX.tar.gz` of that release and cached under the path `/apk/{repository}/{release}/{repo}/{arch}/{file}` serves it from. The file isn't checked by the proxy, as when serving, since the index checksum covers only the package's control section; apk checks it on install. The indexes aren't mirrored, so `apk add` needs the upstream or an index the proxy already cached. Give a version: Alpine repositories keep only the current one.

An RPM package is mirrored from the repositories below `upstream.rpm` that `upstream.rpm_repository_paths` names, in order, as syft writes it in image SBOMs: `pkg:rpm/fedora/curl@8.6.0-10.fc40?arch=x86_64&distro=fedora-40&epoch=0`. The `distro` qualifier gives the release, which replaces `{release}` (`40`, `9.4` for `rocky-9.4`, or `15.6` for `opensuse-leap-15.6`) and `{major}` (`9`, `15`), and `arch` the architecture, which replaces `{arch}`; packages without either are refused. A `noarch` package is looked up in the `x86_64` repositories. By default the paths are Fedora's on dl.fedoraproject.org, its updates and then its release; another distribution's tree needs its own, such as `["{major}/BaseOS/{arch}/os", "{major}/AppStream/{arch}/os"]` for Rocky Linux under `https://dl.rockylinux.org/pub/rocky`. The package is taken from the first repository whose primary metadata lists it, with the `epoch` qualifier picking among builds that differ only in their epoch, checked against its SHA-256 where the metadata gives one, and cached under its path below `upstream.rpm`, which `/rpm/` serves it from. The metadata isn't mirrored. Give a version: Fedora's update repositories keep only the newest build.

| Flag | Default | Description |
|------|---------|-------------|
| `--sbom` | | Path to CycloneDX or SPDX SBOM file |
Expand Down
4 changes: 2 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ require (
github.com/git-pkgs/integrity v0.2.0
github.com/git-pkgs/magic v0.6.0
github.com/git-pkgs/purl v0.2.1
github.com/git-pkgs/registries v0.13.2
github.com/git-pkgs/registries v0.14.0
github.com/git-pkgs/sbom v0.1.9
github.com/git-pkgs/spdx v0.3.6
github.com/git-pkgs/vers v0.7.2
Expand Down Expand Up @@ -75,7 +75,7 @@ require (
github.com/dustin/go-humanize v1.1.0 // indirect
github.com/ecosyste-ms/ecosystems-go v0.5.0 // indirect
github.com/facebookgo/clock v0.0.0-20150410010913-600d898af40a // indirect
github.com/git-pkgs/pom v0.1.8 // indirect
github.com/git-pkgs/pom v0.1.9 // indirect
github.com/git-pkgs/vulns v0.2.4 // indirect
github.com/github/go-spdx/v2 v2.8.0 // indirect
github.com/go-logr/logr v1.4.4 // indirect
Expand Down
8 changes: 4 additions & 4 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -131,12 +131,12 @@ github.com/git-pkgs/integrity v0.2.0 h1:sxpAk7EUllZnTYttOoPTW/Yfe5J2imglpuJiuBUZ
github.com/git-pkgs/integrity v0.2.0/go.mod h1:vPLb2RB+hR/iCMIgqph0PZbQ7iktXDTK+uupoHQikKk=
github.com/git-pkgs/magic v0.6.0 h1:fINPe67FEIaFRVlZzhOIDClMkm+gftBroBENa8bHSxQ=
github.com/git-pkgs/magic v0.6.0/go.mod h1:SXOqcsNmbmpZjJZHEEWnwxprbsFvpwWgTAZrgXp4Jm4=
github.com/git-pkgs/pom v0.1.8 h1:mXCuWw6XkrKKZb6IurRZyIznYLPhV5AYg5ZO38Uj/l8=
github.com/git-pkgs/pom v0.1.8/go.mod h1:sQBeYRzjCUn77fcABPn6I/a1aBHFxm55jTmkNxYtGtw=
github.com/git-pkgs/pom v0.1.9 h1:gP7nqAS+UXITeRXjHhpyrMHRl9K3sRRBzRLV3+TeqHU=
github.com/git-pkgs/pom v0.1.9/go.mod h1:sQBeYRzjCUn77fcABPn6I/a1aBHFxm55jTmkNxYtGtw=
github.com/git-pkgs/purl v0.2.1 h1:b46hRlOMbrG6aXjjFJdYixGi5WTh0TLlAnH9iNeuOS0=
github.com/git-pkgs/purl v0.2.1/go.mod h1:l8rCfvt5iXVrFIcoQ8+vCLOaHhrtEW2id0LENP+IzMw=
github.com/git-pkgs/registries v0.13.2 h1:3fWinubUZbBEJ+kDii3O5kzqHJMJBmJK1X9FefAow+I=
github.com/git-pkgs/registries v0.13.2/go.mod h1:UlX6jcAd4ftRBVVNzB4mQg32tfSNCk/L2pO4Wtplnho=
github.com/git-pkgs/registries v0.14.0 h1:PQbfRFghM6cN2xu1FcDHS2F73yOMATVIBaq+zY/6uSI=
github.com/git-pkgs/registries v0.14.0/go.mod h1:zgtz1lneDi8olIzAJxnSvSMT7PzChp9dsbFxJVbfyfQ=
github.com/git-pkgs/sbom v0.1.9 h1:sfF5cJAwmgrm2N4HTXQWjiBe8tdu8xwhjwfAOyDNMfc=
github.com/git-pkgs/sbom v0.1.9/go.mod h1:KIkZX0v31l62SeFsi2qsQXcZRVrAXvIdPQ4x4en+G2M=
github.com/git-pkgs/spdx v0.3.6 h1:tbkqeYLKgAcDV1qPZNAkNxpNMTx9EfoNQQhcVEHApzs=
Expand Down
41 changes: 41 additions & 0 deletions internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ import (
_ "embed"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"maps"
"math"
Expand Down Expand Up @@ -831,6 +832,13 @@ type UpstreamConfig struct {
// Default: https://dl.fedoraproject.org/pub/fedora/linux
RPM string `json:"rpm" yaml:"rpm"`

// RPMRepositoryPaths are the repository directories below upstream.rpm
// that proxy mirror looks an RPM package up in, in order, with
// {release}, {major} and {arch} taken from the package's distro and
// arch qualifiers. Default: Fedora's updates/{release}/Everything/{arch}
// and releases/{release}/Everything/{arch}/os.
RPMRepositoryPaths []string `json:"rpm_repository_paths" yaml:"rpm_repository_paths"`

// HomebrewAPI is the upstream Homebrew JSON API URL.
// Default: https://formulae.brew.sh/api
HomebrewAPI string `json:"homebrew_api" yaml:"homebrew_api"`
Expand Down Expand Up @@ -1027,6 +1035,39 @@ func (u *UpstreamConfig) Validate() error {
)
}
}
return validateRPMRepositoryPaths(u.RPMRepositoryPaths)
}

func validateRPMRepositoryPaths(paths []string) error {
for _, p := range paths {
if err := validateRPMRepositoryPath(p); err != nil {
return fmt.Errorf("invalid upstream.rpm_repository_paths entry %q: %w", p, err)
}
}
return nil
}

// rpmPathPlaceholder matches the placeholders of an RPM repository path.
var rpmPathPlaceholder = regexp.MustCompile(`\{(release|major|arch)\}`)

// validateRPMRepositoryPath checks a path below upstream.rpm: relative,
// without dot segments or a query, and with no placeholders but {release},
// {major} and {arch}.
func validateRPMRepositoryPath(p string) error {
rest := rpmPathPlaceholder.ReplaceAllString(p, "x")
switch {
case p == "":
return errors.New("empty")
case strings.HasPrefix(p, "/") || strings.Contains(p, "://"):
return errors.New("not a path relative to upstream.rpm")
case strings.ContainsAny(rest, "{}?#\\"):
return errors.New("only {release}, {major} and {arch} can be replaced, and it can't have a query")
}
for _, segment := range strings.Split(p, "/") {
if segment == "." || segment == ".." {
return errors.New("dot segments aren't allowed")
}
}
return nil
}

Expand Down
28 changes: 28 additions & 0 deletions internal/config/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1975,6 +1975,34 @@ func TestGenericUpstreamNamedHosted(t *testing.T) {
}
}

func TestValidateRPMRepositoryPaths(t *testing.T) {
tests := []struct {
name string
paths []string
wantErr bool
}{
{"Fedora", []string{"updates/{release}/Everything/{arch}", "releases/{release}/Everything/{arch}/os"}, false},
{"major release", []string{"{major}/BaseOS/{arch}/os"}, false},
{"empty path", []string{""}, true},
{"absolute path", []string{"/updates/{release}"}, true},
{"URL", []string{"https://example.com/{release}"}, true},
{"dot dot segment", []string{"updates/../{release}"}, true},
{"dot segment", []string{"./{release}"}, true},
{"unknown placeholder", []string{"{version}/{arch}"}, true},
{"unclosed placeholder", []string{"{release/{arch}"}, true},
{"query", []string{"{release}?x=1"}, true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := Default()
cfg.Upstream.RPMRepositoryPaths = tt.paths
if err := cfg.Validate(); (err != nil) != tt.wantErr {
t.Errorf("Validate() error = %v, wantErr %v", err, tt.wantErr)
}
})
}
}

func TestValidateFreeBSDUpstreams(t *testing.T) {
tests := []struct {
name string
Expand Down
8 changes: 7 additions & 1 deletion internal/handler/rpm.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,16 @@ func NewRPMHandler(proxy *Proxy, proxyURL string) *RPMHandler {
}
}

// RPMUpstreamURL returns upstream.rpm, the default Fedora tree when none is
// configured. Package files are cached under their path below it.
func RPMUpstreamURL(configured string) string {
return configuredUpstreamURL(configured, defaultRPMUpstream)
}

// NewRPMHandlerWithUpstream creates an RPM handler with a custom upstream.
func NewRPMHandlerWithUpstream(proxy *Proxy, proxyURL, upstreamURL string) *RPMHandler {
h := NewRPMHandler(proxy, proxyURL)
h.upstreamURL = configuredUpstreamURL(upstreamURL, defaultRPMUpstream)
h.upstreamURL = RPMUpstreamURL(upstreamURL)
return h
}

Expand Down
10 changes: 10 additions & 0 deletions internal/mirror/mirror.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,12 +53,18 @@ type Mirror struct {
// the official Alpine mirror as "alpine".
APKRepositories map[string]string

// RPMUpstream is upstream.rpm, and RPMRepositoryPaths are
// upstream.rpm_repository_paths. Empty ones use Fedora's tree.
RPMUpstream string
RPMRepositoryPaths []string

// RefusePackages, when set, fails every item Images does not take with
// this reason instead of fetching it, so nothing is cached.
RefusePackages string

cran cranIndex
apk apkIndexes
rpm rpmIndexes
}

// New creates a new Mirror with the given dependencies.
Expand Down Expand Up @@ -297,6 +303,10 @@ func (m *Mirror) mirrorOne(ctx context.Context, pv PackageVersion, tracker *prog
m.mirrorAPK(ctx, pv, tracker)
return
}
if pv.Ecosystem == "rpm" {
m.mirrorRPM(ctx, pv, tracker)
return
}
if pv.Ecosystem == "git" {
m.mirrorGit(ctx, pv, tracker)
return
Expand Down
Loading
Loading