Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion .github/workflows/go.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,17 @@ on:
- '**.go'
- 'go.mod'
- 'go.sum'
- 'lfs/go.mod'
- 'lfs/go.sum'
- '.golangci.yml'
- '.github/workflows/go.yml'
pull_request:
paths:
- '**.go'
- 'go.mod'
- 'go.sum'
- 'lfs/go.mod'
- 'lfs/go.sum'
- '.golangci.yml'
- '.github/workflows/go.yml'
env:
Expand All @@ -38,6 +42,7 @@ jobs:
shell: bash
run: |
go mod tidy
(cd lfs && go mod tidy)
STATUS=$(git status --porcelain)
if [ ! -z "$STATUS" ]; then
echo "Unstaged files:"
Expand All @@ -50,6 +55,12 @@ jobs:
with:
version: latest
args: --timeout=30m
- name: Run golangci-lint (lfs module)
uses: golangci/golangci-lint-action@9fae48acfc02a90574d7c304a1758ef9895495fa # v7.0.1
with:
version: latest
working-directory: lfs
args: --timeout=30m

test:
name: Test
Expand All @@ -68,6 +79,7 @@ jobs:
- name: Run tests with coverage
run: |
go test -shuffle=on -v -race -coverprofile=coverage -covermode=atomic -json ./... > test-report.json
(cd lfs && go test -shuffle=on -v -race -coverprofile=coverage -covermode=atomic -json ./...) >> test-report.json
go install github.com/mfridman/tparse@latest
tparse -all -file=test-report.json

Expand All @@ -88,4 +100,7 @@ jobs:
with:
go-version: ${{ matrix.go-version }}
- name: Run tests with coverage
run: go test -shuffle=on -v -coverprofile=coverage -covermode=atomic ./...
shell: bash
run: |
go test -shuffle=on -v -coverprofile=coverage -covermode=atomic ./...
cd lfs && go test -shuffle=on -v -coverprofile=coverage -covermode=atomic ./...
2 changes: 2 additions & 0 deletions .github/workflows/integration.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ on:
- 'integration-tests/**'
- 'go.mod'
- 'go.sum'
- 'lfs/**'
- '.github/workflows/integration.yml'
pull_request:
paths:
Expand All @@ -17,6 +18,7 @@ on:
- 'integration-tests/**'
- 'go.mod'
- 'go.sum'
- 'lfs/**'
- '.github/workflows/integration.yml'

permissions:
Expand Down
1 change: 1 addition & 0 deletions .moon/workspace.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ $schema: 'https://moonrepo.dev/schemas/workspace.json'

projects:
- '.'
- 'lfs'
1 change: 1 addition & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
FROM golang:1.27.4-alpine3.24@sha256:0a5f7fe796599ff487605987432e7619ae45beeb2fba4c2ce0b907879932aebf AS builder
WORKDIR /src
COPY go.mod go.sum ./
COPY lfs/go.mod lfs/go.sum ./lfs/
RUN go mod download
COPY . .
ARG BUILD_DATE
Expand Down
42 changes: 42 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,48 @@ REPO_ALLOWLIST = myorg/**, otherorg/specific-repo

The allowlist is enforced before the forge token check, so a request for a disallowed repo is rejected without a round-trip to the forge.

## Using as a library

The `unknwon.dev/git-lfs-server/lfs` module implements the Batch API and the basic transfer adapter for other Git servers to embed. It is a separate Go module so that importers do not pull in the dependencies of lfsd (PostgreSQL, Flamego, forge clients).

An `lfs.Service` combines an `lfs.Store`, which records objects and the repositories they are linked to, with a `storage.Backend`, which holds the content. A `storage.Proxier` backend (e.g., `storage.FilesystemBackend`) streams content through the server, and a `storage.Presigner` backend (e.g., `storage.S3PresignBackend`) redirects clients to the storage. `PreviousStorages` keeps objects readable after the backend for new uploads changes.

`Service.Serve` handles the HTTP requests of the batch, upload, download, and verify endpoints. The caller routes the requests, authenticates them, and passes an `lfs.Access` with the repository, the base URL of its endpoints, and whether the actor can read and write. `Serve` works with any router that can run a `net/http` handler, e.g., Flamego, Macaron, or `http.ServeMux`:

```go
mux.HandleFunc("/{repo}/info/lfs/objects/", func(w http.ResponseWriter, r *http.Request) {
access, ok := authorize(w, r) // Your authentication and authorization.
if !ok {
return
}
service.Serve(logger, w, r, access)
})
```

The lower-level `Batch`, `Upload`, `Download`, and `Verify` methods are available for servers that need their own HTTP handling.

The module depends only on the OpenTelemetry API. `Service.TracerProvider` receives the spans of LFS operations (`lfs.Batch`, `lfs.Upload`, `lfs.Download`, `lfs.Verify`) and of every call to the storage backend (`storage.Put`, `storage.Head`, etc.), so custom backends are traced without extra work. When it is nil, the global provider from `go.opentelemetry.io/otel` is used.

## Tracing

lfsd exports OpenTelemetry traces over OTLP/HTTP when tracing is enabled:

```ini
[tracing]
ENABLED = true
ENDPOINT = http://otel-collector:4318/v1/traces
SAMPLE_RATIO = 0.1
```

Leave `ENDPOINT` empty to configure the exporter with the standard `OTEL_EXPORTER_OTLP_*` environment variables, e.g., `OTEL_EXPORTER_OTLP_HEADERS` for an API key. `OTEL_SERVICE_NAME` and `OTEL_RESOURCE_ATTRIBUTES` override the default `service.name=lfsd` and `service.version` (the build commit). Requests that carry a sampled W3C `traceparent` are always sampled, so lfsd joins the traces of a reverse proxy or a client that propagates context.

Each LFS request produces a span named after its route, e.g., `POST /{host}/{**}/info/lfs/objects/batch`, with these children:

- `authorize`: whether the permission came from the cache, with the forge API call as a child. Credentials are never recorded, and the trace context is not sent to the forge.
- `lfs.Batch`, `lfs.Upload`, `lfs.Download`, or `lfs.Verify`, with the database queries and storage calls of the operation.

The janitor records a `janitor.Sweep` trace per sweep. `/healthz` is not traced. Log records written during a traced request carry its `traceID` and `spanID`, and error logs are attached to the span as events.

## Local development

This project uses [moon](https://moonrepo.dev/) as its task runner. Install it with Homebrew:
Expand Down
26 changes: 18 additions & 8 deletions cmd/lfsd/auth.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,45 +12,52 @@ import (
"github.com/cockroachdb/errors"
"github.com/flamego/cache"
"github.com/flamego/flamego"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"

"unknwon.dev/git-lfs-server/internal/forge"
"unknwon.dev/git-lfs-server/internal/logx"
"unknwon.dev/git-lfs-server/lfs"
)

func authorize(forges map[string]forge.Provider) flamego.Handler {
return func(c flamego.Context, logger *logx.Logger, cache cache.Cache) {
logger = logger.Scoped("authorize")
ctx := c.Request().Context()

host := hostFromContext(c)
// The span never carries credentials: the forge decides access, and only
// its outcome is recorded.
ctx, span := otel.Tracer("unknwon.dev/git-lfs-server/cmd/lfsd").Start(c.Request().Context(), "authorize")
defer span.End()

provider, ok := forges[host]
if !ok {
writeLFSError(c.ResponseWriter(), http.StatusBadRequest, "unsupported forge host")
lfs.WriteError(c.ResponseWriter(), http.StatusBadRequest, "unsupported forge host")
return
}

username, token, ok := c.Request().BasicAuth()
if !ok || token == "" {
c.ResponseWriter().Header().Set("WWW-Authenticate", `Basic realm="git-lfs"`)
writeLFSError(c.ResponseWriter(), http.StatusUnauthorized, "basic auth credentials are required")
lfs.WriteError(c.ResponseWriter(), http.StatusUnauthorized, "basic auth credentials are required")
return
}

repo := strings.ToLower(c.Param("**"))
if repo == "" {
writeLFSError(c.ResponseWriter(), http.StatusBadRequest, "repository path is missing")
lfs.WriteError(c.ResponseWriter(), http.StatusBadRequest, "repository path is missing")
return
}

if !provider.Allow(repo) {
logger.InfoContext(ctx, "Repository rejected by allowlist", "host", host, "repo", repo)
writeLFSError(c.ResponseWriter(), http.StatusForbidden, "repository is not in the allowlist")
lfs.WriteError(c.ResponseWriter(), http.StatusForbidden, "repository is not in the allowlist")
return
}

key := authCacheKey(host, repo, username, token)
if v, err := cache.Get(ctx, key); err == nil {
if perm, ok := v.(forge.Permission); ok {
span.SetAttributes(attribute.Bool("lfs.auth.cache_hit", true), attribute.String("lfs.permission", string(perm)))
c.Map(perm)
return
}
Expand All @@ -59,13 +66,15 @@ func authorize(forges map[string]forge.Provider) flamego.Handler {
logger.WarnContext(ctx, "Auth cache lookup failed", "error", err)
}

span.SetAttributes(attribute.Bool("lfs.auth.cache_hit", false))
perm, rawTTL, err := provider.Authorize(ctx, logger.Scoped("forge"), repo, username, token)
if err != nil {
if errors.Is(err, forge.ErrTokenInvalid) {
writeLFSError(c.ResponseWriter(), http.StatusForbidden, "token is invalid or lacks repository access")
span.SetAttributes(attribute.Bool("lfs.auth.token_invalid", true))
lfs.WriteError(c.ResponseWriter(), http.StatusForbidden, "token is invalid or lacks repository access")
} else {
logger.ErrorContext(ctx, "Failed to verify repository permissions", "repo", repo, "error", err)
writeLFSError(c.ResponseWriter(), http.StatusInternalServerError, http.StatusText(http.StatusInternalServerError))
lfs.WriteError(c.ResponseWriter(), http.StatusInternalServerError, http.StatusText(http.StatusInternalServerError))
}
return
}
Expand All @@ -80,6 +89,7 @@ func authorize(forges map[string]forge.Provider) flamego.Handler {
}
}

span.SetAttributes(attribute.String("lfs.permission", string(perm)))
c.Map(perm)
}
}
Expand Down
7 changes: 6 additions & 1 deletion cmd/lfsd/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@ import (
"unknwon.dev/git-lfs-server/internal/database"
"unknwon.dev/git-lfs-server/internal/embed"
"unknwon.dev/git-lfs-server/internal/forge"
"unknwon.dev/git-lfs-server/internal/storage"
"unknwon.dev/git-lfs-server/internal/tracing"
"unknwon.dev/git-lfs-server/lfs/storage"
)

// Host is the host of a forge, e.g., "github.com".
Expand All @@ -22,6 +23,7 @@ type Config struct {
Forges map[Host]forge.Provider
Storages map[Host]storage.Backend
Log LogConfig
Tracing tracing.Config
}

type ServerConfig struct {
Expand Down Expand Up @@ -62,6 +64,9 @@ func loadConfig(customPath string) (*Config, error) {
if err := f.Section("log").MapTo(&c.Log); err != nil {
return nil, errors.Wrap(err, `map [log] config`)
}
if err := f.Section("tracing").MapTo(&c.Tracing); err != nil {
return nil, errors.Wrap(err, `map [tracing] config`)
}

backendsByName := make(map[string]storage.Backend)
for _, s := range f.Sections() {
Expand Down
Loading
Loading