Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ mux.HandleFunc("/{repo}/info/lfs/objects/", func(w http.ResponseWriter, r *http.

The lower-level `Batch`, `Upload`, `Download`, and `Verify` methods are available for servers that need their own HTTP handling.

The module depends only on the OpenTelemetry API. `Service.TracerProvider` receives the spans of LFS operations (`lfs.Batch`, `lfs.Upload`, `lfs.Download`, `lfs.Verify`) and of every call to the storage backend (`storage.Put`, `storage.Head`, etc.), so custom backends are traced without extra work. When it is nil, the global provider from `go.opentelemetry.io/otel` is used.
The module depends only on the OpenTelemetry API. `Service.TracerProvider` receives the spans of LFS operations (`lfs.Batch`, `lfs.Upload`, `lfs.Download`, `lfs.Verify`) and of every call to the storage backend (`storage.Put`, `storage.Head`, etc.), so custom backends are traced without extra work. `Service.MeterProvider` receives the `lfs.batch.objects` and `lfs.proxied` metrics. Either provider falls back to the global one from `go.opentelemetry.io/otel` when it is nil.

## Tracing

Expand All @@ -136,6 +136,28 @@ Each LFS request produces a span named after its route, e.g., `POST /{host}/{**}

The janitor records a `janitor.Sweep` trace per sweep. `/healthz` is not traced. Log records written during a traced request carry its `traceID` and `spanID`, and error logs are attached to the span as events.

## Metrics

lfsd serves Prometheus metrics at `/metrics` on a separate listener, so that they are not public along with the LFS endpoints:

```ini
[metrics]
; Defaults to 127.0.0.1:3357. Use 0.0.0.0:3357 to scrape from another host or
; container, or leave empty to disable metrics.
LISTEN_ADDRESS = 0.0.0.0:3357
```

| Metric | Labels | Description |
|--------|--------|-------------|
| `http_server_request_duration_seconds`, `http_server_request_body_size_bytes`, `http_server_response_body_size_bytes` | `http_route`, `http_request_method`, `http_response_status_code` | LFS requests. `/healthz` is excluded. |
| `http_client_request_duration_seconds` | `server_address`, `http_response_status_code` | Forge API calls. |
| `lfs_batch_objects_total` | `lfs_operation` | Objects requested in batch requests. |
| `lfs_proxied_bytes_total` | `lfs_operation` | Object content transferred through the server, i.e., with `filesystem` storage. Presigned transfers go directly between the client and the storage. |
| `lfsd_auth_cache_lookups_total` | `lfs_auth_cache_hit` | Lookups of cached authorization results. |
| `lfsd_janitor_objects_total` | `janitor_pass`, `janitor_result` | Objects swept by the janitor, by whether their content was deleted. |
| `go_sql_connections_*` | | Database connection pool. |
| `go_*`, `process_*` | | Go runtime and process. |

## Local development

This project uses [moon](https://moonrepo.dev/) as its task runner. Install it with Homebrew:
Expand Down
11 changes: 10 additions & 1 deletion cmd/lfsd/auth.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,19 +14,26 @@ import (
"github.com/flamego/flamego"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/metric"

"unknwon.dev/git-lfs-server/internal/forge"
"unknwon.dev/git-lfs-server/internal/logx"
"unknwon.dev/git-lfs-server/lfs"
)

const instrumentationName = "unknwon.dev/git-lfs-server/cmd/lfsd"

func authorize(forges map[string]forge.Provider) flamego.Handler {
cacheLookups, _ := otel.Meter(instrumentationName).Int64Counter("lfsd.auth.cache.lookups",
metric.WithUnit("{lookup}"),
metric.WithDescription("Lookups of cached authorization results, by whether they hit."),
)
return func(c flamego.Context, logger *logx.Logger, cache cache.Cache) {
logger = logger.Scoped("authorize")
host := hostFromContext(c)
// The span never carries credentials: the forge decides access, and only
// its outcome is recorded.
ctx, span := otel.Tracer("unknwon.dev/git-lfs-server/cmd/lfsd").Start(c.Request().Context(), "authorize")
ctx, span := otel.Tracer(instrumentationName).Start(c.Request().Context(), "authorize")
defer span.End()

provider, ok := forges[host]
Expand Down Expand Up @@ -58,6 +65,7 @@ func authorize(forges map[string]forge.Provider) flamego.Handler {
if v, err := cache.Get(ctx, key); err == nil {
if perm, ok := v.(forge.Permission); ok {
span.SetAttributes(attribute.Bool("lfs.auth.cache_hit", true), attribute.String("lfs.permission", string(perm)))
cacheLookups.Add(ctx, 1, metric.WithAttributes(attribute.Bool("lfs.auth.cache_hit", true)))
c.Map(perm)
return
}
Expand All @@ -67,6 +75,7 @@ func authorize(forges map[string]forge.Provider) flamego.Handler {
}

span.SetAttributes(attribute.Bool("lfs.auth.cache_hit", false))
cacheLookups.Add(ctx, 1, metric.WithAttributes(attribute.Bool("lfs.auth.cache_hit", false)))
perm, rawTTL, err := provider.Authorize(ctx, logger.Scoped("forge"), repo, username, token)
if err != nil {
if errors.Is(err, forge.ErrTokenInvalid) {
Expand Down
8 changes: 8 additions & 0 deletions cmd/lfsd/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ type Config struct {
Storages map[Host]storage.Backend
Log LogConfig
Tracing tracing.Config
Metrics MetricsConfig
}

type ServerConfig struct {
Expand All @@ -36,6 +37,10 @@ type LogConfig struct {
Level string `ini:"LEVEL"`
}

type MetricsConfig struct {
ListenAddress string `ini:"LISTEN_ADDRESS"`
}

const (
storageSectionPrefix = `storage "`
forgeSectionPrefix = `forge "`
Expand Down Expand Up @@ -67,6 +72,9 @@ func loadConfig(customPath string) (*Config, error) {
if err := f.Section("tracing").MapTo(&c.Tracing); err != nil {
return nil, errors.Wrap(err, `map [tracing] config`)
}
if err := f.Section("metrics").MapTo(&c.Metrics); err != nil {
return nil, errors.Wrap(err, `map [metrics] config`)
}

backendsByName := make(map[string]storage.Backend)
for _, s := range f.Sections() {
Expand Down
51 changes: 41 additions & 10 deletions cmd/lfsd/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,20 @@ func main() {
if config.Tracing.Enabled {
logger.InfoContext(ctx, "Tracing enabled", "sampleRatio", config.Tracing.SampleRatio)
}
// Metrics stay no-op without an address, like tracing when disabled.
var metricsSrv *http.Server
if config.Metrics.ListenAddress != "" {
metricsHandler, err := setupMetrics(buildCommit)
if err != nil {
logger.FatalContext(ctx, "Failed to set up metrics", "error", err)
}
mux := http.NewServeMux()
mux.Handle("GET /metrics", metricsHandler)
metricsSrv = &http.Server{
Addr: config.Metrics.ListenAddress,
Handler: mux,
}
}

for host, provider := range config.Forges {
logger.InfoContext(ctx, "Forge initialized", "host", host, "type", provider.Type())
Expand Down Expand Up @@ -87,10 +101,10 @@ func main() {
auth := authorize(config.Forges)
const objectsRoute = "/{host}/{**}/info/lfs/objects"
f.Group(objectsRoute, func() {
f.Post("/batch", traceRoute(objectsRoute+"/batch"), auth, serve)
f.Get("/{oid}", traceRoute(objectsRoute+"/{oid}"), auth, serve)
f.Put("/{oid}", traceRoute(objectsRoute+"/{oid}"), auth, serve)
f.Post("/verify", traceRoute(objectsRoute+"/verify"), auth, serve)
f.Post("/batch", instrumentRoute(objectsRoute+"/batch"), auth, serve)
f.Get("/{oid}", instrumentRoute(objectsRoute+"/{oid}"), auth, serve)
f.Put("/{oid}", instrumentRoute(objectsRoute+"/{oid}"), auth, serve)
f.Post("/verify", instrumentRoute(objectsRoute+"/verify"), auth, serve)
})

ctx, stop := signal.NotifyContext(ctx, syscall.SIGINT, syscall.SIGTERM)
Expand All @@ -110,12 +124,26 @@ func main() {
}
stop()
})
if metricsSrv != nil {
routines.Go(func() {
// Metrics are not worth taking the LFS endpoints down for, so a
// failure here does not stop the server.
if err := metricsSrv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
logger.ErrorContext(ctx, "Metrics server exited unexpectedly", "error", err)
}
})
}
routines.Go(func() {
<-ctx.Done()
logger.WarnContext(ctx, "Server shutdown requested")
if err := srv.Shutdown(context.Background()); err != nil {
logger.ErrorContext(ctx, "Failed to shut down HTTP server gracefully", "error", err)
}
if metricsSrv != nil {
if err := metricsSrv.Shutdown(context.Background()); err != nil {
logger.ErrorContext(ctx, "Failed to shut down metrics server gracefully", "error", err)
}
}
})
routines.Go(func() {
janitor.New(db, config.Storages).Run(ctx, logger.Scoped("janitor"))
Expand All @@ -124,6 +152,7 @@ func main() {
logger.InfoContext(ctx, "Server ready",
"buildDate", buildDate, "buildCommit", buildCommit,
"listenAddr", config.Server.ListenAddress,
"metricsAddr", config.Metrics.ListenAddress,
)
r := routines.WaitAndRecover()
// Flush the spans of the requests that finished during shutdown. The
Expand All @@ -136,13 +165,15 @@ func main() {
}
}

// traceRoute names the server span of the request after route, because the
// otelhttp handler wraps the router and does not know which route matched.
func traceRoute(route string) flamego.Handler {
// instrumentRoute records route as the pattern that matched the request, like
// http.ServeMux does. The otelhttp handler wraps the router and reads the
// pattern after the request to name the server span and to label the HTTP
// metrics with the route. It only sets the http.route span attribute when the
// span starts, before routing, so that is set here.
func instrumentRoute(route string) flamego.Handler {
return func(c flamego.Context) {
span := trace.SpanFromContext(c.Request().Context())
span.SetName(c.Request().Method + " " + route)
span.SetAttributes(
c.Request().Pattern = route
trace.SpanFromContext(c.Request().Context()).SetAttributes(
attribute.String("http.route", route),
attribute.String("lfs.forge.host", hostFromContext(c)),
attribute.String("lfs.repo", repoNameFromContext(c)),
Expand Down
29 changes: 25 additions & 4 deletions cmd/lfsd/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,20 +10,25 @@ import (
"github.com/stretchr/testify/require"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
"go.opentelemetry.io/otel/attribute"
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
"go.opentelemetry.io/otel/sdk/metric/metricdata"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/sdk/trace/tracetest"
)

func TestTraceRoute(t *testing.T) {
func TestInstrumentRoute(t *testing.T) {
recorder := tracetest.NewSpanRecorder()
tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(recorder))
reader := sdkmetric.NewManualReader()

const route = "/{host}/{**}/info/lfs/objects/{oid}"
f := flamego.New()
f.Get(route, traceRoute(route), func(c flamego.Context) {
f.Get(route, instrumentRoute(route), func(c flamego.Context) {
c.ResponseWriter().WriteHeader(http.StatusOK)
})
handler := otelhttp.NewHandler(f, "lfsd", otelhttp.WithTracerProvider(tp))
handler := otelhttp.NewHandler(f, "lfsd",
otelhttp.WithTracerProvider(sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(recorder))),
otelhttp.WithMeterProvider(sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader))),
)

r := httptest.NewRequest(http.MethodGet, "/GitHub.com/Org/Repo/info/lfs/objects/abc", nil)
handler.ServeHTTP(httptest.NewRecorder(), r)
Expand All @@ -34,4 +39,20 @@ func TestTraceRoute(t *testing.T) {
assert.Contains(t, spans[0].Attributes(), attribute.String("http.route", route))
assert.Contains(t, spans[0].Attributes(), attribute.String("lfs.forge.host", "github.com"))
assert.Contains(t, spans[0].Attributes(), attribute.String("lfs.repo", "github.com/org/repo"))

var rm metricdata.ResourceMetrics
require.NoError(t, reader.Collect(t.Context(), &rm))
var routes []string
for _, sm := range rm.ScopeMetrics {
for _, m := range sm.Metrics {
if m.Name != "http.server.request.duration" {
continue
}
for _, dp := range m.Data.(metricdata.Histogram[float64]).DataPoints {
route, _ := dp.Attributes.Value("http.route")
routes = append(routes, route.AsString())
}
}
}
assert.Equal(t, []string{route}, routes, "the request duration must be labeled with the route")
}
31 changes: 31 additions & 0 deletions cmd/lfsd/metrics.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
package main

import (
"net/http"

"github.com/cockroachdb/errors"
"github.com/prometheus/client_golang/prometheus/promhttp"
"go.opentelemetry.io/otel"
otelprometheus "go.opentelemetry.io/otel/exporters/prometheus"
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
"go.opentelemetry.io/otel/sdk/resource"
semconv "go.opentelemetry.io/otel/semconv/v1.43.0"
)

// setupMetrics installs the global meter provider and returns the handler of
// the Prometheus endpoint. Metrics are exported to the default Prometheus
// registry, which also collects the Go runtime and process metrics.
func setupMetrics(serviceVersion string) (http.Handler, error) {
exporter, err := otelprometheus.New()
if err != nil {
return nil, errors.Wrap(err, "create Prometheus exporter")
}
otel.SetMeterProvider(sdkmetric.NewMeterProvider(
sdkmetric.WithReader(exporter),
sdkmetric.WithResource(resource.NewSchemaless(
semconv.ServiceName("lfsd"),
semconv.ServiceVersion(serviceVersion),
)),
))
return promhttp.Handler(), nil
}
13 changes: 11 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,18 @@ require (
github.com/flamego/cache v1.5.1
github.com/flamego/flamego v1.9.7
github.com/lib/pq v1.12.3
github.com/prometheus/client_golang v1.25.0
github.com/sourcegraph/conc v0.3.0
github.com/sourcegraph/run v0.12.0
github.com/stretchr/testify v1.12.1
go.bobheadxi.dev/streamline v1.3.2
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.72.0
go.opentelemetry.io/otel v1.47.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.47.0
go.opentelemetry.io/otel/exporters/prometheus v0.69.0
go.opentelemetry.io/otel/metric v1.47.0
go.opentelemetry.io/otel/sdk v1.47.0
go.opentelemetry.io/otel/sdk/metric v1.47.0
go.opentelemetry.io/otel/trace v1.47.0
gopkg.in/ini.v1 v1.67.2
gorm.io/driver/postgres v1.6.0
Expand Down Expand Up @@ -45,6 +49,7 @@ require (
github.com/aws/aws-sdk-go-v2/service/s3 v1.100.1 // indirect
github.com/aws/smithy-go v1.25.1 // indirect
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/charmbracelet/colorprofile v0.4.3 // indirect
Expand Down Expand Up @@ -82,17 +87,22 @@ require (
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/jinzhu/now v1.1.5 // indirect
github.com/klauspost/compress v1.17.8 // indirect
github.com/klauspost/compress v1.20.1 // indirect
github.com/kr/pretty v0.3.1 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/lucasb-eyer/go-colorful v1.4.0 // indirect
github.com/mattn/go-isatty v0.0.22 // indirect
github.com/mattn/go-runewidth v0.0.23 // indirect
github.com/muesli/cancelreader v0.2.2 // indirect
github.com/muesli/termenv v0.16.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/paulmach/orb v0.11.1 // indirect
github.com/pierrec/lz4/v4 v4.1.21 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/prometheus/client_model v0.6.3 // indirect
github.com/prometheus/common v0.72.0 // indirect
github.com/prometheus/otlptranslator v1.0.0 // indirect
github.com/prometheus/procfs v0.22.0 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/rogpeppe/go-internal v1.14.1 // indirect
github.com/segmentio/asm v1.2.0 // indirect
Expand All @@ -101,7 +111,6 @@ require (
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.47.0 // indirect
go.opentelemetry.io/otel/log v1.47.0 // indirect
go.opentelemetry.io/otel/metric v1.47.0 // indirect
go.opentelemetry.io/proto/otlp v1.11.0 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f // indirect
Expand Down
Loading
Loading