Skip to content

lfsd: trace requests end to end with OpenTelemetry - #26

Open
unknwon wants to merge 5 commits into
lfs-libraryfrom
otel-tracing
Open

unknwon wants to merge 5 commits into
lfs-libraryfrom
otel-tracing

Conversation

@unknwon

@unknwon unknwon commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

Summary

Adds OpenTelemetry tracing across lfsd and the lfs module. Stacked on #23.

  • [tracing] config: ENABLED (default false), ENDPOINT (OTLP/HTTP, empty falls back to the standard OTEL_EXPORTER_OTLP_* env vars), SAMPLE_RATIO (parent-based, default 1). When disabled, the global tracer provider stays no-op. Spans are flushed on shutdown.
  • HTTP: otelhttp wraps the router. Each LFS route names its server span, e.g., POST /{host}/{**}/info/lfs/objects/batch, with lfs.forge.host and lfs.repo. An incoming W3C traceparent is continued. /healthz is not traced.
  • Auth: an authorize span records whether the permission came from the cache and the resulting permission. It never records credentials. Forge API calls are client spans, and the trace context is not sent to the forge.
  • lfs module: depends only on the OpenTelemetry API. Service.TracerProvider (nil means global) gets lfs.Batch, lfs.Upload, lfs.Download, and lfs.Verify spans, with a child span for every storage call (storage.Put, storage.Head, etc.). The storage calls are instrumented in Service, so any plugged-in backend is traced without changes. Client errors (*lfs.Error) are recorded as lfs.error.code and do not fail the span.
  • Database: the GORM OpenTelemetry plugin traces queries without bound values. Startup queries share one database.New trace.
  • Janitor: each sweep is a janitor.Sweep trace with a child per pass and its deleted and failed counts.
  • Logs: records in a traced context carry traceID and spanID. Error records are added to the span as events and fail it.

Test plan

  • moon run :lint and go test -race ./... for both projects.
  • lfs: spans go to the injected provider, storage spans are children of the operation, client errors leave the status unset, and internal errors fail it.
  • forge: no traceparent reaches the forge even with a global propagator installed. The test fails when the empty propagator is removed. The token does not appear in any span attribute.
  • logx: trace and span IDs on records, error-only span events and status.
  • lfsd: route span names and http.route.
  • End to end on PostgreSQL 17 with filesystem storage and a local OTLP sink: real git-lfs 3.6.1 push and pull, an upstream traceparent continued, a bad GitHub token traced through authorize to the API call, and spans flushed on SIGTERM. The token and the Authorization header appear nowhere in the exported spans.

Move the transfer logic of lfsd into a separate Go module,
unknwon.dev/git-lfs-server/lfs, so that other Git servers can serve LFS
with the same storage backends. The module has an lfs.Service that
answers batch, upload, download, and verify requests on top of an
lfs.Store for metadata and a storage.Backend for content, and leaves
routing, authentication, and response encoding to the caller.

lfsd now builds one lfs.Service per forge host and keeps its HTTP
responses, except that a proxied upload of an already stored object
must send matching content before the object is linked to another
repository. Previously, any body was accepted in that case, so a
writer of one repository could gain read access to any object of
another repository by its OID and size.
Windows cannot remove the test's temp dir while the object file is open.
Add Service.Serve, which handles the batch, upload, download, and verify
requests of a repository given an Access that the caller resolved after
authenticating the request. It dispatches on the method and the last
path segment, so it works with any router that runs net/http handlers
and any URL layout. lfsd now only maps its forge permissions to an
Access and hands requests to it.

Upload actions now set Content-Type to application/octet-stream so the
Git LFS client does not send a type detected from the file content.
@unknwon unknwon mentioned this pull request Nov 12, 2026
4 tasks done
@unknwon
unknwon added this pull request to stack #28 November 12, 2026 22:31
Service gains a TracerProvider field, defaulting to the global provider,
and starts a span for each of Batch, Upload, Download, and Verify, with a
child span for every call to the storage backend. Instrumenting the calls
in Service traces any plugged-in backend without changes to it. The module
depends only on the OpenTelemetry API.

Client errors (*Error) are recorded as lfs.error.code and leave the span
status unset. Internal errors fail the span.
Add a [tracing] section (ENABLED, ENDPOINT, SAMPLE_RATIO) that installs a
global tracer provider with a parent-based ratio sampler and W3C trace
context propagation. Disabled by default, which keeps every tracer no-op.

- HTTP: otelhttp wraps the router, spans are named after the matched LFS
  route, and /healthz is not traced.
- Auth: an authorize span records cache hits and the permission, never
  credentials. Forge API calls are client spans, and the trace context is
  not sent to the forge.
- Database: the GORM OpenTelemetry plugin traces queries without bound
  values. Startup queries share a database.New trace.
- Janitor: each sweep is a janitor.Sweep trace with deleted and failed
  counts per pass.
- Logs: records in a traced context carry traceID and spanID, and error
  records are attached to the span as events.
@unknwon
unknwon removed this pull request from stack #28 November 13, 2026 04:02
@unknwon
unknwon changed the base branch from ini-v2 to lfs-library November 13, 2026 04:02
@unknwon
unknwon added this pull request to stack #29 November 13, 2026 04:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant