A customer using the Sentry Node Lambda layer sees warm invocations on many functions randomly run until the function timeout, even though the handler has already finished. The only extension attached is the one shipped in the Sentry layer, and CloudWatch shows elevated PostRuntimeExtensionsDuration, so this looks like a possible Sentry Lambda extension hang. This doesn't match the >300s trigger in #24218 (fixed in #24811), since these functions have a 30s timeout.
Environment
@sentry/aws-serverless 10.59.0, layer SentryNodeServerlessSDKv10:73nodejs18.x, CJS output via esbuild (SST v2), 30s timeout, 1024 MB- Hono app exported via
handle(app) from hono/aws-lambda, auto-wrapped by awsLambdaIntegration
- Manual setup:
NODE_OPTIONS="--import ./instrument.js" with a single Sentry.init (tracing on, captureConsoleIntegration, ignoreSpans)
- No other Lambda extensions
Observed (verified from customer logs/metrics)
- Handler logs its final line about 140 ms into the invocation. Nothing is logged after that until
END/REPORT with Duration: 30000.00 ms, Status: timeout.
- Not cold starts. Happens intermittently across many functions.
PostRuntimeExtensionsDuration spikes in the same time window: hourly Maximum of ~14.8s, plus a ~7.4s spike earlier the same day. Normal values are ~0.
Unknown
- Whether the 14.8s
PostRuntimeExtensionsDuration value comes from the timed-out invocation. The metric is aggregated, so it can't be tied to a request ID.
- Why the 14.8s maximum doesn't account for the full ~29.9s gap after the handler finished.
- What the extension is doing during the delay. In
packages/aws-serverless/src/lambda-extension/ the /event/next loop has no error handling, and failures are only logged via debug, which never prints in the extension process. So a stalled or dead poll would leave no trace.
Hypothesis (medium confidence): the layer extension sometimes delays or stops calling /event/next after an invocation, and Lambda holds the invocation open until the timeout. The trigger is unknown, but it isn't the 300s undici headersTimeout described in #24218.
via Daniel Sheccoury.
--
View Junior Session in Sentry
A customer using the Sentry Node Lambda layer sees warm invocations on many functions randomly run until the function timeout, even though the handler has already finished. The only extension attached is the one shipped in the Sentry layer, and CloudWatch shows elevated
PostRuntimeExtensionsDuration, so this looks like a possible Sentry Lambda extension hang. This doesn't match the >300s trigger in #24218 (fixed in #24811), since these functions have a 30s timeout.Environment
@sentry/aws-serverless10.59.0, layerSentryNodeServerlessSDKv10:73nodejs18.x, CJS output via esbuild (SST v2), 30s timeout, 1024 MBhandle(app)fromhono/aws-lambda, auto-wrapped byawsLambdaIntegrationNODE_OPTIONS="--import ./instrument.js"with a singleSentry.init(tracing on,captureConsoleIntegration,ignoreSpans)Observed (verified from customer logs/metrics)
END/REPORTwithDuration: 30000.00 ms,Status: timeout.PostRuntimeExtensionsDurationspikes in the same time window: hourlyMaximumof ~14.8s, plus a ~7.4s spike earlier the same day. Normal values are ~0.Unknown
PostRuntimeExtensionsDurationvalue comes from the timed-out invocation. The metric is aggregated, so it can't be tied to a request ID.packages/aws-serverless/src/lambda-extension/the/event/nextloop has no error handling, and failures are only logged viadebug, which never prints in the extension process. So a stalled or dead poll would leave no trace.Hypothesis (medium confidence): the layer extension sometimes delays or stops calling
/event/nextafter an invocation, and Lambda holds the invocation open until the timeout. The trigger is unknown, but it isn't the 300s undiciheadersTimeoutdescribed in #24218.via Daniel Sheccoury.
--
View Junior Session in Sentry