Skip to content

fix(tracing): handle exceptions raised within traces_sampler and other callbacks#6853

Open
ericapisani wants to merge 9 commits into
masterfrom
py-2617-handle-exceptions-combined
Open

fix(tracing): handle exceptions raised within traces_sampler and other callbacks#6853
ericapisani wants to merge 9 commits into
masterfrom
py-2617-handle-exceptions-combined

Conversation

@ericapisani

Copy link
Copy Markdown
Member

When a user-provided traces_sampler callback raises, catch the exception,
log a warning, and fall back to the parent sample rate or the configured
traces_sample_rate instead of propagating the error.

Wrap the before_send callback invocation in capture_internal_exceptions()
so that exceptions raised by user-provided before_send_log,
before_send_metric, and before_send_span callbacks do not crash the
application. When a callback raises, the original, unmodified telemetry
item is sent.

Fixes PY-2617
Fixes #6850

When a user-provided traces_sampler callback raises, catch the exception,
log a warning, and fall back to the parent sample rate or the configured
traces_sample_rate instead of propagating the error.

Refs PY-2617
Refs #6849
Wrap the before_send callback invocation in capture_internal_exceptions()
so that exceptions raised by user-provided before_send_log,
before_send_metric, and before_send_span callbacks do not crash the
application. When a callback raises, the original, unmodified telemetry
item is sent.

Refs PY-2617
Refs #6849
@ericapisani
ericapisani requested a review from a team as a code owner July 20, 2026 18:57
@linear-code

linear-code Bot commented Jul 20, 2026

Copy link
Copy Markdown

PY-2617

Comment thread sentry_sdk/client.py Outdated
@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Codecov Results 📊

92197 passed | ⏭️ 6304 skipped | Total: 98501 | Pass Rate: 93.6% | Execution Time: 319m 3s

📊 Comparison with Base Branch

Metric Change
Total Tests 📈 +285
Passed Tests 📈 +284
Failed Tests 📉 -1
Skipped Tests 📈 +2

➖ Removed Tests (1)

View removed tests
  • test_circular_references
    • File: tests.tracing.test_misc

All tests are passing successfully.

✅ Patch coverage is 100.00%. Project has 2484 uncovered lines.
❌ Project coverage is 89.66%. Comparing base (base) to head (head).

Coverage diff
@@            Coverage Diff             @@
##          main       #PR       +/-##
==========================================
- Coverage    89.68%    89.66%    -0.02%
==========================================
  Files          193       193         —
  Lines        24007     24030       +23
  Branches      8342      8348        +6
==========================================
+ Hits         21529     21546       +17
- Misses        2478      2484        +6
- Partials      1387      1386        -1

Generated by Codecov Action

@alexander-alderman-webb alexander-alderman-webb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice !

@sentrivana sentrivana left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a couple places where the behavior or tests need some tweaking.

Re: dropping/keeping the metrics/logs/spans that went through a faulty before_send_*: might need more alignment on this. Spans likely shouldn't be dropped, but metrics and logs we might want to drop. There's a convo on slack about this.

Comment thread sentry_sdk/tracing_utils.py Outdated
sample_rate = propagation_context.parent_sampled
else:
sample_rate = client.options["traces_sample_rate"]
sample_rate = _get_fallback_sample_rate(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd change the name of the helper function since in the context of this else branch, it's not really a fallback.

Maybe something like _get_effective_sample_rate?

Comment thread tests/tracing/test_sampling.py Outdated
Comment on lines +275 to +277
# parent said "don't sample", but its propagated sample rate of 0.75
# combined with sample_rand 0.5 should win over both the flag and
# traces_sample_rate=0.0

@sentrivana sentrivana Jul 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually not correct. If the parent sample decision is 0, that should take precedence over anything else.

Can we remove this test? I think I know where this needs fixing (it's not something you've introduced in this PR), and I can follow up with a fix after we've merged this. (I expect some other tests to break so don't want to tack this onto your PR.)

Comment thread tests/tracing/test_sampling.py Outdated
Comment on lines +281 to +283
_experiments={
"trace_lifecycle": "stream",
},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
_experiments={
"trace_lifecycle": "stream",
},
trace_lifecycle="stream",

Comment thread tests/tracing/test_sampling.py Outdated
Comment on lines +256 to +258
_experiments={
"trace_lifecycle": "stream",
},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
_experiments={
"trace_lifecycle": "stream",
},
trace_lifecycle="stream",

Comment thread tests/test_logs.py Outdated
Comment on lines +184 to +188
# The exception in before_send_log is swallowed and the original,
# unmodified log is sent.
assert len(logs) == 1
assert logs[0]["body"] == "This is an error log..."
assert logs[0]["attributes"]["sentry.severity_text"] == "error"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should drop the log in this case

Comment thread tests/test_metrics.py Outdated
Comment on lines +266 to +270
# The exception in before_send_metric is swallowed and the original,
# unmodified metric is sent.
metrics = [item.payload for item in items]
assert len(metrics) == 1
assert metrics[0]["name"] == "test.keep"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here -- the metric should not be sent

"traces_sample_rate,expected_decision",
[(0.0, False), (0.25, False), (0.75, True), (1.00, True)],
)
def test_traces_sampler_raising_falls_back_to_traces_sample_rate_span_streaming(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also add a span streaming test case with no incoming trace propagation headers/no continue_trace and check that we fall back to traces_sample_rate?

assert span.sampled is traces_sampler_return_value


@pytest.mark.parametrize(

@sentrivana sentrivana Jul 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[not related to this specific line or testcase] One case I'm not quite sure of: there is no incoming trace, traces_sampler is defined and throws, and there is no traces_sample_rate defined. We should have a test case (or two: streaming and non streaming) for that too.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will add a couple, where the assertion is that the span/transaction isn't sampled

@sentrivana sentrivana left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving to not block -- as long as the comments have been addressed feel free to merge 🙏🏻

Comment thread sentry_sdk/client.py Outdated
Comment thread sentry_sdk/client.py Outdated
Comment thread sentry_sdk/tracing_utils.py
Comment thread sentry_sdk/client.py
Comment on lines +1249 to +1253
try:
serialized = before_send(serialized, {}) # type: ignore[arg-type]
except Exception:
exception_raised_in_before_send_func = True
raise

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The capture_internal_exceptions context manager suppresses BaseException subclasses like KeyboardInterrupt, preventing process termination when raised in before_send callbacks.
Severity: CRITICAL

Suggested Fix

Modify the __exit__ method of CaptureInternalException to not suppress BaseException subclasses. It should check if the exception type ty is a subclass of Exception before suppressing it, allowing critical signals like KeyboardInterrupt and SystemExit to propagate and terminate the process as expected.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: sentry_sdk/client.py#L1249-L1253

Potential issue: The `capture_internal_exceptions` context manager, used to wrap the
`before_send` callback, has an `__exit__` method that always returns `True`. This
behavior suppresses all exceptions that propagate to it. While a `try...except
Exception` block handles standard exceptions, `BaseException` subclasses like
`KeyboardInterrupt` or `SystemExit` are not caught by this block. Instead, they are
silently swallowed by `capture_internal_exceptions`, preventing the process from
terminating as expected. This is a regression that can make services unresponsive to
termination signals like Ctrl+C.

Also affects:

  • sentry_sdk/utils.py:208~223

The TYPE_CHECKING import of BaseClient from sentry_sdk.client in
tracing_utils.py created a circular import when Sphinx evaluated type
annotations during the API docs build (sphinx_autodoc_typehints evaluates
TYPE_CHECKING imports at runtime). A cold 'import sentry_sdk.client' then
cycled through scope -> tracing -> tracing_utils -> client ->
integrations.dedupe -> partially-initialized scope.

Annotate the client parameter as Any instead.
Comment thread sentry_sdk/tracing_utils.py
Comment thread sentry_sdk/client.py
Comment thread sentry_sdk/tracing_utils.py
Reset the serialized span to its original value when before_send_span
raises, so partial mutations made by the callback before the exception are
discarded and the unmodified span is sent.

Strengthen the raise test to mutate the span before raising, asserting the
mutation does not leak into the sent span.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit b89171c. Configure here.

Comment thread sentry_sdk/tracing.py
Comment thread sentry_sdk/client.py
Comment on lines +1258 to +1260
# functions.
if exception_raised_in_before_send_func:
return

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The try...except Exception block for before_send callbacks doesn't handle BaseException. This causes logs/metrics to be sent instead of dropped when a BaseException is raised.
Severity: LOW

Suggested Fix

Modify the exception handling to catch BaseException instead of just Exception. This will ensure that the exception_raised_in_before_send_func flag is correctly set for all types of exceptions, causing the log or metric to be dropped as intended.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: sentry_sdk/client.py#L1258-L1260

Potential issue: The exception handling for `before_send` callbacks for logs and metrics
is incomplete. The code uses a `try...except Exception` block to set a flag,
`exception_raised_in_before_send_func`, which determines if the log/metric should be
dropped. However, this does not catch `BaseException` subclasses (e.g.,
`KeyboardInterrupt`). When a `BaseException` is raised in a user's `before_send_log` or
`before_send_metric` function, the flag is not set. Consequently, the log or metric is
sent to Sentry instead of being dropped, which contradicts the intended behavior of
dropping events when the callback fails.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants