-
Notifications
You must be signed in to change notification settings - Fork 426
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(span): handle errors on spans when a custom exception overrides __str__ #12307
Conversation
…xception overrides __str__.
|
Datadog ReportBranch report: ✅ 0 Failed, 130 Passed, 1184 Skipped, 3m 30.82s Total duration (25m 39.64s time saved) |
BenchmarksBenchmark execution time: 2025-02-13 15:08:16 Comparing candidate commit 556ed73 in PR branch Found 1 performance improvements and 1 performance regressions! Performance is the same for 392 metrics, 2 unstable metrics. scenario:iast_aspects-ljust_aspect
scenario:iast_aspects-ospathdirname_aspect
|
The backport to
To backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.21 2.21
# Navigate to the new working tree
cd .worktrees/backport-2.21
# Create a new branch
git switch --create backport-12307-to-2.21
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 bada1cd7974515cc63a033b9d2e7361dedaef998
# Push it to GitHub
git push --set-upstream origin backport-12307-to-2.21
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.21 Then, create a pull request where the |
…_str__ (#12307) Today when a custom exception overrides `__str__` by raising an exception, the tracer cannot apply an error message on it because `str(exc_val)` doesn't work (it raises the exception again). As a result the span doesn't close correctly because it's an exception, not a string and thus the span can't get sent. If this condition occurs, this PR sets the error message to the class name of the custom exception so the spans can still get sent and there's a clue to what the exception was. The current logic is this: https://github.com/DataDog/dd-trace-py/blob/6b5f13c2701d55a695afcd4a73db8af2d251b55d/ddtrace/_trace/span.py#L523 When we run the current logic against a custom exception that overrides `__str__`, we can see from the [failed job](https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-py/-/jobs/805643750) that `str(exc_val)` leads to an error: ``` _____________________ test_set_exc_info_with_str_override ______________________ def test_set_exc_info_with_str_override(): span = Span("span") class CustomException(Exception): def __str__(self): raise Exception("A custom exception") try: > raise CustomException() E tests.tracer.test_span.test_set_exc_info_with_str_override.<locals>.CustomException: <exception str() failed> tests/tracer/test_span.py:839: CustomException During handling of the above exception, another exception occurred: def test_set_exc_info_with_str_override(): span = Span("span") class CustomException(Exception): def __str__(self): raise Exception("A custom exception") try: raise CustomException() except Exception: type_, value_, traceback_ = sys.exc_info() > span.set_exc_info(type_, value_, traceback_) tests/tracer/test_span.py:842: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ddtrace/_trace/span.py:523: in set_exc_info self._meta[ERROR_MSG] = str(exc_val) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = CustomException() def __str__(self): > raise Exception("A custom exception") E Exception: A custom exception tests/tracer/test_span.py:836: Exception ``` The fix is to try to catch this exception if `str(exc_val)` and just add the class name as a placeholder for the error message since there isn't an easy way to turn the exception into a string (since it just keeps raising the exception). This way, spans can at least be closed and sent. Thanks to @wiggzz for demonstrating the issue in https://github.com/wiggzz/dd-trace-debug! This is for APMS-14834 . - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) (cherry picked from commit bada1cd)
…_str__ (#12307) [backport 2.21] (#12326) Backport of #12307 to 2.21 Today when a custom exception overrides `__str__` by raising an exception, the tracer cannot apply an error message on it because `str(exc_val)` doesn't work (it raises the exception again). As a result the span doesn't close correctly because it's an exception, not a string and thus the span can't get sent. If this condition occurs, this PR sets the error message to the class name of the custom exception so the spans can still get sent and there's a clue to what the exception was. The current logic is this: https://github.com/DataDog/dd-trace-py/blob/6b5f13c2701d55a695afcd4a73db8af2d251b55d/ddtrace/_trace/span.py#L523 When we run the current logic against a custom exception that overrides `__str__`, we can see from the [failed job](https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-py/-/jobs/805643750) that `str(exc_val)` leads to an error: ``` _____________________ test_set_exc_info_with_str_override ______________________ def test_set_exc_info_with_str_override(): span = Span("span") class CustomException(Exception): def __str__(self): raise Exception("A custom exception") try: > raise CustomException() E tests.tracer.test_span.test_set_exc_info_with_str_override.<locals>.CustomException: <exception str() failed> tests/tracer/test_span.py:839: CustomException During handling of the above exception, another exception occurred: def test_set_exc_info_with_str_override(): span = Span("span") class CustomException(Exception): def __str__(self): raise Exception("A custom exception") try: raise CustomException() except Exception: type_, value_, traceback_ = sys.exc_info() > span.set_exc_info(type_, value_, traceback_) tests/tracer/test_span.py:842: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ddtrace/_trace/span.py:523: in set_exc_info self._meta[ERROR_MSG] = str(exc_val) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = CustomException() def __str__(self): > raise Exception("A custom exception") E Exception: A custom exception tests/tracer/test_span.py:836: Exception ``` The fix is to try to catch this exception if `str(exc_val)` and just add the class name as a placeholder for the error message since there isn't an easy way to turn the exception into a string (since it just keeps raising the exception). This way, spans can at least be closed and sent. Thanks to @wiggzz for demonstrating the issue in https://github.com/wiggzz/dd-trace-debug! This is for APMS-14834 . (cherry picked from commit bada1cd) ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
Today when a custom exception overrides
__str__
by raising an exception, the tracer cannot apply an error message on it becausestr(exc_val)
doesn't work (it raises the exception again). As a result the span doesn't close correctly because it's an exception, not a string and thus the span can't get sent.If this condition occurs, this PR sets the error message to the class name of the custom exception so the spans can still get sent and there's a clue to what the exception was.
The current logic is this:
dd-trace-py/ddtrace/_trace/span.py
Line 523 in 6b5f13c
When we run the current logic against a custom exception that overrides
__str__
, we can see from the failed job thatstr(exc_val)
leads to an error:The fix is to try to catch this exception if
str(exc_val)
and just add the class name as a placeholder for the error message since there isn't an easy way to turn the exception into a string (since it just keeps raising the exception). This way, spans can at least be closed and sent.Thanks to @wiggzz for demonstrating the issue in https://github.com/wiggzz/dd-trace-debug!
This is for APMS-14834 .
Checklist
Reviewer Checklist