Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(span): handle errors on spans when a custom exception overrides __str__ #12307

Merged
merged 6 commits into from
Feb 13, 2025

Conversation

wantsui
Copy link
Collaborator

@wantsui wantsui commented Feb 12, 2025

Today when a custom exception overrides __str__ by raising an exception, the tracer cannot apply an error message on it because str(exc_val) doesn't work (it raises the exception again). As a result the span doesn't close correctly because it's an exception, not a string and thus the span can't get sent.

If this condition occurs, this PR sets the error message to the class name of the custom exception so the spans can still get sent and there's a clue to what the exception was.

The current logic is this:

self._meta[ERROR_MSG] = str(exc_val)

When we run the current logic against a custom exception that overrides __str__, we can see from the failed job that str(exc_val) leads to an error:

_____________________ test_set_exc_info_with_str_override ______________________
    def test_set_exc_info_with_str_override():
        span = Span("span")
    
        class CustomException(Exception):
            def __str__(self):
                raise Exception("A custom exception")
    
        try:
>           raise CustomException()
E           tests.tracer.test_span.test_set_exc_info_with_str_override.<locals>.CustomException: <exception str() failed>
tests/tracer/test_span.py:839: CustomException
During handling of the above exception, another exception occurred:
    def test_set_exc_info_with_str_override():
        span = Span("span")
    
        class CustomException(Exception):
            def __str__(self):
                raise Exception("A custom exception")
    
        try:
            raise CustomException()
        except Exception:
            type_, value_, traceback_ = sys.exc_info()
>           span.set_exc_info(type_, value_, traceback_)
tests/tracer/test_span.py:842: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
ddtrace/_trace/span.py:523: in set_exc_info
    self._meta[ERROR_MSG] = str(exc_val)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = CustomException()
    def __str__(self):
>       raise Exception("A custom exception")
E       Exception: A custom exception
tests/tracer/test_span.py:836: Exception

The fix is to try to catch this exception if str(exc_val) and just add the class name as a placeholder for the error message since there isn't an easy way to turn the exception into a string (since it just keeps raising the exception). This way, spans can at least be closed and sent.

Thanks to @wiggzz for demonstrating the issue in https://github.com/wiggzz/dd-trace-debug!

This is for APMS-14834 .

Checklist

  • PR author has checked that all the criteria below are met
  • The PR description includes an overview of the change
  • The PR description articulates the motivation for the change
  • The change includes tests OR the PR description describes a testing strategy
  • The PR description notes risks associated with the change, if any
  • Newly-added code is easy to change
  • The change follows the library release note guidelines
  • The change includes or references documentation updates if necessary
  • Backport labels are set (if applicable)

Reviewer Checklist

  • Reviewer has checked that all the criteria below are met
  • Title is accurate
  • All changes are related to the pull request's stated goal
  • Avoids breaking API changes
  • Testing strategy adequately addresses listed risks
  • Newly-added code is easy to change
  • Release note makes sense to a user of the library
  • If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
  • Backport labels are set in a manner that is consistent with the release branch maintenance policy

Copy link
Contributor

github-actions bot commented Feb 12, 2025

CODEOWNERS have been resolved as:

releasenotes/notes/fix-custom-error-spans-with-__str__-override-a05726045f8751d6.yaml  @DataDog/apm-python
ddtrace/_trace/span.py                                                  @DataDog/apm-sdk-api-python
tests/tracer/test_span.py                                               @DataDog/apm-sdk-api-python

@wantsui wantsui changed the title handle errors on spans when a custom exception overrides __str__ fix(span):handle errors on spans when a custom exception overrides __str__ Feb 12, 2025
@wantsui wantsui changed the title fix(span):handle errors on spans when a custom exception overrides __str__ fix(span): handle errors on spans when a custom exception overrides __str__ Feb 12, 2025
@datadog-dd-trace-py-rkomorn
Copy link

datadog-dd-trace-py-rkomorn bot commented Feb 12, 2025

Datadog Report

Branch report: wantsui/capture-raise-exceptions-in-__str__
Commit report: 556ed73
Test service: dd-trace-py

✅ 0 Failed, 130 Passed, 1184 Skipped, 3m 30.82s Total duration (25m 39.64s time saved)

@wantsui wantsui marked this pull request as ready for review February 12, 2025 20:33
@wantsui wantsui requested a review from a team as a code owner February 12, 2025 20:33
@wantsui wantsui requested a review from ZStriker19 February 12, 2025 20:33
@pr-commenter
Copy link

pr-commenter bot commented Feb 12, 2025

Benchmarks

Benchmark execution time: 2025-02-13 15:08:16

Comparing candidate commit 556ed73 in PR branch wantsui/capture-raise-exceptions-in-__str__ with baseline commit 47f3bb2 in branch main.

Found 1 performance improvements and 1 performance regressions! Performance is the same for 392 metrics, 2 unstable metrics.

scenario:iast_aspects-ljust_aspect

  • 🟥 execution_time [+1.053µs; +1.156µs] or [+7.436%; +8.162%]

scenario:iast_aspects-ospathdirname_aspect

  • 🟩 execution_time [-651.355ns; -592.588ns] or [-15.118%; -13.754%]

@wantsui wantsui requested a review from a team as a code owner February 12, 2025 20:49
@wantsui wantsui enabled auto-merge (squash) February 13, 2025 14:23
@wantsui wantsui merged commit bada1cd into main Feb 13, 2025
691 checks passed
@wantsui wantsui deleted the wantsui/capture-raise-exceptions-in-__str__ branch February 13, 2025 15:09
Copy link
Contributor

The backport to 2.21 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.21 2.21
# Navigate to the new working tree
cd .worktrees/backport-2.21
# Create a new branch
git switch --create backport-12307-to-2.21
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 bada1cd7974515cc63a033b9d2e7361dedaef998
# Push it to GitHub
git push --set-upstream origin backport-12307-to-2.21
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.21

Then, create a pull request where the base branch is 2.21 and the compare/head branch is backport-12307-to-2.21.

wantsui added a commit that referenced this pull request Feb 13, 2025
…_str__ (#12307)

Today when a custom exception overrides `__str__` by raising an
exception, the tracer cannot apply an error message on it because
`str(exc_val)` doesn't work (it raises the exception again). As a result
the span doesn't close correctly because it's an exception, not a string
and thus the span can't get sent.

If this condition occurs, this PR sets the error message to the class
name of the custom exception so the spans can still get sent and there's
a clue to what the exception was.

The current logic is this:

https://github.com/DataDog/dd-trace-py/blob/6b5f13c2701d55a695afcd4a73db8af2d251b55d/ddtrace/_trace/span.py#L523

When we run the current logic against a custom exception that overrides
`__str__`, we can see from the [failed
job](https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-py/-/jobs/805643750)
that `str(exc_val)` leads to an error:
```
_____________________ test_set_exc_info_with_str_override ______________________
    def test_set_exc_info_with_str_override():
        span = Span("span")

        class CustomException(Exception):
            def __str__(self):
                raise Exception("A custom exception")

        try:
>           raise CustomException()
E           tests.tracer.test_span.test_set_exc_info_with_str_override.<locals>.CustomException: <exception str() failed>
tests/tracer/test_span.py:839: CustomException
During handling of the above exception, another exception occurred:
    def test_set_exc_info_with_str_override():
        span = Span("span")

        class CustomException(Exception):
            def __str__(self):
                raise Exception("A custom exception")

        try:
            raise CustomException()
        except Exception:
            type_, value_, traceback_ = sys.exc_info()
>           span.set_exc_info(type_, value_, traceback_)
tests/tracer/test_span.py:842:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ddtrace/_trace/span.py:523: in set_exc_info
    self._meta[ERROR_MSG] = str(exc_val)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = CustomException()
    def __str__(self):
>       raise Exception("A custom exception")
E       Exception: A custom exception
tests/tracer/test_span.py:836: Exception
```

The fix is to try to catch this exception if `str(exc_val)` and just add
the class name as a placeholder for the error message since there isn't
an easy way to turn the exception into a string (since it just keeps
raising the exception). This way, spans can at least be closed and sent.

Thanks to @wiggzz for demonstrating the issue in
https://github.com/wiggzz/dd-trace-debug!

This is for APMS-14834 .

- [x] PR author has checked that all the criteria below are met
- The PR description includes an overview of the change
- The PR description articulates the motivation for the change
- The change includes tests OR the PR description describes a testing
strategy
- The PR description notes risks associated with the change, if any
- Newly-added code is easy to change
- The change follows the [library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
- The change includes or references documentation updates if necessary
- Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))

- [x] Reviewer has checked that all the criteria below are met
- Title is accurate
- All changes are related to the pull request's stated goal
- Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- Testing strategy adequately addresses listed risks
- Newly-added code is easy to change
- Release note makes sense to a user of the library
- If necessary, author has acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment
- Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

(cherry picked from commit bada1cd)
wantsui added a commit that referenced this pull request Feb 13, 2025
…_str__ (#12307) [backport 2.21] (#12326)

Backport of #12307 to 2.21


Today when a custom exception overrides `__str__` by raising an
exception, the tracer cannot apply an error message on it because
`str(exc_val)` doesn't work (it raises the exception again). As a result
the span doesn't close correctly because it's an exception, not a string
and thus the span can't get sent.

If this condition occurs, this PR sets the error message to the class
name of the custom exception so the spans can still get sent and there's
a clue to what the exception was.

The current logic is this:


https://github.com/DataDog/dd-trace-py/blob/6b5f13c2701d55a695afcd4a73db8af2d251b55d/ddtrace/_trace/span.py#L523

When we run the current logic against a custom exception that overrides
`__str__`, we can see from the [failed

job](https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-py/-/jobs/805643750)
that `str(exc_val)` leads to an error:
```
_____________________ test_set_exc_info_with_str_override ______________________
    def test_set_exc_info_with_str_override():
        span = Span("span")

        class CustomException(Exception):
            def __str__(self):
                raise Exception("A custom exception")

        try:
>           raise CustomException()
E           tests.tracer.test_span.test_set_exc_info_with_str_override.<locals>.CustomException: <exception str() failed>
tests/tracer/test_span.py:839: CustomException
During handling of the above exception, another exception occurred:
    def test_set_exc_info_with_str_override():
        span = Span("span")

        class CustomException(Exception):
            def __str__(self):
                raise Exception("A custom exception")

        try:
            raise CustomException()
        except Exception:
            type_, value_, traceback_ = sys.exc_info()
>           span.set_exc_info(type_, value_, traceback_)
tests/tracer/test_span.py:842:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ddtrace/_trace/span.py:523: in set_exc_info
    self._meta[ERROR_MSG] = str(exc_val)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = CustomException()
    def __str__(self):
>       raise Exception("A custom exception")
E       Exception: A custom exception
tests/tracer/test_span.py:836: Exception
```

The fix is to try to catch this exception if `str(exc_val)` and just add
the class name as a placeholder for the error message since there isn't
an easy way to turn the exception into a string (since it just keeps
raising the exception). This way, spans can at least be closed and sent.

Thanks to @wiggzz for demonstrating the issue in
https://github.com/wiggzz/dd-trace-debug!

This is for APMS-14834 .

(cherry picked from commit bada1cd)

## Checklist
- [x] PR author has checked that all the criteria below are met
- The PR description includes an overview of the change
- The PR description articulates the motivation for the change
- The change includes tests OR the PR description describes a testing
strategy
- The PR description notes risks associated with the change, if any
- Newly-added code is easy to change
- The change follows the [library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
- The change includes or references documentation updates if necessary
- Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))

## Reviewer Checklist
- [x] Reviewer has checked that all the criteria below are met 
- Title is accurate
- All changes are related to the pull request's stated goal
- Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- Testing strategy adequately addresses listed risks
- Newly-added code is easy to change
- Release note makes sense to a user of the library
- If necessary, author has acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment
- Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants