An invalid asynchronous invocation was detected

[jmelosegui]

We’ve recently encountered an issue in our dev environment while running Azure Durable Functions. After some investigation, it seems to be directly related to the Datadog Application Services (AAS) extension that we are using to monitor our functions.

With the extension enabled, our Durable Function orchestrators exhibit errors related to non-deterministic behavior, which causes the orchestrator to fail.
Here is the error message we are seeing:

The orchestrator function completed on a non-orchestrator thread!
"@i":"3192603f",
"@l":"Error",
"@x":"System.InvalidOperationException: An invalid asynchronous invocation was detected. This can be caused by awaiting non-durable tasks in an orchestrator function's implementation or by middleware that invokes asynchronous code.

When we removed the Datadog extension, the Durable Functions performed as expected without any errors or failures. This leads us to believe that the extension might be introducing some asynchronous behavior or thread context switching that conflicts with the deterministic execution requirements of Durable Functions.

Are there any known workarounds or best practices to ensure the Datadog extension can work alongside Durable Functions without impacting their deterministic behavior?
Any configuration or adjustments we could make to minimize the interference?

Thanks in advance.

[bouwkast]

Are there any known workarounds or best practices to ensure the Datadog extension can work alongside Durable Functions without impacting their deterministic behavior?

Unfortunately, at the moment we don't support Durable Functions so I don't think there are any workarounds for the async/thread issues here.
Submitting a feature request or a bug report for this will be the best route at the moment.

Feel free to open either a bug report or feature request here: https://help.datadoghq.com/hc/requests/new?tf_1260824651490=pt_product_type:apm&tf_1900004146284=pt_apm_language:.net&tf_1260825272270=pt_apm_category_feature_request

[RobARichardson]

@jmelosegui about a year ago, my team and I lost about 2 weeks trying to figure out why this was happening with our Durable Functions before figuring out it was the DD Extension. We had to abandon DD and use Application Insights.

[jmelosegui]

@RobARichardson, that is what we are doing for now.
We are waiting for this request to be implemented one day.

Thanks for sharing!

[jcstorms1]

Hey @jmelosegui, thanks for opening this. Unfortunately, we don't currently support durable functions at this time, but this is definitely a feature request we are aware of. I've marked this as an enhancement and will leave the issue open.