Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The active test run was aborted. Reason: Test host process crashed #451

Open
3 tasks
fanyang-mono opened this issue Jul 25, 2023 · 8 comments
Open
3 tasks

Comments

@fanyang-mono
Copy link
Member

fanyang-mono commented Jul 25, 2023

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=351450

Build leg reported

chrome-DebuggerTests.EvaluateOnCallFrameTests.WorkItemExecution

Pull Request

dotnet/runtime#89217

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The active test run was aborted. Reason: Test host process crashed",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

  • Feature changes/additions
  • Bug fixes
  • Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Report

Build Definition Step Name Console log Pull Request
2625793 dotnet-interactive Test / Blame Log
924837 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#76852
2625489 dotnet-interactive Test / Blame Log
2625493 dotnet-interactive Test / Blame Log
923681 dotnet/roslyn Run Integration Tests Log
923659 dotnet/roslyn Run Integration Tests Log
921574 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#75581
2620937 dotnet-interactive Test / Blame Log
2620694 dotnet-interactive Test / Blame Log
917698 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#76752
2619106 dotnet-interactive Test / Blame Log
914993 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#76725
914308 dotnet/roslyn Run Integration Tests Log
914177 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#76692
913987 dotnet/roslyn Run Integration Tests Log
912714 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#76695
2616802 dotnet-interactive Test / Blame Log
2616717 dotnet-interactive Test / Blame Log
2615694 dotnet-fsharp CmdLine@2 Log
2615469 dotnet-interactive Test / Blame Log
2614768 dotnet-interactive Test / Blame Log
909122 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#75970
2614588 dotnet-fsharp CmdLine@2 Log
908618 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#76649
908555 dotnet/roslyn Run Integration Tests Log
908425 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#76646
2614116 dotnet-interactive Test / Blame Log
2614101 dotnet-interactive Test / Blame Log
2613990 dotnet-interactive Test / Blame Log
2613544 dotnet-interactive Test / Blame Log
2612493 dotnet-interactive Test / Blame Log
906308 dotnet/roslyn Run Integration Tests Log
2612433 dotnet-interactive Test / Blame Log
2612333 dotnet-interactive Test / Blame Log
905800 dotnet/roslyn Run Integration Tests Log
2611980 dotnet-interactive Test / Blame Log
2611872 dotnet-interactive Test / Blame Log
2611807 dotnet-interactive Test / Blame Log
905108 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#70455
905102 dotnet/roslyn Run Integration Tests Log
903012 dotnet/roslyn Run Integration Tests Log
Build Definition Test Pull Request
925579 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46186
925697 dotnet/sdk msbuild.Integration.Tests.dll.WorkItemExecution dotnet/sdk#46164
925672 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#45932
925669 dotnet/sdk Microsoft.NET.Publish.Tests.dll.8.WorkItemExecution dotnet/sdk#46171
925655 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46219
925467 dotnet/sdk Microsoft.NET.Build.Tests.dll.23.WorkItemExecution dotnet/sdk#46215
925445 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.14.WorkItemExecution dotnet/sdk#46214
925439 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.14.WorkItemExecution dotnet/sdk#46108
925397 dotnet/sdk Microsoft.NET.Build.Tests.dll.23.WorkItemExecution dotnet/sdk#45932
925430 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46213
925341 dotnet/sdk Microsoft.NET.Sdk.BlazorWebAssembly.Tests.dll.4.WorkItemExecution dotnet/sdk#46211
925350 dotnet/sdk Microsoft.NET.Build.Tests.dll.23.WorkItemExecution dotnet/sdk#45932
925270 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46209
925259 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.14.WorkItemExecution dotnet/sdk#45362
925289 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#46164
925295 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46045
925308 dotnet/aspire Aspire.Hosting.Elasticsearch.Tests.WorkItemExecution dotnet/aspire#7206
925257 dotnet/aspnetcore Microsoft.AspNetCore.Mvc.FunctionalTests--net10.0.WorkItemExecution
925292 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46210
925223 dotnet/aspnetcore Microsoft.AspNetCore.Mvc.FunctionalTests--net10.0.WorkItemExecution
925046 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.20.WorkItemExecution dotnet/sdk#46183
925058 dotnet/sdk Microsoft.NET.Publish.Tests.dll.8.WorkItemExecution dotnet/sdk#45362
924839 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46198
924808 dotnet/sdk Microsoft.NET.Publish.Tests.dll.8.WorkItemExecution dotnet/sdk#46193
925026 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46184
924803 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.14.WorkItemExecution dotnet/sdk#46191
924794 dotnet/sdk Microsoft.NET.Publish.Tests.dll.10.WorkItemExecution dotnet/sdk#46190
924881 dotnet/aspire Aspire.Hosting.Azure.Tests.AzureCosmosDBEmulatorFunctionalTests.Aspire.Hosting.Azure.Tests.AzureCosmosDBEmulatorFunctionalTests.VerifyWaitForOnCosmosDBEmulatorBlocksDependentResources(usePreview: False) dotnet/aspire#7163
924843 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#46200
924723 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46164
924733 dotnet/aspnetcore Microsoft.AspNetCore.Http.Extensions.Tests--net10.0.WorkItemExecution dotnet/aspnetcore#53395
924703 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#46180
919402 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.10.WorkItemExecution dotnet/sdk#46033
924083 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.14.WorkItemExecution dotnet/sdk#46168
2625568 dotnet-roslyn Microsoft.CodeAnalysis.VisualBasic.Features.UnitTests_Microsoft.CodeAnalysis.VisualBasic.ExpressionEvaluator.ResultProvider.UnitTests_Microsoft.CodeAn..._246.WorkItemExecution
924379 dotnet/sdk dotnet-watch.Tests.dll.6.WorkItemExecution dotnet/sdk#46141
924465 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.10.WorkItemExecution dotnet/sdk#46179
924554 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46171
924389 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46177
924466 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#46100
924419 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#46178
924302 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46108
924106 dotnet/sdk Microsoft.NET.Build.Tests.dll.23.WorkItemExecution dotnet/sdk#45362
923505 dotnet/sdk Microsoft.NET.Sdk.BlazorWebAssembly.Tests.dll.4.WorkItemExecution dotnet/sdk#46002
918610 dotnet/sdk Microsoft.NET.Build.Tests.dll.23.WorkItemExecution dotnet/sdk#45883
923901 dotnet/sdk Microsoft.NET.Publish.Tests.dll.10.WorkItemExecution dotnet/sdk#46164
924003 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46055
923981 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#46129
923301 dotnet/sdk Microsoft.NET.Publish.Tests.dll.8.WorkItemExecution dotnet/sdk#46142
923874 dotnet/aspnetcore InMemory.FunctionalTests--net10.0.WorkItemExecution
923698 dotnet/aspnetcore Microsoft.AspNetCore.Mvc.FunctionalTests--net10.0.WorkItemExecution
923746 dotnet/aspnetcore Microsoft.AspNetCore.OutputCaching.Tests--net8.0.WorkItemExecution
923740 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46132
923662 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#46158
923704 dotnet/sdk Microsoft.NET.Publish.Tests.dll.8.WorkItemExecution dotnet/sdk#46160
923643 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46157
923563 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.10.WorkItemExecution dotnet/sdk#46146
923562 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.10.WorkItemExecution dotnet/sdk#46145
923607 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#46155
923127 dotnet/sdk Microsoft.NET.Build.Tests.dll.23.WorkItemExecution dotnet/sdk#46136
2624779 dotnet-aspnetcore Microsoft.AspNetCore.Mvc.FunctionalTests--net10.0.WorkItemExecution
923345 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.10.WorkItemExecution dotnet/sdk#46141
923343 dotnet/sdk Microsoft.NET.Build.Tests.dll.23.WorkItemExecution dotnet/sdk#46137
923312 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46143
923196 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.13.WorkItemExecution dotnet/sdk#46139
923205 dotnet/sdk dotnet-watch.Tests.dll.6.WorkItemExecution dotnet/sdk#46141
922434 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.10.WorkItemExecution dotnet/sdk#45997
923136 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.14.WorkItemExecution dotnet/sdk#46137
923117 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46132
922949 dotnet/aspnetcore Microsoft.AspNetCore.Authentication.Test--net10.0.WorkItemExecution
922941 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46102
922964 dotnet/aspnetcore Microsoft.AspNetCore.Identity.FunctionalTests--net10.0.WorkItemExecution dotnet/aspnetcore#59961
2624486 dotnet-aspnetcore Microsoft.AspNetCore.Identity.EntityFrameworkCore.Test--net10.0.WorkItemExecution
922915 dotnet/aspnetcore Microsoft.AspNetCore.Http.Extensions.Tests--net10.0.WorkItemExecution dotnet/aspnetcore#59959
922680 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.20.WorkItemExecution dotnet/sdk#45983
922832 dotnet/aspire Aspire.Hosting.Redis.Tests.RedisFunctionalTests.Aspire.Hosting.Redis.Tests.RedisFunctionalTests.RedisInsightWithDataShouldPersistStateBetweenUsages(useVolume: False) dotnet/aspire#7168
922797 dotnet/aspire Aspire.Hosting.Azure.Tests.AzureEventHubsExtensionsTests.Aspire.Hosting.Azure.Tests.AzureEventHubsExtensionsTests.VerifyWaitForOnEventHubsEmulatorBlocksDependentResources dotnet/aspire#7163
922574 dotnet/sdk Microsoft.NET.Build.Tests.dll.23.WorkItemExecution dotnet/sdk#46125
922648 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46126
922481 dotnet/sdk Microsoft.NET.Build.Tests.dll.23.WorkItemExecution dotnet/sdk#46124
922193 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46123
922430 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.13.WorkItemExecution dotnet/sdk#46100
922442 dotnet/aspnetcore Microsoft.AspNetCore.Identity.EntityFrameworkCore.Test--net10.0.WorkItemExecution
922383 dotnet/aspire Aspire.Hosting.Redis.Tests.RedisFunctionalTests.Aspire.Hosting.Redis.Tests.RedisFunctionalTests.RedisInsightWithDataShouldPersistStateBetweenUsages(useVolume: False)
920657 dotnet/sdk Microsoft.NET.Build.Tests.dll.23.WorkItemExecution dotnet/sdk#45980
922247 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46110
922194 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46117
922156 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#45932
922116 dotnet/sdk Microsoft.NET.Publish.Tests.dll.8.WorkItemExecution dotnet/sdk#44611
922036 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46122
921985 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#44611
2623549 dotnet-aspnetcore Microsoft.AspNetCore.Identity.EntityFrameworkCore.InMemory.Test--net10.0.WorkItemExecution
921854 dotnet/sdk Microsoft.NET.Publish.Tests.dll.10.WorkItemExecution dotnet/sdk#46099
921824 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.14.WorkItemExecution dotnet/sdk#45990
921762 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.10.WorkItemExecution dotnet/sdk#46112
921806 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.LegacyStaticWebAssetsV1IntegrationTest.Microsoft.NET.Sdk.Razor.Tests.LegacyStaticWebAssetsV1IntegrationTest.BuildProjectWithReferences_WorksWithStaticWebAssetsV1ClassLibraries dotnet/sdk#46021
921796 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46113
921691 dotnet/sdk Microsoft.NET.Sdk.Razor.Tests.dll.11.WorkItemExecution dotnet/sdk#46110
921547 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#45880
921685 dotnet/aspire Aspire.Hosting.Elasticsearch.Tests.WorkItemExecution dotnet/aspire#7136
Displaying 100 of 431 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
45 167 472

Known issue validation

Build: 🔎
Result validation: ⚠️ Validation could not be done without an Azure DevOps build URL on the issue. Please add it to the "Build: 🔎" line.
Validation performed at: 2/8/2024 9:24:30 PM UTC

@andriipatsula
Copy link
Member

Hello @fanyang-mono, could you please update the "ErrorMessage" : "" by following the step by step documentation on how to create a known issue

@fanyang-mono
Copy link
Member Author

Updated.

@pavelsavara
Copy link
Member

pavelsavara commented Nov 7, 2023

This is duplicate of dotnet/runtime#89409, except here it's not only wasm debugger

@dougbu
Copy link
Member

dougbu commented Nov 23, 2023

@pavelsavara and @fanyang-mono, should this issue be closed in favour of dotnet/runtime#94560❔ if not, the error message isn't sufficiently precise to understand what (if anything) we (DNCEng) could do about it.

@pavelsavara
Copy link
Member

@pavelsavara and @fanyang-mono, should this issue be closed in favour of

You mean dotnet/runtime#89409

I think we should keep this one, because this one has some hits which are not just wasm debugger related.
Here we see other repo failures, on roslyn and aspnetcore CI.

if not, the error message isn't sufficiently precise to understand what (if anything) we (DNCEng) could do about it.

I have not investigated it myself yet. But got questions which could help to narrow it down

  • are there crash dump files on the machine ?
  • are there some hints on what's going on in the log files before crash ?
  • is this OOM or stack overflow in disguise ?
  • did we run out of disk space or number of inodes ?
  • hit some limit in docker container ?
  • is this always the same OS, version, image ?
  • could this be hardware issue ?

@fanyang-mono
Copy link
Member Author

This issue was originally created to track issues happened to wasm debugger tests (belonging to dotnet/runtime repo). However, the error message was generic enough to include issues from other repos, such as dotnet/roslyn and dotnet/aspnetcore. We need insights from owners from dotnet/roslyn and dotnet/aspnetcore to confirm that they are aware of these issue. And if they are okay to close this issue as well. We could close then.

@mitchdenny
Copy link
Member

We just got an instance of this in the dotnet/aspire repo. It wasn't a WASM related failure for us. Just looks like the test process crashed and took a few tests with it.

@nagilson
Copy link
Member

nagilson commented Dec 3, 2024

@marcpopMSFT was working on getting the test infrastructure to generate crash dumps when it times out to help with investigation. I think it was generating the dumps but not yet correctly uploading them to the artifacts.

We are waiting to hear from the infra team on an issue with this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants