Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace all test method exclusions with test class exclusions #80

Open
Bouncheck opened this issue Feb 3, 2025 · 3 comments
Open

Replace all test method exclusions with test class exclusions #80

Bouncheck opened this issue Feb 3, 2025 · 3 comments
Assignees
@Bouncheck
Labels
bug Something isn't working

Comments

@Bouncheck
Copy link
Contributor

Bouncheck commented Feb 3, 2025 •
edited
Loading

Likely due to a bug in surefire plugin, we cannot at the same time define specific test groups to be run and exclude some of the test methods (Specifically through the use of args -Dtest.groups='groupA,groupB' and -Dtest='!Class#excludedMethod'). This leads to an error as can be seen for example in https://jenkins.scylladb.com/job/scylla-master/job/driver-tests/job/java-driver-matrix-test/1561/consoleFull
For some reason when excluding a test method, the surefire plugin will also run methods from groups other than those specified. Additionally if such method has @AfterClass defined in its class, the runner will crash.

Starting com.datastax.driver.core.HeapCompressionTest.should_function_with_lz4_compression [Test #140]...
00:52:23      62416  [main] INFO  com.datastax.driver.core.HostConnectionPool - Using advanced port-based shard awareness with /127.0.1.1:38003
00:52:23  SUCCESS: should_function_with_lz4_compression
00:52:23  Test   : 00:00:00
00:52:23  Elapsed: 00:01:02
00:52:23  
00:52:23  [ERROR] l should not be null
00:52:29  [INFO] 
00:52:29  [INFO] Results:
00:52:29  [INFO] 
00:52:29  [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
00:52:29  [INFO] 
00:52:29  [INFO] ------------------------------------------------------------------------
00:52:29  [INFO] BUILD FAILURE
00:52:29  [INFO] ------------------------------------------------------------------------
00:52:29  [INFO] Total time:  01:34 min
00:52:29  [INFO] Finished at: 2025-01-31T23:52:28Z
00:52:29  [INFO] ------------------------------------------------------------------------
00:52:29  [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M6:test (default-test) on project scylla-driver-core: 
00:52:29  [ERROR] 
00:52:29  [ERROR] Please refer to /jenkins/workspace/scylla-master/driver-tests/java-driver-matrix-test/scylla-java-driver/driver-core/target/surefire-reports for the individual test results.
00:52:29  [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
00:52:29  [ERROR] There was an error in the forked process
00:52:29  [ERROR] l should not be null
00:52:29  [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process
00:52:29  [ERROR] l should not be null
00:52:29  [ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:699)
00:52:29  [ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:311)
00:52:29  [ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:268)
00:52:29  [ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1332)
00:52:29  [ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1165)
00:52:29  [ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:929)
00:52:29  [ERROR] 	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
00:52:29  [ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
00:52:29  [ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
00:52:29  [ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
00:52:29  [ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
00:52:29  [ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
00:52:29  [ERROR] 	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
00:52:29  [ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
00:52:29  [ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
00:52:29  [ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
00:52:29  [ERROR] 	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
00:52:29  [ERROR] 	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:956)
00:52:29  [ERROR] 	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
00:52:29  [ERROR] 	at org.apache.maven.cli.MavenCli.main(MavenCli.java:192)
00:52:29  [ERROR] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
00:52:29  [ERROR] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
00:52:29  [ERROR] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
00:52:29  [ERROR] 	at java.lang.reflect.Method.invoke(Method.java:498)
00:52:29  [ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
00:52:29  [ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
00:52:29  [ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
00:52:29  [ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
00:52:29  [ERROR] -> [Help 1]
00:52:29  [ERROR] 
00:52:29  [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
00:52:29  [ERROR] Re-run Maven using the -X switch to enable full debug logging.

Longer stacktrace from local run with -X switch:

[ERROR] There was an error in the forked process
[ERROR] l should not be null
[ERROR] java.lang.AssertionError: l should not be null
[ERROR] 	at org.testng.ClassMethodMap.removeAndCheckIfLast(ClassMethodMap.java:58)
[ERROR] 	at org.testng.internal.TestMethodWorker.invokeAfterClassMethods(TestMethodWorker.java:208)
[ERROR] 	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:114)
[ERROR] 	at org.testng.TestRunner.privateRun(TestRunner.java:767)
[ERROR] 	at org.testng.TestRunner.run(TestRunner.java:617)
[ERROR] 	at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
[ERROR] 	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343)
[ERROR] 	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:305)
[ERROR] 	at org.testng.SuiteRunner.run(SuiteRunner.java:254)
[ERROR] 	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
[ERROR] 	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
[ERROR] 	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
[ERROR] 	at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
[ERROR] 	at org.testng.TestNG.run(TestNG.java:1057)
[ERROR] 	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:151)
[ERROR] 	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:111)
[ERROR] 	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:98)
[ERROR] 	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:155)
[ERROR] 	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:456)
[ERROR] 	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:169)
[ERROR] 	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:595)
[ERROR] 	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:581)
[ERROR] 
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process
[ERROR] l should not be null
[ERROR] java.lang.AssertionError: l should not be null
[ERROR] 	at org.testng.ClassMethodMap.removeAndCheckIfLast(ClassMethodMap.java:58)
[ERROR] 	at org.testng.internal.TestMethodWorker.invokeAfterClassMethods(TestMethodWorker.java:208)
[ERROR] 	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:114)
[ERROR] 	at org.testng.TestRunner.privateRun(TestRunner.java:767)
[ERROR] 	at org.testng.TestRunner.run(TestRunner.java:617)
[ERROR] 	at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
[ERROR] 	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343)
[ERROR] 	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:305)
[ERROR] 	at org.testng.SuiteRunner.run(SuiteRunner.java:254)
[ERROR] 	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
[ERROR] 	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
[ERROR] 	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
[ERROR] 	at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
[ERROR] 	at org.testng.TestNG.run(TestNG.java:1057)
[ERROR] 	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:151)
[ERROR] 	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:111)
[ERROR] 	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:98)
[ERROR] 	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:155)
[ERROR] 	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:456)
[ERROR] 	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:169)
[ERROR] 	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:595)
[ERROR] 	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:581)
[ERROR] 
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:699)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:311)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:268)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1332)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1165)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:929)
[ERROR] 	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:370)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:351)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:215)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:171)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:163)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
[ERROR] 	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
[ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:294)
[ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
[ERROR] 	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
[ERROR] 	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:960)
[ERROR] 	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:293)
[ERROR] 	at org.apache.maven.cli.MavenCli.main(MavenCli.java:196)
[ERROR] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] 	at java.lang.reflect.Method.invoke(Method.java:498)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
[ERROR]

I need to double check the reproducer and if it's definitely a bug in surefire but for the java-driver-matrix the bottom line is that we have to remove method level exclusions and exclude whole classes. If it would skip an important method previously not excluded then if there is no other workaround the methods that are to be excluded can be just removed on a patch level.
@Bouncheck Bouncheck added the bug Something isn't working label Feb 3, 2025
@Bouncheck Bouncheck self-assigned this Feb 3, 2025
@Bouncheck
Copy link
Contributor Author

Bouncheck commented Feb 3, 2025 •
edited
Loading

This also happens with surefire version 3.5.2 so this is not solvable by upgrading the surefire dependency. This could also be something on testNG end.

@Bouncheck
Copy link
Contributor Author

This behavior does not reproduce with scylla java driver 3.10.2.1

@Bouncheck
Copy link
Contributor Author

Upgrading testNG to 7.5.1 prevents the crash, but does not fix the problem with running groups that were not specified.

Bouncheck added a commit to Bouncheck/scylla-java-driver-matrix that referenced this issue Feb 6, 2025
@Bouncheck
Skip whole TabletMetadataTest class instead of method
260d8f2 
Replaces method level exclusions from ignore.yaml for versions:
3.11.4.0, 3.11.5.3, 3.11.5.4
Temporary solution for issue scylladb#80 that should unblock testing.
Whitespace was added before `#` symbols specifying methods for easier
reenablement once cause is resolved.
@Bouncheck Bouncheck mentioned this issue Feb 6, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Bouncheck Bouncheck

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Bouncheck