Make MSQ tests run faster #17718
Make MSQ tests run faster #17718
Conversation
github-actions
bot
added
Area - Batch Ingestion
Area - MSQ
There was a problem hiding this comment.
Seems fine, I'll approve this since it's an improvement. Please do look at the line comments though and fix the spelling errors.
Also, even though this patch is fine, I do want to note that ideally we shouldn't even use MSQWorkerTaskLauncher at all in SQL tests. The ideal setup would be a test-specific WorkerManager for those. MSQWorkerTaskLauncher itself could be tested in more targeted tests (i.e. testing it directly, not using it as part of the SQL setup).
...ore/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQWorkerTaskLauncher.java
Outdated
Show resolved
Hide resolved
...ore/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQWorkerTaskLauncher.java
Outdated
Show resolved
Hide resolved
...ore/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQWorkerTaskLauncher.java
Outdated
Show resolved
Hide resolved
| @@ -353,13 +354,19 @@ public WorkerManager newWorkerManager( | |||
| WorkerFailureListener workerFailureListener | |||
| ) | |||
| { | |||
| MSQWorkerTaskLauncherConfig taskLauncherConfig = new MSQWorkerTaskLauncherConfig(); | |||
| taskLauncherConfig.highFrequenceCheckMillis = 0; | |||
There was a problem hiding this comment.
0 means busy loop, which seems excessive; how about 1?
There was a problem hiding this comment.
setting it to 1 adds 1s for ~500 cases; it would be nice if it would execute a sleep(0) instead of skipping the sleep entirely
set it to 1 for now
most of the time msq tests are not even using the cpu - they are just waiting...
the following are changed to increase their speed to be comparable to the native engine's tests:
MSQWorkerTaskLauncherwas wasting a lot of time waiting a100msbetween checksthis have enabled to execute
*Calcite*in the msq module in52sinstead of the usual6 min 32 sec