Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry for ingestion Resources #356

Merged
merged 22 commits into from
Jan 25, 2024

TC change as per new sdk change

674b6a9
Select commit
Loading
Failed to load commit list.
Merged

Retry for ingestion Resources #356

TC change as per new sdk change
674b6a9
Select commit
Loading
Failed to load commit list.
Azure Pipelines / azure-kusto-spark ci failed Jan 24, 2024 in 6m 9s

Build #86259645 had test failures

1 errors / 4 warnings

Details

Tests

  • Failed: 4 (5.33%)
  • Passed: 71 (94.67%)
  • Other: 0 (0.00%)
  • Total: 75

Annotations

Check failure on line 22842 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / azure-kusto-spark ci

Build log #L22842 

Build failed.

Check failure on line 1 in KustoConnector should write to a kusto table and read it back in default mode

See this annotation in the file changed.

@azure-pipelines azure-pipelines / azure-kusto-spark ci

KustoConnector should write to a kusto table and read it back in default mode 

Error acquiring ApplicationAccessToken
Raw output 
      com.microsoft.azure.kusto.data.exceptions.DataServiceException: Error acquiring ApplicationAccessToken
      at com.microsoft.azure.kusto.data.auth.ConfidentialAppTokenProviderBase.acquireNewAccessToken(ConfidentialAppTokenProviderBase.java:52)
      at com.microsoft.azure.kusto.data.instrumentation.MonitoredActivity.invoke(MonitoredActivity.java:33)
      at com.microsoft.azure.kusto.data.instrumentation.MonitoredActivity.invoke(MonitoredActivity.java:26)
      at com.microsoft.azure.kusto.data.auth.MsalTokenProviderBase.acquireAccessTokenImpl(MsalTokenProviderBase.java:62)
      at com.microsoft.azure.kusto.data.instrumentation.MonitoredActivity.invoke(MonitoredActivity.java:33)
      at com.microsoft.azure.kusto.data.auth.TokenProviderBase.acquireAccessToken(TokenProviderBase.java:32)
      at com.microsoft.azure.kusto.data.ClientImpl.generateIngestAndCommandHeaders(ClientImpl.java:405)
      at com.microsoft.azure.kusto.data.ClientImpl.executeToJsonResult(ClientImpl.java:213)
      at com.microsoft.azure.kusto.data.ClientImpl.executeImpl(ClientImpl.java:173)
      at com.microsoft.azure.kusto.data.ClientImpl.lambda$execute$0(ClientImpl.java:122)
      at com.microsoft.azure.kusto.data.instrumentation.MonitoredActivity.invoke(MonitoredActivity.java:33)
      at com.microsoft.azure.kusto.data.ClientImpl.execute(ClientImpl.java:121)
      at com.microsoft.azure.kusto.data.ClientImpl.execute(ClientImpl.java:116)
      at com.microsoft.kusto.spark.utils.ExtendedKustoClient.$anonfun$executeEngine$1(ExtendedKustoClient.scala:388)
      at com.microsoft.kusto.spark.utils.KustoDataSourceUtils$$anon$2.apply(KustoDataSourceUtils.scala:398)
      at io.github.resilience4j.retry.Retry.lambda$decorateCheckedSupplier$3f69f149$1(Retry.java:137)
      at io.github.resilience4j.retry.Retry.executeCheckedSupplier(Retry.java:419)
      at com.microsoft.kusto.spark.utils.KustoDataSourceUtils$.retryApplyFunction(KustoDataSourceUtils.scala:401)
      at com.microsoft.kusto.spark.utils.ExtendedKustoClient.executeEngine(ExtendedKustoClient.scala:389)
      at com.microsoft.kusto.spark.datasink.KustoWriter$.write(KustoWriter.scala:66)
      at com.microsoft.kusto.spark.datasource.DefaultSource.createRelation(DefaultSource.scala:49)
      at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)
      at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
      at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
      at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90)
      at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:175)
      at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:213)
      at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
      at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:210)
      at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:171)
      at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:122)
      at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:121)
      at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:963)
      at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:100)
      at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160)
      at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:87)
      at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
      at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
      at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:963)
      at org.apache.

Check failure on line 1 in KustoSource should execute a read query on Kusto cluster in single mode

See this annotation in the file changed.

@azure-pipelines azure-pipelines / azure-kusto-spark ci

KustoSource should execute a read query on Kusto cluster in single mode 

Error acquiring ApplicationAccessToken
Raw output 
      com.microsoft.azure.kusto.data.exceptions.DataServiceException: Error acquiring ApplicationAccessToken
      at com.microsoft.azure.kusto.data.auth.ConfidentialAppTokenProviderBase.acquireNewAccessToken(ConfidentialAppTokenProviderBase.java:52)
      at com.microsoft.azure.kusto.data.instrumentation.MonitoredActivity.invoke(MonitoredActivity.java:33)
      at com.microsoft.azure.kusto.data.instrumentation.MonitoredActivity.invoke(MonitoredActivity.java:26)
      at com.microsoft.azure.kusto.data.auth.MsalTokenProviderBase.acquireAccessTokenImpl(MsalTokenProviderBase.java:62)
      at com.microsoft.azure.kusto.data.instrumentation.MonitoredActivity.invoke(MonitoredActivity.java:33)
      at com.microsoft.azure.kusto.data.auth.TokenProviderBase.acquireAccessToken(TokenProviderBase.java:32)
      at com.microsoft.azure.kusto.data.ClientImpl.generateIngestAndCommandHeaders(ClientImpl.java:405)
      at com.microsoft.azure.kusto.data.ClientImpl.executeToJsonResult(ClientImpl.java:213)
      at com.microsoft.azure.kusto.data.ClientImpl.executeImpl(ClientImpl.java:173)
      at com.microsoft.azure.kusto.data.ClientImpl.lambda$execute$0(ClientImpl.java:122)
      at com.microsoft.azure.kusto.data.instrumentation.MonitoredActivity.invoke(MonitoredActivity.java:33)
      at com.microsoft.azure.kusto.data.ClientImpl.execute(ClientImpl.java:121)
      at com.microsoft.azure.kusto.data.ClientImpl.execute(ClientImpl.java:116)
      at com.microsoft.kusto.spark.utils.ExtendedKustoClient.$anonfun$executeEngine$1(ExtendedKustoClient.scala:388)
      at com.microsoft.kusto.spark.utils.KustoDataSourceUtils$$anon$2.apply(KustoDataSourceUtils.scala:398)
      at io.github.resilience4j.retry.Retry.lambda$decorateCheckedSupplier$3f69f149$1(Retry.java:137)
      at io.github.resilience4j.retry.Retry.executeCheckedSupplier(Retry.java:419)
      at com.microsoft.kusto.spark.utils.KustoDataSourceUtils$.retryApplyFunction(KustoDataSourceUtils.scala:401)
      at com.microsoft.kusto.spark.utils.ExtendedKustoClient.executeEngine(ExtendedKustoClient.scala:389)
      at com.microsoft.kusto.spark.utils.KustoDataSourceUtils$.getSchema(KustoDataSourceUtils.scala:175)
      at com.microsoft.kusto.spark.datasource.KustoRelation.getSchema(KustoRelation.scala:145)
      at com.microsoft.kusto.spark.datasource.KustoRelation.schema(KustoRelation.scala:43)
      at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:438)
      at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:297)
      at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:286)
      at scala.Option.getOrElse(Option.scala:189)
      at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:286)
      at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:221)
      at com.microsoft.kusto.spark.sql.extension.SparkExtension$DataFrameReaderExtension.kusto(SparkExtension.scala:22)
      at com.microsoft.kusto.spark.KustoSourceE2E.validateRead(KustoSourceE2E.scala:146)
      at com.microsoft.kusto.spark.KustoSourceE2E.$anonfun$new$2(KustoSourceE2E.scala:158)
      at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      at org.scalatest.Transformer.apply(Transformer.scala:22)
      at org.scalatest.Transformer.apply(Transformer.scala:20)
      at org.scalatest.flatspec.AnyFlatSpecLike$$anon$5.apply(AnyFlatSpecLike.scala:1832)
      at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
      at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
      at org.scalatest.flatspec.AnyFlatSpec.withFixture(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.invokeWithFixture$1(AnyFlatSpecLike.scala:1830)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTest$1(AnyFlatSpecLike.scala:1842)
      at org.scalatest.SuperEngine.runTest

Check failure on line 1 in KustoSource should execute a read query on Kusto cluster in distributed mode

See this annotation in the file changed.

@azure-pipelines azure-pipelines / azure-kusto-spark ci

KustoSource should execute a read query on Kusto cluster in distributed mode 

Error acquiring ApplicationAccessToken
Raw output 
      com.microsoft.azure.kusto.data.exceptions.DataServiceException: Error acquiring ApplicationAccessToken
      at com.microsoft.azure.kusto.data.auth.ConfidentialAppTokenProviderBase.acquireNewAccessToken(ConfidentialAppTokenProviderBase.java:52)
      at com.microsoft.azure.kusto.data.instrumentation.MonitoredActivity.invoke(MonitoredActivity.java:33)
      at com.microsoft.azure.kusto.data.instrumentation.MonitoredActivity.invoke(MonitoredActivity.java:26)
      at com.microsoft.azure.kusto.data.auth.MsalTokenProviderBase.acquireAccessTokenImpl(MsalTokenProviderBase.java:62)
      at com.microsoft.azure.kusto.data.instrumentation.MonitoredActivity.invoke(MonitoredActivity.java:33)
      at com.microsoft.azure.kusto.data.auth.TokenProviderBase.acquireAccessToken(TokenProviderBase.java:32)
      at com.microsoft.azure.kusto.data.ClientImpl.generateIngestAndCommandHeaders(ClientImpl.java:405)
      at com.microsoft.azure.kusto.data.ClientImpl.executeToJsonResult(ClientImpl.java:213)
      at com.microsoft.azure.kusto.data.ClientImpl.executeImpl(ClientImpl.java:173)
      at com.microsoft.azure.kusto.data.ClientImpl.lambda$execute$0(ClientImpl.java:122)
      at com.microsoft.azure.kusto.data.instrumentation.MonitoredActivity.invoke(MonitoredActivity.java:33)
      at com.microsoft.azure.kusto.data.ClientImpl.execute(ClientImpl.java:121)
      at com.microsoft.azure.kusto.data.ClientImpl.execute(ClientImpl.java:116)
      at com.microsoft.azure.kusto.data.ClientImpl.execute(ClientImpl.java:111)
      at com.microsoft.kusto.spark.KustoSourceE2E.$anonfun$new$3(KustoSourceE2E.scala:165)
      at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      at org.scalatest.Transformer.apply(Transformer.scala:22)
      at org.scalatest.Transformer.apply(Transformer.scala:20)
      at org.scalatest.flatspec.AnyFlatSpecLike$$anon$5.apply(AnyFlatSpecLike.scala:1832)
      at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
      at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
      at org.scalatest.flatspec.AnyFlatSpec.withFixture(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.invokeWithFixture$1(AnyFlatSpecLike.scala:1830)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTest$1(AnyFlatSpecLike.scala:1842)
      at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTest(AnyFlatSpecLike.scala:1842)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTest$(AnyFlatSpecLike.scala:1824)
      at org.scalatest.flatspec.AnyFlatSpec.runTest(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTests$1(AnyFlatSpecLike.scala:1900)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
      at scala.collection.immutable.List.foreach(List.scala:431)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:390)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:427)
      at scala.collection.immutable.List.foreach(List.scala:431)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
      at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTests(AnyFlatSpecLike.scala:1900)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTests$(AnyFlatSpecLike.scala:1899)
      at org.scalatest.flatspec.AnyFlatSpec.runTests(AnyFlatSpec.scala:1686)
      at org.scalatest.Suite.run(Suite.scala:1114)
      at org.scalatest.Suite.run$(Suite.scala:1096)
      at org.scalatest.flatspec.AnyFlatSpec.org$scalatest$flatspec$AnyFlatSpecLike$$super$run(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$

Check failure on line 1 in testUpdateAndGetPrevNativeAzureFs

See this annotation in the file changed.

@azure-pipelines azure-pipelines / azure-kusto-spark ci

testUpdateAndGetPrevNativeAzureFs 

TestFailedException was thrown during property evaluation. (KustoAzureFsSetupCacheTest.scala:45)   Message: false did not equal true Initial set is false, refresh is needed   Location: (KustoAzureFsSetupCacheTest.scala:47)   Occurred at table row 0 (zero based, not counting headings), which had values (     now = 2024-01-24T13:35:30.770Z,     checkIfRefreshNeeded = true,     Scenario = Initial set is false, refresh is needed   )
Raw output 
      org.scalatest.exceptions.TableDrivenPropertyCheckFailedException: TestFailedException was thrown during property evaluation. (KustoAzureFsSetupCacheTest.scala:45)
  Message: false did not equal true Initial set is false, refresh is needed
  Location: (KustoAzureFsSetupCacheTest.scala:47)
  Occurred at table row 0 (zero based, not counting headings), which had values (
    now = 2024-01-24T13:35:30.770Z,
    checkIfRefreshNeeded = true,
    Scenario = Initial set is false, refresh is needed
  )
      at org.scalatest.enablers.TableAsserting$$anon$2.indicateFailure(TableAsserting.scala:5397)
      at org.scalatest.enablers.TableAsserting$$anon$2.indicateFailure(TableAsserting.scala:5385)
      at org.scalatest.enablers.UnitTableAsserting$TableAssertingImpl.$anonfun$forAll$12(TableAsserting.scala:592)
      at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:985)
      at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
      at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
      at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
      at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:984)
      at org.scalatest.enablers.UnitTableAsserting$TableAssertingImpl.forAll(TableAsserting.scala:554)
      at org.scalatest.prop.TableFor3.apply(TableFor1.scala:566)
      at org.scalatest.prop.TableDrivenPropertyChecks.forAll(TableDrivenPropertyChecks.scala:423)
      at org.scalatest.prop.TableDrivenPropertyChecks.forAll$(TableDrivenPropertyChecks.scala:422)
      at org.scalatest.prop.TableDrivenPropertyChecks$.forAll(TableDrivenPropertyChecks.scala:1343)
      at com.microsoft.kusto.spark.utils.KustoAzureFsSetupCacheTest.$anonfun$new$3(KustoAzureFsSetupCacheTest.scala:45)
      at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      at org.scalatest.Transformer.apply(Transformer.scala:22)
      at org.scalatest.Transformer.apply(Transformer.scala:20)
      at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
      at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
      at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
      at org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1564)
      at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
      at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
      at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
      at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
      at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
      at org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1564)
      at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
      at scala.collection.immutable.List.foreach(List.scala:431)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
      at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
      at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
      at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
      at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
      at org.scalatest.Suite.run(Suite.scala:1114)
      at org.scalatest.Suite.run$(Suite.scala:1096)
      at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
      at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
      at org.scalatest.SuperEngine.runImpl(Engi
View more details on Azure Pipelines