[BUG] "Workspace capacity assignment in progress" while refreshing existing Fabric workspace resource

[fstraetz]

🐛 What happened?

Sometimes (as in, the problem seems to occur nondeterministic), the Terraform Fabric provider seems to have a problem refreshing the state of existing Fabric workspaces. Terraform / OpenTofu will then log the message "Workspace capacity assignment in progress" until it a timeout occurs. After this happens, the "diagnostic detail" will be "context deadline exceeded".

This seems to happen randomly, even if no changes whatsoever (so neither Terraform / OpenTofu nor GUI) have been made to the workspaces in question.

I am pretty sure this has nothing to do with network problems (on our side), as all other Azure specific requests seem to work fine. Usually, OpenTofu even displays a correct execution plan. It also should have nothing to do with "advanced stuff" like using resource "fabric_workspace_role_assignment" or resource "fabric_workspace_git", as the problem occurs and disappears no matter whether these resources are linked to the respective workspaces or not.

This bug is basically the same as #76, which got closed after the problem (also randomly) disappeared for the user in question.

Especially the nondeterministic nature of this problem makes the Fabric provider (I know, I know, it is in beta state) almost unusable.

🔬 How to reproduce?

1. Query the data of a Fabric capacity resource using data "fabric_capacity".
2. Use this data to create a resource "fabric_workspace".
3. Maybe add some resource "fabric_workspace_role_assignment" resources.
4. Maybe add a resource "fabric_workspace_git" resource.
5. Execute terraform plan / tofu plan multiple times at random times until it fails.
6. Wait for a while.
7. Execute terraform plan / tofu plan and it might work again.

🏗️ Code Sample / Log

module.reporting[0].fabric_workspace.fws_mrcp_reporting: Refreshing state... [id=xxxxxxxx-537e-4508-9119-b302ac7079a1]
2025-02-03T10:51:15.625Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: read timeout configuration not found, using provided default: tf_resource_type=fabric_workspace tf_rpc=ReadResource @caller=github.com/hashicorp/[email protected]/resource/timeouts/timeouts.go:139 @module=fabric tf_provider_addr=registry.terraform.io/microsoft/fabric tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 timestamp=2025-02-03T10:51:15.625Z
2025-02-03T10:51:15.695Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: tf_provider_addr=registry.terraform.io/microsoft/fabric tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 @module=fabric tf_resource_type=fabric_workspace tf_rpc=ReadResource @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 timestamp=2025-02-03T10:51:15.695Z
2025-02-03T10:51:45.762Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: tf_rpc=ReadResource @module=fabric tf_provider_addr=registry.terraform.io/microsoft/fabric tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 tf_resource_type=fabric_workspace @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 timestamp=2025-02-03T10:51:45.762Z
2025-02-03T10:52:15.835Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: tf_rpc=ReadResource @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 @module=fabric tf_provider_addr=registry.terraform.io/microsoft/fabric tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 tf_resource_type=fabric_workspace timestamp=2025-02-03T10:52:15.834Z
2025-02-03T10:52:45.904Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: @module=fabric tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 tf_resource_type=fabric_workspace tf_provider_addr=registry.terraform.io/microsoft/fabric tf_rpc=ReadResource @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 timestamp=2025-02-03T10:52:45.903Z
2025-02-03T10:53:15.970Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 @module=fabric tf_provider_addr=registry.terraform.io/microsoft/fabric tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 tf_resource_type=fabric_workspace tf_rpc=ReadResource timestamp=2025-02-03T10:53:15.970Z
2025-02-03T10:53:46.029Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 tf_provider_addr=registry.terraform.io/microsoft/fabric tf_resource_type=fabric_workspace tf_rpc=ReadResource @module=fabric tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 timestamp=2025-02-03T10:53:46.028Z
2025-02-03T10:54:16.097Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 tf_rpc=ReadResource @module=fabric tf_provider_addr=registry.terraform.io/microsoft/fabric tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 tf_resource_type=fabric_workspace timestamp=2025-02-03T10:54:16.096Z
2025-02-03T10:54:46.158Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: tf_rpc=ReadResource @module=fabric tf_provider_addr=registry.terraform.io/microsoft/fabric tf_resource_type=fabric_workspace @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 timestamp=2025-02-03T10:54:46.158Z
2025-02-03T10:55:16.215Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 tf_resource_type=fabric_workspace tf_rpc=ReadResource tf_provider_addr=registry.terraform.io/microsoft/fabric @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 @module=fabric timestamp=2025-02-03T10:55:16.215Z
2025-02-03T10:55:46.284Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 @module=fabric tf_provider_addr=registry.terraform.io/microsoft/fabric tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 tf_resource_type=fabric_workspace tf_rpc=ReadResource timestamp=2025-02-03T10:55:46.283Z
2025-02-03T10:56:16.356Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 @module=fabric tf_provider_addr=registry.terraform.io/microsoft/fabric tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 tf_resource_type=fabric_workspace tf_rpc=ReadResource timestamp=2025-02-03T10:56:16.356Z
2025-02-03T10:56:46.429Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: tf_resource_type=fabric_workspace tf_rpc=ReadResource tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 @module=fabric tf_provider_addr=registry.terraform.io/microsoft/fabric timestamp=2025-02-03T10:56:46.429Z
2025-02-03T10:57:16.537Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 tf_resource_type=fabric_workspace tf_rpc=ReadResource @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 @module=fabric tf_provider_addr=registry.terraform.io/microsoft/fabric timestamp=2025-02-03T10:57:16.536Z
2025-02-03T10:57:46.631Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 @module=fabric tf_provider_addr=registry.terraform.io/microsoft/fabric tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 tf_resource_type=fabric_workspace tf_rpc=ReadResource timestamp=2025-02-03T10:57:46.631Z
2025-02-03T10:58:16.695Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: tf_resource_type=fabric_workspace @module=fabric tf_provider_addr=registry.terraform.io/microsoft/fabric @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 tf_rpc=ReadResource timestamp=2025-02-03T10:58:16.695Z
2025-02-03T10:58:46.763Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: @module=fabric tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 tf_provider_addr=registry.terraform.io/microsoft/fabric tf_resource_type=fabric_workspace tf_rpc=ReadResource timestamp=2025-02-03T10:58:46.763Z
2025-02-03T10:59:16.817Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: @module=fabric tf_resource_type=fabric_workspace @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 tf_provider_addr=registry.terraform.io/microsoft/fabric tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 tf_rpc=ReadResource timestamp=2025-02-03T10:59:16.816Z
2025-02-03T10:59:46.901Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: @module=fabric tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 tf_provider_addr=registry.terraform.io/microsoft/fabric tf_resource_type=fabric_workspace tf_rpc=ReadResource timestamp=2025-02-03T10:59:46.901Z
2025-02-03T11:00:16.992Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 tf_resource_type=fabric_workspace tf_rpc=ReadResource @module=fabric tf_provider_addr=registry.terraform.io/microsoft/fabric @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 timestamp=2025-02-03T11:00:16.991Z
2025-02-03T11:00:47.081Z [INFO]  provider.terraform-provider-fabric_v0.1.0-beta.8: Workspace capacity assignment in progress, waiting 30 seconds before retrying: tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 tf_resource_type=fabric_workspace tf_rpc=ReadResource @caller=github.com/microsoft/terraform-provider-fabric/internal/services/workspace/resource_workspace.go:605 tf_provider_addr=registry.terraform.io/microsoft/fabric @module=fabric timestamp=2025-02-03T11:00:47.080Z
2025-02-03T11:01:17.086Z [ERROR] provider.terraform-provider-fabric_v0.1.0-beta.8: Response contains error diagnostic: diagnostic_detail="context deadline exceeded" diagnostic_summary="unknown error" tf_req_id=f6bd25b7-8e41-ff2c-d8a3-e73c3a3efae9 @module=sdk.proto diagnostic_severity=ERROR tf_proto_version=6.8 tf_provider_addr=registry.terraform.io/microsoft/fabric tf_resource_type=fabric_workspace tf_rpc=ReadResource @caller=github.com/hashicorp/[email protected]/tfprotov6/internal/diag/diagnostics.go:58 timestamp=2025-02-03T11:01:17.085Z
2025-02-03T11:01:17.087Z [ERROR] vertex "module.reporting[0].fabric_workspace.fws_mrcp_reporting" error: unknown error
2025-02-03T11:01:17.087Z [ERROR] vertex "module.reporting.fabric_workspace.fws_mrcp_reporting (expand)" error: unknown error

📷 Screenshots

No response

📈 Expected behavior

The terraform plan / tofu plan call should run without hangups or errors (assuming all the definitions are correct, of course).

🌌 Environment (Provider Version)

0.1.0-beta.8

🌌 Environment (Terraform Version)

OpenTofu v1.8.8

🌌 Environment (OS)

Linux

📎 Additional context

No response

🔰 Code of Conduct

• I agree to follow this project's Code of Conduct.

[DariuszPorowski]

Hi @fstraetz

Is there a chance that your solution sends more than 50 API requests in 1 minute? (we can assume that each object for a resource or data-source is on average ~2x requests)

The Provider works on top of public Fabric API (https://learn.microsoft.com/en-us/rest/api/fabric/) which has a limit of 50 req/min per principal before the throttling starts (https://learn.microsoft.com/en-us/rest/api/fabric/articles/throttling)

The provider handles throttling and waits based on "Retry-After" value, but it can significantly extend the total execution time and exceed the default timeout value.

[fstraetz]

Thanks for you response @DariuszPorowski

If your estimate of 2 requests per (data) resource is correct, our solution should have quite a bit of leeway. I just checked and concerning the Fabric provider we currently have:

• 2 x data "fabric_capacity"
• 2 x resource "fabric_workspace"
• 7 x resource "fabric_workspace_role_assignment"
• 2 x resource "fabric_workspace_git"
• 3 x resource "fabric_lakehouse"

So, about 32 requests. (My assumption here is that resources of type "fabric_workspace_role_assignment" with principal type "Group" still count as a single resource when it comes to the number of requests, no matter the number of security principals in the given group.)

Taking a look at the log, it seems that the "Workspace capacity assignment in progress" issue blocks for about 10 minutes (more likely longer, as the timeout definitely ends the OpenTofu process / log, but maybe not the issue). So my expectation would be that, even if the throttling of the Fabric API was the culprit, the retrying (which seems to happen regularly, according to the log?) should be successful at some point during these 10 minutes. Unless the Retry-After header can be > 10 Minutes and the retrying which can be seen in the the log is showing something different, independent from the Retry-After mechanism.

Also, considering the message "Workspace capacity assignment in progress", I would not think of this being a throttling issue (though I also have no clue about how all of this is implemented behind the scenes). And it sometimes blocks for hours (like the day before yesterday) and sometimes it works absolutely fine (like yesterday and today when I did some smaller changes). I would expect the throttling to behave in a deterministic manner.

[DariuszPorowski]

Hi @fstraetz

Your assumptions are valid as long as only one API operation is executed at a time. By default, Terraform prepares the execution graph with 10 concurrent operations. You may consider increasing provider timeouts and additionally preventing too many concurrent operations by Terraform. For more information, please refer to: https://registry.terraform.io/providers/microsoft/fabric/latest/docs/guides/troubleshooting#operations-take-too-long-to-complete-or-timeout

Please note, we do not officially support OpenTofu. All diagnostics and troubleshooting are performed exclusively with Terraform binaries.