feat: add support for ADO audit stream resources

[ksh5022]

All Submissions:

• Have you added an explanation of what your changes do and why you'd like us to include them?
• I have updated the documentation accordingly.
• I have added tests to cover my changes.
• All new and existing tests passed.
• My code follows the code style of this project.
• I ran lint checks locally prior to submission.
• Have you checked to ensure there aren't other open PRs for the same update/change?

What about the current behavior has changed?

Add new resources: azuredevops_auditstream_azureeventgrid, azuredevops_auditstream_azuremonitorlogs, and azuredevops_auditstream_splunk. These will create an audit stream at the Organization level to forward logs to.

Issue Number: #363

Does this introduce a change to go.mod, go.sum or vendor/?

• Yes
• No

reason: add audit package from ado go library.

Does this introduce a breaking change?

• Yes
• No

Any relevant logs, error output, etc?

• TestAccAuditStreamAzureEventGrid_CreateAndUpdate
• TestAccAuditStreamAzureEventGrid_CreateDisabled
• TestAccAuditStreamAzureMonitorLogs_CreateAndUpdate
• TestAccAuditStreamAzureMonitorLogs_CreateDisabled
• TestAccAuditStreamSplunk_CreateAndUpdate
• TestAccAuditStreamSplunk_CreateDisabled

Other information

Modeled after serviceendpoint and policy for commons implementation, and agent_pool for resource id of type int.

[ghost]

All CLA requirements met.

[xuzhang3]

@ksh5022 A few changes requested.

1. The import operation looks broken the sensitive datashared_key, access_key etc. import->plan -> apply -> plan , you will find that the diff always exists.
2. Can add more test to cover the import scenario. Ref: https://github.com/microsoft/terraform-provider-azuredevops/blob/main/azuredevops/internal/acceptancetests/resource_serviceendpoint_bitbucket_test.go
3. Also the test case, can you provide the acctest result of the three new resources?
4. Since all the audit stream can be enable/disable, can we add this for the new resources? API: https://docs.microsoft.com/en-us/rest/api/azure/devops/audit/streams/update-status?view=azure-devops-rest-6.0
5. A bit concern about the property days_to_backfill, where can I update this config from UI web page.

[ksh5022]

4. Since all the audit stream can be enable/disable, can we add this for the new resources?

I will implement a setStreamStatusState function to designate status of &audit.AuditStreamStatusValues.Enabled or &audit.AuditStreamStatusValues.DisabledByUser based on the boolean value coming from the schema. This will be run following create and update functions to compare the resource vs. our state declaration.

[ksh5022]

5. A bit concern about the property days_to_backfill, where can I update this config from UI web page.

This is not available via the web UI, only the API. When defined with an API request, we can see an async process when re-enabling the stream from the UI.

[xuzhang3]

@ksh5022 I have fixed the CI issue #512 , please update the latest code from upstream.

[xuzhang3]

=== RUN   TestAccAuditStreamAzureEventGrid_CreateAndUpdate
=== PAUSE TestAccAuditStreamAzureEventGrid_CreateAndUpdate
=== RUN   TestAccAuditStreamAzureEventGrid_CreateDisabled
=== PAUSE TestAccAuditStreamAzureEventGrid_CreateDisabled
=== CONT  TestAccAuditStreamAzureEventGrid_CreateAndUpdate
=== CONT  TestAccAuditStreamAzureEventGrid_CreateDisabled
--- PASS: TestAccAuditStreamAzureEventGrid_CreateDisabled (7.61s)
--- PASS: TestAccAuditStreamAzureEventGrid_CreateAndUpdate (9.22s)
PASS
ok      github.com/microsoft/terraform-provider-azuredevops/azuredevops/internal/acceptancetests        22.671s
=== RUN   TestAccAuditStreamAzureMonitorLogs_CreateAndUpdate
=== PAUSE TestAccAuditStreamAzureMonitorLogs_CreateAndUpdate
=== RUN   TestAccAuditStreamAzureMonitorLogs_CreateDisabled
=== PAUSE TestAccAuditStreamAzureMonitorLogs_CreateDisabled
=== CONT  TestAccAuditStreamAzureMonitorLogs_CreateAndUpdate
=== CONT  TestAccAuditStreamAzureMonitorLogs_CreateDisabled
--- PASS: TestAccAuditStreamAzureMonitorLogs_CreateDisabled (7.47s)
--- PASS: TestAccAuditStreamAzureMonitorLogs_CreateAndUpdate (8.06s)
PASS
ok      github.com/microsoft/terraform-provider-azuredevops/azuredevops/internal/acceptancetests        16.694s
=== RUN   TestAccAuditStreamSplunk_CreateAndUpdate
=== PAUSE TestAccAuditStreamSplunk_CreateAndUpdate
=== RUN   TestAccAuditStreamSplunk_CreateDisabled
=== PAUSE TestAccAuditStreamSplunk_CreateDisabled
=== CONT  TestAccAuditStreamSplunk_CreateAndUpdate
=== CONT  TestAccAuditStreamSplunk_CreateDisabled
--- PASS: TestAccAuditStreamSplunk_CreateDisabled (7.91s)
--- PASS: TestAccAuditStreamSplunk_CreateAndUpdate (8.85s)
PASS
ok      github.com/microsoft/terraform-provider-azuredevops/azuredevops/internal/acceptancetests        18.495s

[ksh5022]

I haven't had much time lately to get back to this, but there is an issue with the flatten function on import not assigning the correct "enabled" value. Hoping to push soon.

[xuzhang3]

@ksh5022 I checked the import function, enabled can be set correctly but the access_key will always have diff if I import the resource. Function secretmemo.IsUpdating(https://github.com/microsoft/terraform-provider-azuredevops/blob/main/azuredevops/internal/utils/secretmemo/secretmemo.go#L38) used to handle the secret value be an empty string returned by service but the access_key here is ************************, there will always be an diff if this resource imported back. The original design for function tfhelper.HelpFlattenSecret used to forbidden the sensitive value be printed in the console but I also accept we remove ***_secet/****_hash properties.

[ksh5022]

enabled can be set correctly

My test flow was terraform apply >> rm terraform.tfstate >> terraform import <resource> <id>. When importing in this fashion it was setting "enabled": false for an enabled auditstream.

access_key will always have diff if I import the resource

The GET request on the auditstream endpoint returns the obfuscated ************************ value in the response, hence the diff. To address both issues should I create a custom import function which better handles the enabled flag and does not set the sensitive values on all streams?

[xuzhang3]

@ksh5022
There is two issues for set enable incorrectly

1. You should use the enabled returns by service not local config. https://github.com/microsoft/terraform-provider-azuredevops/pull/509/files#diff-514b013332963db6fbe5e8f8fac21363cb0d56ddfea01e311c320a8b791d9239R109
2. The enable status update have a state change during the update operation, you should handle this changes to make the status consistent before the read operation. From my test, at least there have a middle state: backfilling
   Ref:

   terraform-provider-azuredevops/azuredevops/internal/service/core/resource_project.go

   Line 325 in d2f8299

   stateConf := &resource.StateChangeConf{

access_key will always have diff if I import the resource:

1. You can update https://github.com/microsoft/terraform-provider-azuredevops/blob/main/azuredevops/internal/utils/secretmemo/secretmemo.go#L13 to

   func isBlank(s string) bool {
       return len(strings.TrimSpace(s)) == 0 || strings.EqualFold("************************", s)
   }

   to handle the obfuscated values or you can remove the SecretHashKey:
   https://github.com/microsoft/terraform-provider-azuredevops/pull/509/files#diff-bc885be60dacee447c40348eb24330b0e503d6137a818fea226828e4d0309988R33

   https://github.com/microsoft/terraform-provider-azuredevops/pull/509/files#diff-ce2ed9e26640187b1428aa12c3a9690321b50dc80679497a2200f3a61568eeedR33

   https://github.com/microsoft/terraform-provider-azuredevops/pull/509/files#diff-b3eb23f267fe0c5924ebcff8b698daf41c962db01c6c83177abe144727254c65R33