fix(aws): CleanupAlarmsAgent cycle to catch exceptions

[christosarvanitis]

A bad account or any exception currently stops the CleanupAlarmsAgent.

Catching the exception and logging it since the purpose of this agent is to cleanup the stale cloudwatch alarms from the aws/ecs accounts in Spinnaker and not to verify if the accounts are valid or not.

[dbyron-sf]

Is there a catch block to add "higher up" (where the run method is called) that would (also) help? Like, what if there exceptions in other agents?

[christosarvanitis]

Higher up there is already a try/catch block in the RunnableAgent initial implementation but i have pushed a try/catch block in the CleanupDetachedInstancesAgent as well which implements a RunnableAgent

[dbyron-sf]

Seems like run is called from RunnableAgentExecution.executeAgent and all the places that call executeAgent are similar to DefaultAgentScheduler. There's a try/catch, but it only updates metrics. So what we get here is some extra logging, but I don't see how it's going to help with agent scheduling.

[christosarvanitis]

This cleanup agents are going through each account and trying to do a cleanup of any stale alarms in cloudwatch or any detached instances etc. When a bad account is present in the credentials repository the agent stops and doesnt go through the rest of the accounts.
By adding a try/catch here we are logging that there was a problem with X account and continue to the next account.

[dbyron-sf]

Aaah yes, the try/catch here and in CleanupDetachedInstancesAgent is inside getAccounts().each. Seems like the consequences of this fix are good...that instead of dying on the first error, we'll continue to clean up other accounts.

[dbyron-sf]

@christosarvanitis if you bring this branch up to date, I think it's ready to go.

[christosarvanitis]

@christosarvanitis if you bring this branch up to date, I think it's ready to go.

Done! Thanks @dbyron-sf