feat(monitoring): add Argo Workflow Controller failure alert

[hazdl]

Create an alert rule for failing Argo Workflows controller pods --- AP-1299

Summary

 
Adds a Prometheus alert for detecting Argo Workflow Controller failures.
 
A temporary test alert was used to verify that alert rules are loaded via the monitoring Helm chart (alerting_rules.yml). After successful validation, it was replaced with ArgoWorkflowControllerFailing.
 

Outcome

 
Implemented and validated an alert that fires when the Argo Workflow Controller remains in CrashLoopBackOff for more than 10 minutes. Testing was completed successfully and the monitoring application was restored to follow HEAD.
Show more lines

[iamvigneshwars]

You don't have to create a PR for testing things and merge it to main. You can simply point the argocd app to your dev branch in staging and test it.

[hazdl]

Thanks Viki, that's what I ended up doing in the end. After the initial push, I realised I could point the monitoring ArgoCD application at my hz/alert-rules branch in staging to validate the change. I used a temporary TestAlert to confirm the rules were being loaded and fired correctly in Prometheus, then replaced it with the final ArgoWorkflowControllerFailing alert and restored the application to track HEAD afterwards.