Grafana alert notifier issue

Grafana Alerting
1

Hey,
I’m using VictorOps and Slack as notification chennal for alerting. I’m having issue with one dashboard as alert failes to send notification to VictorOps but succesfully sends same alert to Slack simultaneously.
Also this happens to only one dashboard as other alerts from different dashboards with same notify chennal work fine.
Most weird thing here is grafana successfully sends resolution alert to both VO and Slack.

t=2019-11-01T07:20:02+0000 lvl=info msg="New state change" logger=alerting.resultHandler alertId=236 newState=alerting prev state=pending

t=2019-11-01T07:20:02+0000 lvl=info msg=Rendering logger=rendering path="d-solo/y6wkHyEik-/dashboard?orgId=1&panelId=32"

t=2019-11-01T07:20:03+0000 lvl=info msg="uploaded screenshot of alert to external image store" logger=alerting.notifier url=https://s3.amazonaws.com/alertimages-000000000000/AnVRKGm.png

t=2019-11-01T07:20:03+0000 lvl=info msg="Executing victorops notification" logger=alerting.notifier.victorops ruleId=236 notification="VO - devops-oncall"


t=2019-11-01T07:20:03+0000 lvl=eror msg="Failed to send Victorops notification" logger=alerting.notifier.victorops error="Post <NOTIFICATION_URL>: read tcp xx.xx.xx.xx:35690->xx.xx.xx.xx:443: read: connection reset by peer" webhook="VO - devops-oncall"


t=2019-11-01T07:20:03+0000 lvl=eror msg="failed to send notification" logger=alerting.notifier uid=7qGw9cvWz error="Post <NOTIFICATION_URL>: read tcp xx.xx.xx.xx:35690->xx.xx.xx.xx:443: read: connection reset by peer"

t=2019-11-01T07:20:03+0000 lvl=info msg="Executing slack notification" logger=alerting.notifier.slack ruleId=236 notification="alerts_grafana - Slack"

Where should I start looking?

By the way I’m using grafana with AWS ECS.
Grafana version 6.1.4

2

With some hope to resolve this issue I upgraded grafana to 6.4.3. Now I’m getting different error,

t=2019-11-26T22:26:33+0000 lvl=info msg="Executing victorops notification" logger=alerting.notifier.victorops ruleId=185 notification="VO - devops-oncall"

t=2019-11-26T22:26:33+0000 lvl=eror msg="Failed to send Victorops notification" logger=alerting.notifier.victorops error="Webhook response status 400 Bad Request" webhook="VO - devops-oncall"

t=2019-11-26T22:26:33+0000 lvl=eror msg="failed to send notification" logger=alerting.notifier uid=********* error="Webhook response status 400 Bad Request"

t=2019-11-26T22:26:33+0000 lvl=eror msg="failed to send notification" logger=alerting.notifier uid=********* error="Webhook response status 400 Bad Request"