Hi,
I am running grafana 4.3.0 on CentOS Linux release 7.3.1611 (Core). I have configured alert notifications for slack and have configured AWS S3 for image upload. When i query the S3 bucket list, i do not see any images and the same are not rendered in the Slack Channel (https://slack.com/)
I am seeing the below error in /var/log/grafana/grafana.log
t=2017-05-25T18:50:33+0530 lvl=info msg=“Starting Grafana” logger=main version=4.3.1 commit=befc15c compiled=2017-05-23T19:20:22+0530
t=2017-05-25T18:50:33+0530 lvl=info msg=“Config loaded from” logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2017-05-25T18:50:33+0530 lvl=info msg=“Config loaded from” logger=settings file=/etc/grafana/grafana.ini
t=2017-05-25T18:50:33+0530 lvl=info msg=“Config overriden from command line” logger=settings arg="default.paths.data=/var/lib/grafana"
t=2017-05-25T18:50:33+0530 lvl=info msg=“Config overriden from command line” logger=settings arg="default.paths.logs=/var/log/grafana"
t=2017-05-25T18:50:33+0530 lvl=info msg=“Config overriden from command line” logger=settings arg="default.paths.plugins=/var/lib/grafana/plugins"
t=2017-05-25T18:50:33+0530 lvl=info msg=“Path Home” logger=settings path=/usr/share/grafana
t=2017-05-25T18:50:33+0530 lvl=info msg=“Path Data” logger=settings path=/var/lib/grafana
t=2017-05-25T18:50:33+0530 lvl=info msg=“Path Logs” logger=settings path=/var/log/grafana
t=2017-05-25T18:50:33+0530 lvl=info msg=“Path Plugins” logger=settings path=/var/lib/grafana/plugins
t=2017-05-25T18:50:33+0530 lvl=info msg=“Initializing DB” logger=sqlstore dbtype=sqlite3
t=2017-05-25T18:50:33+0530 lvl=info msg=“Starting DB migration” logger=migrator
t=2017-05-25T18:50:33+0530 lvl=info msg=“Executing migration” logger=migrator id="copy data account to org"
t=2017-05-25T18:50:33+0530 lvl=info msg=“Skipping migration condition not fulfilled” logger=migrator id="copy data account to org"
t=2017-05-25T18:50:33+0530 lvl=info msg=“Executing migration” logger=migrator id="copy data account_user to org_user"
t=2017-05-25T18:50:33+0530 lvl=info msg=“Skipping migration condition not fulfilled” logger=migrator id="copy data account_user to org_user"
t=2017-05-25T18:50:33+0530 lvl=info msg=“Starting plugin search” logger=plugins
t=2017-05-25T18:50:33+0530 lvl=info msg=“Registering plugin” logger=plugins name="Pie Chart"
t=2017-05-25T18:50:33+0530 lvl=info msg="Initializing Stream Manager"
t=2017-05-25T18:50:33+0530 lvl=info msg=“Initializing HTTP Server” logger=http.server address=0.0.0.0:3000 protocol=http subUrl= socket=
t=2017-05-25T18:50:33+0530 lvl=info msg=“Initializing Alerting” logger=alerting.engine
t=2017-05-25T18:50:33+0530 lvl=info msg=“Initializing CleanUpService” logger=cleanup
t=2017-05-25T18:52:30+0530 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/plugins status=302 remote_addr=103.61.199.2 time_ms=1 size=29
t=2017-05-25T18:52:30+0530 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/login status=302 remote_addr=103.61.199.2 time_ms=26 size=47
t=2017-05-25T22:05:01+0530 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=3 newState=ok prev state=alerting
t=2017-05-25T22:05:01+0530 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=3 sent count=1
t=2017-05-25T22:05:01+0530 lvl=info msg=Rendering logger=png-renderer path="dashboard-solo/db/transaction-metrics?&panelId=2"
t=2017-05-25T22:05:03+0530 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="AuthorizationHeaderMalformed: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘ap-southeast-1’\n\tstatus code: 400, request id: 26EF32AC4DF560E1, host id: eIEk4ZR0zMTgKkrnDCQBS+k/zIiOKHjSe0yjbXFO8Vkn2YM1dM34zkYAWqZG++HkzK2TUnLuOys="
t=2017-05-25T22:05:03+0530 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=2 isDefault=false
t=2017-05-25T22:05:03+0530 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=3 notification="Transactional Metrics"
t=2017-05-25T22:15:01+0530 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=3 newState=alerting prev state=ok
t=2017-05-25T22:15:01+0530 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=3 sent count=1
t=2017-05-25T22:15:01+0530 lvl=info msg=Rendering logger=png-renderer path="dashboard-solo/db/transaction-metrics?&panelId=2"
t=2017-05-25T22:15:03+0530 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="AuthorizationHeaderMalformed: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘ap-southeast-1’\n\tstatus code: 400, request id: 7B891F1F05721452, host id: m6yKA9jNcmHe3ybYHBnbJevRsHIRGfezZHlxtxpID/cw4e2OLNmlT/hDV8CMjt9Uf0j8AnqWtgE="
t=2017-05-25T22:15:03+0530 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=2 isDefault=false
t=2017-05-25T22:15:03+0530 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=3 notification="Transactional Metrics"
t=2017-05-25T22:20:01+0530 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=3 newState=ok prev state=alerting
t=2017-05-25T22:20:01+0530 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=3 sent count=1
t=2017-05-25T22:20:01+0530 lvl=info msg=Rendering logger=png-renderer path="dashboard-solo/db/transaction-metrics?&panelId=2"
t=2017-05-25T22:20:03+0530 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="AuthorizationHeaderMalformed: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘ap-southeast-1’\n\tstatus code: 400, request id: E7E6BA41ED1B4796, host id: sC75Sf9mBh7n6XlFZda5GsNOd1skHuEWsXcc/AsL2wnsCo10el2yGygE61jgW0AkT1Aguoapu0A="
t=2017-05-25T22:20:03+0530 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=2 isDefault=false
t=2017-05-25T22:20:03+0530 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=3 notification="Transactional Metrics"
t=2017-05-25T22:25:01+0530 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=3 newState=alerting prev state=ok
t=2017-05-25T22:25:01+0530 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=3 sent count=1
t=2017-05-25T22:25:01+0530 lvl=info msg=Rendering logger=png-renderer path="dashboard-solo/db/transaction-metrics?&panelId=2"
t=2017-05-25T22:25:03+0530 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="AuthorizationHeaderMalformed: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘ap-southeast-1’\n\tstatus code: 400, request id: A80723A264FEE9FD, host id: NwwrcZVMCHQtVqLYPe9UVqzjGJfkK+g3mEftQUTSzJEVdjedybMN5T5aEMNcc9zMdiYx+KioMl0="
t=2017-05-25T22:25:03+0530 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=2 isDefault=false
t=2017-05-25T22:25:03+0530 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=3 notification="Transactional Metrics"
t=2017-05-25T22:35:01+0530 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=3 newState=ok prev state=alerting
t=2017-05-25T22:35:01+0530 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=3 sent count=1
t=2017-05-25T22:35:01+0530 lvl=info msg=Rendering logger=png-renderer path="dashboard-solo/db/transaction-metrics?&panelId=2"
t=2017-05-25T22:35:03+0530 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="AuthorizationHeaderMalformed: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘ap-southeast-1’\n\tstatus code: 400, request id: 5334578F520E6E6A, host id: pPPhCPNZCXBK9cudHeamMF9cJBaoXSeL+WsMPXiOqc2gv/de3rlSpvnrEF5TkC2AN2Bd/+x8e6E="
t=2017-05-25T22:35:03+0530 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=2 isDefault=false
t=2017-05-25T22:35:03+0530 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=3 notification="Transactional Metrics"
t=2017-05-25T22:55:01+0530 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=3 newState=alerting prev state=ok
t=2017-05-25T22:55:01+0530 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=3 sent count=1
t=2017-05-25T22:55:01+0530 lvl=info msg=Rendering logger=png-renderer path="dashboard-solo/db/transaction-metrics?&panelId=2"
t=2017-05-25T22:55:03+0530 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="AuthorizationHeaderMalformed: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘ap-southeast-1’\n\tstatus code: 400, request id: F643B7DC094BF13E, host id: 8T1x7dlKzhdW/mnNK+bcRfTidcnTtSbDisnB2/i1hNBHw4oA+OqISOZ/l1Sz4jwGtJxXbd6LGIg="
t=2017-05-25T22:55:03+0530 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=2 isDefault=false
t=2017-05-25T22:55:03+0530 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=3 notification="Transactional Metrics"
t=2017-05-25T23:00:01+0530 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=3 newState=ok prev state=alerting
t=2017-05-25T23:00:01+0530 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=3 sent count=1
t=2017-05-25T23:00:01+0530 lvl=info msg=Rendering logger=png-renderer path="dashboard-solo/db/transaction-metrics?&panelId=2"
t=2017-05-25T23:00:03+0530 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="AuthorizationHeaderMalformed: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘ap-southeast-1’\n\tstatus code: 400, request id: 5FF62C65F275CCE4, host id: BzccX8Iveo4U1SGWxH4B4OBupcM3asTT+4faf2VHe+QAyccgw7rVrBcbrhA9U0i/iSsT6NIqk54="
t=2017-05-25T23:00:03+0530 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=2 isDefault=false
t=2017-05-25T23:00:03+0530 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=3 notification="Transactional Metrics"
t=2017-05-25T23:15:01+0530 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=3 newState=alerting prev state=ok
t=2017-05-25T23:15:01+0530 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=3 sent count=1
t=2017-05-25T23:15:01+0530 lvl=info msg=Rendering logger=png-renderer path="dashboard-solo/db/transaction-metrics?&panelId=2"
t=2017-05-25T23:15:03+0530 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="AuthorizationHeaderMalformed: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘ap-southeast-1’\n\tstatus code: 400, request id: 60A4D99F391390A4, host id: PPikGg0PxT/epbqO4YroZxeRJAE9AP0abIGYXtDqsXfd+uzrt0PNE3O5Wr/EKTAcvMLbodRRN3M="
t=2017-05-25T23:15:03+0530 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=2 isDefault=false
t=2017-05-25T23:15:03+0530 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=3 notification="Transactional Metrics"
t=2017-05-25T23:20:01+0530 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=3 newState=ok prev state=alerting
t=2017-05-25T23:20:01+0530 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=3 sent count=1
t=2017-05-25T23:20:01+0530 lvl=info msg=Rendering logger=png-renderer path="dashboard-solo/db/transaction-metrics?&panelId=2"
t=2017-05-25T23:20:03+0530 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="AuthorizationHeaderMalformed: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘ap-southeast-1’\n\tstatus code: 400, request id: 10BC4C55B40E282B, host id: bC1Ez7Y3ZNsCKhXRYMlCmNc2Rmvd8BzwcdszC7QVOpTVpRI7jnQylwyv9r/R4NEt1RyJnzePogU="
t=2017-05-25T23:20:03+0530 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=2 isDefault=false
t=2017-05-25T23:20:03+0530 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=3 notification=“Transactional Metrics”
Any help will be highly appreciable
Regards,
Kaushal