Grafana failed to start after linux update (error in grafana-image-renderer)

Grafana Configuration
,
1

I’m running Grafana 7.0.1 on a local REDHAT 7.9 server, and everything was working correctly. However, we have monthly OS updates, and in the last updates (that were performed yesterday), Grafana stopped booting. From what I saw in the logs (in debug) the problem is related to the “grafana-image-renderer” plugin. By deleting this plugin’s folder, Grafana was able to boot without problems.

As we had the grafana-image-renderer plugin in version 2.0, I updated it to 3.3.0, but the behavior is the same. Grafana only starts without this plugin. In the logs, it shows “exit status 4”.

The logs I get in grafana are the following:

t=2022-01-20T21:46:20+0000 lvl=dbug msg=“Plugins: Adding route” logger=http.server route=/public/plugins/grafana-image-renderer dir=/home/perfteam/opt/grafana-db/data/plugins/plugin-linux-x64-glibc
t=2022-01-20T21:46:20+0000 lvl=dbug msg=“Found old rendered image to delete” logger=cleanup deleted=0 kept=2
t=2022-01-20T21:46:20+0000 lvl=dbug msg=“starting plugin” logger=plugins.backend pluginId=grafana-image-renderer path=/home/perfteam/opt/grafana-db/data/plugins/plugin-linux-x64-glibc/plugin_start_linux_amd64 args=[/home/perfteam/opt/grafana-db/data/plugins/plugin-linux-x64-glibc/plugin_start_linux_amd64]
t=2022-01-20T21:46:20+0000 lvl=dbug msg=“plugin started” logger=plugins.backend pluginId=grafana-image-renderer path=/home/perfteam/opt/grafana-db/data/plugins/plugin-linux-x64-glibc/plugin_start_linux_amd64 pid=11120
t=2022-01-20T21:46:20+0000 lvl=dbug msg=“waiting for RPC address” logger=plugins.backend pluginId=grafana-image-renderer path=/home/perfteam/opt/grafana-db/data/plugins/plugin-linux-x64-glibc/plugin_start_linux_amd64
t=2022-01-20T21:46:20+0000 lvl=dbug msg=“Checking for updates” logger=plugins
t=2022-01-20T21:46:20+0000 lvl=dbug msg=“Initializing Stream Manager”
t=2022-01-20T21:46:20+0000 lvl=info msg=“HTTP Server Listen” logger=http.server address=[::]:3000 protocol=http subUrl= socket=
t=2022-01-20T21:46:21+0000 lvl=dbug msg=“plugin process exited” logger=plugins.backend pluginId=grafana-image-renderer path=/home/perfteam/opt/grafana-db/data/plugins/plugin-linux-x64-glibc/plugin_start_linux_amd64 pid=11120 error=“exit status 4”
t=2022-01-20T21:46:21+0000 lvl=eror msg=“Stopped RenderingService” logger=server reason=“Failed to start renderer plugin: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol.”
t=2022-01-20T21:46:21+0000 lvl=dbug msg=“Stop polling for dashboard changes” logger=provisioning
t=2022-01-20T21:46:21+0000 lvl=dbug msg=“Stopped provisioningServiceImpl” logger=server reason=“context canceled”
t=2022-01-20T21:46:21+0000 lvl=dbug msg=“Stopping plugin” logger=plugins.backend pluginId=grafana-image-renderer
t=2022-01-20T21:46:21+0000 lvl=dbug msg=“Stopped UsageStatsService” logger=server reason=“context canceled”
t=2022-01-20T21:46:21+0000 lvl=dbug msg=“Stopped CleanUpService” logger=server reason=“context canceled”
t=2022-01-20T21:46:21+0000 lvl=dbug msg=“Stopped BackendPluginManager” logger=server reason=“context canceled”
t=2022-01-20T21:46:21+0000 lvl=info msg=“Stopped Stream Manager”
t=2022-01-20T21:46:21+0000 lvl=dbug msg=“Stopped NotificationService” logger=server reason=“context canceled”
t=2022-01-20T21:46:21+0000 lvl=dbug msg=“Stopped RemoteCache” logger=server reason=“context canceled”
t=2022-01-20T21:46:21+0000 lvl=warn msg=“plugin failed to exit gracefully” logger=plugins.backend pluginId=grafana-image-renderer
t=2022-01-20T21:46:21+0000 lvl=dbug msg=“Plugin stopped” logger=plugins.backend pluginId=grafana-image-renderer
t=2022-01-20T21:46:21+0000 lvl=dbug msg=“Stopped InternalMetricsService” logger=server reason=“context canceled”
t=2022-01-20T21:46:21+0000 lvl=dbug msg=“Stopped UserAuthTokenService” logger=server reason=“context canceled”
t=2022-01-20T21:46:21+0000 lvl=dbug msg=“Stopped AlertEngine” logger=server reason=“context canceled”
t=2022-01-20T21:46:21+0000 lvl=dbug msg=“server was shutdown gracefully” logger=http.server
t=2022-01-20T21:46:30+0000 lvl=dbug msg=“Failed to get plugins repo from grafana.com, Get “https://grafana.com/api/plugins/versioncheck?slugIn=grafana-piechart-panel,grafana-image-renderer,jasonlashua-prtg-datasource,grafana-worldmap-panel,input&grafanaVersion=7.0.1”: context deadline exceeded (Client.Timeout exceeded while awaiting headers)”
t=2022-01-20T21:46:30+0000 lvl=dbug msg=“Stopped PluginManager” logger=server reason=“context canceled”
t=2022-01-20T21:46:30+0000 lvl=eror msg=“A service failed” logger=server err=“Failed to start renderer plugin: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol.”
t=2022-01-20T21:46:30+0000 lvl=eror msg=“Server shutdown” logger=server reason=“Failed to start renderer plugin: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol.”

The updates that were install the machine were these:

Transaction performed with:
Updated rpm-4.11.3-46.el7_9.x86_64 @rhel-7-server-rpms
Updated subscription-manager-1.24.48-1.el7_9.x86_64 @rhel-7-server-rpms
Installed yum-3.4.3-168.el7.noarch @rhel-7-server-rpms
Installed yum-metadata-parser-1.1.4-10.el7.x86_64 @anaconda/7.3
Packages Altered:
Updated bash-4.2.46-34.el7.x86_64 @rhel-7-server-rpms
Update 4.2.46-35.el7_9.x86_64 @rhel-7-server-rpms
Updated bind-export-libs-32:9.11.4-26.P2.el7_9.7.x86_64 @rhel-7-server-rpms
Update 32:9.11.4-26.P2.el7_9.8.x86_64 @rhel-7-server-rpms
Updated bind-libs-32:9.11.4-26.P2.el7_9.7.x86_64 @rhel-7-server-rpms
Update 32:9.11.4-26.P2.el7_9.8.x86_64 @rhel-7-server-rpms
Updated bind-libs-lite-32:9.11.4-26.P2.el7_9.7.x86_64 @rhel-7-server-rpms
Update 32:9.11.4-26.P2.el7_9.8.x86_64 @rhel-7-server-rpms
Updated bind-license-32:9.11.4-26.P2.el7_9.7.noarch @rhel-7-server-rpms
Update 32:9.11.4-26.P2.el7_9.8.noarch @rhel-7-server-rpms
Updated bind-utils-32:9.11.4-26.P2.el7_9.7.x86_64 @rhel-7-server-rpms
Update 32:9.11.4-26.P2.el7_9.8.x86_64 @rhel-7-server-rpms
Updated gettext-0.19.8.1-3.el7.x86_64 @rhel-7-server-rpms
Update 0.19.8.1-3.el7_9.x86_64 @rhel-7-server-rpms
Updated gettext-libs-0.19.8.1-3.el7.x86_64 @rhel-7-server-rpms
Update 0.19.8.1-3.el7_9.x86_64 @rhel-7-server-rpms
Updated kbd-1.15.5-15.el7.x86_64 @rhel-7-server-rpms
Update 1.15.5-16.el7_9.x86_64 @rhel-7-server-rpms
Updated kbd-legacy-1.15.5-15.el7.noarch @rhel-7-server-rpms
Update 1.15.5-16.el7_9.noarch @rhel-7-server-rpms
Updated kbd-misc-1.15.5-15.el7.noarch @rhel-7-server-rpms
Update 1.15.5-16.el7_9.noarch @rhel-7-server-rpms
Erase kernel-3.10.0-1160.36.2.el7.x86_64 @rhel-7-server-rpms
Install kernel-3.10.0-1160.49.1.el7.x86_64 @rhel-7-server-rpms
Updated kernel-headers-3.10.0-1160.45.1.el7.x86_64 @rhel-7-server-rpms
Update 3.10.0-1160.49.1.el7.x86_64 @rhel-7-server-rpms
Updated kernel-tools-3.10.0-1160.45.1.el7.x86_64 @rhel-7-server-rpms
Update 3.10.0-1160.49.1.el7.x86_64 @rhel-7-server-rpms
Updated kernel-tools-libs-3.10.0-1160.45.1.el7.x86_64 @rhel-7-server-rpms
Update 3.10.0-1160.49.1.el7.x86_64 @rhel-7-server-rpms
Updated krb5-libs-1.15.1-50.el7.x86_64 @rhel-7-server-rpms
Update 1.15.1-51.el7_9.x86_64 @rhel-7-server-rpms
Updated libsss_idmap-1.16.5-10.el7_9.10.x86_64 @rhel-7-server-rpms
Update 1.16.5-10.el7_9.11.x86_64 @rhel-7-server-rpms
Updated libsss_nss_idmap-1.16.5-10.el7_9.10.x86_64 @rhel-7-server-rpms
Update 1.16.5-10.el7_9.11.x86_64 @rhel-7-server-rpms
Updated mdadm-4.1-7.el7_9.x86_64 @rhel-7-server-rpms
Update 4.1-8.el7_9.x86_64 @rhel-7-server-rpms
Updated nss-3.67.0-3.el7_9.x86_64 @rhel-7-server-rpms
Update 3.67.0-4.el7_9.x86_64 @rhel-7-server-rpms
Updated nss-sysinit-3.67.0-3.el7_9.x86_64 @rhel-7-server-rpms
Update 3.67.0-4.el7_9.x86_64 @rhel-7-server-rpms
Updated nss-tools-3.67.0-3.el7_9.x86_64 @rhel-7-server-rpms
Update 3.67.0-4.el7_9.x86_64 @rhel-7-server-rpms
Updated openssh-7.4p1-21.el7.x86_64 @rhel-7-server-rpms
Update 7.4p1-22.el7_9.x86_64 @rhel-7-server-rpms
Updated openssh-clients-7.4p1-21.el7.x86_64 @rhel-7-server-rpms
Update 7.4p1-22.el7_9.x86_64 @rhel-7-server-rpms
Updated openssh-server-7.4p1-21.el7.x86_64 @rhel-7-server-rpms
Update 7.4p1-22.el7_9.x86_64 @rhel-7-server-rpms
Updated python-perf-3.10.0-1160.45.1.el7.x86_64 @rhel-7-server-rpms
Update 3.10.0-1160.49.1.el7.x86_64 @rhel-7-server-rpms
Updated python-syspurpose-1.24.48-1.el7_9.x86_64 @rhel-7-server-rpms
Update 1.24.50-1.el7_9.x86_64 @rhel-7-server-rpms
Updated rpm-4.11.3-46.el7_9.x86_64 @rhel-7-server-rpms
Update 4.11.3-48.el7_9.x86_64 @rhel-7-server-rpms
Updated rpm-build-libs-4.11.3-46.el7_9.x86_64 @rhel-7-server-rpms
Update 4.11.3-48.el7_9.x86_64 @rhel-7-server-rpms
Updated rpm-libs-4.11.3-46.el7_9.x86_64 @rhel-7-server-rpms
Update 4.11.3-48.el7_9.x86_64 @rhel-7-server-rpms
Updated rpm-python-4.11.3-46.el7_9.x86_64 @rhel-7-server-rpms
Update 4.11.3-48.el7_9.x86_64 @rhel-7-server-rpms
Updated sssd-client-1.16.5-10.el7_9.10.x86_64 @rhel-7-server-rpms
Update 1.16.5-10.el7_9.11.x86_64 @rhel-7-server-rpms
Updated subscription-manager-1.24.48-1.el7_9.x86_64 @rhel-7-server-rpms
Update 1.24.50-1.el7_9.x86_64 @rhel-7-server-rpms
Updated subscription-manager-rhsm-1.24.48-1.el7_9.x86_64 @rhel-7-server-rpms
Update 1.24.50-1.el7_9.x86_64 @rhel-7-server-rpms
Updated subscription-manager-rhsm-certificates-1.24.48-1.el7_9.x86_64 @rhel-7-server-rpms
Update 1.24.50-1.el7_9.x86_64 @rhel-7-server-rpms
history info
Uploading Enabled Repositories Report
Loaded plugins: langpacks, product-id, subscription-manager

Is there anything else I can do to find out the problem? What dependencies does this plugin need on the OS?

2

do you know the exact command or commands you use to update?

Is there something preventing you from upgrading Grafana itself? 7.0.1 is about 18 months old now…

3

Hi,
Updates are performed via AWS, but if the command were executed manually it is equivalent to :

yum update -x=java* -y

Which means update everything except JAVA.

About the Grafana version upgrade, it was planned and we have no problem with that, but from what we tested it does not influence the problem we are having with this plugin.

For example, if I put a version 8 of grafana on this same server, the plugin error still occurs.

4

Is there any other option (or something we can activate) to give more detail in the logs, in order to try to understand (in the plugin) what is the “exit status 4”.

5

try turning on the max verbosity in the server logs:

6

Hello, I’ve managed to get the grafana-image-renderer module working, but I still don’t have an explanation for this behavior.

I had the following in the system environment variables:

export NODE_OPTIONS="–max_old_space_size=4096"

At this point, only if the “max_old_space_size” is removed, the plugin is executed.

Even weirder, if I have this variable “–max_old_space_size=4096” , and run the plugin executable “plugin-linux-x64-glibc/plugin_start_linux_amd64” directly in shell, it doesn’t return any output. If the variable is removed, it already has the expected behavior and returns:

[user@server plugin-linux-x64-glibc]$ ./plugin_start_linux_amd64
Error: This binary is a plugin. These are not meant to be executed directly. Please execute the program that consumes these plugins, which will load any plugins automatically
    at /snapshot/grafana-image-renderer/build/node-plugin/lib/server.js:85:15
    at Generator.next (<anonymous>)
    at /snapshot/grafana-image-renderer/build/node-plugin/lib/server.js:8:71
    at new Promise (<anonymous>)
    at /snapshot/grafana-image-renderer/build/node-plugin/lib/server.js:4:12
    at serve (/snapshot/grafana-image-renderer/build/node-plugin/lib/server.js:78:25)
    at /snapshot/grafana-image-renderer/build/app.js:46:43
    at Generator.next (<anonymous>)
    at /snapshot/grafana-image-renderer/build/app.js:8:71
    at new Promise (<anonymous>)