Server start failure after 8.5.0 upgrade

ebouxirot · April 21, 2022, 5:35pm

Hi everyone ,

I just upgraded my grafana server on my respberrypi to the latest 8.5.0 version…
previously i had the 8.4.7

now, grafana won’t start anymore…
i get a big non-understandable (for me) error…

i have 2 raspberry (1x RPI2B and 1x RPI4)
both installation now are broken…

if anyone can help… ;

thanks a lot !
Eric.

On the Raspberry Pi 2 i get:

raspberrypi:~ $ grafana-server
runtime: unexpected return pc for github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10 called from 0x9
stack: frame={sp:0x4fa3c34, fp:0x4fa3c5c} stack=[0x4f84000,0x4fa4000)
0x04fa3bb4:  0x00000000  0x04fa3bc0  0x00000000  0x05092460
0x04fa3bc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x04922900  0x00000180  0x00000170
0x04fa3bd4:  0x048000f0  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
0x04fa3be4:  0x76cef2a8  0x0000016c  0x04800100  0x00000000
0x04fa3bf4:  0x02478480  0x04312cd0  0x00000000  0x00000000
0x04fa3c04:  0x00000000  0x00000000  0x0007492c <runtime.sigpanic+0x000001e8>  0x02478480
0x04fa3c14:  0x04312cd0  0x02092c5c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000002c>  0x048000f0  0x022d9601
0x04fa3c24:  0x04922900  0x023bc5d8  0x048000f0  0x02092c5c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000002c>
0x04fa3c34: <0x00000009  0x050924b4  0x04971e40  0x04922900
0x04fa3c44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00054bb8 <runtime.(*consistentHeapStats).acquire+0x00000068>  0x00035d20 <runtime.(*mcache).refill+0x00000154>
0x04fa3c54:  0x048513ac  0x0000001d >0x0000001e  0x00000101
0x04fa3c64:  0x0000001e  0x00000000  0x0487e0bf  0x00000000
0x04fa3c74:  0x00000000  0x00000012  0x00000016  0x0000000b
0x04fa3c84:  0x0486fca0  0x0486fca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
0x04fa3c94:  0x000396bc <runtime.SetFinalizer+0x00000064>  0x0003991c <runtime.SetFinalizer+0x000002c4>  0x00000001  0x00000012
0x04fa3ca4:  0x00000001  0x00000000  0x00000000  0x00000000
0x04fa3cb4:  0x00000000  0x00000000  0x00000000  0x00000000
0x04fa3cc4:  0x04810110  0x00000008  0x00000008  0x0236f380
0x04fa3cd4:  0x0261df30  0x00020138 <armCas64+0x00000000>
fatal error: unknown caller pc

runtime stack:
runtime.throw({0x28134c0, 0x11})
        /usr/local/go/src/runtime/panic.go:1198 +0x5c
runtime.gentraceback(0x5858c, 0x4fa3c0c, 0x0, 0x48000f0, 0x0, 0x0, 0x7fffffff, 0x7e8d34b4, 0x0, 0x0)
        /usr/local/go/src/runtime/traceback.go:274 +0x1540
runtime.addOneOpenDeferFrame.func1()
        /usr/local/go/src/runtime/panic.go:751 +0x8c
runtime.systemstack()
        /usr/local/go/src/runtime/asm_arm.s:318 +0x60

goroutine 1 [running, locked to thread]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_arm.s:275 +0x4 fp=0x4fa3b84 sp=0x4fa3b80 pc=0x90abc
runtime.addOneOpenDeferFrame(0x48000f0, 0x5858c, 0x4fa3c0c)
        /usr/local/go/src/runtime/panic.go:750 +0x70 fp=0x4fa3ba0 sp=0x4fa3b84 pc=0x59bf8
panic({0x2478480, 0x4312cd0})
        /usr/local/go/src/runtime/panic.go:998 +0xe8 fp=0x4fa3c0c sp=0x4fa3ba0 pc=0x5aa28
runtime.panicmem()
        /usr/local/go/src/runtime/panic.go:221 +0x4c fp=0x4fa3c18 sp=0x4fa3c0c pc=0x5858c
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:735 +0x1e8 fp=0x4fa3c30 sp=0x4fa3c18 pc=0x7492c
runtime: unexpected return pc for github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10 called from 0x9
stack: frame={sp:0x4fa3c34, fp:0x4fa3c5c} stack=[0x4f84000,0x4fa4000)
0x04fa3bb4:  0x00000000  0x04fa3bc0  0x00000000  0x05092460
0x04fa3bc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x04922900  0x00000180  0x00000170
0x04fa3bd4:  0x048000f0  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
0x04fa3be4:  0x76cef2a8  0x0000016c  0x04800100  0x00000000
0x04fa3bf4:  0x02478480  0x04312cd0  0x00000000  0x00000000
0x04fa3c04:  0x00000000  0x00000000  0x0007492c <runtime.sigpanic+0x000001e8>  0x02478480
0x04fa3c14:  0x04312cd0  0x02092c5c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000002c>  0x048000f0  0x022d9601
0x04fa3c24:  0x04922900  0x023bc5d8  0x048000f0  0x02092c5c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000002c>
0x04fa3c34: <0x00000009  0x050924b4  0x04971e40  0x04922900
0x04fa3c44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00054bb8 <runtime.(*consistentHeapStats).acquire+0x00000068>  0x00035d20 <runtime.(*mcache).refill+0x00000154>
0x04fa3c54:  0x048513ac  0x0000001d >0x0000001e  0x00000101
0x04fa3c64:  0x0000001e  0x00000000  0x0487e0bf  0x00000000
0x04fa3c74:  0x00000000  0x00000012  0x00000016  0x0000000b
0x04fa3c84:  0x0486fca0  0x0486fca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
0x04fa3c94:  0x000396bc <runtime.SetFinalizer+0x00000064>  0x0003991c <runtime.SetFinalizer+0x000002c4>  0x00000001  0x00000012
0x04fa3ca4:  0x00000001  0x00000000  0x00000000  0x00000000
0x04fa3cb4:  0x00000000  0x00000000  0x00000000  0x00000000
0x04fa3cc4:  0x04810110  0x00000008  0x00000008  0x0236f380
0x04fa3cd4:  0x0261df30  0x00020138 <armCas64+0x00000000>
github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10()
        /drone/src/pkg/services/alerting/notifiers/pushover.go:97 +0x2c fp=0x4fa3c5c sp=0x4fa3c34 pc=0x2092c5c

goroutine 114 [select]:
go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0x4b46780)
        /opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:243 +0xf4
go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1(0x4b46780)
        /opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:111 +0x4c
created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor
        /opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:109 +0x1e8

goroutine 13 [select]:
go.opencensus.io/stats/view.(*worker).start(0x4b47140)
        /opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xa0
created by go.opencensus.io/stats/view.init.0
        /opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x64

goroutine 146 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x4588760)
        /opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x70
created by github.com/golang/glog.init.0
        /opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x214

On the Raspberry Pi 4 the error is different:

raspberrypi4:~ $ grafana-server
unexpected fault address 0x27f6b5a
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x27f6b5a pc=0x27f6b5a]

goroutine 1 [running, locked to thread]:
runtime.throw({0x27eb2a9, 0x5})
        /usr/local/go/src/runtime/panic.go:1198 +0x5c fp=0x545bc18 sp=0x545bc04 pc=0x5b244
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:742 +0x1d0 fp=0x545bc30 sp=0x545bc18 pc=0x74914
runtime: unexpected return pc for github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10 called from 0x9
stack: frame={sp:0x545bc34, fp:0x545bc5c} stack=[0x543c000,0x545c000)
0x0545bbb4:  0x0005d1a0 <runtime.gwrite+0x00000034>  0x04668d52  0x029efeac  0x00000001
0x0545bbc4:  0x00000001  0x00000000  0x0005da4c <runtime.printstring+0x00000060>  0x00000002
0x0545bbd4:  0x029efeac  0x00000001  0x00000001  0x00000000
0x0545bbe4:  0x00000000  0x0005b450 <runtime.fatalthrow+0x00000044>  0x0005b244 <runtime.throw+0x0000005c>  0x0545bbf4
0x0545bbf4:  0x0005b468 <runtime.fatalthrow.func1+0x00000000>  0x04c000f0  0x0005b244 <runtime.throw+0x0000005c>  0x0545bc04
0x0545bc04:  0x00074914 <runtime.sigpanic+0x000001d0>  0x0545bc0c  0x0005b25c <runtime.throw.func1+0x00000000>  0x027eb2a9
0x0545bc14:  0x00000005  0x02092c5c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000002c>  0x027eb2a9  0x00000005
0x0545bc24:  0x04e63080  0x027f6b5a  0x04c000f0  0x02092c5c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000002c>
0x0545bc34: <0x00000009  0x05239e18  0x04e03f00  0x04e63080
0x0545bc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00054bb8 <runtime.(*consistentHeapStats).acquire+0x00000068>  0x00035d20 <runtime.(*mcache).refill+0x00000154>
0x0545bc54:  0x04c568ac  0x0000002b >0x0000002c  0x00000101
0x0545bc64:  0x0000002c  0x00000000  0x04caa0bf  0x00000000
0x0545bc74:  0x00000000  0x00000013  0x00000016  0x0000000b
0x0545bc84:  0x04c6eca0  0x04c6eca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
0x0545bc94:  0x000396bc <runtime.SetFinalizer+0x00000064>  0x0003991c <runtime.SetFinalizer+0x000002c4>  0x00000001  0x00000013
0x0545bca4:  0x00000001  0x00000000  0x00000000  0x00000000
0x0545bcb4:  0x00000000  0x00000000  0x00000000  0x00000000
0x0545bcc4:  0x04ca00f8  0x00000008  0x00000008  0x0236f380
0x0545bcd4:  0x0261df30  0x00020138 <armCas64+0x00000000>
github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10()
        /drone/src/pkg/services/alerting/notifiers/pushover.go:97 +0x2c fp=0x545bc5c sp=0x545bc34 pc=0x2092c5c

goroutine 100 [select]:
go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0x4cd49c0)
        /opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:243 +0xf4
go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1(0x4cd49c0)
        /opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:111 +0x4c
created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor
        /opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:109 +0x1e8

goroutine 77 [select]:
go.opencensus.io/stats/view.(*worker).start(0x51e8ba0)
        /opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xa0
created by go.opencensus.io/stats/view.init.0
        /opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x64

goroutine 81 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x4588760)
        /opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x70
created by github.com/golang/glog.init.0
        /opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x214

ebouxirot · April 21, 2022, 6:01pm

i just do a rollback of the raspberrypi4 grafana server to 8.4.7 and now it is running again…
so, something is wrong with the latest 8.5.0

i use repo: https://packages.grafana.com/oss/deb stable InRelease

cyborgxxl · April 22, 2022, 7:55am

Same problem here on Raspi 4…

prussell · April 22, 2022, 9:23am

same here:

install procedure:

wget https://dl.grafana.com/oss/release/grafana_8.5.0_armhf.deb
sudo dpkg -i grafana_8.5.0_armhf.deb

relevant syslog entries (part1, due to size limitation):

systemd[1]: Started Grafana instance.
kernel: [87125.912581] Unhandled prefetch abort: unknown 1 (0x001) at 0x027f6b5a
grafana-server[30606]: SIGBUS: bus error
kernel: [87126.724895] Unhandled prefetch abort: unknown 1 (0x001) at 0x027f6b5a
grafana-server[30606]: PC=0x27f6b5a m=0 sigcode=0
grafana-server[30606]: goroutine 1 [running, locked to thread]:
grafana-server[30606]: runtime: unknown pc 0x27f6b5a
grafana-server[30606]: stack: frame={sp:0x6c1bc34, fp:0x0} stack=[0x6bfc000,0x6c1c000)
grafana-server[30606]: 0x06c1bbb4:  0x00000000  0x06c1bbd0  0x00000000  0x0651cfc0
grafana-server[30606]: 0x06c1bbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06652600  0x00000180  0x00000170
grafana-server[30606]: 0x06c1bbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30606]: 0x06c1bbe4:  0x6596f5a8  0x0000016c  0x00000170  0x76d532e0
grafana-server[30606]: 0x06c1bbf4:  0x00000180  0x045896c8  0x06652600  0x00000000
grafana-server[30606]: 0x06c1bc04:  0x24595180  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30606]: 0x06c1bc14:  0x00000001  0x06652600  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30606]: 0x06c1bc24:  0x06652600  0x023bc5d8  0x06601f80  0x027f6b5a
grafana-server[30606]: 0x06c1bc34: <0x00000009  0x0651d020  0x06595140  0x06652600
grafana-server[30606]: 0x06c1bc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00054bb8 <runtime.(*consistentHeapStats).acquire+0x00000068>  0x00035d20 <runtime.(*mcache).refill+0x00000154>
grafana-server[30606]: 0x06c1bc54:  0x06453dac  0x00000017  0x00000018  0x00000101
grafana-server[30606]: 0x06c1bc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x065120ff  0x00000018  0x00000000
grafana-server[30606]: 0x06c1bc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30606]: 0x06c1bc84:  0x00000002  0x0646aca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30606]: 0x06c1bc94:  0x00000008  0x00000019  0x00000001  0x00000019
grafana-server[30606]: 0x06c1bca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30606]: runtime: unknown pc 0x27f6b5a
grafana-server[30606]: stack: frame={sp:0x6c1bc34, fp:0x0} stack=[0x6bfc000,0x6c1c000)
grafana-server[30606]: 0x06c1bbb4:  0x00000000  0x06c1bbd0  0x00000000  0x0651cfc0
grafana-server[30606]: 0x06c1bbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06652600  0x00000180  0x00000170
grafana-server[30606]: 0x06c1bbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30606]: 0x06c1bbe4:  0x6596f5a8  0x0000016c  0x00000170  0x76d532e0
systemd[1]: grafana-server.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
grafana-server[30606]: 0x06c1bbf4:  0x00000180  0x045896c8  0x06652600  0x00000000
grafana-server[30606]: 0x06c1bc04:  0x24595180  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30606]: 0x06c1bc14:  0x00000001  0x06652600  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30606]: 0x06c1bc24:  0x06652600  0x023bc5d8  0x06601f80  0x027f6b5a
grafana-server[30606]: 0x06c1bc34: <0x00000009  0x0651d020  0x06595140  0x06652600
grafana-server[30606]: 0x06c1bc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00054bb8 <runtime.(*consistentHeapStats).acquire+0x00000068>  0x00035d20 <runtime.(*mcache).refill+0x00000154>
grafana-server[30606]: 0x06c1bc54:  0x06453dac  0x00000017  0x00000018  0x00000101
grafana-server[30606]: 0x06c1bc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x065120ff  0x00000018  0x00000000
grafana-server[30606]: 0x06c1bc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30606]: 0x06c1bc84:  0x00000002  0x0646aca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30606]: 0x06c1bc94:  0x00000008  0x00000019  0x00000001  0x00000019
grafana-server[30606]: 0x06c1bca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30606]: goroutine 101 [select]:
grafana-server[30606]: go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0x66567e0)
grafana-server[30606]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:243 +0xf4
grafana-server[30606]: go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1(0x66567e0)
grafana-server[30606]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:111 +0x4c
grafana-server[30606]: created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor
grafana-server[30606]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:109 +0x1e8
grafana-server[30606]: goroutine 116 [select]:
grafana-server[30606]: go.opencensus.io/stats/view.(*worker).start(0x66d6000)
grafana-server[30606]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xa0
grafana-server[30606]: created by go.opencensus.io/stats/view.init.0
grafana-server[30606]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x64
grafana-server[30606]: goroutine 96 [chan receive]:
grafana-server[30606]: github.com/golang/glog.(*loggingT).flushDaemon(0x4588760)
systemd[1]: grafana-server.service: Failed with result 'exit-code'.
grafana-server[30606]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x70
grafana-server[30606]: created by github.com/golang/glog.init.0
grafana-server[30606]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x214
grafana-server[30606]: trap    0x0
grafana-server[30606]: error   0x1
grafana-server[30606]: oldmask 0x0
grafana-server[30606]: r0      0x6652600
grafana-server[30606]: r1      0x2f1624c
grafana-server[30606]: r2      0x6652600
grafana-server[30606]: r3      0x1
grafana-server[30606]: r4      0x6d184
grafana-server[30606]: r5      0x22d96a8
grafana-server[30606]: r6      0x6652600
grafana-server[30606]: r7      0x23bc5d8
grafana-server[30606]: r8      0x6601f80
grafana-server[30606]: r9      0x180
grafana-server[30606]: r10     0x64000f0
grafana-server[30606]: fp      0x466769c
grafana-server[30606]: ip      0x665277f
grafana-server[30606]: sp      0x6c1bc34
grafana-server[30606]: lr      0x2092c5c
grafana-server[30606]: pc      0x27f6b5a
grafana-server[30606]: cpsr    0x60000010
grafana-server[30606]: fault   0x0
systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 1.
grafana-server[30618]: SIGBUS: bus error
grafana-server[30618]: PC=0x27f6b5a m=0 sigcode=0
grafana-server[30618]: goroutine 1 [running, locked to thread]:
grafana-server[30618]: runtime: unknown pc 0x27f6b5a
grafana-server[30618]: stack: frame={sp:0x6fedc34, fp:0x0} stack=[0x6fce000,0x6fee000)
grafana-server[30618]: 0x06fedbb4:  0x00000000  0x06fedbd0  0x00000000  0x06a82700
grafana-server[30618]: 0x06fedbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06e40780  0x00000180  0x00000170
grafana-server[30618]: 0x06fedbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30618]: 0x06fedbe4:  0x646c0e48  0x0000016c  0x00000170  0x76d28790
grafana-server[30618]: 0x06fedbf4:  0x00000180  0x045896c8  0x06e40780  0x00000000
grafana-server[30618]: 0x06fedc04:  0xe092ddc0  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30618]: 0x06fedc14:  0x00000001  0x06e40780  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30618]: 0x06fedc24:  0x06e40780  0x023bc5d8  0x06a08040  0x027f6b5a
grafana-server[30618]: 0x06fedc34: <0x00000009  0x06a82758  0x0692dd80  0x06e40780
grafana-server[30618]: 0x06fedc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00000000  0x00000000
grafana-server[30618]: 0x06fedc54:  0x00000000  0x00000000  0x00000000  0x00000000
grafana-server[30618]: 0x06fedc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x068a213f  0x00000018  0x00000000
grafana-server[30618]: 0x06fedc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30618]: 0x06fedc84:  0x00000002  0x0686aca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30618]: 0x06fedc94:  0x00000008  0x0000001a  0x00000001  0x0000001a
grafana-server[30618]: 0x06fedca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30618]: runtime: unknown pc 0x27f6b5a
grafana-server[30618]: stack: frame={sp:0x6fedc34, fp:0x0} stack=[0x6fce000,0x6fee000)
grafana-server[30618]: 0x06fedbb4:  0x00000000  0x06fedbd0  0x00000000  0x06a82700
grafana-server[30618]: 0x06fedbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06e40780  0x00000180  0x00000170
grafana-server[30618]: 0x06fedbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30618]: 0x06fedbe4:  0x646c0e48  0x0000016c  0x00000170  0x76d28790
grafana-server[30618]: 0x06fedbf4:  0x00000180  0x045896c8  0x06e40780  0x00000000
systemd[1]: Stopped Grafana instance.
grafana-server[30618]: 0x06fedc04:  0xe092ddc0  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30618]: 0x06fedc14:  0x00000001  0x06e40780  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30618]: 0x06fedc24:  0x06e40780  0x023bc5d8  0x06a08040  0x027f6b5a
grafana-server[30618]: 0x06fedc34: <0x00000009  0x06a82758  0x0692dd80  0x06e40780
grafana-server[30618]: 0x06fedc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00000000  0x00000000
grafana-server[30618]: 0x06fedc54:  0x00000000  0x00000000  0x00000000  0x00000000
grafana-server[30618]: 0x06fedc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x068a213f  0x00000018  0x00000000
grafana-server[30618]: 0x06fedc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30618]: 0x06fedc84:  0x00000002  0x0686aca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30618]: 0x06fedc94:  0x00000008  0x0000001a  0x00000001  0x0000001a
grafana-server[30618]: 0x06fedca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30618]: goroutine 117 [select]:
grafana-server[30618]: go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0x6b086c0)
grafana-server[30618]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:243 +0xf4
grafana-server[30618]: go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1(0x6b086c0)
grafana-server[30618]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:111 +0x4c
grafana-server[30618]: created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor
grafana-server[30618]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:109 +0x1e8
grafana-server[30618]: goroutine 12 [select]:
grafana-server[30618]: go.opencensus.io/stats/view.(*worker).start(0x6ad09c0)
grafana-server[30618]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xa0
grafana-server[30618]: created by go.opencensus.io/stats/view.init.0
grafana-server[30618]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x64
grafana-server[30618]: goroutine 135 [chan receive]:
grafana-server[30618]: github.com/golang/glog.(*loggingT).flushDaemon(0x4588760)
grafana-server[30618]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x70
grafana-server[30618]: created by github.com/golang/glog.init.0
systemd[1]: Started Grafana instance.
grafana-server[30618]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x214
grafana-server[30618]: trap    0x0
grafana-server[30618]: error   0x1
grafana-server[30618]: oldmask 0x0
grafana-server[30618]: r0      0x6e40780
grafana-server[30618]: r1      0x2f1624c
grafana-server[30618]: r2      0x6e40780
grafana-server[30618]: r3      0x1
grafana-server[30618]: r4      0x6d184
grafana-server[30618]: r5      0x22d96a8
grafana-server[30618]: r6      0x6e40780
grafana-server[30618]: r7      0x23bc5d8
grafana-server[30618]: r8      0x6a08040
grafana-server[30618]: r9      0x180
grafana-server[30618]: r10     0x68000f0
grafana-server[30618]: fp      0x466769c
grafana-server[30618]: ip      0x6e408ff
grafana-server[30618]: sp      0x6fedc34
grafana-server[30618]: lr      0x2092c5c
grafana-server[30618]: pc      0x27f6b5a
grafana-server[30618]: cpsr    0x60000010
grafana-server[30618]: fault   0x0
systemd[1]: grafana-server.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
systemd[1]: grafana-server.service: Failed with result 'exit-code'.
systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 2.
systemd[1]: Stopped Grafana instance.
systemd[1]: Started Grafana instance.
kernel: [87127.654566] Unhandled prefetch abort: unknown 1 (0x001) at 0x027f6b5a
grafana-server[30639]: SIGBUS: bus error
grafana-server[30639]: PC=0x27f6b5a m=0 sigcode=0
grafana-server[30639]: goroutine 1 [running, locked to thread]:
grafana-server[30639]: runtime: unknown pc 0x27f6b5a
grafana-server[30639]: stack: frame={sp:0x586dc34, fp:0x0} stack=[0x584e000,0x586e000)
grafana-server[30639]: 0x0586dbb4:  0x00000000  0x0586dbc0  0x00000000  0x050bb180
grafana-server[30639]: 0x0586dbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x0548e900  0x00000180  0x00000170
grafana-server[30639]: 0x0586dbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30639]: 0x0586dbe4:  0x64fc8620  0x0000016c  0x00000170  0x76d7f538
grafana-server[30639]: 0x0586dbf4:  0x00000180  0x045896c8  0x0548e900  0x00000000
grafana-server[30639]: 0x0586dc04:  0xc6042500  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30639]: 0x0586dc14:  0x00000001  0x0548e900  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30639]: 0x0586dc24:  0x0548e900  0x023bc5d8  0x050a0b80  0x027f6b5a
grafana-server[30639]: 0x0586dc34: <0x00000009  0x050bb1d8  0x05042300  0x0548e900
grafana-server[30639]: 0x0586dc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00000000  0x00000000
grafana-server[30639]: 0x0586dc54:  0x00000000  0x00000000  0x00000000  0x00000000
grafana-server[30639]: 0x0586dc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x050a213f  0x00000018  0x00000000
grafana-server[30639]: 0x0586dc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30639]: 0x0586dc84:  0x00000002  0x0506aca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30639]: 0x0586dc94:  0x00000008  0x00000014  0x00000001  0x00000014
grafana-server[30639]: 0x0586dca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30639]: runtime: unknown pc 0x27f6b5a
grafana-server[30639]: stack: frame={sp:0x586dc34, fp:0x0} stack=[0x584e000,0x586e000)
grafana-server[30639]: 0x0586dbb4:  0x00000000  0x0586dbc0  0x00000000  0x050bb180
grafana-server[30639]: 0x0586dbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x0548e900  0x00000180  0x00000170
grafana-server[30639]: 0x0586dbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30639]: 0x0586dbe4:  0x64fc8620  0x0000016c  0x00000170  0x76d7f538
grafana-server[30639]: 0x0586dbf4:  0x00000180  0x045896c8  0x0548e900  0x00000000
systemd[1]: grafana-server.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
grafana-server[30639]: 0x0586dc04:  0xc6042500  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30639]: 0x0586dc14:  0x00000001  0x0548e900  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30639]: 0x0586dc24:  0x0548e900  0x023bc5d8  0x050a0b80  0x027f6b5a
grafana-server[30639]: 0x0586dc34: <0x00000009  0x050bb1d8  0x05042300  0x0548e900
grafana-server[30639]: 0x0586dc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00000000  0x00000000
grafana-server[30639]: 0x0586dc54:  0x00000000  0x00000000  0x00000000  0x00000000
grafana-server[30639]: 0x0586dc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x050a213f  0x00000018  0x00000000
grafana-server[30639]: 0x0586dc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30639]: 0x0586dc84:  0x00000002  0x0506aca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30639]: 0x0586dc94:  0x00000008  0x00000014  0x00000001  0x00000014
grafana-server[30639]: 0x0586dca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30639]: goroutine 89 [select]:

prussell · April 22, 2022, 9:24am

relevant syslog entries (part2, due to size limitation):

grafana-server[30639]: go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0x522f140)
grafana-server[30639]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:243 +0xf4
grafana-server[30639]: go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1(0x522f140)
grafana-server[30639]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:111 +0x4c
grafana-server[30639]: created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor
grafana-server[30639]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:109 +0x1e8
grafana-server[30639]: goroutine 80 [select]:
grafana-server[30639]: go.opencensus.io/stats/view.(*worker).start(0x522f8c0)
grafana-server[30639]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xa0
grafana-server[30639]: created by go.opencensus.io/stats/view.init.0
grafana-server[30639]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x64
grafana-server[30639]: goroutine 134 [chan receive]:
grafana-server[30639]: github.com/golang/glog.(*loggingT).flushDaemon(0x4588760)
grafana-server[30639]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x70
grafana-server[30639]: created by github.com/golang/glog.init.0
systemd[1]: grafana-server.service: Failed with result 'exit-code'.
grafana-server[30639]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x214
grafana-server[30639]: trap    0x0
grafana-server[30639]: error   0x1
grafana-server[30639]: oldmask 0x0
grafana-server[30639]: r0      0x548e900
grafana-server[30639]: r1      0x2f1624c
grafana-server[30639]: r2      0x548e900
grafana-server[30639]: r3      0x1
grafana-server[30639]: r4      0x6d184
grafana-server[30639]: r5      0x22d96a8
grafana-server[30639]: r6      0x548e900
grafana-server[30639]: r7      0x23bc5d8
grafana-server[30639]: r8      0x50a0b80
grafana-server[30639]: r9      0x180
grafana-server[30639]: r10     0x50000f0
grafana-server[30639]: fp      0x466769c
grafana-server[30639]: ip      0x548ea7f
grafana-server[30639]: sp      0x586dc34
grafana-server[30639]: lr      0x2092c5c
grafana-server[30639]: pc      0x27f6b5a
grafana-server[30639]: cpsr    0x60000010
grafana-server[30639]: fault   0x0
systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 3.
systemd[1]: Stopped Grafana instance.
systemd[1]: Started Grafana instance.
kernel: [87128.443909] Unhandled prefetch abort: unknown 1 (0x001) at 0x027f6b5a
grafana-server[30669]: SIGBUS: bus error
grafana-server[30669]: PC=0x27f6b5a m=0 sigcode=0
grafana-server[30669]: goroutine 1 [running, locked to thread]:
grafana-server[30669]: runtime: unknown pc 0x27f6b5a
grafana-server[30669]: stack: frame={sp:0x6c6dc34, fp:0x0} stack=[0x6c4e000,0x6c6e000)
grafana-server[30669]: 0x06c6dbb4:  0x00000000  0x06c6dbc0  0x00000000  0x066ece00
grafana-server[30669]: 0x06c6dbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06724d80  0x00000180  0x00000170
grafana-server[30669]: 0x06c6dbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30669]: 0x06c6dbe4:  0x76d7e768  0x0000016c  0x00000170  0x76d74538
grafana-server[30669]: 0x06c6dbf4:  0x00000180  0x045896c8  0x06724d80  0x00000000
grafana-server[30669]: 0x06c6dc04:  0x71616200  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30669]: 0x06c6dc14:  0x00000001  0x06724d80  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30669]: 0x06c6dc24:  0x06724d80  0x023bc5d8  0x065993c0  0x027f6b5a
grafana-server[30669]: 0x06c6dc34: <0x00000009  0x066ece60  0x06616180  0x06724d80
grafana-server[30669]: 0x06c6dc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00000000  0x00000000
grafana-server[30669]: 0x06c6dc54:  0x00000000  0x00000000  0x00000000  0x00000000
grafana-server[30669]: 0x06c6dc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x064a213f  0x00000018  0x00000000
grafana-server[30669]: 0x06c6dc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30669]: 0x06c6dc84:  0x00000002  0x0646aca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30669]: 0x06c6dc94:  0x00000008  0x00000019  0x00000001  0x00000019
grafana-server[30669]: 0x06c6dca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30669]: runtime: unknown pc 0x27f6b5a
grafana-server[30669]: stack: frame={sp:0x6c6dc34, fp:0x0} stack=[0x6c4e000,0x6c6e000)
grafana-server[30669]: 0x06c6dbb4:  0x00000000  0x06c6dbc0  0x00000000  0x066ece00
grafana-server[30669]: 0x06c6dbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06724d80  0x00000180  0x00000170
grafana-server[30669]: 0x06c6dbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30669]: 0x06c6dbe4:  0x76d7e768  0x0000016c  0x00000170  0x76d74538
grafana-server[30669]: 0x06c6dbf4:  0x00000180  0x045896c8  0x06724d80  0x00000000
systemd[1]: grafana-server.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
grafana-server[30669]: 0x06c6dc04:  0x71616200  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30669]: 0x06c6dc14:  0x00000001  0x06724d80  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30669]: 0x06c6dc24:  0x06724d80  0x023bc5d8  0x065993c0  0x027f6b5a
grafana-server[30669]: 0x06c6dc34: <0x00000009  0x066ece60  0x06616180  0x06724d80
grafana-server[30669]: 0x06c6dc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00000000  0x00000000
grafana-server[30669]: 0x06c6dc54:  0x00000000  0x00000000  0x00000000  0x00000000
grafana-server[30669]: 0x06c6dc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x064a213f  0x00000018  0x00000000
grafana-server[30669]: 0x06c6dc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30669]: 0x06c6dc84:  0x00000002  0x0646aca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30669]: 0x06c6dc94:  0x00000008  0x00000019  0x00000001  0x00000019
grafana-server[30669]: 0x06c6dca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30669]: goroutine 95 [select]:
grafana-server[30669]: go.opencensus.io/stats/view.(*worker).start(0x6721260)
grafana-server[30669]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xa0
grafana-server[30669]: created by go.opencensus.io/stats/view.init.0
grafana-server[30669]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x64
grafana-server[30669]: goroutine 119 [select]:
grafana-server[30669]: go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0x6720360)
grafana-server[30669]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:243 +0xf4
grafana-server[30669]: go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1(0x6720360)
grafana-server[30669]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:111 +0x4c
grafana-server[30669]: created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor
grafana-server[30669]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:109 +0x1e8
grafana-server[30669]: goroutine 125 [chan receive]:
grafana-server[30669]: github.com/golang/glog.(*loggingT).flushDaemon(0x4588760)
grafana-server[30669]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x70
grafana-server[30669]: created by github.com/golang/glog.init.0
systemd[1]: grafana-server.service: Failed with result 'exit-code'.
grafana-server[30669]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x214
grafana-server[30669]: trap    0x0
grafana-server[30669]: error   0x1
grafana-server[30669]: oldmask 0x0
grafana-server[30669]: r0      0x6724d80
grafana-server[30669]: r1      0x2f1624c
grafana-server[30669]: r2      0x6724d80
grafana-server[30669]: r3      0x1
grafana-server[30669]: r4      0x6d184
grafana-server[30669]: r5      0x22d96a8
grafana-server[30669]: r6      0x6724d80
grafana-server[30669]: r7      0x23bc5d8
grafana-server[30669]: r8      0x65993c0
grafana-server[30669]: r9      0x180
grafana-server[30669]: r10     0x64000f0
grafana-server[30669]: fp      0x466769c
grafana-server[30669]: ip      0x6724eff
grafana-server[30669]: sp      0x6c6dc34
grafana-server[30669]: lr      0x2092c5c
grafana-server[30669]: pc      0x27f6b5a
grafana-server[30669]: cpsr    0x60000010
grafana-server[30669]: fault   0x0
systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 4.
systemd[1]: Stopped Grafana instance.
systemd[1]: Started Grafana instance.
kernel: [87129.191852] Unhandled prefetch abort: unknown 1 (0x001) at 0x027f6b5a
grafana-server[30689]: SIGBUS: bus error
grafana-server[30689]: PC=0x27f6b5a m=0 sigcode=0
grafana-server[30689]: goroutine 1 [running, locked to thread]:
grafana-server[30689]: runtime: unknown pc 0x27f6b5a
grafana-server[30689]: stack: frame={sp:0x6c6dc34, fp:0x0} stack=[0x6c4e000,0x6c6e000)
grafana-server[30689]: 0x06c6dbb4:  0x00000000  0x06c6dbc0  0x00000000  0x0671fc00
grafana-server[30689]: 0x06c6dbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06743c80  0x00000180  0x00000170
grafana-server[30689]: 0x06c6dbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30689]: 0x06c6dbe4:  0x76d5ab68  0x0000016c  0x00000170  0x76d4f088
grafana-server[30689]: 0x06c6dbf4:  0x00000180  0x045896c8  0x06743c80  0x00000000
grafana-server[30689]: 0x06c6dc04:  0x0962fc00  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30689]: 0x06c6dc14:  0x00000001  0x06743c80  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30689]: 0x06c6dc24:  0x06743c80  0x023bc5d8  0x0659d240  0x027f6b5a
grafana-server[30689]: 0x06c6dc34: <0x00000009  0x0671fc58  0x0662fb80  0x06743c80
grafana-server[30689]: 0x06c6dc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00000000  0x00000000
grafana-server[30689]: 0x06c6dc54:  0x00000000  0x00000000  0x00000000  0x00000000
grafana-server[30689]: 0x06c6dc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x0644213f  0x00000018  0x00000000
grafana-server[30689]: 0x06c6dc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30689]: 0x06c6dc84:  0x00000002  0x0646fca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30689]: 0x06c6dc94:  0x00000008  0x00000019  0x00000001  0x00000019
grafana-server[30689]: 0x06c6dca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30689]: runtime: unknown pc 0x27f6b5a
grafana-server[30689]: stack: frame={sp:0x6c6dc34, fp:0x0} stack=[0x6c4e000,0x6c6e000)
grafana-server[30689]: 0x06c6dbb4:  0x00000000  0x06c6dbc0  0x00000000  0x0671fc00
grafana-server[30689]: 0x06c6dbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06743c80  0x00000180  0x00000170
grafana-server[30689]: 0x06c6dbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30689]: 0x06c6dbe4:  0x76d5ab68  0x0000016c  0x00000170  0x76d4f088
grafana-server[30689]: 0x06c6dbf4:  0x00000180  0x045896c8  0x06743c80  0x00000000
systemd[1]: grafana-server.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
grafana-server[30689]: 0x06c6dc04:  0x0962fc00  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30689]: 0x06c6dc14:  0x00000001  0x06743c80  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30689]: 0x06c6dc24:  0x06743c80  0x023bc5d8  0x0659d240  0x027f6b5a
grafana-server[30689]: 0x06c6dc34: <0x00000009  0x0671fc58  0x0662fb80  0x06743c80
grafana-server[30689]: 0x06c6dc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00000000  0x00000000
grafana-server[30689]: 0x06c6dc54:  0x00000000  0x00000000  0x00000000  0x00000000
grafana-server[30689]: 0x06c6dc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x0644213f  0x00000018  0x00000000
grafana-server[30689]: 0x06c6dc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30689]: 0x06c6dc84:  0x00000002  0x0646fca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30689]: 0x06c6dc94:  0x00000008  0x00000019  0x00000001  0x00000019
grafana-server[30689]: 0x06c6dca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30689]: goroutine 105 [select]:
grafana-server[30689]: go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0x66350e0)
grafana-server[30689]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:243 +0xf4
grafana-server[30689]: go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1(0x66350e0)
grafana-server[30689]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:111 +0x4c
grafana-server[30689]: created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor
grafana-server[30689]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:109 +0x1e8
grafana-server[30689]: goroutine 111 [select]:
grafana-server[30689]: go.opencensus.io/stats/view.(*worker).start(0x66354a0)
grafana-server[30689]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xa0
grafana-server[30689]: created by go.opencensus.io/stats/view.init.0
grafana-server[30689]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x64
grafana-server[30689]: goroutine 135 [chan receive]:
grafana-server[30689]: github.com/golang/glog.(*loggingT).flushDaemon(0x4588760)
grafana-server[30689]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x70
grafana-server[30689]: created by github.com/golang/glog.init.0
systemd[1]: grafana-server.service: Failed with result 'exit-code'.
grafana-server[30689]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x214
grafana-server[30689]: trap    0x0
grafana-server[30689]: error   0x1
grafana-server[30689]: oldmask 0x0
grafana-server[30689]: r0      0x6743c80
grafana-server[30689]: r1      0x2f1624c
grafana-server[30689]: r2      0x6743c80
grafana-server[30689]: r3      0x1
grafana-server[30689]: r4      0x6d184
grafana-server[30689]: r5      0x22d96a8
grafana-server[30689]: r6      0x6743c80
grafana-server[30689]: r7      0x23bc5d8
grafana-server[30689]: r8      0x659d240
grafana-server[30689]: r9      0x180
grafana-server[30689]: r10     0x64000f0
grafana-server[30689]: fp      0x466769c
grafana-server[30689]: ip      0x6743dff
grafana-server[30689]: sp      0x6c6dc34
grafana-server[30689]: lr      0x2092c5c
grafana-server[30689]: pc      0x27f6b5a
grafana-server[30689]: cpsr    0x60000010
grafana-server[30689]: fault   0x0
systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 5.
systemd[1]: Stopped Grafana instance.
systemd[1]: grafana-server.service: Start request repeated too quickly.
systemd[1]: grafana-server.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Grafana instance.

cebatcheo · April 22, 2022, 2:48pm

I have the same issue. Can I ask how did you do the rollback to the version 8.4.7?

ebouxirot · April 22, 2022, 3:12pm

i simply use:

sudo apt install grafana=8.4.7

pretwick · April 22, 2022, 5:49pm

This is doing the trick. Thank you.

bondskin · April 23, 2022, 6:30am

same here, grafana-server 8.5.0 does not run on my Pi 4.
Rollback to 8.4.7 and all is good

pi@raspberrypidns:~ $ grafana-server
unexpected fault address 0x0
fatal error: fault
[signal SIGBUS: bus error code=0x1 addr=0x0 pc=0x27f6b5a]

goroutine 1 [running, locked to thread]:
runtime.throw({0x27eb2a9, 0x5})
/usr/local/go/src/runtime/panic.go:1198 +0x5c fp=0x5d2dc18 sp=0x5d2dc04 pc=0x5b244
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:732 +0xac fp=0x5d2dc30 sp=0x5d2dc18 pc=0x747f0
runtime: unexpected return pc for github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10 called from 0x9
stack: frame={sp:0x5d2dc34, fp:0x5d2dc5c} stack=[0x5d1e000,0x5d2e000)

florian1984 · April 24, 2022, 10:36am

Exactly same issue on 8.5.0, had to install old version again

schlaubstar · April 24, 2022, 1:51pm

Same here with a Raspberry PI 3B

SIGBUS: bus error
PC=0x27f6b5a m=0 sigcode=0

goroutine 1 [running, locked to thread]:
runtime: unknown pc 0x27f6b5a
stack: frame={sp:0x6c8fc34, fp:0x0} stack=[0x6c80000,0x6c90000)
0x06c8fbb4:  0x00000000  0x06c8fbc0  0x00000000  0x06a84700
0x06c8fbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06630900  0x000001800x00000170
0x06c8fbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sdWindowUpdate32+0x000000f0>  0x027f0100
0x06c8fbe4:  0x76d87378  0x0000016c  0x00000170  0x76d7a088
0x06c8fbf4:  0x00000180  0x045896c8  0x06630900  0x00000000
0x06c8fc04:  0x0c69bac0  0x02092c4c <github.com/grafana/grafana/pkg/services/arting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
0x06c8fc14:  0x00000001  0x06630900  0x0006d184 <runtime.doInit+0x00000114>  022d96a8
0x06c8fc24:  0x06630900  0x023bc5d8  0x068c0f00  0x027f6b5a
0x06c8fc34: <0x00000009  0x06a847bc  0x0669ba80  0x06630900
0x06c8fc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00054bb8 <rtime.(*consistentHeapStats).acquire+0x00000068>  0x00035d20 <runtime.(*mcache)efill+0x00000154>
0x06c8fc54:  0x064513ac  0x0000001d  0x0000001e  0x00000101
0x06c8fc64:  0x0000001e  0x00000000  0x0647e0bf  0x00000000
0x06c8fc74:  0x00000000  0x00000019  0x00000016  0x0000000b
0x06c8fc84:  0x0646fca0  0x0646fca4  0x0000001a  0x00033400 <runtime.findObjec0x00000058>
0x06c8fc94:  0x000396bc <runtime.SetFinalizer+0x00000064>  0x0003991c <runtimeetFinalizer+0x000002c4>  0x00000001  0x00000019
0x06c8fca4:  0x00000001  0x00000000  0x00000000  0x00000000
runtime: unknown pc 0x27f6b5a
stack: frame={sp:0x6c8fc34, fp:0x0} stack=[0x6c80000,0x6c90000)
0x06c8fbb4:  0x00000000  0x06c8fbc0  0x00000000  0x06a84700
0x06c8fbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06630900  0x000001800x00000170
0x06c8fbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sdWindowUpdate32+0x000000f0>  0x027f0100
0x06c8fbe4:  0x76d87378  0x0000016c  0x00000170  0x76d7a088
0x06c8fbf4:  0x00000180  0x045896c8  0x06630900  0x00000000
0x06c8fc04:  0x0c69bac0  0x02092c4c <github.com/grafana/grafana/pkg/services/arting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
0x06c8fc14:  0x00000001  0x06630900  0x0006d184 <runtime.doInit+0x00000114>  022d96a8
0x06c8fc24:  0x06630900  0x023bc5d8  0x068c0f00  0x027f6b5a
0x06c8fc34: <0x00000009  0x06a847bc  0x0669ba80  0x06630900
0x06c8fc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00054bb8 <rtime.(*consistentHeapStats).acquire+0x00000068>  0x00035d20 <runtime.(*mcache)efill+0x00000154>
0x06c8fc54:  0x064513ac  0x0000001d  0x0000001e  0x00000101
0x06c8fc64:  0x0000001e  0x00000000  0x0647e0bf  0x00000000
0x06c8fc74:  0x00000000  0x00000019  0x00000016  0x0000000b
0x06c8fc84:  0x0646fca0  0x0646fca4  0x0000001a  0x00033400 <runtime.findObjec0x00000058>
0x06c8fc94:  0x000396bc <runtime.SetFinalizer+0x00000064>  0x0003991c <runtimeetFinalizer+0x000002c4>  0x00000001  0x00000019
0x06c8fca4:  0x00000001  0x00000000  0x00000000  0x00000000

goroutine 81 [select]:
go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0x65249c
        /opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_an_processor.go:243 +0xf4
go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1(0x65249c0)
        /opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_an_processor.go:111 +0x4c
created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor
        /opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_an_processor.go:109 +0x1e8

goroutine 115 [select]:
go.opencensus.io/stats/view.(*worker).start(0x673df20)
        /opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:27+0xa0
created by go.opencensus.io/stats/view.init.0
        /opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:340x64

goroutine 122 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x4588760)
        /opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23d4e6c14b/glog.go:882 +0x70
created by github.com/golang/glog.init.0
        /opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23d4e6c14b/glog.go:410 +0x214

mdmoore · April 24, 2022, 1:56pm

me too.

/var/log/grafana $ grafana-server
unexpected fault address 0x27f6b5a
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x27f6b5a pc=0x27f6b5a]

goroutine 1 [running, locked to thread]:
runtime.throw({0x27eb2a9, 0x5})
/usr/local/go/src/runtime/panic.go:1198 +0x5c fp=0x549bc18 sp=0x549bc04 pc=0x5b244
runtime.sigpanic()

…etc…

mdebawue · April 25, 2022, 8:29am

After rolling back to a previous version you might want to put the package “on hold” to prevent that during the next update it is replaced with the current version (which does not run):

sudo apt-mark hold grafana

kayd · April 25, 2022, 8:13pm

Same here on Raspi 3B+, Pi OS Bullseye:
$ grafana-server
SIGBUS: bus error
PC=0x27f6b5a m=0 sigcode=0
…

linuxframp · April 26, 2022, 12:17pm

Same here. Raspberry4 running buster. Downgrade fixed the issue.

chphe · April 26, 2022, 12:51pm

The above also is reported as a bug: grafana-server doesn't start after recent update to 8.5.0 on openhabian · Issue #48130 · grafana/grafana · GitHub

I experienced the same problem. Currently there is no fix. Although it seems to work on the 64 bit version of Raspberry Pi OS (formerly known as Raspbian).

aliphm4 · April 26, 2022, 1:39pm

thanks man its work after 5hour

melori.arellano · April 26, 2022, 5:28pm

There is an open issue in github, our developers are actively working on it.

github.com/grafana/grafana

grafana-server doesn't start after recent update to 8.5.0 on openhabian

opened 03:24PM - 22 Apr 22 UTC

Diapolo

type/bug

**What happened**: Did an upgrade to 8.5.0 on openhabian and now grafana-serv…er can't start. > ● grafana-server.service - Grafana instance > Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled) > Active: failed (Result: exit-code) since Fri 2022-04-22 17:18:48 CEST; 1s ago > Docs: http://docs.grafana.org > Process: 7063 ExecStart=/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana->server.pid --packaging=deb cfg:default.paths.logs=${LOG_DIR} cfg:default.paths.data=${DATA_DIR} cfg:default.paths.plugins=${PLUGINS_DIR} cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR} (code=exited, stat > Main PID: 7063 (code=exited, status=2) > >Apr 22 17:18:48 openhabian systemd[1]: grafana-server.service: Main process exited, code=exited, status=2/INVALIDARGUMENT >Apr 22 17:18:48 openhabian systemd[1]: grafana-server.service: Failed with result 'exit-code'. >Apr 22 17:18:48 openhabian systemd[1]: grafana-server.service: Service RestartSec=100ms expired, scheduling restart. >Apr 22 17:18:48 openhabian systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 5. >Apr 22 17:18:48 openhabian systemd[1]: Stopped Grafana instance. >Apr 22 17:18:48 openhabian systemd[1]: grafana-server.service: Start request repeated too quickly. >Apr 22 17:18:48 openhabian systemd[1]: grafana-server.service: Failed with result 'exit-code'. >Apr 22 17:18:48 openhabian systemd[1]: Failed to start Grafana instance. **What you expected to happen**: A smooth upgrade ;). **How to reproduce it (as minimally and precisely as possible)**: If I try to restart via `sudo service grafana-server start`, this doesn't work. **Anything else we need to know?**: Was working fine with the latest 8.4.X version before the upgrade. **Environment**: - Grafana version: 8.5.0 - OS Grafana is installed on: openhabian

danieldiaz · April 27, 2022, 5:53pm

I had the same problem. I’ve downgraded to 8.4.7 and everything works ok!

guillaume1306 · April 28, 2022, 9:01am

Same error here:

-- Logs begin at Thu 2022-04-28 09:35:36 BST, end at Thu 2022-04-28 09:52:38 BST. --
Apr 28 09:35:50 boba systemd[1]: Started Grafana instance.
Apr 28 09:35:59 boba grafana-server[542]: unexpected fault address 0x27f6b5a
Apr 28 09:35:59 boba grafana-server[542]: fatal error: fault
Apr 28 09:35:59 boba grafana-server[542]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x27f6b5a pc=0x27f6b5a]
Apr 28 09:35:59 boba grafana-server[542]: goroutine 1 [running, locked to thread]:
Apr 28 09:35:59 boba grafana-server[542]: runtime.throw({0x27eb2a9, 0x5})
Apr 28 09:35:59 boba grafana-server[542]:         /usr/local/go/src/runtime/panic.go:1198 +0x5c fp=0x5f9bc18 sp=0x5f9bc04 pc=0x5b244
Apr 28 09:35:59 boba grafana-server[542]: runtime.sigpanic()
Apr 28 09:35:59 boba grafana-server[542]:         /usr/local/go/src/runtime/signal_unix.go:742 +0x1d0 fp=0x5f9bc30 sp=0x5f9bc18 pc=0x74914
Apr 28 09:35:59 boba grafana-server[542]: runtime: unexpected return pc for github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10 called from 0x9
Apr 28 09:35:59 boba grafana-server[542]: stack: frame={sp:0x5f9bc34, fp:0x5f9bc5c} stack=[0x5f7c000,0x5f9c000)
Apr 28 09:35:59 boba grafana-server[542]: 0x05f9bbb4:  0x0005d1a0 <runtime.gwrite+0x00000034>  0x04668d52  0x029efeac  0x00000001
Apr 28 09:35:59 boba grafana-server[542]: 0x05f9bbc4:  0x00000001  0x00000000  0x0005da4c <runtime.printstring+0x00000060>  0x00000002
Apr 28 09:35:59 boba grafana-server[542]: 0x05f9bbd4:  0x029efeac  0x00000001  0x00000001  0x00000000
Apr 28 09:35:59 boba grafana-server[542]: 0x05f9bbe4:  0x00000000  0x0005b450 <runtime.fatalthrow+0x00000044>  0x0005b244 <runtime.throw+0x0000005c>  0x05f9bbf4
Apr 28 09:35:59 boba grafana-server[542]: 0x05f9bbf4:  0x0005b468 <runtime.fatalthrow.func1+0x00000000>  0x058000f0  0x0005b244 <runtime.throw+0x0000005c>  0x05f9bc04
Apr 28 09:35:59 boba grafana-server[542]: 0x05f9bc04:  0x00074914 <runtime.sigpanic+0x000001d0>  0x05f9bc0c  0x0005b25c <runtime.throw.func1+0x00000000>  0x027eb2a9
Apr 28 09:35:59 boba grafana-server[542]: 0x05f9bc14:  0x00000005  0x02092c5c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000002c>  0x027eb2a9  0x00000005
Apr 28 09:35:59 boba grafana-server[542]: 0x05f9bc24:  0x059eaf00  0x027f6b5a  0x058000f0  0x02092c5c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000002c>
Apr 28 09:35:59 boba grafana-server[542]: 0x05f9bc34: <0x00000009  0x05dfa3dc  0x05cb2040  0x059eaf00

Raspbian Linux 10 - ARMv7 Processor rev 3 (v7l) @ 270 bMips, 4 cores

Topic		Replies	Views
Server start failure after 9.2.1 upgrade on Raspberry PI 3B Installation raspberry-pi	1	424	November 16, 2022
Grafana server not start (after 8.2.0 upgrade) Installation	14	9890	April 19, 2022
Failed to start Grafana instance Grafana	7	3469	July 31, 2022
Grafana-server fails to start Installation	18	54143	October 19, 2019
Grafana instance fails to start Installation	1	2779	October 10, 2021

Server start failure after 8.5.0 upgrade

Related topics