Bug when stacking time series with gaps

[schnurbel]

Hello,

there’s a bug when stacking time series with gaps: Series on top of the gap start at 0 instead on top of the series below.

Here are two time series without gaps, stacked perfectly on top of each other:

Here is another time series with a gap from 0:00 to 4:00:

If you stack them all, the Top series should be stacked on top of the Middle series, and if there’s no Middle, on top of Bottom like this:

But instead, it’s looking like this: (Top is starting at 0, obscuring Bottom)

This is the example dashboard’s source:

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "description": "Shows how stacking time series with gaps does not work correctly",
  "editable": true,
  "gnetId": null,
  "graphTooltip": 0,
  "id": 158,
  "links": [],
  "panels": [
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "prometheus",
      "fieldConfig": {
        "defaults": {
          "unit": "short"
        },
        "overrides": []
      },
      "fill": 5,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "hiddenSeries": false,
      "id": 2,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 0,
      "nullPointMode": "null as zero",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.5.15",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "exemplar": true,
          "expr": "vector(2)",
          "hide": false,
          "interval": "",
          "legendFormat": "Bottom (no gaps)",
          "queryType": "randomWalk",
          "refId": "A"
        },
        {
          "exemplar": true,
          "expr": "#Series with gap from 0:00 to 4:00\nmin_over_time( hour(vector(time()))[3h:1m] ) > 0",
          "hide": false,
          "interval": "",
          "legendFormat": "Middle (with gaps)",
          "refId": "B"
        },
        {
          "exemplar": true,
          "expr": "vector(3)",
          "hide": false,
          "interval": "",
          "legendFormat": "Top (no gaps)",
          "refId": "C"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Stacking",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "$$hashKey": "object:108",
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "$$hashKey": "object:109",
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    }
  ],
  "schemaVersion": 27,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-12h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "Stacking with Null Values",
  "uid": "rVaiFlw7k",
  "version": 3
}

When changing the nullPointMode from “null” to “null as zero”, the series are stacked correctly, but averages and minimums are not calculated correctly anymore because the zeros are taken into the calculation, so this is not an option for me.

I’m using Grafana version 7.5.15 (7dd69b2442).

I also tested this with Grafana 8.5.0 (6134e3cf3) on https://play.grafana.org/, and the problem is also there.
Even worse, when I migrated the old Graph panel to the new Time series panel, the Top series wasn’t displayed at all where Middle had a gap.

Is this here the right place to report such errors?

Best regards.

[leonsorokin]

yes, stacking with gaps is problematic in both the old Graph panel and the new TimeSeries panel.

i recently refactored the TimeSeries stacking implementation in 8.5.0 (https://github.com/grafana/grafana/pull/47373) and fixed some additional bugs in 8.5.1 (released today).

8.5.1 treats gaps as zero unless the gap is through the entire stack. 8.5.0 vs 8.5.1:

it’s an improvement since the top series’ fill does not disappear, but no longer produces the vertical drop-offs in the yellow’s gap region.

i’ll continue to think of a way to improve this further without also destroying performance for cases with 10+ series and several thousand datapoints each. currently i think it would require writing a separate (slower) renderer specifically for stacked cases.

UPDATE: there is hope! TimeSeries: stacking with gaps PoC · Issue #48473 · grafana/grafana · GitHub

[schnurbel]

Aww, great. Thanks a lot!