How to import/export dashboards from a enviroment to another

jgarciadelanoceda · February 10, 2022, 5:56pm

Hello,

I ask how you can import in a programmatic way dashboards to another environment.

Is related to this issue in Github [Grafana] 8.3.X Copy a Dashboard to another environment by copying it from Json Model · Issue #43440 · grafana/grafana · GitHub

I run a Powershell to Get all Dashboards and Folders and be able to import them in another enviroment(We used to do it for Grafana Version 4.6 and since the datasource was a simple string in the dashboard json, it give no problem).

I paste here the script


$user = 'xxxx'
$pass = 'xxxx'
$port = 8080
$server = 'xxxxxxx'
$outputPath = "$PSScriptRoot\output"
$outputFolderPath = "$([System.IO.Path]::Combine($outputPath,"001_Folder"))";
$outputDashboardPath = "$([System.IO.Path]::Combine($outputPath,"002_Dashboard"))";
$searchForAlarmsDashboards = $false;

rm -Path $outputPath -Recurse
New-Item -ItemType Directory -Force -Path $outputPath > $null;
New-Item -ItemType Directory -Force -Path $outputFolderPath > $null;
New-Item -ItemType Directory -Force -Path $outputDashboardPath > $null;
$Headers = @{
    Authorization = "Basic $([System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes("$($user):$($pass)")))"
}
$folders = Invoke-RestMethod -Uri "http://$($server):$($port)/api/folders" -Method Get -Headers $Headers -ContentType 'application/json';
$folders | % {
    $folder = New-Object System.Object;
    $folder | Add-Member -NotePropertyName uid -NotePropertyValue $_.uid;
    $folder | Add-Member -NotePropertyName title -NotePropertyValue $_.title;
    $folder | ConvertTo-Json | Out-File "$($outputFolderPath)\$($_.uid).folderjson";
}
$apiDashboardUri = "http://$($server):$($port)/api/dashboards";
$dashboards = Invoke-RestMethod -Uri "http://$($server):$($port)/api/search/?type=dash-db" -Method Get -Headers $Headers -ContentType 'application/json';
$dashboards | ? {($searchForAlarmsDashboards -and ($_.tags.Contains("alarms"))) -or (-not $searchForAlarmsDashboards -and (-not $_.tags.Contains("alarms")))} | % {
    $dashboard = Invoke-RestMethod -Uri "$($apiDashboardUri)/uid/$($_.uid)" -Method Get -Headers $Headers -ContentType "application/json";
    $panelBody = New-Object System.Object;
    $panelBody | Add-Member -NotePropertyName dashboard -NotePropertyValue $dashboard.dashboard;
    $panelBody | Add-Member -NotePropertyName folderUid -NotePropertyValue $dashboard.meta.folderUid;
    $panelBody | Add-Member -NotePropertyName overwrite -NotePropertyValue $true;
    $panelBody | ConvertTo-Json -Depth 100 | Out-File "$($outputDashboardPath)\$($_.uid).json";
}

A part of the panelBody has an uid that identifies the Datasource (That usually changes between the different enviroments).
Do I have to export the databases as well in order to change the uid of the equivalent databases in the new enviroment? (Before creating the Dashboards).

This is the only solution that I see but I see that in the database api you can change the uid easily so I think that it is the only solution

$user = 'xxxxxx'
$pass = 'xxxxx'
$port = 8080
$server = 'xxxxxx'
$outputPath = "$PSScriptRoot\output"
$outputFolderPath = "$([System.IO.Path]::Combine($outputPath,"001_Folder"))";
$outputDbPath = "$([System.IO.Path]::Combine($outputPath,"002_DB"))";
$outputDashboardPath = "$([System.IO.Path]::Combine($outputPath,"003_Dashboard"))";
$searchForAlarmsDashboards = $false;
$mainUrl = "http://$($server):$($port)"
rm -Path $outputPath -Recurse
New-Item -ItemType Directory -Force -Path $outputPath > $null;
New-Item -ItemType Directory -Force -Path $outputDbPath > $null;
New-Item -ItemType Directory -Force -Path $outputFolderPath > $null;
New-Item -ItemType Directory -Force -Path $outputDashboardPath > $null;
$Headers = @{
    Authorization = "Basic $([System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes("$($user):$($pass)")))"
}
$folders = Invoke-RestMethod -Uri "$($mainUrl)/api/folders" -Method Get -Headers $Headers -ContentType 'application/json';
$folders | % {
    $folder = New-Object System.Object;
    $folder | Add-Member -NotePropertyName uid -NotePropertyValue $_.uid;
    $folder | Add-Member -NotePropertyName title -NotePropertyValue $_.title;
    $folder | ConvertTo-Json | Out-File "$($outputFolderPath)\$($_.uid).folder";
}
$dbs = Invoke-RestMethod -Uri "$($mainUrl)/api/datasources" -Method Get -Headers $Headers -ContentType 'application/json';
$dbs | % {
    $db = New-Object System.Object;
    $db | Add-Member -NotePropertyName name -NotePropertyValue $_.name;
    $db | Add-Member -NotePropertyName uid -NotePropertyValue $_.uid;
    $db | ConvertTo-Json | Out-File "$($outputDbPath)\$($_.name).db";
}
$dashboards = Invoke-RestMethod -Uri "$($mainUrl)/api/search/?type=dash-db" -Method Get -Headers $Headers -ContentType 'application/json';
$dashboards | ? {($searchForAlarmsDashboards -and ($_.tags.Contains("alarms"))) -or (-not $searchForAlarmsDashboards -and (-not $_.tags.Contains("alarms")))} | % {
    $dashboard = Invoke-RestMethod -Uri "$($mainUrl)/api/dashboards/uid/$($_.uid)" -Method Get -Headers $Headers -ContentType "application/json";
    $panelBody = New-Object System.Object;
    $panelBody | Add-Member -NotePropertyName dashboard -NotePropertyValue $dashboard.dashboard;
    $panelBody | Add-Member -NotePropertyName folderUid -NotePropertyValue $dashboard.meta.folderUid;
    $panelBody | Add-Member -NotePropertyName overwrite -NotePropertyValue $true;
    $panelBody | ConvertTo-Json -Depth 100 | Out-File "$($outputDashboardPath)\$($_.uid).json";
}

jgarciadelanoceda · February 15, 2022, 12:31pm

Finally this is the script that I use for BackUp the Dashboards and Folders from the master environment:

$user = 'xxxxxx';
$pass = 'xxxxxx';
$port = 8080;
$server = 'xxxx';
$outputPath = "$PSScriptRoot\output";
$outputFolderPath = "$([System.IO.Path]::Combine($outputPath,"001_Folder"))";
$outputDbPath = "$([System.IO.Path]::Combine($outputPath,"002_DB"))";
$outputDashboardPath = "$([System.IO.Path]::Combine($outputPath,"003_Dashboard"))";
$searchForAlarmsDashboards = $false;
$mainUrl = "http://$($server):$($port)";
rm -Path $outputPath -Recurse -ErrorAction SilentlyContinue;
New-Item -ItemType Directory -Force -Path $outputPath > $null;
if(-not $searchForAlarmsDashboards){
    New-Item -ItemType Directory -Force -Path $outputDbPath > $null;
    New-Item -ItemType Directory -Force -Path $outputFolderPath > $null;
}
$changelogStringBuilder = [System.Text.StringBuilder]::new();
$changelogStringBuilder.AppendLine("# $packageName") > $null;
$changelogStringBuilder.AppendLine("Package of all default grafana panels") > $null;
$changelogStringBuilder.AppendLine("") > $null;
$changelogStringBuilder.AppendLine("## [$nugetVersion] - $(Get-Date -format yyyy-MM-dd)") > $null;
$changelogStringBuilder.AppendLine("### Changed") > $null;
$changelogStringBuilder.AppendLine("- $nugetVersionReleaseNotes") > $null;
$changelogStringBuilder.AppendLine("### Added") > $null;
New-Item -ItemType Directory -Force -Path $outputDashboardPath > $null;
$Headers = @{
    Authorization = "Basic $([System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes("$($user):$($pass)")))"
}
if(-not $searchForAlarmsDashboards){
    Write-Host "Getting folders";
    $folders = Invoke-RestMethod -Uri "$($mainUrl)/api/folders" -Method Get -Headers $Headers -ContentType 'application/json';
    $folders | % {
        $folder = New-Object System.Object;
        $folder | Add-Member -NotePropertyName uid -NotePropertyValue $_.uid;
        $folder | Add-Member -NotePropertyName title -NotePropertyValue $_.title;
        $folder | ConvertTo-Json | Out-File ([System.IO.Path]::Combine($outputFolderPath,"$($_.uid).folder"));
        $changelogStringBuilder.AppendLine("- Folder $($_.title)") > $null;
    }
    Write-Host "Getting Dbs";
    $dbs = Invoke-RestMethod -Uri "$($mainUrl)/api/datasources" -Method Get -Headers $Headers -ContentType 'application/json';
    $dbs | % {
        $db = New-Object System.Object;
        $db | Add-Member -NotePropertyName name -NotePropertyValue $_.name;
        $db | Add-Member -NotePropertyName uid -NotePropertyValue $_.uid;
        $db | ConvertTo-Json | Out-File ([System.IO.Path]::Combine($outputDbPath,"$($_.name).db"));
    }
}
Write-Host "Getting Dashboards";
$dashboards = Invoke-RestMethod -Uri "$($mainUrl)/api/search/?type=dash-db" -Method Get -Headers $Headers -ContentType 'application/json';
$dashboards | ? {($searchForAlarmsDashboards -and ($_.tags.Contains("alarms"))) -or (-not $searchForAlarmsDashboards -and (-not $_.tags.Contains("alarms")))} | % {
    $dashboard = Invoke-RestMethod -Uri "$($mainUrl)/api/dashboards/uid/$($_.uid)" -Method Get -Headers $Headers -ContentType "application/json";
    $changelogStringBuilder.AppendLine("- Dashboard slug: $($dashboard.meta.slug), updated: $($dashboard.meta.updated), updatedBy: $($dashboard.meta.updatedBy), version: $($dashboard.meta.version) uid: $($_.uid)") > $null;
	$dashboard | ConvertTo-Json -Depth 100 | Out-File ([System.IO.Path]::Combine($outputDashboardPath,"$($_.uid).json"));
}
Add-Content ([System.IO.Path]::Combine($outputPath,"changelog.txt")) $changelogStringBuilder.ToString();

And this is the script that I have for Importing them:

$inputPath = "$PSScriptRoot"

if(-not(Test-Path -Path $inputPath)){
    Write-Error "This path: $inputPath not found"
    return;
}
$Headers = @{
	Authorization = "Basic $([System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes("$($GrafanaAdminUser):$($GrafanaAdminPassword)")))"
}
$mainUrl = "http://$($GrafanaNotificationChannelsServer):$($GrafanaHttpPort)";
$health = Invoke-RestMethod -Uri "$($mainUrl)/api/health" -Method Get -Headers $Headers -ContentType "application/json";
$versionGrafana = $null;
if([version]::TryParse($health.version, [ref] $versionGrafana)){
	Write-Host "Version in Grafana is $versionGrafana";
	$minVersionWithFolders = [version]::new(8,0,0);
	if($versionGrafana.CompareTo($minVersionWithFolders) -ge 0){
		Write-Host "Version in Grafana is $versionGrafana which is greater than $minVersionWithFolders";
		#Reading Folder its necesary to create the dashboard in the correct folder
		$foldersInGrafana = Invoke-RestMethod -Uri "$($mainUrl)/api/folders" -Method Get -Headers $Headers -ContentType 'application/json';
		ls -Path "$inputPath\*.folder" -Recurse | % {
			$folder = [System.IO.File]::ReadAllText($_.FullName) | ConvertFrom-Json;
			$folderInGrafana = $foldersInGrafana | ? {$_.title -eq $folder.title};
			if($folderInGrafana -eq $null){
				Write-Host "Creating folder $($folder.title)";
				Invoke-RestMethod -Uri "$($mainUrl)/api/folders" -Method Post -Headers $Headers -ContentType "application/json" -Body ($folder | ConvertTo-Json);
			}
			else{
				Write-Host "Folder already exist in grafana";
			}
		}
		#Reading datasources its necesary to update uid of the datasource because its written in the dashboard
		$dbsInGrafana = Invoke-RestMethod -Uri "$($mainUrl)/api/datasources" -Method Get -Headers $Headers -ContentType 'application/json';
		ls -Path "$inputPath\*.db" -Recurse | % {
			$db = [System.IO.File]::ReadAllText($_.FullName) | ConvertFrom-Json;
			$dbInGrafana = $dbsInGrafana | ? {$_.name -eq $db.name};
			if($dbInGrafana -eq $null){
				Write-Host "Db $($db.name) does not exist in Grafana";
			}
			else{
				Write-Host "Db $($db.name) does exist in Grafana updating uid";
				$dbInGrafana.uid = $db.uid;
				Invoke-RestMethod -Uri "$($mainUrl)/api/datasources/$($dbInGrafana.id)" -Method Put -Headers $Headers -ContentType "application/json" -Body ($dbInGrafana | ConvertTo-Json);
			}
		}
	}
	else{
		Write-Host "Version in Grafana is $versionGrafana which is lower than $minVersionWithFolders";
	}
	ls -Path "$inputPath\*.json" -Recurse | % {
		$dashboard = [System.IO.File]::ReadAllText($_.FullName) | ConvertFrom-Json;
		$dashboardBody = New-Object System.Object;
		$dashboardBody | Add-Member -NotePropertyName dashboard -NotePropertyValue $dashboard.dashboard;
		if($dashboard.meta.folderUid -ne $null){
			$dashboardBody | Add-Member -NotePropertyName folderUid -NotePropertyValue $dashboard.meta.folderUid;
		}
		$dashboardBody | Add-Member -NotePropertyName overwrite -NotePropertyValue $true;
        $dashboardBody.dashboard.id = $null; #Eliminamos los id, para poder importar.
        Write-Host "Creating dashboard:$($dashboardBody.dashboard.title) Folder:$($dashboardBody.folderUid)";
		Invoke-RestMethod -Method Post -Headers $Headers -ContentType "application/json" -Uri "$($mainUrl)/api/dashboards/db" -Body ($dashboardBody | ConvertTo-Json -Depth 100 );
		Write-Host "Updating dashboard:$($dashboardBody.dashboard.title) Folder:$($dashboardBody.folderUid)";
        ## Invoke twice to make the alarms start
		Invoke-RestMethod -Method Post -Headers $Headers -ContentType "application/json" -Uri "$($mainUrl)/api/dashboards/db" -Body ($dashboardBody | ConvertTo-Json -Depth 100 );
	}
}
else{
	write-Error "Not parsed version in Grafana";
}

I think that both are valid, but I need to know if this is the correct form of importing and exporting dashboards and folders.

Dbs are being created earlier in the process so when you import the dashboards you have to make sure that all the dbs have the same uid

wilfridrussodivito · December 1, 2022, 1:02pm

Hi,
Thanks for your example. I’m new in powershell and grafana.
I need to export all dashboard but when i launch this :
$folders = Invoke-RestMethod -Uri “$($GrafanaURL)” -Method Get -Headers $Headers -ContentType ‘application/json’; , $folders is empty and with a curl i have this message : unauthorized.

Can you help me ?
Many thanks.

Will

jgarciadelanoceda · December 1, 2022, 3:36pm

Hi @wilfridrussodivito
It seems that the user and password are not correct.

So you have a Unauthorized response. Take into account that this was a question and it was never answered (It is not an official solution), if you want to just copy a dashboard to other environment and you have few environments this could not be the solution.

But the company I worked for had over 50 panels and over 20 platforms and we wanted a way to automatize it.

wilfridrussodivito · December 2, 2022, 1:32pm

Hi,

Many thanks.
Is it possible to loop on each grafana’s folder and for each folder, export in json all dashboard ?
i tried to adapt your code but it don’t work.

Thanks:

yosiasz · December 2, 2022, 1:47pm

Are you trying to move dashboards to a new server?

wilfridrussodivito · December 2, 2022, 2:04pm

No. The goal it’s to export all folders (and all dashboards) from grafana in json file (for the dashboards) and import all this in Team Foundatiom server.
On grafana, the structure is like this :
folder 1

dashboard 1
*dashboard 2
folder 2
dashboard 1
*dashboard 2
etc etc

jgarciadelanoceda · December 2, 2022, 2:06pm

Hello @yosiasz,
It is not to move the dashboards to a new server what I originally wanted, what I wanted is to replicate the same dashboards and folders structure in another Grafana deployment.

The company I worked for had over 20 platforms were we did Monitoring using Grafana and InfluxDb, we wanted to have the same dashboards in all of them

yosiasz · December 2, 2022, 2:06pm

Here is what I wrote in python. The basic idea can be done in any other scripting language

github.com

yosiasz/grafanadashboards/blob/main/src/automate/exportalldashboards.py

import requests
import json
from dotenv import load_dotenv
load_dotenv()
import os


TOKEN=os.getenv('TOKEN')
URL=os.getenv('URL')
headers= {
'Accept': 'application/json',
'Content-Type': 'application/json',
'Authorization': 'Bearer {}'.format(TOKEN)
}

r = requests.get(url = URL, headers = headers)

for d in r.json():
    uid = d.get('uid')
    title = d.get('title')

This file has been truncated. show original

yosiasz · December 2, 2022, 2:09pm

If you want to replicate the same structure it sounds like moving to me

In that case what I do is copy the data and conf folder to the secondary server and restart grafana - done

wilfridrussodivito · December 2, 2022, 2:11pm

Thanks,
but how do you put in URL variable : https://yourgrafana/ with api or something else ?
As Javier said, I wanted is to replicate the same dashboards and folders structure in TFS.
Loop on folders is ok, but loop on each dashboard of each folder : i can’t do it

jgarciadelanoceda · December 2, 2022, 2:13pm

Could you explain the solution a little bit more?

I think that this could be interesting for the rest of the community

yosiasz · December 2, 2022, 2:16pm

So we go 2 asks going here. One for TFS and the other for replicating same folder structure in another grafana instance. Lets not conflate those.

For tfs check this out

In the dashboard api, for a specitic dashboard, you can get folderId and folderGuid, you can find the folder name from these 2 values and create the folder accordingly and follower by saving the dashboard json to that folder programatically

Another instance of grafana
Copy data and conf folder

wilfridrussodivito · December 2, 2022, 2:18pm

Yes : i need to import the structure of Grafana (only folders and dashboards) in Team Foundation Server. For the moment i loop on Grafana’s folders and for each folder, i create it on Windows like this :
$folders | % {
$DashboardsDir = $outputFolderPath + "" + $.title
New-Item -ItemType Directory -Force -Name $.title -Path $outputFolderPath;
}
On this loop, i need to get all dashboards of the current folders and create one json file per dashboards found on the current folder but i don’t know how to do this.
Have you got an idea ?

yosiasz · December 2, 2022, 2:21pm

Go the other way around.

Loop on dashboards, find folder name by folderId or folderGuid of dashboard
create folder if it does not exist
dump dashboard json into its folder

wilfridrussodivito · December 2, 2022, 2:23pm

I will try this, thanks so much

jgarciadelanoceda · December 2, 2022, 2:25pm

This was my original Idea
So @wilfridrussodivito check the dashboards importing scripts, because it is what it does (Although if you had already the same folderId with different Uid in the destination I think that the solution could have a problem)

I do not have access to the environments anymore so I am not updating the solution

It is also important to say that the databases in the destination environment were deployed in a previous step of the pipeline

yosiasz · December 2, 2022, 2:26pm

Importing to a different instance of grafana via script?

jgarciadelanoceda · December 2, 2022, 2:27pm

Yes

It was my only goal I had

yosiasz · December 2, 2022, 2:30pm

Why complicate our lives when we can use that time to write this to enjoy a nice bavarian creme filled donut?

Keep it simple. Copy data and conf folder to secondary instance. Couple of lines of code, very easy to maintain, wont break if underlying api changes

I would not recommend the complicated loop to json and import approach, way too complicated

Topic		Replies	Views
Grafana api powershell Developers & API api	12	1034	December 1, 2022
Import dashboard from file via API Developers & API api , dashboards-as-code , python	3	14321	September 29, 2022
Exporting and Importing Dashboards by Name Grafana api	1	260	May 7, 2022
Import Dashboard using provisioning Grafana	1	1547	July 3, 2019
Use API to bulk import/export dashboards Grafana	0	542	March 29, 2019

How to import/export dashboards from a enviroment to another

Related topics