Grafana Series - Unified Display - 4 - AWS CloudWatch Data Source

This article was last updated on: May 17, 2026 am

Series Articles

AWS CloudWatch Data Source

For AWS CloudWatch, the key consideration is the 4 different authentication methods:

  • AWS SDK Default
  • IAM Role
  • AK&SK
  • Credentials file

The recommended approach is to use IAM Role authentication, which avoids the risk of credential leakage.

However, it’s important to note that to read CloudWatch metrics and EC2 tags, instances, regions, and alarms, you must grant Grafana the necessary permissions via IAM. You can attach these permissions to the IAM role or IAM user configured in your AWS authentication.

Example IAM policies are as follows:

Metrics-only:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingMetricsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "cloudwatch:DescribeAlarmsForMetric",
        "cloudwatch:DescribeAlarmHistory",
        "cloudwatch:DescribeAlarms",
        "cloudwatch:ListMetrics",
        "cloudwatch:GetMetricData",
        "cloudwatch:GetInsightRuleReport"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
      "Effect": "Allow",
      "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}

Logs-only:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingLogsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:GetLogGroupFields",
        "logs:StartQuery",
        "logs:StopQuery",
        "logs:GetQueryResults",
        "logs:GetLogEvents"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
      "Effect": "Allow",
      "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}

Metrics and Logs:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingMetricsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "cloudwatch:DescribeAlarmsForMetric",
        "cloudwatch:DescribeAlarmHistory",
        "cloudwatch:DescribeAlarms",
        "cloudwatch:ListMetrics",
        "cloudwatch:GetMetricData",
        "cloudwatch:GetInsightRuleReport"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingLogsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:GetLogGroupFields",
        "logs:StartQuery",
        "logs:StopQuery",
        "logs:GetQueryResults",
        "logs:GetLogEvents"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
      "Effect": "Allow",
      "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}

Cross-account observability:

1
2
3
4
5
6
7
8
9
10
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": ["oam:ListSinks", "oam:ListAttachedLinks"],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

AWS CloudWatch Data Source Configuration Examples

Configuration examples for the different authentication methods are as follows:

AWS SDK (default):

1
2
3
4
5
6
7
apiVersion: 1
datasources:
  - name: CloudWatch
    type: cloudwatch
    jsonData:
      authType: default
      defaultRegion: eu-west-2

Using a credentials file:

1
2
3
4
5
6
7
8
9
10
apiVersion: 1

datasources:
  - name: CloudWatch
    type: cloudwatch
    jsonData:
      authType: credentials
      defaultRegion: eu-west-2
      customMetricsNamespaces: 'CWAgent,CustomNameSpace'
      profile: secondary

Using AK&SK:

1
2
3
4
5
6
7
8
9
10
11
apiVersion: 1

datasources:
  - name: CloudWatch
    type: cloudwatch
    jsonData:
      authType: keys
      defaultRegion: eu-west-2
    secureJsonData:
      accessKey: '<your access key>'
      secretKey: '<your secret key>'

Using AWS SDK Default with an IAM Role ARN to assume:

1
2
3
4
5
6
7
8
apiVersion: 1
datasources:
  - name: CloudWatch
    type: cloudwatch
    jsonData:
      authType: default
      assumeRoleArn: arn:aws:iam::123456789012:root
      defaultRegion: eu-west-2

CloudWatch Built-in Dashboards

The built-in CloudWatch dashboards are not very useful. It’s recommended to use monitoringartist/grafana-aws-cloudwatch-dashboards instead.

Creating Alert Queries

Alerts require queries that return numeric data, and CloudWatch Logs supports this type of query. For example, you can enable alerting by using the stats command.

Here is a valid query for alerting on messages that contain the text “Exception”:

1
2
3
filter @message like /Exception/
    | stats count(*) as exceptionCount by bin(1h)
    | sort exceptionCount desc

Cross-Account Observability

The CloudWatch plugin enables you to monitor and troubleshoot applications across regions and accounts. With cross-account observability, you can seamlessly search, visualize, and analyze metrics and logs without worrying about account boundaries.

To use this feature, configure a monitoring and source account under CloudWatch settings in the AWS Console, then add the necessary IAM permissions as described above.

Observability
#K8S #CloudNative #Monitoring #AWS #Observability #Grafana #Metrics
Grafana Series - Unified Display - 4 - AWS CloudWatch Data Source
https://e-whisper.com/posts/42013/
Author
east4ming
Posted on
May 5, 2023
Licensed under