Setting Up Grafana Annotations using CloudWatch Events and AWS Lambda

March 23, 2017 |

Grafana is an open source metric analytics & visualization suite. It is most commonly used for visualizing time series data for infrastructure and application analytics. Annotations, in Grafana, provide a way to mark points on the graph with rich events. When you hover over an annotation you can get title, tags, and text information for the event. Using Annotations you can easily piece together the effects of Application Activity on your AWS Instance Events.

Setting up visualization templates helps in the quick creation of dashboards, eliminating the need for manual creations. Below we will learn how the architecture of such a system looks.

Understanding CloudWatch Events

Amazon CloudWatch Events delivers a near real-time stream of system events that describe changes in Amazon Web Services (AWS) resources to AWS Lambda functions, Amazon SNS topics, Amazon SQS queues, streams in Amazon Kinesis Streams, or built-in targets. Using simple rules that you can quickly set up, you can match events and route them to one or more target functions or streams.

Events – An event indicates a change in your AWS environment.

Targets – A target processes events.

Rules – A rule matches incoming events and routes them to targets for processing.

The Solution

One of our clients, in the Education Sector, is an American publishing, education and media company known for publishing, selling, and distributing books and educational materials for schools, teachers, parents, and children. They needed visualization and analysis creation for three use cases – Business Events, CI/CD Events, and Operational Events.  REAN Cloud provided evaluations and recommendations for data ingestion and InfluxDB configuration.

The below architecture illustrates how CloudWatch Events triggers a Lambda function, and writes AWS API Event data into InfluxDB.

AWS Lambda blog visual

We determined CloudWatch Event data to not be useful as a graph, as the data is mostly text (event type/instance id). This data would be better displayed using Annotations over an existing graph, such as CPU Load or API calls/min.

Understanding the Architecture

CloudWatch Events

In the Event Selector, for CloudWatch Events, select “AWS API Call” from the dropdown. Then under show advanced options, edit the json manually to the block below. Alternately, you can just choose the API calls you want to watch.



“detail-type”: [

“AWS API Call via CloudTrail”


“detail”: {

“eventSource”: [



“eventName”: [










In the Targets section, Select “Lambda Function” and choose your <Your Lambda Function Name> lambda function.

Lambda Function

This Lambda function we created only accounts for ec2 events, but can be easily modified to work with any other type of CloudWatch Event.

Annotation Queries

Here, we will see how to setup Annotations, a Grafana dashboarding feature.

We want to get all the “RunInstances” events from the series “cloudtrail”.

SELECT * FROM “cloudtrail” where event = ‘RunInstances’ AND $timeFilter

Lambda blog visual 2

We want to get all the “TerminateInstances” events from the series “cloudtrail”.

SELECT * FROM “cloudtrail” where event = ‘TerminateInstances’ AND $timeFilter

Lambda blog visual 3

Note: The “AND $timeFilter” part of the query. This is important because it limits your query to the time range specified in Grafana. If this is left out, the query will look at all events, and depending on the size of the series, this may be very long.


By putting such a system in place, our client, from the education sector, was able to successfully correlate Application Activity to AWS Instance Events for a better understanding. For example, the image below illustrates a “TerminateInstances” event annotated over a graph of Average CPU Load. In this example, you can see that the Instance Terminations did not affect the application’s average CPU load.

Lambda blog visual 4

If you would like to know more about this or similar solutions, and how to architect on AWS using best practices, please reach us at

Other Blog Posts


Top 5 Reasons to Utilize Cloud Computing in Financial Services
Read More

Is Migrating to the Cloud Safe for Financial Sector Companies?
Read More

REAN Cloud is one of the few AWS Premier Partners to achieve both AWS DevOps Competency and MSP Designation
Read More

7 Ways DevOps Can Save Your Company…Time and Money
Read More
Request Consultation
close slider