Exporting CloudWatch Metric Streams via Firehose and AWS Lambda to Amazon Managed Service for Prometheus

In this recipe we show you how to instrument a CloudWatch Metric Stream and use Kinesis Data Firehose and AWS Lambda to ingest metrics into Amazon Managed Service for Prometheus (AMP).

We will be setting up a stack using AWS Cloud Development Kit (CDK) to create a Firehose Delivery Stream, Lambda, and a S3 Bucket to demonstrate a complete scenario.

Note

This guide will take approximately 30 minutes to complete.

Infrastructure

In the following section we will be setting up the infrastructure for this recipe.

CloudWatch Metric Streams allow forwarding of the streaming metric data to a HTTP endpoint or S3 bucket.

Prerequisites

• The AWS CLI is installed and configured in your environment.
• The AWS CDK Typescript is installed in your environment.
• Node.js and Go.
• The repo has been cloned to your local machine. The code for this project is under /sandbox/CWMetricStreamExporter.

Create an AMP workspace

Our demo application in this recipe will be running on top of AMP. Create your AMP Workspace via the following command:

aws amp create-workspace --alias prometheus-demo-recipe

Ensure your workspace has been created with the following command:

aws amp list-workspaces

Info

For more details check out the AMP Getting started guide.

Install dependencies

From the root of the aws-o11y-recipes repository, change your directory to CWMetricStreamExporter via the command:

cd sandbox/CWMetricStreamExporter

This will now be considered the root of the repo, going forward.

Change directory to /cdk via the following command:

cd cdk

Install the CDK dependencies via the following command:

npm install

Change directory back to the root of the repo, and then change directory to /lambda using following command:

cd lambda

Once in the /lambda folder, install the Go dependencies using:

go get

All the dependencies are now installed.

Modify config file

In the root of the repo, open config.yaml and modify the AMP workspace URL by replacing the {workspace} with the newly created workspace id, and the region your AMP workspace is in.

For example, modify the following with:

AMP:
    remote_write_url: "https://aps-workspaces.us-east-2.amazonaws.com/workspaces/{workspaceId}/api/v1/remote_write"
    region: us-east-2

Change the names of the Firehose Delivery Stream and S3 Bucket to your liking.

Deploy stack

Once the config.yaml has been modified with the AMP workspace ID, it is time to deploy the stack to CloudFormation. To build the CDK and the Lambda code, in the root of the repo run the following commend:

npm run build

This build step ensures that the Go Lambda binary is built, and deploys the CDK to CloudFormation.

Accept the following IAM changes to deploy the stack:

Verify that the stack has been created by running the following command:

aws cloudformation list-stacks

A stack by the name CDK Stack should have been created.

Create CloudWatch stream

Navigate to the CloudWatch consoloe, for example https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#metric-streams:streamsList and click "Create metric stream".

Select the metics needed, either all metrics of only from selected namespaces.

Configure the Metric Stream by using an existing Firehose which was created by the CDK. Change the output format to JSON instead of OpenTelemetry 0.7. Modify the Metric Stream name to your liking, and click "Create metric stream":

To verify the Lambda function invocation, navigate to the Lambda console and click the function KinesisMessageHandler. Click the Monitor tab and Logs subtab, and under Recent Invocations there should be entries of the Lambda function being triggered.

Note

It may take upto 5 minutes for invocations to show in the Monitor tab.

That is it! Congratulations, your metrics are now being streamed from CloudWatch to Amazon Managed Service for Prometheus.

Cleanup

First, delete the CloudFormation stack:

cd cdk
cdk destroy

Remove the AMP workspace:

aws amp delete-workspace --workspace-id \
    `aws amp list-workspaces --alias prometheus-sample-app --query 'workspaces[0].workspaceId' --output text`

Last but not least, remove the CloudWatch Metric Stream by removing it from the console.