Exporting CloudWatch Metric Streams via Firehose and AWS Lambda to Amazon Managed Service for Prometheus
In this recipe we show you how to instrument a CloudWatch Metric Stream and use Kinesis Data Firehose and AWS Lambda to ingest metrics into Amazon Managed Service for Prometheus (AMP).
We will be setting up a stack using AWS Cloud Development Kit (CDK) to create a Firehose Delivery Stream, Lambda, and a S3 Bucket to demonstrate a complete scenario.
This guide will take approximately 30 minutes to complete.
Infrastructure
In the following section we will be setting up the infrastructure for this recipe.
CloudWatch Metric Streams allow forwarding of the streaming metric data to a HTTP endpoint or S3 bucket.
Prerequisites
- The AWS CLI is installed and configured in your environment.
- The AWS CDK Typescript is installed in your environment.
- Node.js and Go.
- The repo has been cloned to your local machine. The code for this project is under
/sandbox/CWMetricStreamExporter
.
Create an AMP workspace
Our demo application in this recipe will be running on top of AMP. Create your AMP Workspace via the following command:
aws amp create-workspace --alias prometheus-demo-recipe
Ensure your workspace has been created with the following command:
aws amp list-workspaces
For more details check out the AMP Getting started guide.
Install dependencies
From the root of the aws-o11y-recipes repository, change your directory to CWMetricStreamExporter via the command:
cd sandbox/CWMetricStreamExporter
This will now be considered the root of the repo, going forward.
Change directory to /cdk
via the following command:
cd cdk
Install the CDK dependencies via the following command:
npm install
Change directory back to the root of the repo, and then change directory
to /lambda
using following command:
cd lambda
Once in the /lambda
folder, install the Go dependencies using:
go get
All the dependencies are now installed.
Modify config file
In the root of the repo, open config.yaml
and modify the AMP workspace URL
by replacing the {workspace}
with the newly created workspace id, and the
region your AMP workspace is in.
For example, modify the following with:
AMP:
remote_write_url: "https://aps-workspaces.us-east-2.amazonaws.com/workspaces/{workspaceId}/api/v1/remote_write"
region: us-east-2
Change the names of the Firehose Delivery Stream and S3 Bucket to your liking.
Deploy stack
Once the config.yaml
has been modified with the AMP workspace ID, it is time
to deploy the stack to CloudFormation. To build the CDK and the Lambda code,
in the root of the repo run the following commend:
npm run build
This build step ensures that the Go Lambda binary is built, and deploys the CDK to CloudFormation.
Accept the following IAM changes to deploy the stack:
Verify that the stack has been created by running the following command:
aws cloudformation list-stacks
A stack by the name CDK Stack
should have been created.
Create CloudWatch stream
Navigate to the CloudWatch consoloe, for example
https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#metric-streams:streamsList
and click "Create metric stream".
Select the metics needed, either all metrics of only from selected namespaces.
Configure the Metric Stream by using an existing Firehose which was created by the CDK. Change the output format to JSON instead of OpenTelemetry 0.7. Modify the Metric Stream name to your liking, and click "Create metric stream":
To verify the Lambda function invocation, navigate to the Lambda console
and click the function KinesisMessageHandler
. Click the Monitor
tab and Logs
subtab, and under Recent Invocations
there should be entries of the Lambda function being triggered.
It may take upto 5 minutes for invocations to show in the Monitor tab.
That is it! Congratulations, your metrics are now being streamed from CloudWatch to Amazon Managed Service for Prometheus.
Cleanup
First, delete the CloudFormation stack:
cd cdk
cdk destroy
Remove the AMP workspace:
aws amp delete-workspace --workspace-id \
`aws amp list-workspaces --alias prometheus-sample-app --query 'workspaces[0].workspaceId' --output text`
Last but not least, remove the CloudWatch Metric Stream by removing it from the console.