Using AWS Distro for OpenTelemetry in EKS on Fargate with AWS X-Ray

In this recipe we show you how to instrument a sample Go application and use AWS Distro for OpenTelemetry (ADOT) to ingest traces into AWS X-Ray and visualize the traces in Amazon Managed Grafana.

We will be setting up an Amazon Elastic Kubernetes Service (EKS) on AWS Fargate cluster and use an Amazon Elastic Container Registry (ECR) repository to demonstrate a complete scenario.

note

This guide will take approximately 1 hour to complete.

Infrastructure

In the following section we will be setting up the infrastructure for this recipe.

Architecture

The ADOT pipeline enables us to use the ADOT Collector to collect traces from an instrumented app and ingest them into X-Ray:

Prerequisites

• The AWS CLI is installed and configured in your environment.
• You need to install the eksctl command in your environment.
• You need to install kubectl in your environment.
• You have Docker installed into your environment.
• You have the aws-observability/aws-o11y-recipes repo cloned into your local environment.

Create EKS on Fargate cluster

Our demo application is a Kubernetes app that we will run in an EKS on Fargate cluster. So, first create an EKS cluster using the provided cluster_config.yaml.

Create your cluster using the following command:

eksctl create cluster -f cluster-config.yaml

Create ECR repository

In order to deploy our application to EKS we need a container repository. We will use the private ECR registry, but you can also use ECR Public, if you want to share the container image.

First, set the environment variables, such as shown here (substitute for your region):

export REGION="eu-west-1"
export ACCOUNTID=`aws sts get-caller-identity --query Account --output text`

You can use the following command to create a new ECR repository in your account:

aws ecr create-repository \
    --repository-name ho11y \
    --image-scanning-configuration scanOnPush=true \
    --region $REGION

Set up ADOT Collector

Download adot-collector-fargate.yaml and edit this YAML doc with the parameters described in the next steps.

kubectl apply -f adot-collector-fargate.yaml

Set up Managed Grafana

Set up a new workspace using the Amazon Managed Grafana – Getting Started guide and add X-Ray as a data source.

Signal generator

We will be using ho11y, a synthetic signal generator available via the sandbox of the recipes repository. So, if you haven't cloned the repo into your local environment, do now:

git clone https://github.com/aws-observability/aws-o11y-recipes.git

Build container image

Make sure that your ACCOUNTID and REGION environment variables are set, for example:

export REGION="eu-west-1"
export ACCOUNTID=`aws sts get-caller-identity --query Account --output text`

To build the ho11y container image, first change into the ./sandbox/ho11y/ directory and build the container image :

note

The following build step assumes that the Docker daemon or an equivalent OCI image build tool is running.

docker build . -t "$ACCOUNTID.dkr.ecr.$REGION.amazonaws.com/ho11y:latest"

Push container image

Next, you can push the container image to the ECR repo you created earlier on. For that, first log in to the default ECR registry:

aws ecr get-login-password --region $REGION | \
    docker login --username AWS --password-stdin \
    "$ACCOUNTID.dkr.ecr.$REGION.amazonaws.com"

And finally, push the container image to the ECR repository you created, above:

docker push "$ACCOUNTID.dkr.ecr.$REGION.amazonaws.com/ho11y:latest"

Deploy signal generator

Edit x-ray-sample-app.yaml to contain your ECR image path. That is, replace ACCOUNTID and REGION in the file with your own values (overall, in three locations):

    # change the following to your container image:
    image: "ACCOUNTID.dkr.ecr.REGION.amazonaws.com/ho11y:latest"

Now you can deploy the sample app to your cluster using:

kubectl -n example-app apply -f x-ray-sample-app.yaml

End-to-end

Now that you have the infrastructure and the application in place, we will test out the setup, sending traces from ho11y running in EKS to X-Ray and visualize it in AMG.

Verify pipeline

To verify if the ADOT collector is ingesting traces from ho11y, we make one of the services available locally and invoke it.

First, let's forward traffic as so:

kubectl -n example-app port-forward svc/frontend 8765:80

With above command, the frontend microservice (a ho11y instance configured to talk to two other ho11y instances) is available in your local environment and you can invoke it as follows (triggering the creation of traces):

$ curl localhost:8765/
{"traceId":"1-6193a9be-53693f29a0119ee4d661ba0d"}

tip

If you want to automate the invocation, you can wrap the curl call into a while true loop.

To verify our setup, visit the X-Ray view in CloudWatch where you should see something like shown below:

Now that we have the signal generator set up and active and the OpenTelemetry pipeline set up, let's see how to consume the traces in Grafana.

Grafana dashboard

You can import an example dashboard, available via x-ray-sample-dashboard.json that looks as follows:

Further, when you click on any of the traces in the lower downstreams panel, you can dive into it and view it in the "Explore" tab like so:

From here, you can use the following guides to create your own dashboard in Amazon Managed Grafana:

• User Guide: Dashboards
• Best practices for creating dashboards

That's it, congratulations you've learned how to use ADOT in EKS on Fargate to ingest traces.

Cleanup

First remove the Kubernetes resources and destroy the EKS cluster:

kubectl delete all --all && \
eksctl delete cluster --name xray-eks-fargate

Finally, remove the Amazon Managed Grafana workspace by removing it via the AWS console.