Using AWS Distro for OpenTelemetry in EKS on Fargate with AWS X-Ray¶

In this recipe we show you how to instrument a sample Go application and use AWS Distro for OpenTelemetry (ADOT) to ingest traces into AWS X-Ray and visualize the traces in Amazon Managed Grafana.

We will be setting up an Amazon Elastic Kubernetes Service (EKS) on AWS Fargate cluster and use an Amazon Elastic Container Registry (ECR) repository to demonstrate a complete scenario.

Note

This guide will take approximately 1 hour to complete.

Infrastructure¶

In the following section we will be setting up the infrastructure for this recipe.

Architecture¶

The ADOT pipeline enables us to use the ADOT Collector to collect traces from an instrumented app and ingest them into X-Ray:

ADOT default pipeline

Prerequisites¶

The AWS CLI is installed and configured in your environment.
You need to install the eksctl command in your environment.
You need to install kubectl in your environment.
You have Docker installed into your environment.
You have the aws-observability/aws-o11y-recipes repo cloned into your local environment.

Create EKS on Fargate cluster¶

Our demo application is a Kubernetes app that we will run in an EKS on Fargate cluster. So, first create an EKS cluster using the provided cluster_config.yaml.

Create your cluster using the following command:

eksctl create cluster -f cluster-config.yaml

Create ECR repository¶

In order to deploy our application to EKS we need a container repository. We will use the private ECR registry, but you can also use ECR Public, if you want to share the container image.

First, set the environment variables, such as shown here (substitute for your region):

export REGION="eu-west-1"
export ACCOUNTID=`aws sts get-caller-identity --query Account --output text`

You can use the following command to create a new ECR repository in your account:

aws ecr create-repository \
    --repository-name ho11y \
    --image-scanning-configuration scanOnPush=true \
    --region $REGION

Set up ADOT Collector¶

Download adot-collector-fargate.yaml and edit this YAML doc with the parameters described in the next steps.

kubectl apply -f adot-collector-fargate.yaml

Set up Managed Grafana¶

Set up a new workspace using the Amazon Managed Grafana – Getting Started guide and add X-Ray as a data source.

Signal generator¶

We will be using ho11y, a synthetic signal generator available via the sandbox of the recipes repository. So, if you haven't cloned the repo into your local environment, do now:

git clone https://github.com/aws-observability/aws-o11y-recipes.git

Build container image¶

Make sure that your ACCOUNTID and REGION environment variables are set, for example:

export REGION="eu-west-1"
export ACCOUNTID=`aws sts get-caller-identity --query Account --output text`

To build the ho11y container image, first change into the ./sandbox/ho11y/ directory and build the container image :

Note

The following build step assumes that the Docker daemon or an equivalent OCI image build tool is running.

docker build . -t "$ACCOUNTID.dkr.ecr.$REGION.amazonaws.com/ho11y:latest"

Push container image¶

Next, you can push the container image to the ECR repo you created earlier on. For that, first log in to the default ECR registry:

aws ecr get-login-password --region $REGION | \
    docker login --username AWS --password-stdin \
    "$ACCOUNTID.dkr.ecr.$REGION.amazonaws.com"

And finally, push the container image to the ECR repository you created, above:

docker push "$ACCOUNTID.dkr.ecr.$REGION.amazonaws.com/ho11y:latest"

Deploy signal generator¶

Edit x-ray-sample-app.yaml to contain your ECR image path. That is, replace ACCOUNTID and REGION in the file with your own values (overall, in three locations):

    # change the following to your container image:
    image: "ACCOUNTID.dkr.ecr.REGION.amazonaws.com/ho11y:latest"

Now you can deploy the sample app to your cluster using:

kubectl -n example-app apply -f x-ray-sample-app.yaml

End-to-end¶

Now that you have the infrastructure and the application in place, we will test out the setup, sending traces from ho11y running in EKS to X-Ray and visualize it in AMG.

Verify pipeline¶

To verify if the ADOT collector is ingesting traces from ho11y, we make one of the services available locally and invoke it.

First, let's forward traffic as so:

kubectl -n example-app port-forward svc/frontend 8765:80

With above command, the frontend microservice (a ho11y instance configured to talk to two other ho11y instances) is available in your local environment and you can invoke it as follows (triggering the creation of traces):

$ curl localhost:8765/
{"traceId":"1-6193a9be-53693f29a0119ee4d661ba0d"}

Tip

If you want to automate the invocation, you can wrap the curl call into a while true loop.

To verify our setup, visit the X-Ray view in CloudWatch where you should see something like shown below:

Screen shot of the X-Ray console in CW

Now that we have the signal generator set up and active and the OpenTelemetry pipeline set up, let's see how to consume the traces in Grafana.

Grafana dashboard¶

You can import an example dashboard, available via x-ray-sample-dashboard.json that looks as follows:

Screen shot of the X-Ray dashboard in AMG

Further, when you click on any of the traces in the lower downstreams panel, you can dive into it and view it in the "Explore" tab like so:

Screen shot of the X-Ray dashboard in AMG

From here, you can use the following guides to create your own dashboard in Amazon Managed Grafana:

That's it, congratulations you've learned how to use ADOT in EKS on Fargate to ingest traces.

Cleanup¶

First remove the Kubernetes resources and destroy the EKS cluster:

kubectl delete all --all && \
eksctl delete cluster --name xray-eks-fargate

Finally, remove the Amazon Managed Grafana workspace by removing it via the AWS console.