Tracing on Amazon EKS

Distributed tracing helps you have end-to-end visibility between transactions in distributed nodes. The eks-monitoring module is configured by default to collect traces into AWS X-Ray.

The AWS Distro for OpenTelemetry collector is configured to receive traces in the OTLP format (OTLP receiver), using the OpenTelemetry SDK or auto-instrumentation agents.

Note

To disable the tracing configuration, XrayAdotAddOn for Mixed and Open Source Observability Accelerators from the CDK Observability Patterns

Instrumentation

Let's take a sample application that is already instrumented with the OpenTelemetry SDK.

Note

To learn more about instrumenting with OpenTelemetry, please visit the OpenTelemetry documentation for your programming language.

Cloning the repo

git clone https://github.com/aws-observability/aws-otel-community.git
cd aws-otel-community/sample-apps/go-sample-app

Highlighting code sections

Deploying on Amazon EKS

Using the sample application, we will build a container image, create and push an image to Amazon ECR. We will use the go-sample-app Kubernetes manifest to deploy to EKS cluster.

Warning

The following steps require that you have an EKS cluster ready. To deploy an EKS cluster, please visit our example.

Building container image

amd64 linuxcross platform build

docker build -t go-sample-app .

docker buildx build -t go-sample-app . --platform=linux/amd64

Publishing on Amazon ECR

using docker

export ECR_REPOSITORY_URI=$(aws ecr create-repository --repository go-sample-app --query repository.repositoryUri --output text)
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ECR_REPOSITORY_URI
docker tag go-sample-app:latest "${ECR_REPOSITORY_URI}:latest"
docker push "${ECR_REPOSITORY_URI}:latest"

Deploying on Amazon EKS

apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-sample-app
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: go-sample-app
  template:
    metadata:
      labels:
        app: go-sample-app
    spec:
      containers:
        - name: go-sample-app
          image: "${ECR_REPOSITORY_URI}:latest" # make sure to replace this variable
          imagePullPolicy: Always
          env:
          - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
            value: 'http://adot-collector.adot-collector-kubeprometheus.svc.cluster.local:4317'
          resources:
            limits:
              cpu:  300m
              memory: 300Mi
            requests:
              cpu: 100m
              memory: 180Mi
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: go-sample-app
  namespace: default
  labels:
    app: go-sample-app
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
  selector:
    app: go-sample-app

Deploying and testing

With the Kubernetes manifest ready, run:

kubectl apply -f eks.yaml

You should see the pods running with the command:

kubectl get pods
NAME                              READY   STATUS    RESTARTS        AGE
go-sample-app-67c48ff8c6-bdw74    1/1     Running   0               4s
go-sample-app-67c48ff8c6-t6k2j    1/1     Running   0               4s

To simulate some traffic you can forward the service port to your local host and test a few queries

kubectl port-forward deployment/go-sample-app 8080:8080

Test a few endpoints

curl http://localhost:8080/
curl http://localhost:8080/outgoing-http-call
curl http://localhost:8080/aws-sdk-call
curl http://localhost:8080/outgoing-sampleapp

Visualizing traces

As this is a basic example, the service map doesn't have a lot of nodes, but this shows you how to setup tracing in your application and deploying it on Amazon EKS using our OSS observability patterns.

With Flux and Grafana Operator, the OSS pattern configures an AWS X-Ray data source on your provided Grafana workspace. Open the Grafana explorer view and select the X-Ray data source. If you type the query below, and select Trace List for Query Type, you should see the list of traces occured in the selected timeframe.

You can add the service map to a dashboard, for example a service focused dashboard. You can click on any of the traces to view a node map and the traces details.

There is a button that can take you the CloudWatch console to view the same data. If your logs are stored on CloudWatch Logs, this page can present all the logs in the trace details page. The CloudWatch Log Group name should be added to the trace as an attribute. Read more about this in our One Observability Workshop

Resources

• AWS Observability Best Practices
• One Observability Workshop
• AWS Distro for OpenTelemetry documentation
• AWS X-Ray user guide
• OpenTelemetry documentation