Skip to content

Tracing on Amazon EKS

Distributed tracing helps you have end-to-end visibility between transactions in distributed nodes.

How tracing works in v3

The eks-monitoring module configures traces collection depending on the collector profile:

Profile Traces support Exporter
self-managed-amp Yes (toggle with enable_tracing) OTLP → AWS X-Ray
cloudwatch-otlp Yes (always enabled) OTLP → AWS X-Ray
managed-metrics No (metrics only) —

The OpenTelemetry Collector receives traces via the OTLP protocol (gRPC on port 4317, HTTP on port 4318) and exports them to AWS X-Ray.

Note

To disable tracing in the self-managed-amp profile, set enable_tracing = false in the module configuration.

Instrumentation

Applications send traces to the OTel Collector using the OpenTelemetry SDK. Point your application's OTLP exporter at the collector service:

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://otel-collector.otel-collector.svc.cluster.local:4317"

Note

To learn more about instrumenting with OpenTelemetry, visit the OpenTelemetry documentation for your programming language.

Example: Go sample application

Let's use a sample application that is already instrumented with the OpenTelemetry SDK.

git clone https://github.com/aws-observability/aws-otel-community.git
cd aws-otel-community/sample-apps/go-sample-app

Building and publishing the container image

docker build -t go-sample-app .

docker buildx build -t go-sample-app . --platform=linux/amd64

Publish to Amazon ECR:

export ECR_REPOSITORY_URI=$(aws ecr create-repository --repository go-sample-app --query repository.repositoryUri --output text)
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ECR_REPOSITORY_URI
docker tag go-sample-app:latest "${ECR_REPOSITORY_URI}:latest"
docker push "${ECR_REPOSITORY_URI}:latest"

Deploying on Amazon EKS

eks.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-sample-app
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: go-sample-app
  template:
    metadata:
      labels:
        app: go-sample-app
    spec:
      containers:
        - name: go-sample-app
          image: "${ECR_REPOSITORY_URI}:latest" # replace with your ECR URI
          imagePullPolicy: Always
          env:
          - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
            value: otel-collector.otel-collector.svc.cluster.local:4317
          resources:
            limits:
              cpu:  300m
              memory: 300Mi
            requests:
              cpu: 100m
              memory: 180Mi
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: go-sample-app
  namespace: default
spec:
  type: ClusterIP
  selector:
    app: go-sample-app
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080

Deploy and test:

kubectl apply -f eks.yaml
kubectl port-forward deployment/go-sample-app 8080:8080

curl http://localhost:8080/
curl http://localhost:8080/outgoing-http-call
curl http://localhost:8080/aws-sdk-call

Visualizing traces

Open your Amazon Managed Grafana workspace and add the AWS X-Ray data source. In the Grafana Explorer view, select the X-Ray data source and use Query Type: Trace List to browse traces.

You can also view traces in the CloudWatch console, which provides a service map and trace detail views. If your logs are stored in CloudWatch Logs, the trace detail page can correlate logs automatically.

Resources