Single Cluster Open Source Observability - Istio Monitoring¶

Objective¶

Service Meshes are an integral part of the Kubernetes environment that enables secure, reliable, and observable communication. Istio is an open-source service mesh that provides advanced network features without requiring any changes to the application code. These capabilities include service-to-service authentication, monitoring, and more.

Istio generates detailed telemetry for all service communications within a mesh. This telemetry provides observability of service behavior, thereby empowering operators to troubleshoot, maintain, and optimize their applications. These features don’t impose additional burdens on service developers. To monitor service behavior, Istio generates metrics for all service traffic in, out, and within an Istio service mesh. These metrics provide information on behaviors, like traffic volume, traffic error rates, and request-response latency.

In addition to monitoring the behavior of services within a mesh, it’s essential to monitor the behavior of the mesh itself. Istio components export metrics which provides insights into the health and function of the mesh control plane.

This pattern configures an Amazon Elastic Kubernetes Service (Amazon EKS) cluster with Istio as a service mesh, Amazon Managed service for Prometheus, and Amazon Managed Grafana for monitoring your Istio Control and Data plane metrics

Prerequisites¶

Ensure that you have installed the following tools on your machine.

Deploying¶

Please follow the Deploying instructions of the New EKS Cluster Open Source Observability Accelerator pattern, except for step 7, where you need to replace "context" in ~/.cdk.json with the following:

  "context": {
    "fluxRepository": {
      "name": "grafana-dashboards",
      "namespace": "grafana-operator",
      "repository": {
        "repoUrl": "https://github.com/aws-observability/aws-observability-accelerator",
        "name": "grafana-dashboards",
        "targetRevision": "main",
        "path": "./artifacts/grafana-operator-manifests/eks/infrastructure"
      },
      "values": {
        "GRAFANA_CLUSTER_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/main/artifacts/grafana-dashboards/eks/infrastructure/cluster.json",
        "GRAFANA_KUBELET_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/main/artifacts/grafana-dashboards/eks/infrastructure/kubelet.json",
        "GRAFANA_NSWRKLDS_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/main/artifacts/grafana-dashboards/eks/infrastructure/namespace-workloads.json",
        "GRAFANA_NODEEXP_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/main/artifacts/grafana-dashboards/eks/infrastructure/nodeexporter-nodes.json",
        "GRAFANA_NODES_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/main/artifacts/grafana-dashboards/eks/infrastructure/nodes.json",
        "GRAFANA_WORKLOADS_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/main/artifacts/grafana-dashboards/eks/infrastructure/workloads.json",
        "GRAFANA_ISTIO_CP_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/v0.2.0/artifacts/grafana-dashboards/eks/istio/istio-control-plane-dashboard.json",
        "GRAFANA_ISTIO_MESH_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/v0.2.0/artifacts/grafana-dashboards/eks/istio/istio-mesh-dashboard.json",
        "GRAFANA_ISTIO_PERF_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/v0.2.0/artifacts/grafana-dashboards/eks/istio/istio-performance-dashboard.json",
        "GRAFANA_ISTIO_SERVICE_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/v0.2.0/artifacts/grafana-dashboards/eks/istio/istio-service-dashboard.json"
      },
      "kustomizations": [
        {
          "kustomizationPath": "./artifacts/grafana-operator-manifests/eks/infrastructure"
        },
        {
          "kustomizationPath": "./artifacts/grafana-operator-manifests/eks/istio"
        }
      ]
    },
    "Istio.pattern.enabled": true
  }

Once completed the rest of the Deploying steps, you can move on with the deployment of the Istio workload.

Visualization¶

1. Grafana dashboards¶

Go to the Dashboards panel of your Grafana workspace. You will see a list of Istio dashboards under the Observability Accelerator Dashboards

Open one of the Istio dasbhoards and you will be able to view its visualization

2. Amazon Managed Service for Prometheus rules and alerts¶

Open the Amazon Managed Service for Prometheus console and view the details of your workspace. Under the Rules management tab, you will find new rules deployed.

Note

To setup your alert receiver, with Amazon SNS, follow this documentation

Deploy an example application to visualize metrics¶

In this section we will deploy Istio's Bookinfo sample application and extract metrics using the AWS OpenTelemetry collector. When downloading and configuring istioctl, there are samples included in the Istio package directory. The deployment files for Bookinfo are found in the samples folder. Additional details can be found on Istio's Getting Started documentation

1. Deploy the Bookinfo Application¶

Using the AWS CLI, configure kubectl so you can connect to your EKS cluster. Update for your region and EKS cluster name
```
aws eks update-kubeconfig --region <enter-your-region> --name <cluster-name>
```
Label the default namespace for automatic Istio sidecar injection
```
kubectl label namespace default istio-injection=enabled
```
Navigate to the Istio folder location. For example, if using Istio v1.18.2 in Downloads folder:
```
cd ~/Downloads/istio-1.18.2
```

Deploy the Bookinfo sample application

kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml

Connect the Bookinfo application with the Istio gateway

kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

Validate that there are no issues with the Istio configuration
```
istioctl analyze
```

Get the DNS name of the load balancer for the Istio gateway

GATEWAY_URL=$(kubectl get svc istio-ingressgateway -n istio-system -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

2. Generate traffic for the Istio Bookinfo sample application¶

For the Bookinfo sample application, visit http://$GATEWAY_URL/productpage in your web browser. To see trace data, you must send requests to your service. The number of requests depends on Istio’s sampling rate and can be configured using the Telemetry API. With the default sampling rate of 1%, you need to send at least 100 requests before the first trace is visible. To send a 100 requests to the productpage service, use the following command:

for i in $(seq 1 100); do curl -s -o /dev/null "http://$GATEWAY_URL/productpage"; done

3. Explore the Istio dashboards¶

Log back into your Amazon Managed Grafana workspace and navigate to the dashboard side panel. Click on the Observability Accelerator Dashboards folder and open the Istio Service Dashboard. Use the Service dropdown menu to select the reviews.default.svc.cluster.local service. This gives details about metrics for the service, client workloads (workloads that are calling this service), and service workloads (workloads that are providing this service).

04CBB260-EE0E-405E-BD23-EBCF8333A29D

Explore the Istio Control Plane, Mesh, and Performance dashboards as well.

Control Plane 68AC86D7-4959-4527-A723-A19E8FD9E8F5_1_105_c

AD2E46A3-4BC0-4F5A-BEA0-27D2136E6C06

Mesh 9B04E4D9-6A47-4E4F-AF42-E3A2426B344C

Performance

9429081E-E4CC-4674-81BD-DB7F0B15BC87

Teardown¶

You can teardown the whole CDK stack with the following command:

make pattern single-new-eks-opensource-observability destroy