Viewing Logs¶

By default, we deploy a FluentBit daemon set in the cluster to collect worker logs for all namespaces. Logs are collected and exported to Amazon CloudWatch Logs, which enables you to centralize the logs from all of your systems, applications, and AWS services that you use, in a single, highly scalable service.

Further configuration options are available in the module documentation. This guide shows how you can leverage either CloudWatch Logs or Amazon Managed Grafana for your cluster and application logs.

Viewing Logs in CloudWatch Logs Insights¶

Navigate to CloudWatch, then go to "Logs Insights"

In the dropdown, select any of the logs that begin with "/aws/eks/single-new-eks-mixed-observability-accelerator" and run a query.

Example with "kubesystem" log group:

logs-query

Then you can view the results of your query:

logs-results

You can also find control plane logs in CloudWatch with Control Plane logs visible under the /cluster log group:

control-plane-logs

You can then filter by patterns within the logs to find specific events like so: control-plane-insights

Viewing Logs in Grafana¶

Using CloudWatch Logs as data source in Grafana¶

Follow the documentation to enable Amazon CloudWatch as a data source. Make sure to provide permissions.

All logs are delivered in the following CloudWatch Log groups naming pattern: /aws/eks/$PATTERN. Log streams follow {container-name}.{pod-name}. In Grafana, querying and analyzing logs is done with CloudWatch Logs Insights

Example - ADOT collector logs¶

Select one or many log groups and run the following query. The example below, queries AWS Distro for OpenTelemetry (ADOT) logs

fields @timestamp, log
| order @timestamp desc
| limit 100

logs-1

Example - Using time series visualizations¶

CloudWatch Logs syntax provide powerful functions to extract data from your logs. The stats() function allows you to calculate aggregate statistics with log field values. This is useful to have visualization on non-metric data from your applications.

In the example below, we use the following query to graph the number of metrics collected by the ADOT collector

fields @timestamp, log
| parse log /"#metrics": (?<metrics_count>\d+)}/
| stats avg(metrics_count) by bin(5m)
| limit 100

Tip

You can add logs in your dashboards with logs panel types or time series depending on your query results type.

logs-2

Warning

Querying CloudWatch logs will incur costs per GB scanned. Use small time windows and limits in your queries. Checkout the CloudWatch pricing page for more info.