EC2 Monitoring and Observability
Introduction
Continuous Monitoring & Observability increases agility, improves customer experience and reduces risk of the cloud environment. According to Wikipedia, Observability is a measure of how well internal states of a system can be inferred from the knowledge of its external outputs. The term observability itself originates from the field of control theory, where it basically means that you can infer the internal state of the components in a system by learning about the external signals/outputs it is producing.
The difference between Monitoring and Observability is that Monitoring tells you whether a system is working or not, while Observability tells you why the system isn’t working. Monitoring is usually a reactive measure whereas the goal of Observability is to be able to improve your Key Performance Indicators in a proactive manner. A system cannot be controlled or optimized unless it is observed. Instrumenting workloads through collection of metrics, logs, or traces and gaining meaningful insights & detailed context using the right monitoring and observability tools help customers control and optimize the environment.
AWS enables customers to transform from monitoring to observability so that they can have full end-to-end service visibility. In this article we focus on Amazon Elastic Compute Cloud (Amazon EC2) and the best practices for improving the monitoring and observability of the service in AWS Cloud environment through AWS native and open-source tools.
Amazon EC2
Amazon Elastic Compute Cloud (Amazon EC2) is a highly scalable compute platform in Amazon Web Services (AWS) Cloud. Amazon EC2 eliminates the need for up front hardware investment, so customers can develop and deploy applications faster while paying just for what they use. Some of the key features that EC2 provide are virtual computing environments called Instances, pre-configured templates of Instances called Amazon Machine Images, various configurations of resources like CPU, Memory, Storage and Networking capacity available as Instance Types.
Monitoring and Observability using AWS Native Tools
Amazon CloudWatch
Amazon CloudWatch is a monitoring and management service that provides data and actionable insights for AWS, hybrid, and on-premises applications and infrastructure resources. CloudWatch collects monitoring and operational data in the form of logs, metrics, and events. It also provides a unified view of AWS resources, applications, and services that run on AWS and on-premises servers. CloudWatch helps you gain system-wide visibility into resource utilization, application performance, and operational health.
Unified CloudWatch Agent
The Unified CloudWatch Agent is an open-source software under the MIT license which supports most operating systems utilizing x86-64 and ARM64 architectures. The CloudWatch Agent helps collect system-level metrics from Amazon EC2 Instances & on-premise servers in a hybrid environment across operating systems, retrieve custom metrics from applications or services and collect logs from Amazon EC2 instances and on-premises servers.
Installing CloudWatch Agent on Amazon EC2 Instances
Command Line Install
The CloudWatch Agent can be installed through the command line. The required package for various architectures and various operating systems are available for download. Create the necessary IAM role which provides permissions for CloudWatch agent to read information from the Amazon EC2 instance and write it to CloudWatch. Once the required IAM role is created, you can install and run the CloudWatch agent on the required Amazon EC2 Instance.
Documentation: Installing the CloudWatch agent using the command line
AWS Observability Workshop: Setup and install CloudWatch agent
Installation through AWS Systems Manager
The CloudWatch Agent can also be installed through AWS Systems Manager. Create the necessary IAM role which provides permissions for CloudWatch agent to read information from the Amazon EC2 instance and write it to CloudWatch & communicate with AWS Systems Manager. Before installing the CloudWatch agent on the EC2 instances, install or update the SSM agent on the required EC2 instances. The CloudWatch agent can be downloaded through the AWS Systems Manager. JSON Configuration file can be created to specify what metrics (including custom metrics), logs are to be collected. Once the required IAM role is created & the configuration file is created, you can install and run the CloudWatch agent on the required Amazon EC2 Instances.
Documentation: Installing the CloudWatch agent using AWS Systems Manager
AWS Observability Workshop: Install CloudWatch agent using AWS Systems Manager Quick Setup
Related Blog Article: Amazon CloudWatch Agent with AWS Systems Manager Integration – Unified Metrics & Log Collection for Linux & Windows
YouTube Video: Collect Metrics and Logs from Amazon EC2 instances with the CloudWatch Agent
Installing CloudWatch Agent on on-premise servers in hybrid environment.
In hybrid customer environments, where servers are on-premisis as well as in the cloud. A similar approach of can be taken to accomplish unified observability in Amazon CloudWatch. The CloudWatch agent can be directly downloaded from Amazon S3 or through AWS Systems Manager. Create an IAM User for the on-premise server to send data to Amazon CloudWatch. Install and Start the Agent on the on-premise servers.
Documentation: Installing the CloudWatch agent on on-premises servers