Amazon Managed Prometheus Cross Account Scraping

Amazon Managed Service for Prometheus provides a fully managed, agent less scraper, or collector, that automatically discovers and pulls Prometheus-compatible metrics. You don't have to manage, install, patch, or maintain agents or scrapers. An Amazon Managed Service for Prometheus collector provides reliable, stable, highly available, automatically scaled collection of metrics for your Amazon EKS cluster. Amazon Managed Service for Prometheus managed collectors work with Amazon EKS clusters, including EC2 and Fargate.

An Amazon Managed Service for Prometheus collector creates an Elastic Network Interface (ENI) per subnet specified when creating the scraper. The collector scrapes the metrics through these ENIs, and uses remote_write to push the data to your Amazon Managed Service for Prometheus workspace using a VPC endpoint. The scraped data never travels on the public internet.

To create a scraper in a cross-account setup when your Amazon EKS cluster from which you want to collect metrics is in a different account (Source Account) from the Amazon Managed Service for Prometheus workspace (Target Account), use the procedure below.

High Level Architecture

Figure 1: AMP Managed Collector Cross Account Scraping, Collector Infrastructure is Completely Managed by AWS

In this architecture we create scrapers in the account where the EKS workload exists. The scrapers can assume a role in the target account in order to push data to the AMP workspace in the target account.

1. In the source account, create a role arn:aws:iam::account_id_source:role/Source with STS::AssumeRole permissions and add the following trust policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "scraper.aps.amazonaws.com"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "ArnEquals": {
                    "aws:SourceArn": "$SCRAPER_ARN"
                },
                "StringEquals": {
                    "AWS:SourceAccount": "$ACCOUNT_ID"
                }
            }
        }
    ]
}

You also need an assume role permissions policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "$TARGET_ACCOUNT_ROLE_ARN"
        }
    ]
}

warning

We have to make the IAM constructs before we can actually make the scraper. Therefore at this point the $SCRAPER_ARN is just a placeholder field. After we create the scraper we will go back and update it. The $TARGET_ACCOUNT_ROLE_ARN also does not exist until step 2 is completed.

2. On every combination of source (Amazon EKS cluster) and target (Amazon Managed Service for Prometheus workspace), you need to create a role in the TARGET account of arn:aws:iam::account_id_target:role/Target and add the following trust policy with managed permissions policy for AmazonPrometheusRemoteWriteAccess.

{
  "Effect": "Allow",
  "Principal": {
     "AWS": "arn:aws:iam::account_id_source:role/Source"
  },
  "Action": "sts:AssumeRole",
  "Condition": {
     "StringEquals": {
        "sts:ExternalId": "$SCRAPER_ARN"
      }
  }
}

warning

$SCRAPER_ARN is still just a placeholder. We will update the value after creating the scraper.

3. Create a scraper in the source account (where the EKS cluster exists) with the --role-configuration option.

aws amp create-scraper \
  --source eksConfiguration="{clusterArn='arn:aws:eks:us-west-2:$SOURCE_ACCOUNT_ID:cluster/$CLUSTER_NAME',subnetIds=[$EKS_SUBNET_IDS]}" \
  --scrape-configuration configurationBlob=<base64-encoded-blob> \
  --destination ampConfiguration="{workspaceArn='arn:aws:aps:us-west-2:$TARGET_ACCOUNT_ID:workspace/$TARGET_AMP_WORKSPACE_ID'}"\
  --role-configuration '{"sourceRoleArn":"arn:aws:iam::$SOURCE_ACCOUNT_ID:role/Source", "targetRoleArn":"arn:aws:iam::$TARGET_ACCOUNT_ID:role/Target"}'

warning

Make sure you fill in the $VARIABLES with the values specific to you.

4. Validate the scraper creation (This can take ~20 minutes) and take note of the scraper ARN.

aws amp list-scrapers
{
    "scrapers": [
        {
            "scraperId": "scraper-id",
            "arn": "arn:aws:aps:us-west-2:account_id_source:scraper/scraper-id",
            "roleArn": "arn:aws:iam::account_id_source:role/aws-service-role/scraper.aps.amazonaws.com/AWSServiceRoleForAmazonPrometheusScraperInternal_cc319052-41a3-4",
            "status": {
                "statusCode": "ACTIVE"
            },
            "createdAt": "2024-10-29T16:37:58.789000+00:00",
            "lastModifiedAt": "2024-10-29T16:55:17.085000+00:00",
            "tags": {},
            "source": {
                "eksConfiguration": {
                    "clusterArn": "arn:aws:eks:us-west-2:account_id_source:cluster/xarw",
                    "securityGroupIds": [
                        "sg-security-group-id",
                        "sg-security-group-id"
                    ],
                    "subnetIds": [
                        "subnet-subnet_id"
                    ]
                }
            },
            "destination": {
                "ampConfiguration": {
                    "workspaceArn": "arn:aws:aps:us-west-2:account_id_target:workspace/ws-workspace-id"
                }
            }
        }

5. Go back and update the trust polcies created in steps 1 and 2 with the scraper ARN value from the command in step 4.