Live:CloudOps Webinars & Hands-on Workshops ·Register ↗
跳到主要内容

Using Amazon S3 server access logs for Security, Compliance & Auditing

Overview​

Organizations running workloads on AWS often store their most critical data in Amazon S3, from application data and analytics datasets to backups, logs, and regulated content. With that responsibility comes a fundamental question: how do I know who is accessing my data, what they're doing with it, and whether those actions comply with my security policies?

Amazon S3 server access logs answer that question by recording requests made to your buckets, successful or failed, authenticated or anonymous, with HTTP-level detail not available in other common log sources. With Amazon S3 server access logs now available as a native data source for CloudWatch unified data store, those logs are automatically delivered to CloudWatch without the need to build custom ETL pipelines or transformation workflows. CloudWatch unified data store transforms the raw Amazon S3 server access log data into a structured, queryable format, enabling immediate use of CloudWatch Logs Insights, metric filters, Contributor Insights rules, and Amazon S3 Tables integration. Combined with AWS CloudTrail management events, server access logs become a powerful, cost-effective pillar of your Amazon S3 security and compliance posture.

The Customer Challenge​

The Challenge

Many teams face a common set of questions when monitoring Amazon S3 data-plane activity:

  • Visibility gaps: CloudTrail management events (included at no additional cost) cover bucket-level configuration changes but don't show who is reading, writing, or deleting individual objects.
  • Cost at scale: High-volume buckets (millions of requests per day) need visibility without unbounded cost growth.
  • Pipeline complexity: Traditional server access log setups required bucket policies, lifecycle management, and custom transformation workflows to make the logs queryable in tools like CloudWatch. CloudWatch unified data store now eliminates the need to build custom ETL pipelines to deliver and structure those logs for analysis.
  • Compliance evidence: Frameworks like PCI-DSS, HIPAA, SOC 2, and FedRAMP require proof of encryption-in-transit (TLS version), access controls, and data retention. These fields are uniquely available in server access logs.
  • Operational depth: Request latency, bytes transferred, HTTP status codes, and Amazon S3 lifecycle operations (expirations, transitions) are only captured in server access logs.
The Solution

This guide shows how Amazon S3 server access logs, delivered natively into CloudWatch unified data store, address these challenges and how to use them alongside CloudTrail for defense in depth.

With Amazon S3 server access logs as a native CloudWatch unified data store data source, you get broad, low-cost visibility with built-in transformation and immediate queryability, combined with guaranteed delivery from scoped CloudTrail data events on your most sensitive buckets. No custom ETL pipelines, no additional transformation workflows. Just enable the native integration and start querying, filtering, and alerting on your Amazon S3 access data.

CloudTrail Amazon S3 data events and server access logs: complementary strengths​

Both CloudTrail Amazon S3 data events and Amazon S3 server access logs record object-level activity, and there is meaningful overlap in the operations they capture. Understanding what each source brings to the table helps you design a logging strategy that maximizes coverage while managing costs.

Where They Overlap​

The core data-plane operations (GetObject, PutObject, DeleteObject, CopyObject, HeadObject, multipart uploads, and ListObjects) appear in both log sources when both are enabled. For these operations, you get two independent records of the same request.

What Each Source Uniquely Provides​

Rather than choosing one over the other, consider what each source adds that the other cannot:

CapabilityDetail
Delivery guarantee✅ Every API call is recorded and delivered
Delivery latencyTypically 5–15 minutes
IAM identity detail✅ Full userIdentity block: ARN, session context, assumed role chain, MFA status
Request/response parameters✅ Full JSON: encryption settings, ACL grants, condition keys
Streaming destinations✅ Amazon S3, CloudWatch Logs, or CloudTrail Lake
Selective logging✅ Advanced event selectors filter by bucket ARN, event name, read/write
Per-event costPer-event charge (see pricing)

When to Lean on Each Source​

  • 🔒 Compliance audits requiring complete accounting of every object access. Guaranteed delivery means no gaps in the audit trail.
  • 🔍 Forensic investigations where you need the full IAM identity chain: who assumed which role, whether MFA was used, what session context was active.
  • 📋 Understanding request parameters: what encryption algorithm was applied to a PutObject, what ACL grants were set, what condition keys were evaluated.
  • ⚡ Real-time streaming to CloudWatch Logs or CloudTrail Lake for immediate analysis, alerting, and dashboarding.
  • 🎯 Selective, cost-controlled logging. Advanced event selectors let you log only write operations on specific buckets, so you pay only for what matters most.

The hybrid approach: defense in depth​

Recommended Strategy

A hybrid strategy combines the strengths of both sources to deliver broad coverage at a manageable cost. The Amazon S3 security best practices documentation recommends enabling server access logging and using AWS CloudTrail as separate monitoring and auditing best practices. Using both together gives you defense in depth.

Three-Layer Architecture​

LayerWhat to EnablePurposeCost
1. CloudTrail management eventsTrail (first copy at no additional cost)Bucket-level config changes, PutBucketPolicy, DeleteBucketEncryption, etc. Guaranteed delivery.No additional cost (first copy)
2. Amazon S3 server access logs (via CloudWatch unified data store)Enable on all buckets with CloudWatch unified data store as the destinationBroad data-plane visibility with HTTP-level detail, lifecycle tracking, SigV2 detection, and performance metrics. Logs are automatically transformed into a structured, queryable format by CloudWatch unified data store with no ETL required. Immediate access to Logs Insights, metric filters, Contributor Insights, and Amazon S3 Tables.CloudWatch Logs ingestion + storage (Amazon S3 server access log events are smaller than CloudTrail data events, resulting in lower per-event cost; volume-tiered pricing reduces cost further at scale)
3. CloudTrail data events (scoped)Write-only on sensitive buckets via advanced event selectorsGuaranteed delivery for the most critical mutations, PutObject, DeleteObject, CopyObject on buckets holding PII, financial data, or regulated content.Per-event charge (see pricing), scoped to reduce volume

Scoping CloudTrail Data Events to Control Cost​

The key to making the hybrid approach cost-effective is using advanced event selectors to log only what matters most. For example, logging only write operations on three sensitive buckets:

advanced-event-selector.json
[
  {
    "Name": "S3WriteOnlySensitiveBuckets",
    "FieldSelectors": [
      { "Field": "eventCategory",  "Equals": ["Data"] },
      { "Field": "resources.type", "Equals": ["AWS::S3::Object"] },
      { "Field": "readOnly",       "Equals": ["false"] },
      { "Field": "resources.ARN",  "StartsWith": [
          "arn:aws:s3:::my-pii-bucket/",
          "arn:aws:s3:::my-financial-data/",
          "arn:aws:s3:::my-compliance-archive/"
        ]
      }
    ]
  }
]

This selector logs only write operations (PutObject, DeleteObject, CopyObject, etc.) on three specific buckets. This dramatically reduces cost compared to logging all data events on all buckets, while guaranteeing delivery for the most security-critical mutations. Apply it to your trail or event data store using the CloudTrail console, CLI (put-event-selectors), or CloudFormation.

Cost at Scale​

The cost advantage of the hybrid approach becomes clear when you compare CloudWatch Logs ingestion pricing for the two data sources side by side.

Amazon S3 server access logs are classified as CloudWatch vended logs, which benefit from volume-tiered ingestion pricing. The per-GB rate decreases as your monthly ingestion volume grows (for example, $0.50/GB for the first 10 TB, dropping to $0.05/GB above 50 TB in US East). This makes server access logs progressively more cost-effective at scale. Additionally, Amazon S3 server access log events are significantly smaller in size compared to CloudTrail data events, meaning you ingest fewer GB for the same number of requests.

CloudTrail events (management and data events) delivered to a CloudWatch Logs log group are charged for delivery plus CloudWatch custom logs ingestion pricing. CloudTrail data events provide additional context (see What each source uniquely provides), making them the right choice for your most sensitive buckets where you need complete audit trails with no gaps.

The combination of volume-tiered pricing and smaller event size means that, at high request volumes, Amazon S3 server access logs can cost significantly less per request to ingest into CloudWatch than CloudTrail data events. By scoping CloudTrail data events to write-only operations on your most sensitive buckets (using advanced event selectors), you limit that higher-cost ingestion to only the fraction of traffic that requires guaranteed delivery. Server access logs cover the remaining breadth at a lower per-GB rate.

CloudWatch unified data store also eliminates the operational cost and complexity of building custom ETL pipelines. The logs are natively delivered and automatically transformed into structured JSON upon ingestion, removing the need for destination bucket provisioning, log object lifecycle management, or custom transformation workflows.

Review Current Pricing

Pricing changes over time. Before finalizing your logging strategy, review the latest rates on the Amazon CloudWatch pricing page. Compare the Vended Logs tier (for Amazon S3 server access logs) against the Custom Logs ingestion rate (for CloudTrail events delivered to CloudWatch Logs). Use the AWS Pricing Calculator to help calculate the cost.

Decision Guide​

Use this quick reference to decide your logging strategy per bucket:

Does this bucket hold regulated, sensitive, or business-critical data?

✅ Yes → Enable CloudTrail data events (write-only at minimum) for guaranteed delivery and full identity chain. Also enable server access logs via CloudWatch unified data store for HTTP-level detail, lifecycle tracking, and built-in transformation with no ETL required.

➡️ No → See Q2.

What Amazon S3 server access logs capture​

Every request to an Amazon S3 bucket with server access logging enabled generates a log record with over 25 fields. The fields most relevant to security, compliance, and auditing are grouped below by use case.

FieldValueExample
requesterIAM principal (canonical user ID or assumed-role ARN)arn:aws:sts::123456789012:assumed-role/MyRole/session
remote_ipSource IP of the requester203.0.113.42
authentication_typeHow the request was authenticatedAuthHeader, QueryString, or null (anonymous)
signature_versionSigning protocol usedSigV4 or SigV2 (deprecated)
acl_requiredWhether ACL authorization was needed (boolean)true, false, or null
access_point_arnS3 Access Point ARN (if used)arn:aws:s3:us-east-1:123456789012:accesspoint/my-ap
Why This Matters

These fields answer who accessed your data and how they authenticated. Use them to detect anonymous access (authentication_type IS NULL), deprecated SigV2 clients, and requests routed through Access Points.

Enabling Amazon S3 Server Access Logging for CloudWatch​

Setting up delivery takes a few steps in the Amazon S3 console, or a few API calls with the AWS CLI. You can also standardize delivery across your AWS Organization with CloudWatch Telemetry Enablement Rules.

Option 1: Telemetry Enablement Rules​

CloudWatch Telemetry Enablement Rules let you automatically enable Amazon S3 server access log delivery to CloudWatch Logs for buckets, both new and existing, without per-bucket setup. Rules can be scoped at the individual account level or across an AWS Organization. For organization-wide rules, you must configure them from the management account or a delegated administrator account.

  1. Open the CloudWatch console.
  2. In the navigation pane, choose Ingestion.
  3. Choose the Enablement rules tab, then choose Add rule.
  4. Select Configure telemetry for Amazon S3.
  5. Enter a descriptive Rule name (e.g., S3-Access-Logs-Enablement).
  6. Optionally add Tag Key/Value under Select data source scope to scope the rule to specific resources.
  7. For Target regions, leave the default for the current region or select specific regions. Choose Next.
  8. Configure the Log group name pattern and Retention setting. Choose Next.
  9. Review and choose Configure Amazon S3 Logs.

Once active, CloudWatch begins ingesting Amazon S3 logs from resources within scope. Logs are automatically transformed into structured JSON with no additional Lambda functions or Glue jobs required.

Option 2: Amazon S3 Console (Per Bucket)​

  1. Open the Amazon S3 console, navigate to your bucket, and choose the Properties tab.
  2. In the Log delivery to CloudWatch section, choose Add → To Amazon CloudWatch Logs.
  3. Select a Destination log group. To also query with SQL, select Enable Amazon S3 Tables integration.
  4. Under Service access, leave Auto-create a new role with default permissions selected.
  5. Choose Add.

Option 3: AWS CLI (Automation)​

For scripted or automated deployments across multiple buckets, the AWS CLI gives you full control over each step.

Step 1: Create a delivery source (per source bucket)

aws logs put-delivery-source \
    --name my-sal-source \
    --resource-arn arn:aws:s3:::my-bucket \
    --log-type S3_SERVER_ACCESS_LOGS \
    --region us-east-1

Step 2: Create a delivery destination (once per log group)

aws logs put-delivery-destination \
    --name my-sal-destination \
    --delivery-destination-configuration \
        '{"destinationResourceArn": "arn:aws:logs:us-east-1:123456789012:log-group:/aws/vendedlogs/s3/my-bucket"}' \
    --region us-east-1

Step 3: Create a delivery (per source bucket)

aws logs create-delivery \
    --delivery-source-name my-sal-source \
    --delivery-destination-arn arn:aws:logs:us-east-1:123456789012:delivery-destination:my-sal-destination \
    --region us-east-1

Step 4: Enable the Amazon S3 Tables integration (once per account per Region, optional)

aws observabilityadmin create-s3-table-integration \
    --role-arn arn:aws:iam::123456789012:role/CWLogsS3TableIntegrationRole \
    --encryption '{"SseAlgorithm":"AES256"}' \
    --region us-east-1

Then associate the Amazon S3 server access log data source:

aws logs associate-source-to-s3-table-integration \
    --integration-arn arn:aws:observabilityadmin:us-east-1:123456789012:s3tableintegration/s3tableintegration \
    --data-source '{"name":"amazon_s3","type":"server_access"}' \
    --region us-east-1

Security & Compliance Queries​

The following OpenSearch SQL queries run in CloudWatch Logs Insights against your Amazon S3 server access log group. Select "OpenSearch SQL" from the language dropdown before running. Replace YOUR_S3_ACCESS_LOG_GROUP with your actual log group name.

Detecting Unauthorized Access Attempts​

HTTP 403 and 404 spikes from a single IP address can indicate credential stuffing, object key enumeration, or brute-force access attempts.

query-403-404-errors.sql
SELECT
    remote_ip,
    error_code,
    operation,
    COUNT(*)                   AS error_count,
    COUNT(DISTINCT key_name)   AS unique_keys_targeted,
    COUNT(DISTINCT bucket_name) AS buckets_targeted
FROM `YOUR_S3_ACCESS_LOG_GROUP`
WHERE http_status IN (403, 404)
GROUP BY remote_ip, error_code, operation
ORDER BY error_count DESC
LIMIT 50

Detecting Unusual Data Transfer Patterns​

Aggregating bytes downloaded per requester and IP reveals unusual download patterns that could indicate potentially exposed credentials or internal unauthorized activity.

query-data-exfiltration.sql
SELECT
    remote_ip,
    requester,
    SUM(bytes_sent_size) / 1048576  AS total_MB_downloaded,
    COUNT(*)                        AS request_count,
    COUNT(DISTINCT key_name)        AS unique_objects,
    COUNT(DISTINCT bucket_name)     AS buckets_accessed
FROM `YOUR_S3_ACCESS_LOG_GROUP`
WHERE operation LIKE 'REST.GET.OBJECT'
  AND http_status = 200
GROUP BY remote_ip, requester
ORDER BY total_MB_downloaded DESC
LIMIT 25

Detecting Anonymous (Unauthenticated) Access​

Any request with authentication_type IS NULL means no credentials were presented. This typically indicates a bucket policy or ACL is granting public access.

query-anonymous-access.sql
SELECT
    `@timestamp`,
    bucket_name,
    operation,
    key_name,
    remote_ip,
    http_status,
    bytes_sent_size,
    user_agent
FROM `YOUR_S3_ACCESS_LOG_GROUP`
WHERE authentication_type IS NULL
  AND operation NOT LIKE '%OPTIONS%'
ORDER BY `@timestamp` DESC
LIMIT 100

Detecting Bulk Delete Operations​

Mass deletions from a single requester may indicate unauthorized bulk deletion patterns, insider threat, or a runaway automation pipeline.

query-bulk-deletes.sql
SELECT
    requester,
    remote_ip,
    operation,
    COUNT(*)                   AS delete_count,
    COUNT(DISTINCT key_name)   AS unique_objects,
    COUNT(DISTINCT bucket_name) AS buckets_affected
FROM `YOUR_S3_ACCESS_LOG_GROUP`
WHERE operation LIKE '%DELETE%'
   OR operation = 'REST.POST.MULTI_OBJECT_DELETE'
   OR operation = 'BATCH.DELETE.OBJECT'
GROUP BY requester, remote_ip, operation
ORDER BY delete_count DESC
LIMIT 50

Real-Time Monitoring with Metric Filters & Contributor Insights​

The queries above are powerful for ad-hoc investigation. For continuous, real-time monitoring, create CloudWatch metric filters and Contributor Insights rules on the same log data.

Metric filters emit custom CloudWatch metrics from log patterns. Each metric can drive a CloudWatch Alarm for real-time alerting.

Metric Filters Require Standard Log Class

Metric filters are only supported on CloudWatch Logs log groups using the Standard class. If you chose the Infrequent Access class for lower ingestion costs, use Logs Insights queries and Contributor Insights rules instead.

Metric Filter 1: Access Denied (403) Errors

Counts HTTP 403 responses. A spike may indicate credential probing or misconfigured policies

A sudden increase in 403 errors from a single IP or requester can indicate credential stuffing, brute-force attempts, or a misconfigured IAM policy affecting legitimate workloads.

Filter Pattern:

{ $.http_status = 403 }
  • Metric Namespace: S3ServerAccessLogs
  • Metric Name: AccessDeniedErrors
  • Metric Value: 1
  • Suggested Alarm: > 100 in 5 minutes → SNS notification to security team

Metric Filter 2: Not Found (404) Errors

Counts HTTP 404 responses. High volume may indicate object key enumeration

An attacker attempting to discover object keys will generate many 404 responses. Separating 404s from 403s lets you set different thresholds for each attack pattern.

Filter Pattern:

{ $.http_status = 404 }
  • Metric Namespace: S3ServerAccessLogs
  • Metric Name: NotFoundErrors
  • Metric Value: 1
  • Suggested Alarm: > 500 in 5 minutes → SNS notification to security team

Metric Filter 3: Server Errors (5xx)

Counts HTTP 5xx responses. A spike in 503 SlowDown errors may indicate request throttling

Abnormally high request rates can trigger Amazon S3 throttling (503 SlowDown). This may be caused by a runaway automation pipeline, a DDoS event, or a misconfigured application retrying in a tight loop.

Filter Pattern:

{ $.http_status = 5* }
  • Metric Namespace: S3ServerAccessLogs
  • Metric Name: ServerErrors5xx
  • Metric Value: 1
  • Suggested Alarm: > 50 in 5 minutes → SNS notification to operations team

Metric Filter 4: Anonymous / Unauthenticated Requests

Detects requests with no credentials. Any non-zero value warrants investigation

An anonymous request means no AWS credentials were presented. This typically indicates a bucket policy or ACL is granting public access, which may be intentional (static website) or a misconfiguration.

Filter Pattern:

{ $.authentication_type NOT EXISTS }
  • Metric Namespace: S3ServerAccessLogs
  • Metric Name: AnonymousRequests
  • Metric Value: 1
  • Suggested Alarm: > 0 in 5 minutes (for buckets that should never be public)

Metric Filter 5: Requests Using Deprecated SigV2

Counts requests signed with SigV2. Track progress toward full SigV4 migration

AWS deprecated Signature Version 2. The signature_version field in server access logs is the only way to find clients still using SigV2. Alarm on any occurrence to identify clients that need upgrading.

Filter Pattern:

{ $.signature_version = "SigV2" }
  • Metric Namespace: S3ServerAccessLogs
  • Metric Name: SigV2Requests
  • Metric Value: 1
  • Suggested Alarm: > 0 in 15 minutes → notify application owners for SDK upgrade

Metric Filter 6: Outdated TLS (1.0/1.1)

Counts requests using TLS 1.0 or 1.1. Compliance frameworks require TLS 1.2+

PCI-DSS, FedRAMP, and NIST 800-53 mandate TLS 1.2 or higher. This filter tracks clients that haven't upgraded, providing evidence for compliance reporting and a target list for remediation.

Filter Pattern:

{ $.tls_version = "TLSv1" || $.tls_version = "TLSv1.1" }
  • Metric Namespace: S3ServerAccessLogs
  • Metric Name: OutdatedTLSRequests
  • Metric Value: 1
  • Suggested Alarm: > 0 in 15 minutes → notify compliance team

Metric Filter 7: Large Data Downloads (Potential Exfiltration)

Counts successful GET responses over 100 MB. Sustained high volume may indicate data exfiltration

While large downloads are normal for many workloads, a sudden increase in high-volume GET responses from unexpected requesters or IPs warrants investigation.

Filter Pattern:

{ $.http_status = 200 && $.bytes_sent_size > 104857600 && $.operation = "REST.GET.OBJECT" }
  • Metric Namespace: S3ServerAccessLogs
  • Metric Name: LargeDownloads
  • Metric Value: 1
  • Suggested Alarm: > 50 in 15 minutes → notify security team for investigation
Defense in depth: use all three together
ToolRoleBest For
Metric FiltersEmit CloudWatch metrics → trigger alarmsThreshold-based alerting ("more than 50 errors in 5 minutes")
Contributor InsightsContinuously rank top-N contributorsLive dashboards ("which IP is generating the most errors right now?")
Logs Insights QueriesAd-hoc SQL analysis on raw log dataIncident investigation and forensic deep-dives

Summary​

Amazon S3 server access logs, delivered natively into CloudWatch unified data store, provide a cost-effective, HTTP-level view of every request to your buckets, including fields and operations that no other log source captures. CloudWatch unified data store automatically transforms the raw Amazon S3 server access log data into a structured, queryable format, removing the need for destination bucket provisioning, log object lifecycle management, and custom ETL pipelines. CloudTrail Amazon S3 data events provide guaranteed delivery, rich IAM identity context, and flexible streaming destinations. Together, they form a defense-in-depth strategy that balances cost with assurance.

Key Takeaway

The hybrid approach gives you the broadest coverage at the most manageable cost:

  • Amazon S3 server access logs on all buckets delivered through CloudWatch unified data store for broad visibility, lifecycle tracking, SigV2 detection, and performance metrics with volume-tiered pricing.
  • Scoped CloudTrail data events on sensitive buckets for guaranteed delivery and full identity context on your most critical data.
  • CloudTrail management events (at no additional cost) for configuration change alerting.

CloudWatch unified data store handles the transformation automatically, so you can start querying, filtering, and alerting on your Amazon S3 access data immediately with no additional pipeline setup. CloudTrail's reliable delivery helps ensure you do not miss critical write events on your most important data.

Additional Resources​