Comprehensive Logging For AWS Well-Architected SEC04-BP01

Dec 2, 2025 by Alex Johnson 58 views

In this article, we will explore how to configure comprehensive service and application logging for ARC302 Well-Architected SEC04-BP01. This is crucial for maintaining a secure and compliant environment in AWS. Proper logging enables security teams to investigate incidents, perform root cause analysis, detect unauthorized access, and meet compliance requirements. Let's dive into the details.

Overview

This article addresses the AWS Well-Architected Framework SEC04-BP01: Configure service and application logging. Many implementations lack comprehensive logging across API Gateway, Lambda, DynamoDB, and VPC components, which severely limits security event detection, investigation capabilities, and audit compliance. This article provides a detailed guide on how to implement comprehensive logging to mitigate these risks.

Risk Level: High

Impact: Without proper logging, security teams cannot effectively investigate incidents, perform root cause analysis, detect unauthorized access, or meet compliance requirements. This creates significant security blind spots and prevents effective incident response, potentially exposing the organization to undetected security breaches and regulatory violations. Therefore, implementing a robust logging strategy is paramount for maintaining a secure and compliant AWS environment.

Sub-tasks

To achieve comprehensive logging, we need to address several key areas. Each sub-task outlined below provides a step-by-step guide to enable and enhance logging for different AWS services.

Task 1: Enable API Gateway Access Logging

Location: python/apigw-http-api-lambda-dynamodb-python-cdk/stacks/apigw_http_api_lambda_dynamodb_python_cdk_stack.py

Problem: One of the critical issues is that API Gateway lacks access logging. This prevents the capture of essential information such as caller identity, request/response data, and timestamps, which are crucial for security investigations. Without these logs, it becomes exceedingly difficult to trace and understand the origin and nature of API requests, hindering the ability to detect and respond to potential security threats effectively.

Solution: To address this, we will create a CloudWatch Log Group and configure API Gateway to send access logs with a detailed format. This ensures that all necessary information is captured for security analysis and auditing. Enabling API Gateway access logging involves several key steps. First, you need to create a CloudWatch Log Group, which will serve as the destination for the logs. Then, you configure API Gateway to send access logs to this log group. Finally, you specify the format of the logs to include comprehensive details such as the caller's identity, request and response data, and timestamps. Here's the code snippet to achieve this:

from aws_cdk import aws_logs as logs

# Create log group for API Gateway access logs
api_log_group = logs.LogGroup(
    self,
    "ApiGatewayAccessLogs",
    retention=logs.RetentionDays.ONE_YEAR,
)

apigw_.LambdaRestApi(
    self,
    "Endpoint",
    handler=api_hanlder,
    deploy_options=apigw_.StageOptions(
        access_log_destination=apigw_.LogGroupLogDestination(api_log_group),
        access_log_format=apigw_.AccessLogFormat.json_with_standard_fields(
            caller=True,
            http_method=True,
            ip=True,
            protocol=True,
            request_time=True,
            resource_path=True,
            response_length=True,
            status=True,
            user=True,
        ),
    ),
)

Task 2: Enable API Gateway Execution Logging

Location: python/apigw-http-api-lambda-dynamodb-python-cdk/stacks/apigw_http_api_lambda_dynamodb_python_cdk_stack.py

Problem: Another significant gap in logging is the lack of API Gateway execution logs. Without these logs, detailed troubleshooting of integration latency and errors becomes challenging. Execution logs provide valuable insights into the performance and behavior of your API, enabling you to identify and resolve issues more efficiently. They capture information about the execution flow, including integration requests and responses, which is crucial for diagnosing performance bottlenecks and errors.

Solution: To rectify this, we will add logging level configuration to capture execution details. This involves configuring the deployment options of your API Gateway to include a specific logging level, such as INFO, and enabling data tracing and metrics. By doing so, you can capture detailed information about the API's execution, including request and response payloads, integration latency, and error messages. This comprehensive logging approach significantly enhances your ability to troubleshoot and optimize your API's performance. Here’s how to configure API Gateway execution logs:

apigw_.LambdaRestApi(
    self,
    "Endpoint",
    handler=api_hanlder,
    deploy_options=apigw_.StageOptions(
        logging_level=apigw_.MethodLoggingLevel.INFO,
        data_trace_enabled=True,
        metrics_enabled=True,
    ),
)

Task 3: Enable DynamoDB Point-in-Time Recovery

Location: python/apigw-http-api-lambda-dynamodb-python-cdk/stacks/apigw_http_api_lambda_dynamodb_python_cdk_stack.py

Problem: The DynamoDB table's absence of point-in-time recovery (PITR) is a critical concern. Without PITR, investigating data tampering or recovering from unauthorized modifications becomes exceptionally difficult. This feature is crucial for maintaining data integrity and compliance, as it allows you to restore your table to any point in time within the preceding 35 days. This capability is invaluable for audits, forensic investigations, and recovering from accidental data loss or corruption.

Solution: To enhance data protection and audit capabilities, we will enable PITR. This ensures that you can restore your DynamoDB table to a specific point in time, providing a robust mechanism for data recovery and forensic analysis. Enabling PITR involves a simple configuration change in your DynamoDB table settings. Once enabled, DynamoDB automatically creates and maintains backups of your data, allowing you to restore your table to any point in time within the retention period. Below is the code snippet to enable DynamoDB PITR:

demo_table = dynamodb_.Table(
    self,
    TABLE_NAME,
    partition_key=dynamodb_.Attribute(
        name="id", type=dynamodb_.AttributeType.STRING
    ),
    point_in_time_recovery=True,
)

Task 4: Configure Lambda Function Log Retention

Location: python/apigw-http-api-lambda-dynamodb-python-cdk/stacks/apigw_http_api_lambda_dynamodb_python_cdk_stack.py

Problem: The lack of an explicit log retention policy for Lambda functions poses a significant issue. This can lead to an undefined log lifecycle and potential compliance issues. Without a clear retention policy, logs may be deleted prematurely, making it impossible to conduct thorough security investigations or meet regulatory requirements. Conversely, logs might be retained indefinitely, consuming unnecessary storage and potentially exposing sensitive information over time.

Solution: To address this, we will set an explicit log retention period aligned with security and compliance requirements. This ensures that logs are retained for a sufficient duration to support security investigations and audits while also adhering to data retention policies. Configuring log retention involves specifying the retention period in CloudWatch Logs, which dictates how long log events are stored before being automatically deleted. A common retention period is one year, but this should be adjusted based on your organization's specific needs and compliance obligations. Here's the code to set an explicit log retention period for Lambda functions:

from aws_cdk import aws_logs as logs

api_hanlder = lambda_.Function(
    self,
    "ApiHandler",
    function_name="apigw_handler",
    runtime=lambda_.Runtime.PYTHON_3_9,
    code=lambda_.Code.from_asset("lambda/apigw-handler"),
    handler="index.handler",
    vpc=vpc,
    vpc_subnets=ec2.SubnetSelection(
        subnet_type=ec2.SubnetType.PRIVATE_ISOLATED
    ),
    memory_size=1024,
    timeout=Duration.minutes(5),
    log_retention=logs.RetentionDays.ONE_YEAR,
)

Task 5: Enhance Lambda Application Logging

Location: python/apigw-http-api-lambda-dynamodb-python-cdk/lambda/apigw-handler/index.py

Problem: Minimal Lambda logging is insufficient as it fails to capture security-relevant events, request context, or structured data necessary for analysis. Basic logging often lacks the granularity and context needed to effectively troubleshoot issues or detect security incidents. Without structured logging, it's challenging to parse and analyze log data efficiently, hindering your ability to identify trends, anomalies, and potential threats.

Solution: To improve log analysis and security monitoring, we will implement structured logging with security context and proper error handling. Structured logging involves formatting log messages in a consistent and machine-readable format, such as JSON, which makes it easier to query and analyze log data. Incorporating security context into log messages provides valuable information about the request, such as the source IP address and user agent, which can aid in identifying suspicious activity. Additionally, proper error handling ensures that exceptions and errors are logged with sufficient detail to facilitate troubleshooting. The following code snippet demonstrates how to implement structured logging in a Lambda function:

import json
import logging
from aws_lambda_powertools import Logger

logger = Logger(service="apigw-handler")

def handler(event, context):
    logger.info("Request received", extra={
        "request_id": context.request_id,
        "source_ip": event.get("requestContext", {}).get("identity", {}).get("sourceIp"),
        "user_agent": event.get("requestContext", {}).get("identity", {}).get("userAgent"),
    })
    
    try:
        # Existing logic
        pass
    except Exception as e:
        logger.exception("Error processing request", extra={
            "error_type": type(e).__name__,
            "request_id": context.request_id,
        })
        raise

Task 6: Enable VPC Flow Logs

Location: python/apigw-http-api-lambda-dynamodb-python-cdk/stacks/apigw_http_api_lambda_dynamodb_python_cdk_stack.py

Problem: The absence of VPC Flow Logs prevents network-level security investigation and traffic analysis for Lambda functions. Without VPC Flow Logs, it's difficult to gain visibility into the network traffic flowing in and out of your VPC, making it challenging to detect and respond to network-based threats. These logs provide a record of the network traffic that traverses your VPC, including source and destination IP addresses, ports, and the number of bytes transferred. This information is invaluable for security monitoring, troubleshooting network connectivity issues, and ensuring compliance with regulatory requirements.

Solution: To enable network traffic analysis for security purposes, we will enable VPC Flow Logs. This involves configuring your VPC to capture network traffic metadata and store it in CloudWatch Logs for analysis. By enabling VPC Flow Logs, you can gain insights into the network traffic patterns within your VPC, identify potential security threats, and troubleshoot network connectivity issues more effectively. The following code snippet illustrates how to enable VPC Flow Logs:

from aws_cdk import aws_logs as logs

# Create log group for VPC Flow Logs
vpc_flow_log_group = logs.LogGroup(
    self,
    "VpcFlowLogs",
    retention=logs.RetentionDays.ONE_MONTH,
)

# Enable VPC Flow Logs
vpc.add_flow_log(
    "FlowLog",
    destination=ec2.FlowLogDestination.to_cloud_watch_logs(vpc_flow_log_group),
    traffic_type=ec2.FlowLogTrafficType.ALL,
)

Additional Requirements

To ensure comprehensive logging, consider the following additional requirements:

Import aws_logs module in the stack file.
Consider using AWS Lambda Powertools for structured logging (optional but recommended).
Update Lambda deployment package if using Powertools.
Ensure IAM roles have necessary permissions for CloudWatch Logs (automatically handled by CDK).
Review and adjust log retention periods based on organizational compliance requirements.
Consider enabling DynamoDB Streams if an audit trail of data changes is required.
Implement log encryption at rest using KMS keys for sensitive environments (optional enhancement).

Acceptance Criteria

To verify that comprehensive logging has been successfully implemented, ensure the following acceptance criteria are met:

[ ] API Gateway access logs are enabled and writing to CloudWatch Logs with a comprehensive format.
[ ] API Gateway execution logs are enabled at the INFO level.
[ ] DynamoDB table has point-in-time recovery enabled.
[ ] Lambda function has an explicit log retention policy set (1 year or per compliance requirements).
[ ] Lambda application code implements structured logging with security context.
[ ] VPC Flow Logs are enabled and capturing network traffic.
[ ] All log groups have appropriate retention policies configured.
[ ] Logs are queryable in CloudWatch Logs Insights for security investigations.
[ ] No sensitive data (PII, credentials) is logged in plain text.

Conclusion

Implementing comprehensive logging across your AWS services is crucial for maintaining a secure and compliant environment. By following the steps outlined in this article, you can enhance your security posture, improve incident response capabilities, and meet regulatory requirements. Proper logging provides the visibility and insights needed to detect and address potential security threats effectively.

For further reading on AWS security best practices, consider exploring the AWS Security Documentation.