AWS Lambda CloudWatch Jobs

Need to automate periodic tasks like backups, email sending, or data processing? AWS Lambda combined with CloudWatch Events/EventBridge is the perfect serverless solution to replace traditional cron jobs.

Overview of Lambda Scheduled Jobs#

AWS Lambda is a serverless compute service that allows you to run code without managing servers. When combined with scheduling services, you can:

Automate tasks: Backups, report generation, data cleanup
Save costs: Pay only when code executes
Auto-scale: Handle increased load without configuration
High availability: AWS ensures availability

[!NOTE] AWS currently recommends using EventBridge Scheduler instead of legacy CloudWatch Events for new scheduled jobs.

Method 1: EventBridge Scheduler (Recommended)#

EventBridge Scheduler is the modern scheduling service with enhanced features compared to CloudWatch Events.

Advantages of EventBridge Scheduler#

Timezone support: Not limited to UTC only
Flexible time windows: Run within flexible time ranges
Built-in retry: Automatic retry on failure
One-time schedules: Support for single execution
Dead Letter Queue: Better failure handling

Step 1: Create Lambda Function#

import json
import logging
import boto3
from datetime import datetime

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    """Handler for scheduled job"""
    logger.info(f"Job triggered at: {datetime.utcnow()}")
    logger.info(f"Event: {json.dumps(event)}")
    
    try:
        # Your job processing logic
        result = process_scheduled_task(event)
        
        return {
            'statusCode': 200,
            'body': json.dumps({
                'message': 'Job completed successfully',
                'result': result
            })
        }
    except Exception as e:
        logger.error(f"Job failed: {str(e)}")
        raise

def process_scheduled_task(event):
    """Main job processing logic"""
    # Example: cleanup database, send report, etc.
    logger.info("Processing scheduled task...")
    
    # Your business logic here
    return {"status": "success", "processed_items": 10}

python

Step 2: Create IAM Role#

Lambda needs permissions to execute the job and write logs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

json

Policy permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:*",  # If DynamoDB access needed
        "s3:*"         # If S3 access needed
      ],
      "Resource": "*"
    }
  ]
}

json

Step 3: Create Schedule with EventBridge Scheduler#

Using AWS Console#

Go to EventBridge Scheduler console
Click Create schedule
Select Recurring schedule
Configure schedule expression:

# Run daily at 9:00 AM UTC
cron(0 9 * * ? *)

# Run every 15 minutes
rate(15 minutes)

# Run weekly on Monday 9:00 AM
cron(0 9 ? * MON *)

# Run on first day of each month
cron(0 0 1 * ? *)

bash

Select Lambda function as target
Configure timezone (if needed):

# 9:00 AM Eastern Time
cron(0 9 ? * MON *)
timezone: America/New_York

bash

Configure flexible time window (optional):

# Run within 15 minutes after scheduled time
mode: FLEXIBLE
maximum_window_in_minutes: 15

bash

Setup retry policy:

maximum_retry_attempts: 3
maximum_event_age_in_seconds: 3600

bash

Using AWS CLI#

aws scheduler create-schedule \
  --name daily-job \
  --schedule-expression 'cron(0 9 * * ? *)' \
  --schedule-expression-timezone 'UTC' \
  --target '{
    "Arn": "arn:aws:lambda:us-east-1:123456789012:function:my-function",
    "RoleArn": "arn:aws:iam::123456789012:role/scheduler-role"
  }' \
  --flexible-time-window '{
    "Mode": "FLEXIBLE",
    "MaximumWindowInMinutes": 15
  }'

bash

Using Terraform#

resource "aws_scheduler_schedule" "daily_job" {
  name       = "daily-job"
  group_name = "default"

  flexible_time_window {
    mode                      = "FLEXIBLE"
    maximum_window_in_minutes = 15
  }

  schedule_expression          = "cron(0 9 * * ? *)"
  schedule_expression_timezone = "UTC"

  target {
    arn      = aws_lambda_function.my_function.arn
    role_arn = aws_iam_role.scheduler_role.arn

    retry_policy {
      maximum_retry_attempts       = 3
      maximum_event_age_in_seconds = 3600
    }

    dead_letter_config {
      arn = aws_sqs_queue.dlq.arn
    }
  }
}

hcl

Method 2: CloudWatch Events (Legacy)#

CloudWatch Events is the traditional method, still supported but with fewer features.

Step 1: Create Lambda Function#

Same as Scheduler method.

Step 2: Create CloudWatch Rule#

Using AWS Console#

Go to CloudWatch console
Select Events → Rules
Click Create rule
Select Schedule expression
Enter cron expression:

# 9:00 AM UTC daily
0 9 * * ? *

# Every 5 minutes
rate(5 minutes)

bash

Select Lambda function as target
Configure permission (AWS will auto-create)

Using AWS CLI#

aws events put-rule \
  --name daily-lambda-rule \
  --schedule-expression 'cron(0 9 * * ? *)'

aws events put-targets \
  --rule daily-lambda-rule \
  --targets '{
    "Id": "1",
    "Arn": "arn:aws:lambda:us-east-1:123456789012:function:my-function"
  }'

aws lambda add-permission \
  --function-name my-function \
  --statement-id daily-lambda-rule \
  --action 'lambda:InvokeFunction' \
  --principal events.amazonaws.com \
  --source-arn arn:aws:events:us-east-1:123456789012:rule/daily-lambda-rule

bash

Cron Expression Guide#

EventBridge uses 6-field cron expression:

cron(Minutes Hours Day-of-month Month Day-of-week Year)

text

Example Cron Expressions#

# Daily at 9:00 AM UTC
cron(0 9 * * ? *)

# Weekly on Monday 9:00 AM
cron(0 9 ? * MON *)

# Monthly on day 1 at 00:00
cron(0 0 1 * ? *)

# Every 15 minutes
rate(15 minutes)

# Every 1 hour
rate(1 hour)

# 9:00 AM Monday to Friday
cron(0 9 ? * MON-FRI *)

# Last day of month
cron(0 0 L * ? *)

# First Monday of each month
cron(0 0 ? * MON#1 *)

bash

Field Values#

Field	Values	Special Characters
Minutes	0-59	, - * /
Hours	0-23	, - * /
Day of month	1-31, L, W	, - * / ? L W
Month	1-12, JAN-DEC	, - * /
Day of week	1-7, SUN-SAT, ?, L, #	, - * / ? L #
Year	1970-2199	, - * /

[!WARNING] Use ? when day-of-month or day-of-week is constrained, don’t use both fields simultaneously.

Best Practices#

1. Idempotency#

Ensure your handler is idempotent for safe retries:

def lambda_handler(event, context):
    # Check if job already ran
    if is_job_already_processed(event):
        logger.info("Job already processed, skipping")
        return {"status": "skipped"}
    
    # Process job
    result = process_job(event)
    
    # Mark job as processed
    mark_job_as_processed(event)
    
    return result

python

2. Dead Letter Queue (DLQ)#

Setup DLQ to handle failed invocations:

import boto3
import json

sqs = boto3.client('sqs')

def send_to_dlq(error_message, event):
    """Send failed event to DLQ"""
    sqs.send_message(
        QueueUrl='YOUR_DLQ_URL',
        MessageBody=json.dumps({
            'error': error_message,
            'event': event,
            'timestamp': str(datetime.utcnow())
        })
    )

python

3. Overlap Prevention#

Prevent overlapping executions with DynamoDB lock:

import boto3
from datetime import datetime, timedelta

dynamodb = boto3.resource('dynamodb')
lock_table = dynamodb.Table('job-locks')

def acquire_lock(job_id):
    """Acquire lock to prevent overlap"""
    try:
        lock_table.put_item(
            Item={
                'job_id': job_id,
                'locked_at': datetime.utcnow().isoformat(),
                'expires_at': (datetime.utcnow() + timedelta(minutes=10)).isoformat()
            },
            ConditionExpression='attribute_not_exists(job_id)'
        )
        return True
    except Exception:
        return False  # Lock already exists

def release_lock(job_id):
    """Release lock after completion"""
    lock_table.delete_item(Key={'job_id': job_id})

python

4. Monitoring and Logging#

Use CloudWatch Logs and Metrics:

import time
import logging

logger = logging.getLogger()

def lambda_handler(event, context):
    start_time = time.time()
    
    try:
        # Custom metrics
        logger.info("JOB_STARTED")
        
        result = process_job(event)
        
        # Log duration
        duration = time.time() - start_time
        logger.info(f"JOB_COMPLETED duration={duration}")
        
        return result
    except Exception as e:
        logger.error(f"JOB_FAILED error={str(e)}")
        raise

python

5. Error Handling#

Handle errors gracefully:

def lambda_handler(event, context):
    try:
        result = process_job(event)
        return {
            'statusCode': 200,
            'body': json.dumps(result)
        }
    except ValueError as e:
        logger.error(f"Validation error: {str(e)}")
        return {
            'statusCode': 400,
            'body': json.dumps({'error': str(e)})
        }
    except Exception as e:
        logger.error(f"Unexpected error: {str(e)}")
        # Send to DLQ or alert
        send_alert(str(e))
        raise

python

Comparison: EventBridge Scheduler vs CloudWatch Events#

Feature	EventBridge Scheduler	CloudWatch Events
Schedule types	cron + rate + one-time	cron + rate
Timezone support	✅ Any timezone	❌ UTC only
Flexible windows	✅ Yes	❌ No
Built-in retry	✅ Yes	❌ No (need DLQ)
Cost	$1.00/M invocations	Free (first 5M/month)
Use case	Production jobs	Simple recurring jobs

Practical Example: Daily Database Cleanup#

import boto3
import logging
from datetime import datetime, timedelta

logger = logging.getLogger()
dynamodb = boto3.resource('dynamodb')

def lambda_handler(event, context):
    """Cleanup records older than 30 days"""
    table = dynamodb.Table('user-activity')
    
    cutoff_date = (datetime.utcnow() - timedelta(days=30)).isoformat()
    
    try:
        # Scan and delete old records
        response = table.scan(
            FilterExpression='created_at < :cutoff',
            ExpressionAttributeValues={':cutoff': cutoff_date}
        )
        
        deleted_count = 0
        with table.batch_writer() as batch:
            for item in response['Items']:
                batch.delete_item(Key={'id': item['id']})
                deleted_count += 1
        
        logger.info(f"Deleted {deleted_count} old records")
        
        return {
            'statusCode': 200,
            'body': json.dumps({
                'deleted_count': deleted_count,
                'cutoff_date': cutoff_date
            })
        }
    except Exception as e:
        logger.error(f"Cleanup failed: {str(e)}")
        raise

python

Schedule with EventBridge Scheduler:

# Run daily at 2:00 AM UTC
cron(0 2 * * ? *)

bash

Troubleshooting#

Job not running#

Check schedule expression: Ensure cron syntax is correct
Verify IAM permissions: Lambda has necessary permissions
Check CloudWatch Logs: Look for error messages
Verify target ARN: Ensure Lambda function ARN is correct

Job runs but fails#

Review CloudWatch Logs: Find error messages
Check timeout: Lambda might be timing out
Verify resource permissions: Database, S3, etc. access
Test locally: Run Lambda locally to debug

Overlapping executions#

Implement lock mechanism: Use DynamoDB or Redis
Use reserved concurrency: Limit concurrent executions
Add idempotency: Ensure handler is idempotent

Cleanup Resources#

When no longer needed, clean up resources:

# Delete schedule
aws scheduler delete-schedule --name daily-job

# Delete CloudWatch rule
aws events delete-rule --name daily-lambda-rule

# Delete Lambda function
aws lambda delete-function --function-name my-function

# Delete IAM role
aws iam delete-role --role-name scheduler-role

bash

Conclusion#

AWS Lambda combined with EventBridge Scheduler/CloudWatch Events provides a powerful solution for scheduled jobs:

EventBridge Scheduler: For production jobs with timezone support, retry, flexible windows
CloudWatch Events: For simple recurring jobs with cost efficiency

Key takeaways:

Use EventBridge Scheduler for production
Implement idempotency and error handling
Setup monitoring with CloudWatch
Use DLQ for failed invocations
Prevent overlapping executions

Overview of Lambda Scheduled Jobs#

Method 1: EventBridge Scheduler (Recommended)#

Advantages of EventBridge Scheduler#

Step 1: Create Lambda Function#

Step 2: Create IAM Role#

Step 3: Create Schedule with EventBridge Scheduler#

Using AWS Console#

Using AWS CLI#

Using Terraform#

Method 2: CloudWatch Events (Legacy)#

Step 1: Create Lambda Function#

Step 2: Create CloudWatch Rule#

Using AWS Console#

Using AWS CLI#

Cron Expression Guide#

Example Cron Expressions#

Field Values#

Best Practices#

1. Idempotency#

2. Dead Letter Queue (DLQ)#

3. Overlap Prevention#

4. Monitoring and Logging#

5. Error Handling#

Comparison: EventBridge Scheduler vs CloudWatch Events#

Practical Example: Daily Database Cleanup#

Troubleshooting#

Job not running#

Job runs but fails#

Overlapping executions#

Cleanup Resources#

Conclusion#

References#