Leveraging Lambda Functions (with Amazon SNS, S3 & EventBridge)

For the longest time, I couldn’t quite grasp or fully understand how serverless services on AWS like Lambda functions ACTUALLY work in real-world scenarios and products. Partly because I wasn’t too familiar with event-driven architecture or built projects that fully implemented it.

That has now changed because I finally took some time to explore it with a scenario use-case. In this post, you’ll learn how to use AWS Lambda, S3, and Amazon SNS to build yourself a nice little weather alert system.

Project Summary

I modified the second project of the #DevOpsAllStarsChallenge (mainly because I didn’t want to use the sports data API) so instead, I built a notification system that sends email alerts of weather data for the city of Manchester, UK. It’s made up of:

• a Lambda function that sends a summary of the current weather data for the city to a user email subscribed to an SNS topic; this is hooked up to an EventBridge rule with a morning, afternoon and evening schedule.

• another Lambda function that retrieves the most recently added weather data file to the S3 bucket from this project and sends the summary to the user email subscribed to the same SNS topic; it is triggered whenever a new search is made via the dashboard and saved to storage.

You can read the full project and its feature summary here.

Architecture

One thing I’ve kept in mind since I started the challenge is to not underestimate the power of putting together an architectural diagram. It may not be the best but I quickly realised that drawing up a solution has helped me visualise and build out the projects better.

This was the first diagram I drew but eventually switched to using native S3 Event notifications for the fetch from S3 function for reasons explained further below (but I still included how to set up the approach in the diagram).

Build Process

I first tried building it all out via the AWS console but as with the first project, I wrote scripts to build it all using aws-cli commands instead. The aws-cli documentation was my main source of truth and I learnt a lot navigating my way around for the different services I needed for the project.

I wrote a brief note on how I worked through each section of the architecture here.

Repo structure:

Gotchas

1. I fell into the trap of not specifying the —region option so many times. I was working in the eu-west-2 region but I still haven’t updated my IDE’s credentials and config files from us-east-1, so I spent a lot more time resolving errors that eventually turned out to be region mismatches.

2. Pay extra attention when creating policies and roles, one can easily make mistakes inlining policy documents with JSON formats.

3. It got a bit tricky creating the Lambda functions via aws-cli commands, especially with adding the function code. I had to ensure that I created the .zip files with the most updated versions of the code files; the names used also need to match the handlers defined for the functions + the main execution methods in the code.

e.g For alerts_lambda.py that uses a process() function for the main execution, the zip file was named alerts_lambda and the handler in my scripts was alerts_lambda.process:

 aws lambda create-function --function-name $Function1Name --runtime python3.9 \
        --zip-file fileb://src/alerts_lambda.zip --handler alerts_lambda.process \
        --timeout 10 --role arn:aws:iam::${AWS_ACCOUNT_ID}:role/$Function1RoleName

4. The biggest problem I encountered definitely has to be setting up the EventBridge rules: I couldn’t get the rule using an S3 event pattern to work! Despite trying many fixes, it just wasn’t invoking the Lambda function when an object gets placed in the bucket.

   Turns out that I had completely ignored the alert clearly informing users to turn on the configuration that allows S3 buckets to send notifications to EventBridge for all events that occur within the bucket (perk of using the management console, I guess).

In hindsight, the entire ordeal made me learn about the optimal choice to consider when you need to invoke Lambda functions with simple S3 event triggers: S3 Event Notifications.

S3 Event Notifications

This is a more native approach that can be used to configure what should happen when specified events occur within a S3 bucket. The destinations can be services like Lambda, SNS, or even EventBridge. In this case, I wanted to invoke the Lambda function that handles the data retrieval, preprocessing and sending to SNS, so Lambda it was.

Another benefit of using this approach is its near-instant “activation”. With an EventBridge rule, there might be some delay before it starts working. EventBridge is also better suited for complex event patterns but this was just a simple case of monitoring for the addition of new objects to a S3 bucket.

However, this does not mean it cannot be implemented using EventBridge; just ensure to turn on the configuration mentioned above.

• Setup with S3 Event Notification

aws s3api put-bucket-notification-configuration \
    --bucket $BUCKET_NAME --region $AWS_REGION \
    --notification-configuration '{
        "LambdaFunctionConfigurations": [
            {
                "LambdaFunctionArn": "arn:aws:lambda:'"$AWS_REGION"':'"$AWS_ACCOUNT_ID"':function:'"$FunctionName"'",
                "Events": ["s3:ObjectCreated:*"]
            }
        ]
    }'

• Setup with EventBridge using event pattern

EVENT_PATTERN=$(cat <<EOF
{
  "source": ["aws.s3"],
  "detail-type": ["Object Created"],
  "detail": {
    "bucket": {
      "name": ["$BUCKET_NAME"]
    }
  }
}
EOF
)

aws events put-rule --name $TRIGGER_NAME --event-pattern "$EVENT_PATTERN" --region $AWS_REGION

aws events put-targets --rule $TRIGGER_NAME \
        --targets "Id"="1","Arn"="arn:aws:lambda:$AWS_REGION:$AWS_ACCOUNT_ID:function:$FunctionName"

aws lambda add-permission \
        --function-name $FunctionName \
        --statement-id $EventBridgePermissionStatementId \
        --action "lambda:InvokeFunction" \
        --principal "events.amazonaws.com" \
        --source-arn "arn:aws:events:$AWS_REGION:$AWS_ACCOUNT_ID:rule/$TRIGGER_NAME" \
        --region $AWS_REGION

aws s3api put-bucket-notification-configuration --bucket $BUCKET_NAME --region $AWS_REGION \
    --notification-configuration='{ "EventBridgeConfiguration": {} }'

---

Conclusions

This was a pretty good project and I’m glad I modified the idea and worked on it using both management console and aws-cli commands.

I’ll be skipping Project 3 as I was not that much interested in the services it uses; but definitely saved for when I might want to try it out.

Project 4 is also out now and it looks pretty good: uses ECS (Fargate) that I already tried here, services that I want to refresh my knowledge on (API Gateway, Load Balancing) and some other interesting ones for the enhancements.

Till Next time✨