Process to Create AWS Lambda Function in Python
In this article I am sharing the full and easy process of creating a AWS lambda function using python. So follow step by step process for creating AWS lambda using python 3.
> fill basic information like name, runtime envoironment, permission
In my case I used s3
-> configure triggers basics
-> select s3 Bucket [bucket name on that you want run your trigger]
-> event type {like bucket PUT}
-> prefix
-> suffix [like - .csv, .jpg]
Note:- you can either choose prefix or suffix.
In my case I had to produced csv name as massage that was automatically uploaded in s3 bucket by python program for athena query result. So I produced massage by kafka producer on s3 put event trigger.
For setting up kafka producer on lambda trigger I did below steps..
#Install kafka in your local machine and create .zip of whole package.
Follow below steps:-
#Now you have to create a file for produced massage on trigger as below -
Lets create a file first with name lambda_producer.py
#Now Lets setup thin within Aws lambda:-
-> choose "code entry type" and select the option to upload .zip
-> upload your zip file and click to save
-> your lambda code has been uploaded.
-> in my case I uploaded kafka configuration zip folder
#You have to also change lambda handler function too. as you are using lambda_handler function within lambda_producer.py file so you have to fill "lambda_producer.lambda_handler" within the above input box [shows in right red line above image].
#Now click on save button.
You done the all steps.
********************** process to create lambda function *****************
> fill basic information like name, runtime envoironment, permission
*******************Now add Trigger from the list in left side****************
In my case I used s3
-> configure triggers basics
-> select s3 Bucket [bucket name on that you want run your trigger]
-> event type {like bucket PUT}
-> prefix
-> suffix [like - .csv, .jpg]
Note:- you can either choose prefix or suffix.
*********************** upload your lambda function code ****************
In my case I had to produced csv name as massage that was automatically uploaded in s3 bucket by python program for athena query result. So I produced massage by kafka producer on s3 put event trigger.
For setting up kafka producer on lambda trigger I did below steps..
#Install kafka in your local machine and create .zip of whole package.
Follow below steps:-
"""
# https://pypi.org/project/kafka-python/
# pip install kafka-python -t <FOLDER PATH>
# pip install requests -t <FOLDER PATH>
# README:
* Please Install the Kafka Python & Request Libs in the same folder for it to work
in AWS Lambda. * When Uploading on AWS Lambda, zip the entire folder.
If you are uploading a zip file. Make sure that you are zipping the contents
of the directory and not the directory itself.
Else you will get an Error: aws lambda unable to import module
zip -r ../build_dmp_dashboard_from_athena.zip *
"""
#Now you have to create a file for produced massage on trigger as below -
Lets create a file first with name lambda_producer.py
from __future__ import print_function import sys, time import json import requests from kafka import KafkaProducer ########################### Configurations ############################################# email_api = 'https://api.phpmypassion.com/api/send-mail' email_to = '[email protected]' kafka_server_cluster = ['cluster1', 'cluster2', 'cluster3'] topic = "athena_query_result" """ Send Email """ def sendmail(message): msg = ("{}:{}".format("lambda_producer", message)) data = { 'toEmail': email_to, 'mailSubject': msg, 'mailBody': msg } return requests.post(email_api, data) """ Producer Connection """ exception_counter = 0 for x in range(3): try: producer = KafkaProducer(bootstrap_servers=kafka_server_cluster) break except Exception as e: exception_counter = exception_counter + 1 print(e) print("Error in Connection to kafka cluster: {0}".format(kafka_server_cluster)) time.sleep(2) # 2 Sec if (exception_counter >= 2): message = ('Error: Looks like connection to kafka cluster failed on lambda producer: {0}'.format(kafka_server_cluster)) print(message) response = fnc_sendmail(message) print("Email Sent Response: {0}".format(response)) print("Program exiting") sys.exit(2) """ Lambda Handler. Received the trigger from the Code Commit Repositoy of AWS and Produces a message on kafka Queue """ def lambda_handler(event, context): print("Received event : " + json.dumps(event, indent=2)) # Parse the S3 Trigger and Get the CSV Path & File from the "event" key_file_name = (event['Records'][0]['s3']['object']['key']) print("We have a new file. file name & path : ", key_file_name) try: producer_ret_val = producer.send(topic, key_file_name.encode()) # If Topic Does not Exist then it get created automatically by python Producer record_metadata = producer_ret_val.get(timeout=10) print("sent event to Kafka! topic {} partition {} offset {}".format(record_metadata.topic, record_metadata.partition, record_metadata.offset)) except Exception as e: print(e) print("Some Error in lambda_producer") raise e
#Now Lets setup thin within Aws lambda:-
-> choose "code entry type" and select the option to upload .zip
-> your lambda code has been uploaded.
-> in my case I uploaded kafka configuration zip folder
#You have to also change lambda handler function too. as you are using lambda_handler function within lambda_producer.py file so you have to fill "lambda_producer.lambda_handler" within the above input box [shows in right red line above image].
#Now click on save button.
You done the all steps.
