Tuesday, March 19, 2019

phpMyPassion

Process to Create AWS Lambda Function in Python


In this article I am sharing the full and easy process of creating a AWS lambda function using python. So follow step by step process for creating AWS lambda using python 3.

********************** process to create lambda function *****************


> fill basic information like name, runtime envoironment, permission



*******************Now add Trigger from the list in left side****************


In my case I used s3

-> configure triggers basics
-> select s3 Bucket [bucket name on that you want run your trigger]
-> event type {like bucket PUT}
-> prefix
-> suffix [like - .csv, .jpg]



Note:- you can either choose prefix or suffix.

*********************** upload your lambda function code ****************


In my case I had to produced csv name as massage that was automatically uploaded in s3 bucket by python program for athena query result. So I produced massage by kafka producer on s3 put event trigger.

For setting up kafka producer on lambda trigger I did below steps..

#Install kafka in your local machine and create .zip of whole package.

Follow below steps:-

"""
# https://pypi.org/project/kafka-python/

# pip install kafka-python -t <FOLDER PATH>
# pip install requests -t <FOLDER PATH>

# README:
* Please Install the Kafka Python & Request Libs in the same folder for it to  work
    in AWS Lambda.
* When Uploading on AWS Lambda, zip the entire folder. 
   If you are uploading a zip file.  Make sure that you are zipping the contents 
   of the directory and not the directory itself.  
   Else you will get an Error: aws lambda unable to import module

zip -r ../build_dmp_dashboard_from_athena.zip *

"""

#Now you have to create a file for produced massage on trigger as below -

Lets create a file first with name lambda_producer.py

from __future__ import print_function
import sys, time
import json
import requests
from kafka import KafkaProducer

########################### Configurations #############################################
email_api = 'https://api.phpmypassion.com/api/send-mail'
email_to = '[email protected]'

kafka_server_cluster = ['cluster1', 'cluster2', 'cluster3'] 
topic = "athena_query_result"

"""
Send Email
"""
def sendmail(message):
  msg = ("{}:{}".format("lambda_producer", message))
  data = {
    'toEmail': email_to,
    'mailSubject': msg,
    'mailBody': msg
  }

  return requests.post(email_api, data)


"""
Producer Connection
"""
exception_counter = 0
for x in range(3):
  try:
    producer = KafkaProducer(bootstrap_servers=kafka_server_cluster)
    break

  except Exception as e:
    exception_counter = exception_counter + 1
    print(e)
    print("Error in Connection to kafka cluster: {0}".format(kafka_server_cluster))
    time.sleep(2)  # 2 Sec

if (exception_counter >= 2):
  message = ('Error: Looks like connection to kafka cluster failed on lambda producer: {0}'.format(kafka_server_cluster))
  print(message)
  response = fnc_sendmail(message)
  print("Email Sent Response: {0}".format(response))
  print("Program exiting")
  sys.exit(2)

"""
Lambda Handler. Received the trigger from the Code Commit Repositoy of AWS and  Produces
a message on kafka Queue
"""
def lambda_handler(event, context):
  print("Received event : " + json.dumps(event, indent=2))

  # Parse the S3 Trigger and Get the CSV Path & File from the "event"
  key_file_name = (event['Records'][0]['s3']['object']['key']) 

  print("We have a new file. file name & path : ", key_file_name)
  try:
    producer_ret_val = producer.send(topic, key_file_name.encode())  # If Topic Does not Exist then it get created automatically by python Producer
    record_metadata = producer_ret_val.get(timeout=10)
    print("sent event to Kafka! topic {} partition {} offset {}".format(record_metadata.topic, record_metadata.partition, record_metadata.offset))

  except Exception as e:
    print(e)
    print("Some Error in lambda_producer")
    raise e



#Now Lets setup thin within Aws lambda:-

-> choose "code entry type" and select the option to upload .zip


-> upload your zip file and click to save

-> your lambda code has been uploaded.

-> in my case I uploaded kafka configuration zip folder

#You have to also change lambda handler function too. as you are using lambda_handler function within lambda_producer.py file so you have to fill "lambda_producer.lambda_handler" within the above input box [shows in right red line above image].

#Now click on save button.

You done the all steps.




About Author -

Hi, I am Anil.

Welcome to my eponymous blog! I am passionate about web programming. Here you will find a hug of information on web development, web design, PHP, Python, Digital Marketing and Latest technology.

Subscribe to this Blog via Email :