Configure AWS Lamba programatically with Boto3

Serverless computing has generated a lot of interest in the development community. AWS Lambda is the AWS version of Serverless. In this post, we will look at configuring AWS Lambda programmatically with boto3.  We will configure a lambda function that connects to a Postgres DB on an EC2 instance in a private VPC using sqlalchemy. This would need packaging of the dependencies of sqlalchemy along with the lambda function.

By default, the instance where AWS lambda is running has Boto installed (AWS Python SDK). So, importing a boto3 package in python code will work without any other packaging. But importing a package like “import sqlalchemy” will lead to python error “Unable to import module sqlalchemy”. This article shows how to achieve the same.

Step 1: Write the lambda function

First we need to write the python code which will run as a lambda. This below code creates a python file and zips it. Note down, the python code that we wanted to execute should contain a method called lambda_handler which serves as an entry point for the lambda to start the execution.

One quick aside: SqlAlchemy complains about not being able to find the right provider when connecting to postgres from lambda. So might have to add postgressql + pscycopg2 as part of connection string. Don't forget to open the port in the security group and enable incoming connections from all in the host file of the Postgres server. This can be a cause of a lot of heartaches.
Also, remember if you the system you package the code is windows, then you are in some trouble as the pip will pick windows related drivers. So you need to run this on lambda compatible OS (read as Amazon Linux or compatible OS). Else, you will end up with a lot of wierd run time errors.

Step 2:  Create the lambda function

Now use boto3 client API create_function to map it to the above code. If you take a close look, Handler=‘lambda_function.lambda_handler’ means that it will look for the python file lambda_function.py and inside it it will look for lambda_handler function. Also, you need to pass the right arn for this API to work.

Step 3: Set the frequency of lambda 

Set the rule and the frequency at which you want the lambda to run, it can be a simple expression like rate(1 minute) or rate(5 minutes) or a cron expression too. Also, set the rule target and grant permission for the lambda to execute. In this case, we are configuring the rule to run every minute.

Step 4: Additional Package Deployment

Do this step if we need other packages for our python code to run. For creating deployment folders, you can use this python script create_deployment.py and requirements.txt. Basically, when the script file runs, it picks the packages that need to be installed from requirements.txt and does a pip -install -t and which creates a folder, pull all the packages and zips them. If you need to have a particular version you can say sseclient==0.0.11 or you can omit the version to pull the latest.

Final Step: Putting it together

Now putting everything together yields the following output in the AWS CloudWatch logs, no import module error this time. This can be modified to use whichever package you need for your development purposes.

Hope this saves a few hours trying to jump a few hoops which weren’t really apparent at the onset!

References:

AWS Lambda Functions Made Easy

AWS Lambda: Programmatically scheduling a CloudWatchEvent

 

vCPUs and cost implications in right-sizing of for EC2 instances

vCPUs are the base unit of compute capacity for EC2 instances. They represent one of the main criteria for selecting an EC2 instance type.

What does vCPU really represent on a physical host? and more importantly – are all vCPUs the same?  Let us explore this in a bit of detail. The cost implications will also become clear once we understand it.

Each vCPU is a hyper-thread of an Intel Xeon core (except for T2). More details here

T2 instances have a slightly winding explanation and feature ‘burstable’ or occasional spikes that the originally available compute capacity.

Note that the definition says hyper-threaded instead of virtual core..  Or about 2 hyper-threads per core. Als,o note that doesn’t mean the hyper-threads share 50% of each core.

If you want to get a  good understanding of hyper-threading, take some time to read this excellent article from ArsTechnica about HT or SMT.

The interesting aspect to note ( and something that is often lost to those new to virtualization) is that vCPUs are not a standard measure in a way a ‘meter’ or ‘ mile’ is defined.  This merely indicates that from the underlying host,  ‘n’ vCPUs are virtualized.

Compute capacity also depends upon other host characteristics such as cache, bus speed and QPI of the host itself would change.

E.g. vCPUs based on Intel Xeon 2676 ( up to m4.8x large) and vCPUs based on Intel Xeon 2686 ( m4.10xlarge and greater), m4.10xlarge or above would give better performance per vCPU on account of the difference in the host ( benchmark comparison for both Xeon processors )

The amount of performance you can get out of an instance is also dependent on how much processors state can be controlled via Linux ( e.g. C-State and P-State optimizations )

And this great video:

AWS re:Invent 2017: Deep Dive on Amazon EC2 Instances, Featuring Performance Optimize (CMP301)

 

Let’s consider an example.

Consider C5.2xLarge and C4.2xLarge instance types.

Name

API Name

Memory

vCPUs

Instance Storage

Network Performance

Linux On Demand cost

Linux Reserved cost

Windows On Demand cost

Windows Reserved cost

C5 High-CPU Double Extra Large

c5.2xlarge

16.0 GiB

8 vCPUs

EBS only

Up to 10 Gbps

$0.340000 hourly

$0.214000 hourly

$0.708000 hourly

$0.582000 hourly

C4 High-CPU Double Extra Large

c4.2xlarge

15.0 GiB

8 vCPUs

EBS only

High

$0.398000 hourly

$0.252000 hourly

$0.766000 hourly

$0.620000 hourly

From ec2instances.info

While both have the same 8 vCPUs and almost similar memory.

The C5 family uses 3.0 GHz Intel Xeon Platinum processors with new Intel Advanced Vector Extension 512 (AVX-512) instruction set and C4 family uses Intel Xeon E5-2666 v3. C5 2x large has a better spec, marginally lower cost per hour, better virtualisation model (Nitro) with less variability and delivers up to 25% more performance compared to C4 instance types. However, C5 does need AMIs that include drivers for ENA and NVMe.

On a general note and at risk of broad oversimplification, vCPUs of current generation instances give better compute performance compared to previous generation( C5 better than C4 ) and instances of higher instance types give better performance per unit cost compared to their lower siblings ( e.g. m4.10xlarge better than m4.8xlarge).   A good practical method suggested  by AWS is here

Conclusion

While the vCores give valuable information about the compute capacity, taking the number of cores alone into consideration during selection of instances can trip your application up in unexpected ways. Any choice of instance type should take into all the other options available and benchmark them before finalizing on the instance type.

A service like insisive cloud can help analyze your AWS infrastructure using rule-based policies for cost optimization and help with rapid prototyping at costs up to 60% less.

Sign up for a 14-day no-obligation trial to see if it is suitable for your application needs.

 

Is Cost Optimization on the Cloud essential?

Cloud computing ushered in a new era of low-cost computing and rapid development with the help of managed services that help with undifferentiated heavy lifting. As many enterprises have found out, the road to the promised land has many challenges and barriers. When does Cost Optimization in the Cloud become relevant?  Enterprises typically follow  Cloud Transformation Maturity Model and we can look to it to give a clue.

Stages of Cloud Adoption: 

Enterprises typically follow the 5 stages of cloud adoption. Smaller enterprises and startups typically follow an abridged version of the plan or skip entire steps and take an opinionated approach right from the beginning based on their teams’  prior experience.

Evaluation and Prototyping :

This is the stage when there is an initiative – usually at the CTO office level to start towards migration to the cloud. The focus of the initiative at this stage is on high-level goals and evaluation of multiple cloud vendors to find a good fit at a strategic level.  The initiatives are well-funded at this stage ( compared to the scale of operations). They shortlist a vendor/ vendors, build some small POCs to evaluate the platform, the tooling around it, how much customization is possible and the global reach of the provider. There is no real application development at this stage. Based on the evaluation, they choose a vendor, prepare a roadmap and identify pilot applications to move to the cloud.

Pilot Migration :

They identify one of the teams to migrate their application to the cloud. The focus at this point is to lift and shift the application from on-premise to the cloud. The team is not really skilled at the best practices of the cloud. Once the application starts working in the cloud, they look at non-functional aspects such as privacy, security. This typically takes up a lot of time and effort as the enterprise is initially very cautious (rightly so) with the threat assessments. The R&D team also realizes that there is more to moving to the cloud from a non-functional perspective and starts factoring in that time and effort. At this stage, they will also formulate some extra guidelines in terms of security and privacy policies to follow during development like security by design and privacy by design. Cost is typically not a focus at this point.

Taking a single application to Production. Adoption by multiple development teams:

The team which started development looks at taking the application live on the cloud. Other R&D teams star the process of either new development or moving their applications. The key thing to note is that while there is focus on getting security/privacy right, there isn’t much focus or appreciation on shifts needed to make cloud adoption successful. This is typically the time when the organizations feel the growing pains on the cloud. Since there is no clear-cut guidance, each team takes a slightly different approach. The very reason that motivated the company to choose the cloud provider – the flexibility it offers for a variety of scenarios works in bringing all-around chaos.  The IT team suddenly finds itself overwhelmed by different types of requests. The lack of coherent policies leads to an explosion in costs. This leads to intense scrutiny of the cloud costs. The teams look for better ways of Cost Optimisation.

Cloud Optimization:

The teams formulate a governance mechanism on how and when to use a set of resources, put in place a set of tools to automate actions prescribed in the governance mechanism. The teams also reorganize to best manage their applications in a self-service way.  They institutionalize the use of automation and DevOps. Building Cost-aware applications become a focus for the teams since they own the applications and their management.  There is a broad realization that while the Cloud can deliver massive cost savings, it takes a significant change in the way they architect the applications, how they need to think about their infrastructure about ( e.g programmable and disposable) and how the teams manage the applications. Cost Optimization in the cloud requires a mix of tactical measures in terms of curbing wasted expenditure, putting in place a governance mechanism and re-architecting the applications to be Cloud Native.

Being Cloud Native:

By this stage, the organization is well versed with Cloud Native practices and the starts application development using the  Cloud-Native best practices. The organization institutionalized Cost optimization and it becomes a part of regular DevOps process. There is a high level of automation and continuous data-driven monitoring, analysis, and remedial actions. This keeps the cloud costs under check and the organizations realize the true gains of moving to the cloud.

Cost Optimization:

Enterprises typically see a need for cost optimization once they are in the third stage of the maturity cycle. As the scale of cloud adoption increases, formulating a set of policies, practices and having a tool set for automating many of the activities is essential to manage the cloud adoption complexities.

As the  excellent AWS Cloud Transformation Maturity Model white paper from Amazon indicates,

AWS Cost Transformation Maturity Model
Courtesy: AWS Cloud Transformation Maturity Model

one of the key transformations is to “Implement a continuous cost optimization process – Either the designated resources on a CCoE or a group of centralized staff from IT Finance must be trained to support an ongoing process using AWS or third-party cost-management tools to assess costs and optimize savings“.

and the outcomes to measure the organizational maturity as optimized is

Optimized cost savings – Your organization has an ongoing process and a team focused on continually reviewing AWS usage across your organization, and identifying cost-reduction opportunities.

The cost optimization becomes an essential and continuous activity on the cloud.

 

Visit us at www.insisiv.com to know more on how