• Home
  • Insisive Cloud Insights

Is Cost Optimization on the Cloud essential?

Featured

Cloud computing ushered in a new era of low-cost computing and rapid development with the help of managed services that help with undifferentiated heavy lifting. As many enterprises have found out, the road to the promised land has many challenges and barriers. When does Cost Optimization in the Cloud become relevant?  Enterprises typically follow  Cloud Transformation Maturity Model and we can look to it to give a clue.

Stages of Cloud Adoption: 

Enterprises typically follow the 5 stages of cloud adoption. Smaller enterprises and startups typically follow an abridged version of the plan or skip entire steps and take an opinionated approach right from the beginning based on their teams’  prior experience.

Evaluation and Prototyping :

This is the stage when there is an initiative – usually at the CTO office level to start towards migration to the cloud. The focus of the initiative at this stage is on high-level goals and evaluation of multiple cloud vendors to find a good fit at a strategic level.  The initiatives are well-funded at this stage ( compared to the scale of operations). They shortlist a vendor/ vendors, build some small POCs to evaluate the platform, the tooling around it, how much customization is possible and the global reach of the provider. There is no real application development at this stage. Based on the evaluation, they choose a vendor, prepare a roadmap and identify pilot applications to move to the cloud.

Pilot Migration :

They identify one of the teams to migrate their application to the cloud. The focus at this point is to lift and shift the application from on-premise to the cloud. The team is not really skilled at the best practices of the cloud. Once the application starts working in the cloud, they look at non-functional aspects such as privacy, security. This typically takes up a lot of time and effort as the enterprise is initially very cautious (rightly so) with the threat assessments. The R&D team also realizes that there is more to moving to the cloud from a non-functional perspective and starts factoring in that time and effort. At this stage, they will also formulate some extra guidelines in terms of security and privacy policies to follow during development like security by design and privacy by design. Cost is typically not a focus at this point.

Taking a single application to Production. Adoption by multiple development teams:

The team which started development looks at taking the application live on the cloud. Other R&D teams star the process of either new development or moving their applications. The key thing to note is that while there is focus on getting security/privacy right, there isn’t much focus or appreciation on shifts needed to make cloud adoption successful. This is typically the time when the organizations feel the growing pains on the cloud. Since there is no clear-cut guidance, each team takes a slightly different approach. The very reason that motivated the company to choose the cloud provider – the flexibility it offers for a variety of scenarios works in bringing all-around chaos.  The IT team suddenly finds itself overwhelmed by different types of requests. The lack of coherent policies leads to an explosion in costs. This leads to intense scrutiny of the cloud costs. The teams look for better ways of Cost Optimisation.

Cloud Optimization:

The teams formulate a governance mechanism on how and when to use a set of resources, put in place a set of tools to automate actions prescribed in the governance mechanism. The teams also reorganize to best manage their applications in a self-service way.  They institutionalize the use of automation and DevOps. Building Cost-aware applications become a focus for the teams since they own the applications and their management.  There is a broad realization that while the Cloud can deliver massive cost savings, it takes a significant change in the way they architect the applications, how they need to think about their infrastructure about ( e.g programmable and disposable) and how the teams manage the applications. Cost Optimization in the cloud requires a mix of tactical measures in terms of curbing wasted expenditure, putting in place a governance mechanism and re-architecting the applications to be Cloud Native.

Being Cloud Native:

By this stage, the organization is well versed with Cloud Native practices and the starts application development using the  Cloud-Native best practices. The organization institutionalized Cost optimization and it becomes a part of regular DevOps process. There is a high level of automation and continuous data-driven monitoring, analysis, and remedial actions. This keeps the cloud costs under check and the organizations realize the true gains of moving to the cloud.

Cost Optimization:

Enterprises typically see a need for cost optimization once they are in the third stage of the maturity cycle. As the scale of cloud adoption increases, formulating a set of policies, practices and having a tool set for automating many of the activities is essential to manage the cloud adoption complexities.

As the  excellent AWS Cloud Transformation Maturity Model white paper from Amazon indicates,

AWS Cost Transformation Maturity Model
Courtesy: AWS Cloud Transformation Maturity Model

one of the key transformations is to “Implement a continuous cost optimization process – Either the designated resources on a CCoE or a group of centralized staff from IT Finance must be trained to support an ongoing process using AWS or third-party cost-management tools to assess costs and optimize savings“.

and the outcomes to measure the organizational maturity as optimized is

Optimized cost savings – Your organization has an ongoing process and a team focused on continually reviewing AWS usage across your organization, and identifying cost-reduction opportunities.

The cost optimization becomes an essential and continuous activity on the cloud.

 

Visit us at www.insisiv.com to know more on how

 

Diciphering the arcane aws price list API

AWS released their price list API around 4 years ago, yet, it seems so esoteric in nature. With the SKU values, product codes, and 100 variants of the same product, it isn’t exactly easy to use. As my company deals with AWS cost optimization the price list API is quite imperative for us to understand.

Personally for me, and as I read online for a lot of others too, the API just seems to have too much going on with it for people to use it easily and effectively. For starters, the output is absolute JSON hell. Despite all this, the API comes quite handy when used correctly.

Say you’re a data scientist who wants to know the cost incurred by all the EC2 instances in your EMR cluster, or you’re a DevOps engineer who wants a ballpark figure of all the EC2 instances being provisioned by your development team, this API is a handy addition to your toolkit.

Initially, I tried looking up some examples on how to use the API and I stumbled upon this pricing module written by Lyft, but it wasn’t all that helpful.  After tinkering around with the API for about two weeks I realized how much time I could have saved had I known a more systematic way to get the prices. Since I didn’t find any content online of much use, I thought I’d write a post about it in hopes that it would help someone else.

Now, time to get our hands dirty, but not too dirty, because we just want a quick way to get the price of a product.

Getting the price of an EC2 instance

Creating the pricing client

pricing = boto3.client('pricing')

Filters

The filters I used were:

  1. Operating System
  2. Region Name
  3. Instance Type
  4. Tenancy
  5. Product Family
  6. Usage Type

OS : The os filter takes the values ‘SUSE’, ‘RHEL’, ‘Windows’, ‘Linux’ only.

Region: The regions filter must use the actual region name. For example, for N. Virginia, the filter should be ‘US East (N. Virginia)’ and not ‘us-east-1’. I created this handy dictionary that maps the different names of the regions accordingly.

region_mapping_dict = {'us-east-2': 'US East (Ohio)',
                            'us-east-1': 'US East (N. Virginia)',
                            'us-west-1': 'US West (N. California)',
                            'us-west-2': 'US West (Oregon)',
                            'ap-south-1': 'Asia Pacific (Mumbai)',
                            'ap-northeast-2': 'Asia Pacific (Seoul)',
                            'ap-southeast-1': 'Asia Pacific (Singapore)',
                            'ap-southeast-2': 'Asia Pacific (Sydney)',
                            'ap-northeast-1': 'Asia Pacific (Tokyo)',
                            'ca-central-1': 'Canada (Central)',
                            'cn-north-1': 'China (Beijing)',
                            'cn-northwest-1': 'China (Ningxia)',
                            'eu-central-1': 'EU (Frankfurt)',
                            'eu-west-1': 'EU (Ireland)',
                            'eu-west-2': 'EU (London)',
                            'eu-west-3': 'EU (Paris)',
                            'eu-north-1': 'EU (Stockholm)',
                            'sa-east-1': 'South America (Sao Paulo)'}

Instance Type: The instance type is mentioned in this field. For example, t2.micro, t2.large, etc.

Tenancy: AWS by default offers Shared tenancy unless otherwise specified by the user for it to be Dedicated.

Product Family: The product family in case of EC2 is ‘Compute Instance’.

Usage Type: To be more specific about the kind of instance you want to get the price of the usage type filter is required. Usage types are the units that each service uses to measure the usage of a specific type of resource. For example, the BoxUsage:t2.micro(Hrs) usage type filters by the running hours of Amazon EC2 t2.micro instances. The reason to add this filter is to ensure that you get only on-demand instances as part of our output. This might seem redundant when you see the JSON output as we get this by default without using the filter. But if any changes are made with the pricing and something new is added by AWS this filter still ensures that you get on-demand instances. As of now, this field can be skipped and it would still retrieve the right data but it wouldn’t be advisable to. The filter is as follows,

usage_type = box_usage[region] + instance_type

Again, this field changes region wise. After looking at the pricing API I came up with this simple dictionary that helped solve the region issue for this filter.

box_usage = {'us-east-2': 'USE2-BoxUsage:',
             'us-east-1': 'BoxUsage:',
             'us-west-1': 'USW1-BoxUsage:',
             'us-west-2': 'USW2-BoxUsage:',
             'ap-south-1': 'APS3-BoxUsage:',
             'ap-northeast-3': 'APN3-BoxUsage:',
             'ap-northeast-2': 'APN2-BoxUsage:',
             'ap-southeast-1': 'APS1-BoxUsage:',
             'ap-southeast-2': 'APS2-BoxUsage:',
             'ap-northeast-1': 'APN1-BoxUsage:',
             'ca-central-1': 'CAN1-BoxUsage:',
             'eu-central-1': 'EUC1-BoxUsage:',
             'eu-west-1': 'EUW1-BoxUsage:',
             'eu-west-2': 'EUW2-BoxUsage:',
             'eu-west-3': 'EUW3-BoxUsage:',
             'eu-north-1': 'EUW1-BoxUsage:',
             'sa-east-1': 'SAE1-BoxUsage:'}

Getting the data

Applying the above-mentioned filters to the get_products API we get the JSON data for all the variants that qualify.

data = pricing.get_products(ServiceCode='AmazonEC2', Filters=
[
    {'Type': 'TERM_MATCH', 'Field': 'operatingSystem', 'Value': 'RHEL'},
    {'Type': 'TERM_MATCH', 'Field': 'location', 'Value': 'Asia Pacific (Mumbai)'},
    {'Type': 'TERM_MATCH', 'Field': 'instanceType', 'Value': 't2.micro'},
    {'Type': 'TERM_MATCH', 'Field': 'tenancy', 'Value': 'Shared'},
    {'Type': 'TERM_MATCH', 'Field': 'preInstalledSw', 'Value': 'NA'},
    {'Type': 'TERM_MATCH', 'Field': 'productFamily', 'Value': 'Compute Instance'}
])

The above boto3 code gives us the price t2.micro Red Hat Enterprise Linux instance without any pre-installed software(the ‘preInstalledSw’ is NA) for the Mumbai region.

Getting the price from JSON data

Now, the output of this is not pretty, to say the least. Parsing through the complex JSON alone took me so much time until I stumbled upon this extremely elegant solution to this problem.

def extract_values(obj, key):
    """Pull all values of specified key from nested JSON."""
    arr = []

    def extract(obj, arr, key):
        """Recursively search for values of key in JSON tree."""
        if isinstance(obj, dict):
            for k, v in obj.items():
               if arr not None:
                break
                if isinstance(v, (dict, list)):
                    extract(v, arr, key)
                elif k == key:
                    arr.append(v)
        elif isinstance(obj, list):
            for item in obj:
                extract(item, arr, key)
        return arr

    results = extract(obj, arr, key)
    return results

The above code snippet is taken from https://hackersandslackers.com/extract-data-from-complex-json-python/

The above code parses through the JSON to give us the price of the instance without any additional features that are present in the JSON result. In the above code, ‘obj’ is the JSON data result and the ‘key’ is ‘USD’.

Putting it all together

Here is the sample code I wrote which puts all the above pieces of the puzzle together and gives us a view of the bigger picture.

import boto3
import json


class CostEstimation:

    def __init__(self):
        self.region_mapping_dict = {'us-east-2': 'US East (Ohio)',
                                    'us-east-1': 'US East (N. Virginia)',
                                    'us-west-1': 'US West (N. California)',
                                    'us-west-2': 'US West (Oregon)',
                                    'ap-south-1': 'Asia Pacific (Mumbai)',
                                    'ap-northeast-3': 'Asia Pacific (Osaka-Local)',
                                    'ap-northeast-2': 'Asia Pacific (Seoul)',
                                    'ap-southeast-1': 'Asia Pacific (Singapore)',
                                    'ap-southeast-2': 'Asia Pacific (Sydney)',
                                    'ap-northeast-1': 'Asia Pacific (Tokyo)',
                                    'ca-central-1': 'Canada (Central)',
                                    'cn-north-1': 'China (Beijing)',
                                    'cn-northwest-1': 'China (Ningxia)',
                                    'eu-central-1': 'EU (Frankfurt)',
                                    'eu-west-1': 'EU (Ireland)',
                                    'eu-west-2': 'EU (London)',
                                    'eu-west-3': 'EU (Paris)',
                                    'eu-north-1': 'EU (Stockholm)',
                                    'sa-east-1': 'South America (Sao Paulo)'}

        self.pricing = boto3.client('pricing', region_name='us-east-1')

    def extract_values(self, obj, key):
        """Pull all values of specified key from nested JSON."""
        arr = []

        def extract(obj, arr, key):
            """Recursively search for values of key in JSON tree."""
            if isinstance(obj, dict):
                for k, v in obj.items():
                    if arr != []:
                        break
                    if isinstance(v, (dict, list)):
                        extract(v, arr, key)
                    elif k == key:
                        arr.append(v)
            elif isinstance(obj, list):
                for item in obj:
                    extract(item, arr, key)
            return arr

        results = extract(obj, arr, key)
        return results

    def get_instance_price(self, os, instance_type, region):

        for key, value in self.region_mapping_dict.items():
            if (region == key):
                region_name = value
                break
        price = 0
        try:

            box_usage = {'us-east-2': 'USE2-BoxUsage:',
                         'us-east-1': 'BoxUsage:',
                         'us-west-1': 'USW1-BoxUsage:',
                         'us-west-2': 'USW2-BoxUsage:',
                         'ap-south-1': 'APS3-BoxUsage:',
                         'ap-northeast-3': 'APN3-BoxUsage:',
                         'ap-northeast-2': 'APN2-BoxUsage:',
                         'ap-southeast-1': 'APS1-BoxUsage:',
                         'ap-southeast-2': 'APS2-BoxUsage:',
                         'ap-northeast-1': 'APN1-BoxUsage:',
                         'ca-central-1': 'CAN1-BoxUsage:',
                         'cn-north-1': 'BoxUsage:',
                         'cn-northwest-1': 'BoxUsage:',
                         'eu-central-1': 'EUC1-BoxUsage:',
                         'eu-west-1': 'EUW1-BoxUsage:',
                         'eu-west-2': 'EUW2-BoxUsage:',
                         'eu-west-3': 'EUW3-BoxUsage:',
                         'eu-north-1': 'EUW1-BoxUsage:',
                         'sa-east-1': 'SAE1-BoxUsage:'}

            usage_type = box_usage[region] + instance_type
            pricing = boto3.client('pricing', region_name='us-east-1')

            data = pricing.get_products(ServiceCode='AmazonEC2', Filters=
            [
                {'Type': 'TERM_MATCH', 'Field': 'operatingSystem', 'Value': os},
                {'Type': 'TERM_MATCH', 'Field': 'location', 'Value': region_name},
                {'Type': 'TERM_MATCH', 'Field': 'instanceType', 'Value': instance_type},
                {'Type': 'TERM_MATCH', 'Field': 'tenancy', 'Value': 'Shared'},
                {'Type': 'TERM_MATCH', 'Field': 'preInstalledSw', 'Value': 'NA'},
                {'Type': 'TERM_MATCH', 'Field': 'capacitystatus', 'Value': 'used'},
                {'Type': 'TERM_MATCH', 'Field': 'usagetype', 'Value': usage_type},
                {'Type': 'TERM_MATCH', 'Field': 'productFamily', 'Value': 'Compute Instance'}
            ])

            for value in data['PriceList']:
                json_value = json.loads(value)
            price = self.extract_values(json_value, 'USD')
            ec2_price_per_hour = price[0]

        except Exception as e:
            print(str(e))

        return ec2_price_per_hour


def main():
    os = 'Linux'
    instance_type = 't2.micro'
    region = 'us-west-1'

    test = CostEstimation()
    value = test.get_instance_price(os, instance_type, region)
    cost_for_a_day = float(value) * 24 * 30
    print(cost_for_a_day)


main()

The above code gives the cost of running an ec2 instance of type t2.micro in the N. California region for a month.

The above method to programmatically get the cost of a product can also be applied to other services like load balancers, volumes, etc, with a couple of modifications.

For example, to get the price of a load balancer, you would just have to change the filters to the following

Filters=[
    {'Type': 'TERM_MATCH', 'Field': 'productFamily',
     'Value': 'Load Balancer'},
    {'Type': 'TERM_MATCH', 'Field': 'location',
     'Value': region_name},
    {'Type': 'TERM_MATCH', 'Field': 'groupDescription',
     'Value': 'Standard Elastic Load Balancer'}
]

Once the above filters are applied, you can get the price of the load balancer by following the same steps and mentioned above.

 

The new A1 EC2 Instance

At re: Invent 2018 AWS  introduced A1 instances. Compared to the other instance types they cost 40% lesser ‘per core’. Given that they use the Nitro Hypervisor, they give a better performance as well compared to traditional Xen Hypervisor based instances.

However, before you go and move all your instance types, it is good to know when you can use these instance types and the effort required in moving your workload to use these instances.

ms-clipboard-file://C:/Users/Geetha/AppData/Local/Packages/Microsoft.Office.OneNote_8wekyb3d8bbwe/TempState/msohtmlclip/clip_image001.png

A1 instances have ARM-based processors. If your workloads compile to native code using x86 architecture, then you would need to recompile for ARM platform before you can run on them.

For scripting based languages, it could be negligible ( as long as they do not use some module that is native in their dependency chain).

If you use Docker containers, It is relatively quick as mentioned in https://blog.boltops.com/2018/12/16/ec2-a1-instance-with-aws-homegrown-arm-processor-easy-way-to-save-40.

Amazon Linux, Ubuntu Linux, and Red Hat Enterprise Linux are the initial operating systems with ARMv8 support on EC2.

A1 instances have slightly dated ARM A72 processors (released in 2015. The current generation is A76) that are aimed at high-end smartphones and tablets. So they aren’t meant for same workloads such as Xeon E5 series powering the Cx series instance types for servers. In terms of benchmarks by Phoronix https://www.phoronix.com/scan.php?page=article&item=ec2-graviton-performance&num=1, both Intel and AMD based instances far outperform the current gen A1 instances.

Interestingly enough is the price/performance per dollar

ms-clipboard-file://C:/Users/Geetha/AppData/Local/Packages/Microsoft.Office.OneNote_8wekyb3d8bbwe/TempState/msohtmlclip/clip_image002.png

Courtesy: Phoronix benchmark

In terms of ‘real world’ test of hosting website, they still underperform by about 3.5x ( albeit at a lower cost). (https://www.theregister.co.uk/2018/11/27/amazon_aws_graviton_specs/)

Currently, A1 instances are not meant for general purpose workloads. However, owing to the Nitro system based hypervisor, they will be very useful as part of scale-out workloads, lightweight web servers,

containerized micro-services, caching fleets and such.

However, there is a larger trend at play which will be beneficial to customers in the long run. Amazon bringing their own processor into the mix with Intel and AMD will improve choices and hopefully reduce costs in the long run.

Amazon’s new and Intelligent S3 storage class:

What is S3?

Amazon Simple Storage Service (Amazon S3) is a global Infrastructure as a Service (IaaS) solution. Amazon S3 facilitates highly scalable, secured and low-latency data storage from the cloud.

Yet another storage class?

Prior to this, S3 objects had the following storage classes:

  1. Standard – For frequently accessed data.
  2. Standard-IA – For long-lived, infrequently accessed data.
  3. One Zone-IA – For long-lived, infrequently accessed, non-critical data.
  4. Glacier – For long-lived, infrequently accessed, archived critical data.

By default, the Standard storage class contains all S3 objects unless explicitly specified. Standard-Infrequent Access comes into picture when your object is infrequently accessed. Standard-IA has a lower storage cost associated with it. This may seem ideal initially, Standard offers a storage cost of $0.023 per GB for the first 50 TB data stored per month. In contrast, Standard-IA offers a meager charge of $0.0125 per GB for all the storage per month.

When juxtaposed, the choices seem fairly obvious, right?

Wrong.

The caveat with storing objects in the Standard-IA class is it’s considerably high data retrieval and request prices. Initially, you might store an object in Standard-IA under the assumption that you would barely access it, but as time progresses the need to use the object increases and along with it comes the high data request charges tied to the Standard-IA class. This would presumably lead to a spike in your bill. But, it is also not feasible to keep track of the usage patterns of objects and switch them between the storage classes frequently. Hence, using Standard-IA might end up being more than what you bargained for.

This is where the S3 Intelligent-Tiering storage class comes into play.

What is S3 Intelligent-Tiering Storage?

At re: Invent 2018 AWS introduced the Intelligent-Tiering Storage.

The S3 Intelligent-Tiering storage class is designed for customers who want to optimize their storage costs automatically when data access patterns change, without performance impact or operational overhead. The new storage class offers the same high durability, low latency, and high throughput of S3 Standard or Standard-IA and inherits all of the existing S3 features including security and access management, data life-cycle policies, cross-region replication, and event notifications.

How does it work?

Initially, the S3 objects are by default present in the Frequent Access tier for a period of 30 days. After monitoring data access patterns, objects that have not been accessed for a period of 30 days are moved to the Infrequent Access tier. Once accessed, they’re moved back to the Frequent Access tier.

How are Life-Cycle Policies affected by Intelligent-Tiering?

Source: https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-transition-general-considerations.html

  • The above waterfall model shows the possible storage class transitions supported by S3 life-cycle policies.
  • Transitions supported for Intelligent-Tiering are only to Glacier and OneZone_IA.
  • AWS does not transition objects that are smaller than 128kb to Intelligent-Tiering storage as it is not cost-effective.

Is this right for you?

  • If your data retrieval patterns are erratic and cannot be predetermined then the Intelligent Tiering storage class is for you, if not, the other four classes might be more suited.

What should you keep in mind before using Intelligent-Tiering?

  • This class should contain objects that are present for a minimum of 30 days. Hence, it is not suitable for short-lived objects. Rather, it is more suited for objects whose access patterns change over a longer period of time.
  • If the objects present are deleted, overwritten or transitioned to a different class before the minimum storage duration of 30 days there is a prorated charge per GB for the remaining days.
  • Objects smaller than 128kb could be stored in this storage class but it is not advisable to, because :
    1. Auto-Tiering is applicable only to objects of size greater than 128kb. Hence, the object will always be present in the Frequent Access tier irrespective of change in access patterns.
    2. The pricing for this storage will always be corresponding to that of the Frequent Access tier as it never moves to the Infrequent Access tier.

How to choose the Intelligent-Tiering storage class?

While uploading the object you can choose the Intelligent-Tiering storage class in the Set Properties step.

How is it priced?

(Pricing of US-East(N Virginia) considered in the above image. Pricing varies with region.)

From the above image, we can see that,

  • The billing for storage in the Frequent Access Tier is the same as S3 Standard.
  •  The billing for storage in the Infrequent Access is the same as S3 Standard-Infrequent Access.
  • There is a monthly fee of $0.0025 per 1,000 objects when using this storage class due to monitoring and automation.
  •  There is no cost associated while moving from Infrequent Tier to Frequent Tier and vice-versa.
  • The request pricing is as same as that of S3 Standard.

 

Deployment using AWS CodeDeploy and BitBucket Pipeline

Deploying your python code repo to an EC2 instance using AWS CodeDeploy and BitBucket Pipeline

 

Introduction:

Deploying any code repository manually to an EC2 machines or any dedicated on premises servers is a time consuming job. Maintainability and operating your application becomes difficult. To resolve this problem, there are different tools available in the market like Jenkins, AWS CodeDeploy,.. etc.

In this blog, we are going to deploy python code repository to AWS EC2 machines using AWS CodeDeploy and BitBucket pipeline services.

AWS CodeDeploy is a deployment service that automates application deployments to Amazon EC2 instances, on-premises instances, serverless Lambda functions, or Amazon ECS services. Bitbucket Pipelines is an integrated CI/CD service, built into Bitbucket. It allows you to automatically build, test and even deploy your code, based on a configuration file in your repository.

Prerequisites:

  • Valid AWS Account details. If you don’t have an AWS account then we will have one more blog for explaining
    • Signing Up in AWS
    • Launching an Auto Scaling Group
  • Valid Bitbucket account details with Pipelines access enable. If you don’t have it then upgrade your bitbucket plan.

Deployment Steps:

Configure Deployment settings in below service

  • Configuring AWS Services
  • Configuring Bitbucket pipeline
  • Integrate BitBucket and CodeDeploy
Configuring AWS Services:
Prerequisite:
  1. Login into your aws console (https://aws.amazon.com/console/).
  2. Need a S3 bucket to store the code repository. If you don’t have it then create a S3 bucket.
    • Click on https://s3.console.aws.amazon.com/s3/home?region=us-east-1 .
    • Select the S3 bucket and copy the name of the S3 bucket that you want to use for Code Deploy.
  3. Create a IAM Role for CodeDeploy Service.
    • Go to https://console.aws.amazon.com/iam/home?region=us-east-1#/roles and click on ‘Create Role’.
    • Select ‘AWS Service’ in Trusted Entities.
    • Select ‘CodeDeploy’ in ‘Choose the service that will use this role’.
    • Select ‘CodeDeploy’ in ‘Select your use case’ and click on ‘Next Permissions’.
    • Check whether ‘AWSCodeDeployRole’ is auto selected or not. If it is selected then click on ‘Create Tags’.
    • Enter Name and Value. This is an optional. If you want to skip then just click on ‘Review’.
    • Enter Name of the Role and click on ‘Create Role’.
  4. Create an EC2 Role to access the S3 Bucket.
    • Go to https://console.aws.amazon.com/iam/home?region=us-east-1#/roles and click on ‘Create Role’.
    • Select ‘AWS Service’ in Trusted Entities and select ‘EC2’ in ‘Choose the service that will use this role’ and click on ‘Next Permissions’.
    • Click on ‘Create Policy’ in attach permissions policy.
    • Click on ‘JSON’ and enter below json snippet.
    • Click on ‘Review Policy’ , enter the name of the Policy and click on ‘Create Policy’.
    • Click on ‘Refresh’ in the Create Role – Policy creation window and search for your policy name and click on ‘Next: Tags’.
    • Enter Name and Value. This is an optional. If you want to skip then just click on ‘Review’.
    • Enter Name of the Role and click on ‘Create Role’
  5. Launch an Auto Scaling Group using EC2 Role that was created in Step 4 and attach a valid Load Balancer to the Auto Scaling Group.  (Need steps to launch an Auto Scaling Group. It will be posted shortly)
Configuring AWS CodeDeploy Service:

Note: In this Blog, we are going to cover Blue – Green Deployment

  1. Go to CodeDeploy service. (https://console.aws.amazon.com/codesuite/codedeploy/applications?region=us-east-1).
  2. Click on ‘Create Application’
  3. Enter the name of the Application , Select the Compute Platform as ‘EC2/On-premises’ and click on ‘Create Application’.
  4. Once Application get created, click on ‘Create Deployment Group’.
  5. Enter the deployment name.
  6. Select the service role which we created in Configuring ‘AWS Services – Prerequisites – Step 3’ section.
  7. Select Blue-Green deployment.
  8. Select ‘Automatically copy Amazon EC2 Auto Scaling Group ‘ and select Auto Scaling Group in ‘Choose the Amazon EC2 Auto Scaling group where the current application revision is deployed’ which was configured in ‘AWS Services – Prerequisites – Step 5’
  9. Go to deployment setting and select ‘Reroute traffic immediately’ in Traffic rerouting.
  10. Select Timeline for terminating original instance after successful deployment.
  11. Select Deployment Configuration as ‘CodeDeployDefault.OneAtATime’.
  12. Select the Load Balancer (This is a mandatory field for Blue Green Deployment).
  13. If you need to RollBack to the Original State in case of deployment failure then go to ‘Advanced – Optional’ and select ‘Rollbacks’.
    • Uncheck the ‘Disable Rollbacks’.
    • Select Rollback when deployment fails.
  14. Click on ‘Create Deployment Group’.
Configuring Bitbucket Services:
  1. Create codedeploy_deploy.py in code repo. Find a sample file in (https://bitbucket.org/awslabs/aws-codedeploy-bitbucket-pipelines-python/src/master/).
  2. Create ‘bitbucket-pipelines.yml’ in the same location where codedeploy_deploy.py available.
    • If necessary, update the python version.
    • Each step will be launched in a docker. Mention commands that are to be executed as part of the deployment.
    • Keyword deployment stands to pick proper environment from bitbucket pipeline environment settings.
    • It will zip entire code repo.
  3. Create ‘appspec.yml’ file in the same location. This file will helpful to drive AWS Codedeploy deployment process. 
    • version – Version number of appspec.yml.
    • os        –  Value should be either ‘linux’ nor ‘windows’.
    • files      – Files will be downloaded from S3 bucket and move to the destination folder during ‘Install’ phase in code deployment process.
    • Hooks  – Different events that can be handled via hooks. For example, ApplicationStop, BeforeInstall, AfterInstall, ..etc.
Integrating CodeDeploy and Bitbucket Services:
  1. Login to Bitbucket server.(https://id.atlassian.com/login)
  2. Go to ‘Settings’ of your repository.
  3. Go to ‘Pipelines’ section.
  4. Click on ‘Settings’ and enable the ‘enable pipelines’.
  5. Go to Deployment section and add any variables that require at environment level like development, staging and production. You can specify same in ‘bitbucket-pipelines.yml’
  6. Go to Repository Variables and enter system level variables. Please add all below variable.

Now, It’s time to trigger your deployment process. Once you committed in the code repository, check whether Pipeline get triggered or not.

  • Go to ‘Pipelines’ section in your bitbucket repository.
  • Once ‘codedeploy_deploy.py’ step is in progress, go to AWS CodeDeploy console.
  • Check whether any deployments got triggered or not. Once deployment status changes to ‘success’ then deployment completed successfully.

Open Issues :

  1. AWS CodeDeploy: As part of Blue Green Deployment, codedeploy creates replica of current Auto Scaling Group. Target Groups/Load Balancer are not getting attach to the replace Auto Scaling Group. Need to attach it manually. As of 19-Jan-2019, AWS Team is working on it.
  2. You will receive ‘Rate Exceeded’ error in pipeline with Blue-Green Deployment mode. Please add retry mechanism in ‘codedeploy_deploy.py’ in GetDeployment method. Please find reference below https://docs.aws.amazon.com/general/latest/gr/api-retries.html .

Coming Soon :

  1. How to Launch an Auto Scaling Group.
  2. Configuring AWS CodeDeploy with ‘In-Place’ deployment mode instead of Blue-Green Deployment.
  3. Video that will help for deploying your code repository using AWS CodeDeploy and Bitbucket pipeline.

References:

https://docs.aws.amazon.com/codedeploy/latest/userguide/welcome.html

https://confluence.atlassian.com/bitbucket/get-started-with-bitbucket-pipelines-792298921.html

https://bitbucket.org/awslabs/aws-codedeploy-bitbucket-pipelines-python/src/master/

 

Doing More with AWS EC2 Spot Instances

Given that EC2 accounts for almost 60-70% of AWS spend and spot instances offer discounts of upto 80% , using EC2 Spot Instances is a key pillar in the cost optimization of AWS. However, using spot instances correctly can be challenging at times. We will explore the best practices on using Spot Instances in this introductory post. In the future posts, we will examine why the best practices are important and how best to use Spot for each of the AWS Services.

Here is how EC2 Virtualization model influences Performance

Choosing the right EC2 Virtualization model is becoming increasingly important as part of selecting the right instance type for your workloads.  As a quick recap, an EC2 instance is a virtual machine on top of bare metal with a hypervisor in between.

This can be depicted as below.

So which EC2 Virtualization model should you use? And how does it affect your instance type selection?

Para Virtualization (PV)

Circa 2014, Software-defined virtualization or Para Virtualization was all the rage. It was lightweight, offered benefits over other virtualization models ( like HVM).  However, given that there is a software layer over bare metal, some compute capacity was spent for managing the virtualization layer as well. Also, the guest OS needs some modification before it can be Para Virtualized.  The modifications are essentially replacing instructions that can’t be run natively with a HYPERCALL. ( Read more about Xen and PV  at the https://www.XenProject.org and virtualization spectrum in particular (https://wiki.xenproject.org/wiki/Understanding_the_Virtualization_Spectrum)

From an EC2 perspective, PV has a few issues to note

  1. PV cannot expose the hardware acceleration that the specialized hardware can allow ( e.g. EBS Optimized, Enhanced Networking etc).
  2. Kernel objects are region specific.  This would limit the type of EC2 instance types that can be used for PV images ( especially now in 2018 ).

Hardware Virtual Machine (HVM):

Hardware virtual Machine:  HMV allows the guest OS to be fully virtualized. The guest OS doesn’t know it is sharing Host resources and runs as if all the resources are available natively to it. 

This means specialized hardware on the host can run much faster in HVM mode as if they are running directly on host systems. The underlying host should have the capacity to emulate the hardware for the VMs running on top.  The guest OS also doesn’t need any modifications.

HVM vs PV

In addition some processor features are available on HMV only.

e.g. Intel Advanced Vector Extensions (Intel AVX, Intel AVX2 and Intel AVX-512) – These can be useful for specific applications. e.g  3D-modelling, image processing etc.  More details here 

Nitro

Simply put, it is better bang for your buck. Nitro helps deliver more compute per hypervisor as many of the earlier functions of HVM are offloaded to custom hardware.   Nitro takes many of the functions of the software virtualization to custom ASIC hardware and a bare minimum KVM ( Kernel Virtual Machine). This will keep more compute resources for the guest VMs and also reduce the jitter or variability of performance.  The below figure indicates that all of the hardware is available for the guest OS and the hardware extensions of EBS optimized and Enhanced Networking are handled by custom hardware.  

Another advantage is that it is applicable for Bare metal instances as well as this is at the hardware level.

Nitro Hypervisor for C5 Instances

The Nitro System Consists of

Local NVMe Storage – provides high-speed access to local storage and encryption using dedicated hardware. The new C5d, M5d introduced recently and bare metal EC2 instances feature Nitro local NVMe storage building block,  It also provides hardware-level isolation between storage devices and EC2 instances so that bare metal instances can benefit from local NVMe storage.

Nitro Security Chip – It independently verifies firmware each time a system boots.

Nitro Hypervisor – Hypervisor that manages memory and CPU allocation, and delivers near identical performance to bare metal (Brendan Gregg of Netflix benchmarked it at less than 1%).

Networking – Hardware support for the software-defined network inside of each Virtual Private Cloud (VPC), Enhanced Networking, and Elastic Network Adapter.

As Amazon moves to build purpose-built data centers, hardware, and software for EC2, it is more likely that Nitro will be the future.  C5 instances reportedly give 20% or more performance over C4 instances for similar workloads with less variability for similar price and up to 25% memory efficiency over C4. For more information check out these blogs. A great introduction to it from AWS re:Also, Invent 2017.

A note of caution:

Some of the early adopters were seeing some growing pains especially with EBS volumes and ENI. So, perhaps until the issues stabilize, it is best to test your workloads with C5/M5 instances and then only move to production.

Conclusion:

While EC2 virtualization model isn’t something that we pay a lot of attention to, choosing the wrong model can have implications in terms of availability of instance types for a particular AMI and the net performance you can get out of a particular instance type. As of 2018, HVM is the broadly available virtualization for second generation and above instance types and should be the main stay. PV is fairly limited to the first generation of instances and few second generation of instances. The new kid on the block – Nitro is available for newest C5/C5d/M5d instance types is promised as the next in the evolution of virtualization for EC2 with better performance and performance for unit price.


Configure AWS Lamba programatically with Boto3

Serverless computing has generated a lot of interest in the development community. AWS Lambda is the AWS version of Serverless. In this post, we will look at configuring AWS Lambda programmatically with boto3.  We will configure a lambda function that connects to a Postgres DB on an EC2 instance in a private VPC using sqlalchemy. This would need packaging of the dependencies of sqlalchemy along with the lambda function.

By default, the instance where AWS lambda is running has Boto installed (AWS Python SDK). So, importing a boto3 package in python code will work without any other packaging. But importing a package like “import sqlalchemy” will lead to python error “Unable to import module sqlalchemy”. This article shows how to achieve the same.

Step 1: Write the lambda function

First we need to write the python code which will run as a lambda. This below code creates a python file and zips it. Note down, the python code that we wanted to execute should contain a method called lambda_handler which serves as an entry point for the lambda to start the execution.

One quick aside: SqlAlchemy complains about not being able to find the right provider when connecting to postgres from lambda. So might have to add postgressql + pscycopg2 as part of connection string. Don't forget to open the port in the security group and enable incoming connections from all in the host file of the Postgres server. This can be a cause of a lot of heartaches.
Also, remember if you the system you package the code is windows, then you are in some trouble as the pip will pick windows related drivers. So you need to run this on lambda compatible OS (read as Amazon Linux or compatible OS). Else, you will end up with a lot of wierd run time errors.

Step 2:  Create the lambda function

Now use boto3 client API create_function to map it to the above code. If you take a close look, Handler=‘lambda_function.lambda_handler’ means that it will look for the python file lambda_function.py and inside it it will look for lambda_handler function. Also, you need to pass the right arn for this API to work.

Step 3: Set the frequency of lambda 

Set the rule and the frequency at which you want the lambda to run, it can be a simple expression like rate(1 minute) or rate(5 minutes) or a cron expression too. Also, set the rule target and grant permission for the lambda to execute. In this case, we are configuring the rule to run every minute.

Step 4: Additional Package Deployment

Do this step if we need other packages for our python code to run. For creating deployment folders, you can use this python script create_deployment.py and requirements.txt. Basically, when the script file runs, it picks the packages that need to be installed from requirements.txt and does a pip -install -t and which creates a folder, pull all the packages and zips them. If you need to have a particular version you can say sseclient==0.0.11 or you can omit the version to pull the latest.

Final Step: Putting it together

Now putting everything together yields the following output in the AWS CloudWatch logs, no import module error this time. This can be modified to use whichever package you need for your development purposes.

Hope this saves a few hours trying to jump a few hoops which weren’t really apparent at the onset!

References:

AWS Lambda Functions Made Easy

AWS Lambda: Programmatically scheduling a CloudWatchEvent

 

vCPUs and cost implications in right-sizing of for EC2 instances

vCPUs are the base unit of compute capacity for EC2 instances. They represent one of the main criteria for selecting an EC2 instance type.

What does vCPU really represent on a physical host? and more importantly – are all vCPUs the same?  Let us explore this in a bit of detail. The cost implications will also become clear once we understand it.

Each vCPU is a hyper-thread of an Intel Xeon core (except for T2). More details here

T2 instances have a slightly winding explanation and feature ‘burstable’ or occasional spikes that the originally available compute capacity.

Note that the definition says hyper-threaded instead of virtual core..  Or about 2 hyper-threads per core. Als,o note that doesn’t mean the hyper-threads share 50% of each core.

If you want to get a  good understanding of hyper-threading, take some time to read this excellent article from ArsTechnica about HT or SMT.

The interesting aspect to note ( and something that is often lost to those new to virtualization) is that vCPUs are not a standard measure in a way a ‘meter’ or ‘ mile’ is defined.  This merely indicates that from the underlying host,  ‘n’ vCPUs are virtualized.

Compute capacity also depends upon other host characteristics such as cache, bus speed and QPI of the host itself would change.

E.g. vCPUs based on Intel Xeon 2676 ( up to m4.8x large) and vCPUs based on Intel Xeon 2686 ( m4.10xlarge and greater), m4.10xlarge or above would give better performance per vCPU on account of the difference in the host ( benchmark comparison for both Xeon processors )

The amount of performance you can get out of an instance is also dependent on how much processors state can be controlled via Linux ( e.g. C-State and P-State optimizations )

And this great video:

AWS re:Invent 2017: Deep Dive on Amazon EC2 Instances, Featuring Performance Optimize (CMP301)

 

Let’s consider an example.

Consider C5.2xLarge and C4.2xLarge instance types.

Name

API Name

Memory

vCPUs

Instance Storage

Network Performance

Linux On Demand cost

Linux Reserved cost

Windows On Demand cost

Windows Reserved cost

C5 High-CPU Double Extra Large

c5.2xlarge

16.0 GiB

8 vCPUs

EBS only

Up to 10 Gbps

$0.340000 hourly

$0.214000 hourly

$0.708000 hourly

$0.582000 hourly

C4 High-CPU Double Extra Large

c4.2xlarge

15.0 GiB

8 vCPUs

EBS only

High

$0.398000 hourly

$0.252000 hourly

$0.766000 hourly

$0.620000 hourly

From ec2instances.info

While both have the same 8 vCPUs and almost similar memory.

The C5 family uses 3.0 GHz Intel Xeon Platinum processors with new Intel Advanced Vector Extension 512 (AVX-512) instruction set and C4 family uses Intel Xeon E5-2666 v3. C5 2x large has a better spec, marginally lower cost per hour, better virtualisation model (Nitro) with less variability and delivers up to 25% more performance compared to C4 instance types. However, C5 does need AMIs that include drivers for ENA and NVMe.

On a general note and at risk of broad oversimplification, vCPUs of current generation instances give better compute performance compared to previous generation( C5 better than C4 ) and instances of higher instance types give better performance per unit cost compared to their lower siblings ( e.g. m4.10xlarge better than m4.8xlarge).   A good practical method suggested  by AWS is here

Conclusion

While the vCores give valuable information about the compute capacity, taking the number of cores alone into consideration during selection of instances can trip your application up in unexpected ways. Any choice of instance type should take into all the other options available and benchmark them before finalizing on the instance type.

A service like insisive cloud can help analyze your AWS infrastructure using rule-based policies for cost optimization and help with rapid prototyping at costs up to 60% less.

Sign up for a 14-day no-obligation trial to see if it is suitable for your application needs.