The new A1 EC2 Instance

At re: Invent 2018 AWS  introduced A1 instances. Compared to the other instance types they cost 40% lesser ‘per core’. Given that they use the Nitro Hypervisor, they give a better performance as well compared to traditional Xen Hypervisor based instances.

However, before you go and move all your instance types, it is good to know when you can use these instance types and the effort required in moving your workload to use these instances.

ms-clipboard-file://C:/Users/Geetha/AppData/Local/Packages/Microsoft.Office.OneNote_8wekyb3d8bbwe/TempState/msohtmlclip/clip_image001.png

A1 instances have ARM-based processors. If your workloads compile to native code using x86 architecture, then you would need to recompile for ARM platform before you can run on them.

For scripting based languages, it could be negligible ( as long as they do not use some module that is native in their dependency chain).

If you use Docker containers, It is relatively quick as mentioned in https://blog.boltops.com/2018/12/16/ec2-a1-instance-with-aws-homegrown-arm-processor-easy-way-to-save-40.

Amazon Linux, Ubuntu Linux, and Red Hat Enterprise Linux are the initial operating systems with ARMv8 support on EC2.

A1 instances have slightly dated ARM A72 processors (released in 2015. The current generation is A76) that are aimed at high-end smartphones and tablets. So they aren’t meant for same workloads such as Xeon E5 series powering the Cx series instance types for servers. In terms of benchmarks by Phoronix https://www.phoronix.com/scan.php?page=article&item=ec2-graviton-performance&num=1, both Intel and AMD based instances far outperform the current gen A1 instances.

Interestingly enough is the price/performance per dollar

ms-clipboard-file://C:/Users/Geetha/AppData/Local/Packages/Microsoft.Office.OneNote_8wekyb3d8bbwe/TempState/msohtmlclip/clip_image002.png

Courtesy: Phoronix benchmark

In terms of ‘real world’ test of hosting website, they still underperform by about 3.5x ( albeit at a lower cost). (https://www.theregister.co.uk/2018/11/27/amazon_aws_graviton_specs/)

Currently, A1 instances are not meant for general purpose workloads. However, owing to the Nitro system based hypervisor, they will be very useful as part of scale-out workloads, lightweight web servers,

containerized micro-services, caching fleets and such.

However, there is a larger trend at play which will be beneficial to customers in the long run. Amazon bringing their own processor into the mix with Intel and AMD will improve choices and hopefully reduce costs in the long run.

Amazon’s new and Intelligent S3 storage class:

What is S3?

Amazon Simple Storage Service (Amazon S3) is a global Infrastructure as a Service (IaaS) solution. Amazon S3 facilitates highly scalable, secured and low-latency data storage from the cloud.

Yet another storage class?

Prior to this, S3 objects had the following storage classes:

  1. Standard – For frequently accessed data.
  2. Standard-IA – For long-lived, infrequently accessed data.
  3. One Zone-IA – For long-lived, infrequently accessed, non-critical data.
  4. Glacier – For long-lived, infrequently accessed, archived critical data.

By default, the Standard storage class contains all S3 objects unless explicitly specified. Standard-Infrequent Access comes into picture when your object is infrequently accessed. Standard-IA has a lower storage cost associated with it. This may seem ideal initially, Standard offers a storage cost of $0.023 per GB for the first 50 TB data stored per month. In contrast, Standard-IA offers a meager charge of $0.0125 per GB for all the storage per month.

When juxtaposed, the choices seem fairly obvious, right?

Wrong.

The caveat with storing objects in the Standard-IA class is it’s considerably high data retrieval and request prices. Initially, you might store an object in Standard-IA under the assumption that you would barely access it, but as time progresses the need to use the object increases and along with it comes the high data request charges tied to the Standard-IA class. This would presumably lead to a spike in your bill. But, it is also not feasible to keep track of the usage patterns of objects and switch them between the storage classes frequently. Hence, using Standard-IA might end up being more than what you bargained for.

This is where the S3 Intelligent-Tiering storage class comes into play.

What is S3 Intelligent-Tiering Storage?

At re: Invent 2018 AWS introduced the Intelligent-Tiering Storage.

The S3 Intelligent-Tiering storage class is designed for customers who want to optimize their storage costs automatically when data access patterns change, without performance impact or operational overhead. The new storage class offers the same high durability, low latency, and high throughput of S3 Standard or Standard-IA and inherits all of the existing S3 features including security and access management, data life-cycle policies, cross-region replication, and event notifications.

How does it work?

Initially, the S3 objects are by default present in the Frequent Access tier for a period of 30 days. After monitoring data access patterns, objects that have not been accessed for a period of 30 days are moved to the Infrequent Access tier. Once accessed, they’re moved back to the Frequent Access tier.

How are Life-Cycle Policies affected by Intelligent-Tiering?

Source: https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-transition-general-considerations.html

  • The above waterfall model shows the possible storage class transitions supported by S3 life-cycle policies.
  • Transitions supported for Intelligent-Tiering are only to Glacier and OneZone_IA.
  • AWS does not transition objects that are smaller than 128kb to Intelligent-Tiering storage as it is not cost-effective.

Is this right for you?

  • If your data retrieval patterns are erratic and cannot be predetermined then the Intelligent Tiering storage class is for you, if not, the other four classes might be more suited.

What should you keep in mind before using Intelligent-Tiering?

  • This class should contain objects that are present for a minimum of 30 days. Hence, it is not suitable for short-lived objects. Rather, it is more suited for objects whose access patterns change over a longer period of time.
  • If the objects present are deleted, overwritten or transitioned to a different class before the minimum storage duration of 30 days there is a prorated charge per GB for the remaining days.
  • Objects smaller than 128kb could be stored in this storage class but it is not advisable to, because :
    1. Auto-Tiering is applicable only to objects of size greater than 128kb. Hence, the object will always be present in the Frequent Access tier irrespective of change in access patterns.
    2. The pricing for this storage will always be corresponding to that of the Frequent Access tier as it never moves to the Infrequent Access tier.

How to choose the Intelligent-Tiering storage class?

While uploading the object you can choose the Intelligent-Tiering storage class in the Set Properties step.

How is it priced?

(Pricing of US-East(N Virginia) considered in the above image. Pricing varies with region.)

From the above image, we can see that,

  • The billing for storage in the Frequent Access Tier is the same as S3 Standard.
  •  The billing for storage in the Infrequent Access is the same as S3 Standard-Infrequent Access.
  • There is a monthly fee of $0.0025 per 1,000 objects when using this storage class due to monitoring and automation.
  •  There is no cost associated while moving from Infrequent Tier to Frequent Tier and vice-versa.
  • The request pricing is as same as that of S3 Standard.

 

Deployment using AWS CodeDeploy and BitBucket Pipeline

Deploying your python code repo to an EC2 instance using AWS CodeDeploy and BitBucket Pipeline

 

Introduction:

Deploying any code repository manually to an EC2 machines or any dedicated on premises servers is a time consuming job. Maintainability and operating your application becomes difficult. To resolve this problem, there are different tools available in the market like Jenkins, AWS CodeDeploy,.. etc.

In this blog, we are going to deploy python code repository to AWS EC2 machines using AWS CodeDeploy and BitBucket pipeline services.

AWS CodeDeploy is a deployment service that automates application deployments to Amazon EC2 instances, on-premises instances, serverless Lambda functions, or Amazon ECS services. Bitbucket Pipelines is an integrated CI/CD service, built into Bitbucket. It allows you to automatically build, test and even deploy your code, based on a configuration file in your repository.

Prerequisites:

  • Valid AWS Account details. If you don’t have an AWS account then we will have one more blog for explaining
    • Signing Up in AWS
    • Launching an Auto Scaling Group
  • Valid Bitbucket account details with Pipelines access enable. If you don’t have it then upgrade your bitbucket plan.

Deployment Steps:

Configure Deployment settings in below service

  • Configuring AWS Services
  • Configuring Bitbucket pipeline
  • Integrate BitBucket and CodeDeploy
Configuring AWS Services:
Prerequisite:
  1. Login into your aws console (https://aws.amazon.com/console/).
  2. Need a S3 bucket to store the code repository. If you don’t have it then create a S3 bucket.
    • Click on https://s3.console.aws.amazon.com/s3/home?region=us-east-1 .
    • Select the S3 bucket and copy the name of the S3 bucket that you want to use for Code Deploy.
  3. Create a IAM Role for CodeDeploy Service.
    • Go to https://console.aws.amazon.com/iam/home?region=us-east-1#/roles and click on ‘Create Role’.
    • Select ‘AWS Service’ in Trusted Entities.
    • Select ‘CodeDeploy’ in ‘Choose the service that will use this role’.
    • Select ‘CodeDeploy’ in ‘Select your use case’ and click on ‘Next Permissions’.
    • Check whether ‘AWSCodeDeployRole’ is auto selected or not. If it is selected then click on ‘Create Tags’.
    • Enter Name and Value. This is an optional. If you want to skip then just click on ‘Review’.
    • Enter Name of the Role and click on ‘Create Role’.
  4. Create an EC2 Role to access the S3 Bucket.
    • Go to https://console.aws.amazon.com/iam/home?region=us-east-1#/roles and click on ‘Create Role’.
    • Select ‘AWS Service’ in Trusted Entities and select ‘EC2’ in ‘Choose the service that will use this role’ and click on ‘Next Permissions’.
    • Click on ‘Create Policy’ in attach permissions policy.
    • Click on ‘JSON’ and enter below json snippet.
    • Click on ‘Review Policy’ , enter the name of the Policy and click on ‘Create Policy’.
    • Click on ‘Refresh’ in the Create Role – Policy creation window and search for your policy name and click on ‘Next: Tags’.
    • Enter Name and Value. This is an optional. If you want to skip then just click on ‘Review’.
    • Enter Name of the Role and click on ‘Create Role’
  5. Launch an Auto Scaling Group using EC2 Role that was created in Step 4 and attach a valid Load Balancer to the Auto Scaling Group.  (Need steps to launch an Auto Scaling Group. It will be posted shortly)
Configuring AWS CodeDeploy Service:

Note: In this Blog, we are going to cover Blue – Green Deployment

  1. Go to CodeDeploy service. (https://console.aws.amazon.com/codesuite/codedeploy/applications?region=us-east-1).
  2. Click on ‘Create Application’
  3. Enter the name of the Application , Select the Compute Platform as ‘EC2/On-premises’ and click on ‘Create Application’.
  4. Once Application get created, click on ‘Create Deployment Group’.
  5. Enter the deployment name.
  6. Select the service role which we created in Configuring ‘AWS Services – Prerequisites – Step 3’ section.
  7. Select Blue-Green deployment.
  8. Select ‘Automatically copy Amazon EC2 Auto Scaling Group ‘ and select Auto Scaling Group in ‘Choose the Amazon EC2 Auto Scaling group where the current application revision is deployed’ which was configured in ‘AWS Services – Prerequisites – Step 5’
  9. Go to deployment setting and select ‘Reroute traffic immediately’ in Traffic rerouting.
  10. Select Timeline for terminating original instance after successful deployment.
  11. Select Deployment Configuration as ‘CodeDeployDefault.OneAtATime’.
  12. Select the Load Balancer (This is a mandatory field for Blue Green Deployment).
  13. If you need to RollBack to the Original State in case of deployment failure then go to ‘Advanced – Optional’ and select ‘Rollbacks’.
    • Uncheck the ‘Disable Rollbacks’.
    • Select Rollback when deployment fails.
  14. Click on ‘Create Deployment Group’.
Configuring Bitbucket Services:
  1. Create codedeploy_deploy.py in code repo. Find a sample file in (https://bitbucket.org/awslabs/aws-codedeploy-bitbucket-pipelines-python/src/master/).
  2. Create ‘bitbucket-pipelines.yml’ in the same location where codedeploy_deploy.py available.
    • If necessary, update the python version.
    • Each step will be launched in a docker. Mention commands that are to be executed as part of the deployment.
    • Keyword deployment stands to pick proper environment from bitbucket pipeline environment settings.
    • It will zip entire code repo.
  3. Create ‘appspec.yml’ file in the same location. This file will helpful to drive AWS Codedeploy deployment process. 
    • version – Version number of appspec.yml.
    • os        –  Value should be either ‘linux’ nor ‘windows’.
    • files      – Files will be downloaded from S3 bucket and move to the destination folder during ‘Install’ phase in code deployment process.
    • Hooks  – Different events that can be handled via hooks. For example, ApplicationStop, BeforeInstall, AfterInstall, ..etc.
Integrating CodeDeploy and Bitbucket Services:
  1. Login to Bitbucket server.(https://id.atlassian.com/login)
  2. Go to ‘Settings’ of your repository.
  3. Go to ‘Pipelines’ section.
  4. Click on ‘Settings’ and enable the ‘enable pipelines’.
  5. Go to Deployment section and add any variables that require at environment level like development, staging and production. You can specify same in ‘bitbucket-pipelines.yml’
  6. Go to Repository Variables and enter system level variables. Please add all below variable.

Now, It’s time to trigger your deployment process. Once you committed in the code repository, check whether Pipeline get triggered or not.

  • Go to ‘Pipelines’ section in your bitbucket repository.
  • Once ‘codedeploy_deploy.py’ step is in progress, go to AWS CodeDeploy console.
  • Check whether any deployments got triggered or not. Once deployment status changes to ‘success’ then deployment completed successfully.

Open Issues :

  1. AWS CodeDeploy: As part of Blue Green Deployment, codedeploy creates replica of current Auto Scaling Group. Target Groups/Load Balancer are not getting attach to the replace Auto Scaling Group. Need to attach it manually. As of 19-Jan-2019, AWS Team is working on it.
  2. You will receive ‘Rate Exceeded’ error in pipeline with Blue-Green Deployment mode. Please add retry mechanism in ‘codedeploy_deploy.py’ in GetDeployment method. Please find reference below https://docs.aws.amazon.com/general/latest/gr/api-retries.html .

Coming Soon :

  1. How to Launch an Auto Scaling Group.
  2. Configuring AWS CodeDeploy with ‘In-Place’ deployment mode instead of Blue-Green Deployment.
  3. Video that will help for deploying your code repository using AWS CodeDeploy and Bitbucket pipeline.

References:

https://docs.aws.amazon.com/codedeploy/latest/userguide/welcome.html

https://confluence.atlassian.com/bitbucket/get-started-with-bitbucket-pipelines-792298921.html

https://bitbucket.org/awslabs/aws-codedeploy-bitbucket-pipelines-python/src/master/