Elastic Load Balancer (ELB)

Elastic Load Balancing is load balancer as a service provided by AWS.

    1. Automatically distributes incoming application traffic across multiple targets, such as Amazon EC2 instances, containers, IP addresses, Lambda functions, and virtual appliances.
    2. It can handle the varying load of your application traffic in a single Availability Zone or across multiple Availability Zones.
    3. Elastic Load Balancing offers four types of load balancers that all feature the high availability, automatic scaling, and robust security necessary to make your applications fault tolerant.

Four types

    1. Classic LB (for classic EC2) uses Transport layer (L4) or application layer (L7) of TCP/IP. Classic Load Balancer provides basic load balancing across multiple Amazon EC2 instances and operates at both the request level and the connection level. Classic Load Balancer is intended for applications that were built within the EC2-Classic network.
    2. Application LB uses application layer of TCP/IP. Application Load Balancer is best suited for load balancing of HTTP and HTTPS traffic and provides advanced request routing targeted at the delivery of modern application architectures, including microservices and containers. Application Load Balancer routes traffic to targets within Amazon VPC based on the content of the request.
    3. Network Load Balancer is best suited for load balancing of Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Transport Layer Security (TLS) traffic where extreme performance is required. Network Load Balancer routes traffic to targets within Amazon VPC and is capable of handling millions of requests per second while maintaining ultra-low latencies.
    4. Gateway Load Balancer makes it easy to deploy, scale, and run third-party virtual networking appliances. Providing load balancing and auto scaling for fleets of third-party appliances, Gateway Load Balancer is transparent to the source and destination of traffic. This capability makes it well suited for working with third-party appliances for security, network analytics, and other use cases.

Benefits of ELB

Highly availability and elasticity:

  • Elastic Load Balancing is part of the AWS network, with native awareness of failure boundaries like AZs to keep your applications available across a region, without requiring Global Server Load Balancing (GSLB).
  • ELB is also a fully managed service, meaning you can focus on delivering applications and not installing fleets of load balancers.
  • Capacity is automatically added and removed based on the utilization of the underlying application servers.

Security:

  • Elastic Load Balancing works with Amazon Virtual Private Cloud (VPC) to provide robust security features, including integrated certificate management, user-authentication, and SSL/TLS decryption.
  • Together, they give you the flexibility to centrally manage TLS settings and offload CPU intensive workloads from your applications.
  • ALB also supports integration with AWS WAF, adding a level of protection before bad actors reach the application.
  • Further, S2N and HTTP Guardian have been developed as Open Source solutions to reduce the potential for HTTP-based attacks.

Feature breadth

  • Elastic Load Balancing offers the breadth of features needed by businesses of all sizes, while delivering them in an AWS-native experience.
  • Elastic Load Balancing includes support for features needed in container-based workloads, including HTTP/2, gRPC, TLS offload, advanced rule-based routing, and integration with container services as an ingress controller.
  • ALB provides customers with a native HTTP endpoint for calling Lambda functions, removing the dependency on other solutions.
  • Further, Gateway Load Balancer creates one gateway for routing traffic through fleets of third-party appliances.

Robust monitoring & visibility

  • Elastic Load Balancing allows you to monitor the health of your applications and their performance in real time with Amazon CloudWatch metrics, logging, and request tracing.
  • This improves visibility into the behavior of your applications, uncovering issues and identifying performance bottlenecks in your application stack.
  • ELB helps ensure compliance with application Service Level Agreements (SLAs).

Integration and global reach

  • As a native AWS service, ELB is tightly integrated with other AWS services like EC2, ECS/EKS, Global Accelerator and operational tools such as AWS CloudFormation and AWS Billing.
  • Across the Amazon Global Infrastructure and customer data centers with AWS Outposts and on-premises target support, ELB is available everywhere you run your AWS workloads.

How to Create ELB

    1. Create Load Balancer
      • Choose type of ELB, Name, internal or internet facing, VPC
      • ELB Protocol, ELB Port
        • For Classic choose HTTP/HTTPS/TCP/SSL
        • For application ELB choose HTTP/HTTPS
        • Tor Network ELB choose TCP/Port
      • Instance protocol, Instance port and assign Security Group
      • aws elbv2 create-load-balancer --name my-load-balancer --subnets subnet-0e3f5cac72EXAMPLE subnet-081ec835f3EXAMPLE --security-groups sg-07e8ffd50fEXAMPLE
        
        output: arn:aws:elasticloadbalancing:us-east-2:123456789012:loadbalancer/app/my-load-balancer/1234567890123456
    2. Create Target group
      • For Network ELB and Application ELB, create a new Target Group or use existing target group (for classic ELB, add EC2’s directly to the ELB).
        • TG needs Name, Protocol, Port, Target Type (Instance id or ip)
      • aws elbv2 create-target-group --name my-targets --protocol HTTP --port 80 --vpc-id vpc-0598c7d356EXAMPLE
        
        output: arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/1234567890123456
      • modify-target-group
        --target-group-arn <value>
        [--health-check-protocol <value>]
        [--health-check-port <value>]
        [--health-check-path <value>]
        [--health-check-enabled | --no-health-check-enabled]
        [--health-check-interval-seconds <value>]
        [--health-check-timeout-seconds <value>]
        [--healthy-threshold-count <value>]
        [--unhealthy-threshold-count <value>]
        [--matcher <value>]
        [--cli-input-json <value>]
        [--generate-cli-skeleton <value>]
      • Configure health check:
        • Ping protocol/port/path (example: /test.html)
        • Response timeout (amount of time before response starts)
        • Interval (amount of time between pings)
        • Unhealthy threshold (number of consecutive failures before declaring instance unhealthy)
        • Healthy threshold (number of consecutive pass before declaring instance healthy)
    3. Register Targets to the Target Group: Add EC2 instances (instance id) or private ip/port to the TG
      • aws elbv2 register-targets --target-group-arn targetgroup-arn --targets Id=i-0abcdef1234567890 Id=i-1234567890abcdef0
      • Skip this step if you choose to use this TG as a proxy to an ASG (see next point #8)
    4. Creating ELB listeners (for elbv2)
      • Using CLI, or console you need to create a listener for each target group. The listener can be assigned multiple TGs. Using below CLI command you can create a Listener with a default rule that forwards requests to your target group
      • aws elbv2 create-listener --load-balancer-arn loadbalancer-arn --protocol HTTPS --port 443 --certificates CertificateArn=certificate-arn --default-actions Type=forward,TargetGroupArn=targetgroup-arn
        
        output: listner arn
    5. You can create additional rules in the listener. A rule attaches a listener to a TG. Example: Create a rule to send path-pattern based requests to a special target group:
      • aws elbv2 create-rule --listener-arn listener-arn --priority 10 --conditions Field=path-pattern,Values='/img/*' --actions Type=forward,TargetGroupArn=targetgroup-arn
    6. Final steps
      1. After creating ELB note the DNS name, which looks similar to: elbname.regionname.elb.amazonaws.com. This can be updated in your DNS
      2. You can delete ELB either from console or CLI
      3. aws elbv2 delete-load-balancer --load-balancer-arn loadbalancer-arn

Session management (AWS documentation on ELB session management)

    1. By default, a Classic Load Balancer routes each request independently to the registered instance with the smallest load. However, you can use the sticky session feature (also known as session affinity), which enables the load balancer to bind a user’s session to a specific instance. This ensures that all requests from the user during the session are sent to the same instance.
    2. Stickiness can be setup thru console or CLI
    3. Elastic Load Balancing creates a cookie, named AWSELB, that is used to map the session to the instance.
    4. Duration based session stickiness:
      1. On the Edit stickiness page, select Enable load balancer generated cookie stickiness.
      2. (Optional) For Expiration Period type the cookie expiration period, in seconds. If you do not specify an expiration period, the sticky session lasts for the duration of the browser session.
      3. If there is no cookie (first request), the load balancer chooses an instance based on the existing load balancing algorithm (least loaded EC2). A cookie is inserted into the response for binding subsequent requests from the same user to that instance.  After a cookie expires (after duration from above step), the session is no longer sticky.
    5. Application-Controlled session stickiness:  
      1. The load balancer uses a special cookie to associate the session with the instance that handled the initial request, but follows the lifetime of the application cookie specified in the policy configuration.
      2. The load balancer only inserts a new stickiness cookie if the application response includes a new application cookie.
      3. The load balancer stickiness cookie does not update with each request.
      4. If the application cookie is explicitly removed or expires, the session stops being sticky until a new application cookie is issued.
      5. If an instance fails or becomes unhealthy, the load balancer stops routing requests to that instance, and chooses a new healthy instance based on the existing load balancing algorithm.
      6. The load balancer treats the session as now “stuck” to the new healthy instance, and continues routing requests to that instance even if the failed instance comes back.
      7. On the Edit stickiness page, select Enable application generated cookie stickiness.
      8. For Cookie Name, type the name of your application cookie.
    6. ELB can distribute load across EC2 instances running across AZs within a region only.

Robust monitoring & visibility

You can use request tracing to track HTTP requests from clients to targets or other services. When the load balancer receives a request from a client, it adds or updates the X-Amzn-Trace-Id header before sending the request to the target. Any services or applications between the load balancer and the target can also add or update this header. If you enable access logs, the contents of the X-Amzn-Trace-Id header are logged

Microservices inside a single EC2

You can use a micro services architecture to structure your application as services that you can develop and deploy independently. You can install one or more of these services on each EC2 instance, with each service accepting connections on a different port. You can use a single Application Load Balancer to route requests to all the services for your application. When you register an EC2 instance with a target group, you can register it multiple times; for each service, register the instance using the port for the service.

When you deploy your services using Amazon Elastic Container Service (Amazon ECS), you can use dynamic port mapping to support multiple tasks from a single service on the same container instance. Amazon ECS manages updates to your services by automatically registering and de-registering containers with your target group using the instance ID and port for each container.

Load balancing Databases

If your web servers must be connected to the Internet and database servers are only connected to the web servers

    1. Create an Internet-facing load balancer and register the web servers with it.
    2. Create an internal load balancer and register the database servers with it.
    3. The web servers receive requests from the Internet-facing load balancer and send requests for the database servers to the internal load balancer.
    4. The database servers receive requests from the internal load balancer.

If you have ten instances in Availability Zone 1  and two instances in AZ2, the requests are distributed evenly between the two Availability Zones. As a result, the two instances in us-west-2b serve the same amount of traffic as the ten instances in us-west-2a. Instead, you should have six instances in each Availability Zone

Advantages of ALB over CLB:

    1. Support for host-based routing. You can configure rules for your listener that forward requests based on the host field in the HTTP header. This enables you to route requests to multiple domains using a single load balancer
    2. Support for routing requests to multiple applications on a single EC2 instance. You can register each instance or IP address with the same target group using multiple ports.
    3. Support for registering targets by IP address, including targets outside the VPC for the load balancer.
    4. Support for containerized applications in Amazon Elastic Container Service (Amazon ECS)

Advantages of ALB over NLB:

    1. NLB is at layer 4, does not understand applications, uses simple ICMP ping to determine the health as opposed to actual application health
    2. ALB can distribute traffic to to different web servers with own ips but port 80 running on the same host. NLB can’t

Security Groups (VPC SG)

  1. Security Groups are virtual firewalls at the instance level
  2. SGs only ALLOW traffic so remember this as acronym SAG (security allow groups)
  3. One or more security groups can be assigned to an EC2 instance
    1. When more than one SG is assigned to an EC2, the all the rules (ALLOW) are aggregated
    2. SGs can’t be assigned to subnets or VPCs
  4. Security Groups belong to a VPC. They can’t be shared across VPCs
  5. Traffic
    1. SG rules (IOACPS)
      1. rule is either Inbound or Outbound
      2. rule can specify ALLOW  only
      3. inbound rule has a source which can be a CIDR or SG or PrefixList (list of CIDRs) outbound rule has destination CIDR/SG/PrefixList
    2. Rules will only ALLOW traffic. No DENY rules.
    3. Provide type, protocol/port (example RDP 3389, MySQL/Aurora 3306) , destination for allowed traffic  for inbound and outbound
    4. In default VPC, the default SG has two rules
      1. All inbound traffic is allowed from within the same SG (rule 1)
      2. No inbound traffic is allowed from outside (no rule)
      3. outbound traffic to all destinations (0.0.0.0/0) is ALLOWED (rule 2: All Traffic, ports, protocols ->0.0.0.0/0)
    5. When you create a new SG, one rule is automatically created
      1. No inbound traffic is allowed from outside (no rule)
      2. outbound traffic to all destinations (0.0.0.0/0) is ALLOWED
  6. SG rules are stateful (unlike Network ACL rules), meaning if a protocol (say HTTP) is allowed inbound, then when a request comes in, the corresponding reply packets are allowed outbound irrespective of outbound rules, thus maintaining state.
  7. Any changes to SGs will be effective immediately. No need to stop/start EC2
  8. A prefix list is a set of one or more CIDR blocks. There are two types of prefix lists:
  • AWS-managed prefix list — Represents the IP address ranges for an AWS service. You can reference an AWS-managed prefix list in your VPC security group rules and in subnet route table entries. For example, you can reference an AWS-managed prefix list in an outbound VPC security group rule when connecting to an AWS service through a gateway VPC endpoint. You cannot create, modify, share, or delete an AWS-managed prefix list.
  • Customer-managed prefix list — A set of IPv4 or IPv6 CIDR blocks that you define and manage. You can reference the prefix list in your VPC security group rules, subnet route table entries, and transit gateway route table entries. This enables you to manage the IP addresses that you frequently use for these resources in a single group, instead of repeatedly referencing the same IP addresses in each resource. You can share your prefix list with other AWS accounts, enabling those accounts to reference the prefix list in their own resources.

The following rules apply to customer-managed prefix lists:

  • When you create a prefix list, you must specify the maximum number of entries that the prefix list can support. You cannot modify the maximum number of entries later.
  • When you reference a prefix list in a resource, the maximum number of entries for the prefix lists counts as the same number of rules or entries for the resource. For example, if you create a prefix list with a maximum of 20 entries and you reference that prefix list in a security group rule, this counts as 20 rules for the security group.
  • You can modify a prefix list by adding or removing entries, or by changing its name.
  • A prefix list supports a single type of IP addressing only (IPv4 or IPv6). You cannot combine IPv4 and IPv6 CIDR blocks in a single prefix list.
  • There are quotas related to prefix lists. For more information, see Amazon VPC quotas.
  • When you reference a prefix list in a route table, route priority rules apply. For more information, see Route priority for prefix lists.
  • A prefix list only applies to the Region where you created it. For example, if you create a list in us-east-1, it is not available in eu-west-1.
  • You cannot reference the prefix list in your EC2 Classic security group rules.

The following rules apply to AWS-managed prefix lists:

  • You cannot create, modify, share, or delete an AWS-managed prefix list.
  • When you reference an AWS-managed prefix list in a resource, it counts as a single rule or entry for the resource.
  • You cannot view the version number of an AWS-managed prefix list.

Elastic Block Storage (EBS)

  1. Block level storage volumes that can be attached to EC2 instances as root/boot or secondary volumes
  2. HDD and SSD (root and secondary) are supported
    1. Bootable root volume or secondary volume
      • General purpose SSD (code: gp2): You get 3 IOPS/GB credit every second. The volume is burstable to 3,000 IOPS.
        • What is burstable? A burstable volume is a EBS harddisk/volume that provides a baseline level of performance with the ability to burst to a higher level to support occasional spikes in usage.
        • Each volume receives an initial I/O credit balance of 5.4 million I/O credits, which is enough to sustain the maximum burst performance of 3,000 IOPS for 30 minutes.

If you are expecting an increase in load on the volume at some point in the future, you might want to know how long can that sustain with the existing credit in IOPS.

The burst duration of a volume is dependent on the size of the volume, the burst IOPS required, and the credit balance when the burst begins. This is shown in the following equation:

                             (Credit balance)
Burst duration  =  ------------------------------------
                   (Burst IOPS) - 3(Volume size in GiB)
      • Provisioned IOPS (code: io1): 10,000 and above IOPS provisioned. You pay more based on the provision.
      • Magnetic Standard: Magnetic volumes are backed by magnetic drives and are suited for workloads where data is accessed infrequently, and scenarios where low-cost storage for small volume sizes is important. These volumes deliver approximately 100 IOPS on average, with burst capability of up to hundreds of IOPS, and they can range in size from 1 GiB to 1 TiB.
    1. Secondary Volume only (can’t be used as root volume)
      1. Throughput optimized HDD (code: st1): Data warehousing, log processing
      2. COLD HDD (code: sc1): Infrequently accessed such as file server
  1. Replicated within the availability zone
  2. All volumes except standard HDD can be modified/upgraded even while they are attached as root volumes to a running EC2
    1. Only increase size possible
    2. Can switch from gp2 to provisioned IOPS io1 and vice versa
  3. Snapshots
    1. Saved on S3
    2. Incremental
    3. Snapshots of encrypted volumes are encrypted automatically and vice versa
    4. Can be shared with other accounts only if unencrypted
    5. For unencrypted volumes, you can encrypt a volume’s data by means of snapshot CCR (Create/Copy/Resotre)

      1. Create a snapshot of your unencrypted EBS volume. This snapshot is also unencrypted.
      2. Copy the snapshot while applying encryption parameters. The resulting target snapshot is encrypted.
      3. Restore the encrypted snapshot to a new volume, which is also encrypted.
    6. Convert from unencrypted EC2 boot volume to encrypted boot volume
      1. Create  AMI (Create Image) (unencrypted EC2 –> unencrypted AMI)
      2. Copy this image and check the Encryption box. Or use the CLI, you would use the copy-image mode with the --encrypted flag. (unencrypted AMI -> encrypted AMI)
      3. Re launch a new EC2 from the new encrypted AMI and apply EIP etc and test
      4. shutdown the old EC2
  4. EBS and AZs and Regions
    1. EC2 and its volumes must be in the same AZ. Since latency is important.
    2. To move an existing volume from AZ1 to AZ2, you need to take a snapshot and restore volume from that snapshot in AZ2
    3. You can copy a snapshot from Region 1 to Region 2 and then create an image or volume from that snapshot
  5. AMIs
    1. AMI’s are region specific.
    2. AMIs can be created from snapshots or volumes
    3. You can copy AMI from one region to another and then launch instance from the copied AMI in the new region
    4. Marketplace AMIs are not encrypted at rest
  6. Instance Store (Ephemeral storage)
    1. Instance stores are saved on S3
    2. Only certain EC2 types such as m class support instance store
    3. EC2 instances having root instance store volume can’t be started/stopped. Only rebooted or terminated.
    4. Instance store backed EC2 while terminated, no option to save the volume
  7. Stop/start of EBS backed EC2 instance will provision the new instance on a different hypervisor/host

Elastic Compute Cloud (EC2)

  1. When AWS started their commercial offering back in 2014/15, the only resource that they had was compute. Virtual machines were available to rent intitially and later on AWS started offering other services such as Databases, Queues, Networks etc. They named these VMs as EC2. Elastic meaning these VMs can be scaled up or down i.e., elastic.
  2. Types of instances (I=IOPS G=GPU C=Compute R=memoRy F=Hardware Accelerated).
    1. On demand Pay by the second (Linux) hour (windows)
    2. Reserved Instances 1yr or 3yr
      1. Standard upto 75% cheaper than OD
      2. Convertible (50% cheaper than OD) can be converted to another RI of same or better class (Example: OS change, CPU upgrade, mem upgrade)
      3. Scheduled Reserved Instances: When you need an EC2 for a fraction of a day/week/month
    3. Spot instances
      1. Choose bid price
      2. If spot goes above bid you loose your instances
    4. Dedicated Hosts
      1. Physical server dedicated
      2. On demand or reserved
      3. Useful for regulatory confirmations, licencing requirement of software vendors etc
    5. EC2 Types (acloud.guru Dr Mc GIFT PX acronym). Unless the type ends with letter ‘a’ (stand for AMD), they use Intel CPUs. Only exception is A1 which is AWS Graviton ARM processor based.
      1. General Purpose family
        1. A1 ARM based, General Purpose, fixed performance (non burstable, unlike T)
        2. T Low cost, burstable performance
          1. T3a AMD EPYC CPU
        3. MAC Apple Mac mini based
        4. M fixed performance (non burstable, unlike T)
      2. Compute Optimized family
        1. C type
      3. Storage Optimized family
        1. I IOPS
        2. D Density
      4. Accelerated computing family
        1. G Graphics
        2. F Hardware accelerated (FPGA)
        3. P Graphics
        4. Inf1 machine learning inference
      5. Memory Optimized family
        1. R
        2. X extreme Memory
        3. Z
    6. EC2 Placement Groups
      1. Logical grouping of EC2 instances inside a single AZ (can’t span multiple AZ’s)
      2. EC2 instances inside a group are assured to be connected to each other using a low latency, high network throughput 10 GB/S intranet
      3. Two types
        1. Cluster: Make them close
        2. Spread: Opposite of cluster. Spread as far as possible
      4. Only high end EC2’s such as Memory optimized (M), Compute optimized (C), Storage Optimized(I), GPU Optimized (G) can use placement groups.
      5. Recommended to launch homogeneous (same size/family) EC2s inside a Placement Group
      6. Moving a running EC2 into a PG is not supported. Only while launching, you can choose your PG

Simple Storage Service (S3)

  1. Simple scalable key value Object storage on cloud
  2. Limits
    1. The total volume of data and number of objects you can store are unlimited.
    2. Individual Amazon S3 objects can range in size from a minimum of 0 bytes to a maximum of 5 terabytes.
    3. The largest object that can be uploaded in a single PUT is 5 gigabytes.
    4. For objects larger than 100 megabytes, use Multipart Upload preferably.
    5. Multi-Object Delete API: delete upto 1000 objects: single HTTP request.
  3. Objects are stored in Buckets as key value pairs
    1. Bucket has unique global name
    2. https://s3-us-west-2.amazonaws.com/Bucketname/Folder1/hello.html
    3. arn:aws:s3:::bucket/resourcekey
    4. key: Folder1/hello.html value: the content of that file
    5. Also objects contain metadata and optional version number
  4. Data Consistency
    1. Read after Write consistency for new PUTs (Immediate)
    2. Eventual consistency for DELETE and Overwrite PUT (may take some time)
  5. The data are stored lexicographical/sorted alphabetically
    1. For performance, save objects of random names (add salt before filename if its based on timestamp)
  6. Tiered Storage
    S3 storage classes comparision

    1. Standard
      1. Availability 99.99% (4 nines)
      2. Durability 99.999999999 (11 nines)
    2. Standard-IA Infrequent access
      1. Cheaper than S3 standard but retrieval fee is charged
      2. 99.9 (3 nines) availability
    3. Single Zone -IA or Zone infrequent access (Released April 2018)
      1. Single zone only. No redundancy
      2. 20% cheaper than Standard-IA
      3. Use case: Store reproducible, infrequently accessed data. Example: second or third backup copies for compliance sake.
    4. Reduced Redundancy Storage
      1. Availability 99.99% (4 nines)
      2. Durability 99.99% also
    5. Glacier class
      1. Cheap but takes 4 hours to retrieve
      2. Use “Bulk retrieval” for cheaper cost
      3. Use expedited retrieval for fast retrievals
  7. Life-cycle policies
    1. Specify rules to move across storage classes at specified age and then finally delete.
  8. Versioning
    1. Total data storage across all versions is billed
    2. Once enabled you cannot disable versioning. You can suspend it for future updates. If you want to turn versioning off, you need to delete the bucket and recreate (version id)
    3. Once you delete the delete marker, you can get the file back that you have deleted while versioning on
  9. Access Control Lists
    1. S3 ACLs is a legacy access control mechanism that predates IAM. However, if you already use S3 ACLs and you find them sufficient, there is no need to change. As a general rule, AWS recommends using S3 bucket policies or IAM policies for access control.
    2. An S3 ACL is a sub-resource that’s attached to every S3 bucket and object. It defines which AWS accounts or groups are granted access and the type of access. When you create a bucket or an object, Amazon S3 creates a default ACL that grants the resource owner full control over the resource.
  10. Bucket policies
      1. Use IAM policies if:
        1. You need to control access to AWS services other than S3. IAM policies will be easier to manage since you can centrally manage all of your permissions in IAM, instead of spreading them between IAM and S3.
        2.  You have numerous S3 buckets each with different permissions requirements. IAM policies will be easier to manage since you don’t have to define a large number of S3 bucket policies and can instead rely on fewer, more detailed IAM policies.
        3. You prefer to keep access control policies in the IAM environment.
      2. Use S3 bucket policies if:
        1. You want a simple way to grant cross-account access to your S3 environment, without using IAM roles.
        2. Your IAM policies bump up against the size limit (up to 2 kb for users, 5 kb for groups, and 10 kb for roles). S3 supports bucket policies of up 20 kb.
        3. You prefer to keep access control policies in the S3 environment.
        4. Make it public
  11. S3 is AWS object storage service on the cloud. Lets you store key/value pairs (bucket name, filename is key the content of the object/file is value)
  12. S3 access is global but a bucket will need a region
  13. Encryption
    1. Client side encryption
    2. Server Side encryption
      1. SSE-S3 using S3 managed Keys
      2. SSE-KMS using KMS keys
      3. SSE-C using client provided keys
  14. Security
    1. Control access to a bucket using bucket ACL or bucket policy
  15. All buckets and objects are pvt by default
  16. Two ways to stop people from accidentally delete objects
    1. Enable versioning
    2. Enable MFA delete
  17. Cross region replication
    1. You need to first turn on versioning
    2. Then goto Management and choose cross region replication
    3. create rule to replicate all or some objects to a destination bucket.
    4. You can specify a different storage class for the replication target bucket
    5. Only new objects (not the existing ones) are replicated
  18. S3 transfer acceleration
    1. Lets you copy files to cloud front edge location as opposed to directly copying to s3 bucket thus saving time/latency since the edge location is closer to you than the S3 bucket
  19. Static website hosting on S3
    1. Create a bucket whose name is same as your domain name (without .com)
    2. Go to static website hosting and enable
    3. Grant public read access
    4. URL will be http://your-bucket-name.s3-website-REGION.amazonaws.com where region can be us-east-1 etc.
  20. S3 is global but buckets reside in regions. But no need to provide region in url or arn since they are globally unique
  21. Requester Pays Option: Can be used to pass on request/transfer costs to another AWS account
  22. Events:
    1. The bucket owner (or others, as permitted by an IAM policy) can arrange for notifications to be issued to Amazon Simple Queue Service (SQS) or Amazon Simple Notification Service (SNS) when a new object is added to the bucket or an existing object is overwritten. Notifications can also be delivered to AWS Lambda for processing by a Lambda function.
    2. Following events are supported:  s3:ObjectCreated:Puts3:ObjectCreated:Post , s3:ObjectCreated:Copy, s3:ObjectCreated:CompleteMultipartUpload, s3:ObjectCreated:*,s3:ReducedRedundancyObjectLost.
    3. Each notification is delivered as a JSON object with the following fields: Region, Timestamp, Event Type (as listed above), Request, Actor, Principal ID, Source IP of the request, Request ID, Host ID, Notification Configuration  Destination ID, Bucket Name, Bucket ARN, Bucket Owner Principal ID, Object Key, Object Size, Object ETag, Object Version ID (if versioning is enabled on the bucket).
    4. Notifications are delivered to the target in well under a second.
    5. Cost – There is no charge for this feature.
    6. Regions – The bucket and the target must reside in the same AWS Region.
  23. Optimizing S3 performance: If you consistently exceed 100+ PUT/DELETEs or 300+ GETS, you should optimize your S3.
    1. For GET only performance use CloudFront
    2. For PUT/DELETE performance use a hexadecimal hash as the prefix. This will force S3 to use different bucket partitions which will enhance performance
      1. examplebucket/232a2013-26-05-15-00-00/cust123423/photo1.jpg
      2. examplebucket/7b542013-26-05-15-00-00/cust385742/photo2.jpg
      3. examplebucket/921c2013-26-05-15-00-00/cust124843/photo2.jpg
      4. examplebucket/ba652013-26-05-15-00-00/cust874937/photo2.jpg
  24. S3 price – charged for Storage, number of requests, data transfer (tiered so more you use less charge)
  25. Bucket name has to be all lowercase letters
  26. Individual objects inside the same bucket can have different storage class  and you can turn on server side encryption at object level.
  27. Links
    1. A example bucket link https://s3-eu-west-2.amazonaws.com/myobj
    2. URL for bucket with Static website hosting:  http://mysite.s3-website-eu-west-2.amazonaws.com
  28. you can turn on SSL  https with cloudfront
  29. S3 access
    1. Every non-anonymous request to S3 must contain authentication information to establish the identity of the principal making the request. In REST, this is done by first putting the headers in a canonical format, then signing the headers using your AWS Secret Access Key.
    2. You can use  pre-signed urls
  30. Amazon S3 Select is a new (Apr 2018) capability
    1. designed to pull out only the data you need from an object, which can dramatically improve the performance and reduce the cost of applications that need to access data in S3.
    2. In the past most applications have to retrieve the entire object and then filter out only the required data for further analysis.
    3. Now S3 Select enables applications to offload the heavy lifting of filtering and accessing data inside objects to the Amazon S3 service.
    4. By reducing the volume of data that has to be loaded and processed by your applications, S3 Select can improve the performance of most applications that frequently access data from S3 by up to 400%.
    5. You can use S3 Select from the AWS SDK for Java, AWS SDK for Python, and AWS CLI.
    6. Use SELECT command as opposed to GET command
  31. Amazon Athena
    1. is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL expressions.
    2. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries you run.
    3. Athena is easy to use. Simply point to your data in Amazon S3, define the schema, and start querying using standard SQL expressions.

Cloud Watch

  1. CloudWatch  Metrics represents a time-ordered set of data points that are published to CloudWatch. Think of a metric as a variable to monitor, and the data points represent the values of that variable over time. For example, the CPU usage of a particular EC2 instance is one metric provided by Amazon EC2.
    1. Basic metrics are 5 min intervals and free. Custom metrics are 1 min intervals and cost you extra.
    2. Stored for 2 weeks
    3. Only host level metrics such as below are available free
      1. CPU
      2. Network
      3. Disk
      4. Status Checks
    4. VM level metrics such as memory metrics are not available by default. Its a custom metric done by scripts run in EC2
    5. namespace is a container for CloudWatch metrics. Metrics in different namespaces are isolated from each other, so that metrics from different applications are not mistakenly aggregated into the same statistics.
  2. You can create CloudWatch  Dashboards to customize your view of cloud watch metrics
  3. CloudWatch  Alarms  feature allows you to watch CloudWatch metrics and to receive notifications when the metrics fall outside of the levels (high or low thresholds) that you configure. You can attach multiple Alarms to each metric and each alarm can have multiple actions.
    1. A CloudWatch Alarm is always in one of three states: OK, ALARM, or INSUFFICIENT_DATA. When the metric is within the range that you have defined as acceptable, the Monitor is in the OK state. When it breaches a threshold it transitions to the ALARM state. If the data needed to make the decision is missing or incomplete, the monitor transitions to the INSUFFICIENT_DATA state.
    2. Choose resource (EC2 or ELB or DynamoDB etc)
    3. Threshold: Example: CPU utilization >= 70% for 2 out of 5 data points
    4. Actions: Alarms watch metrics and execute one or more actions by
      1. Publishing notifications to Amazon SNS topics. SNS can inturn deliver notifications using HTTP, HTTPS, SMS, Email, or an Amazon SQS queue.
      2. By initiating Auto Scaling actions
      3. Executing Resource specific actions (Reboot EC2 etc.).
    5. The actions happen only on state transitions, and will not be re-executed if the condition persists for hours or days.
    6. Multiple notifications/actions are allowed when an Alarm is triggered. You can send an email and do a scaling action and do a recovery action, all with single alarm.
    7. You can also add alarms to dashboards.
  4. CloudWatch Events Amazon CloudWatch Events (CWE) is a stream of system events describing changes in your AWS resources. The events stream augments the CloudWatch Metrics and Logs streams to provide a more complete picture of the health and state of your applications. You write declarative rules to associate events of interest with automated actions to be taken. Currently, Amazon EC2, Auto Scaling, and AWS CloudTrail are supported. Via AWS CloudTrail, mutating API calls (i.e., all calls except Describe*, List*, and Get*) across all services are visible in CloudWatch Events.
    1. An event indicates a change in your AWS environment.
    2. AWS resources can generate events when their state changes.
      1. For example, Amazon EC2 generates an event when the state of an EC2 instance changes from pending to running
      2. Update DNS with public ip after EC2 starts successfully
      3. Amazon EC2 Auto Scaling generates events when it launches or terminates instances.
    3. Your applications can emit custom events by using the PutEvents API, with a payload uniquely suited to your needs
    4. You can also set up scheduled events that are generated on a periodic basis. For example you can reboot your EC2 every 24 hours.
    5. Rules 
      1. A rule matches incoming events and routes them to targets for processing.
      2. A single rule can route to multiple targets, all of which are processed in parallel.
      3. Rules are not processed in a particular order. This enables different parts of an organization to look for and process the events that are of interest to them.
      4. A rule can customize the JSON sent to the target, by passing only certain parts or by overwriting it with a constant.
    6. Targets 
      1. A target processes events.
      2. Targets can include Amazon EC2 instances, AWS Lambda functions, Kinesis streams, Amazon ECS tasks, Step Functions state machines, Amazon SNS topics, Amazon SQS queues, and built-in targets.
      3. A target receives events in JSON format.
  5. CloudWatch Logs You can use CloudWatch Logs to monitor, store, and access your log files from EC2, CloudTrail, Route 53, and other sources.
    1. Monitor Logs from Amazon EC2 Instances in Real-time—You can use CloudWatch Logs to monitor applications and systems using log data.
      1. example, CloudWatch Logs can track the number of errors that occur in your application logs (replace ELMAH) and send you a notification whenever the rate of errors exceeds a threshold you specify.
    2. Monitor AWS CloudTrail Logged Events
    3. Archive Log Data—You can use CloudWatch Logs to store your log data in highly durable storage.
      1. You can change the log retention setting so that any log events older than this setting are automatically deleted.
    4. Log Route 53 DNS Queries—You can use CloudWatch Logs to log information about the DNS queries that Route 53 receives.
    5. You can use the CloudWatch Logs agent installer on an existing EC2 instance to install and configure the CloudWatch Logs agent.
    6. After installation is complete, logs automatically flow from the instance to the log stream you create while installing the agent. The agent confirms that it has started and it stays running until you disable it.
    7. In addition to using the agent, you can also publish log data using the AWS CLI, CloudWatch Logs SDK, or the CloudWatch Logs API.
    8. The AWS CLI is best suited for publishing data at the command line or through scripts.
    9. The CloudWatch Logs SDK is best suited for publishing log data directly from applications or building your own log publishing application.
  6. CloudWatch retains metric data as follows:
    1. Data points with a period of less than 60 seconds are available for 3 hours. These data points are high-resolution custom metrics.
    2. Data points with a period of 60 seconds (1 minute) are available for 15 days
    3. Data points with a period of 300 seconds (5 minute) are available for 63 days
    4. Data points with a period of 3600 seconds (1 hour) are available for 455 days (15 months)
  7. The unified CloudWatch agent is a piece of software running on EC2 instances and enables you to do the following:
    1. Collect more system-level metrics from Amazon EC2 instances, including in-guest metrics, in addition to the metrics listed in Amazon EC2 Metrics and Dimensions.
    2. Collect system-level metrics from on-premises servers. These can include servers in a hybrid environment as well as servers not managed by AWS.
    3. Collect logs from Amazon EC2 instances and on-premises servers, running either Linux or Windows Server.

Additional Topics

  1. Bastion host: An AWS bastion host can provide a secure primary connection point as a ‘jump’ server for accessing your private instances via the internet.
    1. Basically bastion host is EC2 running in your public subnet
    2. Allows SSH and RDP only to certain ip ranges
    3. bastion host run in security group that has SSH/RDP permissions to EC2 instances in your Private subnets
    4. You can SSH to bastion using private key and do further SSH to EC2 in private subnet using
      1. Remote Desktop Gateway for windows
      2. Agent forwarding for Linux SSH
  2. AWS Systems Manager is a Management Tool that enables you gain operational insights and take action on AWS resources safely and at scale. Using the run command, one of the automation features of Systems Manager, you can simplify management tasks by eliminating the need to use bastion hosts, SSH, or remote PowerShell.
    1. If you have a running EC2, you can find what role it is using and attach a AWS policy called “AmazonEC2RoleforSSM” to the role. Remember you can attach multiple policies to a single role.
    2. If you are launching a new EC2, you can create a new role for it use it after attaching “AmazonEC2RoleforSSM” policy
  3. NAT Gateways: To create NAT gateway, need to provide VPC, subnet (public subnet)  and EIP
  4. Elastic Map Reduce (EMR)
  5. Simple Email Service (SES) is a cloud-based email sending service designed to help digital marketers and application developers send marketing, notification, and transactional emails. It is a reliable, cost-effective service for businesses of all sizes that use email to keep in contact with their customers. You can use our SMTP interface or one of the AWS SDKs to integrate Amazon SES directly into your existing applications. You can also integrate the email sending capabilities of Amazon SES into the software you already use, such as ticketing systems and email clients.
  6. Quick Site:
    1. AWS service that will aggregate your data from multiple data sources (S3, DynamoDB, RDS, etc.) and provide business intelligence based on this data.
  7. NAS (Network Attached Storage) EFS (Elastic File System):
    1. An Amazon EFS file system is accessed by EC2 instances running inside one of your VPCs.
    2. Instances connect to a file system by using a network interface called a mount target.
    3. Each mount target has an IP address or DNS (fs-xxxxxxxx.efs.us-west-2.amazonaws.com), which AWS assigns automatically or you can specify.
    4. Use linux mount command to mount this to a folder such as /home/mysharedfolder
    5. Cost in .xx US$ per GB/Month units (around 30 cents per GB/hour)
  8. Status Checks
    1. System status check checks the physical host
      1. Examples: Power failure, Network Failure, System software issues, Hardware failure. When this happens, simply stop and restart the VM which will restart it on a different host (hardware)
    2. Instance status check checks the VM/OS
      1. Corrupt memory
      2. Exhausted memory
      3. Misconfigured network
      4. Kernel issues
      5. Reboot will fix
  9. EBS Volume types (16 TB max for all) (burst max 3000 IOPS)
    1. General Purpose SSD: gp2 Can be root/boot volume
      1. General VMs, web servers. Min 1 GB
      2. 3 IOPS/GB max 10,0000 IOPS
    2. Provisioned IOPS SSD: io1 Can be root/boot volume
      1. High volume db server. Min 4 GB
    3. Standard Magnetic HDD: Can be root/boot volume. Magnetic volumes are backed by magnetic drives and are suited for workloads where data is accessed infrequently, and scenarios where low-cost storage for small volume sizes is important. These volumes deliver approximately 100 IOPS on average, with burst capability of up to hundreds of IOPS, and they can range in size from 1 GiB to 1 TiB.
    4. Throughput Optimized HDD:  st1
      1. Can’t be root/boot volume. Min 500 GB
      2. Big data, Data warehousing, Log processing
    5. Cold HDD: sc1 Can’t be root/boot volume
      1. . Min 500 GB
  10. Workplaces
    1. Using AWS WS client one can connect to virtual desktop (windows only)
    2. Workspaces are persistent
    3. data on D drive is backed up every 12 hours
    4. No need to have AWS account
  11. Elasticity vs Scalability and difference between scaling up and scaling out
    1. Elasticity is being able to scale out and scale back (horizontal scaling) within a short period such as hours or days or weeks. You can achieve this by launching additional instances of the same type and closing them after the demand comes down
    2. Scalability is to scale up your systems as the business grows and demand increases over long term (think months and years). You can achieve scale up (vertical scaling) by increasing the memory/CPU by upgrading your instances to a new type (m1 to m2 etc)
    3. Scaling up may not be instantaneous. May need some downtime unlike scaling out which can happen instantaneously.
    4. DynamoDb is inherently scalable,  however you can increase the IOPS and decrease later to achieve elasticity
    5. RDS is not elastic. You can scale it by upgrading to a higher instance type (small to medium etc)
  12. Snowball imports/exports your data to S3. Replaces Import/Export service where you send your hard disk to AWS by courier.
    1. Snowball: 80 TB data can be transferred to AWS using a physical device
    2. Snowball Edge: 100 TB storage plus EC2 running lamda functions all in one box. Use case: On board an aircraft
    3. Snowmobile: Extremely large amounts of data. Mounted on a truck. Capacity 100 PB
  13. Advantage of Direct Connect over VPN
    1. Better bandwidth as DC uses dedicated VLAN connection from your data center to AWS
    2. VPN uses ipsec protocol over internet and can drop while using if internet has problems.
    3. VPN connections can be setup in minutes whereas direct connect takes weeks to setup
  14. Amazon Resource Names (arn)
    1. Uniquely address any resource
    2. arn:PARTITION:SERVICE:REGION:ACCOUNTID:resource
    3. arn:PARTITION:SERVICE:REGION:ACCOUNTID:resourcetype/resource
    4. arn:PARTITION:SERVICE:REGION:ACCOUNTID:resourcetype:resource
    5. PARTITION=aws SERVICE=s3 or iam etc REGION=us-west-2 etc ACCOUNTID is your accountid
    6. For globally unique resources such as S3, no need of REGION or ACCOUNTID  so simply use ::: ex: arn:aws:s3:::bybucket/myfile.txt
  15. Data transfer cost optimization
    1. Always use private ip to transfer data between two instances within a single AZ to avail local transfer rates. Otherwise regional data transfer rates will be applied.
    2. If the instances are in different AZs the regional data transfer rate will be applied regardless or private or public ip is used.
  16. Nitro vs Xen Hipervisor
    1. Nitro reduces software components at host level thus more bare metal access to EC2 hence less wastage or memory and CPU resources
    2. Eventually all XEN will be phased out to Nitro
    3. Nitro allows upto 27 PCI devices to be attached to your EC2 including all your EBS and ENI
  17. AWS Data Pipeline
    1. Web service that helps you reliably process and move data between different AWS compute and storage services, as well as on-premises data sources, at specified intervals.
    2. With AWS Data Pipeline, you can regularly access your data where it’s stored, transform and process it at scale, and efficiently transfer the results to AWS services such as Amazon S3, Amazon RDS, Amazon DynamoDB, and Amazon EMR.
    3. AWS Data Pipeline helps you easily create complex data processing workloads that are fault tolerant, repeatable, and highly available.
    4. You don’t have to worry about ensuring resource availability, managing inter-task dependencies, retrying transient failures or timeouts in individual tasks, or creating a failure notification system.
    5. AWS Data Pipeline also allows you to move and process data that was previously locked up in on-premises data silos.
  18. Only M3 and C3 support para virtualization (PV). Rest use HVM (Hardware Virtual Machine)
  19. To delegate permission to access a resource, you create an IAM role that has two policies attached. The permissions policy grants the user of the role the needed permissions to carry out the desired tasks on the resource. The trust policy specifies which trusted accounts are allowed to grant its users permissions to assume the role. The trust policy on the role in the trusting account is one-half of the permissions. The other half is a permissions policy attached to the user in the trusted account that allows that user to switch to, or assume the role
  20. Trusted adviser provides info about 4 pillars (except operational excellence) and service limits
    1. Cost optimization
    2. Security issues/improvement advise
    3. Performance issues/improvement advise
    4. Fault Tolerance (Eg. If availability zones are used properly etc)
    5. Service Limits (Eg. How many EIPs are used and left)
  21. Launch Configs
    1. You can only specify one launch configuration for an Auto Scaling group at a time, and you can’t modify a launch configuration after you’ve created it.
    2. Therefore, if you want to change the launch configuration for your Auto Scaling group, you must create a new launch configuration and then update your Auto Scaling group with the new launch configuration.
    3. When you change the launch configuration for your Auto Scaling group, any new instances are launched using the new configuration parameters, but existing instances are not affected.
  22.  Difference between bucket policies, IAM policies, and ACLs for use with S3, and examples of when you would use each.
    1. With IAM policies, companies can grant IAM users fine-grained control to their Amazon S3 bucket or objects while also retaining full control over everything the users do.
    2. With bucket policies, companies can define rules which apply broadly across all requests to their Amazon S3 resources, such as
      1. Granting write privileges to a subset of Amazon S3 resources.
      2. Customers can also restrict access based on an aspect of the request, such as HTTP referrer and IP address.
    3. With ACLs, customers can grant specific permissions (i.e. READ, WRITE, FULL_CONTROL) to specific users for an individual bucket or object
  23. Import/Export service lets you import or export data into AWS (S3, EBS) by mailing your devices to Amazon. Replaced by snowball.
  24. Amazon EC2 Auto Scaling provides you with an option to enable automatic scaling for one or more EC2 instances by attaching them to your existing Auto Scaling group. After the instances are attached, they become a part of the Auto Scaling group. The instance that you want to attach must meet the following criteria:
    1. The instance is in the running state.
    2. The AMI used to launch the instance must still exist.
    3. The instance is not a member of another Auto Scaling group.
    4. The instance is in the same Availability Zone as the Auto Scaling group.
    5. If the Auto Scaling group has an attached load balancer, the instance and the load balancer must both be in EC2-Classic or the same VPC.
    6. If the Auto Scaling group has an attached target group, the instance and the load balancer must both be in the same VPC.
    7. When you attach instances, the desired capacity of the group increases by the number of instances being attached.
    8. If the number of instances being attached plus the desired capacity exceeds the maximum size of the group, the request fails.
    9. If you attach an instance to an Auto Scaling group that has an attached load balancer, the instance is registered with the load balancer.
    10. If you attach an instance to an Auto Scaling group that has an attached target group, the instance is registered with the target group.
  25. AWS Database Migration Service (DMS)
    1. helps you migrate databases to AWS quickly and securely.
    2. The source database remains fully operational during the migration, minimizing downtime to applications that rely on the database.
    3. The AWS Database Migration Service can migrate your data to and from most widely used commercial and open-source databases. The service supports homogeneous migrations such as Oracle to Oracle, as well as heterogeneous migrations between different database platforms, such as Oracle to Amazon Aurora or Microsoft SQL Server to MySQL.
    4. It also allows you to stream data to Amazon Redshift, Amazon DynamoDB, and Amazon S3 from any of the supported sources including Amazon Aurora, PostgreSQL, MySQL, MariaDB, Oracle, SAP ASE, SQL Server and MongoDB, enabling consolidation and easy analysis of data in the petabyte-scale data warehouse.
    5. AWS Database Migration Service can also be used for continuous data replication with high-availability.
  26. You can now turn on a CloudTrail across all regions for your AWS account. CloudTrail will deliver log files from all regions to the Amazon S3 bucket and an optional CloudWatch Logs log group you specified. Additionally, when AWS launches a new region, CloudTrail will create the same trail in the new region. As a result, you will receive log files containing API activity for the new region without taking any action.
  27. With cross-zone load balancing, each load balancer node for your Classic Load Balancer distributes requests evenly across the registered instances in all enabled Availability Zones. If cross-zone load balancing is disabled, each load balancer node distributes requests evenly across the registered instances in its Availability Zone only.
  28. The AWS cloud supports many popular disaster recovery (DR) architectures from “pilot light” environments that may be suitable for small customer workload data center failures to “hot standby” environments that enable rapid failover at scale.
  29. AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics. You can create and run an ETL job with a few clicks in the AWS Management Console. You simply point AWS Glue to your data stored on AWS, and AWS Glue discovers your data and stores the associated metadata (e.g. table definition and schema) in the AWS Glue Data Catalog. Once cataloged, your data is immediately searchable, queryable, and available for ETL. AWS Glue generates the code to execute your data transformations and data loading processes.
  30. AWS SDK supports: IOS, Android, Browser (Java scripts), Java, .NET, Node.js,  PHP, Python, Ruby , Go, C++
  31. Import Export
    1. Import / Export Disk
      1. Import to S3, EBS, Glacier
      2. export from S3
    2. Import / Export Snowball
      1. Import to S3
      2. Export to S3
  32. AWS WAF:  Use to control how Amazon CloudFront or an Application Load Balancer responds to web requests. Define your conditions, combine your conditions into rules, and combine the rules into a web ACL.
    1. Conditions define the basic characteristics that you want AWS WAF to watch for in web requests:
      1. Scripts that are likely to be malicious. cross-site scripting or SQL code injection
      2. IP addresses or address ranges or Country or geographical location that requests originate from.
      3. Length of specified parts of the request, such as the query string.
      4.  Strings that appear in the request header (such as user-agent)  or body (query string).
    2. Rules
      1. Regular rules use only conditions to target specific requests. Example: All requests coming from 192.0.2.44 AND contain the value “BadBot”in the User-Agent header.
      2. Rate-based rule
        1. Rate-based rules are similar to regular rules, with a rate limit
        2. count the requests that arrive from a specified IP address every five minutes.
        3. The rule can trigger an action if the number of requests exceed the rate limit.
    3. Web ACLs: You combine the rules into a web ACL. You define an action for each rule—allow (to be forwarded to CloudFront or an Application Load Balancer), block, or count

Amazon Aurora RDS

  1. AWS Aurora is Amazon’s MySQL compatible dB
    1. Only available on cloud
    2. 5X better performance than MySQL
    3. Targeted at Oracle
  2. Scaling
    1. Automatically scales in increments of 10 GB
    2. Two copies of your data will be stored in each AZ. If you have 4 AZs in your region then 2X4=8 copies of your data will be redundantly stored by Aurora
    3. Aurora storage disk blocks are continuously scanned for errors and self healed.
  3. Read Replicas supported:
    1. Up to 15 Aurora Replicas (as opposed to only 5 MySQL read replicas)
    2. Unlike MySQL read replica’s, Aurora Replicas support automatic failover

Elasticache

  1. AWS Elasticache is an in memory cache based db for AWS cloud. Fully managed, easy to deploy, operate and scale as per customers needs
  2. Reduces the disk access by saving the most used and most critical data in the memory thus reduces latency
  3. Supports two popular cacheing engines
  4. Memcached
    1. Not persistent, memory only
    2. Port (11211)
    3. Parameter Group: acts as a container for engine configuration values that can be applied to one or more clusters. 
    4. node type (Eg: cache.r4.large or cache.t2.small)
    5. Specify security groups, subnets (for assigning private ips to nodes)
    6. Specify upto 20 nodes in your cluster. A node is a partition of your data.
  5. Redis is a in memory key-value store (persistent db) that supports structured data sets such as lists, tables. Redis can be used as db, cache and message broker
    1. Cluster mode is optional
    2. Port 6379
    3. parameter group
    4. node type (Eg: cache.r4.large or cache.t2.small)
    5. number of shards. A shard is a partition of your data and is comprised of one primary and up to five read replicas
    6.  number of replicas per shard
    7. Multi AZ with auto failover
    8. Encryption at rest/transit
    9. Backups
  6. For saving web server sessions, use AWS Elasticache (Better than DynamoDB)
  7. Elasticache can be used to
    1. Reduce load on web server, application tier
    2. Game leaderboards Redis sorted lists
    3. Messaging (Redis pub/sub)
    4. Recommendations (Redis counters & hashes)

Redshift

  1. AWS RedShift is a fast petabyte scale data warehouse service on the cloud
  2. OLAP vs OLTP
    1. OLTP is online transaction processing. Use RDS for OLTP. More writes less reads. Example:  E-commerce website with Shopping Cart
    2. OLAP is online analytical processing.
      1. Few writes and many many reads especially those that aggregate an entire column based on conditions.
      2. Example: Query for Sum of all sales in January across all states in the south region.
  3. RedShift uses columnar database
    1. Data is sequentially stored by column on the disk as opposed to by row as in case of RDS/OLTP
    2. Suitable for aggregates across all records in a single column
    3. Suitable for compression since all data in a particular column have data of same type
  4. Configuration of AWS RedShift
    1. Start with single node (max size 160 GB)
    2. You can upgrade to Multi node as your needs grow:
      1. Leader node: manages client connections. Front end t receive queries.
      2. Compute nodes: Stores data, computes queries. Up to 128 compute nodes can be deployed.
      3. Massively Parallel Processing (MPP) via distribution loads across many compute nodes that run parallelly
  5. Pricing
    1. No charges for leader node
    2. Compute nodes are charged per hour per node
    3. Backups are charged
    4. Data transfer is measured and charged
  6. Availability
    1. Available in single AZ (availability is not very important since OLAP systems are only used by few managers)
    2. You can take snapshots and restore to other AZs if needed
Copyright 2005-2016 KnowledgeHills. Privacy Policy. Contact .