A space for tech geek and cloud cloud aspirants

AWS Certified Database – Specialty exam study plan and notes

2021-03-26T00:51:00.009-04:00

I gave my last certification AWS Certified Machine Learning Specialty almost a year ago. In 2020, I took break from certifications and mostly worked to launch my book Solution's Architect Handbook . I got very positive response for the book, which brought by 5000+ aspiring solutions architect across world. You can find more details in latest post - https://getyouralgorithm.blogspot.com/2020/03/the-solution-architects-handbook.html

Finally, I completed AWS Certified Database Specialty Certification with that achieved 10th AWS certifications . As I mentioned earlier , its better to take break between the certification to keep your knowledge fresh and progress with continuous learning.

Here is my study plan for AWS Certified Database Specialty exam:

This exam is easier than either of the Professional exams and harder than any of the associate exams. At high level, this exam have good complexity, Exam expect you to have good understanding of overall database concept , and how to utilize AWS services and tools to solve database use cases. Exam mostly cover Amazon RDS, Amazon Aurora, Amazon Redshift and AWS DMS.

To know more about certification, Visit AWS Database Specialty Certification webpage.

To start the First step

Video Course: Recommend must go through Udemy course "Ultimate AWS Certified Database Specialty 2021" by Stephen Maarek and Ryaz Sayyed . They have done pretty good job to cover all th exam topic in very detail.

Exam Notes:

Read slides came with Ultimate AWS Certified Database Specialty 2021. Its 650 slides covering everything .
As I have put my notes mostly focused on some key points at the bottom of this post .

Practice Test:

Practice test by "Practice Exam | AWS Certified Database Specialty" available in Udemy from Stephen Maarek and Ryaz Sayyed - It has one full exam practice test with 65 question. This is well aligned to exam.
Finally Whizlab launch good practice test around AWS Database Specialty exam having 145 question. You can purchase here.
BrainCert has nice set of 150 questions which is very close to exam . It is must read. You can purchase here.

Lab:

Labs are must pass this exam. Make sure follow along labs for all Udemy course practical videos.

What Exam Covers:

Below are topics you need good knowledge. Topics from notes.

Redshift encryption modify cluster
Secrets manager and DB params in cloud formation
DB end to end encryption needs certificate on user end and request have to signed
Deletion protection for database CloudFormation
DNS caching for aurora.
DynamoDB TTL
RDS audit log
RDS event subscription
Redis requires auth-token
Redis cluster and non-cluster differences
RDS postgres to Aurora migaration is just creating read replica of aurora from RDS (it’s a feature)
Placement of DMS replication instance ( near target always)
Aurora clone and backtrack functions
Changeset in CFN
Redshift copy grant KMS cross region snapshots ( this is one special video in course)
Reshift classic vs elastic resize
RDS performance insights
Aurora replica priority ( you will for sure get question of that)
DMS data validation
DDB global table and stream ( DDB latency 1 Sec and DDB Stream is mili second) so pay close attention what exam is asking
Calculate RCU and WCU DDB
DDB Provisioned capacity vs on-demand (make sure you know diffrences)
DDB Burst capacity and autoscaling ( it can only handle spike for 5 mins)
Non encrypt to encrypt RDS migration from onprem ( snapshot does the magic)
To migrate DW onprem to redshift use SCT to covert schema and agent to migrate to S3 and copy data from s3 to redshift
Oracle LOB migration ( 100% exam question)
DDB backup ( make sure what is not copied over IAM policy and TTL doesn’t copy)
Aurora keep alive setting
When to use Aurora serverless ( 1 question for sure)
Aurora fault injection
Full load with CDC vs patrial load DMS
Lazy loading of read replica

My Exam Notes:

DynamoDB does not have snapshot or automated backup capability nut you can take manual backup. When you restore using point-in-time recovery, DynamoDB restores your table data to the state based on the selected date and time (day:hour:minute:second) to a new table. You restore a table without consuming any provisioned throughput on the table. You can do a full table restore using point-in-time recovery, or you can configure the destination table settings. When you do a full table restore, all table settings for the restored table come from the current settings of the source table at the time of the restore. You can also restore your DynamoDB table data across AWS Regions such that the restored table is created in a different Region from where the source table resides.

When an RDS DB instance is in the incompatible-parameters state, at least one of the parameters in the associated group is set with a value that's not compatible with the current engine version or DB instance class. To resolve this issue, change the value of each incompatible parameter to a compatible value using one of the following options:

Reset all the parameters in the parameter group to the default value.
Reset the values of the parameters that are incompatible.

By default, the time zone for a MariaDB DB instance is Universal Time Coordinated (UTC). You can set the time zone for your DB instance to the local time zone for your application instead. After you set the local time zone, all new connections to the database reflect the change. If you have any open connections to your database when you change the local time zone, you won't see the local time zone update until after you close the connection and open a new connection.

To read data from a table, you use operations such as GetItem, Query, or Scan. Amazon DynamoDB returns all the item attributes by default. To get only some, rather than all of the attributes, use a projection expression.

You can authenticate to your DB cluster using AWS IAM database authentication. IAM database authentication works with Aurora MySQL and Aurora PostgreSQL. With this authentication method, you don't need to use a password when you connect to a DB cluster. Instead, you use an authentication token.

A Redis (cluster mode disabled) cluster always has a single shard with up to 5 read replica nodes. A Redis (cluster mode enabled) cluster has up to 250 shards with 1 to 5 read replica nodes in each. Redis (cluster mode disabled) supports scaling. You can scale read capacity by adding or deleting replica nodes, or you can scale capacity by scaling up to a larger node type.

https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Replication.Redis-RedisCluster.html

If you want to back up or process items that are deleted by Time to Live (TTL) then enable Amazon DynamoDB Streams on the table and processing the streams records of the expired items.

You can use the high-performance Advanced Auditing feature in Amazon Aurora MySQL to audit database activity. To do so, you enable the collection of audit logs by setting several DB cluster parameters. When Advanced Auditing is enabled, you can use it to log any combination of supported events. You can view or download the audit logs to review them.

Amazon RDS supports native backup and restore for Microsoft SQL Server databases using full backup files (.bak files). When you use RDS, you access files stored in Amazon S3 rather than using the local file system on the database server. Alternatively, you can choose to have a new bucket created for you when you add the SQLSERVER_BACKUP_RESTORE option.

For the MariaDB DB engine, automated backups are only supported with the InnoDB storage engine (version 10.2 and later) and XtraDB storage engine (versions 10.0 and 10.1).

The Query operation in Amazon DynamoDB finds items based on primary key values.You must provide the name of the partition key attribute and a single value for that attribute. Query returns all items with that partition key value. If you need to further refine the Query results, you can optionally provide a filter expression. A filter expression determines which items within the Query results should be returned to you. All of the other results are discarded.A filter expression is applied after a Query finishes, but before the results are returned. Therefore, a Query consumes the same amount of read capacity, regardless of whether a filter expression is present.

You can't share a snapshot that's encrypted using the default AWS KMS encryption key. To share an encrypted Amazon RDS DB snapshot- 1/Add the target account to a custom (non-default) KMS key. 2/Copy the snapshot using the customer managed key 3/share the snapshot with the target account 4/ Copy the shared DB snapshot to the target account.

Redis authentication tokens enable Redis to require a token (password) before allowing clients to run commands, thereby improving data security. You can require that users enter a token on a token-protected Redis server. You also need to include it in all subsequent commands to the replication group or cluster.

Multi-AZ deployments for RDS MySQL follow synchronous replication whereas Multi-AZ deployments for Aurora MySQL follow asynchronous replication. Read Replicas can be manually promoted to a standalone database instance for RDS MySQL whereas Read Replicas for Aurora MySQL can be promoted to the primary instance

The primary and standby DB instances are upgraded at the same time for RDS MySQL Multi-AZ. All instances are upgraded at the same time for Aurora MySQL

You can stop and start a DB instance whether it is configured for a single Availability Zone or for Multi-AZ, for database engines that support Multi-AZ deployments. You can't stop an Amazon RDS for SQL Server DB instance in a Multi-AZ configuration.

Connect to your Aurora PostgreSQL DB cluster over SSL.When you connect using SSL, If your connection parameters specify sslmode=verify-ca or sslmode=verify-full, then your client requires the RDS CA certificates to be in their trust store or referenced in the connection URL. You can require that connections to your Aurora PostgreSQL DB cluster use SSL by using the rds.force_ssl parameter. By default, the rds.force_ssl parameter is set to 0 (off). You can set the rds.force_ssl parameter to 1 (on) to require SSL for connections to your DB cluster. For PostgreSQL, if the IAM role (rds_iam) is added to the master user, IAM authentication takes precedence over Password authentication so the master user has to log in as an IAM user.

For MS SQL pass the application connects using SSL, and the server certificate must be verified with "Encrypt=True;TrustServerCertificate=False;"

You can use the Oracle parameter SSL_SERVER_DN_MATCH = true to enforce that the distinguished name (DN) for the database server matches its service name. If you enforce the match verifications, then SSL ensures that the certificate is from the server. If you don't enforce the match verification, then SSL performs the check but allows the connection, regardless if there is a match. If you do not enforce the match, you allow the server to potentially fake it’s identify.

To grant role to document DB user you can use db.grantRolesToUser("user3", [{role: "read", db: "sample-database-2"}])

By using the Global Datastore for Redis feature, you can work with fully managed, fast, reliable, and secure replication across AWS Regions. Using this feature, you can create cross-Region read replica clusters for ElastiCache for Redis to enable low-latency reads and disaster recovery across AWS Regions.

Amazon DynamoDB integrates with Amazon CloudWatch Contributor Insights to provide information about the most accessed and throttled items in a table or global secondary index. DynamoDB delivers this information to you via CloudWatch Contributor Insights rules, reports, and graphs of report data.

For mongo DB to document DB migration

Application continues to write to source
Dump indexes using the Amazon DocumentDB Index Tool
Restore indexes using the Amazon DocumentDB Index Tool
Perform full load and replicate data with AWS DMS
Change application endpoint to Amazon DocumentDB cluster

In Document DB - Specifying a secondaryPreferred read preference ensures that reads are routed to a read replica when one or more replicas are active. You can use the profiler in Amazon DocumentDB (with MongoDB compatibility) to log the execution time and details of operations that were performed on your cluster. The profiler is useful for monitoring the slowest operations on your cluster to help you improve individual query performance and overall cluster performance. To find long running queries that slow down due to a suboptimal query plan, or queries that are blocked due to resource contention, use the currentOp command. You can filter the command to help narrow down the list of relevant queries to terminate. You must have opid associated with the long running query to be able to terminate the query.

AWS CloudTrail does not support logging of DynamoDB data-plane operations, such as PutItem. So you might want to consider using Amazon DynamoDB Streams as a source for these events occurring in your environment. DynamoDB is integrated with AWS Lambda so that you can create triggers—pieces of code that automatically respond to events in DynamoDB Streams. With triggers, you can build applications that react to data modifications in DynamoDB tables.

You can use the LOAD DATA FROM S3 or LOAD XML FROM S3 statement to load data from files stored in an Amazon S3 bucket to RDS.

You can invoke an AWS Lambda function from an Amazon Aurora with MySQL compatibility DB cluster with a native function or a stored procedure. Before invoking a Lambda function from an Aurora MySQL, the Aurora DB cluster must have access to Lambda.

By using the Data API for Aurora Serverless, you can work with a web-services interface to your Aurora Serverless DB cluster. The Data API doesn't require a persistent connection to the DB cluster. Instead, it provides a secure HTTP endpoint and integration with AWS SDKs. You can use the endpoint to run SQL statements without managing connections.

If you delete a source DB instance without deleting its read replicas in the same AWS Region, each read replica is promoted to a standalone DB instance.

You can create an Amazon RDS event notification subscription so you can be notified when an event occurs for a given DB instance, DB snapshot, DB security group, or DB parameter group.

An Aurora global database consists of one primary AWS Region where your data is mastered, and up to five read-only secondary AWS Regions. You issue write operations directly to the primary DB cluster in the primary AWS Region. Aurora replicates data to the secondary AWS Regions using dedicated infrastructure, with latency typically under a second.

To monitor database activity with Amazon Aurora, you can use the Database Activity Streams feature. Database activity streams provide a near real-time stream of the activity in your relational database. When you integrate database activity streams with monitoring tools, you can monitor and audit database activity. A stream of database activity is pushed from Aurora to an Amazon Kinesis data stream that is created on behalf of your Aurora DB cluster. From Kinesis, AWS services such as Amazon Kinesis Data Firehose and AWS Lambda can consume the activity stream. Database Activity Streams is a free feature, but Amazon Kinesis charges for its streams. An activity stream can also be consumed by applications for compliance management.

It is possible to map multiple IAM users or roles to the same database user account.If your application is running on Amazon EC2, then you can use EC2 instance profile credentials to access the database. You don't need to store database passwords on your instance.

When using IAM database authentication with MySQL, you are limited to a maximum of 200 new connections per second. If you are using a db.t2.micro DB instance class, the limit is 10 connections per second.

Conditional writes - DynamoDB optionally supports conditional writes for write operations (PutItem, UpdateItem, DeleteItem). A conditional write succeeds only if the item attributes meet one or more expected conditions. Otherwise, it returns an error.For example, you might want a PutItem operation to succeed only if there is no other item with that same primary key. Or you could prevent an UpdateItem operation from modifying an item if one of its attributes has a certain value. Conditional writes are helpful in cases where multiple users attempt to modify the same item.

You can't convert an unencrypted Aurora DB cluster to an encrypted one. However, you can restore an unencrypted Aurora DB cluster snapshot to an encrypted Aurora DB cluster. You can't restore an unencrypted backup or snapshot to an encrypted Amazon RDS DB instance.To encrypt an unencrypted DB instance, 1/ Take snapshot which will be unencrypted 2/ copy unencrypted snapshot by selecting encryption option 3/ Restore a new RDS DB instance from the encrypted snapshot to deploy a new encrypted DB instance.

Here are some of the ways to prevent deletion or updates to resources in an AWS CloudFormation stack: Set the DeletionPolicy attribute to prevent the deletion of an individual resource at the stack level. Use IAM policies to restrict the ability of users to delete or update a stack and its resources. Enable termination protection to prevent users from deleting the stack from the AWS CloudFormation console or CLI.

A custom endpoint for an Aurora cluster represents a set of DB instances that you choose. When you connect to the endpoint, Aurora performs load balancing and chooses one of the instances in the group to handle the connection. An Aurora DB cluster has no custom endpoints until you create one. You can create up to five custom endpoints for each provisioned Aurora cluster.

AuroraReplicaLag is a measure of lag in milliseconds when replicating updates from the primary Aurora DB instance to the reader nodes in an Aurora DB cluster. To be sure that your changes are propagated among the reader nodes, you must invalidate the cached data for read consistency. In some cases, there can be a delay when propagating changes across the reader nodes. You can observe this through an increase in the AuroraReplicaLag metric in CloudWatch, which leads to an eventual restart.

A read capacity unit (RCU) represents one strongly consistent read per second, or two eventually consistent reads per second, for an item up to 4 KB in size. For example, suppose that you create a table with 10 provisioned read capacity units. This allows you to perform 10 strongly consistent reads per second, or 20 eventually consistent reads per second, for items up to 4 KB. Transactional read requests require two read capacity units to perform one read per second for items up to 4 KB. For 20 RSU

Strongly consistent reads = 20 * 4 = 80KB/sec

Eventually consistent reads = 20 * 4 * 2 = 160KB/sec

Transactional read requests = (20 * 4)/2 = 40KB/sec

To create a Read Replica for a source DB instance, you must enable automatic backups on the source DB instance by setting the backup retention period to a value other than 0. This requirement also applies to a Read Replica that is the source DB instance for another Read Replica.

Automated backups are limited to a single AWS Region while manual snapshots and Read Replicas are supported across multiple Regions. Unlike an Amazon RDS Multi-AZ configuration, failover to a Read Replica is not an automated process. If you are using cross-Region Read Replicas, you should be certain that you want to switch your AWS resources between Regions. Cross-Region traffic can experience latency, and reconfiguring applications can be complicated.

You cannot clone databases across AWS regions. The clone databases must be created in the same region as the source databases. Currently, you are limited to 15 clones based on a copy, including clones based on other clones. After that, only copies can be created. However, each copy can also have up to 15 clones.

DynamoDB backups will backup the following from the source table : Global secondary indexes (GSIs), Local secondary indexes (LSIs), billing mode, Provisioned read and write capacity, Encryption settings.The following must be manually set up on the restored table: Autoscaling policies, AWS Identity and Access Management (IAM) policies, Amazon CloudWatch metrics and alarms, Tags, Stream settings, Time to Live (TTL) settings.

The default backup retention period is one day if you create the DB instance using the Amazon RDS API or the AWS CLI. The default backup retention period is seven days if you create the DB instance using the console. DB Snapshots can be created with the AWS Management Console, CreateDBSnapshot API, or create-db-snapshot command and are kept until you explicitly delete them. Disabling automatic backups for a DB instance deletes all existing automated backups for the instance. If you disable and then re-enable automated backups, you are only able to restore starting from the time you re-enabled automated backups. Manual snapshot limits (of 100 per region) do not apply to automated backups.

Read Replicas can also be created in a different Region than the source database. Using a cross-Region Read Replica can help ensure that you get back up and running if you experience a regional availability issue. Automated backups are limited to a single AWS Region while manual snapshots and Read Replicas are supported across multiple Regions.

You can't stop a DB instance if that instance has a read replica or that instance itself is a read replica. Therefore, you must delete all Read Replicas before stopping the corresponding primary RDS DB instance.

Upgrades to the database engine level require downtime. Even if your RDS DB instance uses a Multi-AZ deployment, both the primary and standby DB instances are upgraded at the same time. This causes downtime until the upgrade is complete, and the duration of the downtime varies based on the size of your DB instance.

Amazon Aurora, Amazon RDS for MySQL and MariaDB allow you to create a second-tier read replica from an existing first-tier read replica. Amazon RDS for PostgreSQL, Oracle and SQL Server do not support Read Replicas of Read Replicas. An Amazon RDS for MySQL or MariaDB read replica will stay active and continue accepting read traffic even after its corresponding source DB instance has been deleted. If you delete an Amazon RDS for PostgreSQL DB Instance that has Read Replicas, all Read Replicas will be promoted to standalone DB Instances and will be able to accept both read and write traffic.

The Aurora cluster cache management (CCM) feature improves the performance of the new primary/writer instance after failover occurs. The replica preemptively reads frequently accessed buffers cached from the primary/writer instance. With CCM, you can designate a specific Aurora PostgreSQL replica as the failover target. CCM ensures that data in the designated replica’s cache is synchronized with the data in the primary DB instance’s cache. To designate a specific replica for CCM, set the promotion tier priority to 0 for that Aurora replica and keep it same instance class.

You cannot change existing Redis cluster from cluster mode disable to enable. Insufficient reserved memory is often the cause of backup failures. Therefore, make sure that you have sufficient reserved memory before attempting a backup. If you have insufficient memory, you can either evict some keys or increase the value of reserved-memory-percent.

Amazon Neptune provides a Loader command for loading data from external files directly into a Neptune DB instance. You can use this command instead of executing a large number of INSERT statements, addV and addE steps, or other API calls.The Neptune Loader command is faster, has less overhead, is optimized for large datasets, and supports both Gremlin data and the RDF (Resource Description Framework) data used by SPARQL.Here are the steps of the loading process:

1. Copy the data files to an Amazon Simple Storage Service (Amazon S3) bucket.

2. Create an IAM role with Read and List access to the bucket.

3. Create an Amazon S3 VPC endpoint.

4. Start the Neptune loader by sending a request via HTTP to the Neptune DB instance.

5. The Neptune DB instance assumes the IAM role to load the data from the bucket

You can view, download, and watch database logs using the AWS Management Console, the AWS CLI, or the Amazon RDS API, however Viewing, downloading, or watching transaction logs isn't supported. In addition to viewing and downloading DB instance logs, you can publish logs to Amazon CloudWatch Logs. With CloudWatch Logs, you can perform real-time analysis of the log data, store the data in highly durable storage, and manage the data with the CloudWatch Logs Agent. AWS retains log data published to CloudWatch Logs for an indefinite time period unless you specify a retention period.

If you enable copying of Amazon Redshift snapshots to another AWS Region, and the source cluster and its snapshots are encrypted using a master key from AWS KMS, you need to configure a grant for Amazon Redshift to use a master key in the destination AWS Region. This grant enables Amazon Redshift to encrypt snapshots in the destination AWS Region.

After you create an Amazon RDS DB instance, you can't modify the allocated storage size of the DB instance to decrease the total storage space it uses. To decrease the storage size of your DB instance, create a new DB instance that has less provisioned storage size. Then, migrate your data into the new DB instance using one of the following methods:

· Use the database engine's native dump and restore method.
Note: This method causes some downtime.

· Use AWS Database Migration Service (AWS DMS) for minimal downtime.

If your database instance runs out of storage, its status changes to storage-full. add more storage space to your instance using the ModifyDBInstance API operation.

You can reduce the lag between updates to a source DB instance and the subsequent updates to the read replica by doing the following:

· Set the DB instance class of the read replica to have a storage size comparable to that of the source DB instance.

· The value for the max_allowed_packet parameter for a read replica is less than the max_allowed_packet parameter for the source DB instance.

· Disable the query cache. For tables that are modified often, using the query cache can increase replica lag because the cache is locked and refreshed often. If this is the case, you might see less replica lag if you disable the query cache. You can disable the query cache by setting the query_cache_type parameter to 0 in the DB parameter group for the DB instance.

To set the retention period for system logs, use the rds.log_retention_period parameter. You can find rds.log_retention_period in the DB parameter group associated with your DB instance. The unit for this parameter is minutes. For example, a setting of 1,440 retains logs for one day. The default value is 4,320 (three days). The maximum value is 10,080 (seven days). Your instance must have enough allocated storage to contain the retained log files.

Connect to Amazon Neptune using the Gremlin Console with Signature Version 4 authentication.

Global tables build on the global Amazon DynamoDB footprint to provide you with a fully managed, multi-region, and multi-active database that delivers fast, local, read and write performance for massively scaled, global applications. Global tables replicate your DynamoDB tables automatically across your choice of AWS Regions. Amazon Aurora global databases span multiple AWS Regions, enabling low latency global reads and providing fast recovery from the rare outage that might affect an entire AWS Region. An Aurora global database has a primary DB cluster in one Region, and up to five secondary DB clusters in different Regions.

You can prevent stack resources from being unintentionally updated or deleted during a stack update by using a stack policy. A stack policy is a JSON document that defines the update actions that can be performed on designated resources.

After you set a stack policy, all of the resources in the stack are protected by default. To allow updates on specific resources, you specify an explicit Allow statement for those resources in your stack policy. You can define only one stack policy per stack, but, you can protect multiple resources within a single policy. A stack policy applies to all AWS CloudFormation users who attempt to update the stack.

The maximum number of connections allowed to an Aurora MySQL DB instance is determined by the max_connections parameter in the instance-level parameter group for the DB instance. You can increase the maximum number of connections to your Aurora MySQL DB instance by scaling the instance up to a DB instance class with more memory, or by setting a larger value for the max_connections parameter in the DB parameter group for your instance, up to 16,000.

You can use the existing Parameters section of your CloudFormation template to define Systems Manager parameters, along with other parameters. Systems Manager parameters are a unique type that is different from existing parameters because they refer to actual values in the Parameter Store. The value for this type of parameter would be the Systems Manager (SSM) parameter key instead of a string or other value. CloudFormation will fetch values stored against these keys in Systems Manager in your account and use them for the current stack operation.

When you opt to use an HSM for management of your cluster key, you need to configure a trusted network link between Amazon Redshift and your HSM. Doing this requires configuration of client and server certificates. The trusted connection is used to pass the encryption keys between the HSM and Amazon Redshift during encryption and decryption operations. There was no way to roll the certs on an existing cluster, you need to spin up an entirely new cluster with new certs, and then migrate over to your new Cluster.

If your MySQL DB instance is currently in use with a production application, you can use the procedure to upgrade the database version for your DB instance. procedure can reduce the amount of downtime for your application.

By using a read replica, you can perform most of the maintenance steps ahead of time and minimize the necessary changes during the actual outage. With this technique, you can test and prepare the new DB instance without making any changes to your existing DB instance.

In Redshift , You can modify an unencrypted cluster to use AWS KMS encryption, using either an AWS-managed key or a customer-managed key (CMK). When you modify your cluster to enable KMS encryption, Amazon Redshift automatically migrates your data to a new encrypted cluster. You can also migrate an unencrypted cluster to an encrypted cluster by modifying the cluster. During the migration operation, your cluster is available in read-only mode, and the cluster status appears as resizing.

server_audit_events - Contains the comma-delimited list of events to log. Events must be specified in all caps, and there should be no white space between the list elements, for example: CONNECT,QUERY_DDL. This parameter defaults to an empty string. You can log any combination of the following events:

CONNECT – Logs both successful and failed connections and also disconnections. This event includes user information.
QUERY – Logs all queries in plain text, including queries that fail due to syntax or permission errors.
QUERY_DCL – Similar to the QUERY event, but returns only data control language (DCL) queries (GRANT, REVOKE, and so on).
QUERY_DDL – Similar to the QUERY event, but returns only data definition language (DDL) queries (CREATE, ALTER, and so on).
QUERY_DML – Similar to the QUERY event, but returns only data manipulation language (DML) queries (INSERT, UPDATE, and so on, and also SELECT).
TABLE – Logs the tables that were affected by query execution.

In RDS , If you need space for additional data, you can scale up the storage of an existing DB instance. If your workload is unpredictable, you can enable storage autoscaling for an Amazon RDS DB instance.

IO:XactSync – is a wait state in PostgreSQL where a session is issuing COMMIT / ROLLBACK and RDS Aurora is waiting for storage to acknowledge persistence

• Can arise when there is a very high rate of commits on the system - modify your application to commit transactions in batches

• If seen along with high CPU waits, it often means DB load exceeds allocated vCPUs - Reduce those workloads or scale up to higher CPUs

Many applications can benefit from using parallel Scan operations rather than sequential scans. For example, an application that processes a large table of historical data can perform a parallel scan much faster than a sequential one. Multiple worker threads in a background "sweeper" process could scan a table at a low priority without affecting production traffic. In each of these examples, a parallel Scan is used in such a way that it does not starve other applications of provisioned throughput resources. A parallel scan can be the right choice if the following conditions are met:

· The table size is 20 GB or larger.

· The table's provisioned read throughput is not being fully used.

· Sequential Scan operations are too slow.

Because a Scan operation reads an entire page (by default, 1 MB), you can reduce the impact of the scan operation by setting a smaller page size. The Scan operation provides a Limit parameter that you can use to set the page size for your request. Each Query or Scan request that has a smaller page size uses fewer read operations and creates a "pause" between each request.

The Solution Architect's Handbook - Kickstart your solution architect career by learning architecture design principles and strategies

2020-03-23T23:20:00.000-04:00

"Solution Architect's Handbook" is available now- https://www.amazon.com/gp/product/1838645640

I wrote this book with my years of experience in the solution architecture area.

The Solution Architect's handbook guide readers to create a robust, scaled, high available, and fault-tolerance solution by learning a different aspect of solution architecture and next-generation architecture design in the Cloud environment. This book will start with detailing solution architecture understanding and how does it fit in an agile enterprise environment. It will further take the reader through the journey of solution architecture design by providing detailed knowledge of design pillars, advance design patterns, anti-patterns, and cloud-native aspect of modern software design. The reader will further dive deep into performance optimization, security, compliance, reliability, cost optimization, and operational excellence of solution design. This book provides an in-depth understanding of the automation of security, infrastructure, DevOps, Disasters Recovery, and documentation of solution architecture. This book also provides a good understating of future proof architecture design with Data engineering, Machine learning, and IoT. As a bonus, this book will also offer a soft skill aspect of solution architect and continuous learning technique.

What this book covers:

Chapter1, The Meaning of Solution Architecture, used as a base to define what solution architecture is and its importance. It explains various benefits of having a solution architecture in place and talks about architecting on the public cloud.

Chapter2, Solution Architect in an Organization, talks about the different types of solution architect roles and how they fit in the organizational structure. It explores the various responsibilities of the solution architect in detail. It further explains the solution architect role fit in an agile organization along with agile processes.

Chapter3, Attributes of Solution Architecture, throw light on various attributes of solution architect such as scalability, resiliency, disaster recovery, accessibility, usability, security, and cost. It explains the co-existence and utilization of these architectural attributes to create an efficient solution design.

Chapter4, Principles of Solution Architecture Design, talks about architecture principles to create scalable, resilient, and high-performance architecture. It explains efficient architecture design by applying security, overcoming constraints, applying changes along with test and automation approach. It explores architecture principals to use design thinking effectively by taking service orient architecture and data-driven approach.

Chapter5, Cloud Migration, explains the benefits of cloud and approach to design cloud-native architecture. It gives an understanding of different cloud migration strategies and migration steps. It talks about hybrid cloud design and explores popular public cloud providers.

Chapter6, Solution Architecture Design Patterns, explore various architecture design patterns such as layered, microservice, event-driven, queue-based, serverless, cache-based, service-oriented, etc. with example. It demonstrates the applicability of solution architecture attributes and principles to design the best architecture as per business requirements.

Chapter7, Performance Considerations, provide an understanding of essential attributes of application performance improvement such as latency, throughput, and concurrency. It explains various technology choices to improve performance at the multiple layers of architecture such as compute, storage, database, and networking, along with performance monitoring.

Chapter8, Security Considerations, talks about various design principles applicable to secure your workload. Security needs to apply at every layer and every component of architecture, and this chapter helps you to get an understanding of the right selection of technology to ensure your architecture secure at every layer. It explores industry compliance applicable to architecture design as needed and explains security in the cloud with a shared responsibility model.

Chapter9, Reliability Considerations, talks about design principles to make your architecture reliable. It explores various disaster recovery techniques to ensure high availabilities of your application and data replication methods for business process continuations. It explains bast practice and role of cloud in application to achieve reliability

Chapter10, Operational Excellence Considerations, talks about various processes and methods to achieve operational excellence for application. It explains best practices and technology selection to apply throughout application design, implementation, and post-production to improve application operability. It explores operational excellence for cloud workload.

Chapter11, Cost Considerations, talks about various techniques to optimizing cost without risking business agility, and outcome. It explains multiple methods to monitor the cost and apply governance for cost control. It helps you to understand cost optimization using the cloud.

Chapter12, DevOps and Solution Architecture Framework, explains the importance of DevOps in application deployment, testing, and security. It explores DevSecOps and its role in the application's continuous deployment and delivery pipeline. It talks about DevOps and best practices and different tools and techniques to implement them.

Chapter13, Data Engineering and Machine Learning, talks about how to design big data and analytics architecture. It explains steps to create a big data pipeline, including data ingestion, storage, processing, and visualization. It helps you to understand concepts and technology for Internet of Things (IoT). It explores details about machine learning, model evaluation technique, and overview of various machine learning algorithms.

Chapter14, Architecting Legacy System, talks about various challenges and modernization drivers for legacy systems. It explains strategy and techniques to modernization legacy systems as the public cloud is becoming a go-to strategy for many organizations, so chapter explores cloud migration of legacy systems.

Chapter15, Documentation of Solution Architecture, talks about solution architecture document with its structure and various details required to accommodate in the documentation. It explores various IT procurement documentation such as RFP, RFI, and RFQ, where solution architecture participates in providing feedback.

Chapter16, Soft Skills to become a Better Solution Architect, talks about various soft skills required for a solution architect to be successful in the role. It helps you to understand methods to acquire strategic skills such as pre-sales and executive communication, develop design thinking and personal leadership skill such as think big and ownership. It explores techniques to establish yourself as a leader and continue improving your skillset.

Order your copy today - https://www.amazon.com/gp/product/1838645640

This book will help you to not only become better solutions architect but also crack tough interview questions.

AWS Certified Machine Learning – Specialty exam notes

2019-12-03T04:06:00.001-05:00

Disclaimer :

This is my personal raw notes so ignore any typo or formatting mistakes.

I have good experience in data engineering so so you will find notes mostly focused on Machine leaning part of exam only.

AWS AI/ML specialty certification notes:

Confusion matrix:

	True (Actual)	False (Actual)
True (Predicted)	True Positive	False Positive
False (Predicted)	False Negative	True Negative

Recall (Sensitivity, completeness, True positive rate) = true positives / (true positives + false negatives) = TP/(TP+FN)

Specificity (True negative rate) = TN/ (TP+FP)

precision (Precision, relevance) = true positives / (true positives + false positives.) = TP/(TP+FP)

Precision is defined as the ratio of True Positives over the sum of all Predicted Positives, which includes correctly labeled trues and those that we predicted as true but were really false (false positives). Another term for False Positives is Type I error and false negative is Type II error.

False Positive Rate (FPR) = FP / (FP + TN)

Accuracy = (TP+TN)/Total

F1 Score= 2x [(Precision x Recall) / (Precision + Recall)]

the Macro F1 Score is an average of all F1 scores and a higher F1 score indicates more accuracy. If the average F1 score is 0.28 and one genre (for example fiction movie or kids book) has 0.9, then this indicates that the model has a much greater accuracy with that single genre. That could mean that we have bias in our training or testing data toward that specific genre or that our data was not sufficiently randomized.

Combination:

TP/TP+FN – TP over actual true – sensitivity for bank = TPR

TN/TP+FP – TN over predicted true – specificity for e-mail spam

TP/TP+FP – TP over predicted true - precision more relevant

FP/TN+FP, TN over actual false, 1- specificity = FPR

TP+TN/TP+TN+FP+FN , all right prediction over all prediction

https://towardsdatascience.com/various-ways-to-evaluate-a-machine-learning-models-performance-230449055f15

AUC Curve – sensitive Vs 1- specificity , TPR vs FPR, for balance data like cat vs dog

PR Curve – Precision Vs Recall , use for imbalance class like cancer detection , absence of TN

The precision-recall curves are not impacted by the addition of patients without disease and with low test results. It is highly recommended to use precision-recall curves as a supplement to the routinely used ROC curves to get the full picture when evaluating and comparing tests.

https://acutecaretesting.org/en/articles/precision-recall-curves-what-are-they-and-how-are-they-used

Pipe mode allows you to stream in data from S3, instead of copying the entire dataset to every machine you are training on. For large data sets this can make a big difference. With LDA, pipe mode is only supported with RecordIO format, and LDA only supports training on a single-instance CPU.

Factorization Machines are unusual in that they expect float32 data, not integers. Factorization machines are relevant to handling sparse data for recommendation, but they don't perform dimensionality reduction per se. Factorization Machines is most often associated with high-dimensional sparse datasets that we are likely to see in one-hot encoded datasets.

Your XGBoost model has high accuracy on its training set, but poor accuracy on its validation set, suggesting overfitting. the "subsample" parameter directly addresses overfitting, but other parameters such as eta, gamma, lambda, and alpha may also have an effect. Refer to https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost_hyperparameters.html .

XGBoost is a CPU-only algorithm, and won't benefit from the GPU's of a P3 or P2. It is also memory-bound, making M4 a better choice than C4. It can be parallelize.

The SageMakerEstimator classes allow tight integration between Spark and SageMaker for several models including XGBoost, and offers the simplest solution. You can't deploy SageMaker to an EMR cluster, and XGBoost actually requires LibSVM or CSV input, not RecordIO.

For XGBoost Hyperparameter :

Optional :

Alpha – adjust L1 regulation terms

Base_Score – set initial predication score for all instances

Eta – prevent overfitting

Gamma – set minimum loss reduction for partition on leaf node of tree

Required are:

Num_round – number of round to run for hyperparamter tuning

Num_class - The number of classes.Required if objective is set to multi:softmax or multi:softprob.

SageMaker Neo provides a way to compile XGBoost models, which are optimized for the ARM processor in the Raspberry Pi.

The Amazon SageMaker Latent Dirichlet Allocation (LDA) algorithm is an unsupervised learning algorithm that attempts to describe a set of observations as a mixture of distinct categories. LDA is most commonly used to discover a user-specified number of topics shared by documents within a text corpus. LDA doesn’t provide context of topic but number of occurrences in set of documents where you can present them in a histogram. LDA is not deep learning.

Neural Topic Modeling (NTM) is deep learning algorithm for topic modeling and able to latent (conclude) topic out of known input. For example , Documents that contain frequent occurrences of words such as "bike," "car," "train," "mileage," and "speed" are likely to share a topic on "transportation,". Amazon SageMaker NTM is an unsupervised learning algorithm that is used to organize a corpus of documents into topics that contain word groupings based on their statistical distribution. Topic modeling can be used to classify or summarize documents based on the topics detected or to retrieve information or recommend content based on topic similarities.

The Amazon SageMaker BlazingText algorithm provides highly optimized implementations of the Word2vec and text classification algorithms. The Word2vec algorithm is useful for many downstream natural language processing (NLP) tasks, such as sentiment analysis, named entity recognition, machine translation, etc. This is not the optimal choice for identifying key topics.

For training SageMaker's supervised BlazingText use file mode. Each line of the input file contains a training sentence per line, along with their labels. Labels must be prefixed with __label__, and the tokens within the sentence - including punctuation - should be space separated. For example __label__4 you are ready for exam , trainer said .

When using SageMaker's BlazingText algorithm in Word2Vec mode, A "continuous bag of words" can be thought of as a jumble of words in a bag. Order doesn't matter. BlazingText doesn't use LSTM or CNN but do embedding (grouping) of text .

Object2Vec is capable of creating embeddings for arbitrary objects, such as Tweets. BlazingText can only find relationships between individual words, not entire Tweets. Amazon SageMaker Object2Vec generalizes the Word2Vec embedding technique for words to more complex objects, such as sentences and paragraphs.

Semantic Segmentation produces segmentation masks that identify classifications for each individual pixel in an image. It uses MXNet and the ResNet architecture to do this.

In addition to language translation Seq-to-seq is good for taking audio and generate sequence of token for IVR response processing . For machine translation you first need to tokenize your words into integers, which refer to vocabulary files you also must provide. Pass training data in RecordIO formant with integer token.

Valid approach for determining the optimal value of k in k-Means clustering is use the elbow method on a plot of the total within-cluster sum of square (WSS) as a function of k. K-means is an unsupervised learning method, and the best we can do is try to optimize the tightness of the resulting clusters. WSS is one way to measure that. The other choices assume a supervised learning environment.

Data Science various graph Summary : https://www.analyticsvidhya.com/blog/2017/09/6-probability-distributions-data-science/

Early stopping is a simple technique for preventing neural networks from training too far, and learning patterns in the training data that can't be generalized. Dropout regularization forces the learning to be spread out amongst the artificial neurons, further preventing overfitting. Removing layers, rather than adding them, might also help prevent an overly complex model from being created - as would using fewer features, not more. Dropout regularization and using fewer layers are two ways to prevent overfitting in a neural network. Shuffling the input data every epoch would also help.

SageMaker Neo is designed for compiling models using TensorFlow and other frameworks to edge devices such as Nvidia Jetson. The low latency requirement requires an edge solution, where the classification is being done within the vehicle itself and not over the air. Rekognition (which doesn't have an "edge mode," but does integrate with DeepLens) can't handle the very specific classification task of identifying different street signs and what they mean.

Since the tuning process learns from each incremental step, too much concurrency can actually hinder that learning. Logarithmic ranges tend to find optimal values more quickly than linear ranges.

Some of the best practice to tune hyper parameter is to choose smallest possible range of hyper parameter and run only one training job at a time.

For reference, see "Best Practices for HyperParameter Tuning" in the SageMaker developer guide at https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-considerations.html . You should understand this document; some of the recommendations surrounding choosing the best number of training jobs and how many instances to use can seem counter-intuitive.

Unless you add a policy with S3FullAccess permission to the role, SageMaker is restricted to buckets with "sagemaker" in the bucket name

When you are deploying your own custom inference container on Amazon SageMaker. Your inference container responds to port 8080, and must respond to ping requests in under 2 seconds. Model artifacts need to be compressed in tar format, not zip.

A "vanishing gradient" results from multiplying together many small derivates of the sigmoid activation function in multiple layers. ReLU does not have a small derivative, and avoids this problem.

In CNN, Transfer learning generally involves using an existing model, or adding additional layers on top of one. Retraining the whole thing isn't transfer learning.

Synthetic samples may be created using SMOTE. With datasets that are not fully populated, the Synthetic Minority Over-sampling Technique (SMOTE) adds new information by adding synthetic data points to the minority class. http://rikunert.com/SMOTE_explained

A regression model on a dataset having many features, and the resulting model appears to be underfitting. L1 effectively removes features that are unimportant, and doing this too aggressively can lead to underfitting. L2 weighs each feature instead of removing them entirely, which can lead to better accuracy.

Quantile binning splits data into a fixed number of buckets, with the same number of observations in each bin.

A learning rate that is too large may overshoot the true minima, while a learning rate that is too small will slow down convergence.

A large batch size can stuck in local minima and may not able to find minimum slop. Small batch size is better to find right local minima but too small size may need increase training.

If loss function is very curvy and has multiple local minima where the training is

getting stuck. Decreasing the batch size would help the Data Scientist stochastically get out of the local minima saddles. Decreasing the learning rate would prevent overshooting the global loss function minimum.

A term frequency–inverse document frequency (tf–idf) matrix using both unigrams and bigrams is built from a text corpus consisting of the following two sentences:

1. Please call the number below.

2. Please do not call us.

There are 2 sentences, 8 unique unigrams, and 8 unique bigrams, so the result would be (2,16).

Please do not call us.” Each word individually (unigram) is “Please,” “call,” ”the,” ”number,” “below,” “do,” “not,” and “us.”

The unique bigrams are “Please call,” “call the,” ”the number,” “number below,” “Please do,” “do not,” “not call,” and “call us.”

The idea behind StandardScaler is that it will transform your data such that its distribution will have a mean value 0 and standard deviation of 1. Given the distribution of the data, each value in the dataset will have the sample mean value subtracted, and then divided by the standard deviation of the whole dataset.

Ordinal Encoder transform is better choice to fill missing value for feature, that has ordinal value like rating H(High)>M(Medium)>L(Low)>N(No) or size data L(Large)>M(Medium)>S(Small)

SimpleImputer is a scikit-learn class which is helpful in handling the missing data in the predictive model dataset. It replaces the NaN values with a specified placeholder.

Labelbinarizer transform a binary label like Y and N in a one Vs all fashion.

MinMaxScaler transformer transforms feature by scaling each feature to a given range.

Data transformation - https://docs.aws.amazon.com/machine-learning/latest/dg/data-transformations-reference.html

The Cartesian product transformation takes categorical variables or text as input, and produces new features that capture the interaction between these input variables.

Textbook	Title	Binding	Cartesian product of no_punct(Title) and Binding
1	Economics: Principles, Problems, Policies	Hardcover	{"Economics_Hardcover", "Principles_Hardcover", "Problems_Hardcover", "Policies_Hardcover"}

Protobuf used for multi-part data for large data set. For file based input use text or csv. For unsupervised learning pass label_size =0 which means there is no target data available. pass label_size =1 means your target have one column with target data and applicable to supervised learning. Example Input data channel specification is – text/csv;label_size=0

If no label size mentioned then it default to 1 which means one column has target data.

Amazon sagemaker require that a csv file doesn’t have header record and that the target variable in first column. For example text/csv. For inference , its assume that csv input has no lable column .

One hot encoder has option None, first or array to drop a feature. None is default.

Cross validations - https://medium.com/datadriveninvestor/k-fold-and-other-cross-validation-techniques-6c03a2563f1e

OSB (Orthogonal Sparse Bigram) create groups of words and output the pair of words that include the first word.

TF-IDF (Term frequency – Inverse document Frequency) determine how important a word in a document by giving weight to the word that are common. This determine just count of the word not importance of the word.

Bag-of-word depict token of input document text and give statically depiction like histogram showing count of each word.

N-gram give multi-word phrase in a text document. 1-gram output individual word while 2-gram give pair of word etc.

Setting low learning_rate like 0.1 means your model will learn slowly and less sensitive to outliers. High learning_rate like 0.8 will make learn quickly and more sensitive to outliers.

Model hosting and inference :

https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-inference-code.html

https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-hosting.html

In your CreateModel request, the container definition includes the ModelDataUrl parameter, which identifies the S3 location where model artifacts are stored. Amazon SageMaker uses this information to determine where to copy the model artifacts from. It copies the artifacts to the /opt/ml/model directory for use by your inference code.The ModelDataUrl must point to a tar.gz file. Otherwise, Amazon SageMaker won't download the file.

Amazon SageMaker strips all POST headers except those supported by InvokeEndpoint. Amazon SageMaker might add additional headers. Inference containers must be able to safely ignore these additional headers.
To receive inference requests, the container must have a web server listening on port 8080 and must accept POST requests to the /invocations endpoint.
A customer's model containers must accept socket connection requests within 250 ms.
A customer's model containers must respond to requests within 60 seconds. The model itself can have a maximum processing time of 60 seconds before responding to the /invocations. If your model is going to take 50-60 seconds of processing time, the SDK socket timeout should be set to be 70 seconds.

Ground Truth provides two features that help improve the accuracy of your data labels and reduce the total cost of labeling your data. The first feature is annotation consolidation. This helps to improve the accuracy of your data object's labels. It combines the results of multiple worker's annotation tasks into one high-fidelity label. The second feature is automated data labeling. This uses machine learning to label portions of your data automatically without having to send them to human workers.
Sagemaker support two kind of hyperparameter tuning :

Random Search: In a random search, hyperparameter tuning chooses a random combination of values from within the ranges that you specify for hyperparameters for each training job it launches. Because the choice of hyperparameter values doesn't depend on the results of previous training jobs, you can run the maximum number of concurrent training jobs without affecting the performance of the search.

Bayesian Search: Bayesian search treats hyperparameter tuning like a [regression] problem. Given a set of input features (the hyperparameters), hyperparameter tuning optimizes a model for the metric that you choose. To solve a regression problem, hyperparameter tuning makes guesses about which hyperparameter combinations are likely to get the best results, and runs training jobs to test these values. After testing the first set of hyperparameter values, hyperparameter tuning uses regression to choose the next set of hyperparameter values to test.

With the new model tracking capabilities in Amazon SageMaker, you can quickly identify the most relevant model by searching through different parameters including the learning algorithm, the hyperparameter settings, and any tags that have been added during training runs. You can also compare and rank training runs based on their performance metrics, such as training loss and validation accuracy, to quickly identify the highest performing models.

You can use various AWS services to transform or preprocess records prior to running inference. At a minimum, you need to convert the data for the following:

Inference request serialization (handled by you)
Inference request deserialization (handled by the algorithm)
Inference response serialization (handled by the algorithm)
Inference response deserialization (handled by you)

https://aws.amazon.com/data-visualization/

When specifying a training or inference container, use the :1 tag at the end of the path to use the stable version. If you want the latest version, use :latest but that might not be backward compatible.

To Creates a model in Amazon SageMaker. In the request, you name the model and describe a primary container. For the primary container, you specify the docker image containing inference code, artifacts (from prior training), and custom environment map that the inference code uses when you deploy the model for predictions. In the CreateModel request, you must define a container with the PrimaryContainer parameter.
To host your model, you create an endpoint configuration with the CreateEndpointConfig API, and then create an endpoint with the CreateEndpoint API. Amazon SageMaker then deploys all of the containers that you defined for the model in the hosting environment.
To run a batch transform using your model, you start a job with the CreateTransformJob API. Amazon SageMaker uses your model and your dataset to get inferences which are then saved to a specified S3 location.

When using a k-fold cross validation method, we want to see that all k-groups have close to the same error rate. Otherwise, this may indicate that the data was not properly randomized before the training process.

If the Hadoop cluster has Spark, you can use the SageMaker Spark Library to convert Spark DataFrame format into protobuf and load onto S3. From there, you can use SageMaker as normal.

We can introduce more data, add early stopping to the training job and reduce features among other things to help return the model to a generalizer.

When both training and testing error is high, it indicates that our model is underfitting the data. We can try to add more details to the dataset, gather more data for training and/or run the training session longer. We might also need to identify a better algorithm.

Residual is the actual value minus the predicted value. If most of our residuals are positive numbers, that means that our predicted values are mostly less than the actual values. This means that our model is consistently underestimating

When using a custom algorithm, you need to ensure that the desired metrics are emitted to stdout output. You also need to include the metric definition and regex expression for the metric in the stdout output when defining the training job.

Root Mean Square Error (RMSE) is the standard deviation of the residuals (prediction errors). Residuals are a measure of how far from the regression line data points are; RMSE is a measure of how spread out these residuals are. The Root Mean Square Error (RMSE) metric is a distance measure between the predicted numeric target and the actual numeric answer (ground truth). The smaller the value of the RMSE, the better the predictive accuracy of the model. A model with perfectly correct predictions would have an RMSE of 0.

Amazon SageMaker Containers is a library that implements the functionality you need to create containers to run scripts, train algorithms, or deploy models that are compatible with Amazon SageMaker.

Algorithm :

Supervised :

· Linear models

· XGBoost (eXtreme Gradient Boosting)

· k-nearest neighbors (k-NN)

· factorization machine

· BlazingText multi-class, multi-label text classification

· DeepAR forecasting

· SageMaker Object2Vec

· Sequence to Sequence

Un-supervised :

· IP Insights

· K-means

· BlazingText Skip-gram and continuous bag-of-words (CBOW)

· Latent Dirichlet Allocation (LDA) algorithm

· NTM (Neural Topic Modeling)

With the BlazingText algorithm, you can scale to large datasets easily. Similar to Word2vec, it provides the Skip-gram and continuous bag-of-words (CBOW) training architectures as unsupervised learning. BlazingText's implementation of the supervised multi-class, multi-label text classification algorithm extends the fastText text classifier to use GPU acceleration with custom CUDA kernels.

For supervised mode, the training/validation file should contain a training sentence per line along with the labels. Labels are words that are prefixed by the string __label__. Here is an example of a training/validation file:

__label__4 linux ready for prime time , intel says , despite all the linux hype , the open-source movement has yet to make a huge splash in the desktop market . that may be about to change , thanks to chipmaking giant intel corp .

The Amazon SageMaker DeepAR forecasting algorithm is a supervised learning algorithm for forecasting scalar (one-dimensional) time series using recurrent neural networks (RNN). When your dataset contains hundreds of related time series, DeepAR outperforms the standard ARIMA and ETS methods. You can also use the trained model to generate forecasts for new time series that are similar to the ones it has been trained on.

Amazon SageMaker trains the DeepAR model by randomly sampling training examples from each target time series in the training dataset. Each training example consists of a pair of adjacent context and prediction windows with fixed predefined lengths. To control how far in the past the network can see, use the context_length hyperparameter. To control how far in the future predictions can be made, use the prediction_length hyperparameter.

You can create training and test datasets that satisfy this criteria by using the entire dataset (the full length of all time series that are available) as a test set and removing the last prediction_length points from each time series for training. During training, the model doesn't see the target values for time points on which it is evaluated during testing.

A factorization machine is a general-purpose supervised learning algorithm that you can use for both classification and regression tasks. It is an extension of a linear model that is designed to capture interactions between features within high dimensional sparse datasets economically. For example, in a click prediction system, the factorization machine model can capture click rate patterns observed when ads from a certain ad-category are placed on pages from a certain page-category. Factorization machines are a good choice for tasks dealing with high dimensional sparse datasets, such as click prediction and item recommendation.

The Amazon SageMaker image classification algorithm is a supervised learning algorithm that supports multi-label classification. It takes an image as input and outputs one or more labels assigned to that image. It uses a convolutional neural network (ResNet) that can be trained from scratch or trained using transfer learning when a large number of training images are not available.

Amazon SageMaker IP Insights is an unsupervised learning algorithm that learns the usage patterns for IPv4 addresses. It is designed to capture associations between IPv4 addresses and various entities, such as user IDs or account numbers. You can use it to identify a user attempting to log into a web service from an anomalous IP address, for example. Or you can use it to identify an account that is attempting to create computing resources from an unusual IP address.

K-means is an unsupervised learning algorithm. It attempts to find discrete groupings within data, where members of a group are as similar as possible to one another and as different as possible from members of other groups. You define the attributes that you want the algorithm to use to determine similarity.

Amazon SageMaker uses a modified version of the web-scale k-means clustering algorithm. Compared with the original version of the algorithm, the version used by Amazon SageMaker is more accurate. Like the original algorithm, it scales to massive datasets and delivers improvements in training time. To do this, the version used by Amazon SageMaker streams mini-batches (small, random subsets) of the training data.

For training, the k-means algorithm expects data to be provided in the train channel (recommended S3DataDistributionType=ShardedByS3Key), with an optional test channel (recommended S3DataDistributionType=FullyReplicated) to score the data on. Both recordIO-wrapped-protobuf and CSV formats are supported for training. You can use either File mode or Pipe mode to train models on data that is formatted as recordIO-wrapped-protobuf or as CSV.

For inference, text/csv, application/json, and application/x-recordio-protobuf are supported. k-means returns a closest_cluster label and the distance_to_cluster for each observation.

Amazon SageMaker k-nearest neighbors (k-NN) algorithm is an index-based algorithm and supervised. It uses a non-parametric method for classification or regression. For classification problems, the algorithm queries the k points that are closest to the sample point and returns the most frequently used label of their class as the predicted label. For regression problems, the algorithm queries the k closest points to the sample point and returns the average of their feature values as the predicted value.

Training with the k-NN algorithm has three steps: sampling, dimension reduction, and index building. Sampling reduces the size of the initial dataset so that it fits into memory. For dimension reduction, the algorithm decreases the feature dimension of the data to reduce the footprint of the k-NN model in memory and inference latency.

Typically, you use dimension reduction for high-dimensional (d >1000) datasets to avoid the “curse of dimensionality” that troubles the statistical analysis of data that becomes sparse as dimensionality increases. The main objective of k-NN's training is to construct the index. The index enables efficient lookups of distances between points whose values or class labels have not yet been determined and the k nearest points to use for inference.

The Amazon SageMaker Latent Dirichlet Allocation (LDA) algorithm is an unsupervised learning algorithm that attempts to describe a set of observations as a mixture of distinct categories. LDA is most commonly used to discover a user-specified number of topics shared by documents within a text corpus. Here each observation is a document, the features are the presence (or occurrence count) of each word, and the categories are the topics. Since the method is unsupervised, the topics are not specified up front, and are not guaranteed to align with how a human may naturally categorize documents. The topics are learned as a probability distribution over the words that occur in each document. Each document, in turn, is described as a mixture of topics.

Linear models are supervised learning algorithms used for solving either classification or regression problems. For input, you give the model labeled examples (x, y). x is a high-dimensional vector and y is a numeric label. For binary classification problems, the label must be either 0 or 1. For multiclass classification problems, the labels must be from 0 to num_classes - 1. For regression problems, y is a real number.

The best model optimizes either of the following:

· Continuous objectives, such as mean square error, cross entropy loss, absolute error.

· Discrete objectives suited for classification, such as F1 measure, precision, recall, or accuracy.

Amazon SageMaker NTM (Neural Topic Modeling) is an unsupervised learning algorithm that is used to organize a corpus of documents into topics that contain word groupings based on their statistical distribution. Documents that contain frequent occurrences of words such as "bike", "car", "train", "mileage", and "speed" are likely to share a topic on "transportation" for example. Topic modeling can be used to classify or summarize documents based on the topics detected or to retrieve information or recommend content based on topic similarities. The topics from documents that NTM learns are characterized as a latent representation because the topics are inferred from the observed word distributions in the corpus. The semantics of topics are usually inferred by examining the top ranking words they contain.

Although you can use both the Amazon SageMaker NTM and LDA algorithms for topic modeling, they are distinct algorithms and can be expected to produce different results on the same input data.

The Amazon SageMaker Object2Vec algorithm is a general-purpose neural embedding algorithm that is highly customizable. It can learn low-dimensional dense embeddings of high-dimensional objects. The embeddings are learned in a way that preserves the semantics of the relationship between pairs of objects in the original space in the embedding space. You can use the learned embeddings to efficiently compute nearest neighbors of objects and to visualize natural clusters of related objects in low-dimensional space, for example. You can also use the embeddings as features of the corresponding objects in downstream supervised tasks, such as classification or regression.

Amazon SageMaker Sequence to Sequence is a supervised learning algorithm where the input is a sequence of tokens (for example, text, audio) and the output generated is another sequence of tokens. Example applications include: machine translation (input a sentence from one language and predict what that sentence would be in another language), text summarization (input a longer string of words and predict a shorter string of words that is a summary), speech-to-text (audio clips converted into output sentences in tokens). Amazon SageMaker seq2seq uses Recurrent Neural Networks (RNNs) and Convolutional Neural Network (CNN) models with attention as encoder-decoder architectures.

The XGBoost (eXtreme Gradient Boosting) is a popular and efficient open-source implementation of the gradient boosted trees algorithm. Gradient boosting is a supervised learning algorithm that attempts to accurately predict a target variable by combining an ensemble of estimates from a set of simpler, weaker models. XGBoost a solid choice for problems in regression, classification (binary and multiclass), and ranking.

he Amazon SageMaker implementation of XGBoost supports CSV and libsvm formats for training and inference:

· For Training ContentType, valid inputs are text/libsvm (default) or text/csv.

· For Inference ContentType, valid inputs are text/libsvm or (the default) text/csv.

Note

For CSV training, the algorithm assumes that the target variable is in the first column and that the CSV does not have a header record. For CSV inference, the algorithm assumes that CSV input does not have the label column.

Learn more details on various algorithm:

https://docs.aws.amazon.com/sagemaker/latest/dg/algos.html

Finally glad to announce that , my book - Solution's Architect Handbook got published . Order your copy here and find more details in latest post - https://getyouralgorithm.blogspot.com/2020/03/the-solution-architects-handbook.html

AWS Certified Machine Learning – Specialty exam study plan and notes

2019-07-04T13:39:00.000-04:00

I gave my last certification AWS Certified Security Specialty in January-2019, and after 10 month gap I completed AWS Certified Machine Learning Specialty Certification with that achieved 9 of 11 certifications . As I mentioned earlier , its better to take 6 month break between the certification to keep your knowledge fresh and progress with continuous learning. Things got bit slow on certification side, as I am working on a book which should be publish in early 2020. Writing 500 page book with full time job is very intense task. However , I used my travel time to watch certification video course and prepared full in final week before exam.

Finally glad to announce that , my book - Solution's Architect Handbook got published . Order your copy here and find more details in latest post - https://getyouralgorithm.blogspot.com/2020/03/the-solution-architects-handbook.html

Here is my study plan for AWS Certified Machine Learning – Specialty exam:

This exam is easier than either of the Professional exams and harder than any of the associate exams. At high level, this exam have pretty much good complexity, It not only test your AWS knowledge but in general machine learning knowledge. Exam expect you to have good understanding of overall machine learning concept and popular algorithm, and how to utilize AWS services and tools to solve machine learning use cases. Data engineering is very important for machine learning so exam cover good amount of data preparation part and test your knowledge on that.

To know more about certification, Visit AWS Machine Learning Specialty Certification webpage.
To start the First step

Video Course: Recommend must go through Cloud Guru Videos first and then Linux Academy. AWS has done pretty good job so must complete AWS trainings listed in Exam preparation page.

ACloudGuru Machine Learning Specialty Course - If you are new to machine learning and data engineering so its recommended to go through their course first. However, aCloudGuru has prepared detail course material but not very deep , I don't think it will be enough to clear exam but very good to build good foundation. This course is covering all data engineering and machine learning aspect like data collection, processing and visualization. Various Machine learning algorithm with example and covered overall machine learning cycle.

Linux Academy Machine Learning Specialty Course - Linux Academy have put excellent course which simplify understanding of complex topic in very creative way . This course will help you to develop machine learning concept in very simple way. A must go course material for exam.

AWS Machine Learning Exam preparation - Make sure to go through this digital training course to build good foundation in data science and general ML concept.

Udemy ML Course by Sundog education - I will say don't go for this course directly until if you have some good experience in ML concept. There are lots of theory and complicated explanation. You can use this as summary of your leaning and take this course after Linux Academy as optional learning.

Make sure to learn detail working of each Sagemaker in-build algorithm from this Youtube playlist
Get dive deep into Amazon Sagemaker with this Youtube Video playlist

Exam Notes:

Read through Linux Academy slides provided in download section of their course.
As I have good experience in data engineering so my notes mostly focused on ML model where I focused more - https://getyouralgorithm.blogspot.com/2019/12/aws-certified-machine-learning.html

Practice Test:

Udemy ML exam practice test by sundog - It has one full exam practice test with 65 question. This is well aligned to exam.
Finally Whizlab launch good practice test around AWS ML Specialty exam having 145 question. You can purchase here. However , This practice exam has question going deep into API and code level , where exam don't go into code level ask. But overall good to build concept.
Take sample practice test from AWS certification portal. It has 20 question and give good idea of exam. However it is easier than final exam.
Linux Academy practice test is good but much simpler than real exam.
ACloudGuru has chapter practice test which is basic.

Lab:

Labs are must pass this exam. Make sure to go through all Linux Academy labs and follow along labs for all acloud guru practical videos.

WhitePaper:

Blogs and AWS UserDocs:

Must go through -
https://docs.aws.amazon.com/sagemaker/latest/dg/algos.html

I will have more link in my exam notes.

What Exam Covers:

Almost 3 to 4 questions on operating Amaszon Sagemaker over VPC and S3 VPC endpoint
Couple of tricky question on confusion matrix . You must understand concept of confusion matrix along with recall and precision . Don't just memorize it.
Lots of question in Data Engineering covering Glue, Athena, EMR and Kinesis
Just couple of questions from managed AI service like comprehend, Polly, Lex etc.
3 to 4 question on building automated data engineering pipeline using lambda funcation and event trigger.
Lots of tricky question on hyper-parameter tuning for deep learning and other supervised/unsupervised algorithm. Develop clear concept on early stopping, L1 and L2 regularization, model over-fitting and under-fitting, Learning rate, Cost function, batch size, reduce model complexity , dimension reduction etc.
5 to 6 question on Data exploration and analysis including charts.
10 to 12 questions on choosing right algorithm as per business problem.

AWS re:Invent 2018: Meet Preston, and Explore Your Digital Twin in Virtu...

2019-01-02T21:02:00.001-05:00

Finally glad to announce that , my book - Solution's Architect Handbook got published . Order your copy here and find more details in latest post - https://getyouralgorithm.blogspot.com/2020/03/the-solution-architects-handbook.html

https://www.youtube.com/watch?v=UH5r1rEz9Gg&t=1266s

AWS Certified Security Specialty Exam Study Plan

2018-09-04T00:51:00.002-04:00

I normally like to take 4 to 6 months breaks between each certification to keep refreshing my knowledge on AWS. I completed all my Associates and Pro certifications in 2017 and targetting to complete Specialities certification one in every 6 months to keep myself up-to-date. In that series, I completed big data certification in Jun-2018 and now completed my AWS Certified Security Specialty exam in January-2019.

Here is my study plan for AWS Certified Security – Specialty exam:

This exam is easier than either of the Professional exams and harder than any of the associate exams. At high level, this exam have pretty much good complexity and include some new feature which you won't find any of the course covering.

I will give Kudos to Linux Academy course for covering the deep concept, which appeared in the exam but you must go through entire course twice and complete all labs to get success. However, some concept you need to focus more, which is not covered in course mainly:

Query logging using Athena
Cloudtrail integration with Athena
Amazon Macie
Glacier Vault lock
Different e-mail protocols in secure port
Quicksight

There is strong focus on KMS , Mutli-account strategy using AWS Organization and many troubleshooting questions. See more details at the end of the blog.

To know more about certification, Visit AWS Security Specialty Certification webpage.
To start the First step

Book:

For any AWS certification , You need to have solid understanding of architecture and how to apply various practice to optimize the solution in real time. I found Solution Architect's Handbook pretty useful . This is 490 pages of book covering all aspects of solution architecture and help you to not only clear the certification but also crack the interview. I will say price is pretty reasonable given that details content . You can order from amazon - https://www.amazon.com/gp/product/1838645640

Video Course: Recommend must go through Cloud Guru Videos first and then Linux Academy.

ACloudGuru Security Specialty Course - aCloudGuru has prepared detail course material but not very deep , however it is covering all security aspect like KMS, IAM, securing S3, CloudTrail, CloudWatch, multiple AWS account access, Config, VPC, security groups, NACLs, and WAF.

Linux Academy Security Specialty Course - This is very detailed , deep and well alinged with exam along with labs. A must go course material for exam. Specially love their security runbook.

Go through this YouTube playlist for AWS re-invent videos.

Exam Notes:

Read through linux academy security runbook.
Read through all questions explanation in Whizlab .

Demo Test:

Whizlab test helped a lot to prepare well on many topics. You can purchase test here with nominal price and 240 questions.

Lab:

Labs are must pass this exam. Make sure to go through all Linux Academy labs and follow along labs for all acloud guru practical videos.

WhitePaper:

Blogs and AWS UserDocs:

You may not need to read all of them but it is great source to clarify concept in depth

What Exam Covers:
Config, Guard Duty, WAF, VPC, NACL, Security groups, VPC Endpoints, Cloudwatch Logs, Penetration testing, STS, Key Pairs, Cognito, Trusted Advisor, Cloudfront, DDOS.

How to Restrict Amazon S3 Bucket Access to a Specific IAM Role
Don’t create users and put your keys on github.
Classic load balancing and perfect forward secrecy especially based on it being essentially phased out.
Using ALB and Cloudfront - how many certs you need and where they are taken from like Region
Compromised EC2 and memory dumps
Scenario based questions around buckets and keys, what key for each bucket and why.
Key rotation questions and KMS 'via-service'
Know how AWS services (DynamoDB, SSM, RDS, Kinesis, etc) integrate with and use KMS for encryption
Use your own Encryption Keys with S3’s Server-Side Encryption
New Amazon S3 Encryption & Security Features
How to Encrypt and Decrypt Your Data with the AWS Encryption CLI
Service Control Policies (SCPs)
Rotating Customer Master Keys
How can I make sure the bucket owner has access to resources that are copied or moved between Amazon S3 buckets owned by different AWS accounts?
Bucket Permission examples
How AWS Systems Manager Parameter Store Uses AWS KMS
How to encrypt and sign DynamoDB data in your application
How Amazon Relational Database Service Uses AWS KMS
Encrypt and Decrypt Amazon Kinesis Records Using AWS KMS

Study Plan for AWS Certified Cloud Practitioner

2018-08-07T22:08:00.003-04:00

I started receiving lots of requests to provide insight and study plan for AWS Certified Cloud Practitioner. There are lots of professionals going for it, as it provides essential credential and knowledge to people, who want to start as AWS Account manager, Cloud project manager, Program Manager and Sales Manager.

I decided to give this exam to gauge the complexity from the point of professionals who are just starting on AWS and overall cloud journey. I have to say this exam has significant complexity, if you are just starting on AWS and cloud space. You need to build a strong foundation and understanding of scalable and fault-tolerance architecture. Also, you need to get well familiar with the breath of services from networking. Storage, compute, security, billing (cost explorer) etc . If not already.

You don't need to go into the details of services, but you should know their role in overall architecture along with a brief overview.

You can find more details regarding certification here.

Here is my lesson learned for AWS Certified Cloud Practitioner exam:

The bar is high, if it is your first time on AWS and solution architecture, as most of the question involves a flavor of scenario. Good news is questioned format is short mostly from one to two lines. I found questions related to Scalability and fault-tolerance could be tricky. Lots of questions from security like protection from DDoS using WAF. Good amount of questions from trusted advisior, consolidated billing, EC2 cost tier (spot, on-demand, reserve), S3 , Glacier, CloudFront, architecture principal, IAM role, VPC. Couple of simple question about Cloudformation, Database, AutoScaling and AWS support. Know Scalability and high-availability in the aspect of Multi-AZ Vs Multi-Region.

You can get an extra 30 minute time by using the following steps, especially if English is not your native language:

From the top navigation, click Upcoming Exams
On the right, click the Request Exam Accommodations button
Click the Request Accommodation button
Select ESL +30 Minutes from the accommodation dropdown
Click Create

To start with the First step

Video Course: Recommend must go through Cloud Guru Videos first and then Linux Academy.

AWS Cloud Practitioner Essentials (Digital) - This training offered by AWS itself so must go through it.
Linux Academy AWS Essential - You will learn AWS services in a very simple manner and this very good training if your first time with AWS. You can subscribe to Linux Academy by clicking here.
Linux Academy AWS Certified Cloud Practitioner - Cover all aspects of AWS Certified Cloud Practitioner Exam.
ACloudGuru Certified Cloud Practitioner Course - aCloudGuru has prepare very detail course, must go to get success in the exam.

Demo Test:

Whizlab test helped a lot to prepare well on any topics. You can purchase test here with nominal price and 240 questions.
Linux Academy Demo test is worth taking a look which comes with 40 questions.

Lab:

Labs are must pass this exam. Make sure to go through all Linux Academy labs and follow along labs for all acloud guru practical videos.

Whitepaper:

Consider this as your exam notes :)

Keep Visiting for updates.

Next Steps:

Migrate your Oracle Exadata workload to Amazon Web Services

2018-06-21T10:53:00.000-04:00

The Oracle Exadata Database Machine is a combined compute and storage system for running Oracle Database software. It is a single piece of hardware that incorporates the database engine, the storage engines and the networking components you need to create a full stack solution.

Oracle Exadata doesn't offer a lot of flexibility and very expensive when its come to scale capacity as you either have to buy a quarter rack, half rack or full rack. Below table showing number of Database and storage servers configure for Oracle Exadata X6-2. Suppose, If your application needs five database servers then you have to buy a full rack, where other three servers sitting idle, while you are paying for it. Also, storage and computes are tightly coupled, so you end up paying for high compute even if you are just looking for more storage.

	Full Rack	Half Rack	Quarter Rack
Database Servers	8 (352 cores )	4 (176 cores )	2 (88 cores )
Exadata Storage Servers	14 (280 cores , 1,344 TB)	7 (140 cores , 672 TB)	3 (60 cores , 288 TB)

Another problem is, Relational databases are not designed for mixed workloads. When discussing mixed workloads, it is helpful to differentiate between the two types of workloads that relational databases are designed to handle—operational and analytical. With relational databases, you have to choose which workload to optimize for. You cannot have both. Also, complex ETL need to create a maintain every time new query comes from business, In most organizations, each time a new question needs to be asked or a new application built, a newer, better model is created—and no model is ever the same.

Why consider migrating your workload from Oracle Exadata to Amazon Web Services?

Amazon Web Services (AWS) has different services available as per your workload requirement. You have options to decouple storage from your compute requirement and save lots of costs while able to achieve high performance at the same time. Your team can be more agile by getting the flexibility to experiment with pay as you go pricing model. You get the advantage of leveraging a wide array of integrated AWS services, which makes your data more discoverable and future proof.

For your relational data, You can orchestrate high performance and highly available application database using Amazon RDS fully managed relation database, Amazon RDS read replicas and Amazon elasticache without going through database administration hassle. Even, you can achieve horizontal scaling with Amazon Aurora Serverless managed database service.

For your analytics data, you can create serverless data lake in Amazon S3 and run an ad-hoc query using Amazon Athena by leveraging schema on read feature. You can process your data without writing complex ETL code using AWS Glue and create a centralized data catalog for all your datastore. Your Data engineers and Data scientists can avail data directly from Amazon S3 store for different machine learning user cases using Amazon SageMaker or by running Spark ML in Amazon EMR. You can use petabyte scale Amazon Redshift data warehousing solution which is highly optimized to run analytics workload.

We will discuss more details as move forward in this post, but being data in the cloud like AWS open ample opportunities for your data discovery and utilization by different means of modern technologies.

Data Discovery to move your workload from Exadata to AWS

To create a migration plan, you need to discover current data situation and design flexibility. You need to define success criteria with the key performance indicators while moving your data to AWS, for example, save cost by XX% and match or beat the current performance metrics observed in the Exadata on-premise platform. You need to collect answers to following discovery questions to plan a robust design :

What is current on-premise Exadata server configuration?

Exadata version with quarter rack, half rack or full rack
How many database and storage server in use and what is excess capacity?
How many environments are there like production, test, and development?
What is disaster recovery plan?

What is the type of data whether it is transaction data (OLTP) or analytics data (OLAP) or both?
How much data need to move for each category OLAP and OLTP.
If your data is Hot (Online and should always available), Warm (Online but less frequent used) or Cold (rarely used and can be archived)
What is Current network latency and performance? What is the % of improvement you are looking for after moving to AWS? For Example – A large report execution completed in 20 minutes, which must be achieved to define migration success criteria.
If your organization is flexible to make some changes in application and performance tune their query while moving to AWS?

Exadata to AWS Services options:

You can consider following AWS services based on amount and type of data. You can dive deep into the suggested solution once choice being made based on amount and nature of data.

OLTP Data Set:

OLTP (On-line Transaction Processing) is involved in the operation of a particular system. OLTP is characterized by a large number of short online transactions (INSERT, UPDATE, DELETE). The main emphasis for OLTP systems is put on very fast query processing, maintaining data integrity in multi-access environments and an effectiveness measured by the number of transactions per second. OLTP data primarily accessed by a software application and hosted in the database where both read and write transaction should be recorded.

Followings are the options to consider in AWS to migrate OLTP data:

· Aurora RDS: Aurora is a high performance and cost-effective database solution created by Amazon. Aurora provides multi-AZ hosting for DR, 15 read replicas which can auto scale for reading load scaling along with quick vertical scaling option for write load. Aurora serverless provide horizontal scaling where the customer can choose an option to pay as you go based on there fluctuating workload. RDS provides auto and manual snapshot option which stored in S3 with very high durability and availability. RDS can help to achieve near-zero RTO and RPO.

· Oracle on RDS: Oracle on RDS provide multi-AZ hosting for DR, 5 read replica and quick vertical scaling option for write load. You can bring your own oracle license or take oracle license from AWS. RDS provides auto and manual snapshot option which stored in S3 with very high durability and availability. RDS can help to achieve near-zero RTO and RPO.

· Oracle Hosted on EC2: if you want to have more granular control to OS for your Database then AWS provides set of the EC2 instance which can be used to install Oracle software and use it same as the on-premise environment. The customer has the flexibility to choose from the vast variety of available EC2 instance. One should evaluate Oracle licensing cost before choosing this option as Oracle has the different licensing model for EC2 instances. Find more details in Whitepaper.

OLAP Data Set:

OLAP (On-line Analytical Processing) deals with the Historical and large volume of data. Queries are often very complex and involve aggregations. For OLAP systems a response time is an effectiveness measure. OLAP applications are widely used by Data Mining techniques. In OLAP database there is aggregated, historical data, stored in multi-dimensional schemas (usually star schema). OLAP data primarily accessed by reporting tools and hosted in data warehouse, where only read access required to execute the complex query.

Followings are the options to consider in AWS to migrate OLAP data:

· Amazon Athena: You can load your data in S3 and use Amazon Athena to query that data using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run. Consider this option if you have limited number of queries to run per day. It is natively integrated with Amazon Quicksight for data visualization.

Athena can be very cost effective after applying proper data partition and data compression as it will reduce the per-query data scan significantly, which will save cost and increase performance significantly. Amazon Athena is Apache Hive metastore compatible and integrated with AWS Glue Data Catalog. You can use Amazon Glue a fully managed ETL service to convert data in the compressed columnar format like Parque with the click of a button without writing any ETL code.

· Amazon Redshift: Amazon Redshift is a fast, fully managed data warehouse which is cost-effective to analyze all your data using standard SQL and your existing Business Intelligence (BI) tools. It allows you to run complex analytic queries against petabytes of structured data, using sophisticated query optimization, columnar storage on high-performance local disks, and massively parallel query execution.

Amazon Redshift also includes Redshift Spectrum, allowing you to directly run SQL queries against exabytes of unstructured data in Amazon S3. No loading or transformation is required, and you can use most of the open data formats.

Redshift is again natively integrated with Amazon quick sight for data visualization.

Exadata to AWS Migration Approach :

To make your workload migration ready, consider following five dimensions :

1. Create Proof Of Concept in AWS and Define baseline –To start, you should identify your data which you can bring in AWS and run a proof of concept to build performance baseline.

2. Tune your Database - It may be the case your database becomes very large and not optimized . as the team may get defocused from optimization when you get a large Exadata machine with excess capacity and start dumping everything there. Think if you have multiple large schemas living in the same database and they can be divided. You can also cold data which is sitting idle in the database and can be archived to save extra cost and increase performance.

3. Go native – Think Cloud native approach and if your database can be converted to managed native AWS database like Amazon Aurora, where you can leverage high performance in very low cost compared to licensed software like Oracle.

4. Tune your infrastructure- Tune your infrastructures like memory, core, and storage to meet or beat your performance. Every application has a different need for performance and you need to find the right fit as per your use cases by making adjustments in your infrastructure setting.

5. Tune your Application - Think about tuning your application by pointing all read transactions to multiple read replicas and add caching layer for complex calculations, which will lead to lessen the burden on master database node and improve overall application performance.

On-premise Oracle Exadata to Disaster Recovery in AWS:

You may also start with creating a disaster recovery plan in AWS while your data living in on-premise Oracle Exadata rack.

For multi-site strategy, You can do replication of database changes from the on-premises setup to Amazon RDS using Oracle Streams.

Oracle Streams replicates information from one Oracle database to another and can capture both data definition language (DDL) and data manipulation language (DML) changes.

For backup and restore strategy, you can use AWS in combination with the data replication mechanism Oracle Data Guard to provide cloud-based infrastructure for disaster recovery processes. You can store DB snapshot in AWS by setting up an Amazon EC2 instance in the AWS region with Amazon EBS Volume and stored EBS snapshots in S3 bucket.

Post Migration High Available deployment Model in AWS:

For enhanced availability and durability for Amazon RDS DB instances, you can do Multi-AZ deployments model. You can deploy your application servers and the database on Amazon RDS across multiple Availability Zones (AZs).

When you provision a Multi-AZ RDS Database instance, Amazon RDS automatically creates a primary DB instance and synchronously replicates the data to a standby instance in a different AZ. In case of an infrastructure failure, Amazon RDS performs an automatic failover to the standby instance, which becomes the new primary. Because the endpoint for your DB instance remains the same after a failover, the application servers can connect to the new primary instance and resume operations without any data loss.

Making your data future proof by migrating to AWS:

After moving your data to AWS, You open up the door to make it future proof for more advanced use cases like Machine Learning and Data Science.

Data lakes in the cloud area key driver of digital transformation initiatives. They enable data and operational agility by enabling access to historical and real-time data for analytics.

You can design your data lake in the layerd model and make it more organized as shown below:

You can ingest the data using service like AWS Data Migration Service (DMS) and design a layered Data Lake architecture. Use AWS Glue to transform your data and load your data in Amazon Redshift for Analytics need and Amazon EMR to perform machine learning like recommendation and predictive analysis.

You can think about the following layers in your enterprise data lake :

Tier 1 Data Lake (Raw Data):

In Tier 1 layer , Data got ingested from various source and landed into Amazon S3 DataLake . Followings are the properties of raw data lake.

· Single Source of Truth for Raw Data

· Use Least Transformations

· Use Lifecycle policies to S3-IA or Glacier

· Use Athena for the ad-hoc query in Raw data.

Tier 2 Data Lake (Proceesed Data):

In Tier 2 layer , You consumed data from Raw data lake to processe it using ETL tool like AWS Glue and store in processed Amazon S3 Data Lake . Followings are the properties of processed data lake.

· Use columnar formats – Parquet/ORC

· Organized into Partitions

· Coalescing to Larger Partitions over time

· Optimized for Analytics

Tier 3 Data Lake (Organized Data):

In Tier 3 layer , You consumed data from processed data lake and organized it for different prupouse and business use cases . Followings are the properties of processed data lake.

· Domain Level DataMart

· Organized by use cases

· Optimized for Specialized Analysis

Above layers make your data clean and ready to use for analytics and machine learning need.

Customer Reference:

Australia Finance Group (AFG) is one of the largest mortgage aggregators in Australia moved their Oracle ExaData workload to Amazon RDS. Please refer below blog to get details on their journey to Cloud Database:

https://aws.amazon.com/blogs/database/australia-finance-groups-journey-to-cloud-databases/

Founded in 1997, Trainline is a website and app dedicated to making train travel easier, cheaper, and smarter. Trainline move their large Exadata workload to AWS . Find more details in below case study link :

https://aws.amazon.com/solutions/case-studies/trainline/

Conclusion:

You can see AWS has a large array of services to fulfill all enterprise data need. AWS scalability, security, compliance and high availability model making it a natural fit for production database workloads. There are many large enterprise customers leveraging migrated their workload from Oracle Exadata to AWS and able to achieve desired performance with low cost. Customers are able to get more agile and innovative with their data by leveraging AWS cutting-edge technologies to get the immense business benefit. Team get more agile for experiment with data and utilize the full capacity of collected data by increasing data discoverability.

Study Based Exam Notes For AWS Certified Big Data – Specialty

2018-06-10T21:58:00.003-04:00

My notes based on the Whizlab test. You can purchase test here with nominal price and 240 questions.

Read my study plan before going through notes.

EMR:

EMR security :

Data at rest

Data residing on Amazon S3—S3 client-side encryption with EMR
Data residing on disk—the Amazon EC2 instance store volumes (except boot volumes) and the attached Amazon EBS volumes of cluster instances are encrypted using Linux Unified Key System (LUKS)

Data in transit

Data in transit from E,MR to S3, or vice versa—S3 client-side encryption with EMR
Data in transit between nodes in a cluster—in-transit encryption via Secure Sockets Layer (SSL) for MapReduce and Simple Authentication and Security Layer (SASL) for Spark shuffle encryption
Data being spilled to disk or cached during a shuffle phase—Spark shuffle encryption or LUKS encryption

https://aws.amazon.com/blogs/big-data/secure-amazon-emr-with-encryption/

EMR Consistent View:EMRFS consistent view is an optional feature available when using Amazon EMR release version 3.2.1 or later. Consistent view allows EMR clusters to check for list and read-after-write consistency for Amazon S3 objects written by or synced with EMRFS. When you create a cluster with consistent view enabled, Amazon EMR uses an Amazon DynamoDB database to store object metadata and track consistency with Amazon S3. If consistent view determines that Amazon S3 is inconsistent during a file system operation, it retries that operation according to rules that you can define. By default, the DynamoDB database has 500 read capacity and 100 write capacity. You can configure read/write capacity settings depending on the number of objects that EMRFS tracks and the number of nodes concurrently using the metadata.
Use Apache Zeppelin as a notebook for interactive data exploration.Apache Zeppelin is an open source GUI which creates interactive and collaborative notebooks for data exploration using Spark. You can use Scala, Python, SQL (using Spark SQL), or HiveQL to manipulate data and quickly visualize results.
The Ganglia open source project is a scalable, distributed system designed to monitor clusters and grids while minimizing the impact on their performance. When you enable Ganglia on your cluster, you can generate reports and view the performance of the cluster as a whole, as well as inspect the performance of individual node instances. Ganglia is also configured to ingest and visualize Hadoop and Spark metrics.
Apache Cassandra and Apache HBase are columnar Database. Compare to Dynamo DB, Apache HBase is much more flexible in terms of what you can store (size and data type wise). Apache HBase gives you the option to have very flexible row key data types, whereas DynamoDB only allows scalar types for the primary key attributes. DynamoDB, on the other hand, provides very easy creation and maintenance of secondary indexes, something that you have to do manually in Apache HBase.
In EMR, The default input format for a cluster is text files with each line separated by a newline (\n) character, which is the input format most commonly used. If your input data is in a format other than the default text files, you can use the Hadoop interface toInputFormat specify other input types.
If you are using Hive, you can use a serializer/deserializer (SerDe) to read data in from a given format into HDFS.

Hue (Hadoop User Experience) is an open-source, web-based, graphical user interface for use with Amazon EMR and Apache Hadoop. Hue groups together several different Hadoop ecosystem projects into a configurable interface. Hue acts as a front-end for applications that run on your cluster, allowing you to interact with applications using an interface that may be more familiar or user-friendly. The applications in Hue, such as the Hive and Pig editors, replace the need to log in to the cluster to run scripts interactively using each application's respective shell. After a cluster launches, you might interact entirely with applications using Hue or a similar interface.
Amazon EMR supports Apache Mahout, a machine learning framework for Apache Hadoop. Mahout is a machine learning library with tools for clustering, classification, and several types of recommenders, including tools to calculate most-similar items or build item recommendations for users. Mahout employs the Hadoop framework to distribute calculations across a cluster, and now includes additional work distribution methods, including Spark.

You can run your EMR cluster as a transient process: one that launches the cluster, loads the input data, processes the data, stores the output results, and then automatically shuts down. This is the standard model for a cluster that performs a periodic processing task. Shutting down the cluster automatically ensures that you are only billed for the time required to process your data.

Spark has micro-batching but can guarantee only-once-delivery if configured. Spark’s four modules are MLlib, SparkSQL, Spark Streaming, and GraphX.

When your cluster runs, Hadoop creates a number of map and reduce tasks. These determine the number of tasks that can run simultaneously during your cluster. Run too few tasks and you have nodes sitting idle; run too many and there is significant framework overhead.
If you want that data to be encrypted in-transit between nodes, then Hadoop encrypted shuffle has to be setup. Encrypted Shuffle capability allows encryption of the MapReduce shuffle using HTTPS. When you select the in-transit encryption checkbox in the EMR security configuration, Hadoop Encrypted Shuffle is automatically setup for you upon cluster launch

Do not use Spark for batch processing. With Spark, there is minimal disk I/O, and the data being queried from the multiple data stores needs to fit into memory. Queries that require a lot of memory can fail. For large batch jobs, consider Hive. Also avoid using Spark for multi-user reporting with many concurrent requests.

RedShift :

When you create a table, you designate one of three distribution styles; EVEN, KEY, or ALL.

Even distribution

The leader node distributes the rows across the slices in a round-robin fashion, regardless of the values in any particular column. EVEN distribution is appropriate when a table does not participate in joins or when there is not a clear choice between KEY distribution and ALL distribution. EVEN distribution is the default distribution style.

Key distribution : The rows are distributed according to the values in one column. The leader node will attempt to place matching values on the same node slice. If you distribute a pair of tables on the joining keys, the leader node collocates the rows on the slices according to the values in the joining columns so that matching values from the common columns are physically stored together.

ALL distribution : A copy of the entire table is distributed to every node. Where EVEN distribution or KEY distribution place only a portion of a table's rows on each node, ALL distribution ensures that every row is collocated for every join that the table participates in.

ALL distribution multiplies the storage required by the number of nodes in the cluster, and so it takes much longer to load, update, or insert data into multiple tables. ALL distribution is appropriate only for relatively slow-moving tables; that is, tables that are not updated frequently or extensively. Small dimension tables do not benefit significantly from ALL distribution, because the cost of redistribution is low.

RedShift UNLOAD: Unloads the result of a query to one or more files on S3, using Amazon S3 server-side encryption (SSE-S3). You can also specify server-side encryption with an AWS Key Management Service key (SSE-KMS) or client-side encryption with a customer-managed key (CSE-CMK).You can manage the size of files on Amazon S3, and, by extension, the number of files, by setting the MAXFILESIZE parameter.

Number of the slice in Redshift Node

o ds2.xlarge , dc1.large, dc2.large = 2

o ds2.8xlarge, dc2.8xlarge = 16

o dc1.8xlarge = 32

For Redshift COPY command, Split your load data files so that the files are about equal size, between 1 MB and 1 GB after compression. The number of files should be a multiple of the number of slices in your cluster.

To validate the data in the Amazon S3 input files or Amazon DynamoDB table before you actually load the data in Redshift, use the NOLOAD option with the COPY command. Use NOLOAD with the same COPY commands you use to actually load the data. NOLOAD checks the integrity of all of the data without loading it into the database. The NOLOAD option displays any errors ins same manner as that would occur if you had attempted to load the data.

There are two types of snapshots: automated and manual. Amazon Redshift stores these snapshots internally in Amazon S3 by using an encrypted Secure Sockets Layer (SSL) connection. If you need to restore from a snapshot, Amazon Redshift creates a new cluster and imports data from the snapshot that you specify. Read more - https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-snapshots.html

You can use a manifest to ensure that the COPY command loads all of the required files, and only the required files, for a data load. Instead of supplying an object path for the COPY command, you supply the name of a JSON-formatted text file that explicitly lists the files to be loaded. The URL in the manifest must specify the bucket name and full object path for the file, not just a prefix. You can use a manifest to load files from different buckets or files that do not share the same prefix.

RedshiftCopyActivity: Copies data from DynamoDB or Amazon S3 to Amazon Redshift. You can load data into a new table, or easily merge data into an existing table. In addition, RedshiftCopyActivity let's you work with an S3DataNode, since it supports a manifest file.

In Redshift, You can efficiently add new data to an existing table by using a combination of updates and inserts from a staging table. While Amazon Redshift does not support a single merge, or upsert, command to update a table from a single data source, you can perform a merge operation by creating a staging table

Redshift cluster resizing - https://docs.aws.amazon.com/redshift/latest/mgmt/rs-resize-tutorial.html

EXPLAIN command Displays the execution plan for a query statement without running the query in Redshift

By default, Amazon Redshift configures one queue with a concurrency level of five, which enables up to five queries to run concurrently, plus one predefined Superuser queue, with a concurrency level of one. You can define up to eight queues. Each queue can be configured with a maximum concurrency level of 50. The maximum total concurrency level for all user-defined queues (not including the Superuser queue) is 50.

In Redshift , If you enable SQA, you can reduce or eliminate workload management (WLM) queues that are dedicated to running short queries. In addition, long-running queries don't need to contend with short queries for slots in a queue, so you can configure your WLM queues to use fewer query slots. When you use lower concurrency, query throughput is increased and overall system performance is improved for most workloads.

In RedShift , Short query acceleration (SQA) prioritizes selected short-running queries ahead of longer-running queries. SQA executes short-running queries in a dedicated space, so that SQA queries aren't forced to wait in queues behind longer queries. With SQA, short-running queries begin running more quickly and users see results sooner.

In Redshift The CASE expression is a conditional expression, similar to if/then/else statements found in other languages. CASE is used to specify a result when there are multiple conditions.

In Redshift , You enable encryption when you launch a cluster. To migrate from an unencrypted cluster to an encrypted cluster, you first unload your data from the existing, source cluster. Then you reload the data in a new, target cluster with the chosen encryption setting. During the migration process, your source cluster is available for read-only queries until the last step. The last step is to rename the target and source clusters, which switches endpoints so all traffic is routed to the new, target cluster

Amazon Redshift logs information about connections and user activities in your database. These logs help you to monitor the database for security and troubleshooting purposes, for database auditing. The logs are stored in the S3 buckets for convenient access with data security features for users who are responsible for monitoring activities in the database.

S3, DynamoDB, and EMR/EC2 instances directly integrate with Redshift using the COPY command.

The UNLOAD ENCRYPTED command automatically stores the data encrypted using client side encryption and uses HTTPS to encrypt the data during the transfer to S3.If you want to ensure files are automatically encrypted on S3 with server-side encryption, no special action is needed. The unload command automatically creates files using Amazon S3 server-side encryption with AWS-managed encryption keys (SSE-S3).
For RedShift data , bzip2 compression algorithm has the highest compression ratio.

To manually migrate an Amazon Redshift cluster to another AWS account, follow these steps:

Create a manual snapshot of the cluster you want to migrate.
Share the cluster snapshot with another AWS account to view and restore the snapshot.
Before you copy a snapshot to another region, first enable cross-region snapshots.
In the destination AWS account, restore the shared cluster snapshot.

Kinesis:

The unit of data stored by Kinesis Data Streams is a data record. A stream represents a group of data records. The data records in a stream are distributed into shards.

A shard has a sequence of data records in a stream. When you create a stream, you specify the number of shards for the stream. Each shard can support up to 5 transactions per second for reads, up to a maximum total data read rate of 2 MB per second. Shards also support up to 1,000 records per second for writes, up to a maximum total data write rate of 1 MB per second (including partition keys). The total capacity of a stream is the sum of the capacities of its shards. You can increase or decrease the number of shards in a stream as needed. However, you are charged on a per-shard basis.

If you have sensitive data, you can enable server-side data encryption when you use Amazon Kinesis Data Firehose. However, this is only possible if you use a Kinesis stream as your data source. When you configure a Kinesis stream as the data source of a Kinesis Data Firehose delivery stream, Kinesis Data Firehose no longer stores the data at rest. Instead, the data is stored in the Kinesis stream.

When you send data from your data producers to your Kinesis stream, the Kinesis Data Streams service encrypts your data using an AWS KMS key before storing it at rest. When your Kinesis Data Firehose delivery stream reads the data from your Kinesis stream, the Kinesis Data Streams service first decrypts the data and then sends it to Kinesis Data Firehose. Kinesis Data Firehose buffers the data in memory based on the buffering hints that you specify and then delivers it to your destinations without storing the unencrypted data at rest.

In Kinesis , To prevent skipped records, handle all exceptions within processRecords appropriately.

For each Amazon Kinesis Data Streams application, the KCL uses a unique Amazon DynamoDB table to keep track of the application's state. Because the KCL uses the name of the Amazon Kinesis Data Streams application to create the name of the table, each application name must be unique.

If your Amazon Kinesis Data Streams application receives provisioned-throughput exceptions, you should increase the provisioned throughput for the DynamoDB table. The KCL creates the table with a provisioned throughput of 10 reads per second and 10 writes per second, but this might not be sufficient for your application. For example, if your Amazon Kinesis Data Streams application does frequent checkpointing or operates on a stream that is composed of many shards, you might need more throughput.

PutRecord returns the shard ID of where the data record was placed and the sequence number that was assigned to the data record. Sequence numbers increase over time and are specific to a shard within a stream, not across all shards within a stream. To guarantee strictly increasing ordering, write serially to a shard and use the SequenceNumberForOrdering parameter.

Details regarding Kinesis records :

For live streaming Kinesis gets ruled out if record size greater than 1 MB , in that case Kafka can support bigger records.

You can trigger One lambda per shard. If you want to use Lambda with Kinesis Streams, you need to create Lambda functions to automatically read batches of records off your Amazon Kinesis stream and process them if records are detected on the stream. AWS Lambda then polls the stream periodically (once per second) for new records.

In Kinesis stream ,the PutRecordBatch() operation can take up to 500 records per call or 4 MB per call, whichever is smaller. Buffer size ranges from 1 MB to 128 MB.

In circumstances where data delivery to the destination is falling behind data ingestion into the delivery stream, Amazon Kinesis Firehose raises the buffer size automatically to catch up and make sure that all data is delivered to the destination.

If data delivery to Redshift fail from Kinesis Firehose , Amazon Kinesis Firehose retries data delivery every 5 minutes for up to a maximum period of 60 minutes. After 60 minutes, Amazon Kinesis Firehose skips the current batch of S3 objects that are ready for COPY and moves on to the next batch. The information about the skipped objects is delivered to your S3 bucket as a manifest file in the errors folder, which you can use for manual backfill. For information about how to COPY data manually with manifest files, see Using a Manifest to Specify Data Files.

If data delivery to your Amazon S3 bucket fails , Amazon Kinesis Firehose retries to deliver data every 5 seconds for up to a maximum period of 24 hours. If the issue continues beyond the 24-hour maximum retention period, it discards the data.

Aggregation refers to the storage of multiple records in a Streams record. Aggregation allows customers to increase the number of records sent per API call, which effectively increases producer throughput. Aggregation Storing multiple records within a single Kinesis Data Streams record while Collection using the API operation PutRecords to send multiple Kinesis Data Streams records to one or more shards in your Kinesis data stream.You can first aggregate stream record and then send them to stream using collection putrecords() in multiple shard.

Spark Streaming uses the Kinesis Client Library (KCL) to consume data from a Kinesis stream. KCL handles complex tasks like load balancing, failure recovery, and check-pointing

The Amazon Kinesis Connector Library helps Java developers integrate Amazon Kinesis with other AWS and non-AWS services. The current version of the library provides connectors for Amazon DynamoDB, Amazon Redshift, Amazon S3, Elasticsearch.

Amazon ML:

Amazon ML Key component :

o Datasources contain metadata associated with data inputs to Amazon ML

o ML Models generate predictions using the patterns extracted from the input data

o Evaluations measure the quality of ML models

o Batch Predictions asynchronously generate predictions for multiple input data observations

o Real-time Predictions synchronously generate predictions for individual data observations

Amazon ML supports three types of ML models: binary classification, multiclass (Categorial) classification, and regression. - https://docs.aws.amazon.com/machine-learning/latest/dg/types-of-ml-models.html

Details on Amazon ML - https://docs.aws.amazon.com/machine-learning/latest/dg/amazon-machine-learning-key-concepts.html

AUC	Area Under the Curve (AUC) measures the ability of a binary ML model to predict a higher score for positive examples as compared to negative examples.
Macro-averaged F1-score	The macro-averaged F1-score is used to evaluate the predictive performance of multiclass ML models.
RMSE	The Root Mean Square Error (RMSE) is a metric used to evaluate the predictive performance of regression ML models.

A lower Area Under Curve (AUC) reduces accuracy of the prediction; AUC values well below 0.5 may indicate a problem with the data. The F1 score's range is 0 to 1. A larger value indicates better predictive accuracy.
Amazon ML is limited to 100 'categorical' recommendations . Amazon ML can support up to 100 GB of data.

You must upload your input data to S3 because Amazon ML reads data from Amazon S3 locations. You can upload your data directly to Amazon S3, or Amazon ML can copy data that you've stored in Redshift or RDS into a .csv file and upload it to S3.

ElasticSearch :

ElasticSearch is suitable to analyze large set of streaming data from Kinesis.

Elastic Search Snapshots are backups of a cluster's data and state. They provide a convenient way to migrate data across Amazon Elasticsearch Service domains and recover from failure. The service supports restoring from snapshots taken on both Amazon ES domains and self-managed Elasticsearch clusters.

Amazon ES takes daily automated snapshots of the primary index shards in a domain, as described in Configuring Automatic Snapshots. The service stores up to 14 of these snapshots for no more than 30 days in a preconfigured Amazon S3 bucket at no additional charge to you. You can use these snapshots to restore the domain.

IoT and Others:

For AWS IoT - Devices connect using your choice of identity (X.509 certificates, IAM users and groups, Amazon Cognito identities, or custom authentication tokens) over a secure connection according to the AWS IoT connection model. Note that KMS is not in list.

AWS IoT rule actions specify what to do when a rule is triggered. You can create actions for the following services: CloudWatch, DynamoDB, Elasticsearch, Kinesis Firehose, Kinesis Streams, Kinesis Firehose, Lambda, S3, SNS, and SQS. Not for any relational store like Aurora and Redshift.

The Rules Engine transforms messages using a SQL-based syntax. The Device Gateway (Message Broker) uses topics to route messages from publishing clients to subscribing clients. Then a message is published to the topic, the SQL statement is evaluated and rule action is triggered, sending the message to another AWS service.

In AWS Data Pipeline, a precondition is a pipeline component containing conditional statements that must be true before an activity can run. For example, a precondition can check whether source data is present before a pipeline activity attempts to copy it. AWS Data Pipeline provides several pre-packaged preconditions that accommodate common scenarios, such as whether a database table exists, whether an Amazon S3 key is present, and so on. However, preconditions are extensible and allow you to run your own custom scripts to support endless combinations.

AWS Data Pipeline supports A JDBC database , A RDS Database and Redshift . computational resource that perform the work on EC2 and EMR cluster.

Data Pipeline integrate with on-premise servers. AWS provides you with a Task Runner package that you install on your on-premise hosts. Once installed, the package polls Data Pipeline for work to perform. If it detects that an activity needs to run on your on-premise host (based on the schedule in Data Pipeline), the Task Runner will issue the appropriate command to run the activity, which can be running a stored procedure or a database dump or another database activity.

A QuickSight Dashboard is a read-only snapshot of an analysis. You can share it with other Quicksight users for reporting. The data in the dashboard reflects the data set that is used by the analysis. If you share a dashboard with users, they can then view and filter the dashboard data, but they cannot save any filters applied to the dashboard.

Dynamo DB:

Number of Dynamo DB partition = readCapacityUnit/3000 + WriteCapacityUnit/1000. 10 GB data per partition after that it create new partition.

Details on GSI Vs LSI in Dynamo DB - https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SecondaryIndexes.html

COPY command used for Loading Data from DynamoDB Into Amazon Redshift.
DAX (DynamoDB Accelerator)is a DynamoDB-compatible caching service that enables you to benefit from fast in-memory performance for demanding application.
A VPC endpoint for DynamoDB and S3 enables Amazon EC2 instances in your VPC to use their private IP addresses to access DynamoDB/S3 with no exposure to the public Internet.
DynamoDB Streams supports the following stream record views:
- KEYS_ONLY—Only the key attributes of the modified item
- NEW_IMAGE—The entire item, as it appears after it was modified
- OLD_IMAGE—The entire item, as it appears before it was modified
- NEW_AND_OLD_IMAGES—Both the new and the old images of the item
DynamoDB supports many different data types for attributes within a table. They can be categorized as follows:

o   Scalar Types – A scalar type can represent exactly one value. The scalar types are number, string, binary, Boolean, and null.

o   Document Types – A document type can represent a complex structure with nested attributes—such as you would find in a JSON document. The document types are list and map.

o   Set Types – A set type can represent multiple scalar values. The set types are string set, number set, and binary set.

Book:

For any AWS certification , You need to have solid understanding of architecture and how to apply various practice to optimize the solution in real time. I found Solution Architect's Handbook pretty useful . This is 490 pages of book covering all aspects of solution architecture and help you to not only clear the certification but also crack the interview. I will say price is pretty reasonable given that details content . You can order from amazon - https://www.amazon.com/gp/product/1838645640

Study Plan For AWS Certified Big Data – Specialty

2018-01-18T18:16:00.000-05:00

After achieving my 2017 resolution by becoming AWS all 5/5 certified. I have setup new challenges for me in 2018 to become big data and AI/ML, expert.

Things are on right track, Now I am working as Big Data SME for AWS customers and passed my AWS Certified Big Data Specialty exam on Jun-2018. My Machine Learning University course started with very deep insight on Machine learning, which will be expected to continue for a year with rigorous learning in Artificial Intelligence and Machine Learning area.

Here is my lesson learn for AWS Certified Big Data – Specialty exam:

Bar is very high as most of the question were scenario based and long format . I found questions related to Kinesis and Dynamo DB pretty complex . For Kinesis , you will get very confusing options between Kinesis stream and Kinesis firehose , so make sure to dive deep on that to know clear difference between them with use cases. For Dynamo DB , make sure to get very deep understanding on best practice to design LSI/GSI and table partition . EMR and Hadoop questions has medium complexity and they are not only limited to AWS, but addressing other Hadoop eco-system open source technologies as well. Machine Learning and Redshift questions were relatively easy , so that is scoring point. Only one question from QuickSight. All security-related questions were very tricky and storage use cases were relatively easy .

I didn't leverage that, but you can get extra 30 minute time by using following steps , specially if English is not your native language:

From the top navigation, click Upcoming Exams
On the right, click the Request Exam Accommodations button
Click the Request Accommodation button
Select ESL +30 Minutes from the accommodation dropdown
Click Create

To know more about certification , Visit AWS Bigdata Certification webpage.

To start with the First step

Book:

For any AWS certification , You need to have solid understanding of architecture and how to apply various practice to optimize the solution in real time. I found Solution Architect's Handbook pretty useful . This is 490 pages of book covering all aspects of solution architecture and help you to not only clear the certification but also crack the interview. I will say price is pretty reasonable given that details content . You can order from amazon - https://www.amazon.com/gp/product/1838645640

Video Course: Recommend must go through Cloud Guru Videos first and then Linux Academy.

ACloudGuru BigData Specialty Course - aCloudGuru has prepare very detail course material by covering all Hadoop and big data technology in addition to Redshift and Dynamo DB.

Linux Academy BigData Specialty Course - For any pro certification , I can not take chance so going to take complete course from Linux Academy , as they have good amount of labs and lots of exam notes with study material which save lots of time. I was expecting more details on Hadoop eco-system but it is not there in this course. Any case, I love their ready to use exam notes , labs, flash card and mobile app for offline reading so many benefit comes along. You can subscribe to Linux Academy by clicking here .

Go through this YouTube play list for AWS re-invent videos.

Exam Notes:

Demo Test:

Whizlab test helped a lot to prepare well on many topics. You can purchase test here with nominal price and 240 questions.
Linux Academy Demo test is worth taking a look which comes with 100 questions.

Lab:

Labs are must to pass this exam. Make sure to go through all Linux Academy labs and follow along labs for all acloud guru practical videos.

Whitepaper:

Blogs and AWS UserDocs:

You may not need to read all of them but it is a great source to clarify concept in depth:

What Exam Covers:

Kinesis (Streams, Firehose, Analytics, data ingestion and export to other AWS services, tuning)
DynamoDB (Performance tuning, partitioning, use patterns and anti-patterns, indexing)
S3 (Patterns and anti-patterns, IA/Glacier and lifecycle, partitioning)
Elastic MapReduce (Products used in conjunction and what they do – Spark, Hadoop, Zeppelin, Sqoop, Pig, Hive, etc.)
QuickSight (Use patterns/anti-patterns, chart types)
Redshift (Data ingestion, data export, slicing design, indexing, schema types)
Instance types (compute intensive, smaller nodes of large instances vs larger nodes of smaller instances)
Compression (performance, compression sizes)
Machine Learning (machine learning model types and when you’d use them)
IoT (understand the basics of AWS IoT architecture)
What services are multi-AZ and/or multi-region and how to work around geographic constraints
Data Import/Export (when to use, options)
Security (IAM, KMS, HSM, CloudTrail)
CloudWatch (log files, metrics, etc.)
Other Topics to cover:
- Kinesis Firehose and Kinesis Stream: Kinesis firehose and Kinesis stream are big data terms associated with streaming real-time data to destinations such as S3, ES, and spunk. You would require to know the importance of Kinesis and combining it with Lambda. Also, learn where the Firehose will unload the content. Also, understand why kinesis ingestion is better. Learn the features offered by KCL and KPL. Also, understand when to use them to send/receive data.
- S3: Amazon S3 is storage for the Internet. It is a simple web service interface which is used to store and retrieve data anywhere anytime. You would require to know how to use S3 buckets when to use S3 standards and S3 glacier.
- DynamoDB: DymanoDB is a NoSQL database services for all application that need consistent, single-digit millisecond latency at any scale. Study the right use case for DymanoDB. How to design LSI and GSI. You might get asked about key optimization.
- Quick sight: It is a fast cloud-powered business analytics service. You may be asked a question on virtual representation with Quicksight.
- Redshift: AWS redshift is a fast, fully managed data warehouse which helps you analyze your data using SQL and BI. You will need to know how to load data using Redshift. You will need to understand the overall architecture. Leader, node slices, compute node. Redshift is an important topic and you would require to know the importance of the COPY command. Also to migrate data using UNLOAD command. Tuning performance use right sort key. At least one question on loading data in Redshift is asked. For this you should know what slices are and instances.
- IoT: The questions will be asked mixed with topics like as DynamoDB and Kinesis Stream. You will require knowledge of IoT rules, Device Registry, Device Gateway, Device Shadow, etc.
- Machine Learning: Knowledge of which types of models are by AWS Machine learning. Also, understand when to select the right model for specific business requirements and scenarios.
- Security: IAM roles, data encryption for S3, Redshift and DynamoDB, HSM.
- Data Pipeline: Some questions will be asked on Data pipeline, Data Pipeline, and Lambda and when to us it. Questions on backup, restore data into other AWS regions.
Topics on which least questions are asked:
- Limited questions on networking.
- Limited questions on ELB and auto scaling groups.
- No CloudFormation, OpsWork questions.
- No Beanstalk questions.
- Limited questions related to SQS and SNS.

Simple VPC Creation with CloudFormation - YAML Made it easy

2017-11-19T23:48:00.001-05:00

In my previous post, I wrote all cloud formation example in json but I am now a big fan of yaml, as it provides more concise cloud formation code write-up compare to json. Specially, I am happy to get rid of the long hierarchical list of curly braces, where keeping track of every bracket become a headache especially if you are using the nested intrinsic function like Fn:: Select. However, sometimes troubleshooting in yaml become tricky as you need to follow proper space, indentation, and tab in order to get things to work.

In this example, I am going to create a complete VPC with two subnets public and private in different availability zone and spin up EC2 instance inside each zone. Of course, all of this in cloud formation yaml template.

The final architecture will look something like below:

Please download my yaml cloud formation template from below Github link :
https://github.com/itsaurabh1/aws/blob/master/VPCCreation.yaml

I will recommend creating complete VPC from the console for a couple of time before start writing cloud formation template. It will give you a very clear idea, what all elements required in order for a complete VPC setup to launch an instance in public and private subnet.

I will explain each section of the template below:

#So You will notice no curly braces and here we define template version and description, both are optional fields.

AWSTemplateFormatVersion: 2010-09-09
Description : Deploy a VPC

#Resources is a mandatory section where we defined our VPC with CIDR block. Make sure to user tab in next line after "Properties:" otherwise yaml will consider this as null value for properties and you will get an unexpected error. As I mentioned you need to be careful with space and indentation when working with yaml.

Resources:
    MyVPC :
        Type: AWS::EC2::VPC
        Properties:
            CidrBlock: 10.0.0.0/16
            EnableDnsHostnames: True
            Tags:
                - Key: Name
                  Value: My VPC

#Here we are creating our IGW and attached it to VPC, you can add appropriate tags if want as I have added for VPC in above

    MyInternetGateway:
        Type: AWS::EC2::InternetGateway

    AttachedGateway:
        Type: AWS::EC2::VPCGatewayAttachment
        Properties:
            VpcId : !Ref MyVPC
            InternetGatewayId : !Ref MyInternetGateway

# In this section we are creating public and private subnet within our VPC. You can see how simple to user "select" intrinsic function in yaml compares to json.

    PublicSubnet1:
        Type: AWS::EC2::Subnet
        Properties:
            VpcId : !Ref MyVPC
            CidrBlock: 10.0.1.0/24
            AvailabilityZone: !Select
                        - '0'
                        - !GetAZs ''

    PrivateSubnet1:
        Type: AWS::EC2::Subnet
        Properties:
            VpcId: !Ref MyVPC
            CidrBlock: 10.0.0.1/28
            AvailabilityZone: !Select
                    - '1'
                    - !GetAZs ''

#Now in this section, we create route table and define a route for IGW to make it public and associated it with the public subnet. Same goes with private subnet where we have private route table but there is no IGW attached to that.

    MyPublicRouteTable:
        Type: AWS::EC2::RouteTable
        Properties:
            VpcId: !Ref MyVPC

    MyPublicRoute:
        Type: AWS::EC2::Route
        Properties:
            RouteTableId: !Ref MyPublicRouteTable
            DestinationCidrBlock: 0.0.0.0/0
            GatewayId: !Ref MyInternetGateway

    PublicSubnetRouteTableAssociation1:
        Type: AWS::EC2::SubnetRouteTableAssociation
        Properties:
            SubnetId: !Ref PublicSubnet1
            RouteTableId: !Ref MyPublicRouteTable

    MyPrivateRouteTable:
        Type: AWS::EC2::RouteTable
        Properties:
            VpcId: !Ref MyVPC

    PrivateRouteTableAssociation1:
        Type: AWS::EC2::SubnetRouteTableAssociation
        Properties:
            SubnetId: !Ref PrivateSubnet1
            RouteTableId: !Ref MyPrivateRouteTable

# In this section I spin up two EC2 instances one in each public and private subnet.

    MyPublicEC2:
        Type: AWS::EC2::Instance
        Properties:
            ImageId: ami-2e64d954
            SubnetId: !Ref PublicSubnet1

    MyPrivateEC2:
        Type: AWS::EC2::Instance
        Properties:
            ImageId: ami-2e64d954
            SubnetId: !Ref PrivateSubnet1

# This is the outputs section where we displayed attributes from   service to cloud formation output tab

Outputs:
    MyVPC:
        Description: My VPC Name
        Value: !Ref MyVPC

    PublicAZ1:
        Description: My Public AZ
        Value: !GetAtt
            - PublicSubnet1
            - AvailabilityZone

    PrivateAZ1:
        Description: My Private AZ
        Value: !GetAtt
            - PrivateSubnet1
            - AvailabilityZone

    public instance:
        Description: My public Web Server
        Value: !Ref MyPublicEC2

    PrivateInstance:
        Description: My Private DB Server
        Value: !Ref MyPrivateEC2

Please find the output tab below:

All create resources from Cloud formation are:

Here is our final result.

           You may want to visit my previous post to learn more about json template of cloud formation.

CloudFormation Primer with Simple Examples for AWS DevOps Engineer Exam

2017-11-06T00:56:00.002-05:00

As Cloud Formation is the pretty big topic and very important for DevOps Engineer Pro Exam, so I decided to prepare separate notes for this with a practical example.

Thorough out this blog , I am going to create simple Cloud formation template to demonstrate basic concept in simplest possible manner. I will run all templates in my aws account and share output and findings .

Note: I am demostrating all example using AWS console but same can be achive by command line using AWS CLI.

Parameters , Ref, and Output:

Download Cloudformation template from my GitHub repository :
https://github.com/itsaurabh1/aws/blob/master/S3StackTemplate.json

In this section we are going to learn about parameter ,Ref and Output by creating a S3 bucket with customer name which will pass through parameters and show that name in output.

{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "Create a Simple S3 bucket with parameter to choose own bucket name",

/* For any Cloudformation template , you must mention "Resources" section and version . Here resources are S3 bucket by defining Type as AWS::S3::Bucket (I find this "::" inline with C++ scope resolution operator ). You can find various properties from aws s3 document , so we will just see structure here . The most important things to note down here is "Ref" which is helping us to link customer parameter to the bucket name */

"Resources" : {
    "Bucket" : {
        "Type" : "AWS::S3::Bucket",
               "Properties" : {
                  "AccessControl" : "PublicRead",
                 "BucketName" : {"Ref" : "S3NameParam" },
                 "Tags" : [ {"Key" : "Name" , "Value" : "MyBucket"} ]
                               }
                 }
         },

/* In above section we learn about ref and how it is refering to parameter "S3NameParam" . Now, Its time to define our parameter, which will show default name as saurabh-dafaultbucket , but give you option to add your own S3 bucket name. I have also added constraint to have bucket name length between 5 to 30 character string .

"Parameters": {
    "S3NameParam" : {
         "Type": "String",
         "Default" : "saurabh-dafaultbucket",
         "Description" : "Enter the Bucket Name",
         "MinLength" : "5",
         "MaxLength" : "30"
                 }
            },

Parameter will look like below in details section during template execution:

I have added just 3 character name and follwoing error showed during review :

/* In below section , we are going to show bucket name in Output tab of Clodformation consol , For which I am using attribute name "Outputs" with Ref value as our S3 bucket parameter name "S3NameParam". By this time you are already familier with Ref so below snippt should be relatively easy

"Outputs" : {
    "BucketName" : {
        "Description" : "BucketName" ,
        "Value" : { "Ref" : "S3NameParam"}
                    }
            }

}

You can see bucket name in Output tab as shown below:

Intrinsic Functions:

In this section , we are going to learn frequently used instrinsic function by example which include Fn::FindInMap, Fn::GetAtt, Fn::Select, Fn::Join, Fn::GetAZs and Fn::Base64.

AWS has done good job for documentation , so to learn details about intrinsic function go thorugh AWS docs.

Download Cloudformation template from my GitHub repository :

https://github.com/itsaurabh1/aws/blob/master/IntrinsicFunction.json

{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "Simple Demo of Intrinsic Function",

/*I can relate this with progrmmaing launguage datatype, where first you need to define data type and in this case it is "Mappings". After that you give map name as "AMIRegionMap" and fill key-value pair. I create 3 copies of AMI in different region and provided ami-id against different region in map below*/

"Mappings" : {
    "AMIRegionMap" : {
             "us-east-1" : { "AMI" : "ami-9c68d5e6" },
             "us-west-2" : { "AMI" : "ami-962afeee" },
             "eu-west-2" : { "AMI" : "ami-66f4e802" }
                      }
              },

/*As now , we have map defined with data so now we will get ami-id using function "Fn::FindInMap" where first parameter is map name "AMIRegionMap" and second paramter is top level key which is current region and best way to get current region by using "AWS::Region" with "Ref". We already define second level key as "AMI" which will help to fetch final value. You will notice to pass Bootstrap script we need to give base64 text as userdata for which we are using
Fn::Base64*/

"Resources" : {
    "MyEC2Ins" : {
         "Type" : "AWS::EC2::Instance" ,
         "Properties" : {
         "ImageId" : {"Fn::FindInMap" : ["AMIRegionMap" , {"Ref" : "AWS::Region"} , "AMI"] },
         "InstanceType" : "t2.micro",
         "UserData" : {"Fn::Base64" : "yum upgrade -y"}
                         }
                 }
             },

/* In above snippet , I kept it simple but we can have multiple values at second level like below:

"Mappings" : {
"RegionMap" : {
      "us-east-1" : { "32" : "ami-6ce4e702", "64" : "ami-9c68d5e6" },
      "us-west-1" : { "32" : "ami-9cf4e778", "64" : "ami-962afeee" },
      "eu-west-1" : { "32" : "ami-3cf4e812", "64" : "ami-66f4e802" }

This map can be called using -

"ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "64"]},

*/

/* In this section we will learn about more use of more intrinsic functions in simplest possible way by displaying outcome in "Output" tab */

"Outputs" : {

/*Below you can see use of "Fn::GetAtt" function where it is helping to get capture value of very specific attribute of EC2 instance like private IP. You can find all supported attribute in aws doc. */

   "EC2PrivateIP" : {
               "Description" : "EC2 Instance Private IP Address",
               "Value" : {
                           "Fn::GetAtt" : ["MyEC2Ins" , "PrivateIp" ]
                           }
                       },
/*In this section , given code snippet demonstrated two intrinsic function , First "Fn::GetAZs" which help to get list of all AZs in a region . This function is very helpful for subnet create where you want cloudformation to create multiple subnet in different AZs within a region. Second function is "Fn::Select" , you can see we are first fetching list of all AZs in a region using "Fn::GetAZs" function and then selecting first AZ with index value "0" and second AZ with value index "1". You can related this with array in C/C++ */

       "MyCurrentAZ" : {
                "Description" : "My Current AZ",
                "Value" : {
                            "Fn::Select" : ["0", {"Fn::GetAZs" : {"Ref" : "AWS::Region"}}]
                            }
                        },
       "NextAZ" : {
               "Description" : "Next Available AZ" ,
               "Value" : {
                           "Fn::Select" : ["1" , {"Fn::GetAZs" : {"Ref" : "AWS::Region"}}]
                           }
               },

/*In this section we are simply joining multiple string using function "Fn::Join" where I have mentioned delimiter as colon (":") to join two string. This can be very helpful to show output by combining multiple outputs together */

       "JoinAZs" : {
                   "Description" : "Join Value of Two AZs by colon" ,
                   "Value" : {
                               "Fn::Join" : [":" , ["MyCurrentAZ","NextAZ"]]
                               }
                    }
       }
}

Console OutPut:

You can see values in console OutPuts tab below:

Learn to Create VPC Cloud formation template using YAML :

http://getyouralgorithm.blogspot.com/2017/11/simple-vpc-creation-with-cloudformation.html

Study Based Exam Notes and Study Plan For AWS Certified DevOps Engineer - Professional

2017-11-02T01:08:00.002-04:00

Finally Achieved AWS Certified DevOps Engineer - Professional Certification with 81% score on 6th-Dec-2017 with this became all 5/5 AWS certified.

As already completed my AWS Certified Solution Architect professional exam 3 months ago , So it helped me to study and re-utilize my knowledge . I did slow study everyday 2 hrs for 1.5 months and studied 7 to 8 hrs daily in last 5 days before exam. Finally , it got paid up by achieving the milestone.

Interestingly, I found DevOps Engineer professional certification has far fewer topics covered compared to Solution Architect professional but the questions were really deep.

Main topic covered in Exam are:

Cloudformation - Lots of questions around rolling up and blue/green deployment
AutoScaling - Lots of question around to address availability
Security - Many tricky questions around storing credential and using in CI pipeline.
Elastic Beanstalk - Update configuration using EB and achieve different deployment type.
OpsWork - Only 4/5 questions were related to this.
Deployment method - Multiple deployment methods and how to achieve them

There was no question from Networking, Lambda, AWS code commit, code deploy or code pipeline.

To know more about certifiication, Visit AWS DevOps Engineer Certification webpage.

I didn't leverage that, but you can get extra 30 minute time by using following steps , specially if English is not your native language:

From the top navigation, click Upcoming Exams
On the right, click the Request Exam Accommodations button
Click the Request Accommodation button
Select ESL +30 Minutes from the accommodation dropdown
Click Create

To start with the First step

Book:

For any AWS certification , You need to have solid understanding of architecture and how to apply various practice to optimize the solution in real time. I found Solution Architect's Handbook pretty useful . This is 490 pages of book covering all aspects of solution architecture and help you to not only clear the certification but also crack the interview. I will say price is pretty reasonable given that details content . You can order from amazon - https://www.amazon.com/gp/product/1838645640

Video Course: Recommend must go through Cloud Guru Videos first and then Linux Academy. Linux Academy wins due to their labs and covering more topics.

ACloudGuru AWS DevOps Engineer Course - While I mentioned in my previous post for SA Pro certification, aCloudGuru covered very high-level content but for DevOps Engg Pro, they did a decent job and provided more in-depth content.I t gives you good study structure and topics where the need to focus more. I am planning to complete this first.

Linux Academy AWS DevOps Engineer Course - For any pro certification, I can not take chance so going to take a complete course from Linux Academy as they have a good amount of labs and lots of exam notes with study material which save lots of time. I love their ready to use exam notes, labs, flash card and mobile app for offline reading so many benefits comes along. You can subscribe to Linux Academy by clicking here.

Go through this YouTube playlist for re-invent videos , I have create list with around 40 Videos you can skip some as each topic has approx 3-4 videos : https://www.youtube.com/playlist?list=PLEF5xKHm-3ZMbypRfRX1CfL49ku9O4DgM

Exam Notes:

Demo Test:

Whizlab test helped a lot to prepare well on many topics. You can purchase test here with nominal price and 400 questions.
Linux Academy Demo test is worth taking a look which comes with 100 questions.

Lab:

Labs are must pass this exam. Make sure to go through all Linux Academy labs and follow along labs for all cloud guru practical videos.

Whitepaper:

Important Notes (Based on Whizlab Test):

To deploy your code from a git repository in Elastic Beanstalk, create a docker file and get the code from gits, then use the docker file to perform deployment on new EB application. Alternatively, You can create AWS Cloud formation template having an EC2 instance with AMI having docker per-installed and bootstrap instance to pull code from git and deploy.

Single container Docker environments can be launched from a Dockerfile (which describes an image to build), a Dockerrun.aws.json file (which specifies an image to use and additional Elastic Beanstalk configuration options), or both. These configuration files can be bundled with source code and deployed in a ZIP file.

The other basic configuration, Multicontainer Docker, uses the Amazon ECS to coordinate the deployment of multiple Docker containers to an Amazon ECS cluster in an Elastic Beanstalk environment. The instances in the environment each run the same set of containers, which are defined in a Dockerrun.aws.json file. Read more details here.

In Elastic Beanstalk, You can save your environment's configuration as an object in Amazon S3 that can be applied to other environments during environment creation, or applied to a running environment. Saved configurations are YAML formatted templates that define an environment's platform configuration, tier, configuration option settings, and tags.
Auto Scaling Default Termination Policy :

Auto Scaling Default Termination Policy

The AWS::AutoScaling::AutoScalingGroup resource supports an UpdatePolicy attribute. This is used to define how an Auto Scaling group resource is updated when an update to the CloudFormation stack occurs. A common approach to updating an Auto Scaling group is to perform a rolling update, which is done by specifying the AutoScalingRollingUpdate policy. This retains the same Auto Scaling group and replaces old instances with new ones, according to the parameters specified. Find more details here
If you suspend AddToLoadBalancer, Auto Scaling launches the instances but does not add them to the load balancer or target group. If you resume the AddToLoadBalancer process, Auto Scaling resumes adding instances to the load balancer or target group when they are launched. However, Auto Scaling does not add the instances that were launched while this process was suspended. You must register those instances manually. Find more details here.
If you are deploying package from Elastic Beanstalk across multiple instances , but you want to execute SQL query only once , then you can use the container_commands key to execute query. Container commands run after the application and web server have been set up and the application version archive has been extracted, but before the application version is deployed. You can use leader_only to only run the command on a single instance. Leader-only container commands are only executed during environment creation and deployments, and not executed due to launch configuration changes, such as a change in the AMI Id or instance type.
A CreationPolicy instructs CloudFormation to wait on an instance until CloudFormation receives the specified number of signals.When you provision an Amazon EC2 instance in an AWS CloudFormation stack, you might specify additional actions to configure the instance, such as install software packages or bootstrap applications. Normally, CloudFormation proceeds with stack creation after the instance has been successfully created. However, you can use a CreationPolicy so that CloudFormation proceeds with stack creation only after your configuration actions are done.
An instance profile is a container for an IAM role that you can use to pass role information to an EC2 instance when the instance starts. It can be used to provide read-only access of production instance to the dev team for debugging of a critical defect in production.
You might want to include resources that aren't available as AWS CloudFormation resource types. You can include those resources by using custom resources. That way you can still manage all your related resources in a single stack.Use the AWS::CloudFormation::CustomResource or Custom::String resource type to define custom resources in your templates. Custom resources require one property: the service token, which specifies where AWS CloudFormation sends requests to, such as an Amazon SNS topic.
Auto Scaling lifecycle hooks enable you to perform custom actions by pausing instances as Auto Scaling launches or terminates them. For example, while your newly launched instance is paused, you could install or configure software on it. Find more details here
The cfn-hup helper is a daemon that detects changes in resource metadata and runs user-specified actions when a change is detected. This allows you to make configuration updates on your running Amazon EC2 instances through the UpdateStack API action.
Auto Scaling Lifecycle :
Auto Scaling Life-cycle

You can manage Amazon SQS messages with Amazon S3. This is especially useful for storing and consuming messages with a message size of up to 2 GB. To manage Amazon SQS messages with Amazon S3, use the Amazon SQS Extended Client Library for Java SDK.
Amazon Route 53 health checks monitor the health and performance of your web applications, web servers, and other resources. Each health check that you create can monitor one of the following:
1. The health of a specified resource, such as a web server
2. The status of an Amazon CloudWatch alarm
3. The status of other health checks
Jumbo frames (9001 MTU) allow more than 1500 bytes of data by increasing the payload size per packet, and thus increasing the percentage of the packet that is not packet overhead. Fewer packets are needed to send the same amount of usable data. However, outside of a given AWS region (EC2-Classic), a single VPC, or a VPC peering connection, you will experience a maximum path of 1500 MTU. VPN connections and traffic sent over an Internet gateway are limited to 1500 MTU. If packets are over 1500 bytes, they are fragmented, or they are dropped if the Don't Fragment flag is set in the IP header.
AWS Elastic Beanstalk provides several options for how deployments are processed, including deployment policies (All at once, Rolling, Rolling with additional batch, and Immutable) and options that let you configure the batch size and health check behavior during deployments. By default, your environment uses rolling deployments if you created it with the console or EB CLI, or all at once deployments if you created it with a different client (API, SDK or AWS CLI).
AWS OpsWorks Stacks provides built-in support for Memcached. However, if Redis better suits your requirements, you can customize your stack so that your application servers use ElastiCache Redis.
Pseudo parameters are parameters that are predefined by AWS CloudFormation. You do not declare them in your template. Use them the same way as you would a parameter, as the argument for the Ref function. This is the complete list of Pseudo Parameters:
AWS::AccountId, AWS::NotificationARNs, AWS::NoValue, AWS::Region, AWS::StackId, AWS::StackName
You can use intrinsic functions only in specific parts of a template. Currently, you can use intrinsic functions in resource properties, outputs, metadata attributes, and update policy attributes. You can also use intrinsic functions to conditionally create stack resources.
AWS OpsWorks Stacks provides a set of stack commands, which you can use to perform a variety of operations on a stack's instances. To run a stack command, click Run Command on the Stack page. Find list of commands here.
You can monitor AWS resources in multiple regions using a single CloudWatch dashboard. For example, you can create a dashboard that shows CPU utilization for an EC2 instance located in the us-west-2 region with your billing metrics, which are located in the us-east-1 region. However you can not aggregate Cloud Watch metrics across region.
When you need to update a stack, understanding how your changes will affect running resources before you implement them can help you update stacks with confidence. Change sets allow you to preview how proposed changes to a stack might impact your running resources.
You can use Run Command from the Amazon EC2 console to configure instances without having to log in to each instance.
You can avoid hitting the limit of web application version by applying an application version lifecycle policy to your applications. A lifecycle policy tells Elastic Beanstalk to delete application versions that are old or to delete application versions when the total number of versions for an application exceeds a specified number.
In OpsWork, for Linux stacks, if you want to associate an Amazon RDS service layer with your app, you must add the appropriate driver package to the associated app server layer.
If a recipe fails in OpsWork, then the instance will end up in the statesetup_failed instead of online. Even though the instance is not online as far as AWS OpsWorks Stacks is concerned, the EC2 instance is running and it's often useful to log in to troubleshoot the issue.
Every instance has an AWS OpsWorks Stacks agent that communicates regularly with the service. AWS OpsWorks Stacks uses that communication to monitor instance health. If an agent does not communicate with the service for more than approximately five minutes, AWS OpsWorks Stacks considers the instance to have failed.
Recommend to creating new Elastic Load Balancing load balancers for your AWS OpsWorks Stacks layers. If you choose to use an existing Elastic Load Balancing load balancer, you should first confirm that it is not being used for other purposes and has no attached instances. After the load balancer is attached to the layer, OpsWorks removes any existing instances and configures the load balancer to handle only the layer's instances.
To use Elastic Load Balancing with a Opswork stack, you must first create one or more load balancers in the same region by using the Elastic Load Balancing console, CLI, or API. You should be aware of the following:
- You can attach only one load balancer to a layer.
- Each load balancer can handle only one layer.
- AWS OpsWorks Stacks does not support Application Load Balancer. You can only use Classic Load Balancer with AWS OpsWorks Stacks.
When you use the AWS Elastic Beanstalk console to deploy a new application or an application version, you'll need to upload a source bundle. Your source bundle must meet the following requirements:
- Consist of a single ZIP file or WAR file (you can include multiple WAR files inside your ZIP file)
- Not exceed 512 MB
- Not include a parent folder or top-level directory (subdirectories are fine)

Important Notes (Based on LinuxAcademy Test):

Fields are reported with Elastic Load Balancer (Classic Load Balancer) access logs:
timestamp, backend:port, backend_processing_time, request_processing_time, received_bytes.
You can specify a publishing interval of either 5 minutes or 60 minutes when you enable the access log for your load balancer. By default, Elastic Load Balancing publishes logs at a 60-minute interval.
Amazon CloudWatch does not aggregate data across regions. Metrics are completely separate between regions.
Stack policies can protect specific resources from specific update actions, You can only have one stack policy per stack. This is a JSON documents that define which update actions can be performed on which resources and applied to all users who have access to the stack.
You can enable basic or detailed monitoring for the instances in your Auto Scaling group when you create a launch configuration. By default, basic monitoring is enabled when you create the launch configuration using the AWS Management Console and detailed monitoring is enabled when you create the launch configuration using the AWS CLI or an API.
When you create an AWS Elastic Beanstalk environment, you can specify an Amazon Machine Image (AMI) to use instead of the standard Elastic Beanstalk AMI included in your platform configuration's solution stack.A custom AMI also allows you to make changes to low level components, such as the Linux kernel, that are difficult to implement or take a long time to apply in configuration files.
Scaling adjustment types are ChangeInCapacity, ExactCapacity, PercentChangeInCapacity
For Amazon EC2 and Auto Scaling resources, recommend to use a CreationPolicy attribute instead of wait conditions. Add a CreationPolicy attribute to those resources, and use the cfn-signal helper script to signal when an instance creation process has completed successfully.
You can use a wait condition for situations like the following:
- To coordinate stack resource creation with configuration actions that are external to the stack creation
- To track the status of a configuration process

Keep Visiting for updates .

My Associates Certification Exams Tips and Notes :

http://getyouralgorithm.blogspot.com/2017/03/my-aws-certifications-lesson-learn-all.html

Refer my Associates Exam notes below:

AWS Certified Solution Architect - Associate:

http://getyouralgorithm.blogspot.com/2017/01/aws-solution-architect-associate.html

http://getyouralgorithm.blogspot.com/2016/11/aws-solution-architect-associate_13.html

AWS Certified Developer -Associate:

http://getyouralgorithm.blogspot.com/2017/02/aws-developer-associates-exam-notes.html

AWS Certified SysOps Administrator Associate:
http://getyouralgorithm.blogspot.com/2017/02/aws-sysops-associates-exam-notes.html

My Professional Certification Exam Tips and Notes:

AWS Certified Solution Architect Professional :

http://getyouralgorithm.blogspot.com/2017/05/exam-notes-for-aws-certified-solutions.html

http://getyouralgorithm.blogspot.com/2017/07/practice-questions-based-exam-notes-for.html

AWS Certified DevOps Engineer Professional :

http://getyouralgorithm.blogspot.com/2017/11/study-based-exam-notes-and-study-plan.html

AWS Certified BigData Speciality:

https://getyouralgorithm.blogspot.com/2018/01/study-based-exam-notes-and-study-plan.html

https://getyouralgorithm.blogspot.com/2018/06

Scale your application - One step at a time

2017-10-17T23:52:00.000-04:00

Often I come across people complaining about scaling problem for their e-commerce or WordPress website and talking about cost and complexity related to scaling. We have the highly popular fancy words "Horizontal Scaling" and "Micro-Service" and looks like people jump into this solution as soon as I ask them, so what is the plan?

Let's talk about different scaling option, if your current workload is in traditional data center then leverage cloud for your on-premise solution , scaling to achive greater efficiency with less cost (You don't want to setup whole powershouse to lightup few bulbs) . If your workload all in cloud then we obviously have out of the box options available.

I will say designing your API in micro-services and adding capability of horizontal scaling is good to have and might be the best choice , but here we are dealing with problem in hand, where my web application is already running in on-premise environment and I am expecting to quickly scale it, due to big launch of new products or late marketing strategy for peak season etc.

So how to handle this situation? I will say let's take one step at a time and might be horizontal scaling is last things I want to do ....What? Are you kidding? Yes, you hear it right !!!

Here, I am taking a very simple example. Assume you have a tech news website and somehow you got the first glimpse of upcoming iPhone launch and one of your tech gig wrote a very good review for yet to launch iPhone which went viral. This blog post includes both Video and bunch of pictures, also allow people to write comments and rate your post. Now your website hosted on the traditional server like a Linux box, let's assume a LAMP stack. So, here we have problem in-hand to address scaling problem immediately.

Continue reading to AWS blog - https://aws.amazon.com/blogs/architecture/scale-your-web-application-one-step-at-a-time/

Practice Questions Based Exam Notes For AWS Certified Solutions Architect - Professional

2017-07-06T14:52:00.000-04:00

In this post , I will list down important points came up while solving the practice questions for SA Pro Certification.

Scalability blog explain different scaling option

Book:

For any AWS certification , You need to have solid understanding of architecture and how to apply various practice to optimize the solution in real time. I found Solution Architect's Handbook pretty useful . This is 490 pages of book covering all aspects of solution architecture and help you to not only clear the certification but also crack the interview. I will say price is pretty reasonable given that details content . You can order from amazon - https://www.amazon.com/gp/product/1838645640

Whizlab Free Test -

To get extremely high I/O throughput, use instance stored back instances and stripe the attached ephemeral devices and use DRDB asynchronous replication. As instance stored back instances directly attached to physical hardware so they give unlimited I/O and DRDB provides durability.

Watch Linux Academy SA Pro Video - Increasing Performance with EBS RAID configuration and study notes from slides 154 to 161.

Amazon S3 hosted website doesn't support HTTPS connection as website endpoint, But Cloudfront can accept user request in HTTP or HTTPS. You can configure Cloudfront to use HTTP or HTTPS or Match Viewer (where Cloudfront use either HTTP or HTTPS based on viewer request ). You need to make sure one of the domain name in your SSL certificate on your origin server must match the domain name you specify for origin domain name ,otherwise viewer will see 502 bad request .

Watch Linux Academy SA Pro Video - Forcing HTTPS to origin and study notes from slides 264.
Read AWS blog : http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html#DownloadDistValuesOriginSSLProtocols

Whizlab Dignostic Test -

To connect multiples AWS accounts to a central VPC for identity validation use VPC peering.
ELB needs to configure multiple listeners from client to ELB and back from ELB to EC2 instances with port and protocol (HTTP ,HTTPS , TCP or SSL).
Both ELB and Cloudfront doesn't support client side SSL certificate or Self signed certificate. However 3rd party CA signed custom certificate supported at server side.
Use Cloud Front Origin Access Identity (OAI) to Restrict Access to Your Amazon S3 Content. This prevent users from accessing your Amazon S3 objects using Amazon S3 URLs.
You can restrict subnet level access at IAM policy level which will help to create zone with-in VPC . Suppose a VPC has 4 subnet then we can create four different IAM user with individual subnet policy restriction to secure access from each other subnet.
Amazon Redshift workload management (WLM) enables users to flexibly manage priorities within workloads so that short, fast-running queries won't get stuck in queues behind long-running queries.
After you create a set of DHCP options, you can't modify them. If you want your VPC to use a different set of DHCP options, you must create a new set and associate them with your VPC. http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_DHCP_Options.html
The Dynamic Host Configuration Protocol (DHCP) provides a standard for passing configuration information to hosts on a TCP/IP network. You can assign your own domain name to your instances, and use up to four of your own DNS servers. To do that, you must specify a special set of DHCP options to use with the VPC.
WAF Sandwich Pattern - There are EC2 instances running the WAF software placed in an Auto Scaling group between 2 Elastic Load Balancers in front of web server fleet. The first load balancer is public-facing and will accept internet traffic. The second one is used to send traffic to the Web EC2 instance once it has been deemed the non-malicious.
To point hard-coded on-premises host-name to ELB , create a private record set in route53 with hardcoded hostname and pointing alias record to ELB.
AWS doesn't support promiscuous mode for IPS/IDS.

Watch Linux Academy SA Pro Video - VPC Network Monitoring and study notes from slides 212 to 218.

AWS doesn't support multicast , To deploy legacy applications that require multicast create a Virtual overlay network that runs on the OS level of the instance.

Watch Linux Academy SA Pro Video - Supporting Legacy Application and study notes from slides 203 to 210.

To attach multiple SSL certification to a single EC2 instance , create instance with multiple network interface with multiple elastic IP address.
To use same EC2 key pair across multiple region , use import key pair feature from AWS web console. However pem key can not be copied with AMI migration and you need to make sure launch new AMI instances with same pem key name in order to use existing pem key as authorization key will copied to AMI and live with OS in new region.
EC2 instances running in private sub-nets of a VPC can have controlled access to S3 buckets using VPC S3 endpoint and it doesn't require any NAT gateway or NAT instance.
Redshift periodically takes snapshots and tracks incremental changes to the cluster.When you restore from a snapshot, Amazon Redshift creates a new cluster and makes the new cluster available before all of the data is loaded, so you can begin querying the new cluster immediately. The cluster streams data on demand from the snapshot in response to active queries, then loads the remaining data in the background.
You must create a virtual interface to begin using your AWS Direct Connect connection. You can create a private virtual interface to connect to your VPC, or you can create a public virtual interface to connect to AWS services that aren't in a VPC, such as Amazon S3 and Amazon Glacier. You can configure multiple virtual interfaces on a single AWS Direct Connect connection. For private virtual interfaces, you need one private virtual interface for each VPC to connect to from the AWS Direct Connect connection.
If you purchase direct connect in region-1 and want to have more reliable connection on region-2 then create a virtual public interface to region-2's public end point from region-1 and secure it over VPN.

Linux Academy AWS PRO Test -

If the application is using TCP, configure proxy protocol to pass the client IP address in a new TCP header. If the application is using HTTP , modify the application code to pull the client IP into the x-forward-for header so the web servers can parse it. see details here : https://aws.amazon.com/blogs/aws/elastic-load-balancing-adds-support-for-proxy-protocol/
RDS instances can replicate to on-premise database servers. It is best practice to first create a dump of the database and copy it down, then enable replication, since this uses the MySQL asynchronous replication feature. Latency is an issue when using replication, so consider using a Direct Connect connection depending the use case for this situation.
The ELB will not work correctly if using Layer 7 HTTP and the application does not respond back with standard HTTP response codes. To support this type of application TCP ports are required.Thus configuration of port 80 over TCP required for health check.
You receive a "400 Error Code:PipelineNotFoundException." while starting data pipeline. This error means that your IAM default roles might not have the required permissions necessary for AWS Data Pipeline to function correctly.
If you need to define latency based routing first and then route users to weighted based routing with-in region , then you will need to use complex routing (nested record sets) and ensure that you define the weighted resource record sets first. Means first region level routing then global level routing on top of that need to configure in Route 53.
To point internal EC2 application to on-premise Database and create fail-over , Use an existing on-premise DNS server to configure hostnames for internal DNS records. Create a new Amazon VPC DHCP Option Set with the internal DNS server's IP address and attach it to VPC and reboot EC2 instaces to reflect that in /etc/resolve.config
If you need to delete Cloud formation but keep data of S3 and RDS then , Set the DeletionPolicy on the S3 resource declaration in the CloudFormation template to retain, set the RDS resource declaration DeletionPolicy to snapshot. Setting the DeletionPolicy on the S3 bucket as retain will ensure the S3 bucket is not removed. Keeping the S3 bucket and the name of the S3 bucket ensures it is easy to relaunch the application later with a template. Setting the RDS DeletionPolicy to snapshot ensures the data can be restored when the application needs to be run again later. Setting the DeletionPolicy on RDS to retain would leave the RDS instance running when it would not be used, thus increasing costs when not required.
CloudFront always caches responses to GET and HEAD requests, and you can also configure CloudFront to cache responses to OPTIONS requests but its not by default. Responses to other requests which use other methods are not cached by CloudFront.
To stream .mp4 file , Upload the MP4 files to S3 and create an Elastic Transcoder job that transcodes the MP4 source into HLS chunks. Store the HLS output in S3 and create a CloudFront download distribution to serve the HLS files to end users. CloudFront streaming distributions only support the Adobe RTMP streaming protocol. HLS a progressive download protocol. Configuring the output of the HLS chunks to be an S3 bucket and using the S3 bucket as the origin for streaming would be the most scalable way to solve the criteria.
To fix 'capacity error' for placement group ,Stop and restart the instances in the Placement Group and then try the launch again.
NAT Gateways already have built-in fault tolerance. From the docs: "Each NAT gateway is created in a specific Availability Zone and implemented with redundancy in that zone." Granted, fault tolerance !=redundancy . So creating multiple NAT gateways in the same subnet doesn't make sense. Instead, you might argue that you need to create multiple gateways in different AZs, so that if one AZ goes down, you have a backup.

Whizlab Practice Test I -

To design Internet connectivity to VPC and Webserver must be available to internet , Assign EIP to all webserver and configure a Route 53 record set with all the EIPs with health check and DNS failover. Place all your webserver behind ELB and configure route 53 CNAME to point to the ELB DNS Name.
When need to provide your own AWS account resource access to third party vendor , always use assume role with external ID parameter if not passing external ID then user corss account access with resource restriction as role will create confuse deputy problem. See details here : https://aws.amazon.com/blogs/security/how-to-use-external-id-when-granting-access-to-your-aws-resources/
Web Proxy mostly refer as Forward proxy put in front of internal network and control request originating from internal clients to Internet , while Reverse proxy contorl traffic coming from internet to internal network , like ELB and CDN can work as reverse proxy . See deatils - http://www.jscape.com/blog/bid/87783/Forward-Proxy-vs-Reverse-Proxy .
To provide role access to an instance via Cloud formation , Refer the role in the instance profile property ("RootInstanceProfile") of instance in cloud formation.
SSL Offloading to ELB- This term used for when you configure SSL/TLS protocol in ELB (HTTPS listener) so that ELB needs to do traffic encryption from client to ELB and ELB to EC2 for any https request.
If you want to provide resource access from Master account to Dev and Test account , then need to create IAM user in master account and cross account access role in Dev/Test Account.
If you routing traffic through Cloudfront , then you can configure cloudfront to create access log which will have detail information regarding every user access and you can provide specific S3 bucket to store those log for analysis purpose.
To Authenticate user from in-premise env to cloud env , Always need to use Identify broker to verify that employees are authenticated within the existing authentication system (AD or LDAP) . Identity broker application has permissions to access IAM's token service (STS) API to create temporary security credentials.
Each gateway configured for gateway cached volumes can support up to 20 volumes and total volume storage of 150 TB.

Whizlab Practice Test II -

An elastic network interface (ENI) is a virtual network interface that you can attach to an instance in a VPC. Network interfaces are available only for instances running in a VPC.
A network interface can include the following attributes:

A primary private IPv4 address
One or more secondary private IPv4 addresses
One Elastic IP address (IPv4) per private IPv4 address
One public IPv4 address
One or more IPv6 addresses
One or more security groups
A MAC address
A source/destination check flag
A description

You can create a network interface, attach it to an instance, detach it from an instance, and attach it to another instance. The attributes of a network interface follow it as it's attached or detached from an instance and reattached to another instance. When you move a network interface from one instance to another, network traffic is redirected to the new instance.
Every instance in a VPC has a default network interface, called the primary network interface (eth0). You cannot detach a primary network interface from an instance. You can create and attach additional network interfaces. The maximum number of network interfaces that you can use varies by instance type.

Temporary security credentials are valid until they expire, and they cannot be revoked. However, because permissions are evaluated each time an AWS request is made using the credentials, you can achieve the effect of revoking the credentials by changing the permissions for the credentials even after they have been issued. If you remove all permissions from the temporary security credentials, subsequent AWS requests that use those credentials will fail.
if you provision a single connection to any Direct Connect location in the US, you can use it to access AWS services in all US Regions and AWS GovCloud (US). You need to do is create a public virtual interface on your Direct Connect connection with the AWS Management Console or AWS Direct Connect API. Once this interface is created, traffic will use your AWS Direct Connect connection to access AWS services in any US Region, or AWS GovCloud (US), instead of the Internet.
You have three mutually exclusive options depending on how you choose to manage the encryption keys:
- Use Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3) – Each object is encrypted with a unique key employing strong multi-factor encryption. As an additional safeguard, it encrypts the key itself with a master key that it regularly rotates. Amazon S3 server-side encryption uses one of the strongest block ciphers available, 256-bit Advanced Encryption Standard (AES-256), to encrypt your data.
- Use Server-Side Encryption with AWS KMS-Managed Keys (SSE-KMS) – Similar to SSE-S3, but there are separate permissions for the use of an envelope key (that is, a key that protects your data's encryption key) that provides added protection against unauthorized access of your objects in S3. SSE-KMS also provides you with an audit trail of when your key was used and by whom. Additionally, you have the option to create and manage encryption keys yourself, or use a default key that is unique to you, the service you're using, and the region you're working in.
- Use Server-Side Encryption with Customer-Provided Keys (SSE-C) – You manage the encryption keys and Amazon S3 manages the encryption, as it writes to disks, and decryption, when you access your objects.
Note : When you list objects in your bucket, the list API will return a list of all objects, regardless of whether they are encrypted.

Study Plan For AWS Certified Solutions Architect - Professional

2017-05-04T00:36:00.000-04:00

Finally Achieved AWS Certified Solutions Architect - Professional certified with 80% score on 4th-Aug-2017 . Its really required lots of study and hard work compare to associate certification .

I did slow study everyday 2 hrs for two months and studied 10 to 12 hrs daily in last 6 days before exam. Finally , it got paid up by achieving the milestone.

Find my Tips below:

You can find more about exam from AWS website SA Pro Exam Page , which include exam blueprint and sample questions.

I didn't leverage that, but you can get extra 30 minute time by using following steps , specially if english is not your native language:

From the top navigation, click Upcoming Exams
On the right, click the Request Exam Accommodations button
Click the Request Accommodation button
Select ESL +30 Minutes from the accommodation dropdown
Click Create

My preparation strategy :

Book:

For any AWS certification , You need to have solid understanding of architecture and how to apply various practice to optimize the solution in real time. I found Solution Architect's Handbook pretty useful . This is 490 pages of book covering all aspects of solution architecture and help you to not only clear the certification but also crack the interview. I will say price is pretty reasonable given that details content . You can order from amazon - https://www.amazon.com/gp/product/1838645640

Video Course : Recommend to use Linux Academy

ACloudGuru AWS SA Pro Course - Looks like these videos cover very high level content with only one lab in entire course . Most of the course track taken from their own associate course , so if you already want through all aCloudGuru associate course then Pro course might not value for money . Overall it will still help you to give a structure and additional advice to develop depth .

Linux Academy AWS SA Pro Course - This course is subscription base with monthly subscription as it include all free labs and each lab last for 2 hrs, so you will have enough time to play with it. For Pro course, running lab from own account could cost more . Over all , I like Linux Academy pro course better than aCloudGuru, as they have good amount of labs and clarify many hybrid cloud concept which is important for Pro. Also they have nice ready to use exam notes , flash card and mobile app for offline reading so many benefit comes along. You can subscribe to Linux Academy by clicking here .

Go through this YouTube playlist for re-invent videos : https://www.youtube.com/playlist?list=PL_RVC-cMNyYTz8zlxq117O1bfji-knooI

Demo Test:

Whizlab SA Pro Course Test - There AWS associates test helped me a lot to score very well in exam so I am going to refer that to develop deep understanding and practice question . It has 400 questions with details explanation . - There AWS associates test helped me a lot to score very well in exam so I am going to refer that to develop deep understanding and practice question . It has 400 questions with details explanation . You can buy test from here .

ProProfs Test - It has 66 question with good quality . Make sure to do your own due diligence for answers as all of them are not correct

AWS Demo Test - This was harder than real exam and helped me to identify gap for my preparation. It cost $40 from. You can buy it from https://www.webassessor.com.

Solve All questions provide in Jayendra patil's blog , specially those makred for Pro Exam.

White paper - Below are important once but try to cover as many as possible.

Lab:

All available in Linux Academy course, You should able to perform all of them without reffering any guide.

Exam Notes:

Refer My practice question based exam notes. Still updating.
Download Linux Academy Exam notes . It is 394 slides having very comprehensive coverage.
Read Jayendra patil's blog has very comprehensive study notes.

Some more notes are here:

Deployment Management :

To configure instances, use CloudFormation::Init
CreationPolicy is a property of an instance or Auto Scaling group. This defines a time limit to wait for the instance(s) to complete configuration. If a CreationPolicy is used as a property of an Auto Scaling group, you can specify a percentage or count of instances.For example, you may need 50% of the instances to be online to consider creation complete.
WaitCondition is a separate resource that allows a delay. A WaitCondition is a separate CloudFormation resource, not a property of an instance or Auto Scaling group.Multiple instances can depend on one WaitCondition, such as a database and a cache, both of which need to be configured before CloudFormation creates application servers.
To remove resources created by CloudFormation, delete the stack. Consider using the DeletionPolicy attribute for resources that should be kept running after a stack deletion. Most CloudFormation resources can have a DeletionPolicy attribute.CloudFormation will not delete the resource during a stack deletion.
The value of Snapshot can be applied to EBS , RDS and RedShift clusters. A final snapshot is created before the resource is deleted during a stack deletion.
CodeDeploy deploys the application revision to EC2 instances identified by tag or by membership in an Auto Scaling group.A deployment configuration can be used to target a portion of instances. For example, CodeDeploy can deploy to a percentage of the fleet at a time.
Amazon EC2 Container Service (Amazon ECS) is a highly scalable, fast, container management service that makes it easy to run, stop, and manage Docker containers on a cluster of Amazon EC2 instances
An ECS Cluster is a logical grouping of container instances that you can place tasks on.
A Container instance is an Amazon EC2 instance that is running the Amazon ECS agent and has been registered into a cluster.
An ECS task definition is a description of an application that contains one or more container definitions (such as CPU and RAM requirements, network port bindings, and storage).
An ECS Service allows you to run and maintain a specified number of instances of a task definition simultaneously
The ECS Scheduler places tasks on container instances, based on resource availability.
The service scheduler ensures that the specified number of tasks are constantly running and reschedules tasks when a task fails .
ECS RunTask randomly distributes tasks across your cluster and tries to minimize the chances that a single instance on your cluster will get a disproportionate number of tasks.

Network Design:

Enhance networking can be achieved through SR-IOV (Single Root Input/Output
Virtualization) available for selected HVM (Hardware Virtual Machine) type .
Put instances in same placement group for maximum network speed. Achieve 10Gbps using Intel VFI (Virtual Function Interface) and 20Gbps using ENA (Elastic Network Adapter).
Placement group don't span AZ , can be span across VPC but performance get impacted.
They can not merge together and all instance should be in stop state to add in placement
group and they should be connected using private IPs.
Data sent across AZ and VPC limited to 1500 MTU , Jumbo frame only allowed inside VPC have 9000 MTU , suitable for placement group.
IpSec is a secure protocol to transfer data packet over network for VPN. It facilitate authentication and IP packet encryption and decryption over network.
Static VPN require all routes IP prefix to be specified . Every VPN establist 2 IPSec tunnel in 2 AZ with pair of security association (Inbound/oubound) per tunnel
Dynamic VPN support 2-bytes autonomous system number (ASN) using Border Gateway protocol (BGP). VPN HA can be achieve using multiple route at CGW and m:m relationship in between VPC to VPN.
AWS VPN always initiated from Customer Gateway (CGW) and IPSec Secure.
VPN Gateway (VPG) determine traffic path based on order of Most specific IP prefix , Statics VPN connection , Direct Connect then Dynamic VPN if all of them are configured.
Customer can host VPN appliance on EC2 if need to manage both end of VPN.

Costing:

If consolidated billing is enable between accounts , then unused RI from one account can be used as On-demand for other accounts given that instance types are identical and in same AZ.

Data Storage :

When you start putting objects into S3, you may be performing PUTs in sequential fashion. In order to increase your throughput to Amazon S3, you can parallelize your PUT requests.
Service support are Initiate multipart upload, upload part , completed multipart upload, Abort multipart upload , List parts ,List multipart upload.
Amazon CloudFront supports range-based GETs to download large files.
Amazon S3 Transfer Acceleration will add additional costs, as in addition to any other costs incurred with the transfer, you will be charged for the amount of data transferred through the edge location that the operation uses.
As enabling S3 Transfer Acceleration will change S3 bucket URL , so It's also important to make sure that your application is actually using the correct URL once transfer acceleration has been turned on, as the URL will change.
After you initiate a snapshot restore to a gateway-stored volume, snapshot data is downloaded in the background. This functionality means that after you create a volume from a snapshot, there is no need to wait for all of the data to transfer from Amazon S3 to your volume before your application can start accessing the volume and all of its data. If your application accesses a piece of data that has not yet been loaded, the gateway immediately downloads the requested data from Amazon S3. The gateway then continues loading the rest of the volume's data in the background.
Retrieving tapes to your VTL will be available in about 24 hours.

Keep Visiting for updates .

My Associates Certification Exams Tips and Notes :

http://getyouralgorithm.blogspot.com/2017/03/my-aws-certifications-lesson-learn-all.html

Refer my Associates Exam notes below:

AWS Certified Solution Architect - Associate:

http://getyouralgorithm.blogspot.com/2017/01/aws-solution-architect-associate.html

http://getyouralgorithm.blogspot.com/2016/11/aws-solution-architect-associate_13.html

AWS Certified Developer -Associate:

http://getyouralgorithm.blogspot.com/2017/02/aws-developer-associates-exam-notes.html

AWS Certified SysOps Administrator Associate:
http://getyouralgorithm.blogspot.com/2017/02/aws-sysops-associates-exam-notes.html

My Professional Certification Exam Tips and Notes:

AWS Certified Solution Architect Professional :

http://getyouralgorithm.blogspot.com/2017/05/exam-notes-for-aws-certified-solutions.html

http://getyouralgorithm.blogspot.com/2017/07/practice-questions-based-exam-notes-for.html

AWS Certified DevOps Engineer Professional :

http://getyouralgorithm.blogspot.com/2017/11/study-based-exam-notes-and-study-plan.html

AWS Certified BigData Speciality:

https://getyouralgorithm.blogspot.com/2018/01/study-based-exam-notes-and-study-plan.html

https://getyouralgorithm.blogspot.com/2018/06/study-based-exam-notes-for-aws.html

My AWS certifications Lesson Learn - All three associate certification

2017-03-12T01:12:00.000-05:00

I achieved all three AWS associate certification last month and thought to share my lesson learned, which can benefit to many AWS aspirants. I wanted to write a good learning lesson because all I have been doing at the various forum(s) for Weeks is using all the wisdom and tips and now it's time to give some back.I Passed all three certification with near 90% score.

Learn more about AWS certification here: https://aws.amazon.com/certification/

Preparation Plan:

I will suggest going for certification, if you have some hands-on in AWS, if not then you need to work hard to develop practical understanding. It's very important to have good hands-on in order to pass the exam. Just theoretical training will not help. My target to prepare certification with most affordable cost and best course. Here are my tips for exam resources:

The Book:

When you are going for AWS Certification , you are inspired to become solution architect. I will recommend to read Solution architect's handbook which provide great details of all architecture aspect .

Book is very highly rated in amazon.com and one of the reviewers describe it as:

"I'm using this book for the certification and needless to say this is the most reader-friendly book with all the necessary explanation. Language in the book is direct and well written. I learned a good deal about how the services work. The balance was reasonable between text and diagrams of architecture. I have read many other good books too, but those are either way descriptive or too short to explain much concepts. So, if I have to choose just one, I would keep this one. It succinctly presents all the important points of every subject in that arena.

I also highly recommend this book as it prepares well for Solutions Architect concepts and interviews at not just at AWS but for any other cloud as well. The way I am using it is identifying my weak areas, read the concepts from this book and also hands on same concept with AWS Console. One more thing about this book is that it covers all new features and services."

Good Luck everyone for their certification and/or job interviews 👍

Here is the book link if you are interested:
https://www.amazon.com/gp/product/1838645640

Video Tutorial:

This is my favorite way of learning and I went through aCloudGuru and Linux Academy tutorial as it is to the point for the exam with practical labs, but again it may be not sufficient to pass the exam.

My opinions on Video Tutorial:

ACloudGuru Course:

ACloudGuru Video Tutorials are good for the associate exam and teaching format is more informal. The main issue is See details review below:

AWS Certified Solution Architect - Associate 2020:
You can buy this course in Udemy in just $10. It is 19 hrs of video learning series with 108 lectures. This is the most up-to-date and best course I came across, as it teaches you all practical hands-on by utilizing free AWS tier. This has a good amount of practice question also, but topic test is much easier than the exam. However, the final practice test is near to exam, but it has only 30 questions. I will suggest some good practice test later in the blog.

AWS Certified Developer - Associate 2020:
Again, this course is also just $10 in Udemy. It is 14.5 hrs of video learning series with 85 lectures. This series has many overlap topic with certified solution architect and cover Dynamo DB in more depth as required. There is serious need to update this course as of Feb-2017 but good enough to cover all exam topic.

AWS Certified SysOps Administrator - Associate 2020:
This course also cost just $10 in Udemy. It is 8.5 hrs of video learning series with 65 lectures. This course doesn't stand alone and you have to buy either developer or solution architect certification course, However , Again it has an overlapping tutorial of networking and storage section which make sense.

Linux Academy Course:

Personally, I liked it more as it is more structured and teach you things to become solution architect in real life also not to just pass the exam. This is subscription based course, you need to pay $18 per month and you will get access for all courses. They have the in-built lab also which is a great thing for first-time learner. You can subscribe here for their course.

White Papers:

You will find video tutorial cover these white paper but I will recommend going through them once before appearing for the exam:

Scalability blog explain different scaling option

Overview of AWS:
https://d0.awsstatic.com/whitepapers/aws-overview.pdf

Security (Must Read):
https://d0.awsstatic.com/whitepapers/Security/Intro_Security_Practices.pdf

Risk and Compliance:
https://d0.awsstatic.com/whitepapers/compliance/AWS_Risk_and_Compliance_Whitepaper.pdf

Architecting Best Practices:
https://d0.awsstatic.com/whitepapers/AWS_Cloud_Best_Practices.pdf

Labs:

AWS Free Tier:

I will recommend creating free AWS account by signing up in a link:
http://console.aws.amazon.com/.

I prepared my certification using this and got charged only $1 while spin up 5 EC2 using autoscaling and configure route 53 to add Apex zone name.

Practice Test:

WhizLab:

They have a practice test available for all associate exam in affordable cost around $15 each. Each practice test consisting of 60 question and total 300 question available for each exam,

The solution Architect practice test is very near to exam and if you can score 75% to 80% in test then good to go for the exam. SysOps exam follow exam pattern well but have some granular level of questions, asking about specific API and command which won't come in exam. They just launched developer exam and looking at their reputation, it should be at par to an exam.

I will recommend going through their practice test before appearing for Solution Architect exam.
Here is the link:

AWS Solutions Architect Associate Exam
AWS SysOps Administrator Associate Exam
AWS Developer Associate Exam

Exam Notes:

Refer my Exam notes below:

AWS Certified Solution Architect - Associate:
http://getyouralgorithm.blogspot.com/2017/01/aws-solution-architect-associate.html
http://getyouralgorithm.blogspot.com/2016/11/aws-solution-architect-associate_13.html

AWS Certified Developer -Associate:
http://getyouralgorithm.blogspot.com/2017/02/aws-developer-associates-exam-notes.html

AWS Certified SysOps Administrator Associate:
http://getyouralgorithm.blogspot.com/2017/02/aws-sysops-associates-exam-notes.html

The Exam:
You can register for the exam using the Kryterion website https://www.webassessor.com. All associate exam cost $150 and you can re-take exam after 2 weeks in $ 75, if not able to clear in the first attempt. Exams conduct in proctored test center and you have to appear in the exam with two proof of ID (e. g. passport or driving license along with credit card).

However, passing marks always fluctuate as AWS follow bell curve to decide the exam pass mark, but if you will score around 70% then it should be good to pass the exam.

AWS Solution Architect Associate exam difficulty is good and it covers almost all major AWS services which organizations use frequently. You need a deep understanding of Storage, EC2, and VPC to pass the exam. Again very near to Whizlab test - AWS Solutions Architect Associate Exam

AWS Developer Associates exam is the easiest exam of all three AWS associate exam but you need a good amount of study to crack it and deep understanding of Dynamo DB. Whizlab test can provide you good insight - AWS Developer Associate Exam

AWS SysOps Administrator exam is a lot harder than the developer exam and almost the same level as the solution architect exam. You need to have a good hold on security and networking to pass this.
You can get benefit from Whizlab test - AWS SysOps Administrator Associate Exam

Hope this will help you to pass the exam. Reach out to me via LinkedIn if need any additional help. Comments and Suggestion are welcome :).

My Associates Certification Exams Tips and Notes :

http://getyouralgorithm.blogspot.com/2017/03/my-aws-certifications-lesson-learn-all.html

Refer my Associates Exam notes below:

AWS Certified Solution Architect - Associate:

http://getyouralgorithm.blogspot.com/2017/01/aws-solution-architect-associate.html

http://getyouralgorithm.blogspot.com/2016/11/aws-solution-architect-associate_13.html

AWS Certified Developer -Associate:

http://getyouralgorithm.blogspot.com/2017/02/aws-developer-associates-exam-notes.html

AWS Certified SysOps Administrator Associate:
http://getyouralgorithm.blogspot.com/2017/02/aws-sysops-associates-exam-notes.html

My Professional Certification Exam Tips and Notes:

AWS Certified Solution Architect Professional :

http://getyouralgorithm.blogspot.com/2017/05/exam-notes-for-aws-certified-solutions.html

http://getyouralgorithm.blogspot.com/2017/07/practice-questions-based-exam-notes-for.html

AWS Certified DevOps Engineer Professional :

http://getyouralgorithm.blogspot.com/2017/11/study-based-exam-notes-and-study-plan.html

AWS Certified BigData Speciality:

https://getyouralgorithm.blogspot.com/2018/01/study-based-exam-notes-and-study-plan.html

https://getyouralgorithm.blogspot.com/2018/06/study-based-exam-notes-for-aws.html

AWS SysOps Associates Certification Exam Notes

2017-02-24T00:30:00.002-05:00

Amazon Web Services - (AWS) Certification is fast becoming the must have certificate for any IT professional working with AWS. This exam is lot harder than developer and almost same level as solution architect . Exam cost $150.

Find more details here:
https://aws.amazon.com/certification/certified-sysops-admin-associate/

Read my Lesson Learn:
http://getyouralgorithm.blogspot.com/2017/03/my-aws-certifications-lesson-learn-all.html

You can buy a cloud guru video or subscribe to Linux Academy course . Make sure to have lots of practice and go through question dumps .

Scalability blog explain different scaling option

Monitoring:

CloudWatch Monitor : ASG, ELB,Route53 Health Check, EBS, Storage Gateway, CloudFront, Dynamo DB , ElasticCache Node , RDS instance , EMR Job flow, RedShift,SNS,SQS,OpsWorks, Cloudwatch logs, AWS Bill.

Cloudwatch Monitor EC2 Metrics :

CPU , Network , Disk , Status Check .
Don't monitor RAM -memory is custom metrics.
By default monitoring 5 minute . Detail Monitoring 1 minute.
By default show 2 week Metrics
Use GetMetricStatistics API to get data longer than 2 week
EC2 or ELB metrics available till 2 week after termination

Default metrics granularity depend upon service can vary from 1 to 5 minute but minimum possible granularity can not be less than 1 minute .
EC2 status check :

System Status Check - Check host , Stop and Start to resolve issue
Instance Status Check - Checks VM , Reboot to resolve issue

General Purpose SSD provide 3 IOPS/GiB. Maximum Volume size 16 TiB and 10,000 IOPS.
Volume up to 1TiB can burst up to 3000 IOPS , so 500 GiB Vol eligible from 500*3 = 1500 IOPS , but can burst to 3000 IOPS if required . However , it will dip into free credit of 5.4M IOPS.
Pre-Warming EBS volumes -EBS volume can consume 5 to 50% of IOPS when attach to first time in instance , to avoid that pre-warm your EBS by

Write all blocks if new EBS volume
Read all blocks if existing EBS volume and have data in it.

EBS volumeQueueLength Metrics show how many read/write request in queue which can slow down performance. Queue length should be zero if this is more , then upgrade to more IOPS.
EBS Volume status check

Ok -Normal
Warning - Degraded/ severely Degraded
Impaired - Stall/Not available
Insufficient Data - Insufficient Data

RDS metrics can be monitor by CloudWatch or by Events itself in RDS
Key RDS metrics are - DatabaseConnections , DiskQueueDepth, FreeStorageSpace, ReplicaLog (Seconds), ReadIOPS, WriteIOPS, ReadLatency, WriteLatency
ELB report metric every minute by default but only if there is traffic.
Key ELB metrics SurgeQueueLength (No. of Request pending for submission) and SpilOverCount (No of Request got rejected as queue was full).
Elasticache Monitoring :

CPU utilization- Memcache is multi-threaded and can handle load up to 90% , Reddis is not multi-threaded and handle low CPU utilization (90/no. of nodes).
Swap Usage - Amount of swap file use , Swap size = RAM size . Memcache swap use should be 0 but not more than 50Mb. Check memcached_connections_overhead if >50Mb . Reddis use reserve memory so no swap usage metrics.
Evictions - If new item added , old must be remove . To mitigate this for Memcache scale up or scale out , decide threshold based on your application. For Reddis only scale up, no standard threshold .
Concurrent Connection - No recommended setting for Memcache and Reddis. Indicate large traffic spike or application is not releasing connection

Centralize monitoring help to monitor whole stack and different server in one place via 3rd party software . Enable ICMP between VPC to specific port or IP range to setup monitoring across server.
Consolidated billing manage by parent account and linked account . Create billing alarm in CloudWatch with amount threshold to monitor billing.
Root access available to Elastic BeanStalk , EMR , EC2 , OpsWork.
To handle sudden spike in ELB it need to pre-warm by raising request to AWS support with details of timeline , request/sec and request/response size .

OpsWork:

Consist of Stacks and Layers and Run of Chef
Stacks or resources such as EC2 , ELB , RDS instances and it could be define as Dev , test or prod stack .
Layers are like Web , Application and Database layer.
Each stack will have 1 or more layer and each layer will have assigned instances. Pre-configured layers are Application , Database , Load Balancers, Caching
Support Sample Stack , Chef 12 stack and Chef 11 stack.
All EC2 must be added by OpsWork stack in ELB , any EC2 added outside of OpsWork will get remove.

Read other associate exam notes:
Cover RDS, EC2 , IAM , Route 53 and VPCs
http://getyouralgorithm.blogspot.com/2017/02/aws-developer-associates-exam-notes.html
http://getyouralgorithm.blogspot.com/2017/01/aws-solution-architect-associate.html

Important points:

CloudWatch:

To simulate CloudWatch Alarm you can use CLI to trigger alram.
Cloudwatch has three alarm state OK (Metrics is within threshold ) , ALARM (Metrics is outside of threshold ), INSUFFICIENT_DATA(Enough data not available). To disable alarm temporary use CLI as console doesn't have option.
CloudWatch doesn't have Aggregate metrics available but it has average , min , max , sum.
You need to enable billing alert to setup billing alarm in CloudWatch.
ClouWatch default time in UTC but you can change it to local timezone from cloudwatch console.
CloudWatch detail monitoring not available for EMR and SNS. You can set only 5 minute interval minimum.
To send your own metrics to cloudwatch use put-metrics-data command with statics-value parameter. The size of HTTP GET request is 8 KB and POST request is 40KB and it is mandatory to pass namespace. In case of large datapoint you can send aggregated data like sum , min ,max with custom namespace. In opposite if you don't have any data for a period then send zero value to cloudwatch.
When sending metrics to Cloudwatch it can have 2 week past time stamp and 2 hrs future time stamp. In case of no timestamp , cloudwatch will mark as current timestamp.To find exact time data , user can use absolute tab under cloudwatch and give exact time range.
You can publish your own metrics in cloudwatch using API or CLI by supplying file as input in command. To see more granular data select interval using mouse and it will zoom metric.
You can change custom bound to Y-axis to see data better in Cloudwatch if graph has mutiple unit in it like CPU and Network IO. Read more : http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/switch_graph_axes.html
You can aggregate CPU utilization data in cloudwatch by AMI ID.
CloudWatch end point format monitoring.us-east-1.amazonaws.com for zone us-east-1

ELB and AutoScaling:

If you delete ELB then your registered EC2 will continue running without any impact.ELB log doesn't have EC2 IP .
ELB connection draining default timeout is 300 sec and max timeout is 3600 (1 hrs) sec.
Make sure to enable health check for ELB in auto scaling group in order to auto scaling terminate unhealthy instance . Default setting is EC2.
You can not schedule two auto scaling process at same time as it will throw conflict error.
Autoscaling process will be suspend if it can not launch resource in 24 hours.You can suspend particular autoscaling process from console to investigate issue like auto scaling alarm notification , launch , termination ,heathcheck,Add to ELB , Schedule action , AZ rebalance etc.
If you suspend termination then Auto scaling can grow 10% beyond max instance size due to AZ rebalance.
in Auto Scaling , If AddToELB is suspend then new instance will not register with ELB and user have to register them manually after resume.
Auto Scaling , ELB and Route 53 health check has free detail monitoring available.
AutoScaling terminate the unhealthy instance first and then launch new instance.
Troubleshoot when Autoscaling fail to launch EC2 : http://docs.aws.amazon.com/autoscaling/latest/userguide/ts-as-instancelaunchfailure.html
You can add more subnet in ELB while it us running using command elb-enable-availability-zone-for-load-balancer
To avoid new instance launch during reboot from Auto Scaling you need to use auto scaling command (set-instance-health) to change instance to healthy.
If detail monitoring is disable for auto scaling then user need to define new launch configuration group with detail monitoring enable and attach it to ASG.
ELB has evaluate target health check which will ensure if one region fail then hop request to another region.
ELB's "x-forwarded-for" log client IP address so you can filter traffic using particular ip address if required .
ELB create one security group by default for EC2-VPC however EC2-classic create 2 security group by default.

EC2 , AMI and EBS:

During AMI creation you don't need to define AMI launch permission. You need AWS account ID, X.509 certificate with private key , and AWS account access key id/secrete access key.
For EC2-Classic , Elastic IP (EIP) get disassociated when you stop it but it remain attached for EC2-VPC. You can have 5 EIP per region for each EC2-Classic and EC2-VPC.
EC2 throw host key not found error if you enter wrong user id.
If EC2 get terminated then you can get reason under state transition reason property from console.
Taking snapshot of a EBS volume can cause performance issue due to increase I/O.
User can set DisableApiTermination attribute to avoid accidental termination of EC2.
EC2 EBS IOPS are network attached storage thus it will consume your network bandwidth to connect the the volumes, so if you have a network intensive application it might affect the connection between your instance to your EBS volumes. EC2 Optimized EBS instances are just instances with an "dedicated" connection to EBS volumes, that means that you will have a dedicated amount of bandwidth just for your EBS volumes.
CloudFormation has capability to add bootstrap script in EC2 instance while spinning up but you can not configure a service using it.
During AMI copy AWS does not copy launch permission , user defined tag or Amazon S3 bucket permission .
A windows based instance store backed AMI can not be converted to EBS backed AMI.
EBS pre-warming required if it got created from snapshot.
We can have additional instance store volume in an instance store backed AMI and block mapping device will have information of additional volume.
For Windows instance, the instance console output displays the last three system events log errors.

S3:

S3 ACL permission can not be provide to individual IAM user . It is available for Everyone, Me, Log delivery and any authenticated AWS User.It provide more granular object level access.
for S3 ,If you don't want to use AWS provided AES-256 encryption then need to send encryption key and algorithm for each API call. It is possible to have different encryption for different version of same object .
Use S3 storage analytics to see storage and access pattern to identify less frequent storage.
S3 bucket policy provide bucket level access to all object in bucket by default.
When using SSE, You can not use AWS console to upload an object and request SSE-C. Also can not use console to update an existing object.
To enable folder upload in S3, user need to enable enhance uploaded option from S3 console.
Once you restore object from Glacier , object class always going to stay as Glacier and will not change to RRS or IA.

DB:

When creating event for RDS Snapshots then event category available are creation, deletion, notification and restoration . Backup event is not available . You can enable/disable a event from console to stop notification temporary .
RDS has default maintenance window scheduled to 30 min if user not provide one , and it use for security patching and scale operation only.
Its better to store MSSQL RDS DB log in a DB table and query use that table for log search. You can monitor error log by default and enable slow query and general log if required. Also MSSQL use native mirroring and don't use AWS RDS technology for HA.
PostgreSQL read replica can not be cross region while MySQL can. While creating read replica for RDS it can impact minor performance due to suspension of IO during snapshot creation.
For MySQL always use InnoDB engine for better backup and restore.MyISAM does not support reliable crash recovery mechanism.
RDS metrics in Cloudwatch are DiskQueueDepth (should be 0) , Write Latency, Replica log.

Others:

To implement failover for hardcoded IP address inside VPC - Create a secondary ENI and assign a secondary private IP address to primary ENI that can be move to failover instance.
CloudFormation use wait condition to resolve dependency between resource creation.
You can delete SQS queue even if it has message in it. All message will get deleted automatically. SQS queue sample URL is : https://sqs.us-east-1.amazonaws.com/789211807855/myfirstqueue.
In case of inactivity in SQS , AWS will delete it after 30 days.
Maximum tag per resource allowed 50 , Max key- 127 char , Max value - 255 char , key value are case sensitive. aws: perfix is reserved so you can not use that. You can also allow permission as per tag value for different instances through IAM policy.
User can setup own billing management for AWS by setting up programmatic access to aws billing and parse that according to his need.
Payee account send request to link account to make them part of consolidated billing.
To Scale your internal DNS server , create DNS option set that include DHCP option with domain-name-server=AmazonProvidedDNS and add internal DNS server to option set. You can add up to four DNS in DHCP along with domain name.
one IAM user can be part of 10 groups and an account can have 100 groups, 500 users and 250 roles .

Here are some of the topics from my SysOps Associate exam:

How best to structure billing alarms to ensure spending across an organization is under control
Resource tagging and whether it allows CloudWatch to aggregate across regions. (It doesn't: "CloudWatch does not aggregate data across regions. Therefore, metrics are completely separate between regions." -- Aggregating Statistics Across Instances . However, "If you manage resources in multiple regions, you can create a resource group to view resources from different regions on the same screen." -- What Are Resource Groups?)
What happens to data on instance stores. (It disappears, of course.)
Services with turnkey automated backups. (For example, EC2/EBS snapshots are not automated by AWS.)
Using an EBS volume during a snapshot. (You can use it normally after the snapshot has started; waiting for it to complete is unnecessary.)
Disabling Source/Destination check for NAT instance was a red herring option in one question.
Several questions assuming that I know all about the different previous generation instance types!
Question expecting to know how Medium Utilization Reserved Instances work. (Sure, the current documentation notes that "Light, Medium, and Heavy Utilization Reserved Instances are no longer available for purchase", but I was particularly surprised by this question because it has been almost a year since these were made unavailable.)

Demo Test:

WhizLab practice Test :

This test provides a deep understanding of concept at API level . They really explain answer well with example and evidence .

Here is the link for test :

AWS SysOps Administrator Associate Exam

Read my Lesson Learn:
http://getyouralgorithm.blogspot.com/2017/03/my-aws-certifications-lesson-learn-all.html

AWS Certified Developer Associates Exam Notes

2017-02-10T00:53:00.000-05:00

Amazon Web Services - (AWS) Certification is fast becoming the must have certificate for any IT professional working with AWS. AWS Developer Associates is easiest exam of all three AWS associate exam but you need good amount of study to crack it and make sure you score around 70% to pass the exam.

Exam cost is $150 and need to give in test center, get more detail here :
https://aws.amazon.com/certification/certified-developer-associate/

Read my Lesson Learn:
http://getyouralgorithm.blogspot.com/2017/03/my-aws-certifications-lesson-learn-all.html

Must Watch :

Buy A Cloud Guru Course from Udemy . It cost only $10.
https://www.udemy.com/aws-certified-developer-associate/learn/v4/overview

Recommend subscribe to Linux Academy. Cost $18 /month.

Practice Exam (Must go through before exam):

WhizLab practice Test :
This practice test is very very near to exam and provides a deep understanding of concept at API level . They really explain answer well with example and evidence .

Here is the link for test : AWS Developer Associate Exam

Make sure you understand each and every question in the sample questions provided by AWS. i thoroughly worked them out and found that 9 out of 10 questions in the sample paper come in the main exam. - http://awstrainingandcertification.s3.amazonaws.com/production/AWScertifieddeveloperassociateexamsample.pdf

Exam Notes:

Link to get EC2 Metadata:
http://169.254.169.254/latest/meta-data/

AWS supported SDK -
IOS, Android, Browser (Java scripts)
Java, .NET,
Node.js, PHP, Python, Ruby
Go, C++

SQS -

message oriented API
Message can contain upto 256KB of text, billed at 64KB chunks
Single request can have 1 to 10 messages upto maximum of 256KB payload
Even though there is one message of 256Kb its basically 4 request for billing since (4 * 64KB)
NO ORDER - SQS messages can be delivered multiple times in any order
Design - you can have 2 priority queues for priority based message one for higher and other for lower priority
EC2 instances always poll for messages from the queue (pull from the queue and not push)
Visibility timeout always start from when the application instance polled the message.
Great design - Visibility timeout expires that means there is a failure somewhere since that message was polled but not processed and hence not deleted so other some other process will poll the message again and visibility timeout starts again.
Visibility timeout by default is 30 Seconds up to 12 hour maximum use function (ChangeMessageVisibility) to change visibility timeout .
Maximum long polling timeout 20 seconds (http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html) —ReceiveMessageWaitTimeSeconds to apply long polling period.
Messages in the Queue can be retained for up to 14 days
First 1 million request ares free, then $0.50 PER EVERY MILLION REQUESTS

SNS:

works on a publish - subscribe model, SNS notifies the message, and hence push based approach. Inexpensive pay as you go
CloudWatch or Autoscaling triggers SNS
SNS can notify to Email, Text / SMS, SQS or any HTTP end point.
protocols: HTTP, HTTPS, EMAIL, EMAIL-JSON, SQS or Application - messages can be customized for each protocol
SNS messages are stored redundantly to multiple AZs
SNS Dataformat - JSON (Subject, Message, TopicArn, MessageId, unsubscribeURL etc..)
$0.50 per 1 million SNS request
Different price for different recipient types

to HTTP: $0.06 / 100,000 notifications deliveries
to EMAIL: $2 / 100,000 notifications deliveries
to SMS: $0.75 / 100 notifications deliveries
http://docs.aws.amazon.com/sns/latest/dg/mobile-push-send-devicetoken.html (CreatePlatformEndpoint API)

SWF -

task oriented API
Simple Work flow - human interaction to complete order or collection of services to complete a work order.
Workers - interact with SWF to get task, process received task and return the result
Deciders - program that co-ordinates the tasks, i.e. - ordering, concurrency and scheduling
Workers and Deciders can run independently
TASK is only assigned ONCE and NEVER DUPLICATED (key difference from SQS where messages can be processed multiple times)
SWF Domain - think of it as a container for the work flow. you can register a domain by Console or API
Maximum workflow processing time can 1 year (equivalent seconds) - SQS is 12 hours processing time

CloudFormation:

Use of CFT, Beanstalk and Autoscaling are free but you pay for the AWS resources that these services create.
Fn::GetAtt - values that you can use to return result for an AWS created resource or used to display in output
By Default - rollback everything on error
Infrastructure as a code, Version controlled, declarative and flexible

ElasticBeanstalk:

Its uses ASG,ELB,EC2,RDS,SNS and S3 to provision things.
Environment Tier - Webserver, Worker
Predefined Configurations - IIS, Node.JS, PHP, Python, Ruby, Tomcat, Go, .NET,
preconfigured docker: Glassfish, Python or generic docker
Environment URL - has to be unique
Dashboard - Recent events, Monitor, Logs, Alarms, Upload and Deploy and Configurations
Configuration - Scaling, Instances (DIRTMCG instance types, key pair), Notifications, Software configuration (e.g. PHP.ini), Networking tier (ELB, VPC config), Data tier(RDS)
Environment properties (Access key and secret key as parameters)

DynamoDB:

fast - flexible No sql database - single digit ms latency, fully managed, supports document and key-value (web, gaming, ad-tech, IOT)..
Table, Item (row), attribute (key - value)
Eventual Consistent Reads vs Strongly Consistent Reads
Read Capacity Units, Write Capacity Units (can be scaled up) - push button scalability
Writes are written to 3 different location / facilities/ datacenter (synchronous) - Amazon DynamoDB synchronously replicates data across three facilities in an AWS Region, giving you high availability and data durability.
Two types of primary key -

(1) Single Attribute (think unique id) - Partition Key (Hash Key) composted of 1 attribute (no nesting allowed here) - Partition key will help determine the physical location of data.
(2) Composite key (think unique id and range) - Partition Key(Hash Key) & Sort Key (Range key - e.g date) - composed of 2 attributes - if two data have same partition key (same location) it must have a different sort key, and they will be stored together on single location.

Secondary Indexes

(1) Local Secondary Index - Same Partion Key + Different Sort Key ( can only be created while creating the table, cannot be added/removed or modified later)
(2) Global Secondary Index - Different Partition Key + Different Sort Key ( can be created during the table creation or can be added later or removed / modified later)

DynamoDB Streams - use to capture any kinda modification to the dynamo db table, Lambda can capture events and push notifications thru SES
Table can be exported to csv (either select all items )
Query vs Scan

Query operation finds item in a table using only primary key attribute values , must provide partition attribute name and the value to search for, you can optionally provide a sort key attribute name and value to refine search results (e.g. all the forums with this ID since last 7 days). By default Query returns all the data attributes for those items with specified primary keys. You can further use ProjectionExpression parameter to only return a selected attributes.
Query results are always sorted by the sort key (ascending for both numbers and string by default). To reverse the sort order set the ScanIndexForward parameter to false
By Default Queries are going to be Eventually consistent but can be changed to StronglyConsistent.
Scan operation is basically examines every item - e.g. dumping the entire table, by default Scan returns all the data attributes but we could use ProjectionExpression parameter to only return a selected attributes.
Query operation is more efficient than scan operation
For quick response time design your table in a way that you can use Query Get or BatchGetItem API (read multiple items - can get upto 100 items or up to 1MB of data) ,
Alternatively design your application to use scan operation in a way that minimize impact of your table’s request rate since it can use up the entire table’s provisioned throughput in a single scan operation

DynamoDB Provisioned Throughput calculations:
Items == rows
Read Provisioned Throughput

All units are rounded up to 4KB increments
Eventual Consistent reads (default) consist of 2 reads per second
Strongly Consistent reads consist of 1 read per second

( Size of Read Rounded to nearest 4KB Chunk / 4 KB * no of items ) / 2 <— if eventual consistency
( Size of Read Rounded to nearest 4KB Chunk / 4 KB * no of items ) / 1 <— if strongly consistency

Write Provisioned Throughput

All units are rounded up to 1KB increments
All writes consist of 1 write per second

( Size of write in KB * no of items ) / 1

When you exceed your maximum allowed provisioned throughput for a table or one or more global secondary index you will get 400 HTTP Status code - ProvisionedThroughputExceededException
AssumeRolewithWebIdentity role
Idempotent conditional write
Atomic counters - always need to increment so its not idempotent
if data is critical and no margin of error then must use Idempotent conditional write.
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-tables- Only Tables(256 table per region) and ProvisionedThroughput(80 K read, 80K write per account for US east, 20K for other regions) limits can be increased
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScanGuidelines.html (Reduce Page Size for Scan operation and Isolate Scan Operation)

S3:

secure, durable, highly scalable object store (0 byte to 5TB), universal namespace (must be unique bucket - regardless of regions),object based key value store, VersionID, Metadata, ACL
The total volume of data and number of objects you can store are unlimited. Individual Amazon S3 objects can range in size from 0 byte to 5 terabytes. The largest object that can be uploaded in a single PUT is 5 gigabytes. For objects larger than 100 megabytes, customers should consider using the Multipart Upload capability. it mean the largest single file into S3 is 5G, but after the 5G files are in S3, they can be assembled into a 5T file,
You can use a Multipart Upload for objects from 5 MB to 5 TB in size (Exam question, scenario where more than 5GB file needs to be uploaded)
object based storage vs block based Storage (EFS)
data is spread out in multiple facilities, you can loose two facilities and still have access to files
For PUTS of New Objects (Read after Write Consistency), For Overwrite PUTS and DELETE (Eventual Consistency)
http://docs.aws.amazon.com/general/latest/gr/awsservicelimits.html#limits_s3 ( Number of S3 bucket limit per account — 100)

Storage Tiers/ Class

S3 Standard - Durability (11 9s), Availability (99.99 %) - reliable regular for just about everything
S3 IA (Infrequent Access) - Durability (11 9s), Availability (99.9 %) - accessed every 1 month to 6 months or so (infrequent) but rapid access and low retrieval time (few ms)
S3 RRS(Reduced Redundant Storage)- Durability (99.99%), Availability (99.99 %) - less durability (data that can easily be regenerated - e.g thumbnails) - cheapest of all s3, less fault tolerant then the other two since you are willing to loose the data, reproducible data
Glacier - for archival only (3 to 5 hours restore time)
S3 price - charged for Storage, number of requests, data transfer (tiered so more you use less charge)
All S3 bucket name has to be all lowercase letters
S3 for static website hosting (Static Website Hosting > Enable website hosting) - no dynamic
Any time you create a bucket nothing is publicly accessible / Any time you add an object to a bucket its private by default (you will get 403) > Make the files public (even for public hosting)
every object inside the bucket can have different storage class (S3 standard, S3-IA, S3-RRS) and you can turn on server side encryption (AES - 256)
regular bucket link: https://s3-eu-west-1.amazonaws.com/saurabhtest <— https
bucket with Static website hosting: http://saurabhbitsite.s3-website-eu-west-1.amazonaws.com <— http (has to be for static hosting), you can turn it into SSL / https with cloudfront though
CROS - Cross Origin Resource Sharing help to avoid the use of proxy
Versioning - once enable you cannot disable versioning / although it can be suspend it , if you want to turn it off delete the bucket and recreate (version id)
Once you delete the delete marker, you can get the file back that you have deleted while versioning on every version is stored separately in the bucket for each version / might not be a good choice for cost perspective for large media files., multiple updates use case also not ideal for versioning.
Versioning’s MFA Delete Capability can be used to provide additional layer of security.
Cross Region Replication - (requires versioning enabled on source and destination buckets). you can enable - need source and destination bucket (create a new bucket, source bucket will not show up on drop down of destination).Existing objects will not be replicated, only new objects will be replicated across the region

Lifecycle management in S3

when versioning is disabled

Transition to IA S3 - min 30 days and has a 128KB minimum of object size
Archive to Glacier - min 1 day if IA is not checked, min 60 day if Transition to IA S3 is checked
Permanently Delete - min 2 day if IA is not checked and 1 is selected for Glacier, min 61 day if IA is selected 30, Glacier is selected 60.

when versioning is enabled you have lifecycle management options to take action on previous version as well as current version.

Security and Encryption in S3

by default newly created buckets are private
Access control using Bucket Policies (entire bucket) and ACL(individual objects and folders)
access logs - all the request made to S3 buckets, to another bucket or another account’s S3 bucket

Encryption

In Transit - SSL / TLS
Data at rest

Server Side Encryption

SSE- S3 Server Side Encryption with S3 managed keys, (amazon AES 256 handled for you) - click on the object and encrypt
SSE - KMS - AWS Key management services , managed keys - additional charges / audit trail of keys, amazon manage keys
SSE - C - Server side encryption with Customer provided keys - you manage encryption keys

Client Side Encryption

you encrypt the data on client side and upload to s3

Every non-anonymous request to S3 must contain authentication information to establish the identity of the principal making the request. In REST, this is done by first putting the headers in a canonical format, then signing the headers using your AWS Secret Access Key.

You can insert a presigned url into a webpage to download private data directly from S3.

The object creation REST APIs (see Specifying Server-Side Encryption Using the REST API) provide a request header, x-amz-server-side-encryption that you can use to request server-side encryption.

S3 Transfer Acceleration

Utilize local edge locations to upload content to S3 - incur extra cost
further away you are the more benefit you get (faster)

Storage Gateway
(1) Gateway stored volumes - entire dataset is stored onsite and asynchronously backed up to S3
(2) Gateway cached volumes - Most frequently used data is stored onsite and entire dataset is stored on S3
(3) Gateway Virtual Taped library - Used for backup if you don’t want to use Tapes, like Netbackups etc..

Import Export
Import / Export Disk

Import to S3, EBS, Glacier
export from S3

Import / Export Snowball

Import to S3
Export to S3

S3 stored data in alphabetical / lexigraphical order. so if you want to spread the load across S3, filename should not be similar (Optimize performance)

CloudFront:

Content Delivery Network - edge locations, reduced latency, traffic serves from the closest nodes
Edge locations - content will be cached (over 50), different from region / AZ. TTL (speed of image // media is quicker - first user suffers the performance), can be not only read only (you can write it)
Origin can be - S3, EC2, ELB, Route53 also NON AWS origin server ,
Distribution - name given to the CDN consist of collection of Edge locations
CDN use for Web Distribution or/and RTMP (media streaming / flash) Distribution - for Adobe flash files only
you can have multiple origins of a distribution using Path Pattern (*)
Restrict viewer access by signed URL or Signed Cookies
Restrict content based on geo location (whitelist and blacklist)
Create invalidate - invalidate TTL (you pay for it) like purge in Akamai

VPC -

logical data centers in AWS
VPC can span multiple AZ, but can’t span multiple regions, PEER VPC, but no Transitive Peering
Custom VPC has to be /16 can’t go higher then that /8 is not allowed. Minimum is /28
When you create Custom VPC it creates default security group, default network ACL and default route table., it doesn’t create default Subnet
One Subnet == one AZ, you can have security group spanning multiple AZ, ACL’s span across AZ (assign sg and ACL to two different subnets)
any CIDR block 5 reserved IPs (.0, .1, .2, .3, .255) so for CIRD block /24: 2^8 - 5 = 256 - 5 = 251 available IP address space
when you create internet gateway, by default its detached, attach it to VPC then, only 1 IGW per VPC
When you create a VPC Default Routetable(Main Routable) is created where the default Routes are,
10.0.0.0/16 Local <— all subnets inside VPC will be able to talk to each other but Don’t touch Main route table
Create another routetable for route out to internet (0.0.0.0/0 IGW) <— route out to the internet
Last thing you associate this new route table to one of the subnet which will make it public. (you can enable auto assign public IP for the public subnet)
1 subnet can have 1 routetable
ICMP protocol is for ping / monitor

NAT instance and NAT gateway:

NAT Instance -

disable source / destination check., always behind security group, must be in public subnet, must have an EIP, ,must be a route out of the private subnet to NAT
Increase the instance size if bottleneck
Change the main route table - add a route (0.0.0.0/0 NAT Instance target)
NAT Instance is a single point of failover (put it behind a ASG- Auto Scaling Group),

NAT gateway

- released in 2016 - amazon handled
Amazon maintains it for you, no need to handle yourself. (security patches applied by AWS)
You can just create the gateway and assign EIP (put it in public subnet) (automatically assigned)
Change the main route table - add a route (0.0.0.0/0 NAT gateway target)
No need for disable source/destination check or no need to put it behind a security group - it handles it for you.
Highly available / redundancy no need for ASG.NAT gateways are little bit costly - always use it in production scale automatically up to 10Gbps

ACL vs SG:
Security groups :

SG are statefull - any inbound rule , applies to outbound as well (Only Allow rules)

by default all inbound deny, all outbound allow
can span across AZ

ACL are stateless -

For default ACL, all inbound and outbound rules are allowed by default - associated with all subnets in VPC by default
for Custom ACL, all inbound and outbound traffic is denied by default - not associated with any subnet
1 subnet is only associated with ACL. granular rules for ACLs, numbered rules (recommended steps of 100)
rule no. 99 takes precedence over rule no. 100 (if 99 is blocked and 100 is allowed) 99 will be executed.
Can SPAN across AZ

Ephemeral port - 1024 - 65535 should be allowed to take traffic.
if you want to BLOCK IP address then must use ACL, because security group doesn’t have deny

Bastion -

keep it in public subnet to allow SSH / RDP into instances into private subnets (High availability - Bation in two public subnets and also ASG - Route 53 running Health checks on those Bastion)

VPC Flowlogs -

to capture all the traffic information into logs - logs everything (create IAM role and create cloud watch log group - and log stream)

VPC Cleanup: can’t delete VPC if you have active running instance or ELB is running

Some More Tips:

" For quick response time design your table in a way that you can use Query Get or BatchGetItem API (read multiple items - can get upto 100 items or up to 1MB of data) ," .... should be upto 16MB

Only Tables(256 table per region) and ProvisionedThroughput(80 K read, 80K write per account for US east, 20K for other regions) limits can be increased this should be 40K per table / 80K per account for US-EAST-1 and for others it is 10K/20K .. also number of tables limited to 256 per region per account , so I guess a single account can have more than 256 spanned in different regions. Read http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html

NAT Gateway isn't actually that expensive (except in South America), it's about the price of 2 t2.micros. That said, we had been using a single t2.mico for our NAT and it had been ok (but am interested to see if perf is better with the NAT Gateway)!
The minimum size of an S3 object is 0 Bytes
S3 request rate. Read up on this, it had the answer to 2 questions I saw in the test. 1 was the best way to name your Keys (files). - https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html
Truly understand SCAN versus Query. I was asked a question that had Parallel SCAN or Query and I also had a question about if Strong reads use twice as much as eventual reads. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScanGuidelines.html
2 questions regarding AWS Ident and Access Management. The question about LDAP and application access to S3 on the sample was exact. If you can't see the Security Token Service in SysOps, read this: https://aws.amazon.com/blogs/aws/aws-identity-and-access-management-now-with-identity-federation/
SNS had at least 5 questions - One question asked you about what is sent to SNS via the API, another was about adding existing device tokens to a SNS topic.
Elastic beanstalk - seriously, know what can and can't be provisioned.
Know the SDK OFFICIAL supported languages. Not just the ones that have support.
Know the 200, 300, 4xx, 500 errors - 2 questions on that
Know the ELB better. namely sticky sessions, application sessions and even proper load balancing if it's not working. There was one question about using ELB and a login keeps popping up after users had already logged in.
There was also a question on my exam about what header to use to enable server-side encryption during an S3 upload. Read : http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingRESTAPI.html

AWS Solution Architect Associate Exam Notes:

http://getyouralgorithm.blogspot.com/2017/01/aws-solution-architect-associate.html

AWS SysOps Administrator Exam notes:

http://getyouralgorithm.blogspot.com/2017/02/aws-sysops-associates-exam-notes.html

Read my Lesson Learn:
http://getyouralgorithm.blogspot.com/2017/03/my-aws-certifications-lesson-learn-all.html

AWS Solution Architect Associate Certification Exam Notes

2017-01-30T01:40:00.000-05:00

AWS Solution Architect Associate Certification has significant market value . Exam difficulty is good and it cover almost all major AWS services which organizations use frequently . You need deep understanding of Storage , EC2 and VPC to pass exam.

Read my Lesson Learn:
http://getyouralgorithm.blogspot.com/2017/03/my-aws-certifications-lesson-learn-all.html

Study Course:

ACloudGuru has done good job to create exam focused courses. You can buy this course in Udemy in just $10

Linux Academy Course is really great and just cost $18 per month . Subscribe here

Read Exam Tips:

http://getyouralgorithm.blogspot.com/2016/11/aws-solution-architect-associate_13.html

Practice Test:

WhizLab practice Test :
This practice test is very very near to exam and provides a deep understanding of concept at API level . They really explain answer well with example and evidence .

Here is the link for test : AWS Solutions Architect Associate Exam

Exam Notes:

Scalability blog explain different scaling option

SQS (Simple Queue Service) :

Message oriented API
SQS Message can contain upto 256KB of text, billed at 64KB chunks,
Single request can have 1 to 10 messages up to maximum of 256KB payload
Even though there is one message of 256Kb its basically 4 request for billing since (4 * 64KB)
SQS messages doesn't have any predetermine order and can be delivered multiple times in any order
You can have 2 priority queues for priority based message one for higher and other for lower priority. So you can put paid content in high priority queue and free subscription in low priority queue for your subscription website .
EC2 instances always poll for messages from the queue (pull from the queue and not push)
Visibility timeout always start from when the application instance polled the message.
Visibility timeout expires that means there is a failure somewhere since that message was polled but not processed and hence not deleted so other some other process will poll the message again and visibility timeout starts again.
SQS timeout parameters:
- Default SQS visibility Timeout : 30 sec
- Longest SQS Visibility Timeout : 12 hrs
- Longest SQS long polling Timeout : 20 sec
- Longest SQS configurable message retention period : 14 days
- Default SQS message retention period : 4 days
- Same Name availability for SNS topic deleted : After 30-60 sec
First 1 million request ares free, then $0.50 PER EVERY MILLION REQUESTS

SNS (Simple Notification Service):

SNS works on a publish - subscribe model, SNS notifies the message, and hence push based approach. Inexpensive pay as you go.
CloudWatch or Autoscaling triggers SNS to notify
SNS can notify to Email , Text / SMS, SQS , Lambda any HTTP end point. Supported protocols: HTTP, HTTPS, EMAIL, EMAIL-JSON, SQS or Application - messages can be customized for each protocol
SNS messages are stored redundantly to multiple AZs
SNS Dataformat - JSON (Subject, Message, TopicArn, MessageId, unsubscribeURL etc..)
$0.50 per 1 million SNS request
Different price for different recipient type

to HTTP: $0.06 / 100,000 notifications deliveries
to EMAIL: $2 / 100,000 notifications deliveries
to SMS: $0.75 / 100 notifications deliveries

SWF (Simple Workflow Service) :

SWF is a task oriented API
It can also involve human interaction to complete order or collection of services to complete a work order.
It has three component :

Starter - Any application that can initiate workflow execution
Workers - interact with SWF to get task, process received task and return the result
Deciders - program that co-ordinates the tasks, i.e. - ordering, concurrency and scheduling

Workers and Deciders can run independently
TASK is only assigned ONCE and NEVER DUPLICATED (key difference from SQS where messages can be processed multiple times)
SWF Domain - think of it as a container for the work flow. you can register a domain by Console or API
Maximum workflow processing time can 1 year (equivalent seconds) - SQS is 12 hours processing time

CloudFormation:

Cloud formation is Infrastructure as a code, Version controlled, declarative and flexible
Use of Cloud formation Template, Beanstalk and Autoscaling are free but you pay for the AWS resources that these services create.
Fn::GetAtt - values that you can use to return result for an AWS created resource or used to display in output
By Default - rollback everything on error

ElasticBeanstalk:

Its uses ASG,ELB,EC2,RDS,SNS and S3 to provision things.
Environment Tier - Webserver, Worker
Predefined Configurations - IIS, Node.JS, PHP, Python, Ruby, Tomcat, Go, .NET,
preconfigured docker: Glassfish, Python or generic docker
Environment URL - has to be unique
Dashboard - Recent events, Monitor, Logs, Alarms, Upload and Deploy and Configurations
Configuration - Scaling, Instances (DIRTMCG instance types, key pair), Notifications, Software configuration (e.g. PHP.ini), Networking tier (ELB, VPC config), Data tier(RDS)
Environment properties (Access key and secret key as parameters)

DynamoDB:

This is fast and flexible No sql database - single digit ms latency, fully managed, supports document and key-value (web, gaming, ad-tech, IOT)
Table, Item (row), attribute (key - value)
Eventual Consistent Reads vs Strongly Consistent Reads
Read Capacity Units, Write Capacity Units (can be scaled up) - push button scalability
Writes are written to 3 different location / facilities/ datacenter (synchronous) - Amazon DynamoDB synchronously replicates data across three facilities in an AWS Region, giving you high availability and data durability.
Two types of primary key -

Single Attribute (think unique id) - Partition Key (Hash Key) composted of 1 attribute (no nesting allowed here) - Partition key will help determine the physical location of data.
Composite key (think unique id and range) - Partition Key(Hash Key) & Sort Key (Range key - e.g date) - composed of 2 attributes - if two data have same partition key (same location) it must have a different sort key, and they will be stored together on single location.

Secondary Indexes-

Local Secondary Index - Same Partion Key + Different Sort Key ( can only be created while creating the table, cannot be added/removed or modified later)
Global Secondary Index - Different Partition Key + Different Sort Key ( can be created during the table creation or can be added later or removed / modified later)

DynamoDB Streams

use to capture any kind of modification to the dynamo db table, Lambda can capture events and push notifications thru SES
Table can be exported to csv (either select all items )

Query vs Scan

Query operation finds item in a table using only primary key attribute values , must provide partition attribute name and the value to search for, you can optionally provide a sort key attribute name and value to refine search results (e.g. all the forums with this ID since last 7 days). By default Query returns all the data attributes for those items with specified primary keys. You can further use ProjectionExpression parameter to only return a selected attributes.
Query results are always sorted by the sort key (ascending for both numbers and string by default). To reverse the sort order set the ScanIndexForward parameter to false
By Default Queries are going to be Eventually consistent but can be changed to StronglyConsistent.
Scan operation is basically examines every item - e.g. dumping the entire table, by default Scan returns all the data attributes but we could use ProjectionExpression parameter to only return a selected attributes.
Query operation is more efficient than scan operation
For quick response time design your table in a way that you can use Query Get or BatchGetItem API (read multiple items - can get upto 100 items or up to 1MB of data) ,
Alternatively design your application to use scan operation in a way that minimize impact of your table’s request rate since it can use up the entire table’s provisioned throughput in a single scan operation

DynamoDB Provisioned Throughput calculations

Read Provisioned Throughput

All units are rounded up to 4KB increments
Eventual Consistent reads (default) consist of 2 reads per second
Strongly Consistent reads consist of 1 read per second
( Size of Read Rounded to nearest 4KB Chunk / 4 KB * no of items ) / 2 <— if eventual consistency
( Size of Read Rounded to nearest 4KB Chunk / 4 KB * no of items ) / 1 <— if strongly consistency

Write Provisioned Throughput

All units are rounded up to 1KB increments
All writes consist of 1 write per second
( Size of write in KB * no of items ) / 1

When you exceed your maximum allowed provisioned throughput for a table or one or more global secondary index you will get 400 HTTP Status code - ProvisionedThroughputExceededException
AssumeRolewithWebIdentity role
Idempotent conditional write , Atomic counters - always need to increment so its not idempotent.if data is critical and no margin of error then must use Idempotent conditional write.
Only Tables(256 table per region) and ProvisionedThroughput(80 K read, 80K write per account for US east, 20K for other regions) limits can be increased

S3 (Simple Storage Service):

secure, durable, highly scalable object store (0 byte to 5TB), universal namespace (must be unique bucket - regardless of regions),object based key value store, VersionID, Metadata, ACL
The total volume of data and number of objects you can store are unlimited. Individual Amazon S3 objects can range in size from 1 byte to 5 terabytes. The largest object that can be uploaded in a single PUT is 5 gigabytes. For objects larger than 100 megabytes, customers should consider using the Multipart Upload capability. it mean the largest single file into S3 is 5G, but after the 5G files are in S3, they can be assembled into a 5T file,
You can use a Multipart Upload for objects from 5 MB to 5 TB in size (Exam question, scenario where more than 5GB file needs to be uploaded)
object based storage vs block based Storage (EFS)
data is spread out in multiple facilities, you can loose two facilities and still have access to files
For PUTS of New Objects (Read after Write Consistency), For Overwrite PUTS and DELETE (Eventual Consistency)
http://docs.aws.amazon.com/general/latest/gr/awsservicelimits.html#limits_s3 ( Number of S3 bucket limit per account — 100)

Storage Tiers/ Class

S3 Standard - Durability (11 9s), Availability (99.99 %) - reliable regular for just about everything
S3 IA (Infrequent Access) - Durability (11 9s), Availability (99.9 %) - accessed every 1 month to 6 months or so (infrequent) but rapid access and low retrieval time (few ms)
S3 RRS(Reduced Redundant Storage)- Durability (99.99%), Availability (99.99 %) - less durability (data that can easily be regenerated - e.g thumbnails) - cheapest of all s3, less fault tolerant then the other two since you are willing to loose the data, reproducible data
Glacier - for archival only (3 to 5 hours restore time)
S3 price - charged for Storage, number of requests, data transfer (tiered so more you use less charge)
bucket name has to be all lowercase letters
S3 for static website hosting (Static Website Hosting > Enable website hosting) - no dynamic
Any time you create a bucket nothing is publicly accessible / Any time you add an object to a bucket its private by default (you will get 403) > Make the files public (even for public hosting)
every object inside the bucket can have different storage class (S3 standard, S3-IA, S3-RRS) and you can turn on server side encryption (AES - 256)
regular bucket link: https://s3-eu-west-1.amazonaws.com/saurabhbittest <— https
bucket with Static website hosting: http://saurshtestsite.s3-website-eu-west-1.amazonaws.com <— http (has to be for static hosting), you can turn it into SSL / https with cloudfront though
CORS - Cross Origin Resource Sharing - to avoid the use of proxy

Versioning :
once enable you cannot disable versioning / although it can be suspend it , if you want to turn it off delete the bucket and recreate (version id)

Once you delete the delete marker, you can get the file back that you have deleted while versioning on
every version is stored separately in the bucket for each version / might not be a good choice for cost perspective for large media files., multiple updates use case also not ideal for versioning.
Versioning’s MFA Delete Capability can be used to provide additional layer of security.
Cross Region Replication - (requires versioning enabled on source and destination buckets)
you can enable - need source and destination bucket (create a new bucket, source bucket will not show up on drop down of destination)
Existing objects will not be replicated, only new objects will be replicated across the region

Lifecycle management in S3

(1) when versioning is disabled

Transition to IA S3 - min 30 days and has a 128KB minimum of object size

Archive to Glacier - min 1 day if IA is not checked, min 60 day if Transition to IA S3 is checked

Permanently Delete - min 2 day if IA is not checked and 1 is selected for Glacier, min 61 day if IA is selected 30, Glacier is selected 60.

(2) when versioning is enabled you have lifecycle management options to take action on previous version as well as current version.

Security and Encryption in S3

by default newly created buckets are private
Access control using Bucket Policies (entire bucket) and ACL(individual objects and folders)
access logs - all the request made to S3 buckets, to another bucket or another account’s S3 bucket

Encryption

(1) In Transit - SSL / TLS

(2) Data at rest

Server Side Encryption

SSE- S3 Server Side Encryption with S3 managed keys, (amazon AES 256 handled for you) - click on the object and encrypt
SSE - KMS - AWS Key management services , managed keys - additional charges / audit trail of keys, amazon manage keys
SSE - C - Server side encryption with Customer provided keys - you manage encryption keys

Client Side Encryption

you encrypt the data on client side and upload to s3

Every non-anonymous request to S3 must contain authentication information to establish the identity of the principal making the request. In REST, this is done by first putting the headers in a canonical format, then signing the headers using your AWS Secret Access Key.

You can insert a presigned url into a webpage to download private data directly from S3.

The object creation REST APIs (see Specifying Server-Side Encryption Using the REST API) provide a request header, x-amz-server-side-encryption that you can use to request server-side encryption.

S3 Transfer Acceleration

Utilize local edge locations to upload content to S3 - incur extra cost

further away you are the more benefit you get (faster)

Storage GateWay

(1) Gateway stored volumes - entire dataset is stored onsite and asynchronously backed up to S3

(2) Gateway cached volumes - Most frequently used data is stored onsite and entire dataset is stored on S3

(3) Gateway Virtual Taped library - Used for backup if you don’t want to use Tapes, like Netbackups etc..

Import Export

Import / Export Disk

Import to S3, EBS, Glacier
export from S3

Import / Export Snowball

Import to S3
Export to S3

S3 stored data in alphabetical / lexigraphical order. so if you want to spread the load across S3, filename should not be similar (Optimize performance)

CloudFront

Content Delivery Network - edge locations, reduced latency, traffic serves from the closest nodes

Edge locations - content will be cached (over 50), different from region / AZ. TTL (speed of image // media is quicker - first user suffers the performance), can be not only read only (you can write it)

Origin can be - S3, EC2, ELB, Route53 also NON AWS origin server ,

Distribution - name given to the CDN consist of collection of Edge locations

CDN type can be Web Distribution or/and RTMP (media streaming / flash) Distribution - for Adobe flash files only
you can have multiple origins of a distribution using Path Pattern (*)
Restrict viewer access by signed URL or Signed Cookies
Restrict content based on geo location (whitelist and blacklist)
Create invalidate - invalidate TTL (you pay for it) like purge in Akamai

VPC -

logical datacenters in AWS
Can span multiple AZ, but can’t span multiple regions, PEER VPC, but no Transitive Peering
Custom VPC has to be /16 can’t go higher then that /8 is not allowed
When you create Custom VPC it creates default security group, default network ACL and default route table., it doesn’t create default Subnet
One Subnet == one AZ, you can have security group spanning multiple AZ, ACL’s span across AZ (assign sg and ACL to two different subnets)
any CIDR block 5 reserved IPs (.0, .1, .2, .3, .255) so for CIRD block /24: 2^8 - 5 = 256 - 5 = 251 available IP address space
when you create internet gateway, by default its detached, attach it to VPC then, only 1 IGW per VPC
When you create a VPC Default Routetable(Main Routable) is created where the default Routes are,
10.0.0.0/16 Local <— all subnets inside VPC will be able to talk to each other
Don’t touch Main route table
Create another routetable for route out to internet (0.0.0.0/0 IGW) <— route out to the internet
Last thing you associate this new route table to one of the subnet which will make it public. (you can enable auto assign public IP for the public subnet)
1 subnet can have 1 routetable
ICMP is for ping / monitor

NAT instance and NAT gateway

NAT Instance -

disable source / destination check., always behind security group, must be in public subnet, must have an EIP, ,must be a route out of the private subnet to NAT
Increase the instance size if bottleneck
Change the main route table - add a route (0.0.0.0/0 NAT Instance target)
NAT Instance is a single point of failover (put it behind a ASG),

NAT gateway -

released in 2016 - amazon handled
Amazon maintains it for you, no need to handle yourself. (security patches applied by AWS)
You can just create the gateway and assign EIP (put it in public subnet) (automatically assigned)
Change the main route table - add a route (0.0.0.0/0 NAT gateway target)
No need for disable source/destination check or no need to put it behind a security group - it handles it for you.
Highly available / redundancy no need for ASG.NAT gateways are little bit costly - always use it in production scale automatically up to 10Gbps

ACL vs SG

Security groups are statefull - any inbound rule , applies to outbound as well (Only Allow rules)

by default all inbound deny, all outbound allow
can span across AZ

ACL are stateless -

For default ACL, all inbound and outbound rules are allowed by default - associated with all subnets in VPC by default
for Custom ACL, all inbound and outbound traffic is denied by default - not associated with any subnet
1 subnet is only associated with ACL. granular rules for ACLs, numbered rules (recommended steps of 100)
rule no. 99 takes precedence over rule no. 100 (if 99 is blocked and 100 is allowed) 99 will be executed.
Can SPAN across AZ

Ephemeral port - 1024 - 65535 should be allowed to take traffic.
if you want to BLOCK IP address then must use ACL, because security group doesn’t have deny

Bastion - keep it in public subnet to allow SSH / RDP into instances into private subnets (High availability - Bation in two public subnets and also ASG - Route 53 running Health checks on those Bastion)

VPC Flowlogs: to capture all the traffic information into logs - logs everything (create IAM role and create cloud watch log group - and log stream)

VPC Cleanup: can’t delete VPC if you have active running instance or ELB is running

Some more Tips :

http://getyouralgorithm.blogspot.com/2016/11/aws-solution-architect-associate_13.html

Practice Exam (Must go through before exam):

This Whizlab practice test is very near to exam, they have 300 practice test and you should pass exam if able to score 75% in those test.
If you want to solve more questions then use http://devcertguru.com/certifications/amazon/exam-questions/aws-saa . However , you don't need to solve them , Whizlab practice test is enough to pass exam .

VPC Peer request/response connection calculation:

VPC peering follow star topology .

So if number of VPC are n then total connection request required = (n-1) *2

Connection required for 4 VPC peering = (4-1)*2 = 3*2 = 6
Connection required for 10 VPC peering = (10-1)*2 = 9*2 = 18

CIDR block total IP allocation :

As IP address is 32 bit so to calculate total IP you need to subtract CIDR block from 32 and put result in power of 2 .

For CIDR /22 = 32-22 = 10 and   2 ^10 (2 To the power 10) = 1024

For CIDR /28 = 32-28 = 4 and   2 ^4 (2 To the power 4) =16 Which is minimum IP range available for VPC.

For CIDR /16 = 32-16 = 16 and   2 ^16 (2 To the power 16) =65536 Which is Maximum IP range available for VPC.

For CIDR /24 = 32-24 = 8 and   2 ^8 (2 To the power 8) =256

IOPS and Throughput Calculation :

To calculate disk throughput from IOPS use formula

Throughput /MBps = (IOPS * KB per IO) /1024

If Disk has 20000 IOPS then with 4K Random IOPS performence Throughput will be :

MBps = (20,000 * 4) / 1024
MBps = 80,000 / 1024
MBps = 78.125MB/s

It provides throughput based on standard 4K Random IOPS performance. The 4K Random IOPS indicate how many 4K (4096 byte) operations the drive will handle per second with each block being read or written to a random position. While Throughput provide sustained MB/s rating is the sequential transfer rate the SSD will maintain continuously, such as over a period of 30 seconds.

Vice versa to calculate IOPS from Throughput:

IOPS = (MBps Throughput / KB per IO) * 1024

To convert the 76.2MB/s to IOPS, we perform the following calculation:

IOPS = (76.2 / 4) * 1024
IOPS = 19.1 * 1024
IOPS = 19,558.4

S3 Bucket Stale data possibility:

Stale data (eventually consistent) scenario -

Any operation after DELETE , like LIST or GET after delete
PUT to existing Key , means GET after override PUT

Consistency data scenario -

GET after PUT for new object
DELETE after PUT for new object

DNS Record Type :

SOA - Start of Authority : Mandatory for all zone files having information about Name of the DNS server , admin of zone and TTL value . Name server resolve domain name to IP address through Zone files.
A : Map a host to IPv4 IP address
AAAA : Map a host to IPv6 IP address
CNAME - Canonical Name : Define alias for CNAME for your server
MX - Mail Exchange : Define Mail Server for a domain
NS - Name Server : Direct traffic to DNS
PTR -Pointer : Map an IP address to DNS
SPF - Sender Policy Framework : Used by Mail Server to Stop Spam
TXT - Text : Human readable information about server , network , data center etc.
SRV - Service : Define location information like host name and port number

AWS Trusted Advisor provide recommendation for 4 category :

Cost Optimization
Security
Performance
Fault-Tolerance

Trusted Advisor available through AWS support API.

Elastic Load Balancer (ELB) is configurable for following OSI layer :

Layer 7 : Application Layer - HTTP/DNS
Layer 4 : Transport Layer - TCP/UDP

Kinesis Firehouse Destination for Data ingestion:

Amazon S3
Amazon Red Shift
Amazon Elastic Search Service

Note : Amazon EBS is not in list

My Associates Certification Exams Tips and Notes :

http://getyouralgorithm.blogspot.com/2017/03/my-aws-certifications-lesson-learn-all.html

Refer my Associates Exam notes below:

AWS Certified Solution Architect - Associate:

http://getyouralgorithm.blogspot.com/2017/01/aws-solution-architect-associate.html

http://getyouralgorithm.blogspot.com/2016/11/aws-solution-architect-associate_13.html

AWS Certified Developer -Associate:

http://getyouralgorithm.blogspot.com/2017/02/aws-developer-associates-exam-notes.html

AWS Certified SysOps Administrator Associate:
http://getyouralgorithm.blogspot.com/2017/02/aws-sysops-associates-exam-notes.html

My Professional Certification Exam Tips and Notes:

AWS Certified Solution Architect Professional :

http://getyouralgorithm.blogspot.com/2017/05/exam-notes-for-aws-certified-solutions.html

http://getyouralgorithm.blogspot.com/2017/07/practice-questions-based-exam-notes-for.html

AWS Certified DevOps Engineer Professional :

http://getyouralgorithm.blogspot.com/2017/11/study-based-exam-notes-and-study-plan.html

AWS Certified BigData Speciality:

https://getyouralgorithm.blogspot.com/2018/01/study-based-exam-notes-and-study-plan.html

https://getyouralgorithm.blogspot.com/2018/06/study-based-exam-notes-for-aws.html

AWS Solution Architect Associate Certification Exam Tips

2016-11-13T23:55:00.001-05:00

Exam cost $150. Find more details here :
https://aws.amazon.com/certification/certified-solutions-architect-associate/

Read my Lesson Learn:
http://getyouralgorithm.blogspot.com/2017/03/my-aws-certifications-lesson-learn-all.html

Must Read Exam Notes :

http://getyouralgorithm.blogspot.com/2017/01/aws-solution-architect-associate.html

Tutorial :
ACloudGuru has done great job putting nice course together .You can buy this course in Udemy in just $10.

Exam Tips :

Scalability blog explain different scaling option

1. Know what instance types can be launched from which types of AMIs. In particular, I needed to know that T2 instances must be launched using an HVM AMI.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/virtualization_types.html

2. Know that when you use a bastion host, it lives on the public subnet. Here's a good, brief summary of bastion hosts taken from http://cloudacademy.com/blog/aws-bastion-host-nat-instances-vpc-peering-security/ - "Bastion hosts are instances that sit within your public subnet and are typically accessed using SSH or RDP. Once remote connectivity has been established with the bastion host, it then acts as a ‘jump’ server, allowing you to use SSH or RDP to login to other instances (within private subnets) deeper within your network. When properly configured through the use of security groups and Network ACLs, the bastion essentially acts as a bridge to your private instances via the Internet."

http://cloudacademy.com/blog/aws-bastion-host-nat-instances-vpc-peering-security/

3. Know the difference between Directory Service's AD Connector and Simple AD. "Use Simple AD if you need an inexpensive Active Directory–compatible service with the common directory features. AD Connector lets you simply connect your existing on-premises Active Directory to AWS."

http://docs.aws.amazon.com/directoryservice/latest/admin-guide/what_is.html

4. Know how to enable cross-account access with IAM. Specifically, know that you need to use a permissions policy and trust policy. "To delegate permission to access a resource, you create an IAM role that has two policies attached. The permissions policy grants the user of the role the needed permissions to carry out the desired tasks on the resource. The trust policy specifies which trusted accounts are allowed to grant its users permissions to assume the role. The trust policy on the role in the trusting account is one-half of the permissions. The other half is a permissions policy attached to the user in the trusted account that allows that user to switch to, or assume the role."

http://docs.aws.amazon.com/IAM/latest/UserGuide/idrolesterms-and-concepts.html

5. Have a good understanding of how Route53 supports all of the different DNS record types, and when you would use certain ones over others.

http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-choosing-alias-non-alias.html

https://aws.amazon.com/route53/faqs/

6. Know which services have native encryption at rest within the region, and which do not. For example, Storage Gateway and Glacier do, but DynamoDB, CF, and SQS do not.

http://jayendrapatil.com/aws-storage-gateway/

7. Know which services allow you to retail full admin privileges of the underlying EC2 instances, OR just know which ones definitely do not. Specifically, EMR and Beanstalk do allow this, while RDS, Elasticache, and DynamoDB do not.

8. Know that Elastic IPs are free if you have only one EIP per instance and the associated instance is running. "If you associate additional EIPs with that instance, you will be charged for each additional EIP associated with that instance per hour on a pro rata basis. Additional EIPs are only available in Amazon VPC. To ensure efficient use of Elastic IP addresses, we impose a small hourly charge when these IP addresses are not associated with a running instance or when they are associated with a stopped instance or unattached network interface."

https://aws.amazon.com/ec2/pricing/

9. Know what four high level categories of information Trusted Advisor supplies: Cost Optimization, Performance, Security, and Fault Tolerance

https://aws.amazon.com/premiumsupport/trustedadvisor/

10. Know what might be wrong if you are getting a connection time out error when trying to connect to an instance in your VPC. "You need a security group rule that allows inbound traffic from your public IP address on the proper port, you need a route that sends all traffic destined outside the VPC (0.0.0.0/0) to the Internet gateway for the VPC, the network ACLs must allow inbound and outbound traffic from your public IP address on the proper port," etc.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesConnecting.html#TroubleshootingInstancesConnectionTimeout

11. Know which services can/should be used to preserve state information in which cases (DynamoDB, S3, EFS, or SWF). Read the section on "Stateless Components" in the AWS Cloud Best Practices white paper.

https://d0.awsstatic.com/whitepapers/AWS_Cloud_Best_Practices.pdf

12. Know the various use cases for SWF as a coordination of tasks, including media processing, web application back-ends, business process workflows, and analytics pipelines. Be able to identify multiple possible use cases and eliminate non-use cases.

https://aws.amazon.com/swf/faqs/

13. Understand how you might set up consolidated billing and cross-account access such that individual divisions' resources are isolated from each other, but corporate IT can oversee all of it. (For example, use Consolidated Billing to link the divisions' accounts to a parent corporate account and enable IAM cross-account access for the corporate IT admins in each child account.)

http://jayendrapatil.com/aws-consolidated-billing/

14. For confidence's sake, just be aware that the S3 200 code comes with an MD5 checksum in the success message.

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html

15. Know how you would go about making changes to an Auto Scaling group, fully understanding what you can and can't change. "You can only specify one launch configuration for an Auto Scaling group at a time, and you can't modify a launch configuration after you've created it. Therefore, if you want to change the launch configuration for your Auto Scaling group, you must create a launch configuration and then update your Auto Scaling group with the new launch configuration. When you change the launch configuration for your Auto Scaling group, any new instances are launched using the new configuration parameters, but existing instances are not affected."

http://docs.aws.amazon.com/autoscaling/latest/userguide/LaunchConfiguration.html

16. Know that to run a script upon launching your instance, you would enter the script in the instance's User Data field.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-shell-scripts

17. Know how DynamoDB (durable, and you can pay for strong consistency), Elasticache (great for speed, not so durable), and S3 (eventual consistency results in lower latency) compare to each other in terms of durability and low latency.

https://d0.awsstatic.com/whitepapers/AWS%20Storage%20Services%20Whitepaper-v9.pdf

18. Know the difference between bucket policies, IAM policies, and ACLs for use with S3, and examples of when you would use each. "With IAM policies, companies can grant IAM users fine-grained control to their Amazon S3 bucket or objects while also retaining full control over everything the users do. With bucket policies, companies can define rules which apply broadly across all requests to their Amazon S3 resources, such as granting write privileges to a subset of Amazon S3 resources. Customers can also restrict access based on an aspect of the request, such as HTTP referrer and IP address. With ACLs, customers can grant specific permissions (i.e. READ, WRITE, FULL_CONTROL) to specific users for an individual bucket or object."

https://aws.amazon.com/s3/faqs/

19. Know when and how you can encrypt snapshots. "Public snapshots of encrypted volumes are not supported, but you can share an encrypted snapshot with specific accounts."

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html

20. Understand how you can use ELB cross-zone load balancing to ensure even distribution of traffic to EC2 instances in multiple AZs registered with a load balancer.

http://jayendrapatil.com/tag/elastic-load-balancer/

Practice Exam:

Take Whizlabs quiz, 300 question , really good and cost just $15 , very near to actual exam:

Here is the link for test : AWS Solutions Architect Associate Exam

Read my Lesson Learn:
http://getyouralgorithm.blogspot.com/2017/03/my-aws-certifications-lesson-learn-all.html

CIDR (Classless Inter Domain Routing) Block IP allocation made Simple

2016-11-13T23:38:00.001-05:00

CIDR (Classless Inter Domain Routing) Block made Simple :

CIDR is notation is a simplified way to show specific range of IP. Amazon VPC can use CIDR ranges between /16 and /28 , So what does it means?

There are several article how CIDR block allocate range of IP , here I am juts going to explain how to know calculate total number of allocated for VPC using CIDR notation .

A VPC with CIDR /22 include total 1024 total IP.

As IP address is 32 bit so to calculate total IP you need to subtract CIDR block from 32 and put result in power of 2 .

For CIDR /22 = 32-22 = 10 and   2 ^10 (2 To the power 10) = 1024

For CIDR /28 = 32-28 = 4 and   2 ^4 (2 To the power 4) =16 Which is minimum IP range available for VPC.

For CIDR /16 = 32-16 = 16 and   2 ^16 (2 To the power 16) =65536 Which is Maximum IP range available for VPC.

For CIDR /24 = 32-24 = 8 and   2 ^8 (2 To the power 8) =256

Similarly you can calculate total IP available for any CIDR block.

Note that always 5 IP reserved by AWS .

The first four IP addresses and the last IP address in each subnet CIDR block are not available for you to use, and cannot be assigned to an instance. For example, in a subnet with CIDR block 10.0.0.0/24, the following five IP addresses are reserved:

10.0.0.0: Network address.
10.0.0.1: Reserved by AWS for the VPC router.
10.0.0.2: Reserved by AWS for mapping to the Amazon-provided DNS.
10.0.0.3: Reserved by AWS for future use.
10.0.0.255: Network broadcast address. AWS not support broadcast in a VPC, therefore we reserve this address.

Serverless computing with Amazon Web Servcie - AWS Lambda

2016-09-29T23:08:00.002-04:00

Technology advanced beyond imagination in today's world . Just couple fo year ago you can never think of doing code without maintaining a server . It's really fascinating how today's cloud service making developers life easy and eliminating complication of maintaining server for business thus saving cost .

Here we will learn about AWS Lambda

AWS Lambda :

AWS Lambda is a compute service that runs your code in response to events and automatically manage the compute resources for you , making it easy to build applications that respond quickly to new information . AWS lambda starts running your code within milliseconds of an event such as web page click , in-app activity , comment submission or output from a connected device .

You can also use AWS lambda to create new back-end services where compute resources are automatically triggered based on custom request

You can handle S3 (Simple Storage Service) event using AWS lambda for example if you are uploading product image in S3 bucket and want you code to consume and display image in website. All lambda functions can be monitored through Amazon cloud watch which give a robust way to collect performance metrics and set up alarm .

Uses :

You can trigger manual auto-scaling using Lambda and cloud watch by automating instance tagging. Also can launch small dynamic website feature very quickly where AWS Lambda function writes to a DynamoDB table and publishes a notification to an Amazon SNS topic which end user can consume.

a Lambda serverless architecture processes logs for a variety of uses:

The Lambda function pulls a log stored as soon as it's been stored into a designated Amazon S3 bucket. From there, it could process the data in multiple useful ways.
It could read the log and store important findings into a DynamoDB table, where the data could be read by another application.
It could convert the logs into a different format and store the new log file back into Amazon S3.
It could also send pre-processed data to an Amazon Redshift cluster, which can then use it to perform analytics operations on the logs, providing insight for end-users in a targeted manner.

AWS Lambda could also be used to handle a high volume real-time messaging system.

The Lambda function detects new messages and starts running within milliseconds.
After interpreting the contents of the message, it can then send information as needed to another topic in Amazon SNS, ready for another part of the application to use it.
The function could also send information to Amazon Kinesis, which is designed to handle high throughput streaming data, such as a high volume messaging system.

Take an example of serverless architecture, a live streaming company replaces all of its complicated video transcoding servers with AWS Lambda:

First, the live streaming application encodes the video and uploads it to the closest edge location in Amazon CloudFront. This reduces the latency between the user and the AWS environment.
CloudFront then pushes that content into an Amazon S3 bucket.
When a new stream is detected by the bucket, it sends an event notification to an Amazon SNS topic.
When the topic receives a notification, that triggers the Lambda functions
The Lambda functions pull the video from the Amazon S3 bucket and perform their designated processing.
Each processed video file is stored into its own Amazon S3 bucket.
Those videos are made available via a CloudFront streaming distribution.
Clients can then stream the video from CloudFront to their devices.

Other examples of a serverless architecture serves, delivers, and processes secure surveys, all on managed services.

First, a customer requests the website over HTTPS. The webpage is served directly from Amazon S3.
The customer's survey is submitted via an AJAX call to Amazon API Gateway
Amazon API Gateway logs this to Amazon CloudTrail. In case a survey's results are lost, or one of the AJAX calls includes malicious activity of some sort, these logs may be helpful in identifying and fixing the problem.
Amazon API Gateway then turns the AJAX call into an event trigger for an AWS Lambda function, which pulls the survey data and processes it.
The results of the survey are sent by the AWS Lambda function to an Amazon S3 bucket, where they are secured with Server-Side Encryption.
Metadata from the survey which does not include any personally identifiable information is then written and stored in a DynamoDB table. This could be used for later queries and analysis.

I will update more details uses here as learn more but Lambda has limitation also . Currently , it is applicable to use for small functions which means you need to break your features into smaller functions and if your code execution will go beyond 5 minutes then lambda will automatically terminate them.

You can learn more for AWS Lambda from free available resources :
https://aws.amazon.com/lambda/

Get rid of Windows and Install Ubuntu by creating bootable USB

2016-02-18T01:08:00.001-05:00

My windows 7 got dead slow and it was not speeding up even after running various performance tuning and clean up , so I have to shut it down . And decided to move over and break15 years old relationship with windows . Yeah !!! I fell in love with my 17 inch Mac Book Pro.

While my windows laptop buried for six month , so I thought why not to give it a new life , Specially when I have put good amount of dollars into it to have i7 process , 8 GB RAM and yes!! fancy and fast 256 GB SSD (Yes, HDD or outdated and stone aged now at least for me).

Oh! Forgot to mentioned .. On top of that I forgot my windows admin password so that made matter worst . Now even I can not log-in to my windows 7 .

So I started looking for option and reached out to our best adviser .. and yes!! you got it right , that is google.com :) , Also in my current job at Amazon we are big fan of ubuntu , so I thought lets give a try to Linux GUI and the biggest motivator of all .. its Free!! Free!! Free!!

Now cut the crap , here are the steps to get rid of your slow windows OS and install ubuntu even if you don't have windows password and your system is completely locked up .

Download Ubuntu from http://www.ubuntu.com/download/desktop
I am big fan of LTS version given that it has longer supply of security , update and maintenance , so downloaded Ubuntu 14.04.3 LTS
Again CD/DVD is something out of fashion so I prefer to use USB and created bootable USB using steps listed here http://www.ubuntu.com/download/desktop/create-a-usb-stick-on-windows
I am also putting link to download bootable USB creator software here , its pretty simple and fast http://www.pendrivelinux.com/universal-usb-installer-easy-as-1-2-3/#button
Now its time to install Ubuntu , So reboot your windows laptop and keep pressing F12 Key in case your laptop not starting USB boot automatically
So here you get positive sign , It will ask if you want to try Ubuntu or Install Ubuntu and you will have 10 second time to select install Ubuntu
After that pretty much it is self explanatory and you just need to do next-->next-->next and follow instruction on screen
For those who want to be extra careful , here is the link with all steps and screenshot http://www.ubuntu.com/download/desktop/install-ubuntu-desktop

After 10 minute , Hurry!!! Here you go with super fast and free Ubuntu . Hope it will help everyone and happy Ubuntu .

Link List Operation - From Creation to Deletion

2015-10-04T19:50:00.001-04:00

This program has all link list related operations:

#include "stdafx.h"
#include <iostream>
#include <string>
#include <sstream>

using namespace std;

void traverse();
void insert_from_front();
void insert_from_end();
void insert_after_specified_num();
void delete_from_front();
void delete_from_end();
void delete_specified_node();
void sort_list();

typedef struct node
{
int data;
node *next;
};

node *head = NULL;
int option = 0;

void insert_from_front()
{
node *temp;
temp = new node;

cout << "Enter data: ";
cin >> temp->data;

temp->next = head;
head = temp;
}

void traverse()
{
node *temp;
temp = head;

if(head == NULL)
cout << "End of list.\n";
else{
while(temp != NULL) {
cout << temp->data << " ";
temp = temp->next;
}
}
}
void insert_from_end()
{
node *temp, *temp1;

temp = new node;

cout << "Enter data: ";
cin >> temp->data;
temp->next = NULL;

if(head == NULL)
head = temp;
else{
temp1 = head;
while(temp1->next != NULL)
temp1 = temp1->next;
temp1->next = temp;
}
}

void insert_after_specified_num()
{
int node_num;
node *temp1;
node *temp;

//temp1 = new node;
temp1 = head;

cout << "Enter node number: ";
cin >> node_num;

for(int i=1; i<node_num;i++)
{
if(temp1 == NULL) {
cout << node_num << " does not exist \n";
break;
}
}

temp = new node;
cout << "Enter data for new node:";
cin >> temp->data;

temp->next = temp1->next;
temp1->next = temp;
}

void delete_from_front()
{
node *temp;

if(head == NULL)
cout << "List is empty. \n";
else{
temp = head;
head = temp->next;
delete temp;
}
}

void delete_from_end()
{
node *node2del;
node *temp;
//node2det = head;

if(head == NULL)
cout << "List is empty.\n";
else{
node2del = head;
if(node2del->next == NULL) // single node list
{
delete node2del;
head = NULL;
cout << "Only one was there. Deleted. Now empty list.\n";
}
else{
while(node2del->next != NULL) {
temp = node2del;
node2del = node2del->next;
}
temp->next = NULL;
delete node2del;
}
}
}

void delete_specified_node()
{
node *N2D ; //node to delete
node *temp ; // keep track of the node before deleting
int node_num;

N2D = head;

cout << "Enter node num:";
cin >> node_num;

for(int i=1; i< node_num;i++)
{
if(N2D == NULL) {
cout << node_num << " does not exist \n";
break;
}
temp = N2D;
N2D = N2D->next;
}
temp->next = N2D->next;
delete N2D;
}

void sort_list()
{
node *temp, *temp1, *temp2;
temp = new node;
temp1 = new node;
temp2 = new node;

char choice;

cout << "Ascending order or Descending order? (A/D)";
cin >> choice;

switch(choice)
{
case 'A':
for(temp1 = head; temp1 != NULL; temp1 = temp1->next)
for(temp2 = head; temp2 != NULL; temp2 = temp2->next)
{
if(temp1->data > temp2->data)
{
temp->data = temp2->data;
temp2->data = temp1->data;
temp1->data = temp->data;
}
}
break;

case 'D':
for(temp1 = head; temp1 != NULL; temp1 = temp1->next)
for(temp2 = head; temp2 != NULL; temp2 = temp2->next)
{
if(temp1->data < temp2->data)
{
temp->data = temp2->data;
temp2->data = temp1->data;
temp1->data = temp->data;
}
}
}
}

int main()
{
head = NULL;

do{
traverse();
cout << endl;
cout << "Please select an option: " << endl;
cout << "0. Exit the program." << endl;
cout << "1. Insert a node to the head of the list." << endl;
cout << "2. Insert a node to the end of the list." << endl;
cout << "3. Insert after specified num in the list." << endl;
cout << "4. Delete from front of the list." << endl;
cout << "5. Delete node from end." << endl;
cout << "6. Delete specified_node. " << endl;
cout << "7. Sort the list." << endl;
cout << endl << " >> ";
cin >> option;

switch(option)
{
case 1: insert_from_front();
break;
case 2: insert_from_end();
break;
case 3: insert_after_specified_num();
break;
case 4: delete_from_front();
break;
case 5: delete_from_end();
break;
case 6: delete_specified_node();
break;
case 7: sort_list();
break;
}
}while (option != 0);

return 0;
}