Musings of Rodos

Which instances are available in my region?

2016-03-10T18:17:00.001+11:00

Have you wondered what the best way is to quickly list which instances are available in an AWS region? When the t2.nano was added to the Sydney region this week, I thought exactly this.

First place you would look would be the pricing page and select your region. However this only lists the latest instance types, you have to click through to another page to see the older generations. As a technical person you think "surely this can be done from the CLI"?

First port of call would be the trusty AWS CLI which I just love, its so handy. However there is no command which lets you extract the instances types. The closest you can get is to list the currently available reserved instances, for example

aws ec2 describe-reserved-instances-offerings --query "ReservedInstancesOfferings[?AvailabilityZone=='ap-southeast-2a'] [InstanceType]" --output text --region "ap-southeast-2" | sort -u

Problem with this command is that there may be an instance type which does not have an RI available. When I was doing this t2.nano and hi1.4xlarge were not available for RI in Sydney. So the CLI is probably not the best solution here.

The only other possibility is the relatively new pricing file. This was created to provide a programatic interface to the pricing data instead of the nasty scraping hacks people previously performed. The pricing file for EC2 lists all of the instances available and looks something like this.

{
"formatVersion" : "v1.0",
"disclaimer" : "This pricing list [...]",
"offerCode" : "AmazonEC2",
"version" : "20160126001708",
"publicationDate" : "2016-01-26T00:17:08Z",
"products" : {
"DQ578CGN99KG6ECF" : {
"sku" : "DQ578CGN99KG6ECF",
"productFamily" : "Compute Instance",
"attributes" : {
"servicecode" : "AmazonEC2",
"location" : "US East (N. Virginia)",
"locationType" : "AWS Region",
"instanceType" : "hs1.8xlarge",
"currentGeneration" : "No",
"instanceFamily" : "Storage optimized",
"vcpu" : "17",
"physicalProcessor" : "Intel Xeon E5-2650",
"clockSpeed" : "2 GHz",
"memory" : "117 GiB",
"storage" : "24 x 2000",
"networkPerformance" : "10 Gigabit",
"processorArchitecture" : "64-bit",
"tenancy" : "Shared",
"operatingSystem" : "Windows",
"licenseModel" : "License Included",
"usagetype" : "BoxUsage:hs1.8xlarge",
"operation" : "RunInstances:0002",
"preInstalledSw" : "NA"
}
},

So with some use of jq magic to manipulate the json data we can extract only the instance flavors. Notice I am filtering for the region I am interested in.

curl -s https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/index.json | jq -r '.products[].attributes | select(.location == "Asia Pacific (Sydney)" and .tenancy == "Shared") | .instanceType' | sort -u
c1.medium
c1.xlarge
c3.2xlarge
c3.4xlarge
c3.8xlarge
c3.large
c3.xlarge
c4.2xlarge
c4.4xlarge
c4.8xlarge
c4.large
c4.xlarge
d2.2xlarge
d2.4xlarge
d2.8xlarge
d2.xlarge
g2.2xlarge
g2.8xlarge
hi1.4xlarge
hs1.8xlarge
i2.2xlarge
i2.4xlarge
i2.8xlarge
i2.xlarge
m1.large
m1.medium
m1.small
m1.xlarge
m2.2xlarge
m2.4xlarge
m2.xlarge
m3.2xlarge
m3.large
m3.medium
m3.xlarge
m4.10xlarge
m4.2xlarge
m4.4xlarge
m4.large
m4.xlarge
r3.2xlarge
r3.4xlarge
r3.8xlarge
r3.large
r3.xlarge
t1.micro
t2.large
t2.medium
t2.micro
t2.nano
t2.small

If you are just after the instance families a little more manipulation will get that too.

curl -s https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/index.json | jq -r '.products[].attributes | select(.location == "Asia Pacific (Sydney)" and .tenancy == "Shared") | .instanceType' | sort -u | cut -f1 -d. | sort -u
c1
c3
c4
d2
g2
hi1
hs1
i2
m1
m2
m3
m4
r3
t1
t2

Note you have to use the text descriptor of the region names and not the codes such as "ap-southeast-2". If you want to list all the region descriptors you can do.

curl -s https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/index.json | jq -r '.products[].attributes.location | select(. != null)' | sort -u
Asia Pacific (Seoul)
Asia Pacific (Singapore)
Asia Pacific (Sydney)
Asia Pacific (Tokyo)
AWS GovCloud (US)
EU (Frankfurt)
EU (Ireland)
South America (Sao Paulo)
US East (N. Virginia)
US West (N. California)
US West (Oregon)

There you go, a nice and easy way to pull instance types for a region. Well its easy if you cut and paste, its a relatively long command.

Enjoy.

Rodos

P.S. Thanks to the Solution Architects in the Sydney team at AWS who thought of using describe-reserved-instances-offerings in the CLI, turns out it is not as fruitful as the pricing file though.

Killer interview question on success

2016-02-27T18:26:00.000+11:00

I do a lot of interviewing, close to 400 of them over my 3 years working at Amazon. I am also in the process of becoming one of those mysterious "bar raisers" which you can read a description of in this WSJ article. Whilst I feel I still have a lot to learn about interviewing I find it a fascinating topic.

Previously I wrote some advise on interviews but figured I should write some more. Lifehacker.com.au likes to write articles on Killer Interview Questions and there was one this month that I thought was a good one.

One good question to ask is “Who succeeds in this position?” or, to phrase it carefully, “How would you define success for this position?”

What a great question. You are probably thinking "Great with the answer to this question I can just tell them what they want to hear! For the rest of the interview or for subsequent interviews I will reinforce my alignment to these". If that's what you are thinking you have missed a great insight.

What insight should you gain from this question? The insight you will gain is if you want to work for this company. Often people forget that interviews are a two way thing. The interviewers are trying to understand you and if you will be able to perform the role and be a good cultural fit for the company. Likewise you are trying to figure out if this role and company is something that you are willing to commit a portion of your life too. The answer to this question is going to give you good insight into the role and how to be successful. If characteristics expressed do not align with your personal goals/desires or how you want to work then you really need to consider is this the right job for you.

For example, what if part of the answer was, "To be successful in this role you really need to be curious and have a deep desire to learn new things and experiment on your own with new technologies and concepts. The people who have done very well in this role are like sponges when it comes to technology." That might sound fantastic to you and describe just how you like to operate in your personal and work life. On the other hand it might make you really uncomfortable. Maybe you just finished your MBA after previously doing years of university and you really want to find opportunities to practise your learning rather than embarking on something that is going to require you to learn and develop lots of new knowledge. Maybe you are the type of person that likes formal training and structure and experimenting on your own is just not your thing.

Another example, what if part of the answer was, "Success is easily defined here, if you don't hit your monthly quota you get zero commission. If you miss three in a row you will be moved onto a performance plan. We are a results driven company and there is no easier way to measure success than hitting your quota". Does that sound fantastic to you? Is that an environment where you think you will thrive? Some people might love that environment, they are result driven and like clear measurements. They have a track record of results to know they can achieve the task and they like it when everyone around them is held to the same bar. Of course many people, including me, would not find such an environment a cultural fit. This answer would give me some good insight to determine if this was a role I would be good at and enjoy.

Remember the questions you ask in the interview are important. As advised in the previously mentioned article, have some good questions prepared. But it's important to know why you are asking the question and what you are going to do with the answer. You want to gain insight into the role and the company. Some questions are better at achieving this than others.

Happy interviewing!

Rodos

P.S. Shameless plug. Remember Amazon in always hiring. See http://amazon.jobs/ for open roles in Australia. If you apply for a role in Solution Architecture you may end up having an interview with me! Wouldn't that be fun!

Using an architectural review for improving site reliability

2015-06-16T22:24:00.002+10:00

I stumbled across another AWS Blogger, Eric Hammond who blogs at https://alestic.com

One of the recent things which Eric has done is his Unreliable Town Clock (UTC) which you can use to schedule triggering of AWS Lambda functions. Its a cool idea.

Eric certainly knows what he is doing, he not only launched a service he sat down and ensured "this service is as reliable as I can reasonably make it". No wonder he is a AWS Community Hero!

Of course reliability is only one of the elements of an architectural review of an AWS environment. You should cover off such things as Security, Availability, Scalability and Cost Efficiency. Eric has covered some of this. Check out what he has done to ensure UTC is always up and running, there are some great tips in there.

What if you wanted to do a architectural review of your AWS environment. How would you go about that? What questions would you ask? What things require focus? Maybe post in the comments. Saying I will call my friendly AWS Solution Architect is cheating, although its a great idea.

Two items that will really help you get started with a review are these whitepapers.

What would you do beyond this? Here is some very small things I would investigate.

Auditing. Is CloudTrail, Config and VPC Flows all turned on? Its hard to do debugging or forensics on something in the past when you were not capturing the data. Is all the activity from the instance logged to CloudWatch Logs?
What dependancies are there that might stop a failed employment? That autoscaling group may relaunch an instance if it fails. What AMI is it using? Is it your own AMI sitting in the account or are you launching from a public one? What if the public ones goes away because a new one is released? How is the code deployed into that AMI? Is it baked in, coming from S3, does it need to download software from github, what if it can't?
Monitoring. There are 4 metrics in CloudWatch for SNS. Are there any alarms that could be created to provide alert of failure? What if the number of published messages dropped below a certain rate? An alarm like that could replace what Eric is using Cronitor.io for. You can even create those alarms with CloudFormation!
Turning on MFA is always a great idea.

This is the simplest of examples. For your typical system there are hundreds of review items to assess. But you get the idea.

Doing an architectural review is something you should do periodically in your AWS environment. As AWS keeps releasing new features there is frequently new things you can do to improve your setup.

If only everyone was like Eric! Also, anyone use builds everything in CloudFormation is a winner in my book!

Rodos

Shortcuts in the AWS Console

2015-06-15T21:43:00.000+10:00

Here is something that I did not know you could do for ages, shortcuts inside the AWS console that appear on the top bar.

See this animated Gif for how to add them and then use them. I think the Edit button used to be a lot less obvious.

Its very handy to have the links for your most frequently accessed services always there.

Enjoy.

Rodos

A quick first look at AWS VPC Flow Logs

2015-06-11T23:50:00.002+10:00

I woke up this morning to yet another new AWS feature, VPC Flow Logs, as described by Jeff Barr.

Jeff did a great job of providing an overview so make sure you read that before continuing.

Its really interesting to think what you can do with network flows logs. A lot of Enterprise customers ask for this so they can perform various security activities. Many of those security activities are really not needed in the new world of Cloud. However there are some valid ones that you may want to consider. There are also some good reasons to have flows available so you can perform some troubleshooting of your Security Groups or NACLs.

I suggest people turn them on, capture the data and set a retention period on the destination Cloud Watch Log Group, say 3 days up to 6 months. The data is then there if you need it. Just like Cloud Trail data. Its to late after the fact!

A great little use case would be some general visualization of network flows on a dashboard. Its not real time but its going to give you a general indication. You could analyze the amount of traffic by category, such as incoming, outgoing, cross AZ and within AZ (by reverse engineering the subnet ranges). You could even track it down to traffic to AWS regional based services such as S3. You may want to track these patterns over time, looking for trends. You could also look at top talker hosts internally or externally. I suspect it will be of interest to people at first, and then it will be a colorful screen to show visitors. After all, AWS handles all that heavy lifting of operating and scaling the networking.

Many will be interested in monitoring rejected traffic and if they see a lot if it starting, wonder if there is something else going on they should look at or take precessions on. Generally you probably don't care, nothing to see here, its just dropped traffic.

Be great to see what AWS Partners do in the visualization space, I sense some eye candy coming.

I quickly turned VPC Flow Logs on in my account this morning.

Here is my Cloud Watch console showing the Log Groups.

Notice I have set the expire at 6 months. You can see below that when I look at my Log Group each of my Elastic Network Interfaces (ENIs) is shown.

I have 4 ENIs. Some of those are for my Workspaces instances which is cool.

If I look at the instance I launched this morning by clicking on the eni-981db9fc-all here is the data displayed.

Notice how I have applied a filter. Nice hey. Here is what that filter looks like in that text box.

[version, accountid, interfaceid, srcaddr, dstaddr, srcport, distport=23, protocol, packets, bytes, start, end, action=REJECT, logstatus]

Notice that by putting the field names separated by commas and between brackets you can parse out the text. This is a general feature of Cloud Watch Logs. The field list is in the VPC Flow Logs documentation.

There is lots of filters you can apply, here you can see I am just checking for matching values of a destination port of 23 (telnet) and where the action was to reject the packets. You can see all of those machines which have attempted to telnet into my little server. Thats why it has a correctly configured Security Group!

There is documentation in CloudWatch for the filter patterns syntax. It supports both string and numeric conditional fields. For string fields, you can use = or != operators with an asterisk (*). For numeric fields, you can use the >, <, >=, <=, =, and != operators.

If someone asks you which hosts are communicating with the database at the moment you can quickly jump into the console and answer it by look at traffic to the right port.

The other nice thing you can do is create a metric on this filter to pull out the data. Here is one that creates a metic on the number of bytes accepted as SSH traffic into the ENI.

I created a few of these for my machine, here is the metrics display after I pushed some data its way. I am using the sum function to get the sum of bytes.

During this time period there were a few rejected telnet sessions, some SSH traffic and lots of general traffic. If you can write a filter on it, you can graph it.

Of course this only gets you so far. You have to know the ENI etc.

You will probably want to extract all of the data into something easier. If you want to roll your own a good way would be to create a Subscription on the whole Log Group, see http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/Subscriptions.html and push all the data to a Kinesis stream (it will handle the scale). How do you get data out of Kinesis? Well you use Lambda functions of course, see http://docs.aws.amazon.com/lambda/latest/dg/walkthrough-kinesis-events-adminuser.html. You Lambda function could dump it to S3 and from there you load into Redshift (which can be automated too) or start writing some EMR jobs. Now thats the power of AWS.

Hope that little bit of a first look helps you understand a bit more about VPC Flow Logs. I am really interested to see what people are going to do with it. The main uses will be those occasional operations or forensic events.

Enjoy.

Rodos

P.S. Remember, I might work for AWS but these posts are my own ramblings late at night. Its the geek speaking.

How to rock re:Invent 2015, Rodos style

2015-06-10T22:50:00.000+10:00

I confess I am a conference junkie. Not any conference, but the conference that fertilizes the roots of my current IT thinking.

Back in the day this was VMware. I was a VMworld junkie. I think I may have done six in a row. I collected the t-shirts year after year and even blogged about it. I would stay up till 3am recording nightly video summaries of the days events.

Today Matt Wood (@mza) from AWS, a rock star, did a post on his personal blog on How to Rock re:Invent. He lists things like how to prepare, what gear to bring, what sessions to see. It was music to my ears. Of course you should go an read all of his post.

Matt asked for any other suggestions. Of course I responded with a career limiting move (CLM) and started tweet bombing him with my deep experience and insight, okay random ideas. Maybe not my finest moment but he was in my wheel house!

So here are my random and not as well thought out additions to Matt's list.

Get there

First stop, just get a ticket and a hotel room booked. This is the hard part. One year I was between two jobs and had to take annual leave, pay for my own flights from Australia to the US, scrounge a conference ticket and beg a spare bed in a friends room. People do more to go and watch a Rugby game, so you can do this for something as amazing as re:Invent.

When it comes to hotels try and stay close to the convention. I once stayed at the other end of the strip in Vegas and it was horrible having to walk back and forwards each day. Its so great being able to quickly visit your room to drop something off or pick something up on the way to something else. This means getting your booking in early.

Try and arrive the day before or even two days before. If you want to play tourist, don't do it after the event, you will be exhausted and just want to sleep. Being adjusted to the timezone and having a bit of R&R before the week of full days and little sleep gives you the best conference experience.

Also register the day before the event. Registration always opens the day before at these things and there is less crowds. Its madness the morning of the first day, no matter how well organized it is.

Prepare

There is lots you can do before hand. As Matt says go through the agenda, think about what equipment to bring. I would add to his list business cards (old skill but useful when you are in a hurry and want to pass details). I also don't recommend you bring two laptops!

Bring comfortable clothes and especially shoes, you will be walking a lot! Think about what bag you will carry. You may get a conference bag or you may not. I always prefer my own bag. Ensure you have enough spare room in your luggage for any swag, conference materials or shopping you pick up.

What evening activities will you go to? There is always the exhibition opening which is usually packed and full of people trying to grab swag. If you don't like big crowds and mayhem you may need to skip this one. The conference party is always awesome and not to be missed (I did one year and regretted it). But what other parties are on and can you get an invite? Who are the cool vendors that will be having an event? The more you get into the community the harder this is as there are often multiple on per night and you have to choose or jump from one to another.

Shameless plug for Amazon.com, but this is also something I did before I was an employee. If you live overseas place an order on Amazon.com for all those things you want and get them shipped to the hotel to bring home. I bank up my wish list all year and empty it each trip. You save a lot on shipping and its a bit like Christmas when you arrive. However check with the hotel for extra package handling charges as they can sting you. If you want to bring gifts home for your kids this is a great method, as you are not going to find a lot of gift items around Vegas unless you go to the outlet malls and who wants to do that!

Extra activities

There are extra activities that people often miss. The day before the event starts there are Bootcamps. You pay extra for these but they are really worth it, IMHO. They are either half or a full day and are presented by the best subject matter experts from AWS. Bootcamps have a large hands on component. I ran one my first year at re:Invent and I know people who have run them or will be this year. The instructors put a huge amount into making these relevant and worth while, so check them out.

Certifications can be done onsite, what a great time to get your first or that extra AWS Certification. Go and book in and get one out of the way. Past years there has been a certification lounge where you get a private space, a bit of swag and some snacks and power outlets. So its worth being certified.

There are usually a number of Hackathons. I have some friends who went to them last year and I dropped in and they were amazing. If you want to have fun with others and test your skills these and something to check out.

Hands on labs are fun to do. These are run by training and certification and are a great way to quickly get some experience with an AWS Service or a Partner product.

Engage

Matt called it out as "The Corridor", meaning have those conversations with people that matter. To me this is one of the most valuable things to do. When you sit at a table for breakfast or lunch, talk to the people at the table. Introduce yourself, ask what people do. Ask why they came, ask what they have learnt. You will gather so many nuggets of information, tips and ideas from these conversations. Get out of your comfort zone and engage with people who are just as obsessed with this stuff as you are.

Sessions

People have different approaches to sessions and I have evolved mine over the years. One thing I am positive on is don't miss the keynotes. You want to get in early and get into the main room and not be 15m late and end up in the overflow. You want to experience the vibe, you want to sit with >10,000 other people. You want to live tweet it from in the room!

For the breakout sessions pick what is key to you. Yes they will be recorded and available later online, but you probably will not find the time to do it. Sitting in the sessions gives you time and permission to think about the topic, to digest, to ponder. So go to sessions rather than thinking this is something I can do later. When you are conflicted go to the one that is furthest from your comfort zone or that has a speaker you want to meet. If the session is something you are really familiar with or passionate about you are more likely to watch and digest the recoding afterwards. As Matt mentioned, keep an eye out for those secret ones for new services which may be announced in the keynotes.

Exhibition Hall

This will be huge and take you a LONG time to get around. Plan to visit it each day and cover a portion. Go and see the small vendors, see the startups and not just the big guys. Engage with the vendors and find out how they help their customers and if they could help you. If there is no fit between you and them politely move on, but give them a chance.

There should be a part of the AWS stand that has Solution Architects, Support and Training & Certification. Visit each of these. Ask the Solution Architects the hard questions that you have just not been able to figure out. Check in on a support case, or log one, or just say thanks to the support guys. Maybe ask the support people how best to utilize them or the types of cases you can log. Lastly discuss the training and certification options with the team, what should you consider doing?

Give Feedback

Amazon and AWS is a customer obsessed company. This is your chance to give feedback. If you see a staff member (check their badge and possibly lanyard color) give them feedback both good and bad. If they cover a particular speciality (training, sales, architecture, support etc) and you have interacted with that group tell them about your experience and how they can do better. If you are talking with someone from a service team or someone who knows something about a service (bootcamp instructor or assistant, lab assistant, breakout speaker) then give them feedback on that specific service. Approximately 95% of all those features that AWS releases are based on customer feedback. Staff will be super keen for this feedback and will really appreciate you taking the time.

Hope that helps with some of your re:Invent preparation. If you have your own ideas you may want to leave a comment on this blog entry, hit me and Matt on Twitter or contact Matt (see his post).

AWS Re:Invent 2015 is taking place at The Venetian in Las Vegas from 6th to the 9th October. Registration is now open. If you go, see if you can bump into me and say hello, that would be awesome!

Rodos

Two books on changing the "how"

2015-06-09T20:30:00.000+10:00

In my role I have the great privilege to get to speak to a lot of really interesting people and companies that are on their Cloud journey. The bits and bytes of Cloud are not that hard for most technologists and companies, the hard part is the cultural change.

For a long time I have been recommending that people read the book, "The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win" by Gene Kim, Kevin Behr & George Spafford. You can get it on Kindle for under $10 and you can read it in a weekend.

This book can be a little confrontational to read if you have been in IT for quite some time. Its written as a narrative and you may recognize yourself or others you have closely worked with in the characters. But what this book does do is provide a great vision that will challenge your thinking about how IT can be done in todays world.

Since reading the book I have read quite a few others that have helped in my thinking about how software and architecture should be done. For example "Release It!: Design and Deploy Production-Ready Software (Pragmatic Programmers)" by Michael T. Nygard. Its a little dated but you can tell how it is really sitting on the cusp of public Cloud.

But this last weekend I stumbled across a new book, "The Practice of Cloud System Administration: Designing and Operating Large Distributed Systems, Volume 2" by Thomas A. Limoncelli , Strata R. Chalup & Christina J. Hogan.

I view this book as a great follow up to get one thinking about the "how" of the Phoenix Project. One of the authors, Thomas, worked at Google for 7 years and I see many parallels in the operational aspects described and those which are in place inside Amazon.com and AWS.

The book covers a lot of topics from how to build large scale distributes system to the important bit of how to operate them. Yes there is a chapter on DevOps but this is not a DevOps book.

For people who are keen to understand the possible way to perform operations in the new world of Cloud this book is a great primer and full of great tips and examples from the real world.

An interesting quote, "cloud or distributed computing was the inevitable result of the economics of hardware. DevOps is the inevitable result of needing to do efficient operations in such an environment." (p. 171). Again, lets not overdo the DevOps model but I think the premise is that cloud presents an amazing new way to architect systems. The parallel of that is we need new ways to operate these new architectures and reading this book will help give you insights on how people have been doing that.

If you have read the book, or once you have, would love to hear what you think in the comments.

I have been thinking a lot about operations in the Cloud over the last few weeks so expect some more posts coming up on this topic.

Till the next post.

Rodos

Deep Linking into the AWS Console

2015-06-08T19:55:00.002+10:00

Last year Jeff Barr did a blog post on improvements to the AWS Console. One of these was about "Deep Linking Across EC2 Resources".

"The new deep linking feature lets you easily locate and work with resources that are associated with one another. For example, you can move from an instance to one of its security groups with a single click."

This deep linking is something you can use yourself when creating operational dashboards or your own interfaces. For example here is a little dashboard that I have which displays a quick interactive console for an Auto Scaling Group. You can see that the instance IDs are links, which take you to the specific instance in the EC2 console.

If you look in the AWS Console documentation you will not find any documentation for the deep linking URLs.

Here is a list of what I have used which works, your millage may vary.

Instance : https://console.aws.amazon.com/ec2/home?region=<region>#Instances:search=<instance id>
Volume : https://console.aws.amazon.com/ec2/home?region=<region>#Volumes:search=<volume id>
Load Balancer : https://console.aws.amazon.com/ec2/home?region=<region>#LoadBalancers:search=<lb name>
Auto Scaling Group : https://console.aws.amazon.com/ec2/autoscaling/home?#AutoScalingGroups:id=<autoscaling group name>

There are lots of others. To reverse engineer one you can use the Tag Editor which shows them, see below.

If you are keen to find out about a specific deep link but can't find it, post in the comments.

Rodos

2 minutes and counting

2015-06-05T22:59:00.000+10:00

Do you find that you clean at home the fastest when you know you have an imminent visit by a friend? Its amazing how fast you can whip around the house and get things ready in time for the knock on the door.

Spot instances are one of my many favorite things about AWS. They are for people who really want to cost optimize and have a workload that can match its model. The main element to be aware of with Spot is that it is a market where you place a bid for how much you are willing to pay for the resource, if the market changes and the amount you are willing to pay is no longer high enough your instance will be taken away (or reclaimed if you want a nice way of saying it).

How fast is the instance reclaimed? Well before the start of the year it was reclaimed fast, essentially no notice. Now you get a two minute warning before they are terminated. Enough time to tidy the house before that knock on the door. Many people have not noticed this change as it occurred very early in January whilst many people were away on their Christmas holidays. Who reads the AWS blog on their holidays?

The notice period provides you the opportunity to automate performing tasks such as saving state, coping off data such as logs, or numerous other things which may be part of stopping processing.

How do you know when your two minutes notice period starts? You can monitor for the existence of the metadata field http://169.254.169.254/latest/meta-data/spot/termination-time. Its recommended you check every 5 seconds, but thats really up to you.

If you need to be aware of the status from outside of the instance each spot request has a bid status which will change to marked-for-termination during the notice period. See all the details at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-bid-status.html

You can read more about all of this on the blog announcement. For details on Spot see the documentation at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html.

Rodos

Using AML to predict weather

2015-06-05T00:07:00.001+10:00

You can't really predict the weather can you? Well I have been thinking quite a bit about Lambda and Amazon Machine Learning lately and just yesterday I posted about AWS bloggers. Well todays post combines two of those topics.

Arne Sund at http://arnesund.com just did a post on "Using Amazon Machine Learning to Predict the Weather". Its a good read about how you can get started with AML. I have no idea if this is a good model for weather prediction but could it be any worse? Will let our data scientist friends way in on that one. Certainly for the simple machine learning I have been doing its been working great.

You may want to follow Arne's AWS feed. This is his first post on AWS, nice work and would be great to see some more.

Rodos

AWS Bloggers

2015-06-03T23:53:00.004+10:00

I love reading blogs as I believe that a great way to learn is to listen to people who have spent lots of time investigating something or experiencing it. For good bloggers this is what they do, take their hours of learning and share it with you in a digestible format.

Back in 2008 and onwards I was really into learning about the new world of server virtualization and a great way to do so was through bloggers (see http://vlp.vsphere-land.com for how the space has grown). Since I have been in the AWS world for the last few years I have not seen a lot of individual bloggers out there diving into AWS. Maybe I am just looking in the wrong places.

One day I would love to collate a feed of the AWS specific bloggers that people can follow. However, here are two blogs I do know of that cover interesting stuff that is usually related to AWS. You may want to subscribe to their feeds.

Of course there are the large scale blogs that you are probably already following.

If you know of others, please post in the comments. Even better, if you are using AWS, why not start your own blog and share you experiences.

Cheers

Rodos

Remember to make your Lambda functions idempotent

2015-06-02T23:40:00.000+10:00

Todays post is about an AWS service I have been having some fun with, Lambda.

Essentially Lambda its a service which executes your code within millisecond of an "event" happening. An event may be your own action or it can be triggered by actions in other AWS services such as S3, DyamoDB or Kinesis. The great thing is there is no infrastructure to build or run and you pay only for the requests served and the compute time required to run your code. Billing is metered in increments of 100 milliseconds! Its "way cool". You can read all about it on the product page if you need an introduction. But this post is not about whats so cool about Lambda.

What I wanted to cover was that you need to make sure your functions that you write are idempotent. Idempotency in software "describes an operation that will produce the same results if executed once or multiple times". "It means that an operation can be repeated or retried as often as necessary without causing unintended effects."

Why is this important to remember with Lambda? Well there is some text in the documentation and FAQ that sort of explains why.

From the documentation. [highlight is mine]

Your Lambda function code must be written in a stateless style, and have no affinity with the underlying compute infrastructure. Your code should expect local file system access, child processes, and similar artifacts to be limited to the lifetime of the request, and store any persistent state in Amazon S3, Amazon DynamoDB, or another cloud storage service. Requiring functions to be stateless enables AWS Lambda to launch as many copies of a function as needed to scale to the incoming rate of events and requests. These functions may not always run on the same compute instance from request to request, and a given instance of your Lambda function may be used more than once by AWS Lambda.

Also from the FAQ.

Q: Will AWS Lambda reuse function instances?
To improve performance, AWS Lambda may choose to retain an instance of your function and reuse it to serve a subsequent request, rather than creating a new copy. Your code should not assume that this will always happen.

Today Lambda functions are written in Node.js. Here is my Lambda function which returns Twitter data combined with Amazon Machine Learning Predictions to tell me if those tweets are on topic (aka SPAM) or not. My use case was creating a tweet board that filtered junk message based on machine learning. It actually worked really well. But back to our code, you want to jump right to the end, not need to read it all.

getTweetsError = function (err, response, body) {
    console.log('ERROR [%s]', err);
};

function retrieveATweetPrediction(tweet) {

    // This is an async operation and we are going to have lots. Therefore we
    // will use a promise which we will
    // return for our caller to track. When we do our actual work we will mark
    // our little promise as resolved.

    var deferred = Q.defer();

    var req = aml.predict(
    {       
     MLModelId: '',
     PredictEndpoint: 'https://realtime.machinelearning.us-east-1.amazonaws.com',
     Record: { 
         text: tweet['text'].toString(),
         id: tweet['id'].toString(),
         followers: tweet['user']['followers_count'].toString(),
         favourites: tweet['favorite_count'].toString(),
         friends: tweet['user']['friends_count'].toString(),
         lists: tweet['user']['listed_count'].toString(),
         retweets: tweet['retweet_count'].toString(),
         tweets: tweet['user']['statuses_count'].toString(),
         user: tweet['user']['screen_name'].toString(),
    source: tweet['source'].toString(),
   }
    });

    // We did not pass a function to predict so we can call the .on function and 
    // get access to the complete response data. This allows us to look up the original request and 
    // tie this async call back to our original data. If we call it the normal way we dont have access
    // to that, just the response and can't tie it back!
    req.on('success', function(response) {
     if (response.error) {
      console.log(response.error)
     } else {
      var t = "";
   if (response.data.Prediction.predictedLabel == "0") {
          t += 'ON';
    } else {
       t += 'OFF';
         }
            returnData[response.request.params.Record.id]['prediction'] = t;

    var val = response.data.Prediction.predictedScores[response.data.Prediction.predictedLabel];
    if (val < 0.5 ) {
       val = 1 - val;
    }   
            returnData[response.request.params.Record.id]['probability'] = Math.round(val*100000)/1000;
            deferred.resolve(); // This task can now be marked as done
            
     }
    });
    req.send();
    return deferred.promise;
};

function extractTweets() {

    var deferred = Q.defer();

    twitter.getSearch({'q':'#aws','count': 15}, getTweetsError, 
    
        function (data) {

            var tweets = JSON.parse(data)['statuses'];

            // We need to create a list of tasks as we are going to fire off a bunch of async calls to 
            // do a prediction for each tweet.
            var tasks = [];

            for (i in tweets) {

                var id = tweets[i]['id'];
                returnData[id] = {}; 
                returnData[id]['text']       = tweets[i]['text'];
                returnData[id]['name']       = tweets[i]['user']['name'];
                returnData[id]['screen_name']= tweets[i]['user']['screen_name'];
                returnData[id]['followers']  = tweets[i]['user']['followers_count'];
                returnData[id]['friends']    = tweets[i]['user']['friends_count'];
                returnData[id]['listed']     = tweets[i]['user']['listed_count'];
                returnData[id]['statuses']   = tweets[i]['user']['statuses_count'];
                returnData[id]['retweets']   = tweets[i]['retweet_count'];
                returnData[id]['favourites'] = tweets[i]['favorite_count'];
                returnData[id]['source']     = tweets[i]['source'];
                returnData[id]['image_url']  = tweets[i]['user']['profile_image_url'];

                // The prediction return a promise which we will push into our list of tasks.
                // When the prediction is returned it will mark its little task as resolved.
                tasks.push(retrieveATweetPrediction(tweets[i]));
            }

            // We have a list of tasks which are happening. Lets wait till ALL of them are done.
            Q.all(tasks).then(function(result) { 
                // Woot woot, all predicitons are returned and we have our data!
                // We are therefore resolved ourselves now. Whoever is waiting on us is going to 
                // now get some further stuff done.
                deferred.resolve();
            });
        }
    );
    return deferred.promise;
};

// End of Functions, let look at out main bit of code.

// Setup AWS SDK
var aws = require('aws-sdk');
aws.config.region = 'us-east-1';
var aml = new aws.MachineLearning();

// Setup Twitter SDK
var Twitter = require('twitter-node-client').Twitter;
var twitter = new Twitter({
    "consumerKey": "",
    "consumerSecret": "",
    "accessToken": "",
    "accessTokenSecret": "",
    "callBackUrl": ""
});

// Setup Q for our promises, we have lots of calls to make and we need to track when they are all done!
var Q = require('q');

var returnData = {};

// This is the function required by Lambda
exports.handler = function(event, context) {

    returnData = {}; // We may be reincarnated so ensure we are idempotent 
    
    Q.allSettled([extractTweets()]).then(
        function(result){
            // Return our data an end the Lambda function
            context.succeed(returnData);
        },
        function(reason){
            console.log("Opps : " + reason);
        });

};

See how there are lots of functions then some code which sets up some variables, Q and returnData, and then the main function which Lambda will call when an event occurs, exports.handler. Notice how I am not a great coder and I used a global variable to store some data which is used by all of the functions. Well if exports.handler gets called over and over again in the same environment those global variables will not be re-created or cleared. I did not quite realize this at first and wondered why I was sometimes getting weird data back from Lambda, not always, just sometimes.

To fix my problem I simple ensured that I cleared the key variable each time the handler function was called, so you can see that the first thing it does above is the "returnData = {}; // We may be reincarnated so ensure we are idempotent". Fixed. Of course I know I could just code better, but this was my first ever time writing node.js. You can tell me how to improve my function in the comments.

I will probably do another writeup on my Amazon Machine Learning experiment and how I trained it to filter tweets, it was really easy and I have no servers involved, thanks to Lambda to execute my application logic, so I just have S3, Lambda and AML Live Prediction for a highly scalable site.

Hopefully you won't get caught by the same mistake.

Rodos

interviews

2015-06-01T21:35:00.000+10:00

Wow, its been so long since I did my last blog post. Over the last weeks I have felt that I really miss the days where I was blogging frequently. Hence I decided I would do a month of blogging and force myself to get something small out more often. Lets see how it goes.

Today's topic is interviews.

I see a lot of interview tips on sites like Lifehacker (http://www.lifehacker.com.au/tags/interviews/) such as Killer Questions, why not to Humblebrag or how to answer questions such as Why for a role or What Motivates You. I find these interesting to read and sometimes there is some good insight.

As someone who has done close to 300 interviews at Amazon I thought I would share my very non-official quick list of tips for a interview. Some of them may slant to how Amazon interviews or my personal preferences. I am generally interviewing for technical roles but I also do lots for sales staff, operations and so on.

Here is what I think is important when it come to interviews.

Be yourself - you may have a perception of what the company is looking for but there is little use putting on a show. You may assume wrong and you probably wont be be able to maintain the facade for the duration of your employment. If you never intend to wear a suit, don't wear one to the interview. People say "Dress for the job you want", I say "Be who you are." I am not really talking about dress code here, although that is one element. Show your personality and what you will be like to work with, what you will be like with customers. The interviewers are thinking, "Is this someone I want to spend my days with?", so be yourself.
Be articulate - The interview is a key circumstance where you want to be on your game when it comes to communication. This means body language, pace of speech, active listening and providing short clear answers. Try to ascertain early on the style of conversation the interviewer is using and match this. Is it a friendly conversation, is it a list of quick fire question and answer rounds? Also note that the style may change through the interview. Many people talk way too fast in an interview. If you do this normally then practice slowing down, as this can be hard for people who are listening to you for the first time. Nerves are no excuse IMHO. If you think an interview is stressful, trying having a conversation with a senior executive at a customer when you have a senior executive from the new company with you, that's stress. Listen to cues. If the interviewer says "So tell me about your high level career background. But lets cover this in less than 5 minutes in order to get onto other topics", then you really should answer within 5 minutes. If after 10 minutes you are still going through the subjects you did in high school there is a problem. Listen to the question and provide enough information and colour to answer it, that's all. Don't keep talking on and on and on until the interviewer needs to interrupt you. If more information is required the interview will ask a followup up question. Very long answers to the one question are not adding a lot of value to your answer and removing time for giving answers to other questions which can give greater insight into you and your skills.
Use examples and stories - This may be influenced by my time at Amazon but try to use examples and stories (short ones) for answers. It not only provides interesting colour and is easier to remember but it also provides great insight into what you have actually done and achieved rather than a general assertion. For example, if asked "So how do you learn new things?" you might answer "I like to read books, I love reading. I don't find classes that effective as they move slowly." but compare to "I usually learn through reading. Last year I had to learn Ruby so I read the O'Reilly book on Ruby and then hacked away. After a few months I wanted to go further so read Eloquent Ruby which really helped me understand the nuances of the language." The second version really provides some demonstration of how you applied or practiced whatever the question is about. However, don't be tempted to make something up, a good interviewer will ask you a detailed followup question which may just catch you out.
Do your research and improve during the process - do some research on the company and understand who they are and what they do. As you pick things up during your interviews do more research, dive in more. If you don't know the answer to a question in one interview remember what it was and do some research, you never know you may get a similar question by another interviewer.
Have some good questions to ask - You will often get asked if you have your own questions. In my opinion unless you are now convinced you are not going to take the job there has to be something you want to ask. You can ask questions about the role, the company, the culture. You will be spending a lot of time working for this company and with these people, surely you want to know more about them. Also, avoid common questions if they are not really that meaningful. I started getting a few "What's The Most Frustrating Part Of Working Here?" questions after the Lifehacker post. Its a fine question if you really are interested in the answer, but avoid just asking filler questions.

I can't say I have been the interviewee many times in my career, but I have survived two rounds of Amazon interviews (first externally and a second for an internal role change). What I did find was that if you are a good fit for the role and the company (which is really what you and the employer wants), then the interview should not be like a visit to the dentist. It should be like a first date, a little nerve racking, some fun, a chance to learn more about someone else and yourself, and a good start to what you hope could be a long and rewarding relationship. If the interview is like that dentist visit, maybe you are not made for each other, that's okay.

There you go, first post. Lets hope I can throw out some more random ones this month!

Rodos

P.S. Shameless plug. Remember Amazon in always hiring. See amazon.jobs for open roles in Australia. If you apply for a role in Solution Architecture you may end up having an interview with me! Wouldn't that be fun!

Wake on LAN in AWS

2014-03-06T16:49:00.000+11:00

Someone asked the question. Is Wake-on-LAN supported in Amazon Web Services.

The answer is no. But it also shows not thinking of infrastructure as code.

How would you approach this in AWS? You can fire of an API call to start any instance, but what if you wanted to make this easier? Simply tag your instances with an identifiable tag, such as "WakeOnLAN" and then run the following script (I prefer Ruby).

#!/usr/bin/ruby
require 'aws-sdk'

AWS.regions.sort_by(&:name).each do |region|
  next if region.name.match('cn-')
  puts region.name
  region.ec2.instances.each do |instance|
    if instance.status == :stopped and
       instance.tags.to_h.has_key?('WakeOnLAN')
      puts "\t#{instance.id} started"
      instance.start
    end
  end
end

That results in

[ec2-user@ ~]$ ./wake.rb 
ap-northeast-1
ap-southeast-1
ap-southeast-2
i-7c444f41 started
eu-west-1
sa-east-1
us-east-1
us-west-1
us-west-2
[ec2-user@ ~]$

The script simply goes through all of your instances in each region, finding those that have the WakeOnLAN tag and that are stopped, then starts them. If you run it make sure it has privilege to perform the actions, a role on an EC2 instance makes this easy.

I am a big fan of the AWS CLI too. Here is how to do the same on one line, all be it only within a single region. Its one command line but I have wrapped for formatting.

:~ rodos$ aws ec2 describe-instances 
--query 'Reservations[*].Instances[*].[InstanceId]'
--filters "Name=instance-state-name,Values=stopped"
"Name=tag-key,Values=WakeOnLAN" --output text 
| xargs aws ec2 start-instances --instance-ids
{
    "StartingInstances": [
        {
            "InstanceId": "i-7c444f41", 
            "CurrentState": {
                "Code": 0, 
                "Name": "pending"
            }, 
            "PreviousState": {
                "Code": 80, 
                "Name": "stopped"
            }
        }
    ]
}
:~ rodos$

This uses two very powerful features of the CLI. One is the --query option which lets you pull data out of the returned JSON data. The second is the --filters option which, as the name implies, lets you filter the results based on a lot of criteria. You can see all of the filters for the describe-instances command in the documentation. There are 78 different filters you can use (based on my quick count)!

Enjoy the world of infrastructure as code!

Rodos

Parse OnDemand pricing for AWS in Ruby

2013-08-04T11:45:00.000+10:00

Well I have been doing a LOT more Ruby programming since my last "Hello World" post. I figured that I should probably start sharing some of the more useful code.

First is this class which gives you a quick and easy way to determine the current OnDemand pricing for an instance type. There is a Gem out there for this already but I wanting something that was small which I understood myself (thats how you learn). The class is not really useful by itself, but I am writing some analysis of Spot pricing (will post when its fully complete) and it really need to be able to dynamically get the current pricing.

What that the code does is pull the current pricing from the Amazon web site, which returns a json file. This file has a deep data structure that is really overboard for most usage. Also, the naming for instance types is very different to the normal 'm1.small' format. Therefore the code does a mapping between the two.

All you need to do is copy and paste the Class into your file and then instantiate the instance and use the price method to get a specific value. For example

puts OnDemandPricing.new.price('ap-southeast-2','t1.micro','linux')

Use at your own risk.

#!/usr/bin/ruby

require "rubygems"
require "net/http"
require "uri"
require "json"
require "yaml"

class OnDemandPricing

  REGION_MAPPING = {
    'us-east'    => 'us-east-1',
    'us-west'    => 'us-west-1',
    'us-west-2'  => 'us-west-2',
    'eu-ireland' => 'eu-west-1',
    'apac-tokyo' => 'ap-northeast-1',
    'apac-sin'   => 'ap-southeast-1',
    'apac-syd'   => 'ap-southeast-2',
    'sa-east-1'  => 'sa-east-1'}

  TYPE_MAPPING = {
    'uODI'            => 't1',
    'stdODI'          => 'm1',
    'secgenstdODI'    => 'm3',
    'hiMemODI'        => 'm2',
    'hiCPUODI'        => 'c1',
    'clusterComputeI' => 'cc1',
    'clusterHiMemODI' => 'cr1',
    'clusterGPUI'     => 'cg1',
    'hiIoODI'         => 'hi1',
    'hiStoreODI'      => 'hs1'}

  SIZE_MAPPINGS = {
    'u' => 'micro', 
    'sm' => 'small',
    'med' => 'medium',
    'lg' => 'large',
    'xl' => 'xlarge',
    'xxl' => '2xlarge',
    'xxxxl' => '4xlarge',
    'xxxxxxxxl' => '8xlarge'}

  def autovivifying_hash
    # A little helper to save initialising the hash as we go, make Ruby more like Perl!
    # http://en.wikipedia.org/wiki/Autovivification
    Hash.new do |hash, key|
      hash[key] = autovivifying_hash
    end
  end

  def initialize
    uri = URI.parse(
      "http://aws.amazon.com/ec2/pricing/pricing-on-demand-instances.json")
    request = Net::HTTP::Get.new(uri.request_uri)
    http = Net::HTTP.new(uri.host, uri.port)
    response = http.request(request)
    data = JSON.parse(response.body)

    @price_table = autovivifying_hash

    # Walk the json structure finding the key data and load into an easy to
    # access hash of hash of hashes
    # being price_table[REGION][INSTANCE TYPE][OS linux/mswin] returns price
    data['config']['regions'].each do |reg|
      reg['instanceTypes'].each do |type|
        type['sizes'].each do |size|
          size['valueColumns'].each do |val|
            @price_table[REGION_MAPPING[reg['region']]] \
                        [
                         TYPE_MAPPING[type['type']] +
                         "." + 
                         SIZE_MAPPINGS[size['size']] 
                        ] \
                        [val['name']] = val['prices']['USD']
          end
        end
      end 
    end
  end

  def price (region, instance_type, os)
    return @price_table[region][instance_type][os]
  end
end # Class

#An example
puts OnDemandPricing.new.price('ap-southeast-2','t1.micro','linux')

Enjoy. Any feedback use the comments.

Hello World of AWS API with Ruby

2013-01-31T00:18:00.000+11:00

After years of writing Perl I need to start learning Ruby. The most comprehensive SDK for Amazon Web Services (AWS) looks to be the Ruby SDK. It even contains interfaces for the new Elastic Transcoder service released overnight.

It is pleasing to find just how little it takes to create a "Hello AWS" style Ruby program. For my first test I decide that listing out my S3 buckets would be sufficient.

Here are four simple steps to get you started.

Step 1. Get Ruby on your machine.

Well of course you are using a recent Mac and Ruby is already installed. To confirm

machine:~ rodos$ ruby -v
ruby 1.8.7 (2012-02-08 patchlevel 358) [universal-darwin12.0]
machine:~ rodos$

Step 2. Install the AWS Ruby SDK

This is a simple command

machine:~ rodos$ sudo gem install aws-sdk
Password:

Building native extensions. This could take a while...
Building native extensions. This could take a while...
Successfully installed uuidtools-2.1.3
Successfully installed nokogiri-1.5.6
Successfully installed json-1.7.6
Successfully installed aws-sdk-1.8.1.1
4 gems installed
Installing ri documentation for uuidtools-2.1.3...
Installing ri documentation for nokogiri-1.5.6...
No definition for parse_memory
No definition for parse_file
No definition for parse_with
No definition for get_options
No definition for set_options
Installing ri documentation for json-1.7.6...
Installing ri documentation for aws-sdk-1.8.1.1...
Installing RDoc documentation for uuidtools-2.1.3...
Installing RDoc documentation for nokogiri-1.5.6...
No definition for parse_memory
No definition for parse_file
No definition for parse_with
No definition for get_options
No definition for set_options
Installing RDoc documentation for json-1.7.6...
Installing RDoc documentation for aws-sdk-1.8.1.1...
machine:~ rodos$

Step 3. Create your .rb file with the code.

Create a file with the text below. I named my file "hello-aws.rb". Of course I still use vi for some silly reason to write code! You could always try pico or one of those fancy graphical text editors, even Xcode.

# List you S3 buckets

require 'rubygems'
require 'yaml'
require 'aws-sdk'

AWS.config(
:access_key_id => 'your.access.key.here',
:secret_access_key => 'your.secret.here')

s3 = AWS::S3.new

s3.buckets.each do |bucket|
puts bucket.name
end

Don't forget to enter your access key and secret for your account.

Step 4. Execute your file.

You can now run your code and watch it list your buckets.

machine:~ rodos$ ruby s3list.rb
rodos.singapore.bucket1
rodos.singapore.bucket2
rodos.sydney.bucket1
machine:~ rodos$

There you go, it lists all of my buckets. From here it is developing your knowledge of the AWS SDK for Ruby alongside general Ruby programming skills. As I write some interesting code, I will share my experiences here.

Why don't you go and try your first automation of the Cloud with AWS and Ruby! Its fun and easy.

Rodos

P.S. It is a bad idea to leave your account access key and secret locked away in your code file. I have done it here to show a complete working example in a single file.

AWS re:Invent 2012 session index and links

2013-01-28T18:00:00.002+11:00

It is great that the sessions from Amazon re:Invent 2012 have been made publicly available. The video recordings are on YouTube and the slides are on SlideShare.

Typically I want to search for a session based on the description, the speaker or maybe a company, then quickly view the content. Without a index that links all the sources this can be painful. So for my benefit and especially yours, below is a list of the sessions, speakers, session description and the link to the YouTube and SlideShare content.

Enjoy.

Rodos

P.S. Any errors in the links please post in the comments.

ARC201 - AWS Database Tier Architecture Best Practices by Matt Tavis - Solutions Architect - AWS, Siva Raghupathy - Enterprise Solutions Architect - AWS
AWS provides a rich database environment comprising of a wide selection of In-memory data stores, SQL & NoSQL databases, Search, and Data Warehousing technologies. Plus there are many storage options such as EBS, EC2 Instance Storage, HDFS, Amazon S3, Amazon Glacier etc. with different performance and durability characteristics. AWS Solutions architects have helped a wide variety of customers build successful database solutions on AWS. In this presentation we'll outline various database and storage options and discuss AWS DB Architecture Patterns & Best Practices. You will be able to learn how to architect your database tier by using the right database and storage technologies to achieve the required functionality, performance, availability and durability - at the right cost.
ARC202 - Architecting for High Availability & Multi-Availability Zones on AWS by Attila Narin - Sr Manager, Solutions Architecture - AWS
Amazon Web Services (AWS) provides a platform that is ideally suited for building highly available systems, enabling you to build reliable, affordable, fault-tolerant systems that operate with a minimal amount of human interaction. This session covers many of the high-availability and fault-tolerance concepts and features of the various services that you can use to build highly reliable and highly available applications in the AWS Cloud: architectures involving multiple Availability Zones, including EC2 best practices and RDS Multi-AZ deployments; loosely coupled and self-healing systems involving SQS and Auto Scaling; networking best practices for high availability, including Elastic IP addresses, load balancing, and DNS; leveraging services that inherently are built with high-availability and fault tolerance in mind, including S3, Elastic Beanstalk and more.
ARC203 - Highly Available Architecture at Netflix by Adrian Cockcroft - Director of Architecture, Cloud Systems - Netflix
This talk describes a set of architectural patterns that support highly available services that are also scalable, low cost, low latency and allow agile continuous deployment development practices. The building blocks for these patterns have been released at netflix.github.com as open source projects for others to use.
ARC204 - AWS Infrastructure Automation by Chris Munns - Solutions Architect - AWS, Paul Duffy - Sr. Manager, Solutions Architecture - AWS
So, youve got your solution deployed and have so many things to managenow what? Come to this session to learn how you can scale operations with solutions deployed in the AWS cloud. We take a look at services like AWS CloudFormation and tools like Chef and Puppet. See an overview of these services and tools, and we show you how they might be used in real-life scenarios and how you might incorporate these services and tools into your own environment.
ARC205 - Building Web-Scale Applications With AWS by James Hamilton - Vice President & Distinguished Engineer - AWS, Simon Elisha - Principal Solution Architect - AWS
As both new and established businesses work to increase their customer numbers, revenue and relevance to the market they are working to deliver software that scales larger than ever before. The challenge of being the "victim of your own success" be it from viral marketing, social media or simply dramatic uptake of a new service; is something that troubles the minds of CIOs and Engineers alike. This session will focus on ways to avoid creating "technical debt" during initial development, and will share well established practices and approaches to building applications that can tolerate and revel in the challenges of scaling to "web scale". Working through a range of architectural dimensions, patterns and pithy examples attendees will leave this session with useful ideas on how to design new applications, as well as the "retro-fitting" that can be done to existing applications to enable them to scale on AWS.
ARC206 - Extend Your Existing Data Center to the Cloud with Amazon Virtual Private Cloud by Steve Morad - Solutions Architect - AWS
Amazon Virtual Private Cloud (VPC) is fast becoming the networking option of choice for enterprise and government customers because it provides a powerful set of virtual networking capabilities. VPC allows you to isolate, control, connect, and empower your systems at the network level. Learn the common architectural patterns for building virtual networks in the cloud, interconnecting your virtual and physical networks, and securing your AWS cloud environment. Come learn about the extensive set of features specific to VPC that you should know about before your next cloud deployment.
ARC301 - Intro to Chaos Monkey & the Simian Army by Ariel Tseitlin - Director, Cloud Solutions - Netflix
Join the product and cloud computing leaders of Netflix to discuss why and how the company moved to Amazon Web Services. From early experiments for media transcoding, to building the operational skills to optimize costs and the creation of the Simian Army, this session guides business leaders through real world examples of evaluating and adopting cloud computing.
ARC302 - AWS Cloud Design Patterns (CDP) by Ken Tamagawa - Sr. Manager, Solution Architecture - AWS, Akio Katayama - Solutions Architect - AWS
AWS Cloud Design Patterns (a.k.a. CDP) are generally repeatable solutions to commonly occurring problems in cloud architecting. In this session, we introduce CDP and explain how you can apply CDPs in practical scenarios such as photo sharing, e-commerce, and web site campaigns.
ARC303 - Dissecting an Internet-Scale Application by Paul Duffy - Sr. Manager, Solutions Architecture - AWS, Umesh Sampat - Solutions Architect - AWS , Peter-Mark Verwoerd - Solutions Architect - AWS
In this session, we take an Internet-scale application built on AWS and dissect it. We start by looking at the problem we want to solve and finish with a design. We walk through the various architectural decisions taken for each tier and explain our choices for appropriate AWS services and building blocks to ensure the security, scale, availability and reliability of the application. In addition to learning about the architecture of the application, you see demos along the way.
ARC304 - Solutions in Action by David Rocamora - VP DevOps - Control Group , Ben Cochran - Sr. Software Architect - Autodesk , Jeremy Przygode - CEO - Stratalux, Brian Besterman - MD, Chief Information Officer & Co-Founder - Pronia Medical Systems
You've had a chance to hear from AWS Solutions Architects about how you might architect a solution which would run in the AWS cloud and learned how you might better scale your operations. Come to this session if you'd like to hear some real-world stories from customers such as Autodesk and Pronia and partners such as Control Group and Stratalux. You'll learn how Autodesk has used the AWS cloud to revolutionize the architecture of their solutions to meet their customers' needs and from Stratalux you'll see some pragmatic real world examples for increasing operational efficiency. You'll also hear how Pronia worked with Control Group to deploy a HIPAA compliant application on AWS.
BDT101 - Big Data with Amazon Elastic MapReduce by Peter Sirota - Sr. Manager, Software Development - AWS, Steve Mardenfeld - Data Engineer - Etsy , Jim Blomo - Engineering Manager - Data Mining - Yelp
Big data technologies let you work with any velocity, volume, or variety of data in a highly productive environment. This session seeks to answer questions such as "what is big data," "how can I use unstructured data," and "how can I integrate data collections from different sources" using Hadoop with Amazon Elastic MapReduce. Join general manager of EMR, Peter Sirota, on a journey through real-world use cases of data-driven discovery.
BDT102 - Algorithms, Machines, and Crowdsourcing by Michael Brown - CTO - Comscore , Sharon Chiarella - Vice President - AWS
In this session, join the Vice President of Mechanical Turk to explore how businesses are marrying human judgment with distributed data processing, improving accuracy of Big Data analytics without sacrificing efficiency or scalability. Well highlight real world examples and introduce Mike Brown, CTO of Comscore, to discuss how the combination technologies such as Hadoop and Mechanical Turk are driving large scale systems to cleanse and categorizes business critical data from unstructured and inconsistent data sources.
BDT201 - AWS Data Pipeline: A Guided Tour by Kathryn Shih - Sr. Product Manager - AWS
In this session, we'll review the features and architecture of the new AWS Data Pipeline service and explain how you can use it to better manage your data-driven workloads. We'll then go over a few examples of setting up and provisioning a pipeline in the system.
BDT202 - The Hadoop Ecosystem by Ted Dunning - Architect - MapR , Jon Einkauf - Sr. Product Manager - AWS , Ronen Schwartz - VP Products B2B Data Exchange, VP Marketing Ultra Messaging - Informatica
The Hadoop ecosystem is blossoming. In this session, learn how to take advantage of tools such as Mesos, Spark, Shark and Mahout on Amazon Elastic MapReduce. Senior Product Manager, Jon Einkauf, discusses the optimizations which make Hadoop sing on EMR, and describes how to use different Hadoop distributions and tools such as Hbase and Hparser with your big data analytics pipelines.
BDT203 - Accelerating Research: Spotlight on Science by Deepak Singh - Principal Product Manager - AWS, Chris Dagdigian - Co-Founder, Principal Consultant - Bio Team , Steve Litster - Global Lead for Scientific Computing - Novartis , Michael DeAngelo - Deputy CIO - State of WA
Learn more about AWS, for science! In this session well introduce some of the world changing research being carried out on the AWS Cloud, including computer science, biology, astronomy and high energy physics. Join leaders in the field to hear more about how cloud computing is accelerating research.
BDT204 - Awesome Applications of Open Data by Matt Wood - Product Manager - AWS, Lisa Green - Director - Common Crawl , Ravi K Madduri - Fellow, Computation Institute - Globus Online , Matthew Berk - Founder & CEO - Lucky Oyster, Inc.
Dive into the world of big data as we discuss how open, public datasets can be harnessed using the AWS cloud. With a lot of large data collections (such as the 1000 Genomes Project and the Common Crawl), join this session to find out how you can process billions of web pages and trillions of genes to find new insights into society.
BDT205 - Solving Big Problems with Big Data by David A Patterson - Pardee Professor of Computer Science - AMP Lab, UC Berkeley
The problem of big data is not only that it is capacious, but that it is also heterogeneous, dirty, and growing even faster than the improvement in disk capacity. One challenge is then to derive value by answering ad hoc questions in a timely fashion that justifies the preservation of big data. A group of us from databases, machine learning, networking, and systems just started a new lab at University of California, Berkeley, to tackle this challenge. The AMPLab is working at the intersection of three trends: statistical machine learning (Algorithms), cloud computing (Machines), and crowdsourcing (People). One of the driving applications for the AMP Lab is cancer genomics. Over the next several years, gene-sequencing technologies will begin to make their way into medicine, offering the most complex tests available. This advance brings a new type of data with tremendous promise to help elucidate physiological and pathological functions within the body, as well as to make more informed d
BDT206 - A Seismic Shift in Education by Steven Halliwell - Education & S/L Gov't Director - AWS, Bill Howe - Director of Research, Scalable Data Analytics, eScience Institute - UW, Anant Agarwal - President - EdX , Michael L. Chasen - President and CEO - Blackboard
Join some of the education pioneers who are using cloud computing to drive new methods of delivery and evaluation across the world. Well discuss how the AWS cloud has helped students from all disciplines learn more, faster, using the innovative programs and technologies. Join this session to learn more from EdX, Blackboard and the University of Washington on how the cloud is accelerating modern education and research.
BDT207 - Big Data and the US Presidential Campaign by Miles Ward - Solutions Architect - AWS, Harper Reed - CTO - Obama For America, Leo Zhadanovsky - Director of Systems Engineering - Democratic National Committee, Brian Holcomb - DevOps Engineer - Obama For America, Jay Edwards - Lead Database Engineer - Obama For America, JP Schneider - DevOps Engineer - Obama For America, Scott VanDenPlas - Lead Dev Ops - Obama For America, Ben Hagen - Security Engineer - Obama For America
Obama For America, the organization behind Barack Obamas successful campaign for re-election as President of the United States, designed, built, deployed and now dismantled an election-winning technology system which heavily leveraged Amazon Web Services. Join us to meet the OFA "TECH" team, core members of the visionary group that delivered the applications, analytics, and DevOps tools that made such a big impact. We'll discuss the unique data system "Narwhal" at the center of this operation, their forward-minded use of SQS, RDS, and DynamoDB, and cloud power tools like Auto-Scaling, Asgard, CloudOpt, and New Relic that helped them deliver on election day.
BDT301 - High Performance Computing in the Cloud by Matt Wood - Product Manager - AWS, Dave Fellows - CTO - Green Button , Michael Driscoll - CEO - Metamarkets
Join Matt Wood to discuss the story of high performance computing on Amazon Web Services, from an introduction to EC2 Cluster Compute Instances all the way to tips and tricks, like optimizing your code for the Intel Xeon processor E5 family. We'll welcome guest speakers on stage to discuss real world case studies on driving more teraflops for your applications.
BDT302 - Enterprise HPC in the Cloud: Fortune 500 Use Cases by Jason Stowe - CEO - Cycle Computing , Steve Litster - Global Lead for Scientific Computing - Novartis , Robert Nordlund - Assistant Vice President - The Hartford, Amitabh Shukla - Director of Software and Computer Engineering - Life Technologies, David Chang - Assistant Vice President, Risk Management Actuary - Pacific Life Insurance Company, Kurt Prenger - IT Senior Analyst - Johnson & Johnson, Taylor Hamilton - ITLDP Analyst - Johnson & Johnson
This talk will describe the use of utility supercomputers to solve real world problems for Genentech, Novartis, Schrodinger, and many others in Life Science, Finance, Manufacturing, Energy, and Insurance. We will talk about the latest and greatest workloads that are helping push industries forward through the insights gained from big data and high performance computing, as well as a look at what's coming next.
BDT303 - Data Science with Elastic MapReduce by Kurt Brown - Director, Data Science & Engineering Platform - Netflix
In this talk, we dive into the Netflix Data Science & Engineering architecture. Not just the what, but also the why. Some key topics include the big data technologies we leverage (Cassandra, Hadoop, Pig + Python, and Hive), our use of Amazon S3 as our central data hub, our use of multiple persistent Amazon Elastic MapReduce (EMR) clusters, how we leverage the elasticity of AWS, our data science as a service approach, how we make our hybrid AWS / data center setup work well, and more.
BDT304 - Big Data Masterclass by Peter Sirota - Sr. Manager, Software Development - AWS, Pradeep Ananthapadmanadhan - CTO - Vivaki , Rick Farnell - Co-Founder & President - Think Big Analytics
Learn how engineers at startups and larger enterprises use data to drive greater insight into their operations, customers, and business in this lively discussion of big data techniques and tools. From Hadoop to data warehouses, this panel discusses the tools, techniques, tips, and tricks for building data driven teams and delivering cost optimization at scale.
BDT305 - Transforming Big Data with Spark and Shark by Michael Franklin - Siebel Professor of Computer Science - UC Berkeley , Matei Zaharia - PhD Student, AMP Lab, UC Berkeley - UC Berkeley
The Berkeley AMPLab is developing a new open source data analysis software stack by deeply integrating machine learning and data analytics at scale (Algorithms), cloud and cluster computing (Machines) and crowdsourcing (People) to make sense of massive data. Current application efforts focus on cancer genomics, real-time traffic prediction, and collaborative analytics for mobile devices. In this talk, we present an overview of this stack and demonstrate key components: Spark and Shark.
CPN101 - Revving up Your Application by Deepak Singh - Principal Product Manager - AWS
Amazon EC2 offers a broad array of instance types that provide customers with a varying set of resources for their applications. From the diminutive Micro instance to the powerful Cluster Compute Eight Extra Large instance, AKA "The Beast", customers are presented with a number of options for deploying their applications. This talk dives into the various Amazon EC2 instance families and walks through a variety of use cases that demonstrate the capabilities and ideal use cases for different instance types. You should leave with enough information to understand how to choose the instance type most appropriate to meet the requirements of your applications.
CPN102 - Your First Week with Amazon Elastic Compute Cloud by David Brown - Sr. Manager, Software Development - AWS
Amazon Elastic Compute Cloud (Amazon EC2) provides resizable compute capacity in the cloud. It gives you complete control over your computing resources how you can run your applications in the AWS Cloud. In this session, we explore the fundamentals of Amazon EC2, providing you with all that you need to begin deploying your applications to the cloud. Come prepared with your Amazon EC2 questions and have them answered in this session!
CPN202 - Run More for Less by Dave Ward - Sr. Manager, Amazon EC2 Pricing Services - AWS
Amazon EC2 provides you several pricing options that can help you significantly reduce your overall AWS bill, including On-Demand Instances, Spot Instances, Reserved Instances, and the Reserved Instance Marketplace. This session covers high-level architectures and when to use and not to use each of the pricing models for components of those architectures. We walk through several customer examples to illustrate when to use each pricing option. Additionally, we walk through tools that may be useful to determine when to use each pricing model. This session is aimed at technically savvy managers and engineers who need to reduce their cloud spending.
CPN203 - Saving with EC2 Spot Instances by Dave Ward - Sr. Manager, Amazon EC2 Pricing Services - AWS, Lynwood Bishop - President - MapLarge , Nigel Duffy - CTO - Numerate , Nimrod Hoofien - SVP of Engineering - Ooyala , Chris Doherty - Tech Lead, Video Ingestion & Processing - Ooyala
In this session, we provide an quick overview of how some customers leverage Spot Instances. Join us as we hear from Spot Instance customers, including Ooyala, Numerate, and MapLarge, about how they got started, how they architect for the potential of interruption, how they maximize their savings using Spot, and what best practices they have learned. These customers also provide a high-level architectural overview their Spot solutions including media encoding (Ooyala), drug research (Numerate), and analytics (MapLarge). This session is aimed at customers interested in learning how to maximize their savings using Spot Instances. Come with your questions and get ready to be amazed at how easy it is to save on your Amazon EC2 bill.
CPN204 - Windows on Amazon Elastic Compute Cloud by Tom Rizzo - General Manager, Windows EC2, AWS Windows Business - AWS
If you are interested in learning more about how you can use Windows technologies in the AWS environment, this session is for you! We explore running Microsoft Windows and Windows-based workloads on AWS. We focus on the key topics that will help you get started on Windows to expert topics like building applications using .NET and Visual Studio and everything in-between. Filled with tons of demos, this session shows you firsthand how you can use Windows to bridge your on-premises network with AWS and quickly deploy enterprise workloads such as Exchange Server, SQL Server, SharePoint Server, and Windows ISV applications. Come prepared with all your Windows questions and get them answered at the session!
CPN205 - Zero to Millions of Requests by Khawaja Shams - Software Engineer - NASA , Spencer Dillard - Sr. Product Manager - AWS , Brett George - Engineering Applications Software Engineer - NASA
Regardless of the size of your application, you need to ensure it is scalable and highly available. You want a simple, cost-effective solution that makes it easy to securely run your site on the Amazon EC2 platform. This session starts with an overview of Elastic Load Balancing (ELB), its key features, and how to get started. Learn how ELB can fit into your application architecture and best practices for managing load balancers in EC2. NASA / JPL presents a real world example of how ELB enabled millions of users to view the Curiosity Mars rover landing.
CPN206 - Learning From the Masters by Deepak Singh - Principal Product Manager - AWS, Adrian Cockcroft - Director of Architecture, Cloud Systems - Netflix , Peter Esposito - Systems Manager - USTA , Steve Litster - Global Lead for Scientific Computing - Novartis
Please join us for a conversation with Adrian Cockroft (Netflix), Pete Esposito (USTA) and Stephen Litster (Novartis) as they share insights into their organizations journey to the cloud. You will learn more about lessons they have learned from successes and failures as they made Amazon EC2 a component of their IT infrastructure. Find out how Netflix scales their infrastructure and enables teams to innovate, how the USTA made its journey to the cloud, and how Novartis takes advantage of the elasticity and scale of Amazon EC2 to find new drug targets.
CPN207 - Virtual Networking in the Cloud by Eric Schultze - Principal Product Manager - AWS, Steve Morad - Solutions Architect - AWS
Take Amazon EC2 to the next level and create a virtual network in the AWS cloud using our API-defined networking solutions. Learn how to create networks that closely resemble those used in a traditional data center, enhance your knowledge of elastic network interfaces and multiple IP addresses for EC2 instances, and learn how to leverage egress filtering and network ACLs for an additional layer of security for your network. In addition to discussing virtual network security appliances, internal load balancing, and site to site VPN connectivity, we also discuss the past, present, and future for Amazon virtual networking.
CPN208 - Failures at Scale and How to Ignore Them by James Hamilton - Vice President & Distinguished Engineer - AWS
At scale, rare and unexpected events will happen. Things eventually will go wrong. This talk dives into what can go wrong at scale and how to architect applications to ride through disaster obliviously. Well talk about AWS infrastructure design including Regions and Availability Zones and show how applications can be written and operated to best exploit this industry-unique infrastructure redundancy model. Believing that experience is one of the best teachers, we will go through some of the more interesting and educational industry post mortems including some experienced at AWS to motivate these application design decisions and show how they can mitigate the damage of the truly unexpected.
CPN209 - Your Amazon Linux AMI by Max Spevack - Manager, Linux Kernel & Operating Systems - AWS
If you are a Linux user in Amazon EC2, this is a session you want to attend. Join us for a discussion related to the Amazon Linux Amazon Machine Image (AMI). Leave the session with an understanding of why the Amazon Linux AMI exists, how its packages and repositories are composed, and information about its rolling release cycle and security updates. Additionally, we discuss the niche that the Amazon Linux AMI fills in the larger Linux-in-EC2 space. Come with your questions, feedback, and requests related to the Amazon Linux AMI.
CPN210 - Defining an Enterprise Cloud Strategy by George Reese - CTO - Enstratus
Few terms are more confusing than the term "cloud." While we've moved beyond the age of "defining the cloud," there's still a significant amount of confusion in understanding the role of the public cloud in an enterprise IT infrastructure. This presentation defines the elements of a mature enterprise cloud computing strategy that includes all components of a hybrid cloud, how to build out an integrated public/private infrastructure, and strategies for when and where to deploy new systems, and when it makes sense to migrate existing systems.
CPN211 - My Data Center Has Walls that Move by Deepak Singh - Principal Product Manager - AWS
How do you think about computing resources in a world where you can launch and terminate computational capacity in minutes? Amazon EC2 provides a powerful platform to access vast computational resources at the click of a button or a simple API call. It is also very different from operating your own data center or having to managed fix assets in a co-location facility. This talk walks you through examples of how the cloud enables more efficient capacity planning, provides guidance in how developers and organizations can manage thousands of instances efficiently, and highlights tools that make it easy for you to plan your capacity needs, even when those needs might require you to provision the equivalent of a small data center at short notice.
CPN301 - The Best Amazon EC2 Features You Never Knew About by Deepak Singh - Principal Product Manager - AWS, Miles Ward - Solutions Architect - AWS, Scott VanDenPlas - Lead Dev Ops - Obama For America
You might think you make Amazon EC2 dance to the strokes of your keyboard and bend to the will of a mouse click. If you think you know everything about Amazon EC2, think again. Learn some advanced EC2 tricks that will save you time, money and development effort. Secure your automation, scale naturally with ease, flexibly redistribute load and GO FAST!
CPN401 - Packet Plumbing in Amazon Virtual Private Cloud by Kevin Miller - Software Development Manager - AWS
In this session, we put on our network tool belt and explore the depths of Amazon Virtual Private Clouds API-defined networking! Learn how to isolate and fix networking problems; we cover such things as instance routing tables, Elastic Network Interfaces, VPC subnets and route tables, network ACLs, security groups, route propagation, and remote connectivity via VPN and Direct Connect. Join as we discover how to build and troubleshoot VPC networks meeting complex requirements, and pick up ideas for some unexpected things you can do. This session is aimed at engineers that understand the fundamentals of VPC networking and are looking for a deeper dive of how VPC engineers diagnose network problems.
DAT101 - Understanding AWS Database Options by Sundar Raghavan - General Manager - AWS, Andy Skalet - CTO and Co-Founder - BrandVerity , Jay Edwards - Lead Database Engineer - Obama For America, Kimo Rosenbaum - Senior Systems Administrator - Edmodo
When you're handling big data in the modern world, you will come to a point where you can't just pick a one size fits all approach anymore. However, to get the results you want, you also dont have to spend big money on fire breathing hardware, or expensive software. AWS offers a beautiful array of open and commercial database choices, from do-it-yourself to fully managed services which handle scaling, and gives you powerful tools to choose the right architecture. You could choose from MySQL, RDS, Oracle, SQL Server, MongoDB, DynamoDB, Cassandra, ElastiCache, Redis, and SimpleDB, and our customers use them for different use cases. Each has different strengths, and this session highlights when you would want to choose each, with examples of how we use each to solve our big data challenges and why we made those decisions. We profile the some of the choices available to you - MySQL, RDS, Elasticache, Redis, Cassandra, MongoDB and DynamoDB and three customer case studies on RDS, ElastiCache and DynamoDB.
DAT102 - Introduction to Amazon DynamoDB by Dave Lang - Sr. Product Manager, Amazon DynamoDB - AWS, Eric Nadalin - Co-Founder & CTO - Nexmo, Eddie Dingels - Lead Software Engineer - Earth Networks
Learn why Amazon DynamoDB is the fastest-growing service in AWS history. DynamoDB is a NoSQL database service that lets you scale from one to hundreds of thousands of I/Os per second (and beyond) with the push of a button. It's designed to give you scalability and high performance with minimal administration and enables you to scale your app while keeping costs down. You also learn about the services design principles, its history, and about how some of our customers are using DynamoDB in their applications.
DAT103 - Amazon Redshift: Amazon's New Data Warehousing Service by Anurag Gupta - Director, Database Engines - AWS
Amazon Redshift is a fast and powerful, fully managed, petabyte-scale data warehouse service in the cloud. Starting today, you can sign up for an invitation to the limited preview of the service. Come to our session for an overview of the service, how it delivers fast query performance on data sets ranging from hundreds of gigabytes to a petabyte or more, and its pricing.
DAT201 - Migrating Databases to AWS by Grant McAlister - Senior Principal Engineer - AWS, Rodney Grilli - Solution Architect - College Board
In this session, learn how to move your existing database applications to the cloud. We cover the best practices for planning your migrations, moving your data over, sizing your AWS deployment appropriately, and minimizing downtime. You also hear from some of our customers who have successfully migrated their applications about the techniques they used and the reasons they moved onto the cloud.
DAT202 - Optimizing your Cassandra Database on AWS by Gregg Ulrich - Manager, Cassandra DevOps - Netflix , Ruslan Meshenberg - Director, Platform Engineering - Netflix
For a service like Netflix, data is crucial. In this session, the Director of Cloud Platform Engineering at Netflix details how they chose and leveraged Cassandra, a highly-available and scalable open source key/value store. In this presentation they discuss why they chose Cassandra, the tools and processes they developed to quickly and safely move data into AWS without sacrificing availability or performance, and best practices that help Cassandra work well in AWS.
DAT203 - Optimizing Your MongoDB Database on AWS by Miles Ward - Solutions Architect - AWS, Jared Rosoff - Director of Product Marketing - 10gen
MongoDB is one of the fastest growing NoSQL workloads on AWS due to its simplicity and scalability, and recent product additions by the AWS team have only improved those traits. Join us for a deep-dive on MongoDB best practices, including installation, configuration, orchestration, performance, and durability optimization, as well as operational management using tools from AWS and 10gen.
DAT301 - Accelerating Amazon Relational Database Service Performance with Amazon ElastiCache by Rahul Pathak - Sr. Product Manager - AWS, Ori Zaltzman - Founder & CTO - Gogobot.com, Khaled Alquaddoomi - SVP of Technology - HealthGuru
Learn how you can use Amazon ElastiCache to easily deploy a Memcached-compatible, in-memory caching system to speed up your application performance. We show you how to use ElastiCache to improve your application latency and reduce the load on your database servers. We'll also show you how to build a caching layer that is easy to manage and scale as your application grows.
DAT302 - Under the Covers of Amazon DynamoDB by Matt Wood - Product Manager - AWS, Michael Laing - Systems Architect - New York Times , Andrew Canaday - Lead Software Engineer - New York Times
Learn about the thought and decisions that went into designing and building DynamoDB. We'll talk about its roots and how we can deliver the performance and throughput you enjoy today. Well also show you how to model data, maintain maximum throughput, and drive analytics against the data with DynamoDB. Finally, you'll hear from some of our customers on how they've built large-scale applications on DynamoDB and about the lessons they've learned along the way.
DAT303 - Amazon Relational Database Service Best Practices by Grant McAlister - Senior Principal Engineer - AWS, Eli White - Founding Partner & CTO - mojoLive
Learn how to set up, operate, and scale mission critical MySQL, Oracle, and SQL Server database deployments using Amazon RDS. Grant McAlister, our senior principal engineer, shares best practices for deploying mission critical systems on Amazon RDS. We show you how to architect for security, durability, and high availability. You also learn how easy it is to scale your compute capacity, storage, and the IOPS associated with your database. Finally, some of our customers share what they've learned about building scalable applications on RDS.
ENT101 - Embracing the Cloud by Yury Izrailevsky - Vice President, Cloud Computing and Platform Engineering - Netflix , Neil Hunt - Chief Product Officer - Netflix
Join the product and cloud computing leaders of Netflix to discuss why and how the company moved to Amazon Web Services. From early experiments for media transcoding, to building the operational skills to optimize costs and the creation of the Simian Army, this session guides business leaders through real world examples of evaluating and adopting cloud computing.
ENT102 - Application Rationalization by Michael Gephart - Business Development Manager - AWS, Srikanth Geddada - Head - Public Cloud Infrastructure - Wipro, Erica Eatmon - Strategic Initiatives Leader - McGraw Hill, Harsha Rao - Principle - Booz Allen Hamilton, Oliver Alvarez - Lead Enterprise Security Architect - International Finance Corporation
Understanding the requirements and dependencies of your existing applications is an important component of a successful migration to Amazon Web Services. In this session, hear best practices from customers in rationalizing your applications to understand the easiest path to migrate and take advantage of the on-demand, elastic infrastructure of the AWS Cloud.
ENT103 - Making the Case for Cloud by Markku Lepisto - Principal, Cloud Computing - Nokia , Jinesh Varia - Manager, AWS Evangelism - AWS, Jay Chakrapani - VP & General Manager, Digital - McGraw Hill
In this session, AWS technology evangelist Jinesh Varia discusses common best practices for adopting cloud computing in your business. Tap into the shared learning of hundreds of thousands of customers, as we talk about how to create the right technical and business policies for adopting the cloud, migration hints and tips, and how to develop employees skills around the cloud computing ecosystem. This session includes customer examples from McGraw Hill and Nokia Siemens on how they successfully grew their usage of cloud computing while staying agile and keeping costs low.
ENT201 - How Much Can You Save with the Cloud? by Dan Rogers - Head of Global Demand Gen - AWS, Vladimir Mitevski - Vice President, Product Management Core Services - Thomson Reuters , Kris Bliesner - CEO & Co-Founder - 2nd Watch
There isnt an IT department out there that isnt under pressure to reduce costs. For thousands of enterprises, the AWS cloud has become part of that lower cost strategy. But how much could you really save with AWS? Where will those savings come from, and how does shifting to a model where you pay only for what you use impact your IT spend? In this session, we are joined by Kris Bliesner, chief executive officer of 2nd Watch, who have successfully helped over one hundred organizations reduce their IT costs. We share detailed best practices on how to calculate detailed apples-to-apples comparisons, how to build the models, and where to look to identify the biggest cost saving opportunities. We are also joined by Vladimir Mitevski, Vice President Product Management Core Services at Thomson Reuters who will walk through by line item how and where their actual operating and capital expenses changed when they migrated to the cloud, so you can learn from their experiences, and take home some
ENT202 - 69% and Falling: Lowering the TCO of Enterprise Apps by Ralph K. Treitz - CEO & Co-Founder - VMS
In this presentation, Ralph Treitz from enterprise consulting legend, VMS AG, dives deep into a detailed cost analysis of running SAP on AWS using best practices from more than 2600 environments. If you want to know more about understanding the components of TCO models for enterprise software (including SAP), this is the session for you.
ENT203 - Integrating On-Premise Resources by Paul Ortiz - Principal Architect - Apollo Group , Ryan Shuttleworth - Technical Evangelist - AWS
The general purpose computing and storage environment of Amazon Web Services integrates perfectly into your existing ecosystem. Join customers who have taken advantage of this environment in parallel to their on-premise infrastructure to hear tales, tips, and tricks of best practices of integrating AWS with existing resources securely using services such as Amazon Virtual Private Cloud, AWS Direct Connect, and AWS Storage Gateway.
ENT204 - From Science Fiction to Reality: NASA's journey into the Cloud by Tom Soderstrom - IT Chief Technology Officer - NASA , Khawaja Shams - Software Engineer - NASA
This talk describes how cloud computing is now used for everything that JPL does, including the mission to Mars and beyond. We describe JPL's journey from prototyping to full operational status on mission critical systems, including what parts of the AWS Cloud JPL uses. The speakers provide practical lessons learned and recommendations on how to bring everyone along on the journey.
ENT205 - Drinking our own Champagne: Amazon.com's Adoption of AWS by Laura Grit - Principal Technical Program Manager - AWS
Amazon retail websites around the world run on AWS, affording us greater control over capacity, performance, and availability than we have ever had in the past. In this session, we explain how we managed the process of migrating the worlds largest e-commerce sites to the cloud and operationally the benefits we get daily from using AWS technology.
ENT206 - Introduction to Benchmarking with AWS by Robert Barnes - Director of Benchmarking - AWS
Whether you are building a new application or migrating an existing application, benchmarking can help you establish a baseline of your application's ability to support business requirements. The cloud is transforming benchmarking as it is not only scalable but also cost-effective to conduct benchmark experiments. In this session, Robert Barnes the Director of Benchmarking at AWS will give you an overview of benchmarking in the cloud and discuss a framework for benchmarking. He will discuss 3 ways to use benchmarks to answer specific capacity and performance questions and also share some examples of mistakes that people make when conducting their own benchmark tests or when consuming 3rd party benchmarking reports. This session is targeted at those who are looking for high-level guidance on how benchmarking can help your business. For a more in-depth practical review of my recommended tests join me at ENT 303 A practical guide to benchmarking with AWS.
ENT301 - Enterprise IT Customer Panel by Dan Rogers - Head of Global Demand Gen - AWS, Bharat Shyam - CIO - State of Washington , Troy Otillio - Cloud Strategist - Intuit , Darren Person - CTO, Chief Architect & Online Media Executive - Elsevier , Sean Perry - Chief Information Officer - Robert Half International
In this session you will hear IT leaders of some of the most innovative and dynamic enterprises, answer the most burning topics that we are all dealing with today: What is the right mix of on-premise and cloud infrastructure for your company? How can you get clear visibility into all the cloud solutions being implemented today by your distributed development teams? How should you think about budgeting and billing in a world of utility, monthly usage charges versus upfront capital expenses? How can cloud infrastructure unleash new models of IT innovation? What is the role of IT as we transition to cloud computing?
ENT302 - Evaluating and Deploying Microsoft Exchange Server and SharePoint Server on AWS by Tom Rizzo - General Manager, Windows EC2, AWS Windows Business - AWS
Have you wondered about migrating your on-premises Windows Server, Active Directory, Exchange Server or SharePoint Server deployment to AWS? Looking for the best way to get started? This session will step you through the evaluation steps you should take in looking at migrating your Microsoft workloads to AWS. Well do a deep dive into the architecture, performance, availability and functionality running these workloads on AWS. Well also explore using value added services in AWS to extend your Microsoft workloads from both an IT Pro and Developer standpoint. Finally, well look at how to integrate your on-premises deployments with AWS so you can continue to run some workloads on-premises and others in AWS. This session is full of demos and come armed with all your questions about running Microsoft workloads on AWS.
ENT303 - A Practical Guide to Benchmarking with AWS by Robert Barnes - Director of Benchmarking - AWS
Benchmarking not only helps identify areas of concern and bottlenecks, but also acts as the foundation for future productivity improvements. In this session, Robert Barnes the Director of Benchmarking at AWS will provide practical tips and examples of conducting benchmarks of AWS compute, storage, network and application services AWS. Using specific examples, he will provide helpful hints and tips on how to configure and execute benchmarks tests for evaluating your application's performance so you can enhance application performance over time with AWS. This session provides more detailed operational tips, for a more general overview of why to benchmark and how to use benchmarks, join me at ENT 206 Introduction to Benchmarking at AWS"
ENT401 - Oracle Enterprise Performance Management Applications in the AWS Cloud by Larry Hill - Business Development Manager - Full 360
Oracle Hyperion applications - Essbase, Planning and Financial Management (Consolidation) - are the most widely-used financial analytic applications in the world. They all experience extreme peaks in usage, primarily during period-end close. Few corporate IT operations provision enough server capacity to provide good response time to Finance Departments during these close periods, which tends to make a challenging time even more difficult. Because of these factors, Hyperion applications are ideally suited for cloud computing in an enterprise setting. Join Larry Hill from Full 360 to discuss his firms deep experience with the benefits and best practices related to this or key workload running in the Amazon cloud.
GMG203 - Meteor Entertainment by Sarah Novotny - CIO - Meteor Entertainment
Game developers need to spend their time building new games and features, not managing infrastructure. Meteor Entertainment has learned how-to minimize the time they spend managing infrastructure by automating deployments, monitoring systems through log analysis, and by making their data tier easy to scale. Attend this session to hear all about Meteors best-practices.
GMG204 - TinyCo by Gabi Ghimis - Product Manager - TinyCo
TinyCo is a game studio that powers and monetizes hit titles such as Tiny Village and Tiny Pets. In this session they will share their best practices for developing engaging titles that work across mobile platforms. TinyCo has learned how-to scale their AWS app servers and databases to handle viral demand, and they will talk about what they learned while they were developing their gaming platform and code libraries. Additionally, TinyCo was successful marketing and monetizing their game with the Amazon Appstore and Kindle Fire, and they will explain how-to integrate with Amazons in-app purchasing service.
GMG301 - Developing Mobile Games in the Cloud by Patrick Prendergast - Business Development Manager - AWS
The convergence of social gaming and mobile has changed the game for social game developers combining old and new challenges. Many mobile social game developers are leveraging AWS for their backend infrastructure because it allows them to execute in the face of these challenges. How can you reduce deployment time from weeks to hours? How can you build a back-end that can serve 10 users or 10,000,000 without failing or paying for what youre not using? Come learn from us how to architect your mobile social game on AWS and change the game of mobile social deployment in your favor.
GMG302 - Scaling Social Games with Ubisoft by Read Maloney - Product Manager - AWS, Tsvetan Petkov - Online Architect - Ubisoft
Game success leads to rapid and unpredictable growth, and high-levels of read and write traffic can make scaling games even more difficult. In this session Ubisoft and AWS cover the architectural best practices for building scalable apps and the best ways to adjust your app if it begins to slow down. This session mainly focuses on scaling databases, and we focus on the following technologies: Couchbase, Cassandra, MySQL, Amazon RDS, and Amazon DyanmoDB.
MBL101 - Distributing Apps through Kindle Fire and the Amazon Appstore for Android by Aaron Rubenson - Director - AWS
Interested in offering your apps and games to Amazon customers? Learn how to grow your mobile app or gaming business by offering your app to millions of Amazon and Kindle Fire customers. This talk will provide an overview of selling your app on Amazon and resources to help you engage customers and monetize, including an overview of the GameCircle and In-App Purchasing APIs. Plus, hear tips for building relationships with Amazon customers and creating the killer app for Kindle Fire by optimizing your mobile apps and games for Kindle Fire tablets.
MBL202 - One-Click Mobile Cloud Services Using AWS by Matthew Schmulen - Lead Solutions Engineer - Appcelerator
Create awesome mobile apps for iPhone and Android using the new integration of AWS Mobile SDK with Appcelerators Titanium Platform. Tap into more than 5,000 native APIs to create cross-platform mobile apps using a single JavaScript code base. Also, take advantage of Appcelerator Cloud Services, including more than 20 of the most high-demand mobile back-end services like push notification, photo sharing, and check-ins - a superior alternative to Microsofts Azure Mobile Services, running on AWS and supporting iOS/Android.
MBL203 - Building a Mobile Application Platform on AWS by Ilya Sukhar - Founder - Parse.com
Get under the hood with Parse.coms founder to see how they used AWS to build their mobile Platform as a Service. In this session, you learn how Parse is using a variety of AWS services including Amazon EC2, S3, ELB, EBS and Route53 to build data storage, push, and easy upload services for mobile developers.
MBL204 - Enhancing Quality & Performance for Vonage Mobile VOIP and Messaging Application using AWS by Guy Fighel - Director of Research & Development - Vonage
Mobile communication applications have become a growing trend in the last few years. They allow users to build global virtual communities, reducing the need for the classic carrier while using alternative data channels like WiFi or 4G. Such applications offer phone call features, interactive chats, SMS messages, video, and more for free or a very low cost. Companies are being challenged daily to provide best in class, high quality, over-the-top communication applications. In this session, we discuss the challenges in building a global quality network. We present how Vonage built a Mobile Global architecture using Amazon EC2, developed geo-location algorithms for optimal routing, and implemented iPhone/Android AWS SDKs for secure attachments storage using Amazon S3.
MBL205 - Monetizing Your App on Kindle Fire: In-App Purchasing Made Easy by Mekka Okereke - Manager, Software Development - AWS
The Amazon In-App Purchasing API makes it easy for you to offer digital content and subscriptions such as in-game currency, expansion packs, upgrades, magazine issues and more for purchase within your apps. Within minutes you can be up and running, ready to give millions of Amazon customers the ability to purchase engaging digital content using their Amazon 1-Click settings. Discover how in-app purchasing can help you monetize your apps on Kindle Fire and learn how to integrate the Amazon In-App Purchasing API into your mobile apps.
MBL301 - Data Persistence to Amazon DynamoDB for Mobile Apps by Bob Kinney - Mobile SDE at AWS - AWS
Object modeling is a common practice in mobile applications. We present two methods for modeling objects backed by Amazon DynamoDB, the AWS Persistence Framework for Core Data and DynamoDBMapper. We cover the benefits and limitations of these two solutions and demonstrate sample applications built with both technologies.
MBL302 - Solving Common Mobile Use Cases with the AWS Mobile SDKs by Bob Kinney - Mobile SDE at AWS - AWS, Glenn Dierkes - Software Development Manager - AWS
The AWS Mobile SDKs can be used to build thick-client architecture apps for iOS and Android devices. An overview of the SDKs will be presented as well as demos and code for storing data in Amazon S3 and sending emails via Amazon SES. You will also learn how to manage AWS credentials in a mobile environment.
MBL303 - Scalable Mobile and Web Apps by Kingsley Wood - APAC Business Development Manager - AWS, Aritra Ghosh Dastidar - Software Engineer - Intuit
AWS offers an array of products and services to handle the unprecedented volumes of traffic, enormous user numbers and vast amounts of data being experienced by a successful mobile app that takes off. Learn how with new found agility and amazingly low time to market, these must-know best practices and techniques in the rapidly evolving and highly demanding mobile landscape can ensure success. Featuring Intuit's txtWeb architecture as a case study.
MED101 - Introduction to Amazon CloudFront by Mark Ramberg - Media Strategy and Business Development - AWS, Brian Kaiser - CTO - HUDL
End users expect to be able to view media content anytime, anywhere, and on any device. Amazon CloudFront is a web service for content delivery used to distribute content to end users around the globe with low latency, high data transfer speeds, and no commitments. In this session, learn what a content delivery network (CDN) such as Amazon CloudFront is and how it works, the benefits it provides, common challenges and needs, performance, pricing, and examples of how customers are using CloudFront.
MED201 - Media Ingest and Storage Solutions with AWS by Joe Lyons - Manager, Global Storage Business Development - AWS, Alan Schaaf - Founder & CEO - Imgur
In this session we will discuss the numerous ways to ingest data into AWS including options such as physical media import & direct connect. We also talk about policy-based Hierarchical Storage Management (HSM) in the cloud, total cost of ownership, the importance of storage durability, and the infinite scalability of Amazon S3. Also, the founder of photo-share sensation IMGUR, Alan Schaaf, speaks about their migration to AWS.
MED202 - Netflix by Kevin McEntee - Vice President, Digital Supply Chain - Netflix
Netflix designed a massive scale cloud based media transcoding system from scratch for processing professionally produced studio content(to meet the unique scale and time constraints of our business). We bucked the common industry trend of vertical scaling and, instead, designed a horizontally scaled elastic system using AWS to meet the unique scale and time constraints of our business. Come hear how we designed this system, how it continues to get less expensive for Netflix, and how AWS represents a transformative opportunity in the wider media owning industry.
MED203 - Scalable Media Processing with AWS by Usman Shakeel - Solutions Architect - AWS
This session walks through approaches for large-scale media processing scenarios. We cover hybrid and cloud-based transcoding, file transfer, media preparation, and media management. We expect attendees to come away with an understanding of best practices for architecting and deploying hybrid and cloud-based systems for media processing.
MED204 - High Performance Content Delivery by Alex Dunlap - Senior Manager, Amazon Web Services - AWS, Andy Rosenbaum - Director of Development - Earth Networks
In this hands-on session, we demonstrate how you can use Amazon CloudFront to help architect your site to deliver both static and dynamic content (portions of your site that change for each end-user). We walk through how you can configure multiple origin servers for your Amazon CloudFront distribution providing you the flexibility to keep your content in different origin locations without the need to create multiple distributions or manage multiple domain names on your website. We also show you how you can use query string parameters to help customize your web pages for each viewer and how you can configure multiple cache behaviors for your download distribution based on URL patterns on your website.
MED301 - Is My CDN Performing? by Nathan Dye - Software Development Manager - AWS, Jeff Miccolis - Engineering - MapBox
This presentation provides practical guidance using external agent-based measurements and real user monitoring techniques. We review common content delivery network (CDN) architectures and how they relate to performance measurement. Finally, we walk through real-world CDN performance monitoring implementations used by MapBox, Amazon.com, and Amazon CloudFront.
MED303 - Addressing Security in Media Workflows by Usman Shakeel - Solutions Architect - AWS, Heidi Kujawa - CEO - KODE Compliance
Are your media assets secure? For media companies, security is paramount. Few things can more directly impact your companys bottom line. As the move to store, process and distribute digital media via the cloud continues, it is imperative to examine the relevant security implications of a multi-tenant public cloud environment. This talk is intended to answer questions around securely storing, processing, distributing and archiving digital media assets on the AWS environment. AWS also enables customers to achieve compliance with the MPAA security best practices with minimal effort. Learn how AWS complies with the MPAA security best practices and how media companies can leverage that for their media workloads.
RMG202 - Rainmakers: How Netflix Operates Clouds for Maximum Freedom and Agility by Jeremy Edberg - Reliability Architect - Netflix
In this session, learn how Netflix has embraced DevOps and leveraged all that Amazon has to offer to allow our developers maximum freedom and agility.
RMG203 - Cloud Infrastructure and Application Monitoring with Amazon CloudWatch by Derek Pai - Sr. Product Manager, Monitoring - AWS, Darren Lee - Sr. Quantitative Engineer - Bizo , Henry Hahn - Sr. Product Manager - AWS
Amazon CloudWatch provides AWS customers the monitoring platform for keeping tabs on their cloud infrastructure and applications. In this session, we show you how to use CloudWatch to monitor vital operational resource data such as EC2 Instance CPU Utilization, ELB Request Counts, RDS Read Throughput and much more. Learn how to configure CloudWatch Alarms to alert you any time services are operating outside of ranges you define. Finally, see how you can monitor applications on your EC2 instances or outside of AWS.
RMG204 - Optimizing Costs with AWS by Coburn Watson - Manager, Cloud Performance Engineering - Netflix
Find out how Netflix, one of the largest, most well-known and satisfied AWS customers, develop and run their applications efficiently on AWS. A member of the Netflix Cloud Performance Engineering team outlines the Netflix common-sense approach to effectively managing AWS usage costs while giving the engineers unconstrained operational freedom.
RMG205 - Decoding Your AWS Bill by Matt Leonard - Manager, Tech Program Management - AWS, Ranjit Prabhu - Sr. Manager, Software Development - AWS
AWS billing has many features to help you manage and control your costs in the AWS cloud. In this session, we walk through the mechanics of AWS bill computation focusing on consolidated billing, detailed billing reports, programmatic access, cost allocation, billing alerts, and IAM access. We provide an overview of these features and demo how they are used in your own account setup.
RMG206 - Introduction to AWS Elastic Beanstalk by David McArthur - Software Development Engineer - AWS, Saad Ladki - Sr. Product Manager - AWS
Are you looking to build the next viral Facebook application or mobile game? Are you worried about the viral growth of your web application? Are you tired of managing servers and installing software? This session introduces AWS Elastic Beanstalk, the easiest way to deploy and manage web applications on AWS. Well show you how you can write your application and let Elastic Beanstalk do the rest.
RMG207 - Introduction to AWS CloudFormation by Avinash Jaisinghani - Software Development Manager - AWS, Chris Whitaker - Senior Software Development Manager - AWS
AWS CloudFormation makes it easy to deploy and manage the lifecycle of applications running in AWS. This session walks through an end-to-end scenario, creating a stack with a set of AWS resources and deploying the application files and packages. Once deployed well walk through how you can change the stack to reflect operational changes or application requirements and finally use CloudFormer to create a CloudFormation template from AWS resources already running in your environment.
SEC101 - A Guided Tour of AWS Identity and Access Management by Jim Scharf - Director - AWS
Learn what AWS Identity and Access Management (IAM) technologies are available for you to manage users and their access to your AWS environment. We present a high level discussion of the benefits and functionality IAM provides to control secure access to your AWS environment. We discuss how you can manage users and their permissions when using IAM, how roles makes it simpler for you delegate access, and how to use Multi-Factor Authentication (MFA) to require additional proof of identity.
SEC102 - Security and Compliance in the AWS Cloud by Chad Woolf - AWS Compliance Leader - AWS
To properly evaluate cloud computing services, there are several industry resources to manage cloud provider security, risk, and compliance. This session discusses AWS collateral you can use to accomplish this and allow you to build an environment that can conform to a wide range of compliance and security requirements. If youre already using AWS and need to perform an audit on your cloud assets, this session demonstrates a feasible validation approach that works for AWS.
SEC201 - Security Panel by Jason Chan - Cloud Security Architect - Netflix , Khawaja Shams - Software Engineer - NASA , Andrew Doane - Director, Systems - AWS, Rahul Sharma - Founder and VPE - Averail
Learn from fellow customers, including Jason Chan of Netflix, Khawaja Shams of NASA, and Rahul Sharma of Averail, who have leveraged the AWS secure platform to build business critical applications and services. During this panel discussion, our panelists share their experiences utilizing the AWS platform to operate some of the worlds largest and most critical applications.
SEC202 - Federal Government Compliance Best Practices by CJ Moses - GM, Government Cloud Services - AWS
Federal Agencies seeking a secure and compliant cloud platform have migrated to AWS GovCloud (US). Dont miss this session as CJ Moses, general manager of Government Cloud Solutions for AWS, will share his insights into government compliance in the cloud and how your agency can utilize AWS GovCloud to host and process your agencies Critical Unclassified Information (CUI), including data requiring compliance with International Traffic In Arms Regulation (ITAR) and Export Administration Regulations (EAR). CJs background with both the FBI and the Department of Defense, as well as his work with AWS Public Sector customers, allows him the ability to share a unique perspective derived from experiences on both sides of todays compliance issues. CJ also announces, for the first time, the availability of many new services and features within GovCloud (US).
SEC203 - AWS Security for Microsoft Shops by Miles Ward - Solutions Architect - AWS, Tom Stickle - Sr. Manager, Solution Architecture - AWS
Attend this session to learn best practices for advancing your security posture to include cloud-based deployments of Windows and Windows-based applications. We will put some of the more complex AWS security controls into a more Windows-centric context. Expect deep dives on federation between AWS Identity and Access Management and Active Directory, Routing, Security Groups and Network ACL's in Virtual Private Cloud, leveraging HOSM authentication from IIS applications, and direct guidance for Sharepoint and Exchange workloads.
SEC206 - Security OF the AWS Cloud by Stephen Schmidt - Chief Info Security Officer - AWS
Security must be the number one priority for any cloud provider and thats no different for Amazon Web Services. Stephen Schmidt, vice president and chief information officer for AWS, will share his insights into cloud security and how AWS meets the needs of todays IT security challenges. Stephen, with his background with the FBI and his work with AWS customers in the government and space exploration, research, and financial services organizations, shares an industry perspective thats unique and invaluable for todays IT decision makers. At the conclusion of this session, Stephen also provides a brief summary of the other sessions available to you in the security track.
SEC301 - Security IN the AWS Cloud by Max Ramsay - Principal Security Solutions Architect - AWS
There are so many different thoughts about how to secure your applications running in AWS that it can be confusing to know where to start. In this session, we cover tips, tricks, and emerging best practices for securing your applications. We discuss topics ranging from how to configure your AWS resources to options for logging and intrusion detection. Discover that running your applications in AWS gives you a great head start.
SEC302 - Delegating Access to Your AWS Environment by Jeff Wierer - Sr. Product Manager - AWS
At times you may have a need to provide external entities access to resources within your AWS account. You may have users within your enterprise that want to access AWS resources without having to remember a new username and password. Alternatively, you may be creating a cloud-backed application that is used by millions of mobile users. Or you have multiple AWS accounts that you want to share resources across. Regardless of the scenario, AWS Identity and Access Management (IAM) provides a number of ways you can securely and flexibly provide delegated access to your AWS resources. Come learn how to best take advantage of these options in your AWS environment.
SEC303 - Top 10 Identity and Access Management Best Practices by Anders Samuelsson - Technical Program Manager - AWS
Learn about best practices on how to secure your AWS environment with AWS Identity and Access Management (IAM). We discuss how you best create access policies, manage security credentials (i.e., access keys, password, Multi Factor Authentication devices, etc.), how to set up least privilege, minimizing the use of your root account, and more.
SEC304 - Building Security from Scratch in AWS by Alex Stamos - CTO - IT Sec Partners
Ask any two cloud "experts" about whether you can trust cloud providers for running your security sensitive systems and you'll likely get three opinions. When our group of security experts turned to the task of building a robust, reliable, and secure infrastructure, we chose to disregard the conventional wisdom, ignore the FUD, and design controls that allow us to confidently build on AWS, Salesforce, and other cloud providers. This talk walks you through the steps necessary to build a trustworthy cloud infrastructure. We outline how you can deconstruct your security needs into specific technical goals, map those goals onto controls that are available in the cloud, and discuss what risks need to be accepted while others are mitigated. The talk includes detailed discussion of cryptographic, network and logical controls, and is best enjoyed by those with advanced knowledge of AWS.
SPR101 - The Future of Cloud Security (Presented by Intel) by Jason Waxman - General Manager - Intel
Direct engagement with enterprise end users and industry partners like Amazon Web Services has helped Intel define its vision for future cloud infrastructure. In this session, Jason Waxman, Intel's Cloud Infrastructure Group GM, will share the basis for Intel's vision, the five pillars of future cloud infrastructure and the collaboration between Intel & AWS.
SPR201 - Recursive Clouds: Building Platforms on Platforms (Presented by Twilio) by Jeff Lawson - Co-Founder & CEO - Twilio
The economic and technological implications of elastic infrastructure are rippling through every layer of enterprise software delivery. Starting at the hardware layer, the Cloud is enabling new platform infrastructure business models that fundamentally change the pricing and delivery of software applications. Jeff will explore the way the Cloud is disrupting industry after industry by enabling platforms upon platforms of flexible, pay as you go, virtualized infrastructure.
SPR202 - The Convergence of IaaS and PaaS (Presented by RightScale) by Thorsten von Eicken - Founder & CTO - Rightscale
In the six years since AWS launched, pundits have categorized cloud offerings into IaaS, PaaS and SaaS. However, the lines between thes categories have started to blur. As IaaS solutions offer more platform and application services, this traditional view is becoming outdated. In this talk, you will learn about the shift to a new world of cloud services, where companies can pick and choose from a menu of options and assemble a solution that fits their needs. Find out how to architect your solutions to leverage a wide variety of cloud services and get a glimpse into what it means for your customers and your business.
SPR203 - Cloud Security is a Shared Responsibility (Presented by Trend Micro) by Allan Macphee - Sr. Product Manager - Trend Micro
Securing workloads in the cloud is a shared responsibility between you and your cloud service provider. With an Infrastructure as a Service (IaaS) cloud, such as AWS, the service provider is responsible for securing the underlying hosting infrastructure, but businesses are expected to secure their virtual machines and their applications and data built on top of it. But what security controls are required? And can traditional security tools be applied to cloud environments? This presentation answers these questions and provides guidance on what security controls are needed to protect your cloud workloads. We also highlight the features and services that make AWS a trusted partner in securing your cloud environment.
SPR204 - Securing Your Cloud Across Your On-Premise and AWS Instances (Presented by Trend Micro) by Dave Asprey - VP Cloud Security - Trend Micro
Public cloud adoption is a natural progression from virtualized data centers and on-premise deployments. For most businesses, workloads will reside in both their data centers and in public clouds, resulting in mixed environments. This presentation introduces the new security management and policy framework required to support these environments. Consistent security policy enforcement is needed regardless of VM location with additional security controls applied when VMs migrate to the public cloud. And this security should be implemented through automatic provisioning and enforcement to support the elastic nature of public clouds. Examples will be given on how this security can be deployed in an AWS environment.
SPR205 - Privileged Identity and Access Management (Presented by Xceedium) by Ken Ammon - Chief Strategy Officer - Xceedium
Xceedium discusses best practices for protecting your infrastructure from the risks that privileged users and credentials pose to systems and data in cloud-only and hybrid architectures. Topics include: privileged user and account risks, applying AWS security best practices to the "customer side" of the shared security equation (from the hypervisor up), how Xsuite Cloud mitigates the privileged user risks for both AWS-only or hybrid architectures with infrastructure across cloud and traditional data centers (including unified policy management), how Xsuite Cloud's deep AWS integration adds: 1) additional protection for the AWS Management Console, 2) auto discovery and provisioning for highly elastic environments, and how can Xsuite help control access when it is scripts, not individual users, doing management console like functions using AWS APIs.
SPR206 - Bridging the Desktop to the Cloud (Presented by Citrix) by Joe Vaccaro - Director, Product Management - Desktop and Apps Group - Citrix, Peder Ulander - Vice President of Product Marketing - Citrix
The datacenter is rapidly evolving and cloud services are leading the transformation towards IT-as-a-Service. In order to meet the growing demands of employees, IT is turning to desktop virtualization and cloud computing to empower workers with anytime, anywhere access to their desktop, apps, and data on any device. However, is it possible to enable desktops as a service from the cloud? In this session, you'll learn how enterprises and service providers can transform key workloads including Windows desktops and apps as an IT delivered service on AWS. You'll get insight into the technical details and cloud architecture required to provide the orchestration, self-service, elasticity, and flexibility necessary to deliver desktops and apps as a true cloud service.
SPR207 - Your Enterprise Network and AWS (Presented by Citrix) by Morgan Gerhart - Director - Citrix
Citrix and Amazon Web Services together enable enterprises to optimize the delivery of cloud-based applications. Maximizing the availability and performance of these application workloads requires that you eliminate disparities between AWS and your on-premise data center. Recent AWS enhancements now make it possible to deploy the same network services that power your own data center, such as application delivery and load balancing, directly into an AWS environment. It's time to learn how Citrix NetScaler VPX virtual appliances make AWS a seamless extension of your enterprise network infrastructure. In this session you will learn: The best way to seamlessly connect AWS and your enterprise network; the advantages of using virtual appliances to build a common set of network services spanning your data center and AWS; how to leverage advanced availability, acceleration, visibility and control capabilities to provide a seamless network experience; and how the right network infrastructur
SPR208 - Hitting Your Cloud's Usage Sweet Spot (Presented by Newvem) by Eric Hammond - Internet Startup Technologist - Alestic.com, Shane Meyers - Operations Engineer - SmugMug, Andrew Kenney - VP of Platform Engineering - Acquia, Chemi Katz - VP of Technical Operations - DoubleVerify, Inc, Dan Feld - VP of Sales - Newvem
Join Newvems All-Star panel of AWS expert users as they discuss their experiences, insights, and best practices on how usage analytics can save a ton of money, make your CIO and CFO really happy, and ensure business performance. Learn how usage analytics can enable you to accomplish optimal resource usage across your cloud adoption/deployment life-cycle: deterministic planning, right sizing, elasticity, compliance, and cost effectiveness.
SPR209 - On Demand IT with Cloud360 (Presented by Cognizant) by Ramesh Panuganty - Managing Director, Cloud360 Solutions - Cognizant
In this new age of IT, say goodbye to siloed, monolithic application environments, operations, and governance. Embrace the dynamic, agile application environments, and future-proof your virtual data centers. As businesses strive to tackle unrelenting information growth, increasing customer demands, cost pressures, and complex application environments, embrace an unprecedented ability to manage every aspect of your services; quality, SLAs, performance, and compliance from the operations, governance, and cost perspective. Through this session, discover how to use the service management and service governance layers to automate and manage your application environments. Learn how to apply strategic points of control in your organization and deploy automation and intelligencewhether in provisioning new application resources, migrating data, adjusting capacity, supporting new hypervisor, securing resources, or integrating AWS cloud into your operations. Understand how Cognizant's Clou
SPR210 - How Zumba Delivers Superior Application Performance to Distributors & Customers Around the Globe (Presented by New Relic) by Chris Kelly - Developer / Evangelist - New Relic, Douglas Jarquin - Technical Operations Manager - Zumba
Introduction: Chris Kelly, New Relic; Speaker: Douglas Jarquin, Technical Operations Manager, Zumba. Zumba is one of the largest branded fitness programs in the world, with more than 12 million weekly class participants, in over 140,000 locations, across more than 150 countries. Zumba uses New Relics PHP agent to help them deliver the scalability, superior performance, and mobile access for both their B2B and B2C customers around the globe. With over 1 million unique visitors and 35 million total visitors each month, features like Real User Monitoring, Server Monitoring, and App Map help the IT team monitor and tune performance in every Zumba location. In this session Douglas Jarquin, Zumba Technical Operations Manager, discusses how he keeps the application and technical infrastructure that are critical to Zumbas business results running at peak performance, his approach to ongoing performance management, and how the team uses New Relic to help prioritize their performance workload
SPR211 - Enterprise in Motion: Convergence of The Big 4 (Presented by Wipro) by Atul Sood - General Manager and Global Practice Head, Integrated Cloud Services - Wipro Technologies
In todays fast paced and challenging business environment, IT priorities are increasingly being integrated with Business Strategy. Technology is powering business innovation and establishing a culture of agility. Social and Mobility have led the way to a connected planet and this bears relevance across industries undergoing rapid transformation. Hear from Wipro on how you can harness the power of the Social, Analytics, Mobility on public cloud to create an Enterprise in Motion!
SPR212 - Conquering Challenges to Company Wide Cloud Adoption @Adobe (Presented by Cloudability) by Tim Prendergast - Senior SaaS Infrastructure Architect - Adobe Cloud Services
The convergence of company and cloud happens at more than just the engineering/ops tier. Without good processes and communication, cloud can quickly become a four-letter word in your company. Cloudability has invited the Adobe Cloud Services Team to show you how they work with finance, operations, engineering, and legal teams to ensure that everyone loves the cloud as much as they do.
SPR213 - Leverage the Cloud with Mapping and Geographic Data (Presented by Esri) by Marwa Mabrouk - ArcGIS Server Product Manager - Esri
The cloud has opened new capabilities for location analytics. If youre already using the cloud, how can you leverage those capabilities? In this session, we cover how location analytics' need for processing power and storage capacity has put the cloud to use. We also discuss how users who want to leverage location analytics can get started in the AWS cloud today.
SPR214 - Enterprise Grade, Secure Big Data Analytics (Presented by Intel) by Greg Khairallah - Business Development Manager for Big Data - Intel
Analysts predict that enterprises who embrace extreme information management architectures that deliver real time, actionable analytics will outperform competitors by 25%. This requires mastery of not only Hadoop software data analysis & processing paradigms- it requires focus on access control, data compliance, interoperability with cloud, and optimized hardware infrastructure across storage, network, and compute processing. Intel is a key contributor to the enablement of Apache Hadoop and is focused on delivering optimizations that span a whole Big Data deployment from hardware to software. This session will focus on the software and hardware engineering efforts to bring security, performance and stability to the cloud ecosystem.
SPR215 - Limitless IT: Control for the unbounded datacenter (Presented by BMC) by Kia Behnia - Senior Vice President and Chief Technology Officer - BMC
While many large enterprises are embracing Amazon Web Services for the promise of unlimited capacity, flexibility, and reduced capital expenditure, IT departments are just starting to engage and manage AWS as a strategic vendor. In order to accelerate adoption of AWS, IT needs a clear strategy that extends their existing operations, controls, and governance models to enterprise workloads running on AWS. Working with AWS, BMC Software is delivering the technology and process changes that enable enterprise IT to embrace AWS while maintaining the oversight their business requires. In this session, Kia Behnia, chief technology officer of BMC Software, explores the balance of running IT in traditional data centers and running in AWS. Kia discusses how you can unlock value for your business with AWS, and how you can address the management challenges of provisioning, billing, change management, and monitoring in mixed AWS and on-premise IT environments.
SPR216 - Migrating a Large SAP Landscape to the Amazon Cloud by Joe Coyle - North American Chief Technology Officer - Capgemini
A real life case study of migrating existing global SAP instances from a traditional datacenter infrastructure to the Amazon Web Services Cloud. This presentation will cover the business case, how the planning was executed, major challenges and how to navigate them, and how we adapted SAP to a consumption based Cloud environment. We will also cover how we performed Heterogeneous system copies to the Cloud. Finally we will cover the Capgemini Cloud Framework used and how we tackled the Legal and Audit implications of the migration.
SPR217 - Real World Privileged Identity, Security and Operational Experiences Panel (Presented by Xceedium) by Craig Sutherland - Principal Engineer - Booz Allen Hamilton, Balaji Balakrishnan - CISO - International Finance Corporation, John Suit - VP of Product Management - Xceedium, Sri Vasireddy - Chief Cloud Officer - 8KMiles.com, Slawek Ligier - CTO and Corporate VP of WW Product Development - SafeNet
Xceedium and guests discuss real world privileged identity, security and operational experiences when building or migrating applications to the AWS Cloud.Topics: 1) What did you perceive as the key risks before you started? What are the actual risks?; 2) How are the models/approaches to IT operations, Managing Privileged Identities/Accounts, and Security different?; 3) What did you do well? What would you do differently if starting over again? 4) Thoughts on managing/securing across hybrid cloud/data center architectures?
SPR219 - Manage for Peak Performance with AWS (Presented by RightScale) by Brian Adler - Senior Professional Services Architect - RightScale
The possibilities are endless with AWS. You can spin up more servers than you ever imagined across the globe and across your organization. But how will you manage all your AWS usage, especially as it continues to grow? How will you ensure that your deployments on AWS are rock-solid? In this session we will discuss and demonstrate how to use RightScale cloud management to: Bullet-proof your application: how to structure a build and deploy methodology across AWS regions. Well demo how to replicate configurations across EC2 availability zones to enable disaster recovery. Automate and organize: well show you how to automate redundant tasks so you can organize, clone, and deploy entire systems efficiently. Forecast cloud costs: well demo our new tool that gives you the ability to model deployments and usage patterns with AWS.This session will be presented by Brian Adler, RightScale Senior Services Architect. RightScale customers, Coupa and Pearson, will join the
SPR220 - Portability and Content Management When Moving from On-Premise to AWS (Presented by Red Hat) by Todd Sanders - Director, Software Engineering - Red Hat
In this session attendees will learn about open source software solutions sponsored by Red Hat that help to create a framework for distributed cloud services, focusing initially on content and entitlement (CaaS and EaaS, respectively) of Red Hat Subscriptions. Utilizing PKI for authorization and authentication, the audience will learn about technologies that are creating the plumbing and laying the groundwork which will allow for more ubiquitous and pervasive use of their Red Hat Subscriptions both on-premise and on AWS, while addressing the business, operational and technical issues that such subscription movements can entail. There will be a live demonstration and technology preview showing Red Hat's new content management solution running inside of Amazon EC2.
SPR221 - Learn How Enterprises Can Manage their AWS Cloud Services with CA Automation Suite for Clouds Powered by Amazon Web Services (Presented by CA) by Jeff Williams - Principal Product Manager, Alliance Solutions, Cloud Management - CA Technologies
Enterprises are looking to effectively consume and manage Amazon cloud based services. CA Automation Suite for Clouds incorporates the full benefits of CA Service Catalog interface combined automation connectors to enable AWS services into an enterprise service delivery and support platform. The enterprise grade solution provides administrative and governance for the entire Amazon cloud lifecycle which includes; aggregating existing machine images to a single Amazon account, building standard corporate machine images, publishing services, managing request approval, service provisioning (via AWS CloudFormation APIs), access provisioning, and image de-provisioning. Join our session to learn how CA Technologies management tools can provision and manage AWS compute services in an enterprise environment to establish standards, reduce cost, enable service delivery and improve service support.
SPR301 - Optimizing Enterprise Applications and User Access in the Cloud (Presented by F5 Networks) by Siva Mandalam - Product Management Executive Director for Cloud, Virtualization and Management - F5 Networks
Are you interested in delivering enterprise or enterprise class applications and user access and various workloads like SharePoint on AWS? This session gives you an overview of issues surrounding application delivery in Cloud, and presents a number of architectural options for workloads in AWS. The session also discusses approaches to architecting networks that can dramatically simplify deployment and ongoing management or applications, particularly as they migrate to AWS from another cloud or they operate in active/passive deployment clouds for business continuity. The session includes guidance on deployment through examples, and presentation of reference architectures, tools and strategies for delivering and security applications on AWS.
SPR302 - Migrating to the Cloud Without Breaking Compliance (Presented by SafeNet) by Slawek Ligier - CTO and Corporate VP of WW Product Development - SafeNet
Organizations who have locked down compliance in the traditional data center are now struggling with how to migrate to the cloud with compliance intact. Ownership and liability, lack of transparency from the cloud provider, an almost complete absolution of liability in contracts, and lack of clear guidance on required controls have been creating confusion in cloud migrations. Plainly put, regulatory compliance is typically enabled through a combination of certified infrastructures, and protection and control of PAN data and the same principles apply whether youre in the Data Center and in the Cloud. In this presentation, we guide you through understanding audit scope, selecting the right infrastructure, and preparing you with whats in store - from a new class of privileged users to shared responsibility and contract implications. We also arm you with checklist for enforcing and enabling controls that will ensure critical protection and compliance of your data no matter where it
STG201 - Understanding AWS Storage Options by Joe Lyons - Manager, Global Storage Business Development - AWS
AWS provides multiple storage options to meet your varying needs. We provide an overview of how AWS storage services can be used to support application development and delivery, backup, archive, disaster recovery, and virtualized compute.
STG202 - Parmigiano, a Monastery, Love and Faith: Technical Lessons on how to do Backup and Disaster Recovery in the Cloud by Simone Brunozzi - AWS Evangelist - AWS, Augusto Rosa - Manager, Server Operations - Shaw Media
IT systems provide tremendous value, and maintaining data integrity and guaranteeing business continuity is of utmost importance for any organization. However, in today's world, those systems have grown in complexity and cost, while the business demands IT agility and lower costs. In this talk, AWS technology evangelist, Simone Brunozzi, joined by AWS customers, will explore how organizations should approach backup and disaster recovery, and how these two aspects can be implemented in the cloud to improve efficiency and flexibility. The talk starts with general concepts, and then dives into technical details, culminating in real customer examples that showcase some tips and tricks and the benefits of a cloud-based approach.
STG203 - Cloud Storage War Stories: From the front lines of some of the biggest battles by Stephanie Cuthbertson - Sr Manager, Product Management - AWS, Don MacAskill - Co-Founder, CEO & Chief Geek - SmugMug , Peter Esposito - Systems Manager - USTA , Greg Arnette - Founder & CTO - Sonian , Jeff Kimsey - Head of U.S. Product Management - Global Data Products NASDAQ OMX , James Petts - Manager Database Automation - Amazon.com
Here is your chance to learn from others' front line experiences in building game changing cloud storage implementations. We've put together a panel of battle-hardened cloud storage customers. They've gone through, over, and around obstacles to come out smarter, faster, better. They've been where you are and theyre going to share their challenges and their lessons learned with their cloud storage implementations, how they've learned to leverage AWS as their vendor, and their future plans for using the latest storage services and features.
STG204 - Using AWS Storage Gateway by Arun Sundaram - Sr. Product Manager - AWS
Organizations are increasingly looking to cloud storage to address many of their corporate IT use cases, but prefer to do so in a way that minimizes the impact to their existing on-premises applications. This presentation provides an overview of AWS Storage Gateway, a service that connects an on-premises software appliance with cloud-based storage for seamless integration between on-premises IT environments and AWSs storage infrastructure. The presentation covers the key use cases of the service, discusses how AWS Storage Gateway works, and provides guidance on getting started.
STG205 - Amazon S3: Reduce costs, save time, and better protect your data by Khawaja Shams - Software Engineer - NASA , Dan Winn - Sr Manager, Software Development - AWS
Amazon Simple Storage Service (S3) offers an expanding set of capabilities for protecting and managing data, as well as hosting content. We'll cover the breadth of the service's features and show you how you can take advantage of them.
STG301 - Using Amazon Elastic Block Store by Miles Ward - Solutions Architect - AWS, Steve Newman - Scalyr - Cloud Engineer, JP Schneider - DevOps Engineer - Obama For America
This session will walk through best practices on architecting Amazon EBS for high performance workloads.
STG302 - Archive in the Cloud with Amazon Glacier by Mark Seigle - Software Dev Manager - AWS
Archive in the cloud explores the rapidly growing amount of infrequently accessed (or "cold") data being stored by enterprises, and how IT organizations can use the cloud to dramatically reduce the cost of storing that data while improving durability. The talk gives a technical overview of Amazon Glacier, including coding samples, and tips and tricks for data indexing and reducing cost.
STG303 - Building Scalable Applications on Amazon Simple Storage Service by Jim Sorenson - Principal Engineer - AWS
Want to build an application that requires minimal up-front investment, and will seamlessly scale from hundreds to millions of users? Amazon S3 is a powerful building block that can enable you to focus your time on the value and functionality of your application, rather than the challenges of scaling it. In this session we'll cover techniques to best take advantage of the platform. We'll discuss structuring your key naming convention to maximize consistency of performance, as well as ways to optimize your upload and download throughput. We'll learn how to eliminate proxies between your application and Amazon S3, and use the platform for your logging needs. Finally, we'll cover simple techniques for efficiently managing the billions of objects your highly scaled application may accumulate.
STP101 - What Can You Do With $100? by Josiah Carlson - Chief Architect and Co-Founder - ChowNow , Jason Gurwin - Co-Founder - PushPins , Kevin Gibbon - Co-Founder & CEO - SmartAisles , Jeff Barr - Technical Evangelist - AWS
This panel, led by Senior AWS Evangelist Jeff Barr, shares their build-with-no-cash experiences. Find out how these companies creatively built prototypes on AWS to secure slots in top accelerators and/or investment from top firms. From efficient use of engineering resources, to fully leveraging AWS to remain agile, scale as needed, and even pivot with little or no infrastructure cost.
STP102 - Ahead in the Clouds by Don MacAskill - Co-Founder, CEO & Chief Geek - SmugMug
Session Information coming soon.
STP201 - Efficiency at Scale by Ammon Bartram - Co-Founder - Socialcam , Guillaume Luccisano - Co-Founder & VP of Engineering - Socialcam
In May of 2012, Socialcam exploded, gaining tens of millions of new users in just a few weeks. At the time, the service ran on 15 servers in a co-location facility in San Francisco. To meet new user traffic demands and continue to deliver maximum user satisfaction, Socialcam made the move to cloud services. With only two engineers and a constant barrage of users, there was limited time for technical transition, but Socialcam endured with no significant downtime. In this technical session, Socialcam co-founders Guillaume Luccisano and Ammon Bartram talk about their experience scaling Socialcam. They present the challenges they encountered, how they addressed them, and the technologies they used in the process. They focus particularly on how they used Amazon services in conjunction with their own hardware to keep Socialcam active with no significant downtime and no costly system redesign.
STP202 - Driving to Success: Fueling Growth and Innovation by Bob Van Nortwick - Business Development Manager - Venture Capital and Startups - AWS, Meir Morgenstern - Co-Founder and VP of Engineering & Operations - CloudOn , Sam Parnell - CTO - Bleacher Report , Gregarious Narain - Co-Founder & CTO - Chute, Jack Murgia - Senior DevOps Engineer - Edmodo, Robert Kotredes - Head of Engineering - TinyCo
This cutting edge panel explores the approaches of companies managing rapid customer adoption and innovation cycles. Many of these companies are at the forefront or in the middle of explosive growth curves- they are continuing to innovate as a vehicle for accelerating that growth. These companies will share how they were able grow quickly, maintain market advantage and in some cases position themselves for acquisition. You will learn from the scoop from the inside- what these companies experienced and how they were able to leverage AWS to capitalize on growth and push innovation without compromising customer experience or breaking the bank.
STP204 - Pinterest Pins AWS! Running Lean on AWS Once You've Made It by Ryan Park - Operations Engineer - Pinterest
When youre starting out, its not worth putting a lot of effort into optimizing your costs; you should put your time into growing your business. But as your infrastructure grows, it becomes worthwhile to optimize your use of AWS resources. Ryan Park from Pinterests technical operations team presents how they have optimized their infrastructure costs as their site has exploded in popularity.
STP205 - Making it Big Without Breaking the Bank by Jim Blomo - Engineering Manager - Data Mining - Yelp , Ray Bradford - Investment Partner - Kleiner Perkins Caufield & Byers, Gregory Scallan - Chief Architect - Flipboard
Join Ray Bradford from Kleiner Perkins in a frank discussion with Yelp Engineering Manager Jim Blomo, and Flipboard Chief Architect Greg Scallan , as they explore how they are optimizing their costs with AWS, and how they think about owning vs. renting hardware as they grow. Ray will also share observations and trends on how successful VC funded companies think about IT costs and the right things to be spending money on.
STP301 - Building in the Cloud Best Practices by Sarah Novotny - CIO - Meteor Entertainment , Ralph Gootee - CTO - PlanGrid , Joe Ziegler - Technology Evangelist - AWS, Bill Platt - Sr. VP of Operations - Engine Yard , Kate Matsudaira - VP of Engineering - Decide , Adam D'Amico - Director of Technical Operations - Okta
Amazon Web Services are fundamentally changing the way organizations are building new applications and services by enabling them to focus on their customers needs, and not on the underlying infrastructure and technology. Organizations are dramatically accelerating their agility and time to market, enabling developers to rapidly deliver new applications and then quickly iterate based on customer and market response. Businesses no longer have to focus on scaling databases, the unique challenges of big data and the complexities of highly available global applications and instead invest directly in building their market offerings. This panel will speak to the best practices for creating applications on AWS from decision makers of companies who have successfully built enterprises on top of AWS and are benefiting from the advantages of running in the cloud.
STP302 - Top Venture Capitalists Discuss Investing in the App and Cloud Economy by Ann Winblad - Managing Director - Hummer Winblad Venture Partners , Michael Skok - Partner - North Bridge Venture Partners , Brad Steele - Business Development Manager - AWS, Matt Mcllwan - Managing Director - Madrona, Matthew Miller - Partner - Sequoia Capital
This panel of top venture capitalists discusses how their investment thesis and strategies have evolved over the past several years in light of rapid and low cost development cycles made possible by the cloud. They share insights into how they evaluate investments in various vertical markets and stages, what metrics they use to monitor progress and success of their portfolio companies, provide advice and best practices to entrepreneurs looking to raise capital, and share what they see on the cloud horizon that will change the world. This panel is moderated by Brad Steele of Amazon Web Services.
SVC101 - Building Search into Your App by Jon Handler - CloudSearch Solutions Architect - AWS, Matt Norton - Director of Application Development - Public Broadcasting Service
Amazon CloudSearch is a fully-managed search service in the cloud that allows customers to easily integrate fast and highly scalable search functionality into their applications. In this session, we cover the basics of search and search engines. We take an introductory look at CloudSearch along with a deep dive showing how to build a CloudSearch-based web application.
SVC103 - The Whys and Hows of Integrating Amazon Simple Email Service into your Product or Website by Chris Wheeler - Sr. Technical Program Manager - AWS, Rohan Deshpande - Software Development Engineer - AWS, James LaPlaine - Vice President, Technology Operations - AOL
If you're already building your website our application on AWS, using Amazon SES is a quick and cost-effective way to send your email. This session will talk about what Amazon SES is and why you would want to use it. Then we will dig into the most common ways our customers use Amazon SES with their current systems and give you the tools you need to do the same.
SVC104 - AWS Marketplace by David Zipkin - Sr. Manager, AWS Marketplace - AWS, Barry Russell - Sr. Manager, AWS Marketplace - AWS, Paul Wallace - Director, Product Marketing, Stingray Business Unit - Riverbed Technology, Misha Govshteyn - Co-Founder & Vice President of Emerging Products - Alert Logic
AWS Marketplace is an online store that helps customers find, buy, and immediately start using the software and services they need to build products and run their businesses. Visitors to the marketplace can use AWS Marketplaces 1-Click deployment to quickly launch pre-configured software and pay only for what they use, by the hour or month. AWS handles billing and payments, and software charges appear on customers AWS bill. The first half of this session talks about what AWS Marketplace offers to AWS customers buying software and the second half of this session discusses why AWS Marketplace is a great place to sell software.
SVC105 - AWS Messaging by Jon Turow - Sr. Product Manager - AWS, Clay Magouyrk - Software Development Engineer - AWS
Amazon SNS and Amazon SQS are important systems for sending, managing, and queuing system notifications. This session details the fundamentals on how to use these services, demonstrates the value for application developers, and covers some common use cases and customers where they have solved a critical business problem.
SVC201 - Distributing Work in the Cloud with Amazon by Rick Sears - Software Development Manager - AWS
Applications today can span on-site and off-site environments, as well as across multiple compute resources in the cloud. Come learn how to simplify your applications state management, asynchronous tasks and work distribution with Amazon Simple Workflow (SWF). During this session, you will learn how to use the SWF Flow Framework to define your application logic in workflows that are managed at high-scale and with fault-tolerance by Amazon SWF.
SVC202 - Scaling Your Application by Andrew Kerr - Principal Engineer - Fluid.com , Andrew Guldman - VP of Product Engineering - Fluid.com , Mai-lan Tomsen Bukovec - Director, EC2 Compute - AWS
Dealing with scale and concurrency in todays web and mobile services can require complex business logic in your application. To achieve high scale in the cloud, often developers have to coordinate and track state for steps in application processes distributed across remote data centers. Come to this session to learn how Amazon Simple Workflow (SWF) manages and coordinates your application sequences in workflows by our AWS pay-as-you-go service. We will walk through real-world examples of customers who are basing their high-scale, fault-tolerant applications on Simple Workflow today.
TLS301 - Deploying to the AWS Cloud with Visual Studio by Norm Johanson - Software Development Engineer - AWS, Steve Roberts - Software Development Engineer - AWS
The AWS Toolkit for Visual Studio contains many features to help you deploy and maintain your applications in the AWS cloud. In this session, learn how easy it is to deploy and manage your SQL Server powered ASP.NET application from the IDE and command-line tools using AWS Elastic Beanstalk and Amazon RDS. The session also includes some discussion of automated deployment with continuous integration build systems.
TLS302 - Being Productive with the AWS SDK for Java by Jason Fulghum - Software Development Engineer - AWS
The AWS SDK for Java includes several higher level APIs that make working with AWS simpler. Learn more about higher level APIs such as TransferManager, which allows developers to easily manage asynchronous uploads and downloads from Amazon S3; the AWS DynamoDB Object Persistence Layer, which allows developers to annotate their Java classes to specify how the SDK should map them to AWS DynamoDB tables when they are saved and loaded; and other higher level APIs such as the Amazon Simple Email Service JavaMail provider.
TLS303 - How to Deploy Python Applications on Elastic Beanstalk by James Saryerwinnie - Software Development Engineer - AWS
Learn how to configure, deploy and scale a Python application running on Amazon Elastic Beanstalk. This talk uses two samples, a simple url shortening API built using Flask, and an image processing app built using Django, to demonstrate how to quickly get up and running on Amazon Elastic Beanstalk. In addition to learning best practices, the talk covers performance tweaks, and options for scalable data storage including S3, DynamoDB and RDS.
TLS304 - Getting Productive with the AWS SDK for Ruby by Trevor Rowe - Software Development Engineer - SDK and Tools - AWS
Learn best practices for using the AWS SDK for Ruby, including configuration, logging, debugging, consuming high and low level interfaces, collections, memoization, Rails integrations, AWS::Record and more.
TLS305 - Using Amazon DynamoDB Effectively with the AWS SDK for PHP by Jeremy Lindblom - PHP Software Engineer - AWS, Michael Dowling - Software Development Engineer - AWS
Learn how to work with Amazon DynamoDB using the AWS SDK for PHP. Attendees learn about the AWS SDK for PHP including how to install and configure the SDK and how to perform operations with DynamoDB. Advanced discussion topics include tips for effective DynamoDB usage, request batching, performance tuning, configuring event listeners, and setting up the included DynamoDB session handler for session storage.
TLS306 - Develop, Deploy and Debug with Eclipse and the AWS SDK for Java by Zach Musgrave - Software Development Engineer - AWS
The AWS SDK for Java and the AWS Toolkit for Eclipse enable developers to easily manage AWS resources, quickly build web scale Java applications that interact with AWS services, and deploy those applications to the AWS platform. In this session, learn what functionality the AWS SDK for Java and the AWS Toolkit provide, see common usage scenarios with the AWS SDK for Java, and discover how to use the management, deployment, and debugging capabilities in the AWS Toolkit for Eclipse.

What is the meaning of "Cloud for the Enterprise"?

2013-01-21T12:23:00.001+11:00

What is the meaning of “Cloud for the Enterprise”, or its synonym “Enterprise Cloud”? The terms is used often, mostly as a throw away phrase. What is different about Cloud Computing which is “Enterprise” in nature, and that which is not? The phrase also comes up frequently in the discussions around “Enterprise Cloud adoption”.

This has been a topic that I have been wrestling with in the back of my mind over the last month, probably due to me starting my new role at Amazon Web Services this past week, specifically to work within the Enterprise customers. Therefore over the break I have really been considering, what on earth is “Cloud for the Enterprise”?

In thinking “Cloud for the Enterprise” there are three areas that we need to consider. Who is the enterprise, what are their applications and what are their requirements.

Who is the Enterprise?

What do we mean when we are referring to the Enterprise?

The term Enterprise can sometimes refer to simply being commercial in nature. For example you will often hear people discussing services such as DropBox being for consumers. Where as alternative services, such as Huddle or Syncplicty, are characterized as being for the Enterprise. Likewise some companies also have different editions or service levels of their Cloud services. For example Crashplan have consumer, business and enterprise editions.

In our understanding then we need to acknowledge firstly that the phrase is referring to Cloud services which are targeted to business use and not the consumer space.

Next the term Enterprise can often refer to the size of the business, where size either equates to the number of employees or the dollar value of turnover. Typical the business sizes are broken into categories of Small, Medium and Enterprise. Often the Small and Medium are joined to form a category Small-Medium Business (SMB) or Small-Medium Enterprise (SME). Whilst there are no standard on the splits especially across countries, typically if you have less than 500 employees you are a SME. These cutoffs are the stuff that Sales people loose sleep over as they have a big impact on their territories. If you are interested to know the number of businesses of different employee sizes in the US, they can easily be found in the census data tables.

Therefore in our understanding we want to acknowledge that we are discussing large scale entities. Yes, small and medium enterprises can and are consumers of Cloud services, however I believe when we hear the term “Cloud for the Enterprise” we are not discussing them.

To summarize our first area; when we discuss “Cloud for the Enterprise” we are discussing:

Business rather than consumer based services.
Where that business is large, typically 500 or more employees.

What are Enterprise applications?

The second area that we should be thinking about in “Cloud for the Enterprise” is Enterprise applications or Enterprise software. Often when the phrase is used, it is attempting to make a distinction in the nature of the application workloads. I believe that the reference is to the breadth and the nature of Enterprise applications.

First, breadth.

In the business world there is usually a differentiation when things become “Enterprise” in nature. Anecdotally the most obvious is that the price tag has extra zeros on the end. I think that Mark McDonald stated what this means really well.

People talk about the ‘enterprise’ all the time, particularly in IT. There are enterprise solutions, enterprise architecture, enterprise portals, enterprise security, etc. In this context the term enterprise is more often meant to mean all encompassing, across business units or geographies, the whole of our business operations.

This breadth aspect of the “Enterprise” attribute, particularly in IT is very helpful. We know that there is great adoption of Cloud services within businesses that are Enterprise in size. Many of these deployments have not been concerned with finding Cloud services that are labeled as Enterprise in nature. We have all heard the stories of Cloud deployments that were skunk works funded through a corporate credit card. Likewise there have been significant deployments which have been blessed by or even driven by IT, but they have remained as isolated deployments, special projects, or have been given approval to live outside of the normal regime.

My perception is that when we are discussing “Cloud for the Enterprise” we are also talking about deployments and solutions that are broad. Where the usage is the normal rather than the unknown or the exception. Where the use case runs across the breadth of the business and not for isolated solutions.

Second, nature.

Enterprise applications are software that is used to run the business rather than software used by an individual. So whilst a user might use Microsoft Excel to track and manage business functions it is not typically viewed as an Enterprise “application”. A helpful definition from Wikipedia is

Enterprise application software is application software that performs business functions such as order processing, procurement, production scheduling, customer information management, energy management, and accounting. It is typically hosted on servers and provides simultaneous services to a large number of users, typically over a computer network. This is in contrast to a single-user application that is executed on a user's personal computer and serves only one user at a time.

Given the nature of these systems they are usually large and complex, much of the business process is integrated and intertwined to the application. It is typically expensive and time consuming to implement or change from one application to another. These are the critical application that the whole business relies on to generate its revenue and service its customers.

Many Enterprise applications, due to their legacy have certain architectures. Often these architectures are very different from the new paradigm that is Cloud computing. Traditional Enterprise applications are often (but not always):

Scale up rather than scale out
Have specific requirements in hardware and software compatibility
Mandate specific and consistent performance from underlying infrastructure
Are intolerant of failure in any underlying infrastructure and rely on the high availability of all components

It is a hyper exaggeration but a characterization could be made that modern Cloud applications are scale out, distributed systems built on Open Source Operating Systems and software, such as Linux. Enterprise applications on the other hand, are scale up, monolithic systems built on proprietary Operating Systems and software such as that from Microsoft or Oracle.

Enterprise applications, even leaving aside the way they are architected, are often sitting in a different class from others within the business. These are your “bet the business” workloads.

To summarize our second area; when we discuss “Cloud for the Enterprise” we are discussing:

Where the solution has breadth and depth across the business rather than being an exception.
Critical application that the whole business relies on to generate its revenue and service its customers.
Often the applications are based on a traditional rather than cloud architecture.

What are the requirements of an Enterprise?

The third area of consideration is that of Enterprise requirements. Because we are dealing with much larger businesses and the workloads are critical in nature the requirements placed on the service are different.

Requirements are an area where there will be great variation across different businesses and different applications. Even without introducing Cloud computing into the mix there is great variance in business maturity, execution and industry regulation. For example a business in the financial services industry is more regulated and manages data of greater sensitivity and risk, than a business in the media industry.

However there will be requirements that are shared across Enterprise solutions. The list of requirements below outlines my starting thoughts, but I am sure these will grow as others provide some valuable insight.

Enterprise requirements for Cloud, include:

Data sovereignty - there will be more concern over the geographical storage of data.
Private networking – whilst many may interface to the public Internet there will be a dominance of deployment onto private networks, integrated with corporate wide area networks.
Authentication, authorization and accounting (AAA) - with more staff and larger deployments sophisticated access controls and tracking is required, often integrated to existing systems.
Security – including third party certifications, which validate procedures and policies that the Cloud provider claims to be operating.
Detailed billing - which can also be interfaced for analysis and chargeback.
Reference cases and best practices - a propensity to not be the first at adoption and that best practices and deployment guidelines are readily available.
Support services - that can sustain the service levels the Enterprise is required to deliver to the business for the applications.
Maturity in feature set - as the use cases are border and diversity across the business needs to accommodate the greatest amount of consolidation.
Application vendor support - where the Cloud provider and vendor have worked together to test functionality and that cross company support arrangements are in place to resolve incidents in a timely manner.
Maturity of the Cloud provider - that is of sufficient size to engage with large business and meet their unique needs.

Conclusion

Ultimately it will be customers to define for the industry what “Cloud for the Enterprise” means. As Cloud practitioners and as an industry we should be clear in our usage of terms and what they mean. We should not be Enterprise Cloud Washing, taking any Cloud and throwing an “Enterprise” label on it. When we are discussing Cloud for the Enterprise, we want to know what that generally means, as well as what it does not mean. Most importantly we want to understand what might make it different.

“Cloud for the Enterprise” is for large business, for those strategic services that support the key functions of the business operations. The service will deliver to the requirements demanded by these entities and workloads.

Appreciate your comments and contribution.

Rodos

P.S. As always, remember the views expressed are my own person ones and do not assert the view of my current or past employers.

P.P.S. This is my starting position. I don’t think I have locked it all down yet. Over the next weeks I expect to have some great conversations with some really smart people at work who I am sure will have some golden nuggets on the topic. Just like a meeting I had many years ago with Kunjal Trivedi whilst he was at Cisco; his insight in Cloud way back then gave me many epiphany moments, which I milked for insight for years. Also there was some discussion of this topic on a recent SpeakingInTech podcast.

Architecture effects cost

2012-10-30T18:55:00.001+11:00

"Architecture effects cost", thats a simple and somewhat obvious statement. Those of us who architect systems know that there are outcomes for the design decisions that are made. Often there are tradeoffs and no environment lives in a world where money is unlimited. Well okay the unlimited funds projects probably do exist somewhere, but I have not had the pleasure of working in one of those environments in my career.

The concept of architecture effecting costs is a key element when you are architecting for Cloud solutions. One of the reasons this is key, is that your ongoing costs have a greater potential to be variable. This is a good thing, one of the benefits of moving to Cloud models is the elastic nature and you want your applications to be right sized and running efficiently both technically but also financially.

There are two considerations when thinking about costs for Cloud.

First there is the selection cost. As an example, if you were using Amazon S3 there are two durability options available, one is lower durability and also lower cost. If you have scratch data or data that can be regenerated you might simply choose to use the less expensive storage.

Second is then understanding your elastic cost, based on your predicted demand. Based on your forecast demand, what will your peak and average expenditure be over time? What size of cheque are you signing up to if you implement your selected solution?

An example of this crossed my mind this last week as a read a blog post from VMware on their vFabric blog. It was entitled "ROBLOX: RabbitMQ, Hybrid Clouds, and 1 Billion Page Views/Month".

Don't be distracted by the 1 billion number. Its an interesting article on how the application architecture for this company needed to be changed as they started to scale out. The example was an AWS customer and in order to speed up their service they introduced a message queue to decouple the synchronous relationship between the web services and the back-end services (as you do). Here is a pertinent bit of text.

To dive deeper, ROBLOX implemented RabbitMQ to help deal with the volume of database requests and slow response times. The queue is managing 15 million requests a day. The example scenario is when there is an update to the ROBLOX catalog content, which needs to update the search index. Before RabbitMQ, the web server would have to wait for an update to complete. Now, the loosely coupled architecture allows the web server to update the message queue, not the database, and move on (technically, the update is enqueued and picked up by the consumers). At some point, the message is picked up by a service that updates their search platform index.

Wonderful stuff. But but it hit me, why bother implementing RabbitMQ? It does not sound like they were using a lot of its sophisticated functions, surely they could have used the AWS Simple Queue Service (SQS). Thats when the "architect for cost" mantra kicked in. 15 million requests a day, that does not seam like much but lets see what that would cost for SQS. SQS is $0.01 per 10,000 requests and there are no data charges if the transfer is within a single region. Thats $15 a day which is reasonable. Thats about $465 a month. Thats about the same prices as a full license over a year potentially for RabbitMQ (or you could use the free version) but that is now a fixed cost and you have to factor in the cost for running the server to execute it along with the effort to maintain it (no cheating on the hidden costs). Looks like others have pondered this SQS vs RabbitMQ question as well (1, 2). However this is just an example of the point which jumped out this week.

So when architecting for Cloud, don't forget "architecture effects costs". With a Cloud there is not only the selection cost but also consider your elastic cost.

Rodos

Secure Multi-Tenancy

2012-10-17T19:22:00.002+11:00

Edward Haletky (blog) once asked me what I thought secure multi-tenancy was in relation to Cloud Computing. I am willing to admit that at first I gave an answer reminiscent of a Ronald Reagan quote.

Due to the painful memory of this event the actual definition of what security multi-tenancy is has always had my attention. I work with a lot of vendors and despite what their marketing and sales people declare their understanding of this topic is wide ranging in maturity and reality. Just as there is Cloud washing there is a lot of secure multi-tenancy washing going on.

This week I actually had to sit down and document in a paper what secure multi-tenancy is. How would we know when it had been achieved?

Of course one usually starts such an activity with Wikipedia and Google, just like my children do for their school work.

Given how much you hear this phrase these days its a surprise that there is no clear and concise definition out there. One of the earlier ones is from the Cisco Validated Design that they produced with Netapp. It states

... the capability to provide secure isolation while still delivering the management and flexibility benefits of shared resources. Both private and public cloud providers must enable all customer data, communication, and application environments to be securely separated, protected, and isolated from other tenants. The separation must be so complete and secure that the tenants have no visibility of each other.

That is reasonable but personally I did not feel it catered as a measure of success. So I sad down and tried to summarise what I have learnt and have been implementing over the last few years.

Secure multi-tenancy ensures that for a shared service:

No tenant can determine the existence or identity of any other tenant.
No tenant can access the data in motion (network) of any other tenant.
No tenant can access the data at rest (storage) of any other tenant.
No tenant can perform an operation that might deny service to another tenant.
Each tenant may have a configuration which should not be limited by any other tenant's existence or configuration. For example in naming or addressing.
Where a resource (compute, storage or network) is decommissioned from a tenant the resources shall be cleared of all data and configuration information.

I have tried to keep it as succinct as possible. The last item regarding the clearing of all data and configuration is most familiar to people as the "dirty disk" problem. You could consider that this item is a duplication of the 3rd point, that no data of one tenant can be accessed by that of any other. Yet people tend to forget about the residual information of both configuration or information that may remain thus introducing vulnerabilities. Thanks to my colleague Jarek for contributing this 6th item to my original list.

Do you consider the environments that you use meet all of this criteria? Does this criteria cover the required elements or does it cover to much? Appreciate your comments.

Rodos

Drobo Update - My experience of a failed disk drive

2012-10-10T20:48:00.001+11:00

Back in November 2009, thats 3 years ago, I did a post about Drobo. What made this interesting was that it included a video of my then 12 year old son doing the unboxing and deployment of the device.

Well 3 years later a lot has changed. My son Tim is more of a geek than I am and a heck of a lot taller, and the range of Drobo hardware has changed. However the Drobo box has been faithfully running those last three years without incident. That is until last week when it suffered a drive failure.

When I originally installed the Drobo I put in 3 x 1TB drives. Then after about two years, as usage started to increase, I picked up another 2TB drive to insert into the remaining slot. No config required, just inserted the drive and let it do its thing. Thats the benefit of BeyondRAID.

Last week at work I get a text from Tim.

The drobo lights are flashing green then red, its got about 480g free. I dont have the drobo dashboard but if you tel me which drobo it is I can get it and see what it says.

Of course I remember, unlike Tim, that there is the indicator light cheat sheet on the back of the front cover. I ring Tim and he reads out what the lights mean.

The chart reveals the problem, the new 2TB drive has failed and a rebuild is occurring on the remaining 3 drives. Thats good. We have not lost any data, the unit is still operating, we can read and write data. The only change is that the available free space on the unit has reduced, as indicated by the blue lights on the front.

After quite a few hours, Tim and I start to get impatient about the rebuild time. I of course am expecting the rebuild to take a while, I know that even in big enterprise storage arrays this can take a while and that the older Gen2 Drobo that we have is does not have much grunt in its processing power.

To try and determine the time I browse a few sites on the Internet to read what others have experienced. It was a little disturbing to read so many horror stories about peoples rebuild times, sometimes its weeks and other stories of their units failing. Sounded a little ominous so we installed the Drobo Dashboard onto the Mac Mini it was now connected to in order to determine the rebuild time.

The estimate for the rebuild was another 30 or so hours and it had probably been running around 12 hours already. We went to bed thinking we had a long wait ahead. In the end the rebuild finished ahead of schedule and probably took around the 24 hour mark. For the age of the device and the fact that we lost the largest drive in a box and that was reasonably utilised, I think that is a reasonable time.

Turned out that the drive that failed was still under warranty (thank you serial numbers) but we figured we might as well go and get another 2TB drive and get back some free space, when the RMA arrives we can swap out one of the smaller drives with the larger one and get some new space.

After a fun trip to a computer store we slotted in the new drive. The crazy thing was that within under a minute the indicator lights started flashing again and one of the original 1TB drives was showing red, another failed drive. I have no idea if it was good or bad luck! But the drive was certainly failed. The rebuild was much faster this time.

A day or so later we found a spare 1TB hard drive in the study and threw it in the slot of the failed drive. All great. We are now back to where we were, plenty of space and redundancy. Here is the current state.

Now once that 2TB drive returns from the RMA we will still swap out one of the original drives.

So whats my thoughts on the Drobo after 3 years and experiencing a real life drive failure?

Everything works as advertised.
The setup was easy, we experienced that 3 years ago.
When a drive failed it was great that everything continued to operate as normal, we could still read and write data as we wanted.
It was great that it did not matter if we had the Drobo Dashboard software up to date or even installed, the unit took care of everything itself.
The rebuild after a failure did take time, but in our experience that time was reasonable and as it did not really on the computer, and that we could still utilise the device, the only thing inconvenienced was our patience.
Even thought the unit is starting to age, the software and firmware updates are still available.
Never trust a single device. All of data that is critical such as family photos, but also general data we would be inconvenient to loose is also backed up to the Cloud using CrashPlan. I don't care how highly available a storage unit is, it is not backup if its your primary copy! That is one concept I think a few of those people complaining about their Drobo experiences need to take heed of.

There is sometime attractive about technology that is so smart it can make things so simple to use.

Rodos

IDWS - Come hang out with the storage geeks

2012-10-05T08:45:00.001+10:00

If you are in Sydney next Tuesday (9th of October) come hang out with the storage and information systems geeks.

IDWS, or Information & Data World Symposium is on and a great place to catch up with vendors and providers, share experience with colleagues and industry, and of course learn new things.

Here is the blurb

A comprehensive technical symposium designed for data management professionals and IT practitioners to broaden their knowledge into all facets of building and maintaining their information infrastructures.

The symposium will be educational and technical, targeted to all IT levels from CIOs to the skilled staff responsible for managing and protecting their companies greatest asset, it's data.

Be engrossed as key industry players battle it out against each other at the 'Great Debate'. Throw in live tweets and feeds from the floor including voting and see who will be crowned champion on a range of key topics.

The symposium will feature a Technical Lab for vendors to demonstrate real information infrastructure solutions as well as technical workshops to suit all delegates.

This one-day event will cover Big Data Analytics, Cloud Storage & Services, Infrastructure Convergence, Data Management & Protection, Storage Security, Virtualisation and a lot more.

Registration is free, head to http://www.idws.com.au/ to learn more and register.

See you there.

Rodos

P.S. As you know I am a Board Member of SNIA who are part of this event.

Crashplan gets Australia based service

2012-09-01T13:09:00.001+10:00

Crashplan from Code42 is one of those services that are so great that I just tell everyone about it. You never have a problem recommending something that has worked so well for yourself and that you know others will really benefit from.

I have always recommended crashplan to colleagues, friends and family for many reasons.

It is free if you want to do local backup or backup to a friend.
Their optional Cloud backup destination is all you can eat and competitively priced.
You can backup to multiple locations with different data sets, which is a key feature.
It's very secure when backing up locally or remotely.
It's automated and just works, set and forget.
You can access the files you have backed up via an iPad app.
I have tried other services and they were not as good.

So I was really please when at VMworld this week I ran into the CrashPlan stand.

On the stand was their chief marketing officer and he happened to mention that they were launching an Australian based presence for their service. My accent must have given away that I was Australian, go figure.

Here is what their press release says about it

"We've always had a strong customer base in Australia. Now, with the opening of our Australian office, we’re positioned to serve this fast-growing market much more effectively," commented Matthew Dornquast, co-founder and CEO, Code 42 Software. "In addition, having a local data centre means we can deliver even better performance for our cloud backup customers located across the entire Asia-Pacific region."
The new data centre will provide state-of-the-art, cloud-based backup services to Australian users of Code 42‘s CrashPlan products:
• CrashPlan+ - the award-winning computer backup solution for home users.
• CrashPlan PRO - the innovative backup system for small and mid-sized businesses.
• CrashPlan PROe - the world's most advanced backup and disaster-recovery solution for enterprises.
A local data centre also enables Code 42 to extend its popular U.S. “seeding” option to Australian customers. With this option, Australian customers have the option of shipping their initial backup to Code 42 where it’s then loaded directly onto CrashPlan servers. “This approach is extremely beneficial because it can save our customers a lot of time, especially those with large initial backups or where bandwidth is limited,” Dornquast explained.

Latency always adds some overhead to these network applications so having an Australian presence is going to help all of us Australian users. Plus the ability to send them you're data for seeding is great for those with slow Internet links.

However, if you are already a happy customer, like myself, you will continue to use the US servers, for the moment at least. Only new Australian customers will use the new site. According to the person I spoke to they are working on the process for moving existing customers on shore and their was no timeframe for when this might be done by. Definetely something we need to all encourage them to do. I don't want to start a new service and have to push my 0.5TB back again plus loose all of my regions and deleted files.

So if you were looking for a backup service for your personal machine, you have even more reason to give Crashplan a look!

Rodos

P.S. I will have to see if I can go visit their Australian office and interview their local staff to find out more about their environment. I should also write up my best tips for using Crashplan.

Posted with Blogsy

BigData. So what?

2012-06-05T10:52:00.001+10:00

Sometimes it takes a bus trip to connect the dots. In my case today these were BigData and a Wired Magazine article.

We have all been hearing a lot about big data lately. If a vendor has little to say, or possibly said everything they can, about Cloud then they just search and replace the marketing materials with the phrase "Big Data". We are not at the stage where McDonalds has decided to replace the BigMac with the BigData burger so the consumer world is safe for the moment, but most CIOs are probably getting their in tray full of promotions and case studies.

Whilst I get big data and see its value, I have personally struggled with the realities of execution. We have been reading about the increasing demand of developers skilled in Hadoop and I have a college who is a CCIE, got into Cloud and is now chasing the Hadoop angle. But to me BigData itself brought no real shift in ability to execute here. It might be cheaper and easier to store and process big data these days, but the insights have always been a human effort. the human effort to develop the analytics takes intellect and scale. There was the rub, not all humans have the same intellect and humans don't scale in the specialist areas. I have a friend who works for Oracle in demand planning. His is real smart at building data mining for global companies that need to forecast all sorts of whacky things. Yet he is very specialised and uses some real high end software. The gap between those people with big data, and those who can do something with it, has always irked me.

So today I am on the bus reading Wired on my iPad, as you do, and read an article "Can an Algorithm Write a Better News Story than a Human Reporter?". Have a scan thru the article but the premise is that given large amounts of statistical data companies such as Narrative Science and turn it into a news story that is very insightful. They started out doing this with children's baseball games. Feed in the play by play data and it generates a story such as

Friona fell 10-8 to Boys Ranch in five innings on Monday at Friona despite racking up seven hits and eight runs. Friona was led by a flawless day at the dish by Hunter Sundre, who went 2-2 against Boys Ranch pitching. Sundre singled in the third inning and tripled in the fourth inning … Friona piled up the steals, swiping eight bags in all …

Baseball, financial markets, they can do some amazing stuff. Many companies are actually using machines to find insights and produce prose. As mentioned

Once Narrative Science had mastered the art of telling sports and finance stories, the company realized that it could produce much more than journalism. Indeed, anyone who needed to translate and explain large sets of data could benefit from its services. Requests poured in from people who were buried in spreadsheets and charts. It turned out that those people would pay to convert all that confusing information into a couple of readable paragraphs that hit the key points.

and

And the subject matter keeps getting more diverse. Narrative Science was hired by a fast-food company to write a monthly report for its franchise operators that analyzes sales figures, compares them to regional peers, and suggests particular menu items to push. What’s more, the low cost of transforming data into stories makes it practical to write even for an audience of one. Narrative Science is looking into producing personalized 401(k) financial reports and synopses of World of Warcraft sessions—players could get a recap after a big raid that would read as if an embedded journalist had accompanied their guild. “The Internet generates more numbers than anything that we’ve ever seen. And this is a company that turns numbers into words,” says former DoubleClick CEO David Rosenblatt, who sits on Narrative Science’s board. “Narrative Science needs to exist. The journalism might be only the sizzle—the steak might be management reports.”

This is where the dots connected and I became a lot more relaxed about big data. Here we have the birth of what can start to give reality to big data capture and processing. Whether you view it as AI, clever algorithms or plain ole automation does not matter. Looking forward you can see how companies can cheaply and easily generate business insights from the data they collect.

Until these analytic services mature you might want to brush up on your hadoop skills, but in the future you might just start getting more emails from a automated account like the following.

"Rodos, yesterday there was a flood of traffic on the fibre channel network that was generated from workloads in the Melbourne IaaS availability zone. Looks like this was mostly from a specific customer and I also picked up that their Unified Communications workloads in the UCaaS nodes in Singapore peaked. The company in question just listed on the stock exchange in Hong Kong and forecast interest in their services, if it continues at the rate, will cause increased workload that will take the Melbourne availability zone B to 90% capacity. Last time zone B hit 88% capacity (Sept 2014) SLAs for 2 customers were broken. Just a heads up, regards Siri".

VFD2 - Xangati

2012-02-24T19:44:00.000+11:00

Third and last on the first day of Virtualisation Field Day #2 was Xangati.

There was great food presented before the start of the session, ice-cream and a variety of bacon in different flavours. The idea was not to mix them (hey they do weird things with food in America) but to provide choice. The bacon was very well received by the delegates. For some reason bacon is very popular.

Jagan Jagannathan, the founder and CTO grabbed the whiteboard pen and started to explain things. It worth repeating (and it gets repeated a lot), this is exactly the type of engagement and insight that delegates at TFDs respect and value. You can tell the difference. The room is quiet, the questions and interruptions are minimal, people listen intensity. I have seen this again and again at these days. It staggers me how some vendors do it and succeed and other ignore the advise and struggle. In a similar way, Xangati provided a list to each person of who was present, their names, titles and twitter handles. When you are writing up notes and posting on twitter, this easy access to info is so very helpful. You would think that the PR people at Xangati had not only read the advise given them (http://techfieldday.com/sponsors/presenting-engineers/) but they actually attempted to put it into practice!

I was engaged with what was being presented and ended up not taking many notes.

Some of the interesting things discussed was how in the performance monitoring world you have triage vs postmortem. For triage you need real time, at this minute information, anything older than that and its not as useful. Older data such as five minutes later, thats used for postmortem analysis.

One of the key things that Xangati does is take all of the incoming data and process/analyse it in memory, rather than writing it to a database for analysis. This allows them to give very timely and detailed information in their UI and alarming. The interface has a slider and you can wind back the clock a little and see what was just happening prior to now. You can also record the detailed real time information you are looking at for later analysis. This recording links in with their alerting. That is when an alert is created it records the associated real time info for that short time period so you can see what was happening. Of course the data, is also written to database in a summarised form for later analysis. This uses a reporting interface that is not as nice or as interactive as the real time interface. I would like to see the two much more similar, its feels a little strange to have them so different. However given that they work of different data models and server different purposes you can see the reasons why.

They have self calculating thresholds but you can create your own.

Xangati have been doing a lot in the VDI monitoring space but they were keen to point out that they are not a VDI monitoring company, they do straight virtualistion too. I think they don't like being tarred to much with the pure VDI brush.

The do have some great VDI features though. If you are looking into desktop performance you can launch into a WMI viewer as well as a PCoIP or Citrix HDX viewer to see a lot more detail about whats going on inside the desktop and the display protocols. They even have a neat feature where and end user can self service a recording of their performance for a help desk to analyse. The user can go to a form and request a recording for their environment, it records the previous 1 minute before the submission. Thats nice.

Here is a look a the demo environment I interacted with.

Where there are reports that have sub elements (such as a protocol list) you can drill down to those. At first I thought the reports were not interactive, but I was wrong about that and shown the error of my ways.

It was a good session. I certainly got the impression that for real time performance trouble shooting Xangati is a real player worth investigating. I did not get enough of a chance to look at the product or discuss with them its suitability as an comprehensive monitoring solution. I think there are a few things that an overall monitoring solution requires that I did not see in the product, for example inventory data. Maybe a more in-depth look at the features and functions would help nut this out more. Hard to do in our limited time. They do have the free version which is popular and evils are available, so its easy to check these out for yourself.

Well its been a long day, lots to see and think about. Looking forward some brief sleep before another day of it all tomorrow.

Rodos

P.S. Note that I am at this event at the invite of GestaltIT and that flights and expenses are provided. There is also the occasional swag gift from the vendors. However I write what I want, and only if I feel like it. I write nice things and critical things when I feel it is warranted.