Chris Moyer

Splunk Storm: Another Cloud Logging alternative?

2012-02-02T10:16:00.000-08:00

In December, I posted an article on My Great Search for Syslog services. Since then, We've started using Papertrail to log and manage our numerous cloud-based servers. Papertrail so far seems great, it gives us a live tail of our logs, as well as search ability and even the ability to archive all of our log content to S3. (I love when services offer exporting to S3).

One major drawback of Papertrail, however, is that it's really only useful to tech-savy users. You can only spot issues if you know exactly what you're looking at. It doesn't really have a way to map what your logs say into any meaningful segments. It does do a decent job most of the time trying to pick some common things out, but my logs are very structured, it would be nice if I could explain my structure to Papertrail.

I recently received a Beta invite to the SplunkStorm project (I can only assume it was in relation to my blog post). I never looked at Splunk before, since it really is an "in-house" solution, not a service. This new model, however, is just what I'm looking for: A true Syslog service offering a nice pay-as-you-go structure (or so they're planning). It's still in beta so no prices are out yet, but from what I can tell the pricing structure will be based on the data that's searchable. In Papertrail, you pay for how much log data you send into the system, whereas SplunkStorm meters by how much data you retain in the system. Once there's too much info in your Project, old data is removed to make way for the new stuff. This means you always can send data into your project and continue to search it. They don't simply start dropping logs after a certain period of time.

Ok, so that's the pricing structure (or so it appears), but what about functionality?

Well, they offer a very complex search system, which includes being able to graph (something Papertrail folks are still working on), and the ability to do other very complicated behaviors.

But it gets better...

They have a unique "Dashboard" system, which lets you set up custom graphs based on specific events, splitting on any field in your logs.

Yes... Field, not hostname, not application name, field.

So what is a field?

They have several "templates" that pull out common information (for example, one template parses an Apache log), but if yours doesn't quite fit? no problem, just use their Template system and you can give it any regular expression and mapping out to custom fields.

First you set up a thing called a Field Transformation. Under Manage->Fields->Field Transformations you can clone an existing one, or start from scratch. Setting up a Field Transformation is relatively simple, you give it a regex and a mapping. My boto config specifies this logging format for my logs that go to syslog:

%(name)s[%(process)d]: %(levelname)s %(message)s

Unfortunately, when using multiple processes/threads with the logging module, you'll often get crap in your log messages:

<11>Feb 2 18:06:09 hostname <14>botoweb.url_mapper[3000]: INFO [username] GET: /

these <14> and often <11> appear in the logs just before the date and log name in most (but not all) of my log lines.

Ok, so lets parse this out. We'll start off with a Transformation named syslog-boto.

Enter my massive regex:

^(<\d+>)?([A-Z][a-z]+\s+\d+\s\d+:\d+:\d+)\s([a-zA-Z0-9\.\_]+)\s+(<\d+>)?([a-zA-Z\.\_\-]+)\[(\d+)\]:\s+([A-Z]+)\s+(\[([a-zA-Z0-9_]+)\]\s+)?(.*)$

And my "format":

date::$2 host::$3 process::5 pid::$6 level::$7 username::$9 message::$10

And tada! We now have a proper way to map these logs to their different field names.

But wait, it's still not quite completed! You have to then set up a Field Extraction that tells your project when to use this template. I set mine up like this:

Name: syslog-boto
Apply to: sourcetype
Named: syslog
Type: Uses transform
Extraction/transform: syslog-boto

Save, and now check your logs, you'll see they're parsed out now. You can even search by field name such as this:

level=info AND username=cmoyer

There's so much more you can do with SplunkStorm. You can create dashboards now based on these fields. For example to chart out activity on your botoweb server by username:

level=INFO | timechart count(_raw) by username

What else are you using SplunkStorm to do?

Amazon DynamoDB

2012-01-23T07:09:00.000-08:00

Last week, Amazon announced the launch of a new product, DynamoDB. Within the same day, Mitch Garnaat quickly released support for DynamoDB in Boto. I quickly worked with Mitch to add on some additional features, and work out some of the more interesting quirks that DynamoDB has, such as the provisioned throughput, and what exactly it means to read and write to the database.

One very interesting and confusing part that I discovered was how Amazon actually measures this provisioned throughput. When creating a table (or at any time in the future), you set up a provisioned amount of "Read" and "Write" units individually. At a minimum, you must have at least 5 Read and 5 Write units partitioned. What isn't as clear, however, is that read and write units are measured in terms of 1KB operations. That is, if you're reading a single value that's 5KB, that counts as 5 Read units (same with Write). If you choose to operate in eventually consistent mode, you're charged for half of a read or write operation, so you can essentially get double your provisioned throughput if you're willing to put up with only eventually consistent operations.

Ok, so read operations are essentially just look-up operations. This is a database after all, so we're probably not just going to be looking at looking up items we know, right?

Wrong.

Amazon does offer a "Scan" operation, but they state that it is very "expensive". This isn't just in terms of speed, but also in terms of partitioned throughput. A scan operation iterates over every item in the table, It then filters out the returned results, based on some very crude filtering options which are not full SQL-like, (nothing close to what SDB or any relational database offers). What's worse, a single Scan operation can operate on up to 1MB of data at a time. Since Scan operates only in eventually consistent mode, that means it will use up to 500 Read units in a single operation (1,000KB items/2 (eventually consistent) = 500). If you have 5 provisioned Read units per second, that means you're going to have to wait 100 seconds (almost 2 minutes) before you can perform another Read operation of any sort again.

So, if you have 1 Million 1KB records in your Table, that's approximately 1,000 Scan operations to perform. Assuming you provisioned 1,000 Read operations per second, that's roughly 17 minutes to iterate through the entire database. Now yes, you could easily increase your read operations to cut that time down significantly, but lets assume that at a minimum it takes at least 10ms for a single scan operation. That still means the fastest you could get through your meager 1 Million records is 10 seconds. Now extend that out to a billion records. Scan just isn't effective.

So what's the alternative? Well there's this other very obscure ability that DynamoDB has, you may set your Primary Key to a Hash and Range key. You always need to provide your Hash Key, but you may also provide the Range Key as either Greater then, Less then, Equal To, Greater then or equal to, Less then or equal to, Between, or Starts With using the Query operation.

Unlike Scan, Query only operates on matching records, not all records. This means that you only pay for the throughput of the items that match, not for everything scanned.

So how do you effectively use this operation? Simply put, you have to build your own special indexes. This lends itself to the concept of "Ghost Records", which simply point back to the original record, letting you keep a separate index of the original for specific attributes. Lets assume we're dealing with a record representing a Person. This Person may have several things that identify it, but lets use a unique identifier as their Hash key, with no Rage key. Then we'll create several separate Ghost records, in a different table. Lets call this table "PersonIndex".

Now if we want to search for someone by their First Name, we simply issue a query with a Hash Key of property = "First Name", and a range Key of the first name we're looking for, or even "Starts With" to match things like "Sam" to match "Samuel". We can also insert "alias" records, for things like "Dick" to match "Richard". Once we retrieve the Index Record, we can use the "Stories" property to go back and retrieve the Person records.

So now to search for a record it takes us Read operation to search, and 1 Read operation for each matching record, which is a heck of a lot cheaper then one million! The only negative is that you also have to maintain this secondary table of Indexes. Keeping these indexes up to date is the hardest part of maintaining your own separate indexes. however, if you can do this, you can search and return records within milliseconds instead of seconds, or even minutes.

How are you using or planning to use Amazon DynamoDB?

What's coming up for Amazon Web Services Cloud?

2012-01-13T06:58:00.000-08:00

Amazon announced today the new upcoming Live streaming event for What's up and coming in AWS for 2012. It's not surprising since they've recently hired a ton of new people, including Mitch Garnaat, creator of Boto and former co-worker of mine. They've recently been launching new services all over the globe, including new regions and CloudFront/Route53 Endpoints. With the additional new Direct Connect locations, it's also easier then ever to build a Hybrid cloud.

They've also been launching new services, such as ElastiCache, a memcached service offered up directly from Amazon. And of course it wouldn't be amazon without dozens of improvements to existing services, such as finally enabling console logins for IAM users, and vast improvements to Amazon SNS.

So what then really can we expect at this new announcement? A few things have been on the radar for quite a while now:

Location-Aware Route53 (DNS) - Make sure that users get the closest server to where they are
Two-Factor Auth for IAM Users
APN (Apple Push Notifications) support for SNS
"Pass Through" option for CloudFront (i.e. don't cache certain URLs)
Lots of improvements for SimpleDB, such as SQL functions and cross-domain joins.

What do you think? What other improvements have you been waiting on for AWS?

Monitor your SDB Domains

2012-01-05T11:45:00.000-08:00

This recently came up in the Boto-Users mailing list, so I thought I'd post here a few quick details on how to monitor your SimpleDB domains to prevent them from hitting maximum capacity before you know it.

As you should be aware, SimpleDB has a limit of 10GB per domain. This limit is calculated as a sum of the bytes used by Item Names, Attribute Names (unique), and Attribute Values. Fortunately, Attribute Names are only stored once per name, so you don't pay for each name being used multiple times, they just charge you per unique name.

You can get all of the Usage information about your SDB Domain using the get_metadata function of a boto domain.

>>> import boto
>>> sdb = boto.connect_sdb()
>>> db = sdb.lookup("my-domain")
>>> md = db.get_metadata()

This "md" object then contains the following elements:

md.item_names_size
md.attr_names_size
md.attr_values_size
md.item_count

I wrote a simple script to check my domains, which takes an optional list of arguments for domain names to check. If you dont' pass in a domain name, it will iterate over all of them and show you any domain that uses more then 3GB:

#!/usr/bin/env python
"""
Check script to make sure none of our domains are close to the size limit
"""
import boto
if __name__ == "__main__":
import sys
sdb = boto.connect_sdb()
if len(sys.argv) > 1:
query = [sdb.lookup(n) for n in sys.argv[1:]]
limit = 0
else:
query = sdb.get_all_domains()
limit = 3000000000
for db in query:
md = db.get_metadata()
total = int(md.item_names_size) + int(md.attr_names_size) + int(md.attr_values_size)
if total > limit:
print db.name
print "\tItems:", md.item_count
print "\tItem Name Size:", md.item_names_size
print "\tAttribute Name Size:", md.attr_names_size
print "\tAttribute Values Size:", md.attr_values_size
print "\t---------------------------------------------"
print "\tTOTAL:", total

If your domain is using more then 10GB of space, you can use this to track down what's using a lot of space. In my case, I was adding a lot of unnecessary items that were almost completely blank, so my item_count was huge, and my item_names_size was over 7GB.

Of course, if you do have need for all these items, you should consider Sharding your domain into multiple sub-domains. This process is usually handled by taking one attribute that nicely splits your items into different segments, and using that value as the domain name. Unfortunately, you can not query across multiple domains, so you have to be very careful what you choose as your Shard Key.

The Great search for Syslog services

2011-12-30T07:07:00.001-08:00

I've been spending a lot of time lately looking for a good replacement for Loggly, ever since they've been having so many problems with uptime and availability. The most important feature of any log management platform is obviously to make sure it's available when I need it, and always collecting my logs. If the service drops 50% of my logs, then it's not very useful in tracking down those little bugs, I still have to log into all of my servers to see everything.

Thanks very much to Jordan Sissel for his post on Shipping Some Logs, we've decided to switch away from Loggly, to something else

Features of Log Management Solutions

What we really need out of a log management solution is something that easily integrates with our services transparently. What I mean by that is not something that will take extra code-level development work in order to use. We use Python with the standard logging module, which we then push to syslog. Syslog (or rsyslog as we use now) allows us to ship off logs to another remote server, which is really what we like to do. That means that no matter what language we use, all of our logs are able to be stored both locally and shipped off to the log management solution, without any integration with our actual programs. We also like to see logs from native linux apps like SSH, so the only real solution for us is something that integrates with syslog

Scale and Search

It also Needs to scale. We follow the rule of Log everything, even if you don't think you'll need it. Its not abnormal to have over 1 million log events, or 1GB of log data, in a single day. If you log everything, you can sort through things later to more easily find out what happened.

Since we do log everything, what we really need is the ability to search for something across all of our systems. We need to be able to trace down something that may have gone wrong and figure out exactly where it went wrong

Alerts

We also want to be able to use our logging solution to alert us if something is going wrong. Specifically we have two types of alerts:

If more then X events appear in Y minutes (error threshold)
If LESS then X events appear in Y minutes (heartbeat)

Specifically, we deliver to clients using FTP, HTTP, or other methods, and we need to be alerted if we haven't delivered in over 10 minutes, or if we've received more then a few errors per client in an hour. It's not abnormal to have a client system go down for a few minutes, or receive just a periodic error (the internet is not perfect, after all), but if there say 500 errors delivering to client Z in an hour, then someone should be alerted. It's also nice to be able to schedule timeframes for when these alerts run, but that's just icing on the cake

Graphing

Another feature we would like to have is the ability to graph events. Take for example graphing how many errors you've had over the past day. Even better, how many deliveries have you had over the past month? This could show us what days we have the heaviest amount of load on our services, and we could then use that information to determine when we need to have more servers available (after all, we're in the cloud so that can be really automated).

Uptime is key

The most important piece of any log management solution though, is consistent uptime and availability. If your logs are lost, or if you can't get to them in the time of an emergency, or if alerts suddenly fail because the service isn't available, then the service is completely useless

What services have we looked at?

We've looked at quite a few different services to solve our logging problems. Obviously we don't want to roll our own service, since that's really not our core business, and we don't want to maintain yet another system that doesn't make us any money. Here's a list of what we've looked at

Loggly

We previously were using Loggly. It's cheap, only about $200/month for up to 1GB/day and 90 days of search history. It gives you graphs, search, and it's got the easy integration. With the addition of AlertBirds, it also has very robust alert management. AlertBirds is actually not integrated directly with Loggly, but it's a free service that they do also provide.

So why did we recently switch away from them? They violated the first rule of Log management, they had a horrible uptime. Whenever we went to their service, they always had a message about "Sorry, we're working on backfilling the servers with your logs". This type of transparency is nice, but it doesn't make up for constant issues. The final straw was A huge outage which lost all logs for several days and a weak apology where they basically blamed AWS for reboots, which Amazon announces up front that are not uncommon. Amazon has always had the policy that they may have to reboot your instance at any time. While they try to let you know about it, sometimes it's just not possible (what if a server suddenly dies?). They weakly tried to say it was their fault while simultaneously blaming AWS. It's one of the major points that anyone seriously interested in cloud computing should handle from the beginning. It's one of the few rules of doing business in the cloud

Loggr.net

While digging around, I also took a look at Loggr.net. While at first glance it does appear very nice, they offer tons of analytics tools, it again violates one of my primary rules. It requires you to use their API to push logs, it doesn't integrate seamlessly with Syslog or any other common standard. Additionally, they charge per log event, and the highest plan they appear to have is for 20,000 log events per day. We blow through that in a few minutes, so obviously they're not designed to scale to what we need.

Papertrailapp

Papertrailapp is the current log management system of choice. They have one feature that I really wanted from the beginning with Loggly, being able to see a live tail of your logs. They do lack alerting and graphing, but they also offer up a very nice API which means you could build it yourself, or simply wait for them to build it as they seem to be very concerned about their customers. I've had several email interactions with the folks there, which is quite frankly the entire reason we're still staying with them. What's even nicer is that unlike Loggly, I don't have to keep logging in every 5 minutes to view my logs. Automatic login that actually works is very nice.

Although you wouldn't think much of it, being able to look at a live tail of a saved search is very important to debugging issues, or simply watching how well an upgrade went. Although Papertrailapp doesn't support more then 4 weeks of search, having this live tail of events is very nice to be able to see to get a "live look" at how well things are going in the system. They do also offer a nice archival to your S3 bucket which means you could do your own work with your log events after the 30 days of search results in Papertrailapp are gone

What other solutions are people looking at?

Have a better solution or use something better? Please let me know in the comments!

ROI: Why aren't we building this ourselves?

2011-12-30T07:04:00.001-08:00

I've been going through a lot of services recently to simply purchase solutions to some oddball needs we have at Newstex. We've gone out and used Pingdom for monitoring, Papertrailapp for log management, Localytics for mobile analytics, and Google Analytics for website analytics. I often stop and think "I could probably write this better", (and so do many of my employees), but the question is then why don't I?
The answer can be summed up in 3 words, Return on Investment. What exactly is the ROI for me building it myself, and how much would it actually cost to build and maintain the service, vs purchasing an external service that does the same thing?

Your time may be better spent elsewhere

Most likely your company, like mine, isn't based around Log management. If the task you're thinking about doing yourself is solved by other solutions out there already, and it's not part of your core business, then you always have to remember that your time could be better spent doing things that improve your product.

Lets take a look at Log Management. Yes, I could probably build a service that would be better then Papertrailapp, providing me the features that I want and need without having other features that maybe I don't (although Papertrailapp doesn't have a lot of extra features that I wouldn't need). Maybe I could add in some nifty graphs, and other sorts of alert management, but how much time would that take me? At a minimum, it would take me several months to build a system to do all that paper trailapp does that I need, and it would take me several more months to add in everything I really want that they don't have. But what would that give me? Would our core product be any better? What value could I have added to our core products that I couldn't because I was working on this project instead?

Quite simply, you always have to ask yourself before you start on a project, would my time be better spent elsewhere? If the answer is yes, you probably shouldn't be building it yourself.

It's probably not cheaper to build it yourself

Lets face it, no project is ever completed even after you're "finished" with it and it's being used. There's always regular maintenance, unexpected events, and updates that need to be done. That log management service may be costing you $200/month, but what's that in hours of your time? Lets say you get paid $20/hour, that's 10 hours of your time per month. That's half an hour a day on average of your time. Would it cost you more then that to maintain and update the system? What about adding new features? Even if you ignore the fact that you had to initially build the system, you're still getting quite a deal. How can they do this cheaper then you can? Quite simply, they have more use of the same product. You don't. If you think it would take you less then half an hour a day on average just to maintain the system even after it's built, you've probably never actually worked with a system this big. Do you think they spend less then that per day maintaining the system?

Leave it to the experts

You don't do this for a living, those that offer these services are focused completely on providing a service like this just for clients like you. If they're any good, they're also asking you for input on how you use the system and what new features you might be looking for. What happens if the system goes down? Are you monitoring this daily to make sure you respond quickly to an outage? If it's running for months and then suddenly breaks, then you have to stop working on your core business just to fix a problem with your log management system. Unless you're constantly working with it, it'll probably take you longer to "switch gears" and get back into the mode to fix and update the system. Leave it to the experts that work with their technology on a regular basis

Anytime you think "I could write this better myself", don't forget to ask the question "but is it worth it?"

A quick note on Papertrailapp, they are missing a few features I would love to see, but they offer a very substantial API which can easily be used to add the features you may want, or integrate with your other systems.

Design for Failure

2011-12-21T12:07:00.000-08:00

Failures are inevitable. No system is entirely without them, it's how you deal with them that makes the difference. While many people will tell you that you should build your system so that it does not fail, in practice that's really just not possible. Instead of always trying to prevent any sort of failure, it's often better to be prepared for failure, and have a plan for what to do when your system fails. It's also important to minimize the impact of a failure. You'll also want to determine the risk of failures, and mitigate those risks using appropriate means.

Split your system into Modules

The first thing to do with any system is to make sure you split appart different features into different self-reliant pieces. In this way you can minimize impact of an outage for a particular system. Making an online game? Make sure that if your website goes down (say, due to a DDOS attack), it doesn't take down the system that serves up your game to your clients. Writing an online video conversion tool? Make sure that even if your conversion servers are overloaded (or go down entirely), your users can still submit videos.

Monitor the health of your systems

Another very common practice to minimize the impact of failures is to make sure that you know when your service is down. You may have several different methods of determining the "health" of your system, but in the end you need to make sure that you're monitoring the end-to-end result of your system.

But who monitors the monitors? If you're writing your own checks to monitor your systems, make sure there's a third party also involved. If your monitors are running on the same hardware or platform as your service, what happens if you have a hardware/platform failure? How are you being notified? If you're notification system goes down, do you find out about it?

At Newstex, we use several different methods to monitor the health of our systems.

Pingdom

Pingdom offers simple port-based checks. They have monitor servers located all across the world, and allow you to check things like web servers, or any generic TCP port. We use this to monitor the health of all of our web-accessible services, as well as check some services that simply respond to HTTP just to verify the health of the system. Pingdom provides a nice tie-in with other services like Pagerduty to notify your staff when there's a problem.

Papertrail

Papertrail is a nice solution which offers log management. Although they don't currently have much in the way of alerts, they can notify you via Email or Pagerduty if certain events appear within your log events. They have future plans to also include checking for the lack of events, as well as thresholds, (for example, at least 15 events within the last 5 minutes). Papertrail also has a nice API that you can tie into to make your own custom alert monitoring, just make sure this isn't the ONLY way you get notified of a problem.

Custom Scripts

For those things that are custom issues we like to track, we also use our own custom scripts. We verify these scripts are working through other services (such as pingdom), but even if the system itself is working, that doesn't mean there might not be other minute problems. For example, you may want to track and log if it's taking a particularly long amount of time to render your webpage, or if a user receives a 5xx level error for any reason. There's nothing wrong with using your own custom monitoring scripts, just as long as that's not the only method you use to check your system.

Determining Risk

Risk Mitigation is one of the key aspects of a disaster recovery plan. Risk can be calculated as a function of how likely a system is to go down, as well as the impact that outage would cause you. If a system is very fragile and likely to go down, but that outage wouldn't cause any negative results for a few days, then it's probably not worth investing a lot of time and money into making it more stable. What if you do get hacked? What information will the hacker get for such and such account? If all they get to see is your name, it's probably not very important to spend a lot of time securing that system. If that system also contains phone numbers, social security numbers, or any sort of financial data, then it's more important to make it less likely that someone can hack in. Make sure you put the proper effort where it's needed. Don't spend 6 months making a secure system for people to see your company logo, or read what the lunch specials for the day are.

Have a Plan

Ok, so your system failed and you've been notified. Do you know what to do about it? Who do you contact? What actually failed? What's the procedure for fixing it and who do you have to notify about it?

Monitoring your system is only half of a recovery plan. You also need to have detailed documentation on how to fix problems, and make sure everyone knows or has a way to contact the person who does know. It's also extremely important that if your system is down, you have access to this documentation. If your system is a wiki, don't keep the only copy of your disaster recovery documentation in your wiki. Plans can vary from system to system, but when an on-call support person is paged in the middle of the night what they really want to know is these few questions:

1. Is there really a problem?

2. Does this need to be fixed now?

3. How can I fix this?

4. If I can't fix it, who do I contact to fix it? (escalation)

The third question How can I fix this, often will include a series of steps of how to identify the problem, and what to do to fix it. It's important to have this decently documented so that you can avoid having to wake up everyone in the office for something that can be solved by a simple reboot of the servers. You also need to have monitoring in place to make sure that once it's resolved, the on-call can verify it has been fixed.

The most important thing to remember is that when there is an outage, it doesn't matter who's fault it is, what matters most is minimizing the impact and downtime of the outage.

Questions you should ask yourself (or your IT staff)

So how do you really know if you're prepared for an outage? Can you (or your IT staff) answer these questions?

1. What would you do if your database suddenly died?

2. What if your entire datacenter went down (or fell into the ocean)?

3. What if your user base suddenly increased by an order of magnitude?

4. What would happen if your head of operations or IT were to suddenly quit, or died?

5. If your system went down, how would you find out? (if your customers notice before you do, there's a problem).
6. Who would respond right now if your entire system was down?
7. What if an employee went rogue, how much damage could they do and how quickly could you recover?

Remember, the most important thing isn't to prevent failures, but to handle them, and be ready for when they happen.

Bridging the communication barrier: describing what you do to your boss.

2011-12-17T09:20:00.000-08:00

I don't blog a lot, but when I do blog it's always about something that I have real-world knowledge of, and I'm very passionate about. This is one thing that I'm very passionate about. It's not very useful to know something if you can't describe it to others. If you really want to get ahead in life, you're going to have to explain things to people that have absolutely no clue what you're talking about, and don't have any knowledge of the terminology you use with your colleagues. Bridging this communication barrier can be quite complex, and is something that few people really understand. It's the reasons most companies have several layers of management, simply because an executive top-level person wouldn't ever be able to understand anything a bottom-level engineer is describing. If you're one of those middle management folks, it's probably because you're very good at bridging the communication barrier.

I recently read a post by Jordan Sissel, a former classmate of mine at college, where he details how to Speak the same language with others. While I do and always will have a lot of respect for Jordan, there's one thing that I strongly disagree with on this point. Effective communication isn't always about speaking the same language, it's usually about how to speak to someone who doesn't understand your language. Describing to a business person all the details they need to know to speak your language is usually impossible to do within any reasonable period of time (but don't worry, them teaching you to speak their language would be equally as challenging). So how really do you communicate effectively to people who don't speak your same language?

Find a common language

When two or more people meet and try to communicate to each other, the first step is always to find the common language that both people can understand. This is true not only of difference in language of two people that both speak english, but also of people that speak literally different languages. If you came across someone that spoke German, for example, and you really needed to use the bathroom, how would you go about describing that to them? Obviously you trying to teach them English wouldn't work, and them teaching you German before you pee yourself probably wouldn't work either. What you want to do is find some way to explain to them that you really need to go to the bathroom. You start off saying words like "I need to pee!" and they don't understand, ok, so that's not the common language you both speak. Then you start making hand gestures, maybe even that doesn't work. Finally you start dancing up and down holding your crotch and crossing your legs. They finally catch on and direct you to the bathroom. They don't tell you in German how to get there, they actually show you. Why? Because you've established a common language that you can both speak.

The same is true with any communication experience. You can speak in multiple different forms of communication, so what you do in any meeting is try to find a common language that you can speak and the intended audience can understand. I'm doing it right now; I'm typing out my thoughts into a written form which, hopefully, you as the reader will understand and be able to use in the future.

So how, really, do you go about finding a common language? You have to study your target audience. The first part of being able to communicate to your audience is to listen. If you're trying to explain something to your boss, find out what he likes. If he's a big baseball fan, and you know about the rules of baseball, you can speak the same language. If he likes to go fishing, learn more about fishing, and you can speak that language. If you're trying to say something to anyone else, the first thing you have to do is listen to what they say and then you can find a common language. Speaking louder, slower, or faster, doesn't help.

Use analogies and examples

The very title of this post is an example. No this isn't just how you talk to your boss, but it's something many people have a hard time doing. So what did I do here? I found a common language we can all understand (talking with our boss), and I've given that out as an example

Analogies, no matter how silly they may seem, are generally very effective if done well. Take, for example, the long criticized analogy The internet is a series of tubes. Yes, for you the reader this may seem like a horrible analogy, and a great reason why not to use analogies. Ok, so that's how you feel about this analogy, but you weren't the target audience. Take this analogy to your grandparents, or your redneck cousins who don't understand anything about the internet. After they read it, I bet they have a much better understanding of what the internet really is.

You have to target your analogies at your audience, once you've found the common ground, speak it. Study it, and make sure you have it right.

The best example of an analogy in recent times that I've heard was actually (and I'm hesitant to say this) while watching the coverage of SOPA. I apologize in advance for not having the exact quote, or even the name of the senator who made it (and if you do please let me know so I can quote/add it here). For those of you who don't know, SOPA is the government act that is their attempt to prevent online piracy of copywritten works that are being distributed through the internet, by using DNS Blocking and filtering.

The basic gist of it is this:

When we identify a crack house, we send in a raid and shut it down, taking those responsible into custody and making sure that they are punished for selling crack. What this bill is essentially suggesting is that instead of arresting those responsible, we change the street signs and take their address out of the GPS and map systems. The house will still be there, and anyone who's smart enough to figure out how to get there will still have their crack. The problem isn't actually solved. Worse yet, those normal law-abiding citizens now have to deal with the fact that their street signs were all changed, and the GPS systems were all completely messed with. This is essentially what we're doing with DNS in SOPA.

It really is a great analogy if you think about it. By using DNS filters and blocking, we're not removing the site, it still has a public IP address, and it now has impact on the law-abiding normal citizen more so then those who would be seeking out to reach that site. They went on to talk about how even a ten year old could follow instructions on how to change their DNS servers to use something housed outside of the US that has the right information. Back to the analogy, anyone could still get a copy of a map that has that crack house's address on it, and go off of that instead of anything official.

Listen to and make sure your audience understood

The final piece of any communication session is to listen again to your audience and make sure they understood what you said. If you gave an analogy or example that they didn't understand, you may need to explain things a little more clearly, give more details, or try a different analogy all together. While switching analogies can be quite confusing, if you start out talking about baseball and they don't understand anything about baseball, it's time to switch and try something else. It's important at this stage to wrap up your point and make sure they come away with a better understanding of what you were trying to explain. Don't think of this as a quiz time, but it's good to allow them to follow up with questions. Be open minded and answer their questions with the same common language you've been using. Don't assume that if they don't understand it's their fault, in fact if they don't understand it's because you didn't communicate effectively.

There are always going to be some times where your target and you simply can't find any common language to speak, and even after you think they understood, they just look at you completely puzzled. If this happens, it's time to find someone who can bridge the gap. If you have trouble explaining something to your grandparents, try first explaining it to your parents, then perhaps they can explain it to your grandparents. You don't always have the ability to speak to everyone, in fact those that do usually end up in public jobs.

Interviewing for a Job

2011-10-28T06:06:00.000-07:00

I've done a fair amount of interviewing from the hiring side, and what I've noticed is that most people simply don't understand the concept of how to really respond to questions in an interview. Now, to be completely upfront, all of my interviews are over the phone, and thus rely almost entirely on the verbal communication that goes on within the interview. I also hate "formal" interviews, and so none of my interviews are very formal.

Understand why you're being interviewed

You're being interviewed because the company you applied to wants to know if you're the "right fit" for the job. Most likely you put something on your resume that triggered a keyword or just generally said something in an email, cover-letter, or resume that triggered word recognition (not always automatic, this is also how most hiring managers work). This doesn't mean that we've actually read your resume, it probably just means we've skimmed it.

Do your research before the interview

Make sure you research the company and workspace that the company you're working for is in. Googling for the company's name should turn something up, and you should be able to find a bit of information about what exactly the company does before the interview. This is important for both you and the company, to make sure that you're not wasting each other's time. It's ok to ask questions like "so what exactly do you do?", but it's generally a good idea to have researched the general field that the company is in. If you're applying to a company like mine, you don't want to go in thinking that you'll end up in an office writing games for the 360.
If I'm really interested in hiring someone, before the interview, I'll check to see if they have a twitter account, facebook, G+, or just generally type their name into google and see what comes up. It's a good idea to do this about your interviewer too. For example, if I was interviewing you, you could have come up with this guide and then know exactly how to nail the interview with me!

Communication is Key

If you want to move on from the first interview (which, for me, usually means we bring you on part-time on a trial basis), you need to be able to communicate well in the interview. It's really a test of how well you interact with the interviewer more then it is about the knowledge you have. Try not to ramble on too much, and whatever you do don't ever make up an answer. It's perfectly acceptable to say "I'm sorry, but I really don't know the answer to that". If you lie about something you know or are comfortable doing, it will only turn around to bite you later on in the job. Expectations are set on the day you interview, so don't make them something you won't be able to live up to if you do get the job.

Be excited, but not cocky

It's important to show enthusiasm for the work you're about to perform. Nobody wants someone that's just willing to be a drone, they want someone who's actually going to work because it's something they like to do. If you answer with just one-word answers, you're probably not going to get the job. I personally look for people that can ramble on about what they've done and how much they like doing it, but that doesn't mean you should ramble on too long. It's a good thing to have a job you absolutely love doing.
However, this doesn't mean you should be cocky, or simply tout how good you are because of what you've done. Yes, be proud, but don't think you're better then the interviewer. You don't want to be stuck-up, or sound like you're belittling the interviewer because he/she doesn't know something that you do. Never say "Well, it was on my resume", don't assume that your time is worth more, or even as much as, the interviewers time. It's generally a good idea to end the interview by saying "thanks for taking the time to interview me".

Don't be discouraged if you don't get the job

Just because the interviewer doesn't give you the job doesn't mean you're not capable of doing the job, it just means you're not the right fit for the position, or perhaps just the company. When I interview people, I'm not just looking for someone who has the knowledge or ability to do the work, I'm looking for someone who will be able to fit within the work environment we have. We're a completely virtual company, and it takes a special breed of person to be able to handle that. Don't take it personally

Don't blame the economy for being unemployed

2011-10-05T06:40:00.000-07:00

Every time we turn on the news people are complaining about the "horrible unemployment rates" and how "jobs are incredibly hard to find". Yet, for some reason, whenever anyone I know goes looking for employee's, they can't find any that aren't happily employed elsewhere.

The problem isn't that there are less jobs, the problem is that the US is becoming smarter and doing things more efficiently, thus we require the right type of people, those that know certain things, not just jobs that any electronic monkey can do.

Eric Hammond's Post on how companies have to work harder to get quality employee's just shows that it's even the big boys out in LA that are having problems. Don't blame the economy if you can't find a job.

Even those that don't have the skill set for the kind of work that Eric and I are looking for still have options. Companies are always hiring, even if your current company goes away, there's always another place that will hire people doing something.

Here's my three simple rules for getting, and keeping, a job.

Rule #1 Don't be picky

I'm not saying that you will always be able to find a job immediately, in fact it often takes some work, but there are always companies looking for work. Just because you were doing plumbing at your previous job, doesn't mean that if you get laid off you can't go work at WalMart.

If you're in need of a job to get by, don't complain that there's no jobs for you. There are a multitude of jobs out there, maybe not in your area, but that doesn't mean you can't move.

The worst thing to do in an interview is turn down a job based on the amount of money it's going to provide for you. Even if you're moving down to $10/hour from $20/hour, at least it's a job. Take it and move up in the company, don't set your initial expectations so incredibly high that you don't find anything. Remember, even $10/hour is better then $0/hour. Unless you have several job's lined up already, you can't afford to turn down any offer, no matter how bad it may seem.

Don't be afraid to go back to an old employer. Just because you left them doesn't mean that they won't take you back. Again, don't expect to just jump right in where you left off, but any job is better then no job.

Rule #2 Make a real Resume

I don't know when it became no longer standard practice to include a decent 1 page resume with specific details about the job you're applying for, but as someone who's hired several employees and waded through hundreds of resumes, I'll tell you that if your resume isn't focused and short, you're not going to get a job. The Resume is a 1 page representation of who you are and what you can bring to the company. If you're applying for a job doing software programming and you have 3 pages of job experience, mostly detailing your work in retail, then you're going right in the trash.

The most important part of the Resume is to make it focused on the kind of job you're applying for. The best resume's I've seen were from people that often kept several copies of their resume, each tailored for specific job positions. For example, you may have one resume for software programming, and another for project management, and maybe even another for sales (remember rule #1).

Rule #3 Be Honest

The worst thing that you can do in any job interview, or any part of the job hiring process, is to lie. If a prospective employer asks you about how you would go about solving a particular issue, don't bumble around and make something up, be honest and tell them you don't know. Eventually, these things will catch up to you. If you put on your resume that you speak Spanish but really all you know is how to say "I need to use the bathroom", you're going to get fired when they put you in a meeting with all spanish speaking people and you can't understand anything they're saying.

Don't make things up, if you honestly have no idea or have trouble understanding what the interviewer is asking for, just ask for clarification and explain the situation. Don't be afraid to say things like "I'd have to do more research on that topic". The only time in your life that you need to memorize answers to complex questions is when you're in school, in the real world we have Google.

Be upfront about any issues you may have. I was interviewing for a company about a year ago that didn't want to hire someone who was looking for anything more then a 6 month "job". Since I already had a decent job that I was working at, I could afford to break rule #1, so I told them right upfront that I was looking for a career, not just a job. This may have meant that I lost the higher paying job, but by being upfront about it, I avoided the long-term issue of being out of a job in 6 months.

iTunes Match vs Spotify

2011-09-02T07:34:00.000-07:00

As you may remember from this year's WWDC conference, Apple is releasing iTunes Match, part of iCloud which is designed to look up the music you have in your iTunes Library, and then allow you to listen to that music from any of your apple devices, including iPhone, iPod touch, and iPads.

While at first this seems like a really cool idea, what you don't realize is that there's already a much better service out there called Spotify, which costs more (roughly $60/year vs $25/year), but allows you to listen to any song from any of your devices. This includes iOS devices as well as PCs and MACs.

Spotify does also offer a free version of their service which simply adds in audio advertisements (similar to Pandora), but the free version doesn't allow you to stream to your mobile devices, or download songs for playing offline later. Still, if you're looking for that one-off song that you just can't get out of your head, or looking for similar songs that you don't already own, Spotify fits the ticket.

So, if Apple is to compete, they'd need to lift this restriction and allow you to listen to any song even if you don't own a copy of it, from any device, and allow you to listen to them offline as well. If they added that (and kept the lower price), surely it would easily beat off Spotify. The iTunes interface is way more refined then Spotify's, but you can't beat being able to listen to any song ever produced (and no, I have not found a single song that isn't on their system).

Sorry Apple, looks like you fell just short of the mark this time.

Another Hiccup in the AWS Data Centers

2011-08-09T05:32:00.000-07:00

Most of you probably remember reading about The Great EC2 Outage of 2011, so when it happened again, many people begain to question if Amazon is the best way to host our services.

Let me first state that Newstex (the company where I manage the infrastructure hosted entirely within AWS), was almost completely unaffected by this outage. Yes, we had all of our servers in US-East become disconnected from the rest of the internet, but no, this did not negatively impact our core business. Why? because we use Multi-Region Deployment.

Multi-Region Deployment is mostly useful when you're serving up services to people in multiple locations. It helps to create a more direct line to your customers by centering your servers closer to them. For Newstex's Core business, we place servers in both US-East and US-West. This means that while servers in US-East were down completely, the servers in US-West continued to function at full capacity, allowing us to pull in and deliver our content without any interruptions.

Now to say that we were completely unaffected would be a lie. We did temporarily lose our management console during the brief 40 minute outage, but this was a risk that we agreed wasn't worth spending an extra $200/month to cover. Statistically speaking, since AWS has only been down a total of about 2 hours this year, that would mean that it would have cost us roughly $1,600 for an additional 2 hours of uptime. That's just not economical for something that's only used to modify our configurations. What's important is that our core business was secured and protected against a region-wide outage such as this.

For more information on how to build your own Multi-Region Deployment, see my book:

This is not a cloud

2011-05-05T06:15:00.000-07:00

After seeing my eleventy-billionth "cloud" product, I've decided to make a post about what is not a cloud product.

This is not a cloud:

And, although very cool, this is still not a cloud:

This is a cloud:

... although probably not what you're referring to when you think of clouds these days.

Still, lets take a look at the cloud, the real cloud, and try to describe it.

It's big

Clouds are big, and generally not shaped. When you're within a cloud (fog when it's at ground level), you can't really see the boundaries.

It just floats there

It's not tied down to anything, it's just up and floating around in the sky.

It's elastic

It's not constrained to a certain size, and often they grow and shrink seamlessly while you're not even looking.

So what is a Cloud in terms of computing?

But really, what does all this crazy talk have to do with cloud computing?

Quite simply, to be a cloud service, it has to be big, floating around, and elastic. To put it directly,

The internet is not just a big truck you can dump onto

... but a cloud is.

Post Mortem of EBS Outage

2011-04-29T06:30:00.000-07:00

Now that Amazon has fully recovered from the EBS outage last week, they've reased an official explanation of what happened, and why.

Within this explanation is burred a reference that this really did not drastically impact more then one Availability Zone. Due to this they're going to be offering a series of webinars to help developers better understand how to properly architect their applications for cloud computing

But what really happened?

Most users of Amazon's Cloud Computing really want to know what happened, or more specifically we want assurances that it wont happen again. In reality, what happened can be broke down into the following failures which compounded into a complete and total collapse of EBS's services across the entire us-east-1 region:

A bad network upgrade caused several EBS volumes to think they were alone, and panic
These panic'd EBS nodes caused lots of trouble in the regional EBS control system
Due to this failure, Amazon had to disable the entire EBS API
Users trying to launch new EBS volumes to replace the ones that had failed could not, because the EBS API had been disabled for the duration of the fix

How do I know it wont happen again?

Amazon is attacking the problem on several fronts, but the biggest bonus that I saw in their entire post is a true admission that mistakes will happen. Although they're going to try to not make mistakes and are putting in lots of automation and controls to hopefully prevent the original network problem that caused the outage, they're also attacking each step along the way which caused the outage.

Controls will be put in place to prevent bad network upgrades in the future
EBS clusters will be modified to back-off gracefully instead of wildly trying to recover from a failure causing the entire system to break
The EBS Control system will be modified to be more zone-based, so that failures in one zone wont cross over to others
Amazon is making more services (such as VPC) available in all availability zones
Amazon is offering up a series of webinars to help developers architect their systems more effectively

But why should I still trust Amazon?

The reality of this event is that yes, Amazon did make a mistake, but all those effected by this mistake, even some that didn't get effected, will be getting a credit equal to 10 days worth of EC2 and EBS usage on their account. Although the monetary compensation isn't the big part of this, and is very little comfort, what is comforting is that Amazon is really announcing that they screwed up, and they quickly updated everyone to acknowledge there was a problem.

When my Cable goes out (which happens almost every month), I get no notification other then from my own systems. Whenever I call my cable company, all I get told is if they know there's a problem, never any compensation in return, and never even an ETA or apology. The same is true with most hosting providers. What's even more comforting is that really only those who limited themselves to a single availability zone were effected. Although you couldn't quickly launch in another AZ to solve your problems, you could launch within another Region, and more importantly if you had already had instances in other zones, you were most likely still operational.

What can we learn from this?

The biggest thing to learn from this is that nobody, not even Amazon, is infallible. That being said, in Amazon's case they always give you the option to recover from a failure quicker then any other solution. For example, if I had a single server that was broken in a traditional hosting environment, I couldn't quickly launch a clone in another datacenter to serve out of while that server was repaired. With Amazon, it's just a matter of a few API calls.

Cloud Computing is not dead, in fact this is a shining moment for it. It shows that you'll get exactly the stability that you plan for.

Troubles in US-East, migrating to US-West

2011-04-28T09:01:00.000-07:00

I've been having increasing amounts of problems with my servers in the US-East-1 of Amazon Web Services. Fearful that this may be some fallout or aftershock of the Great EC2 Outage of 2011, I quickly sprung into action, migrating my servers over to US-West-1 (located in California).

It's important to note, however, that this had almost no impact on any of my production systems, as I already base my core services in two regions. These services continued to operate taking the full load of the system while us-east was having problems.

However, I took this as an opportunity to review my disaster recovery plan, and build up a new image that automatically updates from bitbucket repositories using an SSH key on reboot.

I launched my old instance, updated it quickly, and then used the following commands to bundle it from within the instance (as root):

# ec2-bundle-vol -c my_cert_file.pem -k my_key_file.pem -u MY_EC2_OWNER_ID -p my-ami-prefix -s 102400 -d /mnt

You can grab your EC2_OWNER_ID by using ElasticFox, double click on any instance you own, and copy the value for "Owner ID" which should be a number.

The next step is to upload this bundle to EC2:

ec2-upload-bundle -m /mnt/my-ami-prefix.manifest.xml -b my-ami-bucket -a MYACCESSKEYID -s MYSECRETACCESSKEY

Note that your bucket must be in the same region that you're bundling from. This means that if you want to actually migrate that manifest to another region, you have to use the ec2-migrate-manifest tool:

ec2-migrate-manifest -m /mnt/my-ami-prefix.manifest.xml \
   -c my_cert_file.pem \
   -k my_key_file.pem \
   -a MYACCESSKEYID \
   -s MYSECRETACCESSKEY \
   --region=us-west-1

And then you can re-upload it to a us-west-1 bucket using the ec2-upload-bundle command above. When you do this, you will be asked to make sure you want to upload to the other region, since it's not in the current region. Just say "y" for this since you've already migrated it using the ec2-migrate-manifest tool. This will take significantly longer since you're submitting cross-region

Lastly, you'll need to register this image with EC2 (or one copy in each region if you uploaded it to each region). I like to do this using boto:

>>> import boto.ec2
>>> conn = boto.ec2.connect_to_region("us-west-1")
>>> conn.register_image(name="my-ami-name", image_location="my-ami-bucket/my-ami-name.manifest.xml")
u'ami-XXXXXX'

Then you're good to go, you'll get the AMI ID back that you can use to launch new instances in us-west.

The Great EC2 Outage of 2011

2011-04-22T06:31:00.000-07:00

There's been a lof of posts recently about the Massive Outage on EC2 that's now spanning two days of Amazon acknowledging that they have an all-out failure. Guess what folks, it's not all of EC2 that's down, it's a small part of EC2 called Elastic Block Storage (EBS). Unfortunately, this system was developed because most people can't live by the disposable instances theory, and wanted something more like traditional Virtual Machines (VMs).

I've always been a huge fan of using Cloud Computing, specifically Amazon Web Services mostly because it's pretty much the only game in town. But, as I've always said, you have to prepare for something to go wrong. In most cases, this outage is more related to Network Providers then AWS itself, but in this case it was actually an outage caused by some pretty massive failures at the Amazon side.

Amazon splits up their systems into multiple Regions, each Region is then split up into multiple Zones. The major outage effected only one region, and furthermore it's now limited down to just one Zone within that region that's still a problem. Amazon has always had the policy that any single zone within a region may be down without effecting their SLA (they always tell you to use multi-zone deployments), but in this case they've actively marked themselves as down. What this means is that this outage will be counted against their SLA. Unlike traditional providers who try to cover things like this up, Amazon has been very transparent and open about all the problems they've been having.

What I find even more Amazing is that everyone yells that all of Amazon Web Services is down when they have this kind of outage, yet nobody even noticed that Amazon never went down in Japan even with all the recent Tsunami and Earthquakes, Nuclear Disasters, and even rolling blackouts that have been occurring. They survive acts of God which anyone would forgive for them being down, but there's no forgiveness for one piece of AWS being down for any length of time.

It's not hard to launch systems in multi-region deployments for redundancy situations. In fact, Amazon has even offered up some suggestions for how to do this, and there's even more help available on the Forums. I find it Amazing that many people seem to simply call it a day when something like this happens, rather then try to figure out how to improve your DR situation. And if you really "dont have the resources to make a DR solution", then you probably don't have enough resources to matter in an outage like this.

Using Amazon $NAME with Boto

2011-04-01T07:01:00.000-07:00

Today, Amazon announced it's new $NAME service available in all five AWS regions. This new service automatically integrates with your existing systems with no configuration required. As requested, we've integrated it into boto. You don't even have to upgrade your installation, it's there for all versions by default.

Usage of $NAME is simple, you first have to sign up for the service, and then you'll notice a new connect_ method is available to you, connect_me().

>>> import boto
>>> me = boto.connect_me()
>>> me.hardware()
[u'sofa', u'closet', u'barn', u'dishwasher', u'pillow']
>>> me.services()
[u'ec2', u'sqs', u'input', u'output']

You can interact with hardware by name:

>>> lr_sofa = me.get_sofa("livingroom")
>>> lr_sofa.is_clean
False
>>> lr_sofa.vacuum()
u'Pending'

Most actions, like vacuum() take some time, so we can queue up a bunch of commands at once:

>>> lr_sofa.shampoo()
u'Queued'
>>> lr_sofa.dry()
u'Queued'
>>> lr_sofa.add(me.get_pillow())
u'Queued'

You'll also note here that we can call get_pillow with no arguments to just grab a random pillow available, instead of a specific one.

Next, we can see all the actions that are pending, as well as an estimated time to completion (ttc):

>>> lr_sofa.get_tasks()
[(u'clean', u'Running'), (u'vacuum', u'Queued'), (u'shampoo', u'Queued'), (u'dry', u'Queued'), (u'add(pl-012573)', 'Queued')]
>>> lr_sofa.get_ttc()
1789

Note that the ttc is returned as a number of seconds from now, in this case it's just under 30 minutes before we can expect all these tasks to be completed.

That's it! Have fun using the new $NAME service from Amazon!

Tagging EC2 Instances using Boto

2011-03-30T08:07:00.000-07:00

For a while, I have been creating command line tools provided right with boto which I used to manage AWS. Recently, others have become interested in these tools as well, and I've seen several other contributors adding to these tools to make them even more useful to others. One recent submission by Ales Zoulek added some nice features to my list_instances command, which I use on a regular basis to list out the instances that are currently active for my account in EC2.

Amazon now lets you add Tags to EC2 objects such as Instances and Snapshots. This allows you to actually "Name" your EC2 instance, as well as add some metadata that could be used for AMI initialization, etc. Ales added the ability to list these tags by name within the list_instances command line application:

% list_instances -H ID,Zone,Hostname,T:name
ID             Zone           Hostname                                   T:name                        
-------------------------------------------------------------------------------                        
i-xxa          us-east-1a     xxxx.compute-1.amazonaws.com               www  
i-xxb          us-east-1b     xxxx.compute-1.amazonaws.com               ftp
i-xxc          us-east-1c     xxxx.compute-1.amazonaws.com               webdav 
i-xxd          us-east-1d     xxxx.compute-1.amazonaws.com

This is great for when you're launching instances, since you can also specify a tag at launch time:

% launch_instance -z us-east-1a -t m1.small -a my-ami -T name:mobile boto.cfg

But what if you've already got a few instances running and you want to label (or re-label) them?

Simple, use the create_tags functionality of boto.ec2:

>>> import boto
>>> ec2 = boto.connect_ec2()
>>> ec2.create_tags(["i-xxd"], {"name": "mobile"})
True

You can actually specify any number of tags to be applied to any number of instances (or snapshots) with this command. The first argument is the list of objects (in this case we just had one), and the second is a dictionary of key-value pairs to tag the objects with.

That's it! when you go back in and check your instances, they'll be tagged properly with the new tags you just set:

% list_instances -H ID,Zone,Hostname,T:name
ID             Zone           Hostname                                   T:name                        
-------------------------------------------------------------------------------                        
i-xxa          us-east-1a     xxxx.compute-1.amazonaws.com               www  
i-xxb          us-east-1b     xxxx.compute-1.amazonaws.com               ftp
i-xxc          us-east-1c     xxxx.compute-1.amazonaws.com               webdav 
i-xxd          us-east-1d     xxxx.compute-1.amazonaws.com               mobile

It's a dangerous time to read Gizmodo

2011-02-21T14:16:00.000-08:00

Gizmodo Recently decided to publicly attack Apple about it's recent change in policy, designed to protect mobile users as well as earn a little more cash for Apple.

Gizmodo's been pretty famous recently for changing their entire website to use a new frame-like UI with fixed-width formatting that pretty much makes everyone upset. They've also added massive amounts of advertisements all over the top half of their stories, making all of their readers scroll through a page of ads before they can even read the content.

Now they're attacking Apple publicly, something that's been known not to work, and have very negative repercussions.

And why, after all, do they claim you shouldn't buy Apple products right now? They're coming out with them too quickly. That's right, Apple is releasing too many products now... Wait... how many new Android phones came out this month alone? Even Best Buy knows that their the big culprits, advertising that they will buy back your old technology (note the primary devices are Androids on their ad), for roughly 50% if it's before 6 months.

Seriously Gizmodo, take a clue, Apple isn't going to let you back into the fan club.

Amazon T1 Micro instance With Apache Primer

2011-02-09T12:32:00.000-08:00

Although there's been a lot of buzz about the new Amazon t1.micro instances and the "Free Tier" for amazon, I haven't seen a good tutorial on how to just get started using the Amazon Dashboard console, so I decided to make this quick-and-dirty "get your site up and running in 15 minutes" bit. For those of you who don't know, Amazon launched a micro instance type which is perfect for most "simple" websites that don't generate a ton of traffic. If you don't need MySQL, but do need some server-side processing like PHP, this may be the solution for you.

Signing Up for the Services

First things first, head on over to http://aws.amazon.com and click the big "Sign Up Now" button.

You'll want to sign up for at least EC2 and EBS, but most other services aren't used in this tutorial.

Sign into the AWS Management Console

Everything really happens under the AWS management console, so after you've gone through the riffraft of verifying your account, you'll want to click the "sign in to the AWS Management Console" link at the top of the http://aws.amazon.com page.

Once here, you'll want to click on the "EC2" tab:

Launching your instance

Click on the "Launch Instance" button.

This wizard will guide you through launching a new instance. The key things to keep in mind here are that you want to make sure you're launching a "t1.micro" instance and using Amazon's 64 bit image.

On the next page, make sure you choose the "t1.micro" instance:

The next page you can ignore, just click continue:

The next page just lets you set some labels for your instance, in general you probably just want to at least set the "Name" tag to something in the event you launch multiple instances, you can use this to identify which is which:

For example, I chose to name mine "ApacheServer"

Setting up your Keypair

Click continue to the next page, where you'll be prompted to create a keypair. If you've done this before, you can use an existing key pair, but if you're reading this tutorial I'll assume you haven't set one up yet.

KeyPairs are how you log into your server, using SSH. It's important to note that you'll need to download this key pair and keep it in a safe place. For security reasons, you can never download your keypair again after this setup procedure.

Enter a name for your keypair and click the "Create and Download your Key Pair" link:

Setting up your Security Group

Click continue and you'll be taken to a page allowing you to set up your security group. It's generally a good idea to set up a new security group for each new "group" of servers you want to create. For example, this is for apache, so lets name it "apache" and fill out a description:

Now we see by default that SSH is authorized, but we also want users to be able to access this server via HTTP, and probably HTTPS as well. We add those in by choosing the appropriate fields from the row at the bottom:

Make sure to click the "Add Rule" button after adding each rule:

Before moving on, make sure you see something like this:

Review and launch

Now we look over our information, and make sure everything's ok before launching the instance:

After launching, you should see a page like this:

Setting up your Elastic IP

Of course, Amazon is very volitile, so we'll want to make sure we set up an elastic IP for our instance before we change over our DNS name.

On the left side of your console, you should see a link to "Elastic IPs":

Once here, you can allocate a new Address:

This requires you to confirm that you wish to allocate a new EIP.

You're currently limited to 5 EIPs per account, and you pay for any EIPs which are not associated to instances. If your EIP is associated to an instance, you don't pay for it. From here, you'll see your new EIP:

Right-click on it and choose "Associate Address":

Choose your instance:

Assign your DNS to your EIP

This all depends on who you bought your domain from. You can simply associate your domain with this IP address and your users will be forwarded along to this server automatically.

Logging in

Of course, now you'll need to log in and set up your software, specifically Apache. From a terminal or shell, use the keypair you downloaded to SSH into your instance by your elastic IP address:

$ ssh -i myKeyPair.pem ec2-user@174.xxx.xxx.xxx

You'll be greeted with a friendly banner:


       __|  __|_  )  Amazon Linux AMI
       _|  (     /     Beta
      ___|\___|___|

See /usr/share/doc/amzn-ami/image-release-notes for latest release notes. :-)

If instead you see something like this:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Then you need to first change your keypair permissions:

$ chmod 0600 myKeyPair.pem

Installing Apache

The instance you've just logged into is based on Redhat, so you've got the Yum package manager. Install apache using the following command:

$ yum install httpd

Confirm all the prompts with a "y", and continue on to setting up your apache configuration

Configuring Apache with WebDav

The easiest thing to do next is to configure your Apache server to allow you to connect via webdav. Do so by first creating an .htpasswd file for your user:

sudo htdigest -c /var/www/.htpasswd webdav myAwesomeUserName

It'll prompt you for a password, make sure this is secure since you'll be using it to gain full access to your website files.

Next you'll want to change the ownership of the /var/www folder to the apache user:

sudo chown apache:apache /var/www

Finally, set up the DAV server in a /etc/httpd/conf.d/000-default.conf file. You can use whatever basic editor you want here, but I use vim:

vim /etc/httpd/conf.d/000-default.conf

Make sure this file looks like the following:

        DocumentRoot /var/www
        <Directory /var/www>
                Options Indexes MultiViews
                AllowOverride All
                Order allow,deny
                allow from all
        </Directory>

        Alias /webdav /var/www
        <Location /webdav>
           DAV On
           AuthType Digest
           AuthName "webdav"
           AuthUserFile  /var/www/.htpasswd
           Require valid-user
        </Location>

Starting Apache

You can start apache by simply running:

$ sudo service httpd start

Setting up your WebFiles

You can now connect to your WebServer via WebDav. Simply use the following URL, replacing with your hostname:

http://example.com/webdav/

And connect using your username and password you set up with the htaccess command. Note that this will take you directly into your web root, so feel free to deposit your index.html file directly here.

Boto and Amazon Route53

2011-02-04T17:44:00.000-08:00

For a while now, I've been working on making the Route53 module for boto much more simple and easy to use. While Route53 doesn't yet provide anything special that you can't find in other DNS providers, it's still quite useful to be able to programmatically change DNS records from within your instances. For example, you could set up an instance to boot and automatically associate itself as another server on a given domain for the purposes of Round Robin DNS, or for setting up Memcache servers where you need to have your servers contact all of the possible memcache servers to determine if there's a cached version of your data available.

Creating your first hosted Zone

Before you begin, you'll need to make sure you sign up for Route53 from Amazon Web Services. It's free to sign up, and you only pay for what you use, so if you have no hosted zones you'll pay nothing for being signed up. It only costs $1/month per hosted zone, plus a minimal charge for each hit against your domain.

To create your domain, use the route53 command line client included with boto.

% ./bin/route53 
Usage: route53 [command]
 add_record - Add a new record to a zone
 cmd - Prints this help message
 create - Create a hosted zone, returning the nameservers
 delete_zone - Delete a hosted zone by ID
 get - Get all the records for a single zone
 help - Prints this help message
 ls - List all hosted zones

We want to use the create command to make your first domain.

% ./bin/route53 create example.com 
Pending, please add the following Name Servers:
 ns-xxx.awsdns-22.com
 ns-xxx.awsdns-32.net
 ns-xxxx.awsdns-12.co.uk
 ns-xxxx.awsdns-60.org

After you create the zone, the script will tell you where to point your domain. This must be done from wherever you purchased your domain.

As you can see, Amazon's DNS servers are located around the globe so that no matter where your end-user is, they can have a DNS server close by.

Adding a record to a Zone

Of course, without any records, this zone configuration is relatively useless, so lets set up our records. Before we can add a new record, we need to look up the ID for the zone we just added:

% ./bin/route53 ls         
================================================================================
| ID:   ZXXXXXXXXXXXXXX
| Name: example.com.
| Ref:  xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx
================================================================================
{}

Next we use that to add our A record, using the Command line function:

% ./bin/route53 add_record ZXXXXXXXXXXXXXX example.com. A 127.0.0.1    
{u'ChangeResourceRecordSetsResponse': {u'ChangeInfo': {u'Status': u'PENDING', u'SubmittedAt': u'2011-02-05T01:34:20.912Z', u'Id': u'/change/XXXXXXXXXXXXXXXXX'}}

This change request will take a while to actually occur, so just sit back and wait for the change to happen, or continue on adding records. You can add any type of record, A, AAAA, CNAME, MX, TXT... etc..

Adding records through boto

Of course, the real advantage is probably not through the command line functionality, but through the ability to change your records programmatically. For example, you could create the following script to automatically add the current instance to your zone if it's a standard boto-based image:

import boto
from boto.route53.record import ResourceRecordSets
conn = boto.connect_route53()
changes = ResourceRecordSets(conn, "ZXXXXXXXXXXXXXX")
change = changes.add_change("CREATE", boto.config.get("dns", "name"),"CNAME")
change.add_value(boto.config.get("Instance", "public-hostname"))
changes.commit()

What's next?

Eventually, Amazon plans to add in Geographic support which will direct your end users to different IPs based on where they're located. This means you can have EC2 servers (or load balancers) set up in different Regions, and depending on where your user is sitting, they'll get the server closest to them!

Boto, Loggly, and rsyslog

2011-02-03T09:41:00.000-08:00

Recently, I discovered an interesting new "Cloud Logging" service provided by a small start-up company called Loggly. This service promised to offer a nice web interface to a logging server that would essentially comprise of a set of syslog servers that could read in all of my standard logs that I would normally get on the vast amounts of cloud-based servers I use, and keep them in one central searchable location.

UPDATE:

Loggly has recently had a horrible track record of downtime, although they're by far the cheapest and most feature-filled service that we've found, the record of about 50% uptime is really a killer. After a 3 day outage, we've switched to papertrailapp, and are looking around for any other alternatives.

Do you have an alternative solution to Loggly? Please let me know in the comments!

Installing rsyslog

Obviously this sparked my attention, as I'd love to not have to log into 5 different servers just to find out what was going on in my boto.log file. I began researching and discovered that I'd have to switch to using rsyslog, which proved to be a slightly more difficult task then you'd expect when running on the Amazon Linux AMI based on RedHat, but with a bit of googling I found an easy solution:

yum shell
install rsyslog
remove sysklogd
run

All this eventually results in replacing the built-in syslog server with rsyslog, which not only handles multiple threads of execution at once, it also handles sending logs to a remote syslog server.

Setting up boto logging

The next thing I tried was to use rsyslogs built-in method of simply polling a file and piping that through its logging mechanisms. There were several problems with this, but the most important and deadly issue was that no matter what I did, the syslog server seemed to crash overnight. As it turned out, this was a problem with rsyslog not handling the logrotation of that file, which caused it to lose place in the file and freak out.

As a solution, I decided to look into simply piping the logs directly to syslog, bypassing the intermediary step of using a file at all. This proved to be relatively easy once I found the right configuration directives. If you recall, boto has the ability to assign logging configuration automatically from the boto.cfg, so simply replace your existing logging configurations with this:

[loggers]
keys=root,boto

[handlers]
keys=syslog

[formatters]
keys=syslog

[logger_boto]
level=INFO
handlers=syslog

[logger_root]
level=INFO
handlers=syslog

[handler_syslog]
formatter = syslog
class = handlers.SysLogHandler
args = ('/dev/log',handlers.SysLogHandler.LOG_USER)
level = INFO

[formatter_syslog]
format = [%(name)s] %(levelname)s %(message)s

The most important thing to note is that you should NOT use any date formatting in the formatter, I've noticed this causing bizarre issues that make the logging not work.

Piping all logs to loggly

The last step is provided to you with a configuration line given by loggly itself. When you set up an input, you get a configuration line to add to your rsyslog.conf file, something like:

*.*    @@logs.loggly.com:123456

Using this, you'll send all your logs directly to the loggly servers, including your newly added boto logs!

Test this using:

import boto
boto.log.infO("Test log from BOTO!")

Allow about a minute for the log to appear in loggly, although it may appear much quicker then that.

The real difference between Verizon and AT&T

2011-01-26T09:16:00.000-08:00

There's been a lot of controversy recently about the recent announcement that Verizon will now be selling the iPhone. While hundreds of thousands (if not millions) of users switched from AT&T to Verizon when the iPhone first came out, it's not clear as to if, or why, we should switch to Verizon. The truth is that there are many differences in the technology behind each of these companies that will greatly effect how you decide where to stay.

First of all, I'd like to say that all of my information comes from a time when I worked at a retail store selling Verizon phones, and then eventually Cingular (which was purchased by AT&T) phones later. There's a mass confusion about the technologies and which one is "faster" and which one covers more area, and the truth is that they're almost dead even in both aspects. The bottom line comes down to a few points.

Coverage
AT&T has more 3G cellular towers then any other provider in the US. Verizon has more overall towers then any other provider in the US. That means while you're more likely to have 3G coverage by AT&T, you're more likely to have any cellular reception by Verizon. Obviously this depends on where you are, but in general Verizon targets to have the most ground coverage, and AT&T targets where people live.

Signal Frequencies
As a fundamental law of physics, lower frequency signals tend to penetrate solid objects, while higher frequency signals tend to bounce off of them. Which is better is really a matter of opinion, and terrain, but there are some undeniable facts.

Verizon uses lower frequencies which tend to penetrate solid objects better. AT&T uses higher frequency signals wich tend to bounce off of solid objects better.

A good example of where you can see this is in basements. In my college days, I was riding down an elevator talking on my cell phone. Since this was before the iPhone days, I was still on Verizon. Another passenger on the elevator was talking on his phone, which used Cingular (now AT&T). I knew that both towers were actually on the same exact building, about 5 yards apart, so this next step came as a surprise to both of us.

As soon as we passed into the basement (which was solid concrete plus a metal frame), he lost his call. I continued on my call the entire way through the basement, picked up my laundry, and returned to my dorm room while still on the call. The only difference was the carrier, and more specifically the frequency they use.

Multi-Tasking Compatibility
One of the biggest complaints that iPhone users will have of Verizon is that they can no longer talk on the phone while browsing the web, or using data of any kind. This is because Verizon towers do not let you connect to both their data network an their cellular network at the same time. Although Verizon has indicated that this should be fixed soon, it's almost unthinkable that you'd not be able to be on a call, hit your menu button, look something up, and talk about it with the person you're calling. This area clearly will need to be fixed before most iPhone users should switch.

Multi-Tower Compatibility
The biggest complaint that iPhone users had of AT&T was dropped calls. This is not due to the problem of the phone itself, but due to the technology that is provided by AT&T. Specifically, AT&T only allows any given phone to be connected to one tower at a time. Verizon phones, while only being allowed to connect to one network at a time, can be connected to up to 3 towers at any given time.

This functionality is known as soft handover, which essentially means that when you're driving down the road and switch which towers you're in range of, the Verizon phone is more likely to be able to handle the switchover, since it's already using 2 other towers.

Data Transfer Speeds
The last point I'm going to touch on here about AT&T vs Verizon is speed of the networks. Most of us are familiar with Cable internet, specifically that Cable uses a shared medium which means that the performance you get will be directly proportional to the number of people on your same line.

Cellular towers are no different. The more people on a given tower, the less overall throughput that tower will have. Although there's a difference in technology coming up with the new 4G networks, in general the more people in your area using your same network, the less throughput you'll have.

Why I don't use EBS root instances

2010-09-13T08:30:00.000-07:00

Recently Amazon announced a new Micro Instance for EC2 which costs significantly less then a standard small, and contains the ability to "burst" CPU usage. This new instance type leads developers more and more to using many instances to perform tasks since it's far easier to scale out.

This also marks the first Instance that doesn't contain any ephemeral storage at all, meaning you have to set up each instance on an EBS volume, and you can't simply run one off of an AMI. Additionally, if your instance dies, you'll have to fix or replace that EBS volume, instead of simply tossing it away.

Amazon points out that this method of booting from EBS is faster then copying from S3, and does provide a few advantages. While it's true that the Initial boot process will be faster, it's been tested and proved that the Ephemeral disks are faster then EBS volumes. This means that if you're doing any sort of heavy disk IO, especially temporary writes, EBS will be much slower for you, and you've got no other option for Micro instances.

I love the idea of Micro instances, and I understand that Amazon was saving money (and passing that savings on to the consumer) by not proving a large Ephemeral disk with each instance, but really not even a 10GB root partition? This new step in Amazons recent history seems to be a major step backwards, showing that EBS-backed instances will be their future supported instance of choice, instead of providing support for the disposable instance model.

If anyone at Amazon is listening, I'm not asking for much, just let us have 10GB of free Ephemeral storage for these Micros. This Ephemeral storage gives us a lot of usability for free, which is exceptionally useful for things that are temporary.

Search for Books on Amazon using Boto!

2010-09-08T14:15:00.000-07:00

Today I added some minor support for ECS into boto. Specifically, I've implemented the ItemSearch API from the Product Advertising API. Usage is simple:

>>> from boto.ecs import ECSConnection
>>> conn = ECSConnection()
>>> books = conn.item_search("Books", Author="John Resig")
>>> for book in books:
... print book.Title
... print book.ASIN

The above example should print out the title and ASIN of all books currently on Amazon authored by John Resig, including Secrets of the JavaScript Ninja.

Happy searching!