The topics covered include:

• What is cloud computing?
• How your startup can take advantage of cloud computing to gain a competitive advantage
• How to know if cloud computing is a good fit for your startup
• Answers to common myths about cloud computing
• Guidance through the decisions required to prepare your startup for taking advantage of the cloud
• Tips on how to move your existing startup to the cloud

Whether you are new to cloud computing or just want to learn more about how it applies to startups, this eBook will provide some new and interesting insights for you.

Learn more about the eBook or go straight to the download page.

Watch the full interview

• 43% said they’re not interested in cloud storage
• Another 43% were interested but have no plans to adopt
• 3% plan to implement a cloud storage platform in the next year
• 5% plan at least a year out
• 3% have already switched to cloud storage
• Only 1% are expanding an existing implementation

Some consider these stats as a bad thing for cloud computing. I see it as a non-issue. Here is why:

1. Many likely prefer to have local storage for fast access and security, even for general purpose data
2. They also would prefer to have flexibility in managing cost, with lower cost storage for least used data and higher cost put toward data that is heavily used or where performance is important
3. Using storage APIs from Amazon S3 and other service provides is similar to writing support for accessing your hard drive using low-level operating system calls – it takes considerable effort for managing something as simple as files

The survey did find that companies are more interested in using these services for backup. This makes sense, as it provides an option for offsite storage of data and often comes with higher level management of files and volumes (time differential logic, storage, and restoration).

From my research, most consumers of cloud storage are from those fully embracing the cloud architecture, namely startups. Startups are finding that it can provide elastic storage, redundancy, and can offload data transfer costs by utilizing smarter APIs to create temporary access keys to send owners directly to the data (where the cost is cheaper than costs at the server).

The conclusion? Using cloud storage services for backup is an easy decision; using it for general purpose storage is great to help reduce costs. Otherwise, local storage will still reign for some time to come.

Many Ruby/Rails Job Processing Libraries

There are quite a few Ruby and Rails job processing libraries – in fact, too many to list. Most of the libraries can be divided into the following groups:

1. Spawners – spawn a new process and wait or fork and forget
2. Single Process Workers – launch a single process, watch a queue or database table, and run each job in turn
3. Multiple Process Workers – launch as many processes as you want and let each one run a job. Coordination between the processes is done by a) a managing process, b) a message broker, or c) database tables

Since the workload was going to be heavy and the hardware we had was capable of running multiple processes, I opted for the multiple process worker type. The next step was to figure out which library to use.

Why Skynet for Ruby on Rails Job Processing?

Enter Skynet, a Ruby-based library written by Adam Pisoni. Adam uses the Map Reduce framework, patented by Google and powering many of Google’s services, including search. Here is Adam’s summary of Map Reduce:

At its simplest level, a MapReduce job defines a data set, a map method and a reduce method. It may also define a partition method. The MapReduce server evenly splits up (partitions) the data given to it and sends those chunks of data, along with a copy of the code in the map method, to workers that execute the map method against the data it was given. The output from each worker is sent back to the MapReduce server. At  this point the MapReduce server evenly partitions the RESULT data returned from the workers and sends those chunks of data along with the reduce code to the workers to be executed. The reducers return the final result which is returned to the process that requested the job be done in the first place. Not all jobs need a reduce step, some may just have a map step.

You can also read the full paper from Google at http://labs.google.com/papers/mapreduce.html.

So, How Does It Work?

Skynet works by launching one or more processes that are either a) masters (coordinators), b) workers, or c) both. Any master pulls a job off the queue (stored in the database) and dispatches the work or runs it locally (if it can perform both tasks). The worker performs the mapping step first and then the reduce step, if provided. When the job has completed, the results are stored back into the database for later retrieval, or returned back to the caller if one is waiting.

The Project Specifics

For this project, we have long-running jobs that can be broken into smaller units of work and then reassembled. The nice thing about Skynet is that it allows us to break up this work without having to track the results and assemble them ourselves. This allowed us to define a map step of processing the smaller work unit, then a reduce step of reassembling the pieces back together. We could launch as few or as many workers as we needed, to balance memory consumption and time-to-completion.

The results were quite impressive: a 5 minute job would be broken into approx 10 smaller units of work. The total running time to create the job, let Skynet map and reduce it, and receive the results was approximately 45 seconds on a dual quad core server with plenty of memory. We further optimized this by using memcached to cache calculations and prevent redundant work, allowing us to reduce the jobs further to as low as 15 seconds.

Tips for Using Skynet with Ruby on Rails

Using Skynet can have a large learning curve. We’ve had over 10 years of distributed computing experience and it still took a little time to familiarize ourselves with the gem. Here are some tips and suggestions:

1. Skynet requires a restart after a code change. So, write your job code in a separate class and test it. This may require you to design for testability, but it will allow you to ensure your code is stable before testing it live within Skynet
2. By default, Skynet will start 2 dual master/workers and some worker-only processes. If your masters get overwhelmed, it can cause your map/reduce jobs to stall as there is no master to coordinate the effort. Launch master-only jobs to prevent this
3. Skynet will dispatch work as it sees fit. If you want to ensure that your jobs are processed in a certain order, you may need to throttle the jobs into Skynet by fronting it with a message broker and queue processor. This is often overkill for most projects, but can help ensure that a job that has been split into smaller jobs is fully completed before the next one starts
4. The last version we used required that we refresh all ActiveRecord connections at the start of our job. Otherwise, AR would receive a timeout from our database connection and fail to process
5. Use good logging. Skynet provides a logging mechanism, so use it. You may need to include specific identifiers to provide context, as the log will fill quickly with the more processes you have running at once

You can view the full Skynet documentation at its Rubyforge home, or grab the latest source code from Github.

Finally, there seems to be a fork of the Skynet project by Brendan Baldwin. He looks to be making some really nice improvements, including Nagios monitoring support and dispatching jobs to Skynet from any object with a single call. We hope to evaluate his project fork soon.

Here is what we needed to deliver for our project:

1. Track a list of known points
2. Track the location of a specific item at a point in time
3. Determine a list of those known points that were within a specific radius of that item during that time

The MySQL Spatial Solution

After performing some extensive research on the current methods for geofencing, I opted to utilize MySQL’s spatial extensions to help store and calculate possible known points within the perimeter of a specific location. This allowed us to store each point as a MySQL geometry, then perform a special calculation to detect known points within the radius.MySQL even offers indexing of geometric columns if you opt for the MyISAM engine for the table.

Geofencing with ActiveRecord

Since ActiveRecord doesn’t have direct support for the MySQL spatial extensions, the handy geokit-rails gem by Andre Lewis was used. With this gem, we were able to create the proper migrations to save and restore the geometry locations for our models. After writing a helper method to make the query as fast as possible, our work on the Rails side was complete. We had a little MySQL-specific code, but it was isolated within the migration (via the gem extensions) and the model itself.

Results

I also performed some optimizations to our MySQL configuration and to the table itself. After that, I roughly measured the performance for our queries to <= 0.05 seconds against 4+ million known points stored in MySQL. More than enough for our performance needs!

Don’t Forget the Database!

Many developers forget about the power of using a database’s native extensions. Many architects and managers don’t want to bind to vendor extensions, preventing them from saving time and increasing performance.

MySQL’s spatial extensions, while following some industry standards regarding spatial data storage and support, wasn’t exactly a SQL standard. For this project, we made the right decision and saved considerable time of building the solution ourselves.

We were also reminded that sometimes spending time reviewing and selecting your architecture solutions during a discovery phase can pay off considerably.

1. Slice cloning from running slices – You can now create a new slice from the image of a running slice. Previously backups were the only way to clone a slice
2. Bandwidth has been increased as well – you can now push more data out on the public and private networks

Read the full post on the Slicehost blog

Auditing your application is an important step to take at regular intervals. I suggest conducting an audit every 3-6 months. This provides enough elapsed time for you to have learned something about your customers since the last audit, but prevents too much time from elapsing and your application from getting stale.

To conduct a customer experience audit:

1. Prepare a testing environment – this may require setting up a second copy of the system or using a staging system to prevent your auditing procedures from disrupting a production system. Some applications may not have this issue and may be able to use the production environment
2. Use a different workstation or browser – from experience, I’ve noticed that I often take the same paths through a system because I let my browser remember my last form entries. This saves me lots of time during development testing, but it also causes me to take the same paths and make the same assumptions. Use a different browser on your machine or switch to a different machine. Using virtualization technologies, such as VMWare or Xen Server, allows you to setup a fresh sandbox for testing your app if spare hardware isn’t available
3. Start by putting yourself in the place of a new customer and ask yourself some simple but effective questions:
   • What kind of problem am I experiencing that makes me want to look for this solution?
   • What kind of keywords, questions, or websites might I visit to find a solution?
   • What is compelling about your website to make them signup?
4. Walk through each screen in your application, asking yourself if the first-time user of your system will understand what is happening and what they need to do next
5. Capture each improvement as a separate ticket in your bug tracking system – be as detailed as possible about the problem and include one or more possible solutions for future consideration
6. (Bonus) Take a look at some new web apps to see if they have come up with innovative ways to deal with the same kind of workflow/data entry problems

Make time to audit your application on a regular basis. It will reset your understanding for what a new customer sees and how you can make your product even better.

Launching a startup requires time and patience. Most startups take 2-3 months to get a great application launched that is useful to their customer base. It also requires another 3-6 months to continue to refine it as customer feedback is obtained from those outside the friends and family network.

Seth Godin explains it best by calling it the patience trap:

The trap: Use all your money to build a fancy website and leave no money or patience for the hundred revisions you’ll need to do.

Planning on launching a startup? Be prepared to take the time and money to make those hundreds of small revisions as you learn different ways to grow your company.

Design carries a lot of baggage related to aesthetics. We say something is well-designed if it looks good. There are great designs that don’t look good, certainly, but it’s really easy to get caught up in a bauhaus, white space, font-driven, Ideo-envy way of thinking about design.

So I reserve “architect” to describe the intentional arrangement of design elements to get a certain result.

He goes on to talk about businesses that actively architect:

Stew Leonard’s, which used to be my favorite supermarket example, is architected to extract large amounts of money from customers. One example: there’s only one route through the store. You start at the beginning and work your way to the end. No one goes there to buy a half-gallon of milk. And he’s not going to win any design competitions either…

Or consider the architecture of the pricing at 37signals or the architecture of Hotmail’s viral marketing campaign years ago.

Architecture, for me anyway, involves intention, game theory, systems thinking and relentless testing and improvement. Fine with me if you want to call it design, just don’t forget to do it.

I agree with Seth. Architecting your startup requires more than making a few decisions about technology up front. It also requires startups to make decisions early on about their revenue model, marketing, and other aspects of their business.

Over time, these things may change as the startup learns more about what works and what doesn’t. When that change happens, you need to have a startup architect available to listen to where you want to go and help you get everything in place to get you there. They need to understand not only how to make technology decisions, but also to understand what kind of impact those decisions have for your business going forward.

That is what we do.

MySQL optimizations: We’ve been working with a MySQL performance consultant to help us optimize our database calls and queries. We’re still early in the process but we’ve learned a lot so far.

As with most applications, optimizations are application-specific. This means that developers have to make certain assumptions about how their application works when they optimize their code.

However, Rails is quite opinionated already, so do you think they’ll take some of the more general optimization techniques and wrap it into the MySql adapter in ActiveRecord? Perhaps they will offer some optimization mixins to allow developers to quickly improve performance when using MySql.

What do you think?