Watch the video to find out how it all came together using MyLocalReporter (via Facebook).

I love it when our hard work to launch a startup ends up helping others through that startup! That is what we are all about!

Want to know more about the technical details of we put the site together? Read on…

How We Launched MyLocalReporter.com

Dave approached us with a deep understanding of PR and a vision of being able to help small businesses target local media outlets without spending thousands of dollars in PR. After a few hours of discussion, we put together a plan for a Minimum Viable Product (MVP) that met both the business goals of MyLocalReporter and the desired budget.

To accelerate the delivery of the project, we selected a few key technologies:

1. Ruby on Rails – a web application development framework that is great for both small and large-scale web sites
2. MongoDB – a database that provides a high-performance, flexible, schema-less database for rapid development
3. Heroku – a low cost Platform-as-a-Service (PaaS)  that provides easy deployment of any Ruby on Rails, Java, or JVM-based language with on-demand scaling

Combining these technologies, along with our test-driven development process, we were able to accelerate the development of the MVP. We then integrated the design provided by The MOD Studio, who had already engaged with Dave in a previous project. They delivered both the design and the WordPress site for the marketing and sales site, while Blue Jazz delivered the web application. This provided Dave with the ability to engage with his existing design team, who already understood his business needs, while leveraging Blue Jazz to launch the first version of his web application.

The work for MyLocalReporter isn’t done yet. Dave has more planned and we will be there to help him launch his ideas. In the mean time, visit his web site and discover how your business can take advantage of targeted, hyper-local press releases.

The Approach

With rvm once again ready to do my bidding, I proceeded to install the following gems into each of the versions of Ruby installed:

1. fastercsv (1.5.3)
2. mysql (2.8.1)
3. sequel (3.24.1)
4. jdbc-mysql (5.1.13) (JRuby only)

I then generated 10 files that each contain 10k rows of CSV data. I inserted these into a MyISAM table 5 times, for a total of 500k rows. Now, the results…

The Results

I was amazed at how much faster Ruby 1.9.2 with the MySQL gem was compared to JRuby with the same gem. However, once you move to the JDBC MySQL gem with JRuby, it edges out Ruby 1.9.2 slightly.

The Analysis

What do these results really mean? Here are a few take aways:

1. Ruby 1.9.2 can insert 2x as many rows per second than Ruby 1.8.7
2. JRuby with the MySQL JDBC gem can insert nearly 3x as many rows per second than Ruby 1.8.7
3. JRuby with the MySQL JDBC gem performs 20% faster than Ruby 1.9.2 with bulk inserts
4. (bonus) Rubinius 1.2.x and 2.0.0pre both have known issues with poor file I/O performance at the moment

Need to insert large amounts of bulk inserts where performance is key? Consider moving to JRuby and using the JDBC MySQL gem, or at a minimum move to Ruby 1.9.2 if you have not done so already.

I was working on my first entrepreneurial adventure: I had an idea and thought I needed a technical person to build it…I met some technical folks, but none were available. As a result, I began by really focusing on the business plan and how the solution I dreamed up was going to make money.

…

One of my mentors suggested building a prototype. After some quick research I settled on Ruby/Rails given that the developer community seemed to be very excited about it and made it easy for newbies to learn…After a few months I had a prototype and it actually worked.

…

He goes on to talk about how he eventually met a co-founder through the process of learning and talking to developers during this process. While this path may not be best for everyone, it does shed some light on the issue of how to get your venture started when you don’t have a technical co-founder. For Will, it meant doing it himself. For others, it might mean one of the following paths:

1. Working with a designer that can help provide an interactive, clickable PDF for pitches
2. Building a small prototype (often best if a clickable PDF is not expressive enough or a designer isn’t available)
3. Bringing in an outside development team to build the first version

Now, while I may be a little biased to option 3, I have seen each of these steps applied in turn to help solidify the idea and generate creative revenue streams that had not been thought out while developing the requirements and business plan.

So, why consider an outside development team rather than hiring your own? Several reasons:

1. Unless you or a founding member have technical experience, you may not know what kind of skills to look for in your first hires
2. You become too focused on using your developers because you have them, distracting you from the business of generating revenue
3. More difficult skills can be obtained for shorter periods of time to address a particular development need

Need help selecting your development team? You can find my top recommendations on how to select a development team at LaunchAny.com

Project Background

My current project is a massive data generation tool, targeted at generating from gigabytes to terabytes and beyond for very large databases (VLDBs) and Big Data projects. Having recently completed a VLDB project that applied some areas of Big Data to the solution, I quickly realized that we are entering a new world. I’ve worked with large datasets before, up to 10+ TB in size. In the past, these kinds of projects were rare. Today, I’m seeing them over and over. I realized the need to generate large datasets for optimizing areas of the codebase and data storage is necessary so that I don’t have to spend cycles trying to contrive and load enough data to verify my architecture and implementation. Thus, by new project to generate large datasets in a deterministic and realistic fashion.

Comparing Ruby Virtual Machines

Given this background, I proceeded to mockup a very naive prototype of the application. Something that could be contained within one source file and easily benchmarked. While the prototype won’t reflect anything close to reality (yet), it gave me an excuse to explore the most current Ruby VMs available. With RVM at hand, I proceeded to install Ruby 1.8.7, Ruby EE, Ruby 1.9, Rubinius, and JRuby to see what would happen. I then built the prototype to include the majority of what I needed: random data generation and file I/O for 30 millions of rows. Here is what I found:

As you can see, JRuby outperformed the other VMs for my project. Obviously, this isn’t an exhaustive comparison of all areas of the VMs, but it does show that the majority of what my application will be performing is much faster on JRuby than other VMs at this time.

Making the Decision for JRuby

As I reviewed these results, my conclusion became easy: JRuby. But my decision wasn’t just focused on the numbers. Before Ruby, I put in nearly 15 years of Java in either a full-time or part-time capacity. I used to know details about the Java VM and bytecode structure that most didn’t want to know. So, going back to Java for the speed may have been the right choice, but these days I prefer Ruby. I enjoy writing applications in Ruby. It makes me happy. And a happy developer is a productive one, especially in a language that is productive for me. JRuby gives me the best of both worlds, allowing me to take advantage of the Java VM and HotSpot compiler while allowing me to write Ruby and dropping to Java for specific portions of the application when necessary.

Things I’m looking forward to exploring further with JRuby: compiling Ruby into Java classes using the JRuby compiler, JRuby performance tuning, and JRuby local and remote monitoring

Final Thoughts

Before you jump into a platform, consider reviewing the landscape. Try building some prototypes that will exercise the majority of your application’s logic and compare the results. Every Ruby VM has a different personality as well, so determine if you are comfortable with the VM before moving to it. Also, look for other opportunities on the new platform that didn’t exist before, such as better debugging, profiling, and technology integrations. This will help you make a more informed decision.

FYI, expect more updates soon on my experiences around JRuby on VLDB and Big Data projects, as time allows.

As with anything, understanding each subsystem and identifying the key performance issues will often cut through the confusion and help to construct the most optimum solution. Here is my simple introduction and guide to selecting the best components for your next development machine purchase or build, while saving you time and anguish trying to sort through the variety of information available.

Buying vs. Building a Machine

So, should I just buy a new machine? Often, this can solve your immediate problem. Apple offers a solution for powerful desktops, as do Dell, HP, and others. Simply buy a machine with a faster processor, more memory, and multiple drives and you can get far.The problem lies in the fact that many of these builders don’t offer the most powerful or current hardware available, as they tend to lag by 3-6 months. This typically provides an advantage, as the builders will determine what parts work the best and then combine them together. However, unless you go for a high-end machine from these builders, they will often substitute one part for another, based on their current inventory. The result is a machine that is not quite to spec, or lagging behind the most current technology available. If you prefer to find the best machine performance possible, then the best option is to build a custom machine.

If you are reading this article, my assumption is that you plan on working with large amounts of data using multiple processes or virtualized servers within a development environment, and will therefore benefit from the best performance you can buy. To do this, you must have a basic understanding of how to select the best parts for your machine that will work together properly. Either way, this guide will help you better understand your options before you buy or build your next development workstation.

Selecting the Processor and Memory

Processors are commonly thought of as throw-away parts. Find the fastest and install it. However, with the current selection of processor families, there are often several choices. On the Intel side, there are currently options for the i5 and i7 families. Within each family are options for the number of cores (often 2, 4, or 6 cores), the L2 and L3 cache size, and the QPI rating. The fastest i7 processor with the most cores, L2 and L3 cache, and the highest QPI will often provide the most processing power when it comes to working with Big Data solutions like Hadoop or distributing computing applications.

Memory can be a little more confusing. Since the i7 processor can talk to the memory using triple channels, it is best to select the proper memory to complement your processor selection. Memory is commonly rated with four numbers, such as 7-7-7-21, indicating the latency of the memory (deeper research about these numbers is interesting but left as an exercise for the reader). This means finding the fastest DDR3 triple channel memory with the lowest latency and buying enough to handle your development needs. Often, you have to inform the BIOS of your motherboard about the speed of your memory, otherwise it will default to a slower rating for stability. More on that later.

Motherboard

If you have every walked into a store like Fry’s, you will see a large selection of motherboards. They vary is sizes, features, and capability. Most motherboards are designed for the average price-conscious consumer that wants to see the most number of features for the dollar. In reality, development desktops need something different. They need to be reliable for long-term use, with 24×7 uptime and stability. I have found that Asus is often a good choice as a motherboard manufacturer, particularly their Sabretooth X58 motherboard. Do some research for the operating system you will be running to ensure the best compatibility, as even the best manufacturer may take a short-cut occasionally and install a controller chip that isn’t as friendly for some operating systems (especially Ubuntu/Linux).

Hard Drives and RAID

If you own a modern laptop, you may already have a solid state drive (SSD drive) installed. They are silent, having no moving parts, and are quite fast to boot your favorite operating system as some can achieve over 300MB/s during read access. These drives are sufficient for most uses and for boot drives for the OS, but often they are not as efficient when it comes to mass storage requirements. Big Data and VLDBs often require large amounts of throughput and storage during development, which is currently beyond the capacity of modern SSDs. This means that traditional spindle-based SATA drives are still the best option for our needs. This does not, however, prevent you from mixing your storage subsystem with a combination of SSD drive(s) for the OS and spindle-based drives for mass data storage. The primary concern when selecting your hard drives are speed, reliability, and capacity.

SATA drives are commonly available in different speeds: 5400, 7200, and 10k rpm. The faster and smaller the drive, the faster data can be accessed and written. SAS drives are a newer form of spindle-drive commonly found on higher end server machines and can achieve speeds of up to 15k rpm and even faster seek times. They are very expensive, however, and therefore often out of the reach of most developers. Another factor is the mean time between failure (MTBF) rating, which is an estimate of the number of hours before the drive is likely to fail. Lower-grade consumer drives often have lower MTBF and are more likely to fail while server-class drives have a much higher MTBF. They also run cooler, which is another important factor for long life and reliability. The best choice at this time is the Western Digital Velociraptor 10k rpm drives, currently found in 300 and 600GB sizes. They have a higher MTBF, run cool, and are generally a good choice for high performance desktop use.

RAID controllers allow multiple drives to be combined and viewed as a single device by the operating system. RAID 0 allows two or more drives to be striped (written to and read from in parallel), although if any one fails the entire system fails. RAID 1 allows for two or more drives to be mirrored, providing a simple form of redundancy and parallel reads (assuming the controller allows it). RAID 5 and 6 provide a more economical way to provide data redundancy but can sometimes hinder performance when it comes to the way some database servers read and write data to the filesystem. I’ve found that the most optimum RAID choice for development machines is RAID1+0 (sometimes called RAID 10). This means that there are two RAID 0 drive arrays that are then combined into a RAID 1 pair, allowing for very fast reads and redundant data storage when writing.

When selecting a controller for your RAID configuration, many utilize the onboard RAID controller from their motherboard. While many motherboards are now using more popular RAID controllers from known vendors, they still require the use of the processor to perform the RAID logic. Before you choose to use the onboard controller, do some research for issues during RAID recoveries and rebuilding after drives fail. Often, these controllers are great when they work but terribly slow or unreliable during a rebuild after a drive failure. My recommendation is to select a separate RAID controller card that has a dedicated processor, onboard memory (sometimes with a battery backup), and support for 4, 8, 16, or 32 drives. These cards are typically $300 or more in price, while lower priced cards are likely no different than the controllers available on today’s motherboards. I prefer the Adaptec 5805 series RAID controller. These cards can sometimes be obtained at a discount due to an open box or overstock. Caution: be sure to purchase the kit, not just the standalone card, or else you will be missing the cable sets that are often required to connect the card to the drives.

One final note about hard drive selection: the larger the hard drives selected, the longer it takes to rebuild a RAID device when a drive fails. If you have a drive failure and replace your drive, you may still be running in degraded mode (one or more drives are unavailable and being rebuilt) for some time. This can be a huge factor in the middle of a development cycle, so minimizing the rebuild time is a must.

Case and Power Supply

Selecting a case is mostly an aesthetic decision, although there are a few factors to consider: number of available hard drive bays, location of the power supply, and airflow. The number of available drive bays must meet or exceed the number of drives you intend on installing and may require that you locate a case that supports the smaller SSD drives if you plan on installing one without an adapter for the larger SATA-sized bays. Power supplies are traditionally located at the top of the case, though more are starting to locate them at the bottom for easier access, a lower likelihood of tipping over, and aesthetics. Finally, identifying a case that has good airflow from front-to-back, plenty of locations for fans of 120mm or larger, and even CPU  funnels will help extract the heat produced from your processor, video card, and hard drives efficiently. I prefer the Antec Nine Hundred v3 (not the v2) or the Twelve Hundred, as they provide plenty of fans, great airflow, plenty of hard drive bays, and have filters on the front fans for easily removing dust.

Power supplies are often the most overlooked and underrated part of the desktop. One is just like the other, right? Wrong! The parts inside the power supply can vary in quality and power efficiency, and can easily take down your entire machine if they fail. Find a power supply from a well-rated manufacturer that offers the appropriate rating in watts for your machine. Most i7 processors with up to 4 hard drives and a moderate video card can utilize a gold series 800W or better power supply. Also, look for modular power supplies, as they allow you to select only the cables you need for your installation, reducing the number of cables you have to deal with inside your desktop case. Finally, ensure that the cables supplied with the power supply will reach all devices in your selected case, from the motherboard power sockets to the hard drives and DVD.

My Setup

To conclude this guide, I’ll outline the setup I built in the fall of 2010, to help get you started. I found that this was the best combination of power and low noise for a pleasant yet powerful work environment:

• Intel i7 980X Extreme Edition Processor (6 core) w/ stock CPU fan on quiet mode
• G.Skill 7-7-7-21 1600 DDR3 Memory
• ASUS Sabretooth X58
• Adaptec 5805 RAID controller (up to 8 drives)
• 4 x Velociraptor 600GB 10k rpm drives (6GB/s) in RAID 10
• Antec Nine Hundred v3 case
• ASUS AHA5450 Silent ATI video card
• Ubuntu Linux 10.10 w/ native ATI drivers installed (I prefer Gentoo but Ubuntu is much easier to maintain on a daily basis)

Memory Performance Results

Memtest86+ results before and after I adjusted the BIOS to use the 7-7-7-21 memory settings:

1. Mem stable at 9-9-9-24 (BIOS default):
   BLCK 133 Mhz // QPI 6.4 GT/s // RAM 668 Mhz // L1=65509 MB/s // L2=22574 MB/s // L3=6449 MB/s // Memory speed 4949 MB/s
2. Mem stable at 7-7-7-21 (spec):
   BLCK 141 Mhz // QPI 6.7 GT/s // RAM 705 Mhz // L1=69176 MB/s // L2=22328 MB/s // L3=15679 MB/s // Memory speed 9069 MB/s

Hard Drive Performance Results

Finally, here are some quick performance tests with the 4 drives in RAID 10. While I could add 4 more drives to achieve higher results, these results are great and provide a nice balance between the performance vs. cost, heat, and power consumption.

ubuntu@ubuntu:~$ sudo hdparm -tT /dev/sda

/dev/sda:
Timing cached reads:   17128 MB in  2.00 seconds = 8570.32 MB/sec
Timing buffered disk reads:  1562 MB in  3.00 seconds = 520.29 MB/sec

Conclusion

I hope that this article has been beneficial and will assist you in the key decisions needed to build a powerful desktop machine for high performance development. If you have any questions, please drop me a comment or a note using the contact form and I’ll do my best to answer any questions you may have.

The topics covered include:

• What is cloud computing?
• How your startup can take advantage of cloud computing to gain a competitive advantage
• How to know if cloud computing is a good fit for your startup
• Answers to common myths about cloud computing
• Guidance through the decisions required to prepare your startup for taking advantage of the cloud
• Tips on how to move your existing startup to the cloud

Whether you are new to cloud computing or just want to learn more about how it applies to startups, this eBook will provide some new and interesting insights for you.

Learn more about the eBook or go straight to the download page.

Watch the full interview

• 43% said they’re not interested in cloud storage
• Another 43% were interested but have no plans to adopt
• 3% plan to implement a cloud storage platform in the next year
• 5% plan at least a year out
• 3% have already switched to cloud storage
• Only 1% are expanding an existing implementation

Some consider these stats as a bad thing for cloud computing. I see it as a non-issue. Here is why:

1. Many likely prefer to have local storage for fast access and security, even for general purpose data
2. They also would prefer to have flexibility in managing cost, with lower cost storage for least used data and higher cost put toward data that is heavily used or where performance is important
3. Using storage APIs from Amazon S3 and other service provides is similar to writing support for accessing your hard drive using low-level operating system calls – it takes considerable effort for managing something as simple as files

The survey did find that companies are more interested in using these services for backup. This makes sense, as it provides an option for offsite storage of data and often comes with higher level management of files and volumes (time differential logic, storage, and restoration).

From my research, most consumers of cloud storage are from those fully embracing the cloud architecture, namely startups. Startups are finding that it can provide elastic storage, redundancy, and can offload data transfer costs by utilizing smarter APIs to create temporary access keys to send owners directly to the data (where the cost is cheaper than costs at the server).

The conclusion? Using cloud storage services for backup is an easy decision; using it for general purpose storage is great to help reduce costs. Otherwise, local storage will still reign for some time to come.

Many Ruby/Rails Job Processing Libraries

There are quite a few Ruby and Rails job processing libraries – in fact, too many to list. Most of the libraries can be divided into the following groups:

1. Spawners – spawn a new process and wait or fork and forget
2. Single Process Workers – launch a single process, watch a queue or database table, and run each job in turn
3. Multiple Process Workers – launch as many processes as you want and let each one run a job. Coordination between the processes is done by a) a managing process, b) a message broker, or c) database tables

Since the workload was going to be heavy and the hardware we had was capable of running multiple processes, I opted for the multiple process worker type. The next step was to figure out which library to use.

Why Skynet for Ruby on Rails Job Processing?

Enter Skynet, a Ruby-based library written by Adam Pisoni. Adam uses the Map Reduce framework, patented by Google and powering many of Google’s services, including search. Here is Adam’s summary of Map Reduce:

At its simplest level, a MapReduce job defines a data set, a map method and a reduce method. It may also define a partition method. The MapReduce server evenly splits up (partitions) the data given to it and sends those chunks of data, along with a copy of the code in the map method, to workers that execute the map method against the data it was given. The output from each worker is sent back to the MapReduce server. At  this point the MapReduce server evenly partitions the RESULT data returned from the workers and sends those chunks of data along with the reduce code to the workers to be executed. The reducers return the final result which is returned to the process that requested the job be done in the first place. Not all jobs need a reduce step, some may just have a map step.

You can also read the full paper from Google at http://labs.google.com/papers/mapreduce.html.

So, How Does It Work?

Skynet works by launching one or more processes that are either a) masters (coordinators), b) workers, or c) both. Any master pulls a job off the queue (stored in the database) and dispatches the work or runs it locally (if it can perform both tasks). The worker performs the mapping step first and then the reduce step, if provided. When the job has completed, the results are stored back into the database for later retrieval, or returned back to the caller if one is waiting.

The Project Specifics

For this project, we have long-running jobs that can be broken into smaller units of work and then reassembled. The nice thing about Skynet is that it allows us to break up this work without having to track the results and assemble them ourselves. This allowed us to define a map step of processing the smaller work unit, then a reduce step of reassembling the pieces back together. We could launch as few or as many workers as we needed, to balance memory consumption and time-to-completion.

The results were quite impressive: a 5 minute job would be broken into approx 10 smaller units of work. The total running time to create the job, let Skynet map and reduce it, and receive the results was approximately 45 seconds on a dual quad core server with plenty of memory. We further optimized this by using memcached to cache calculations and prevent redundant work, allowing us to reduce the jobs further to as low as 15 seconds.

Tips for Using Skynet with Ruby on Rails

Using Skynet can have a large learning curve. We’ve had over 10 years of distributed computing experience and it still took a little time to familiarize ourselves with the gem. Here are some tips and suggestions:

1. Skynet requires a restart after a code change. So, write your job code in a separate class and test it. This may require you to design for testability, but it will allow you to ensure your code is stable before testing it live within Skynet
2. By default, Skynet will start 2 dual master/workers and some worker-only processes. If your masters get overwhelmed, it can cause your map/reduce jobs to stall as there is no master to coordinate the effort. Launch master-only jobs to prevent this
3. Skynet will dispatch work as it sees fit. If you want to ensure that your jobs are processed in a certain order, you may need to throttle the jobs into Skynet by fronting it with a message broker and queue processor. This is often overkill for most projects, but can help ensure that a job that has been split into smaller jobs is fully completed before the next one starts
4. The last version we used required that we refresh all ActiveRecord connections at the start of our job. Otherwise, AR would receive a timeout from our database connection and fail to process
5. Use good logging. Skynet provides a logging mechanism, so use it. You may need to include specific identifiers to provide context, as the log will fill quickly with the more processes you have running at once

You can view the full Skynet documentation at its Rubyforge home, or grab the latest source code from Github.

Finally, there seems to be a fork of the Skynet project by Brendan Baldwin. He looks to be making some really nice improvements, including Nagios monitoring support and dispatching jobs to Skynet from any object with a single call. We hope to evaluate his project fork soon.

Here is what we needed to deliver for our project:

1. Track a list of known points
2. Track the location of a specific item at a point in time
3. Determine a list of those known points that were within a specific radius of that item during that time

The MySQL Spatial Solution

After performing some extensive research on the current methods for geofencing, I opted to utilize MySQL’s spatial extensions to help store and calculate possible known points within the perimeter of a specific location. This allowed us to store each point as a MySQL geometry, then perform a special calculation to detect known points within the radius.MySQL even offers indexing of geometric columns if you opt for the MyISAM engine for the table.

Geofencing with ActiveRecord

Since ActiveRecord doesn’t have direct support for the MySQL spatial extensions, the handy geokit-rails gem by Andre Lewis was used. With this gem, we were able to create the proper migrations to save and restore the geometry locations for our models. After writing a helper method to make the query as fast as possible, our work on the Rails side was complete. We had a little MySQL-specific code, but it was isolated within the migration (via the gem extensions) and the model itself.

Results

I also performed some optimizations to our MySQL configuration and to the table itself. After that, I roughly measured the performance for our queries to <= 0.05 seconds against 4+ million known points stored in MySQL. More than enough for our performance needs!

Don’t Forget the Database!

Many developers forget about the power of using a database’s native extensions. Many architects and managers don’t want to bind to vendor extensions, preventing them from saving time and increasing performance.

MySQL’s spatial extensions, while following some industry standards regarding spatial data storage and support, wasn’t exactly a SQL standard. For this project, we made the right decision and saved considerable time of building the solution ourselves.

We were also reminded that sometimes spending time reviewing and selecting your architecture solutions during a discovery phase can pay off considerably.