Big Fast Blog

Deploy The Tornado Chat Demo On The Stackato PaaS Sandbox

Phil Whelan — Tue, 10 Jan 2012 17:57:08 +0000

In this blog post I'll go through an example, in repeatable steps, of how to get up and running with the Tornado chat demo on ActiveState's public sandbox for Stackato. ActiveState's Stackato PaaS (Platform-as-a-Service) is based on VMware's open-source PaaS, Cloud Foundry, and offers an enterprise PaaS solution that will run on any public cloud, private cloud, laptop or desktop.

Geohash Intro

Phil Whelan — Fri, 16 Dec 2011 07:53:15 +0000

Geohashing is a simple way to encode latitude and longitude and grouping nearby points on the globe with varying resolutions. It was created by Gustavo Niemeyer. This blog post looks at how it's implemented any why it is such an elegant solution for encoding and managing location-based data.

Rollout, Degrade, Metrics and Capacity

Phil Whelan — Wed, 14 Dec 2011 00:16:36 +0000

Last week I attended a presentation by James Golick, CTO of fetlife.com. Intriguingly advertised as “Scaling a Website to 300 Million Page Views Per Month with Scala” I thought it was worth a break from building cooq.com to attend. Fetlife have been heavy users of Ruby, but are transitioning to Scala and having been using [...]

Summify’s Technology Examined

Phil Whelan — Mon, 07 Mar 2011 10:11:57 +0000

In this post I will look at the technology infrastructure behind Summify.com, a website that strives to make our lives easier and helps us deal with the information overload we all experience every time we sit down at our computers. Summify has aggregated over 200 million stories from the web and serves them up on-demand through a series of different mediums. The website uses Tornado to push real-time updates out to the users and they have developed over a dozen backend systems, some of which I will cover in this blog post.

Other People’s Data – An Interview With “Drawn To Scale”

Phil Whelan — Mon, 14 Feb 2011 19:13:56 +0000

In this blog-post Bradford Stephens, Drawn To Scale's founder, answers a series of technical, business and personal questions to give an overview of what Drawn To Scale is and where it is going. Who are the founders? What is their background, technology and business model? How were they going to manage other people's big data? Can one tool fit the demands from a broad range of data challenges that different businesses are seeing?

Quora’s Technology Examined

Phil Whelan — Mon, 31 Jan 2011 20:37:19 +0000

In this blog post I will delve into the snippets of information available on Quora and look at Quora from a technical perspective. What technical decisions have they made? What does their architecture look like? What languages and frameworks do they use? How do they make that search bar respond so quickly?

Run The Latest Whirr And Deploy HBase In Minutes

Phil Whelan — Tue, 25 Jan 2011 00:17:48 +0000

In a few of my recent posts I have covered the ease of deploying clusters of Hadoop and Cassandra using Whirr. With Whirr you can simply write a configuration file specifying which cloud provider you are using, your credentials and the definition of the cluster you desire and it will build it for you. In [...]

Quickly Launch A Cassandra Cluster On Amazon EC2

Phil Whelan — Sat, 22 Jan 2011 02:58:39 +0000

If you have read my previous post, “Map-Reduce With Ruby Using Hadoop“, then you will know that firing up a Hadoop cluster is really simple when you use Whirr. Without even ssh’ing on the machines in the cloud you can start-up your cluster and interact with it. In this post I’ll show you that it [...]

De-volatile Your Memcached. Upgrade to Membase

Phil Whelan — Fri, 21 Jan 2011 20:16:23 +0000

If you have used Memcached and have gotten carried away with how it can make everything faster, you may have felt some pain when that volatile memory disappeared into the ether due to a restart or a failure. Your house of cards comes tumbling down as suddenly nothing is in cache! Your application is scrambling around trying to fully serve your users and the database is screaming at the application, "what the *%$@ is going on up there?". All your lack-of-performance implementation sins are exposed in one fell swoop and your boss stops inviting you to play golf. It's tough, but a few hours later Memcached is singing again with a full belly of cache and your application layer is putting the kettle on to make your exhausted database a nice cuppa tea. You start to wonder... how can I persist this "caching" data?

SQLShell. A Cross-Database SQL Tool With NoSQL Potential

Phil Whelan — Wed, 19 Jan 2011 01:07:26 +0000

In this blog post I will introduce SQLShell and demonstrate, step-by-step, how to install it and start using it with MySQL. I will also reflect on the possibilites of using this with NoSQL technologies, such as HBase, MongoDB, Hive, CouchDB, Redis and Google BigQuery. SQLShell is a cross-platform, cross-database command-line tool for SQL, much like psql for PostgreSQL or the mysql command-line tool for MySQL.

Hosting Images With Google Storage Manager

Phil Whelan — Sat, 15 Jan 2011 00:32:21 +0000

Today I received my invite from Google "Google Storage for Developers". Yes, like most of us, my life if wrapped in layers of googliness. In this post I'm going to review briefly what Google Storage is and upload the image files, for the images you see in this post, using the Google Storage Manager.

Zero-Copy. Transfer Data Faster In Ruby

Phil Whelan — Fri, 14 Jan 2011 21:27:12 +0000

In this post I will explain the concept behind "zero-copy", which is feature of the Linux allowing for faster transfer of data between pipes, file-descriptors and sockets. I will demonstrate how you can use this functionality in your Ruby projects using a code example. This functionality has been implemented in C, Java, Ruby, Perl and nameless other languages, but in this blog I will focus on the Ruby usage.

The Apache Projects – The Justice League Of Scalability

Phil Whelan — Tue, 11 Jan 2011 23:48:00 +0000

In this post I will define what I believe to be the most important projects within the Apache Projects for building scalable web sites and generally managing large volumes of data.

Landsliding Into PostGIS With KML Files

Phil Whelan — Wed, 05 Jan 2011 17:25:39 +0000

In this post I will show, in repeatable steps, how to install PostGIS, load in geospatial data found in a KML file and run queries against that data. The focus of this geospatial data will be landslides and our resulting database will allow us to query, using longitude and latitude co-ordinates, the landslide status of a specific geographical point.

Embed Base64-Encoded Images Inline In HTML

Phil Whelan — Mon, 03 Jan 2011 19:23:17 +0000

Here is how you can embed an image in HTML inline. This is similar to how you embed an image in a HTML email message.

Map-Reduce With Ruby Using Hadoop

Phil Whelan — Fri, 31 Dec 2010 17:38:35 +0000

Here I demonstrate, with repeatable steps, how to fire-up a Hadoop cluster on Amazon EC2, load data onto the HDFS (Hadoop Distributed File-System), write map-reduce scripts in Ruby and use them to run a map-reduce job on your Hadoop cluster. You will not need to ssh into the cluster, as all tasks are run from your local machine. Below I am using my MacBook Pro as my local machine, but the steps I have provided should be reproducible on other platforms running bash and Java.

Homebrew – Intro To The Mac OS X Package Installer

Phil Whelan — Thu, 09 Dec 2010 09:51:15 +0000

I thought I should mention the very easy Homebrew package installer for Mac OS X. I’ve found it quite simple to use and have not turned back since I blew away my MacPorts. You can find Homebrew on gitub and you can find the recommended installation instructions there which are as follows. ruby -e "$(curl [...]

How To Get Experience Working With Large Datasets

Phil Whelan — Wed, 08 Dec 2010 19:50:28 +0000

There are data sources out there, but which data source you choose depends on which technology you wish to get experience working with. The experience should be of the technologies you are using, rather than what the data is. Certain datasets pair better with certain technologies. Simulating the data can be another approach. You just need a clever way of generating and randomizing your fake data. Thirdly, you can use a hybrid approach. Take real data and replay it on a loop, randomizing it as it goes through. Simulating the Twitter fire-hose should not be too hard, should it?

Find The Road To Your Happiness By Helping Others

Phil Whelan — Mon, 06 Dec 2010 21:28:13 +0000

This is a common theme I've heard in many of the books I've read. Although, in the books I've read, this pearl of wisdom is phrased a little differently. The way to build a successful business is to help as many people as you can. Apparently, the cash will follow, if you concentrate on the helping part. The number of people you help is also important. The more people you help, the better. For instance, Facebook "helps" 250 million people per day, whereas Google only helps around 90 million people per day. Helping all those people has become very profitable for these two companies and many more. It's all about changing the focus from "how do I make money" to "how do I help more people".

Highchart Vs Flot.js – Comparing JavaScript Graphing Engines

Phil Whelan — Fri, 03 Dec 2010 22:55:44 +0000

In previous projects at MailChannels I have used Flot.js for graphing. There were many reasons I chose this originally. The graphs are interactive and can be manipulated within the browser without having to communicate back to the server. You can retrieve additional data from the server which enables you to be able to scan left [...]

Install Gitolite To Manage Your Git Repositories

Phil Whelan — Fri, 03 Dec 2010 19:03:46 +0000

Recently, netSIGN asked me to setup gitolite to give external developers controlled access to git repositories. Gitolite enables easy management of this access control. In this post I will detail how I set this up. The first thing to note about the gitolite install is that the installer is run remotely. Therefore, you will want [...]

To Show Ads Or Not To Show Ads

Phil Whelan — Thu, 02 Dec 2010 22:47:05 +0000

I was just reading “An appeal from Wikipedia founder Jimmy Wales”, where he is asking for donations for supporting the continuation of Wikipedia, ad-free. I do not need to tell you what an amazing website Wikipedia is. It’s a vast pool of information with a very large volume of visitors. It would be a perfect [...]

Tools for Testing Ruby On Rails

Phil Whelan — Wed, 20 Oct 2010 18:05:15 +0000

I’m getting stuck into learning Ruby On Rails and the topic of which tools / frameworks to use for testing came up. Here’s some notes. I’ll be reading up on these testing frameworks in the following order. If you’ve got an thoughts, comments or other ideas, please let me know in the comments. Test::Unit Vanilla [...]

Great And Real Customer Service

Phil Whelan — Wed, 06 Oct 2010 04:20:37 +0000

Customer service has been on my mind recently. Funny, because I do not currently have any customers. I’ve been reading (listening to an audiobook, but it sounds funny if I say “listening to”) the Zappos: Delivering Happiness book. Last night I spoke with Alex Toul on Twitter who has just written a post entitled The [...]

Location, Location, Location

Phil Whelan — Fri, 01 Oct 2010 22:22:43 +0000

I’ve been meeting other Vancouver entrepreneurs at various networking events and discussing ideas with them. A common theme I’m seeing is that most of them are very focused on building technology with location-based data at the heart of the application. This is the next (or current) big thing and it feels like everyone is jumping [...]

Data Mining Without Hadoop

Phil Whelan — Thu, 30 Sep 2010 21:41:06 +0000

This is a follow-up to my recent blog-post on Working With Large Data Sets. That post had some interest, so I thought it would be a good idea to go through the methodologies I had used for processing this data. I entitled this “Data Mining Without Hadoop”, because I have experience using Hadoop, and although [...]

Bootstrappers Unite!

Phil Whelan — Wed, 29 Sep 2010 16:26:35 +0000

“A situation in which an entrepreneur starts a company with little capital. An individual is said to be bootstrapping when he or she attempts to found and build a company from personal finances or from the operating revenues of the new company.” – “Bootstrapping” by Investopedia I just got back from another session [...]

Working With Large Data Sets

Phil Whelan — Thu, 23 Sep 2010 18:35:57 +0000

For the past three and a half years I have been working for a start-up in downtown Vancouver. We have been developing a high performance SMTP proxy that can scale to handle tens of thousands of connections per second on each machine, even with pretty standard hardware. For the past 18 months I’ve moved from [...]

iPhone 3G’s iOS 4.1 Upgrade Cannot Come Soon Enough

Phil Whelan — Sun, 05 Sep 2010 17:48:12 +0000

When I signed up to Apple’s iPhone Developer programme I had full intention of developing some really cool apps. It has not happened yet, but I have been able to get the latest and greatest iPhone OS betas installed on my phone and have been able to start using the latest features early, which is [...]

Gmail Priority Inbox – The Email Marketer’s Nightmare

Phil Whelan — Fri, 03 Sep 2010 14:42:53 +0000

Gmail Priority Inbox is a great idea. Prior to this the Inbox was split into 2 levels – Inbox and Spam. This Priority Inbox creates a third middle level, which is essentially “ham”. Ham is all the things that waste your time, but come from organisations that you are affiliated with in some way, such [...]

Diaspora – Coming back out of the cloud will be painful

Phil Whelan — Wed, 01 Sep 2010 21:05:26 +0000

$200,641 has been pledged for the development of Diaspora, the decentralized web implementation, which intends to replace Facebook with a version that is distributed. Each user will retain their own data on their own machine. I see a huge flaw in this. They’re moving in the wrong direction. The web is moving to the cloud, [...]

Is Your Next Phone a Nokia?

Phil Whelan — Fri, 07 May 2010 22:25:42 +0000

This week I attended Mobile Geeks Vancouver where we were given a presentation by Samir Agarwal, Head of Maemo Operations at Nokia in Mountain View, California. The presentation by Samir was definitely eye opening. After winning an Android phone at the Vancouver Android Developers Meetup last week I thought I’d covered all the bases of [...]

An Android Developer, I Am

Phil Whelan — Sun, 02 May 2010 00:49:12 +0000

Well, I guess I have to be now, because I have just won a Samsung Galaxy Spica Android phone from Rogers. They kindly donated it to the Vancouver Android Developers Meetup. It’s running 1.5, so I want to get it updated to 2.1 and multi-touch-ness as soon as possible. More info to follow…

Is iChat the future of iPhone 4.0?

Phil Whelan — Thu, 22 Apr 2010 04:47:32 +0000

After seeing the iPhone 4G on Engadget and the front facing camera that will be included, I started to think about what Apple intends as the future of the device. For the iPad Wifi+3G Apple has negotiated some great deals with cellphone networks. These are basically no contract data plans with no phone service (iPad [...]

Mac OSX’s Preview Annotations

Phil Whelan — Tue, 13 Apr 2010 18:56:39 +0000

This is what I’ve been looking for all my life. Well, all my take-a-screenshot-to-help-explain-which-button-to-click-on life, anyway. It doesn’t happen very often, but when it does you want to do it quickly. I’m talking about taking a screenshot and adding some simple annotations. Draw a circle around this button, add a arrow to point to that [...]

Sclogging – A Different Slant On Blogging

Phil Whelan — Mon, 29 Sep 2008 00:32:00 +0000

This week I had an idea for a different form of blogging. This stemmed from the fact that we’ve been seeing a great response with opt-in email and responsiveness to email campaigns. Communicating directly with your audience via opt-in email gives you such a closer and detailed view of who your are communicating with than [...]

Welcome To Rogers – The Story Of The Phantom Caller

Phil Whelan — Sat, 13 Sep 2008 18:18:00 +0000

I recently purchased my first iPhone. Here in Canada, like every other country, there’s only one provider of the iPhone, which is Rogers. I’ve really not enjoyed my experience with Bell and I had heard that Rogers was just the same so it made no difference in switching. When I switched our numbers over the [...]

iPhone 3G – Battery Dies Twice As Fast

Phil Whelan — Wed, 10 Sep 2008 20:42:00 +0000

I’m finding the battery 3G iPhone is dying very fast. Hopefully the 2.1 iPhone firmware update being released on Friday will do as they say it will and make the battery life better.

"Simplify Media" For The iPhone – Too Simple?

Phil Whelan — Mon, 08 Sep 2008 04:26:00 +0000

Like most people nowadays my computer’s music collection has grown out of all proportion. I have more music on there than I could ever hope to listen to. Gigabytes on top of gigabytes. Luckily there are plenty of great apps out there like iTunes that can manage this and let me search, rate, sort, structure [...]

Google Chrome In Parallels – Flash Is Too Slow

Phil Whelan — Sun, 07 Sep 2008 00:38:00 +0000

I use Windows on my iMac using Parallels and Firefox runs very well. There’s not really anything I cannot do, since it is actually the fasted experience of Windows I’ve ever had. This is mainly due to using Macs and having not needed Windows for a long time, whilst the speed to computers and virtualization [...]

Pandora Genome Meets Apple iTunes?

Phil Whelan — Fri, 05 Sep 2008 15:09:00 +0000

With iTunes 8 just around the corner and some rumored changes that include iTunes suggesting songs you might like or in some way to grouping your music better, I saw a potential acquisition for Apple or possibility or partnership with the Internet radio station Pandora. Pandora uses what it terms “the music genome” that it [...]

Amazon Buys Shelfari. Will They Buy Delicious Library Too?

Phil Whelan — Thu, 04 Sep 2008 03:51:00 +0000

No not Del.ici.ous, or is it De.li.cio.us? Can never remember. Anyway, neither of those. I’m talking about the Mac app “Delicious Library”. I thought of this because the design of Shelfari looks just like Delicious Library. Delicious Library is really impressive (when the scanner works). You wave the bar code of any book, CD or [...]

Ubuntu is great

Phil Whelan — Tue, 08 May 2007 23:16:00 +0000

Ok, so I just updated to the latest version of Ubuntu – “Feisty Fawn”. Couldn’t notice a whole lot of difference on my work machine, and the effects didn’t work. But on my laptop it seems much much faster and that’s with the wobbly windows and other effects running full whack. The update process wasn’t [...]

Things to look out for when building a large application.

Phil Whelan — Mon, 29 May 2006 15:26:00 +0000

Things to look out for when building a large application. Joshua Schachter, del.icio.us A great list of advice for anyone building a large scale website.

"Ads By Microsoft"

Phil Whelan — Sun, 07 May 2006 03:14:00 +0000

Microsoft to Put More Money Into MSN – New York Times I’m surprised so many large companies are so slow to duplicate Google’s highly successful “Google Ads”. We developed the system for selecting appropriate ads to display on our livedoor.com pages and the biggest repository of Japanese blogs. Where we failed was not developing the [...]

AOL to Add Free Phone to Instant Messaging Feature – New York Times

Phil Whelan — Sun, 07 May 2006 02:33:00 +0000

AOL to Add Free Phone to Instant Messaging Feature – New York Times This is a great idea. Quite an investment for AOL, but is definately a way to catch up with Skype. The main problem I have with AOL Instant Messenger at the moment is how ugly it is. I like the fact it [...]

Share your internet connection and join the free wifi club

Phil Whelan — Tue, 07 Feb 2006 00:28:00 +0000

Many large companies such Skype, eBay and ICQ have just investing in a new intiative called FON to allow anybody with a home broadband internet connection to install update software on their router (the little internet box, that that allows you to plug in multiple computers to the internet and have wireless internet in your [...]

Lunarpages – Amazing support!

Phil Whelan — Sat, 05 Nov 2005 04:03:00 +0000

For nearly 3 years now I’ve been using Lunarpages web hosting, and every now and then I have to contact their support. Rarely has it ever been a problem with service. Mostly it’s to install a new Perl module I need to use. But everytime, no matter what day or time it is over in [...]

GP2X Game System (GPX2)

Phil Whelan — Fri, 21 Oct 2005 09:56:00 +0000

The GP2X Game System (GPX2) looks like a good cheap alternative to the video iPod offered by Apple, especially if you already have a 1Gb SD card lying around like me. It’s got linux on it, plays lots of games emulators, has lots of video codecs, and easy to replace batteries with 2 AA’s if [...]

TWiT – "This Week In Tech" – The most popular podcast?

Phil Whelan — Mon, 17 Oct 2005 19:40:00 +0000

I’m enjoying listening to these fun hour long discussions from Leo Laporte’s gang on anything tech related. Currently they’re doing it just for the love of it, but with a quarter of a million listeners and a growing fan-base will they sell out and go commercial? They don’t think so. You can subscribe to the [...]