Last week, at the EGaming Review 2012 Operator Awards, BetDash took home the Innovation in Gaming award. Presented to the company with the most disruptive innovation, the award is a great testament to the contributions of everyone on the BetDash team and within the Paddy Power organization.

Three years ago, we had an early prototype, some grand visions, but very little background in what is a huge industry outside of the US. Now we are poised to disrupt a part of that industry. Lots of exciting potential for BetDash!

Just to give you a sense of what these villages in Ethiopia are like, the kids (and most of the adults) there have never seen a word. No books, no newspapers, no street signs, no labels on packaged foods or goods.

“We left the boxes in the village. Closed. Taped shut. No instruction, no human being. I thought, the kids will play with the boxes! Within four minutes, one kid not only opened the box, but found the on/off switch. He’d never seen an on/off switch. He powered it up. Within five days, they were using 47 apps per child per day. Within two weeks, they were singing ABC songs [in English] in the village. And within five months, they had hacked Android. Some idiot in our organization or in the Media Lab had disabled the camera! And they figured out it had a camera, and they hacked Android.”

Ethiopian kids hack OLPCs in 5 months with zero instruction

Originally from MIT Technology Review

One of the biggest improvements at BetDash.com in October was our migration into a new production environment. We had outgrown our previous environment and took the opportunity to improve many aspects of our production setup as we built out the new systems.

The majority of our servers are now virtualized using VMWare ESX. Of the BetDash servers, only our database remains on physical hardware – this is at the recommendation of the Oracle MySQL consultant we worked with on our deployment.

This is what our infrastructure looks like in our new environment and how our stack is distributed across the different tiers:

• Citrix Netscaler load balancers
• Web servers running Apache, serving the contents of our Rails app’s public folder, i.e. all static assets (CSS/JS/Images). This content is also replicated into our CDN.
• Passenger Enterprise on top of Apache for our front-end application servers, serving our primary Rails 3.2 app. These boxes are presently 8 virtual CPUs and 8 GB RAM
• Resque, Resque Scheduler, and other proprietary backend applications run on our backend application servers – these are 16 virtual CPUs and 32 GB RAM.
• Dedicated instances for our Admin panel application, Redis, Memcached, and other internal applications utilized by our system.
• Redis is currently deployed in a master/slave replicated setup, however, we’re keeping a close eye on Redis Sentinel emergence from beta and are also looking to deploy Redis Failover on top of Apache Zookeeper.
• MySQL is on physical hardware, configured as a RedHat Conga Cluster with the database files themselves sitting on a SAN. Each node in the cluster has 12 physical CPUs and 96 GB RAM.
• Dedicated MySQL instances for reporting and data warehousing

The instances are built with Puppet and the applications are deployed out of git using a combination of Capistrano and Paddy Power’s proprietary release tool.

One of the most common comments in regard to our stack and environment is the question of why we’re running on our own VMs and hardware as opposed to hosting in the AWS cloud like everyone else. The regulated nature of real-money gaming requires that our systems sit in the Isle of Man and thus this rules out the IaaS options. It was extremely interesting to see the number of leading Rails consultancies, which we were speaking with in regards to our deployment strategy and tuning our application for the new environment, that haven’t worked with non-IaaS systems in years.

With our move to these systems, we’re well positioned as we continue to scale the BetDash platform, thanks to the hard work of a large cross-functional project team. Our next step will be to move to run in an active/active configuration across Paddy Power’s multiple data center sites.

There was an excellent nugget of management advice in Bob Sutton’s Stanford Entrepreneurial Thought Leaders podcast, What Great Leaders Do. Dr. Sutton, Professor of Management Science and Engineering in the Stanford Engineering School, referenced work by Rob Cross, from the University of Virginia. Apparently, the findings were derrived from a network analysis study done for a large management consulting firm. (If you want to listen to the section, it starts around 43:25 in the podcast.)

In this study, Cross included a question in the study that had interesting results:

After you talk with your boss, do you have more or less energy?

When analyzing the results from the study and subsequent studies using the same question, they found a strong correlation between whether the employee was promoted, management’s performance evaluation of the employee (whether the employee was promoted/fired) and whether the manager is surrounded with a network of energized innovators.

Therefore, Dr. Sutton suggests that the most salient question for managers to ask themselves is:

How do people feel after they interact with you?

I think this is a great way of boiling down what can be difficult to evaluate for yourself (how am I doing as a manger) into something straightforward to focus on. In your interactions with your employees, energize them. Empower them.

I think we all know from personal experience how difficult it is when you have a manager that drains your energy. Where interacting with them sucks your enthusiasm for the job away. It’s not a good experience for either party.

Make sure you’re energizing & empowering your employees each and every time you interact with them.

(More on this from Bob Sutton here)

There’s been an increasing amount of conversation recently about how difficult it is for startups to find qualified engineers. And yet at the same time, recent graduates with strong engineering credentials are saying that they can’t find jobs.

What’s going on here?

(Let’s set aside those who aren’t looking to work for a startup. Nothing wrong with that – but the hiring process for corporations is governed more by the macroeconomic situation and waiting for executives to feel comfortable that the analysts on Wall Street are comfortable with increased hiring expenditures. I can’t provide much insight into that. But I can offer some thoughts for recent graduates who are chomping at the bit to get out there and put the skills that they’ve spent years honing to good use, and they’re hungry for any opportunity, especially at a startup where their contributions will actually move the needle.)

I think the immediate reaction is to try to attribute this apparently misalignment of supply and demand to some generalization that startups won’t hire recent graduates.

In December, Vivek Wadhwa wrote a post on TechCrunch, Shortage of Engineers or a Glut: No Simple Answer. The post made a lot of useful points and quoted some smart people (including a comment that I left on his blog).

After thinking about this again recently, I went back and read some of the comments on the post.

Most of the comments seemed to be along one of the following themes:

1) Schools don’t train me in what’s really being used in industry. All of the job requirements that I read ask for X years of experience with Y skill (that I don’t have). What is wrong with education?

2) Startups don’t just hire anyone. They are looking for engineers who have strong fundamental skills, can pick up new technologies and run with them, and are willing to tackle any problem.

There is truth in both of these themes.

School doesn’t teach me what I need to know for industry

Perhaps the Information Sciences and Technology program at Penn State was unique in this regard, but the Dean and professors would regularly get up in front of students and say something along the lines of, “Technology is an industry which requires lifelong learning. We will teach you how to learn. Then you can use that skill to learn what you need for your career.” Sure, we learned some useful things along with the theories, but it was always made clear to us that we shouldn’t expect the curriculum to teach us everything we would ever need for a career in technology.

Good thing, too. Because the vast majority of the skills that I use on a day-to-day basis in my COO/CTO role were things that I’ve learned myself.

One of the comments on the post spoke to this point: “I was shocked that so many of my classmates didn’t apply what they learned — about how to learn — to themselves. In the time between my CS degree and my first industry job, I’d taught myself version control, design patterns, unit tests, embedded systems programming, user interface design, and so on. It’s easy to get a good job when so many of my classmates thought that college was the last time they’d ever have to learn something!”

This is why so many companies look for candidates who contribute to Open Source. Because contributions aren’t something that you regularly do as a part of school or as a part of most jobs. Instead, it’s something you do on your own accord, and it demonstrates both initiative and the ability to pick up something and run with it.

Startups don’t hire just anyone

That’s true – they don’t. But it doesn’t mean that you aren’t already or can’t become the kind of person that they will hire.

I make this point in my comment on Vivek Wadhwa’s personal blog, that led to the subsequent TechCrunch article:

Just because engineers are graduating, or there are engineers on the bench with decades of experience, this doesn’t immediately solve the issue that a startup CTO faces in trying to staff their team. At the start, an entrepreneur is looking for a driven technologist who can come in and partner with them to translate their vision into reality. If you’re making hire #2 or beyond, you’re looking for specific needs – startups often don’t have the luxury to train someone into their technology stack and are looking for folks who can hit the ground running and get stuff done. As a result, simply having an engineering degree or decades of experience from BigCompany clearly does not make an engineer an immediate fit.

Try to find an engineer with the right experience and the ability to get stuff done — this is what entrepreneurs are struggling with and is what drives the entrepreneurs and investors to complain that there isn’t enough talent in the market.

Startups are looking to fulfill a need. There’s no leadership development program to train a pipeline of technologist for future needs three years away. Startups have a problem. They need to grow their product. Now. Faster. If you’re not the solution to that problem, you won’t get in.

So, become the solution.

Pick the area that you want to work in. Ruby on Rails web development? Great. Objective C iPhone development? Sounds good. Now start writing code. Show that you get stuff done (code on GitHub, contribute to open source, blog!). If you have experience already, find ways to make it relevant to the position that you’re looking for.

And to really get a gold star, don’t limit yourself to the one technology that you start with. If you’re doing Ruby on Rails development, it’s likely that you may need to do some basic systems administration on a Linux server to get your stack up and running. Embrace that opportunity as a chance to prove that you can do whatever it takes to get the job done. Same thing when you need to setup memcached or Redis or whatever additional technologies might come onto your radar. Because a startup will expect you to pickup new technologies, and to become productive rapidly.

Finally, you need to get in touch – I think this point is under-appreciated: startups don’t have big recruitment search budgets (or a lot of time) – if we can’t find you easily, we can’t hire you. Promote yourself. You are a brand. Figure out how to get found.

I sent some very similar advice to my summer development interns at the end of August. I closed with the comment that, if you put some time into really considering the skillets that you’ll need in the job that you want, and then really make an effort to build those skillets (and any supporting skillets on your own), and then do the extra work to publicize what you’re doing, you will be easily heads and shoulders above 90% of the other students you”re graduating with who could do the work, but won”t bother.

Much has been said about the Gen Y folks who are eager to get out there and start making a difference in the world. You don’t need anyone’s permission to go get started. What are you waiting for?

A security update for Ruby has been released to respond to multiple vulnerabilities. All versions of the interpreter are affected, including the Ruby Enterprise Edition interpreters.

Fortunately, it’s easy to upgrade with RVM:

Tell RVM to upgrade your specific REE installation to the latest:

rvm update                                                   #upgrade RVM
rvm upgrade ree-1.8.7-2011.02 ree-1.8.7-2010.02              #upgrade REE

Accept Y to all options to move elements of your current install.

Then reset REE to be your default:

rvm ree-1.8.7-2011.02
rvm --default ree-1.8.7-2011.02

Following this, confirm that the new install of Ruby is active for your shell session (which ruby). Then, you will likely need to install bundler again to make sure you have 1.0.10 for the upgraded copy of gem 1.5. Also, run bundle install from the app root to make sure your gemset is up to spec.

Despite telling the upgrade process to move everything, I still needed to reinstall Passenger (3.0.2) and reconfigure it for Apache:

gem install passenger
passenger-install-apache2-module

With this reinstallation, you’ll also need to edit your Apache config file (on Mac OS X: /etc/apache2/httpd.conf ) to point to the new Passenger gem.

Restart Apache from System Preferences > Sharing > Web Sharing and you should be good to go.

Supporting Internet Explorer for your web app is a pain. At least, trying to test your Ruby on Rails (or other localhost-based site) in IE while developing on the Mac should be easy, given the prevalence of virtual machines. Right?? You’d think. However, trying to setup our front-end developer to access a local instance of Passenger running on Mac OS X from a Windows VM proved to be more of an issue than it needed to be. Adding to this, most of the top Google results for relevant keywords for this configuration made the process more overly difficult than it needed to be.

So, to help ease your pain in testing your Mac-based Rails site from a Windows virtual machine, here is an easy configuration for the three main VMs.

We need to do two main things:

1. Trying to serve port-based or subdomain-based sites over to the Windows VM is going to be more challenging. Let’s make things easy – we’ll set the default http://localhost site for our Mac to our Rails site.
2. We need to get the VM’s network to communicate with the Mac’s network correctly so that we can access this site.

On my team, we’re using Passenger on the Mac (running on top of the Mac’s base install of Apache). We’re using Passenger Pane to configure Apache easily. We also have this hooked into RVM, but that won’t be relevant to what we’re doing here. For the purposes of this post, I’ll assume that you have Passenger successfully serving a site that you can access via http://somedomain.local I’ll also assume that for whichever VM you’re opting to use, you have Windows installed and running in it.

** Do yourself a favor. Make sure that VM Ware Tools, Parallel Tools, or VirtualBox Guest Additions are installed – this ensures that the network connectivity will work as expected.

Let’s get http://localhost serving our Rails site

At the bottom of /etc/apache2/httpd.conf when using Passenger Pane, you’ll find the following configuration:

<IfModule passenger_module>
  NameVirtualHost *:80
  <VirtualHost *:80>
    ServerName _default_
  </VirtualHost>
  Include /private/etc/apache2/passenger_pane_vhosts/*.conf
</IfModule>

To make our Passenger site the default localhost site, go into passenger_pane_vhosts/, view the Apache config file for the site you want to be the default, and you’ll see something along the lines of the following:

$ cat yoursite.local.vhost.conf
<VirtualHost *:80>
  ServerName yoursite.local
  DocumentRoot "/Users/username/rails/yoursite/public"
  RackEnv development
  <Directory "/Users/username/rails/yoursite/public">
    Order allow,deny
    Allow from all
  </Directory>
</VirtualHost>

Copy the lines from inside the VirtualHost block and paste them back into /etc/apache2/httpd.conf so that it looks like the following:

$ cd ..
$ sudo mate httpd.conf
# Added by the Passenger preference pane
# Make sure to include the Passenger configuration (the LoadModule,
# PassengerRoot, and PassengerRuby directives) before this section.
<IfModule passenger_module>
  NameVirtualHost *:80
  <VirtualHost *:80>
    ServerName _default_
	####
	DocumentRoot "/Users/username/rails/yoursite/public"
	RackEnv development
	<Directory "/Users/username/rails/yoursite/public">
	  Order allow,deny
	  Allow from all
	</Directory>
	####
  </VirtualHost>
  Include /private/etc/apache2/passenger_pane_vhosts/*.conf
</IfModule>

Restart Apache by going to System Preferences > Sharing and uncheck & recheck Web Sharing.

Also, if you’re using VMWare Fusion or Parallels, note the IP address that’s shown here.

Configuring VirtualBox

Let’s start with Virtual Box, which was the biggest pain to configure. Not because there’s a lot of work, but because they one critical detail doesn’t seem to be well known (most of the posts you’ll read online try to guide you through using the command-line network configuration tool). I should note that this was my first real opportunity to use Virtual Box, having only used Parallels and VMWare Fusion beforehand. It’s a little rough around the edges in a few places, but overall very impressive.

1) First, ensure that network settings for VirtualBox should be set to NAT – no port forwarding needed

2) Now, here’s what took a while to dig up: VirtualBox connects http://10.0.2.2 to the Mac’s localhost (thanks to this forum post) That is, just typing that into IE within the VM should connect you to http://localhost running on the Mac.

Configuring Parallels

1) For Parallels, you want to make sure that your VM is running on a Shared Network.

2) If you’re setup with this network configuration, you should be able to directly access the local IP address of your Mac, as shown in the System Preferences > Sharing screen:

Configuring VMWare Fusion

1) VMWare Fusion is much like Parallels. Configure your VM to “Share the Mac’s network connection” via NAT.

2) If you’re setup with this network configuration, you should be able to directly access the local IP address of your Mac, as shown in the System Preferences > Sharing screen, as with Parallels.

Hopefully, if you’ve made it to this point, for your respective VM, you have IE accessing your Passenger-based site. Success!

Hopefully, at least now you’ll be able to spend your time on all of the issues that poor users confined to IE are suffering through and get bug fixing quickly.

We switched from using MacPorts to Homebrew for package management on our Mac OS X development machines. Recently, when building two machines and using brew to install MySQL on them, we ran into an issue where MySQL would only partially install and we’d get a mysql database with an empty user table. This was with mysql-5.1.52:

mysql> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from user;
Empty set (0.00 sec)

We tried a couple of things to solve this, but were not able to come up with a workable way to insert a root user, as you can’t create users when running with the –skip-grant-tables option.

I also tried the binary download from MySQL.com and hit the same snag.

After a couple of rebuilds and testing with a fresh install of Snow Leopard, we traced the issue back to the following error message from the MySQL setup script:

$mysql_install_db
Installing MySQL system tables...
ERROR: 1049  Unknown database 'mysql'
101121 19:21:38 [ERROR] Aborting
101121 19:21:38 [Note] ./bin/mysqld: Shutdown complete
Installation of system tables failed!  Examine the logs in
./data for more information.
"¦

Error 1049 proved to be the key to solving this. It led me to this MySQL bug report, which discusses that this error can be generated when the default temporary directory doesn’t exist.

Passing the /tmp temporary directory to mysql_install_db as a param seems to resolve error 1049 and allow us to get the default mysql system database setup with a working root user.

Most of the instructions for installing MySQL with Homebrew don’t mention any additional params needed, which leads me to believe this is something recent in the recent builds of MySQL. We didn’t have this issue when installing MySQL on our other Snow Leopard machines (which have 5.1.49). However, this is the sequence of commands that we found that worked for us – hopefully, this will help others hitting similar snags:

brew install mysql
mysql_install_db --user=mysql --tmpdir=/tmp
mysql.server start
#server must be running first
mysql_secure_installation

Maybe you’ve run into this, too? You’re taking a look at the tables generated by your Rails migration and you see this really odd column.

#<ActiveRecord::ConnectionAdapters::TableDefinition:0x10245fd78>

Here’s another look:

Yuck!

Searching the web didn’t turn up much in this case, so I thought I’d post what I found when figuring out the root cause.

See the issue in this migration?

No? I didn’t the first time I looked, either.

It’s the comma – at the tail end of the last column definition before t.timestamps.

This little typo doesn’t look like much, but it’s the cause of that wacky column.

Here’s what the table looks like when you run the migration with the comma:

And here’s what it looks like when you fix your typo:

Ah! Much better.

As for creating a migration to fix this, I just resolved the typo and dropped the column manually. It’s a little difficult to script this because the column name will be different in each database, so, it’s some variation of this (the part in bold on each database that the original typo ran on):

ALTER TABLE `friendships` DROP `#<ActiveRecord::ConnectionAdapters::TableDefinition:0x10245fd78>`;

Hope this helps someone else avoid wasting time on this!

When I last posted in June, we were heading to Europe to continue growing the FanGamb product. I’ve taken a bit of a vacation from blogging while I’ve been focusing on all that’s been involved in this.

I moved to Dublin in July and we’ve been rapidly rebuilding our team and the product. I’ll be able to go into more detail on the changes when the product re-launches in the near future, but things are going well. We’re having a great time in Ireland and have really enjoyed working abroad.

Since I can’t yet delve into the details of the move and transition, or the next version of the product, I’ll be focusing for a while on what it’s like to working abroad, and also some topics that I haven’t given as much “airtime” to in the past. Most of my posts to this point have been on various startup/entrepreneurship topics, which I’ve enjoyed writing about. A large part of my job, though, as VP of Product & Technology for Three Screen Games, has me deeply involved with both the technical product development and customer-centric product management efforts. These are petty broad areas and I look forward to exploring them further through future blog posts.

And of course, an update to the seed stage accelerator list is overdue – it’s amazing how this space has continued to grow in the last few months.

Anyway, it’s good to be back.