Facebook’s HipHop

HipHop takes normal PHP code and transforms it into C++, which is then compiled in machine code. Code compiled by HipHop runs significantly faster than normal PHP run with the Zend Engine.

It has its disadvantages though. HipHop is only capable of transforming a subset of the PHP language. Some important (and some may say evil) features, such as eval, have been removed to make it possible to transform from PHP to C++. It also has a few other limitations, such as only supporting PHP 5.2 and 64-bit operating systems.

IBM’s WebSphere sMash

WebSphere sMash is a Java implementation of a PHP runtime environment. The PHP code is compiled into Java bytecode which runs on the Java Virtual Machine. The major benefit of running things in Java is that it is portable to more operating systems and allows you to use things like database connection pooling.

While it does support more PHP functionality than HipHop, it is still missing some features. However, it is capable of running a few major open source packages out of the box, including SugarCRM, Mediawiki, and WordPress.

Caucho’s Quercus

Quercus is another Java implementation of a PHP runtime environment.It too allows you to use Java features like connection pooling, and supports major open source packages like MediaWiki and WordPress.

The paid version allows you to pre-compile your PHP scripts, which allows them to run super-fast. Unfortunately, this is cost-prohibitive for users that have more than 1 CPU in their server (like me) or plan on having more than 1 CPU in the future (most anyone with a vaguely popular website).

Phalanger

Phalanger is a strange creature. It is sort of like PHP.NET. It takes your normal PHP code and compiles it into MSIL, which can then be run by .NET or Mono. The major advantage to using Phalanger is that it makes it possible to access .NET classes in PHP (for example, VB.NET or C#).

Thanks to Mono, .NET isn’t limited to Windows. This means you can write Gtk applications in PHP. I hope to have something released in the near future that uses Phalanger.

So I searched the internet and discovered I could run git fsck –full to figure out what the problem was:

$ git fsck --full
broken link from    tree e79ab5efffca93a784afdb9cef73801eb1fa9db4
              to    tree fb9d707c76dde3f8cb672d1c07ca046615175587
dangling tree 1003e43ee3725b0482193aea9d82ca527c2903cc
dangling blob 361211b552b7910299e63b6627944ff72b0577eb
dangling blob 8e79df8dbe87431fbec8f25a4c6527fabe4d9e6f
dangling blob 9b3a891e5094920f743c36a49d1647d9aad2b2f0
missing tree fb9d707c76dde3f8cb672d1c07ca046615175587

After digging around the internet, I discovered that my repo was missing a tree, and that I could get it out of a cloned copy of the repo, assuming the cloned copy wasn’t corrupt, too.

I created a new empty repo:

$ mkdir new-repo
$ cd new-repo
$ git init

Next, I copied my pack file from the good repo to the new empty repo (there was only one in my pack folder):

$ cd /www/path/to/good/repo
$ cp .git/objects/pack/pack-c376dc3b9d2ce5f4d9ab9269d29ddc82f6f6a822.pack /path/to/new-repo

I unpacked the pack file to see if I could find the missing tree:

$ cd /path/to/new-repo
$ git unpack-objects < pack-c376dc3b9d2ce5f4d9ab9269d29ddc82f6f6a822.pack

My missing tree is fb9d707c76dde3f8cb672d1c07ca046615175587, so I looked to see if my file was there:

$ cd .git/objects/fb/
$ ls -l

9d707c76dde3f8cb672d1c07ca046615175587 showed up in the list, so yay! My missing tree was there! Now to copy it to the corrupt repo (my corrupt repo was a bare repo, so no .git in the path):

$ mkdir -p /path/to/corrupt/repo/objects/fb
$ cp /path/to/new-repo/.git/objects/fb/9d707c76dde3f8cb672d1c07ca046615175587 /path/to/corrupt/repo/objects/fb/9d707c76dde3f8cb672d1c07ca046615175587

That was it for me! I ran git gc and everything worked without a problem.

Hopefully this helps someone, or at least will help me remember what the heck to do next time my repo gets corrupted.

This short Perl script (less than 350 lines of actual code) is capable of turning your lowly desktop computer into a server killing monster. Traditional denial of service attacks use several clients (hundreds, sometimes thousands) to overwhelm the target server. The clients make as many requests as they can from the target server, causing it to use all its resources responding to the requests. Slowloris has a different approach.

Instead of using hundreds of clients, a Slowloris attack can often be successful run from a single client. And instead of overloading the server and utilizing a pile of bandwidth, Slowloris leaves the load on the target server at near zero and uses almost no bandwidth. It still makes a pile of requests, but it makes .…. them .…. nice .…. and .…. slow. Most web servers are only capable of handling a certain number of requests at a time (say, 150) so if you start 150 requests to the target server and leave those requests open for 10, 20, or even 30 minutes, you’ve effectively made it impossible for anyone else to get a valid request through to the web server. The target server isn’t actually doing any work, so the load doesn’t go up. What makes it even more annoying is the requests don’t show up in the log until they fail, if they ever do, which makes it more difficult to figure out what the heck is going on.

There are ways to mitigate the effectiveness of this type of attack, the most common being the use of a proxy to sit between the client and the web server (like haproxy or CloudFlare) and/or installing a bunch of Apache modules. Fortunately I’m doing both so hopefully I’ll be fine if someone decides they don’t like me much and wants to take down my server.

As a final note, I wholeheartedly discourage you from trying out the software unless you only do it on your local network. Attacking someone’s server is probably illegal and is likely to get your internet service shutoff. In other words, don’t be stupid.

Enter meld, an awesomely awesome GUI-fied tool for Linux that not only lets you compare files, it also lets you compare directories full of files and three files at the same time. Very cool. It is also in the Ubunutu repos which is a plus.

In less time than it would take to interpret a diff on a single file, I quickly saw which files only existed in the original directory, which only existed in the new directory, and which were in both but different. The diff for each file was only a double-click away.

Pretty cool app. If you are on an Ubunutu based system check it out by running: sudo apt-get install meld

So what is a zone file, you ask? Basically this file keeps track of how to access every .com domain name in the entire world. Well, technically not all of them, just the ones that have name servers associated with them, but practically all of them. So how many .com’s are there in the zone file? Over 88 million! Holy cow that is a lot of domains!

So the first problem is how to use the zone file. It is a 6.5 GB text file so you can’t just open it up and say “hey is jacoballred.com taken?” and expect a quick reply. On top of that, it isn’t even designed to give you a list of taken domain names, that is just a happy side effect of keeping track of how to access all the .com’s in the world.

My solution was to preprocess the data using a few Linux utilities, then load it into MySQL.

The zone file looks a little like this:

 NS E.GTLD-SERVERS.NET.
 NS M.GTLD-SERVERS.NET.
$TTL 172800
ENERCONTECHNOLOGIES NS NS1.BIZ.RR
ENERCONTECHNOLOGIES NS NS2.BIZ.RR
SELF-DRIVE-CAR-RENTAL NS NS3.IZP

None of the domains in the file have .com on the end. Each of these lists a nameserver after it. There are also non-TLD domains (nameservers) that I don’t care about, and other random markers in the file ($TTL). All I want are the domain names, so I use Linux to strip out the stuff I don’t want:

sed -e '/^[^A-Z0-9]/d' -e '/^$/d' -e 's/ .*$//' -e /[^A-Z0-9\-]/d com.zone \
| sort -u \
| awk -F "" '{close(f);f=$1}{print > "com.zone.split."f}'

Wow doesn’t that look fun? So lets go over it.

That first line uses sed to load in the zone file (com.zone) and remove all lines that don’t start with A-Z or 0–9 (the only valid characters for the first character of a .com domain), then it removes blank lines, then it removes all but the first word on each line (gets rid of the nameserver after the domain name), and finally removes any line that has characters that aren’t allowed in a domain (anything other than A-Z, 0–9, or a dash). This gets a list of JUST the domain names (without the .com), but has duplicates and they aren’t in any particular order.

The next line sorts the list of domains and removes duplicates.

The last line uses awk to split the list of domains into 36 separate files, one for each starting character (A-Z, 0–9). This isn’t technically needed but makes things more convenient.

My server is pretty wussy (1GB of RAM) so I’m preprocessing on my fast 8GB of RAM desktop at home. So I kick off that command and 20 minutes later I have files ready to be loaded into MySQL.

My table structure is pretty basic. I have 1 table for each letter/number (for performance) that has a numeric primary key and a varchar for the domain name. So I run this for each letter and number:

DROP TABLE IF EXISTS `com_A`;

CREATE TABLE `com_zone`.`com_A` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY ,
    `name` VARCHAR( 255 ) NOT NULL
) ENGINE = MYISAM CHARACTER SET utf8 COLLATE utf8_general_ci;

Next I use LOAD DATA INFILE to quickly pull in the data:

LOAD DATA INFILE '/path/to/file/com.zone.split.A' INTO TABLE `com_A` (name);

This step took about 5 minutes total for all the processed files. It is super super fast, but we still have one step left. Without an index on the name field, queries are really slow (about 2 seconds for a single domain). So we add an index to each table:

ALTER TABLE `com_A` ADD UNIQUE `name` ( `name` ( 255 ) );

This step was painfully slow, about 40 minutes, but once it was done I could do pretty much any query in a fraction of a second.

The final step was to turn off MySQL on my desktop, copy the MyISAM files to my server, then restart MySQL on my server so it could use them. Woot! I know have nearly every .com in the world on my server, ready to tell any web app I want if a domain is available or not with a high degree of confidence. I have a couple really fun webpages in the works that will use this.

Well that was a bit of a ramble but should be enough to get someone else in my position on the road to domain goodness!

So, not really understanding exactly what it was, I signed up and configured it for the Fake Name Generator.

After some poking around and letting it do its thing for a few days, this is what I’ve discovered it does for me:

• Provides a free DNS management. This is included for free with many registrars, but it just so happened that the domain I’m trying this out on didn’t come with DNS management so I’ve been paying $10 a year for it. This alone makes CloudFlare worth using for me.
• Serves my content on a CDN-like intrastructure. This makes my site faster to some users, which is always a good thing.
• Caches my static content (like images and JavaScript). This dramatically reduces my server load, and makes my site faster. My LAMP server with only 1GB of RAM is currently serving about 100,000 pageviews per day and running millions of queries in offline processes. With CloudFlare, my load average rarely goes above 0.10.
• Blocks bad guys. This is a huge deal for me. Everyone and their mom thinks it is okay to scrape my site for data. Bots love to hit my site to try to find exploits. CloudFlare does a great job at identifying these people and blocking them for me, or providing a way for them to enter a captcha to prove they aren’t a bot.
• Provides geolocation data on all visitors. I haven’t started using this yet, but CloudFlare adds a request header with the visitor’s geographic location. This makes it easier to target content to visitors from certain parts of the world.
• Makes me more profitable. All around, CloudFlare has made my business more profitable. My site requires less server resources, which means I can keep my site running on my relatively cheap tiny server. Fewer bots are loading my ads, which means my click thru rates are higher, which means I get paid more. My pages respond faster, which means I’m ranked higher in the search engines, which means I get more visitors.

One problem I ran into, however, is occasionally a screen scraper gets through their blocks and starts hitting my site. In the past I would use iptables to block them, but the way CloudFlare works makes that impossible (at least with my limited knowledge of iptables). CloudFlare provides a way to block a specific IP, but it can take several minutes to go into effect.

The solution I came up with is to use Apache to give visitor’s from the offending IP a 403 error:

<VirtualHost *>

 SetEnvIf CF-Connecting-IP 98.17.241.185 GoAway=1

 <Directory "/path/to/your/website">
 Order allow,deny
 Allow from all
 Deny from env=GoAway
 </Directory>

</VirtualHost>

This snippet, properly placed in the Apache config file, will cause Apache to look at a header set by CloudFlare, and if it matches the offending IP (in this case 98.17.241.185), it denies access to the site. You can add a nearly unlimited number of SetEnvIf statements to block any number of IPs.

Anyways, if you get an invite to CloudFlare, check it out! It is definitely worth it!

Disappointingly, I couldn’t find anybody that was scraping my site, which means I had to dig deeper. The next step was to use top to figure out what processes are stealing all my resources. To my surprise (and exceedingly great alarm) I saw that there were about a dozen sshd processes running. For those that are not Linux server savvy, there should not be about a dozen sshd processes running.

SSH is the protocol that Linux server admins use to connect to their servers. When connecting, an sshd process will run. When a dozen are showing up, that means a dozen people are connected or trying to connect, which is very very disturbing for a server like mine where I’m the only one that should ever be on it.

I quickly turned to the logs and found thousands of failed login attempts. Someone was trying to hack my box. Yikes!

I quickly used iptables to block the most flagrantly offending IP, but I knew that wouldn’t hold back a committed attacker. Enter my hero: DenyHosts!

DenyHosts is a free chunk of code written in Python that periodically scans your log files, determines if someone looks like they are trying to break in, and blocks them. If you are really paranoid then you can even have it talk to other servers to find out who is trying to hack them, so you can preemptively block the bad guys.

Installation and configuration literally took about 3 minutes, and is even easier to setup if you are using Ubuntu or Linux Mint because it is in the repos. As soon as I started it all the bad guys were blocked and my load averages started to drop. I highly recommend it for anyone that administers Linux servers.

I’ve been wanting to make a Firefox add-on FOREVER, but have never found something worth making that hasn’t already been made (or isn’t way beyond my abilities to make). So I decided I’d start simple and make a search provider add-on for ABA Number Lookup.

I found a page in the Mozilla developer wiki that explains how to make an OpenSearch plugin. It is super easy. Basically it is just a snippet of XML that tells the browser where to send queries to, and how to get search suggestions. It also contains basic information like the name of the search engine and who wrote it. Easy stuff.

It took about 30 minutes to modify ABA Number Lookup to be able to return search suggestions and to send the proper MIME header for the OpenSearch XML file. After that was done, I went to the “submit an addon” page, followed the instructions, and wham! My search engine add-on is available to the world!

Mozilla puts all new add-ons in a sandbox to make it a little harder to push malicious code. This is just a search add-on so I think it should pretty quickly get approved and the download page will look a little less scary.

Anyways, it was fun! I might write one for Rhyta Arcade just for fun, though I doubt anyone would actually use that one.

Google has this awesome alert service that lets you set up search queries and have new search results from them emailed to you automatically. For example, I have “jacob allred” as a query, so anything new with my name in it pops up in my email.

On Saturday night, I got an alert email from Google for “fake name generator”. Turns out somebody made a Fake Name Generator Android application and put it in the Android Market. “Well that is strange,” I thought to myself. “I don’t remember giving anyone permission to steal my content.” The guy stole my product name, logo, look and feel of my site, layout of my site, and my content. And to make it worse, his app has an ad on it so he is making money off my product while I’m stuck paying the bill for the server that powers it.

I decided to try to be a good person first, and try to contact the developer and see if we can work something out. Turns out the Market doesn’t really require developers to provide contact information. Lame. I managed to find an email address but I’m not sure if the developer actually reads it. I sent an email to him letting him know I’m not thrilled about his app and that I’d prefer to work out a way for him to continue using my services (using the API instead of screen scraping) in a way where we can both get paid.

Next step is to try to notify Google of the copyright infringement. Google has detailed instructions on how to file a Digital Millennium Copyright Act infringement notification, but it requires either faxing (I don’t have a fax) or mailing (which takes days). Neither is a great option.

My last option is to try to break the app. This requires downloading the app and then viewing my logs to see what it looks like when it scrapes my site. Should be easy, right? Not exactly. I don’t have an Android phone and apparently the Market isn’t available on the emulator. Lame.

Luckily, with much searching, I found this blog post that explains how to get the Market working in the emulator (albeit with an older API version). I ended up having to download the 1.6 image he posted on 4shared.com, but at least it works. Once I got the Market working, I quickly downloaded the app and broke it. That should get the developer’s attention. ;)

Anyways, if you are an Android, iPhone, or Palm app developer and want to make a legitimate fully-sanctioned application, shoot me an email. I’m interested in paying someone to write apps for all of those platforms, or in working out a revenue sharing option.

Cisco VPN settings

For months I have been trying to get a Cisco VPN connection working with the NetworkManager applet in Ubuntu with no success.

I’ve had to resort to using a Bash script that connects using the vpnc command line utility, and then using sed to manipulate the /etc/resolv.conf file so I can use both my name servers and the VPN name servers. This works okay, but things get wonky several times a day forcing me to disconnect/reconnect.

Well, I finally got it working the right way. How did I do it? Sorry, not really sure. It definitely had something to do with the gnome keyring though.

This may be what made it finally work, but no guarantees:

• Make sure you have the required software (some of this may not be needed, but it won’t hurt anything to have it installed): sudo apt-get install network-manager-vpnc network-manager-vpnc-gnome vpnc
• Open your VPN connection settings (right click on NetworkManager icon, click Edit Connections, click VPN tab, click your VPN connection then click Edit). Check the “Available to all users” box at the bottom, then click Apply. Close all the NetworkManager windows. Go back to your VPN connection settings, and uncheck the “Available to all users” box and click Apply. Try connecting to your VPN.