Viktors Rotanovs

Understanding Amazon EC2 failures and redundancy

2011-09-22T00:00:00+02:00

I’ve been looking after several websites on Amazon EC2, and I’d like to share some thoughts about how Amazon EC2 can fail.

Shared servers and avoiding SPOF

Only two instance types run on dedicated servers: c1.xlarge and m2.4xlarge. Other instance types use shared servers.

If you start multiple instances for redundancy, there’s a good chance they will land on the same server, therefore not providing redundancy. For example, I’ve just tried launching eight instances, and three of them landed on the same physical server (as confirmed by traceroute).

Therefore, to avoid having a single point of failure, most likely you’ll need to start instances in more than one Availability Zone.

EBS durability

EBS does not provide real durability because of writeback caches. Some people claim that’s an advantage.

Replicating data to a second EBS volume in the same Availability Zone will not prevent disasters:

in case of data center failure multiple EBS volumes can get corrupted simultaneously,
there is no guarantee that two EBS volumes won’t land on the same physical hardware.

The only solution is to replicate data to another Availability Zone.

The devil is in the details

Having perfectly running system today does not guarantee you’ll be able to run it tomorrow.

In case of forced server shutdown by Amazon, hardware failure, or stopping and starting an EBS root instance, the following may happen:

Physical server may change. It’s possible to get different CPU models with different extensions (e.g. E5507 may change to E5430 or even to an Opteron)
Xen version may change. For example, 3.0.3-rc5-8.1.14.f may become 3.4.3-2.6.18.

There is no guarantee that your OS will support these changes.

Also, some Linux kernel bugs exist only when running on EC2.

Conclusion

Creating reliable systems on EC2 can be hard, because critical services may need to span multiple Availability Zones. While most of the time systems will run fine, extensive sysadmin skills may be required when things go wrong.

P.S.

Here are two common setup mistakes when you take factors above into account:

Hadoop HDFS with replication factor set to 2 for reliability
Database replication for reliability with all database servers in the same availability zone

Do you know more examples?

HBase Dump and Restore

2010-05-13T00:00:00+02:00

Cookbook-type snippets to export data out of HBase or import dumps into HBase.

Export a table from HBase into local filesystem:

bin/hbase org.apache.hadoop.hbase.mapreduce.Driver export \
table_name /local/path

Export a table from HBase into HDFS:

bin/hbase org.apache.hadoop.hbase.mapreduce.Driver export \
table_name hdfs://namenode/path

Import a table from a local dump into existing HBase table:

bin/hbase org.apache.hadoop.hbase.mapreduce.Driver import \
table_name /local/path

It’s a good idea to count and compare number of rows before exporting and after importing:

bin/hbase org.apache.hadoop.hbase.mapreduce.Driver \
rowcounter table_name

Number of rows is visible in Hadoop counter called ROWS, like in output below:

mapred.JobClient:   org.apache.hadoop.hbase.mapreduce.RowCounter$RowCounterMapper$Counters
mapred.JobClient:     ROWS=103821

Alternatively, you can use Hadoop Tool interface, but it may complain about missing classes if hadoop-env.sh is not configured properly. For example, when launched without arguments, it displays available options:

hadoop jar hbase-0.20.3.jar

An example program must be given as the first argument.
Valid program names are:
export: Write table data to HDFS.
hsf2sf: Bulk convert 0.19 HStoreFiles to 0.20 StoreFiles
import: Import data written by Export.
rowcounter: Count rows in HBase table

HBase dump is one ore more Hadoop SequenceFiles, you can inspect its contents with something like:

hadoop fs -fs local -text table_name/part-m-00000

Hitgeist.com rewritten using HAML and Compass

2009-06-12T00:00:00+02:00

Hitgeist, a ranking service for finding recently popular sites in each country, received a minor upgrade - now it uses several new libraries to make development of new features easier, and code cleaner and nicer.

I built Hitgeist using Merb, a Ruby framework which brings fun back into web development. In fact, the fun is taken by Symfony, which just doesn't feel elegant anymore. Merb also has less magic than Ruby on Rails, and is a lot better architected. RoR developers liked Merb so much they started to unify both frameworks.

So, what are the updates?

First, I rewrote all templates from ERB to HAML. I should have done this before. Code becomes so much cleaner—tags are closed automatically, and visual clutter is reduced:

ERB (before, 9 lines)

<div id="container">
  <div id="sitemap">
    <span class="letters">
      <% ('A' .. 'Z').each do |letter| %>
        <a href="/trends/<%=letter%>"><%=letter%></a>
      <% end %>
    </span>
  </div>
</div>

HAML (after, 5 lines)

#container
  #sitemap
    %span.letters
      - ('A' .. 'Z').each do |letter|
        %a{ :href => '/trends/' + letter }= letter

Second, I changed site structure to use grid system, a method to construct visually appealing page layouts using columns and rows which was used in newspapers for decades, but started getting its way to web only around 2005. Usually grids are implemented using one of CSS frameworks, which mostly solve cross-browser and printing issues and add niceties such as vertical rhythm and beautiful typography.

There are several CSS frameworks available to implement grids, most famous ones being Blueprint (which is used by sites like QuantCast), 960.gs, and Yahoo UI Grids. There are also others, like this high-quality small CSS framework.

I chose Blueprint because it’s complete and integrates nicely with Compass, a CSS meta-framework for making CSS shorter, much like HAML does for ERB. It also allows to do computations, for example:

.bright
  :font-color !main_color + #663300

Did these changes improve the website? Visit new version, and send your feedback.

PS: Do you know anything like HAML but for PHP?

Canon 50D, or Truth About DPReview

2008-12-18T00:00:00+01:00

I like DPReview. I thought after Amazon bought it it will become more corporate, but it looks it’s still as biased as it used to be:

Nikon D90 - described as photographer’s camera
Canon 50D - highly recommended, just</li>

Let’s get things straight. None of D90 features are very different to Canon 50D, but there are several quite important differences:

	Nikon D90	Canon 50D
Viewfinder magnification	0.94x	0.95x
Histogram in live view	None	RGB
Live view	Slow mode only (contrast-detect)	2 modes (contrast detect and phase detect)
Focus points	11 focus points (1 cross-type sensors)	9-point TTL CMOS sensor, all cross-type for lenses of F5.6 or faster
Lens correction	Chromatic aberrations	Vignetting, distortion and chromatic aberrations
Viewfinder dioptric adjustment	–2.0 to +1.0	-3.0 to +1.0

As you can see, these 2 cameras are not even in the same class, although the price difference is very small - around $250.

So, after weighting all pros and cons, I chose Canon 50D, and I must say it’s an excellent camera. Photographic experience is totally different than it used to be with my old Canon 350D.

Viewfinder probably as good as it can get with 1.6x crop factor
Super-sharp, super-bright 640x480 LCD - you can re-shoot bad photos without having to check them on a computer first
Live view for precise focusing - I couldn’t imagine the camera is still shaking a lot when my hands are rock sturdy after making “stabilization exhale”. Perhaps this technique works for old cameras, but 15MP requires to do things by the book: use a tripod, or a stabilized lens. Apparently IS lens are even more important for getting sharp picture than good glass.
Live view for black and white photography - all picture styles work also in live view. You can start training your eye for B/W without having to download to a computer.
Focusing is great - success rate a lot bigger now. New auto-focus system totally rocks.
Superb menu - very logical and easy to use now. Almost Apple of cameras.

Best Nature Photography 2007

2008-02-06T00:00:00+01:00

Some time ago I was asked if super-expensive ($5k+) lens make any sense for creative photography, given the fact that the most famous photographers of the past were using relatively simple equipment. Fortunately, NatureScapes.net has announced 2007 Images of the Year awards, and pictures speak better than words.

Have a look at the list of cameras and lenses used by winners:

Birds category: Canon EOS 20D + 500mm f/4; Canon EOS 1D MkII + 300mm f/2.8 L IS + 1.4X Teleconverter; Canon EOS 1D MkII + 500mm f/4 IS + 2X Teleconverter
Wildlife: Nikon D2x + micro Nikkor AF-S VR 105mm; Nikon D2x + 200-400mm f/4 VR; Canon EOS 1Ds MarkII + EF 500mm f/4.5L
Landscapes: Canon 1Ds MkII + Canon 300mm f/2.8; Canon EOS 20D + Canon 17-40mm; Canon EOS 40D + Canon 10-22mm
Flora & Macro: Canon 30D + Canon 100mm macro; Canon 40D + Tamron 180mm; Nikon D200 + Sigma 150mm f/2.8 macro
Elements of Humankind: Nikon D2x + 35-70mm; Canon EOS 5D + Canon 16-35mm f/2.8; Canon EOS 5D + 70-200mm f/4L + 1.4X Teleconverter

As you might have imagined, bird photography is indeed very expensive.

Facebook Copycat Becomes #3 in Russia

2008-01-13T00:00:00+01:00

Something strange is happening in Russia. For several years, social networking didn’t really pick up, although many have tried, and about one year ago new Russian Facebook clone appeared, and became #3 now.

The site itself doesn’t look special - design is almost except copy of digg at time vkontakte.ru was created:

All features are standard: friends, photos, videos, messages, notes (blog), groups, and events. There is support for Russian declinations, including gender-specific inflection with support for names, which is quite difficult to implement, for example: Друзья Наталии Искорки (Friends of Nataly Iskorka).

Former soviet republics are recognized, so perhaps website may become popular among Russian-speaking audience in Baltics or Central Asia, and 5.9% of its users already come from Ukraine.

Website owner is Pavel Durov, with user’s id of 1. I’m wondering who’s got id of zero.

To view pages of other users, you have to fill your profile to at least 30%. Perhaps that’s a contributing factor to site’s growth, in addition to quite good picture filtering - you won’t find Olsen Twins photos in profiles here, at least not at first sight. Updating profile photo may be not super-easy for users who got high-megapixel camera or don’t know the difference between BMP and JPG - upload is limited to 5 MB.

I’ve heard rumors that the site grew so quickly because creators took their hands on database of students, which is not impossible given current situation with intellectual property protection in Russia.

Many other Russian Web 2.0 sites copy ideas as well, just check the list at Hitgeist: Trends in Russia. In particular, RuTube, Odnoklassniki.ru and Smotri.com are worth mentioning.

Rising Web Sites 2008 – HitGeist.com Relaunch

2008-01-13T00:00:00+01:00

Today I’ve relaunched HitGeist.com, with fresh data from Alexa and much improved algorithm.

Visitor popularity trends have been not what TechCrunch usually writes about - there are many overseen websites which grow at astonishing rates. Many new companies emerge from around the world, for example, Russian vkontakte.ru is about to become #2 by daily unique visitors at local market, and Scandinavian countries have their own set of traffic-rich national players.

While sudden traffic surges are interesting, new HitGeist focuses on discovering websites which have shown steady growth, and assumes that when traffic is unstable, website is more likely to use non-natural means to attract traffic, such as advertising, PPC campaigns, spam, or spyware.

There are still some bugs to fix, but new stats were so breathtaking that I decided to share them as soon as possible.

Ruby 1.9 released

2007-12-26T00:00:00+01:00

Great Christmas gift from matz and other contributors: Ruby 1.9. Some highlights:

Around 2x performance increase in many cases, thanks to YARV-based code
Nice syntax enhancements, like (1..10).reduce(:+) # => 55
JavaScript-style literal hashes: {var: "value"}
Lots of other enhancements

Ruby 1.9 is a development release, and contains a lot of experimental features. For production, it’s recommended to continue using 1.8 branch.

jQuery-style Chainability in PHP, Using SPL

2007-10-29T00:00:00+01:00

What could be better than chainable operators in jQuery, .addClass("beautiful").show("slow")-style? Almost nothing, but adding those to PHP would make code cleaner and life easier.

At first sight, implementing these looks almost impossible, but after a quick look at extensions we notice that SPL could offer something similar, if combined with magic methods. Let’s implement ArrayAccess interface and see how it works:

<?php
$fruits = new Chain(array(
           new String('apple'),
           new String('mango'),
           new String('kiwi')
));

echo "First fruit: ";
echo $fruits[0];
echo "\n";

echo "All fruits: ";
echo $fruits->toUpperCase()->join();
echo "\n";
?>

Outputs:

First fruit: apple
All fruits: APPLE, MANGO, KIWI

Beautiful, eh?

Here’s complete source code:

<?php
class Chain implements ArrayAccess {
    private $items = array();

    public function __construct($items = array()) {
      if (!is_array($items)) return;
      $this->items = $items;
    }

    public function add($element) {
      $this->items[] = $element;
    }

    public function __call($method, $args) {
      if (count($this->items)
            && !method_exists($this->items[0], $method)) {
        throw new BadMethodCallException();
      }

      $returnChain = new Chain();
      foreach($this->items as $item) {
        $returnChain->add(call_user_func_array(array($item, $method), $args));
      }
      return $returnChain;
    }

    public function rewind() {
      reset($this->items);
    }

    public function current() {
      return current($this->items);
    }
   public function key() {
      return key($this->items);
    }

    public function next() {
      return next($this->items);
    }

    public function valid() {
      return $this->current() !== false;
    }

    public function offsetExists($offset) {
      return isset($this->items[$offset]);
    }

    public function offsetGet($offset) {
      return $this->items[$offset];
    }

    public function offsetSet($offset, $value) {
      return $this->items[$offset] = $value;
    }

    public function offsetUnset($offset) {
      unset($this->items[$offset]);
    }

    // convenience method
    public function join($separator = ', ') {
      return implode($this->items, $separator);
    }
}

class String{
    private $s;

    public function __construct($s) {
      $this->s = $s;
    }

    public function toUpperCase() {
      $this->s = strtoupper($this->s);
      return $this;
    }

    public function __toString() {
      return $this->s;
    }

}

$fruits = new Chain(array(
              new String('apple'),
              new String('mango'),
              new String('kiwi')
));

echo "First fruit: ";
echo $fruits[0];
echo "\n";

echo "All fruits: ";
echo $fruits->toUpperCase()->join();
echo "\n";
?>

Canon 40D vs Nikon D300

2007-10-29T00:00:00+01:00

What a difficult choice—both leaders just released excellent prosumer models. Both dSLRs offer top-notch features like self-cleaning sensor units and low-noise CMOS sensors, the latter being exclusive Canon selling point for several years. But how do they compare now in key characteristics, after Nikon catched up with the competition?

	Canon 40D	Nikon D300
Megapixels	10.1 MP	12.3 MP
Sensor	CMOS, sensitivity up to ISO 1600, 14-bit A/D converter	CMOS, sensitivity up to ISO 3200, 14-bit A/D converter
TTL Auto-Focus	9 cross-type sensors	15 cross-type sensors
Live View Auto-Focus	Yes, turns off LCD while focusing	Yes, contrast detect (like on P&S cameras)
Viewfinder	Pentaprism, 95% frame coverage, 0.95x magnification, diopter adjustment: -3.0 to +1.0	Pentaprism, 100% frame coverage, 0.94x magnification, diopter adjustment: -2.0 to +1.0
LCD Monitor	3.0", 320x240 pixels	3.0", 640x480 pixels
B&H Price	$1,299.95 (~900 EUR)	$1,799.95 (~1250 EUR)

For pixel peepers, neither of those are interesting—the real winner is Pentax K10D, which has very light anti-alias filter and offers the best per-pixel sharpness and unique non-processed, completely photographic 3D look—check out some examples. At about $700, with image stabilisation built in, this camera can become a great choice for photography geeks—imagine its potential after processing photos with DxO Optics v5.

Belgium: Brussels, Bruges

2007-09-16T00:00:00+02:00

It was very refreshing since it was my first trip outside of Catalonia since I arrived in Barcelona. After rather minimalistic Iberian airlines flight I arrived to BRU, and walked, and walked, and walked before I’ve got my luggage.

Old city was beautiful, with beautiful buildings dancin’ and singin’ at night.

View my Brussels & Bruges Photo Gallery

What I enjoyed:

Great food, especially mussels at Chez Léon, and some Asian cuisine in Bourse area
Beautiful old city, ceremony near Manneken Pis
Chinese pavilion (Japanese tower was closed for reconstruction works)
Comics on walls in many places

What I didn’t like so much:

Weather
Strange people on streets after 10 p.m.
Coffee!

Also, I must say Belgian chocolate is over-hyped. I’ve tried 6 different brands of chocolate, it’s all great, but not better than any other top-notch chocolate in other countries. My favorite is still Emila Gustava chocolate from Latvia.

Bruges was really nice, very Flemish, lots of nice people and almost everybody speaks good English. And, of course, there were canals and houseboats - what an idilly!

HITgeist – realzeit stats (gainers & losers by country)

2007-04-08T00:00:00+02:00

After being noted as an interesting project by Web 2.0 entrepreneurs and SEO specialists for several months, AlexaRadar becomes HITgeist today. New name demonstrates focus on international coverage and its European origin, as well as allows to show data from multiple providers of statistics.

ProgrammableWeb chooses AlexaRadar as a 5-star mashup

2006-10-06T00:00:00+02:00

Yesterday I submitted AlexaRadar to ProgrammableWeb.com, a largest collection of mashups that currently exists, and - surprise! - today AlexaRadar is not only a mashup of the day, but it also received 5-star rating from users of ProgrammableWeb.com. Thanks everyone who rated the website!

For those who aren't into mashups yet, ProgrammableWeb.com provides absolutely unique Web 2.0 Mashup Matrix - it definitely deserves a look.

Alexa Radar – find successful websites around the world

2006-09-30T00:00:00+02:00

I needed some real project to try Ruby on Rails, so here’s Alexa graph analyzer with some additional analytic features. It’s not finished yet, but already useful for finding what websites show steady growth in all countries supported by Alexa. Check out what’s popular in Estonia, what do they grow in Netherlands, and what do they eat in South Korea.

Zoom Magazine – Free Issue

2006-02-04T00:00:00+01:00

Dead or alive? I had the same question when browsing Zoom Magazine's website - last issue mentioned there was dated January 1, 2005.

The magazine features a lot of great works (you can see past issues free of charge after registration), so I was eager to know if magazine is still there and e-mailed them. They replied shortly thereafter that the magazine is still alive and I can receive free issue if I send them my postal address. So today is a great day - while checking my mailbox I noticed big pack there - surprise surprise it was super-printed, large-format Zoom Magazine January-February 2006 issue.

There was also a letter there offering subscriptions Zoom and five other magazines published by the same company from Milano, Italy - and the signature there was real (wow!), not one of these inkjet signatures you usually see on that kind of letters. Oddly enough the pack was sent from France, not from Italy. So, it’s nice Friday evening!

BetterPhoto.com Second Place, Again!

2005-10-24T00:00:00+02:00

Another e-mail from Heather Young, nice as always:

Congratulations! You have won Second Place in the Travel & Place category of this month's BetterPhoto photo contest!
Way to go! With over 20400 entries in the contest, it is a big honor to be selected among the winners.

They even make a press release for each winner, for consumption by local papers and radio stations. So, here’s my photo and other September contest winners.

BetterPhoto Finalist, Second Time in a Row!

2005-10-22T00:00:00+02:00

Another e-mail from BetterPhoto:

Congratulations... you have placed as a Finalist in this month's BetterPhoto Contest! Over 20900 photos were entered into the competition; it is an honor to have your image selected among the top 877 Finalist photos. Your Finalist photo: Catching Light to Shine at Night (#1253746)

See my photo or other finalists.

Best Ajax Example Out There

2005-10-18T00:00:00+02:00

ProtoPage.com is probably the most beatiful Ajax application to show in 5 minutes to your boss and colleagues. Another interesting ~~poem~~code: Fish Eye (be patient - it’s loading very slowly).

BetterPhoto Second Place!

2005-09-26T00:00:00+02:00

My first award in the photo contest! More than a year passed since photography became my hobby, and it finally happened!

My photograph Woman in Black was chosen as one of 10 second place winners in the category “Travel & Place”. During this month, tens of thousands of photos were submitted to the contest - 23400 to be exact, so it’s a big pleasure to be among the winners.

You’re welcome to see all the winners or just my photo.

BetterPhoto Finalist!

2005-09-25T00:00:00+02:00

Wow! What a day! Got this email:

Congratulations… you have placed as a Finalist in this month’s BetterPhoto Contest! Over 23900 photos were entered into the competition; it is an honor to have your image selected among the top 988 Finalist photos.

View The Photo

What’s interesting, this particular photo isn’t the best among photos I submitted to the contest last month (at least in my opinion) - see other contest entries.

Full text of e-mail follows:

Hi Viktors

Congratulations… you have placed as a Finalist in this month’s BetterPhoto Contest! Over 23900 photos were entered into the competition; it is an honor to have your image selected among the top 988 Finalist photos.

Your Finalist photo: Old Woman In Black (#1073943)

This image can be viewed - along with all of the Finalist images - at: http://www.betterphoto.com/gallery/dynoGall2.asp?catID=7679

During the next few days, the judges select the top winning photos. If your image is selected as a winner, you will be emailed again.

Again, congratulations on placing as a Finalist - keep up the great work!

Heather Young heather@betterphoto.com

Learn photography with Jim Miotke’s “BetterPhoto Guide to Digital Photography” And the award-winning DVD - “Digital Photography Unleashed” Get your autographed copies today at http://www.betterphoto.com

www.BetterPhoto.com The better way to learn photography!

Lighttpd vs Tux: nice() patch

2005-03-25T00:00:00+01:00

Lighttpd by default can be slower than TUX. Why?

Because TUX runs inside Linux kernel and has the highest possible priority, while lighttpd is just ordinary userspace process. Solution? Make lighttpd not so nice:

nice -20 /opt/lighttpd/sbin/lighttpd ...

Lighttpd’s Adaptive Spawning users will immediately notice a problem: all processes spawned by lighttpd will also run with the highest priority. That’s not good, since any PHP or other FastCGI process will be able to slow down lighttpd. Solution is quite simple: to restore back niceness level before launching a child process. This can be achieved by lighttpd nice patch.

Any other ideas how this could be done?

Lighttpd initgroups()

2005-03-22T00:00:00+01:00

While converting my website to lighttpd, I noticed that php processes launched by adaptive spawning can’t access certain files anymore. Small investigation has shown that the problem was caused by user httpd not joining groups specified in /etc/groups. Adding initgroups() call before calling setuid() helped.

lighttpd initgroups patch

Please let me know if you’ve experienced the same problem with spawn-fcgi program.

Web 2.0

2005-03-21T00:00:00+01:00

Bored by making websites? Not anymore. Almost overnight big players made 99% of the web obsolete. Less than 10 seconds page load time? 1 second? Forget about it. Everyone now wants INSTANT results.

Here are a few links for those who want to make their own Google Suggest, Gmail or Flickr:

Sajax: allows to call PHP functions from JavaScript
JavaScript O Lait: make XML-RPC and JSON-RPC (its own proprietary protocol) calls to server; some SVG examples too
DWR: Direct Web Remoting, for Java
JSRS: JavaScript Remoting, with PHP and Perl samples

Do you know other frameworks with similar functionality?

Lighttpd Config Includes

2005-02-16T00:00:00+01:00

Some of us techies have to maintain tens of webservers with different configurations. With Apache, you can easily throw in .htaccess for each website without having to change web server config. It’s not so simple with Lighttpd, that’s why I needed some script to join global configuration file with a local one. Some perl coding, and lighttpd config merge script is ready.

Installation and usage

Install Tie-IxHash perl module:
- Gentoo users: emerge Tie-IxHash
- Other distros: perl -MCPAN -eshell cpan> install Tie::IxHash
Backup your configuration
Call joincfg like follows, then start lighttpd: joincfg global.conf local.conf > new.conf lighttpd -f new.conf

As usual, NO WARRANTIES!

Lighttpd DoS Tool

2005-02-01T00:00:00+01:00

After seeing lighttpd grow to hundreds of megabytes I couldn’t sleep. It was like a tiny chicken has grown into a huge monster. After some experimenting I managed to grow monsters myself:

Lighttpd DoS Tool

The source of the problem is that lighttpd ignores maximum request size setting and reads data until it fills all available memory.

Return to Apache? No way - lighttpd bugs are usually fixed lightning fast.

HttpOnly in PHP (Anti-XSS)

2004-09-05T00:00:00+02:00

Wondering how to prevent JavaScript from stealing session cookie? Major browsers got the answer: just add HttpOnly to cookie to protect it from malicious JavaScript code. Full details are available at MSDN.

Since this option is well-supported (AFAIK MSIE, Mozilla/Firefox/Gecko and Konqueror support it) it’s unclear why PHP doesn’t support it yet. I’ve written a patch to add HttpOnly support to PHP. After applying the patch, add this string to your php.ini:

session.cookie_httponly = 1

Enjoy!