Coding Aloud

Tony Robbins And A Gun To Your Head

2009-08-29T15:05:00.004+03:00

Frank Kern is my hero. If I were to line him up with the rest of the Übermensch, he'd be standing there between Batman and Bill Hicks, waving back. There are only a handful of people who can speak and behave like Kern and still get away with it. He's the surfer boy you just can't hate.

I've been following Frank's work and talks over the years. Unlike his other colleagues in the make-money-online niche, his presentations are informative, helpful and really hilarious.

A few days ago I bookmarked a link to Tony Robbins' interview with Frank Kern and John Reese. I've heard of Tony Robbins' years ago, but never actually watched any lecture or listened to his talks. I must say (and he admits,) that his look is misleading. The guy is super inspirational, great motivational talker and very down to earth.

Kern and Reese drove to Robbins' hotel, looking for an advice. The let-me-help-you-do-better niche is crumbling from the inside. I've heard it so many times that it so obvious: people buy self-help products, but never actually go through. It's proven that just the mere act of purchasing such an item makes you fell warm and fuzzy that you think you are half way there already. I don't know anybody who bought a Idiot's Guide to Piano and is playing well just because it's collecting dust on the shelf.

My point is, during the talk both Reese and Kern agree that they started out making money online because they had guns to their heads. They were poor and needed the money. From there to where there are now is a very long story, but it would never begin if they had proper day jobs and security.

Most of my friends are making money online. Not enough to retire. It's a dream many people have, and it doesn't require relocating to Hollywood and look for acting gigs. But I think what we all miss is the gun to our heads. The point of make or break. If any of us doesn't work for a year, we'll still be fine. It won't be fun or easy, but I think we will get along. Comfort-zone has become a curse word. I intend to fight back.

As this blog is about my journey in my online business, I would like to share some of the ideas I use to try and break my comfort zone (CZ). As stupid as it sounds, it feels great. Go through the list, if you think an idea is idiotic, then it means you are cozy there in your bubble. Try to burst it.

* Live in another apartment/city/house for a month+. Bring only one backpack of stuff. Don't go living on the streets, but realize that you are not your stuff. You are not your guitar or your Playstation 3. I travel a lot in the world, and I feel great satisfaction that everything I need I can carry on my back.

* Go broke. Don't give away your money, but try to see how it feels without a credit card. Take very little cash and try to live with that for a week. Suddenly shelling out $20 for a dinner looks strange; why did you ever do it in the first place?

* Change your working environment completely: if you are working from home, try and work in a cubicle for a couple of works. If you are working from an office, try and work from home/cafe. How do you handle the new surrounding? Would you switch if you could?

Again, the interview can be seen at Tony Robbins Blog. I highly recommend it, and it's only 30 minutes long.

5 Tips For Boosting Up Productivity or: How Cheezburgers Killed The Productivity Star

2009-07-29T21:00:00.000+03:00

Productivity is the name of the game and I am losing. Back in the day, when I started Dumpr, I recall developing the habit of OCD by "tail -f"'ing my access log and just waiting for a visitor to come by. If you build it, they will come. Fast-forward a couple of years: I check icanhasdoublecheezburger.com for new lolcats 27 times a day, refresh my inbox 53 times an hour and just hang around tweetdeck, flickr, mobwars and youtube enough to rebuild the pyramids myself.

No more!

Starting ~~today~~ tomorrow, I am limiting my hours and planning my time as I see fit. Earlier this year, I did an experiment: using Google's SMS service, I configured Google Calendar to txt me every day at 7pm a single sentence: "Shut down the computer". I have to admit that it worked pretty well. My life evolves around computers. I've been sitting in front of a crt since the age of 6. Forcing myself to not use a computer (not even for watching movies) made other activities more available. Watch a movie with friends, read a book, spend hours on the guitar, learn a new language, or the best of all: go to sleep early!

This post is turning into a Seinfeld's Post About Nothing kind of deal. So I'll make it educational. Here are 5 tips guaranteed to increase your productivity. Learned the hard way:

* Define Productivity - what is that you want to achieve? YouTubing 18 hours a day is considered productive if your goal is to watch each and every movie on the internets. My goals in life are learning Japanese and practicing guitar enough to be able to perform in front of friends.

* Make Daily Timetables. Each morning, sit down for 5-10 minutes and write your schedule for today. I find it very difficult to plan a whole week ahead, so I'm always planning the night before. Allocate in granularity of 30-minutes, to allow tasks to be executed in a worry-free zone. In other words: when you know you have 30 minutes to complete a task, there is nothing to worry about missing other opportunities; don't waste brain cycles worries, you have planned your day already.

* Leave Meetups For After Work - business at day, pleasure at night. Fun is harder to cap in hours. What if your breakfast buddy is very interesting today, will you part early or postpone all the other tasks? Plan meetups, dinners, beers after workhours.

* Cheezburgers Before The Zone - getting into the productivity zone is difficult. There are plenty of interruptions: blogs, emails, twitter. Part of my day includes a visit to I Can Has Double Cheezburger for some lolcats to brighten my day. I won't fight this, so I might as well accept it. If I finish my emails/funny-cats early, then I don't need to stop my work later to check up on them. Pleasure before business, I always say.

* Make Daily/Weekly Audits - check up on yourself, sit for a few minutes to review how your week worked out. What did you do right, what would you like to improve? Don't punish yourself and feel down if something is out of order. Just be honest with yourself.

Got tips that work for you? Care to share?

I murdered the Fail Whale! by nickbilton

Вы запах смешно - The Pain of Website Internationalization

2009-01-24T17:14:00.000+02:00

Let's face it, internationalization is a pain in the ass. Zero fun. Often termed i18n and l10n, these two are God's punishment for trying to reach for the heavens.

I have recently started an effort of translating one of my sites http://www.dumpr.net to a range of languages, while maintaining a clean and loose code that can be later be put in use for other projects.

My plan is to make a 3-parts post. First, I will talk about what are my goals from the translation system. In a later post, we will review how technically other websites managed to both internationalizate and keep sanity. Third and last post will conclude: pitfalls you should know about, pains I've been going through and some tips for a better life.

As a rule of thumb, when I face with such a tremendous problem, I first head out and seek who has already achieved this goal. While there are plenty of websites that manage to translate several keywords a page (zooomr, photofunia,) there are two distinct websites I look up to: flickr und facebook.

I consider Flickr as the mother of all website architectures. In his bible book: "Building Scalable Websites", Cal Henderson explains lots of the architectural redesign and growing pains they had with Flickr. Nowadays Flickr supports a wide range of 8 languages, all left-to-right.

Facebook, on the other hand, has always impressed me with their amazing development framework and ability to adapt fast. They make sure their effort can then be helpful for other programmers. Their Translation application is so powerful and easy to use, that 20,000 people from France localized the entire website within 24 hours. (wow!) Also, Facebook supports right-to-left languages as well (ie, hebrew and arabic,) which also affect the website layout (completely mirrored.)

Drilling down, the following bullet points make up the bigger picture:

* Translating is done online. I have tried to maintain a csv or a google document for translating, it's just unbearable. Being a man-in-the-middle and transferring strings back and forth to translators is absolutely a no-no and can result in a brain damage.

* Strings list generated automatically. The gettext approach demands that the person in charge updates a list of translatable strings. Rapid development will be hurt and wounded if I have to update the same string in two places.

* No keywords or tagging, see what you are translating. It's hard to write and design an html with placeholders and copywriting is strictly impossible.

* No translation goes online without approval. Some authorization and permission-levels are required. Whether you are crowd-translating or using a hired professional, you still can't trust every person you meet on the internet. Facebook breaks this problem into finding the most fit translation (via voting,) and publishing the selected translation online.

* Person translating might not know html. Strings should not contain any markup text. Don't rely a 3rd party not to break your html compatibility.

* Rapid-translation (tm). Translators must find this task easy and fun. They should be able to see results in real-time, and not be dependent on anybody else.

Have you ever made your website appeal to other locales? Got tips to share? What problems did you face? I want to hear all about them! efterlade en kommentar!

Tower of Babel by fimoculous (cc).

Get Your Gig Going: Social Media For Entrepreneurs

2009-01-16T12:36:00.010+02:00

eClub HUJI had a great speaker last night. Kfir Pravda of Pravda Media presented a kickass lecture titled "Social Media For Entrepreneurs".

Pravda laid out a list of dos and don'ts for using the latest buzz bingo term "Social Media" for your own gain. From using the available services to network with people in your field, through being accepted as an expert in the subject. This media can make or break your business, and it's very important for us entrepreneurs (and more importantly, solo-entrepreneurs) to learn how to utilize this for our needs.

The talk was taped, thanks to our friends at israelhightech.tv. I will post a link to the video when it's up online and narrated in English.

So what have I learned from this. Have you ever been in a situation when somebody tells you something, and it seems so goddamn obvious to you, and yet, you haven't done it yourself? This is how it felt. There is some kind of enlightenment when a respectable person not only reawaken your dead cells, but also gives proofs (again and again) and real life examples how these tricks have changed his business and life.

1. A Business Card Is Just The Beginning of a Relationship
More often than not, I'd visit a conference or a lecture, network with a bunch of interesting people, exchange cards and then shelf them for a time in need. Now, why would you do that? A person gave you his business card, she has done so because she is interested in hearing more from you. Go home now, take all of your business cards, and add each one to your Facebook friends, LinkedIn connections and Twitter friends. Cause you know what? A week after meeting her, she had probably forgotten and there goes a valuable contact.

And here is an example from my life: a while ago I was at the Amazon AWS Meetup In Tel Aviv. After the meetup, I started talking to this guy. Now, I consider myself a great listener, but I just couldn't get the guy to talk much. Eventually we parted after exchanging business cards. That same night, he added me on Facebook and LinkedIn.

A while later, I attended the Facebook Developer Garage meetup, and there he was again. Just this time, he was accompanied with other important people from the field. Having me as his friend on these platforms, he checked my cv, updated on what I'm doing and realized the potential. He introduced me (now he did all the talking) to his partners in such an enthusiastic manner, that I was blushing immediately. Now I'm friends with important contacts closer to what I am doing.

2. LinkedIn Answers Is Your Greatest Tool
LinkedIn Answers has been there for quite a while. From time to time, I'd get these questions on my email. Basically, it's a platform for LinkedIn users to ask their peers a question. The question is published publicly, and other people can participate. I always answer the questions that I receive.

After Kfir's talk (and again, proving how important it is for his business,) I decided to give it a try. I asked a question in Startups And Small Businesses category. I have to admit I was stunned. It took 45 seconds for the first answer to arrive, and from a person I have never known before. It has been less than 12 hours, than I already got 15 different replies. It is really an amazing tool, not only to get help with decisions, but also to find people who are interesting and influential in your field of expertise.

3. Get Your Story Straight, Kiddo
There is a distinct different between, "what's your story" and "what's your job". I have learned this the hard way. If you were to ask me 5 years ago, who are you, I would reply "Gil Megidish: Programmer". Ask me the same question today, and I would tell you "I'm an entrepreneur filled with motivation and creative ideas. I am a proud owner of dumpr.net, a hobby that turned into a successful business. I am very open about my work, I share my knowledge with others and help other startups get on their feet".

With all modesty, I must say that the later answer invokes more questions and interested.

Kfir lectures quite often (and all over the world), so make sure you follow him on twitter and maybe attend his upcoming talks.

A Programmers Guide to Outsourcing

2008-12-13T18:19:00.010+02:00

Outsourcing needs no introduction -- it has been covered by news media extensively for the past couple of years. Everybody's doing it, and at some point in life, you figured it's worth a try.

But it's not all shine and glamor. Outsourcing your programming tasks can be a real headache and even a burden. This guide is here to help you identify the pitfalls before you're even making the first step. It's all based on experience and I'm sure many of the readers will agree what what is said here.

There are two types of outsourcing: one-shot (projects) and continual (semi-hired programmer.) Projects are posted on sites such as Elance, where freelancers send you quote for your consideration. Semi-hiring programmers can be done through oDesk, where you get to interview programmers and share svn access.

1. It might take 3 hours to explain a 10 minutes task
When you figure it would take you several hours to explain, listen, pack sources and then merge back the results, ask yourself this question: is it worth it?. When you're outsourcing a single task with no further interaction with that specific programmer, then the answer is simply: no. The only reason that you would do such a thing, is to educate your programmer (semi-hiring freelancers.) Add task, send bug reports, review code, integrate. The more you iterate this, the shorter it will become, so don't lose your head prematurely.

2. It's mentally difficult if you can do it yourself
The person who knows my framework and setup best is me. Moreover, I love my coding conventions, my automatic tests and how well my new code fits with the existing code base. It's awfully hard to look at somebody else's code and not go "I would have done this better." I'm sure you would. The easiest tasks I have outsourced are translating and copywriting. Why? Because I know I'm weak there and I can easily assess other's work. Learn to accept other people's code or shape them as needed.

3. Language barrier is no fun
Not everybody speaks English, and not everybody speak the same English. During my trial-and-error period, I have hired East Asians (India and China) programmers. I know I'm making a bad generalization here, but all of my projects failed. I received code that was either useless or had absolutely nothing to do with my requirements. Bouncing emails back and forth for a long time concluded this as hopeless and as awful as it sounds, I choose not to work with programmers from these countries again. It goes without saying that the same problem might happen with people from other nations as well. It helps to talk to the person, ask her questions about the weather or sports, just to get an idea how well you two communicate.

Here are a few tips I have picked up along the way. They help me calm down and do this process (more) smoothly:

1. When posting a project, make sure the price offers are sealed (hidden from the eyes of other freelancers.) I have noticed the second bidder base their price by the first bidder. Sealing it will make freelancers think and bid as they see fit, without the competition against other programmers.

2. Start small. It's easy to get hooked with the infinite opportunities. Don't shell out more than you can afford, and don't handle more than one task at the time. Use the first few projects for learning from mistakes. They will happen, there is no way around it.

3. Remember why you are outsourcing in the first place. The reason why you are reading this article is because you are not scalable. You are a very talent and creative person, but there is only one of you. Your to-do list is 10 pages long, but you only have one pair of hands. The sooner you realize this, the faster you will get around all of the above and start getting productive.

Got something to tell? I am very interested in your opinion and story. Learning from other people's mistakes is definitely a bonus. I'd be more than happy if you could share your experience. What worked and what didn't.

Good luck!

Amazon AWS Meetup in Tel Aviv

2008-12-04T13:42:00.003+02:00

Boy, was that awesome or what! Amazon AWS team from Luxembourg invited us tiny start-ups to an introduction (and networking) meet-up in Tel Aviv. Prior to that, there was a 2-day convention about grid computing with folks from AWS, eBay, Yahoo! and others.

After socializing for over an hour with plenty of new upcoming startups (lots of interesting things are going on right now!,) we were gathered for an evangelistic talk with Martin Buhr (@tallmartin) and Simone Brunozzi (@simon). When asked, almost everybody in the room raised hands for already trying out/working with Amazon's web services. It's quite amazing that it has become a common use among startups and based companies.

Following their talk, Guy Nirpaz of Gigaspaces introduced their incredible framework over EC2, and handed out free try-out coupons for everybody who was interested in hands-on trial of their service.

Were you there?

Martin is explaining about AWS EMEA

Simone is answering a million and one questions

$83.58 Discount on Pingdom!

2008-10-29T20:24:00.005+02:00

My server just got to 100% iowait again. Luckily I have friends who are constantly visiting dumpr to see what's new, so I got a cellphone notification (ie- phone call) that the site isn't working.

Those manual human techniques, time for an upgrade!

I just signed up for pingdom. It's a pretty sweet service. They have tons of servers scattered all over the planet, and they ping your configured server every few minutes to see that it's alive. You define what alive means. For me, it's a special php on my server that returns OK or FAIL. In case of FAIL, I get an SMS to my cellphone, telling me to rush into the nearest public telephone booth and change to my Superman costume -> website is down.

One thing I noticed just now on pingdom, is that you can save $83.58 (yearly) easily! I'm not affiliated with them, I'm just bootstrapping like most of you guys. On the index page, if you start immediately with a Pingdom Basic package, it costs you $119.40 (yearly.) It's a good price for such a terrific service, but if I can have it for less, why not. I discovered by mistake, that if you start with a 30-days Free Basic plan, you get a 70% discount if you upgrade within that month. The new upgrade plan box shows a yearly use of pingdom will now cost $35.82, that's $83.58 in pure savings!

Now, that's an upgrade!

All This CSRF Nonsense

2008-10-20T11:28:00.009+02:00

There is a lot of discussion lately about Cross-Site Request Forgery. Honestly, it's just an old and well known problem getting some media attention.

Consider the following example. It's known that YouTube uses ajax to add a video to the currently-logged-in-user's favorites. For the sake of simplicity, say the endpoint url is: http://www.youtube.com/fave?id=1. Now let's say I'm an underdog director with a killer new indie movie that I want to promote. I go through flickr's explore pages and shove <img src="http://www.youtube.com/fave?id=1" width="1" height="1" /> as a comment to all popular photos. The next day I'm on YouTube's top directors, and the road to Hollywood is shorter than ever.

Of course the example above can be put to banking transactions, election votes and befriending strangers on social networks.

Working with POST instead of GET does not fix the problem, as javascript can create a form and submit it without user interaction.

So, how do you solve it?


global $user_id;    # currently logged in user
global $video_id;   # currently watched video

# a secret that is not shared outside the server
$secret = "here-be-a-secret-nobody-knows";

# url to be sent to browser
$auth = base64(sha1($secret . $user_id . $video_id));
$add_fave_url = "/fave?id=" . $video_id . "&uid=" . $user_id . "&auth= " . $auth;

When processing the ''fave'' request itself, just check that the auth signature matches the string you reconstruct from the parameters $user_id, $video_id and the server $secret. If they don't match, you have encountered an attempt of forgery.

By adding the id of the user that this url will affect, you are eliminating the possibility that Bob will add his video to other users' favorites but his own. You might also consider adding expiration (such as time()+86400) to your request, making it void after a certain period.

And an apology for all rising indie movie directors. You will just have to work harder. :)

Apple Lifting Strict NDA Restrictions!

2008-10-04T17:50:00.003+03:00

In an anti-big-brotherish act, Apple has finally lifted the why-was-it-there-in-the-first-place strict NDA restrictions preventing from iPhone developers in discussing about the SDK, App Store approval process, and to participate in forums.

What seemed like a silly code of silence has now been removed when tight competition from Google and Nokia arrived. God bless them.

iPhone developers rejoice! Now, I need help with AudioQueue..

"Buy this new thing, suckers" photo by Jarod_Uses_Film

Stripping Dada Mail

2008-09-21T11:53:00.006+03:00

DaDa Mail is a great mailer program for Unix. I use it to send newsletters to over 50,000 subscribers and it's doing it's job very well. A while ago I gave MailChimp a go. Why? Because administering sendmail and MX-records is pretty nasty task. Cross mail agent compatibility? Pfft. SpamAssassin scores? Ugh! MailChimp delivers it all, using their own templates, they create emails that are both readable by all (major) agents AND with a very low possibility of false-positive spam detection. With that said, having 50,000 subscribers with a weekly newsletter would have cost me $2,000 a month. I decided to sit down, learn SpamAssassin and Sendmail, and make my Dada look more like my Chimp.

Removing the "Mailing List Powered by Dada Mail" banner

Whenever you're sending out an email, Dada adds a banner with a link back to their site. One way to remove it, is to pay $50 (per year) and get a new set of scripts. Another is just to change configuration and get it done today.

Edit DADA/Config.pm, and update these 4 variables to zero:

$GIVE_PROPS_IN_EMAIL = 0;
$GIVE_PROPS_IN_HTML = 0;
$GIVE_PROPS_IN_ADMIN = 0;
$GIVE_PROPS_IN_SUBSCRIBE_FORM = 0;

Having your own Subscribe and Unsubscribe pages

By default, Dada configures the subscribe and unsubscribe headers (List-Subscribe and List-Unsubscribe) to point to the mail.cgi script. I suggest that mail.cgi will be blocked behind a password protected directory. There is no reason for others to know which version you have installed, or to even to brute force their way in.

For dumpr, I have a nicer subscribe and unsubscribe pages than Dada offers. It knows the logged-in user, and if it has been subscribed before.

To configure the List-Subscribe/List-Unsubcribe and List-URL headers, first edit DADA/Config.pm again. Scroll all the way down to %EMAIL_HEADERS and change the undef default to your liking. For example, my List-Subscribe now has the value of "http://www.dumpr.net/subscribe.php".

For some reason these values are never referenced when sending out emails. So, now edit DADA/Mail/Send.pm and change these lines:

$lh{'List-URL'} = '<' . $DADA::Config::PROGRAM_URL . '/list/'.$self->{list}  . '/>';
$lh{'List-Unsubscribe'} = '<' . $DADA::Config::PROGRAM_URL . '/u/' . $self->{list} . '/>';
$lh{'List-Subscribe'} = '<' . $DADA::Config::PROGRAM_URL . '/s/' . $self->{list} . '/>';

To:

$lh{'List-URL'} = '<'.$DADA::Config::EMAIL_HEADERS{'List-URL'}.'>';
$lh{'List-Unsubscribe'} = '<'.$DADA::Config::EMAIL_HEADERS{'List-Unsubscribe'}.'>';
$lh{'List-Subscribe'} = '<'.$DADA::Config::EMAIL_HEADERS{'List-Subscribe'}.'>';

The problem with this method, is that it will use one set of urls for all of your newsletters. So if you have more than one on the same dada instance, you'll have to get more creative.

Creating your own html template

When Dada sends out mail, it wraps it with some bad copy about the server and the owner of the list. This information is important for CAN-SPAM and I don't encourage you to just remove it. But when it comes down to it, it's just purely ugly. Dada lets you change the layout of emails, if you want your email to be identical to the html page you're sending, just Manage Copy -> Email Templates and change both Plain Text Message and HTML Message to this single line:

[message_body]

Controlling the Subscriber List Yourself

Dada's admin page allows you to easily add and remove subscribers. You can modify the list without transmitting an opt-in email (but be nice, don't abuse this.) I like to keep a single copy of my subscribers list. Under dada_files you will find a file suffixed with .list, mine is newsletter.list. It's a simple one-email-per-line flat database of your subscribers list. To make things easier for me, I have a short shell scripts that runs once a day and creates this list-file, to contain only valid emails of subscribed users.

Additional Changes: Tracking Bounces

Since Dada is a mailer that interfaces with sendmail or with a SMTP server, it cannot be notified back when an email it just sent has bounced or found invalid. I still haven't patched Dada to handle these cases, but I'll be working on it, and when I get it done I will post the sources here.

Additional Changes: Tracking Opens & Clicks

Another thing I'm interested in when it comes to my newsletter, is tracking and better understanding how my readers get my newsletter. Will my readers click more if I post 5 images or 3? Will my open rate be higher if I choose this rather than that. MailChimp has A/B testing feature that I haven't played with yet, but as I said, MailChimp is way expensive if you have over 50,000 subscribers. I am going to use Google Analytics for this, and use Urchin Tracker to create a custom channel per each image and copy. Then I will be able to better understand which copy and layout is better than the other, and see the Goal Funnel through email clicks. I will post sources when I'm done.

Image by Warm 'n Fuzzy

A Hacker Goes to a Restaurant

2008-06-24T02:26:00.006+03:00

This is a short story about the time when a hacker (me) walked into a restaurant. See, I'm on vacation now in Australia. I took the time off, and traveled to the other end of Earth, to visit the lovely city of Katoomba (at the Blue Mountains.) My friend and I stepped into Papa Dino's place for some Italian food. Running through the menu, I was looking for something to feed my huge appetite. At the very bottom, I saw these glyphs. Immediately recognized as being part of the same Wingdings family of fonts. With what feels as all eyes on me, I quickly scribbled a copy onto a piece of unused napkin and shoved it into my pocket as if a part of national security secret.

Thousands of miles and some days afterwards, I re-united with my preciousss laptop. I took out the napkin from my coat and fired up Word. Running through the As to Zs, I was hoping to find the exact of neighboring planet, the secret to zero gravity or at least the answer for ''why did the chicken cross the road''.

Finally, I got the code deciphered. I looked and it and hit with shame. It said "May 2003" written in the wrong font. Damn you Irony! You win again!

And that what happens when a hacker goes to a restaurant (true story!).

Amazon EC2, 97% Complete!

2008-04-14T14:06:00.004+03:00

Amazon AWS just fired up a private beta of persistent storage for EC2. After the public announcement of Elastic IP Addresses a couple of weeks ago, Amazon does it again with a feature we have all been crying for.

I've been intending on writing a post about EC2 web-hosting for quite a while. I'm a grumpy programmer (but I stay optimistic!), and usually my comparisons contain more CON and less PRO.

With that being said, I barely hold any case against /not/ hosting at Amazon. Where a month ago I could have shouted about dns annoyances, now we have elastic addresses. When I screamed about volatile disks, we ~~now~~ will soon have persistent storage.

So why am I still not kicking my dedicated server and getting 5 xen boxes for the same price? Support. Take a minute to review the most used keywords on the Amazon ec2 developers' forums, it's people crying for help. Mostly it's when their instance is inaccessible. I, for one, have been emailing support when an Amazon bug hit and disabled my account (3 times). Amazon _does_ have a form where you can email in a support ticket, but it takes days to get a reply (which is still better than freaking Paypal!.)

Support is the most important thing when you're putting your eggs in one basket. Many startups now completely rely on AWS, and when their instance is down, so is their business.

I'm not worried at all though. I've been a loyal Amazon buyer since 1998, and when panic hit and I needed help with my shipment, they provided immediate response. Amazon is terrific, and I'm sure AWS support phone is just a matter of time.

Regression Testing A Facebook Application

2008-04-02T02:32:00.002+03:00

You know how it feels like when you launch a facebook app, and it breaks the minute you hit the 50,000 users mark? Well, I do. Facebook applications are hard to keep because they require constant maintenance. With an ever-changing API, and users from all over the world and every thinkable configuration accessing your code, it's just a matter of time till it collapses under. and. slowly. dies.

Luckily! You don't have to repeat the same mistakes and run a daily black box test using this nifty PHP class I wrote. Simply put, it's a class that extends PHPUnit_Framework_TestCase for use with PHPUnit toolkit. First, make a new testcase that extends FacebookTestBase. You will have to initialize it with your API_KEY, an email and a password of a test user (on facebook).

All FaceookTestBase.class.php does is to login into the configured Facebook user's account before fetching any pages from urls prefixed with http://apps.facebook.com. Call $this->loginFacebook() once, and subsequently to $this->fetchFacebook($url). Note that if loginFacebook() fails, the testcase fails as well. You are here to debug your application, not facebook.

Another minor note, I chose to use wget as it has a nice working session cookies and a cookie jar. Feel free to change the implementation to PEAR or Zend_Framework if you see a need. It works for me, so I'm quite happy with it :)

FacebookTestBase.class.php source at code.google.com

How Zend_Framework Is Losing The Wrong War

2008-03-15T23:14:00.018+02:00

Unless you have been living under a rock for the past year, you have surely heard of the war between Rails and PHP. There are no casualties, no land mines and no tv coverage. Zend_Framework is fighting back with full power, but shooting in the wrong direction.

I've been a bystander and just minding my own business. You see, I'm fairly comfortable with php; Over the years I have built up a nice framework of my own, with my own tools and my own reusable classes. Idea converts into a prototype in 60 seconds.

I am subscribed to Zend's rss feed, and I just grab my head and scream (silently) "what is wrong with them" with every new post. Zend is clearly trying to impress everybody with their new toolkit. After all, it has millions of lines, thousands of unit tests, and enough documentation to put insomniac to sleep. Framework talks to Flickr, Google and whatnotapi. Screencasts, conferences, hey, team with us, we're perfect!

But this is where they get it all wrong. People who were enlightened by Ruby don't care for all of this. There's one and only reason why they moved from one language and framework to another. Forget screencasts and forget about 3rd party apis. Ruby'kers are after one thing, how long it takes them to get an app up and running, from scratch to full implementation.

Although I'm not Rails user myself, I've always been stunned by their great propaganda. When visiting their homebase, you are welcome with a peaceful slogna Web development that doesn't hurt. Pan left across the border, and you have Zend's read-documentation-and-attent-webinars front.

(sorry, blogspot killed my <table>)

Zend_Framework: Can do anything but make coffee; Strongest point: RSS, 3rd Party api; Super flexible code; Step 1: download or read documentation; Used in projects you never heard of; Quite techie; Monolithic package; LAMP
Ruby on Rails: Minimal framework, core with database; Strongest point: ActiveRecord; Let's write the shortest code possible, okay?; Step 1: hey, see how I write a weblog in 58 lines!; Riding the whole 37 Signals / "Getting Real" success story; (it matters!) All samples are sexy with great webdesign; Install additional gems at will; Err, nothing out of the box?

While Ruby on Rails converts web developers to their religion, Zend is fighting with Zend_Framework against PHP+PEAR. PEAR has great classes, why did you have to reinvent the wheel?

Can't there ever be peace?
:)

THAT Amazon AWS Bug Again!

2008-02-19T18:20:00.003+02:00

OMFG. I shrieked like the Nintendo-64-Kid when it first happened (and those weren't tears of happiness, I tell ya.) It's a big slap in the face when you upload hundreds of GBs to S3, and then see a "You Do Not Have an Amazon Web Services Account" message.

Third time around, I capable of keeping calm and going through this safely. One of the things that most scared me about AWS (and the reason why I will never host my httpd at ec2,) is that there's no phone contact. There's a single email address that you can dig up on aws homepage for all your support needs.

So, what really happened here? Amazon denies me access to my aws account, while my api credentials work fine!. My webserver is operational, new files are being uploaded every minute. I get monthly bills and all is well, but the web is inaccessible.

A couple of months ago, I had the same problem. A nice Amazon support techie sent me an email that he heard from a co-worker, that if you have an Associates account by the same email, then aws dashboard breaks. Being the lazy person that I am, I don't wish to move my aws account to a different user (and mess around with custom user privileges on S3,) nor do I wish to claim my check from Associates and change all my links.

Wish I had a rabbit to pull from a hat and tell you how to get around it. But there isn't. So until Amazon fixes this bug, or at least restores my account, remember you have been warned. Don't mix business (aws) with pleasure (associates.) :D

Moving To My New Server in 6 Steps (and 0 mistakes)

2008-02-10T13:51:00.002+02:00

(or, how to move hosting and servers without your users noticing..)

I am writing this post during an ongoing server upgrade. You would assume I can do this blindfolded by now, after all, this is my 6th server move in 2 years. I attract bad hosting programs like flies to a banana. It's really a touchy subject, I don't want to talk about it. I said, I don't want to talk about it!

Moving servers always gets my hands sweaty. My server hosts several websites with hundreds of thousands of daily page views. I will be quoted on this, but "downtime is money". Every second your website is down, it is driving your users away. In this post I will cover how to move to a new server with as short period of downtime as possible.

The straightforward procedure would be:

Take webserver A offline
Copy files and backup database (snapshot)
Deploy files and database onto server B
Start server B
Make dns record point to B
(wait 24-48 hours for dns to update)

It's not going to work. Twenty-four hours of downtime? Better make a farewell party and say byebye to your userbase. You don't want that, so let's get down to business:

Prerequisites:

Regression test: prepare a sanity check script. I know you can do this manually, but it is a lot of work to check that new server handles the load, has SpamAssassin score that is less or equal to original server, has sendmail configured and working properly, that pages render successfully, database is secure, server is firewalled. I can go on forever with this.

Seperate servers access: make sure you are able to access both servers at the same time. You can use ip address or server alias. I name the old server www1. and the new server www. That way, I can always access the old server if I'm missing files or something went wrong.

Backup backup backup: better safe than sorry. If you are an administrator, you probably have a backup script that creates a restore-point. This is a good time to run it.

Server-is-being-upgraded copy: let your users know that you are currently upgrading and apologize in advance for the inconvenience.

Now that you have the ingredients, you are ready to begin cooking. Here is the procedure I use myself, and it goes something like this:

Copy files from A to B
Redirect all traffic to a static page (server is being upgraded)
Make server read only (nothing is written to database, no new files are created)
Export database on A and import onto database on B
Change www to point to B
Change .htaccess on A to redirect properly to

After 24-48 hours, all traffic to A will slow down to a halt, and it is safe to turn the server down (which is now only hosting an .htaccess with one redirect.)

How Logwatch Saved My Life

2008-01-27T22:22:00.000+02:00

Logwatch is an uber little collection of scripts for unix which can really save your life. It dodges bullets and stops a speeding train. It slows down global warming, and it can simply tell you what has been going on with your server.

Day by day, logwatch is sending me an email. I love getting emails, and mail from a daemon are no different. Logwatch's scripts run through my system logs and tell me who logged in from where, fresh installed packages, free disk space and other goodies like that -- the stuff that go behind the scenes and normally goes uninformed.

Yesterday I installed a bittorrent client package using yum; Just thought I'd try Amazon S3's bittorrent support, see how it works. I usually trust yum if were my dog. But today I got me a nice report:


--------------------- Connections (secure-log) Begin ------------------------

New Users:
  torrent (101)

New Groups:
  torrent (102)


---------------------- Connections (secure-log) End -------------------------

What's this? I have a new user on my server? That's odd, I don't remember inviting anybody! Luckily, there were no logins using this user. But this is how logwatch saved the day (no speeding locomotion required). Clearly the user was removed and threat level is down to White.

Going Bankrupt With Amazon S3

2008-01-13T22:40:00.000+02:00

You had everything planned out: your new video hosting service rocks and people love you from day one. You chose S3 for its cheap traffic and figured ads would cover all expenses. Your traffic grows rapidly and every day more sites link in, bringing swarms of new visitors. Life is just great. Your TV set has just been repossessed by the bank. Hold on, what the hell just happened?

Let's rewind and replay slowly.

Since its introduction, S3 has been used widely by new players for file storage and hosting due its cheap costs and no upfront payments. But not taking AWS Pirates into consideration, S3 can end up very expensive and hazardous to your health.

5 Rules of Thumb: ignore at your peril.

1. Never give anonymous access to your files on S3
There is never a reason to, is there? This directly translates to letting people you don't know consume bandwidth you pay for, without being able to defend yourself. It's not worth it. (Scary thought: what if a competitor wants to drive you broke?)

2. Enable access-logs on your S3 bucket
Track down leechers as soon as possible. Amazon's access log are similar to apache's and are on best-effort only. Keep a record of bandwidth used by each file you are hosting. Block those that go over-quota.

3. When possible, pass Expires to S3
Limit access to S3 storage by signing url and appending an Expires parameter. This will require users to request files through your servers, and not directly from S3; and will give you more control of who gets what and when.

4. Serve files from your own servers
Most hosting packages are equipped with a large bandwidth quota, which can also be expanded later if required. GoDaddy offers additional 500gb traffic for $20 (traffic is calculated as rx and tx combined), that's $0.04/gb, instead of $0.18/gb out.

5. Use reverse-proxy against S3
Harness the power of S3 as a secondary storage platform. Configure a reverse-proxy to download locally unavailable files from S3, and serve locally. Squid has a killer solution with lru caching.

Vote For Leah Culver!

2008-01-10T11:50:00.001+02:00

Wired is having their annual Sexiest Geek contest again. It's not a newsflash or anything, just wanted to make sure you voted for the Übersexy Leah Culver.

Have a better afterlife, vote for The Sexiest Geek of 2007.

Atime in No Time

2008-01-05T20:49:00.000+02:00

Consider the following for example:

I created an empty file called "tempfile" by running:

[root@ec2-fx1 tmp]# touch tempfile

I checked the ext3 filesystem status by running:

[root@ec2-fx1 tmp]# stat tempfile
File: `tempfile'
Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 801h/2049d      Inode: 340709      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2008-01-05 07:54:38.000000000 -0500
Modify: 2008-01-05 07:54:38.000000000 -0500
Change: 2008-01-05 07:54:38.000000000 -0500

Notice the line marked in blue. It indicates the file's access time, known also as atime.

A few moments later, I attempted to read that empty file by running:

[root@ec2-fx1 tmp]# cat tempfile

When I ran stat again, this is what was shown:

[root@ec2-fx1 tmp]# stat tempfile
  File: `tempfile'
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 801h/2049d      Inode: 340709      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2008-01-05 07:55:53.000000000 -0500
Modify: 2008-01-05 07:54:38.000000000 -0500
Change: 2008-01-05 07:54:38.000000000 -0500

Oh oh, reading a file results in writing to disk. Yeah, but that can't be much, right? Wrong! For 100,000 http requests, your webserver will cause (at least) 100,000 disk io writes. Multiply that by io-wait penalty and (possibly) raid mirror lag. A lot!

No worries, ext3 has a mount flag to disable this feature. Luckily there are two ways to remove this annoyance. One is at boot time (when kernel mounts the filesystem,) and the second is via command line.

To disable this feature for the next boot, edit your /etc/fstab and append ",noatime" to the 4th field of the ext3 mounting point of your choice. Change will not take effect until filesystem is mounted on the next reboot.

You can also disable this right now by running (e.g. for root file system):

mount / -o remount,noatime

Happy Serving!

Faster Google Analytics For A Better Life

2008-01-04T12:04:00.000+02:00

I love Google Analytics, I really do. It provides me vital information about my service: where are people coming from and what are they doing on my website.

Recently I worked on optimizing my site: the faster I make the service available for its users, the happier they are. While at process, I realized that some of my javascript code depended on a callback of onload. Only when the dom document is ready, the event is fired. And the most disturbing thing was that I realized that the bottleneck is urchin.js, the file served by Google!

Why is it slow?

As fast as Google's servers are, they still need to process millions of requests

urchin.js is hosted at www.google-analytics.com, which requires a dns lookup by client

Client is required to open a new tcp connection

Why local serving is faster?

Clients already have an established connection with you (you enabled keepalive in apache, right?)

One less dns lookup

How is it done?

It would seem a lot of work, but the solution is rather simple and consists of two things:

Get a daily version of urchin.js
Change tracker code to serve local copy instead of one from Google

For your own sake, I hope you have a single point of change for the second item.

Getting a daily update of urchin.js requires a scheduled task. If you are using a dedicated or a vps hosting, the straightforward solution would be via crontab. Most cPanel users have access to their crontab, but if not, do not despair, I have a solution for you as well.

Step 1: Log in onto your shell account, and edit your crontab file. This is done either by running "crontab -e" or by editing /etc/crontab.

Step 2: In your crontab file, append this line:

0 0 * * * apache (sh /usr/local/bin/fetch-urchin.sh) >/dev/null 2>&1

Note: if you are using a different user for apache, be sure to replace that value. I use /usr/local/bin for my scripts, but feel free to change this as you see fit.

Step 3: Create a file called /usr/local/bin/fetch-urchin.sh with this content:

#!/bin/sh

wget --quiet --output-document /tmp/.urchin.js "http://www.google-analytics.com/urchin.js"
if [ "$?" -eq "0" ]; then
    chmod 0644 /tmp/.urchin.js
    mv /tmp/.urchin.js /var/www/html/urchin.js
fi

Again, if you're hosting your files from a different path, make sure you change it.

After creating that file, change permissions by typing "chmod 0755 /usr/local/bin/fetch-urchin.sh"

Step 4: Run this script manually, to get a copy of the latest urchin.js.

Step 5: Reload crond (daemon that runs the jobs) by running "/etc/init.d/crond reload"

Alternative solution: shared-hosting users, and users who don't have write access to crontab can use this trick: in one of your scripts, check the modification date of "urchin.js", if it's more than 24 hours ago, use file_get_contents and then file_put_contents to fetch and store the new version.

Fast & Happy Serving, Everyone!

Sanitazing User Generated Content With lib_filter

2008-01-03T17:54:00.000+02:00

Your new killer webapp has spawned millions of followers, TechCrunch has blogged you, and Google loves you. Now spammers worldwide got you on their radar. Oh OH.

Flickr's uberhacker Cal Henderson has published lib_filter, a php library for keeping user submitted text friendly and clean. Taking care of:

Tag Balancing (e.g. <b><i>oops!</b>)

Tag Formation (e.g. <<script>script<script>>, what happens if you strip <script>?)

Allowed Protocols (e.g. remove all "javascript:" href references)

Allowed Attributes (e.g. strict <a> to contain only href and target attributes)

Sources are very friendly and are distributed with a thorough harness test. I strongly recommend spending the hour it takes to implement it on your website. It has saved my life already.

Anybody wants to share a success story?

Hello World \n

2008-01-01T23:35:00.000+02:00

I guess there is no easier way to start a blog. For the past few days, I have been firing up a new browser window and stare at a blank page wondering what counts as a good first-post? Next to my monitor I keep a handwritten list of articles to write and items to discuss. It's quite long, so lots of fun ahead. Now that I broke the ice, I can just dive right in.

Before we depart, let me introduce myself. My name is Gil Megidish, and I'm the guy behind numerous projects, such as dumpr.net, shokolada.com, four facebook applications, three game rewrites, several unix projects, and many project contributions. I have been coding for most processors, and believe programming is a way of modern religion. My other blog is megidish dot net.

This weblog is where I will share my experiences with designing, maintaining, programming, optimizing, and networking websites and projects. Feel free to introduce yourself; Let be the first to greet: "Nice to meet you".