Sea Breeze and Biz

Multiple Data Centers and the Scalability Step Function

2009-11-21T15:19:05-08:00

This week outbrain launched the second Data Center in the US west coast.

Staying in one cozy Datacener is always more comfortable but there are lot's of reasons also to open another one as you grow.

I will name a few:

Disaster recovery - you want to be able to still server your clients even if a 'plane crushed into your datacenter building' (not that such things happen :).
Global load balancing and users proximity - you want to serve your clients from places that are close to them.
Data replication - you always have your data replicated to more then one physical location so it is harder to lose it.

I can probably name some more reasons but I want to write now about another interesting reason which is the scalability step function. usually when a web business scales, there are 2 ways to handle it. Scaling UP and Scaling OUT. By Scaling UP you always buy bigger, faster and more expensive machines that can handle your load and size of data. By Scaling OUT you build your application in such way that you can always add more machines to handle coming load. The big problem of Scaling UP it the "Step function" in cost. If your system top capacity is X and it cost Y, when you need to grow your system to 1.1X the next jump in cost is probably to 2 (or 4)Y. When scaling OUT, when the demand grow to 1.1X you only add 0.1Y to the cost and there is no need to put redundant cost.

What does this have to do with multiple Data Centers? Actually... A lot.

If you have the ability to manage more then one datacenter, you have the ability to scale your system OUT in much lower costs.

Let's see what happens in datacenters. you start with a single rack, then you expand to 2 then after the 3rd there is no rack in proximity and you start wiring all around. Before the 5th rack you want all of your racks to be in proximity so you get a "cage" which is a leased area to put all your racks in proximity. (ooops - a step in the cost). then on the 30th rack you need a bigger space and you do a "migration project to space 4 times bigger (another big step).

Apart from the space you allocate for the servers, keeping them on the same LAN turns to be very costly in the bigger numbers. Connecting 100 servers to a LAN demands more then 1 switch and if you want ports redundancy it becomes even harder. then you jump to higher level switches that complicates the administration and when you get really big the next "Step" is core switches that makes Cisco's salesman smile and your CFO cry.

Instead of that, when you have the ability to manage multiple datacenters, when your setup hits it's limit - you just open another datacenter, can be even next door to your current one and you start racking this one with servers as you grow. Your step functions become smaller and most important, linear to your growth.

You can even shrink your business if you like. Lost some clients, did a business change or optimized your system? one datacenter became redundant? leave it. this is something you can't do when committing to one big space an after doing the network investment.

I guess you've got my point.

Lastly - I wan to thank 2 of our service providers that enabled this ability:

Atlantic Metro Communication - our CoLocation provider in both sites.
Cotendo - our global DNS provider.

They are both highly recommended.

Now everybody can be heard - OutLoud

2009-11-14T14:30:16-08:00

This post is not for the big Co's. It's even not for the big media buyers.

It's for the small businesses, for the millions of one-man, two-man shows that runs their nice little business. These little businesses are doing great things but have very little chance to get nice exposure and expand the business.

I'm talking about Artists, Musicians, Writers, Technicians, Free Lance software engineers, eBay sellers, Etsy artists, etc... all these are paying one or two salaries but usually don't have the budget to buy the advertising money to get a banner in a news site.

What they usually have is good WOM (Word Of Mouth), can be a blog post written about them or about the work they do by some enthusiastic customer. When they see it they say: "WOW it's great to see somebody think that about my work, I wish every body could read it". Now its possible.

Now everybody can be heard - OutLoud!

Small businesses, Artists, Free Lancers, Internet sellers like eBay or Etsy,... everybody who gives products and services. If you have a good story written about you on the net. If the story is authentic and interesting to read. all you need to do is submit the link in outbrain site and for only $10 a month your story will be shown below relevant stories on sites like USAToday.com, Slate, Chicago Tribune and many many other sites and blogs.

The system will drive traffic to the story about you continuously and get more people to know you or your product. People that otherwise were never expose to the good service you give.

Now... you are heard! Say it OutLoud

Wor(l)d of Mouth

2009-10-07T04:56:09-07:00

Found the term "World of Mouth" in a presentation in this post by Gal Mor (Hebrew).

The slides in the presentation moves very quickly but you can clearly see the section speaking about WOM marketing. That's basically what we do in outbrain.

Almost every product now has WOM speaking about it, almost every business have it's group of fans that share their excitement on the social media.

All you have to do to promote the content speaking about your great product or service is to send it to us in outbrain and for only $10/month we will promote it for relevant audience.

here is the presentation, It tells better then me why it's the right thing to do.

About Data Center hosting vendor.

2009-10-03T14:35:31-07:00

As I mentioned in one of my previous posts, from the early days of outbrain we chose not to base our system serving on a cloud solution and establish our own serving farm. I guess it is not because we did not know that cloud exist or was afraid to try cloud. I think it was because we were somehow funded and as graduates of big internet companies we thought that if we want this company be as big we have to start our infrastructure the way the big ones start - with our Owned and Operated data center.

That doesn't mean you own the facility but when you start it you buy your own hardware and host it with some Colocation facility vendor. There are lots of such vendors with different sizes and locations. Most of them gives very much the standard service of power, network, security, power redundancy,etc... but I want to talk here about 2 aspects that are most important and makes our Colocation supplier exceptional - Serviceability and Professionalism.

It's been almost 2 years now that we are working with Atlantic Metro Communication (AMC) as our Colocation vendor. We have been hosted somewhere else before and forced to move in a short notice. We have also tried Amazon AWS for short period and got disappointed but now after long enough with AMC and handling almost 20X growth in our traffic I can recommend these guys to you all.

The reasons is the 2 factors that made them so exceptional and a good fit for company like outbrain:

Best customer service - Like outbrain, for the AMC staff, customer satisfaction is a core value. Like an integral part in the DNA. Yes, they do have a proper ticketing system and a proper NOC like any other vendor, but you can also submit a ticket or send a mail and get an immediate reply, in the middle of the night from Steve Klenert, their CEO, taking care of the problem or answering one of your questions. You can see that with all AMC staff which is always eager to help and share from their experience and knowledge, even when the case is not in their direct responsibility. For a CTO of a company located across the ocean from his Datacenter, this is super helpful.
Being professional- knowledge in the technology zone is always something you can argue about. I believe no one is perfect and knows it all. However, there is something that makes a professional staff... mania for solving problems. When it comes to handling internet WAN and LAN networks, it is such a huge domain that nearly every problem is different from the other. Experts with good understanding in this domain and enthusiasm for solving each problem will eventually nail it and solve it. That's one of the things I see in AMC staff, these are guys I can count on for solving problems and will not give me answers like "That's vendor X problem, nothing we can do about it".

outbrain is growing fast and so does our infrastructure I'm pretty sure AMC will keep being on our help in the foreseeable future.

One word for you, when you want to choose a colocation vendor, contact @klenert and see what he has to offer, you'll not regret it.

Rolling Rolling Rolling

2009-09-22T02:43:17-07:00

Under water shooting is a new trend in our paddling club.

It also enable you to follow what's going on below water while doing an Eskimo Roll.

here is me:

New Office for outbrain Israel - what have changed?

2009-09-02T13:06:52-07:00

In the last 2 years outbrain Israel had a lovely little office in Natanya. It was actually right below Shopping.com (which is the company I came from) office so I had warm feeling to the area. I also hoped that a good Karma will fall down in gravity and give us an Exit luck as Shopping.com had :).
Our lease agreement for that warm and cozy office was about to expire and since we were growing, there was no point in extending that agreement.
We have been looking for another space in Natanya for very long time and did not find anything that will fit our need and company culture. As always, the luck is the biggest factor in such cases. Saifun, an Israeli semi-conductors company, was looking to sub-lease one of their building's floors. long story short... we took it.

I wanted to draw some of the differences and changes between the old and the new office. Things that we did, to make our engineers more comfortable and of course, more productive.

Standard upgrade: our old office was nice and warm but had lower standard of building. As example, we had only single toilet (unisex) which was not the most "ventilated" place, so for #2 you had to go outside quite a way. Mess! Here there are 3 cabins for each sex and shower for the sportive ones that want to use bicycles and ride to work. Light, sunlight, acoustics, and air condition is by far better then the old office. I'm now connecting my Ipod to a sound system in our Saloon and my Hebrew music does not interrupt the engineers around.
All of the furniture are also part of the sub-lease from Siafun and are ergonomics and well designed to the comfort of the engineers that spend long hours next to the computer.

Open Space: we had long debate if it's the best to keep engineers in offices or have them in open space. Our old office had 2 large rooms for the engineers and in them there were 6 work stations for the engineers. We kind of liked that, it enabled the guys to interact freely from their chairs within the teams but also not interrupt from one team to the other on separate discussions. Luckily, the new office have similar design, it is actually cubicals of 4 people, (the perfect size of a software team)that have ~180 cm height of walls. Still keeping the level of collaboration within the team. The acoustics of the Teknion separating walls is just great so, although we are kind of noisy company, it is still pretty quiet in the office.

Communication: One of the biggest productivity factors in an Internet company is the speed and quality of communication. We did not save on that! We've got a 4Mbps symmetric line from Cellcom (the leading Israeli cell provider) to keep the bandwidth to the internet backbone fast and wide. As outbrain is located both in Israel and in NYC, it is very important for us to keep a clear channel between the offices. this "wide" line will use us to connect the telephony systems and most of all for fluent video/audio conferencing. I believe this is crucial to the general feel of "one company" instead of "us and them" that usually get developed in remote sites. A nice project that was done by Shai, our IT consultant, was to put Asterisk telephony system which is totally free and open sourced. it will also allow us to cut many communication expenses significantly. Soon, when our NY office will move to a new place we will have both Asterisk machines connected and act like one, cross Atlantic, phone system which will simply cut all the cost of long distance calls.

Location: people who know me, knows I'm the kind of a country guy that ocean view and green fields out of the window just make my day. Some of the options we had for offices did fulfill my aspirations for that. I have to admit in the new office this was a compromise for me. However, as someone that knows the Israeli High-tech scene, there is nothing more important for employees then proximity to restaurants for lunch. For that matter, our new location is a Home-Run. It's right above lots of good places to eat.

Atmosphere: well, one thing that bothered me when first seeing the property was that it looked too much like an office, It wasn't as warm as our previous, wooden floor, office. It was somehow... Cold. So, right before we moved we did slight changes that made it look much more fun and homy.

First of all we butchered the first cubical and made it a big Saloon - with our well known green sofa, some decorations and our famous Wii equipment. Guitar hero stands in the corner waiting for the next talent to come.

Actually, having no designer to this office made me and Eytan act as designers getting some decorations from IKEA and buying some cushions for the guys to take some rest on. We made a fond of ourselves but the feedback are great so Eytan is considering to change profession now :)

The office is much bigger then what we actually need. However, if you like it and you are a small company that is looking to lease a space and work in nice atmosphere, our new, cool office is ready for partial sub-lease. The main criteria is to be a company with fun people DNA.

More photos of the office move, with some artistic touch of Vered, our Data Analyst and home photographer, can be found here

Family Sailing Vacation - The Full Guide

2009-06-26T14:10:09-07:00

We just got back from a sailing vacation with the kids.
The destination, as always, Gocek in southern Turkey.
The party was, Alona, my wife, our 2 kids (8 and 5) and another friends family with their kids in the ages of 4 and 2. And of-course the skipper... me.
Sailing vacation in Gocek is always fun but with 4 kids in varying ages is, somehow, a challenge to get prepare to. As with every sailing vacation it is essential that you will pick your party adults carefully and make sure you all can get along together. If you miss that... especially with the kids - it can ruin a lot of the boat atmosphere. On our case, the choice was terrific and we all got along very well.
Unlike many other vacations, sailing vacations put people in slightly different conditions like boat movement, lack of "personal space" and privacy. It is sometimes tough for adults but fortunately kids are much more "adjustable" and get along very well.
Some tips for boat behavior with family:

kids that experience sailing the first time are "unpredictable" in terms of sea sickness. better have the right drugs that you know may help them and if you feel they are getting "unhappy", get a bucket next to them.
Yacht toilets are not the standard they know so they might need an escort every time they go. always good to have a bucket with a bit of water at the cockpit in case of "emergency need".
In tougher conditions it is better for kids to not wonder below deck. moving objects and closing doors might harm them. The last thing you want is a bandage over kid's finger.
Kids are happy when busy, bring them toys, books and other games.
Boat furnishing is rather expensive - try to use toys like books, cards and figures that wont scratch and hurt furnishing.
Going a shore for a little trip or having dinner in a restaurant can get beyond the kids hours. Get yourself a stroller for just in case.
The water in the turkish coast are simply great for swimming so take all the inflatables you have at home and ask for foam Noodles.
Important info, every restaurant in Turkey serves Schnitzel and fries so... you are safe with kids dinner. If you like to try seafood - turkey is a great place for it.

Here are some tips for the skipper.

The first rule for family vacation skipper is... be patient!!!
It also worth to plan ahead with your adults crew who will do what and when. Brief them about anchor handling and mooring procedure.
Be aware that at least one of your crew members will have to stay in the cockpit watching the kids so 2 hands are off your stuff.
Don't push yourself to the limits. You have limited crew that can help and some don't have any experience so try to keep out of tough maneuvers like, windy anchorages or tough sails handling.
Jib riffing is much easier then main riffing. If you want to set sails (recommended) prefer Jib.
When sea is a bit wavy from the side, I always prefer to set the main sail to prevent the boat from rolling all the time. That's usually make people feel better.
In Turkey, at many anchorages you use anchor and a stern line to shore. Usually the guy that swims to shore is doing a knot around a rock or a tree. If your man is a good swimmer but not that experienced in knots you better use my trick. I brought with me a spring shackle with a little floating buoy which made the whole maneuver much easier and faster.

Most important - Don't miss the Turkish Ice-cream show. the ice-cream itself is not that good, but the whole show makes the kids lough a lot.

The film below shows some more photos from this great time we had:

My take on cloud services

2009-05-18T02:48:00-07:00

Cloud services are a big hype now in the tech/internet industry.
Engineers love it - very easy to get an instance of machine and run development tasks and testing.
VCs love it - it makes development cycles shorter and turns CAPEX to OPEX.
IT engineers - well, they will have to get used to it but eventually will also master this also.

Lots of people keep asking me what is my take on cloud services. Does outbrain use it?
So... here it is:
I think cloud services with Amason AWS as the leader in this field are doing a great technical job. they truly made a revolution in the IT space and made computing resources cheap and available on demand. This is not only a big capital investment but also a huge engineering effort that requires lots of time of great IT talent.
To be honest, I think there is nothing technical that cannot be done on cloud service. Take as an example the adoption of Map/Reduce method. It was initially created by Google, then shifted into the open source as Hadoop project and about a year later, as the demand for it grew AmazonAWS also offered it as Elastic MapReduce. Today i noticed they released first Beta of CloudWatch - the missing piece of monitoring and load balancing.

According to everything I said above it would have been obvious for outbrain to use such cloud service. Actually... things are not that simple.
First, when growing your own application you need to optimize all your stack for the benefit of your application's needs. With virtually shared machines you cannot choose and optimize your hardware to do stuff like this. You just can't.
Second and most important is reliability. I know you open your eyes wide open and ask... 'does he think he can get better up-time and reliability then Amazon?' and the answer for it is... 'probably not'. A start up company, with limited resources, cannot get higher up-time then amazon AWS. The question that should be asked is 'What happen when the shit hits the fan?'
Here is one rule of the nature since the invention of the wheel - technological solutions will *always* break!!! sadly, even plains and submarines fail. The more the impact of the failure is critical the less common failures are but you can agree that no technological system is 100% fail safe.
I can assure you that even mighty AWS will fail to give it's service at least once a year. From my experience they will not even tell you there is a problem and you will hear a complete silence not to mention an ETA for the problem.
The bigger question is what do you say to your clients? can you give them ETA? can you tell them 'It's not us... its Amazon?'.
Running your own setup is defiantly harder and yes... you need IT experts in order to do it right. sometimes it even cost you more but such failures as we had with the cloud service could have cost us by the life of the company if we were not leaving AWS.

I think technically cloud services will always be a bit behind the cutting edge but will eventually close the gaps. The downside that they will never be able to solve is customer tailored solutions - just because there is no 'one size fits all in this business' and most important, support, support support. The AWS salesman says I can buy a Gold or platinum support which eventually give me the ability to pick up the phone and yell at somebody at Amazon, but it will not necessarily fix the problem. Having a great team of engineers that can control the setup will give you the fix faster and without yelling.
Your BD guy can always say to the customer, 'we have a problem- all our engineers are on it even it's past midnight in Israel, their ETA is one hour'. so the customer knows what to expect. With cloud service.... what can you say? And this can really harm your business.

Photo by DeZeen magazin

Amplifying Word of Mouth

2009-05-05T16:13:01-07:00

At some points people get to ask me 'So... what does outbrain do?' - well that I already know quite well how to explain. The second question that comes up is usually 'and... how does that makes money?'. Here things get a bit complicated to explain.
It usually starts with me saying 'We have this concept called Sponsored But Good' and I start explaining all about it (as you can read in the link above).
That usually brings up the look of nodding pig having a swine flu and they change the subject immediately.
I bet you still want to know what outbrain does for a living. well... we are 'Amplifying Word of Mouth'.
Yes, we take pieces of content (from blogs or new sites) which represent a good Word of Mouth (WOM) about a product or service and amplifying it to a reach which is much broader then what it originally had.
Imagine a HydePark speaker that have his own small audience of listeners - there is no way his massage is going to be delivered to everybody who might be interested. But how about we give this speaker a radio or television show at prime time. What do you think? Will he get his maggase through? I bet he will.
outbrain actually take it one step farther, the message will not get to all the audience in the reach but to the users at the network that might be interested in this message and get it to them without them getting to the HydePark.
In our case this message is the WOM about a product service or a brand.
In case you are an advertiser and have a branding message to get to the world and some reliable WOM is going around your product- outbrain is for you. Let us Amplify it.

#memthem - Israeli Memorial Day

2009-04-26T23:02:01-07:00

Today is the Israeli Memorial Day for all those who sacrifice their life for the defense of the state of Israel and those who were killed in terrorist attacks and wars.
Unfortunately we have too many of those in Israel and this day is touching almost anyone of us here.
I was thinking, if any one of us will contribute a little twitt with the name of the person to remember and a link to a site, blog post, web page or even upload a picture to twitpic. Just add the tag #memthem.
I believe that way we can create a very big memory album for those we honor their memory.