Amr Awadallah Blog

Last chance to signup for Hadoop World [Discount Code]

amr — Tue, 29 Sep 2009 09:08:41 +0000

This is pretty much your last chance to signup for Hadoop World, it is happening this Friday (Oct 2nd) in Newyork City. The registration prices are higher now since these are the last few days, but if you are interested in going then shoot me an email, I have a special deep discount code for “friends of amr”.

Below I highlight a number of news/blog articles that covered some of the use cases that will be talked about at Hadoop World:

“Hadoop buzz continues to excite the cloud”. CNET, Dave Rosenberg, September 4th, 2009.
“What’s the New York Times doing with Hadoop?”. InfoWorld, Savio Rodrigues, September 11th, 2009.
“Telecom operators get the scoop on Hadoop”. Telephony Online, Rich Karpinski, September 15th, 2009.
“Q&A: Visa dips a toe into the Hadoop pool”. CNET, Matt Asay, September 17th, 2009.
“The Tech Behind eHarmony”. PC World, Eric Lai, September 19th, 2009.
“Hadoop World Shows Diverse Uses”. SD Times, Alex Handy, September 21st, 2009.
“Cloudera and Facebook Shed More Light on Hadoop Integration”. TechCrunch, Leena Rao, September 23rd, 2009.
“Welcome to Hadoop World”. Network World, Jon Olstik, September 24th, 2009.
“FeatherCast - Aaron Kimball on Hadoop”. Apache Feathercast, Rich Bowen, September 24th, 2009.

I look forward to meeting you there,

– amr

Share it: related | del.icio.us | digg it! | reddit!

The Essentiality Criterion

amr — Thu, 30 Apr 2009 09:31:25 +0000

First, I can’t believe it has been three months since I last blogged, all I can say is that a startup will suck the life out of you

I still see many startups creating free online services and expecting them to generate hundreds of millions of dollars from advertising. I might be stating the obvious here, but that will only work out if these services get tens of millions of users, with those users returning to the service almost every day, and that will only happen if the service meet the essentiality criterion. Essentiality is the litmus test that differentiates a life-style business (e.g. a super popular blog that makes a few million $/year max) versus a venture business (i.e. a business which generates hundreds, if not thousands, of million $/year). By essentiality I mean both the ability to amass a large population of users, but become necessary to these users such that they need to visit the service almost every day (if not many times a day).

For example, here are some essential online services:

E-Mail/Chat/IM: It is essential that you communicate, almost continuously
Search: We all need to find stuff many times a day
News: Almost everybody needs to know what is happening. Facebook is news about your friends, it is communication too.
Music/Movies/TV/Videos/Games/Sports: A decent population needs their daily entertainment “fix”, however very few destinations are able to aggregate a large enough audience due to content fragmentation

I was then going to list the many services/sites that I don’t think will make it due to failing the essentiality criterion. Many of these services will collapse due to fact that they are small/life-style businesses but their VCs expect them to hit venture scale (the ones structured as small businesses will live and prosper). But, to avoid alienating such businesses (since some of them are potential customers for Cloudera) I decided not to list them here . That said, the common pattern among them are: either they have a lot of users but are not essential to any of them, or they are useful but to a very small group of users, in both cases this means they will not be able to amass the huge volume of pageviews required to make venture-scale money based on advertising.

Cheers,

– amr

Share it: related | del.icio.us | digg it! | reddit!

Diagnosoft raises $4M and hires Firas BenAchour as CEO & President

amr — Fri, 30 Jan 2009 06:04:58 +0000

I seed funded a company called Diagnosoft a few years back, mainly because one of the cofounders (Nael Osman) is a life-long friend of mine, but also because the technology was very intriguing and came across as something that could save my life one day

In layman terms, Diagnosoft develops a program that analyzes cardiac MRI video sequences to generate quantities that show where the heart is weak. These measurements greatly augment the “eye balling” that physicians do today, which leads to faster and more precise detection of problems that can otherwise go undetected. The software also does visualization and integrated reporting, but the core differentiator is the proprietary quantification algorithms. The founders are from John Hopkins University and are the leading experts in this field.

The funding for Diagnosoft comes from a VC firm called “Technology Development Fund (TDF)“, which is based in Egypt. BTW, many of the life sciences VCs that we contacted in the US wussed out from investing in Diagnosoft, the reason they usually gave us was “it is too risky”! Well, yeah, inventing a new disruptive technology is risky, but Diagnosoft is ready to cross the chasm, it is already being used by early adopters at hundreds of research institutions all over the world. Diagnosoft HARP is now the de facto standard for cardiac tagged MRI analysis (more than 22 published papers involving more than 3000 test subjects).

The key premise behind this market opportunity goes as follows: the previous wave of heart treatment was heavy cardiac surgery, the current wave is non-invasive surgery, and the future wave will be pro-active treatment for which precise detection tools like Diagnosoft HARP will be a must. You can find out more by reading the press release announcing the funding and Diagnosoft’s new CEO:

http://news.prnewswire.com/DisplayReleaseContent.aspx?ACCT=104&STORY=/www/story/01-29-2009/0004963069

I am very passionate about Diagnosoft, not just because of the market size potential, but rather because it can truly revolutionize prediction and prevention of cardiac diseases, the number one killer of humans in the United States.

Congrats to the Diagnosoft team on this very important milestone, onward and upward.

Cheers,

– amr

Share it: related | del.icio.us | digg it! | reddit!

My Hajj (Pilgrimage)

amr — Sat, 20 Dec 2008 07:43:06 +0000

I just returned from the most amazing trip in my life, it was two weeks with my wife in Saudi Arabia to perform the Hajj (Pilgrimage). Many of my American friends asked me about the trip and what it entailed, this blog posting tries to summarize that.

First, the Hajj is the fifth and final pillar of Islam, the other four being: (1) Conviction that there is only one God, and that Mohammed is his prophet, (2) Five daily prayers (to continuously remember God), (3) Fasting the month of Ramadan (to show obedience to God, but also to remember the less fortunate), and (4) Giving out charity to the poor and needy (about 2.5% of your wealth every year).

Hajj is only required once of Muslims who can afford it both in terms of wealth and health (it is a very tiring and expensive trip). In many ways, the Hajj is like confession, once you perform Hajj all of your sins are erased and you start with a new slate. But Hajj is also a time when God is most likely to answer your prayers, and hence we also ask God for things we want, in addition to asking for forgiveness (Cloudera got many prayers during this trip ).

The Hajj takes about five days, and it goes as follows:

You dress in just two white towels, one that covers your upper half, and one that covers your lower half (and that’s it, no underwear ). The idea is twofold: (1) rid yourself of worldly belongings so you can focus on just your worship, and (2) eliminate difference between rich and poor. However, women get to wear normal clothes. There is also a long list of dos and donts once you wear the Hajj costume and declare your intention to perform Hajj (e.g. you are not allowed to kill insects, fight with others, remove hair, clip your nails, get angry, or make love).

Photo 1: I and Shirin during Hajj, I was only wearing two towels

Next you go to Mecca, and walk seven laps around the Kabaa, the holiest site in Islam (Men uncover their right shoulder and jog during the first three laps). The Kabaa was originally built by Adam, then later reconstructed by prophet Ibrahim (Abraham). The Kabaa is considered to be linked to the house of God in the heavens.

Photo 2: I took this photo from the roof, the place was packed even though it was 4am in the morning!

Beside the Kabaa there is the “station of prophet Ibrahim” which has a rock with the feet of prophet Ibrahim engraved in it. After finishing the seven laps around the Kabaa (aka Tawaf), you pray behind the station of prophet Ibrahim, then you drink Zamzam water (holy water), and make prayers for long health (or any thing else you want).

Photo 3: In the station of prophet Ibrahim there is a stone with traces of his feet from the time he reconstructed the Kaaba.

Next you do seven trips between the Safaa and Marwa, these are the two mounts that Hajar (Hagar) ran between after prophet Ibrahim, her husband, left her there alone with her newborn, Ismael (Ishmael). Hajar made seven trips between those two mounts while seeking water mirages to try and quench her newborn’s thirst. After her last trip God surprised her by opening the Zamzam well right between Ismael’s baby feet. Every year during Hajj more than two million Muslims imitate what this faithful mother did thousands of years ago. We also get to drink Zamzam water, which is considered to be holy water (it does taste very special due to the minerals in that well).

Photo 4: The Safa and Marwa area is now air conditioned and has three levels with escalators between them. It was interesting to see some folks scared of the escalators since it was their first time to see one!

The next step is to stay from noon till sunset making prayer at the valley of mount Arafaat (aka mount Mercy). You confess your sins (directly to God), repent, and ask for forgiveness. You also make prayers asking for whatever you wish, e.g. cure from a disease, happy family, children, success, wealth, paradise, etc. There are no authentic stories for why mount Arafaat is so special, some unconfirmed stories are that this is where Adam and Eve found each other after they were banished to Earth, another tells that this is the mountain from which prophet Ibrahim preached the oneness of God. At the end of the day of Arafaat God forgives you and you become as sinless as the day your mother gave birth to you (assuming you were sincere in repenting and asking for forgiveness).

Photo 5: Praying at mount Arafaat (Mt Mercy) and surrounding valley (Photo by Faisal Mirza, who was in our group)

After you are done at Arafaat you head to a place called Muzdalifa, there you collect seven pebbles for stoning the large Jamara (devil symbol), you also perform the night prayer there. We witnessed the worst traffic jam ever during our trip from Arafaat to Muzdalifa, the reason being two million pilgrims all wanting to travel those six miles at the same time (the bus took six hours to travel six miles!)
The next step is to go to Minna, the city with the devil symbols, there you pelt the largest of the devil symbols with the seven pebbles you collected from Muzdalifa. The historic story behind pelting the devil comes from Satan being furious that prophet Ibrahim had total belief in God and was willing to sacrifice his own son to prove that. Satan tried to sway prophet Ibrahim away from obeying God’s orders, but prophet Ibrahim did not listen and started throwing stones at the devil.

Photo 6: Rows and rows of tents are erected in Minna just for the Hajj week, then all of them are removed after it is done.

After the stoning comes the sacrifice, and this is where you donate a large sheep to poor people, this is a holiday for Muslims, it is called Eid Al-Adha, or Feast of Sacrifice (it is analogous to Thanksgiving in the US, with Sheep instead of Turkey). In the old days the sacrifice had to be done inside Mecca, but with two million Muslims making Hajj these days it does not make sense for all the sacrifices to be made in Mecca. So what the Saudi government does is collect money for the sacrifice and distribute that across the world (the money still has to be used to provide a sheep sacrifice). The story of the sacrifice comes from God ordering prophet Ibrahim to sacrifice his son Ismael. After prophet Ibrahim put his son down and tried to slice his neck, God sent a sheep to prophet Ibrahim since he passed the test and asked him to slaughter the sheep instead.
Now, if you are a man then you shave one inch of your head hair (which means shaving all of your hair if you got short hair), and if you are a woman then you just cut off a lock of hair from the end. Once you have done that then you can take off the towels and put on normal clothes, but the Hajj is not over yet.

Photo 7: Look ma, no hair In this photo I am standing in front of the devil symbol (it is the large wall in the background).

For next two, or three days, you throw more pebbles at the devil symbols. You throw seven at each of them (the small, medium, and large symbols), so twenty one on each day.
Next, you go back go Mecca and perform another set of seven laps followed by a prayer behind the station of prophet Ibrahim (no walking between Safa and Marwa, just seven laps around the Kabaa), this is called tawaf al-ifadah and it completes the Hajj
Finally, before leaving Mecca, i.e. on day of your departure, you do another set of seven laps, essentially saying farewell to the holiest site in Islam. This was the most crowded day, as all pilgrims finished the other actions and descended on the Kaaba to make the farewell laps. I was simply awe struck to see all these people from all corners of the earth united with the same purpose. To put things in perspective, the largest events at the Beijing 2008 Olympics only had 100,000 people.

Photo 8: This photo is from a set of really nice Hajj photos at Boston.com’s “The Big Picture” blog. The Kabaa is the black dot in the middle of the structure with lots of pilgrims around it, the Safa and Marwa mounts are at opposing ends of the stretch covered with blue on the left (they are building another floor, hence the cover). The super huge hotel behind the haram mosque is called “Grand Zamzam”, I was staying at the hotel to the right of it, which is called “Makkah Hilton Towers”.

Before performing Hajj in Mecca, we also got to visit Madina, home of the Masjid Nabawy (Prophet’s Mosque), the second holiest site in Islam (the first being the Kaaba, and the third being the Al-Aqsa Mosque in Jerusalem). Prophet Mohammed (pbuh) is buried inside this mosque. We also visited a number of interesting sites in Madina, like areas of famous early Islamic battles and the graveyards associated with them.

Both Mecca and Madina reminded me of Las Vegas in a weird way: they are in the middle of the desert and the downtown of the city is comprised of super large hotels, however, instead of gambling, drinking, and nudity there is prayer, charity, and fasting

Photo 9: The Prophet’s mosque is on the left side of this photo, on the right side there is hotel after hotel after hotel, all with mega shopping malls inside them.

In conclusion, performing this pilgrimage is something that I will never forget, and it was double rewarding to experience all this with my wife by my side while the kids stayed back at home under my mom’s care (thanks mom).

Salam (aka Peace),

– amr

PS: It is customary to put (pbuh) after the name of prophet Mohammed, it means “Peace Be Upon Him”. If you are curious to learn more about prophet Mohammed, then I highly recommend “The Sealed Nectar“, which is a biography available at Amazon.com and many other places.

Share it: related | del.icio.us | digg it! | reddit!

What happened to Obama’s brother in Cairo?

amr — Thu, 06 Nov 2008 02:04:01 +0000

First, congratulations to the US and the world for Obama becoming the 44th president of the USA.

A few months back an Egyptian columnist, Ibrahim Eissa, wrote a hilarious column depicting a hypothetical scenario in which Hussein Obama stopped by Cairo before going to the US. In that scenario he married an Egyptian girl and she delivered Mabrook Obama, but after two years he traveled to the US and left his wife and Mabrook Obama behind. In the states he married an American girl, then got blessed with Barack Obama. The column continues to follow the lives of the Egyptian born son, Mabrook, versus the US born son, Barack.

Of course by now you can see where this is going, the US son went on to take the seat of the most important position in the world, but the Egyptian son didn’t even get a toilet seat . In fact, no body in Egypt, black or white, can dare to dream to get the seat of the president. And that, my friends, is why the US is the envy of the world, it is truly the land of dreams, the land where you are measured by how talented you are, versus your “nepotistic” connections, the amount of your “bribes”, or your race/religion/ethnicity/sex, etc.

That said, I do dream of the day that Egyptian democracy catches up to the US, after all, I do love Egypt, it is my motherland.

Cheers,

– amr

PS: I had to write that last sentence to be allowed to visit

Share it: related | del.icio.us | digg it! | reddit!

Twitter mentions also predict an Obama win.

amr — Wed, 05 Nov 2008 01:02:14 +0000

Following up on my previous post regarding Facebook lexicon predicting an Obama win, the mentions for Obama in Twitter also extremely outrank McCain (this comes from a service called Twist which monitors Twitter mentions):

Go Obama

– amr

Share it: related | del.icio.us | digg it! | reddit!

Facebook Lexicon Predicts Obama Win.

amr — Sun, 02 Nov 2008 19:56:20 +0000

The Facebook Lexicon is a very nifty tool which analyzes the frequency and associations of words in Facebook wall messages for profiles, events, and groups. The first graph below shows that Obama is mentioned significantly more frequently than Mccain, and, more importantly, the second graph shows that the mentions for Obama have more positive sentiment (e.g. “I love Obama”), versus negative sentiment (e.g. “I hate McCain”).

Figure 1: Mention frequency for Obama vs McCain on Facebook walls

Figure 2: Mention sentiment for Obama vs McCain on Facebook walls. A 70% score means that 70% of the mentions had a positive sentiment and 30% had a negative sentiment.
I am not eligible to vote this year since I procrastinated in applying for citizenship (I am due to apply this year since I had the Green Card for more than five years now). That said, if I was eligible to vote, I would vote for Obama, and the reason would be this video from Colin Powell:

Frankly, the US made a mistake by electing Bush for president in 2000, but re-electing him in 2004 was like giving the finger to the rest of the world. Electing Obama shows that the US is (1) sorry for re-electing Bush, and (2) getting over racial/religious/ethnic intolerance, these two factors would open the door for the world to accept the US back as an unbiased super power. Last, but not least, Obama just comes across as smart guy, and we didn’t have an intelligent US president for some time now

Cheers,

– amr

Share it: related | del.icio.us | digg it! | reddit!

Cloudera Hadoop and Big Data Blog is now live.

amr — Mon, 27 Oct 2008 07:03:48 +0000

Cloudera just launched a Hadoop and Big Data blog, you can access it at the following URL:

http://www.cloudera.com/blog

The purpose of the blog will be to post tips and tricks on how to tune Hadoop system performance and how to work with big data in general.

Jeff Hammerbacher just posted his latest “Open Source Data Management Software” presentation on the blog. In the presentation he covers Thrift, Scribe, Hive, and Cassandra, you should check it out.

Cloudera also launched Hadoop Support, Hadoop Resources and Hadoop Training sections, we are still in the process of beefing them up.

Cheers,

– amr

Share it: related | del.icio.us | digg it! | reddit!

The Startup is Cloudera, the Business is Hadoop MapReduce

amr — Tue, 14 Oct 2008 03:10:08 +0000

The new company I am starting is called Cloudera, it will be offering support for Hadoop, think RedHat for Hadoop, but that is just the beginning.

My co-founders are a bunch of really interesting folks:

Mike Olson was CEO of Sleepycat, which commercialized Berkeley DB, the open source embedded database engine. Mike spent two years at Oracle after they acquired SleepyCat in 2006. You can read this article about Mike from InformationWeek to find out more.
Christophe Bisciglia just left Google, he created and managed their Academic Cloud Computing Initiative, which is a public hadoop cluster for academia/research. You can read this article about Christophe from BusinessWeek, his photo was featured on the cover of that issue.
Jeff Hammerbacher just left Facebook, where he started and managed the Data Team. The team made a lot of contributions to Hadoop under his stewardship, most important of which is Hive (a SQL structured data layer on top of Hadoop). You can read Jeff’s Mainly Data Blog here, and you can also watch a video of him giving a talk about Facebook at Yahoo.

In addition to these folks, we also started building out our founding engineering team: Aaron Kimball joined us as a full-time software developer, and so did Tom White. Alex Loddengaard joined us as an intern, and Matei Zaharia joined us as a part-time consultant. We are hiring a couple more engineers, but that’s it for now. The skills required are a solid understanding of large-scale distributed systems and strong C++/Java programming experience, we also have one position for an outstanding front-end developer, specifically Javascript/AJAX.

We are in the process of wrapping up our funding (we don’t need any more cash at this point), and will soon announce our investors and technical advisors.

Cheers,

– amr

Share it: related | del.icio.us | digg it! | reddit!

Lessons I learned while growing at Yahoo!

amr — Fri, 03 Oct 2008 05:47:38 +0000

I spent 8.5 years at Yahoo, and grew up the corporate ladder from a senior engineer to VP engineering. I had many wonderful bosses at yahoo (Qi Lu, Tim Cadogan, Andrew Braccia, and Venkat Panchapakesan), whom helped mentor and grow me. I also attended many career development training classes (an excellent benefit at Yahoo!) and read books like “The Seven Habits Of Highly Effective People” and “First, Break All The Rules”.

Below I list some of the key lessons I learned and tried to practice while at Yahoo:

Lead By Example: Don’t ask your troops to do something that you aren’t willing to do, if you want them to work hard, then you gotta work even harder than them.
Keep Your Hands Dirty: Don’t be a pure people manager, roll up your sleeves and jump in to help the troops. This helps you earn the respect of your team, but also allows you to call BS when you see it .
Eat Your Own Dog Food: To really work on a product and make it better, you need to use that product at least once. I saw many people working on Yahoo products that never used those products, that is wrong.
Think Solutions, Not Problems: I saw many people complaining about problems and getting completely hung up. As soon as you encounter an obstacle you should move on to think what is way around it, how to resolve it, as opposed to getting locked up on fact that the problem is there.
Never Be Shy to Ask And Learn: It is ok to admit that you don’t know something and ask your reports to educate you about it. I sometimes saw people with egos that forced them into pretending they knew when they didn’t, that made them look even more foolish.
Be Honest To Yourself: You gotta have your own opinions about things, even if they contradict what your boss or company wants to hear. You shouldn’t be rude, or just contradict for the sake of it just be honest and stand by your principles and passion.
We Are Statistically Insignificant: Reaching product decisions by using proper analytics of user traffic is much better than going with your instinct because you’re statistically insignificant in the grand scheme of things, no matter how smart you are. I’m not saying never use your instincts, you should, but if the numbers say otherwise then be comfortable changing your opinion.
Be Self Motivated, Go The Extra Mile: If you want to grow career wise, you have to be self motivated, you have to come up with ideas on how to improve things without being asked to do so. You also have to go the extra mile when needed, work extra without being asked to, work on something that other neglected even if it was not your assignment to do it, etc.
Don’t Just Go With The Flow: It is very important you know where you want to go, and what you want to be working on, don’t just sit there and wait for things to happen. Similarly, know what projects and what things you want to accomplish in a given quarter and have a score sheet to measure your self against.
Multitask and Prioritize: It is important to know how to work on more than one thing at the same time, keep a todo list and highlight the important things that need to be done first. Nothing is more satisfying than scratching done items off a sheet of paper.
Listen, Reflect, Think, then Talk: When somebody tells you something that you don’t necessarily agree with, don’t be impulsive and jump back directly with a counter-response. You should digest what they said, repeat it back at them in your own words to make sure you understood what they really meant (this also helps them feel you heard them), think about your response, then finally express your opinion. This was one of the hardest lessons for me to learn, took a lot of mentoring in the early years, even at the end still .
Take Care Of Your Peeps: A good manager should know all of his reports by name and take good care of them, he should remove all obstacles in front of them so they can excel. He should realize their strengths then put them in roles that leverage that, conversely, he should realize their weaknesses then help them improve or move them out of roles that depend on those skills. That said, if after all that they do not deliver, then they don’t deserve to be one of your peeps .
Rally The Troops: It is very important that your group has a clear mission that every body is excited about and is building towards, even if they are only contributing little towards that goal. Clear goals like that draw people together, motivate them, and give them a sense of pride (e.g. “The Moon Landing Mission” ).

Cheers,

– amr

Share it: related | del.icio.us | digg it! | reddit!