Why isn’t the Apple TV Social?

Apple just launched its own social network – Ping. Why isn’t Ping data featured on the Apple TV?
I’d like see what my friends watched, get recommendations and share stuff I Like.

Where are the Apps?!

The iPhone, iPod and iPad are thriving on a vibrant Apps market. Why can’t the Apple TV do the same?
I was really expecting an iOS driven device where developers could enhance the Apple TV capabilities by providing apps specifically designed to be run on TV set with limited controls.

You could browse social networks (Facebook, Twitter etc.), or watch stock market information, or browse content from your favorite feeds and content providers (imagine something like the iPad’s Flipboard your big LCD TV), or cook with your TV’s help (imagine Jamie Oliver’s iPhone app on the big screen) or get apps from content providers (like local news channels for example) that’ll be able to stream their own content…
There’s tons of stuff developers could do with the Apple TV if given the option.

Apple could have really taken the streamers to a whole new level with this…

In short, I wish Apple TV would be more like Boxee

Similar Posts:

• What would Twitter do with $100 million?
• 3 Takeaways from the Apple iPad Launch
• iPhone vs. Droid
• Why Coming Up With an iPhone Killer Will Be Tough…
• High Performance at Massive Scale – Lessons learned at Facebook

Its not a geek device, its a computing appliance. The iPad is a computer with an iPhone OS. Not a full fledged Os like we’re used to, no multitasking, terminal, filesystems… 
Just a list of Apps that can be installed and updated from the net. Simple, elegant, and exactly what most people who aren’t computer geeks need.
The perfect “laptop” for mom and dad…

Apple Sells Relationship, Not Hardware

The iPad’s amazing pricing just shows that Apple is really counting on people using the heck out of their device… that means buying apps, books, music and videos. Its building a relationship with its customer that gets stronger and stronger with each purchase on iTunes.

Customers who have all their applications, games, books, music and videos on iTunes are locked in the Apple ecosystem. They’re vested in it…
They’re not going to easily switch to an Android or a Zune…

HP\Dell\Asus\etc. only make money the moment you purchase their hardware (and on support and stuff) they make no difference if you use it or not. Microsoft too, only makes money when you buy your Windows license.

Apple on the other hand keeps monetizing its customers way after they left the Apple Store with their latest new device – when they buy content for their device. They keep and nurture a profitable relationship with their customers and thats a way better business than a one-off hardware\license sale…

(btw, Microsoft is learning about making profit from a relationship too… that’s what drives its Xbox business)

iBooks Can Change the Publishing Industry

Before the iTunes Music store, buying songs at a ridiculous 99c price was inconceivable. The iPhone App Store did the same to applications, changing the the economy (checkout this excellent App Economy graphic) of application taking prices down.

Same could (and would probably) happend with books…  but there’s more!

Unlike Amazin’s Kindle (and Sony’s reader and the rest of the bunch) which support a very limited interaction – text and some grey imagery – the iPad comes with a big, colorful touchscreen and a CPU that can handle 3D gaming.
The iPad’s hardware is perfect for interactive content and the Times Magazine demo showed during the launch shows a glimpse of  how our future books and magazines should look and feel on electronic media.
Electronic “print” is going to be much more interactive and rich which means the entire process of book production changes.
The publishers’ role changes from mass printing and delivery to production of interactive content, the way authors work changes, and distribution costs drop…

Similar Posts:

• What the new Apple TV is really missing…
• Why Coming Up With an iPhone Killer Will Be Tough…
• iPhone vs. Droid
• Building an iPhone Application
• Moving Your Application to Amazon’s Cloud

Store and Serve – Files can be uploaded and stored as blobs, to be served later in response to user requests. Developers can build their own organizational structures and access controls on top of blobs.

The way this API works is pretty simple. To upload files you can an API that manufactures a POST URL that web forms requests containing files data are submitted to. App Engine processes the POST request and created the blobs in its storage (and BlobInfo objects – readonly datastore entities containing the metadata on each blob). It then rewrites the request, removing the uploaded files data and replacing them a Blobstore key pointing to the stored blob in the App Engine Blobstore, and calls your handler with this data.

To serve an existing blob in your app, you put a special header in the response containing the blob key. App Engine replaces the body of the response with the content of the blob.

Now this is pretty straightforward but there are few concerns with this approach:

1. What about request validation (authentication\authorization etc.)?

When uploading files, the request reaches your code only after blobs have already been processed and stored. This means that you can only handle authentication\authorization or even form validation after data has been stored.

This means you’ll have to write code to clean the relevant blob entries in case of failed authentication\authorization\validation – more datastore API calls, more CPU…

It also means that without taking care of these special cases any newbie hacker with a simple snifter (or FireBug) can start uploading (and potentially) serving files off your service (see update).

2. No way to preprocess data

As the files data is already stored prior to the program’s handler being called, there’s no way to preprocess submitted data other than reading it from the store, processing it and storing it again.

There’s also no straightforward API to access or store blob data in code, so the above process has to be implementing using URL fetching (fetch the image via http call, process it, store it again using http POST call)

There must be a way for the Google App Engine team to wrap this app nicely and provide a clean API for this to be done efficiently (along with solving the validation problem described before)

As the Blogstore API is still in experimental phase I guess we’ll see some quick progress made on its development and hopefully the Google team will solve the issues above.

Atleast now there’s a beginning of an alternative to Amazon S3 for AppEngine applications.

Update:

Bret Slatkin notes that when the API manufactures the POST URL to be used for uploading the files, it creates a unique one-time URL which which mitigates any potential sniffing.
This fits perfectly for the scenario when you’re rendering a web form to be submitted by the user. But, it makes things harder if you’re trying to provide a REST API that allows uploading files (think of something like TwitPic for example). In this case you’ll have to write your own render that simulates what a web form would do (get the files, create random POST URL, call it, …)

Similar Posts:

• Building an iPhone Application
• Moving Your Application to Amazon’s Cloud
• ASP.NET MVC RSS Feed Action Result
• Microsoft’s Next Killer OS is… SharePoint?
• Microsoft calls OpenID a De Facto Login Standard

I found the following comparison between the iPhone and the Droid ads hilarious.

Especially, the following Droid bullets:

• It is fast and it despises aesthetics.
• It is packaged inside missiles launched by stealth jets. (*)
• It is a robot and should mostly be handled by other robots.
• Droid is to be used with robotic hands in a low-lit hi-tech laboratory or warehouse.

Actually, in one of the ads they say they Droid is like a Scud, a soviet missile that’s not known for its accuracy…

In any case, I’ve used an Android device before (the Samsung Galaxy) and I can’t really find anything good to say about the device or the Android OS (and its Apps Market).
I can’t believe there are people in the blogsphere calling the Galaxy and Droid an iPhone alternative… 

Similar Posts:

• What the new Apple TV is really missing…
• Why Coming Up With an iPhone Killer Will Be Tough…
• 3 Takeaways from the Apple iPad Launch
• Facebook, Hadoop and Hive
• Creating an Attractive Internet Company – It’s All About Emotions

There are three questions you have when you’re hiring a programmer (or anyone, for that matter):

• Are they smart?
• Can they get stuff done?
• Can you work with them?

Someone who’s smart but doesn’t get stuff done should be your friend, not your employee. You can talk your problems over with them while they procrastinate on their actual job.

Someone who gets stuff done but isn’t smart is inefficient: non-smart people get stuff done by doing it the hard way and working with them is slow and frustrating.

Someone you can’t work with, you can’t work with.

I think its a much better and more effective approach than the traditional method of asking cheesy annoying riddles and problems…

Similar Posts:

• No Hope for Traditional Media Companies?
• 99 Ways to Become a Better Developer
• Give Up Control, Think Distributed – DLD 2009 Summary
• Office Web Applications
• A Visit to Maraboo, Restaurants as Tribes in the Digital Age

(via Ido Keinan)

Similar Posts:

• WordCamp Israel 2008
• High Performance at Massive Scale – Lessons learned at Facebook
• Weekend Inspirations – Big Data Visualizations, Innovative Man-Machine Interactions
• Microsoft Can Clone Twitter?!
• The Morning After (Stressed Out Blogger Lashes Out On Twitter)

First some information about the team

…because technology choices are affected by the team’s technical skillset.

• We’re 3 developers (Yosi, Udi and myself) and one designer (the awesome Naor Suki).
• We’ve allocated two developers for the iPhone and one for the backend APIs & website.
• We’re all veteran developers with experience mostly on Microsoft’s Development stack. This project meant going out of our comfort zone to a whole new set of technologies. Experience does make a difference easing the learning curve…

iPhone Development

• iPhone SDK: This one is obvious right? We looked for alternatives for writing Objective-C. Unfortunately, Flash isn’t available for the iPhone (yet?) and MonoTouch looked promising but isn’t quite there…
  Besides its always better to be developing on the platform most developers are using which means there’s a big community that can help when you get stuck. Being on Apple’s official stack also means we get the latest features without having to wait for a 3rd party to convert them…
  To be perfectly honest, I’m not on the iPhone side of the development and did not actually write a single line of Objective-C code but I noticed it took my teammates 1-2 weeks to get the hang of it.
  To me, the fact that the iPhone App store is so successful and has so many apps which Objective-C as the development language (which is definitely harder than modern languages – Java etc.) makes Apple’s achievement even more amazing…
  • If you’re beginning iPhone development you have to checkout the Stanford iPhone development class on iTunes University – http://www.stanford.edu/class/cs193p/cgi-bin/index.php
• Three20: Handful of UI extractions from the Facebook iPhone app and open-sourced by the developer – joehewitt. His announcement blog post details the libraries it contains, shows some demos etc.
  The source code is also a great learning tool for how stuff is done on the iPhone.
• json-framework: This is a pretty slick JSON parser for the iPhone. Hand parsing JSON in obj-C would not have been fun. This made it easy. I’m pretty sure I followed this tutorial to get it up and running.
• ASIHttpRequest: A Nice Http framework that enabled easily handling asynchronous Http requests.
• Stackoverflow is an invaluable resource for asking question and solving all sort of problems. As Yosi, who’s been concentrating on the iPhone side of our development, puts it “I dont think any iPhone development could be done without StackOverflow”
• MGTwitterEngine: an awesome objective-c wrapper for twitter api which we based our api on.
• Google Analytics SDK: A library enabling sending information to Google Analytics from the iPhone. This is important for measuring the ways users interact with certain flows on our program. For example, it helps us measure the conversion on our signup flow – how many users go through the signup flow and finish, and if they dont, what steps makes them go away?
  This kind of functionality is essential to measuring and improving UX flows…
• Google Toolbox for Mac: A library for working with the different services exposed by Google.

Backend Development

After playing around with ASP.NET MVC (which we all had background with having come from the Microsoft ecosystem) and Ruby on Rails (because its cheaper to host than ASP.NET and way simple, faster, more fun to use IMHO) we’ve finally settled for Google AppEngine and django (Python).

We made the decision to base our development on django rather than on Google’s own webapp framework for the following reasons:

• Lots of out of-of-the-box features. django has been out there for quit a while and is bundled with lots of features (like an easy to build admin interface, authentication system, validation system etc.)
• Big community. There are lots of people doing django out there lots of open source libraries and samples available. As a rule of thumb its always better to be on the majority side…
• Not specific to Google AppEngine. django is a standalone Python web development platform. While some parts of our code has to be AppEngine specific it would still be considerably easier to move away from AppEngine (if we ever decide to do so) than if we were entirely Google specific.

Google AppEngine also have Java support. But using Python with django is way easier and has a lot more support when it comes to both AppEngine and web development. Seriously, if you’re thinking of using Java, Don’t! take the leap and go with Python…

Libraries we’ve used:

• app-engine-patch: This library is absolutely amazing and a must if you’re using django on AppEngine. Since the AppEngine data store API is not compatible with django’s API, a lot of the really cool time-saving features of django will simply not run on AppEngine (such as the admin UI, authentication and basically anything that requires data access). app-engine-patch loads django and patches it so it is compatible with the AppEngine API making all those cool django features work. This one is a must! You just download their project template and start developing your application on top of it.
• PyDev: An Eclipse plugin for editing and debugging Python and AppEngine applications. It might sound obvious but I was using was actually using Notepad++ (on Windows)  for development until I found out there’s a decent IDE I could use…
• Piston: a django library for developing REST-APIs. While its not entirely compatible with AppEngine it took a simple forking to edit those parts out…
• GeoModel: provides basic indexing and querying of geospatial data on Google AppEngine.

Also, I would recommend taking the time to learn and understand how the AppEngine datastore works so you’ll understand how to build your datamodel to run efficiently on Google’s platform.
The following two presentations from Google I/O are invaluable:

• Building Scalable, Complex Apps on App Engine (pdf)
• App Engine Datastore Under the Covers
• Building Scalable Web Applications with Google App Engine
• Working with Google App Engine Models

So what do you think? If you’re developing an iPhone app, I’m very interested to know what were your technology choices and reasoning…

Oh, and if you have an iPhone and you live in Israel (its a local app so we’re limiting our efforts to Israel at the moment) please head over to our beta signup form and signup

Similar Posts:

• The New Google App Engine Blobstore API – First Thoughts
• Google Chrome is a Failure… Surprised?
• A Year’s Worth of Popular Posts
• Why Coming Up With an iPhone Killer Will Be Tough…
• 99 Ways to Become a Better Developer

Facebook has grown into one of the largest sites on the Internet today serving over 200 billion pages per month. The nature of social data makes engineering a site for this level of scale a particularly challenging proposition. In this presentation, I will discuss the aspects of social data that present challenges for scalability and will describe the the core architectural components and design principles that Facebook has used to address these challenges. In addition, I will discuss emerging technologies that offer new opportunities for building cost-effective high performance web architectures.

I’m halfway through watching it and there are already several interesting point worth a detailed post later on.
If you want to learn how Facebook manages 30K+ machines, 300 million active users, 20 billion photos, and 25TB/day of logging data you should go and watch the talk’s webcast.

Similar Posts:

• Facebook, Hadoop and Hive
• What the new Apple TV is really missing…
• Data Mining – Handling Missing Values the Database
• Weekend Inspirations – Big Data Visualizations, Innovative Man-Machine Interactions
• New Features for Amazon EC2 – Now You Can Truly Scale Applications

For context, that is almost double the market capitalization of Domino’s Pizza, which has 10,500 employees and had $1.4 billion in sales last year. Twitter has some 60 employees, and although it is experimenting with running advertisements on its Web site, Biz Stone, a Twitter founder, said this week at an industry conference that the company had no plans to begin widely running ads until 2010.

Twitter previously raised $55 million and has said it still has $25 million of that in the bank. So the question is, what will it do with these $100 million? Or, as I see it, who will it acquire now?

As part of it efforts to find a business model, Twitter will most likely acquire companies that’ll help it form that model.I’m thinking\betting on two major trends for such a business model:

Manage Companies Presence on Twitter

Twitter’s most obvious business model is by helping companies manage their presence on Twitter and monitor how their brands are being discussed.

This makes companies who provide all sort of analytics information, CRM integration and even url-shorteners as potential acquisitions for a future Twitter business package…

Local Markets, Local Social Network

The minute I read about Twitter’s $100 million round I thought of companies like Foursquare. I wasn’t really surprised when I read today’s Techmeme and noticed Twitter’s co-founder Jack Dorsey invested in Foursquare.
Also, the twitter team has been working very hard lately to make Twitter location aware, allowing users to share their location via their tweets and browse stuff that is happening around them.

Twitter is great in forming local communities (just checkout the local tweetups everywhere), gather news and provide all sort of local information. Fouresquare as well as other location based social networks doing can really help twitter tap into the long-tail local businesses market and take on companies such as Yelp.

So, what do you think Twitter’s latest valuation? What will it do with its newly raised $100mil?

Similar Posts:

• No Hope for Traditional Media Companies?
• Microsoft Can Clone Twitter?!
• Give Up Control, Think Distributed – DLD 2009 Summary
• What the new Apple TV is really missing…
• A Visit to Maraboo, Restaurants as Tribes in the Digital Age

One of the important stages of data mining is preprocessing, where we prepare the data for mining. Real-world data tends to be incomplete, noisy, and inconsistent and an important task when preprocessing the data is to fill in missing values, smooth out noise and correct inconsistencies.

If we specifically look at dealing with missing data, there are several techniques that can be used. Choosing the right technique is a choice that depends on the problem domain – the data’s domain (sales data? CRM data? …) and our goal for the data mining process.

So how can you handle missing values in your database?

1. Ignore the data row

This is usually done when the class label is missing (assuming you data mining goal is classification), or many attributes are missing from the row (not just one). However you’ll obviously get poor performance if the percentage of such rows is high.

For example, lets say we have a database of students enrollment data (age, SAT score, state of residence, etc.) and a column classifying their success in college to “Low”, “Medium” and “High”. Lets say our goal is do build a model predicting a student’s success in college. Data rows who are missing the success column are not useful in predicting success so they could very well be ignored and removed before running the algorithm.

2. Use a global constant to fill in for missing values

Decide on a new global constant value, like "unknown", "N/A" or minus infinity, that will be used to fill all the missing values.
This technique is used because sometimes it just doesn’t make sense to try and predict the missing value.

For example, lets look at the students enrollment database again. Assuming the state of residence attribute data is missing for some students. Filling it up with some state doesn’t really makes sense as opposed to using something like “N/A”.

3. Use attribute mean

Replace missing values of an attribute with the mean (or median if its discrete) value for that attribute in the database.

For example, in a database of US family incomes, if the average income of a US family is X you can use that value to replace missing income values.

4. Use attribute mean for all samples belonging to the same class

Instead of using the mean (or median) of a certain attribute calculated by looking at all the rows in a database, we can limit the calculations to the relevant class to make the value more relevant to the row we’re looking at.

Lets say you have a cars pricing database that, among other things, classifies cars to "Luxury" and "Low budget" and you’re dealing with missing values in the cost field. Replacing missing cost of a luxury car with the average cost of all luxury cars is probably more accurate then the value you’d get if you factor in the low budget cars.

5. Use a data mining algorithm to predict the most probable value

The value can be determined using regression, inference based tools using Baysian formalism , decision trees, clustering algorithms (K-Mean\Median etc.).

For example, we could use a clustering algorithms to create clusters of rows which will then be used for calculating an attribute mean or median as specified in technique #3.
Another example could be using a decision tree to try and predict the probable value in the missing attribute, according to other attributes in the data.

I’d suggest looking into regression and decision trees first (ID3 tree generation) as they’re relatively easy and there are plenty of examples on the net…

Additional Notes

• Note that methods 2-5 bias the data as the filled-in value may not be correct.
• Method 5 uses the most information available in the present data to predict the missing value so it has a better chance for generating less bias.
• Missing value may not necessarily imply an error in the data! forms may contain optional fields, certain attributes may be in the database for future use.

Similar Posts:

• Introduction to MapReduce for .NET Developers
• Bill Gates and Jerry Seinfeld, Take Two
• Moving Your Application to Amazon’s Cloud
• Facebook, Hadoop and Hive
• Amazon S3 Storing 29 Billion Objects

Keeping in mind that there are a lot of applications out there who can benefit from moving to the cloud (including my own) I’ve made a list of tasks\considerations to make when preparing for such a move:

Step One: Move Static Content to S3

The first and easiest step is to move all your static content – images, CSS, JavaScript files, etc. – to Amazon S3. Let Amazon worry about storage, backups and availability for you.

Things to consider:

• GZIP content. S3 does not support serving GZIPed content natively so you’ll have to upload both GZIPed and plaintext version of each file and figure out in code which one to use according to the headers sent by the user’s browser.
  The following post describes how its done: Using Amazon S3 as a CDN
• Amazon’s S3 service is “eventually consistent” which means that files uploaded to S3 may not be immediately available to read.
• Use separate sub-domains for content.

Once your content is on S3 you can also use CloudFront, Amazon’s CDN (Content Delivery Network), to serve the files and improve your application’s performance.

Step Two:  Move Web Servers and Backend Servers to EC2

Move your web server code and backend services – database, memcached, etc. – to run on Amazon EC2 instances.

Consider using Amazon’s availability zones to setup servers in different availability zones. This can help your serve customers at different parts of the worlds better, while making your infrastructure tolerant to the unlikely event of a datacenter failures at Amazon.

The Web Servers

Moving your web servers to EC2 should be fairly simple. You can setup EC2 images that are configured exactly the same way your current web servers are.

If you require a queuing service as part of your architecture, consider switching to Amazon’s SQS to make administration easier.

The Database

Moving your database to EC2 is probably the hardest part of the move to AWS. If you plan on keeping your database (as opposed to migrating to a cloud solution like SimpleDB) you should use EBS (Elastic Block Storage) so that your storage is persists independently from the life of your EC2 instance.

Backup. Figure out how to take scheduled snapshots of your EBS and store them on S3.

Consider replication and sharding. If you’re using availability zone you should consider sharding your data. For example, store data European accounts data in Europe only. You should also consider replication between the different availability zones to ensure keep your site available even when one of the datacenter is unavailable.

Related Links:

• Running MySQL on EC2 with EBS
• EC2 SQL Server backup strategies and tactics
• Create a MySQL-EBS Database Setup

Step Three: Scale. Take advantage of the cloud services

Now that your application is entirely running on Amazon’s platform it’s time to take the full advantage of the platform and make it scale.

Setup Monitoring to keep up with what’s going on on your system. Amazon provides a service called CloudWatch that allows you to monitor your machines and applications.

Based on the monitoring metrics you should start using Amazon’s auto-scaling and load balancing capabilities to be able to consume and release computing resources according to demand.

At this point you should also investigate reducing your dependency on relational databases (RDMS) as much as possible (as its the most complex, and hardest to scale, component in the system) and try to move as much functionality as possible to S3 and SimpleDB.
S3 is suitable for storing large objects while SimpleDB is ideal for small stubs of data.

Important notes:

• Amazon’s load balancer doesn’t support SSL. This can be a showstopper for some applications…
• Simple DB has a max row size limitation. If your data exceeds that limit you should consider using SimpleDB as a metadata store that references the full data stored on S3.

Related Links:

• Load Balancing, Auto Scaling and CloudWatch resources on the AWS blog.

Similar Posts:

• New Features for Amazon EC2 – Now You Can Truly Scale Applications
• Amazon S3 Storing 29 Billion Objects
• Microsoft’s Next Killer OS is… SharePoint?
• The New Google App Engine Blobstore API – First Thoughts
• Cloud Avenue

The following two posts shed some more light on why Facebook chose the Hadoop\Hive path, how they’re doing it and the challenges they’re facing:

Facebook, Hadoop, and Hive on DBMS2 by Curt Monash discusses Facebook’s architecture and motivation.

Facebook decided in 2007 to move what was then a 15 terabyte big-DBMS-vendor data warehouse to Hadoop — augmented by Hive — rather than to an MPP data warehouse DBMS…

The daily pipeline took more than 24 hours to process. Although aware that its big-DBMS-vendor warehouse could probably be tuned much better, Facebook didn’t see that as a path to growing its warehouse more than 100-fold.

Hive – A Petabyte Scale Data Warehouse using Hadoop by Ashish Thusoo from the Data Infrastructure team at Facebook discusses Facebook’s Hive implementation in details.

… using Hadoop was not easy for end users, specially for the ones who were not familiar with map/reduce. End users had to write map/reduce programs for simple tasks like getting raw counts or averages. Hadoop lacked the expressibility of popular query languages like SQL and as a result users ended up spending hours (if not days) to write programs for typical analysis. It was very clear to us that in order to really empower the company to analyze this data more productively, we had to improve the query capabilities of Hadoop. Bringing this data closer to users is what inspired us to build Hive. Our vision was to bring the familiar concepts of tables, columns, partitions and a subset of SQL to the unstructured world of Hadoop, while still maintaining the extensibility and flexibility that Hadoop enjoyed.

Similar Posts:

• Yahoo Releases Its Own Hadoop Distribution
• High Performance at Massive Scale – Lessons learned at Facebook
• Introduction to MapReduce for .NET Developers
• Weekend Inspirations – Big Data Visualizations, Innovative Man-Machine Interactions
• Microsoft calls OpenID a De Facto Login Standard

Hadoop is a distributed file system and parallel execution environment that enables its users to process massive amounts of data.
In response to frequent requests from the Hadoop community, Yahoo! is opening up its investment in Hadoop quality engineering to benefit the larger ecosystem and to increase the pace of innovation around open and collaborative research and development.
The Yahoo! Distribution of Hadoop has been tested and deployed at Yahoo! on the largest Hadoop clusters in the world.

Hadoop is free Java software framework born out of an open-source implementation of Google’s published computing infrastructure and fostered within the Apache Software Foundation.
Yahoo! has been the primary developer and contributor to Apache’s Hadoop.
In 2006, Hadoop founder Doug Cutting joined Yahoo, which provided a dedicated team and resources, to lead the project of developing the open-source software and turn Hadoop into a system that ran at web scale. Today, Yahoo! is running the largest Hadoop cluster in the world, which includes more than 25,000 servers and provides the framework for many Yahoo properties including Yahoo Search, Yahoo Mail, and several content and ad services.

Yahoo says its opening up the source code to Hadoop to “benefit the larger ecosystem increase the pace of innovation around open and collaborative research and development.”.
As Nigel Daley, Quality and Release Engineering Manager at Yahoo! Grid Technologies, summarizes:

Hadoop is helping us solve key science and research problems in hours or days instead of months. It provides us a platform to solve extreme problems requiring massive amounts of data processing. It underpins major revenue-generating systems. Opening our distribution enables a faster pace of innovation for the entire Hadoop ecosystem and broadens the use — and ultimately the quality — of this key platform across the industry.

Similar Posts:

• Facebook, Hadoop and Hive
• 99 Ways to Become a Better Developer
• Cloud Avenue
• High Performance at Massive Scale – Lessons learned at Facebook
• Introduction to MapReduce for .NET Developers

While Amazon’s EC2 has been great so far in allowing companies to run large, computational-heavy distributed tasks on the background, it has been really lacking on allowing companies to run online services in a manner that’s efficient and reliable.

The reason for this lack of support is basically because there was no way to do load balancing on EC2 machines allowing a service to efficiently scale across multiple EC2 instances.

With the launch of Amazon CloudWatch, Auto Scaling and Amazon Elastic Load Balancing, Amazon is effectively making EC2 a viable host for online services, not just background jobs. Using these services you can not only scale a service across machines for reliability, you can also automatically add and remove computing resources as demand increases\decreases driving your operation costs down.

You can find more information at the detail pages for Amazon CloudWatch, Auto Scaling and Elastic Load Balancing and on the AWS developer weblog. Also, check out Werner Vogel’s blog post for some background on how Amazon is horizontally scaling its services.

Similar Posts:

• Moving Your Application to Amazon’s Cloud
• Amazon S3 Storing 29 Billion Objects
• Microsoft Updates Its Windows Live Services
• Introduction to MapReduce for .NET Developers
• Why Coming Up With an iPhone Killer Will Be Tough…

MapReduce, inspired by these concepts, was developed as a method for writing processing algorithms for large amounts of raw data. The amount of data is so large that it can’t be stored on a single machine and must be distributed across many machines in order to be processed in a reasonable time.
In systems with such data distribution, the traditional central processing algorithms are useless as just getting the data to the centralized CPU running the algorithm implies huge network costs and months (!) spent on transferring data from the distributed machines.
Therefore, processing such massive scales of distributed data implies the need for parallel computing allowing us to run the required computation “close” to where the data is located.
MapReduce is an abstraction that allows engineers to write such processing algorithms in a way that is easy to parallelize while hiding the complexities of parallelization, data distribution, fault tolerance etc.

This value proposition for MapReduce is outlined in a Google research paper on the topic:

MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper.

Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program’s execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.

Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: hundreds of MapReduce programs have been implemented and upwards of one thousand MapReduce jobs are executed on Google’s clusters every day.

The MapReduce Programming Model

As explained earlier, the purpose of MapReduce is to abstract parallel algorithms into a map and reduce functions that can then be executed on a large  scale distributed system.
In order to understand this concept better lets look at a concrete map reduce example – consider the problem of counting the number of occurrences of each word in a large collection of documents:

map(String key, String value):
// key: document name
// value: document contents
for each word w in value:
  EmitIntermediate(w, "1"); 

reduce(String key, Iterator values):
// key: a word
// values: a list of counts
int result = 0;
for each v in values:
  result += ParseInt(v);
Emit(AsString(result));

The map function goes over the document text and emits each word with an associated value of “1”.

The reduce functions sums together all the values for each word producing the number of occurrences for that word as a result.

First we go through the mapping phase where we go over the input data and create intermediate values as follows:

• Records from the data source (lines out of files, rows of a database, etc.) are fed into the map function as <key,value> pairs.For example: <filename, file content>
• The map function produces one or more intermediate values along with an output key from the input

After the mapping phase is over, we go through the reduce phase to process the intermediate values:

• After the map phase is over, all the intermediate values for a given output key are combined together into a list and fed to the reduce function.
• The reduce function combines those intermediate values into one or more final values for that same output key

Notice that both the map and the reduce functions run on independent set of input data. Each run of the map function process its own data source and each run of the reduce function processes the values of a different intermediate key.

Therefore both phases can be parallelized with the only bottleneck being the fact that the map phase has to finish for the reduce phase to start.

The underlying system running these method is in takes care of:

• Initialize a set of workers that can run tasks – map or reduce functions.
• Take the input data (in our case, lots of document filenames) and send them to the workers to map
• Streamline values emitted by map function to the worker (or workers) doing the reduce. Note that we don’t have to wait for a certain map run to finish going over the entire file in order to start sending its emitted values to the reducer, so that the system can prepare the data for the reducer while the map function is running
  (In Hadoop – send the map values to the reducer node and andle grouping by key).
• Handle errors – support a reliable, fault tolerant process as workers may fail, network can crush preventing workers from communicating results, etc.
• Provides status and monitoring tools.

A Naive Implementation in C#

Lets see how we can build naive MapReduce implementation in C#.

First, we define a generic class to manage our Map-Reduce process:

public class NaiveMapReduceProgram<K1, V1, K2, V2, V3>

The generic types are used the following way:

• (K1, V1) – key-value types for the input data
• (K2, V2) – key value types for the intermediate results (results of our Map function)
• V3 – The type of the result for the entire Map-Reduce process

Next, we’ll define the delegates of our Map and Reduce functions:

public delegate IEnumerable<KeyValuePair<K2, V2>>   MapFunction(K1 key, V1 value);
public delegate IEnumerable<V3>                     ReduceFunction(K2 key, IEnumerable<V2> values);

private MapFunction _map;
private ReduceFunction _reduce;

public NaiveMapReduceProgram(MapFunction mapFunction, ReduceFunction reduceFunction)
{
    _map = mapFunction;
    _reduce = reduceFunction;
}

(Yes, I realize I could use .NET’s Func<T1,T2,TResult> instead but that would just result in horribly long ugly code…)

Now for the actual program execution. The execution flow is as follows: We take the input values, pass them through the map function to get intermediate values, we group those values by key and pass them to the reduce function to get result values.

So first, lets look at the mapping step:

private IEnumerable<KeyValuePair<K2, V2>> Map(IEnumerable<KeyValuePair<K1, V1>> input)
{
    var q = from pair in input
            from mapped in _map(pair.Key, pair.Value)
            select mapped;

    return q;
}

Now after we got the mapped intermediate values we want to reduce them. The Reduce function expects a key and all its mapped values as input so to do that efficiently we want to group the intermediate values by key first and then call the Reduce function for each key.

The output of this process is a V3 value for each of the intermediate K2 keys:

private IEnumerable<KeyValuePair<K2, V3>> Reduce(IEnumerable<KeyValuePair<K2, V2>> intermediateValues)
{
    // First, group intermediate values by key
    var groups = from pair in intermediateValues
                 group pair.Value by pair.Key into g
                 select g;

    // Reduce on each group
    var reduced = from g in groups
                  let k2 = g.Key
                  from reducedValue in _reduce(k2, g)
                  select new KeyValuePair<K2, V3>(k2, reducedValue);

    return reduced;
}

Now that we have the steps code the execution itself is simply defined as Reduce(Map(input)) :

public IEnumerable<KeyValuePair<K2, V3>> Execute(IEnumerable<KeyValuePair<K1, V1>> input)
{
    return Reduce(Map(input));
}

The full source code and tests can be downloaded from here:

Map-Reduce Word Counting Sample – Revisited

Lets go back to the word-counting pseudo code and write it in C#.

The following Map function gets a key and a text value and emits a <word, 1> key-pair for each word in the text:

public IList<KeyValuePair<string, int>> MapFromMem(string key, string value)
{
    List<KeyValuePair<string, int>> result = new List<KeyValuePair<string, int>>();
    foreach (var word in value.Split(' '))
    {
        result.Add(new KeyValuePair<string, int>(word, 1));
    }
    return result;
}

Having calculated a <word, 1> key-pair for each input source, we can group the results by the word and then our Reduce function can sum the values (which are 1 in this case) for each word:

public IEnumerable<int> Reduce(string key, IEnumerable<int> values)
{
    int sum = 0;
    foreach (int value in values)
    {
        sum += value;
    }

    return new int[1] { sum };
}

Our program code looks like this:

MapReduceProgram<string, string, string, int, int> master = new MapReduceProgram<string, string, string, int, int>(MapFromMem, Reduce);
var result = master.Execute(inputData).ToDictionary(key => key.Key, v => v.Value);

The result dictionary contains a <word, number-of-occurrences> pairs.

Other Examples

Distributed LINQ Queries. One of POCs I’m working on using the above naive, LINQ-based implementation, is running a distributed LINQ query. Imagine you have a system where raw data is distributed across several SQL Servers. We can have our map function run a LINQ-to-SQL query on multiple DataContexts in parallel (the value input for the map function – V1 – can be a DataContext) and then reduce it to a single result set. This is probably a naive\simplified implementation of what the guys at Microsoft’s Dryad team are doing.

Count URL Visits. Consider you have several web servers and you want to produce the amount of visits for each page on your site. You can produce pretty much the same way the word-counting example works. The map function parses a log file and produce a <URL, 1> intermediate value. The reduce function then sums the values for each URL and emits <URL, number of visits>

Distributed Grep. You can run a grep search on a large amount of files by having the map function emits a line if it matches a given pattern. The reduce function in this case is just an identity function that copies the supplied intermediate data to the output.

Map-Reduce in the Real World

The real complexity and sophistication in MapReduce is in the underlying system takes care of running and managing the execution of MapReduce jobs. Real world MapReduce implementations, like Google’s system, Hadoop or Dryad have to go beyond the naive implementation shown here and take care of things like resource monitoring, reliability and fault tolerance (for example, handle cases where nodes running map\reduce jobs crush, or go offline due to network problems).

The following resources are worth checking out:

• Cloudera’s Hadoop Training
• Google Code University – Distributed Systems

Similar Posts:

• Facebook, Hadoop and Hive
• ASP.NET MVC RSS Feed Action Result
• Data Mining – Handling Missing Values the Database
• Yahoo Releases Its Own Hadoop Distribution
• Moving Your Application to Amazon’s Cloud

On the past couple of week we’ve been working on a new, food related venture (more details on that in the very near future).  Apparently, we raised enough twitter noise to get noticed, and thanks to Maya we’ve been contacted by Channel 10 and invited to visit Maraboo as part of the promotions for the season finale of the Israeli version of Icon Chef featuring the Maraboo team vs. the Hudson Brasseri – you should tune in on Channel10 this Thursday (April 30, 2009) at 21:00.

At the restaurant we met the owners, Haim Amshalom and Chef Yoav Bar
who showed us around the place and introduced us to each and every dish we got. We had a great time…

* On the left Haim Amshalom and on the right Chef Yoav Bar

The fact that were at Maraboo as bloggers, that Maraboo has its own group on Facebook and that its owner is on Twitter made me think about the opportunities the web has to offer restaurants.

The Web is About Openness… Can Restaurants be Open?

The Web is about openness, its about empowering online communities and individuals to create, remix, share and make things themselves.
Restaurants are essentially small communities, or tribes of people with a common interest (in our case – food) a leader (the Chef) and a platform to communicate (the restaurant). They can use the web as a platform to develop a personal relationship with its customers and the facility itself (the restaurant) as a social hub.

Restaurants, like any other business, can use Twitter or a Facebook group etc. to keep in touch with their diners but what’s more interesting is that they can actually use it to become a platform, or a hub, for its tribe members (diners) to interact and their common interests (food).

The Menu as a Wiki. Ok, the Chef builds the menu and we wouldn’t want to change that… that’s the whole point of the restaurant. But what if he could collaborate or include the diners into that process?

Maybe people think the Calamari dish is lacking? or maybe they just hate Calamari…  wouldn’t you want to know?

It can put recipes online and invite diners to make suggestions. Some users might go through the trouble of preparing the dish at home. They might make all sort of adaptations and report back, discuss them online with their fellow diners (For example, a lot of people tried to make the same wings sauce as Dixie’s famous sauce and they were discussing it online). Maybe some of these adaptations are actually worth putting in the menu?

In fact, what if browsing a restaurant’s menu was more like browsing Amazon?
You could see which dishes people like more, what they think of it. You could see what your friends like (or not) or filter dishes by “spec” (like calories intake?).

Restaurants can become the stars. The popularity of kitchen-based reality shows is soaring (heck, we’re here as a promotion for such a show) because there’s a public interests in restaurants, in chefs, in food making.
Chefs could blog about their taste, inspirations, trends they see, cooking practices and tutorials and become local stars (take Gary Vaynerchuk’s for example)

Restaurant as a community meeting place. People in online communities like to meet. Just see all the Tweetup events where communities of people who know each other from talking on Twitter gather to talk offline. As its community members all share a passion for food, a restaurant can organize bakeups- where people could learn about cooking\cook. Maybe even have a community member as Chef for a night…

Final Thoughts and.. Food Photos!

As Seth Godin puts it, managing a tribe is a whole different concept than the old way of just managing a brand using traditional PR:

Tribe management is a whole different way of looking at the world.

It starts with permission, the understanding that the real asset most organizations can build isn’t an amorphous brand but is in fact the privilege of delivering anticipated, personal and relevant messages to people who want to get them.

It adds to that the fact that what people really want is the ability to connect to each other, not to companies. So the permission is used to build a tribe, to build people who want to hear from the company because it helps them connect, it helps them find each other, it gives them a story to tell and something to talk about.

Anyway, I can’t end this post without some photos of some of the amazing dishes we had at Maraboo:

And finally, the entire dinning team

Similar Posts:

• Microsoft Can Clone Twitter?!
• No Hope for Traditional Media Companies?
• What would Twitter do with $100 million?
• Is Windows Live Still Alive?
• Give Up Control, Think Distributed – DLD 2009 Summary

Designing “Big Data”

Jeff Veen from Small Batch Inc. gave a 20-minutes talk at the Web2.0 Expo at San Francisco.
During the talk he focuses on some classic examples for information visualization (John Snow pump, Minard‘s map, the tube map, and so on), the challenge of making data more accessible and understandable vs. just “decorating” it and the emerging challenge in Web 2.0 to empower users to find and create their own stories using the data.

And on the same topic of dealing with “big data”, check out the AlloSphere. A system developed by scientists and artists at the University of California for exploring huge sets of data as a visual and audible 3D world:

Cool New Computer Interaction Technology

I saw this amazing UI concept watching the TED podcast:

The concept as computers as tiny blocks that can react to each other is just amazing and the possibilities here, especially in education, are endless…

Kiva Robots Invade the Warehouse

If a bunch of tiny computers interacting can change personal computing here’s what it can do in the enterprise – optimize warehouses supply chain:

The servers work in real-time, receiving orders, immediately dispatching robots to bring the required pods to the worker fulfilling the order, and then returning the pods to their storage locations. The robots receive their orders wirelessly, while using cameras to read navigational barcode stickers on the warehouse floor.

Have a great weekend!Similar Posts:

• Facebook, Hadoop and Hive
• iPhone vs. Droid
• Developing a Robust Data Driven UI Using WPF – An Overdue Summary (and full source code)
• A Visit to Maraboo, Restaurants as Tribes in the Digital Age
• High Performance at Massive Scale – Lessons learned at Facebook

I started documenting the proof-of-concept in a series of posts but unfortunately after leaving SAP my interests (and work) shifted away from WPF and I didn’t find the time to finish the series.

I’ve received numerous requests to release the source code but I couldn’t do so because it was part of a larger infrastructure code I wrote at SAP which basically ads a lot of noise to the sample (an d probably ads legal issues for me sharing it).
Anyway, I took some time off this afternoon to re-write the sample independently so that I could share it:

It can be found on my SkyDrive

This, I guess is the long overdue ending for the series:

• Introduction – introduces the concept of M-V-VM and the reasoning behind it.
• The DataModel – describes how to write the Model part of our application.
• Stock DataModel Sample – provides a conrete implementation of a Stock model and its view.. 

However, If you’re interested in M-V-VM in WPF, there are numerous topics worth mentioning that I didn’t get to cover and are definitely worth checking out:

Unit Testing

As I said in the introduction post, one of the most important benefits of seperating the logic code from the presentation (XAML) is that its straightforward to unit test. In fact, my next post following the Stock DataModel Sample was going to be about unit testing – specifically, how to test the DataModel its provider which, because of the use of threading, is a bit tricky.

This post is actually 99% done in the comments of the unit test code that’s in DefaultStockQuoteProviderTest.cs in the  provided source code. So do yourself a favor and go over the code. It’s not long and very well documented…

Using Lambda Expression for DataBinding

Data-binding is pretty much at the heart of the M-V-VM concept and it makes us write Value Converters which is pretty tedious and annoying.
Wouldn’t it be great if we could replace writing lots of IValueConverter classes like this:

<TextBlock Foreground="{Binding Change, Converter={StaticResource StockForegroundConverter}}" … />

[ValueConversion(typeof(double), typeof(Brush))]
public class StockChangeToBrushConverter : IValueConverter
{
    public object Convert(object value, Type targetType, object parameter, CultureInfo culture)
    {
        double change = (double)value;
        if (change == 0) return Brushes.Black;
        return (change < 0) ? Brushes.DarkRed : Brushes.Green;
    }

    public object ConvertBack(object value, Type targetType, object parameter, CultureInfo culture)
    {
        return double.NaN;
    }
}

To just the following XAML statement that embeds the conversion logic:

<TextBlock Foreground="{Binding Change,

Converter={ change=> if (change == 0) return Brushes.Black; return (change < 0) ? Brushes.DarkRed : Brushes.Green; }}" … />

M. Orçun Topdağı wrote an excellent series on using Lambda Expressions for data-binding in WPF to achieve just that:

• WPFix Part 1 (Lambda Converter Extension)
• WPFix Part 2 (Binding Extension)
• WPFix Part 3 (Extension Methods)

Reference Applications and Guidance

I haven’t seen a lot of sample WPF LOB reference applications out there but here are some interesting links for further learning:

• Prism: patterns & practices Composite Application Guidance for WPF and Silverlight site
• M-V-VM Reference Application – A new project that aims to “created reference application for M-V-VM frameworks to use for demonstration purposes, similar in concept to Pet Shop for web frameworks.”

Similar Posts:

• ASP.NET MVC RSS Feed Action Result
• Introduction to MapReduce for .NET Developers
• Weekend Inspirations – Big Data Visualizations, Innovative Man-Machine Interactions
• Cloud Avenue
• A Visit to Maraboo, Restaurants as Tribes in the Digital Age

…But I’d argue Microsoft could simply do a Twitter clone — the same way that it has built its own Facebook-notification-like news stream into Windows Live — and reap similar results. In fact, the Softies are hinting they’ve already been experimenting with adding Twitter-like functionality to its business software (possibly via SharePoint). I’d bet the Xbox and maybe the Pink/Danger mobile teams have been looking at doing their own Twitter-like services too.

Seriously?! Microsoft could simply clone Twitter?!

Sure, Microsoft certainly has some brilliant tech folks that can surely implement or clone anything. It also has the resources to do that.
And yet, its mostly following the pack with its online offering pretty stagnant. Last time it tried anything like that was Live Home with its Facebook\FriendFeed like functionality and that’s pretty much dead…
It can certainly clone Twitter, probably even do a better job at it than the original, but will anyone bother using it?

Twitter has a fast growing, huge and vibrant community. Its also a well known brand name that’s getting a huge amount of media attention right now.
It survived its own fail whales, upgrade owls and all sorts of other nasty service downtimes simply because no one else can beat that…
You can clone technology but you can’t clone a community and brand strength…Similar Posts:

• Is Windows Live Still Alive?
• What would Twitter do with $100 million?
• A Visit to Maraboo, Restaurants as Tribes in the Digital Age
• The Morning After (Stressed Out Blogger Lashes Out On Twitter)
• Microsoft Updates Its Windows Live Services

The discussion over Google’s latest move into behavioral ad targeting is all over TechMeme. Basically what this means is that Google will start selling users (or more specifically, clusters of users, like football fans for example) in addition to words:

Today we are launching “interest-based” advertising as a beta test on our partner sites and on YouTube. These ads will associate categories of interest – say sports, gardening, cars, pets – with your browser, based on the types of sites you visit and the pages you view. We may then use those interest categories to show you more relevant text and display ads. (…)
So if you visit an online sports store, you may later be shown ads on other websites offering you a discount on running shoes during that store’s upcoming sale.

Google’ is not the first to embrace behavioral targeting as other companies (including Google’s own DoubleClick) were already offering advertisers these features, but it is certainly the biggest.
Behavioral targeting on Google massive network of sites (just think of anyone who’s using Google Analytics, which is pretty much everyone) is certainly a game changing event to the online advertising industry.

The responses online (and ones I usually get talking to people about the topic as we’re doing stuff in this area at Nuconomy) range from simple skepticism of behavioral targeting effectiveness (like Google’s own Matt Cutts…) to Big Brother concerns.

To the latter group, concerned with privacy and worried about its actions online being tracked I have to say – NEWS FLASH! you don’t have privacy on the internet… everything you do is already being tracked, analyzed and can be linked back to you even if stored anonymously (ever heard of Thelma Arnold?), the only question is what do you get out of it?

I hope that with behavioral targeted ads I will see less irrelevant ads offering me low mortgages and seduction courses and more ads geared towards tech gear, maybe even making online ads actually useful (can’t remember the last time I clicked an ad. with time I just learned to automatically ignore these ad banners…). But I think change is not going to hit just the advertising market…

Google’s move can (hopefully) marks the beginning of a personal internet era where sites will customize their entire layout and content based on the preferences, or habits of the browsing users.
My favorite news site will know I care about tech and biz. and never read sports news and will modify its homepage layout based on these preferences, the recommendation widget on my blog will show post recommendation based on the viewers’ recent browsing history rather than based on traffic, and so on…

Behavioral targeting opens the door on a conceptual change, not just in advertising…

Related Resources:

• The Borg’s Cookies Are Delicious, Indeed – Zoli discusses privacy concerns over at Cloud Avenue.
• Google AdSense Starts Behavioral Targeting- Google Blogscoped doesn’t like behavioral targeting…
• Making ads more interesting– Google’s announcement on their blog.
• Behavioral Targeting in Google AdSense 

Similar Posts:

• No Hope for Traditional Media Companies?
• Microsoft Updates Its Windows Live Services
• A Year’s Worth of Popular Posts
• Give Up Control, Think Distributed – DLD 2009 Summary
• Google Chrome is a Failure… Surprised?

But not just any fish, its a Siamese fighting fish, also knows a “betta fish” (or just “betta”).
A subtle Microsoft joke? Hope it does a better job selectively breeding this one (and get rid of some mutations… err… SKUs) …

Similar Posts:

• Is Windows Live Still Alive?
• Office Web Applications
• Microsoft Updates Its Windows Live Services
• Google Chrome is a Failure… Surprised?
• Microsoft Can Clone Twitter?!

As defined by Steffi Czerny "DLD is interdisciplinary, creating interfaces and connecting people from the most different worlds." the conference’s “New Realities” theme encourages participants to discuss and formulates perspectives on markets following an eventful year, setting the agenda for 2009.

Or as Yossi Vardi defines it:

I told people that of my generation no one will understand instant messaging. And I was told my generation was already dead. But our generation has experiences the new generation does not have: real handshakes and real warmth of the hand. This still has to  move into social networks. And this is what we provide here: This old style feeling.
…
We invite you for 4 things:

• to get food for thought
• to make new friends
• to play
• to laugh

Think Distributed

So what is the new reality we’re living in? the repeating theme in most panels I’ve been to was the same:
We’re going through a fundamental change from centralized control (over content, decision making, …) to a distributed structure.

This change becomes very clear when you go over the following panels (and most other panels too):

• 100 Million Uniques
• New Media Models
• On Leadership
• Cloud Computing

All these panels talk about a new reality where, in order to survive, you have to give up on centralized control and go distributed.
On the 100 Million Uniques and New Media Models we hear about new distributed media networks that distribute information rather than creating and controlling it. On cloud computing we learn about technologies allowing companies to lose control over infrastructure for increased agility and lower costs, and on On Leadership we have the story of Best Buy that transformed its culture from “communicating at employees” to empowering their employees using social media tools.

Distributed Media

On New Media Models we’ve heard from Jeff Jarvis who says that the future of media distributed – aggregating content from a lot of independent sources rather than controlling the creation of content in-house.
On his latest book, “What Would Google Do?” (which I got signed at the event. Thanks Jeff! )  discusses the importance of building a platform for content, handing over control to users to build and distribute content, making your distributed network larger.

Traditional media companies think centralized. They create and control content and then spend a fortune on advertising in order to try and attract us, the consumers, to their content. This worked pretty well for big online media companies, like Yahoo for example, so far.
However, consumer today are relying more and more on social, self organization tools, to collect, sort, filter and rate content. Tools like Digg, Twitter, delicious and Facebook replace the functions that content portals used to provide.

In this new reality media companies need to stop thinking of themselves as an end – a closed content site (or portal) users come to in order to find content – and start distributing themselves.
Google distributes itself. It has its widgets, ads, maps and videos embedded in millions of web pages that it doesn’t own and it is making its profit of this vast distribution network.
Another example for distributed content is Glam. On 100 Million Uniques, Samir Arora, CEO of Glam described his company the following way:

“Glam is a distributed network that recognizes the fact that people go to many sites as opposed to a few, and that is the fundamental change that is happening today. So instead of bringing people to one place, we find out where people go and are there.”

“with every day that was passing, fueled by Google, its easier to find more sites, as opposed to one portal.

Instead of producing women targeted content and competing with all the other women content producers out there, Glam finds the best women content producers out there (some are lone bloggers and other can be bigger media companies) and invites them to join its network. It then sells ads on those sites and shares it revenue with them. It also aggregates the best content of its network to Glam.com, where it sells ads at a higher rate and shares its revenues too.
Glam helps sites on its network by sharing technology and content, and delivering traffic (and advertisers) to its member sites.

Distributed Leadership

Changing from centralized control to a distributed structure is not limited to the media and advertising world.
On the On Leadership panel we see how Best Buy made a strategic decision to distributing control to its employees allowing them to self organize and interact using social Web 2.0 tools.
By changing the company culture  “Less control from the top, more taking responsibility” the role of the CEO changes from  to defining a common goal, not way to get there, and relying the companies human capital to do what it takes to get there.
So some control over what’s going on is lost, but according to Best Buy CEO Brad Anderson, “this stuff builds energy. if people have self accountability and can do something that they helped created and love they got more energy. what a customer would get when he gets into that store is that he could tell whether that store has got energy and engaged employees serving him or not”.

So if there’s one thing I have to take from my  time at DLD 2009 its this: give up control, think distributed.

More on DLD:

• No Hope for Traditional Media Companies?
• Creating an Attractive Internet Company – It’s All About Emotions

Recommended session videos:

• 100 Million Uniques
• Reflections on a Crisis
• Predictably Irrational
• On Leadership
• New Media Models
• Software

Similar Posts:

• No Hope for Traditional Media Companies?
• What would Twitter do with $100 million?
• Creating an Attractive Internet Company – It’s All About Emotions
• Is Windows Live Still Alive?
• What the new Apple TV is really missing…

This vast new world of content is taking the lead from traditional media. Newspapers, Magazines and TV Channels are all loosing their audience in favor of the internet. Those traditional media companies who do develop strong internet presence are still having troubles as income from online advertising does not cover the decrease in income from traditional advertising and subscription fees.

The New Media Models (video) panel, on last week’s DLD conference, tries to deal with the question of business models that media companies can use to profit on the web.

When asked about what kind of new media outlet they’d start today, non of the panel’s participants would start a print newspaper or a magazine.

“I wouldn’t start a newspaper. There might be a place for a magazine to sit on my coffee table, but not for a NewsWeek or a BusinessWeek or a daily newspaper I think its absurd… I understand why the do it, revenues on print are so great vs. online but it has to go way” said Michael Arrington. According to Arrington we’re going to be consuming our news and other content online or on digital devices and once a certain threshold is reached it’ll no longer be profitable to print. “It doesn’t make any sense for news to be on paper because of its just the cost structure” he summarizes.

Jeff Jarvis said he would start a distributed content network, like Glam. According to Jeff not owning and controlling the data is what allowed Glam to literally explode to more than 110 Million unique users in 3 years. While content owners have to spend a fortune advertising to bring people in, distributed networks just go to where the people are. People today reach content via rating sites like DIGG or Twitter messages and no longer require central content portals to collect, sort, filter and rate content them. Jeff says that media companies need to start asking “How can we build platforms on which others succeed?”, they have to become platforms for content distributed around the web (like YouTube for example) rather than producing and controlling their own content.

On that same note, Arrington added that traditional media sites, who’s cost structure includes all kinds of expenses that do not produce content – like programmers, office space – can’t compete with an army of bloggers who blog from their home using their laptop and free software.

Carolyn McCall, who is the CEO of Guardian Media Group (representing the “traditional” media companies) confirmed the fact that income from the printing business is going down while internet income, even for a huge company like the Guardian Group, do not compensate for these losses. However, she also mentions that 25 million users use the Guardian’s web and that its income is 300 million dollars a year and that both numbers continue growing.

The panelist could not explain why advertising budget that are taken from the print versions of the media do not move in their entirety to the online advertising world, and they could also not find a definite business model to run media company online (even Jeff’s example for a distributed network – Glam – is still not profitable and taking VC money) but they all agree that print is phasing out and that there’s a need for a new model for online media and journalism.

Similar Posts:

• Give Up Control, Think Distributed – DLD 2009 Summary
• Creating an Attractive Internet Company – It’s All About Emotions
• Google’s New Behavioral Ad Targeting Should be Excellent for All
• 3 Takeaways from the Apple iPad Launch
• What would Twitter do with $100 million?

That’s the tough question that Yossi Vardi’s was trying to find an answer to on his panel at this year’s DLD conference – 100,000 Million Uniques (video) – together with Glam’s co-founder Samir Arora, YouTube’s co-founder and CEO Chad Hurley, CEO of the Mozilla Foundation Mitchell Baker and Toby Coppel who is a VP at Yahoo!.

The answer, according to Yossi and the panel speakers, who all run companies with over 100 million users, is that having a good product is not enough; there has to be an emotional connection between the product and its users.

After discussing the process of starting, marketing and improving their products the panel’s conclusion was that users engagement is the result of an emotional involvement between the users and the product and others users and a feeling of participating in creation of something new.

A good example for that is Yahoo’s Q&A service that, according to Coppel, has more than 140 million users a month. Although most users are passive and only read the questions and answers, 5%-7% of the users write answers, and they do so without getting paid – they share their knowledge with the world.
According to Coppel, the emotional connection to a product is achieved by providing a platform to self organize allowing people to create and share knowledge and experiences and find other like minded people.

Baker explained that Firefox’s success is based on the sense of mission – to build a piece of the internet that is a public assert – which turns out to be an emotional goal for a lot of people.

And what about YouTube? Chad Hurley explains its success on the focus on creating a simple-as-possible user experience and allowing people to distribute content by embedding videos  – “We try to associate our easy to use service with our brand – YouTube – and to allow people to take that experience, take that video code to embed and place on their own websites and blogs so that people can experience it there as well and drive traffic back to our service. Beyond creating a service that adds value people need to connect emotionally to the brand that you’re creating”.

To demonstrate the importance of emotion, Yossi Vardi showed two YouTube videos of the aria Nessum Dorma. The first video, sang by Pavarotti only received about 9 million views while the second video, sang by Paul Potts, a hobbyist opera singer, received well over 40 million views.

Vardi later asked Hurley why the second video was so popular. “Well he’s good” was Hurley’s answer.

Similar Posts:

• Give Up Control, Think Distributed – DLD 2009 Summary
• Microsoft Updates Its Windows Live Services
• No Hope for Traditional Media Companies?
• What would Twitter do with $100 million?
• A Visit to Maraboo, Restaurants as Tribes in the Digital Age

There’s a better (shorter) way for achieving the same result while leveraging on the Syndication mechanism built into .NET’s WCF.
WCF exposes the SyndicationFeed, SyndicationItem, SyndicationPerson classes which represent our data model.
In order to render this model WCF also exposes the Atom10FeedFormatter, and RSS20FeedFormatter classes that can render the feed to a stream, so all we need to do is integrate that into the ASP.NET MVC pipeline.

The ASP.NET MVC framework introduces a concept of returning an ActionResult instance as the result of Controller Actions.
This ActionResult object indicates the result from an action (a view to render, a URL to redirect to, another action/route to execute, etc).

ASP.NET MVC ships with several Action Results:

• ContentResult – Simply writes the returned data to the response.
• EmptyResult – Returns an empty response.
• HttpUnauthorizedResult – Returns Http 401 code for non authorized access.
• JsonResult – Serializes the response to Json.
• RedirectResult – Redirects to another Url.
• RedirectToRouteResult – Redirects to another controller action.
• ViewResultBase (abstract) – Renders an HTML content as a result.
  • PartialViewResult (inherits from ViewResultBase) – Renders a partial HTML response.
• BinaryResult (abstract) – Returns a binary response.
  • BinaryStreamResult (inherits from BinaryResult) – Writes a binary stream as a result.

So basically, to return a feed result all we need to do is define our own ActionResult implementation by deriving from ActionResult:

public abstract class ActionResult
{
    protected ActionResult();

    public abstract void ExecuteResult(ControllerContext context);
}

All we need to do is override the ExecuteResult method and write our data model to the output http stream using RSS20FeedFormatter:

public class RssActionResult : ActionResult
{
    public SyndicationFeed Feed { get; set; }

    public override void ExecuteResult(ControllerContext context)
    {
        context.HttpContext.Response.ContentType = "application/rss+xml";

        Rss20FeedFormatter rssFormatter = new Rss20FeedFormatter(Feed);
        using (XmlWriter writer = XmlWriter.Create(context.HttpContext.Response.Output))
        {
            rssFormatter.WriteTo(writer);
        }
    }
}

Now we can simply return RssActionResult as a result of our controller’s action.

Here’s a simple example:

public ActionResult Feed()
{
    SyndicationFeed feed =
        new SyndicationFeed("Test Feed",
                            "This is a test feed",
                            new Uri("http://Contoso/testfeed"),
                            "TestFeedID",
                            DateTime.Now);

    SyndicationItem item =
        new SyndicationItem("Test Item",
                            "This is the content for Test Item",
                            new Uri("http://Contoso/ItemOne"),
                            "TestItemID",
                            DateTime.Now);

    List<SyndicationItem> items = new List<SyndicationItem>();
    items.Add(item);
    feed.Items = items;

    return new RssActionResult() { Feed = feed };
}

… and that’s it!

A more elegant solution that leverages existing framework capabilities.

Related Posts

• RSS Feed with the new ASP.NET MVC Framework (Brad Abrams)
• ASP.NET MVC RSS Feed Action Result (Guy Burstein)
• RSS Feeds in ASP.NET MVC – StackOverflow

Similar Posts:

• Is Windows Live Still Alive?
• Developing a Robust Data Driven UI Using WPF – An Overdue Summary (and full source code)
• The New Google App Engine Blobstore API – First Thoughts
• Introduction to MapReduce for .NET Developers
• What would Twitter do with $100 million?