DeveloperZen

Building an iPhone Application

ekampf — Thu, 29 Oct 2009 16:25:00 +0000

On the past few weeks I’ve been working on a new venture centered around the iPhone. The process of building our app has been quite an adventure and we’ve experimented with several technologies that were new to us before reaching our current technology stack.
As we’ve finally got our stuff together and made an initial release to a group of testers I thought I’d share some of the technology choices we’ve made and the reasons behind them.

First some information about the team

…because technology choices are affected by the team’s technical skillset.

We’re 3 developers (Yosi, Udi and myself) and one designer (the awesome Naor Suki).
We’ve allocated two developers for the iPhone and one for the backend APIs & website.
We’re all veteran developers with experience mostly on Microsoft’s Development stack. This project meant going out of our comfort zone to a whole new set of technologies. Experience does make a difference easing the learning curve…

iPhone Development

iPhone SDK: This one is obvious right? We looked for alternatives for writing Objective-C. Unfortunately, Flash isn’t available for the iPhone (yet?) and MonoTouch looked promising but isn’t quite there…
Besides its always better to be developing on the platform most developers are using which means there’s a big community that can help when you get stuck. Being on Apple’s official stack also means we get the latest features without having to wait for a 3rd party to convert them…
To be perfectly honest, I’m not on the iPhone side of the development and did not actually write a single line of Objective-C code but I noticed it took my teammates 1-2 weeks to get the hang of it.
To me, the fact that the iPhone App store is so successful and has so many apps which Objective-C as the development language (which is definitely harder than modern languages – Java etc.) makes Apple’s achievement even more amazing…
- If you’re beginning iPhone development you have to checkout the Stanford iPhone development class on iTunes University – http://www.stanford.edu/class/cs193p/cgi-bin/index.php
Three20: Handful of UI extractions from the Facebook iPhone app and open-sourced by the developer – joehewitt. His announcement blog post details the libraries it contains, shows some demos etc.
The source code is also a great learning tool for how stuff is done on the iPhone.
json-framework: This is a pretty slick JSON parser for the iPhone. Hand parsing JSON in obj-C would not have been fun. This made it easy. I’m pretty sure I followed this tutorial to get it up and running.
ASIHttpRequest: A Nice Http framework that enabled easily handling asynchronous Http requests.
Stackoverflow is an invaluable resource for asking question and solving all sort of problems. As Yosi, who’s been concentrating on the iPhone side of our development, puts it “I dont think any iPhone development could be done without StackOverflow”
MGTwitterEngine: an awesome objective-c wrapper for twitter api which we based our api on.
Google Analytics SDK: A library enabling sending information to Google Analytics from the iPhone. This is important for measuring the ways users interact with certain flows on our program. For example, it helps us measure the conversion on our signup flow – how many users go through the signup flow and finish, and if they dont, what steps makes them go away?
This kind of functionality is essential to measuring and improving UX flows…
Google Toolbox for Mac: A library for working with the different services exposed by Google.

Backend Development

After playing around with ASP.NET MVC (which we all had background with having come from the Microsoft ecosystem) and Ruby on Rails (because its cheaper to host than ASP.NET and way simple, faster, more fun to use IMHO) we’ve finally settled for Google AppEngine and django (Python).

We made the decision to base our development on django rather than on Google’s own webapp framework for the following reasons:

Lots of out of-of-the-box features. django has been out there for quit a while and is bundled with lots of features (like an easy to build admin interface, authentication system, validation system etc.)
Big community. There are lots of people doing django out there lots of open source libraries and samples available. As a rule of thumb its always better to be on the majority side…
Not specific to Google AppEngine. django is a standalone Python web development platform. While some parts of our code has to be AppEngine specific it would still be considerably easier to move away from AppEngine (if we ever decide to do so) than if we were entirely Google specific.

Google AppEngine also have Java support. But using Python with django is way easier and has a lot more support when it comes to both AppEngine and web development. Seriously, if you’re thinking of using Java, Don’t! take the leap and go with Python…

Libraries we’ve used:

app-engine-patch: This library is absolutely amazing and a must if you’re using django on AppEngine. Since the AppEngine data store API is not compatible with django’s API, a lot of the really cool time-saving features of django will simply not run on AppEngine (such as the admin UI, authentication and basically anything that requires data access). app-engine-patch loads django and patches it so it is compatible with the AppEngine API making all those cool django features work. This one is a must! You just download their project template and start developing your application on top of it.
PyDev: An Eclipse plugin for editing and debugging Python and AppEngine applications. It might sound obvious but I was using was actually using Notepad++ (on Windows) for development until I found out there’s a decent IDE I could use…
Piston: a django library for developing REST-APIs. While its not entirely compatible with AppEngine it took a simple forking to edit those parts out…
GeoModel: provides basic indexing and querying of geospatial data on Google AppEngine.

Also, I would recommend taking the time to learn and understand how the AppEngine datastore works so you’ll understand how to build your datamodel to run efficiently on Google’s platform.
The following two presentations from Google I/O are invaluable:

So what do you think? If you’re developing an iPhone app, I’m very interested to know what were your technology choices and reasoning…

Oh, and if you have an iPhone and you live in Israel (its a local app so we’re limiting our efforts to Israel at the moment) please head over to our beta signup form and signup

Similar Posts:

High Performance at Massive Scale – Lessons learned at Facebook

ekampf — Tue, 27 Oct 2009 07:39:00 +0000

Jeff Rothschild, Vice President of Technology at Facebook gave a great presentation at UC San Diego on "High Performance at Massive Scale – Lessons learned at Facebook". The presentation’s abstract:

Facebook has grown into one of the largest sites on the Internet today serving over 200 billion pages per month. The nature of social data makes engineering a site for this level of scale a particularly challenging proposition. In this presentation, I will discuss the aspects of social data that present challenges for scalability and will describe the the core architectural components and design principles that Facebook has used to address these challenges. In addition, I will discuss emerging technologies that offer new opportunities for building cost-effective high performance web architectures.

I’m halfway through watching it and there are already several interesting point worth a detailed post later on.
If you want to learn how Facebook manages 30K+ machines, 300 million active users, 20 billion photos, and 25TB/day of logging data you should go and watch the talk’s webcast.

Similar Posts:

What would Twitter do with $100 million?

ekampf — Mon, 28 Sep 2009 21:35:00 +0000

Last week the NY Times reported that Twitter has raised about $100 million of new funding, making the company’s value to be $1 billion. Just to put things in perspective, they also provide an example:

For context, that is almost double the market capitalization of Domino’s Pizza, which has 10,500 employees and had $1.4 billion in sales last year. Twitter has some 60 employees, and although it is experimenting with running advertisements on its Web site, Biz Stone, a Twitter founder, said this week at an industry conference that the company had no plans to begin widely running ads until 2010.

Twitter previously raised $55 million and has said it still has $25 million of that in the bank. So the question is, what will it do with these $100 million? Or, as I see it, who will it acquire now?

As part of it efforts to find a business model, Twitter will most likely acquire companies that’ll help it form that model.I’m thinking\betting on two major trends for such a business model:

Manage Companies Presence on Twitter

Twitter’s most obvious business model is by helping companies manage their presence on Twitter and monitor how their brands are being discussed.

This makes companies who provide all sort of analytics information, CRM integration and even url-shorteners as potential acquisitions for a future Twitter business package…

Local Markets, Local Social Network

The minute I read about Twitter’s $100 million round I thought of companies like Foursquare. I wasn’t really surprised when I read today’s Techmeme and noticed Twitter’s co-founder Jack Dorsey invested in Foursquare.
Also, the twitter team has been working very hard lately to make Twitter location aware, allowing users to share their location via their tweets and browse stuff that is happening around them.

Twitter is great in forming local communities (just checkout the local tweetups everywhere), gather news and provide all sort of local information. Fouresquare as well as other location based social networks doing can really help twitter tap into the long-tail local businesses market and take on companies such as Yelp.

So, what do you think Twitter’s latest valuation? What will it do with its newly raised $100mil?

Similar Posts:

Data Mining – Handling Missing Values the Database

ekampf — Fri, 14 Aug 2009 17:50:54 +0000

I’ve recently answered Predicting missing data values in a database on StackOverflow and thought it deserved a mention on DeveloperZen.

One of the important stages of data mining is preprocessing, where we prepare the data for mining. Real-world data tends to be incomplete, noisy, and inconsistent and an important task when preprocessing the data is to fill in missing values, smooth out noise and correct inconsistencies.

If we specifically look at dealing with missing data, there are several techniques that can be used. Choosing the right technique is a choice that depends on the problem domain – the data’s domain (sales data? CRM data? …) and our goal for the data mining process.

So how can you handle missing values in your database?

1. Ignore the data row

This is usually done when the class label is missing (assuming you data mining goal is classification), or many attributes are missing from the row (not just one). However you’ll obviously get poor performance if the percentage of such rows is high.

For example, lets say we have a database of students enrollment data (age, SAT score, state of residence, etc.) and a column classifying their success in college to “Low”, “Medium” and “High”. Lets say our goal is do build a model predicting a student’s success in college. Data rows who are missing the success column are not useful in predicting success so they could very well be ignored and removed before running the algorithm.

2. Use a global constant to fill in for missing values

Decide on a new global constant value, like "unknown", "N/A" or minus infinity, that will be used to fill all the missing values.
This technique is used because sometimes it just doesn’t make sense to try and predict the missing value.

For example, lets look at the students enrollment database again. Assuming the state of residence attribute data is missing for some students. Filling it up with some state doesn’t really makes sense as opposed to using something like “N/A”.

3. Use attribute mean

Replace missing values of an attribute with the mean (or median if its discrete) value for that attribute in the database.

For example, in a database of US family incomes, if the average income of a US family is X you can use that value to replace missing income values.

4. Use attribute mean for all samples belonging to the same class

Instead of using the mean (or median) of a certain attribute calculated by looking at all the rows in a database, we can limit the calculations to the relevant class to make the value more relevant to the row we’re looking at.

Lets say you have a cars pricing database that, among other things, classifies cars to "Luxury" and "Low budget" and you’re dealing with missing values in the cost field. Replacing missing cost of a luxury car with the average cost of all luxury cars is probably more accurate then the value you’d get if you factor in the low budget cars.

5. Use a data mining algorithm to predict the most probable value

The value can be determined using regression, inference based tools using Baysian formalism , decision trees, clustering algorithms (K-Mean\Median etc.).

For example, we could use a clustering algorithms to create clusters of rows which will then be used for calculating an attribute mean or median as specified in technique #3.
Another example could be using a decision tree to try and predict the probable value in the missing attribute, according to other attributes in the data.

I’d suggest looking into regression and decision trees first (ID3 tree generation) as they’re relatively easy and there are plenty of examples on the net…

Additional Notes

Note that methods 2-5 bias the data as the filled-in value may not be correct.
Method 5 uses the most information available in the present data to predict the missing value so it has a better chance for generating less bias.
Missing value may not necessarily imply an error in the data! forms may contain optional fields, certain attributes may be in the database for future use.

Similar Posts:

Moving Your Application to Amazon’s Cloud

ekampf — Sat, 25 Jul 2009 08:02:22 +0000

I’ve been dealing a lot with Amazon’s AWS platform lately. Mostly doing offline data processing using Hadoop but the latest load balancing features finally opened the door for frontend applications to take advantage of Amazon’s cloud computing platform – making it easier for developers to make application more cost efficient an scalable.

Keeping in mind that there are a lot of applications out there who can benefit from moving to the cloud (including my own) I’ve made a list of tasks\considerations to make when preparing for such a move:

Step One: Move Static Content to S3

The first and easiest step is to move all your static content – images, CSS, JavaScript files, etc. – to Amazon S3. Let Amazon worry about storage, backups and availability for you.

Things to consider:

GZIP content. S3 does not support serving GZIPed content natively so you’ll have to upload both GZIPed and plaintext version of each file and figure out in code which one to use according to the headers sent by the user’s browser.
The following post describes how its done: Using Amazon S3 as a CDN
Amazon’s S3 service is “eventually consistent” which means that files uploaded to S3 may not be immediately available to read.
Use separate sub-domains for content.

Once your content is on S3 you can also use CloudFront, Amazon’s CDN (Content Delivery Network), to serve the files and improve your application’s performance.

Step Two: Move Web Servers and Backend Servers to EC2

Move your web server code and backend services – database, memcached, etc. – to run on Amazon EC2 instances.

Consider using Amazon’s availability zones to setup servers in different availability zones. This can help your serve customers at different parts of the worlds better, while making your infrastructure tolerant to the unlikely event of a datacenter failures at Amazon.

The Web Servers

Moving your web servers to EC2 should be fairly simple. You can setup EC2 images that are configured exactly the same way your current web servers are.

If you require a queuing service as part of your architecture, consider switching to Amazon’s SQS to make administration easier.

The Database

Moving your database to EC2 is probably the hardest part of the move to AWS. If you plan on keeping your database (as opposed to migrating to a cloud solution like SimpleDB) you should use EBS (Elastic Block Storage) so that your storage is persists independently from the life of your EC2 instance.

Backup. Figure out how to take scheduled snapshots of your EBS and store them on S3.

Consider replication and sharding. If you’re using availability zone you should consider sharding your data. For example, store data European accounts data in Europe only. You should also consider replication between the different availability zones to ensure keep your site available even when one of the datacenter is unavailable.

Step Three: Scale. Take advantage of the cloud services

Now that your application is entirely running on Amazon’s platform it’s time to take the full advantage of the platform and make it scale.

Setup Monitoring to keep up with what’s going on on your system. Amazon provides a service called CloudWatch that allows you to monitor your machines and applications.

Based on the monitoring metrics you should start using Amazon’s auto-scaling and load balancing capabilities to be able to consume and release computing resources according to demand.

At this point you should also investigate reducing your dependency on relational databases (RDMS) as much as possible (as its the most complex, and hardest to scale, component in the system) and try to move as much functionality as possible to S3 and SimpleDB.
S3 is suitable for storing large objects while SimpleDB is ideal for small stubs of data.

Important notes:

Amazon’s load balancer doesn’t support SSL. This can be a showstopper for some applications…
Simple DB has a max row size limitation. If your data exceeds that limit you should consider using SimpleDB as a metadata store that references the full data stored on S3.

Facebook, Hadoop and Hive

ekampf — Tue, 16 Jun 2009 04:37:50 +0000

Facebook has the second largest installation of Hadoop (a software platform that lets one easily write and run distributed applications that process vast amounts of data), Yahoo being the first. It is also the creator of Hive, a data warehouse infrastructure built on top of Hadoop.

The following two posts shed some more light on why Facebook chose the Hadoop\Hive path, how they’re doing it and the challenges they’re facing:

Facebook, Hadoop, and Hive on DBMS2 by Curt Monash discusses Facebook’s architecture and motivation.

Facebook decided in 2007 to move what was then a 15 terabyte big-DBMS-vendor data warehouse to Hadoop — augmented by Hive — rather than to an MPP data warehouse DBMS…

The daily pipeline took more than 24 hours to process. Although aware that its big-DBMS-vendor warehouse could probably be tuned much better, Facebook didn’t see that as a path to growing its warehouse more than 100-fold.

Hive – A Petabyte Scale Data Warehouse using Hadoop by Ashish Thusoo from the Data Infrastructure team at Facebook discusses Facebook’s Hive implementation in details.

… using Hadoop was not easy for end users, specially for the ones who were not familiar with map/reduce. End users had to write map/reduce programs for simple tasks like getting raw counts or averages. Hadoop lacked the expressibility of popular query languages like SQL and as a result users ended up spending hours (if not days) to write programs for typical analysis. It was very clear to us that in order to really empower the company to analyze this data more productively, we had to improve the query capabilities of Hadoop. Bringing this data closer to users is what inspired us to build Hive. Our vision was to bring the familiar concepts of tables, columns, partitions and a subset of SQL to the unstructured world of Hadoop, while still maintaining the extensibility and flexibility that Hadoop enjoyed.

Similar Posts:

Yahoo Releases Its Own Hadoop Distribution

ekampf — Wed, 10 Jun 2009 22:52:33 +0000

Yahoo! is releasing its own distribution of Hadoop:

Hadoop is a distributed file system and parallel execution environment that enables its users to process massive amounts of data.
In response to frequent requests from the Hadoop community, Yahoo! is opening up its investment in Hadoop quality engineering to benefit the larger ecosystem and to increase the pace of innovation around open and collaborative research and development.
The Yahoo! Distribution of Hadoop has been tested and deployed at Yahoo! on the largest Hadoop clusters in the world.

Hadoop is free Java software framework born out of an open-source implementation of Google’s published computing infrastructure and fostered within the Apache Software Foundation.
Yahoo! has been the primary developer and contributor to Apache’s Hadoop.
In 2006, Hadoop founder Doug Cutting joined Yahoo, which provided a dedicated team and resources, to lead the project of developing the open-source software and turn Hadoop into a system that ran at web scale. Today, Yahoo! is running the largest Hadoop cluster in the world, which includes more than 25,000 servers and provides the framework for many Yahoo properties including Yahoo Search, Yahoo Mail, and several content and ad services.

Yahoo says its opening up the source code to Hadoop to “benefit the larger ecosystem increase the pace of innovation around open and collaborative research and development.”.
As Nigel Daley, Quality and Release Engineering Manager at Yahoo! Grid Technologies, summarizes:

Hadoop is helping us solve key science and research problems in hours or days instead of months. It provides us a platform to solve extreme problems requiring massive amounts of data processing. It underpins major revenue-generating systems. Opening our distribution enables a faster pace of innovation for the entire Hadoop ecosystem and broadens the use — and ultimately the quality — of this key platform across the industry.

Similar Posts:

New Features for Amazon EC2 – Now You Can Truly Scale Applications

ekampf — Mon, 18 May 2009 10:14:40 +0000

The Amazon Elastic Compute Cloud (Amazon EC2) allows customers build secure, fault-tolerant applications that can scale up and down with demand, at low cost. One of the core features for achieving this kind of efficiency and fault-tolerant is the ability to acquire and release computing resources in a matter of minutes according to demand.

While Amazon’s EC2 has been great so far in allowing companies to run large, computational-heavy distributed tasks on the background, it has been really lacking on allowing companies to run online services in a manner that’s efficient and reliable.

The reason for this lack of support is basically because there was no way to do load balancing on EC2 machines allowing a service to efficiently scale across multiple EC2 instances.

With the launch of Amazon CloudWatch, Auto Scaling and Amazon Elastic Load Balancing, Amazon is effectively making EC2 a viable host for online services, not just background jobs. Using these services you can not only scale a service across machines for reliability, you can also automatically add and remove computing resources as demand increases\decreases driving your operation costs down.

You can find more information at the detail pages for Amazon CloudWatch, Auto Scaling and Elastic Load Balancing and on the AWS developer weblog. Also, check out Werner Vogel’s blog post for some background on how Amazon is horizontally scaling its services.

Similar Posts:

Introduction to MapReduce for .NET Developers

ekampf — Wed, 06 May 2009 16:08:21 +0000

The basic model for MapReduce derives from the map and reduce concept in functional languages like Lisp.
In Lisp, a map takes as input a function and a sequence of values and applies the function to each value in the sequence.
A reduce takes as input a sequence of elements and combines all the elements using a binary operation (for example, it can use “+” to sum all the elements in the sequence).

MapReduce, inspired by these concepts, was developed as a method for writing processing algorithms for large amounts of raw data. The amount of data is so large that it can’t be stored on a single machine and must be distributed across many machines in order to be processed in a reasonable time.
In systems with such data distribution, the traditional central processing algorithms are useless as just getting the data to the centralized CPU running the algorithm implies huge network costs and months (!) spent on transferring data from the distributed machines.
Therefore, processing such massive scales of distributed data implies the need for parallel computing allowing us to run the required computation “close” to where the data is located.
MapReduce is an abstraction that allows engineers to write such processing algorithms in a way that is easy to parallelize while hiding the complexities of parallelization, data distribution, fault tolerance etc.

This value proposition for MapReduce is outlined in a Google research paper on the topic:

MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper.

Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program’s execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.

Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: hundreds of MapReduce programs have been implemented and upwards of one thousand MapReduce jobs are executed on Google’s clusters every day.

The MapReduce Programming Model

As explained earlier, the purpose of MapReduce is to abstract parallel algorithms into a map and reduce functions that can then be executed on a large scale distributed system.
In order to understand this concept better lets look at a concrete map reduce example – consider the problem of counting the number of occurrences of each word in a large collection of documents:

map(String key, String value):
// key: document name
// value: document contents
for each word w in value:
  EmitIntermediate(w, "1"); 

reduce(String key, Iterator values):
// key: a word
// values: a list of counts
int result = 0;
for each v in values:
  result += ParseInt(v);
Emit(AsString(result));

The map function goes over the document text and emits each word with an associated value of “1”.

The reduce functions sums together all the values for each word producing the number of occurrences for that word as a result.

First we go through the mapping phase where we go over the input data and create intermediate values as follows:

Records from the data source (lines out of files, rows of a database, etc.) are fed into the map function as pairs.For example:
The map function produces one or more intermediate values along with an output key from the input

After the mapping phase is over, we go through the reduce phase to process the intermediate values:

After the map phase is over, all the intermediate values for a given output key are combined together into a list and fed to the reduce function.
The reduce function combines those intermediate values into one or more final values for that same output key

Notice that both the map and the reduce functions run on independent set of input data. Each run of the map function process its own data source and each run of the reduce function processes the values of a different intermediate key.

Therefore both phases can be parallelized with the only bottleneck being the fact that the map phase has to finish for the reduce phase to start.

The underlying system running these method is in takes care of:

Initialize a set of workers that can run tasks – map or reduce functions.
Take the input data (in our case, lots of document filenames) and send them to the workers to map
Streamline values emitted by map function to the worker (or workers) doing the reduce. Note that we don’t have to wait for a certain map run to finish going over the entire file in order to start sending its emitted values to the reducer, so that the system can prepare the data for the reducer while the map function is running
(In Hadoop – send the map values to the reducer node and andle grouping by key).
Handle errors – support a reliable, fault tolerant process as workers may fail, network can crush preventing workers from communicating results, etc.
Provides status and monitoring tools.

A Naive Implementation in C#

Lets see how we can build naive MapReduce implementation in C#.

First, we define a generic class to manage our Map-Reduce process:

public class NaiveMapReduceProgram

The generic types are used the following way:

(K1, V1) – key-value types for the input data
(K2, V2) – key value types for the intermediate results (results of our Map function)
V3 – The type of the result for the entire Map-Reduce process

Next, we’ll define the delegates of our Map and Reduce functions:

public delegate IEnumerable<KeyValuePair>   MapFunction(K1 key, V1 value);
public delegate IEnumerable                     ReduceFunction(K2 key, IEnumerable values);

private MapFunction _map;
private ReduceFunction _reduce;

public NaiveMapReduceProgram(MapFunction mapFunction, ReduceFunction reduceFunction)
{
    _map = mapFunction;
    _reduce = reduceFunction;
}

(Yes, I realize I could use .NET’s Func instead but that would just result in horribly long ugly code…)

Now for the actual program execution. The execution flow is as follows: We take the input values, pass them through the map function to get intermediate values, we group those values by key and pass them to the reduce function to get result values.

So first, lets look at the mapping step:

private IEnumerable<KeyValuePair> Map(IEnumerable<KeyValuePair> input)
{
    var q = from pair in input
            from mapped in _map(pair.Key, pair.Value)
            select mapped;

    return q;
}

Now after we got the mapped intermediate values we want to reduce them. The Reduce function expects a key and all its mapped values as input so to do that efficiently we want to group the intermediate values by key first and then call the Reduce function for each key.

The output of this process is a V3 value for each of the intermediate K2 keys:

private IEnumerable<KeyValuePair> Reduce(IEnumerable<KeyValuePair> intermediateValues)
{
    // First, group intermediate values by key
    var groups = from pair in intermediateValues
                 group pair.Value by pair.Key into g
                 select g;

    // Reduce on each group
    var reduced = from g in groups
                  let k2 = g.Key
                  from reducedValue in _reduce(k2, g)
                  select new KeyValuePair(k2, reducedValue);

    return reduced;
}

Now that we have the steps code the execution itself is simply defined as Reduce(Map(input)) :

public IEnumerable<KeyValuePair> Execute(IEnumerable<KeyValuePair> input)
{
    return Reduce(Map(input));
}

The full source code and tests can be downloaded from here:

Map-Reduce Word Counting Sample – Revisited

Lets go back to the word-counting pseudo code and write it in C#.

The following Map function gets a key and a text value and emits a key-pair for each word in the text:

public IList<KeyValuePair<string, int>> MapFromMem(string key, string value)
{
    List<KeyValuePair<string, int>> result = new List<KeyValuePair<string, int>>();
    foreach (var word in value.Split(' '))
    {
        result.Add(new KeyValuePair<string, int>(word, 1));
    }
    return result;
}

Having calculated a key-pair for each input source, we can group the results by the word and then our Reduce function can sum the values (which are 1 in this case) for each word:

public IEnumerable<int> Reduce(string key, IEnumerable<int> values)
{
    int sum = 0;
    foreach (int value in values)
    {
        sum += value;
    }

    return new int[1] { sum };
}

Our program code looks like this:

MapReduceProgram<string, string, string, int, int> master = new MapReduceProgram<string, string, string, int, int>(MapFromMem, Reduce);
var result = master.Execute(inputData).ToDictionary(key => key.Key, v => v.Value);

The result dictionary contains a <word, number-of-occurrences> pairs.

Other Examples

Distributed LINQ Queries. One of POCs I’m working on using the above naive, LINQ-based implementation, is running a distributed LINQ query. Imagine you have a system where raw data is distributed across several SQL Servers. We can have our map function run a LINQ-to-SQL query on multiple DataContexts in parallel (the value input for the map function – V1 – can be a DataContext) and then reduce it to a single result set. This is probably a naive\simplified implementation of what the guys at Microsoft’s Dryad team are doing.

Count URL Visits. Consider you have several web servers and you want to produce the amount of visits for each page on your site. You can produce pretty much the same way the word-counting example works. The map function parses a log file and produce a intermediate value. The reduce function then sums the values for each URL and emits

Distributed Grep. You can run a grep search on a large amount of files by having the map function emits a line if it matches a given pattern. The reduce function in this case is just an identity function that copies the supplied intermediate data to the output.

Map-Reduce in the Real World

The real complexity and sophistication in MapReduce is in the underlying system takes care of running and managing the execution of MapReduce jobs. Real world MapReduce implementations, like Google’s system, Hadoop or Dryad have to go beyond the naive implementation shown here and take care of things like resource monitoring, reliability and fault tolerance (for example, handle cases where nodes running map\reduce jobs crush, or go offline due to network problems).

The following resources are worth checking out:

Cloudera’s Hadoop Training
Google Code University – Distributed Systems

Similar Posts:

A Visit to Maraboo, Restaurants as Tribes in the Digital Age

ekampf — Thu, 30 Apr 2009 07:34:10 +0000

Its a rare occasion that I go out of a restaurant (and I go to many) with an absolute feeling of “WOW!”… This Monday was one of these occasions.

On the past couple of week we’ve been working on a new, food related venture (more details on that in the very near future). Apparently, we raised enough twitter noise to get noticed, and thanks to Maya we’ve been contacted by Channel 10 and invited to visit Maraboo as part of the promotions for the season finale of the Israeli version of Icon Chef featuring the Maraboo team vs. the Hudson Brasseri – you should tune in on Channel10 this Thursday (April 30, 2009) at 21:00.

At the restaurant we met the owners, Haim Amshalom and Chef Yoav Bar
who showed us around the place and introduced us to each and every dish we got. We had a great time…

* On the left Haim Amshalom and on the right Chef Yoav Bar

The fact that were at Maraboo as bloggers, that Maraboo has its own group on Facebook and that its owner is on Twitter made me think about the opportunities the web has to offer restaurants.

The Web is About Openness… Can Restaurants be Open?

The Web is about openness, its about empowering online communities and individuals to create, remix, share and make things themselves.
Restaurants are essentially small communities, or tribes of people with a common interest (in our case – food) a leader (the Chef) and a platform to communicate (the restaurant). They can use the web as a platform to develop a personal relationship with its customers and the facility itself (the restaurant) as a social hub.

Restaurants, like any other business, can use Twitter or a Facebook group etc. to keep in touch with their diners but what’s more interesting is that they can actually use it to become a platform, or a hub, for its tribe members (diners) to interact and their common interests (food).

The Menu as a Wiki. Ok, the Chef builds the menu and we wouldn’t want to change that… that’s the whole point of the restaurant. But what if he could collaborate or include the diners into that process?

Maybe people think the Calamari dish is lacking? or maybe they just hate Calamari… wouldn’t you want to know?

It can put recipes online and invite diners to make suggestions. Some users might go through the trouble of preparing the dish at home. They might make all sort of adaptations and report back, discuss them online with their fellow diners (For example, a lot of people tried to make the same wings sauce as Dixie’s famous sauce and they were discussing it online). Maybe some of these adaptations are actually worth putting in the menu?

In fact, what if browsing a restaurant’s menu was more like browsing Amazon?
You could see which dishes people like more, what they think of it. You could see what your friends like (or not) or filter dishes by “spec” (like calories intake?).

Restaurants can become the stars. The popularity of kitchen-based reality shows is soaring (heck, we’re here as a promotion for such a show) because there’s a public interests in restaurants, in chefs, in food making.
Chefs could blog about their taste, inspirations, trends they see, cooking practices and tutorials and become local stars (take Gary Vaynerchuk’s for example)

Restaurant as a community meeting place. People in online communities like to meet. Just see all the Tweetup events where communities of people who know each other from talking on Twitter gather to talk offline. As its community members all share a passion for food, a restaurant can organize bakeups- where people could learn about cooking\cook. Maybe even have a community member as Chef for a night…

Final Thoughts and.. Food Photos!

As Seth Godin puts it, managing a tribe is a whole different concept than the old way of just managing a brand using traditional PR:

Tribe management is a whole different way of looking at the world.

It starts with permission, the understanding that the real asset most organizations can build isn’t an amorphous brand but is in fact the privilege of delivering anticipated, personal and relevant messages to people who want to get them.

It adds to that the fact that what people really want is the ability to connect to each other, not to companies. So the permission is used to build a tribe, to build people who want to hear from the company because it helps them connect, it helps them find each other, it gives them a story to tell and something to talk about.

Anyway, I can’t end this post without some photos of some of the amazing dishes we had at Maraboo:

And finally, the entire dinning team

Similar Posts:

Weekend Inspirations – Big Data Visualizations, Innovative Man-Machine Interactions

ekampf — Sat, 25 Apr 2009 10:54:28 +0000

A bunch of interesting/inspiring topics for the weekend…

Designing “Big Data”

Jeff Veen from Small Batch Inc. gave a 20-minutes talk at the Web2.0 Expo at San Francisco.
During the talk he focuses on some classic examples for information visualization (John Snow pump, Minard’s map, the tube map, and so on), the challenge of making data more accessible and understandable vs. just “decorating” it and the emerging challenge in Web 2.0 to empower users to find and create their own stories using the data.

And on the same topic of dealing with “big data”, check out the AlloSphere. A system developed by scientists and artists at the University of California for exploring huge sets of data as a visual and audible 3D world:

Cool New Computer Interaction Technology

I saw this amazing UI concept watching the TED podcast:

The concept as computers as tiny blocks that can react to each other is just amazing and the possibilities here, especially in education, are endless…

Kiva Robots Invade the Warehouse

If a bunch of tiny computers interacting can change personal computing here’s what it can do in the enterprise – optimize warehouses supply chain:

The servers work in real-time, receiving orders, immediately dispatching robots to bring the required pods to the worker fulfilling the order, and then returning the pods to their storage locations. The robots receive their orders wirelessly, while using cameras to read navigational barcode stickers on the warehouse floor.

Have a great weekend!Similar Posts:

Developing a Robust Data Driven UI Using WPF – An Overdue Summary (and full source code)

ekampf — Tue, 14 Apr 2009 21:48:14 +0000

I wrote the stocky application more than a year ago as a research project aimed at proving that using WPF we can separate presentation metadata (XAML) from program logic. The goal was to provide the Duet team at SAP with a document reference sample for using M-V-VM to achieve this separation.

I started documenting the proof-of-concept in a series of posts but unfortunately after leaving SAP my interests (and work) shifted away from WPF and I didn’t find the time to finish the series.

I’ve received numerous requests to release the source code but I couldn’t do so because it was part of a larger infrastructure code I wrote at SAP which basically ads a lot of noise to the sample (an d probably ads legal issues for me sharing it).
Anyway, I took some time off this afternoon to re-write the sample independently so that I could share it:

It can be found on my SkyDrive

This, I guess is the long overdue ending for the series:

Introduction – introduces the concept of M-V-VM and the reasoning behind it.
The DataModel – describes how to write the Model part of our application.
Stock DataModel Sample – provides a conrete implementation of a Stock model and its view..

However, If you’re interested in M-V-VM in WPF, there are numerous topics worth mentioning that I didn’t get to cover and are definitely worth checking out:

Unit Testing

As I said in the introduction post, one of the most important benefits of seperating the logic code from the presentation (XAML) is that its straightforward to unit test. In fact, my next post following the Stock DataModel Sample was going to be about unit testing – specifically, how to test the DataModel its provider which, because of the use of threading, is a bit tricky.

This post is actually 99% done in the comments of the unit test code that’s in DefaultStockQuoteProviderTest.cs in the provided source code. So do yourself a favor and go over the code. It’s not long and very well documented…

Using Lambda Expression for DataBinding

Data-binding is pretty much at the heart of the M-V-VM concept and it makes us write Value Converters which is pretty tedious and annoying.
Wouldn’t it be great if we could replace writing lots of IValueConverter classes like this:

<TextBlock Foreground="{Binding Change, Converter={StaticResource StockForegroundConverter}}" … />

[ValueConversion(typeof(double), typeof(Brush))]
public class StockChangeToBrushConverter : IValueConverter
{
    public object Convert(object value, Type targetType, object parameter, CultureInfo culture)
    {
        double change = (double)value;
        if (change == 0) return Brushes.Black;
        return (change < 0) ? Brushes.DarkRed : Brushes.Green;
    }

    public object ConvertBack(object value, Type targetType, object parameter, CultureInfo culture)
    {
        return double.NaN;
    }
}

To just the following XAML statement that embeds the conversion logic:

<TextBlock Foreground="{Binding Change,

Converter={ change=> if (change == 0) return Brushes.Black; return (change < 0) ? Brushes.DarkRed : Brushes.Green; }}" … />

M. Orçun Topdağı wrote an excellent series on using Lambda Expressions for data-binding in WPF to achieve just that:

Reference Applications and Guidance

I haven’t seen a lot of sample WPF LOB reference applications out there but here are some interesting links for further learning:

Prism: patterns & practices Composite Application Guidance for WPF and Silverlight site
M-V-VM Reference Application – A new project that aims to “created reference application for M-V-VM frameworks to use for demonstration purposes, similar in concept to Pet Shop for web frameworks.”

Similar Posts:

Microsoft Can Clone Twitter?!

ekampf — Sun, 05 Apr 2009 01:03:41 +0000

In a response to Microsoft watcher Todd Bishop’s post saying Microsoft should buy Twitter, Mary Jo-Foley wrote Microsoft Shouldn’t Buy Twitter saying:

…But I’d argue Microsoft could simply do a Twitter clone — the same way that it has built its own Facebook-notification-like news stream into Windows Live — and reap similar results. In fact, the Softies are hinting they’ve already been experimenting with adding Twitter-like functionality to its business software (possibly via SharePoint). I’d bet the Xbox and maybe the Pink/Danger mobile teams have been looking at doing their own Twitter-like services too.

Seriously?! Microsoft could simply clone Twitter?!

Sure, Microsoft certainly has some brilliant tech folks that can surely implement or clone anything. It also has the resources to do that.
And yet, its mostly following the pack with its online offering pretty stagnant. Last time it tried anything like that was Live Home with its Facebook\FriendFeed like functionality and that’s pretty much dead…
It can certainly clone Twitter, probably even do a better job at it than the original, but will anyone bother using it?

Twitter has a fast growing, huge and vibrant community. Its also a well known brand name that’s getting a huge amount of media attention right now.
It survived its own fail whales, upgrade owls and all sorts of other nasty service downtimes simply because no one else can beat that…
You can clone technology but you can’t clone a community and brand strength…Similar Posts:

Google’s New Behavioral Ad Targeting Should be Excellent for All

ekampf — Thu, 12 Mar 2009 02:29:47 +0000

The discussion over Google’s latest move into behavioral ad targeting is all over TechMeme. Basically what this means is that Google will start selling users (or more specifically, clusters of users, like football fans for example) in addition to words:

Today we are launching “interest-based” advertising as a beta test on our partner sites and on YouTube. These ads will associate categories of interest – say sports, gardening, cars, pets – with your browser, based on the types of sites you visit and the pages you view. We may then use those interest categories to show you more relevant text and display ads. (…)
So if you visit an online sports store, you may later be shown ads on other websites offering you a discount on running shoes during that store’s upcoming sale.

Google’ is not the first to embrace behavioral targeting as other companies (including Google’s own DoubleClick) were already offering advertisers these features, but it is certainly the biggest.
Behavioral targeting on Google massive network of sites (just think of anyone who’s using Google Analytics, which is pretty much everyone) is certainly a game changing event to the online advertising industry.

The responses online (and ones I usually get talking to people about the topic as we’re doing stuff in this area at Nuconomy) range from simple skepticism of behavioral targeting effectiveness (like Google’s own Matt Cutts…) to Big Brother concerns.

To the latter group, concerned with privacy and worried about its actions online being tracked I have to say – NEWS FLASH! you don’t have privacy on the internet… everything you do is already being tracked, analyzed and can be linked back to you even if stored anonymously (ever heard of Thelma Arnold?), the only question is what do you get out of it?

I hope that with behavioral targeted ads I will see less irrelevant ads offering me low mortgages and seduction courses and more ads geared towards tech gear, maybe even making online ads actually useful (can’t remember the last time I clicked an ad. with time I just learned to automatically ignore these ad banners…). But I think change is not going to hit just the advertising market…

Google’s move can (hopefully) marks the beginning of a personal internet era where sites will customize their entire layout and content based on the preferences, or habits of the browsing users.
My favorite news site will know I care about tech and biz. and never read sports news and will modify its homepage layout based on these preferences, the recommendation widget on my blog will show post recommendation based on the viewers’ recent browsing history rather than based on traffic, and so on…

Behavioral targeting opens the door on a conceptual change, not just in advertising…

Related Resources:

The Borg’s Cookies Are Delicious, Indeed – Zoli discusses privacy concerns over at Cloud Avenue.
Google AdSense Starts Behavioral Targeting- Google Blogscoped doesn’t like behavioral targeting…
Making ads more interesting– Google’s announcement on their blog.
Behavioral Targeting in Google AdSense

Similar Posts:

Playing with the Windows 7 Fish

ekampf — Sat, 07 Feb 2009 18:24:00 +0000

I just finished installing Windows 7 Beta on my home machine to find a fish swimming on my desktop:

But not just any fish, its a Siamese fighting fish, also knows a “betta fish” (or just “betta”).
A subtle Microsoft joke? Hope it does a better job selectively breeding this one (and get rid of some mutations… err… SKUs) …

Similar Posts:

Give Up Control, Think Distributed – DLD 2009 Summary

ekampf — Fri, 06 Feb 2009 11:19:00 +0000

I was very fortunate to get invited to the 2009 DLD Conference as a participant (Thanks to Yossi Vardi!). This was my first time at DLD and I can definitely say it has been the most amazing conference I have been to so far.

As defined by Steffi Czerny "DLD is interdisciplinary, creating interfaces and connecting people from the most different worlds." the conference’s “New Realities” theme encourages participants to discuss and formulates perspectives on markets following an eventful year, setting the agenda for 2009.

Or as Yossi Vardi defines it:

I told people that of my generation no one will understand instant messaging. And I was told my generation was already dead. But our generation has experiences the new generation does not have: real handshakes and real warmth of the hand. This still has to move into social networks. And this is what we provide here: This old style feeling.
…
We invite you for 4 things:

to get food for thought

to make new friends

to play

to laugh

Think Distributed

So what is the new reality we’re living in? the repeating theme in most panels I’ve been to was the same:
We’re going through a fundamental change from centralized control (over content, decision making, …) to a distributed structure.

This change becomes very clear when you go over the following panels (and most other panels too):

All these panels talk about a new reality where, in order to survive, you have to give up on centralized control and go distributed.
On the 100 Million Uniques and New Media Models we hear about new distributed media networks that distribute information rather than creating and controlling it. On cloud computing we learn about technologies allowing companies to lose control over infrastructure for increased agility and lower costs, and on On Leadership we have the story of Best Buy that transformed its culture from “communicating at employees” to empowering their employees using social media tools.

Distributed Media

On New Media Models we’ve heard from Jeff Jarvis who says that the future of media distributed – aggregating content from a lot of independent sources rather than controlling the creation of content in-house.
On his latest book, “What Would Google Do?” (which I got signed at the event. Thanks Jeff! ) discusses the importance of building a platform for content, handing over control to users to build and distribute content, making your distributed network larger.

Traditional media companies think centralized. They create and control content and then spend a fortune on advertising in order to try and attract us, the consumers, to their content. This worked pretty well for big online media companies, like Yahoo for example, so far.
However, consumer today are relying more and more on social, self organization tools, to collect, sort, filter and rate content. Tools like Digg, Twitter, delicious and Facebook replace the functions that content portals used to provide.

In this new reality media companies need to stop thinking of themselves as an end – a closed content site (or portal) users come to in order to find content – and start distributing themselves.
Google distributes itself. It has its widgets, ads, maps and videos embedded in millions of web pages that it doesn’t own and it is making its profit of this vast distribution network.
Another example for distributed content is Glam. On 100 Million Uniques, Samir Arora, CEO of Glam described his company the following way:

“Glam is a distributed network that recognizes the fact that people go to many sites as opposed to a few, and that is the fundamental change that is happening today. So instead of bringing people to one place, we find out where people go and are there.”

“with every day that was passing, fueled by Google, its easier to find more sites, as opposed to one portal.

Instead of producing women targeted content and competing with all the other women content producers out there, Glam finds the best women content producers out there (some are lone bloggers and other can be bigger media companies) and invites them to join its network. It then sells ads on those sites and shares it revenue with them. It also aggregates the best content of its network to Glam.com, where it sells ads at a higher rate and shares its revenues too.
Glam helps sites on its network by sharing technology and content, and delivering traffic (and advertisers) to its member sites.

Distributed Leadership

Changing from centralized control to a distributed structure is not limited to the media and advertising world.
On the On Leadership panel we see how Best Buy made a strategic decision to distributing control to its employees allowing them to self organize and interact using social Web 2.0 tools.
By changing the company culture “Less control from the top, more taking responsibility” the role of the CEO changes from to defining a common goal, not way to get there, and relying the companies human capital to do what it takes to get there.
So some control over what’s going on is lost, but according to Best Buy CEO Brad Anderson, “this stuff builds energy. if people have self accountability and can do something that they helped created and love they got more energy. what a customer would get when he gets into that store is that he could tell whether that store has got energy and engaged employees serving him or not”.

So if there’s one thing I have to take from my time at DLD 2009 its this: give up control, think distributed.

More on DLD:

Recommended session videos:

Similar Posts:

No Hope for Traditional Media Companies?

ekampf — Wed, 04 Feb 2009 16:01:00 +0000

The Internet is a media platform based on content from traditional media companies (Times Magazine, etc.) as well as content produced by its users – blogs, forums and other social communication platforms.

This vast new world of content is taking the lead from traditional media. Newspapers, Magazines and TV Channels are all loosing their audience in favor of the internet. Those traditional media companies who do develop strong internet presence are still having troubles as income from online advertising does not cover the decrease in income from traditional advertising and subscription fees.

The New Media Models (video) panel, on last week’s DLD conference, tries to deal with the question of business models that media companies can use to profit on the web.

When asked about what kind of new media outlet they’d start today, non of the panel’s participants would start a print newspaper or a magazine.

“I wouldn’t start a newspaper. There might be a place for a magazine to sit on my coffee table, but not for a NewsWeek or a BusinessWeek or a daily newspaper I think its absurd… I understand why the do it, revenues on print are so great vs. online but it has to go way” said Michael Arrington. According to Arrington we’re going to be consuming our news and other content online or on digital devices and once a certain threshold is reached it’ll no longer be profitable to print. “It doesn’t make any sense for news to be on paper because of its just the cost structure” he summarizes.

Jeff Jarvis said he would start a distributed content network, like Glam. According to Jeff not owning and controlling the data is what allowed Glam to literally explode to more than 110 Million unique users in 3 years. While content owners have to spend a fortune advertising to bring people in, distributed networks just go to where the people are. People today reach content via rating sites like DIGG or Twitter messages and no longer require central content portals to collect, sort, filter and rate content them. Jeff says that media companies need to start asking “How can we build platforms on which others succeed?”, they have to become platforms for content distributed around the web (like YouTube for example) rather than producing and controlling their own content.

On that same note, Arrington added that traditional media sites, who’s cost structure includes all kinds of expenses that do not produce content – like programmers, office space – can’t compete with an army of bloggers who blog from their home using their laptop and free software.

Carolyn McCall, who is the CEO of Guardian Media Group (representing the “traditional” media companies) confirmed the fact that income from the printing business is going down while internet income, even for a huge company like the Guardian Group, do not compensate for these losses. However, she also mentions that 25 million users use the Guardian’s web and that its income is 300 million dollars a year and that both numbers continue growing.

The panelist could not explain why advertising budget that are taken from the print versions of the media do not move in their entirety to the online advertising world, and they could also not find a definite business model to run media company online (even Jeff’s example for a distributed network – Glam – is still not profitable and taking VC money) but they all agree that print is phasing out and that there’s a need for a new model for online media and journalism.

Similar Posts:

Creating an Attractive Internet Company – It’s All About Emotions

ekampf — Mon, 02 Feb 2009 13:33:00 +0000

How do you create an internet company\product that can get the attention of more than 100 million unique viewers a month?

That’s the tough question that Yossi Vardi’s was trying to find an answer to on his panel at this year’s DLD conference – 100,000 Million Uniques (video) – together with Glam’s co-founder Samir Arora, YouTube’s co-founder and CEO Chad Hurley, CEO of the Mozilla Foundation Mitchell Baker and Toby Coppel who is a VP at Yahoo!.

The answer, according to Yossi and the panel speakers, who all run companies with over 100 million users, is that having a good product is not enough; there has to be an emotional connection between the product and its users.

After discussing the process of starting, marketing and improving their products the panel’s conclusion was that users engagement is the result of an emotional involvement between the users and the product and others users and a feeling of participating in creation of something new.

A good example for that is Yahoo’s Q&A service that, according to Coppel, has more than 140 million users a month. Although most users are passive and only read the questions and answers, 5%-7% of the users write answers, and they do so without getting paid – they share their knowledge with the world.
According to Coppel, the emotional connection to a product is achieved by providing a platform to self organize allowing people to create and share knowledge and experiences and find other like minded people.

Baker explained that Firefox’s success is based on the sense of mission – to build a piece of the internet that is a public assert – which turns out to be an emotional goal for a lot of people.

And what about YouTube? Chad Hurley explains its success on the focus on creating a simple-as-possible user experience and allowing people to distribute content by embedding videos – “We try to associate our easy to use service with our brand – YouTube – and to allow people to take that experience, take that video code to embed and place on their own websites and blogs so that people can experience it there as well and drive traffic back to our service. Beyond creating a service that adds value people need to connect emotionally to the brand that you’re creating”.

To demonstrate the importance of emotion, Yossi Vardi showed two YouTube videos of the aria Nessum Dorma. The first video, sang by Pavarotti only received about 9 million views while the second video, sang by Paul Potts, a hobbyist opera singer, received well over 40 million views.

Vardi later asked Hurley why the second video was so popular. “Well he’s good” was Hurley’s answer.

Similar Posts:

ASP.NET MVC RSS Feed Action Result

ekampf — Sun, 11 Jan 2009 15:41:25 +0000

Guy wrote a post about rendering an RSS feed on ASP.NET MVC using custom feed model classes and a view that renders the feed XML.

There’s a better (shorter) way for achieving the same result while leveraging on the Syndication mechanism built into .NET’s WCF.
WCF exposes the SyndicationFeed, SyndicationItem, SyndicationPerson classes which represent our data model.
In order to render this model WCF also exposes the Atom10FeedFormatter, and RSS20FeedFormatter classes that can render the feed to a stream, so all we need to do is integrate that into the ASP.NET MVC pipeline.

The ASP.NET MVC framework introduces a concept of returning an ActionResult instance as the result of Controller Actions.
This ActionResult object indicates the result from an action (a view to render, a URL to redirect to, another action/route to execute, etc).

ASP.NET MVC ships with several Action Results:

ContentResult – Simply writes the returned data to the response.
EmptyResult – Returns an empty response.
HttpUnauthorizedResult – Returns Http 401 code for non authorized access.
JsonResult – Serializes the response to Json.
RedirectResult – Redirects to another Url.
RedirectToRouteResult – Redirects to another controller action.
ViewResultBase (abstract) – Renders an HTML content as a result.
- PartialViewResult (inherits from ViewResultBase) – Renders a partial HTML response.
BinaryResult (abstract) – Returns a binary response.
- BinaryStreamResult (inherits from BinaryResult) – Writes a binary stream as a result.

So basically, to return a feed result all we need to do is define our own ActionResult implementation by deriving from ActionResult:

public abstract class ActionResult
{
    protected ActionResult();

    public abstract void ExecuteResult(ControllerContext context);
}

All we need to do is override the ExecuteResult method and write our data model to the output http stream using RSS20FeedFormatter:

public class RssActionResult : ActionResult
{
    public SyndicationFeed Feed { get; set; }

    public override void ExecuteResult(ControllerContext context)
    {
        context.HttpContext.Response.ContentType = "application/rss+xml";

        Rss20FeedFormatter rssFormatter = new Rss20FeedFormatter(Feed);
        using (XmlWriter writer = XmlWriter.Create(context.HttpContext.Response.Output))
        {
            rssFormatter.WriteTo(writer);
        }
    }
}

Now we can simply return RssActionResult as a result of our controller’s action.

Here’s a simple example:

public ActionResult Feed()
{
    SyndicationFeed feed =
        new SyndicationFeed("Test Feed",
                            "This is a test feed",
                            new Uri("http://Contoso/testfeed"),
                            "TestFeedID",
                            DateTime.Now);

    SyndicationItem item =
        new SyndicationItem("Test Item",
                            "This is the content for Test Item",
                            new Uri("http://Contoso/ItemOne"),
                            "TestItemID",
                            DateTime.Now);

    List<SyndicationItem> items = new List<SyndicationItem>();
    items.Add(item);
    feed.Items = items;

    return new RssActionResult() { Feed = feed };
}

… and that’s it!

A more elegant solution that leverages existing framework capabilities.

RSS Feed with the new ASP.NET MVC Framework (Brad Abrams)
ASP.NET MVC RSS Feed Action Result (Guy Burstein)
RSS Feeds in ASP.NET MVC – StackOverflow

Similar Posts:

Is Windows Live Still Alive?

ekampf — Mon, 05 Jan 2009 19:38:05 +0000

Originally published on Cloud Avenue.

About a month ago, Microsoft rolled out its previously announced 3rd wave of Windows Live Services refresh.
The purpose of the release was to position the Live Services as the central hub for everything you do online – the new Live Home shows input from your various services (Hotmail, SkyDrive, etc.) as well as an activity stream composed of your friends’ activities on both Live Services as well as other external services such as Twitter, Flickr and blogs…

What initially seemed like a promising new start for Microsoft, leveraging on its already existing Live Messenger social network now seems, a month or so after its rollout is pretty much dead. Most of the people I know (including myself) that tried using the new Live Home quickly gave it up as its simply as there’s simply nothing to do there – with an anemic design and an activity feed that does not allow any interaction, Microsoft’s new homepage is simply a “Read-only Social Network”.

Microsoft missed out on what makes a social site successful – social interaction! duh!

It should have encouraged users to interact using its platform by allowing rating / commenting / sharing / replying / etc. on feed items.
By not doing so Microsoft is actually making users leave its homepage platform to use other services (Twitter, FriendFeed, etc.) when they actually want to do something with the content – rendering its platform pretty useless.

All that is left now is to wait for the next wave, expected at the end 2009 Q4, and hope for a more mature product that can… well.. Live.Similar Posts:

A Year’s Worth of Popular Posts

ekampf — Wed, 31 Dec 2008 17:04:00 +0000

It’s the end of the year, and like most bloggers I decided to take a look at my blog’s performance over the last year…

My Top Posts (According to Google Analytics)

Here’s are my top 10 posts from 2008, as ranked by Google Analytics:

Personal Favorites

The following are a couple of my personal, somewhat overlooked, favorites:

Looking For New Adventures – On May of this year I left my (pretty cool) job with the Duet team at SAP to face the unknown…
Google Chrome is a Failure… Surprised? – Originally published on Cloud Avenue and triggered an interesting discussion worth going through.
Are You Designing for Bigfoot?
Is Google Replacing My OS Again?! which was followed by On Google’s Chrome and The Future of Cloud Computing.
Introducing Kampyle – The Next Generation of Online Feedback. Just because I got to be the first to write about my friends company and I think this post written a lot better than the many posts that followed on TechCrunch etc.

That’s it. All wrapped up and ready for 2009!

Similar Posts:

99 Ways to Become a Better Developer

ekampf — Fri, 05 Dec 2008 19:04:33 +0000

I encountered this post on my weekend reading. 91 Surefire Ways to Become an Event Greater Developer contain a comprehensive guide linking to all sort of blog posts providing insights on improving your skills as a developer.

While the list is very long and sometimes debatable it does have some interesting pointers. If you do nothing else, delve into item #8: Learn Programming by Not Programming referring to the following post by Jeff Atwood.

The topic in question is why some developers outperform their peers regardless of their accumulated experience:

But the dirty little secret of the software development industry is that this is also true even for people who can program: there’s a vast divide between good developers and mediocre developers.

A mediocre developer can program his or her heart out for four years, but that won’t magically transform them into a good developer. And the good developers always seem to have a natural knack for the stuff from the very beginning.

The answer lies in the quotes taken from Bill Gates remarks:

“The nature of these jobs is not just closing your door and doing coding, and it’s easy to get that fact out. The greatest missing skill is somebody who’s both good at understanding the engineering and who has good relationships with the hard-core engineers, and bridges that to working with the customers and the marketing and things like that.”

Eric Sink makes the distinction even clearer in You Need Developers, Not Programmers drawing a distinction between Programmers who are only excited about writing code and basically only care about doing that, and Developers who contribute to the software product in many ways.

The Great Programmer\Hacker Stereotype

You all know that guy (hell, most of us were that guy when we just started out, I know I was) – he has great technical skills, likes writing code and can spend hours within his IDE writing code that’ll make most of us scratch our head. Yet, he views the world only in one dimension – code. Business? that’s for the managers to figure out. Sales\Marketing? annoyances for others to take care of. Documentation? but the code is so obvious…Builds? Deployment? Configuration? …

Passion for code is a great quality. But as a specialist its all too easily digging yourself deeper and deeper into a skill you’ve already proven yourself to be capable at when you’d be better of using the time to cultivate other skills that are part of the process of making software – rendering yourself obsolete over time…

The great hacker is a one trick pony – he writes great code but that’s about it…
Most of these guys end up working alone as consultants or freelancers where they don t have to care about that other stuff, or they end up as programmers at some big firms where there’s more room for specialists doing specific jobs (Architects to architecture, PMs do project management, Programmers code…).
On the other hand, those who truly like making software, open up to the other aspects of software development.
When that change in mindset happens, that’s when you can truly grow exponentially…

So what do I do?

Ok, I guess you got the point… But how do you get started? Here are my own 5 cents on the topic…

Read, Read and Read Some More…

We’re in an industry that is moving forward at a fast pace. Technology becomes obsolete every year and a half or so and as developers we have to constantly struggle to keep up. Books are not only great to help you keep up but also to expand your knowledge to other fields.
There are plenty of interesting books and blogs about, well, pretty much everything.
Here are some recommendations to get you started:

The Inmates Are Running the Asylum	The Pragmatic Programmer	Made to Stick	Crossing the Chasm	The Innovator’s Dilemma	Eric Sink on the Business of Software
					(most of it is available online here)

Oh and one word about programming books: the best ones are timeless, transcending choice of language, IDE and platform.
I try to stay away from them thick, heavy, language\platform specific references – most of them go out of date after a year or so anyway and most of the information there could be easily obtained elsewhere (online – Google, the product’s docs, blogs…)

Most programming big are just a waste of your time (and money…)

Contribute to an Open Source Project

Back in the days of Delphi I was involved in Project JEDI dedicated to exposing different APIs (especially the Win32 API) to Delphi developers.
I learned a lot working with the JEDI code base, documentation, samples and other team members.
Later when it was time to get drafted to the Israeli Army (we all have to do it at 18 here) the experience, credit and code samples help me land a (very) exclusive position as a programmer. Who knows where I’d be today if I didn’t qualify and had to serve as a combatant…

Contributing to an open source project is a great way to gain experience, learn and get better.
There are no job interviews to pass, degree requirements or commitment to working hours or schedule required – you can just join in and start submitting patches or contribute in ways other than code (submit bugs, docs, support, …).

You can learn a lot just from studying the code and interacting with your peers…

Contributing to open source shows dedication and passion – its a walking talking resume.

Get a mentor

Find yourself a mentor or mentors who can teach you about different aspects of the business. I’ve had several at SAP and talking with them proved to be an invaluable asset (If you’re reading, thanks! )

It doesn’t have to be official mentoring which is part of the person’s goals or job description. Many of your peers are experts in their field and they’ll be happy to show you around if you just show some interest…

Become a Mentor

Great developer are eager to learn… and teach. Can you pass you passion and knowledge to others?

You can also…

Open a blog about your experience, opinions, etc.
Start answering questions at stackoverflow.com and collect achievements

Land an Internship

Try getting an internship in a different role. When I was in SAP they had a special program allowing employees to apply for a ~6 month position somewhere within the company. The reason behind it was to get employees familiar with different aspects of the company. Maybe product management, marketing or sales in not really your first choice of profession but why not try it for a couple of month without the risk of going through a career change? How cool is that? I’m sure many large corporations has something similar and even if not, it can’t hurt if you come up with such an interesting offer to your boss…

Own a Product Area

Get ownership on some part of the product your team is working on. Weather a specific component or a vertical (like Security) you should be in charge of getting it done – from getting the definition done with the product\sales\business team, through UX, development, QA, etc…
There’s nothing better than learning about the process of software development through experiencing the entire cycle…

Innovate

Start something new. When working on Duet we’ve had many issues getting the thing deployed. So I made a tool for (myself mainly) our QA and RIG (regional implementation group – the guys who work with customers) to help diagnose problems. This later became the official Duet Support Tool and got its own dedicated development time. Is your product, development environment perfect? I’m sure not… find a need a feel the gap…

Why? If by owning a product area you learned about the entire development cycle, here you’ll learn about defining and “selling” to the team…

Bonus Reading…

Another link worth visiting is the one about the Metrosexual Developer. Funny and true…

Related:

So you want to be a great developer?

Similar Posts:

WordCamp Israel 2008

ekampf — Sat, 15 Nov 2008 10:14:07 +0000

We’ll be at the WordCamp Israel 2008 convention tomorrow showing off some of the latest Nuconomy developments (and of course, our WordPress plugin).

Here’s Niv surveying last year’s visitors:

See you there!

Similar Posts:

Touch Panning (Kinetic Scrolling) in Windows 7

ekampf — Fri, 14 Nov 2008 15:38:19 +0000

If you like the scrolling functionality as implemented in the iPhone (and Zune etc.) you’ll be happy to know its part of the Windows 7 operating system and implemented for anything with a scrollbar.

More Windows 7 goodies on Rafael Rivera’s blog…

Similar Posts:

Microsoft Updates Its Windows Live Services

ekampf — Thu, 13 Nov 2008 21:30:16 +0000

(Cross posted from CloudAve)

Microsoft announced today its rollout plans for the 3rd wave of Windows Live services.

The goal of this latest release wave, according to company officials, is to simplify the use of the offered services and unify the user’s entire online experience into the Windows Live interface.
Microsoft is planning to rollout the new services, currently in beta, to the public within the 1-2 months timeframe.

Windows Live Goes Social

One of Microsoft’s main emphasis with the current wave of services is on social networking between users using its services.

Microsoft finally figured out that its Live Messenger with about 268 million users worldwide, is by far the most popular instant messaging software in the world, is actually a social networks. With the new release, your Live Messenger contacts are now your Friends and you can see aggregated information about their activities on the net.

Very much like Plaxo, FriendFeed etc. Microsoft allows users to bring into their profile content they create in all sorts of services on the web (Live Services, Flickr, LinkedIn, blogs and RSS feeds, …) and share it with their friends and colleagues.
When users add photos, write reviews, and update their profiles directly on Live.com, that content will be put into their activity stream as well.
This activity stream is exposed in all sort of ways throughout Microsoft’s services interface.

For example, Microsoft’s new Live Home portal shows the latest events in your social network. When emailing a friend or chatting on Messenger you’re also able to interact with that friend’s activity stream and more…

Not just for private consumers…

I’ve been told that all these new service updates will not skip Windows Live Domains used by universities and organizations to create a personalized version of Microsoft’s services.
If that really the case, having all these new social capabilities as part of its domain offering can be amazing for collaboration and communication inside the organization.
While Google doesn’t seem to care about its Google Applications for Your Domains customers its good to see that Microsoft is going forward with Live Domains.
This latest update may just be the final straw I need to make the switch to Live Domains…

Where’s Live Mesh?!

It will be really interested to see where Live Mesh comes into the picture in regards to all of these Live services.
Live Mesh should be the glue bridging between Microsoft’s online services and its offline applications and devices (S+S) allowing users to sync all their content- contacts, photos, events, favorites, etc. – across devices and services.
Unfortunately, there’s no clear answer for that…

During the launch we’ve only heard about Live Sync allowing users to sync photos across computers. Some sources say its an incarnation of FolderShare and in any case it doesn’t seem to be based on Live Mesh technology.
With Live Mesh being one of Microsoft’s core platform offering its really hard to understand why we need to have Live Sync too…

Other notes…

All the services are released simultaneously in all countries and in 48 (!) languages.
Windows Live Skydrive size limit has changed from 5GB to 25GB
Windows Live Hotmail looks and feels a lot better to use.
I’ve uploaded all the screenshots of the new services to my SkyDrive:

Similar Posts:

DeveloperZen

Building an iPhone Application

First some information about the team

iPhone Development

Backend Development

High Performance at Massive Scale – Lessons learned at Facebook

What would Twitter do with $100 million?

Manage Companies Presence on Twitter

Local Markets, Local Social Network

Data Mining – Handling Missing Values the Database

1. Ignore the data row

2. Use a global constant to fill in for missing values

3. Use attribute mean

4. Use attribute mean for all samples belonging to the same class

5. Use a data mining algorithm to predict the most probable value

Additional Notes

Moving Your Application to Amazon’s Cloud

Step One: Move Static Content to S3

Step Two: Move Web Servers and Backend Servers to EC2

The Web Servers

The Database

Step Three: Scale. Take advantage of the cloud services

Facebook, Hadoop and Hive

Yahoo Releases Its Own Hadoop Distribution

New Features for Amazon EC2 – Now You Can Truly Scale Applications

Introduction to MapReduce for .NET Developers

The MapReduce Programming Model

A Naive Implementation in C#

Map-Reduce Word Counting Sample – Revisited

Other Examples

Map-Reduce in the Real World

A Visit to Maraboo, Restaurants as Tribes in the Digital Age

The Web is About Openness… Can Restaurants be Open?

Final Thoughts and.. Food Photos!

Weekend Inspirations – Big Data Visualizations, Innovative Man-Machine Interactions

Designing “Big Data”

Cool New Computer Interaction Technology

Kiva Robots Invade the Warehouse

Developing a Robust Data Driven UI Using WPF – An Overdue Summary (and full source code)

Unit Testing

Using Lambda Expression for DataBinding

Reference Applications and Guidance

Microsoft Can Clone Twitter?!

Google’s New Behavioral Ad Targeting Should be Excellent for All

Playing with the Windows 7 Fish

Give Up Control, Think Distributed – DLD 2009 Summary

Think Distributed

Distributed Media

Distributed Leadership

No Hope for Traditional Media Companies?

Creating an Attractive Internet Company – It’s All About Emotions

ASP.NET MVC RSS Feed Action Result

Related Posts

Is Windows Live Still Alive?

A Year’s Worth of Popular Posts

My Top Posts (According to Google Analytics)

Personal Favorites

99 Ways to Become a Better Developer

The Great Programmer\Hacker Stereotype

So what do I do?

Read, Read and Read Some More…

Contribute to an Open Source Project

Get a mentor

Become a Mentor

Land an Internship

Own a Product Area

Innovate

Bonus Reading…

WordCamp Israel 2008

Touch Panning (Kinetic Scrolling) in Windows 7

Microsoft Updates Its Windows Live Services

Windows Live Goes Social

Not just for private consumers…

Where’s Live Mesh?!

Other notes…