Unfolding the Web

MapReduce related patterns

Bruno Pedro — Mon, 26 Sep 2011 21:16:15 +0000

Here’s a list of possible MapReduce related patterns I’ve been thinking about:

map-update: update each mapped document and emit its updated or original version;
map-delete: delete each mapped document;
map-reduce-map: map results of a map-reduce;
map-map: map results of a map;
map-recurse: apply recursion to the mapping function until a stop condition occurs;
any combination of patterns, e.g., map-reduce-map-update.

Example usage:

get the e-mail address of every customer with a negative balance: map-reduce-map. First, map-reduce to get the aggregate balance for each customer, then map again to get only customers with negative value and emit their e-mail address;
delete all documents older than one month: map-delete. First, map to get all documents older than one month and then delete each one;
get a list of documents and mark them as read: map-update. First, map to get the list of documents according to a given criteria, then update each document marking it as read;
and so on…

Share now!

node-fs npm package

Bruno Pedro — Thu, 14 Apr 2011 10:30:14 +0000

node-fs is now available as an npm package. Just run npm install node-fs, require(‘node-fs’) and have fun.

Share now!

Blog posts firehoses

Bruno Pedro — Wed, 13 Apr 2011 23:40:55 +0000

Three more firehoses to get all blog posts from the following platforms:

blogger posts, through their changes.xml;
tumblr posts, through superfeedr‘s track feature;
WordPress.com posts, through their firehose feature.

While the first feed is free of charge, the other two have an attached price tag.

Also worth investigating is Paul Kinlan’s faux firehose for blogger.

Share now!

Drinking from the firehose

Bruno Pedro — Tue, 12 Apr 2011 15:22:27 +0000

Some firehoses to ~~drink~~ read from, other than twitter’s:

Google Buzz, through their activity firehose API methods;
FriendFeed, using their real-time updates methods.

While the first works via pubsubhubhub, the second uses a combination of long polling and a cursor that helps you make subsequent calls.

Share now!

Are we inside a bubble?

Bruno Pedro — Mon, 11 Apr 2011 16:03:36 +0000

Very interesting post by Brad Feld about technology cycles.

If you recognize that we are in a strong, positive, upward segment of the current “tech company creation cycle”, that’s more than enough. You should accept that we’ll be back in a downward part of the cycle at some point, but that we don’t know if it’ll be in a week, month, year, or decade. — I Encourage Entrepreneurs To Ignore The Word “Bubble”

Share now!

FPC: a floating-point compressor

Bruno Pedro — Fri, 08 Apr 2011 13:14:59 +0000

FPC is a very fast lossless compressor with better results than most traditional algorithms:

FPC compresses linear sequences of IEEE 754 double-precision floating-point values by sequentially predicting each value, xoring the true value with the predicted value, and leading-zero compressing the result.

Share now!

Three things to remember

Bruno Pedro — Fri, 08 Apr 2011 00:00:32 +0000

Three simple things that can make all the difference.

talk about things you’ve done, not about things that you wish you did;
measure reactions to everything you do, react by changing things if needed;
stop planning and just do it!

Share now!

node-fs

Bruno Pedro — Wed, 05 Jan 2011 19:20:30 +0000

Following my previous post about recursive directory creation with nodejs, I created the node-fs repository on github. node-fs is a way to extend nodejs original fs library, adding recursive directory creation to the mkdir() and mkdirSync() functions.

Share now!

Is OAuth Really Secure?

Bruno Pedro — Wed, 05 Jan 2011 01:01:27 +0000

“Is OAuth Really Secure?” is the title of a talk I gave at the IBWAS’10 conference, last December.

Is the OAuth protocol really secure? Even though the OAuth authorization protocol has been published as the RFC 5849 and is being widely adopted by large Internet companies, it’s important to stress out its possible security vulnerabilities.

This talk focuses on the OWASP Top 10 Application Security Risks and how OAuth is affected by them. While some of the security risks are mitigated by OAuth, developers need to take some action to prevent other risks from affecting their implementations.

If you have time to watch, there’s also a video of the talk.

Share now!

MapReduce literature

Bruno Pedro — Wed, 22 Dec 2010 13:13:45 +0000

Just a quick note to point to two valuable MapReduce related resources:

“MapReduce: Simplified Data Processing on Large Clusters“, by Jeffrey Dean and Sanjay Ghemawat: the original paper describing Google’s MapReduce strategy;

MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper.

“Data-Intensive Text Processing with MapReduce“, by Jimmy Lin and Chris Dyer: a fantastic book covering almost every topic related with MapReduce.

This book focuses on MapReduce algorithm design, with an emphasis on text processing algorithms common in natural language processing, information retrieval, and machine learning. We introduce the notion of MapReduce design patterns, which represent general reusable solutions to commonly occurring problems across a variety of problem domains.

Do you suggest any other interesting publications?

Share now!