atbrox

A large-scale in-memory storage example – social network data

Amund Tveit — Wed, 16 May 2012 08:25:22 +0000

This posting is a follow-up to the large-scale low-latency (RAM-based) storage related price estimates in my previous posting Main takeaways from Accel’s Big Data Conference. Assume you were to store and index large amounts of social network updates in-memory, e.g. tweets. 1) fetch some tweets curl https://stream.twitter.com/1/statuses/sample.json?delimited=length -uAnyTwitterUser:Password > yourfilename 2) gather some stats about [...]

Main takeaways from Accel’s Big Data Conference

Amund Tveit — Mon, 14 May 2012 07:24:36 +0000

Attended Accel Partners Big Data conference last week. It was a good event with many interesting people, a very crude estimate of distribution: 1/3 VCs/investors, 1/3 startup tech people, 1/3 big corp tech people +-. My personal 2 key takeaways from the conference: Realtime processing: hot topic with many companies creating their own custom solutions, [...]

atbr now has Apache Thrift support

Amund Tveit — Wed, 02 May 2012 10:06:10 +0000

atbr (large-scale and low-latency in-memory key-value pair store) now supports Apache Thrift for easier integration with other Hadoop services. Thrift Example Checkout and install atbr Prerequisite Install/compile Apache Thrift – http://thrift.apache.org/ Compile a atbr thrift server and connect using python client Python thrift api example Stay tuned for other updates on atbr. Rough roadmap Increased [...]

atbr – supports websocket-based sharding

Amund Tveit — Tue, 01 May 2012 07:38:50 +0000

atbr (large-scale and low-latency in-memory key-value pair store) now supports websocket-based sharding for parallel deployments. Websocket Sharding Example Checkout and install atbr Start 3 servers loaded with data Start shard server talking to shards Connect to shard server and lookup key=key1 Stay tuned for other updates on atbr, here is a rough roadmap. Increased concurrency [...]

atbr – large-scale in-memory hashtables (in Python)

Amund Tveit — Wed, 25 Apr 2012 14:13:57 +0000

Large-scale in-memory key-value stores are universally useful (e.g. to load and serve tsv-data created by hadoop/mapreduce jobs), in-memory key-value stores have low latency, and modern boxes have lots of memory (e.g. EC2 intances with 70GB RAM). If you look closely many of the nosql-stores are heavily dependent on huge amounts of RAM to perform nicely [...]

Monodroid with Sencha Touch for App Development

Amund Tveit — Wed, 07 Mar 2012 13:28:09 +0000

Selecting development tools for an app depend on several criteria, e.g. Should the app run on multiple mobile/tablet/pad platforms? (e.g. Android, iOS, Windows 8 etc.) Access to native features Is the UI “game like” (e.g. custom graphics) or “business like” (e.g. standardized forms/input fields etc.) Mono – the opensource version of C#/.net platform – seems [...]

Distributed Tracer Bullet Development

Amund Tveit — Tue, 24 Jan 2012 10:18:42 +0000

Tracer Bullet Development Tracer Bullet Development is finding the major “moving parts” of a software system and start by writing enough code to make those parts interact in a real manner (e.g. with direct API-calls, websocket or REST-APIs), and as the system grows (with actual functionality and not just interaction) keep the “tracer ammunition” flowing [...]

Workshop: Mapreduce’12 – 3rd International Workshop on Mapreduce and its applications)

Amund Tveit — Wed, 30 Nov 2011 08:16:22 +0000

If you are interested in Mapreduce or Hadoop I recommend submitting to or attending the following workshop. The Third International Workshop on MapReduce and its Applications (MAPREDUCE’12) June 18-19, 2012 HPDC’2012, Delft, the Netherlands. http://graal.ens-lyon.fr/mapreduce/ SCOPE ===== Since its introduction in 2004 by Google, MapReduce has become the programming model of choice for processing large [...]

Workshop: Searching for fun

Amund Tveit — Wed, 23 Nov 2011 20:59:59 +0000

If you are interested in search I recommend you to consider submitting a paper to or attending the Searching 4 fun workshop* (I just joined as a program committee member) which is going to be held in Barcelona in April 2012. Call for Papers The topics of the workshop will be evaluation focused and include [...]

Mapreduce & Hadoop Algorithms in Academic Papers (5th update – Nov 2011)

Amund Tveit — Wed, 09 Nov 2011 12:26:27 +0000

The prior update of this posting was in May, and a lot has happened related to Mapreduce and Hadoop since then, e.g. 1) big software companies have started offering hadoop-based software (Microsoft and Oracle), 2) Hadoop-startups have raised record amounts, and 3) nosql-landscape becoming increasingly datawarehouse’ish and sql’ish with the focus on high-level data processing [...]