atbrox

Combining Hadoop/Elastic Mapreduce with AWS Redshift Data Warehouse

Amund Tveit — Mon, 25 Feb 2013 12:13:25 +0000

There are currently interesting developments of scalable (up to Petabytes), low-latency and affordable datawarehouse related solutions, e.g. AWS Redshift (cloud-based) [1] Cloudera’s Impala (open source) [2,3] Apache Thrill (open source) [4] This posting shows how one of them – AWS Redshift – can be combined with Hadoop/Elastic mapreduce for processing of semi/unstructured data. 1. Processing [...]

Mapreduce Algorithms – Presentation held at O’Reilly Strata Conference

Amund Tveit — Mon, 01 Oct 2012 15:24:20 +0000

My presentation held at O’Reilly Strata Conference in London, UK, October 1st 2012 Best regards, Amund Tveit

Atbrox @ O’Reilly Strata Conference in London

Amund Tveit — Wed, 26 Sep 2012 07:28:25 +0000

Atbrox is participating and holding a Hadoop/Mapreduce algorithm related presentation at the O’Reilly Strata Conference in London October 1st and 2nd. If you are there and would like to meet Atbrox send an email to info@atbrox.com

A large-scale in-memory storage example – social network data

Amund Tveit — Wed, 16 May 2012 08:25:22 +0000

This posting is a follow-up to the large-scale low-latency (RAM-based) storage related price estimates in my previous posting Main takeaways from Accel’s Big Data Conference. Assume you were to store and index large amounts of social network updates in-memory, e.g. tweets. 1) fetch some tweets curl https://stream.twitter.com/1/statuses/sample.json?delimited=length -uAnyTwitterUser:Password > yourfilename 2) gather some stats about [...]

Main takeaways from Accel’s Big Data Conference

Amund Tveit — Mon, 14 May 2012 07:24:36 +0000

Attended Accel Partners Big Data conference last week. It was a good event with many interesting people, a very crude estimate of distribution: 1/3 VCs/investors, 1/3 startup tech people, 1/3 big corp tech people +-. My personal 2 key takeaways from the conference: Realtime processing: hot topic with many companies creating their own custom solutions, [...]

atbr now has Apache Thrift support

Amund Tveit — Wed, 02 May 2012 10:06:10 +0000

atbr (large-scale and low-latency in-memory key-value pair store) now supports Apache Thrift for easier integration with other Hadoop services. Thrift Example Checkout and install atbr Prerequisite Install/compile Apache Thrift – http://thrift.apache.org/ Compile a atbr thrift server and connect using python client Python thrift api example Stay tuned for other updates on atbr. Rough roadmap Increased [...]

atbr – supports websocket-based sharding

Amund Tveit — Tue, 01 May 2012 07:38:50 +0000

atbr (large-scale and low-latency in-memory key-value pair store) now supports websocket-based sharding for parallel deployments. Websocket Sharding Example Checkout and install atbr Start 3 servers loaded with data Start shard server talking to shards Connect to shard server and lookup key=key1 Stay tuned for other updates on atbr, here is a rough roadmap. Increased concurrency [...]

atbr – large-scale in-memory hashtables (in Python)

Amund Tveit — Wed, 25 Apr 2012 14:13:57 +0000

Large-scale in-memory key-value stores are universally useful (e.g. to load and serve tsv-data created by hadoop/mapreduce jobs), in-memory key-value stores have low latency, and modern boxes have lots of memory (e.g. EC2 intances with 70GB RAM). If you look closely many of the nosql-stores are heavily dependent on huge amounts of RAM to perform nicely [...]

Monodroid with Sencha Touch for App Development

Amund Tveit — Wed, 07 Mar 2012 13:28:09 +0000

Selecting development tools for an app depend on several criteria, e.g. Should the app run on multiple mobile/tablet/pad platforms? (e.g. Android, iOS, Windows 8 etc.) Access to native features Is the UI “game like” (e.g. custom graphics) or “business like” (e.g. standardized forms/input fields etc.) Mono – the opensource version of C#/.net platform – seems [...]

Distributed Tracer Bullet Development

Amund Tveit — Tue, 24 Jan 2012 10:18:42 +0000

Tracer Bullet Development Tracer Bullet Development is finding the major “moving parts” of a software system and start by writing enough code to make those parts interact in a real manner (e.g. with direct API-calls, websocket or REST-APIs), and as the system grows (with actual functionality and not just interaction) keep the “tracer ammunition” flowing [...]