Basic node setup

In this brief section I will explain what needs to be defined in the puppet recipe in order to perform basic setup of a machine (getting a dedicated user account to run hstack and setting the correct ssh permissions). The recipes we’re going to use are stored in the manifests/virtual.pp and manifests/environment.pp.

First up, adding a new user to the system. The recipe for creating users does not work fully out of the box – you need to give it your own ssh keys to use. To do so, replace the files under templates/ssh_keys/keys with actual content (they’re just dummy files right now). You might also want to change the password – you need to write the already encrypted version into manifests/virtual.pp. The default password is hadoop, although it might not seem so.

Another customization you can make is to the system settings. These are controlled through the environment.pp file. In this file you can change the maximum number of file descriptors (default 200.000), aliases for commands and the tmpwatch(http://linux.about.com/library/cmd/blcmdl8_tmpwatch.htm) deamon configuration.

Once this is done, you can add your first node, run the configuration and enjoy the results – a brand new user ready to take on Hadoop. :)

Deploying Hadoop

The hadoop module has options and configurations you need to change to make it suit your setup. The most important ones you need to take care of are: in the modules/hadoop/templates/conf/ folder you can create a sub-folder for each separate environment you want to manage. This allows you to keep completely different templates and use the same code to push them to their respective locations. This is where the hadoop (core, hdfs and map-reduce) configuration templates are stored. By default only the critical properties are filled in, some with values, but most are using variables (you can spot them by the <%= %> marks). These variables can be set on a per-node basis, but I just define a base-node with the common values and have all the other nodes in a cluster extend that. Some of the variables that you need to set are:

• $hadoop_version – the version of hadoop to deploy. The recipes are assuming as version the part of the tar name after hadoop-. For example if your generated archive is called hadoop-core-0.21.0-31, the version is “core-0.21.0-31″. Another assumption is that the archive contains a folder with the same name, which will be unpacked in the user’s home (/home/hadoop) and then symlinked to the final $HADOOP_HOME destination. This tactic allows you to retain all versions ever deployed, and you can easily switch back between them by just re-creating the symbolic link to an older folder.
• $hadoop_home – final directory to which the unpacked hadoop will be symlinked to
• $environment – match this with the subfolder in the templates/conf that you want to use
• $hadoop_default_fs_name – the uri to the NameNode; this gets pushed into hdfs-site.xml and core-site.xml
• $hadoop_namenode_dir – the folder where the NameNode stores its files
• $hadoop_datastore – the folders where the datanode stores its files. This is expressed as a list: ["/var/folder1", "/var/folder2"]
• $mapred_job_tracker – uri to the Hadoop Map-Reduce job-tracker.
• $hadoop_mapred_local – the local folder where the TaskTrackers should store its files. This is also a list of folders.

Hadoop HDFS High Avalability

To use this you need to dedicate two machines to the NameNode role and set some other variables (on both machines):

• $virtual_ip – the common IP address that both computers will be sharing
• $hostname_primary | $hostname_secondary – the hostnames of the two machines to use
• $ip_primary | $ip_secondary – the IP addresses of both machines
• $disk_dev_primary | $disk_dev_secondary – the device to use as the starting partition for drbd
• $hadoop_namenode_dir – where the namenode will store its files; this will also be the mount point for the newly created /dev/drbd

Deploying Zookeeper

A standalone zookeeper deployment is recommended for HBase. You should have at least 3 servers running zookeeper – if your cluster is low on load, you can just use 3 of your regionservers. To define a zookeeper node, import the zookeeper module and set these variables:

• $zookeeper_version – same as for hadoop, the version to deploy
• $zookeeper_datastore – where to store the zookeeper data files
• $zookeeper_datastore_log – where to store the zookeeper log
• $zookeeper_myid – this needs to be configured for each individual node, as it assigns an unique id to the machine

Deploying HBase

The HBase deployment is very similar to the Hadoop one. You need to adjust the HBase configuration stored under modules/hbase/templates/conf/$environment to add or remove any properties specific to your installation. In the configuration files you will spot some of the variables you have to set, like:

• $hbase_version – what version to deploy; similar to the Hadoop case, this is part of the archive and folder name. If you use the default build script, it should pack it just right
• $hbase_home – final directory to which the unpacked hbase will be symlinked to
• $zookeeper_quorum – comma separated list of zookeeper nodes
• $hbase_rootdir – HBase directory in HDFS, usualy /hbase

This was a quick run through some, but not all of the options that each recipe provides and requires. I strongly encourage you to go look in the provided nodes.pp which has a sample node configured with all of the options. Also, as a best practice, when adding your own properties to the template configuration files, you should try to use variables and set those on a per-node basis where applicable.

Down and up again

At first we had erratic results: No two performance runs were the same, and our servers were almost idling. We started to get rid of the moving pieces in order to identify the hot spot. First we removed thrift, than the entire testing framework and wrote a simple test case (basically a long Java loop) which ran that on the cluster. Then we got rid of the cluster and ran the test on a single node. We profiled the code, and then got rid of the large table too. We ended up with running the test program, locally, on a table with one row (got rid of the I/O overhead in the equation, everything was served from memory).

Finally, after looking at the profiler logs and with the help of the guys in #hbase, we got to HBASE-2180. After applying the patch, we went back up the stack, and added all the layers of complexity back, one by one. In the end server load increased, mean request time decreased and the results were consistent.

One problem with this approach is that by trying to pinpoint the issue, you can lose track of the big picture and might even end up hiding the initial cause. You have to measure performance every step of the way to make sure the problem still reproduces.

The second big problem was that sometimes the throughput went down to 0 req/s for short periods of times (blackouts). We first looked at all logs on all the nodes, seeing some weird timeouts between them. At the time, the servers and network were not under heavy load. After a lot of debugging we turned to the OS logs (there was nothing left :): in the dmesg output, iptables was screaming out ip_conntrack: table full, dropping packet.. It turns out that on busy network servers, the firewall can get to a point where it will merrily drop connections ⁶, if the length of the connection queue is bigger than the default cap.

We were so fixated on identifying problems in the upper layers, that we totally forgot about the operating system. It usually IS your code, but don’t take anything for granted when identifying performance problems. Low-level components, like network and file system play a big part in performance tests.

Performance numbers: Hard Mode

How do you determine when you hit the limits of your system? More than that, how do you identify the actual problems starting from the initial results? Simply pushing on until something crashes or stops won’t do it. What’s worse, it won’t tell you anything about whether you are walking through the hardware sweet spot, or if you are spending money in the right place.

In order to identify the limits or bottlenecks you need a way to look inside the system – something like an x-ray. For code there are profilers, instrumentation and log files. Linux and derivatives have plenty of tools that you can use to get information on how CPU, memory and I/O cycles (both HDD and network) are spent. We use an ad-hoc combination of top, htop, iostat, dstat, netstat, iptraf, sar. Most of these are frontends to /proc, and functionality is duplicated between them. We look at different things, like overall network bandwidth used, CPU %, number of page faults, I/O queues average length, etc.

Cost

We are very careful what we spend our money on. Under-dimensioned hardware is a performance killer, but over-dimensioned one increases our costs. There’s no reason in sticking 20 hard drives in a server if the CPU / mainboard can’t handle it. In the same vein, having huge processing power and memory is useless when your network communication is strangled by a 100Mbps network board, or even worse, by another poorly dimensioned equipment down the road (core switch, etc). All the components of a machine (and by extension, of the cluster) have to have the correct size.

Our configuration uses 24 hard disks per server. Increasing the number of spindles significantly improves I/O by parallelization. But there’s always a push / pull relation between failover, performance and costs.

For lots of nodes, processing power (MapReduce) is higher, but you eat more power / network traffic in the data center (which translates in bigger costs). On the other hand, with smaller number of beefier servers, power utilization is smaller, but you run into the risk of having big churn during failover (a machine dies)¹¹, because there’s a lot of data that need to get re-balanced in HDFS.

The point is that we’re careful about costs. We try to be thorough, get an in-depth look on how our hardware is performing, and decide whether we need to upscale or downscale our components for optimal usage.

Therefore in our production cluster, we decided to go with 16 2.5″ disks instead of 24, trading spindles for less power usage. 16 disks are enough for our current usage, and we can always add more when we run out of space.

Hammers, anvils and other assorted sundries

We use a bunch of great tools that really made life easier for us:

The main performance tool that we use is Grinder. Grinder lets you write your tests in Jython, and plug them in with whatever you want to test(DNS speed or weird protocols ? No problem). HTTP is built in; it has a console that lets you orchestrate your tests across multiple injector nodes. Grinder has a steeper learning curve, but it’s worth it, and we highly recommend it.

I already mentioned all the system monitoring tools that we use. When the performance tests are finished, we use R to load the data sets. R is great because it allows for an incremental approach to data analysis. It’s really fast, and you can dice the data, look at the distributions, eliminate outliers, etc. Also highly recommended.

More specific details on how we actually use everything, with recipes, in a later post. Also, now that YCSB just came out as open-source, we’ll give that a go as well.

How we work

We usually develop against trunk code (for both Hadoop and HBase) using a mirror of the Apache Git repositories. We don’t confine ourselves to released versions only, because we implement fixes, and there are always new features we need or want to evaluate. We test a large variety of conditions and find a variety of problems – from HBase or HDFS corruption to data loss etc. Usually we report them, fix them and move on. Our latest headache from working with unreleased versions was HDFS-909 that causes the corruption of the NameNode “edits” file by losing a byte. We were comfortable enough with the system to manually fix the “edits” binary file in a hex editor so we could bring the cluster back online quickly, and then track the actual cause by analyzing the code. It wasn’t a critical situation per se, but this kind of “training” and deep familiarity with the code gives us a certain level of trust regarding our abilities to handle real situations.

It’s great to see that it gets harder and harder to find critical bugs these days, however, we still brutalize our clusters and take all precautions when it comes to data integrity¹.

Testing distributed systems is hard. There aren’t many tools or resources. There are some promises for performance and scalability benchmarking tools, thanks to Yahoo! (we too await the open-sourcing of the YCSB tool), but right now you have to roll your own, and it takes time. There’s no clear test plan for distributed systems, no failover benchmarking tool, nothing on reliability, availability or data consistency.

Your system can fail no matter how well you thought you tested it, even if it’s sunny outside and you’re throwing a party (especially then). Google, Twitter, Amazon – all have had downtime. Everyone fails once in a while. It’s only a matter of time and users tend to tolerate a short downtime or performance degradations. On the other hand, what users will not tolerate is losing their data². We are completely paranoid about losing data. If other failure scenarios resulting in degraded performance or even a little downtime are bearable, losing data is not.

We try to learn our systems by heart and be able to fix anything fast or even while the system is running. After more than a year, we’re OK with keeping data for our clients³, but we’re still testing and taking precautions.

Really thorough testing of ALL the solutions that we use has paid off. It’s a lot of work, because you have to build the scaffolding for it, but going back and forth keeping up with the changes and pushing fixes is a sure way to know the system in depth.

Demystifying HBase Data integrity, Availability and Performance

It’s good to know the strengths of a system, but it’s more important to be aware of and understand its limitations. We have extensive suites of tests covering both of them. Testing performance is pretty straightforward, but testing data integrity is the hardest, and here we spend most our time.

Full Consistency

HBase is an inherently consistent system. After you write something, modifications are immediately available. You can’t get stale data, or have to reason about quorum reads. We think consistency is good, for a multitude of reasons: if you write an application over a consistent system, application logic is much simpler. One doesn’t have to take into account stale data, it’s just like single threaded programming: you’re going to read what you’ve written earlier¹⁴. Also, consistency is a solid base to build more complex primitives: transactions and indexes, increment operations, test-and-set semantics, etc.

It all comes down to engineering choices: it’s a good exercise for the reader to determine if a system which defaults to eventual consistency, that can accept writes at any moment on any node (e.g. using consistent hashing) and has data fragmented across the cluster, can perform optimally when it comes to sequential reads. How much network chat is needed to do a table scan?

It’s all about what we think is a sensible default: availability and partition tolerance deal with relatively isolated scenarios: you can compute the probability of losing a node or getting your network split in two. It’s relatively low. However, consistency is something you deal with in every operation you do.

This is all getting a little philosophical, but here’s a list of questions (not rhetorical ones), related to this:

An eventual consistent system could be configured to support full consistency and/or data ordering. Would this impact or degrade other attributes like availability and performance? A system that can juggle with C, A and P, is quite flexible. But what part of CAP do you want to support by default, and what’s the impact when you change it afterwards?

Our assumption is that building on consistency is an appealing and sound decision, and any architecture that doesn’t handle this in its default design will lose the performance and availability when forcing it later on. Partition tolerance is not something that we think is worth handling within a single datacenter (redundant datacenter equipment investments are pretty much the norm for both electricity and networking). We do however care about partitioning when doing multi-datacenter replication.

Which of C, A or P do you think will hold the greatest impact? (Hint: for us it’s C :D)

HBase’s edge is in the H

We created a fair amount of tests that we maltreat our system with. It takes effort to implement correct fault tolerance and there’s an advantage in relying on Hadoop for it. Also, in the last 4 years, there was a large client base¹⁵ that validated Hadoop by using it in their production systems (especially Yahoo!). This had a real impact in the stability and fault tolerance of the system.

Now consider a different system, built from scratch. It has to enable and test all that Hadoop does starting from 0 (again, architecture is a promise, but implementation is what you USE). In a best case scenario, this system will gather a critical mass, a community will be created, and it will evolve organically, etc. Hadoop and HBase have that today, and it’s a big advantage.

We pride ourselves in keeping up with new technologies, but we think that Hadoop and HBase are over the “safety” threshold, for what we need to do.

HBase has an “edge” in Hadoop over other technologies, in that, just like Hadoop, it fills the gap between storage scalability, fast processing and cost-efficiency. Why was Hadoop successful? We think it’s because it didn’t rely on a narrow vertical need. Hadoop did not build something that was impossible before. We had NAS systems, and OLAP cubes for data processing, but Hadoop made this possible for any development group, with little initial¹⁶ investment, hence democratizing scalable data processing.

About support

Many Hadoop developers are paid by companies which use Hadoop and see its value. Corporate sponsorship is a catalyst for progress in open-source systems (see MySQL, Eclipse etc.). Hadoop got started as a component in the Nutch search engine, but it was Yahoo that invested resources, and helped make it a success story.

You can dig into Hadoop’s architecture and learn how it works (just like we did), or you could take advantage of the large ecosystem around it. The community report bugs and help people get started, there are books, and even paid support (look at Cloudera).

How does this relate to HBase? HBase is the Hadoop database. It has the best Hadoop integration. It uses HDFS for storage and MapReduce for distributed processing. Once you have a Hadoop cluster, you already have one half of an HBase cluster. It’s only natural that companies that are using Hadoop will be looking at HBase, if they aren’t already using it. And, following Hadoop’s model, they will invest resources and money, adding to HBase’s momentum. The companies that use HBase today sustain the core HBase development team. We too, are contributing back to both HBase and Hadoop. It’s only natural to invest in something that supports your business.

Complexity

HBase is more complex than other systems (you need Hadoop, Zookeeper, cluster machines have multiple roles). We believe that for HBase, this is not accidental complexity and that the argument that “HBase is not a good choice because it is complex” is irrelevant. The advantages far outweigh the problems. Relying on decoupled components plays nice with the Unix philosophy: do one thing and do it well. Distributed storage is delegated to HDFS, so is distributed processing, cluster state goes to Zookeeper. All these systems are developed and tested separately, and are good at what they do. More than that, this allows you to scale your cluster on separate vectors. This is not optimal, but it allows for incremental investment in either spindles, CPU or RAM. You don’t have to add them all at the same time.

The HStack can be a pain to deploy. We took some time to understand the problem, and now we have Puppet recipes for everything. We can set up a cluster completely unattended. We’ll try to push all these back to the community and help other users have it easier, so stay tuned.

Zookeeper, Hadoop etc., let us implement transactions, simple queries and data processing. We want to have a system with such capabilities (these are requirements for large applications). Yes, you can drop some of them but you can’t drop them all. We don’t want a tool that’s missing too much. We want the good parts from an RDBMS, like queries and transactions, while still having distributed processing and cheap scalability. We don’t drink the “NoSQL” kool-aid. We’re not running away from SQL, we’re running towards something that is built from scratch on the premises of scalability and high availability.

About Community and Leadership

This is the biased part of the article, and it should be, because it’s about our relation with the HBase development team. Stack, Ryan, JD were always very receptive. They always help with the issues that we have, whether it’s a bug or a new feature that we need. There’s an open and democratic decision process when prioritizing work with HBase. The team is well balanced and there’s not a single company that drives HBase’s direction.

They are genuinely passionate about their work and strive to have it used by people. We attended one of their regular developer meetups and it was eye opening that developers coming from different backgrounds and companies are working together as a team. We think open-source projects benefit from good leadership and Michael Stack has done a great job with HBase.

Another aspect that appeals to us is the maturity of the development team. They focus on long term benefits. For example, the current focus is to improve the architecture of HMaster and multi-datacenter replication. However, in light of recent performance benchmark reports they took the time to understand the situation, validated with the community that it’s OK to stick to the current plan and didn’t switch focus.

Maturity is also shown in the way that the team positions HBase in relation to other competing projects; they let facts speak rather than opinions. They don’t engage in holy wars and this, to us, seems the right way to build a healthy community.

In the end we’d hope technologies wouldn’t be dismissed based on superficial or biased perception, FUD, or tweets. We don’t like it when talks are based on assumptions without knowing ALL the details of a certain problem or technical choice, and this seems to become a vicious trend in some circles. Hopefully, the reasons explained in this article can help you make your own informed assessments, and see what works for you.

Cosmin.

Lucky shot

If one would have asked me a couple of days ago why or how we chose HBase, I would have answered in a blink that it was about reliability, performance, costs, etc.(a bit brainwashed after answering “correctly” and “objectively” too many times). However, as the subject has become rather popular lately¹, I reflected deeper about “how” and “why”.

The truth is that, in the beginning, we were attracted to working with bleeding edge technology and it was fun. It was a projection of the success we were hoping to have that motivated us. We all knew stories about Google File System, Bigtable, GMail and what made them possible. I guess we wanted a piece of that, and Hadoop and HBase were one logical step to reach that.

We didn’t even have a cluster when we started. I begged and bribed for hardware from teams that had extra cycles on their testing machines. We were going to use them just as SETI@Home does, well sort of. Once we got 7 machines, we had a cluster² running Hadoop and HBase stack (HStack³). We even went on and refurbished some old broken machines to work as extra test agents, besides our own laptops.

Technology driven decisions tend to fall over when assessed from a business perspective. I never thought about costs, data loss, etc. We were somehow assuming that these were all fine. If others ran it why wouldn’t we be able to do it? We knew this architecture would enable scalability, but we didn’t challenge whether the implementation actually works.

Scalability and performance “lured” us in. But in reality it’s the implementation that dictates costs, consistency, availability and performance. A good and scalable architecture is just a long term promise, unless it is backed up by the implementation. In our case the architecture choice paid off, as you’ll see.

Once we realized the potential of HBase through early experiments, we subjected it to a full analysis. It took a while to get an objective opinion, but after all the tests, we really knew we were on to something.

The 40M mark

We had already scaled MySQL, so denormalization, data partitioning and replication weren’t all that new to us. When mid-2008 one of our clients asked us to provide a service that could handle 40M records, real time access, aggregation and all that, we thought we had an answer. This was our first step towards doing “big data”.

There were no benchmarking reports⁴ then, no “NoSQL” moniker, therefore no hype :). We had to do our own objective assessment. The list was (luckily) short: HBase, Hypertable and Cassandra were on the table. MySQL was there as a last resort.

We abstracted the implementation details, and made a stub (so that the clients could start developing), and we started testing each technology stack.

Cassandra was out first. It had just come out, was barely usable, lacked any decent resources or active mailing lists and it could keep only one table per instance. This has changed a lot in time, but we had a deadline then.

Funny enough, HBase was the second one out :).

When we started pushing 40 million records, HBase⁵ squeaked and cracked. After 20M inserts it failed so bad it wouldn’t respond or restart, it mangled the data completely and we had to start over. It was performing bad and seemed to lose data.

I literally dreamt logs for a week, trying to identify the issues. It was looking as if we would discard HBase, but I insisted we should be able to switch later even if we went ahead with Hypertable. The team agreed, even though it didn’t look as a viable choice in comparison with Hypertable that handled the data rather well.

HBase community turned out to be great, they jumped and helped us, and upgrading to a new HBase version fixed our problems. Hypertable⁶ on the other hand seemed to perform better.

Plan for failure

When testing failover scenarios, HBase started to gain ground, handling node failures gracefully and consistently. Hypertable had problems bringing data back up after node failures and required manual intervention. We were, once more, left with a single option.

But there were more questions to be answered, concerns that we never had with MySQL:

What’s the guarantee that every bit you put in comes back in the same form no matter what?

We had to be able to detect corruption and fix it. As we had an encryption expert in the team (who authored a watermarking attack), he designed a model that would check consistency on-the-fly with CRC checksums and allow versioning. The thrift serialized data was wrapped in another layer that contained both the inner data types and the HBase location (row, family and qualifier). (He’s a bit paranoid sometimes, but that tends to come in handy when disaster strikes :). Pretty much what Avro does.

Another scenario we had to cover was a total failure of the Hadoop Distributed File System (HDFS) that HBase relies on.

We adapted an outdated HBase export tool to ensure data consistency during and after backup. We kept backups in 3 places: locally, HDFS and another distributed file system. We also had it prepared in MySQL format so we could switch to a MySQL cluster in case of disaster. We had scripts that would take the latest backup and bring up an alternative storage cluster (HBase or MySQL).

Things went pretty smoothly. We created MapReduce jobs to compute recommendations out of the data that was stored, we had an automated backup, and a mechanism for disaster recovery.

In October 2008 our system went live on time.

Disasters WILL happen

On the 3rd of December 2008, around midnight⁷, sanity alerts started pouring in: the service was running in degraded mode. Our HBase cluster would write data but couldn’t answer correctly to reads. Following the procedure, I was able to make another backup and restore it on a MySQL cluster. Then I enabled the MySQL cluster backup. We were paranoid enough to have a backup procedure for the MySQL “backup” cluster as well :)

We were up and running about half an hour later. MySQL had saved the day, and we congratulated ourselves. Except that our thorough, tested backup plan had a glitch: the master-master replication was not setup for the new tables. We shortly got two different data sets on each server. We had to stop the replication, switch to a single node and fix the consistency issue. Master-master replication is pretty much a hack and needs proper care. Otherwise, it’s pretty easy to screw up your data.

After a thorough investigation⁸, a postmortem and a couple of patches that brought both HBase and HDFS to the latest released version, we switched back to HStack⁹ on the 5th of December. We had no data loss and our clients only experienced a short interruption. Our client team congratulated us for the fast response.

Most importantly, this was our first reality check and a new lesson learnt. The system still runs today and we had no problems with it ever since. After just a few months our biggest client switched direction and never got to the 40 million records.

The system never reached its planned capacity. Even though we took up new clients on board and implemented new services on top of it, we weren’t yet the “stars” we’d imagined to be.

In reality, all we had would have been easy to handle with a MySQL cluster and just a little operational overhead.

The Billion Mark

We decided to switch focus in the beginning of 2009. We were going to provide a generic, real-time, structured data storage and processing system that could handle any data volume. By this time we caught the attention of bigger potential clients and the requirements changed a bit. 40 million became 1 billion, with access times under 50ms and serious processing power. All this with no downtime and definitely no data loss.

This time, we were going to do it right:

We wrote down all the failure scenarios¹⁰ that we could think of: bad disk, bad memory, packet loss, network card failure, machine and rack power failure, disk failure, raid controller failure, etc. Nothing was “sacred”. At one point, in sprint demos we would ask for two random numbers (two machines in the cluster) and unplugged the power cable, unplugged the network cable, or just randomly took out hard drives while running.

We used DRBD and Heartbeat to have HA for the Hadoop NameNode, because this was the last Single Point of Failure (SPOF) in our system.

We initially failed to reach 1B records as we filled up all the disks after less than 500M records. However, the performance and resilience to failures under rough conditions helped us “bootstrap” for something bigger. So, we got a new cluster¹¹ installed, that could handle the necessary capacity.

We also looked at the operational overhead; we always try to automate as much as possible. Our latest deployment system took any number of barebone machines (using the IPMI interface) and would deploy the OS, partition everything and then deploy and configure Hadoop, HBase along with our systems, completely unattended. Everything was set up using a combination of Kickstart scripts and Puppet for service deployment. Even now, deploying a new cluster is basically a git push away.

We had a 3B record table, on which we ran all our benchmarks: cold start (no memory caches), reads, writes, combined tests in different proportions. Random reads under 15 ms, huge throughput, all the cool stuff you’d imagine.

We started contributing to HBase, HDFS and Map-Reduce to support all our failover and performance scenarios and make appropriate fixes when necessary.

Right now, our system is currently “beta” inside the organization. We’re still testing it, but already supports distributed data import, random access, distributed data processing, we have the APIs, web interfaces and it’s running fast.

This is a historical account on how we started working with Hadoop and HBase. The second part of this article will show more practical and objective aspects on why we stick with this technology stack.

Cosmin.