The Data Quality Chronicle

What data quality is (and what it is not)

2019-12-26T10:23:00.002-06:00

Like the radar system pictured above, data quality is a sentinel; a detection system put in place to warn of threats to valuable assets. Keeping with the analogy, strategic leaders, not the tactical resources, decide where and when to place these systems. Data quality is strategic, not tactical. It has tactical applications, but it is very much a calculated implementation of a system used to guard critical assets.

Prior to that implementation, organizations decide on which assets are critical, when to start detecting threats, and where is the most effective way to solicit threat warnings. That is data governance. A strategic outline of what is critical (note: not important, critical), when the need for warning is most effective, and where is it most likely to provide ample warning.

Data quality supports data governance. And since you cannot support that which does not exist, you need data governance before having effective data quality. In other words, you need a well defined strategy before you can have a tactical plan which supports that strategy.

I have read many articles, whitepapers, blog posts, and so forth that get into the minutiae of data quality. I have written about the minutiae myself. But before you understand that data quality is the implementation and early warning detection system for a more comprehensive and strategic data governance plan, all that detail will likely be misplaced and guard a non-critical asset.

If your governance plan centers around data security, point your early warning system at entry and end points which can threaten that secure strategy. If your data governance plan centers around data accuracy, patrol the landscape of systems that threaten accuracy.

Hopefully, subtly, I have demonstrated that a governance plan needs to be centered around a theme. Too many alarms going off in too many directions provides no more clear a direction than no alarms at all. Another reason why having a strategy is more important than tactics. Find out what threatens the end goal the most and focus on that.

In short, data quality is not a plan. It is an implementation of a plan; an early warning detection system to monitor threats to the plan.

Data Validation: Data Quality's Doorman

2015-03-23T16:58:00.004-05:00

Assumptions and Issues Abound

Often data quality seems to be an assumption. The business assumes that IT has controls in place, IT assumes that data supplied is in order and everyone assumes they'd know about it if these assumptions weren't correct.

Fact is, most often, these assumptions are false and there are data quality issues. The costs, both tangible and intangible, associated to data quality issues are extremely high so you'd think people would assume less and attack data quality root causes more often. But for some reason data quality remains something people assume is either inherent or addressed.

Maybe this lack of diligence is because data quality root causes are so varied in nature and often tough to trace. I know of many instances where the source produced issues and there was nothing the data consumer could do to drive down, much less stop, the constant production of erroneous data.

If this situation sounds familiar and you are looking for relief, there is an important step that can give you back control of your data, stop the proliferation of issues, and define root causes for these issues.

Check it out the rest of this article on LinkedIn here

Data Quality (Data Profiling) Tutorial: Talend Open Studio for Data Qual...

2015-02-16T19:44:00.001-06:00

Talend is an interesting vendor. I've watched them grow and harden all their product offerings over the past few years.

I wanted to share this tutorial from youtube as an introduction to their data quality offering.

Personally I have used the profiler and it is hard to beat the no configure, out-of-the-box reports.

Enjoy!

No budget, No problem! - Data Quality on the cheap

2014-10-12T08:34:00.000-05:00

While most organizations have data quality issues, not every organization has a budget for software to monitor, report and remedy data quality. However just because there is a lack of funding that doesn't mean there needs to be a lack of quality.

With some SQL skills and creativity, a data quality program can thrive. It can even have slick dashboards that measure quality trends.

What you really need

In order to build any data quality program only one thing is absolutely necessary; a business rule that differentiates good quality data from bad. Preferably, and inevitably, there would be many rules, but the point is still the same.

The main components of a data quality program are business rules

Walking the walk
So now that we've talked the talk, let's walk the walk. As I mentioned, all you really need is a rule.

Rule: Every bill must be associated with a customer

To understand the rest of this article we need to quickly go over the framework of a data quality scorecard.

Data Quality Dimension: Integrity is the dimension we will use in our example
Metric: That's the name we give to the rule
Rule: Our essential component
Total Records: In our example this would be the total number of bill records
Total Records Violating the rule: In our example this would be the total number of bill records not associated to a customer
% Violating: Violations / Total Records
Target: Number of violating records we are willing to accept
Target Description: Explanation of the target derivation
Count Trend: Are the number of violating records increasing or decreasing
% Trend: Is the percentage of violating records increasing or decreasing

Step 2. Step one was get the rule from the business. Step two involves writing the SQL code that will give you numbers to plug into the scorecard framework above.

Total Records SQL

-------------------------------------------------------- Total BILL COUNT

SELECT COUNT(*) AS TOTAL_BILL_COUNT FROM dbo.TBL_Bill_Detail AS C

Total Records Violating Rule SQL

-------------------------------------------------------- TOTAL Bill Orphans

SELECT COUNT(*) AS TOTAL_BILL_Orphans FROM dbo.TBL_Bill_Detail AS C WHERE C.Customer_Key IS NULL

% Violating SQL

-------------------------------------------------------- ORPHANED BILL % VIOLATING

DECLARE @TOTAL_BILLING AS decimal (38,2)

SELECT @TOTAL_BILLING = COUNT(*) FROM dbo.TBL_Bill_Detail AS C

DECLARE @TOTAL_BILLING_VIOLATING AS decimal (38,2)

SELECT @TOTAL_BILLING_VIOLATING = COUNT(*) FROM dbo.TBL_Bill_Detail AS C WHERE C.Customer_Key IS NULL

SELECT SUM(@TOTAL_BILLING_VIOLATING / @TOTAL_BILLING*100) AS ORPHAN_BILLING_PERCENTAGE_VIOLATING

Target SQL

-------------------------------------------------------- ORPHANED BILL TARGET (HARD CODED AT 10% FOR TESTING PURPOSES)

DECLARE @TARGET_PERCENT AS NUMERIC(2,2)

SELECT @TARGET_PERCENT = 0.10

SELECT COUNT(*) * @TARGET_PERCENT as TOTAL_BILL_TARGET FROM dbo.TBL_Bill_Detail AS C

Now that we have our SQL coded, let's take a look at the Excel formulas for determining the trends. One thing to keep in mind is that trends are a comparison of the current total and percent violating against the a previous total and percent violating measurement. In this case I have chosen a week over week measurement, or this week's execution against last week's execution.

Here is what it looks like in Excel to the user:

Here are the Excel formulas

=SUM(E15-'DQ SCORECARD 8292014'!E15)

In this example, E15 is the current week count violating the rule and 'DQ SCORECARD 8292014'!E15 is the previous week count violating the rule.

I use conditional formatting to create the traffic light icon using a rule like this:

Once I have the SQL and Excel formulas setup, I simply take the query results and plug them into the Excel spreadsheet.

The end result looks like this ...

Interpreting the results

Only 0.13 % of Bills are not associated to a customer. This provides a measure of the severity of the issue relative to the number of bills
The count trend is heading in the wrong direction indicated by the red traffic light icon. 59 bills were processed this week that are not associated with a customer
The % trend is headed in the right direction in that more bills are being processed each week and the overall percentage of bills not associated with a customer is not growing
In the end, there is an issue. It is a small one. The issue is not growing relative to the growth in bills processed

With this structure you can quickly add metrics by writing new SQL and then leveraging the Excel formulas to expand your data quality program and scorecards.

That's how you do data quality on the cheap!

Graph DB Comparison

2014-09-04T13:28:00.001-05:00

OrientDB vs Neo4j

http://www.orientechnologies.com/orientdb-vs-neo4j/

I'm glad I read this and I think you will be too!

It was beneficial in that it outlined many of the enterprise limitations of Neo4j.

Here are a few ...
1. One master server which creates a bottleneck on write operations when distributed
2. Cypher. Easy to learn but who needs to constantly covert sql to cypher?
3. Inability to reclaim storage space on deletes without a restart
4. Schema less architecture. Inhibits design for a lot of use cases

I'm looking forward to working with Orient some more. I'll write about that soon!

Graph Database Observations

2014-06-22T13:53:00.000-05:00

I have been ramping up (maybe a little late to the dance?) on graph database technologies, specifically Neo4j and Titan.

I really like modeling for graph databases. It is much more intuitive and direct in terms of moving toward a solution. I can see where graph dbs fit into the Agile development methodology really well (about time something fit into Agile well).

Even as a novice, data modeling for a graph database feels intuitive, fast and fun!

I have seen some promise in Talend Big Data Integration tool in migrating data into Neo4j. I think that is a HUGE plus! Way to go Talend!!

However what I do see, at least initially, is a large amount of coding to migrate data from database to a graph database. Maybe this is a good thing? Graph dbs seem to shrink the design time, document in an intuitive manner and require more time for developers to code the solution. For years I have felt strongly that too much time is wasted designing and documenting solutions and not enough time is spent actually coding them. Graph databases seem to tip the scale in favor of the developer with regard to this dynamic.

Graph databases seem to reduce design and documentation durations and give that time back to the developer!

Here are five things I have learned so far that are worth passing on:

Nodes = vertices = records (in the relational world)

Relationships = edges = constraints (in the relational world)

Nodes/Vertices = records = you will have a lot of nodes/vertices (learn how to create 'em, you'll need it)

Nodes/Vertices have properties which hold values (kind of like table attributes)

Relationship/edges have direction and properties allowing for the developer to program the strength of the relationship and make the relationship bi or uni directional

I mentioned modeling for graph databases so I will share my first graph model. This model is just a draft and a high level depiction of a customer centric solution. However I feel it is a good example of how intuitive they are to read, develop from and iterate through. This only took about 15 minutes to work out.

[caption id="attachment_3491" align="aligncenter" width="747"] high level graph model of a customer centric solution[/caption]

That's it for now. I'll be adding to this topic as I develop more graph skills!

Helpful links

http://nosql.mypopescu.com/post/10162152437/what-is-a-graph-database#about-blog

http://thinkaurelius.github.io/titan/

https://github.com/thinkaurelius/titan/wiki/Getting-Started

http://www.neo4j.org/

http://www.neo4j.org/learn

Eventual Consistency Models: Release Consistency

2014-04-16T15:29:00.000-05:00

As a data quality professional, eventual consistency is a concern. Don't get me wrong, it is a necessary and incredible programmatic feature. However, the fact that a system can store different values for the same attribute is concerning.

Before we get into the gritty details let's define some key terms.

Node: In a distributed system a node represents hardware which contains data and code

Memory Object: Medium for data storage. Could be a JSON document, graph node or cached memory location

In this post, let's look at the release consistency model. This model enforces, or maybe implements is the better word, consistency through two main operations; acquire and release.

In order to write data, a node must acquire a memory object. When the write is complete, the node releases the memory object.

The model is implemented by distributing all the writes across the system before another node can acquire the memory object.

There are two methods to implement release consistency; eager and lazy. In the eager implementation writes are distributed after the node releases the memory object. In the lazy implementation writes are distributed before a node acquires the memory object.

Eager release consistency is (potentially) more costly to the system in that it is data not necessarily required by the current users. The upside of eager release consistency is that when the user does request the data, there is a low latency period.

Lazy release consistency essentially represents the reverse scenario. Only required data is distributed, conserving system resources, however request latency can be high.

Hope you enjoyed the post! I enjoyed writing it. Stay tuned for more write ups on eventual consistency models.

Finding Orphaned Nodes in a Graph Database

2014-04-16T06:51:00.000-05:00

The Problem

Orphaned records represent missed opportunity

Regardless of the data store, unrelated data cannot be mined or analyzed and often goes undetected until there is a problem.

The Solution

Profile your data

Data profiling is probably the most under utilized practice is all of data management. I can't think of a data related issue where the resolution methodology doesn't start with profiling the data. Detecting and resolving orphaned data is no exception.

In the relational database world there are many ways to profile data and detect orphans. From packaged solutions with dashboard reports to simple SQL queries and Excel exports, profiling relational data is a mature practice.

But what if you are working with graph database technologies? Prepackaged solutions for graph technology is a growing market, but not as mature as the RDMS market and those SQL queries are not exactly going to translate well into the graph world.

The Situation [ref]No, not some abdominal flashing kid from Jersey. I'm referring to a scenario.[/ref]

Let's say you have a graph database which you are using to analyze which of your customers are placing orders. You will likely have a graph model that looks something like the figure below.

In this model you have a customer node [ref] in graph terms a node is a record [/ref] and an order node, related to each other by the placed relationship [ref] in graph terms a relationship joins to nodes and describes their relationship [/ref].

Using this model you can determine (quickly) what orders a customer has placed with very little code.

Cypher Query

 match (c:Customer), (o:Order) where (c)-[:PLACED]->(o) return c,o;

Graph Result

Here we can see that customer 0 placed orders 1133 and 1132. Not insightful enough? Here is the result in a more descriptive table form.

So Bill Smith placed order number 47 on 10/7/2013 and order number 90 on 10/9/2013.

This is great when the nodes are all related correctly, however, this post is about when everything is in fine working order. This post is about orphans, right?

Orders without Customers

Let's look at the code required to find what orders are not related to a customer.

match (o:Order), (c:Customer) where not ((c)-[:PLACED]->(o)) return distinct o;

Here we are saying give me the distinct list of orders which are not placed by a customer.

Here is the more readable table layout results

Summary

So by using the relationship between customers and orders and specifying where it does not exist, we can find the 'orphaned' orders. [ref] Here is the cypher query in order to find customers without orders.

match (o:Order), (c:Customer) where not ((c)-[:PLACED]->(o)) return distinct c;

Technically a customer should be someone who places an order of some kind (otherwise they'd be a prospect). [/ref]

Is there a better ETL approach? Data Warehouse is all about loading data from source to target. Traditionally, source system produces fil...

2014-04-01T12:25:00.000-05:00

My answer to a curious ETL question

Answer by William Sharp:

Connect directly and schedule the etl jobs to run at a rate that matches the update frequency of the source. you can profile the source update dates to figure that out.

View Answer on Quora

What aspects of Data Quality influence the impact on Business Value?

2014-03-21T03:56:00.000-05:00

Answer by William Sharp:

when I read your question the one word that stood out to me was documents. to me, this means data entry deviations. In the context of the 6dimensions of data quality this means consistency checks (is a value referred to/spelled the same way across all the documents), completeness checks (is all the required information present in every document), accuracy checks (is each value from ever document correctly represented), duplication checks (is the information in the documents redundant)

View Answer on Quora

Why should I use an existing ETL vs writing my own in Python for my data warehouse needs?

2014-03-09T09:04:00.000-05:00

Answer by William Sharp:

Unless you want to enter the ETL software market, you will spend a lot of time writing software instead of migrating / managing data (which I am assuming is your real job). If this tool is going to be used by many developers in an enterprise setting, you will spend a lot of time writing code to manage the ETL code. Probably more than you think

This is classic buy over build ... buy because there are people dedicated to solving issues you haven't even thought of yet. Build if your needs are so unique that there isn't a solution out there (extremely rare)

View Answer on Quora

What things does a data scientist do?

2014-03-09T06:10:00.000-05:00

Answer by William Sharp:

Put succinctly it is data management and statistics. collect, clean and migrate data (just like everybody else does) and perform statistics on that data. data science is a new marketing term that goes along with big data. I'm not down playing statistics. I love stats. the 'science' part is a little (a lot) pretentious imho

View Answer on Quora

What are some good resources for learning about distributed computing? Why?

2014-02-25T14:20:00.000-06:00

Answer by Alex Kamil:

Prerequisites
Unix shell basics: http://www.amazon.com/Unix-Progr...
C: http://www.amazon.com/Programmin...
OS basics: http://www.amazon.com/Andrew-S-T... , http://www.amazon.com/Linux-Kern...
Unix Programming: http://www.kohala.com/start/
Networking Basics: http://www.amazon.com/Computer-N... , more advanced: http://web.mit.edu/dimitrib/www/...
Sockets: http://www.amazon.com/TCP-Socket... , http://www.amazon.com/Foundation... , and network programming: http://www.kohala.com/start/
RPC: Remote Procedure Calls (RPC)
XMPP: http://web.sarathlakshman.com/Ar...
Transmission of information: http://www.amazon.com/Informatio... , http://www.inference.phy.cam.ac.... and network coding
Intro to concurrency: http://www.amazon.com/Practical-...
Java: What questions are Java Software Engineers seeing the most of on technical interviews?
Data structures and algorithms: Learning Algorithms: What are the most learner-friendly resources for learning about algorithms?

Courses
CS525: http://www.cs.uiuc.edu/class/sp1...
6.824: http://pdos.csail.mit.edu/6.824/...
6.828: http://pdos.csail.mit.edu/6.828/...
CS264: http://www.cs264.org/lectures/le...
CS294: http://www.cs.berkeley.edu/~oded...
CS 707: http://www.cs.gmu.edu/~setia/cs7...
Advanced Computer Science Courses: http://the-paper-trail.org/blog/...
CS696: http://www.eli.sdsu.edu/courses/...
Google Code University: http://code.google.com/edu/paral...
CS7960: http://www.cs.utah.edu/~jeffp/te...
6.829 http://ocw.mit.edu/courses/elect...
Textbook: http://www.tacc.utexas.edu/~eijk...

Notes
BitTorrent: http://en.wikipedia.org/wiki/Bit...
Other:
Start with Lin, Data-Intensive Text Processing with MapReduce, ISBN 1608453421, http://www.umiacs.umd.edu/~jimmy...
See Tom White, Hadoop: The Definitive Guide, ISBN 1449389732, http://www.amazon.com/Hadoop-Def..., Sean Owen et al., Mahout in Action, ISBN 978193518268, http://www.manning.com/owen/
HDFS under the hood: http://assets.en.oreilly.com/1/e...
Zookeper: http://highscalability.com/zooke...
Download Hadoop (http://hadoop.apache.org/) and run some MapReduce jobs on your laptop in pseudo-distributed mode (see
MapReduce: What's the best way to come up to speed on MapReduce, Hadoop, and Hive? )
Learn about Google technology stack (MapReduce, BigTable, Dremel, Pregel, GFS, Chubby, Protobuf, Snappy, Ganeti, Tenzing, Sawzall, BigQuery, F1, Spanner, Jingle, GCM, Google Talk, etc). (See
Google Research: What are the most interesting Google Research papers?
also http://www.columbia.edu/~ak2834/... , http://www.cs.rutgers.edu/~muthu... , http://the-paper-trail.org/blog/...)
Setup account with Amazon AWS/EC2/S3/EBS and experiment with running Hadoop on a cluster with large data sets (you can use Cloudera or YDN images, but in my opinion you can better understand the system if you set it up from scratch, using the original distribution). Watch the costs: http://www.networkworld.com/news...
Try out Hadoop alternatives, specifically the minimalist frameworks such as BashReduce: http://github.com/erikfrey/bashr... and CloudMapReduce: http://code.google.com/p/cloudma... (see
What are some promising open-source alternatives to Hadoop MapReduce for map/reduce?)
See Machine Learning: What are some good class projects for machine learning using MapReduce?
Run Bryan Cooper's Cloud Serving Benchmark on AWS, compare Hbase vs
Cassandra performance on a small cluster (6-8 nodes): http://wiki.github.com/brianfran... also see Pete Warden's tests: http://petewarden.typepad.com/se... , Hbase book: http://hbase.apache.org/book.html
Run LINPACK benchmark: http://www.datawrangling.com/on-...
Run some experiments with MPI
(http://www.mcs.anl.gov/research/...) try to implement a
simple clustering algorithm
(e.g http://en.wikipedia.org/wiki/K-m...) with MPI vs
Hadoop/MapReduce and compare the performance, fault tolerance, ease of
use etc. Learn the differences between the two approaches, and when it
makes sense to use each one.
Check out Dongarra' papers: http://www.netlib.org/utk/people... , works by Gibbons: http://www.bell-labs.com/org/112... , Lamport: http://research.microsoft.com/en... , Blelloch: http://www.cs.cmu.edu/~guyb/pubs... , also see What are the seminal papers in distributed systems? Why?
There is a new library called MPI-Mapreduce
(http://www.sandia.gov/~sjplimp/m...) see how it works and how
it compares to other MapReduce implementations
Run some tests with Scalapack (http://www.netlib.org/scalapack/), try to port one of the routines to Hadoop, compare the performance and scalability. See how stability of numerical algorithms is evaluated: http://portal.acm.org/citation.c...
Write your own simplified MapReduce runtime in C or any other programming language.
Try xargs -P and GNU Parallel, also see What are some lesser known but useful Unix commands?
Check out http://www.cascading.org/, http://clojure.org/ , http://www.bloom-lang.net/features/
Learn about distributed hash tables (http://en.wikipedia.org/wiki/Dis... , http://www.linuxjournal.com/arti...), run some experiments with Paxos (Algorithm) http://the-paper-trail.org/blog/..., Kademlia: http://en.wikipedia.org/wiki/Kad... , See Wolf Garbe's answers on Peer-to-Peer Technology. Also see Petar Maymounkov's answer to Computer Science Research: Have there been any new advances in distributed hash tables?
Download Nutch (http://nutch.apache.org/) or Solr
(http://lucene.apache.org/solr/), run a crawl on Wikipedia. Analyze the
collected data with R (see item 2 above) or Python
(http://www.nltk.org/)
Write you own simplified crawler/indexer, test the performance and
scalability, look at the Lucene source for ideas, look
at http://infolab.stanford.edu/~bac... for inspiration. You
can probably build it as a term project in either Information Retrieval
or Search Engines course.
Learn about prefix-sum: http://en.wikipedia.org/wiki/Pre... ,parallel
matrix multiplication: http://www.cs.berkeley.edu/~yeli... ,streaming: http://infolab.stanford.edu/stream/ and
BSP: http://en.wikipedia.org/wiki/Bul... , DSM: http://www.amazon.com/Distribute... and "The Use of Name Spaces in Plan 9"
Check out Persistent Linda: http://www.google.com/#sclient=p... , if you find it interesting see Linda and Friends (an article by Sudhir Ahuja from 1986), also search for "Linda in Context", "tuple space", Javaspaces, Gigaspaces. Read "How to write parallel programs: a guide to the perplexed" by Nicholas Carriero and David Gelernter: http://portal.acm.org/citation.c...
Learn about Compute Unified Device Architecture (CUDA): http://www.amazon.com/CUDA-Examp... , Graphics Processing Unit and Field Programmable Gate Arrays accelerators, PlayStation 3 programming: http://en.wikipedia.org/wiki/Cel... , http://www.hotchips.org/hc21/mai...
Pick one of the PGAS languages
(http://en.wikipedia.org/wiki/Par...), e.g.
X10 (http://en.wikipedia.org/wiki/X10..., go
through the tutorials
(http://ppppcourse.ning.com/forum...),
run some HPC benchmarks (LU, FFT) and the examples (the streaming
example in particular): see how it scales on a cluster/AWS, compare to
sequential and Hadoop/MapReduce implementation, see what kind of
performance/scalability gains it gives you on multicore boxes.
Some good references on parallel programming: Herlihy& Shavit, The art
of multiprocessor programming:
http://www.amazon.com/Art-Multip... , Blelloch, Vector models for data-parallel computing:
http://citeseerx.ist.psu.edu/vie... , Valiant, A bridging model for parallel computation:
http://portal.acm.org/citation.c... ,Hillis & Steele, Data
Parallel Algorithms: http://portal.acm.org/citation.c... , Miller & Boxer, Algorithms sequential and parallel: http://www.amazon.com/Algorithms... , Leighton, Introduction to Parallel Algorithms and Architectures: http://www.amazon.com/Introducti... , JaJa: Introduction to Parallel Algorithms: http://www.amazon.com/Introducti...
You should probably start with Dijkstra, Cooperating Sequential Processes: http://www.cs.utexas.edu/users/E... and Ben-Ari, Principles of Concurrent and Distributed Programming: http://www.amazon.com/Principles... (I have the older edition from 1982, which is an excellent intro)
Take a course in Parallel Computer Architecture: http://www.eecs.berkeley.edu/~cu... , http://www.amazon.com/Parallel-C... , http://people.engr.ncsu.edu/efg/...
Check out Cilk: http://software.intel.com/en-us/... and
Matlab Parallel computing toolbox:
http://www.mathworks.com/product...
For some theoretical background on distributed algorithms, information decomposition and complexity see: Feldman et al., On the Complexity of Processing Massive, Unordered, Distributed Data: http://arxiv.org/abs/cs/0611108, Traub, An introduction to information based complexity: http://octopus.library.cmu.edu/C... and Is Nancy Lynch's book still the best intro to distributed algorithms?
Parallel Distributed Processing (PDP) by Rumelhart and PDP research group: http://www.amazon.com/Parallel-D... - look up the computing architectures for Artificial Neural Networks, e.g. http://www.amazon.com/Parallel-A...
Run some experiments with Weka (http://www.cs.waikato.ac.nz/ml/w...) or RapidMiner (http://rapid-i.com/), pick a simple algorithm and port it to MapReduce, see how it scales on a cluster/AWS
Experiment with distributed 'NoSQL' data stores (Voldemort, Hbase, Redis, Tokyo, Cassandra etc). Figure out what is CAP theorem all about
(http://www.allthingsdistributed.... , http://www.cloudera.com/blog/201... ).
Create a simple app with key-value or column-based store as a back-end.
Import several GBs of interesting data into it and run some simple
clustering/KNN algos (http://en.wikipedia.org/wiki/Clu..., http://en.wikipedia.org/wiki/Nea...).
Optimize your algo to better utilize random access patterns, experiment
with various tuning options. Build a frond-end visualization for the
results (Check out Protovis or similar visualization
package: http://vis.stanford.edu/protovis/)
A good resource on 'NoSQL': Daniel Abadi's publications: http://cs-www.cs.yale.edu/homes/... and Varley, No Relation: The Mixed Blessings of Non-Relational Databases: http://ianvarley.com/UT/MR/Varle...
Doozer: http://xph.us/2011/04/13/introdu...
Learn about main-memory
databases: http:YouTube: What is YouTube's architecture?//en.wikipedia.org/wiki/In-memory_database , http://scholar.google.com/schola..., http://monetdb.cwi.nl/ , http://hstore.cs.brown.edu/ , Microsoft Trinity - a graph database over distributed memory cloud: http://research.microsoft.com/en...
Write a distributed hash table in C, here is a good reference: http://pdos.csail.mit.edu/papers... or use node: https://github.com/stbuehler/nod...
Networking: http://www.amazon.com/Unix-Netwo... , http://www.amazon.com/TCP-Illust... , Network Programming: What are some good resources for learning about network programming?
Write a distributed file system in C. See git for inspiration: http://apenwarr.ca/log/?m=200801#31 , Frangipani http://portal.acm.org/citation.c... . For a good intro see the Tanenbaum's series: http://www.amazon.com/Distribute... , http://www.amazon.com/Modern-Ope... and http://www.stanford.edu/class/cs... , The Amoeba Distributed OS: http://www.cs.vu.nl/pub/amoeba/a...
Graph databases, etc: http://nosql-database.org/ , http://www.graph-database.org/ , GraphLab: http://graphlab.org/
Facebook Engineering: What is Facebook's architecture? , What are the most interesting Facebook Data papers/projects?
Hadoop/Hbase at Facebook: http://borthakur.com/ftp/Realtim...
YouTube: What is YouTube's architecture?
Justin.tv: How does justin.tv work?
Scalability: How does Heroku work?
Netflix: What is Netflix's architecture?
Hulu: What is Hulu's architecture?
eBay: What is eBay's architecture?
Dropbox: What is Dropbox's architecture?
What are the core technologies that Twitter uses for their platform and what is the Twitter Macro architecture?
LinkedIn SNA: http://sna-projects.com/sna/
What is LinkedIn's database architecture like?
Quora Infrastructure: How does LiveNode work?
Scaling LiveJournal: http://danga.com/words/2007_06_u...
Content Delivery Networks: How does Akamai CDN work?
GitHub architecture: https://github.com/blog/530-how-...
Twitter Rainbird: http://www.slideshare.net/kevinw..., http://www.slideshare.net/nkalle...
Yahoo! S4: https://github.com/s4/core , http://docs.s4.io/
IBM Infosphere streams/System S: http://www-01.ibm.com/software/d...
BackType Storm: http://news.ycombinator.com/item...
Octobot, a distributed task queue worker: http://octobot.taco.cat/
F* by Microsoft: http://research.microsoft.com/en...
Google Realtime Search: Is real-time search fundamentally different from the various search services Google already provides?
HN thread on the architecture of backend systems: http://news.ycombinator.com/item...
The secrets of Node's success: http://radar.oreilly.com/2011/06...
Druid: A Distributed, In-Memory OLAP Store: http://metamarketsgroup.com/blog... (some dissing here: http://news.ycombinator.com/item...)
FathomDB response to AWS outage: http://news.ycombinator.com/item...
Google Ganeti - Cluster-based virtualization management software: http://code.google.com/p/ganeti/ , http://k1024.org/~iusty/papers/i...
Google GO: http://www.theregister.co.uk/201...
Erlang/OTP: http://learnyousomeerlang.com/co...
Cloud Haskell: http://research.microsoft.com/en...
NASA Nebula: http://nebula.nasa.gov/
Platform MR: http://www.platform.com/Products...
Fast 2011: http://www.usenix.org/events/fas...
The history of consensus: http://betathoughts.blogspot.com... (via http://the-paper-trail.org )
Distributed Linked List: http://www.google.com/search?scl...
GIbbons, Synopsys Data Structures For Massive Data Sets: www.cs.princeton.edu/courses/archive/spring04/cos598B/bib/GibbonsM-syn.pdf
Lock-Free Linked Lists and Skip Lists: http://www.cse.yorku.ca/~ruppert...
Maekawa's lock: http://www.google.com/#sclient=p...
Crossbow, searching for SNPs with cloud computing: http://www.biomedcentral.com/con...
Distributed Caching: Hazelcast, Ehcache, Terracotta (company), Memcached, Oracle Coherence
Bitcoin, A Peer-to-Peer Electronic Cash System: www.bitcoin.org/bitcoin.pdf
Distributed computing with JS: BitCoin miner: http://news.ycombinator.com/item..., MapRejuice: https://github.com/ryanmcgrath/m... , http://www.igvita.com/2009/03/03...
OpenCirrus - Cloud Computing Research Testbed: https://opencirrus.org/content/r...
Antonio Piccolboni, A Comparison of Eight MapReduce
Languages: http://www.dataspora.com/2011/04...
Caching and processing 2TB in memory with Hazelcast: http://highscalability.com/blog/...
Dapper: a Large-Scale Distributed Systems Tracing Infrastructure: http://static.googleusercontent....
On the performance of distributed lock-based synchronization: http://portal.acm.org/citation.c...
Tonika: social routing with organic security: http://pdos.csail.mit.edu/~petar...
http://www.linuxvirtualserver.org/
It's time for low latency: http://www.matt-welsh.blogspot.c...
OS Research Wanted: http://surriel.com/research_wanted
FlightPath: Obedience vs. Choice in Cooperative Services: http://www.usenix.org/event/osdi...
Piccolo: Building Fast, Distributed Programs with Partitioned Tables: http://piccolo.news.cs.nyu.edu/p...
CIEL: a universal execution engine for distributed data-flow computing: www.usenix.org/event/nsdi11/tech/full_papers/Murray.pdf
Directed Edge: On building a stupidly fast graph database: http://blog.directededge.com/200...
What is the best tutorial for Python's Twisted framework?
Node.js: What are the best resources to learn Node.js?
Parallelism /= Concurrency: http://ghcmutterings.wordpress.c...
PyCon 2011: Handling ridiculous amounts of data with probabilistic data structures: http://blip.tv/pycon-us-videos-2...
Go (programming language) at Heroku: http://blog.golang.org/2011/04/g...
Meijer & Lamport, Mathematical Reasoning and Distributed Systems: http://channel9.msdn.com/Shows/G...
Concurrency's Shysters: http://blogs.oracle.com/bmc/entr...
Horton: Online Query Execution On Large Distributed Graphs: http://www.graph-database.org/20...
DataDomain: http://www.datadomain.com/
WebRTC: https://sites.google.com/site/we...
Bloom: http://www.bloom-lang.net/ and http://boom.cs.berkeley.edu/pape...
Jini tutorial: http://jan.newmarch.name/java/ji...
java distributed cache for low latency, high availability: http://stackoverflow.com/questio...
Scalable, Distributed Data Structures for Internet Service Construction (2000): http://usenix.org/events/osdi200...
Cloud Programming: From Doom and Gloom to BOOM and Bloom: http://neilconway.org/talks/boom...
SEDA: An Architecture for Highly Concurrent Server Applications: http://www.eecs.harvard.edu/~mdw... (http://matt-welsh.blogspot.com/2... )
Scalable Network programming: http://bulk.fefe.de/scalable-net...
Protocol Buffers: http://news.ycombinator.com/item...
Distributed Systems: What is the future of file systems?
What are some current directions in operating system research?
Distributed Systems: What are the best resources for learning about distributed file systems?
How do I approach building a distributed queue architecture?
What are some good resources for learning about data compression? Why?
Information Retrieval: What are some good resources to get started with Information Retrieval? Why?
What are good resources to learn about search engine architecture?
What are the good resources to learn about distributed, scalable, robust software architecture/infrastructure building?
What are some common approaches to error aggregation, alerting, and analysis in distributed systems?
What are some good research papers and articles on fault-tolerant systems design?
Big Data: What are the most influential papers in the world of big data? Why?
Large Scale Learning: What are some introductory resources for learning about large scale machine learning? Why?
Computer Science Research: Which CS areas have the most low-hanging fruit for research?
Big Data: Why the current obsession with "big" data?
Distributed Systems: Which conferences are the best to follow for Distributed Systems?
What are the best recommended research topics on databases according to edge technologies and recent research trends?
Distributed Systems: What are the most interesting research projects related to the management of distributed systems?
TCP/IP: What are some high performance TCP hacks?
Pike, Systems Software Research is Irrelevant: http://herpolhode.com/rob/utah20...
Concurrency in Go (programming language): http://golang.org/doc/effective_...
Communicating Sequential Processes (CSP): http://www.usingcsp.com/
Scalable Joins: http://research.microsoft.com/en...
Kestrel, tiny queue system based on starling, in scala: https://github.com/robey/kestrel
Disruptor - concurrent programming framework: http://code.google.com/p/disruptor/
DataTurbine streaming engine: http://www.dataturbine.org/
The Task Parallel Library (TPL) in .NET: http://msdn.microsoft.com/en-us/...
A crash course on modern hardware: http://www.infoq.com/presentatio...
Infinispan data grid on top of JGroups: http://www.jboss.org/infinispan
Memcached distributed cache on top of Jgroups: http://www.jgroups.org/memcached...
Scalable Application Layer Multicast: http://pages.cs.wisc.edu/~suman/...
Hadapt: Efficient Processing of Data Warehousing Queries in a split execution environment: http://portal.acm.org/citation.c...
Danga's open source projects: http://danga.com/
Hadoop on MPI: http://hadoopbi.com/index.php/te...
Hadoop on Pallet: http://sritchie.github.com/2011/...
Oracle Grid Engine: http://en.wikipedia.org/wiki/Ora...
Ejabberd: a scalable XMPP instant messaging server: http://www.ejabberd.im/
Cheetah - Circuit-switched High-speed End-to-End Transport ArcHitecture:
http://www.ece.virginia.edu/chee...
PVM: http://www.snakebytestudios.com/...
Systems at ETH Zurich: http://www.systems.ethz.ch/resea...
OpenCL: http://www.khronos.org/opencl/
Cloud Haskell: http://research.microsoft.com/en...
Concurrent programming in Erlang: http://www.erlang.org/erlang_boo...
Concurrent programming in Occam 2: http://www.amazon.com/Programmin...
Concurrent and Real-Time Programming in Ada: http://www.amazon.com/Concurrent...
Distributed Programming in Ruby: http://www.amazon.com/dp/0321638...
REST: http://rest.elkstein.org/
Gearman: http://gearman.org
Drizzle: https://launchpad.net/drizzle
Distributed logging with syslog: https://wiki.archlinux.org/index...
Logs are streams, not files: http://adam.heroku.com/past/2011...
Utilizing Redis in distributed Erlang systems (Heroku): http://erlang-factory.herokuapp....
MS Command Shell: http://arstechnica.com/business/...
MS PowerShell: http://en.wikipedia.org/wiki/Win...
DRb - Distributed Ruby: http://segment7.net/projects/rub...
God - The Ruby Framework for Process Management: https://github.com/mojombo/god
Taco Bell programming: http://teddziuba.com/2010/10/tac...
Rush - the Ruby Shell: http://rush.heroku.com/
CloudCrowd: https://github.com/documentcloud...
Coda: http://www.coda.cs.cmu.edu
Tenzing: http://research.google.com/pubs/...
GNTP: http://www.growlforwindows.com/g...
CycleCloud : http://blog.cyclecomputing.com/2...
GNU Parallel: http://www.gnu.org/software/para...
Torque: http://www.adaptivecomputing.com...
Chef: http://www.opscode.com/chef/
Dempsy - Nokia's Distributed Elastic Message Processing System: http://dempsy.github.com/Dempsy/
F1 -The Fault-Tolerant Distributed RDBMS Supporting Google's Ad Business: http://research.google.com/pubs/...
Galaxy - a distributed in-memory data grid by Parallel Universe: http://blog.paralleluniverse.co/...
Spanner: Google's Globally-Distributed Database: http://research.google.com/archi...
Jingle: http://code.google.com/p/libjingle/
Paxos Made Live: http://www.eecs.harvard.edu/cs26...
TeleHash: https://github.com/quartzjer/Tel...
Adobe RTMP: http://en.wikipedia.org/wiki/Rea...
UPnP: http://en.wikipedia.org/wiki/Uni...
IP multicast: http://en.wikipedia.org/wiki/IP_...
Reliable multicast: http://en.wikipedia.org/wiki/Rel...
JGroups: http://www.jgroups.org/
LDPC Codes: http://en.wikipedia.org/wiki/Low...
Erasure codes: see the last chapter in D.J.C MacKay's http://www.inference.phy.cam.ac.... (note that some of these are heavily patented, e.g. http://www.inference.phy.cam.ac.... )
Blaze: Next-gen NumPy on which to build out-of-core and distributed algorithms: https://speakerdeck.com/sdiehl/b...
Spark cluster computing by UC Berkeley AMPLab
NSQ by bitly | ♥ your bitmarks: NSQ: realtime distributed message processing at scale
Celery: Distributed Task Queue
Go'Circuit by Tumblr: Paradigm for developing and sustaining Big Data apps
List of popular backend stacks: ragingwind/backend-architectures.md
CRAN Task View: High-Performance and Parallel Computing with R
Apache Accumulo by NSA
alternative-internet
OpenFlow for programmable networks: Page on Openflow, switch spec: http://archive.openflow.org/docu...
On data center scale, OpenFlow, and SDN
Docker, The linux container engine , and one use case at Etsy: LXC - Running 14,000 tests per day and beyond! (Part 1)
Apache Mesos (A Platform for Fine-Grained Resource Sharing in the Data Center, tech report: Page on Berkeley )
Apache Spark: Lightning-Fast Cluster Computing via https://news.ycombinator.com/ite...
Google's MillWheel: Fault-Tolerant Stream Processing at Internet Scale: Page on Googleusercontent
Google's Sibyl, a distributed learning system: www.magicbroom.info/Papers/Ladis10.pdf‎
Berkeley amplab - SQL Benchmark: Redshift vs Hive vs Impala vs Shark : Big Data Benchmark based on Page on Brown
The Log: What every software engineer should know about real-time data's unifying abstraction | LinkedIn Engineering by Jay Kreps (also Kafka: a Distributed Messaging System for Log Processing )
(This is a live list. Edits and additions welcome)

View Answer on Quora

Why is ETL associated mostly with data warehousing? Isn't ETL a procedure that eventually may apply when loading raw data to a transactio...

2014-02-25T00:11:00.000-06:00

Why is ETL associated mostly with Data Warehousing?
Isn't ETL a procedure that eventually even may apply when loading raw data to a transactional databse?

View Question on Quora

Eventual consistency in simple terms

2014-02-03T01:37:00.000-06:00

Unless you live under a rock, you know 15 minutes can save you 15% and big data is going to make life super awesome!

I don't know if GEICO has consistency issues but big data does and I get the feeling the IT media and software vendors are trying to hide that under a rock.

Big data does a few things really well. Specifically, write data and scale. Sounds awesome, huh? Well, hold on. In order to do this it has many, many copies of the data stored all over the place. Sometimes the copy A is updated and sometimes copy B is updated. The point is to be able to write data at amazing scale and speed and distributing the data all over the place achieves this very difficult task.

But, but ...

Don't worry, eventually these copies of data are synchronized to represent a single version of the truth.

But, but ...

The word eventually scare you? It should. You see, in order to write data at all times and scale infinitely, something had to give. That something is the synchronization of distributed data.

Put simply, big data distributes multiple copies of a database, writes to whichever copy is available and synchronizes the data when it can.

What the ...

I know, right? I'm curious if there is a distributed system out there that has ever reached consistency. If the point is to write at all costs because you have such a high volume of updates, when is there time to synchronize?

For those who prefer a more technical definition of eventual consistency:

Eventual consistency is a consistency model used in distributed computing that informally guarantees that, if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value.[1]

How does this necessity of invention sit with you?

[polldaddy poll=7768766]

Microsoft Data Quality Services: Market Impact?

2014-01-23T08:30:00.000-06:00

I have been getting my feet wet with Microsoft's Data Quality Services and as I read more and more I cannot help but think about how this feature introduction will impact the DQ market. I know there will be some that scuff at this suggestion, but I think there a ripple forming in Seattle (and I am not talking about Richard Sherman) that is going to stir the market.

Here is why I think Microsoft has the potential to disrupt the DQ market ...

most DQ solutions on the market cost more than the SMB market can afford

they require skill sets that are hard to come by and demand considerable compensation

many SMBs own MS SQL Server for some, if not all, their database needs

what do you think?

[polldaddy poll=7739976]

relationship fail

2013-10-23T03:06:00.000-05:00

trust me

this is not a relationship strategy

Would you read my Big Data book?

2013-09-26T05:58:00.000-05:00

As you my know, I am pretty critical of the "big data" hype in the information technology media. One of my criticisms is the very term "big data". Frankly, I hate the term as much as I hate the outrageous claims that are out there.

I prefer to talk about distributed systems which is really the differentiator in this space, in my humble [ref] degrees of humility can easily be argued[/ref] opinion.

As I researched distributed systems (DS), I came across a topic that struck a chord with my passion for data quality. The topic was the CAP Theorem and notion of eventual consistency. At this point, I am becoming obsessed with the notion of eventual consistency.

As a result, I am convinced I need to write a book on the topic and its implications on deploying distributed systems to a business and its decisions.

However, before I get too far down the line, I want to know is ...

would you read my book about eventual consistency and how that would impact a distributed system's ability to aid in the process of making business decisions?

Please vote and let me know!

[polldaddy poll=7427832]

MySQL: loading data from an external file

2013-09-25T04:54:00.000-05:00

Path of least resistance

When you need to get something done quickly, the community edition of MySQL is a great option. It has just what you need in terms of a management studio and requires very little setup.

Yesterday I was in that exact circumstance. I have some data files that I need to analyze and not much time to do it. So I downloaded the community edition of MySQL (found here).

Data time!

As I said, I needed to get some analysis done quickly, so I created a few tables with your basic ANSI create table statements and I was ready to load my data.

I was loading data from delimited sources not located in the default MySQL data directory so I had to include a few wrinkles in my load data statement. By the way, load data is the command for loading data from a file into a MySQL table.

The wrinkles

Since I was loading data from an external file I had to include the following statement in my load data statement:
Load data LOCAL INFILE 'INSERT FILE PATH HERE'
I was using a comma delimited file so I had to include the following statement as well:
FIELDS TERMINATED BY ','
And finally, I had column headers in external file so the following statement was necessary:
IGNORE 1 LINES
The load script, when complete, looked like this:


Use SCHEMA NAME;
Load data LOCAL INFILE Load data LOCAL INFILE 'INSERT FILE PATH HERE'
into table TARGET TABLE HERE
FIELDS TERMINATED BY ','
IGNORE 1 LINES;

Nothing too magical, but I wanted to pass it along and, hopefully, save some of you the time in looking up the statements.

More information on MySQL load data parameters can be found here

a thought on brand loyalty

2013-09-15T03:16:00.000-05:00

dance with who brought you

Want brand loyalty? Take your customers to a dance!

How to design a Source to Target Matrix

2013-09-01T05:16:00.000-05:00

After working a Source to Target Matrix (STTM) for weeks I came to a rather disappointing conclusion ... we laid out the matrix by sources and a developer can't pick that up and start developing. To say the least I was upset at myself and frustrated that we needed to reorganize this document again. So I decided to pass on this piece of advice to you so you don't have to go through this like did.

Organize your STTM by targets and not sources

The target is what the developer is using as a starting point. As a result, the developer needs to have his work, the mappings, organized by targets. I'm not suggesting deviating from the left to right convention of source to target, but rather building a STTM for each target. In many cases this means having a Excel tab for each target.

Sounds simple enough, right? In reality it is simple but the complexity comes in how the information is gathered or relayed from clients. If you start conversations with the client in the context of a target the approach is straight forward. But that's not reality. Clients start by talking about sources because that is what they know. At the end of that session you end up with one source and multiple targets and a document you need to reorganize.

Have migration conversations with clients in the context of the target and save everyone the time of reorganizing and confirming the requirements

This approach is rooted in the principle of beginning with the end in mind. In my experience, when everyone understands the goal in detail it is much easier, efficient and accurate to hit the target (no pun intended! Well maybe a little).

comprehension

2013-08-23T08:09:00.000-05:00

If you can't explain it simply, you don't understand it well enough.

The dependent relationship of data quality and data governance

2013-07-23T07:56:00.000-05:00

In my experience adding data quality to a data governance initiative is critical to an efficient, effective and demonstrative program. In this presentation I outline the ways in which data quality can enable and increase the effectiveness of the three core data governance activities.

[slideshare id=24547684&doc=thedependentrelationshipofdataqualitydata-130723125048-phpapp02]

K9 Wisdom that applies to many situations

2013-07-03T03:24:00.000-05:00

once a dog bites, he is a biting dog

Anatomical clue

2013-06-04T03:20:00.000-05:00

you have 2 ears and 1 mouth ... listen twice as much as you talk

Salesmen, product managers, consultants, experts and developers take heed ... you are boring and are being paid to listen not talk.

There I said it, were you listening or waiting to talk?