Featured Blog Posts - AnalyticBridge

When to use a view and when a table, the fuzz about it for analysis

2013-05-20T10:50:53.000Z

When to use a view and table?
Is it an art? I doubt it. Only when you cannot explain it and depends on feeling, it is an art.
Recently we are revising our codes and this question of giving guidance.on when to use table and view come to me. I consider the following questions:
1. when does it justify a table/view?
Having worked in a team setting on modeling and reporting works, the key to me is readability. leveraging my programming learning on object oriented concept, every object exists for a purpose, with set of inputs, preconditions, processes and output. When you feel that the same processes can be reused for other reports or analysis, I created a view or table for it. For those of you from the database admin, put down your judgement adhereing to normalization, listen: the cost of screwing the model or analysis leading to wrong business decision outweights the cost of harddisk.
2. View or table?
To choose between, view is always preferred as it takes less disk space and one less program that creates the table. (now we are talking admin :)). Only when the same data is queried very often or very big in size, when performance in speed is needed,I use a table.
I believe that our task is to define as clearly as possible our programming style for data analysis, even now it is empty in the programming literature.

R (Web Server) Solutions - Amplifying Artichokes

2013-05-20T04:52:38.000Z

Every month I see one or more new R based web server solutions coming into the market, sight seeing some of them thought of sharing one of my old architecture map manifested to the client long back in early 2009 (good to see quick spreading of scalable and customizable open source statistical computing tool in the market).

Weekly Digest - May 20

2013-05-17T17:30:00.000Z

Selected articles, blog posts and forum questions from DSC, AnalyticBridge, BigDataNews

The application of Propensity Score Matching

2013-05-16T09:00:39.000Z

Propensity Score Matching is a statistical matching technique that attempts to estimate the effect of a treatment, policy or other intervention by accounting for the covariates that predict receiving the treatment. It helps to reduce bias due to confounding and can be used to estimate the counterfactual outcome.

For example, many of you will have been to a particular university or school and achieved a certain result. But have you ever wondered what could have been the result if you had attended somewhere else (the counterfactual outcome) ? To determine this you would need to account for the covariates using information on people like yourself who studied the same course. Then, you could estimate this counterfactual outcome using Propensity Score Matching.

I have put various resources (including SAS code) on my blog. These have allowed me to do Propensity Score Matching - See blog post here: What could propensity score matching do for you ? (with examples from justice, medicine, education and finance).

Ian Morton has built propensity scoring models for the financial services sector, for a utility company, and for the public sector. He has given a number of presentations on the technique of propensity score matching, and has also co-authored a forthcoming peer-reviewed journal article.

Strategy for building a “good” predictive model

2013-05-16T04:00:00.000Z

By Ian Morton. Ian worked in credit risk for big banks for a number of years. He learnt about how to (and how not to) build “good” statistical models in the form of scorecards using the SAS Language.

Read original post and similar articles here. I thing Ian's list below is a good starting point. I would add a few steps such as deployment, maintenance at the end, and gathering requirements, understanding goal and success metrics at the top.

Initial investigations

1. Look at the data dictionary to see which data is available

2. What is the outcome ? is it yes / no ? is it continuous ?
3. Decide upon the model required (logistic ! for yes / no outcome)

Getting the data ready

4 cross tabulations on categorical variables to understand the coding and volumes

5. summary statistics to understand the distribution of the continuous variables
6. Ask questions about data quality:

remove these variables from any potential models ? or,
think about imputation ? or,
obtain accurate data ?

7. Convert continuous variables into categorical variables

Modelling

8. Check for multi-colinearity / correlation between variables (variance inflation factors), or correlation tests

9. Check for interactions
10. Choose type of logistic approach (e.g. forward, backward, stepwise)
11. Choose the baseline attribute for each categorical variable
12. Create a random variable – mustn’t step into the model - something is wrong if it does step into the model
13. Split the dataset into two parts (ratio 80%/20%)

using random selection without replacement
the larger sample is the build dataset
the smaller sample is the test dataset

14. Put all variables from the build dataset (including interactions and the random variable) into the model and run it

Check odds ratios – do they make sense ?, and
Check the coefficients – do they make sense ?

Check the model

15. Do diagnostic checks and plots of the fit (e.g. Somers D, residuals etc., etc.)

16. Put all variables from the test dataset (including interactions and the random variable) into a new model and run it

Are the coefficients the same as the model it was built on ? and
Are the odds ratios the same as the model it was built on ?

Start again

17. Back to the start, fine tune the grouping of the data, put variables in or take variables out.

Related articles

Analytical Skills Development Spending.

2013-05-13T08:04:00.000Z

Challenged with the acquisitiveness for adaptability and agility, analytical service organizations are turning to real-world work & emergent technologies/methods to develop next-generation analytical skills (talent). However, corporate bosses seems to playing it cautious when coming to their spending on analytical skills development.

According to our survey conducted by us across a good number of analytical services organizations in India, most expect to increase their current spending on analytical skills development (43.8%) in next six months. Around 27.2% organizations expect to spend aggressively in next six months. Only 6.7% expect spending to decrease in next six months. While 22.3% of organizations spending on analytical skills development continue to be frozen in next six months.

The End of Theory: The Data Deluge Makes the Scientific Method Obsolete | Wired

2013-05-13T01:30:00.000Z

Here's my rebuttal to this article published in Wired in 2008.

Vincent's rebuttal:

A lot can be done with black-box pattern detection, where patterns are found but not understood. For many applications (e.g. high frequency trading) it's fine as long as your algorithm works. But in other contexts (e.g. root cause analysis for cancer eradication), deeper investigation is needed for higher success. And in all contexts, identifying and weighting true factors that explain the cause, usually allows for better forecasts, especially if good model selection, model fitting and cross-validation is performed. But if advanced modeling requires paying a high salary to a statistician for 12 months, maybe the ROI becomes negative and black-box brute force performs better, ROI-wise. In both cases, whether caring about cause or not, it is still science. Indeed it is actually data science - and it includes an analysis to figure out when/whether deeper statistical science is really required. And ill all cases, it always involves cross-validation and design of experiment. Only the statistical theoretical modeling aspect can be ignored. Other aspects, such as scalability and speed, must be considered, and this is science too: data and computer science.

Related article

Use PRESS, not R squared to judge predictive power of regression

2013-05-12T15:00:00.000Z

R squared, also known as coefficient of determination, is a popular measure of quality of fit in regression. However, it does not offer any significant insights into how well our regression model can predict future values. Instead, the PRESS statistic (the predicted residual sum of squares) can be used as a measure of predictive power. The PRESS statistic can be computed in the leave-one-out cross validation process, by adding the square of the residuals for the case that is left out. As a reminder, in the leave-one-out cross validation, one case of the data set is used as the testing set and the remaining are used as the testing set. We iterate this process, until all cases have served as the testing set.

Here is an example implemented in R, on the gala dataset in the faraway package:

> gala[1:3,]

Species Endemics Area Elevation Nearest Scruz Adjacent

Baltra 58 23 25.09 346 0.6 0.6 1.84

Bartolome 31 21 1.24 109 0.6 26.3 572.33

Caldwell 3 3 0.21 114 2.8 58.7 0.78

Model1:

>model1<-lm(Species~Endemics+Area+Elevation)

>summary(model1)

....

Residual standard error: 27.29 on 26 degrees of freedom

Multiple R-squared: 0.9492, Adjusted R-squared: 0.9433

F-statistic: 161.8 on 3 and 26 DF, p-value: < 2.2e-16

Model2:

> model2<-lm(Species~I(Endemics^2))

> summary(model2)

...

Residual standard error: 27.1 on 28 degrees of freedom

Multiple R-squared: 0.946, Adjusted R-squared: 0.9441

F-statistic: 491 on 1 and 28 DF, p-value: < 2.2e-16

Model3:

> model3<-lm(Species~Endemics+I(Endemics^2))

> summary(model3)

.....

Residual standard error: 22.94 on 27 degrees of freedom

Multiple R-squared: 0.9627, Adjusted R-squared: 0.9599

F-statistic: 348.5 on 2 and 27 DF, p-value: < 2.2e-16

Here are now the AIC (Akaike test criterion), BIC (Bayesian information criterion), and PRESS statistic of the three models:

Model 1:

>AIC(model1)

289.243

> BIC(model1)

296.249

PRESS(model1)=259520.5

Model 2:

> AIC(model2)

287.0325

> BIC(model2)

291.2361

PRESS(model2)=26382.22

Model 3:

> AIC(model3)

277.9558

> BIC(model3)

283.5606

PRESS(model3)=22567.03

As we can see, the PRESS statistic is significantly smaller (better) for models 2 and 3, while R squared has a trivial improvement for model 3. So, according to PRESS, model 3 has the highest predictive power. It is interesting to note that the AIC and BIC also get their best values for model 3.

If you are interested in how I computed the PRESS statistic doing cross-validation in R, please check my next blog post.

Using Big Data to Improve Customer Experience

2013-05-09T10:24:41.000Z

This week we are happy to introduce and welcome our guest blogger Alastair Kane. Alastair is a freelance writer and has provided this article on behalf of Acxiom a company providing big data analytics services.In this post he talks about how Big Data can enhance Customer Experience.

What is Big Data, and why is it different from other data

As the world becomes increasingly digitised, access to reams of information about customers, including data detailing buying behaviours, preferences and dislikes is advancing. These useful, but separate bits of information arrive in a variety of formats, from an equally varied number of sources, as in addition to traditional CRM information, data is now being exponentially captured from social posts, blogs, via smartphones and other diverse digital sources. Formed of small pockets of brand-useful information, plus an explosion of other data 'noise', this voluminous and varied information is what's known as 'big data'.

What Makes ‘Big Data’ Different?

‘Big Data’ is different from traditional data as it is unstructured and scattered, making it impossible for traditional SQL databases to make sense of. And of course, there’s just so much more of it! That’s why new ways of analysing big data (big data analytics) are required - to assess and monetise huge pools of information collected from customers - from the web, call centers in fact from just about anywhere you can think of.

Building a good predictive model for credit risk

2013-05-09T16:00:00.000Z

A colleague of mine wanted to understand how to build predictive models, and asked if I had a strategy for building them. I thought it would be useful to share this. For more details about each stage see my personal blog (My suggested strategy for building a “good” predictive model).

Stage 1 – Perform initial investigations

Stage 2 - Getting the data ready

Stage 3 - Modelling

Stage 4 - Check the model

Stage 5 - Start again

The bottom line: It’s an iterative process and it might take some time to get a model that’s acceptable in terms of fit, and acceptable to business users. Always, always, always and at each stage consult with the business to check on ethical issues, applicability of the model, and that the model can be implemented.

Ian Morton worked in credit risk for big banks for a number of years. He learnt about how to (and how not to) build “good” statistical models in the form of scorecards using the SAS Language.

George E. P. Box “Essentially, all models are wrong, but some are useful”

Wiley's list of leading and interesting blogs to follow

2013-05-08T16:00:00.000Z

Here are the top 10, in alphabetical order. Wiley's full list mentions many interesting statistical blogs.

The Analysis Factor -offers statistical consulting, resources, and training that researchers need to conduct quality work.
Analytic Bridge - A group blog written by data scientists about statistics and data analysis.
AnnMaria’s Blog - The blog of Dr. AnnMaria De Mars, President of the online statistics education company The Julia Group.
Bad Science - The blog of Dr. Ben Goldacre, an epidemiologist who uses statistics to debunk bad science.
Beyond the Box Score - A blog using statistics to analyse the game of baseball.
Blog About Stats - Blogstats creates a network for dissemination professionals mainly of statistical institutions. It is a meeting point where colleagues from Statistical Organizations share their experiences, successes and failures, focus attention on (new) developments and stay informed in the vast domain of disseminating (statistical) information.
bLog-Normal Distribution - 'The Corner Office blog', where SAS executives post their thoughts on global business, analytics and technology
By the Numbers - The blog of New York Times visual Op-Ed columnist Charles Blow conducts discussions about all things statistical — from the environment to entertainment — and their visual expressions.
Byte Mining - A blog written by Ryan Rosario, an ex-Statistician, now turned Computer Scientist, even though he is a Ph.D. candidate in Statistics at UCLA.
CoolData blog - A blog written by Kevin McDonell, Business Analyst for the Office of External Relations at Dalhousie University in Halifax, Nova Scotia where he concentrates on data mining and predictive modelling for many Advancement functions.

Read full list at http://stats.cwslive.wiley.com/details/tools/13ae5fb1aa6/A-list-of-leading-and-interesting-blogs-to-follow.html

Related articles

What Agile Desktop BI Tool Really Means?

2013-05-08T08:15:02.000Z

BI (Business Intelligence) refers to the intelligence and ability to enhance the enterprises competitiveness, involving report presentation, reporting result calculation, OLAP analysis, business data calculation, data mining and predication. Among these, there are both the technician-oriented high-level systems, and the business user-oriented agile desktop BI tools. In this post, we only talk about agile desktop BI tools. But what is agile desktop BI tool and what are their standards?

Agile desktop BI software should have the following features:

Common BI features support:

The agile desktop BI software should fit for those business personnel to prepare the static report by themselves, even if they are inexperienced in IT. The friendly reporting design interface is necessary. It also supports the rapid reporting, the business personnel-oriented data processing, and high fidelity report preparation. In other words, it is capable to keep reports consistent in the stages of design, preview, pagination & printing, and export.

For example, make a product sales situation report for specific products to present the monthly sales of 3 products, and their link relative ratio, and monthly year-on-year growth.The agile desktop BI software supports the calculation on the result of common reporting tools. Excel and the plain txt are the export formats supported by most reporting tools. It can import and calculate them directly; supports the data pasting from the report result directly on the clipboard; provides the calculator-style operation for business personnel; capable to conduct any process on data and the calculation between steps can be transited smoothly.

For example, the reporting tools generate the below report result:

Find the big clients who account for 60% of the total sales of the company based on the above data. The result is as follows:

The agile desktop BI tools support the true OLAP analysis. It is capable to perform the interactive analysis arbitrarily and intuitively, decompose and simplify the obscure analysis goal. It provides the basic analysis methods that are both simple and easy to use. Then, lots of advanced analyses can be implemented through the free combination. esCalc is such kind of desktop BI software. With an Excel-style, esCalc becomes relatively easier to understand and learn. Moreover, esCalc also provides a range of powerful advanced functions to solve the complex problems regarding OLAP.

Agile installation deployment:

The size of the agile BI tool should be very small and easy to install and uninstall. For example, esCalc installer is only dozens of MB and only requires a few clicks to install and run, which is the typical example of agile desktop BI software. Agile desktop BI tools are capable to run on most desktop computers independently, not having to deploy the additional server:

Agile desktop BI software supports various mainstream databases, like esCalc which can manipulate data from different databases, including MSSQL, Oracle, Access, MySQL, DB2, and Sybase. This desktop BI software also supports the local data files, for example, Txt, Log, tab, and other text files; Excel 97, Excel 2010, and the Excel of other versions. It also supports the interactive calculation between various data sources, such as the calculation between Oracle and Excel.

Agile formula functions:

Agile BI software also provides the agile formulas and functions, so that the business personnel can easily represent the relatively complex calculations, such as comparison on year-on-year basis, link relative ratio, set operations, ranking and row number calculations.

For instance, calculate the players whose rankings are among the top 5 in every game. We take esCalc, the agile desktop BI software as an example. The data available is as follows:

Simply input "={A3}" in E2, and the top 5 players of each game will be calculated out in E2, E8, and E14. Input "={E2}.isect ()" in E1, then the players whose rankings are among the top 5 in every game will be calculated automatically. In which, the function "isect" is to calculate the intersection of sets, and "{E2}" is a set to indicate "cells shares the same meaning with E2 regarding business” (homocell by name), that is, E2, E8, and E14.

It also has the similar function like:

diff(): Calculate the difference set of a group of data; for example, calculate the employee who made a full attendance in this quarter. You can calculate through the formulas like [set of employees, employee who ever absent in the 1st month, employee who ever absent in the 2nd month, employee who ever absent in the 3rd month].diff() to calculate.

There are also other advanced functions available, such as the sum(~*~) to calculate the sum of squares of a certain group of data; cumulate() to calculate the cumulative value of a group of data, ord() to calculate the relative row number in the calculation hierarchy, and ranki() to calculate the ranking of a certain number in a group of numbers.

All in all, agile desktop BI software supports the common BI functions, and is business personnel-oriented in the respect of installation deployment, and formula function.

Predictive Analytics in Campaign Management

2013-05-08T01:00:00.000Z

One of the most popular application of predictive analytics is optimization of marketing campaign management. I've implemented it several times for clients in four different industries and here is my solution.

Introduction to campaign management

Marketing departments are usually responsible for management the marketing campaigns. Campaigns always have some triggers - may be planned, triggered by customer events and various alerts. Campaigns are run for a purpose and usually carry strong call-to-action message. Most common call-to-actions are acquisition, x-sell, retention or win-back. Campaigns are delivered through various channels. There are channels for Above The Line (ATL) and Bellow The Line (BTL) communication.

Customer Management Solution with Predictive Components

Feedback loop is very important. In my solution consists of four components: responses collection & reporting, evaluation across customer segments, predictions what happens next and ROI analysis / projection.

Role of predictive analytics

Predictive analytics is used for campaign optimization. Can identify the best customers for campaign with specific goal. This approach saves the most of budget or lead to better results with the same budget. For example with 20% of budget you achieve 80% of all possible sales in a campaign. Or you can identify customers at risk of churn and run very precise campaigns with "stay with us" call-to-action.

Customer segmentation provides better insight. One campaign can work well in one segment and fail completely in another. The segmentation unveils such fails and you never repeat the same failure twice in given segment. And finally we got to the profitability. It's always great to measure it backwards and see what happend. And it's even better to simulate what will happen next! In this circle you have all required data for projections. To certain point the customer behavior is well predictable. So you can make your go / no go decision well informed.

Conclusion

Predictive analytics is optional and for companies whose mastered it delivers great value. The majority of companies is not using it at all. Some companies use predictive analytics and don't run any campaigns. They let results in the drawer. For various reasons. The most common reason is lack of good infrastructure what makes the process manageable for business people without hard skills. That is the topic for my next post. Stay tuned.

Joseph Kovac

CEO of 7SEGMENTS

http://www.7segments.com

Choosing a BI Vendor - Making the Short List

2013-05-05T13:17:19.000Z

There is no shortage of business intelligence vendors out there. They all claim to be powerful, easy-to-use, flexible and affordable. So how do you pick the one that is right for you?

In order to be able to choose the right BI vendor from the abundance out there, the best way is to follow high-level, yet restrictive, criteria and only then compare them on a feature-by-feature basis. Here are a few tips that will help you do that, as well as avoid common mistakes typically made when choosing a BI solution. This is the 21st century, and BI solutions are completely different than what you may be used to. If you follow these tips, you’ll end up with a very short list of vendors, and then it’ll just be a matter of choosing the one you feel most comfortable with in terms of specific features, pricing, support, etc:

Find a Complete Solution, Not Just Pretty Visualization.

The visualization of data is important, of course, but the biggest mistake you can make is judge the BI vendor based on the pretty dashboard samples they show you on their website or during a demo. Every BI vendor can do that because visualization software components are a dime a dozen. The real challenge is customizing these dashboards to your own needs and having them show your own data. This part usually takes most vendors months, and costs you bundles. If the BI vendor cannot get your own data to show the way you like it within just a few days, you could probably find a better one.

Beware of the Data Warehouse.

A data warehouse is a centralized database filled with all the business’s data, and for years it’s been making a ton of money for BI vendors and bringing nothing but grief to customers. Today’s BI technology does not require a data warehouse, even when there are multiple data sources involved, large amounts of data or multiple users querying the data. There are very specific scenarios where a data warehouse is a good idea, but they are most likely not relevant to you. If the vendor requires a data warehouse to proceed with implementation, it is most likely you should keep looking.

Beware of the OLAP Cube.

OLAP, which stands for Online Analytical Processing, is 20 year old technology designed to improve query performance over medium to large datasets. OLAP is also very lengthy and costly to implement, and there is really no need for it anymore. Today’s BI technology can handle even huge amounts of data without OLAP, at fractions of the time or cost. If the BI vendor requires OLAP to assure you acceptable query performance, you should probably move on.

Refuse to Make Significant Upfront Investments.

Many BI vendors will promise you the world, but will demand significant upfront investment in preparation projects, hardware and software before you even get to run a single report on your actual data. Do not agree to this, and demand to have at least one solid report or dashboard running over your own data before you commit to anything significant in advance. If the vendor is not willing to do so, it’s probably because they would have to spend weeks on development before they can reach that point. That typically means this vendor is either using very old technology or is simply trying to pull one over you.

Be wary of Vendors whose Business is Prof. Services.

Vendors who sell real home-grown BI software products (in contrast to OEMing someone else's software) do not like engaging in long professional services projects because it hurts their margins. That is why they prefer to create software that is easy enough to be used directly by the customer or through a third party (which usually lives off these professional services contracts). If you choose a BI vendor who makes most of his business off professional services (as opposed to software sales), you can pretty much be sure that they will take their time building your solution. These types of BI vendors also live off on-going maintenance services, so what you initially pay for the solution is actually only the beginning. Whenever possible, try to choose a BI vendor that focuses on selling BI software to the end customer, not to the professional services community.

Make the Vendor Prove it To You.

The most important thing is to make the vendor prove what they claim prior to investing too much money upfront. This proof must be in the form of reports, dashboards or analytics in real life scenarios, running on real data, used by the actual end users and within a reasonable amount of time. If a vendor is not willing to accommodate this simple request, you really should find one that does. Many vendors provide free trial versions, as well as utilize technology that speeds up implementation tremendously. If the one you're in contact with now doesn't, they shouldn't make your short list.

See more at www.sisense.com

Weekly Digest - May 6

2013-05-03T17:00:00.000Z

Announcements and articles from the last couple of days:

Viral program for data science authors and bloggers - Read updated version.
R code for model-free, data-driven confidence intervals
Choosing a BI Vendor - Making the Short List
How do I become a data scientist?
What are the best blogs about data?
My O'Reilly book collection
I could not resist to adding this book
Data Science Apprenticeship update - New study material added: check out the starred items.
Shooting stars: new R code to produce new, great videos
When data flows faster than it can be processed
Big Data, Big Relevance
New LinkedIn group about Data Science Training - 3,000 members in the first 24 hours.
Predicting the reality show winner using big data
How to find First and Second Order effects and their significance/causality in Data Analysis
Can we automate data mining?
How to install MapR M3 on Ubuntu through Ubuntu Partner Archive.
Rajesh Ramamurthy tells us how he made a career shift into analytics
6 Advantages of Analytics-driven Hiring Technology
New press releases

Previous digest | Recent jobs | Top Links | Data Science eBook | Apprenticeship

6 Advantages of Analytics-driven Hiring Technology

2013-05-02T16:30:00.000Z

By now you are probably familiar with the business value analytics can provide in a myriad of business contexts. If not, you’ll want to read The Value of Business Analytics by Evan Stubbs as it’s one of my favorite analytics books for the business community.

One opportunity that cuts across all industries, functions and levels where analytics-driven technology can provide a tremendous advantage is by improving how we hire. Hiring is a critical decision for any analytics manager because it represents arguably the most direct impact on business performance – the people doing the work. At a high level, improved hiring decisions is what analytics-driven hiring technology offers.

Yet, according to the well-publicized expected analytic talent shortage, many firms are finding it increasingly difficult to attract and retain top talent.

If we drill in a bit further, there may be some ways to address analytic talent hiring challenges by using analytics-driven hiring technology. What follows are some key advantages that make analytics-driven hiring technology a sound strategic investment for innovative hiring managers responsible for analytics teams, with a focus on optimizing both short and long term business performance.

Analytics-driven Hiring Technology Advantages

1. Improve hiring decision effectiveness by reducing risk. Bad hires are an expensive, yet pervasive and well documented business problem. While no one can claim hiring perfection, analytics-driven technology that can reduce the risk of a bad hire is an invaluable ally for any hiring manager.

2. Reduce ramp-up time and business disruption. No one enjoys a disruptive and time-consuming hiring process, whether it’s you as the hiring manager, your team, your candidates or the business. Analytics-driven hiring technology shortens disruption and ramp-up time by speeding up collaboration through pre-boarding with Team Playbooks. The new hire hits the ground running and knows how to best work together with their team.

3. Google for your candidate pool. The pool of candidates for the role you are hiring for may have dozens or even hundreds of candidates, making it time consuming to find candidates that closest match what the role requires. Just like a search engine uses algorithms behind the scenes to instantly shortlist search results for you, analytics-driven technology enables you to benchmark the specific role and let the analytics engine shortlist job candidates that most closely match its pre-defined requirements in context of the culture of your team and organization.

4. Optimize interview time. Analytics-driven technology enables innovative hiring managers to conduct candidate gap analysis prior to the interview. This allows interview time to be spent better understanding specifically how the candidate will perform in the role, giving managers valuable data to support their decision they would otherwise learn after the decision has been made.

5. Reduce interview bias. Since two different interviewers may have two completely opposite opinions of how a candidate did in the interview, analytics-driven technology provides an consensus on key areas to focus on. This is an easy way to take add more objectivitY into the candidate rating process by taking the interviewer out of it.

6. Improve employee/manager relationship. If the common saying “employees join companies and leave managers” is true, it underscores the importance of getting off on the right foot. Analytics-driven technology allows you as hiring manager to quickly aggregate the communication preferences of your direct reports from day one (and vice versa) by quickly understanding opportunities for synergies and conflict. This lets both you as manager and your employees customize communication and develop stronger rapport, which in turn yields greater engagement and productivity.

This is just a quick look at a few opportunities analytics-driven hiring technology provides.

What other advantages can analytics-driven hiring technology provide? Feel free to comment below!

I did it! Jigsaw Academy student Rajesh Ramamurthy tells us how he made a career shift into analytics.

2013-05-02T08:33:55.000Z

Many of Jigsaw’s students join our various courses because they can hear the rising crescendo within the analytics industry. They yearn to be a part of this challenging and exciting career, one that is fast paced, has rich monetary returns and can fast track your career path. For those of you out there, who also have such dreams, are you skeptical and nervous because you have no previous experience in analytics? Well, be reassured, many have walked a similar path. Having no previous analytics experience, they took that first step and contacted Jigsaw, registered for one of our analytics courses, worked hard at their assignments and upon completion of the course, were able to find a job in the analytics sector.

This week we would like to showcase one of our bright and talented students who after successfully completing one of our courses, has been able to break into the analytics arena.

Shooting stars

2013-04-30T01:00:00.000Z

This is a follow up to our video series From chaos to clusters, made with data points moving over time to form clusters, and produced with open source and home-made data science algorithms.

See below two frames from the new video, now featuring line segments connecting a current point to its location in the previous frame. These line segments are overwritten and change constantly from iteration to iteration, creating a "shooting stars" visual effect when you watch the video.

Towards the end of the video, the clusters are well formed (though they are also moving, especially the one at the bottom right corner) and points coming from outside are progressively attracted to the nearest cluster: you can see them quickly getting close and then get absorbed.

Here are the two new videos:

Video #4 (orange, more visually pleasant | Youtube version)
Video #5 (blue, better shooting star effect | Youtube version)

Download the data file rfile3.txt used to produce these videos, (also available in compressed format). These videos are based on the following R script (a more complex version of our initial R script):

R Source code

vv<-read.table("c:/vincentg/rfile3.txt",header=TRUE);
iter<-vv$iter;
for (n in 1:199) {
x<-vv$x[iter == n];
y<-vv$y[iter == n];
z<-vv$new[iter == n];
u<-vv$d2init[iter == n];
v<-vv$d2last[iter == n];
p<-vv$x[iter == n-1];
q<-vv$y[iter == n-1];
u[u>1]<-1;
v[v>0.10]<-0.10;
s=1/sqrt(1+n);
if (n==1) {
plot(p,q,xlim=c(-0.08,1.08),ylim=c(-0.08,1.09),pch=20,cex=0,col=rgb(1,1,0),xlab="",ylab="",axes=TRUE );
}

points(p,q,col=rgb(1-s,1-s,1-s),pch=20,cex=1);
segments(p,q,x,y,col=rgb(0,0,1));
points(x,y,col=rgb(z,0,0),pch=20,cex=1);
Sys.sleep(5*s);
segments(p,q,x,y,col=rgb(1,1,1));
}

segments(p,q,x,y,col=rgb(0,0,1)); # arrows segments
points(x,y,col=rgb(z,0,0),pch=20,cex=1);

Related articles

Predicting the reality show winner using big data

2013-04-29T15:25:27.000Z

I wonder, using big data and predictive analytic, can we predict the winner of x-factor or American Idols from the start of their audition performance? I think we might have a good chance to predict the winner right away.

What if we could only have the information from their first performance, what should be the variables to be used in the predictive model? Here’s from what I could think of:

The voice: quantified timbre, energy and rhythm
Song selection: how popular the song was, type of music (pop/jazz/country)
The singer appearance: body mass, hair color, skin color, clothing, type of shoes, color contrast, etc (some of these variables might not be legal to use)
The early response from panel of judges: number of yes/no
Wisdom of the crowds: mentions at twitter, number goods vs bad sentiments, number of videos uploaded to YouTube, number of download from iTunes, etc.
Audience claps: number of decibels from audience claps

Once we have all of these variables, we might be able to predict the winner this coming season X-factor/American Idols/The voice.

However, if the model is able to predict the winner, who should be benefited from this algorithm? The producer of the show could be one. Once he/she knows who should be the winner, he/she can play with the TV viewers’ emotions by altering some of the significant variables. The audience could be more attached if their favorite singer is about to lose and need more support. With more viewer’s getting more attached, the TV can have higher rating and higher advertising income. Hear the KA-CHING?

Can we do this project just for fun? :)

Predicting the reality show winner using big data

Hadoop Herd : When to use What...

2013-04-26T00:55:43.000Z

8 years ago not even Doug Cutting would have thought that the tool which he's naming after the name of his kid's soft toy would so soon become a rage and change the way people and organizations look at their data. Today Hadoop and BigData have almost become synonyms to each other. But Hadoop is not just Hadoop now. Over the time it has evolved into one big herd of various tools, each meant to serve a different purpose. But glued together they give you a powerpacked combo.

Having said that, one must be careful while choosing these tools for their specific use case as one size doesn't fit all. What is working for someone might not be that productive for you. So, here I am trying to show you which tool should be picked in which scenario. It's not a big comparative study but a short intro to some very useful tools. And, I am really not an expert or an authority so there is always some scope of suggestions. Please feel free to comment or suggest if you have any. I would love to hear from you. Let's get started :

1- Hadoop : Hadoop is basically 2 things, a distributed file system(HDFS) which constitutes Hadoop's storage layer and a distributed computation framework(MapReduce) which constitutes the processing layer. You should go for Hadoop if your data is very huge and you have offline, batch processing kinda needs. Hadoop is not suitable for real time stuff. You setup a Hadoop cluster on a group of commodity machines connected together over a network(called as a cluster). You then store huge amounts of data into the HDFS and process this data by writing MapReduce programs(or jobs). Being distributed, HDFS is spread across all the machines in a cluster and MapReduce processes this scattered data locally by going to each machine, so that you don't have to relocate this gigantic amount of data.

2- Hbase : Hbase is a distributed, scalable, big data store, modelled after Google's BigTable. It stores data as key/value pairs. It's basically a database, a NoSQL database and like any other database it's biggest advantage is that it provides you random read/write capabilities. As I have mentioned earlier, Hadoop is not very good for your real time needs, so you can use Hbase to serve that purpose. If you have some data which you want to access real time, you could store it in Hbase. Hbase has got it's own set of very good API which could be used to push/pull the data. Not only this, Hbase can be seamlessly integrated with MapReduce so that you can do bulk operation, like indexing, analytics etc etc.

Tip : You could use Hadoop as the repository for your static data and Hbase as the datastore which will hold data that is probably gonna change over time after some processing.

3- Hive : Originally developed by Facebook, Hive is basically adata warehouse. It sits on top of your Hadoop cluster and provides you an SQL like interface to the data stored in your Hadoop cluster. You can then write SQLish queries using Hive's query language, called as HiveQL and perform operations like store, select, join, and much more. It makes processing a lot easier as you don't have to do lengthy, tedious coding. Write simple Hive queries and get the results. Isn't that cool??RDBMS folks will definitely love it. Simply map HDFS files to Hive tables and start querying the data. Not only this, you could map Hbase tables as well, and operate on that data.

Tip : Use Hive when you have warehousing needs and you are good at SQL and don't want to write MapReduce jobs. One important point though, Hive queries get converted into a corresponding MapReduce job under the hood which runs on your cluster and gives you the result. Hive does the trick for you. But each and every problem cannot be solved using HiveQL. Sometimes, if you need really fine grained and complex processing you might have to take MapReduce's shelter.

4- Pig : Pig is a dataflow language that allows you to process enormous amounts of data very easily and quickly by repeatedly transforming it in steps. It basically has 2 parts, the PigInterpreter and the language, PigLatin. Pig was originally developed at Yahoo and they use it extensively. Like Hive, PigLatin queries also get converted into a MapReduce job and give you the result. You can use Pig for data stored both in HDFS and Hbase very conveniently. Just like Hive, Pig is also really efficient at what it is meant to do. It saves a lot of your effort and time by allowing you to not write MapReduce programs and do the operation through straightforward Pig queries.

Tip : Use Pig when you want to do a lot of transformations on your data and don't want to take the pain of writing MapReduce jobs.

5- Sqoop : Sqoop is a tool that allows you to transfer data between relational databases and Hadoop. It supports incremental loads of a single table or a free form SQL query as well as saved jobs which can be run multiple times to import updates made to a database since the last import. Not only this, imports can also be used to populate tables in Hive or HBase. Along with this Sqoop also allows you to export the data back into the relational database from the cluster.

Tip : Use Sqoop when you have lots of legacy data and you want it to be stored and processed over your Hadoop cluster or when you want to incrementally add the data to your existing storage.

6- Oozie : Now you have everything in place and want to do the processing but find it crazy to start the jobs and manage the workflow manually all the time. Specially in the cases when it is required to chain multiple MapReduce jobs together to achieve a goal. You would like to have some way to automate all this. No worries, Oozie comes to the rescue. It is a scalable, reliable and extensible workflow scheduler system. You just define your workflows(which are Directed Acyclical Graphs) once and rest is taken care by Oozie. You can schedule MapReduce jobs, Pig jobs, Hive jobs, Sqoop imports and even your Java programs using Oozie.

Tip : Use Oozie when you have a lot of jobs to run and want some efficient way to automate everything based on some time (frequency) and data availabilty.

7- Flume/Chukwa : Both Flume and Chukwa are data aggregation tools and allow you to aggregate data in an efficient, reliable and distributed manner. You can pick data from some place and dump it into your cluster. Since you are handling BigData, it makes more sense to do it in a distributed and parallel fashion which both these tools are very good at. You just have to define your flows and feed them to these tools and rest of things will be done automatically by them.

Tip : Go for Flume/Chukwa when you have to aggregate huge amounts of data into your Hadoop environment in a distributed and parallel manner.

8- Avro : Avro is a data serialization system. It provides functionalities similar to systems like Protocol Buffers, Thrift etc. In addition to that it provides some other significant features like rich data structures, a compact, fast, binary data format, a container file to store persistent data, RPC mechanism and pretty simple dynamic languages integration. And the best part is that Avro can easily be used with MapReduce, Hive and Pig. Avro uses JSON for defining data types.

Tip : Use Avro when you want to serialize your BigData with good flexibility.

The list is actually pretty big, but I have covered only the most significant tools. Over time if I feel like something else should be mentioned here I would definitely do that. Comments and suggestions are welcome.

From chaos to clusters - statistical modeling without models

2013-04-25T05:00:00.000Z

Here I provide the mathematics, explanations and source code to produce the data and moving clusters in the From chaos to clusters video series.

A little bit of history on how the project started:

Interest in astronomy, visualization and how physics models apply to business problems
Research on how urban growth could be modeled by the gravitional law
Interest in systems that produce clusters (as well as birth and death processes) and in visualizing cluster formation with videos rather than charts
Creating art: videos with sound and images synchronized and both generated using data (coming soon). Maybe I'll be able to turn your business data into a movie (either artistic or insightful or both)! I'm already at the point where I can produce the video frames faster than they are delivered on the streaming device. I called it FRT for faster than real time.

What is a statistical model without model?

There's actually a generic mathematical model behind the algorithm. But nobody cares about the model, the algorithm was created first without having a mathematical model in mind. Initially, I had a gravitational model in mind, but I eventually abandoned it as it was not producing what I expected.

This illustrates a new trend in data science: we care less and less about modeling, but more and more about results. My algorithm has a bunch of parameters and features that can be fine-tuned to produce anything you want - be it a simulation of a Neyman-Scott cluster process, or a simulation of some no-name stochastic process.

It's a bit similar to how modern rock climbing has evolved: focusing on big names such as Everest in the past, to exploring deeper wilderness and climbing no-name peaks today (with their own challenges), to rock climbing on Mars in the future.

You can fine tune the parameters to

Achieve best fit between simulated data and real business (or other data), using traditional goodness-of-fit testing and sensitivity analysis. Note that the simulated data represents a realization (an instance for object-oriented people) of a spatio-temporal stochastic process.
Once the parameters are calibrated, perform predictions (if you speak statistician language) or extrapolations (if you speak mathematician language).

So how does the algorithm work?

It starts with a random distribution of m mobile points in the [0,1] x [0,1] square window. The points get attracted to each other (attraction is stronger to closest neighbors) and thus over time, they group into clusters.

The algorithm has the following components:

Creation of n random fixed points (n=100) on [-0.5, 1.5] x [-0.5, 1.5]. This window is 4 times bigger than the one containing the mobile points, to eliminate edge effects impacting the mobile points. These fixed points (they never move) also act as some sort of dark matter: they are invisible, they are not represented in the video, but they are the glue that prevents the whole system from collapsing onto itself and converging to a single point.
Creation of m random mobile points (m=500) on [0,1] x [0,1].
Main loop (200 iterations). At each iteration, we compute the distance d between each mobile point (x,y) and each of his m-1 mobile neighbors and n fixed neighbors. A weight w is computed as a function of d, with a special weight for the point (x,y) itself. Then the updated (x,y) is the weighted sum aggregated over all points, and we do that for each point (x,y) at each iteration. The weight is such that the sum of weights over all points is always 1. In other words, we replace each point with a convex linear combination of all points.

Special features

If the weight for (x,y) [the point being updated] is very high at a given iteration, then (x,y) will barely move.
We have tested negative weights (especially for the point being updated) and we liked the results better. A delicate amount of negative weights also further prevents the system from collapsing and introduce a bit of chaos.
Occasionally, one point is replaced by a brand new, random point, rather than updated using the weighted sum of neighbors. We call this event a "birth". It happens for less than 1% of all point updates, and it happens more frequently at the beginning. Of course, you can play with these parameters.

In the source code, the birth process (for point $k) is simply encoded as:

if (rand()<0.1/(1+$iteration)) { # birth and death
$tmp_x[$k]=rand();
$tmp_y[$k]=rand();
$rebirth[$k]=1;
}

In the source code, in the inner loop over $k, the point ($x,$y) to be updated is referenced as point $k, that is, ($y, $y) = ($moving_x[$k], $moving_y[$k]). Also, in a loop over $l, one level deeper, ($p, $q) referenced as point $l, represents a neighboring point when computing the weighted average formula used to update ($x, $y). The distance d is computed using the function distance which accepts four arguments ($x, $y, $p, $q) and returns $weight, the weight w.

Click here to view source code.

Related articles

Finance Analysis by Empowering Spreadsheet with SQL Ability

2013-04-24T01:29:04.000Z

The spreadsheet can implement the visualized calculation to some extent, and the nontechnical people can perform some rather complex calculations without having to learn the SQL. However, as the core of SQL, the relational query is unable to be implemented through spreadsheet, which adds complexity to the apparently simple problems of multi-table join.

For example, the Finance department needs to calculate the salary, and the relevant data is stored in ”standard sheet”, ” Absence sheet”, and ” performance sheet”, as shown in the below figure:

If these three sheets can be joined, then you can compute it easily via the standardWages*(1+Evaluation-Absence/40)+Bouns, as shown below:

However, the common business spreadsheet software like Excel is usually quite inconvenient for such Union and Join actions. The manual data copying is error-prone, and it will be even more exhausting if the data volume is huge. Considering these factors, composing formula is a great method, for example, in D2, E2, and F2, respectively compose the 3 formulas:

=IFERROR(INDIRECT("'Absence'!"&ADDRESS(MATCH(A2,'Absence'!$A:$A,0),2)),0)

=IFERROR(INDIRECT("'Performance'!"&ADDRESS(MATCH(A2,'Performance'!$A:$A,0),2)),0)

=IFERROR(INDIRECT("'Performance'!"&ADDRESS(MATCH(A2,'Performance'!$A:$A,0),3)),0)

The above-mentioned formula requires the strong technical competence and rich user experience with spreadsheet. In fact, the qualified capable people would rather import the data to the database and use a simple statement of “relation query” to solve the problem, because this formula is hard to understand and error-prone.

Isn’t there any better method for non-technical users? Actually, we’ve got one – esCalc, an innovative desktop BI tool which is capable for the relation query. To join the Absence sheet with the standard sheet, simply use the Join function, as shown in the below figure:

Similarly, you are only allowed to perform the Join action for once for the Performance sheet. The ultimate result is the just what we have expected:

The “perform for once” even includes the formula to calculate the salary. In G2, just enter the formula for once and the formula will be automatically copied to G3, G4, and other cells sharing the common business sense. We call such cells Homocell.

The Join action is dependent on the homocell to some degree. The advantage of group table at multi-levels is to join the data correctly, even those data at various levels. Similarly, in the grouping table at multi-levels, the formula will be copied and pasted to the homocells. For example, the formula in the summary section will be copied and pasted to other summary section, and the data in the details section will not be affected. Therefore, for the huge amount of workload needing adjust before in the business spreadsheet software has been automated in esCalc, the smarter desktop BI tool.

The Cost of BI In Perspective

2013-04-21T13:55:10.000Z

_{The Cost of Starting Too Big… or Too Small}

After talking to business managers, executives and other stakeholders, you’ve determined that this BI solution you’re considering has the potential of serving 100 users. How would you then go about calculating your project costs? This is where things get tricky, and where most BI buyers fail to protect their wallets. Making the wrong decision here is far more significant than any decision you make on software licenses or even hardware.

Even if the development stage of your BI project goes without a hitch, getting a hundred users to use any kind of software, in any company, is a challenge that is not at all easier than any technical challenge you will encounter during the various stages of the project. You could easily find yourself spending tons of money on the development and deployment of a complicated 100 user solution, only to find that only 15 of them are actually using it.

So instead of your total cost per user being reduced due to the ‘volume-pricing’ model, you actually paid much more – because each one of these 15 users absorbs the cost of the 85 others who find it utterly useless, too difficult to use or completely misaligned with their business objectives. You'd be surprised how often this happens.

The obvious way of dealing with this common problem is to start off small (10-20 users), and expand as usage of the system grows (assuming it will). But when it comes to traditional business intelligence solutions, there’s a catch - deploying a solution for 10-20 users and deploying a solution for 100 users are utterly different tasks and require significant changes in solution architecture.

Following this path will save you some cost on the software licenses you did not purchase straight off. However, if demand for the solution grows inside the business, you will have to re-design your solution – which would probably end up costing more than it would have initially.

Tip: The correct way of dealing with this challenge is to seek a solution that scales without having to re-architect the solution as usage grows. Buying more software and upgrading hardware when the time comes is relatively easy and inexpensive, while rebuilding the entire solution from scratch every year or two costs way more.

Find more at http://www.sisense.com/prism

Hadoop+Ubuntu : The Big Fat Wedding.

2013-04-21T02:10:40.000Z

Now, here is a treat for all you Hadoop and Ubuntu lovers. Last month, Canonical, the organization behind the Ubuntu operating system, partnered with MapR, one of the Hadoop heavyweights, in an effort to make Hadoop available as an integrated part of Ubuntu through its repositories. The partnership announced that MapR's M3 Edition for Apache Hadoop will be packaged and made available for download as an integrated part of the Ubuntu operating system. Canonical and MapR are also working to develop a Juju Charm that can be used by OpenStack and other customers to easily deploy MapR into their environments.

The free MapR M3 Edition includes HBase, Pig, Hive, Mahout, Cascading, Sqoop, Flume and other Hadoop-related components for unlimited production use. MapR M3 will be bundled with Ubuntu 12.04 LTS and 12.10 via the Ubuntu Partner Archive. MapR also announced that the source code for the component packages of the MapR Distribution for Apache Hadoop is now publicly available on GitHub.

MapR is the only distribution that enables Linux applications and commands to access data directly in the cluster via the NFS interface that is available with all MapR Editions. The MapR M5 and M7 Editions for Apache Hadoop, which provide enterprise-grade features for HBase and Hadoop such as mirroring, snapshots, NFS HA and data placement control, will also be certified for Ubuntu.

Now, as you get Hadoop integrated natively with Ubuntu, it's a lot easier to install it and go. No more unnecessary downloads and wacky configuration steps. And the best part is the NFS interface available with MapR's distribution that enables other Linux commands and application to access the cluster data directly. The Ubuntu/MapR package will be available through the Ubuntu Partner Archive for 12.04 LTS and 12.10 releases of Ubuntu on the official website starting from April 25, 2013.

For more info you can get the Ubuntu and Hadoop: the perfect match white paper from here.

Take-aways from IE's first Predictive Analytics Summit in the Asia-Pacific

2013-04-21T01:00:00.000Z

Having asked for a budget for it with special approval, I, of course, would take whatever the Event offered in the short-but fruitful 2 days in Hong Kong during 18th to 19th April, 2013. My overall feedback is positive and will recommend companies to spend their training budget for Analytics people to come to here instead of staying in the classroom to learn Stat 101. This event meant to be for particioners

The Analytics market in Asia-Pacific

The market is divided into:

i. Data collector: those with different fancy technologies or incentive to capture your footprint online, on social network, on the toucscreen at the backseat of the cab. These people are moving fast in the retail sector. Nothing done with the B2B market.

ii. Data integrator: those link information from various databases in house, over the social network. There are media companies doing all these, but have problem in identifying the customers. However, financial industry are struggling more about managing their internal data as there are a sea of high quality data inside the organization.

iii. Analyst of the big data patterns: Algorithms are shown but what is key is that algorithm focuses on capturing two trends: the mass trend and the outliers. In Asia, the outliers or emerging trends are one of the key topics for Risk Analytics, not so for the Marketing or Operation side.

iv. Implenmentator of the findings: applications areas involves risk(Fraud, Credit Risk), B2C marketing(Identifying the profitable customers with the right resources), finance (forecasting materials for an online shop or pricing correctly for the auto industry), operational efficiency via workforce planning (in quantifying /finding out the key performers' characteristics).

v. Speeding up the overall cycles or spreading out the processes: There are IT guys working hard to give real-time output and spreading the iii. and iv. to as many people as possible.

What the participants are thinking?

From what I observed, there are three types: 1. the Start-up Analytics looking for or developing ideas/knowledge; 2. The successful camp which think they are the best and want to continue doing so; 3. The sales and consultants who came to pitch their products/services. Having a mix of these people would ensure the industry is alive and the conversation is healthy. Unfortunately, I do not feel that 3. the products and services of Predictive Analytics are doing good in the Region. You can say this is a gap or an opportunity.

What I am looking forward to?

There is a lack of development of Predictive Analytics in the B2B markets - If I have to see more potential for development in Asia-Pacific and if what GDP said was right, even in the US, over 50% of the business/revenues are generated in the B2B market, why so few development out there?

I have not been to other Region but I feel that it would be interesting to see more exchange like this type for the Professionals.

Disclaimer: As my sample size is bias towards the participants coming to the Hong Kong event, those centered around Singapore or Shanghai may not be within my reach.

Weekly Digest - April 22

2013-04-21T00:00:00.000Z

Sponsored Announcements

Invitation to the Pivotal launch - this Wednesday. A new platform for a new era.
Enterprise Apache Hadoop: Best Practices and Tools - Hortonworks Webinar
Calling all travel, hospitality and tourism technology professionals! Join us at The Smart Travel Technology Show this July 18-19 in Boston to meet 150-200 of the best IT and Tech minds in the space. We’re discussing data, cloud, wifi, payment efficiency, mobile, strategic IT concerns for travel and more. Plus! Meet leading travel brands such as Travelocity, Expedia, Boston Logan Airport, Los Angeles World Airports, Target Vacations, Visit California, Atlas Travel, Roomkey, Louvre Hotels and more.
Predictive Analytics World Chicago - June 2013. Hear how top practitioners deploy predictive modeling, and what kind of business impact it delivers. Will feature over 30 sessions with case studies across 2 tracks: 1) All Audiences and 2) Expert/Practitioner.

Staff Postings

Selected Postings from Members

Forum Questions

Previous digest | Recent jobs | Top Links

Should business and engineering schools develop joint programs?

2013-04-19T16:30:00.000Z

Businesses are increasingly using data-driven methods to make business decisions. Hence, there is a need for people with both good business skills and programming/quant skills. Finance/Accounting PhDs and other business PhDs do have such skills, but they are few in number, are costly to hire, and the majority anyway prefer academia. This limits businesses to mainly hire bachelors or masters level candidates.

However, a majority of the business students (MBAs) do not have sufficient quantitative and programming skills to be able to adequately do the quant part of the job. A single course teaching introductory statistics using basic Excel for analysis is nowhere near adequate, as far as quant skills go. It is difficult for a quant neophyte to do the quant analysis, understand the results from the analysis and use it effectively to make business decisions.

On the other hand, most folks with programming experience do not have the expertise to adequately do the business part of the job. While they may be able to do a great job of programming, they often find it difficult to understand the business side of things, what is the business decision being analyzed and what analysis would be appropriate to address these issues, and how to leverage the results of the quant analysis to make good business decisions.

To address this gap, should business and engineering schools come together and develop a joint program, where the graduates have both sets of skills? Some schools may already be doing this.

The amateur data scientist and her projects

2013-04-17T23:00:00.000Z

With so much data available for free everywhere, and so many open tools, I would expect to see the emergence of a new kind of analytic practitioners: the amateur data scientist.

Just like the amateur astronomer, the amateur data scientist will significantly contribute to the art and science, and will eventually solve mysteries. Could the Boston bomber be found thanks to thousands of amateurs analyzing publicly available data (images, videos, tweets etc.) with open source tools? After all, amateur astronomers have been able to detect exoplanets and much more.

Also, just like the amateur astronomer only needs one expensive tool (a good telescope with data recording capabilities), the amateur data scientist only needs one expensive tool (a good laptop and possibly subscription to some cloud storage/computing services).

Amateur data scientists might earn money from winning Kaggle contests, working on problems such as identifying a Bonet, explaining the stock market flash crash, defeating Google page ranking algorithms (contact me regarding this upcoming, paid project, helping find new complex molecules to fight cancer (analytical chemistry), predicting solar flares and their intensity. Interested in becoming an amateur data scientist? Here's a first project for you, to get started:

First project: do large meteors cause multiple small craters or a big one?

If meteors usually break up into multiple fragments, or approach the solar system already broken down into several pieces, they might be less dangerous than if they hit with a single, huge punch. That's the idea, although I'm not sure if this assumption is correct. Even if the opposite is true, it still worth asking the question about frequency of binary impact craters.

About to hit Earth

Eventually, knowing that meteorites arrive in pieces rather than intact, could change government policies and priorities, and maybe stop spending money on projects to detect and blow up meteors (or the other way around).

So how would you go about estimating the chance that a large meteor (hitting Earth) creates multiple small impacts. And how many impacts on average: 2, or 3? An idea consists in looking at Moon craters and check how many of them are aligned. Yet what causes meteors to explode before hitting (an thus create multiple craters) is Earth's thick atmosphere. Thus Moon would not provide good data. Yet Earth's crust is so geologically active that all crater traces disappear after a few million years. Maybe Venus would be a good source of data? Nope, even worst than Earth. Maybe Mars? Nope, just like moon. Maybe some moons from Jupiter or Saturn would be great candidates.

Double impact, seen million years later

Once a data source is identified and the questions answered, deeper questions can be asked, such as, when we see a binary crater (two craters, same meteor), what is the average distance between the two craters. This will also help better assess population risks, and how many billion dollars should NASA spend on meteor tracking programs.

In any case, as a starter, I did a bit of research and found the following data, with a map showing impact craters on Earth.

Visually, with the naked eye, it looks like multiple impacts (e.g. binary craters), and crater alignment, is the norm, not the exception. But the brain can be very lousy at detecting probabilities. So a statistical analysis is needed. Note that the first step consists in processing the image to detect craters and extract coordinates, using some software or writing your own code. But this is still something a good amateur data scientist could do, I'm sure you can find the right tools on the Internet.

A better idea might be to use some public data I published a while back: it is more comprehensive, already in Excel format, and also has dates attached to craters, so that binary craters created on the same year are likely to be true twins (that is, coming from a single piece of space rock that broke into two fragments sometimes in the past, before hitting Earth).

Google 'binary craters on Earth' for additional info.

Related articles

Is Open Source Really Free?

2013-04-17T08:00:00.000Z

Open Source and the Cost of Software Licenses

People often think that the answer to this question lies in software costs, but in fact software costs are usually the red herring in the process of business intelligence costing.

It is obvious that the more users your solution has the more software licenses are going to cost. Therefore, you might be tempted to choose a vendor that sells software for 30% less than another vendor – but basing a decision solely on this is a big mistake as license costs have little bearing on the total cost of a BI solution, and hardly any impact on ROI.

Some proof to this can be found in open source. Open source BI provides (by definition) free software, and there is no shortage of open source BI tools/platforms. However, none of them are doing as well as the established non-open source vendors, even though they have been around since the beginning of the century. They’re having trouble acquiring customers, at least compared to commercial vendors. It is very easy to assume that if software costs were significant inhibitors in the BI space, open source solutions would be much more prominent than they actually are.

Another hint at this can be found in the ‘commercial’ (non-open source) world, where BI vendors do charge for licenses but will usually provide significant discounts on purchasing of large volumes of licenses. BI vendors do it for reasons that go beyond the obvious attempt to motivate potential buyers to expand their purchase orders. They do it because they realize the total cost of the solution – to the customer – grows significantly as the number of users grows, regardless of license costs (preparation projects, IT personnel assignment, etc). They need to take this into account when they price their software.

For more info visit http://www.sisense.com/product

How do you solve your "Pre-Etl" Source to target mapping problems?

2013-04-15T20:30:00.000Z

It's one of the integration problems that most of the big palyers in the industry have pretty much left untouched, Anyone working in the data integration / data warehousing industy understands that when you build a data warehouse, you have to create these complex pre-ETL source mappings before the ETL developers start work. The way most organizations do this is with spreadsheets. Every organization has an exorbitant amount of spreadsheets that they use to document this stuff. Once they've handed themoff to the ETL developers, they're never maintained.

Using spreadsheets is definitively non-authoritative: source mappings change as you design and test your ETL jobs. A spreadsheet that once functioned as the single or authoritative catalog of all source mappings might not get updated -- or (just as likely) might get updated with incorrect or incomplete information -- as the ETL design process evolves.