clario Analytics was founded back in 2002 largely by folks from Fingerhut. The team had been working on mailstream optimization – how to manage catalogs. The best customers of a catalog marketer can get literally 100 catalogs per year and this is not good. Initially a consulting company they raised money in 2006 and launched clario Stream, a product for developing customized/personalized outbound contact streams – contact optimization or the evolution of mailstream optimization. Now they have launched their generic data mining and analytics platform, clario.

clario was originally designed as an interface for internal analysts to work on the models in clario Stream. Like most analysts the clario Analytics analysts spent 60-70% of their time on data integration / manipulation / quality issues so this was an initial focus. With these data handling capabilities and some statistical / modeling nodes added it became clear it was usable as a standalone tool leading to the launch this year.

clario is a web-based, cloud-backed data mining and analytics tool. It is a general purpose tool with a node-driven workbench (allowing you to assemble the model development process) that is intended for analysts to build models. The user of clario works through the web browser (the front-end is in Adobe Flex) and connects to an application server that controls access/state. Secure FTP is used to put data in the cloud and analysts can then develop workflows to process the data. All the heavy duty work is done on the amazon.com compute cloud using standard capabilities, though clario has developed the dynamic scaling needed to manage multiple execution engines in support of data mining tasks.

The cloud based approach has a couple of advantages. First it allows for better collaboration –letting outsourced analysts and worldwide clients share an analytic workspace. Second it takes advantage of the cloud to allow analyst teams to have a lot of computing power when they are building a model (typically a very compute-intensive exercise) without paying for it while they are thinking about the model.

clario processes flat files and nodes include a variety of data manipulation nodes (sort, transform, join, filter, score, rank, append, aggregate), statistics (univariate, sample, bivariate, logistic, linear, factor) and best practices (kind of macro nodes for things like detecting outliers, handling missing values, eliminating unhelpful data elements etc). Once the model is done clario either outputs CSV files (scored data) or displays visual results. Modelers can export to pseudo code or to Excel. Right now they haven’t decided on a model output format – personally I hope they will support PMML soon.

Pricing is cool – $300/month to get 100GB of data transfer and 50GB of secure storage with 720 hours of processing. Typically users run 4-6 workflows simultaneously during the workday and still stay below the total. They also have a 30-day free trial where you can “kick the tires”.

The product is still fairly new and their immediate plans are to add decision tree (CART, CHAID and MARS) and clustering (EM and k-means) nodes over the next 6-9 months. Usability improvements around the user interface to make it easier to edit existing workflows are planned as are scheduling and workflow templates as well as some new collaboration tools. They are also taking advantage of being cloud-based to add some web services nodes too like the USPS address cleaning service.

The cloud seems like a perfect fit for data mining, with its ability to deliver lots of compute power when you need it without requiring you to own those computers. I think data mining in the cloud is going to be a hot space…

Syndicated from ebizQ

I have recently been a reviewer for Michal Bali’s new book on Drools (Drools JBoss Rules 5.0 Developer’s Guide), which will soon be generally available. In the meantime you could check out an example of the style by reading this two part article on Drools JBoss Rules 5.0 Flow – Part 1 and Part 2. Michal’s book is intended for developers getting up to speed on Drools.

Now if you are looking for a business friendly introduction to Drools you should definitely check out Paul Browne’s book JBoss Drools Business Rules. This an excellent and very approachable book. There’s an article extracted from the book on testing here.

I plan to write longer reviews of both books over the summer but thought I would give them a quick plug now.

Of course, if you want to understand how to use rules in the broader context of Decision Management you should read Smart (Enough) Systems, the book I wrote with Neil Raden.

Syndicated from BeyeNetwork

John Elder of Elder Research is well known in data mining circles and speaks/teaches regularly. Not only has John recently released a new book (Handbook of Statistical Analysis and Data Mining Applications), he has now released his great seminar on the top 10 mistakes in data mining on YouTube! Highly recommended for anyone working with data mining (or data miners)
Check out:
Part 1: Top 10 data mining mistakes
Part 2: Don’t rely on only one technique
Part 3: Don’t extrapolate
Part 4: The path to data mining success

Optimization is one of the core technologies for Decision Management and an increasingly important one as the technology grows to handle more operational problems. Dynadec is one of the new players in this space and has what it regards as a game-changing optimization platform (Comet). Their intent is not simply to provide a new optimization tool but to drive value for large companies in vehicle routing, workforce management and resource scheduling. Keeping to these focus areas allows them to build more complete solutions, though the underlying technology has other potential uses. Dynadec sees an opportunity (and I agree with them) because the economic environment means that companies are trying to drive out costs and better utilize assets. Add in the massive growth in RFID/sensors/GPS and real-time data and there is an opportunity to use this data to solve some complex problems. Dynadec believes they can help companies make decisions they could not make before – quickly and under uncertainty.

Dynadec’s Comet supports what can be thought of as the whole spectrum of optimization – from mathematical programming, to constraint programming, constraint based local search and now what they call “dynamic stochastic combinatorial optimization.” This last is unique to them and combines predictive models and optimization so it can adapt model-based on predictions of the future. Comet supports these four kinds of optimization and allows access to/from the standard enterprise platform elements – databases for historical data, SOA, C++ etc. Built on this platform are the three solutions for vehicle routing, workforce management and resource scheduling. Dynadec provides hosted or on-site solutions as well as embedding the engine into ERP systems or providing it as a black box service. As time passes they are increasingly packaging these applications.

Today they have two broad categories of customers. Some use a proof of value pilot approach where Dynadec spend time working with them to show how it works. This is aimed at companies without the optimization expertise to use the platform and the end result is that Dynadec hands over a solution. Others work on an application development pilot where the customer’s optimization staff is trained on Comet and Dynadec focuses on skills transfer and helping with integration. These customers end up licensing the technology as a development platform.

Several demos followed:

• Vehicle routing – their example was for an oil company where people move from shore to platforms, between platforms and back to shore. Objective is to minimize the number of helicopters and it is complex because the helicopters are close to their range limit, must therefore be refueled, and people on multiple rigs must be scheduled. Dynadec did the pilot in about 3-4 weeks and the schedule was getting calculated in about 20 minutes – much faster – while reducing the amount of helicopter time required.
• Hot Strip Mill scheduling was the second one. Change over time was a problem and the mill had lots of constraints – wear, oxidation, grouping etc. This was an experienced optimization user that had hit the wall on their use of the existing technology. Using Dynadec they were able to satisfy all their constraints (something their existing solution could not achieve), to improve urgency and transition costs significantly.  All of this was done in under two minutes.
• Rostering. Another sophisticated user of mathematical programming and other optimization technology. Hospital scheduling is tricky with different skills in the staff pool and complexity in assigning them to different roles in different time slots. This hospital had too many constraints and was just trying to find a solution. It took Dynadec 45 seconds to solve the problem more effectively.
• Last example is a company that needs to make a decision every 2 minutes around customers in a region who need to be visited. They have no advance knowledge of where customers are, when they will need service or what they will need. Essentially customers get added to the list of those who must be visited when they call in. The company can accept customers or reject them when they call but once accepted they must be served. This is an exercise both in continuous optimization – after every call – and in the integration of predictive modeling. Dynadec improves the rate of coverage by integrating their optimization with a model predicting the likelihood of a particular region having a customer at a particular time. Afterwards they have all the data and can compare how the engine would have optimized with perfect data with what it actually recommended. In other words you can assess how well you would have done if you had perfect advanced knowledge. Dynadec is using some machine learning algorithms to automate this feedback loop.

Dynadec’s technology is targeted at an interesting subset of the optimization / operational decisioning market. If the problem is too small then customers don’t care about optimizing it (or people can do the optimization). If the problem is very large then customers don’t care because the problem “grays out” – the average works well enough all the time. In between is the sweet spot – lots of data but not so much that blurs from one day to another. As an example Saturday delivery for a delivery company would be in the sweet spot while a weekday might not be. Air taxis might be another example or restoring the power grid after problems by dispatching skilled techs.

One of the features I most liked about Dynadec’s solution is that it attempts to be robust in the face of uncertainty by integrating predictive analytics and optimization. Their ability to get to a good answer (if not the best) quickly and the potential for using the solution in new classes of operational decisions becomes clear. This is critical as optimization technology typically iterates multiple times looking for the best answer. In operational decisions, though, you want the best answer you can get quickly so an optimization engine that gets most of the way to the answer early in its iterations is more useful. For example in one large scale vehicle routing problem – 90 vehicles and a thousand customers – they could find an optimal solution in an hour but could drop the total travel time almost to the optimal level after just 5 minutes. All this means that even high-volume operational decisions can be optimized one at a time.

Syndicated from ebizQ

Mike Gualtieri is always interesting over at the Forrester Blog For Application Development & Program Management Professionals. This week he has a post called Do Application Developers Need To Change Their Ways? In the post he asks developers to look at the person in the mirror (he’s been listening to Michael Jackson’s song “Man in the Mirror“) and suggests four ways for them to change.

If you read this blog regularly you will know that I often spend time trying to get application developers to adopt business rules and to focus on how business rules technology and a decision management approach can help develop smarter, simpler, more agile processes and systems. With this in mind I thought I would comment on how my recommendations would support Mike’s four changes:

• Understand the business in your bones
  First, understand in particular the business decisions that matter to the systems you are building. These decisions are critical flex points in the system because they rely on both business know-how and external regulation and because decision making changes constantly, making decisions a critical point for agility. Secondly make sure to adopt business rules and a business rules management system (BRMS) because that will let you bring the business folks who already understand the business into the development process. If the business is writing rules that you and they can both read, the system will be more accurate and you will gain understanding.
• Be a developer not a coder
  The use of business rules, a higher level abstraction for business logic, is a key enabler for being a developer not a coder. Writing atomic, declarative business rules instead of wads of code reduces the time to market for your ideas and reduces the maintenance they will need later. Business rules and other decision management technologies let you develop the logic your business needs.
• Use new technologies, but only when they make a difference
  And use old technologies that you have been neglecting, especially if the performance or reliability of those technologies was a reason for neglecting them. Business rules have been around a while. Many developers saw expert systems years ago and rejected them for perfectly valid reasons at the time. But times change, technologies evolve and business rules have become a robust, proven, reliable technology for managing the decision making logic in your systems.
  Meanwhile analytics and optimization technologies are ready for use even in transactional and real-time systems. And while the algorithms are not always new, the performance and scalability of modern implementations is very different.
• Become architects again
  Focus on the kinds of things your systems must do – execute processes, make decisions, manage information – and use the right technology and approach for each aspect. SOA makes it reasonable to use the right tool/approach for each element and then assemble a composite application. Take advantage of this to use the right kinds of technology for each kind of service. And Decision Services need to be built on declarative, atomic business rules and decision models.

Another great post from Mike. Adopting decision management technologies like business rules are not the only change application developers need to make but they are one such change.

If you work at a Financial Services organization there is a great new source of free information. Chris Pratt, formerly of Tower Group and Financial Insights has started her own firm – Pratt Associates. She covers credit solutions, scoring/analytics, core banking and BPM solutions. Chris really knows the space and understands how banks work and what they need from software and analytic solutions.

Her research is free to those who work at Financial Services Institutions and I would encourage those of you who do to contact her through her website so you can receive her Quarterly Intelligence Report – a nice report centered around market intelligence for providers, such as new deal announcements, management and strategy changes.

And if you work at an FSI vendor you should seriously consider contacting her for independent analyst services. Chris was always great to work with when I was briefing her as a vendor and I am confident she would give you great advice.

Syndicated from BeyeNetwork

The Evidence Based Management blog had a post on Why experts are so often wrong that discusses a book by Philip Tetlock (Expert Political Judgment: How Good Is It? How Can We Know?)

In a world filled with expert predictions that are mostly incorrect, and filled with people who eagerly seek such predictions even though they are incorrect, Tetlock’s book explores why experts are so often wrong and why we listen to them anyway. There is no more evidence-based subject matter than forecasting. This book provides an excellent overview of the perils and pitfalls in making forecasts.

But if we can’t rely on experts to synthesize information and pass on judgments to us, can we make our own? Perhaps not, according to another post, this time from the Institute for the Future. In So much information, such limited ability to understand it all Vivian Distler quotes Stephan Dahl:

we make the assumption that they will be able to keep up with and synthesize the abundance of information that may be relevant to their health. 21% of adult Americans have only rudimentary skills, leaving them unable to extract even simple information from printed material. A further 25% can perform simple reading functions but “cannot integrate or synthesize several facts” from documents.

and she goes on to ask

Information will have to be made accessible and understandable.  Are we ready to take on that obligation?

Personally I don’t think it is the information that needs to be made accessible and understandable, but the decisions that must be made with that information.We should not allow experts to make judgments without process (as I discussed here) and we cannot rely on consumers (or front line staff) to have the necessary analytic skills. Instead we should focus our experts on understanding how we might make a good decision and build that expertise (those rules) into a decison making system that also allows those impacted to add constraints or additional rules. These rules could take advantage of sophisticated analytics without just handing over decision-making power to the analytic models.

The end result could and should be a transparent system that makes decisions consistently based on the data and on robust analysis of that data using rules that come from regulations, policies, expertise and personal preference. An action support system not just a decision support system. Just a thought…

This week’s event calendar is below. The intent of this weekly post is to focus on web events coming in the next few weeks and conferences in the coming months. If you know about web or physical events around business rules, analytics, optimization or decision management, please let me know – james@decisionmanagementsolutions.com.

Web Events:

• Silverlink: Applying Analytics to Drive Behavior
  Series – July 9 through July 28
  Join best-selling authors and industry leaders for this engaging, four-part webinar series. You’ll hear fresh insights and proven methodologies about how to engage and drive healthcare consumers. Experts will describe how applied analytics enable healthcare enterprises to motivate behavior, improve health outcomes and drive member loyalty.
  Register
• FICO: New Advances in Blaze Advisor for .NET
  July 8, 9:00am ET AND 12:00pm ET
  Version 6.7 of FICO™ Blaze Advisor® is now available for .NET users. Find out what business rule system enhancements have been made and how they can benefit you and your business users.
  Register

Physical Events:

• Gartner BPM Summit - October 5-7, Orlando FL
  I am speaking on Tuesday morning
• Predictive Analytics World – October 19-21, Washington DC
  I am giving a tutorial on the 19th and speaking
• Information on Demand 2009 – October 25-26, Las Vegas NV
• Business Rules Forum/Decision Management Summit – November 1-5, Las Vegas NV
  I am giving a tutorial and keynote

Syndicated from ebizQ

Jeff Attwood had a great post over on Coding Horror – All Abstractions Are Failed Abstractions in which he discussed a Joel Spolsky article in which that states

All non-trivial abstractions, to some degree, are leaky.

At some level, of course, this is true and Jeff goes on to say

But I’d also argue that virtually all good programming abstractions are failed abstractions. I don’t think I’ve ever used one that didn’t leak like a sieve. But I think that’s an awfully architecture astronaut way of looking at things. Instead, let’s ask ourselves a more pragmatic question:

Does this abstraction make our code at least a little easier to write? To understand? To troubleshoot? Are we better off with this abstraction than we were without it?

And that, of course, is the critical question. And that brings me to business rules. Business rules, especially when business users are brought into the picture to collaborate on their creation and maintenance, are an abstraction for the hard-code logic that will be executed. And they almost certainly meet this definition of a failed, leaky abstraction. Many programmers reject the use of business rules because of this.

When a programming blog linked to my article on using business rules rather than soft-coding lots of programmers responded. Underlying the reasons they gave for rejecting business rules (which I discussed in a series of posts – 1,2,3) was a sense that business rules are an unnecessary and failed/leaky abstraction. But if you ask the question Jeff asks – does it make the business logic easier to write, to understand and to trouble shoot – then business rules are, as I argued, a compelling abstraction. In particular:

• Business rules use fewer words, fewer lines of “code” to describe complex logic than procedural languages do
• Business rules are easier for those who understand the business to understand and thus to validate (and who else is really in a position to tell if you if the logic is what the business needs)
• Because they are atomic and independent, business rules either fire or don’t fire and so troubleshooting is easier.

So are business rules a “failed” and “leaky” abstraction? Probably. Should you use them anyway? Yup.

Syndicated from BeyeNetwork

In 2005, Mr Ahmadinejad got 17 million votes and in 2009 he got 24 million.
The question is, where did all those extra votes come from?
The answer, according to this study, is not at all clear.

I don’t write political or personal posts and, despite first appearances, this is not one either. When I saw the BBC News post from which the quote above is taken (Iran: Where did all the votes come from?) I was inspired to blog not so much by the specifics of the situation as by the process followed by the folks who investigated the situation. They took a result, one in dispute, but then looked past the simple facts to see how likely the result was to be reasonable and a truthful representation of the voters’ intent.

For instance they went beyond the facts that the vote percentage for Mr Ahmadinejad only rose by 1% and that the poll some weeks before the election also showed him winning. They drilled in to ask questions like “how many more votes does this 1% swing represent” and “are the regional variations the same or similar in the two elections” and “how would voting patterns have to have changed to generate this result”. All these questions, and the statistical analysis that backs them, result in interesting conclusions.

But, like I said, this is not a political post about the election in Iran. What I want to ask you is how often you do this kind of analysis when someone presents a conclusion? How often is the data that has been used to base decisions in your company put through this kind of analysis? Is anyone asking the hard questions about the data that drives your company?