But that wasn’t the number one conversation.  The one topic that everyone was interested in – and I talked to nearly 100 customers and partners over four days – was creating new reports faster, or Agile BI. 

I’m not talking about agile as in scrum, but agile as in greater business agility. A little over a year ago, Forrester found that 66% of BI requirements change each month, but 77% of respondents said it takes days to months to get their requests fulfilled. Most of the people I talked to at the conference said the majority of new reports required data outside the warehouse and typical turnaround times were one month or more.

To some, one month may seem OK, but not to most in the business. The business wants the results in days.  For eHarmony, that often meant in one day.  If you think one month is OK, ask yourself why there are so many ‘spreadmarts’ (spreadsheets and data marts) created by the business. It’s because the way you increase profitability or add new revenue streams is with new insights, not the old ones. If you wait a month, you lose a month of benefits or worse, the entire opportunity.

So why does it take so long?  If you look at BI as a business process, there are two reasons.  The first is that business and IT have to talk. A business  analyst has to explain to a developer what they need and that can often take weeks.  The second reason is that the BI team often takes all their data from the data warehouse. If new data is needed that is not in the warehouse, it may take a month or more to add that data, to change and test the warehouse.  So IT operates in batch.  They gather many requirements and make many changes at once in the warehouse to be efficient.  That’s efficient for IT, but not necessarily what the business needs.

The good news is that companies are now creating reports in days, not months.  How did they do it?  They eliminated the conversation with IT and accessed data directly from the sources.  IT analysts at companies like HealthNow are using a Web-based tool called Informatica Analyst to directly access, profile, transform and cleanse data live. A BI developer adds a report on top of the view, which looks just like it came from a data warehouse – except it didn’t. It’s a live or federated query across many source systems and the warehouse.  This is called Data Virtualization. It’s implemented with Informatica Data Services, which along with Informatica Data Explorer, Informatica Analyst and Informatica PowerCenter are all an integral part of the Informatica Platform, and part of the new PowerCenter Data Virtualization Edition. So the business gets their report in days.  A developer can take this work, captured in metadata and decide whether to keep it virtual or move the data into the warehouse as an ETL movement with PowerCenter. The report doesn’t change.

So, have that conversation. If you’re in the data warehousing team, go ask your BI team, your analysts, whether they need to get new reports in days, rather than months. Go find the people with spreadmarts and ask why they created them.  Show them they could get reports in days and you’ll hear what I heard from nearly 100 people, that it’s critical to the business and it’s time for Agile BI.

A recent article by Chris Matyszczyk of CNET highlights the often conflicting and confusing nature of monitoring social media.  A 26-year old British citizen, Leigh Van Bryan, gearing up for a holiday of partying in Los Angeles, California (USA), tweeted in British slang his intention to have a good time:  “Free this week, for quick gossip/prep before I go and destroy America.” Since I’m not too far removed the culture of youth, I did take this to mean partying, cutting loose, having a good time (and other not-so-current definitions.)

The US Department of Homeland Security (DHS) thought otherwise (from Matyszczyk’s article):

Many Brits will feel confident in explaining that the use of the word “destroy” here refers to, well, partying. Brits like to do this. They are good at it. They enjoy exporting it.

Leigh Van Bryan's Twitter profile picture. (Credit: Screenshot: Chris Matyszczyk/CNET)

Sadly, the Department of Homeland Security agents in Los Angeles were not up to the vernacular. Even more sadly, they appear not to have aficionados of “Family Guy”, as Van Bryan had also tweeted: “’3 weeks today, we’re totally in LA p****** people off on Hollywood Blvd and diggin’ Marilyn Monroe up!”

Perhaps if the authorities had chosen to google that line, they might have felt more at ease. Perhaps, though, they would have received some strange results from Google+.

It seems sure that they didn’t get this American cultural reference as they reportedly checked Van Bryan’s luggage for shovels.

This story does not end happily, as Van Bryan and his friend Emily Bunting were arrested and then sent back to Blighty.

Bad luck for Van Bryan, and bad press for DHS.  This highlights that context is as important in social media monitoring as sentiment.  Context is not just about variations in speech patterns, age-specific slang, industry jargon, or use of emoticons (“<3” anyone?).  Context is how the dialog around a word or statement can shed light on its meaning.  Context is using interrelated (correlated) events to give you a deeper understanding into the meaning and intent.  This, ultimately, will help you decide how to respond appropriately with the most benefit.

Let’s put this in the context of enterprises and how you can get additional context in your social media monitoring programs.  Imagine someone tweeting bad things about your company’s service.  What do you know about that person?  Is she a current customer?  How much has she spent with your firm in the past 90 days?  Has she logged any customer complaints with your customer service desk recently?  Is she just complaining for the sake of complaining (as some do on the Internet), or does she have a valid customer issue?  Understanding this context by correlating her tweets with internal enterprise data about her customer history can better equip your social media service representatives to diffuse the situation, fix the issue, and, hopefully, turn a critic into a raving (very positive) fan.

Informatica RulePoint complex event processing, in conjunction with the Informatica data integration platform and single view of customer Master Data Management (MDM), can give you that picture, that context, as social media mentions are being published in real-time.

Providing this contextual analysis to your social media monitoring initiatives will not only give you an advantage in your markets, but also make sure your social media efforts “don’t go all Pete Tong”.

The first approach that comes to mind is architecture by exception. This is where you only solve part of the problem by defining the problem scope very narrowly. For example, to address the need to get sales data into the data warehouse, you could say that you ONLY need sales data and proceed to architect a solution that meets that specific need; in other words you treat it as an exception that is not related to other data integration needs. The end result of this approach is well known – the integration hairball – which is filled with waste and inefficiency. So the first characteristic of a lean architecture is that it must be holistic.

A second non-lean approach is architecture by PowerPoint. This is where the focus of architecture activities is to produce nice-looking architecture artifacts like Visio models and standards documents. Don’t get me wrong – models and standards are extremely useful tools, but if they are static snapshots that are custom crafted by individuals which soon collect dust on a shelf. Architectures are dynamic and evolving, so they need to be represented as structured data in metadata or EA repositories with appropriate role-based user interfaces to allow different stakeholders to see the view that is relevant. The second characteristic of a lean architecture therefore is automation of architecture artifact rendering.

A third approach is architecture by committee. Consensus and collective intelligence of a group of people can be good, but it can also result in groupthink which Wikipedia defines as “The mode of thinking that happens when the desire for harmony in a decision-making group overrides a realistic appraisal of alternatives. Group members try to minimize conflict and reach a consensus decision without critical evaluation of alternative ideas or viewpoints.” The result of consensus is often an overly complex solution that satisfies everyone’s interests, but as a result ends up being a heavy-weight solution that performs poorly, is expensive to maintain and contains functionality that is never used. The third characteristic of lean architecture therefore is simplicity – generally developed by only one architect. In the world of lean this roles is satisfied by the Chief Engineer which brings together technical, functional, business and operational skills into one person. These people are hard to find, but not impossible once you start looking.

A fourth approach is architecture by tribal knowledge. Basically, this is where teachings in the form of anecdotes, personal experiences and rules of thumb are handed down to junior staff from experienced architects. This is useful in part, but it is not systematic and repeatable and therefore two architects can easily end up with totally different architectures because their personal experiences (tribal knowledge) are different. The fourth characteristic of a lean architecture therefore is the use of systematic (scientific, data-based) methods such as Design Structure Matrix[1] which results in a mutually exclusive and comprehensive collection of architecture components which represent the whole enterprise.

All of these (and other) non-lean approaches to architecture end up creating a huge amount of waste – software functions that aren’t used, tables with no transactions, reports that no one reads and data that is stale and not adding any value.  All this waste clogs up application systems, servers and networks; consumes IT budgets; distracts management and staff; and acts as a drag on transforming the business to remain competitive over time.

Informatica may not have ALL the IT architecture answers, but when it comes to enterprise information management, the company is a global leader and has a collection of industry best practices.  Read more at Informatica Best Practices. Or if you want to learn more in person, why not attend Informatica World in May 2012 www.informaticaworld.com.

Social networking usage has increased dramatically over the last decade and social network accounts are now commonplace in millions of homes.  By harnessing and integrating social media data into CRM, marketing and other business applications, you can gain real time insight into new product and promotional offers and help identify the right customers for your business.  But leveraging social media data requires integration, and that integration can be lengthy and complex. IT organizations need to be able to access and integrate social media data for their big data projects quickly without additional IT infrastructure or resources to help turn big data integration projects into big opportunities for the business.

When we  introduced Informatica 9.1, we enabled businesses to tap into the social media data stream and integrate it with data from mainframes, databases, appliances, applications and even Hadoop. Social media has become a very important source of feedback for businesses and Informatica equips you with the ability to leverage the data it produces, increase the value it brings and maximize your return on data.

Social media is here and it impacts every business from big global conglomerates to the corner mom and pop shop.

Is your company embracing the challenges social media brings forth or ignoring it and hoping for the best?

But another interesting coincidence is that there are two aspects to our customers’ desire to “fix” their data problems. One is their perception of the solution, and the other is the reality of the root causes.

More to the point, the way a project is envisioned from the customer’s perspective is driven by an expectation of the solution. An example might be “guidance for designing and implementing a master data management program.” Multiple applications refer to the same data concepts but don’t talk to each other; instituting MDM will make the business processes talk to each other, therefore the project is envisioned as an MDM activity.

Once we start to peel the onion we begin to see things a little bit differently. For example, the business processes are impeded because they do not talk to each other. Implementing MDM might bring the data together into a single location, but it won’t necessarily change the communications between processes. Similarly, other root causes of issues that lead one into drawing conclusions about a potential solution need to be both identified and evaluated before moving forward. As part of that, I have started to look at the issues that originally motivate the presumption of a solution, and, more interestingly, ways to categorize their root causes.

One example of the benefits that this high performance architecture provides to capital markets firms is message delivery load balancing that is an inherent function of Ultra Messaging’s queuing capabilities. Load balancing enables efficient scalability as message queue instances persist messages from sending applications and pass them on to multiple instances of a receiving application according to load balancing configurations such as round robin or least busy. Each application instance processes a set of messages in parallel with other instances, and additional instances can be easily added as business volumes increase. An excellent example of the need for this capability can be found in financial exchanges.

One key requirement for scalability within an exchange is to enable trading on any number of financial instrument symbols. Scalability in this area requires that each symbol be assigned a matching engine process to manage the order book for it. The order books for many symbols may be managed by a single multi-threaded matching engine, and many matching engine processes may be running on a given server. The union of all order books/matching engines across multiple servers makes up the total current trading volume at that particular exchange.

The key to scaling therefore is the ability of an exchange to seamlessly move a symbol across matching processes or machines as capacity needs dictate. As an exchange trades more symbols, more matching engine memory is needed to store the order books for these symbols. However, performance may be hindered if too many symbols share the same matching engine server because of contention for its CPUs. Therefore, exchanges clearly need the flexibility to allocate symbols to matching engine servers and CPU cores dynamically in response to changing trading rates and as new symbols are traded.

The use of Ultra Messaging allows design of matching engines to be independent of their physical machine locations, thus enabling the horizontal scaling and load balancing described above.  The scaling of symbols per matching engine processes and machines is managed via the Ultra Messaging queuing process, and any required load balancing changes are transparent to the applications.

To learn more about how Informatica helps exchanges compete more effectively, please read the  whitepaper titled “Ultra Messaging Benefits for Financial Exchanges”.

Learn more about CEP in Scott’s first video: http://www.youtube.com/watch?v=AUmveP07Ea8.

Fast forward to 2012.  Reports aren’t sent out on paper anymore due to ubiquitous networks, BI tools and desktop PCs, but it seems that many IT organizations still have not addressed the 30-year-old challenge; how do you know when business users no longer need a given report?  As a result we see enterprise data warehouses with exponentially growing multi-terabyte or petabyte capacities. The amount of IT budget dollars being consumed for data that no-one uses is staggering. It’s not just storage for the data warehouse, it is also the network and server resources to extract, transform and move the data. Experts that frequently work on DW clean-up projects say that 60%-70% waste is not uncommon. As per this article by Bill Inmon, unused data can be as high as 99%!

It doesn’t have to be this way.  There are in fact simple, and cost-effective, ways to monitor data usage in a data warehouse as Claudia Chandra recently wrote about in Optimize Data Warehouses with Data Usage Monitoring and Data Warehouse Archiving. If no-one is using the data, simply stop producing it, or at a minimum remove it from expensive DW infrastructure and put the information in a highly compressed form on low-cost storage.

By applying simple measurement tools along with a few lean practices, you can achieve a Lean Data Warehouse.  For example:

• Respond rapidly to new business analytics needs with lean and agile data self-service profiling and mapping tools for business analysts.
• Analyze usage of production data warehouse to determine what end users are actually querying and systematically remove data or reports that are no longer of use.
• Keep aggregates and frequently accessed transaction data in the data warehouse.  Purge data and eliminate loading information that is not being used at all and that has no value for compliance purposes. 
• For infrequently used data that has some long term analytic value, or where retention is required for compliance purposes, move it to lower cost storage.

This is not rocket science – it just needs a bit of focus and discipline and the financial payback is worthwhile!  To learn more, check out Lean Data Management. Or better yet, come to Informatica World 2012 in May.  Check it out at www.informaticaworld.com.

In his blog Top 10 Business Intelligence Predictions For 2012, Boris Evelson of Forrester Research, Inc., states that traditional BI approaches often fall short for the two following reasons (among many others):

• BI hasn’t fully empowered information workers, who still largely depend on IT
• BI platforms, tools and applications aren’t agile enough

Data virtualization is an agile data integration approach that architects are adopting to provide fast and direct access to new critical data and reports the business needs and trusts. However, with Data Virtualization solutions not created equal, here is a list of criteria typically requested by architects.

As a leader in data virtualization, Informatica takes its responsibility to offer the highest quality of thought-leadership, very seriously. In 2011, Informatica hosted a series of interactive webinars with industry architects on architecture and best practices, as well as the annual Data Virtualization Experts Forum.

For 2012, Informatica is announcing an all-new program starting with a series of expert roundtables with industry analysts and customers. These roundtables are designed to engage the audience in live conversations with the experts, on data virtualization and its role in business intelligence.

Informatica is also bringing together industry architects and BI professionals to participate in the largest open online community on this subject. Over 1800 strong, the Data Virtualization and Data Services Architecture group is fast becoming the de facto platform for architects to meet (virtually) and discuss.

Needless to say – data virtualization is top of mind for architects, and Informatica is proud to be part of the ongoing dialogue.

I invite you to sign-up for the 2012 program and keep the conversations going.

In covering that topic they made a point of saying that due to severe flooding in Thailand, they expect storage to become in short supply (as much as a 29% global shortfall) through the end of 2012. It is expected that the price of storage/GB will increase as a result and supplies will fall short of demand. They recommended finding alternatives to purchasing storage to keep costs down.

Purchasing more storage to manage database data growth is really a “band aid” solution to structured data growth management as one has to consider the fully burdened cost of storage, including electricity. Structured data growth management aka structured data archiving is an ideal solution as it solves the root cause of having to continue to increase your storage footprint and costs, in addition to several other benefits that it provides. Storage used for production environments typically have the best performance, availability, and disaster recovery capabilities and (as a result) are usually the most expensive storage devices.

Archiving moves inactive or dormant data off production storage onto less expensive storage.  There are a few options with robust data archive solutions. One is to have “seamless access” where you archive directly to a database sitting on a less expensive tier of storage, with lower level SLAs (such as less frequent backups) applied to the archive database environment. Your users can log in to their native application using standard navigation (thus “seamless access”) and have access to both production and archived data. One of the more common use cases for this configuration is that the production database has an extremely high data growth rate and you want to be as aggressive as possible with your retention policies, leaving as little data in production as your users can live with.  Your users will be more agreeable in that aggressive approach if you provide them “seamless access”.

You can then implement a true multi-tiered storage ILM solution by applying retention policies to the archive database and archiving it to a highly compressed (up to 98%) optimized archive file. As a file, the archive can sit on a relatively inexpensive of storage devices as it doesn’t need storage that supports a database. You can also archive directly to the highly compressed optimized archive file from production without implementing the “seamless access” configuration, eliminating the need to support the archive database (and the “middle” tier), if that scenario works best for you and your users.  Access to the optimized archive file is done via ODBC/JDBC compliant reporting tools (most reporting or Business Intelligence tools support this protocol).  A robust data archive solution has unique compliance features as well, such as automated enforcement of data retention/disposal policies.  With storage prices expected to increase, you can archive to inexpensive storage AND reduce the data footprint by as much as 98%. With an industry best data compression of up to 98%, feature rich data archive solutions offer the best alternative to the “band aid” approach of throwing more storage at explosive data growth.

Once you have an archiving program in place and your production database has been optimized in size by archiving, there is a ripple effect out to any production clones you use for testing, development, etc. As an example, if you have a 1TB production database plus 4 copies, the total footprint is 5TB. After archiving off the dormant or inactive data, production is now 500 GB. Once you replicate the optimized production database out to the other instances, you now have a total footprint of 2.5 TB. This “ripple” effect provides additional operational and cost benefits in the non-production copies.  There are also Test Data Management solutions that take this reduction in database footprint one step further. There are subset products that filter data according to the requirements of the user community (testers, developers, etc.) and can create minimally sized, optimal environments for non-production use that can significantly further reduce the database footprint of non-production instances.  These solutions are relatively small maintenance and typically provide rapid return on investment (especially with storage cost on the rise – usually 6-12 months ROI).

By incorporating data archiving and subsetting (test data management), you will be optimized for structured data growth management minimizing the impact of storage costs and/or supply on your organization.

Links

• Data Archive
• Application Retirement
• Data Subset
• Application ILM