The Next Big Thing

Data Virtualization: Essential but Approach with Caution

Fred Cummins — Mon, 02 Nov 2009 07:34:00 GMT

Data Virtualization is the current marketing banner for Enterprise Information Integration (EII). It is an important adjunct to SOA, but must be undertaken with caution.

David Linthicum, Linthicum Group, and Bradley Wright, Progress DataDirect, recently presented an ebizQ webinar on "Putting Your Data to Work for Your Cloud, BPM, MDM and SOA Project." The thrust of this presentation was that data virtualization will provide consistent, cross-enterprise access to data in heterogeneous data stores. This is not a new capability, but was introduced as EII a number of years ago.

When asked the difference between data virtualization and EII, Bradley Wright indicated that data virtualization includes the capability to perform updates. While not all EII products supported updates, some did, so it appears the primary difference is marketing. At the same time, as I will discuss, below, data virtualization should not be used for updates.

The fundamental concept of data virtualization and EII is that data is accessed from multiple, heterogeneous databases through a virtual database that provides an integrated, consistent view of data from these multiple sources. Queries are expressed in terms of the virtual database schema and translated as required, and data from multiple sources is transformed and integrated to provide a response that is consistent with the virtual database schema.

SOA increases the importance of data virtualization because SOA is likely to increase the number and diversity of databases. I discussed this in my blog last year entitled, "Data Management for SOA." A service should be loosely coupled and its data stores should be hidden from the service users to maintain flexibility in the implementation of the service. This conflicts with needs for cross-enterprise views of data for planning and decision-making. Data virtualization can provide such visibility; however, there are certain realities that must be understood when using data virtualization. Loraine Lawson touched on some limitations in an interview with Peter Tran and Bob Reary of Composite Software two years ago entitled, "When Data Virtualization Works - And When It Doesn't," but there are additional concerns.

The following paragraphs outline key limitations of data virtualization that must be considered when setting expectations and when using data virtualization to obtain composite views.

Data inconsistencies. A data virtualization product can perform data conversions (e.g., feet to meters), but it can't create data that isn't stored. For example, if one organization maintains weekly production figures and another maintains monthly figures, these two different measures cannot be reconciled. If one organization tracks numbers of defects in one set of categories, and another uses a different set of categories, the figures cannot be compared or added.

Such problems are fundamental to the business, and if it is important to examine such data across the enterprise, then there must be a transformation initiative to make the data collection and storage consistent with a common scheme.

Process inconsistencies. Some enterprises will have similar business operations that are in different geographies or produce different categories of products or services. What they do may be similar, but they may have business processes that cannot be compared. There may be different stages of production or service delivery that are of interest to top management. The different operations may use the same terminology for phases, but the terms are not applied consistently to the business processes. This may lead to top management comparing apples to oranges. Such discrepancies might extend to inconsistent metrics such as the definition of rework, and inconsistencies between sales and the cost of goods sold.

Timing inconsistencies. An enterprise does not operate instantaneously and in lock step. The orders being received are not the same as the orders being filled and the orders being shipped. The engineering change issued by the engineering department may be delayed until current inventories are consumed. Payment is not due on orders shipped but not yet delivered. A query that combines data from different operations will not represent a consistent view of the enterprise. That requires the definition of cut-off-points and the time for various activities and transactions to reach and record consistent points in their operations. This is why financial information is not immediately available at the end of a period.

It is not practical to eliminate all such inconsistencies or wait to accumulate consistent results. Users of data virtualization must understand such limitations when using the composite data.

Resource overload. A data virtualization service will access data from various production databases. These databases are not necessarily configured to handle an increased volume of queries. Some queries may add unexpected workload to a database, or the workload from many potential users may be quite unpredictable. This ad hoc resource demand could interfere with mainstream business application performance. In cloud computing, the resource may be available on demand, but there could be unacceptable increases in costs.

Update errors. If data virtualization is used to update databases, it will bypass the applications designed to validate, control and coordinate the updates. The updates may also be inconsistent with the current state of the production operations. Furthermore, updates normally performed by associated applications may require coordination and propagation to related operations and applications. It is very dangerous to bypass the responsible organizations and their applications to update their databases-it should not happen. Any update should go through the appropriate processes for validation, authorization, control and coordination that are the responsibility of those business operations and their applications.

I think data virtualization (a.k.a., enterprise information integration) is an important technology that should be part of a SOA strategy, but users must adopt it with their eyes wide open. It's a long-term investment, and it is likely there will always be the need to understand and allow for inconsistencies in the data.

It must be getting closer to the end of the year than I thought

Charlie Bess — Sat, 31 Oct 2009 16:29:00 GMT

I am seeing more of the “technologies you can’t ignore” or “technologies that will change everything” articles. I usually don’t see a rush of these until December

Windows 7, now the ball is rolling…

Charlie Bess — Fri, 30 Oct 2009 19:46:00 GMT

Many home users have Windows 7 in their hands (and more importantly on their machines), almost everything I've heard has been positive. I know HP has been working on preparing for deployments and various support services for Windows 7, including the 7 days of 7 Enterprise clients have their activities in the works and I have it on good authority that there are some eminent deployments. So the images are built...

It makes me wonder about the gap between the early adopters and the mainstream.

I'll try to add some more information as news comes out over the next week or so.

Robotics capabilities for the next decade

Charlie Bess — Fri, 30 Oct 2009 16:30:00 GMT

This video provides some impressive examples of the kinds of sensing and robotics capabilities being demonstrated today (e.g. throwing and catching a cell phone). If it takes a few years to make it into production, there are still some tremendous advances that will take place early in the next decade in the business environment.

There are all kinds of predictions about the use of robotics in surgery as well as the plant floor. They will definitely be more robots in our houses than just roombas or helping us in our cars.

Revisiting 3D printing

Charlie Bess — Thu, 29 Oct 2009 11:26:00 GMT

I've done entries on personal fabrication and 3D printing (printing organs...). Until this point, these techniques have been applied mostly to software materials. Now it's been extended to printing 3D objects directly in metal. The current process is called Electron Beam Freeform Fabrication, or EBF3. This technique is essentially computer controled arc welding.

3D fabrication is definitely continuing to advance and even has its own open source project (RepRap) that started back in 2005 and continues to progress. Commercial 3D printers have broken the $5K barrier and are starting to be used more commonly, for interesting applications like allowing people to have their own WoW figures created in 3D.

One of the themes for sustainability is to move bits not atoms. 3D printing allows you to move the plans to the location and then create the part -- another example where IT techniques are effective in a new way to improve efficiency -- green IT.

Tech jobs replaced by cloud

Charlie Bess — Wed, 28 Oct 2009 21:00:00 GMT

Over the past few months I've seen numerous articles talking about tech jobs being replaced as organizations implement cloud, like this one in CIO magazine.

This is really not anything new. When the mainframe came on the scene in the 50s, it replaced a whole other kind of computers -- people. Yes, they were likely to say their profession was "computer" if they worked on calculating actuarial tables... Some of those jobs went away, but many shifted to a higher value function. When the terminals and PC's came on the scene, many of the keypunch operators or card deck loader jobs went away. This is par for the course for individuals working in high tech fields.

The better question would be: What kind of new positions and opportunities will be opened up by the new technologies? What new value will be generated that allows organizations to expand and impact the market?

The IT jobs most at risk are those focused on configuring and maintaining infrastructure, since those will be centralized and automated. On the other hand, I've written numerous times about how data centers will be used and what new kinds of computing can take advantage of the new computing capabilities.

It is a time to evaluate and think about the implications of the future and then prepare yourself (or your organization) to maximize your ability to surf that new wave of technology.

The Challenge of Analytics

Kas Kasravi — Sun, 25 Oct 2009 01:46:00 GMT

Since I got involved with analytics in the early 90s, I had asked myself why we don’t see more organizations use analytics and business intelligence.

Early on, I thought the answer was that the algorithms and tools were inadequate, then I thought there was a lack of awareness, and then I figured the ROI wasn’t well established. Now, in hind sight, I think I was wrong. In fact, I now see the answer somewhere outside of analytics altogether. There are many good BI and advanced analysis tools on the market, there is a fair amount of awareness, lots of skills, and the ROIs look very good. So, what’s missing? I think the answer is “data preparation”.

Data preparation is the seemingly trivial task of representing the data in a usable form. In many cases, well-structured transactional data don’t lend themselves well to analytics applications. The contextually rich data often seems to be in disparately formatted spreadsheets, different types of databases, word processors, presentation, and even in the minds of the subject matter experts. As I look back, I think we had turned down more opportunities for analytics due to inadequacies in source data than any other reason. The analytics projects we completed seemed to consume more energy for data preparation than any other activity, including modeling and the actual analysis. The data that is suitable for analytics is often hard to pull together; not only expensive, but usually impractical. It’s interesting that there have been so many more advances in tools/algorithms, and not enough in the initial phase of data preparation. Do you agree?

Mashup standards

Charlie Bess — Sat, 24 Oct 2009 23:36:00 GMT

I was reading SD Times the other day while riding an exercise bike at the gym (hey, you have to do something to keep your mind occupied) and I came across an article on the Open Mashup Alliance. I had quite a number of questions about the OMA. Although it is not really a standards body, OMA is working to create a standard for the creation of mashups.

This standard is called Enterprise Markup Mashup Language (EMML).

"EMML is a Domain Specific Language (DSL) that was designed specifically to address the important characteristics that make mashups easier to create and reuse:

EMML is not meant to be a General Purpose Language and does not compete with them. In fact, EMML was designed to be complimentary to and integrated with popular languages like JavaScript, Java, Groovy, Ruby and others.
EMML is a declarative XML-based language and, as such, leverages and complements existing XML capabilities inherent in XQuery, XPath, and XSLT.
EMML has been in development for several years. It also has already been tested in dozens of real-world production mashup implementations.
EMML is an open language specification. This type of common and free-to-use language (and technologies that embed or use it) have a much better chance of meeting the needs of enterprise developers than a proprietary language."

I was surprised and pleased to see that HP was a member, working on this. I guess I'll have to track those folks down.

There are some people who view this consortium positively, while others are a bit more cautious. Mashups have become a standard technique to generate useful information from various sources across an enterprise or a business. Even though formal mashups are relatively immature, having a standards effort should help improve the quality and predictability of results. It's definitely something I'll keep watching.

Is Business-IT Alignment Suicide for the CIO?

fredcummins — Wed, 21 Oct 2009 13:51:00 GMT

Last week, Bob Evans published an article entitled, "Suicide Strategy For CIOs: Aligning IT With The Business." In it he asserts that the CIO must abandon efforts to align IT to the business and instead align with customers. This does not make business sense. Information technology enables business capabilities to deliver value to the customer, but the CIO should support the business by ensuring that the business capabilities make effective use of information technology. The enterprise and its business activities are the customers of the CIO.

The role of the CIO should be to optimize the enterprise information systems and their use of information technology to enable the enterprise to achieve competitive advantage in each of its lines-of-business. That does not mean the CIO is a technology geek, but rather that the CIO must know about the design of the business to determine how technology can empower the business.

Alignment of IT to the business should be put in the context of delivering value to the enterprise customers. Value chain analysis as introduced by Michael Porter in his 1985 book, Competitive Advantage: Creating and Sustaining Superior Performance, promotes business alignment with the customer value. A value chain identifies those capabilities that contribute directly to the value of a line-of-business product or service delivered to the customer. The IT organization represents a supporting capability that is necessary for the effective performance of the enterprise value chains.

The CIO should focus not on the implementation of computer and communication systems, but on management of the technical capability, much of which may be outsourced. This capability includes not only the development of applications and operation of computers and networks, but it includes support for the design of the enterprise.

The enterprise itself in a complex information system. SOA is a design paradigm for enterprise optimization. In SOA:A Matrix Architecture I described how SOA leads to a matrix organization where value chains are managed by line-of-business managers, and services are managed by shared capability managers. The CIO provides the cross-enterprise perspective to develop value chain models, design shared services and provide the applications that support the implementation and integration of the capabilities. Where shared capabilities are provided as services, Value Chain Modeling Is Essential for SOA Management. The value chain models, as well as other information services managed by the CIO, support management planning, decision-making and governance. By focusing on value chains, the CIO should help determine priorities for investment in technology and business systems design that will improve customer value. This is alignment of IT with the business.

In The Next Generation CIO I describe this key business role of the CIO. The CIO is responsible for a critical aspect of the business, just as the CFO is responsible for the financial aspect and the HR executive is responsible for the personnel aspect. There is no discussion about how the CFO and HR executive align to customer values. Their jobs are to serve the shared financial and personnel needs of the enterprise.

The CIO should focus on optimizing the enterprise as an information system through managing the information technology capability. That includes modeling, designing and transforming the business systems. In that context, the CIO will play in indispensable role in the future of the enterprise, and alignment of IT to the business is a measure of CIO success.

Technical trends from an HR perspective

Charlie Bess — Mon, 19 Oct 2009 11:55:00 GMT

A few weeks ago I participated in a technical trend conversation aimed mainly at the oil and gas organizations. Last weekend I did something similar but this time talking with some individuals from various HR organizations - at an HR Roundtable Summit that SMU held in San Antonio.

Their rankings came out it four tiers:

The highest level of interest was:

Predictive Analytics
Web 3.0/Semantic web

The next highest grouping was:

Digitized voice & and transfer to system
IT Virtualization
Social Media Monitors
Content Analytics
Sustainability/optimization
Event Driven Architectures

The lowest grouping receiving votes were:

Process Modeling
Context-aware computing
Offline Web applications
Cloud computing
Sensor Networks and Real-world Integration
Simulation/Modeling
Dynamic business applications
Web 2.0
Extended Enterprise/federation

And those with no votes:

3D Visualization
Interoperability standards
Reference Data Models
Enterprise Master Data Management
Complex Event Processing
Intelligent software assistant

These are quite different ratings than the previous set so it shows how important it is to know your organizations definition of innovation.

Dealing with information overload - email

Charlie Bess — Tue, 13 Oct 2009 11:31:00 GMT

In the October IEEE Spectrum there was an article on How to Beat Information Overload.that I found very timely since, during this past week, there were a number of cases where I saw a question go out to a distribution list asking for technical insight and numerous answers coming back using "reply all", followed (enviably) by numerous people using reply all to say "stop using reply all".

I personally don't mind that much when the answers come back with a "reply all" since many times I am as interested in the answer as I am with the question. I do get disturbed with the folks who say "take me off this list" or "stop using reply all" while using "reply all". My view is "that's why most modern mail readers have rules and filters". If you don't want to see the answer "take care of it yourself". It's not like the other mail wasn't useful and addressing the question being asked. But the follow-up attempt at self-selection definitely serves no business purpose. I've thought about writing an auto-reply rule for this but that might be considered a DoS. ;-)

I easily receive 200-300 emails a day and now have a significant hierarchy of rules to sort and filter email into categories that I get to:

junk - skim through once a week
once a week
once a day
as soon as possible
immediately - these I have sent to my phone as well as an instant message

These capabilities are available to everyone (at least where I work) so people should learn to use them. I've thought about writing a white paper on the topic, just because it is so annoying.

Unfortunately, my once a week pile has lately grown to being over 1000 email messages a week and that takes a chunk out of my weekend to address. On the other hand, using this technique has helped me be more focused during the week. There is no free lunch.

I've thought about trying to have a "no e-mail" day (hey, the federal government does it all the time with regular mail) or even limit myself to 20 outgoing emails a day but part of my job is to communicate with others and so that doesn't seem prudent.

This article goes on to talk about similar corporate activities where there are "no e- mail" times during the day or other techniques where people can try to limit their disruptions. Unfortunately, this may point to an issue where IT is viewed as not having any value for the organization or at least interfering with value creation.

I guess the point I am trying to get across is that we all need to be responsible for our own attention span and not just try to put the blame on others.

Tuesday, October 13th is "National Face Your Fears Day"

Charlie Bess — Mon, 12 Oct 2009 15:36:00 GMT

It even has an official site to help folks confront and overcome the fears that hold them back. There are many things that keep people (and IT organizations) from doing what they need to do to move their organization forward. One of the first is an inability to face up to what's going on around them that scares them.

Many of the technologies on the horizon (like cloud computing, predictive analytics or event-based processing) shake the value of IT organizations down to their core. It's time to face those shifts and think strategically about what they mean to your corporation as well as your organization.

Will your organization's focus be more on service portfolio management or on the analysis of the business rather than the actual management of the hardware itself or will your organization need to turn into an service provider (with deep service management industry knowledge) -- essentially managing leveraged services across the corporation or industry??

Carpe diem

Technical trends from an Oil and Gas Perspective

Charlie Bess — Sat, 10 Oct 2009 10:53:00 GMT

A week or so ago I spoke at an SAP User Group meeting in Houston about technical trends. It wasn't until I was there that I realized I was going to be third out of four presenters talking about technical trends. Needless to say, by the time I presented I had to rearrange my examples on the fly since some of them were already discussed. I think the audience was satisfied, especially since I was the only presenter who wasn't from SAP so it's likely I didn't use the word SAP more than once in my presentation.

One exercise we did was to have everyone in the room place a marker on various technical trends that they believe their organization will be acting upon in the next 3 years.

Here is a summary of the results:

IT Virtualization
Cloud computing
Sensor Networks and Real-world Integration
Predictive Analytics
3D Visualization
Interoperability standards
Event Driven Architectures
Reference Data Models
Process Modeling
Enterprise Master Data Management
Complex Event Processing
Web 2.0
Social Media Monitors
Intelligent software assistant
Content Analytics
Web 3.0
Simulation
Context-aware computing
Offline Web applications
Dynamic business applications
Sustainability/optimization
Extended Enterprise/federation
Digitized voice & and transfer to system

With a breakdown of voting: 11 votes, 7-9 votes, 4-6 votes, 1-3 votes

Topics that no one showed interest in were:

Audio Mining
X Internet
Software Defined Networking
Multi-agent Systems
HashCache
Modeling Surprise
Augmented Reality
The Information Workplace
Semantic Web (I found this strange since many people view Web 3.0 as being Semantic Web)
Ontologies
Microformats/Linked Data

Web 2.0 four years later and five years out

Charlie Bess — Fri, 09 Oct 2009 13:19:00 GMT

There is a good interview of Tim O'Reilly that validated many of the posts we've made in this blog over the last 5 years. He also published a white paper that goes into more details.

The way business value is generated in the future will be changing, as we have the intersection of sensors and edge computing with large databases of information and context. The kinds of skills needed by the personnel will be changing as well. As value is generated by having people focus on the unique aspects of the business instead of being involved in "normal". Some may think this mimics science fiction, but really it will just become the new normal.

The interview provides some good examples and greater understanding of a holistic view of the intersection of technologies that we'll likely see coming out of this downturn.

Tech is leading the way???

Charlie Bess — Thu, 08 Oct 2009 05:55:00 GMT

In an article stating that we should Look to tech to see the economy's future, it appears that demand for high tech items is on the upswing and that Tech will prosper in 2010. It's been stated before that HP is bullish on 2010.

There are numerous reasons to look at 2010 in a positive light, ranging from the aging technology infrastructure that some organizations have not replace since the end of the dot com era and now must, to the outstanding benefits of newer solutions that have come on the scene. System 7 and Office 2010 may also cause some organizations to think about getting new desktops. You can only fight Moore's law for so long before it becomes clear that you're missing something.

Hopefully with this spending organizations will move away from "paving the cow path" to looking at the new business possibilities from IT, and taking that new road. With cloud and SaaS options and various complex event processing options out there, having a Enterprise Architecture holistic perspective can't hurt.