Attend this seminar to learn best practices in emerging technologies and techniques.

This seminar will feature Krish Krishnan, President and CEO, Sixth Sense Advisors, Inc. Along with Bill Inmon, Krishnan is the coauthor of “Building the Unstructured Data Warehouse.” He is also a faculty member of The Data Warehouse Institute (TDWI) and Chief Technology Officer at Forest Rim Technology, LLC.

Learn more…

The post Big Data Seminar 2013, Featuring Krish Krishnan appeared first on .

Frank Fillmore, Founder and President of The Fillmore Group, discusses the intersection of big data and data warehousing, and offers advice to CIOs just embarking on the big data journey. Fillmore spoke to Wikibon’s Jeff Kelly inside theCUBE.

The post Big Data and the Evolution of Data Warehousing appeared first on .

IBM Director of Database Software and Systems with IBM Software Group talks with Wikibon’s Dave Vellante about how IBM’s big data solutions—including BLU Acceleration—are helping customers find faster answers and make better decisions.

The post Big Data in Memory appeared first on .

Communications Services Providers (CSPs) are at the heart of their customers’ digital universe. They are sitting on a gold mine of digital data that enables them to understand their customers at an unparalleled level. It is a key corporate asset, much needed in a hyper-competitive landscape.

Big data has the potential to place CSPs in a prime position to win the battle for customers and create new revenue streams. It provides them with a wealth of information about their customers’ behaviors, preferences and movements. Yet, many CSPs still struggle to fully derive the greatest value big data.

Though CSPs have handled large amounts of data for years, the game-changing aspect of big data today lies in using data to derive new insights—mostly real-time or near real-time—to become more competitive and to create business value. The current focus is on supporting the CSP’s core business, such as improving customer experience, driving new products, increasing productivity, and optimizing networks.

This IBM Institute for Business Value executive report shows how CSPs can acquire the fundamental capabilities to capitalize on the opportunity big data presents, such as:

• Building customer-centric outcomes
• Developing an enterprise-wide big data blueprint
• Using existing data to achieve near-term results
• Building analytics capabilities based on business priorities
• Creating a business case for analytics based on measurable outcomes

IBM’s enterprise-class big data platform enables CSPs to address the full range of big data challenges. Learn more about the opportunities big data presents to CSPs by downloading the IBM Institute for Business Value executive report.

Download the report now…

The post Analytics: Real-World Use of Big Data appeared first on .

One question that keeps coming up in my conversations with customers—despite my best efforts to guide people away it—is “how much data do I need to use a big data solution?” As I’ve written previously, data sizing is usually a lousy way to choose whether to use big data technologies. While there are some cases—for example, if know you are going to have 6+ PB of data under management, as some of our customers do—where it makes sense to choose your technology based on data size, most big data projects are driven by the need for flexibility well before scale comes into the picture.

The flexibility of these systems has many dimensions, but I’d like to focus on one of the most important ones here: the idea of “schema on read.” The structure of the data is definitively determined before any data arrives for us, and we apply the schema to the data store at the time the data is written. Most of us are deeply familiar with schema on write, where we use a traditional (and still vital) relational database to store the data with a predetermined schema in mind, but we generally accept it as the only way to do things. So to help change how we look at this, let’s quickly remind ourselves of the pro/cons to this approach.

There are some non-trivial benefits to schema on write, including:

• In traditional data ecosystems, most tools (and people) expect schemas and can get right to work once the schema is described
• The approach is extremely useful in expressing relationships between data points
• It can be a very efficient way to store “dense” data

However, schema on write isn’t the answer to every problem. Downsides of this approach include:

• Schemas are typically purpose-built and hard to change
• Generally loses the raw/atomic data as a source
• Requires considerable modeling/implementation effort before being able to work with the data
• If a certain type of data can’t be confined in the schema, you can’t effectively store or use it (if you can store it at all)
• Unstructured and semi-structured data sources tend not to be a native fit

We’ve lived with these tradeoffs for a long time now, partially because there aren’t many good alternatives. The emergence of big data technologies poses an alternative—a schema on read approach—that changes the equation since it allows us more flexibility in matching the approach to the problem/maturity/nature of the patterns we are serving.

Schema on read is dramatically simpler up front: you just write the information to the data store. Unlike schema on write, which requires you to expend time and effort before loading the data, schema on read involves very little delay and you generally store the data at a raw or atomic level. In other words, you store what you get from the source systems—as it comes in from those systems. Schema on read means you can write your data first and then figure how you want to organize it later.

So why do it that way? The key drivers: flexibility and reuse. With a schema on write approach, it is hard to support applications, reporting, and analytics that don’t understand your schema, need changes to it, or have ad hoc usage patterns. With a schema on read approach, you define the schema at the time of interaction so it can be (with some constraints) pretty much anything you want or need it to be.

You may want to consider taking a schema on read approach for several reasons:

• Gives you massive flexibility over how the data can be consumed
• Your raw/atomic data can be stored for reference and consumption years into the future
• The approach promotes experimentation, since the cost of getting it “wrong” is so low
• Helps speed the time from data generation to availability
• Gives you flexibility to store unstructured, semi-structured, and/or loosely or unorganized data

But there are some drawbacks to schema on read too:

• Can be “expensive” in terms of compute resources (then again, these big data engines were built to handle that)
• The data is not self-documenting (i.e., you can’t look at a schema to figure out what the data is)
• You have to spend time creating the jobs that create the schema on read

One area where we see the advantages far outweighing the drawbacks is in environments where multiple LOBs all try to hit the same source systems for their own copy of the data. The schema on read approach involves having a data “landing zone” where the raw or atomic data is written out. After getting the data once, all the LOB systems make their schema on read requests against the landing zone. This —prevents the source systems from having to deal with all the LOB requests and provides a one-to-many approach of serving up data. We’ll talk about the landing zone pattern more in future columns.

Remember, no one approach works for all needs. I’d encourage you to add this topic to your Fit For Purpose discussions. Let me know what you think in the comments, and thanks as always for reading.

The post Why is Schema on Read So Useful? appeared first on .

DB2 for z/OS is celebrating its 30^th anniversary this year. When released, DB2 was designed to serve as a data warehouse. With time and development, DB2 evolved into the state-of-the-art Online Transaction Processing (OLTP) and data warehousing engine that we know today. But very few changes have impacted DB2 in its history as much as the introduction of accelerators. Accelerators integrate with existing DB2 for z/OS servers to create a hybrid database. Today, DB2 is a powerful software package that can deliver excellent performance for both transactional and analytic processing.

But what is a DB2 accelerator? How do accelerators work? And why you should consider an accelerator for your organization? This article will provide direct answers to the most frequent and fundamental questions that DB2 users are asking today.

What are DB2 accelerators?

A DB2 accelerator, or an accelerator for DB2 for z/OS queries, is a technical solution that executes SQL queries on behalf of DB2 for z/OS. It is designed to be a high-performance appliance. It can help some types of queries, such as those commonly found in business intelligence or analytic applications, run fast—sometimes much faster.

The very nature of the appliance concept provides transparent integration with DB2 for z/OS. An accelerator is not another database; it is an extension of DB2. The maintenance and administration tasks are done through DB2. Performance metrics are included in the traditional DB2 instrumentation facility, and there is no new user interface to learn.

Both isolation and integration are transparently exposed to end users as well. No application changes are necessary to accelerate eligible queries.

IBM announced the availability of IBM DB2 Analytics Accelerator for z/OS V3.1 (IDAA) in October 2012. In its third generation, IDAA provides improvements that significantly increase its business value. It provides an enhanced scope for query offload, meaning that more SQL is now eligible for acceleration. It supports low-latency incremental data updates, which makes recent changes available for acceleration. The new high-performance storage server feature helps to reduce disk storage costs by allowing you to store archive data on IDAA only, under DB2 control, where it can benefit from hyper-speed analytics processing.

How do DB2 accelerators work?

The IDAA appliance connects to z/OS and DB2 using a private network. The DB2 objects to be accelerated are defined and loaded in IDAA using DB2 stored procedures and a graphical user interface. Just configure a few settings and choices, and the installation is ready.

Once an accelerator is installed and activated, the DB2 optimizer treats the appliance as a new access path and will offload SQL processing transparently when it is more efficient to do so. Not every query may be accelerated. Some may have incompatible SQL statements, or the DB2 optimizer may decide that offload is not the optimal choice (for more information, read this IBM Redbook: Optmizing DB2 Queries with IBM DB2 Analytics Accelerator for z/OS). Alternatively, you can also configure the optimizer to send every eligible query to the accelerator—regardless of any costing or heuristic consideration.

Why you should consider using a DB2 accelerator?

Most of the business value of IDAA could be summarized in two key concepts: speed and cost savings. IDAA also helps deliver transparency for users and applications, and it is designed for fast and easy deployment.

The potential benefits of IDAA can be estimated before purchasing and installation. A study based on the information provided by the DB2 dynamic SQL statement cache can help you assess the SQL queries and the CPU that would be offloaded if you deployed an accelerator. You can use a virtual accelerator in combination with EXPLAIN to understand how your queries would behave with a real accelerator. For example, the DSN_QUERYINFO_TABLE table provides reasons why ineligible queries are not eligible, as well as information about accelerated queries.

Conclusion

IDAA helps transform DB2 into a state-of-the-art transactional and analytic database engine while minimizing installation and maintenance efforts. By accelerating SQL queries in a way that is transparent to applications, IDAA can provide both high performance and a reduction in total cost of ownership (TCO).

Have you considering using IDAA in your organization? How has it changed your operations? Let me know—or leave your questions—in the comments.

The post The What, How, and Why of DB2 Accelerators appeared first on .

The CIO walked with me down a short concourse to a large conference room, where every wall was a whiteboard. From floor to ceiling, in all directions, the walls were filled with drawings, scribbles, graffiti, and a lot more. Interestingly, of the two large walls and two short walls, 90 percent of the content described the company’s firefighting and tactical remediation efforts. Only a few short feet of one wall contained notes on new initiatives.

Several senior technologists browsed each wall, as if the contents of their section had been assigned solely to them. Huddled around the new initiatives section, eight sales people feverishly recorded notes. I was struck by how odd it seemed that so many of the sales people were interested in that tiny piece of real estate, while the remainder was assigned to a relative handful of folks. The CIO had intended to show me the problem at hand by directing me to the “owner” of one of the tactical areas. But I realized immediately that the problem at hand was much larger than this. His primary problem was capacity—too little of it.

When we invoke humans as data processors because the machine processors are overworked or underpowered, we witness strange cultural turns. Humans will leverage part of their workday for data processing—pulling, integrating, formatting, and collating information about their various points of interest. Once that process is completed, they start consuming the information or share it with others. Spreadsheets are a common technology foundation for this. In short, the lack of capacity for repetitive, tedious data reconstruction (which should have been automated long ago) is instead offloaded to the humans.

In these conditions, it is common for whole departments to train their staffs in two primary disciplines. The first is data processing—and the second is the discipline associated with the top bullets on their individual resumes. You know, the business reason why each person was hired.

One leader of a customer services department lamented that he now had to specify SQL skills as a prerequisite for hiring someone in his department, because the various internal data systems were too underpowered and poorly architected to put a useful reporting interface on them. At another site, we reported to the CEO that his corporate staff would actually have time to do the jobs they were hired to do if only such a large portion of their hours weren’t being conscripted for individual data processing activities. Imagine his expression when we multiplied the individual hours by his headcount, and applied an average hourly rate to them—the company’s inefficient processes were costing them millions of dollars. The organization was bleeding productivity, but the CEO failed to see it because it was an intangible loss. In three years time, at least two of his senior staff had put in over four years worth of working hours. (Imagine working an entire additional year for free!)

When data processing systems are underpowered and humans take over, they start to develop cultural sideshows like SWAT teams racing to and fro between data system problems. Rather than fix the root cause of the problems—lack of capacity—the company simply tries to help the SWAT teams move faster. Anyone walking into the situation from a more right-sized environment would get culture shock, and wonder why the organization’s culture has so devolved away from the core business mission.

But back to the CIO in the whiteboard room. Can we now imagine why the environment is so fractured and dysfunctional? Technology is an enabler—but underpowered technology is a false economy. It leads senior leaders into tacit acceptance that they have spent enough and done enough, because on the surface things seem to be operational. But they are not operating through automated power. They are operating thanks to human discipline.

Just as an aircraft is a collection of non-flying parts, so is a business is a collection of functional pieces that must all work in harmony and efficiency to achieve the overall goal of the organization. When a business requires processing power but is starved for it, the CIO sees chaos. The CEO, on the other hand sees something else: lack of capacity. The CEO has plans, you see. He wants to take the business higher. But he cannot seem to get it out of the tactical tar pit.

In Part 2 of this article, we’ll take a look at how to apply capacity to get out, up and over these problems. In the meantime, let me know if you have any feedback or questions in the comments.

The post Dream Machines: Part 1 appeared first on .

Join senior-level executives from Pfizer, AstraZeneca, Takeda, GlaxoSmithKline, Johnson & Johnson, and many more at the Pharma Data Analytics Conference on July 10-11, 2013 in Philadelphia, Pennsylvania.

Delegates will benefit from a dynamic presentation format consisting of workshops, panel discussions, and industry-specific case studies on the recent trends in big data management. This conference will also address how industry leaders are using analytics to gain insight and take immediate action in a competitive market. Hot topics will include real-world data sharing and best practices in business forecasting.

For more information, please visit http://www.marcusevansch.com/MDGL2013_IBM.

Seats are limited! For information on registering, please contact Robin Yegelwel at (312) 540-3000 ext. 6483 or robiny@marcusevansch.com.

The post Attend the marcus evans Pharma Data Analytics Conference appeared first on .

BLU Acceleration in DB2 10.5 for Linux, UNIX andWindows delivers results from data-intensive analytic workloads with speed and precision that is termed ʺspeed of thoughtʺ analytics.

Join IBM Distinguished Engineer and DB2 expert Sam Lightstone for an in-depth discussion of the all-new BLU Acceleration features in DB2 10.5.

In this deep dive, Sam will explain capabilities such as dynamic in-memory analytics, parallel vector processing, enhanced columnar storage techniques, actionable compression, and more.

Register for this webcast now!

The post DB2 TechTalk: Deep Dive on BLU Acceleration in DB2 10.5, Super Analytics Super Easy appeared first on .