The new BLU Acceleration and Columnar Data store are great new technologies and should prompt every installation to do a DB2 10.5 upgrade immediately. I discussed some of the features before and these features were highlighted in many of the IDUG presentations. The presentations really brought out all the advantages of these new technologies. Below are the 10 best reasons that I found in a variety of IDUG presentations that demonstrate the case for upgrading to DB2 10.5 as soon as possible.

1.  DB2 Columnar offers massive compression

• With the new DB2 BLU Acceleration and DB2 Columnar data store there are three levels of compression within DB2 10.5: standard table compression, adaptive compression and the new DB2 columnar data store compression.  All compression saves I/Os and improves all aspects of performance. Some customers using DB2 10.5 columnar tables are getting 90-95% compression of their data.

2.  Leveraging CPU hardware

• The Single Instruction Multiple Data (SIMD) BLU technology is important because instead of having to issue multiple instructions to interrogate data within the chip’s buffer, only one instruction needs to be issued.  By leveraging SIMD hardware capabilities DB2 is able to interrogate data simultaneously throughout the entire CPU chip hardware, speeding processing substantially.

3.  Parallelism through the CPU core

• By leveraging SIMD hardware DB2 is able to bring multiple data references into the computer chip CPU and memory hardware and interrogate them in parallel, providing improved performance for SQL processing. 

4.  Columnar efficiencies minimizes I/O

• With the ability to store only one columnar data value for each row occurrence a minimum of I/O is needed to interrogate and retrieve data.  Huge compression and a minimum of I/O are reached, especially important for data warehouse types of repeating data applications.

5.  Data Skipping

• Data Skipping is a tremendous performance boost brought about by skipping data not needed by the SQL result set or overall processing.  For example, if a SQL query has WHERE CUST_LASTNAME IN (‘BEULKE’, ‘KATZNELSON’,’ ZIKOPOULOS’), data skipping can skip through the data. doing a minimum of I/Os and only interrogating a minimum of data beginning with  ‘B’, ‘K’ and ‘Z’ to realize  the answer set extremely fast with a minimum of resources.

6.  Better WLM capabilities

• The Workload Manager (WLM) algorithms within DB2 10.5 have been improved under the DB2_WORKLOAD=ANALYTICS setting to improve the workings of the DB2 Kernel to moderate the number of queries that consume resources.  This helps improve performance by minimizing database and resource contention.

7.  Maximizes internal CPU and RAM operations on the data

• Through better WLM algorithms the DB2 Kernel is able to maximize the CPU, memory and I/O devoted to the various SQL workloads.  Since DB2 is leveraging all aspects of the computing CPU, memory and I/O hardware infrastructure, maximizing performance and reducing contention is best left to the IBM lab and DB2 optimized algorithms by leveraging the WLM algorithms.

8.  New “Not Enforced Uniqueness” constraint

• Similar to the Not Enforce Referential constraints DB2 provides the ability to not enforce a unique constraint over your data.  This avoids requiring the creation of an index over a database table key to enforce a unique constraint.  Remember DB2 Columnar does not need indexes because it is so fast.

9.  Automatic space reclaim

• DB2 automatically reclaims space in data extents that have had their data deleted.  You no longer need a separate REORG or DBA management to get unutilized space within your storage.

10.  Simple to implement

• All of these features, especially the DB2 Columnar data store, are seamlessly built into DB2 10.5 so no application changes are necessary.  The DB2 columnar tables can be used within the same database, application and SQL statement as any other DB2 tables.

Also my article entitled “Five Imperatives for Superior Java Application Performance” was published during the IDUG conference in the latest Enterprise Systems Journal.  You can link to it here.

________________________________________________________

I am presenting my Big Data presentation about how a 22 billion row data warehouse was built in only six months in the DB2 Night Show on May 17th. You can sign up at the DB2 Night Show web page.

________________________________________________________
Dave Beulke is an internationally recognized DB2 consultant, DB2 trainer and education instructor. Dave helps his clients improve their strategic direction, dramatically improve DB2 performance and reduce their CPU demand saving millions in their systems, databases and application areas within their mainframe, UNIX and Windows environments.

For example, IBM and the software groups are focusing on delivering better business outcomes.  The IBM software team is collaborating across the world with over 280,000+ employees and customers to incorporate all types of ideas.  Through this collaboration IBM developed five main themes for delivering value at the point of impact from the very detailed and wide ranging ideas.

• Exploration:  Deliver solutions that help data exploration for delivering better data relationships and insights.
• 360° View of the Customer: Provide solutions that help me understand all aspect of my customers better and faster
• Deliver Security and Intelligence: Protect my data assets and enhance fraud detection.
• Deliver Operational Efficiencies: Achieve the best total cost of ownership infrastructure.
• Deliver data Augmentation: Help me uncover relationships between any data types anywhere in my enterprise.

With these five themes IBM is focusing on delivering value to companies so they can know their market, know their customer, know what to do and know the best next action to deliver better solutions for their customers.

The IDUG conference provides in-depth technical information and IBM Fellow Curt Cotner has delivered many IDUG technical speeches describing DB2 technology advances over his 35-year DB2-focused career.  This year Curt delivered his  last keynote announcing his retirement in summer later this year.  He is retiring on a high note, as he along with Bill Bireley, detailed the new DB2 data stores within both the DB2 for z/OS and DB2 LUW products by the end of 2013.  These new DB2 Family data stores provide interoperability between all types of new and interesting platforms, products and protocols.

During Curt’s speech he announced that the DB2 Family will be embracing the following data stores by the end of 2013:  

• Key Value Data Store
• Document Data Store
• Columnar Data Store
• Graph Data Store

Along with embracing and handling these new data stores, the DB2 Family is announcing support for interfacing with these data stores through JSON, SPARQL and OWL languages capabilities.  These fast and flexible new data management data interchange, web ontology, and query languages provide interfaces and some interrogation capabilities into these new data stores.  These advanced technologies  are being leveraged for all types of applications and help DB2 leverage and interface with new mobile phones, machine, medical and mechanical sensors and all types of new schema-less data stores.

Also, check out the updates to my blog post from a few weeks ago.  The beta program for DB2 Version 10.5 Kepler has gone well and the team has been working hard on the new DB2 Version 10.5 Columnar data store.  Development has gone so well that the testing and integration of new features has been completed.  Check out the new features for DB2 Version 10.5 Kepler in my updated blog here.

This was only the start to the many great technical presentations during the week, proving once again that IDUG continues to be the premier DB2 Family technical conference.  Make sure you put money in the budget now for IDUG North America 2014 next year in Phoenix May 12-16th.

I am presenting my Big Data presentation about how a 22 billion row data warehouse was built in only six months in the DB2 Night Show on May 17th.  You can sign up at the DB2 Night Show web page.

DB2 and IDUG have prospered for many reasons and the following three reasons are I believe some of the most important reasons that DB2 and IDUG will continue to endure and prosper for many more years to come.  

First, IBM and the DB2 development team put performance first.  While data integrity is a given within any database, DB2 continues to use its optimizer, algorithms and performance patterns to reduce the amount of CPU and I/O to get answers from the database.  Every release and version has raised the bar on performance making sure to continue to lead the industry and stay ahead of the competition.  

Performance and reducing the total cost of ownership for DB2 has been a common theme for every new version. The buzz coming out of the DB2 11 Beta and especially DB2 Version 10.5 Kepler that I detailed here continue the great tradition of DB2 leading the database industry in performance.  I expect at this IDUG, like previous years, more details of these new DB2 versions will be made public so I can share them here in the coming weeks.  That is another reason IDUG is so important and continues to be a leading database industry conference for getting the latest and greatest DB2 information along with networking with the people that design and work with DB2 in companies around the world.

Next, DB2 continues to be open and embrace any new data types.  When DB2 was first introduced it started supplying answers only against standard data types.  With subsequent releases DB2 has put its optimizer and processing algorithms to use for all varieties of new data types: unstructured, video, audio, XML, and lately the new graph data stores.  All of these new data types and their related processing patterns have had the IBM researchers and scientists studying the data type usage and I/O characteristics.  Each I/O pattern characteristic of these new data types has been embraced and optimized so that DB2 can get the best performance possible and continue to lead the industry.  

Also, DB2, unlike several other popular industry databases, provides the freedom to have user defined data types (UDT) and user defined functions (UDF).  These UDTs and UDFs provide DB2 users the capabilities to experiment and define their own new data types and processing functionality.  This openness to any new data type imaginable helps the users build any business system solution and is another critical attribute of why DB2 continues to be embraced for all types of new applications and systems.  Seeing presentations on these new data types UDTs and UDFs is another reason IDUG is special because it caters to all aspects of DB2.

Third, IBM DB2 continues to acquire, incorporate and leverage new software and hardware technology at an amazing rate.  Through the last thirty years IBM has studied and sometimes acquired startup database technology companies to incorporate their technology inside the DB2 family.  Many of these new software techniques and processing patterns have come from open source efforts.  IBM has sponsored or been involved with academic or other industry open source efforts.  This is why you see IBM incorporating the Hadoop graph store type technology while jettisoning its “R” application language in favor of standard SQL.

These many acquisitions over the years as well as the recent Netezza technology have been great examples of how quickly IBM, due to the flexibility of the DB2 family, is able to incorporate these new technologies.  The Netezza technology especially has been a huge success after being incorporated into the IDAA z/OS solution providing dramatic query execution time reductions.  Hardware gating and I/O interrogation via the hardware chip technology continues to help DB2 provide industry leading performance  and will for years to come as this technology becomes more sophisticated.  

Getting information on these new software and hardware technologies along with application development and database designs techniques has always been IDUG’s strength. This year like other years, IDUG is having DB2 users, beta customers, IBM DB2 developers and third party vendors explain and present all this new technology and processing patterns.  

These are the reasons DB2 continues to be an industry performance leader and  IDUG continues to give you the best information directly from the sources.  See you there!
______________________________________________
I will be speaking at the conference presenting Big Data and Analytics Session F07 – “Data Warehouse Designs for Big Data Performance” Wed, May 01, 2013 (02:15 PM – 03:15 PM) in Bonaire 5&6.
_____________________________________________________
Dave Beulke is an internationally recognized DB2 consultant, DB2 trainer and education instructor.  Dave helps his clients improve their strategic direction, dramatically improve DB2 performance and reduce their CPU demand saving millions in their systems, databases and application areas within their mainframe, UNIX and Windows environments.

Since the response to the new DB2 BLU Acceleration dramatic query performance improvements has been so tremendous, everyone is planning on using DB2 BLU Acceleration and the DB2 Columnar data store for their new Big Data initiatives as soon as it becomes available.  But within every new leap in performance and version of database there are always considerations. With DB2 Version 10.5 Kepler there are three important considerations that need to be put into your plans for leveraging the new DB2 Columnar Big Data capabilities.

DB2 version 10.5 Columnar data store is mainly for read only types of workloads such as Big Data analytics, data warehouse, marketing, auditing and other reporting type applications.  Since it leverages the new technology mentioned earlier, its main mode in this first release is read only.  Insert, update, and delete SQL statements are not allowed yet but are rumored to be coming soon.

The main way of getting all your Big Data into the DB2 Version 10.5 Columnar data store is through the LOAD utility.  The LOAD utility with the REPLACE and INSERT options are the main ingestion processing options.  Again rumor has it that more LOAD utility options and other SQL statements and functions will be coming soon.

IDUG UPDATE:  The DB2 Version 10.5 Columnar data store now handles SQL Inserts, Update and Delete statements.  Also, the LOAD utility options have been improved to include all of the other standard LOAD utility options such as INGEST, IMPORT, INSERT and others.

Since DB2 Columnar is leveraging new types of table definitions and other new database engine access techniques, the Big Data table space table storage needs to be handled through the DB2 LUW Automatic System setting.  In this first release the Big Data DB2 Columnar table requires that its table space has Automatic Storage enabled so that the table space can manage all the extents, compression and data storage attributes.  Your plans should probably use a table space Automatic Storage setting or separate table spaces and buffer pools to get the best performance possible for your new Big Data DB2 Version 10.5 Kepler Columnar tables.

The new BLU Acceleration DB2 Kepler Columnar table data store shows great performance for the beta customers and it will be released very soon so everyone can take advantage of this new exciting technology for all their new Big Data projects.  Follow me on Twitter for IDUG highlights and comments from the conference.

__________________________________________________

Have you made your plans for IDUG in Orlando this year?  Also make sure to register early and get the IDUG early bird discount. Sign up today!

I look forward to speaking at the IDUG DB2 Tech Conference 2013 North America conference.  The conference will be held in Orlando, Florida on April 29-May 2, 2013.  Get more information at www.idug.org.

Also I am involved in the Thursday night IDUG Dine Around.  Sign up at IDUG.org.

I will be speaking at the conference presenting Big Data and Analytics Session F07 – “Data Warehouse Designs for Big Data Performance” Wed, May 01, 2013 (02:15 PM – 03:15 PM) in Bonaire 5&6.
_____________________________________________________
Dave Beulke is an internationally recognized DB2 consultant, DB2 trainer and education instructor.  Dave helps his clients improve their strategic direction, dramatically improve DB2 performance and reduce their CPU demand saving millions in their systems, databases and application areas within their mainframe, UNIX and Windows environments.

One of the things that struck me was that everyone talked about their Big Data integration initiatives and how the businesses were going to outsourcers, consultants and non-standard IT outsiders to start their Big Data projects.  One analyst talked about asking a client company to count how many Big Data projects were active within their company.  Auditing the IT department found only two Big Data projects on the books.  Further asking around of the various departments showed that nine other Big Data projects had been started by the business units which were actively doing Big Data integration.  With all your Big Data being copied everywhere; here are four security improvements for your environment so that your Big Data projects, both official and unofficial, are protected within your company.

First, research and remove the PUBLIC access to your Big Data.  During the previously mentioned audit to find the Big Data projects it was discovered that someone had downloaded the complete 6TB data warehouse of a financial firm and had it stored on two PCs next to their desk.  They accessed and loaded the data through a data warehousing tablet/mobile reporting interface over a long holiday weekend.  Personal Identifying Information (PII), credit card information and financial firm account numbers were downloaded because of a new tablet/phone report interface that had been GRANTed to PUBLIC. Worse yet, the two PCs were available on the shared company network so everyone could play with their Big Data Analytics.  Do you think the employee should be rewarded for initiative or fired for being derelict with protecting the critical company data?

Next, audit your BYOD, user and partner integration and access points.   A lot of end-user access is from “bring your own device” (BYOD) set-ups, with tablet access, PC access and mobile interfaces the most commonly used access points.  Making sure that these BYOD and other connections are made through only authorized interfaces where some type of authentication is necessary is critical for data security.  All interfaces should use some type of authentication, even web browsing if possible.  If no authentication is required, allow only very strict limited access with limited capabilities with limited data amounts. No Big Data downloads should be allowed over an extended weekend.

Third, define the same or better security, usage and governance within your Big Data project. Big Data projects are going to use and access customer, product and other types of PII or sensitive information.  Those Big Data insights need to be matched up to your systems and data to gain those analytic insights.  To ensure the on-going protection and safeguards are in place, make sure your security department and its procedures are fully involved and engaged with the security profiles used for execution and access of the Big Data. Since another type of database, like Hadoop or any of the other NoSQL vendors’ products might be involved, it is always good to contrast and document the security for your IT management and showcase the robust and flexible security within your main DB2 databases.  This is a good management and security department education and is a great discussion point advantage for using DB2 for your Big Data project.

Fourth, use encryption by default or upgrade to use encryption as soon as possible for every connection.  The latest hacker Android phone application exploit highlighted here shows that it is easy to take over the control of a plane because there is no encryption in the flight systems of older planes.  Encryption for your connections or better yet encryption of your data is vital to protect your Big Data.  The hacking exploits are becoming more complex and some are even using Big Data dumps of monitor logs to analyze and pinpoint the possible viable attack strategies against your environment and configuration.  Encryption of your access points for both remote and local communications to your users is vital for the survival of all your systems and applications.

The business units are starting and funding their own Big Data project without IT involvement.  By downloading open source Hadoop and configuring two discarded PCs with extra Best Buy hard drives make a Big Data project easy to do. The Big Data download can begin without IT knowledge, help, security, standards and procedures. Security usually doesn’t get better when data is transported to another platform especially a new open source one.  Audit your company and environment now for any Big Data projects before your Big Data project gets your company on the front page of the New York Times. Pass this on to your IT VP of the new Big Data projects and see if they are surprised how many Big Data projects that have already been started without IT’s involvement.

__________________________________________________

Have you made your plans for IDUG in Orlando this year?  Also make sure to register early and get the IDUG early bird discount. Sign up today!

I look forward to speaking at the IDUG DB2 Tech Conference 2013 North America conference.  The conference will be held in Orlando, Florida on April 29-May 2, 2013.  Get more information at www.idug.org.

Also being an IBM Data Champion I get to sponsor an attendee for the IDUG conference through the Mentor program.  This enables them to get a reduced registration fee.  Email me if you are interested.

I will be speaking at the conference presenting Big Data and Analytics Session F07 – “Data Warehouse Designs for Big Data Performance” Wed, May 01, 2013 (02:15 PM – 03:15 PM) in Bonaire 5&6.
_____________________________________________________
Dave Beulke is an internationally recognized DB2 consultant, DB2 trainer and education instructor.  Dave helps his clients improve their strategic direction, dramatically improve DB2 performance and reduce their CPU demand saving millions in their systems, databases and application areas within their mainframe, UNIX and Windows environments.

1. First IBM continues to develop the IBM DB2 Family and it continues to pull away from the competition.  This is confirmed through the growing number of companies that are converting their databases from other database vendors, especially Oracle to DB2, for better performance and to reduce overall hardware and software licensing costs.  Disappointing strategy and quarterly results from Oracle show this trend in almost every category as highlighted in this Yahoo article. IBM continues to pull away from the competition through its DB2 Version 11 for zOS and DB2 LUW “Kepler” Betas that are progressing nicely.  Usually, there are DB2 Beta news items and updates debuted at IDUG conferences and I look forward to hearing more about these new releases.  
2. Because I’m involved with the DB2 LUW “Kepler” Beta, I’ve seen that IBM has some new great DB2 LUW features that every OLTP, Big Data and data warehousing system can leverage for even more infrastructure savings.  Experiencing these features first hand through the Beta program has been very exciting because it’s a chance to see IBM DB2 development leverage all the latest hardware and software capabilities.  Pay close attention to the IBM DB2 LUW announcements in the coming weeks as the new technology really is a game changer for your database designs and performance tuning thinking.
3. In addition to DB2 Version 11 for zOS and DB2 LUW “Kepler” IBM continues to enhance their Big Data platforms and software offerings.  IDUG this year has many great sessions planned for Big Data and IBM’s coming announcements will only add to the excitement at the IDUG conference.  IBM understands Cloud and Big Data from its traditional and new clients and continues to learn about these areas through a variety of initiatives.  IBM recently expanded the number and use of IBM Watsons, the computer that won Jeopardy, for different types of applications.  Read more about it in this great New York Times article. If you had a Watson what analytics or questions would ask it?
4. IDUG is also a great place to see the innovations from the other vendors.  There are always new releases and versions to talk about with their performance products, maintenance improvements and special features.  Finding out about all these is critical to make sure your shop leverages the latest and greatest tools for availability and performance.  Sometimes you find a specific tool that addresses your database situation and it can make all the difference in the world for the success of your application.   
5. IDUG is also the place to network, make new friends and see old friends from the 25 years of  the conference.  It is always great catching up with everyone and hearing about our common database design, performance and maintenance struggles.  Talking about the application development languages, their techniques and performance issues is always helpful for avoiding the same problems.  Also discussing the management by magazine, database trends, the best Dilbert and the best new beer is always good.

There are many more great reasons to come to IDUG this year and every presentation is another great opportunity to learn from the users of DB2 about the on-going real world success stories of the DB2 Family. With this year also being the 30th anniversary of DB2 and the 25th anniversary of IDUG, you will want to hear how the DB2 Family continues to evolve and be the best in the database industry.
___________________________________________________

Have you made your plans for IDUG in Orlando this year?  Also make sure to register early and get the IDUG early bird discount. Sign up today!

I look forward to speaking at the IDUG DB2 Tech Conference 2013 North America conference.  The conference will be held in Orlando, Florida on April 29-May 2, 2013.  Get more information at www.idug.org.

Also being an IBM Data Champion I get to sponsor an attendee for the IDUG conference through the Mentor program.  This enables them to get a reduced registration fee.  Email me if you are interested.

I will be speaking at the conference presenting Big Data and Analytics Session F07 – “Data Warehouse Designs for Big Data Performance” Wed, May 01, 2013 (02:15 PM – 03:15 PM) in Bonaire 5&6.
_____________________________________________________
Dave Beulke is an internationally recognized DB2 consultant, DB2 trainer and education instructor.  Dave helps his clients improve their strategic direction, dramatically improve DB2 performance and reduce their CPU demand saving millions in their systems, databases and application areas within their mainframe, UNIX and Windows environments.

1. Is the database design normalized?  Database normalization using at least third normal form and maybe up to fifth normal form is the starting critical evaluation criteria.  The reason the database design normalization processes have been endorsed forever is because they are effective for identifying all the insert, update and delete data anomalies and support the integrity of the application data. 

   Look at the number of tables and the unique keys within each of the tables.  Too many tables with the same unique key(s) can be a sign of over normalization.  Over normalization can be a database design performance killer.  If your team doesn’t know about database design normalization search the web; there are many great deep resources.

2. Do the database design tables leverage the business’ natural keys?  Good database design leverages, accesses and fully filters using the natural keys within the business.  Make sure to confirm these natural business keys are used within the database design.  Too often recently with some of the new Big Data NoSQL databases or some of the Object Relational Mapping products being used for web development, a ROWID, SEQUENCE or IDENTITY column is being used or required.  These types of keys will limit the database integration within the rest of the business environment and are usually detrimental for getting good SQL application performance by requiring a business key to ROWID translation for every database activity.

3. Does the database have good column definitions?  Database column names and definitions are going to be used for as long as the database will be active so confirm the proper names, abbreviations and short standard names are used for your column names.  Also analyze the database column data type definitions for proper business use, domain and range.  Having all your columns as a single data type such as VARCHAR, CHAR or XML does not reflect the business diversity of data.

   Even though column definitions can be easily changed, make sure to reflect the numeric type and range of values so that the proper DECIMAL, SMALLINT, INT or BIGINT is used.  Having a database design with tables that are defined with good numeric business centric keys is preferred if possible.

4. Are all the columns NULLable?   Within the database columns definitions good data domains, ranges and values should be analyzed, evaluated and prototyped for the business application.  Having good default values, a limited scope of values and always a value are best for performance and application logic.  NULLable columns are only good when data is unknown or doesn’t have a value yet.  Someone’s death date data is the classic example of a NULLable column because it is unknown unless they are already dead.  Make sure your database design represents data that is known and only uses a minimum of NULLable columns.
5. Do you leverage the correct amount of Referential Integrity (RI)?  I endorse RI defined within the database schema definitions because it enforces the business policies, is usually more efficient than application or multiple applications enforcing it and database defined RI will always be there years later to validate the data relationships. 

   RI needs to be defined within the database schema during the development phase of the application so the coders can understand the RI and how to efficiently minimize the RI overhead by referencing and performing all operations in the proper RI sequence. 

   The RI should be developed to the correct depth of the business application and the various transactions within the schema.  Good RI database structures can usually be built within five to ten levels.  Using over ten levels of RI usually leads to database performance issues, indicates table design over normalization or business transactions with a scope requires too many tables.  

These are only five simple database design performance tips and there are many more that I thought about while writing this article.  Use these first five database design performance tips as the starting criteria for your next database development and your system will be designed with a good perspective on performance. 

_____________________________________________________

Have you made your plans for IDUG in Orlando this year?  Also make sure to register early and get the IDUG early bird discount. Sign up today!

I look forward to speaking at the IDUG DB2 Tech Conference 2013 North America conference.  The conference will be held in Orlando, Florida on April 29-May 2, 2013.  Get more information at www.idug.org.

I will be speaking at the conference presenting Big Data and Analytics Session F07 – “Data Warehouse Designs for Big Data Performance” Wed, May 01, 2013 (02:15 PM – 03:15 PM) in Bonaire 5&6.
_____________________________________________________
Dave Beulke is an internationally recognized DB2 consultant, DB2 trainer and education instructor.  Dave helps his clients improve their strategic direction, dramatically improve DB2 performance and reduce their CPU demand saving millions in their systems, databases and application areas within their mainframe, UNIX and Windows environments.