Syncsort blog

A Week at Insight in Macau, China

Peter Eicher — Wed, 22 Feb 2012 04:21:46 +0000

I’ve been spending the week at the NetApp Insight event in Macau, China, talking to a host of NetApp partners and sales engineers from all over the Asia-Pacific region about the NetApp Syncsort Integrated Backup (NSB) solution.

While NSB was launched in 2010, Syncsort has only recently begun to deliver the solution in the region via our distribution partner, Distribution Central. Based on the conversations I’ve been having, it’s not a moment too soon!

I spoke to dozens of folks yesterday, and over and over it’s the same theme. Users need better data protection solutions and the partner community here is excited about how NSB can take NetApp’s world-class data protection features (Snapshots, FlexClone, SnapMirror) and extend them into non-NetApp storage environments.

At one point during last night’s cocktail reception, I spoke to four Australian NetApp sales engineers in a row and they all confirmed that they are currently working on opportunities to solve customer problems with NSB. They were all eager to learn more about the solution’s unique value proposition and how it works!

When I share information about how NSB has helped take an 18-hour backup window and reduce it to 45 minutes or less, people’s eyes really light up. The problem is real, and folks here are ready to take NSB out to their users in a big way.

Good thing too, because it was not easy to get here. Just for fun, here’s the journey I took to get to the event:

* Two hour flight from New York to Detroit
* Four hour layover in Detroit
* 16 hour flight to Hong Kong (I watched three movies and we weren’t even close to arriving!)
* 30 minute cab ride from Hong Kong airport to the Macau ferry terminal
* One hour ferry ride, Hong Kong to Macau
* 20 minute shuttle bus from ferry terminal to hotel

On Friday, I get to do it all over again only this time in reverse. While the travel is not easy, the trip and the great conversations here at Insight have been well worth it!

Enabling Your BI Strategy

Jorge A. Lopez — Wed, 15 Feb 2012 19:04:53 +0000

“What happens in Vegas, stays in Vegas”… at least according to the popular tourism campaign. Pardon me for breaking this rule, but I believe an exception is in order when it is the TDWI World Conference that brings you to the Entertainment Capital of the World.

As seems to be the case at every TDWI event, I’m finding a vibrant, exciting blend of IT professionals eager to share and learn about the latest industry trends. However, what has really caught my attention this time around is the increasing level of awareness and interest in data integration.

There is no doubt in my mind that “Big Data” and mobile devices have been instrumental in catapulting business intelligence (BI) to the forefront. Information that was once reserved for a few business analysts and techies has become readily available and open to the entire organization via simple, easy-to-use mobile interfaces. This “democratization” of BI is demanding more information than ever before, and creating a significant need for faster, simpler, and more agile data integration solutions.

This is exactly what I’m hearing over and over again this week at the show. It seems like people are increasingly realizing that when all is said and done, data integration is the core that makes data available to the organization. Mobility has allowed BI tools to deliver on user demands for agile, self-service BI. Therefore, organizations have a greater need than ever before for fast, easy ETL development to minimize “time to BI.” This requires a level of performance and scalability capable of delivering all of the data that users are demanding. Interestingly enough, this was one of the common themes I heard during my visit to MicroStrategy World last month.

I had the opportunity earlier this week to speak with Shawn Rogers, an industry analyst and research vice president at Enterprise Management Associates. Syncsort has worked closely with Shawn for some time, and he made a great point about the role of data integration as an “enabler.” I could not agree more with Shawn, and it seems as though more and more IT professionals are coming around to this way of thinking too. In fact, we have seen more and more organizations turning to Syncsort to enable and accelerate BI initiatives.

Are you seeing this too? Is your data integration tool enabling your BI strategy?

Are You “Syncsorting” Your Data?

Peter Eicher — Thu, 09 Feb 2012 19:40:19 +0000

We recently held our annual sales kickoff and it was quite an event. Three days of meetings, trainings sessions, hundreds of conversations, and lots of fun mixed in.

A favorite moment for me was having one of our customers speak to the assembled group about their experience with NetApp Syncsort Integrated Backup (NSB).

This particular customer (who I don’t have permission to identify publicly) is an organization of more than 20,000 employees spread across 1,500+ locations. The company’s IT environment is pushing 200 servers (physical and virtual), 50 terabytes of data and a Microsoft Exchange system of more than 10,000 mailboxes. Their backup environment was giving them a serious headache – long backup windows, slow restores, lots of tape to handle. Backup was also a full time job. They literally had a contractor on staff five days a week just to handle it! They were using a conventional backup product (you’d know the name if I said it).

Enter NSB. Let’s get right to what changed.

Backup windows went from 18 hours to 30-45 minutes! Yes, you read that correctly. A 96% decrease in time!
Restore times changed from 2-4 hours to restore a system to being back on line in 5 to 10 minutes.
Messy tape shipments are gone, replaced by data replication (using SnapMirror). While the customer thankfully has not had to do a major restore, testing shows they could get all their tier 1 applications up and running in 1 to 2 hours should they lose their main data center.
They no longer have a full-time contractor to run backups. Instead, data protection takes about 30 minutes of attention every few days, a huge boon to a busy IT organization.

I’d call that a major success at modernizing a conventional backup architecture!

My favorite part of the story though is how this customer came up with a new word for recovery. When the IT team restores information using NSB, they say we’re “Syncsorting” the data!

You can be “Syncsorting” your data too, assuming you like the idea of a 96% reduction in backup time and recovery in ten minutes or less. If you want to find out just how NSB can help you, please drop us a line.

Data Protection Q&A with Evaluator Group’s Randy Kerns

Peter Eicher — Fri, 03 Feb 2012 00:57:26 +0000

This week, the Syncsort team from around the world has gathered in Newport Beach, CA for our 2012 Global Sales Kickoff. It has been a great event to say the least with presentations from NSB customers, channel partners, NetApp executives, and our own solution experts. In conjunction with the event, we were also able to recognize great NSB partners like SwishData, Zumasys and Voyant Strategies as part of our Partner of the Year Awards program.

One of the presentations that delivered a lot of value to the Syncsort data protection team was from Evaluator Group’s Randy Kerns. Randy has had relationships with members of the Syncsort executive team for many years, and it was great to have him on hand to share his views on the data protection market and the benefits that NSB can deliver to customers. We had the chance to catch up with Randy following his presentation and wanted to share an excerpt from that. Enjoy!

Q. What demands do you see server virtualization placing on data protection today?

RK: Virtualization creates both a storage bottleneck and a great deal more systems to be backed up. A new approach is needed. New approaches may need a better backup software solution that can handle the virtualized environment.

Q. What do you hear from organizations in terms of the impact of Big Data on data protection?

RK: Right now, it is mostly a case of fear of dealing with the projected amount of machine to machine data they are going to have to deal with. They are looking for approaches and tools to deal with this now.

Q. What are you observing from companies regarding business continuity and disaster recovery planning, as well as hearing about the priorities in those areas?

RK: Many companies have put BC/DR plans on hold the last few years and are now starting to catch up with what is needed. BC/DR has many facets but the basis for all of it is data availability. These companies are starting to execute on their delayed plans.

Q. Having spent time with the Syncsort team this week, are there any key takeaways on the NSB solution and its value to customers?

RK: I think one thing that struck me is the wealth of capabilities there are in the virtualization area. There are some great opportunities there for improving the administration and operations.

Big Data Q&A with ESG’s Julie Lockner

Keith Kohl — Fri, 03 Feb 2012 00:25:50 +0000

It has been a great week at Syncsort’s 2012 Global Sales Kickoff in Newport Beach, CA. As you might expect, there has been a lot of discussion about Big Data, Hadoop, ETL, data analytics, business intelligence, and more.

In addition to some great networking with colleagues from around the world, we have been fortunate to have several of our partners join us. During the event, we also announced the winners of our Partner of the Year Awards program.

However, the presentation that had everyone excited and talking was from Enterprise Strategy Group’s Julie Lockner. We’ve interacted regularly with Julie since she joined the ESG team a little more than a year ago, and find her to be really in touch with the end user community and the challenges they face. I had the chance to chat with Julie for a bit after her presentation and wanted to share a portion of the conversation. Thanks again to Julie for participating in our event!

KK: When you speak with end users, what do they tell you about the challenges they face with data integration?

JL: What is interesting is that everyone I speak with regarding data integration state that they have challenges. Aside from the common challenges associated data quality and data consistency, most talk how the development team has challenges responding to new requirements in a timely fashion. More data, shorter windows, and new applications to be integrated all contribute to the challenges.

KK: What do you see organizations doing to overcome data integration challenges?

JL: Unfortunately, according to recent ESG research, many are still investing in more hardware and software to address the performance challenges. What is interesting, however, is more than 1/5th of the research respondents are using technology classified as data integration accelerators to complement their ETL strategies.

KK: Are you hearing more about data integration accelerators?

JL: As I mentioned in your previous question, this is absolutely becoming a viable option. Data integration accelerators can play an important role in data transformations – especially when you consider that 80% of the tasks involve simple to complex data flows with sorts being a large part of what consumes hardware and compute resources. Organizations I speak with that are using solutions such as DMExpress love the fact that it is fairly straightforward to implement and with little effort. They see significant performance improvements in their ETL jobs.

KK: What are you seeing in terms of Hadoop and the profiles of early adopters?

JL: Hadoop has a lot of promise. There is a lot of excitement because it is new and with that comes hype. What I am seeing, and what is being confirmed by ESG research, is that many are still trying to figure out how to leverage Hadoop in their environment. This year seems to be the year of experimentation with Hadoop. Those that are using Hadoop had an obvious need for something like Hadoop to deal with a seriously big data challenge – and Hadoop made economical sense. The time and effort associated with coming up to speed with Hadoop and deploying a dedicated compute cluster needs to be vetted out in a business case before taking the plunge. In the meantime, I like the idea that there are a lot of “skunks work” projects out there. This will give the technical team a head start when the business case arrives at their door.

KK: Syncsort is based in New Jersey (home of the New York Giants), but has a number of employees (like myself) that either live in New England or have roots there. Do you want to go on the record with your Super Bowl prediction?

JL: I would love to see the New England Patriots win the Super Bowl this year. With these two great teams going head to head, it will be a great game.

Congratulations to the Syncsort Partner of the Year Award Winners

Josh Rogers — Wed, 01 Feb 2012 13:29:36 +0000

Today, we announced some very exciting news recognizing our partners for their commitment to Syncsort’s solutions. The program is a testament to the important role partners play in Syncsort’s go-to-market strategy. Across both our data integration and data protection businesses, we continue to expand our partner ecosystem and forge deep relationships as we navigate the increasingly diverse and competitive industry landscape.

The past year has brought a lot of attention to the industries we serve. The data integration business saw discussions around the opportunities Big Data represents for companies and how Hadoop and the evolution of ETL tools, or ETL 2.0 as we call it, make capitalizing on these opportunities a reality. In the data protection world, snapshot technology came to the forefront and organizations of all sizes faced increased pressure from the C-suite to modernize backup and implement more robust disaster recovery plans. This has certainly been a positive for solutions like NSB, our integrated data protection offering with our partners at NetApp.

Across the board, our partners demonstrated a tremendous commitment to serving customers and using Syncsort technology to create cost-effective solutions that address some of the most complex IT challenges that exist today. Our partners also showed great enthusiasm for the new Syncsort certification and training programs that we rolled out throughout the year. While we appreciate the many contributions made by all of our partners, the award winners really stood out across multiple dimensions. They include:

Data Integration

Global Systems Integrator Partner of the Year: Cognizant
ISV Partner of the Year: Clerity Solutions
Reseller Partner of the Year: Amdocs

Data Protection

Channel Partner of the Year: SwishData
Channel Partner Rookie of the Year: Zumasys
NSB Technical Innovator Award: Voyant Strategies

We’ve had a tremendous response from all of the award winners and couldn’t be more enthusiastic about working closely with each of them in the year ahead. We appreciate their hard work and it’s been fantastic for partners to make statements like:

“Our strategic relationship with Syncsort enables us both to offer clients unique solutions that address the most complex data integration challenges at a fraction of the time and cost. We appreciate the recognition as Syncsort’s GSI Partner of the Year, and look forward to building on our work and success together in the marketplace in the year ahead.” – Hal Lavender, Chief Architect and AVP, Cognizant

“Our partnership with Syncsort provides significant benefits to organizations, such as Singapore Exchange, looking for effective ways to reduce IT operating expenses and enhance flexibility through mainframe migration to better prepare for future growth. By collaborating with Syncsort and utilizing their powerful data integration and sorting technology, we can bring greater value to enterprises with large datasets and batch requirements.” – Cameron Jenkins, COO, Clerity

“Syncsort data protection solutions and their partnership have been a key piece of SwishData’s success since our founding in 2006. As virtualization continues to disrupt various data center strategies, Syncsort products not only provide our customers and partners with an integrated, simple way to protect data, but also enable them to return to production operation time by recovering applications in minutes instead of the hours commonly required with other products. Fundamentally, SwishData adds value to our customers and Syncsort solutions enhance our ability to do just that.” - Bob Kerr, Vice President, SwishData Corporation

Congratulations again to this year’s winners!

Live at MicroStrategy World: Where Business Intelligence Meets Data Integration

Jorge A. Lopez — Wed, 25 Jan 2012 22:59:08 +0000

Each January for the last 8 years, I have had the opportunity to escape the cold winter weather of the Northeast to visit warmer places like Las Vegas and Miami (where I currently am!) to catch up on the latest BI trends at MicroStrategy World. (Full disclosure, I’m an ex-MicroStrategist).

With more than 2,000 attendees, I can honestly say this year has been one of the most exciting, energizing events I’ve attended in a long time. IT professionals from all around the world are thrilled about the seemingly endless possibilities that market disrupting forces like Big Data, cloud, social media and mobile technologies are producing. When all is said and done, these things will have made a profound impact on the way we do business as well as the way we live, communicate and interact with our world. In fact, they already have!

During my many conversations with BI professionals here at MicroStrategy World and elsewhere, they often cite a fundamental challenge that is preventing them from fully realizing the benefits of their BI applications. That challenge is how to build a strong data integration infrastructure that enables IT to capitalize on the opportunities of Big Data. This is exactly where BI meets DI.

It is clear to me that a solid data integration infrastructure not only accelerates BI initiatives, but also helps maximize the benefits by making more data, that is accurate and relevant, available in much less time. However, this is a story that is much more powerful when told by one of our customers. For those of you in Miami at MicroStrategy World, don’t miss the chance on Thursday, Jan. 26 at 11:30 a.m. to hear directly from a leader in the healthcare industry. Part of the Big Data track, the presentation will be held at the Intercontinental Miami and will focus on using Syncsort’s DMExpress to reduce the cost and complexity of ETL for better, faster BI.

For the lucky ones in Miami, see you there! For everyone else, please feel free to leave a comment. I’m interested in your thoughts and experiences.

The Proof is in the Pudding

Keith Kohl — Tue, 24 Jan 2012 20:50:15 +0000

The majority of technology sales, particularly in software, require some sort of proof of concept (POC) intended to prove out the product(s) based on a customer’s requirements. Syncsort is no stranger to POCs and we have a record of producing some really impressive results. Recently, we had the opportunity to present some of these results to a respected industry analyst. He suggested we share some of our POC results on a regular basis on the Syncsort blog. What a great idea!

In writing the first in what will be a series of posts throughout the year on POC results, I was inspired by my colleague and fellow Syncsort blogger Dave Nahmias. One of the phrases that those of us who work with Dave have no doubt heard him speak at one time or another is, “the proof is in the pudding.” We have seen time and time again situations where prospects are pleasantly surprised (and even amazed!) when they get their hands on DMExpress and an up close and personal look at just how fast, efficient and simple it really is to use.

Let’s start with a relatively straight forward POC performed on a Windows machine with 8 cores. Clearly this is not a large, powerful box. This will be important to keep in mind as I share the results. This particular job joined two data sources, performed two aggregations, and then loaded the data into SQL Server and Oracle as well as wrote to a compressed file.

Here are some of the specifics:

One of the data sources consisted of more than 100 million records (15GB of compressed data). The second data source was small (1,200 records). The reading of both files and the join took 2 minutes 25 seconds (about the same as the amount of CPU time). Only 35MB of memory was used!
The first aggregation took just under 19 seconds, and used only 3 of the cores and 50 seconds of CPU time. This included the write to the compressed file and the load into Oracle.
The second aggregation took 40 seconds, using only 2 cores and 45 seconds of CPU! This included the load into SQL Server.

Total job time: 3 minutes, 5 seconds!

So, what were we trying to beat? How about almost 4 hours of processing that was running in the database! Not only did we beat the times by orders of magnitude, the customer can now use a graphical interface to build and maintain the ETL. Perhaps more importantly, the customer can also offload expensive database cycles and staging tables.

Since we do this for a living and see results like this from DMExpress all the time, it is easy to lose sight of the impressive results consistently coming from POCs. However, what we believe makes these results even more impactful is that they were achieved without consuming the entire box.

Stay tuned for more results in the days, weeks and months ahead. We’d also love to hear from anyone interested in learning more or who has seen similar results with their tools (please feel free to post a comment). We are also willing to take on those interested in challenging us to a benchmark!

Data Protection Survey Series (Part 3): Recovery is at Risk

Peter Eicher — Wed, 18 Jan 2012 22:39:20 +0000

It’s been a while since Part 2 of our “Data Protection Survey Series.” I’ve been very busy preparing for Syncsort’s sales kickoff coming up in a couple of weeks and also doing some early prep for NetApp Insight in Macau, China in February (hope to see lots of NetApp partners there!). Kickoff should be a great affair with lots of interesting guest speakers, and I’ll be sure to provide some reports from the event. Meanwhile, back to our survey!

Our final topic is recovery, and the short version of our survey results is that data recovery is truly at risk for many organizations. Systems are not being protected as they should be, and confidence levels are not high.

We started by asking what percent of servers were being backed up each night (broken out into physical and virtual). For physical servers, only 29 percent of respondents were backing up 100 percent of their servers. This means that 71 percent had some amount of exposure to unprotected data.

On the virtual side, results were both better and worse.

A slightly higher percentage of users (31 percent) were backing up 100 percent of their virtual machines (VMs), but there were more users protecting less than 50 percent of their VMs.

The first problem around recovery is that a lot of data (roughly 30 percent) is not even being backed up on any given night. However, the question specifically asked respondents to only consider their backup schedules. In other words, what percentage of your servers are you even trying to back up? It didn’t take into account backup success rates, so that was our next question. What percentage of your backups complete successfully each night?

Only 18 percent of users are seeing 100 percent nightly success rates. The bulk of respondents (57 percent) were getting what is typically considered a reasonable success rate of between 91 and 99 percent. However, a full 25 percent of respondents were at 90 percent or less success, adding a significant amount of data exposure to their organizations each night.

With all these issues around backup, we wanted to see how confident users were about data protection. The answer: not very. We wanted to know how people would view a major disaster where an entire data center was lost, so we asked: “In the event that you lost an entire data center, how confident are you that you could restore application services in a timely manner?” Here are the responses.

Only 14 percent considered themselves “totally confident” with another 33 percent “very confident” (defined as: “I expect most recoveries will succeed but I am not convinced I can achieve 100 percent recovery of all systems”).

More than 50 percent of users had a significant degree of uncertainty. In fact, the results are worse than shown here because 14 percent of total respondents said they didn’t have disaster recovery in place at all! They responses were excluded from the chart. So, well over half of our survey participants are effectively risking their businesses in the event of a major disaster.

Our final question was around disaster recovery (DR) testing. DR testing is usually a rather difficult affair, often involving long hours on weekends spent trying to bring up systems. But testing is critical: it’s the only way to know you can actually restore your data when you need to.

Again, we see a lot of potential exposure. A little over half of respondents test DR at least once a year. The rest range from less than once a year to never, or they have no DR to test. When we correlated “confidence” with “testing,” it was not a big surprise to find that among the group that were “totally confident” they could restore data, 60 percent of them said they test more than once a year and 29 percent tested once a year. That’s a huge correlation of 89 percent of the “totally confident” users testing their DR once a year or more.

On the flip side, of those that were “reasonably confident” they could restore data, only 9 percent tested more than once a year and 32 percent tested once a year. You couldn’t ask for a clearer indication that “testing equals confidence,” and that’s why one of the things I like to emphasize about NetApp Syncsort Integrated Backup is that it makes DR and testing your DR so easy. It’s no wonder that more than 90 percent of NSB customers deploy at least two NetApp FAS units, one for local backup and recovery and the other for remote-site disaster recovery.

Data protection and recovery are serious concerns and can’t be taken lightly. I certainly don’t think that all the risk exposure our survey uncovered is because users are indifferent to the problem. What they are is overwhelmed. Too much data plus disruptive new technologies like virtualization have made conventional backup models obsolete.

As our survey showed, this has led to a mix of problems: too many products being used, backups taking too long, and recovery at risk. Backup needs to be modernized and it can’t happen too soon!

The Relativity of Einstein, Elephants, Air Mattresses and ETL

Steven Totman — Thu, 12 Jan 2012 14:45:05 +0000

According to Albert Einstein, the definition of insanity is repeating the same actions and expecting a different result. While I won’t go quite as far as to call it insanity, it has always bothered me that people keep tuning ETL tools that can’t handle larger data volumes. Over Christmas I had an experience which helped me understand at least some of the logic behind it.

It was Christmas Day and I was staying at the home of my fiancé’s parents. I had taken an inflatable bed so that we could stay the night after indulging in way too much turkey. Having managed to shoehorn the bed into a room that was entirely too small for it, I settled down to sleep. Shortly thereafter at about 3 a.m., I woke to find that I was being swallowed by the mattress. It had developed a slow puncture. For those of you that haven’t experienced it, moving around on a deflating air mattress is not easy or fun!

Knowing that if I got up and off the mattress it was going to deposit my fiancé onto the floor, I had little choice but to inflate the mattress from where I was (waking up everyone else in the house in the process). From that point forward, I spent nearly every hour repeating the same process of inflating the mattress until it was time to get up for the day. Needless to say, I was grumpy and the rest of the house was irritable that entire day. There was also a large air mattress deposited directly into the rubbish bin!

This whole situation got me thinking. Even though I knew it wouldn’t help for more than an hour, why did I continue to inflate the mattress throughout the night?

For starters, I didn’t think that I had any other options (although the 4 hours I spent sleeping on the sofa the next day while Boxing Day chaos continued around me proved that wrong). I also thought (at least for the first inflation at 3 a.m.) that inflating the mattress would permanently solve the problem. It was after the second time (okay, probably the third) that I got wise.

Bringing my crazy story back to ETL, the vast amount of people out there “tuning” ETL tools are likely working on this same logic. The first time they do it, there is probably an assumption made about only needing to do it once. The second time, they maybe think that they just didn’t get it quite right last time and this time will work exactly right. The third time, the harsh reality of their situation starts slowly seeping in as they realise they could be doing this for the rest of eternity and not get the result they are seeking.

However, here is the thing. Ultimately, I knew I only had to keep inflating the bed that one night. The next day the leaky air mattress would be in the bin and I’d be at home sleeping in my own bed. People who “tune” ETL tools don’t have that luxury. They know data volumes are increasing (between 10% and 500% a year depending on which customer I talk to) and fundamentally their ETL tools aren’t going to help. Sure, they can try and buy more hardware (a bigger air mattress), but that’s just a temporary (and very expensive) measure because that leak is definitely going to reappear.

In fact, given all the discussion about Hadoop and Big Data, I am now picturing an elephant standing on a deflating mattress! For those of you that made it to this point in my post, thank you for sticking with me. Now it is your turn. I’d love to hear about your thoughts and experiences tuning ETL tools to handle larger data volumes. Comments are welcome!

Syncsort’s VP of Worldwide Sales Josh Rogers Talks Big Data

Kevin Ruane — Thu, 05 Jan 2012 18:22:40 +0000

For those of you that regularly visit the Syncsort Blog, you no doubt are familiar with our data protection blogger extraordinaire Peter Eicher as well as our smart, insightful team of data integration bloggers including Steven Totman, Jorge A. Lopez and Keith Kohl. To kick off 2012 right, we’ve even added Dave Nahmias to the mix (welcome, Dave!). Syncsort bloggers always value feedback from their readers. Please don’t be shy about commenting on posts, letting us know what information is valuable and what is not, and challenging us when you have a different perspective.

One aspect of the blog that we plan to devote more time to in 2012 is the “Corner Office” section. This is intended to be a place where insights from Syncsort’s executive leadership team are shared with readers. At times, this will take the form of posts directly from the executives. Other times, I will be sharing links to things like media articles or significant news announcements where our executives are quoted (all part of my day job here at Syncsort!).

In the spirit of getting 2012 started on the right foot, I wanted to share a couple of media articles that you might have missed in December. Both feature extensive commentary from Syncsort’s Vice President of Worldwide Sales Josh Rogers on the topic that everyone seems to be speaking and writing about at the moment…Big Data.

Josh’s interview with Forbes’ Tom Groenfeldt includes some historical information about Syncsort and focuses on how the company today helps many of the world’s largest financial services firms process large volumes of data. Josh also had the opportunity recently to speak with BeyeNETWORK’s Ron Powell. The result is an interesting Q&A that explores why enterprises of all sizes are struggling to scale their data integration environments cost effectively to tap into the insights within Big Data.

We wish everyone a happy and prosperous 2012, and are looking forward to interacting with many of you in the blogosphere!

Data Protection Predictions for 2012

Peter Eicher — Tue, 03 Jan 2012 15:59:37 +0000

It’s the time of year to make predictions (a bit late really) about the year ahead. On the subject of prophecy, George Eliot wrote in Middlemarch that “Among all forms of mistake, prophecy is the most gratuitous.” That’s not a sentiment I would argue with, so let’s just say I’m willing to make mistakes!

In general, I’m not one for making predictions. They usually come down to stating the obvious or throwing a random long ball and hoping the receiver yanks it out of the air, David Tyree style for any of you New York Giants fans out there. However, if I must put on the hat and play Carnac the Magnificent, I will stick to broad trends rather than highly specific predictions.

In the data protection world, I can predict with confidence that:

#1 — Traditional Backup Will Remain Broken

This prediction is a lock because it can’t be otherwise. Traditional backup – by which I mean file-based backup – has been broken by massive data growth and increasing virtualization of applications. It won’t fix itself because the architecture is fundamentally flawed. Many users still struggle with it and take a Band-Aid approach to “fixing” it by adding more tape drives or replacing tape drives with a dedicated deduplication disk target. Doing this only drags out the pain. Sooner or later you’ll have to replace what you’re doing. Moving from “Traditional Backup Vendor A” to “Traditional Backup Vendor B” isn’t going to solve your problems either.

Don’t worry though, because as my next prediction states:

#2 – Modernized Backup Will Remain Unbroken

The thing about backup is that we’ve already solved that problem. Now we just have to convince you that there really is a better way! I’ve written about this many times, how it’s possible to make your backup dreams come true by modernizing backups with snapshots.

The only thing standing between you and a fast, efficient and reliable data protection environment are old technologies and outdated approaches. But the key is to modernize your backup from end to end, because:

#3 – Deduplication Will Continue to Be Insufficient

Data deduplication is a great technology, but it’s a single ingredient in what has to be a full recipe for modernized data protection. You can’t just replace your tape with a deduplication target and think it will solve all your problems (despite the many vendors trying to sell you on this story). Does deduplication still matter? Yes it does, but by itself it will leave you disappointed.

The final thing to think about is disaster recovery. In other words, how do I get my data from “here” to “there” so I’m protected when something bad happens “here”? Disaster recovery (DR) isn’t always cheap or easy, and many organizations struggle to do it well (or at all) for a variety of reasons. This brings me to my final prediction:

#4 – Disaster Recovery Will Continue to be a Challenge Except When It’s Not

Like backup, DR is a problem that’s been solved if you re-examine how you do things. With NetApp Syncsort Integrated Backup, we’ve simplified DR by letting you consolidate all your backup data onto a single platform, and then replicate that data using SnapMirror. Add to that a number of different recovery technologies all from the same backups, and you’ve got your disaster recovery all done.

On the other hand, you can keep handling DR with five different technologies, massive outdated run-books, an Ouija Board and who knows what else. It’s easy to predict that something bad will happen eventually if your DR plans aren’t in order.

When it comes to New Year’s Resolutions, fixing your DR plans should be high on the IT department’s list of things to do in 2012 if it isn’t already. If you need some help with that and some new ideas, please let us know.

The Importance of Being Ordered

Dave Nahmias — Tue, 03 Jan 2012 15:17:57 +0000

Automobiles and ETL – what could they possibly have in common? What if I told you it was fuel injection. Remember the muscle cars of yesteryear? They got their power through bigger and bigger engines. Back then, no one would have believed that a four cylinder car could be faster and more efficient. That was until fuel injection. Fuel injection focused on the most inefficient part of the process – gas consumption. By improving this one aspect, engines could be smaller, faster and cheaper. And once an on-board computer could automatically tune the process, fuel injection became the norm. At the time it required a radical re-orientation, but today, nearly every vehicle uses fuel injection.

I believe we are at a similar inflection point in ETL. What is the most inefficient part of the ETL process? Ordering records! Joins, aggregations, rankings, database loads, etc. all depend on ordering to complete their processing. You may be surprised to learn that upwards of 75 percent of all ETL processing is spent ordering. If you need to join two heterogeneous sources (for example a file and a table), the records have to be lined up. If you need to group records in order to create subtotals, the records have to be ordered. If you need to create rankings, you guessed it; the records have to be ordered.

Current thinking is to attack the need to order data with the equivalent of a muscle car. More hardware, more parallelism and more cost. Most ETL tools don’t innovate in this area and simply use parallelism to address the problem. This looks good on a slide with perfect streams of data being operated on concurrently. Unfortunately, reality is not so perfect.

First, these streams are only perfect if the source is partitioned exactly like the target (which is almost never the case). To line up the data, either a sort has to be performed on each node or each node has to exchange records with every other node in order to get them lined up properly for the target. Pick your poison, either a lot of simple sort tasks feeding a simple merge or lots of network traffic and resources to pass records around. And what about all those additional barely-used CPUs that are required to address what is essentially an I/O problem?

A second approach is to load the data into a database. Unfortunately, a database is not very good at ordering records that aren’t indexed. So each time a task is pushed down to the database, a temporary table has to be created to hold the data, the data has to be loaded into that temp table, and then an index has to be created (by the way, creating the index requires ordering also).

This approach is able to keep up with demand as long as there is excess capacity in the database and the ETL process is nightly. However, as volume, currency and usage grows, ETL starts colliding with the business user. Add to that the cost of constantly expanding database hardware and software licenses to try to keep up with the Big Data explosion and many organizations are beginning to consider a more “radical” approach.

This approach, like fuel injection, is surprisingly simple. Rather than continue to build bigger and bigger engines – how about just improve the record ordering process. By injecting intelligence into the most inefficient part of the process, ETL can be performed faster with less hardware. Suddenly, complex joins can be done in seconds without the need for staging tables and indexes. ETL can once again be performed on commodity hardware with minimal impact on the environment.

Intelligent ordering is also finding rapid acceptance with the emergence of Hadoop. The Map-Reduce paradigm requires ordering in a number of its steps and the distributed, file-base architecture requires a small footprint. Because of this, Hadoop implementations are finding efficient ordering to be even more important.

So the next time you consider an ETL tool, make sure it is designed on a second-generation engine that understands the importance of being ordered.

Data Protection Survey Series (Part 2): Backup Takes Too Long

Peter Eicher — Tue, 20 Dec 2011 15:31:52 +0000

Now that we’ve explored the challenge of too many products in Part 1 of the series, it is time to turn our attention to a familiar topic. I’m sure you are all “shocked” by the title and wondering if this blog has turned into the “Captain Obvious” report. Indeed, we all know backup takes too long, but sometimes it’s good to compare yourself to others to see how your experiences stack up.

As someone who has been around the backup world for quite some time (I started off at Cheyenne Software, which my fellow old-timers will remember well), the one constant in this business has been that backups take too long. Technology has changed quite a bit over the years, but the complaint remains the same.

Lately, the problem has increased exponentially as data growth has exploded. So as part of our IT survey we asked users about it, and 26 percent answered, “Yes, data growth has had a very significant impact on our data protection and is causing problems.” That number may seem a bit low, but it is a very serious statement about a problem that exists right now for those respondents. The scope of the problem is fleshed out by the next response, where 51 percent answered, “Yes, data growth has had some impact on our data protection but we are able to handle it.”

That’s not a statement that data growth isn’t a problem, but rather that data growth isn’t a problem yet. It also doesn’t reveal what the users may have done to deal with it. There may be quite a bit of radical backup re-engineering behind the comment that “we are able to handle it.” And for others, it’s very likely just a matter of time.

A final 23 percent said that “data growth has not had an impact on our data protection.” That is partly a function of company size. When we broke out the data between companies above and below the 500 employee limit, it turns out that only 16 percent of the larger companies said “no impact.”

Well, what about actual backup times? We broke out those questions in various forms. Since most people are still stuck in the mold of nightly incremental backups and weekly full backups, we started with “How long does it take your nightly incremental backups to complete?” and got these results:

We see that 32 percent of respondents are in what I would call the “danger zone” of taking more than 7 hours to get a nightly backup done. This is becoming an excessive length of time. What about full backups?

This is where things go from bad to worse. More than half of respondents are taking more than 11 hours for a full backup cycle, and a significant number are in the 24 hours and beyond range. This is cause for serious alarm.

One last cut at the data was to ask about the length of the single longest backup job? This is that one server that stands out amongst all the others as the toughest backup nut to crack.

We have 39 percent of respondents having one server taking more than seven hours to protect. And what happens if that particular backup job fails right towards the end? Trouble is what happens!

Backup times are unreasonable for a significant number of the respondents in our survey. Data growth has overwhelmed legacy technologies and the problems this creates are obvious.

The answer to this isn’t putting in more tape drives or adding more media servers (which you should get rid of anyway). The answer is fundamentally changing your approach to backup by moving less data and not backing up the same data over and over again.

Reading and moving data only once is a core technology strategy of the NetApp Syncsort Integrated Backup (NSB) solution, and it’s the only realistic way to deal with massive data growth. That’s how our users such as Campbell Alliance can reduce backup times from 13 hours to “an hour or two”. You can do the same and we’d be happy to show you how.