Constant Thinking

Getting Started with Amazon Web Services

constant — Tue, 24 Jul 2012 20:34:49 +0000

In the last few articles, I shared a few thoughts on how I think the world of IT is changing, which became the context for my good-bye to the world of physical IT altogether.

As of last week, I started working for Amazon Web Services (AWS) as a Solutions Architect, helping customers architect systems and solve technical problems using the latest cloud computing technologies. I'm very thankful to be able to work here, as it brings me back to the very center of IT innovation and gives me the opportunity to do lots of new and interesting things.

In the last weeks, I've been digging around AWS and its services, playing with stuff and meeting lots of inspiring people. So I thought I'd put together a few links for those interested in exploring the world of the AWS cloud computing platform for you to learn more about AWS:

Amazon Web Services Overview

If you haven't heard much about AWS before, or haven't had the chance to check it out yet, I suggest you start with the AWS home page and check out the list of AWS Products & Services. I was amazed at how much more AWS has to offer in addition to virtual machines and storage in the cloud. There's a page full of Case Studies that includes some well known brands and their stories on how they use AWS to save money, be more flexible and scale their infrastructure securely in the cloud.

Digging Deeper

If you're like me, you'll want to try out some of the AWS services and see how they work. It's really easy: Sign up using your normal Amazon credentials and start playing right away! Signing up is for free and you'll only pay for what you use. Many of the services come with a free usage tier that helps you learn stuff at no cost.

In addition to documentation, whitepapers and sample code & libraries, there's a variety of articles & tutorials and videos & webinars that show you step-by-step how to do cool stuff in the cloud.

Examples include launching a Wordpress website in 10 minutes, backing up your Oracle database into Amazon S3 using RMAN or running your own Big Data operations in the cloud by using Amazon Elastic MapReduce.

For the architectually inclined, there's the AWS Architecture Center with overview diagrams, architecture whitepapers and webinars on architecting complex systems in the cloud. Eye-opening.

Books, Blogs and Birdfeed

Like every good geek, I bought myself a good O'Reilly book to get started, which is this one: Programming Amazon EC2*. It provides a good overview of basic AWS services and guides you through the design, development and practicalities of a few example web services running on AWS. I haven't read Programming Amazon Web Services* and Host Your Web Site In The Cloud* yet, but they're high up on my reading list.

Here are some blogs to read:

Werner Vogels, CTO at Amazon, blogs about building scalable and robust distributed systems in his blog All Things Distributed.
You can stay in touch with AWS News and Announcements by reading the Amazon Web Services Blog.
My dear Sun colleague Adrian Cockcroft now works at Netflix, a prominent AWS customer. He blogs at the Netflix Tech Blog and his own personal blog. Check this out if you want some real-world stories about operating a multi-million dollar business with tens of millions of users from the cloud.

And some Twitter accounts to follow:

Jeff Barr (@jeffbarr) is the Senior Evangelist for AWS. He tweets about AWS news, announcements and other cool stuff.
If you prefer German, here's a brand new Twitter account for you: @AWS_Aktuell is the German language Twitter account for AWS News and Announcements. Be among the first to follow!

Jobs

Amazon Web Services is hiring. If you're interested, check out our AWS open positions page and the Amazon careers page. And for those in EMEA, the AWS EMEA Jobs Twitter feed is a good account to follow.

In Germany, we're hiring Solutions Architects, Sales Representatives and a Business Development Manager in Munich. In Berlin, we're looking for a Solutions Architect, a Technical Evangelist, Software Development Engineers and a Systems Engineer.

One of the key things that make Amazon unique as a company and that convinced me to join are the Amazon Leadership Principles. I wish more companies would embrace these the way Amazon does.

Have fun!

There you have it: Lots of links to get started. Before I leave you to trying out your own stuff on AWS, here's a fun video that introduces you to one of my favourite database services: Amazon DynamoDB.

What will you build with AWS?

P.S.: This blog now uses Amazon CloudFront for blazingly fast static content delivery. It took me only 15 minutes to set it up for my two Drupal-based blogs and I'm paying less than 2 dollars per month for it.

*Affiliate link: Buy cool stuff and support my blog at no extra cost. We both win!
Here are the books featured in this entry:

Walking my Talk

constant — Thu, 14 Jun 2012 19:19:00 +0000

A while ago, I argued that the world of IT is changing, and that change is good. And that as a result of change, many people would need to change their jobs.

Well, I did it. Last Tuesday was my last working day at Oracle.

14.5 years ago, on January 15th, 1998, I joined Sun Microsystems, directly from university. Back then, I was immersed in the world of the world-wide web: I used to be my university's web master, and I already worked with Sun as a temp, administering the web server of ARD, the biggest German public television network. (Check my CV for details.) Those were the fun days when entire web sites ran on a single processor, single disk Netra i system (arguably an Engineered System already) that was sitting in a small rental office somewhere in Hamburg.

14.5 Years of Learning and Meeting Great People

During my time at Sun, I had the privilege to learn a lot about UltraSPARC, Solaris, System design, web protocols and standards, Java, big systems, small systems, really big systems, clusters (both for high availability and for performance, as in HPC), batch scheduling, workstations, visualization, 3D, real graphics cards, Grid Computing, Web 2.0, blogging, podcasting and finally, Cloud Computing.

Then, after Oracle took over, I continued learning: Databases (only a little bit, I confess), Weblogic, ITIL, Enterprise Architecture and TOGAF, and of course Engineered Systems.

I also had the privilege of meeting great people, both at Sun and at Oracle: Great engineers, system engineers, field engineers, sales people, marketing people, managers, directors, VPs, even Scotty and Jonathan. And above all: Customers after customers, partners, ISVs and all other roles that the IT ecosystem has to offer. All of you provided helpful insight, inspiration, impulse, wake-up calls, viewpoints, nudges and the occasional kick in the butt.

Thank You!

The Road Ahead

My next job will be very different, though still IT centric. I hope I'll continue learning a lot while meeting great people. Almost everything will change for me, but one thing remains the same: Helping customers get the most out of exciting new technology.

The Blog

Of course, I'll continue to blog about interesting technology topics. Just bear with me while I settle down at my new job and figure out a few things before I start blogging about my job again. Meanwhile, I have plenty of stuff on my "to blog" list that I plan to share over time.

BTW, I actually run two blogs: This one and Paleosophie, a German blog about Paleo Nutrition and Lifestyle, which is the best way I know to really understand health, as well as HELDENfunk, a podcast for German sysadmins that a few friends and I produce semi-regularly. If you're fluent in German, feel free to check them out.

Stay Tuned

I'll now wrap up a few things, clean up my to-do list and go on a quick vacation, then start a new chapter in my professional life.

Meanwhile, feel free to connect with me on LinkedIn, Xing, Facebook and/or Twitter, and subscribe to my newsletter or my RSS feed.

To put it in the words of Austrian-American economist Peter F. Drucker:

“The overwhelming majority of successful innovations exploit change.”

Get Ready to Change your Job

constant — Wed, 25 Apr 2012 14:35:10 +0000

The universe is change; our life is what our thoughts make it.
(Marcus Aurelius)

If you have a job in IT (and who among my readers hasn't?), then it is going to fundamentally change soon.

Why?

In my own job, I see the full spectrum from where IT innovation is created to the very last laggards who are still depending a lot on mainframes and other ancient technology. Some things in IT are new (like, every week there's a new startup/technology/trend that is shaking up the industry), and some things are just repetitions of stuff that has happened before, albeit in slightly different colors.

So now, the world of IT as we know it is changing (again) and this time, change will impact organizations, roles and jobs.

Let's dive a little bit into what's happening. Don't worry, change is good, but only if you prepare for it.

There are three trends that are hitting the IT landscape right now. They are more or less independent, but they also complement each other:

Trend #1: Climbing Up the IT Stack

Information technology is just a stack of technologies. In its simplest form, we see hardware at the bottom, and software at the top. Over time, the stack has diversified into a more complex building where IT is composed of:

Applications and Services
Software:
- Middleware
- Database
"Glue":
- Operating Systems
- Virtualization
- Networking
Hardware:
- Servers
- Storage

For each of these components, every IT department needs to decide whether to build something from scratch, or just buy a pre-engineered solution.

A long time ago, IT shops built their own storage and server components, or they built their own databases and middleware, but that was very quickly delegated to IT vendors. Today, IT shops build systems out of components.

But that is changing:

When was the last time your team built a file server? Right. Home-grown file servers today are obsolete, the network attached storage industry is doing that now, and IT shops buy ready-to-use NAS-devices.
Engineered Systems are doing the same for database and middleware servers: Soon, there won't be a lot of motivation to build yet another database server cluster from scratch or yet another web farm etc. Even if you leave out Oracle's Engineered Systems, there are other vendors now catching up with their own versions of “building blocks” or “integrated systems”, all with the same goal: Do the hardware integration for you.

So if your job/role/task has been to evaluate host bus adapters and their compatibilities with individual servers and SAN components, chasing down storage firmware patches so your server's OS doesn't hiccup too badly when confronted with a certain type of disk or if you run a test lab that regularly evaluates storage, server or networking components, then it's time to rethink what you're doing and whether this is going to continue being a good use of your time.

The trend here is obvious: Smaller parts are more and more integrated at the factory level, bigger parts are delivered to IT organizations, the point of deployment and integration is rising up the stack over time.

Today's part granularity is the rack, not the component. And if racks are too much of a hassle for you, you can get rid of all this hardware stuff altogether by using cloud computing.

Trend #2: Disintermediation

Ultimately, the whole purpose of running IT is to provide business services: A credit card billing service or maybe a blogging service or even an online dog food store. Every such service is composed of components across the whole IT stack.

Unfortunately, this creates a lot of dependencies: If you want to upgrade your storage, you need to plan for it, otherwise your credit card service would suffer, if your server needs a patch, you need to make sure the blogging software running on top of this server is moved elsewhere first, if you're running a consolidation project, you better schedule a few meetings with your dog food store department to make sure the transition is accurately planned, managed and executed without disrupting your service and so on.

But: The more stuff is dependent on more other stuff, the stiffer your organization is, the longer stuff takes time to complete and the more errors can happen along the way. Agility suffers.

The solution is disintermediation:

Volume managers helped storage admins disintermediate their work from their server colleagues so they can manage storage without being dependent on their server departments. Server people liked that, too, because they could "get a new volume" by just calling their buddies from the storage department without having to wait until new hardware is rolled in and connected to their servers.
The same thing is happening in server-land through virtualization: Running the hardware is now becoming more and more independent from running the OS on top of the hardware. OS admins now request a new server in much the same way server admins request some new storage. The physical hardware layer can now operate separately from the OS and all the other software layers above.

And disintermediation is hitting again, this time thanks to Engineered Systems and cloud computing. Whether you're operating a private cloud (based on Engineered Systems or not) or you're using a public cloud, your IT operations are now more and more independent from your IT services:

No more coordination between departments for hardware consolidation, faulty hardware replacement or hardware modernization.
Processes closer to the hardware level are more automated and require less manual intervention.
But on the other side, developers, architects and other people closer to the IT service are now doing more and more operations work: They are now responsible for the OS within the VMs that they get from IT, they have to come up with the golden images that IT ops runs, they have to design scalability, durability and the whole service lifecycle in the virtual world that IT offers them. Or they simply plug together and run their own service, out of PaaS components.

While IT ops is focusing more and more on running a private cloud infrastructure that is application agnostic (or maybe their company has decided to entirely offload IT into the cloud), the remaining pieces of IT that are closer to developing and managing IT at the application/service level enjoy more independence, more control but also more responsibility for their services.

This trend has been known as “DevOps” for quite a while (and I highly recommend Ben Rockwood's blog posts on DevOps, or this piece by Adrian Cockcroft of Netflix). Werner Vogels, Amazon CTO put it into his now famous words: “You build it, you run it!”.

What does this mean for your job? Well, if you're working in IT and you're now evolving into a more cloud based model, you need to decide whether you want to be part of the cloud operations bit (which is going to be much more automated and hence will offer less room for creativity), or maybe it is time to huddle up closer with your developers and service operators: They now have a need for some operating system and HA and network savvy guys that help them do this “you run it” thing, now that the cloud is here.

Trend #3: There's a Lot of New Stuff to Learn!

With all this change (from assembling components by hand to automated data centers, from silo IT to cloud computing and so on), there comes a lot of new opportunity: New stuff needs to be figured out which requires creativity, new tools and technologies emerge which need to be explored, new roles and responsibilities are created almost daily.

Here's some food for thought:

The web is reinventing itself almost every year: If you thought Blogging, Podcasting, Facebook and Twitter was something new, think again. The new kid on the block is now Pinterest and I still have to figure out what it may be useful for (at least to me...). Try to stay on top of current web developments and learn how they play the game of popularity, influence and monetization.
The rules of work are changing: Mobile tools like laptops and mobile phones have enabled a more mobile workforce in the 90's. Today, the workplace is entirely virtual: Offices are no longer assigned to individuals, instead we now see flexible offices and drop-in centers almost everywhere. Not to mention work-from-home, which has become a great way of increasing productivity while promoting work-life balance. Other ancient rules of working are challenged as well: As older companies downsize and lay off employees, they learn how to become independent contractors while an entire new generation is growing up to be small to medium entrepreneurs instead of joining a regular job. On the other hand, innovative employers try to attract talent through workspaces that look more like living rooms than offices, relying on self-organization vs. hierarchies (have you seen the Valve Employee Handbook?) and even letting employees choose how many days of vacation they want. The more open and flexible you are about work concepts, the better you can adapt to change. Try to be more unconventional.
New programming methods and languages are emerging, almost on a daily basis. Whether triggered by the need to better leverage the web, cloud computing or simply by playing around with new concepts. Three years ago, I would never have thought of becoming a mobile sensor network that feeds Big Data or using a client-side scripting language to program the cloud. As automation hits data centers and services move into the cloud, high-level scripting has emerged as a new "must have" skill. Even the rules of programming and scripting have changed: Learning how to program today is much more about learning patterns and concepts, while individual languages can vary by taste, requirements or simply whatever environment your cloud provider currently offers. Try to learn a new programming paradigm, API or language at least every year to keep your mind fresh in terms of what the software world has to offer.

Conclusion

As new IT concepts like virtualization, Engineered Systems, Cloud Computing, DevOps, new services, patterns and languages emerge, they force IT organizations to re-think and adapt roles, responsibilities and jobs to the new reality. Change is a constant in IT, and the current times are likely to see a lot more change than we have seen before.

Try to climb up the IT stack towards places that are closer to where your company creates value, and where your creativity makes a difference, as commoditization, automation and outsourcing in the industry climbs up behind you and is now hitting the system integration level.

Re-think how your company's value creation processes map to business processes and ultimately IT processes and how new ways of disintermediation affect your processes and your architecture methodology. If DevOps is the new ITIL, what would a new version of TOGAF look like?

Finally, try to learn something new from the rapidly changing web culture, be it a new way to look at work, a new scripting language or a new API.

Nobody knows how the future is going to look like, but the more you dig into new stuff, the closer you'll be to whatever the winners of the future are.

Useful Stuff to Read

Here are a few books that may be interesting, helpful or just fun to read as you re-invent your job (and yes, these are affiliate links. They help me pay for hosting costs and they'll cost you nothing, we both win!):

(If you don't see anything, then you have an ad blocker enabled. Trust me, at least this time, it'll be ok to switch it off :).)

Your Turn

What changes do you see in your organization as your IT department evolves?

How do you prepare for change?

What other tips, resources and books do you recommend to stay on top of changes in IT?

Share your insights in the comments!

Street sign photo by Flickr-user CBS_Fan, used and modified under CC license.

How to Avoid Your Next 12-Month Science Project

constant — Tue, 10 Apr 2012 12:54:43 +0000

While most customers immediately understand how the magic of Oracle's Hybrid Columnar Compression, intelligent storage servers and flash memory make Exadata uniquely powerful against home-grown database systems, some people think that Exalogic is nothing more than a bunch of x86 servers, a storage appliance and an InfiniBand (IB) network, built into a single rack.

After all, isn't this exactly what the High Performance Computing (HPC) world has been doing for decades?

On the surface, this may be true. And some people tried exactly that: They tried to put together their own version of Exalogic, but then they discover there's a lot more to building a system than buying hardware and assembling it together. IT is not Ikea.

Why is that so? Could it be there's more going on behind the scenes than merely putting together a bunch of servers, a storage array and an InfiniBand network into a rack? Let's explore some of the special sauce that makes Exalogic unique and un-copyable, so you can save yourself from your next 6- to 12-month science project that distracts you from doing real work that adds value to your company.

Engineering Systems is Hard Work!

The backbone of Exalogic is its InfiniBand network: 4 times better bandwidth than even 10 Gigabit Ethernet, and only about a tenth of its latency. What a potential for increased scalability and throughput across the middleware and database layers!

But InfiniBand is a beast that needs to be tamed: It is true that Exalogic uses a standard, open-source Open Fabrics Enterprise Distribution (OFED) InfiniBand driver stack. Unfortunately, this software has been developed by the HPC community with fastest speed in mind (which is good) but, despite the name, not many other enterprise-class requirements are included (which is less good).

Here are some of the improvements that Oracle's InfiniBand development team had to add to the OFED stack to make it enterprise-ready, simply because typical HPC users didn't have the need to implement them:

More than 100 bug fixes in the pieces that were not related to the Message Passing Interface Protocol (MPI), which is the protocol that HPC users use most of the time, but which is less useful in the enterprise.
Performance optimizations and tuning across the whole IB stack: From Switches, Host Channel Adapters (HCAs) and drivers to low-level protocols, middleware and applications. Yes, even the standard HPC IB stack could be improved in terms of performance.
Ethernet over IB (EoIB): Exalogic uses InfiniBand internally to reach high performance, but it needs to play nicely with datacenters around it. That's why Oracle added Ethernet over InfiniBand technology to it that allows for creating many virtual 10GBE adapters inside Exalogic's nodes that are aggregated and connected to Exalogic's IB gateway switches.
While this is an open standard, it's up to the vendor to implement it. In this case, Oracle integrated the EoIB stack with Oracle's own IB to 10GBE gateway switches, and made it fully virtualized from the beginning. This means that Exalogic customers can completely rewire their server infrastructure inside the rack without having to physically pull or plug a single cable - a must-have for every cloud deployment.
Anybody who wants to match this level of integration would need to add an InfiniBand switch development team to their project.
Or just buy Oracle's gateway switches, which are conveniently shipped with a whole server infrastructure attached!
IPv6 support for InfiniBand's Sockets Direct Protocol (SDP), Reliable Datagram Sockets (RDS), TCP/IP over IB (IPoIB) and EoIB protocols. Because no IPv6 = not very enterprise-class.
HA capability for SDP. High Availability is not a big requirement for HPC, but for enterprise-class application servers it is. Every node in Exalogic's InfiniBand network is connected twice for redundancy. If any cable or port or HCA fails, there's always a replacement link ready to take over. This requires extra magic at the protocol level to work. So in addition to Weblogic's failover capabilities, Oracle implemented IB automatic path migration at the SDP level to avoid unnecessary failover operations at the middleware level.
Security, for example spoof-protection. Another feature that is less important for traditional users of InfiniBand, but very important for enterprise customers.
InfiniBand Partitioning and Quality-of-Service (QoS): One of the first questions we get from customers about Exalogic is: “How can we implement multi-tenancy?” The answer is to partition your IB network, which effectively creates many networks that work independently and that are protected at the lowest networking layer possible. In addition to that, QoS allows administrators to prioritize traffic flow in multi-tenancy environments so they can keep their service levels where it matters most.
Resilient IB Fabric Management: InfiniBand is a self-managing network, so a lot of the magic lies in coming up with the right topology and in teaching the subnet manager how to properly discover and manage the network. Oracle's Infiniband switches come with pre-integrated, highly available fabric management with seamless integration into Oracle Enterprise Manager Ops Center.

In short: Oracle elevated the OFED InfiniBand stack into an enterprise-class networking infrastructure. Many years and multiple teams of manpower went into the above improvements - this is something you can only get from Oracle, because no other InfiniBand vendor can give you these features across the whole stack!

Exabus: Because it's not About the Size of Your Network, it's How You Use it!

So let's assume that you somehow were able to get your hands on an enterprise-class IB driver stack. Or maybe you don't care and are just happy with the standard OFED one? Anyway, the next step is to actually leverage that InfiniBand performance. Here are the choices:

Use traditional TCP/IP on top of the InfiniBand stack,
Develop your own integration between your middleware and the lower-level (but faster) InfiniBand protocols.

While more bandwidth is always a good thing, it's actually the low latency that enables superior performance for your applications when running on any networking infrastructure: The lower the latency, the faster the response travels through the network and the more transactions you can close per second.

The reason why InfiniBand is such a low latency technology is that it gets rid of most if not all of your traditional networking protocol stack: Data is literally beamed from one region of RAM in one server into another region of RAM in another server with no kernel/drivers/UDP/TCP or other networking stack overhead involved!

Which makes option 1 a no-go: Adding TCP/IP on top of InfiniBand is like adding training wheels to your racing bike. It may be ok in the beginning and for development, but it's not quite the performance IB was meant to deliver.

Which only leaves option 2: Integrating your middleware with fast, low-level InfiniBand protocols. And this is what Exalogic's "Exabus" technology is all about. Here are a few Exabus features that help applications leverage the performance of InfiniBand in Exalogic:

RDMA and SDP integration at the JDBC driver level (SDP), for Oracle Weblogic (SDP), Oracle Coherence (RDMA), Oracle Tuxedo (RDMA) and the new Oracle Traffic Director (RDMA) on Exalogic. Using these protocols, middleware can communicate a lot faster with each other and the Oracle database than by using standard networking protocols,
Seamless Integration of Ethernet over InfiniBand from Exalogic's Gateway switches into the OS,
Oracle Weblogic optimizations for handling massive amounts of parallel transactions. Because if you have an 8-lane Autobahn, you also need to improve your ramps so you can feed it with many cars in parallel.
Integration of Weblogic with Oracle Exadata for faster performance, optimized session management and failover.

As you see, “Exabus” is Oracle's word for describing all the InfiniBand enhancements Oracle put into Exalogic: OFED stack enhancements, protocols for faster IB access, and InfiniBand support and optimizations at the virtualization and middleware level. All working together to deliver the full potential of InfiniBand performance.

Who else has 100% control over their middleware so they can develop their own low-level protocol integration with InfiniBand? Even if you take an open source approach, you're looking at years of development work to create, test and support a whole new networking technology in your middleware!

The Extras: Less Hassle, More Productivity, Faster Time to Market

And then there are the other advantages of Engineered Systems that are true for Exalogic the same as they are for every other Engineered System:

One simple purchasing process: No headaches due to endless RFPs and no “Will X work with Y?” uncertainties.
Everything has been engineered together: All kinds of bugs and problems have been already fixed at the design level that would have only manifested themselves after you have built the system from scratch.
Everything is built, tested and integrated at the factory level

. Less integration pain for you, faster time to market.

Every Exalogic machine world-wide is identical to Oracle's own machines in the lab: Instant replication of any problems you may encounter, faster time to resolution.
Simplified patching, management and operations.
One throat to choke: Imagine finger-pointing hell for systems that have been put together using several different vendors. Oracle's Engineered Systems have a single phone number that customers can call to get their problems solved.

For more business-centric values, read The Business Value of Engineered Systems.

Conclusion: Buy Exalogic, or get ready for a 6-12 Month Science Project

And here's the reason why it's not easy to "build your own Exalogic": There's a lot of work required to make such a system fly. In fact, anybody who is starting to "just put together a bunch of servers and an InfiniBand network" is really looking at a 6-12 month science project.

And the outcome is likely to not be very enterprise-class. And it won't have Exalogic's performance either.

Because building an Engineered System is literally rocket science: It takes a lot of time, effort, resources and many iterations of design/test/analyze/fix to build such a system. That's why InfiniBand has been reserved for HPC scientists for such a long time.

And only Oracle can bring the power of InfiniBand in an enterprise-class, ready-to use, pre-integrated version to customers, without the develop/integrate/support pain.

For more details, check the new Exalogic overview white paper which was updated only recently.

P.S.: Thanks to my colleagues Ola, Paul, Don and Andy for helping me put together this article!

The Business Value of Engineered Systems

constant — Tue, 13 Mar 2012 12:51:30 +0000

If I had to formulate in one sentence what my job and that of my teammates is, I'd say something like:

"To show our customers the business value of Oracle's Engineered Systems"

Because at the end of the day, customers pay real money only if there's some real value they see in a solution.

And that is the problem most people in IT struggle with: How is what you do in IT related to your company's total value chain?

Most of the time, people, both those working in IT and those selling and supporting into IT departments are consumed with functions and features, tech specs, standards and other tech stuff. Worse yet: Some people look at Oracle's Engineered Systems like Exadata and Exalogic and they only see a bunch of servers in a rack, because all they know is components, servers, tech stuff.

This is dangerous terrain: Because if you can't show the business value of your IT to your company, you're going to be put on the list of cost centers to be squeezed, and budget cuts are never a good motivator for your job.

So what is the value of IT to the business? Or more specifically, what is the value of Engineered Systems for our customers' businesses?

Two Kinds of Value

Essentially, there are two kinds of business values to look for when considering Engineered Systems: Saving money and making more money.

Saving money is what most people look for in IT departments. That is a historic, almost Pavlowian reaction to the question of business value, because IT budgets have been reduced for so many times due to dot com, economic or other crises.

The truth is that hardware and software are only a small fraction of total IT budget (somewhere in the 25% range depending on who you ask), while the bulk of IT spending is in implementing IT and operating it. The good news is that Engineered Systems are exactly all about helping you cut costs at exactly these biggest two factors of IT spending: Implementation and Operation.

However, IT budgets are limited and being able to save N% of them is even more limited. Therefore, the next category is much more interesting:

The other source of creating business value out of Engineered Systems is by making more money. By leveraging the breakthrough performance and other innovations of Engineered Systems, new things become possible that enable new business possibilities or provide a competitive edge that others simply don't have. Business opportunities are only limited by your market size, so this is a much more fruitful way of looking at the business value of your Engineered System.

Save money with Engineered Systems

Let's get the easy part out of the way first. Oracle's Engineered Systems are the ultimate standardized IT building block. Everything is integrated into a single system: Storage, servers, networking, virtualization, OS, database, middleware, applications. One building block, one point of management, one vendor, one throat to choke, one vision (insert song here).

This results in a number of cost savings opportunities, with varying amounts of potential depending on how your particular IT shop is set up:

Time to market savings: Since everything is pre-engineered, pre-tested, pre-assembled, and delivered in one go, there's a lot of project time to be saved with an Engineered System. No more testing, validation, evaluation, integration or other painful work to be done. How much is it worth to you if you could complete your IT project 3 months earlier?
TCO savings: We see a lot of consolidation projects where customers consolidate databases onto Exadata or SuperCluster and middleware and apps onto Exalogic or SuperCluster. Here's your laundry list of savings opportunities that this approach creates:
- Power, cooling, space: Put together all of your existing database/middleware servers, then calculate their total power consumption. Multiply by a factor of 2 for cooling and HVAC, then add any costs related to space that you pay inside your datacenter. Now compare with the equivalent database/middleware power sitting in Exadata, Exalogic and/or SuperCluster. This is where raw performance of Engineered Systems translate into savings for you: More database/middleware power with less machinery, every month, every year.
- Less cores means less licenses: The other good news of consolidation is that if you use more powerful hardware, you'll end up using less cores for what you have to do. This directly translates into license cost savings, which can be as high or higher than any cost you save on pure hardware.
- Less stuff to save: By using Oracle's Advanced Compression option in the database, or (in a more data warehouse related setting) Exadata's Hybrid Columnar Compression or the ZFS Storage Appliance's Compression/Deduplication features, you can save a lot of Terabytes in raw disk space. And every Terabyte saved on the database side means one extra Terabyte saved on the backup side, and one Terabyte less to recover in a DR scenario!
- Less stuff to administer: Count the number of administrators devoted to keeping your lights on in the datacenter. Subtract two. Multiply by 100k per year. That's the cost you'll save as a result of only needing two administrators to keep your hardware and OS running on Exadata, Exalogic or SuperCluster. Because everything on the infrastructure side is fully automated using Oracle Enterprise Manager Ops Center and Oracle Enterprise Manager Grid Control.
  Why two administrators? So one can go on vacation.
  Before you cry out something about killing IT jobs, let me remind you that it's no real fun to be the guy who's just sitting there, trying to keep stuff running between multiple vendors, endless hardware swaps, firmware incompatibilities, or installing an OS for the umpteenth time. You've always felt that this is just grunt-work that should be automated because you're meant to do something more interesting. Now it is! Go and learn about how you can leverage Infiniband for faster backups to a ZFS Storage Appliance, how Hybrid Columnar Compression can help your data warehouse environment or how you can introduce automation at a higher level into your datacenter.
- Less stuff to test/QA/patch: Even Exadata, Exalogic, SuperCluster and other Engineered Systems have to be patched and upgraded from time to time. The good news is that there's only one patch to apply to the whole system, and it has been already tested, quality-assured and certified for you. This means: Less work, less people and less time to test, QA and certify patches. Because this time, you don't need to check whether the database patch works with the OS, whether the OS patch works with the FC driver, whether the SAP patch works with the database and so on. Even SAP bundle patches now come together with the Exadata patch for single patch bliss. Again, this approach not only saves you time and headaches and personell, it also reduces the risk that results out of having to qualify many components from multiple vendors together.
Less risk: Never underestimate the monetary effect of having less risk. A colleague of mine knows about a partner who deployed an Oracle database stack on a standard Red Hat Linux machine on some standard hardware. There was a case of driver/OS/database patch mismatch that brought them down for a long time and later on they calculated that this incident cost them more than 40k Euros. Just because some storage driver didn't work with the OS and the database very well. This is exactly the stuff that is taken out of the equation at the engineering level, so the machine leaves the factory without this kind of problem. I'm not saying here that Engineered Systems are perfect, 'cause nothing is. But a huge chunk of potential failures has already been eliminated before your machine hits production. And that is sometimes, as some credit card company likes to say: Priceless.
Remove Bottlenecks: Many times, the value of Engineered Systems at our clients revolves around fixing bottlenecks in their IT. Such bottlenecks can quickly become business bottlenecks and that is never fun to deal with. Case in point: Huge telco company, recently merged with another one. Overloaded the call center. Batch jobs that were designed to run overnight couldn't complete. And as a result, call center operators didn't have any upsell opportunities prepared for their customers (= Less inceremental business) and customers started to cancel contracts due to declining service quality. Ouch. In this case, all that was needed was an Exadata environment for the data warehouse that reduced the database size by a factor of 8x and that accelerated the batch run tremendously so that all the bottlenecks could be removed. Now the call centers can work again much more efficiently and customer satisfaction plus new business due to upselling goes up.

Make Money Fast with Engineered Systems!

(sorry for the "fast" part, I just couldn't resist...)

On to the more interesting aspects of Engineered Systems: How to help your business make more money!

Oracle likes to highlight the breakthrough performance of Engineered Systems in our advertising. Just visit any airport and you'll see what I mean. For tech people, this is indeed impressive: 10x faster here, 20x faster there, and so on.

But how does a statement like "10x" faster translate into more business? The key is in finding things that were impossible before, but that you can now do as a result of using Engineered Systems:

Data Warehousing: In a typical data warehouse, batch runs often take a long time. By using Exadata, you can often shrink batch run times by an order of magnitude. How would your business decision process change, if you got your analysis daily, instead of weekly?
Business Intelligence: The new Exalytics In-Memory Machine comes with one Terabyte of RAM and it supports Hybrid Columnar Compression. This means you can now analyze Multi-Terabyte datasets in seconds that used to take days to analyze before. How would that kind of BI power change your day to day business decisions?
High Frequency Trading: Oracle Coherence is an in-memory, distributed Java database that is designed to scale across many nodes for very large, in-memory datasets. It's quite popular in the Financial trading scene because there, you try to analyze the risk of an investment in real-time so you can reach trading decisions very quickly. Like, milliseconds-quickly. In fact, each millisecond you can shave off of your risk analysis can be worth millions of Euros/Dollars/Yens as it enables you to reach an investment decision a bit earlier than your competition. Thanks to the Infiniband interconnect in Oracle Exalogic, that extra millisecond (or more) is now easier to get!
Moving into a Cloud Model: Cloud Computing is all about massive standardization, self-service and metering/billing. Engineered Systems are solving the standardization part, self-service, metering and billing are solved through Oracle's Middleware and management solutions. Now, you can turn your infrastructure into a Database-as-a-Service (DBaaS), PaaS and SaaS environment. Why is this not just consolidation but a new business generator? Because for the first time in IT history (actually, the second, if you count the mainframe), you can show the value of your IT services to your departments in Dollars and Pennies! And through self-service APIs, you can enable your departments to reduce their time to market, try out new stuff faster and generate better business models. This creates a nice upward spiral: The next time your company celebrates a business success, you'll be able to show exactly how your IT contributed to it, and how more IT helped close that business faster, better and with less risk. That will help you justify a bigger IT budget and so on.
New Business Insights and Opportunities through Big Data: Like I said in this article, Big Data is coming! The combination of Oracle's Big Data products like the Big Data Appliance, Exadata, Exalytics and a suite of useful software on top will enable you to make sense out of social media mentions, mobile sensor networks, real-time sales data and other very large sources of information with gold nuggets waiting for you to discover.

Now these are just a few examples of how Engineered Systems can help create new business opportunities or optimize existing ones. The rest is up to you, because you know better what your company is doing: What would you like to do from a business perspective, if you had enough IT power at your disposal?

Bonus: Fun Big Data Movies to Watch!

I guess we earned ourselves a fun video to watch now, right? Oracle has recently produced a series of advertising videos around big data that I think are quite good. Here's the first one:

See how our heroes' BigData story unfolds, including unusual renditions of their new "Big Data" colleague in this Oracle Big Data Video Playlist with 4 more videos waiting for you!

Your Business Experience with Engineered Systems

How have Engineered Systems helped you either save money, or create new business?
What is your biggest pain point in IT where more capacity or performance would help?
What is the biggest opportunity your business is facing that you would like to grab if you just had enough IT power?

Leave a comment below and share your experience!

Introducing Sparse Encrypted ZFS Pools

constant — Mon, 27 Feb 2012 13:39:57 +0000

Ever since I've been using a Mac, I enjoy using Sparse Encrypted Disk Images for a variety of tasks, for instance securely storing data that can be backed up somewhere else, say on a hosting server.

In fact, most of my project/personal data on my Mac sits on sparse encrypted disk images that are regularly rsynced to an external storage service, Strato's in particular.

The beauty of this solution lies in it simplicity:

Sparse encrypted disk images show up just like any other hard drive. But on the back end, they translate into a bunch of flat files that store all the data in an encrypted manner. By rsyncing the backing store, sparse encrypted disk images can be easily backed up across the net, while ensuring privacy and convenience.

Here's how to do similar things with Solaris and ZFS, including some extra data integrity magic:

So what we're looking for is a solution that is:

Encrypted, so data remains private,
Robust, so data survives even if the transmission/rsync/backup process is flawed,
Convenient, so data can be written or read easily on the home server side,
Finely-grained, so we can make our storage as big as we want, without worrying about file sizes.

The solution is to leverage a couple of Solaris mechanisms. Notice that this has not been blessed by Solaris Engineering, but it works for me and is fun to play with:

Using mkfile, we can create individual files of 1GB of size to store our data in.
We will mount these files using lofiadm as encrypted block devices, so they can be accessed by the system just like a regular disk would be accessed.
We can then combine a bunch of such lofi devices into a zpool using any of the redundancy schemes (RAID-Z2 in this particular case) to turn a bunch of files into a regular ZFS zpools
Later, we can either backup or rsync our backing store files to some cloud service without worrying about security (since all data is encrypted at the block level) and data integrity (since we leverage ZFS RAID-Z in an end-to-end manner). Even if our cloud storage service or whatever we use to backup out stuff with screws up the occasional file, we can get our data back since it's being RAID-Z2 protected.

Here's what to do:

Create a bunch of 1GB files to serve as backing stores:

  mkdir /export/szpools
  cd /export/szpools
  mkfile 1g szpool_1 szpool_2 szpool_3 szpool_4 szpool_5 szpool_6

Use lofiadm to turn those files into encrypted block devices:

  lofiadm -c aes-256-cbc -a /export/szpools/szpool_1
  Enter passphrase: 
  Re-enter passphrase: 
  /dev/lofi/1
  lofiadm -c aes-256-cbc -a /export/szpools/szpool_2
  Enter passphrase: 
  Re-enter passphrase: 
  /dev/lofi/2
  lofiadm -c aes-256-cbc -a /export/szpools/szpool_3
  Enter passphrase: 
  Re-enter passphrase: 
  /dev/lofi/3
  lofiadm -c aes-256-cbc -a /export/szpools/szpool_4
  Enter passphrase: 
  Re-enter passphrase: 
  /dev/lofi/4
  lofiadm -c aes-256-cbc -a /export/szpools/szpool_5
  Enter passphrase: 
  Re-enter passphrase: 
  /dev/lofi/5
  lofiadm -c aes-256-cbc -a /export/szpools/szpool_6
  Enter passphrase: 
  Re-enter passphrase: 
  /dev/lofi/6

Create a zpool out of those encrypted block devices:
zpool create szpool raid-z2 /dev/lofi/1 /dev/lofi/2 /dev/lofi/3 /dev/lofi/4 /dev/lofi/5 /dev/lofi/6

Now you can access the pool through regular ZFS means and at the same time backup or rsync the backing store files to some cloud/web hosting service.

We can now create more devices and add them to the pool to grow its capacity and set up a regular cron job that rsyncs all data into an external service for backup.

Should your local system go up in flames, you can restore the data on a new system, recreate the lofi devices and import the pool.

A few questions and answers:

Why not use ZFS encryption? Yes, ZFS encryption would work as well, but I prefer to use block encryption in this case because it more effectively hides the structure of the pool to the backup service. They should not know what's on the files, not even that they're being used as virtual block devices.
Can the asynchronous nature of rsync and other mechanisms compromise data integrity? Yes and now. As long as there's a regular rsync process running that backs up all block devices, data can be restored reliably. There are a couple of mechanisms in ZFS that help import the pool even if the underlying devices have been backed up/restored in an unknown order.
Why go through this trouble if you can also use zfs send/receive for backup? zfs send/receive is fine, but it creates very large files, which are unwieldly for your typical hosting service. They're also not encrypted. This is a much more compatible solution because all it does is map ZFS magic onto a bunch of 1GB files.
Shouldn't this be scripted? Yes, it should.

Three Enterprise Architecture Principles for Building Clouds

constant — Thu, 23 Feb 2012 10:49:00 +0000

After having gone through TOGAF training and certification, I've now caught the Enterprise Architecture bug, as you can probably tell by this article. It is a really neat way to add structure to the IT development process and to better understand what it really means to solve business problems with IT.

One of the first things TOGAF recommends architects do when establishing an Enterprise Architecture practice within a company is to formulate Architecture Principles that guide the development of solutions. During the last few workshops and during some discussions with other architects, three principles in particular struck me as being key to successfully developing a Cloud solution:

Cloud Architecture Principle #1: Standardization

Principle

When establishing a cloud solution, there should be only one standard, and it's the point of service where you standardize the kind of cloud service you provide to your cloud customers. No other standards are necessary or useful after that.

Rationale

As explained here, a standard is really only an interface you define for the purpose of providing a service to your customers. Nothing more, nothing less. Anything below that point is your responsibility and should not concern your customers, anything above the point of standardization is your customer's business and not yours.

It's the clear separation of Cloud provider and Cloud consumer through a standardized Cloud interface that enables the necessary independence you need in order to maximize efficiency of your Cloud service.

Implications

The standardized interface should be minimal: Just enough service to be useful and no more, to ensure flexibility and independence. The standard should also be implementation agnostic, so you're free to choose how to implement your Cloud service without being dependent on what your Cloud consumers do with it.

This is the magic behind Amazon-style Cloud optimization: They only give you a simple API, then optimize the heck out of it below their chosen API level.

Cloud Architecture Principle #2: Optimization

Principle

Optimize as much as you can below the point of standardization.

Rationale

This is key to your cloud's success: Cloud computing, from the point of view of the infrastructure supplier is nothing more than a ruthlessly optimized way of supplying a well defined (see point of standardization above) service. All bets are off here, as long as you focus on providing the most efficient implementation of your Cloud service.

Commercial break: This is the reason Engineered Systems exist: They're ruthlessly optimized cloud building blocks for well defined Database, J2EE and other services.

Implications

There's no need for or utility in adhering to limitations here. As long as you satisfy you Cloud service requirements, you're free to choose whatever technology, architecture or other means are useful in providing your service. This is good news for those who always wanted to break out of their "This is how it has always been done" mentality and like to start something new.

Cloud Architecture Principle #3: Separation

Principle

The implementation details of your Cloud service should always be hidden from your customers. They should never know how exactly you provide your J2EE, database or other service, and certainly never depend on the particular way you do it.

Rationale

This is basic layer separation hygiene: As soon as the consumers of your service start depending on some detail of your implementation, you lose the ability to change that detail because now you have to satisfy backward compatibility to a feature you haven't committed to in the first place. Therefore it's best to make it clear upfront what the minimum set of supported features are, and to hide any extra features that may or may not be present, so you're free from entanglement between service and implementation.

Implications

This also means you need some sort of filter that ensures any specifics of your Cloud service implementation are hidden from your consumers. If your J2EE service is implemented using, say, WebLogic, your consumers should never know, and in particular should never be able to tell. This is one of the beauties of Solaris Containers: You know what you get, but you never know how and where that Container has been implemented, so you can't really take advantage of any machine specifics.

Conclusion

So here they are: Standardization, Optimization and Separation. Three principles that you should keep in mind when designing a Cloud solution, and adhering to which can save you a lot of headaches.

The Difference Between a Standard and a Preferred Vendor

constant — Tue, 21 Feb 2012 14:39:30 +0000

Recently, I attended a customer workshop where the customer declared that they standardized on x86, VMware and Linux.

That got me and my colleague thinking about what standardization really means and whether that actually makes sense.

The workshop was actually about defining a PaaS platform for the customer, and early in the process they just said: Fine, but it's gonna be x86, VMware and Linux, because that's our standard. WTF?

What Standards are Not

x86 is not a standard, it's a processor architecture. VMware is not a standard, it's a hypervisor implementation. And Linux is not a standard either, it's an operating system.

J2EE is a standard, and the only one that is relevant to the particular project we were talking about: The goal was to create a Private PaaS Cloud for the customer, and J2EE is the standard that establishes the interface between the PaaS cloud as a service provider and their customers. Nothing else.

Once the interface between you and your internal customers is defined through a standard, it doesn't matter what processor architecture, what hypervisor or what OS you are using to deliver against that standard. At least not to the consumers of your service, because the purpose of defining an interface is to abstract away all implementation details from your customer so you get to choose the best possible way to provide your service!

Standardization is a Common Language

Standardization is about agreeing on a common language so you're free to use whatever you want to do your job. Standardization should not dictate your choice of CPU, Hypervisor or OS. It's that simple.

So the next time you say "We have standardized on x86", you're really saying: We have limited our choices of CPUs and stopped thinking about them. Think twice because you may be missing a lot of opportunity.

And that brings us to another important principle in building clouds: After you define the standard that abstracts your service away from your customers, there's no point in using any type of standardization at a lower level than that. This is best explained in this blog post about PaaS is not Middleware over IaaS.

Which means: If you're about to create a PaaS service, you should do whatever is necessary to make it as efficient as possible, without resorting to someone else's service. Because that would limit your choices, limit your service level and limit the opportunities you have in making your approach to delivering your service as efficient as possible.

So here are some good architecture principles for building clouds and establishing standards:

Standardize on one thing only: Your interface between you, the service supplier and your customers, the service consumers.
Don't accept any standardization below the level of the interface you provide, because it limits your choices and your opportunity to deliver quality.
Learn to distinguish standardization from choosing a preferred vendor. They are different things with different purposes.

And here's the plan on how to build clouds:

Define the service that you want to deliver as a standard. It could be Database as a Service, J2EE as a service or even dogfood as a service.
Make that service definition as product agnostic as possible. It should survice at least ten years of usage.
Stop caring about your service consumers (they have the manual, right?) and start optimizing the heck out of your service delivery stack, without any restrictions on what technologies, vendors or stuff you use.

Really, it's that simple. But it's not "We have standardized on x86, VMware and Linux."

Iron Sky: The Trailer is Here!

constant — Wed, 08 Feb 2012 22:18:50 +0000

In Summer 2010 I learned about a cool new geeky movie called Iron Sky that was crowd-funded. I decided to help finance it with a small sum.

Then the second Iron Sky teaser came out and I thought: Wow, this is looking really good! And I invested some more.

In December 2010, I was invited to attend the Iron Sky shootings in Frankfurt together with my brother. This is where we got to meet the crew, the actors, the people behind the movie and other investors. And I thought: Wow, this is not only cool, this is for real! Everybody was 100% determined to make this the coolest movie ever, and everybody put in so much attention to detail, love and true craftmanship that I thought: "Yep, this is going to be a true movie milestone!" And I also got to shoot an interview with the director and the inventor of Iron Sky. This time, my brother and I together invested a bit more to help this baby fly.

Fast forward to December 2011 (regrettable, a second trip to Iron Sky shootings in Australia didn't come together as I was already booked out from my regular job). This time, I got to see a beta version in Helsinki, and I brought home a Behind the Scenes Video Interview with the producer and the social media mastermind of Iron Sky.

Now, Iron Sky has been accepted for the 2012 Berlinale Festival starting next weekend. What a milestone!

Today, the official Iron Sky Trailer has been released:

In March, I'll be visiting the Iron Sky team again, for the Finnish Premiere in Tampere, Finland.

And the official world-wide release is going to be April 4th, 2012.

If you like Mars Attacks!, Inglourious Basterds and/or American Dreamz, then this is the movie for you! If you don't know/like these three movies, go see them, then think again.

Expect:

Glorious space battles with lovingly detailed animation.
Fun dialogues that will make you laugh from the very beginning.
Classy evil villains that are too cool to be good.
Very smart heroines that will seize the situation and rip their male counterparts' breaths.
Endless fun with ongoing American Presidency Election Campaigns.
To finally find out what "a true computer" looks like!
A new meaning to the Dawn of the gods ("Götterdämmerung").
...and much more.

If you're in Berlin next week for the Berlinale: Have fun!

If you're going to the Finnish Premiere in Tampere: Let me know!

If you're in Munich near April 4th, let me know, too!

Meanwhile, watch the trailer above and leave your comments.

I am a Mobile Sensor Network, Collecting Big Data

constant — Mon, 23 Jan 2012 15:40:48 +0000

Don’t worry, this is not a desperate attempt at SEO for my blog (although I do appreciate your likes, Tweets, RSS subscriptions and other ways you help me reach a wider audience), nor is this my entry into the latest contest of IT BS Bingo.

It just occurred to me yesterday that Big Data is everywhere. Even during your weekend jogging run.

Collecting Fitness Data, Step by Step, Heartbeat by Heartbeat, on Your Phone

For Christmas, I bought myself a Wahoo Fitness Key* and its matching ANT+ heart rate monitor (HRM)*. The key plugs into your iPhone and provides connectivity to the ANT+ wireless sensor protocol. The HRM is another dongle that straps around your chest and electrically registers every heart beat, then transmits the data to the Wahoo key. If you have an iPhone 4S, you can do without the key and just buy a Bluetooth HRM like the Wahoo BlueHR, because iPhone 4 supports Bluetooth 4.0 which includes a low power version of the protocol that supports sensor collection devices such as HRMs that run off of a coin cell.

So iPhone + Wahoo + HRM = Wireless Sensor Network. And if your idea of a network involves more than two participants, Wahoo also sells an ANT+ pedometer* to measure your stepping frequency along with heart beat data as well.

(Android users: I'm sure you'll find a similar solution for yourselves as well. I just happen to prefer quality over popularity.)

Running 2.0

Thanks to modern gadgetry, apps like iSmoothRun on my phone can now tell me how I'm doing while I’m running, including time, distance (thanks to GPS, which is another sensor), pace, cadence (using the phone’s accelerometer or a wireless pedometer*) and heart rate. I can also set up a target running profile (like “No more than 70% of max. heart rate so I can stay in the aerobic zone, please.”) and my phone will duck the music and tell me to slow down whenever I go beyond target heart rate.

Pretty cool.

Social Network Running

But we live in the age of web 2.0 so there's obviously more to do if you want to maintain your running geek-cred: The iPhone also collects all data (position, heart-beats, and steps) over time and at the end of the run, it will not only present me with my running statistics, possibly spiced up with current weather data etc., it will also offer to upload the data to one of the emerging fitness social networks, such as RunKeeper.com.

Sites like Runkeeper take the data and create web maps with my running path, complete with nice graphs that I can dive into for analyzing my own running behavior including altitude, pace, heart rate, cadence etc. They also collect other data such as weight and body fat percentage (yes, using a Withings Scale* for example, you can track weight/bodyfat data too, even data from a sleep tracking system* can be collected!) and show you your running (or fat loss) progress over time.

And thanks to social network goodness, you can run with friends over the network and compare statistics even if you’re not physically running at the same time. Or the same place.

And this is where Big Data comes into play, but what is it and how does it work?

The Advent of Big Data

The first time I heard about big data was during an internal workshop about the Sun Cloud in 2009 (you know, the old Sun habit of being way before our time). While we contemplated the implications of cloud computing for enterprises, someone mentioned that this would be nothing compared to the implications of Big Data. Back then, Big Data was reserved to web giants like Google and Yahoo! and the occasional large research institute such as CERN.

Big Data is the art of handling (surprise!) large amounts of data. “Large” can be anywhere starting at a dozen of Terabytes or a couple of Petabytes or any large number that no-one in their right mind would place into a single database on a single server.

Big Data has been made popular by innovations from web companies like Google, Yahoo, Facebook or Twitter, who pioneered new ways of handling huge amounts of data.

Today, Big Data is about to cross the chasm* from the domain of a few innovators and early adopters to the early majority, as businesses start to realize its value.

The Four V’s of Big Data

Big Data is typically associated with four V’s:

Volume, meaning lots and lots of data from sensors, devices, social networks, the web, retail offices, mobile fleets, etc.,
Variety, meaning there’s no predefined structure in the data that one can rely on: Unstructured data. This is the main differentiator against classic data ware-housing, which is strictly structured.
Value, meaning that somewhere within that data, there is some valuable information to extract, though most of the pieces of data individually may seem valueless.
Velocity, meaning quick turnaround cycles, quick, almost real-time processing and also short innovation cycles. Fail fast, fail often is the mantra, until you hit data gold.

RunKeeper and Big Data

Let’s come back to our running example: RunKeeper is a Big Data company because it collects GPS, heart rate, cadence and other data from its millions of users. Assuming that only half of their 6 million users actually use the service for real, and that they run once a week and assuming a data size of 50 KByte per run (including GPS positions), we get 7.8 TByte of data per year. This is not a lot by Big Data standards, and it is quite structured, but when you combine this data with Tweets, Facebook status updates, other exercise data and nutrition/sleep data (RunKeeper does all of the above), then data volume easily increases to more than 10 TB per year, which is quite a lot to wade through.

And if you start counting records, the complexity is overwhelming: Each GPS sample is about 100 Bytes, which means that RunKeeper’s 10TB per year translates into roughly 100 Billion records to correlate, analyze and create meaning from.

What meaning?

The Meaning of Big Data

And that is the goal of Big Data: To create meaning out of billions of records that seem so innocent, if looked at individually. In the RunKeeper example, they create graphs of your running history and help you analyze and optimize your fitness either for free or as a paid, “pro” service. And thanks to their Health Graph API, an eco-system of other applications and companies emerges who slice and dice RunKeeper’s data in other creative ways, trying to create valuable (and monetizable) meaning out of it. Example: World-Rank.in collects data from RunKeeper and Twitter, then ranks runners into its own top 30 lists.

Other companies use Big Data to identify patterns in their customer’s behavior, find threats or opportunities to act upon, or simply alert hospitals that a new flu epidemic is about to hit them.

How Big Data Works

Most Big Data use cases work around the same pattern:

Aquire: Data is collected. Speed and scalability is critical here, not necessarily high availability. It’s ok to be offline for a few minutes, even hours, or to lose the occasional data record, but it’s important to catch as much as possible. The Hadoop framework and file system and/or NoSQL databases are key tools at this step.
Organize: To make large amounts of data manageable, a divide and conquer approach is taken: Data is mapped to some interesting nomenclature/metric/attribute (For example: is this a positive of a negative tweet? What company is this tweet about? Was that run a new record or a below-average result?), then the data is reduced into a more condense form (“Number of negative tweets that mention our company”, “Fastest 10k runs per country”, etc.). These two steps are the key in the MapReduce framework and can be used repeatedly and creatively to compose a new, more valuable data source, like: “Top 10 keywords associated with negative feelings towards your company.”, or “Top runners per age category and country”, or “Best training improvement over the last year”, or even crazy stuff like “Fastest music to run with”.
Analyze: With that kind of data and processing at your fingertips, new kinds of insight are possible that can be analyzed and acted upon. Which flights are delayed and carry unhappy social media celebrities so you better take good care of them? How about sponsoring your top runner in a certain category and create a celebrity out of her/him? Of course, valuable analysis can also be monetized. Perhaps some research institute or some sports company is interested in getting access to all that heart beat, speed and nutrition data (anonymized of course)?

Oracle and Big Data

Don’t worry, this commercial break will be brief, but interesting:

Oracle’s big strengths of course are in handling commercial data warehouses and analyzing business information data, as well as building Engineered Systems that remove the pain of setting up an IT shop while optimizing the usefulness you get from your systems.

Big Data’s strength lies in its innovation to handle and organize unstructured, large data sets, through the Hadoop filesystem, the MapReduce framework, the R statistical language and other emerging technologies. But analyzing data after these steps is still in it infancy.

By combining the worlds of Big Data, Data Warehousing and Business Intelligence, running on Engineered Systems, Oracle can offer unique value to businesses who want to leverage Big Data for their benefit, without going through the trial/error/research of running their own Big Data development operations.

Learn more from Oracle’s Big Data White Paper, it’s really good, and check out Oracle's Big Data home page.

Building your own Sensor Driven Big Data Collection Network

As you can see, Big Data is fun and healthy. Here are some gadgets* to get your own Sensor Network based Big Data collection infrastructure set up that feeds into RunKeeper and other Big Data collecting social networks for your analytical pleasure:

Big Data and You

What are your favorite Big Data examples? Do you see Big Data being used in your company? Have you played with collecting, organizing and analyzing Big Data yourself? Leave a comment and share!

Finally, here's a video that shows the beauty of collecting, organizing, analyzing and visualizing of Big Data:

And if you want to see my own small chunks of running data, feel free to join my Street Team on RunKeeper.

Disclaimer: Neither me nor Oracle are affiliated with RunKeeper (Not that I know of). I just think it’s a cool service.

*Disclosure: Some product links in this article are affiliate links. If you buy through them, I’ll get a small kickback to help with hosting costs for this blog at no extra charge to you.

Engineered Systems and Enterprise Architecture (or: How to Sell Dog Food Online)

constant — Mon, 16 Jan 2012 16:21:45 +0000

One of the first things that customers and sales teams realize when dealing with Engineered Systems is: They fundamentally change the IT architecture of a business.

Change is good, it means progress. But change is sometimes seen as a bad thing: Change comes with fear.

The truth is that Engineered Systems really empower IT architects to add value to their business, application and data architectures, without worrying about the technology architecture.

To understand this, we need to dig a bit deeper into Enterprise Architecture, specifically the TOGAF flavor of it.

One of my first tasks as a member of my new team was to get TOGAF certified. TOGAF is “The Open Group Architecture Framework” which is a fancy name for a giant document full of definitions, models, and methods for creating IT architecture out of a given business capability need.

Let’s look at a simple example: Suppose you’re the owner of a big dog food business empire. Your core competency is to know all about creating the best dog food, knowing your customer’s dog feeding preferences and habits, and having the best suppliers of dog food ingredients lined up from partners to help you excel at selling dog food.

And now you want to take your business to the next level and start selling dog food online. Here’s the typical progression of steps, assuming your company has successfully introduced TOGAF as your method of doing Enterprise Architecture, according to the TOGAF Architecture Development Method (ADM):

TOGAF ADM for Dummies

Architecture Vision: Your boss told you to come up with an architecture for adding an online shop to your dog food business. The first step then is to create an architecture vision: How does it look like to sell dog food online? Who are the stakeholders for such a capability and what are their concerns?
Business Architecture: Your enterprise architects sit down through numerous meetings with your business people, identifying a business architecture for selling dog food online. This is a bunch of diagrams, catalogs and matrices describing exactly how your online shop capability ties into the rest of your dog food business: What’s your online dog food business model? What are the use cases you want to support through your new capability? What existing processes does the new capability tie into? What new processes need to be defined to add the online shop capability?
The output of this stage uses business building blocks: Business processes and capabilities, modeled at a very high level, and technology-agnostic, specific to your enterprise.
Information Systems Architecture: Your enterprise architects translate the business architecture created in step 2 into an information system representation, which is composed of two architectures, a data architecture and an application architecture:
1. The Data Architecture describes what kind of data you need to take care of for your online shop capability. Customer data, product data, payment data, shipping data, dog pictures, food pictures, dog food ingredient lists, all of it. It also describes where the data comes from, where it flows to, who needs what data when and so on. We see two types of data here: Structured data (what you'd place into a database) and unstructured data (i.e.: flat files).
2. The Application Architecture describes what applications you need to handle your data: Web portals, CRM applications, supply chain management applications, shipping and tracking applications, order management applications, payment processing applications, dog food personalization and mixing order processing applications etc.
  More diagrams, more catalogs and more matrices are created to describe this stage of building your dog food online shopping empire. Notice that there are no particular technologies and no products considered at this stage either.
The building blocks at this stage: Applications, database schemas, flat file specs. All particular to your enterprise.
Technology Architecture: The next step is to translate the information systems architecture into a technology architecture. Now we’re getting somewhere: This is where our enterprise architecture heroes get to describe what systems are needed where, how to map applications and data to those systems, what networks to use, which servers are connected how at what locations, etc.
And here comes the cut: The building blocks are now generic! A database server is a database server, no matter what it stores, be it dog food recipes or car configurations. An app server is an app server, no matter what app runs on top of it, same goes for a file server, a firewall or a router. All components of Technology Architecture are generic, there's hardly anything specific about selling dog food online here.
Opportunities and Solutions: This is the earliest stage at which particular technologies (such as the Oracle database) come into play and where specific instances of a technology ("Our main CRM system running on Oracle on SPARC SuperCluster.") are mentioned.
Migration Planning: Here's where the hand-over from the enterprise architects to the project managers occurs: Implementation and migration plans are forged, projects are created and prioritized, resources are allocated, hands become dirty.
Implementation Governance: This stage ensures synchronization between Enterprise Architecture and its implementation, as carried out by you project managers, through running reviews and monitoring progress.
Architecture Change Management: Finally the project is done, the dog food online shop goes live and hopefully the first orders come in. Parties are thrown and people get drunk. The next day, lessons are learned and captured, and fed into the next ADM cycle, starting again with an architecture vision. Perhaps the beta users identified a few bugs or RFEs, or the product managers have new ideas for what else the online dog food store could sell, but that would require a new capability to be added to the architecture: The cycle starts again.

The Limits of Creating Business Specific Value

What does this have to do with Oracle's Engineered Systems? The answer is that it's all about who creates unique value at what level of the architecture and who doesn't:

Steps A-C are highly specific to the enterprise: This is where enterprises truly innovate: The first online shop exclusively for dogs! Online dog food customization! Geo-aware local dog food sourcing! The value at these stages is unique to the particular enterprise, or at least the vertical business it operates in.
But step D (Technology Architecture) is different: All building blocks here are generic, and there's hardly any value for a typical enterprise to create here that differentiates themselves from other enterprises. How is a database cluster developed for a dog food database different in design than a database cluster developed for a car configuration database? Or why would a file server for hosting dog pictures be significantly different from a file server for hosting book covers?
All other steps starting from 4 follow the same pattern: Building blocks, best practices and implementation details are the same, there's nothing specific to the particular business that they support. They all need to be secure, performant, reliable and more or less highly available. All business use the same trucks to deliver their goods, only very specialized business may occasionally need a special truck, but they're quite a minority.

And yet, businesses spend enormous amounts of resources for coming up with their own custom made database cluster design, their own custom made app server farm design, their own custom made file server infrastructure design, their own custom made web infrastructure design and so on.

This is where the rise of Engineered Systems takes place.

The Engineered System: TOGAF Stage D in a Box

Oracle's Engineered Systems (you know, Exadata, Exalogic, SPARC SuperCluster, ZFS Storage Appliance and so on) are nothing more than the TOGAF stage D of the ADM in a few, easy to plan, buy, install, manage boxes, without the usual headaches that used to occur when planning Technology Architecture.

Want a new database for your dog food recipes? Just ask the DBA to allocate one for you out of an Exadata. Want to host the entire chain of app servers and portal that forms the application layer of your dog food shopping empire? Push a button on your PaaS portal to instantiate a virtual assembly on one of your Exalogic boxes. Want something more powerful? Perhaps with more transactional crypto-oomph? Ask you friendly PaaS portal for a slice off of your shiny new SPARC SuperCluster. Wanna analyze heaps of personalized dog food recipes and correlate them with mailman assault reports, dog flu epidemic data and weather forecasts? That's what the big data appliance and Exalytics are for.

The point here is: No business should spend time, money and resources for creating yet another personalized flavor of a RAC cluster, an app server tier or a web portal infrastructure. This is what IT companies like Oracle can do better than anyone else: Design Technology Architecture.

Instead, use your energy and resources to create real business value. What's your online dog food architecture?

P.S.: Please forgive my dog food analogy. Here's the Guy who planted it into my mind:

TOGAF ADM picture taken from Wikipedia, used under NASA public domain policy.
Dog picture by digital_image_fan on Flickr, used under Creative Commons license.

Iron Sky Update: A Behind the Scenes Video Interview

constant — Tue, 27 Dec 2011 19:58:13 +0000

If you've been following my blog or the Systemhelden.com podcast for some time, then you probably know that I'm a big fan of and small investor in the movie Iron Sky, a crowd-funded science-fiction comedy about the Nazis who went to the dark side of the moon in 1945 and come back to conquer Earth in 2018.

A few weeks ago, I had the privilege of attending a pre-screening of the movie's beta version in Helsinki, Finland. What can I say? It exceeded my high expectations!

The script is ingeniously written, just the right mix of humor, action, character and political satire. All actors are brilliant: Götz Otto, Christopher Kirby, Tilo Prückner and Udo Kier gave the movie a unique blend of coolness, style, creepiness and comedy while Julia Dietze and Peta Sergeant stood out in excellently played character roles. Special kudos to Stephanie Paul for being the most hilarious US President in movie history!

Director Timo Vuorensola found just the right balance between comedy, action and the occasional emotional moment. The whole production has a very high quality feeling, despite the limited (by hollywood standards) budget of about 7.5 Million Euros.

Video Interview

After the pre-screening, I had the opportunity to interview Tero Kaukomaa, producer of Iron Sky and Pekka Ollula, Social Media Manager. We talked about the audience's first impressions, a day in the life of a movie producer, and some cool new side projects such as the Iron Sky Comic Book or the brand new Iron Sky board game.

Here’s the full interview (sorry for the slightly shaky camera):

Wanna see more videos? Check out an interview with the fathers of Iron Sky from last year. Or go watch some cool Iron Sky teasers.

Participate in Iron Sky!

Iron Sky is scheduled for world-wide release in April 4, 2012. If you want to participate, you still can:

There’s some room for a few more fan investors: Check out the Iron Sky Finance page for prerequisites to becoming an investor in Iron Sky and become part of the biggest crowd-funding effort in movie history!
For only EUR 1 (or more), you can take a sneak peek at the full production process of the first 5 minutes of the movie, from script to storyboard to shooting to special FX and final result.
If you want to see your company logo on Times Square within the movie, there may still be an Iron Sky Times Square sponsoring opportunity left!
Every T-Shirt, board game or other fan item bought in the Iron Sky Store helps promote the movie while contributing to the production budget.
Spread the word! Visit the “Demand Iron Sky” page and help sell the movie to distributors, like the Iron Sky page on Facebook, follow Iron Sky on Twitter or just tell all your friends. With more than 80,000 fans on Facebook, the movie is already a social media phenomenon, let's reach 100k fans on Facebook by the time it reaches the theaters!

Special Constant Thinking Iron Sky T-Shirt Contest!

Win an official Iron Sky T-Shirt by promoting this interview! Just tweet (or retweet) a link to this article along with the #ironsky hashtag and enter a draw to win a t-shirt!

On January 15th, I’ll randomly draw the lucky winner out of all participants and send her/him an official Iron Sky t-shirt.

Disclaimers, Disclosures and Stuff

Standard disclaimers apply for the T-Shirt contest, this is no official promo, no legal hassles please, this is just for fun among friends.

Disclosure: I'm a proud investor in Iron Sky which means that I'm biased, have a commercial interest in this project and that I want you to see the movie and tell all your friends about it.

Update (2012-01-01): A stupid bug killed the intro part of the article, restored from a draft. Also corrected the praise for Julia and Peta since my poor English didn't do them enough justice before :).

Update (2012-01-06): Corrected the year to 2018. In 2038, earth will face a different problem. Thanks to Max who pointed that out in the comments!

Update (2012-01-20): The results are in!

Today I determined the lucky winner of the Twitter #ironsky retweet contest. 15 different people retweeted this article before or on January 15th, here's the list:

00 @TAKIM0T0
01 @sascha242
02 @MThorr
03 @zarkon
04 @Okona
05 @bayoda
06 @sunian314
07 @ironskyfilm
08 @optionstrict
09 @BrodyK91
10 @shift_plusone
11 @jenksho
12 @digisubculture
13 @apfelgriebs72

Special mention to @deltabps who retweeted this article, but was a bit late. So 14 people altogether qualified for the t-shirt draw.

Here's the first winner, determined by a roll of Unix dice:

constantins-macbook-pro(2):~ constantin$ expr `cat /dev/random | head -c 1 | od -t u1 | head -1 | cut -c 11-` % 14
4

Congratulations @Okona!

Now, I personally know Okona who is a regular guest at the HELDENfunk podcast, so I didn't feel good about keeping the t-shirt in the family so to speak, so I decided to give away a second t-shirt! Here's the second roll of dice:

constantins-macbook-pro(2):~ constantin$ expr `cat /dev/random | head -c 1 | od -t u1 | head -1 | cut -c 11-` % 14
12

Congratulations @digisubculture!

Damn, again someone I know personally from my university days in Clausthal :).

Ok. I'm rolling again until I get someone I don't know. Here's lucky winner #3:

constantins-macbook-pro(2):~ constantin$ expr `cat /dev/random | head -c 1 | od -t u1 | head -1 | cut -c 11-` % 14
13

Congratulations @apfelgriebs72!

That's it, congratulations to you all! I'll send you some DMs now and ask for your shipping addresses so I can send you your T-Shirts.

But wait! Since Twitter and I aren't infallible, there's a slight chance that someone who re(tweeted) a link to this article with the #ironsky hashtag before or on January 15th didn't come to my attention and so wasn't included in the above list by pure bad luck. If that is you, then send me a link to your Tweet using the contact form of this site to prove it and I'll send you another T-Shirt!

(Just to avoid getting broke due to very unlikely events, this offer is only valid for the first three people to qualify for this, and only for the remainder of January 2012.)

The Rise of Engineered Systems

constant — Wed, 07 Dec 2011 14:11:38 +0000

I changed into a new role at Oracle: I now work for the EMEA Engineered Systems Architecture Team (ESAT). We support Oracle’s EMEA Engineered Systems business by engaging with customers, enabling our field organization with trainings and through evangelization.

You can call me biased towards Engineered Systems now, but that would be like accusing a Mac fanboy of suffering from the Stockholm Syndrome, when it’s actually the other way round.

The other side of the “biased” medal really is that I have a choice of where I want to work, and one of the reasons I changed from my cozy SPARC/Solaris Technology camp to the Engineered Systems crowd is: I believe the world of IT is changing.

Let me explain.

Back to the Future

Once upon a time, when computers were new, they came as a whole: You bought yourself a “computer” and it came with everything you needed to get started: Hardware to compute with, storage to store data with and software to work with. Those times culminated in the rise of mainframes that offered everything a business wanted to do with computers as one whole, integrated, single package.

Even through the 70s, when mainframes got replaced by minicomputers, they still came as an integrated whole: Storage, Computing hardware, OS and development tools with libraries, what we would today call “Middleware”. (For a fascinating travel in time back to the minicomputer era, read “The Soul of a New Machine”*).

But then the integrated approach was seen as too proprietary and too limiting, which gave rise to a new wave of systems in the 80s: Open Systems. Now, vendors offered the pieces (You know: Storage, Servers, OS, Tools and Middleware, Applications, Networking and so on), while businesses (or system integrators, for a fee) put the pieces together into a customized whole.

30 years later, we’re still living in the 80s, when it comes to running an IT project.

A Typical IT Project

So how does a typical IT project work?

Someone has an idea for a new business capability.
“We need an app store for our iToaster company!”
Bright people create an IT architecture to support that new capability.
“Here's our blueprint for a private cloud based, full-featured, interactive, augmented reality, HTLM5, AJAX iToaster App Store system, complete with CRM integration, web portal, BI engine and ToastHabits™ loyalty card integration!”
A round of RFPs goes out to procure the components for the new system: Storage, Servers, Networking, Virtualization, OS, Database, Middleware and some Applications.
Some of these are standardized, but we still end up with more than a handful of them, multiplied by our company’s list of n>1 favorite vendors to play against each other.
RFP responses are evaluated.
Technologies are studied, claims verified, proof-of-concepts are run, pilots are conducted and so on. Lots of traveling, playing, arguing.
Finally, the components of the IT system are decided on.
The result tends to look like a camel (which allegedly is an animal that was created by a committee): The number of vendors involved is only slightly smaller than the number of components across the storage-to-application stack, thanks to elaborate purchasing rules, standardized buying practices, old relationships or just habits of the people involved in the process.
Then the pieces come in and the IT department gets to play.
Storage is set up and plugged to servers through some network, servers are set up and provisioned with virtualization and operating systems, databases are set up and connected to middleware which is being rolled out to support the actual applications that matter and so on.
In the process of putting the components together, problems emerge.
The storage doesn't like to play well with the server, which turns out to be a firmware issue after weeks of finger-pointing between vendors and countless hours on their hotlines. The OS needs a specific patch and some settings in /etc/system to support the right version of the database so it can optimally talk to the middleware, but there’s still a network latency or routing issue which drags down performance because the NAS device tends to overload specific segments more than the middleware communication to the database so different networking paths need to be set up, and so on.
Finally, after months of delay the new IT system is running.
Kinda. A bright guy came up with a creative solution so the project doesn’t have to be delayed again. The “solution” also compromises security a little bit, but that’s a small price to pay for cutting weeks or months off time-to-market and reach the huge potential of hundreds of thousands of new customers (and their credit cards) and turn them into revenue through the magic of the new system. And we’ll fix the security bugs later. When we have the time.
Only, it is not running very well, yet.
Performance problems arise, once more than 10,000 customers try to place orders at the same time, which happens on the very first day because the new service was announced with a huge marketing campaign. On the management floors, biological end products hit rotational dynamics which only worsens the problem because now even more people try to use the new system just to see what the problem is. The press isn’t helping either. The test department guy says they shouldn’t have brought the system online without running any further tests while the business people point at charts with "opportunity costs" due to the already mind-blowing project delays.
I’ll spare you the details of the first successful hacking attempt.

True, this is exaggerated for reasons of entertainment, but things like this happen all the time in current, “modern” IT shops. I've seen them all in two decades of working in IT and I’m sure you’ll recognize a few of them, too.

The point is:
Building IT systems is complicated, time-consuming, error-prone, unpredictable, resource-intensive, expensive and risky.

Or, more shortly:
The way we build IT today is broken.

The Return of the Engineered System

Wouldn’t it be nice if you could just buy a full system that comes with everything you need to get started on your new business idea? A system where storage, networking, servers, virtualization, operating system, database and middleware, applications are engineered together, tested together, certified together, delivered together, managed and patched together?

Unpack, set up and get started in days, not months. Roll out your application without the risk of it blowing up due to unforeseen incompatibilities. Profit from better performance because all components are pre-integrated, so they can use a faster networking technology than the generic standard. Less suffering from bugs because most of them have been fixed already at the factory level rather than at the customer site. Less support suffering because the guy at the other end of the phone has access to exactly the same system and can therefore understand your issue much better. Better security because everything has been designed together with security in mind and in a pre-packaged way with controlled interfaces in and out of the system as a whole. Shorter time-to-market because steps 3-10 of the above process are reduced into one step that only takes a few days: Unpack and configure your Engineered System.

That's what Oracle’s Engineered Systems are about.

It sounds similar to the early times of computing, where the 10-step process from above was just two:

Buy computer.
Get to work.

And that’s the goal of Engineered Systems: Get rid of the maddening complexity of building IT systems and let the people sort it out that do this actually for a living, 100% of the time and with much better expertise.

And the goal for you is to get to work on what you are best qualified for. Building App Stores. Selling stuff through e-commerce. Collecting big data and finding sense therein. Running social networks. Figuring out where the economy is going. Making sure that parcels go from A to B on time during the christmas season. And so on.

But isn't this going back to monolithic, proprietary and closed systems?

Of course it is not. The thing is that we're getting the best of both worlds:
Simple IT deployment, and open standards with open interfaces across all components of the stack.

There is no lock-in: All software that comes with Engineered Systems is identical to the software that you can buy independently as a component of your 10-step home-brew IT deployment process. If you don't like the Engineered System, just build your own IT and move your stuff onto it, while keeping the same OS, middleware or database software.

Yes, there’s the occasional tool that is specific to the Engineered System it ships with, but these are extra components for convenience, like the customized dashboard on your Volkswagen that doesn't make sense if you duct-tape it onto your Toyota. Which brings us to:

Analogies Galore!

Engineered Systems are often compared to cars. And the analogy is striking: When was the last time you ran RFPs for tires, engines, transmission, seats, dashboards, chassis and bodywork?

Indeed, when cars were invented, each car was individually built to the driver’s specs. Until Henry Ford came along, and the rest is history:

Any customer can have an Engineered System painted any color that he wants as long as it is gray.

A friend of mine came up with a much better analogy: Beer!
Ever tried brewing your own beer? Sure, a liter of home-brew is cheaper than a liter of your favorite beer brand, but good luck going through the elaborate process of setting up, cleaning, roasting, fermenting, brewing and bottling your own beer. One word: Hygiene.

The result can taste great, but often enough, something (usually bacteria related) goes wrong and you end up with a beverage that tastes, well, home-brewn. And then you're sitting on 20 liters of something that your friends politely called “interesting” and that you’re too proud of to dump into the sewer for a few bottles of good, proper German beer.

Which is not far from “interesting” IT solutions we see a lot, their creators being too proud to replace them with something that actually works.

Engineered Systems are the Bavarian beers of IT.

You can find a great analogy much closer to traditional IT: Just walk into any coffee shop, better yet, visit any conference and look at the number of MacBooks vs. laptop PCs, iPhones, vs. Android phones, and of course iPads. Particularly their users and the amount of “work” vs. “fix” they do on their devices.

The iPad is the ultimate consumer Engineered System: Hardware and software and applications merged into a single, optimized system that lets you “just work”.

Change is Coming

Expect professional IT to change the way the car industry changed, or the way Apple changed the consumer industry: Companies are starting to build IT out of large, pre-engineered building blocks, vs. re-inventing wheels at the component level.

When was the last time you pondered whether to use Hitachi, Seagate or Conner drives for your storage project? Do you really care about the chip sitting inside your networking card or storage controller? Are you still busy testing, sourcing and integrating hardware and software components while your colleagues are treating the system as a component?

The focus of professional IT has now been elevated: From building systems out of components to building clouds out of pre-engineered systems. That’s where IT is going in 2012.

So What Do They Look Like?

Oh, I almost forgot. This is not a product pitch, so I’ll be brief:

Exadata is the Engineered System for running Databases.
Exalogic is the Engineered System for running Middleware and Applications, including SAP. (And one of my first duties as part of my group was to write a white paper about “Running SAP NetWeaver on the Oracle Exalogic Elastic Cloud”)
The Oracle Database Appliance is the easiest to use, all-in-one database appliance for small and medium enterprises or individual departments.
The Big Data Appliance makes it easy to pre-process Big Data for your database, NoSQL included.
Exalytics is the Business Intelligence System that leverages 1 TB of RAM to slice and dice huge amounts of data at lightning speed.
SPARC SuperCluster is the most flexible Engineered System: It comes with both Exadata and Exalogic technologies to boost database and middleware performance within the same box, and it can run any other application, too. A full Enterprise IT stack in a single box. I can’t wait to see SAP certified on this baby!
The ZFS Storage Appliances are part of Oracle’s Engineered Systems, too. The one-stop shop for a wide range of storage requirements. It's also used within Exalogic and SPARC SuperCluster as a common repository for compute nodes.

Your Take on Engineered Systems

What do you think? Do you see how Engineered Systems change IT? Or do you still see value in building stuff from scratch, over and over again?

What are your experiences with Engineered Systems? What aspects of Engineered Systems would you like to see covered in this blog?

Drop me a few lines of comment below, or send me mail. Let’s discuss!

*: Amazon affiliate link. Buy cool books at no extra charge, and I’ll get a small kickback. We both win!

Car picture by Flickr user flamesworddragon, used under Creative Commons License.

Solaris 11 Launch Blog Carnival Roundup

constant — Sat, 12 Nov 2011 16:31:27 +0000

Solaris 11 is here!

And together with the official launch activities, a lot of Oracle and non-Oracle bloggers contributed helpful and informative blog articles to help your datacenter go to eleven.

Here are some notable blog postings, sorted by category for your Solaris 11 blog-reading pleasure:

Getting Started/Overview

A lot of people speculated that the official launch of Solaris 11 would be on 11/11 (whatever way you want to turn it), but it actually happened two days earlier. Larry Wake himself offers 11 Reasons Why Oracle Solaris 11 11/11 Isn't Being Released on 11/11/11.

Then, Larry goes on with a summary: Oracle Solaris 11: The First Cloud OS gives you a short and sweet rundown of what the major new features of Solaris 11 are.

Jeff Victor has his own list of What's New in Oracle Solaris 11.

A popular Solaris 11 meme is to write a blog post about 11 favourite features: Jim Laurent's 11 Reasons to Love Solaris 11, Darren Moffat's 11 Favourite Solaris 11 Features, Mike Gerdt's 11 of My Favourite Things! are just three examples of "11 Favourite Things..." type blog posts, I'm sure many more will follow...

More official overview content for Solaris 11 is available from the Oracle Tech Network Solaris 11 Portal. Also, check out Rick Ramsey's blog post Solaris 11 Resources for System Administrators on the OTN Blog and his secret 5 Commands That Make Solaris Administration Easier post from the OTN Garage.

(Automatic) Installation and the Image Packaging System (IPS)

The brand new Image Packaging System (IPS) and the Automatic Installer (IPS), together with numerous other install/packaging/boot/patching features are among the most significant improvements in Solaris 11.

But before installing, you may wonder whether Solaris 11 will support your particular set of hardware devices. Again, the OTN Garage comes to the rescue with Rick Ramsey's post How to Find Out Which Devices Are Supported By Solaris 11. Included is a useful guide to all the first steps to get your Solaris 11 system up and running.

Tim Foster had a whole handful of blog posts lined up for the launch, teaching you everything you need to know about IPS but didn't dare to ask: The IPS System Repository, IPS Self-assembly - Part 1: Overlays and Part 2: Multiple Packages Delivering Configuration. Watch out for more IPS posts from Tim!

If installing packages or upgrading your system from the net makes you uneasy, then you're not alone: Jim Laurent will tech you how Building a Solaris 11 Repository Without Network Connection will make your life easier.

Many of you have already peeked into the future by installing Solaris 11 Express. If you're now wondering whether you can upgrade or whether a fresh install is necessary, then check out Alan Hargreaves's post Upgrading Solaris 11 Express b151a with support to Solaris 11. The trick is in upgrading your pkg(1M) first.

Networking

One of the first things to do after installing Solaris 11 (or any operating system for that matter), is to set it up for networking. Solaris 11 comes with the brand new "Network Auto-Magic" feature which can figure out everything by itself. For those cases where you want to exercise a little more control, Solaris 11 left a few people scratching their heads. Fortunately, Tschokko wrote up this cool blog post: Solaris 11 manual IPv4 & IPv6 configuration right after the launch ceremony. Thanks, Tschokko!

And Milek points out a long awaited networking feature in Solaris 11 called Solaris 11 - hostmodel, which I know for a fact that many customers have looked forward to: How to "bind" a Solaris 11 system to a specific gateway for specific IP address it is using.

Steffen Weiberle teaches us how to tune the Solaris 11 networking stack the proper way: ipadm(1M). No more fiddling with ndd(1M)! Check out his tutorial on Solaris 11 Network Tunables.

And if you want to get even deeper into the networking stack, there's nothing better than DTrace. Alan Maguire teaches you in: DTracing TCP Congestion Control how to probe deeply into the Solaris 11 TCP/IP stack, the TCP congestion control part in particular. Don't miss his other DTrace and TCP related blog posts!

DTrace

And there we are: DTrace, the king of all observability tools. Long time DTrace veteran and co-author of The DTrace book*, Brendan Gregg blogged about Solaris 11 DTrace syscall provider changes. BTW, after you install Solaris 11, check out the DTrace toolkit which is installed by default in /usr/dtrace/DTT. It is chock full of handy DTrace scripts, many of which contributed by Brendan himself!

Security

Another big theme in Solaris 11, and one that is crucial for the success of any operating system in the Cloud is Security. Here are some notable posts in this category:

Darren Moffat starts by showing us how to completely get rid of root: Completely Disabling Root Logins on Solaris 11. With no root user, there's one major entry point less to worry about.

But that's only the start. In Immutable Zones on Encrypted ZFS, Darren shows us how to double the security of your services: First by locking them into the new Immutable Zones feature, then by encrypting their data using the new ZFS encryption feature.

And if you're still missing sudo from your Linux days, Darren again has a solution: Password (PAM) caching for Solaris su - "a la sudo".

If you're wondering how much compute power all this encryption will cost you, you're in luck: The Solaris X86 AESNI OpenSSL Engine will make sure you'll use your Intel's embedded crypto support to its fullest. And if you own a brand new SPARC T4 machine you're even luckier: It comes with its own SPARC T4 OpenSSL Engine. Dan Anderson's posts show how there really is now excuse not to encrypt any more...

Developers

Solaris 11 has a lot to offer to developers as well. Ali Bahrami has a series of blog posts that cover diverse developer topics: elffile: ELF Specific File Identification Utility, Using Stub Objects and The Stub Proto: Not Just For Stub Objects Anymore to name a few.

BTW, if you're a developer and want to shape the future of Solaris 11, then Vijay Tatkar has a hint for you: Oracle (Sun Systems Group) is hiring!

Desktop and Graphics

Yes, Solaris 11 is a 100% server OS, but it can also offer a decent desktop environment, especially if you are a developer. Alan Coopersmith starts by discussing S11 X11: ye olde window system in today's new operating system, then Calum Benson shows us around What's new on the Solaris 11 Desktop. Even accessibility is a first-class citizen in the Solaris 11 user interface. Peter Korn celebrates: Accessible Oracle Solaris 11 - released!

Performance

Gone are the days of "Slowaris", when Solaris was among the few OSes that "did the right thing" while others cut corners just to win benchmarks. Today, Solaris continues doing the right thing, and it delivers the right performance at the same time. Need proof? Check out Brian's BestPerf blog with continuous updates from the benchmarking lab, including Recent Benchmarks Using Oracle Solaris 11!

Send Me More Solaris 11 Launch Articles!

These are just a few of the more interesting blog articles that came out around the Solaris 11 launch, I'm sure there are many more!

Feel free to post a comment below if you find a particularly interesting blog post that hasn't been listed so far and share your enthusiasm for Solaris 11!

*Affiliate link: Buy cool stuff and support this blog at no extra cost. We both win!

Join the Solaris 11 Launch Party!

constant — Wed, 02 Nov 2011 19:36:06 +0000

In about a week, on November 9th, 2011, the long-awaited final version of Solaris 11 will be launched. If you happen to be near New York that day (and assuming there'll be no power outages), you're invited to join the official Solaris 11 launch party!

Solaris 11 has been in the making since 2005, when Solaris 10 was launched. In fact, every major Solaris release is just a fork of the ongoing Solaris development train, so the very first uber-pre-release of Solaris 11 was actually generated only weeks after Solaris 10 hit the shelves.

Since then, Solaris 11 (or: Project Nevada as it was called) has seen a lot of OS history: An open source adolescence called OpenSolaris, growing adoption and community work, a broad range of ground-braking new features, long overdue re-writes, brand new concepts, controversial discussions, a major acquisition, rules changed and rules kept, siblings and offsprings, lots of investments, entire companies built on top of its source code, generations of processors and hardware, lots of systems in production, the Cloud and what not.

And all that before it was even born. Quite an achievment, eh?

So when you finally hold that DVD in your hand... Nah, DVDs are last century. Let me try again.

So when you finally finish that download, here are a few things about Solaris 11 to try out and dig into:

Brand New Package Management, a Brand New Root File System and a Brand New Installer

This is perhaps the biggest change from Solaris 10 to Solaris 11: The way Solaris is installed in the first place. Over time, the old System V package manager and patching system has become quite a burden, so it just had to be re-written:

A modern, network-based, finely-granular package manager called IPS.
No more patching, just updates of existing packages.
No more entanglement between installation and configuration.
A new, powerful concept for root file systems: Boot environments.
Everything is neatly integrated with ZFS.
A new, network-centric installation mechanism that works hand in hand with IPS and ZFS: Automated Installer (AI).
Backward-compatibility with existing System-V packages.

I know some people had their issues with the new ways things are handled with IPS, boot environments and AI, but I'm sure they'll eventually get over it: Every change is difficult at first, but when the power of the new concepts start shining through, we'll all be glad we did it. And over time, and with more experience from the installed base, I'm sure that the Solaris team will iron out any missing bits.

Learn more about this by visiting the Installation and Packaging Community on OpenSolaris.org.

A Brand New Way to Support Existing Installations

Whenever a new release of Solaris came out in the past, it always took a long time until it would finally be adopted in production. The reason was simple: Migration. Granted, there's a 100% binary compatibility guarantee with Solaris that (technically, not legally) dates back to the ancient times of Solaris 2.6 and that only one competitor can remotely compare itself with (can you name it?), but still there was quite some work involved in moving a Solaris release to its successor, plus formal certification, of course.

Now, with Solaris 11, we'll get Solaris 10 Containers: Automatically pack your existing Solaris 10 installation into a Solaris 10 Container, then import into Solaris 11 as a branded zone, done. "Brand" new, indeed. Sorry for the pun.

No need for re-scripting stuff. No need to re-certify. No need to re-install or re-configure. Take the existing stuff and run unchanged, with a new OS under the covers. Migration heaven.

Yes, there may be the occasional detail to observe, but for the utter majority of existing installations, I expect the migration process to be smooth and slick: Containerfy, then import, done. A huge step in terms of ease-of-migration, and one that I expect will greatly help adoption of Solaris 11.

More about Virtualization in Solaris 11 is available in this OTN feature.

Brand New Networking

Solaris 10 changed the way Applications and Services are run and introduced virtualization at just the right level and with no friction: Zones.

Now, Solaris 11 will do the same to the network: Thanks to Project Crossbow, Solaris 11 users can enjoy limitless network virtualization including virtual NICs, virtual switches and routers, a built-in load-balancer and including full resource management.

Check out this introductory article with examples by Sunay Tripathi to get a better feel about Crossbow.

And if you're looking for 10x faster latency and 4x more bandwidth, then check out the new Infiniband stack that is included with Solaris 11. This will give your Exadata and Exalogic (and any other Infiniband based hardware) a nice, OS-level upgrade. Catch up with Ted Kim's Weblog if you want to learn more about InfiniBand in Solaris 11.

A whole new way of looking at networks.

Brand new Storage

Solaris 11 comes with a brand new version of ZFS, including two new major, much anticipated features: De-Duplication and Encryption, plus lots of new and improved stuff under the hood.

But there's a lot more to discover: The new COMSTAR framework will turn any Solaris 11 system into a first-class block-based storage engine citizen, be it on a SAN, an Infiniband or an Ethernet network: Create arbitrarily large volumes in seconds with ZVOLs, export them via FC, iSER or iSCSI, then enjoy the beauty of ZFS data integrity, snapshots, deduplication and storage replication with easy administration. Better than the big storage boys, but without the cost.

If you happen to be at the DOAG 2011 Conference in Nürnberg, Nov. 15th-17th, feel free to show up at my talk on "Neue ZFS-Funktionen in Solaris 11" on Tuesday, Nov. 15th, 15:00.

More DTrace

Solaris 10 brought us the ultimate X-Ray tool for the kernel and the rest of the OS: DTrace. more than 30,000 probes for your observation and analytical pleasure.

With Solaris 11, DTrace extends to the whole stack: With a total of 77982 probes (at least on my plain vanilla system, give or take a few dozen), you'll be able to observe your application throughout the whole stack: From kernel to OS to the web server, middleware (including Java and PHP) up to the browser.

Full stack observability, at run-time, in production, without any code change. What more can developers and administrators ask for?

If you want to refresh your DTrace know-how in time before Solaris 11 comes out, check out this postfrom the Observatory with a real-life use-case of DTrace, including an example on how to analyze a web application with MySQL DTrace probes.

Of course, the real authority in terms of DTrace literature is Brendan Gregg's and Jim Mauro's book DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X and FreeBSD (Oracle Solaris Series)* which is highly recommended. If you want to get a preview, here's a series of cool videos about the DTrace book and its content. Their favorite DTrace scripts are included in every Solaris 11 copy in the /opt/DTT directory, and this is where you'll get much more!

More Security

Similarly to DTrace and ZFS, the new security concepts from Solaris 10 have been enhanced in Solaris 11. Most notably, root is no longer the all-powerful user (though you can still make it so if you want), but just a role. This introduces accountability to the way sysadmins handle root passwords: No more anonymity by "just becoming root for a while".

We already saw encryption being part of the ZFS overhaul and in addition, Solaris 11 includes more and better implementations of common and new encryption algorithms, plus TPM support and of course support for hardware encryption in new processors, both from Intel and from Oracle.

Finally, Security is now deeper ingrained in other sub-systems such as SMF, IPS, Kerberos, Networking and Trusted Extensions.

Here'a a short overview of some new security features in Solaris 11.

Tune In and Follow the Launch

I'm sure the Solaris people have a few more good news to share during the November 9 event, so make sure to tune in and register for the event, be it in-person or through the live webcast.

Your Take on Solaris 11

What are your expectations about Solaris 11? Which new feature do you expect to deliver the biggest benefit to you and your organization? What are your plans for Solaris 11? Drop me a line in the comments and get yourself heard!

Update: Added a reference to the DTrace book. How could I have forgotten? Sorry, was a late night post...

* Affiliate link: Buy cool stuff here and I'll get a small kickback at no extra cost for you. We both win!