I talked to Stephen about how he would characterize today’s world of developers and what he feels project Sputnik needs to deliver on to be successful.

Updated March 22

• Stephen’s post on Sputnik a few days after the interview: Why a developer laptop?

Extra-credit reading

• The Register: Dell puts Sputnik open-source laptop on launch pad
• Engadget: Project Sputnik: Dell’s Ubuntu-based XPS13 laptop for developers
• WSJ: Dude, Dell’s Building an Open Source Laptop
• The Verge: Dell’s Project Sputnik pairs Ubuntu with the XPS 13 to help create a developer’s dream
• Ars Technica: New Dell Ubuntu ultrabooks a step in the right direction for Linux support

Pau for now…

-> Weigh in on Dell IdeaStorm: Project Sputnik

Extra-credit reading

• DevOpsAngle: Dell Announces A Linux Laptop Designed for Developers
• GigaOm: Dell tests open-source laptop for developers
• VentureBeat: Neckbeards, rejoice! Dell’s new Linux ultrabook was developed just for you
• Cote’s Blog: Project Sputnik
• Hacker News: Dell Announces A Linux Laptop Designed For Developers (devopsangle.com)
  

Made possible by an internal innovation fund, project Sputnik is a 6 month effort to explore the possibility of creating an open source laptop targeted directly at developers.  It is based on Ubuntu 12.04 and Dell’s XPS13 laptop.

To put it in context, Sputnik is part of an effort by Dell to better understand and serve the needs of developers in Web companies.  We want to finds ways to make the developer experience as powerful and simple as possible.  And what better way to do that than beginning with a laptop that is both highly mobile and extremely stylish, running the 12.04 LTS release of Ubuntu Linux.

Why a developer laptop

When we first started setting up the web vertical to focus on companies who use the internet as their platform, we brought in Stephen O’Grady of Redmonk to learn as much as we could about the needs of developers.  One of the ideas that Stephen tossed out was a Dell laptop running Ubuntu, targeting developers.  We thought the idea was pretty cool and filed it away.

As we continued talking to customers and developers the topic of Ubuntu kept coming up and we came across a fair number of devs who were asking for a Dell laptop specifically based on it.  To my knowledge, no other OEM has yet made a system specifically targeted at devs and figured it was time to see what that might mean.  When the XPS13 launched we realized that we found the perfect platform to start with and when Dell’s incubation program was announced we knew I had the vehicle to get the effort kick started.

I should also add that Ubuntu was a natural choice not only because of its popularity in the Web world but Dell has quite a bit of experience with it.  In fact Dell has enabled and pre-installed out-of-the-factory Ubuntu on more computer models than any other OEM.

What’s Sputnik actually running?

The install image available for Sputnik contains

1. drivers/patches for Hardware enablement
2. a basic offering of key tools and utilities (see the complete list at the end of this entry)
3. coming soon, a software management tool to go out to a github repository to pull down various developer profiles.

Hardware enablement

In putting together the project, the area that we focused on first was hardware enablement.  As Linux users are all too painfully aware, Linux drivers are not always available for various platforms.  We have been working hand in hand with Canonical, the commercial sponsor behind Ubuntu and identified three main areas on the XPS13:

1. An issue with brightness
2. The Wifi hotkey
3. The touchpad and multi touch support

The first two have been resolved but the last one re the touchpad is still at large.  The issue is a bit of a pain particularly the lack of palm rejection support which can cause your cursor to jump by mistake.  We have contacted the vendor who makes the touchpad and they are sizing the effort to fix this and at the same time we are working with Canonical to find an interim solution.

Developer profile management

Hardware enablement is table stakes but where Sputnik starts to get interesting is when we talk about profiles.  No two developers are alike so instead of stuffing the system with every possible tool or app a developer could possibly want, we are trying a different approach.  As mentioned above, the actual “stuff” on the install image is pretty basic, instead we are working with a few developers to put together a tool that can go out to a github repository and pull down various developer profiles.  The first profiles we are targeting are Android, Ruby and JavaScript.

As a one of our alpha cosmonauts, Charles Lowell, explained (we have been working with three local developers in Austin, Charles, Mike Pav and Dustin Kirkland to put together our initial offering together.   And yes I know Sputnik was unmanned but its our project and we wanted to call the testers “cosmonauts.” )

What I’d like to see is not only a gold-standard configuration, but also a meta-system to manage your developer configuration… The devops revolution is about configuration as code. How cool would it be if my laptop configuration were code that I could store in a source repo somewhere?

After we build the management tool and some basic profiles to get the effort started, we are hoping that the community will take over and began creating profiles of their own.

Getting Feedback and UDS activities

The idea is to conduct project Sputnik out in the open.  There is a Storm Session that went live this morning on Dell Idea Storm for people to discuss the project and submit feedback, comments and ideas.  Later today here at the Ubuntu Developer Summit, Dustin, Mario Limonciello of Dell and I will be hosting a UDS session to discuss Sputnik.  Additionally at UDS there is a coding contest that has been kicked off.  The three people who write the best Juju charms will each get an XPS13.

The Vision: a Launchpad to the cloud

As mentioned at the start, Sputnik is currently a 6month project to investigate an Ubuntu laptop.  If successful, we have big plans for the effort.

When we initially pitched Sputnik to Ubuntu’s founder Mark Shuttleworth a couple months ago he really liked the idea.  In his eyes however, he saw something bigger.  Where it got really interesting for him was when this laptop was optimized for DevOps.  In this scenario we would have a common set of tools from client, to test, to production, thereby tying Sputnik via a common tool chain to a cloud backend powered by OpenStack.  Developers could create “micro clouds” locally and then push them to the cloud writ large.

We see a lot of potential in Sputnik to provide developers with a simple and powerful tool.  Only time will tell however so stay tuned to this blog, check out the Sputnik Storm session and weigh in on the project, what you’d like to see and how you think it can be made better.

Pau for now…

Extra-credit reading

• Ubuntu 12.04 LTS (Precise Pangolin) Review | Desktop OS | ZDNet
• Ferreting out Innovation

Links and notes

•  Sputnik Storm session on idea storm

• Sputnik install image

Touchpad issue

Current behavior is that the touchpad functions as a basic pointing device, but lacks some desirable functionality:

1. Pinch, zoom, & flick
2. Palm negation (esp. important on larger touchpads where it’s easy for your hand to inadvertently touch the surface)
3. Side/edge scrolling
4. Two finger scrolling

We have contacted the vendor who makes the touchpad and they are sizing the effort to fix this and at the same time we are working with Canonical to find an interim solution.

Basic Install

== standard meta packages ==

ubuntu-desktop^

standard^

== scm ==

git

git-core

bzr

bzr-gtk

bzr-git

python-launchpadlib

== utilities ==

screen

byobu

tmux

meld

juju

charm-tools

charm-helper-sh

euca2ools

puppet

chef (available post install)

== editors ==

emacs

vim

vim-gnome

== browsers ==

chromium-browser

firefox

== common build tools/utilities & dependencies == fakeroot build-essential crash kexec-tools kvm makedumpfile kernel-wedge fwts devscripts

libncurses5

libncurses5-dev

libelf-dev

asciidoc

binutils-dev

Innovation Incubation

At the beginning of the year Dell launched an internal incubation fund with the idea of rooting out innovation throughout the company.  The thought was that there are probably a bunch of cool ideas tucked away in the heads of employees at all different levels of the company, ideas that would ultimately benefit our customers if given a little protection and help to get off the ground.

The three Dell employees behind it, Nnamdi Orakwue, Michale Cote and Matt Baker act like a mini  internal VC firm hearing pitches from employees.  Those projects they approve get a small amount of cash to build out their idea for six months.  At the end of six months the projects will either get their training wheels taken off and folded into business units, apply for more funding or disbanded.

Enter Sputnik

The program is just getting started and the first project green lighted was Project Sputnik.  I’m very excited about it and am looking forward to be able to talk all about it in the near distant future.  Stay tuned…

Extra-credit reading

• Forbes.com: Real Innovation: What It Really Is, and How to Really Do It (Really)

Pau for now…

Extra-credit reading

• Crowbar on github
• Rob’s blog
• DevOpsDays Austin kicks off, A short history

• DevOpsDays Austin — The provisioning panel starring JuJu, Crowbar, Puppet, Chef and Pallet

Some of the details published were the actual PUE numbers around Dell’s Modular Data Center:

Project Mercury gets free cooling year round, even in the heat of summer. On Aug. 23, 2011 — a 119-degree day — one of eBay’s Dell units had a partial-PUE score 0f 1.044 while drawing 520 kilowatts of power. On January 17, 2012, while drawing 1 megawatt, the same unit had consistent partial PUE of 1.018 while the rest of the data center was doing between 1.26 and 1.35.

And the winner is

Today the Uptime Institute announced that this Modular Data Center Product Deployment by eBay and Dell were named recipients of the 2012 Green Enterprise IT Award.

Here is a video I did at the opening back in December with Dell’s GM for our Data Center Solutions group, Roy Guillen.  Roy talks about what eBay was looking for and how we answered the challenge.

Some of the ground Roy covers:

• The challenge that eBay issued
• How Dell got involved
• The evolution of Dell’s Modular Data Center and what eBay proved about the ability to group a whole bunch of workloads that don’t require tier 4 resiliency

Extra-credit reading

• Making the web more efficient, a thousand servers at a time – GigaOm

Pau for now…

The panel on the first day,  moderated by John Willis, was entitled: Provisioning Panel – Meet Juju, Crowbar, Puppet, Chef, Pallet + discussion.  After the panel I caught up with each of the members for a follow-up chat.  Here they are:

Juju – Mark Mimms of Canonical

Crowbar – Rob Booth of Zenoss

Puppet – Dan Bode of Puppet Labs

Chef – Matt Ray of Opscode

Pallet – Antoni Batchelli of Pallet Ops

Stay tuned for more DevOpsDays goodness in the days to come!

Extra-credit reading

• DevOpsDays Austin kicks off, A short history – Barton’s Blog
• What Is This Devops Thing, Anyway? – Patrick Debois

Pau for now…

DCS was founded originally to chase the world’s top 20 hyperscale data center operators, and creates stripped-down, super-dense, and energy-efficient machines that can mean the different between a profit and a loss for those data center operators.

This team, which now represents a greater than $1 billion dollar business and has expanded beyond just custom systems to include standard systems built for the “next 1000,”  all started on a simple napkin.

The origin of DCS -- Ty’s Sonic sketch - November 2, 2006

From napkin to “Frankenserver,” to today

Ty Schmitt who was one of the original team and now is the executive director of Dell’s modular infrastructure team within DCS, explains:

This was sketch I made over drinks with Jimmy Pike late one night after visiting a big customer on the west coast.  We we were working on a concept for a 1U system for them based on their requested requirements.   As you can see by the date (Nov 2006) it was actually before DCS became official … we were a skunk works team called “Sonic” consisting of a hand full of people.   We wanted to take an existing chassis and overhaul it to fit 4 HD’s, a specific MB, and SATA controller.  When we got back to Austin, I modified the chassis in the RR5 machine shop (took parts from several different systems and attached them together) and Jimmy outfitted it with electronics, tested it, and it was sent to the customer as a sample unit.

This first proto was described by the customer as “Frankenserver” and was the beginning of the relationship we have with one of our biggest customers.

A little over five years later, Dell’s DCS team has gone from Frankenserver to commanding 45.2 percent revenue share in a market that IDC estimates at $458 million in sales last quarter.  Pretty cool.

Extra-credit reading:

• Dell’s bespoke server unit pushes over $1bn of tin – The Register
• IDC starts tracking the hyperscale server market – Barton’s Blog
• Dells Data Center Solutions group turns Four! – Barton’s Blog
• Dell’s “Custom Tailor” to Internet Stars turns three — Expands to serve the next 1000 – Barton’s Blog
• A Data Center that fits in the Overhead Bin – Barton’s Blog

Pau for now…

The Austin event was maxed out the day after it was announced a couple months ago and there was a waiting list of over one hundred people.   About half the crowd was from out of town with a big contingent from the Valley and New York.   Near the end of the day yesterday I caught up with one of the organizers, Damon Edwards to learn more about the event and how it came to be.

Some of the ground Damon covers:

• How “DevOps” and DevOps Days came to be: How John Allspaw‘s talk at Velocity to Patrick Debois to set up the first event
• 3:36:  How the event has transformed over time:  from a focus on people to a focus on tools and now back to a focus on people
• 4:56  Who attends DevOps Days and how the mix between developers and operations folks has shifted over time.

Extra-credit reading

• What Is This Devops Thing, Anyway? – Patrick Debois
• What is DevOps? – Damon Edwards
• Velocity 09: John Allspaw, “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr” – Video
• 10 deploys Per day: dev and ops Cooperation at Flickr – Slide Deck

Pau for now…

Some of the ground Sean covers:

• What is EnterpriseDB and what is their business model
• 1:10 Where does EnterpriseDB fit in the overall database landscape and where is it used
• 2:00 The release of the Postgres Plus cloud database on EC2 and soon OpenStack
• 2:44 What EnterpriseDB has got up its sleeve in the way of features and functionality in the next year

Extra-credit reading

• EnterpriseDB launches PostgreSQL Plus Cloud Service – ZDNet
  
• EnterpriseDB Blog

Pau for now…

These are minimalist server designs that resemble blades in that they have skinny form factors but they take out all the extra stuff that hyperscale Web companies like Google and Amazon don’t want in their infrastructure machines because they have resiliency and scale built into their software stack and have redundant hardware and data throughout their clusters….These density-optimized machines usually put four server nodes in a 2U rack chassis or sometimes up to a dozen nodes in a 4U chassis and have processors, memory, a few disks, and some network ports and nothing else per node.

Source: IDC -- Q3 2011 Worldwide Quarterly Server Tracker

Here are the stats that Prickett Morgan calls out (I particularly like the last bullet :

• By IDC’s reckoning these dense servers accounted for $458 million in sales, up 33.8 percent in a global server market that fell by 7.2 percent to $14.2 billion in the quarter.
• Density optimized machines accounted for 132,876 servers in the quarter, exploding 51.5 percent, against the overall market, which comprised 2.2 million shipments and rose 2 percent.
• Dell, by the way, owns this segment, with 45.2 percent of the revenue share, followed up by Hewlett-Packard with 15.5 percent of that density-optimized server pie.

Extra-credit reading

• Dell dominates in new Density-Optimized Server category

• See the IDC press release.
• Read the Dell press release.

Pau for now…

If you’re not familiar with it, OpenStack is an open source cloud project built on a foundation of code initially donated by NASA and Rackspace.  The project kicked off a little over a year and a half ago here in Austin and it has gained amazing traction since then.

Dell’s offering

Dell’s OpenStack cloud offering is an open source, on premise cloud solution based on the OpenStack platform running on Ubuntu.  Its composed of:

• The OpenStack cloud operating system
• PowerEdgeC servers: C6100, C6105, C2100 and, coming soon, Dell’s new C6220 and R720
• The Crowbar deployment and management software framework – developed and coded by Dell
• Dell’s OpenStack reference architecture
• Dell Services

Crowbar software framework

To give a little more  background on the Crowbar software framework, its an open source project developed initially at Dell and you can grab it off github.  The framework, which is under the Apache 2.0 license,  manages the OpenStack deployment from the initial server boot to the configuration of the primary OpenStack components, allowing users to complete bare metal deployment of multi-node OpenStack clouds in hours, as opposed to days.

Once the initial deployment is complete, you can use Crowbar to maintain, expand, and architect the complete solution, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.   Beyond Dell, companies like VMware, Dreamhost and Zenoss have built “barclamps”  that allow them to utilize Crowbar’s modular design.  Additionally, customers who buy the Dell OpenStack-Powered Cloud Solution get training, deployment, and support on Crowbar.

So as of today, customers in the UK, Germany and China can purchase the Dell OpenStack-Powered Cloud Solution.  As customer demand grows in other regions we will be adding more countries so stay tuned.  If the first 18 mos of the project are any indication of whats the pace is like to come, we are all going to be in for a lot more excitement.

For more info, email: OpenStack@Dell.com

Extra-credit reading

• Blog: Dell OpenStack-Powered Cloud Solution Now Available in Europe and Asia
• Blog: World Hosting Day: Dell OpenStack-Powered Cloud Solution with Ubuntu 
• Blog: Dell announces availability of OpenStack solution; Open sources “Crowbar” software framework
• Blog: Introducing OpenStack — an open source cloud platform

Pau for now…

The final meet up was held on Sunday evening at the Hilton bar, Finn and Porter.  Here is a mini-montage from the event:

I asked the folks to say who they are, where they’re from, who they work for and what they hope to get out of SXSW.

The line-up

• Jonathan Bryce of OpenStack/Rackspace, from three hours outside of Austin
• Neelan Choksi of Tasktop, from Austin
• Ben Kepes of Diversity Limited from Waipara, New Zealand
• Sandeep Parikh of 10gen, from Austin
• Yoram Heller of Morph Labs, from Los Angeles, CA
• Ernest Mueller of Bazaar Voice, from Austin
• Matt Brutsche of Present Now, from Austin

Extra-credit reading

• SXSW: Our first developer Chill and Chat
• SXSW: Our Saturday night Dev/tech meet up
• Gazzang – One of the 10 Austin startups to check out at SXSW

Pau for now…

Here is a small sampling of the folks who dropped by (notice the atmospheric lighting, for half of them they were literally lit by candle light):

The Line up

• Sam Ramji of Apigee, from Palo Alto, CA – talks about the launch of UserGrid
• Ed Saipetch of Joyent, from Indianapolis, IA
• Polly Squires of T-Systems, from Houston, TX
• Alex Williams of SiliconAngle from Portland, OR
• Robert Duffner of Apigee from Seattle, WA
• Gina Rosenthal (Minks) of Dell from Austin, TX

Extra-credit reading

• Gazzang – One of the 10 Austin startups to check out at SXSW
• SXSW: Our first developer Chill and Chat

Pau for now…

Given the  proximity and the promise of free beer, I was able to twist the arms of four members of their development team and get them to join us.  Here is a quick video featuring Dustin Kirkland, Sergio Pena, Hector Acosta, and Eddie Garcia.

Pau for now…

As for the Web|Tech vertical we have taken our own guerrilla approach to participation in this shindig in our own backyard.  Besides going to parties that customers and partners are throwing, Cote and I have organized a series of informal “chill and chat” meet ups for developers and tech types.

Last night we held our first soiree at Opal Divines.  Here is a mini-montage I made featuring a few of the attendees:

I asked the folks to say who they are, where they’re from, who they work for and what they hope to get out of SXSW.

The line-up

• Nosh Petigara of 10gen, from NYC
• Michael Cote of Dell, from Austin
• Rob Smith of 3M, from Minnesota
• Maggie Fox of Social Media Group, from Toronto
• Charlie Wood of Spanning cloud apps, from  Austin

Our next events

If you’re around here’s where we’ll be tonight and tomorrow:

• Saturday, March 10th  7-9:00pm  – Speakeasy – Downstairs (412 Congress Avenue, Austin, TX)
• Sunday, March 11th 5:30-7:30pm – Hilton Bar (Finn and Porter)  (500 East 4th Street, Austin, TX)

Extra-credit reading

• Dell Officially Launches Web|Tech vertical — the 100 second overview
• The World of Web and Developers, getting to know it better

Pau for now…

Some of the ground Laura covers

• An intro to SugarSync: what it is and who it’s targeted at
• 0:43 — How do you get SugarSync and what’s their business model
• 1:32 — How Laura got involved with the company and how they’ve been doing
• 2:06 — How does SugarSync differ from something like Dropbox, how does it work and the power of cross-platform solutions
• 4:09 — What’s next for the company and the product

Extra-credit reading

• PC Magazine: Editor’s Choice Award: SugarSync
• WSJ/All things Digital: Samsung Smart TVs Get Sweeter With SugarSync
• Laura’s blog: The Kitchen Sync

At the show, which was held in Vegas, we had booths on the expo floor to talk about our solutions.  We also delivered breakout sessions to present an overview of customer needs and concerns for each of the six — Retail, Manufacturing, Financial Services, Web|Tech, Energy and TME (Telco, Media & Entertainment), as well as for our existing three verticals in the Public space, Government, Education and Healthcare.

Here is a quick overview I did of the Web|Tech vertical from the show floor in a mocked up developer’s cube in our booth.

Extra-credit reading

• Web|Tech page on Dell.com – (Note: This is for a general audience, we will be developing a separate location targeted at developers)
• The World of Web and Developers, getting to know it better – My blog

• Dell Gets It – Developers are the new Kingmakers – ServicesAngle
• Dell’s golden opportunity is not in the servers – GigaOm

Pau for now…

Some of the ground Prabhakar covers:

• Where he got the idea
• From laptop to github to Rackspace
• Desktop and mobile HTML5 version
• Finding the right tools
• Lessons learned

Extra-credit reading

• SiliconAngle — Kanban2Go: A Drag and Drop Zen-Like To Do App That Actually Works
• Press Release — Simple Idea Labs Launches Free Open Beta Of kanban2go To-Do List App for Smart Phones, Tablets, and PCs
• Blog — One experiment out the door: kanban2go is live

Pau for now…

Extra credit reading

• Open source Crowbar code now available for Hadoop
• Dell.com/Hadoop

Pau for now…

Growing beyond geek fest

The event itself has picked up significant momentum over the last three years going from 500 attendees, to 900 the second year, to over 1400 this past year.  The tone has gone from geek-fest to an event focused also on business problems e.g. one of the keynotes was by Larry Feinsmith, managing director of the office of the CIO at JP Morgan Chase.  Besides Dell, other large companies like HP, Oracle and Cisco also participated.

As a platinum sponsor, Dell  had both a booth and a technical presentation.   At the event we announced that we would be open sourcing the Crowbar barclamp for Hadoop and at out booth we showed off the Dell | Hadoop Big Data Solution which is based on Cloudera Enterprise.

Cutting’s observations

Doug Cutting, the father of  Hadoop, Cloudera employee and chairman of the Apache software foundation, gave a much anticipated keynote.  Here are some of the key things I caught:

• Still young: While Cutting felt that Hadoop had made tremendous progress he saw it as still young with lots of missing parts and niches to be filled.
• Big Top: He talked about the Apache “Bigtop” project which is an open source program to pull together the various pieces of the Hadoop ecosystem.  He explained that Bigtop is intended to serve as the basis for the Cloudera Distribution of Hadoop (CDH), much the same way Fedora is the basis  for RHEL (Redhat Enterprise Linux).
• “Hadoop” as “Linux“: Cutting also talked about how Hadoop has become the kernel of the distributed OS for big data.  He explained that, much the same way that “Linux” is technically only the kernel of the GNU Linux operating system, people are using the word Hadoop to mean the entire Hadoop ecosystem including utilities.

Interviews from the event

To get more of the flavor of the event here is a series of interviews I conducted at the show, plus one where I got the camera turned on me:

•  Hadoop World:   Cloudera CEO reflects on this year’s event
•  Hadoop World:   Accel’s $100M Big Data fund
•  Hadoop World:   Battery Ventures EIR Todd P.
•  Hadoop World:   Talking to Splunk’s Co-founder
•  Hadoop World:   NoSQL database MongoDB
•  Hadoop World:   Talking HBase with Facebook’s Jonathan Gray
•  Hadoop World:   Ubuntu, Hadoop and Juju
•  Hadoop World:   Karmasphere and big data intelligence
•  Hadoop World:   Learning about NoSQL database Couchbase
•  Hadoop World:   O’Reilly Strata conference chair, Ed Dumbill

• Hadoop World:   What Dell is up to with Big Data, Open Source and Developers

Extra-credit reading

• Cloudera Teams With O’Reilly Media to Merge Hadoop World and Strata Conferences

Blogs regarding Dell’s crowbar announcement

• Dell to open source software to ease Hadoop install & management
• Developers: How to get involved with Crowbar for Hadoop
• Open source Crowbar code now available for Hadoop
• How to create a Basic or Advanced Crowbar build for Hadoop

Hadoop Glossary

• Hadoop ecosystem
  • Hadoop: An open source platform, developed at Yahoo that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.  It is particularly suited to large volumes of unstructured data such as Facebook comments and Twitter tweets, email and instant messages, and security and application logs.
  • MapReduce: a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in parallel on large clusters of commodity hardware in a reliable, fault-tolerant manner.  Hadoop acts as a platform for executing MapReduce.  MapReduce came out of Google
  • HDFS: Hadoop’s Distributed File system allows large application workloads to be broken into smaller data blocks that are replicated and distributed across a cluster of commodity hardware for faster processing.

• Major Hadoop utilities:
  • HBase: The Hadoop database that supports structured data storage for large tables.   It provides real time read/write access to your big data.
  • Hive:  A data warehousing solution built on top of Hadoop.  An Apache project
  • Pig: A platform for analyzing large data that leverages parallel computation.  An Apache project
  • ZooKeeper:  Allows Hadoop administrators to track and coordinate distributed applications.  An Apache project
  • Oozie: a workflow engine for Hadoop
  • Flume: a service designed to collect data and put it into your  Hadoop environment
  • Whirr: a set of libraries for running cloud services.  It’s ideal for running temporary Hadoop clusters to carry out a proof of concept, or to run a few one-time jobs.
  • Sqoop: a tool designed to transfer data between Hadoop and relational databases.  An Apache project
  • Hue: a browser-based desktop interface for interacting with Hadoop

The idea behind the glossary is to help our teams get a better understand of the wild and wacky world of the Web and Web developers as we move forward with our Web|Tech vertical.  I figured I might as also share it with a few friends.

Today’s focus, having worked our way down from the top, is the infrastructure tier (with a short catch-all bucket at the end , “Misc.”)

Infrastructure

General Terms

• DevOps:  The goal of the DevOps movement is to drive out inefficiency in web shops by bridging the gap (and lessening conflict) between traditional development activity and operations activity.  It seeks to address this issue by providing tools and practices to bring these two groups closer together and provide for greater automation of processes.  Key tools in this effort are Opscode’s Chef and Puppet lab’s Puppet which automate the set-up and management of infrastructure.
• PUE: Power Usage Effectiveness is a measure of how efficiently a computer data center uses its power; specifically, how much of the power is actually used by the computing equipment (in contrast to cooling and other overhead).   PUE is the ratio of total amount of power used by a computer data center facility to the power delivered to computing equipment.  The closer to 1.0, the better the PUE.
• Distributed management: refers to the setup, provisioning, maintenance and management of the scale-out infrastructure (either physical or virtual) that has historically been characteristic of web firms and is increasing typical within traditional enterprise customers.  This includes players like Chef and Puppet for provisioning and configuration, New Relic and Splunk for monitoring and management, and Loggly/Eucalyptus/OpenStack/ VMware for management monitoring.

Projects/Entities

• Crowbar:  Crowbar is a Dell-developed open source software framework designed to speed up the installation and configuration of open source cloud software onto bare metal systems.  By automating the process, Crowbar can reduce the time needed for installation from days to hours.  The software is modular in design so while the basic functionality is in Crowbar itself, “barclamps” sit on top of it to allow it work with a variety of projects.  There have been barclamps built for OpenStack, Hadoop, CloudFoundry and Dreamhost.
• Ubuntu:  The most popular desktop linux distribution.  On the server side they are supporting OpenStack and have an offering called the Ubuntu Enterprise Cloud.   Backed by the commercial company Canonical.
• Puppet: a configuration management tool designed to automate the set up and management of infrastructure.  A key DevOps tool.  It is produced by Puppet labs
• Chef: a configuration management tool designed to automate the set up and management of infrastructure.  A key DevOps tool.  It is produced by Opscode, who hosts a cloud-based version of Chef called the Opscode Platform.
• Nagios: a popular open source computer system and network monitoring software application. It watches hosts and services, alerting users when things go wrong and again when they get better.
• Ganglia: an open source scalable distributed monitoring system for high-performance computing systems such as clusters and grids.

Misc

• LAMP stack:  Open source stack that provides a viable general purpose web server.  The name comes from the first letters of its components: Linux, Apache web server, MySQL and PHP (or Perl or Python).   LAMP has become a de facto development standard and is an excellent example of how open source software has made its way into enterprise environments through unofficial channels.
• Apache Software Foundation: A decentralized group of developers that produce open source software under the Apache license.  Notable projects include: Apache web server, Hadoop, CouchDB, Cassandra, Tomcat, Subversion
• Nginx: an open source web server that recently has been gaining considerable traction
• Recipes:  They encapsulate collections of software resources which are executed in the order defined to configure a system.

Extra-credit reading

• The World of Web and Developers, getting to know it better
• Web Glossary part one: Application tier
• Web Glossary part two: Data tier

Pau for now…

Enjoy

General terms

• Structured data: Data that can be organized in a structure e.g. rows or columns so that it is identifiable. The most universal form of structured data is a database like SQL or Access.
• Unstructured data:  Data that has no identifiable structure. Unstructured data typically includes bitmap images/objects, text and other data types that are not part of a database. Most enterprise data today can actually be considered unstructured. An email is considered unstructured data.
• Big Data: Data characterized by one or more of the following characteristics:  Volume – A large amount of data, growing at large rates; Velocity – The speed at which the data must be processed and a decision made;  Variety – The range of data, types and structure to the data
• Relational Databases (RDBMS) Management Systems: These databases are the incumbents in enterprises today and store data in rows and columns.  They are created using a special computer language, structured query language (SQL), that is the standard for database interoperability.  Examples:  IBM DB2, MySQL, Microsoft SQL Server, PostgreSQL, Oracle RDBMS, Informix, Oracle Rdb, etc.
• NoSQL: refers to a class of databases that 1) are intended to perform at internet (Facebook, Twitter, LinkedIn) scale and 2) reject the relational model in favor of other (key-value, document, graph) models.  They often achieve performance by having far fewer features than SQL databases and focus on a subset of use cases.  Examples: Cassandra, Hadoop, MongoDB, Riak
• Recommendation engine:  A recommendation engine takes a collection of frequent itemsets as input and generates a recommendation set for a user by matching the current user’s activity against the discovered patterns. The recommendation engine is on-line process, therefore its efficiency and scalability are key,  e.g. people who bought X often also bought Y.
• Geo-spatial targeting: the practice of mapping advertising, offers and information based on geo location.
• Behavioral targeting: a technique used by online publishers and advertisers to increase the effectiveness of their campaigns.  Behavioral targeting uses information collected on an individual’s web-browsing behavior, such as the pages they have visited or the searches they have made, to select which advertisements to display to that individual.
• Clickstream analysis: On a Web site, clickstream analysis is the process of collecting, analyzing, and reporting aggregate data about which pages visitors visit in what order – which are the result of the succession of mouse clicks each visitor makes (that is, the clickstream). There are two levels of clickstream analysis, traffic analysis and e-commerce analysis.

Projects/Entities

• Gluster: a software company acquired by Red Hat that provides an open source platform for scale-out Public and Private Cloud Storage.
• Relational Databases
  • MySQL:  the most popular open source RDBMS.  It represents the “M” in the LAMP stack.  It is now owned by Oracle.
  • Drizzle:  A version of MySQL that is specifically targeted the cloud.  It is currently an open source project without a commercial entity behind it.
  • Percona:  A MySQL support and consulting company that also supports Drizzle.
  • PostgreSQL: aka Postgres is is an object-relational database management system (ORDBMS) available for many platforms including Linux, FreeBSD, Solaris, Windows and Mac OS X.
  • Oracle DB – not used so much in new WebTech companies, but still a major database in the development world.
  • SQL Server – Microsoft’ s RDBMS

  NoSQL Databases

  • MongoDB:  an open source, high-performance, database written in C++.  Many Linux distros include a MongoDB package, including CentOS, Fedora, Debian, Ubuntu and Gentoo.  Prominent users include Disney interactive media group, New York Times, foursquare, bit.ly, Etsy. 10gen is the commercial backer of MongoDB.
  • Riak: a NoSQL database/datastore written in Erlang from the company Basho. Originally used for the Content Delivery Network Akamai.
  • Couchbase: formed from the merger of CouchOne and Membase.  It offers Couchbase server powered by Apache CouchDB and is available in both Enterprise and Community editions. The author of CouchDB was a prominent Lotus Notes architect.
  • Cassandra: A scalable NoSQL database with no single points of failure.   A high-scale, key/value database originating from Facebook to handle their message inboxes. Backed by DataStax, which came out of Rackspace.
  • Mahout: A Scalable machine learning and data mining library. An analytics engine for doing machine learning (e.g., recommendation engines and scenarios where you want to infer relationships).

• Hadoop ecosystem
  • Hadoop: An open source platform, developed at Yahoo that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.  It is particularly suited to large volumes of unstructured data such as Facebook comments and Twitter tweets, email and instant messages, and security and application logs.
  • MapReduce: a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in parallel on large clusters of commodity hardware in a reliable, fault-tolerant manner.  Hadoop acts as a platform for executing MapReduce.  MapReduce came out of Google
  • HDFS: Hadoop’s Distributed File system allows large application workloads to be broken into smaller data blocks that are replicated and distributed across a cluster of commodity hardware for faster processing.

• Major Hadoop utilities:
  • HBase: The Hadoop database that supports structured data storage for large tables.   It provides real time read/write access to your big data.
  • Hive:  A data warehousing solution built on top of Hadoop.  An Apache project
  • Pig: A platform for analyzing large data that leverages parallel computation.  An Apache project
  • ZooKeeper:  Allows Hadoop administrators to track and coordinate distributed applications.  An Apache project
  • Oozie: a workflow engine for Hadoop
  • Flume: a service designed to collect data and put it into your  Hadoop environment
  • Whirr: a set of libraries for running cloud services.  It’s ideal for running temporary Hadoop clusters to carry out a proof of concept, or to run a few one-time jobs.
  • Sqoop: a tool designed to transfer data between Hadoop and relational databases.  An Apache project
  • Hue: a browser-based desktop interface for interacting with Hadoop
• Cloudera: a company that provides a Hadoop distribution similar to the way Red Hat provides a Linux distribution.  Dell is using Cloudera’s distribution of Hadoop for its Hadoop solution.
• Solr: an open source enterprise search platform from the Apache Lucene project. Backed by the commercial company Lucid Imagination.
• Elastic Search: an open source, distributed, search engine built on top of Lucene (raw search middleware).

Extra-credit reading

• The World of Web and Developers, getting to know it better
• Web Glossary part one: Application tier

Pau for now…

Over the next several entries I will be posting the glossary.  Feel free to bookmark it, delete it, offer corrections, comments or additions.

Today I present to you, the Application tier.

enjoy

General terms

• Runtime: A programming language e.g. Java, .NET, JavaScript, PHP, Python, Ruby…
• Application framework : Provides re-usable templates, methods, and ways of programming applications. Often, these frameworks will provide “widgets” and “libraries” that developers use to create various parts of their application – they may also include the actual tools to create, deploy, and run the final application. Some application frameworks create whole sub-cultures of developers, such as Rails which supports the Ruby programming language.  Most application frameworks are open source and free, though there are also many closed source, not-free ones.
• Continuous code development lifecycle: releasing software at more frequent intervals (30 days or less) by (a.) doing smaller batches of code, and, (b.) using tools and processes that enable a more lean approach to development. Software released in such a cycle tends to release many small features instead of, in contrast, “traditional” development where 100s of features are bundled up in one version of the software and released every 1-2 years.

Programming languages

• Java/.NET:  The incumbent enterprise development languages.  Very powerful but relatively difficult to learn and take time to program in.
• Dynamic languages: e.g. PHP, Perl, Python, JavaScript, and Ruby.  They are popular for creating web applications since they are both simpler to learn and faster to code in than traditional enterprise standards like Java. This offers a substantial time to market advantage, particularly for smaller projects for which the benefits of Java are less applicable.
  • PHP: a server-side scripting language originally designed for web development to produce dynamic web pages.  WordPress is written in PHP, as well as Facebook and countless web sites. PHP is infamous for being very quick and easy to get started with (which it is) but turning into a mess of “spaghetti code” after years of work and different programmers.   PHP is open source, though Zend, the patron company behind PHP, and others sell “commercial” versions.
  • Perl:  One of the original programming languages of the web, Perl emphasizes a very “Unix way” of programming. Perl can be quick and elegant, but like PHP can result in a pile of hard to maintain code in the long term.  While Perl was extremely popular in the first Internet bubble, it has sense taken a back-seat to more popular development worlds such as PHP, Java, and Rails. Perl is open source and there are few, if any, commercial companies behind it.
  • Python: Like all dynamic languages, Python emphasizes speed of development and code readability. Its an object-oriented language. Python is something of an evolution of Perl, but it not that closely tied to it. Python emphases broadness of functionality while at the same time being a proper, object oriented programing language (not just a way to write “scripts”). Python enjoys steady popularity; Google uses Python as one of its primary programming languages.
  • JavaScript: once a minor language used in web browsers, JavaScript has become a stand-alone language on its own known and used by many programmers. Most web applications will include the use of JavaScript.
  • Ruby: Ruby and Python are very similar in ethos: emphasizing fast coding with a more human-readable syntax. Ruby became famous with the rise of Rails in the mid-2000s which was a rebellion against the “heavy weight” practices that Java imposed on web development.  Ruby is still very popular.  Ruby can also be run on-top of the Java virtual machine (via JRuby), providing a good bridge to the Java world.  Salesforce’s acquired PaaS, Heroku, uses Ruby, and most modern development platforms use Ruby.
  • Ruby on Rails: a popular web application framework written in Ruby.  Rails is frequently credited with making Ruby “famous”.
  • Scala:  A somewhat exotic language, but it has quite a buzz around it. It’s good for massive scale systems that need to be concurrent (lots of people changing lots of things, often the same things, at the same time).  Erlang is another language in this area.  Scala runs on the Java Virtual Machine and Common Language Runtime.  In April 2009 Twitter announced they had switched large portions of their backend from Ruby to Scala and intended to convert the rest.  In addition, Foursquare uses Scala and Lift (Lift is a framework for Scala much in the same way Rails is a framework for Ruby.)
• R:  a programming language and software environment for statistical computing and graphics.
• Node.js:  (aka “Node”) What’s interesting about Node.js is the idea that it is taking JavaScript which was originally designed to be used in web browsers and using it as a server-side environment.  It is intended for writing scalable network programs such as web servers.  It was created by Ryan Dahl in 2009, and its growth is sponsored by Joyent, which employs Dahl.
• Clojure: A recent dialect of the Lisp programming language and is good for data intense applications.  It runs on the Java Virtual Machine and Common Language Runtime

Runtimes and Platforms

• Common Language Runtime (CLR):  is the virtual machine component of Microsoft’s .NET framework and is responsible for managing the execution of .NET programs.
• Java Virtual Machine (JVM) – the underlying execution engine that the Java language runs on-top of.  It controls access to the hardware, networks, and other “infrastructure” and services outside of the main application written in Java. Of special note is that many languages other than Java can run on the JVM (as with the CLR), e.g., Scala, Ruby, etc. There are many JVMs and ISVs (IBM, Oracle, etc.) will use their custom JVMs as key differentiators for middle ware, mostly around performance, scale-out, and security.

Projects/Entities

• Openshift: Red Hat’s Platform as a Service (PaaS) offering.  More specifically, OpenShift is a PaaS software layer that Red Hat runs and manages on top of third party providers – Amazon first with more to follow.
• Heroku:  A Platform as a Service (PaaS) offering that was acquired by Salesforce.com.  It supports development of Ruby on Rails, Java, PHP and Python.
• CloudFoundry: A Platform as a Service (PaaS) offering and VMware-led project. Cloud Foundry provides a platform for building, deploying, and running cloud apps using the Spring Framework for Java developers, Rails and Sinatra for Ruby developers, Node.js and other JVM languages/frameworks including Groovy, Grails and Scala.
• Joyent: Offers PaaS and IaaS capabilities through the public cloud.  Dell resells this capability as turnkey solution under the name The Dell Cloud Solution for Web applications.  Joyent also sponsors the development of node.js and employs its creator.
• GitHub: a web-based hosting service for software development projects that use the Gitrevision control system. GitHub offers both commercial plans and free accounts for open source projects.

But wait there’s more…

Stay tuned for the next couple of entries when I will cover first the Database tier and then the Infrastructure tier.

Extra-credit reading

• The World of Web and Developers, getting to know it better

Pau for now…

Based on this success, a little while ago we kicked off a similar effort in our commercial business.  The first six verticals we are setting up are: Retail, Manufacturing, Financial Services, Web|Tech, Energy and TME (Telco, Media & Entertainment).  Web|Tech is the group I belong to (I lead marketing for the group).

Developers, Developers, Developers

In the Internet space we have already had a fair amount of success through our DCS group.  The idea with the new Web vertical is to learn even more about the customer set, companies that use the internet as their platform, and take this knowledge along with our accumulated experience, to a wider audience.  Two of the key areas of focus of this new vertical will be developers and open source software.

Look it up

One of the ways we are helping our teams get a better understand of the wild and wacky world of the Web and Web developers is via a glossary we’ve created.  In compiling this I pulled information from various and sundry sources across the Web including wikipedia, community and company web sites and the brain of Cote.

The glossary is organized into the following sections:

[Update Feb 1: I've gone back and linked the entries below]

• Application tier
• Data tier
• Infrastructure tier

Over the next several entries I will be posting the glossary.  Feel free to bookmark it, delete it, offer corrections, comments or additions.

Extra-credit reading

• Dell Gets It – Developers are the new Kingmakers – Services Angle
• Dell’s golden opportunity is not in the servers – GigaOm

Pau for now…