Kevin Hegg

What have I been doing on my sabbatical?

noreply@blogger.com (Kevin Hegg) — Sun, 22 Jun 2008 19:59:00 +0000

Since I stopped working on Feb. 29, 2008 a lot of people have asked what I have been doing so I thought I would write something up. Here is what I have been doing to keep busy.

The objectives I set were to learn more about (1) computer security, (2) Windows Vista/Server 2008 platform, (3) .NET 3.5, specifically WCF and LINQ, (4) concurrency, parallelism, and scalability, (5) agile software development, (6) software design, and (7) advanced debugging.

Books Read

Writing Secure Code for Windows Vista
Windows Server 2008 Security Resource Kit
Ethical Hacking and Countermeasures 6.0, Volumes 1, 2, and 3
Windows Vista Inside Out
Windows Server 2008 Inside Out
Inside SQL Server 2005: Query Tuning and Optimization
Pro C# 2008 and the .NET 3.5 Platform (Ch. 10, 11, 14, 20)
Silverlight 1.0 Unleashed (don't recommend)
Essential Silverlight 2
Learning WCF
Essential Windows Communication Foundation (don't recommend)
Windows Via C/C++ (Ch. 6 - 11)
The Enterprise and Scrum (2nd time read)
Agile Project Management with Scrum (2nd time read)
Domain-Driven Design
Working Effectively with Legacy Code
Software Factories (don't recommend)
I.M. Wright's "Hard Code" (don't recommend)
Facts and Fallacies of Software Engineering
Programming Robotics Studio (don't recommend)

Books Currently Reading

Events Attended

MIX, March 2008
Devscovery Conference, April 2008
ALT.NET Seattle, April 2008
Edward Tufte's Presenting Data and Information, May 2008
Hacker Halted USA 2008 Conference, May/June 2008
DC ALT.NET Meeting, Monthly

Events Planned

Black Hat USA 2008, August 2008
Microsoft Professional Developers Conference, October 2008

Software I have been playing with

Microsoft Robotics Studio and Lego Mindstorm NXT
ASP.NET AJAX
ASP.NET MVC
xUnit.net 1.0
Enterprise Library 4.0, specifically Unity
Windows Communication Foundation
Web Service Software Factory - Modeling Edition Visual Studio 2008
LINQ
.NET Asynchronous Programming Model and Jeffrey Richter's Threading Tools
Parallel Extensions to the .NET Framework
JetBrains Resharper 4.0
JetBrains dotTrace 3.1

Certification Exams Passed

Certified Ethical Hacker
Microsoft .NET Framework, Application Development Foundation (70-536)
Windows Vista, Configuring (70-620)
Windows Server 2008 Applications Infrastructure, Configuring (70-643)

Hacker Halted Conference Summary

noreply@blogger.com (Kevin Hegg) — Sun, 22 Jun 2008 18:58:00 +0000

I attended the Hacker Halted USA 2008 Conference May 28 - June 3, 2008. My objective in attending this conference was to get a different perspective on cyber security and to evaluate the new Certified Ethical Hacker (CEH) Version 6 curriculum/certification at the request of a friend.

The first three days I attended the CEH course. This is normally a 5-day course. I think it was a mistake to try to jam 5 days of material into 3 days. Too much information was skimmed over or not presented. This was the first presentation of the Version 6 material. I knew that going into the course and expected more issues than were encountered. There were surprisingly few problems with the course material and labs. The instructor, on the other hand, was horrible. He was unfamiliar with the material, read straight from the slides, was not a very good speaker, and injected a lot of personal anecdotes that were laced with inaccuracies. I strongly recommend against taking any course from this instructor, Chuck Swanson. The only saving grace from this experience is that the CEH course material and labs are very good, in my opinion. The course material is over 4,000 pages. After the conference I went through the course and labs in-depth and I learned a lot.

The Hacker Halted conference was held jointly with the Techno Security and AccessData Users' Conferences. In total, there were probably 1,000 people in attendance. Attendees could go to any of the sessions, but I didn't take advantage of that. I attended the Hacker Halted sessions exclusively. I thought the Hacker Halted agenda was good, unfortunately 3 of the best speakers cancelled and their replacements were very underwhelming. The other presentations were good.

The target audience for the three conferences is forensic investigation, computer security, network administration, and law enforcement people. There were very few developers in attendance. I had the opportunity to talk with a lot of people with different backgrounds which is something I was looking forward to. I met some interesting people and had interesting conversations.

I am not sure that I would recommend this conference. It had lots of potential and if it lived up to its potential it would have been a very good conference.

How do I convince my company/client/boss/team/co-worker to do X?

noreply@blogger.com (Kevin Hegg) — Thu, 08 May 2008 18:02:00 +0000

This topic comes up every now and then, but over the last couple of months I have had a lot of conversations with a bunch of folks about "How do I convince my company/client/boss/team/co-worker to do X?" or some variation of that. Since I spend a lot of my professional life trying to influence others and because I am mostly successful in doing so, I sometimes take it for granted that it is just common-sense. People seem to respond positively to my advice so I thought I would share it.

1. Credibility. First and foremost, if you want to influence people you have to be credible. If you are not viewed by others as being credible then it is going to be difficult or impossible to influence others. It doesn't matter whether you believe that you are credible. Credible people are the first ones to admit when they don't know something, so the first lesson is to admit that you can't know everything. If you don't know something then keep your mouth shut. The second lesson is that it can take years to gain credibility and an instant to lose it. Before opening your mouth think about whether there is a great risk of losing credibility and if so then keep your mouth shut.

2. Execution. It is very important that others view you as someone who successfully executes. You can have many flaws, but others are more tolerant of the flaws if you successfully execute. Successful execution helps to build credibility. Also, successful execution is often what grants you entry into a conversation that provides an opportunity to influence others. Don't confuse successful execution with showing up for work and not getting fired. Understanding your role is very important. There are plenty of smart people who lose focus and think they are helping by pointing out the flaws in others. Unless it is your role to do so, and it rarely is, then stay focused on your role. If you don't like your role then change it, but once you are in a role then you need to play the role the best you can.

3. Listening. Often the most influential people are the ones who say the least. If you don't listen then you can't understand others. If you don't understand others then how can you possibly respond to their questions or issues? Even if you know the other person is wrong it is important to let them have their say and it is important for you to listen to what they are saying. Listening can provide you with important information needed for negotiation.

4. Facts. It is very important to distinguish between facts and opinions when trying to influence others. If you are offering an opinion and someone else has another opinion then be tolerant of that opinion. If someone disagrees with your opinion then acknowledge the difference in opinion, but don't fight about it. Healthy debate at the appropriate times is OK, but constant debate about everything can lead to you not being invited into key discussions. When pushing facts (a) make sure your facts are correct, (b) offer supporting data if available, and (c) don't exaggerate. If someone disagrees with facts that you present to them then don't get dragged into a never-ending debate. State the facts, counter the other person's objections once, and end down the conversation. In time, you will often be proven correct.

5. Importance. Don't win the battles, but lose the war. You have a limited opportunity to influence others. You need to understand which battles are worth fighting and which ones aren't. First, if you focus on the most important issues and leave the rest to others then you are going to find yourself being more influential than them. Second, you need to let others participate in the process. You don't want to be viewed as being intolerant, inflexible, or dictatorial. Letting others win the minor things, especially the things that are just opinions of little consequence, can lead to you being more influential. Third, people need to learn and sometimes you learn through making mistakes. Letting people learn through mistakes on the minor things is better than the alternative.

6. Negotiation. Everything is a negotiation. Many non-influential people will convince themselves that they don't have to sell or compromise. They are dead wrong. If you understand that everything is a negotiation then you can come prepared for a negotiation. It has been my experience that most people come into a discussion unprepared for any kind of negotiation. The person who is most prepared is often the most influential.

7. Religion. Avoid religious discussions. It is very difficult to change a zealot's opinions. Zealots often self-destruct and they like to drag people down with them.

8. Permission. Don't ask for permission unless you want to be told no. It is important to understand that people initially react to change negatively. This is human nature. If you want to introduce a new tool or process then just do it. It is better to ask for forgiveness than it is to ask for permission. It is easier to introduce tools or process changes for yourself than it is for an entire team/organization. Become successful by yourself instead of trying to influence the team/organization to change. Over time, if the tool or process makes you significantly more successful than others then it will become apparent. Successful results provide a much stronger negotiating position. Success is also a stronger magnet than failure. People will gravitate towards success.

9. Demeanor. You will be judged by your demeanor. If you come across as lacking confidence or experience then you are going have a more difficult time influencing others. No matter how anxious, angry, depressed, etc. you are you need to hide this from others.

Edward Tufte, "Presenting Data and Information"

noreply@blogger.com (Kevin Hegg) — Wed, 07 May 2008 06:58:00 +0000

I attended Edward Tufte's "Presenting Data and Information" course this week. This course was on my To-Do list for too long, so I finally decided to attend. It helps that the course was held very close to home and only lasts one day. The course fee includes Tufte's four books: "The Visual Display of Quantitative Information", "Envisioning Information", "Visual Explanations", and "Beautiful Evidence". Overall, it was a good course and I recommend it to others.

The course topics included:

fundamental strategies of analytic design
evaluating evidence used in presentations
statistical data: tables, graphs, and semi-graphics
business, scientific, research, and financial presentations
complexity and clarity
effective presentations: on paper and in person
use of PowerPoint, video, overheads, and handouts
multi-media, internet, and websites
credibility of presentations
animation and scientific visualizations
design of computer interfaces and manuals

This is a non-technical course that provides valuable guidance to anyone working in a technical job. I found The (Six) Fundamental Principles of Analytical Design in "Beautiful Evidence" to be immediately helpful to some projects that I am working on. Once you read and understand the principles it just makes sense.

ALT.NET Seattle Summary

noreply@blogger.com (Kevin Hegg) — Mon, 21 Apr 2008 18:02:00 +0000

I attended the ALT.NET Seattle event this weekend. I had mixed emotions about attending and didn't have very high expectations. I am fairly new to ALT.NET (3 months). My first couple of weeks on the ALT.NET discussion list just about soured me on the group. There were a bunch of petty, uncivil discussions and there wasn't a lot of useful information coming out of the discussions. Luckily, I was patient and the signal-to-noise ratio has improved recently. The ALT.NET Seattle event greatly exceeded my expectations. There were a lot of intelligent, passionate people who attended. Also, a couple of the ALT.NET leaders spent a lot of effort working to keep everyone civil. I think those efforts made a huge difference. Thanks to the organizers for making this a memorable event.

Friday evening started with individuals proposing topics. The heavy posters on the ALT.NET discussion list and prolific ALT.NET bloggers dominated this activity. That wasn't surprising nor is it a criticism. I just found it interesting to watch. Next, there was a fishbowl conversation on polyglot programming. Fishbowl conversations were new to me, so it was interesting to observe. I think it was the right thing to do because there were some people who would of and could of dominated the conversation. The result is that a lot of people got to say a little bit about polyglot programming, but no one dominated the conversation. The downside is that there wasn't a lot of substance to the conversation. That's OK because it set the mood for the weekend, civil conversations where everyone gets an opportunity to be heard.

The first session that I attended on Saturday was led by John Lam and was about IronRuby and the DLR. He talked a bit about IronRuby progress. They have a ways to go. He mentioned they are using the Rubinius tests as one measure of done-ness and they have passed 89% of the tests. He talked about working with code at the meta and meta-meta level and how quickly things can get complex and hard to maintain.

The second session that I attended on Saturday was led by Dustin Campbell and was about functional programming. Most of the people in the room had very little experience with functional programming so the conversation didn't go very deep. Functional programming conversations very quickly get into how much better suited FP languages are than non-FP languages for handling concurrency. I understand how FP languages have the possibility of handling concurrency well, but I haven't seen a lot of real-world examples to prove it. Concurrent computing is complex. FP languages may help with the complexity, but they aren't going to eliminate it any time soon and they certainly aren't going to push concurrent computing into the hands of junior/intermediate programmers any time soon.

The third session that I attended on Saturday was about ASP.NET MVC. Many of the attendees use ASP.NET MVC. Phil Haack and Brad Abrams from Microsoft attended and fielded a lot of questions. We talked a bit about Microsoft's 5% adoption of MVC comments. The number was just pulled out of someone's butt and Microsoft has no idea how much MVC will be used. I think Microsoft is also very nervous about upsetting their ISV's and large corporate partners. If MVC is widely adopted then it will be very disruptive to the web control vendors. Microsoft is always fearful of incurring the wrath of large corporations who will ask "Why am I building applications with X when you are moving to Y?". The attendees asked Microsoft to stop using the 5% adoption number because it will scare a lot of managers away from adopting MVC. Some attendees also asked Microsoft to make it as easy as possible to leverage open source Javascript libraries and client-side controls. They seemed receptive to the message.

The fourth session that I attended on Saturday was led by Scott Bellware and was about Behavior Driven Development (BDD) / Context Specification. Part of the discussion was about BDD and whether Bellware had hijacked the term to push something else. He admitted that may be the case. He presented an interesting way of testing software against specifications. While I liked some of the things I saw I doubt that what he is proposing will gain any traction. While his ideas were interesting I don't think he presented a strong case for improving anything. His approach was different, not better.

The fifth session that I attended on Saturday was led by Scott Hanselman and was about whether the .NET community innovates or contributes anything back to the open source community. It seems to me that a fair amount of ALT.NET'ers are very bothered by the image that all of the innovation is happening outside of the .NET platform or only in the open source community. I don't understand the insecurity. Why does it matter where innovation happens? I contributed two comments to the conversation. First, I think open source is not as prevalent in the Microsoft space because there is a rich ecosystem of commercial companies whereas in the non-Microsoft space there isn't. Many of these commercial companies provide source code with their products, but they aren't open source companies. Second, while the .NET community didn't invent something, they have a long list of things they greatly improved. I used JUnit vs. NUnit as an example. Scott Hanselman translated my point into "innovation doesn't necessarily equal invention" which accurately summarizes my point. Someone else made the point that much of the innovation claimed by the open source community happened some time in the past. What innovation has there been recently?

I skipped the two Sunday sessions. My son and two of my grandchildren live in Seattle, so I decided to spend the morning with them instead. There were a couple of sessions that looked interesting, but nothing I couldn't live without.

The best part of the event was the socialization. Unlike a conference, user group, or Code Camp where people are expecting to be passively taught, most people came to this event to talk/socialize. I had a number of interesting conversations with people in between sessions, at dinner, and at the bar.

Devscovery April 2008 Conference Summary

noreply@blogger.com (Kevin Hegg) — Sat, 05 Apr 2008 10:23:00 +0000

I attended the Devscovery Conference last week. Overall, it was a good conference and I recommend attending it in the future. I think it is one of the best value-for-the-money conferences.

Scott Hanselman kicked off the conference with a discussion on ASP.NET 3.5 Extensions. Usually Scott's presentations are very good, but I walked out of this one somewhat disappointed. He started off by asking a couple of "How many of you use X?" questions. I don't remember what he said, but he made some comment that came across to me as the attendees were a bunch of dopes. This completely shut down the audience participation which in turn threw off Scott because he likes to engage the audience. Also, I thought Scott's material was a bit dated.

There really wasn't anything that excited me for the first session so I sat in the "Translating Architectures to Technologies" presentation by Roger Dahlman. This was supposed to be about design patterns and .NET. This presentation was horrible. Dahlman was a very poor speaker, his slides were littered with spelling mistakes, and he came across as not knowing design patterns at all. I got nothing of value from this presentation. At this point I was starting to think that I wasted my money and time attending this conference.

The second session was "C# 3.0" by Jeffrey Richter. Since I feel like I have a pretty good grasp of C# 3.0 I was expecting to maybe get a couple pearls of wisdom out of this presentation. I really liked this presentation. Jeffrey told a nice story about all of the C# 3.0 features that concluded with why all of the new features were necessary to enable LINQ. That was fine, but I knew that already. What I didn't know was that this presentation provided the foundation for the next day's threading presentation.

The third session was "Performance of Every Day Things" by Jeffrey Richter. This was an excellent presentation about how different .NET programming constructs and techniques can affect performance. I already knew most of what he presented, but there were a couple of new things that I learned about measurement that I will use immediately.

On day two I attended the "Day of Threading" presentation by Jeffrey Richter. This was four sessions on threading. I was expecting this presentation to be a refresher for much of what I already knew about threading. I was pleasantly surprised when it wasn't. Jeffrey spent most of the day discussing the Asynchronous Programming Model (APM) that he wrote about in CLR via C#, Chapter 23 and the MSDN Magazine March 2007 and November 2007 issues. One of the issues that he discussed was how difficult the APM was for most developers to use. During the presentation he knocked down the obstacles to using the APM. By leveraging the new features in C# 3.0 he greatly simplified the APM. Throughout the day he ran a number of tests to measure the performance of different threading techniques. His final implementation provided substantially better performance, lower resource utilization, and lower code complexity. He said to expect another MSDN article in the June 2008 timeframe that completes the discussion on APM. He mentioned that the Microsoft Robotics team tried to prevent this article from being published because it provides a vastly superior solution to the Concurrency and Coordination Runtime (CCR). Also, he mentioned a couple of teams in Microsoft that are using APM and are seeing major improvements in scalability. Some of the tests that he ran showed the scalability improvements and they were substantial. I didn't fully appreciate the APM, but I have the religion now. This will play a major role in my development from now on.

The first session on day three was "The Microsoft AJAX Library" by Jeff Prosise. This presentation focused on the client side entirely. He said that he presented the server side of AJAX the previous day. He did a good job presenting the internals of Microsoft's AJAX libraries and discussed how to extend/modify them.

The next three sessions by Jeff Prosise were on Silverlight, "Building Great Applications with Silverlight 1.0", "Building Great Applications with Silverlight 2.0", and "Silverlight Tips, Tricks & Best Practices". All of these presentations were good. I was a little leery about sitting through a presentation on Silverlight 1.0, but Jeff did a good job on focusing on the features that remained in 2.0. Jeff provided lots of sample code that will be helpful in further use of Silverlight.

I was a little disappointed that I couldn't attend any of John Robbins' sessions, but since I did a day of training with him back in December 2007 it wasn't a big deal. I didn't attend any of the non-Wintellect sessions since they seemed to be more focused on the mechanics of basic things. I would have been OK if Wintellect didn't co-host with Infragistics. I think the audiences that both attract are completely different.

Throwing out the first pitch at a Major League Baseball game

noreply@blogger.com (Kevin Hegg) — Mon, 31 Mar 2008 13:20:00 +0000

My 9-year nephew, Charlie Costa, is a die-hard St. Louis Cardinals baseball fan. I have fun talking trash with him because I have been a die-hard Chicago Cubs fan since about his age.

A little over one year ago Charlie was diagnosed with a pretty serious brain tumor. The doctors treated it right away, but in the fall of last year they determined they hadn't gotten all of it. At the beginning of this year he went in for another 6 weeks of treatment. This time the doctors think they got all of it, but only time will tell. This last year has been very tough on Charlie, his mother (my sister), and their family. There have been a lot of trips from Springfield, Illinois to St. Louis for treatment.

The hospital that Charlie has been treated at has an affiliation with the St. Louis Cardinals. When the opportunity for a patient at the hospital to throw out the first pitch came up, Charlie was the first one everyone thought of. There is no bigger fan than him. He will be throwing out the first pitch when the St. Louis Cardinals play the Washington Nationals on April 6. This is such a huge thrill for him and a nice end to a difficult period in his life. So, if you are watching the baseball game and you see a little kid throwing out the first pitch you know the story behind it.

High functioning teams

noreply@blogger.com (Kevin Hegg) — Mon, 24 Mar 2008 14:37:00 +0000

Over the last week on the ALT.NET list there has been a discussion about the "expert" ALT.NET'ers joining forces to form the best IT consulting business. This idea is not a new one. Many have discussed this in other venues and some have acted on it. Recently, Martin Fowler wrote about PreferDesignSkills. This is not a new idea either. What is common between both of these discussions is the focus on the individual. For the first 20 years of my career I fell into the "best individual" camp, however over the last 10 years I have been in the "best team" camp. This change in thought can lead to very different hiring practices and company culture.

When interviewing a candidate I start by evaluating the person on the things they claim to know. If you don't know what you claim to know or are as good as you claim to be in that area then that is grounds for immediate rejection. Next, I evaluate the candidate for capacity and willingness to learn. This is where Fowler and I start to disagree. Fowler values design skills above anything else, as far as I can tell. I take a much broader view. I value candidates with a strong foundation. Design skills are important, but they are just one part of the foundation. Regarding capacity to learn, I am looking for someone who can reason about a new method, technology, process, tool, etc. and develop an informed opinion about if and when to use it. If the candidate only believes in the "one true way" or "one true tool" then they have lost their capacity to learn, at least in that area. Regarding willingness to learn, I am trying to determine how self-motivated the candidate is to learn new things. Next, I evaluate the candidate on their ability to make a team better. Is this candidate willing to play a specific role on the team? Will this candidate help make other team members better? Will this candidate be respectful to other team members? Will this candidate show loyalty to the company, project, and customer? Finally, I try to form an assessment of how much of an investment is required before the candidate is a contributing member of the team.

Getting back to Fowler's design vs. platform skills choice, there are a couple of problems with his conclusions when you look at it from a team point of view. First, what are the skills that are required to round out the team this candidate will be put on? If the team doesn't have all of the skills necessary for the project to succeed then I believe the most important thing is to fill the holes in the team. It is easy to say that you are hiring the person for future and not for the project, but if you put a person on a project that they aren't a good fit for then you haven't done the candidate, company, project team, or customer any favors. Putting any butt in any seat doesn't work, in my opinion. Second, I don't believe that you can make a blanket statement "that a good programmer should be able to pick up a new platform relatively quickly". Some projects require solving hard problems and that may require a deep understanding of the platform. Third, Fowler shows clear bias in design or platform skills. He implies that platform skilled people will never be able to acquire design skills or at least not easily. After a couple of years working in Microsoft Consulting Services I have a very different perspective. I can match his horror stories one-for-one like ripping out 1,000 lines on non-working Spring.NET code and replacing it with 20 lines of .NET platform optimized code or reliance on Rational XDE design-generated code resulted in a sub-optimal .NET solution that required substantial performance tuning or a sub-optimal data access layer that failed to leverage any of the features of the DBMS. My point is not that platform is preferred over design, it's that both play an equally important role in the success of a project.

NOTE: I have nothing against Spring.NET. It is something that I have used successfully, but in this case it was used when it wasn't necessary and it was used incorrectly. The person who wrote this code was a design-focused person who was looking implement IoC and AOP everywhere and into places where it wasn't needed to meet the requirements.

There is a perpetual debate that goes like "1 top-2% developer can outperform 10 bottom-50% developers". You can replace any of the four numbers with whatever you want, but the story is the same. A highly-talented person can outperform a bunch of below-average people. As Craig Andera said at our last ALT.NET meeting, by its very definition we are always going to have many more below-average than highly-talented developers. I've been there and done that. I've been the top developer that outperformed X other developers. So, what good came from it? Certainly, my customers appreciated the efforts, but the team didn't. There are only so many hours in the day that a highly-talented person can work and because there are so few of them it isn't a scalable practice. When I left a project the team wasn't able to continue on the same path or velocity because they didn't get it. When the customer finally realizes the maintenance issues it is too late. Is there something better than this? I think so and have been increasingly working differently over the last 10 years. Instead of stating that I can outperform X developers I ask "How can I make X developers and the team as a whole more productive?". I don't have any numbers to support this claim, but I believe this change in attitude can lead to higher functioning teams. I believe the "outperform" attitude causes you to write off a bunch of people unnecessarily. As in any industry, the IT profession has some useless individuals, but I think the number is a lot less than elitists believe it is.

Back to the original question, can a bunch of experts join forces to form the best IT consulting business? It is certainly possible, but I don't believe it is probable and its certainly harder than forming a corporate structure. I believe the question is too focused on the individual. It takes a lot more than individual effort, no matter how expert, to build the best company. It takes a lot more than individual effort, no matter how expert, to grow beyond a small company. High functioning teams are an essential ingredient to building the best company.

Are we going to see major changes in data management?

noreply@blogger.com (Kevin Hegg) — Fri, 14 Mar 2008 17:50:00 +0000

Row-oriented Relational Database Management Systems (R-RDBMS) have grown in popularity since the early 1980's to the point where the overwhelming majority of data management by the early 2000's was handled by RDBMS's. There have always been alternatives to R-RDBMS's, but until recently none of the alternatives have provided sustainable technical advantages or gained significant market share. So, what's different now and is it going to impact the life of a software developer?

Early in my career I had the opportunity to program IBM's SQL/DS and shortly after that Oracle (2.0 or 3.0, I forget). Relational database development was simple for me. In late 1986 Sybase released SQL Server 1.0. I was assigned to a project that was going to use it, but since the project had large-scale data and significant performance requirements I needed special training. I spent a couple of weeks at Sybase learning SQL Server internals and there was no going back. Since then I have kept up with Sybase, Microsoft SQL Server, and Oracle database internals and have spent a lot of time designing and tuning databases. At this point in my career I feel that I can squeeze every drop of performance out of any R-RDBMS. Now that we have established the I am an experienced database guy, let's resume the discussion. :-)

As R-RDBMS's matured vendors began throwing everything into a single product. Michael Stonebraker discusses the "one size fits all" approach and the problems with the approach here and here. He lays the foundation for when and why R-RDBMS's start to fall apart. From my personal experience building applications to process sensor data and financial data feeds it took a lot of expertise, effort, and cost to tune commercial R-RDBMS's to meet the performance requirements.

With the rapidly decreasing cost and increasing capacity of CPU, RAM, and disk storage the alternatives to R-RDBMS start to become much more attractive. This increased the appetite to process and store substantially more data and this is now testing the limits for R-RDBMS technology. As solutions scale into the petabyte range many of the traditional data modeling techniques like Ralph Kimball's dimensional modeling are less successful. The same can be said for indexing (bit-map) and physical partitioning schemes. What worked for 10 GB - 10 TB doesn't work near as well for 10 TB - 10+ PB. Also, the problems become more severe as your processing requirements approach real-time.

Over the last couple of years the number of columnar storage solutions has increased noticeably, led by Google's BigTable, Sybase IQ, Stonebraker's research with C-Store, etc. Much of the benefit from columnar storage comes from the ability to compress data and to substantially reduce the I/O's needed to complete a query. Google decided not to use SQL for a query language while other columnar solutions stuck with SQL, but what this showed is that SQL is also reaching its limits.

Next, Stonebraker published his research on H-Store. He proposes that pure OLTP applications can see dramatic improvements in performance from massive simplification of the database engine and performing in-memory, distributed processing of data. Also, he proposes to do away with SQL as the query language. Werner Vogels, who is well-respected in the distributed systems community, expresses some scepticism. While he likes Stonebraker's challenge to provide 50x improvements, he is worried that Stonebraker is only solving the scale-up problem when instead he should be focused on the scale-out problem, similar to what Google and Amazon do. That was my concern initially, but the more that I think about it I don't think that H-Store is necessarily unable to scale-out. I think it is just a matter of time. Now that there is a working implementation of H-Store, they can now focus on scaling-out.

This led into a (contentious, if you read the blog comments) debate between Curt Monash and Stonebraker about how many different types of databases should be supported. It doesn't matter so much on the exact categorizations. What does matter is that the R-RDBMS world is "hitting the wall" with increasingly regularity with the "one size fits all" solution and that is driving the database market to come up with a variety of alternate solutions for each category of data management. I don't believe that R-RDBMS's offer a good solution for managing XML, semi-structured, and unstructured data, especially as the amount of data increases and the processing requirements approach real-time. Also, I don't believe that SQL is the correct language for processing this data.

I disagree with a some of Stonebraker's comments though. He thinks the low-end OLTP market will go almost entirely to open-source databases. I don't believe it is that simple. First, brand loyalty should not be underestimated. The cost difference between low-end commercial and open-source R-RDBMS's isn't significant enough to drive people in one direction or another. Second, the cost and complexity of swapping out R-RDBMS's for legacy systems far outweighs the license cost savings. Third, I think that solutions like Amazon's SimpleDB and Microsoft's SQL Server Data Services will be a more attractive option for the low-end than open-source. Not only do they eliminate the software license fees, but they eliminate the hardware and system/database administration costs. Head-count reduction is far more important to many organizations than software license reduction.

I agree with Stonebraker that the current R-RDBMS vendors are at risk of getting caught in the middle as we undergo change in the data management market. Also, I agree that the R-RDBMS's are getting too complex and this complexity is unnecessary. Finally, there is one conclusion that I would like to add. If solutions like H-Store are able to eliminate transactions, concurrency management, and other complexities then the benefits will be so great that they will quickly permeate into the mid-range solutions.

How is the developer's life going to change? First, they will potentially have to unlearn a lot of database relational design and programming habits. For an H-Store type of solution, this could result in a substantial reduction in the amount and complexity of the code. It will bring us a lot closer to being able to automatically generate the data access layer from a data model since much of the programmer intervention is due to transaction and concurrency management. Second, if we find a suitable replacement for SQL then we can potentially eliminate another big pain, Object/Relational Mapping. Am I the only who thinks that every O/RM tool completely sucks? I know what problem they are trying to solve. I just don't think they are solving the problem. Yes, I can build a working application with them, but I feel so dirty afterwards. I feel like you are just trading one problem for another. Third, in the area of columnar storage and non-relational data solutions I think we will have to be prepared for more developer effort in the short-term. The Google-imitators are just now learning that BigTable and MapReduce type of solutions are no free ride. The lack of tool support and best practices is something that will be fixed over time, but in the short-term it will be an issue.

When are the major changes going to happen? They already are happening. When is it going to impact the average customer or developer? I am not smart enough to accurately predict this, but I think it is close enough that I am paying attention. If and when changes start to happen, I want to be ready.

MiX 2008 Conference Summary

noreply@blogger.com (Kevin Hegg) — Wed, 12 Mar 2008 15:26:00 +0000

I attended the MiX 2008 Conference last week. Overall, it was a good conference and I recommend attending it in the future.

The keynote speeches were pretty dry. The best part was the pre-keynote entertainment by Vince Mira. He did an incredible job singing Johnny Cash songs. Ray Ozzie, Steve Ballmer, and Guy Kawasaki were boring. Dean Hachamovitch provided a good overview of IE 8. Scott Guthrie talked about Silverlight, ASP.NET futures, and the Visual Studio 2008/Windows Server 2008/SQL Server 2008 launch. Deep Zoom (previously called Seadragon) looks interesting.

The "Crossing the Usability Chasm - Advanced and Adaptive User Interfaces" talk was well-done. Gil Hupert-Graff provided good advice on user interface development based upon his research.

The "Building Rich Internet Applications Using Microsoft Silverlight 2, Parts 1 & 2" talks provided an introduction into Silverlight 2.0 development. Since I haven't done much with Silverlight this provided me with exactly the introduction I was looking for.

The "RESTful Data Services with the ADO.NET Data Services Framework" and "Building RESTful Real World Applications with ADO.NET Data Services" talks provided a good overview of the ADO.NET Data Services Framework. I liked what I heard and plan on diving into this deeper over the next month.

The "Introducing SQL Server Data Services" talk provided a decent overview of SQL Server Data Services (SSDS). Contrary to the SSDS team's blog the current state of SSDS makes it a very comparable product to Amazon's SimpleDB. I am sure that over time Microsoft will expose more of SQL Server through SSDS and that will allow it to surpass SimpleDB. I am waiting for my account so that I can experiment with SSDS. I am interested in seeing how complex of a data model you can build or how large of a database you can create before it no longer makes sense to use SSDS. Just as SimpleDB, SSDS should be able to satisfy the small, simple database niche well. As Microsoft rolls out more of their Software plus Services offerings I think they are uniquely positioned to benefit from product integration. I wonder how long it will take before we hear more anti-trust complaints.

The "Building Great AJAX Applications from Scratch Using ASP.NET 3.5 and Visual Studio 2008" talk provided an introduction to AJAX development features in .NET 3.5 and Visual Studio 2008. Even though .NET 3.5 and Visual Studio 2008 were just released this information has been out in various forums for many months. Since I have already been playing with these bits for a couple of months there wasn't any new information for me in this talk.

The "Cross-Browser Layout with Internet Explorer 8" talk provided a lot of useful information. The most important thing that was discussed (also in the keynote) was the commitment that Microsoft is making to CSS 2.1 compliance. This should calm a lot of web developers down if compliance is mostly achieved. Microsoft seems to be doing a lot to build unit tests to prove compliance. It will be interesting to see how much of the non-Microsoft community contributes to the suite of unit tests. The performance improvements in IE 8 look promising too.

The "Developing ASP.NET Applications Using the Model View Controller (MVC) Pattern" talk was probably the best at this conference. Scott Hanselman is always entertaining, but I think the MVC bits are going to be a huge hit for architects. Scott made a point of saying that MVC is optional, but if the MVC bits evolve as I believe they will evolve then there is not going to be much of a reason to develop ASP.NET applications any other way. I am looking forward to digging into the MVC bits.

The "Using an Internet Service Bus to Build Next Generation Applications and Services" talk provided a good overview of BizTalk Services, which is not BizTalk Server. Just like SSDS, BizTalk Services is going to address a niche set of problems. I look forward to playing with these bits.

The "Using the Microsoft Sync Framework and FeedSync" talk provided an excellent overview of the Sync Framework. I think this is going to be a huge hit with developers. I look forward to playing with these bits.

Leaving Microsoft

noreply@blogger.com (Kevin Hegg) — Wed, 12 Mar 2008 12:37:00 +0000

I joined Microsoft in 2005 to work in the Intelligence and Homeland Security practice of Microsoft Services. I resigned from Microsoft on Feb. 29, 2008.

I am grateful for the opportunity I had to work for Microsoft. During my stay at Microsoft I worked as a software development consultant, architect, and manager, sometimes all at the same time. My managers were always very supportive of me. They put me in charge of growing the business and managing the project delivery in a couple of key areas and gave me a lot of freedom to do so. My team and I didn't disappoint. We had very significant revenue growth driven by successful project delivery due to a lot of hard work.

If everything was so great then why did I leave Microsoft? I am at a point in my life where I can do things differently than most. I am 49 years old, my children have been raised, I am virtually debt-free, I have a decent amount of money saved, and I live very frugally. So, I am leaving Microsoft because I can, not because of any specific negative reason. I have a couple of itches that need to be scratched. Who knows, after I have scratched my itches I might consider rejoining Microsoft.

What's next? First, I am not retiring even though I can afford to. Anyone who knows me knows that I like to work hard. I will probably continue working until I drop. I figure I have at least 20 years of productive work life remaining. Second, I am considering whether to resume my Ph.D. studies. Over 20 years ago I dropped out of the Computer Science Ph.D. program at University of Michigan because I had a wife and three children to support. While I can handle the academics I am not sure if I have the temperament to go through a formal program. Third, I like entrepreneurial work. I enjoyed the couple of start-ups that I have been involved with. I see one or more new ventures in my future.

At a minimum, I am taking the month of March off. I have a long list of technical topics that I want to dig into. After March I have no specific timeline for my next venture.

Comments on "Thinking Inside the Box and Premature Optimization"

noreply@blogger.com (Kevin Hegg) — Wed, 27 Feb 2008 21:45:00 +0000

Here are my comments on Matt Podwysocki's post on premature optimization. Since I was also at the last DC ALT.NET we are just continuing our discussion.

All programming languages, frameworks, operating systems, databases, etc. have known best practices and many of the best practices are related to improving performance. I don't buy into the argument that following best practices is premature optimization. If you are going to implement something and it takes no more effort to use the best practice than it does to use a non-optimized approach then it makes no sense to me to implement things sub-optimally. Poor development habits beget poor development habits. Unless you are building trivial applications the aggregation of non-optimized software can have substantial negative impact to the application.

Unless you are new to software development you have some knowledge and experiences to bring to bear on the new thing you are learning. If you know the .NET framework and you are learning the Java framework, then you can apply lessons learned from .NET to Java even though it may not always be an apples-to-apples comparison. Implementing efficient algorithms, managing resources efficiently, minimizing I/O, building secure code, etc. are things that most software applications needs to worry about. It isn't too difficult to apply what you know about .NET to learn the equivalent thing in Java, or vice versa.

Should you think five steps ahead or not? I am a strong believer in defining a project's vision and requirements and letting them determine where the design and implementation should go. You should only design and implement the minimum features needed to meet the customer's requirements, however not taking into account the project's vision early in the project can result in some very costly redesign later in the project's lifecycle. Some applications are complex and require a certain amount of thought about the future. I agree with Matt that developers tend to carry implementation from project to project, unfortunately this isn't a good idea unless it satisfies the project requirements. I find that the less experienced developers are the more they practice the "whatever I did on my last project is the one true way" style of development.

So, what is it about the belief that learning "insert language or framework here" is a solution to design and implementation issues? I view this as the Survivor-style of development. Put yourself on a stranded island with minimal supplies and you will learn to survive, sometimes very well. Having learned a lot of programming languages and frameworks in my career I am very skeptical that there is a panacea to design and implementation issues. Most developers know how to implement complex solutions (or poor, complex solutions). It takes a lot of talent to come up with a simple solution. There was a time when Java, .NET, and other frameworks were considered to be lean. If Ruby on Rails achieves a substantial following then I am sure that we lament the bloat in it someday. I think it is inevitable that popular development frameworks will have additional frameworks added or ported. You shouldn't get frustrated by this. Instead, you should learn to use the minimal features needs to solve the problem you are working on.

I like some of the things I have read about Domain-Driven Design, but unfortunately I feel like it provides us with a small step forward, but not any major improvements. In general, I am still very disappointed with all of the modeling approaches: data, object, domain, etc. First, models are difficult to map back to requirements so it is very hard to have a meaningful discussion about the model with users. Second, models have a hard time accurately and completely describing solutions, especially complex and/or large solutions. Third, keeping the model and code synchronized is very difficult and costly. As much as we would like models to remain valid throughout the entire lifecycle of an application, the reality is that very few projects maintain their models past the early stages of the project. Fourth, I believe that many of the most successful solutions built were built without much modeling, certainly without a complete system model. So, where am I going with this? We need to glean whatever tidbits of knowledge that we can from the design gurus, but we need to be realistic about how much time we spend in the code vs. the model.

First blog

noreply@blogger.com (Kevin Hegg) — Mon, 18 Feb 2008 22:12:00 +0000

I am going to dip my toe in the blog pond. I spend a lot of time reading blogs, but I have always made the excuse of being to busy to blog myself. I have been in the computer industry 30 years. I have had the fortune to work on a lot of interesting projects, played with a lot of interesting technologies, and met a lot of interesting people along the way.

Here are some things I am interested in.

Large-scale data management. I am a child of the RDBMS era and have used Oracle since 3.0, Sybase since 1.0, and SQL Server since its first release. I have a lot of experience squeezing every drop of performance out of these products. There are lots of large-scale data management issues that interest me, but the current RDBMS products are having increasing difficulty playing in this space.
Internals. I love to tinker with operating systems internals. Early in my career I spent a lot of time in the Unix kernel, but in the latter part of my career I have spent more time in Windows internals than Unix/Linux. Since .NET was released I have spent as much time as possible learning the .NET framework internals. I never spent any meaningful time learning Java VM internals, not because of any dislike of Java, but just due to lack of time.
Project recovery. I have spent a significant amount of my career helping to bail out failing projects. Anyone who does this more than once or voluntarily is a warped individual. I get a sense of satisfaction knowing that I found and solved problems that no one else wanted to touch. Project recoveries involve:
- Identifying and correcting systems lifecycle dysfunction.
- Diagnosing and solving hard software correctness problems.
- Diagnosing and solving performance issues.
High productivity. I enjoy studying tools, processes, and personalities to determine what makes individuals and teams highly productive.
Computer industry trends. I feel fortunate that my job is also my hobby. I can't imagine ever getting bored with my job. There are thousands of new things to learn every day.
Chicago Cubs. I have been a Cubs fan almost from the day I knew how to spell baseball. I am used to "waiting until next year". One of these years ...

Kevin Hegg