KwBlog

DDD is Not About Perfection

2009-11-10T07:30:00.005-05:00

The main practice of DDD (Domain Driven Design) is refactoring to deeper insight. The idea is very similar to much of what the Agile practices preach. Namely, when you find you have

misunderstood something or made a mistake
been given bad information
learned something new that changes previous assumptions

You go back and you update your code to reflect your new understanding. In Agile, this is called embracing change, in DDD its refactoring to deeper insight.

This is one of those principles that seems like it shouldn't need to be said. If someone knows that what they've done isn't right anymore who WOULDN'T go back and fix it?! But it turns out this is one place where real life and theory don't line up. The thinking usually goes something like this:

We've spent a lot of time working on this feature, we're out of budget, it works fine in 90% of the cases, and we can just add this little hack that will take care of the other 10%. Therefore it makes more sense for us to just do the little hack.

I'm sure you've seen this line of thinking before, so you're already anticipating that I'm going to say this is stupid. But hold on. Its not stupid. This is actually totally sensible thinking, but there are two problems:

You made up that thing about "90% of the cases." You actually have no idea how often the "edge cases" that are a problem for you will crop up. And for all you know, those may be the more important cases, which your system can now only accommodate with a weird hack.
You haven't considered changes that might arise in the future, or insights you may have in the future. If any do crop up that are related to the "10%" edge cases, you're now going to be forced to build hacks on top of hacks (on top of hacks, on top of hacks, on top of hacks...).

So obviously you should never write the hack, you should always embrace the change and refactor to deeper insight.

Nope, sorry, wrong again. Unfortunately we are here living in the real world. We have real world constraints: Time and Money. If we always took the time to fix everything we discovered we got slightly wrong we would never deliver a product, ever. And shipping is a feature.

So... sometimes we're going to have to hack, which might get us into a world of trouble? And sometimes we're going to have to blow our budget refactoring? How do we know when to do which?

In Strategic Design - Responsibility Traps Eric Evans (the founder of DDD) says "the whole system will not be well designed." This is an inescapable fact if you're working on a large complex system. If you're working on a small simple system, maybe you can pull it off, but even then I doubt it.

How can the guy who's whole development technique revolves around refactoring to deeper insight say the whole system will not be well designed? Well, he has an answer of sorts. Evans says that what you need to do is identify your application's Core Domain. What is it that your application does that sets it apart, makes it important, or provides the most value for the users? That's your Core Domain. Now if a change crops up in the Core Domain, you refactor to deeper insight. This is the most important part of your app! It's the part you'll be building on for everything else in your app. This is the part of your app has to be perfectly designed.

But what about parts that are NOT the Core Domain? Or what if it's not so easy to define your app's Core Domain? Then what do you do? Well, you weigh the options, and take your best guess.

Is there time in the budget?
How hard will it be to refactor to account for this change?
How much better will the app be if you do change it (time saved? # people affected? usability? performance? correctness?)?
Can you convince yourself it is unlikely other changes will have to be made in the future that will be related to this change?

Once you have answers to these questions that are as accurate as you can manage, then you have to just guess. Because your answers to these questions are NOT scientific. And you have no way to predict the future. But you have to try anyway, so you guess.

If you think it's going to be very hard to accommodate the change and you think the change wont make your app all that much better, then hack it. If that's reversed, refactor it.

I'm a programmer, and I'm hungry for perfection, so I always lean toward wanting to refactor it. I think refactoring to make it correct anytime you have the time and ability is the right move. You'll end up with an app you're proud of, that works better for your users, and is easier to maintain and update. But we have to face facts! Sometimes the cost is simply too high. Sometimes we're forced to hack it now, and pay the consequences later.

The good news is that DDD helps tremendously with this. If you can define a Core Domain, you're that much better off because you have now decided what your application is all about. This will help you in every decision you need to make.

Further more, DDD is drastically easier to refactor and maintain than spaghetti code. So the cost of refactoring to deeper insight is lowered when you're writing DDD.

But in the end, we have to realize that DDD is not about writing perfect code. Its about writing good code, that makes the complexity of your application manageable. But we know that despite all the benefits DDD can bring, it doesn't promise perfection. The simple truth is that software is very very hard to write, and designing enterprise applications is even harder. So the best we can do is write Good Enough software, like they say in The Pragmatic Programmer. But hopefully DDD will help us to raise the bar on how good that software is.

Knowledge in the Head and in the World

2009-11-02T21:20:00.001-05:00

A looooooooong time ago I wrote a post called Theory of Software Usability. This post was primarily about the tradeoff between “Ease of Learning” and “Ease of Use.”

I used my favorite example of Vim vs. Notepad. Vim is an advanced modal editor that a n00b wont even be able to get text into if they don’t know what they’re doing. Whereas Notepad is just about the simplest application you can imagine that anyone can figure out how to use.

My argument was that Vim is extremely usable but difficult to learn while Notepad is extremely learnable, but not really all that usable. So there is an implicit tradeoff between Learnability and Usability.

Recently I have been reading The Design of Everyday Things and I can across a concept that is an interesting corollary to the Usability vs. Learnability issue. The book presents “The Tradeoff between Knowledge in the World and in the Head”. Knowledge in the world is simply information that is readily available in the world, so you don’t have to learn it, or at least you don’t have to learn too much. In The Design of Everyday Things an example of a typist is used.

“Many typists have not memorized the keyboard. Usually each letter is labeled, so nontypists can hunt and peck letter by letter, relying on knowledge in the world and minimizing the time required for learning. The problem is that such typing is slow and difficult… But as long as the typist needs to watch the keyboard, the speed is limited.

If a person needs to type large amounts of material regularly, further investment is worthwhile: a course, a book, or an interactive computer program… It takes several hours to learn the system and several months to become expert. But the payoff of all this effort is increased typing speed, increased accuracy, and decreased mental load and effort at the time of typing.”

At the end the book presents some tradeoffs between knowledge in the head and in the world in terms of 5 properties, retrievability, learning, efficiency of use, ease of use at first encounter, and aesthetics. It breaks down like this, knowledge in the world is retrievable, requires little to no learning, is not efficient, is easy to use at first encounter, and can be unaesthetic and inelegant. On the other hand, knowledge in the head is not retrievable, requires lots of learning, is efficient, is not easy at first encounter, and can lead to better aesthetics. So basically, they are at odds with each other.

Bringing back my Vim vs Notepad example, we can see how this fits right in. Notepad puts all the knowledge you need “in the world.” All the labeled keys on your keyboard do exactly what you’d expect and the other functions are clearly labeled in the menus. In Vim on the other hand, you can’t even enter text until you learn the “i” command. The knowledge must be in your head. All the tradeoffs listed above apply perfectly to this example.

I think this is a very important concept to keep in mind when doing software design. Who is your user? What job are they doing? Often often will they be doing that job? Will new people need to figure it out on the fly, or will the same people always do it over and over again? The answers to these questions will help you decide if you should emphasize knowledge in the world or knowledge in the head. If you are building a public facing website that many people will visit, you want to emphasize knowledge in the world. If you are building an application for something like data entry you may want to emphasize knowledge in the head.

The important thing to take away from this is that there is a tradeoff and you have to make a decision one way or the other. Knowledge in the world is not always better than knowledge in the head, and vice versa. Pay attention to what you are building and who you are building it for and design accordingly.

Intangible Value

2009-10-20T08:02:00.001-04:00

Rory Sutherland is a marketer who recently gave a TED talk. The talk is about 16 minutes long and is hilarious, you should totally watch it.

He opens the talk with this gem, "if you want to live in a world in the future where there are fewer material goods, you can either live in a world which is poorer (which people generally don't like), or you can live in a world where actually intangible value constitutes a greater part of overall value."

Seth Godin is another marketer who has a short and sweet blog post called Creating sustainable competitive advantage in which he argues that competitive advantage rarely comes from proprietary technology or technological barrier to entry. In other words, technology alone will not allow a business to succeed because its competitors will quickly be able to copy the technology.

He has a list of things you can do to gain competitive advantage, 3 of which apply here:

You can build a network (which can take many forms--natural monopolies are organizations where the market is better off when there's only one of you).
You can build a brand (shorthand for relationships, beliefs, trust, permission and word of mouth).
You can create a constantly innovating organization where extraordinary employees thrive.

Tying in with Sutherland, these are about adding intangible value. You can gain intangible value by building a network around your product, or by building trust and a name for yourself ("brand"), or by being a constantly innovating organization.

The last is half intangible, half tangible. The actual innovations produced are tangible, but being innovative adds its own intangible value, both to your customers as well as your own employees and even to job applicants! Emphasizing extraordinary employees is a relatively intangible thing which can produce tangible benefits across the board (better products, faster delivery, lower employee turn over rate, and better employees). Thus the competitive advantage.

I think this concept of Intangible Value can be extended into software itself. Possibly the best example is usability. Some software usability concerns are tangible: how long does it take someone to accomplish a task, how many clicks are required, etc. But other usability concerns are intangible: does the user enjoy using the software or does it make them want to shoot themselves in the face.

Seth Godin's post talks about things you can do today to gain competitive advantage, but Rory Sutherland's talk is about how we as a people need to learn to value intangible things more. This is much harder than it sounds.

When you're evaluating two products, you look at the feature lists. If one product has more features, you're likely to decide that it is the better product. But while it may have more features, it may also make people want to shoot themselves in the face. Can we include that on the list of features?

As an example, compare Microsoft Visio to Balsamiq Mockups. Visio is a very full featured product which is ridiculously flexible and powerful compared to Balsamiq. But everyone I know likes Balsamiq better. Why? It's the shoot myself in the face factor. Balsamiq is faster and easier to use. In fact, it's a joy to work with. That is a relatively intangible benefit, but it's real.

As another example, take 37 signals. I have not personally used their products, but I know from what I've heard and from what they've said that their focus as a company is on building slim lined and usable software. There are big box alternatives to their products that have been around for much longer and are far more "configurable," but people love 37 signals. Again, for mostly intangible reasons I think.

So intangible value is a real thing which is often overlooked by the "deciders" but always appreciated by the users. The challenge for those of us who design software is to figure out how to add that intangible value into our products, and how to make potential users aware of it. The challenge for people in general is to recognize intangible value when they come across it and not dismiss it as unimportant.

What a Programmer Wants in a Manager

2009-10-14T11:40:00.000-04:00

Management is an oddly fascinating subject. It's kind of dirty word these days, but when you distill out all the nonsense and get down to what it's really about, it's interesting. Managing programmers, or "knowledge workers," is, in many ways, a special case and requires special consideration. This is because you can't break a programmer's job into a series of reproducible steps. Programming is an inherently creative job, which is why it's often compared to craftsmanship.

In thinking about how to manage programmers, I tend to empathize with the programmers more than the managers. The way I see it, a programming shop's number one expense AND number one asset is its programmers. So it seems pretty clear to me that you aught to do everything you possible can to keep those programmers working well. The challenge is of course figuring out how you actually do this.

The book First, Break All the Rules, suggests that a manager's job is to discover the talents of his people and direct those talents to the business's goals. I like looking at it this way because it indicates that the manager should recognize what his people are good at, and then let them go be good at it. The trick then is to make sure the things they're being good at are also the things the business needs to be good at.

Joel Spolsky goes even further and says that a manager's "most important job is to run around the room, moving the furniture out of the way, so people can concentrate on their work." At least, he says that's how managers at Microsoft behaved. Again, there is a large focus on getting out of the way so your people can work.

Recently Fog Creek announced a series of training videos they are selling. The video series is $2000 and a little conceited if you ask me... But in the promo video, one of the Fog Creek dudes says, "Developers are assumed to know the right answer. So you don't start from a position of negotiating about whether or not you could possibly have the right answer. You assume they have the right answer and that's there job to explain to you why its the best solution, not why its the wrong solution or the right solution." In this we're seeing some of that "get out of the way" mentality but also a certain amount of built in trust.

I have my own theory on the best way a manager should behave, which is strongly influenced by these references as well as just about every word in Peopleware. I think it breaks down like this:

Trust your people
Value your people's time over just about everything

When I say trust, I'm not talking about blind trust. I'm not a complete idiot! But I am talking about a change in tone. A manager needs to set the goals and objectives people are working toward, and a manager needs to ensure those goals and objectives are being met correctly and effectively. But a manager should trust their people to do the work right and in the best way possible. As a manager, you can ask for proof of why what people are doing is the BEST way. But you need to be careful that you don't demand proof they are not completely wrong. There is a subtle difference here which has a huge effect on morale.

I think this is very important. Programmers want to be treated like experts. They want their opinions to matter. If you stifle that, you will end up with frustrated programmers, and frustrated programmers don't work as well. Worse, since programming is a creative activity, if you're actually stifling creativity you're ruining your product. The best way to improve morale and encourage creativity is to offer up some trust and treat people like adults.

If your people know that they are trusted, and that their ideas will be seriously considered, they're more likely to bring ideas to you. They're more likely to think outside the box. These can only be good things. If instead they feel untrusted and are afraid their ideas will always be shot down, they wont bring anything to you.

Programming can be detailed tedious work. Nothing pisses a programmer off more than the impression that the time and effort they have spent was wasted or simply unappreciated. This is why programmers hate dropping a project in the middle to work on something "new" and "higher priority" that "just came up." New higher priority things DO come up at the last minute. But if a manager just tosses it on a programmer's desk and says, "get this done first" they're not valuing the programmers time.

I firmly believe a manager's job is to push the furniture out of the way so the programmers can get their work done. But, we're not really talking about furniture. We're talking about all the complexity of a project. The inter-relations between different teams, the ever changing client demands, the relative priorities of different assignments. These are things the managers should be focused on working out so that programmers don't have to spend so much time worrying about them.

Again, much of this is just tone. When a manager is laying out the work that needs to be done and the order it needs to be done in, and who needs to do what work when... they can treat this as a power opportunity for themselves. To hand out assignments from on high with little regard for explaining the circumstances to the programmers and a simple expectation that the programmers should take it and get it done. Or they can treat this as an opportunity to indicate that the goal is to optimize the developers time and so they focus and do their best work instead of dealing with the sticky details of the real world that programmers simply don't like.

A manager has to do the same work either way. In the end, much of management really comes down to politics. And since I believe the programmers are the most important asset going for a business, I believe the politics should be oriented around keeping the programmer's morale high, avoiding frustration as much as possible, and engendering a corporate culture where the programmers feel their work is valued and important.

Responsibility Traps

2009-10-08T07:30:00.002-04:00

Over at InfoQ there is a presentation by Eric Evans called Strategic Design - Responsibility Traps. This presentation is full of all sorts of gems like "one irresponsible programmer can keep any 5 top notch programmers busy."

His presentation is all about how to take legacy systems and apply DDD (Domain Driven Design) to them when doing new development. In the process he addresses lots of really important issues like, "the whole system will not be well designed," which I may dedicate a whole post to in the future.

What I want to mention here is what he calls Responsibility Traps. These are traps that responsible programmers (read: good programmers) fall into. In the presentation he mentions two:

Building a platform to make others productive
Cleaning up other people's mess

The reason why these are traps is that "you are not the one who finally delivers the sexy new capability." Instead, what you've done is "made the irresponsible programmers look even better". Ultimately Evans believes this leads to this fact: "Because the best programmers are busy making the platform strong, the actual delivery of the Core Domain is being done by irresponsible programmers."

I had never really thought about this, but I think he's dead on. Your real star players, over time, gravitate to working on more abstract projects like frameworks and platforms and very technology oriented things. This migration makes sense on the surface, because these things are harder, so you want your best people working on them.

The problem with this is that these are not the projects that make the biggest difference to the product as a whole. These things are not part of the Core Domain. They may be important. In fact they may be absolutely essential, but it doesn't change the fact they are not the MOST important. They are not what your application is all about. So what happens is your irresponsible developers end up working on the MOST important parts of the application.

Evans consistently uses the word "irresponsible" instead of "bad" or "weak". I think this is more than a political move on his part. The reason why it's a problem that the irresponsible developers are writing the core domain is that they write it irresponsibly, not that they write it badly. What does that mean?

They hack through it
They leave it a mess
They don't question the design when it stops working well
They keep bolting new stuff on top instead of refactoring
They introduce performance and maintenance problems

These things are "bad", but for the most part they are not outwardly noticeable. Irresponsible developers may be fully capable of delivering a project with few to no bugs that looks just like what the business people asked for. But because they wrote it irresponsibly it will be a thorn in the side of the project from then on. Unfortunately, the business people wont know that. And if the responsible people tell them, the business people either a) wont believe them or b) wont understand the severity.

There is a certain amount of "the sky is falling!" here. The responsible people say there is a problem in something that has been written. The business people say, ok, go fix it. The responsible people toil away for awhile and return with the issues resolved. The business people don't see a difference, usually there ISN'T an immediate noticeable difference. So it looks like these responsible people keep shouting about the falling sky and then spinning their wheels on nothing for weeks, while the irresponsible people are off getting things done (and creating more problems).

The point Evans is trying to make with this is that the responsible developers are actually being somewhat irresponsible by allowing this to happen. This isn't about being political and trying to make yourself look good. I mean, it's partly about that. But it's really about being truly responsible and embracing the fact that the whole system will not be perfect and focusing on making the most important parts be the parts that are the highest quality. Those are the parts the responsible people should be working on, those are the parts YOU should be working on!

Reading List

2009-10-05T07:41:00.004-04:00

I've noticed that most programmers don't read books on programming. This has in fact included me, until recently. I was willing to read books on things tangentially related to programming, like management books or usability books, but I wouldn't read books about the art of actually programming.

I think the reason for this is most books that actually talk about programming are usually about some specific technology. Whether it be a language book on C#, or a framework book on WCF, or a technology book on SQL Server. These books can be useful, but their usefulness is always limited. And many times, these books can be down right horrible. This means most people's impression of these books is they really aren't worth the money. Especially when most of the information in them can be gleaned from Google searches.

However, there is another class of programming book which doesn't fall into this specific technology category but is still definitely about programming. These books are timeless in nature because they deal with the actual art of programming instead of specific syntax or frameworks or tools. These books are also ridiculously valuable because they have the ability to dramatically expand your programming horizons, making you a much better developer.

This reading list will only contain books which I feel have a certain timeless quality to them and are mostly independent of language or framework. This list is also not complete. I'm really only just beginning to discover the wealth of knowledge available in printed media, so I'll keep coming back and updating this as I discover more.

The Art of Programming

Domain-Driven Design: Tackling Complexity in the Heart of Software - Eric Evans

This is the most comprehensive book on enterprise application development I've ever encountered. For me it was a complete game changer. The book itself presents you with concepts and examples and patterns, but it doesn't get bogged down with implementation issues. The result is after reading it you know you HAVE to start writing code this way, but you really don't know how to write it just yet. Even given that, I still highly recommended it to anyone who writes applications with any non-trivial business logic in an Object Oriented language.

Patterns of Enterprise Application Architecture - Martin Fowler

The Evans DDD book references this Fowler book on just about every other page. If you've ever read the "Gang of Four" patterns book, or any books that repackaged those patterns you know what a Design Pattern is all about and you're probably bored with them. The patterns in this book are so much more influential and important than the GoF patterns, so don't let the word "patterns" scare you off. Think of this book as the text book for anyone developing multi-user "business" web apps or rich clients. It contains all the possible answers to nearly ever major problem you'll be faced with.

The Pragmatic Programmer: From Journeyman to Master - Andrew Hunt and David Thomas

This book wont give you dramatic new ways to write code. Instead it will give you dramatic new ways to think about code and your responsibilities as someone who writes code. It includes what to my mind is the beginnings of Agile Programming and many of the SOLID design principles. It is also packed with parables that seem obvious until you realize they've happened to you at work. It should be required reading for any developer.

Managing Programmers

Peopleware: Productive Projects and Teams (Second Edition) - Tom DeMarco and Timothy Lister

This book is some interesting cross between a book for managers and a book for programmers. Its a great read and is likely one of the most influential books in our industry. It has clearly defined much of the culture of companies like Fog Creek and Microsoft and even Google. You should read it, but if you don't work for one of those companies be warned you might get a little depressed.

First, Break All the Rules: What the World's Greatest Managers Do Differently

If you are in any kind of "Management" role you should absolutely read this book. If you're not, you should still read this book, cause it will help you manage your manager. Even if management doesn't directly affect you at work, you still should read this book, simply because it's interesting and will give you new outlook on all the places you spend money. There is nothing specific about programming in this book, just a really solid and entertaining book on the result of a giant Gallup poll on managers.

Software Development Related

Don't Make Me Think: A Common Sense Approach to Web Usability, 2nd Edition - Steve Krug

This book is short and a ridiculously fast read. The content is so common sense you might trick yourself into thinking you already knew it. And the truth is you probably DID, but you hadn't thought about it consciously. And for that reason alone, this book is worth reading. I think the most valuable thing about this book is it shows you that you can work on Usability without spending a fortune on a usability lab or outside consultants or long term studies with thousands of volunteers.

Blogosphere: Duct Tape Programmer

2009-10-01T07:53:00.000-04:00

Joel Spolsky is a famous blogger, partly because he knows how to get people so riled up that they'll talk about him... He's done that again which a recent post called The Duct Tape Programmer. As much as I hate to fall into his trap, the debate around this has been really interesting, so I'm gonna go ahead and talk about it!

The point of Joel's post seems to be in this quote from Jamie Zawinski:

"It’s great to rewrite your code and make it cleaner and by the third time it’ll actually be pretty. But that’s not the point—you’re not here to write code; you’re here to ship products."

Joel goes on to say that "Shipping is a feature. A really important feature. Your product must have it."

So, you have to ship a product to be worth a damn. Got it. Of course, the blogging world at large took his post as an attack on TDD, Agile development practices, and striving for Quality. This appears to me to be a classic case of Black Or White Disease. Either you are a "Duct Tape Programmer" who works fast and ships early but who's code is crap, or you are an "Architecture Astronaut" who over designs everything and never ships anything.

But this is not a black and white distinction. Richard Dingwall makes that point in Duct-tape programmers ship... once.

Ayende responds with disbelief that anyone would seriously suggest people abandon good practices in preference of relentless hacking just to ship a product. I'm not sure that was the message Joel was trying to send, though he was certainly saying that Unit Testing isn't worth the added cost.

Most of the debate therefore boils down to Quality, which I've touched on before in The Paradox of Quality. Quality is tricky. Jack Charlton hits on this by saying that quality is really just one dimension of many that must be balanced when developing a product. If fast to market with lots of features is a requirement, the only thing left to give up is quality.

But when you sacrifice code quality you are sacrificing product quality as well, and you are certainly adding long term cost. What shipped project ships once and never has to be dealt with again? Furthermore, what shipped product stays as simple as it was when it was shipped? If you sacrificed quality you're going to end up in a world of hurt when it comes time to dive back into that code and enhance it in unpredictable ways. But maybe the benefits of being first to market or shipping fast will mean that NOW you have the time to go back and raise the quality bar. Its harder to do it that way, but its not impossible.

This is what Domain Driven Design (DDD), and Test Driven Development (TDD) or Behavior Driven Development (BDD) is all about. Attempting to raise the quality of your code to the point where your product doesn't become legacy the day it is shipped (or even before its shipped...). But also attempting to do that without inordinate cost so that you can still ship your product.

This is about real world decisions with real world trade-offs. Developers don't like real world trade-offs, its part of what makes them good developers. They are given to all or nothing, Black Or White type thinking. Trade-offs are unpleasant and unwanted. But trade-offs come with real world issues, so as usual the answer of when to sacrifice some quality to ship a product earlier is: It Depends.

PS. This is the first post in a series of "Blogosphere" posts I intend to start where I write about interesting things I've seen out in the Blogosphere. If you like it, let me know!

DDD: Performance Trade Offs

2009-09-29T08:21:00.001-04:00

Domain Driven Design is a powerful technique composed of a lot of powerful patterns. It's focus is on designing rich object models that help control Domain complexity, like I talked about previously.

Before I stumbled on DDD I was trying to switch from stored procedure/record set based development to NHibernate/object model based development. One of the first stumbling blocks I encountered was what I refer to as "loading the object web." In a typical domain model your objects will have a lot of associations with each other. If you represent each of these with a property traversal (ex: Employee.Company.Branches.Sales...) things start to get messy.

The first problem you run into is in loading your Model from the database. To load any one Entity you have to load all of it's associations, and all those associations' associations, etc. This is a no go. The first way to handle this is through lazy loading, so the associations aren't loaded until you try to use them. That works, but it's a bit sloppy and it is not expressed in the Domain Model.

Another problem with this object web is determining how to persist changes and what records to lock. I have found database locking issues to be ridiculously complicated and amazingly ignored in the blog-o-sphere. For more on locking and its various solutions you should read Martin Fowler's Patterns of Enterprise Application Architecture. Fortunately DDD has a pattern we can apply to help address the web of objects problem: Aggregates.

The idea is simple, an Aggregate is a group of tightly related Entities. Evans says that all objects within the Aggregate should be loaded together and persisted together. He also says that objects outside the Aggregate can only have a reference to the Aggregate Root.

Roots are now explicitly indicated in our Model and they indicate where the boundaries between the objects in our object web are. This simplifies life considerably and it will also help with our database locking issues.

That's a lot of talk and I haven't even started talking about performance trade offs yet. Well, fortunately that really wont take long. If you are coming from the ad-hoc, off the cuff, procedural style, "Transaction Script" (as Martin Fowler would say) programming world like I am you may be alarmed by the Aggregate concept. You are used to executing a diverse array of procedures (filled with business logic) that return a specific subset of data: only what you need right now. But with DDD and the Aggregate, you're now going to return all the data for the Aggregate. This may be more data than you think you NEED at this point in your code.

There's your performance trade off. Retrieving more data when you don't necessarily need it. Why is this worth it? One simple reason is it allows you to enforce all your invariants (read: consistency rules) in the model all the time. You load all that data so that you can make sure all your data is in a consistent state at all times.

This also allows you to avoid triggers. You don't need a trigger in the database because your Model will do whatever the trigger would have done. Avoiding triggers is a performance benefit of DDD. Ironically, I've even seen that loading the Aggregate at once would have REDUCED the number of trips to the database in my code. This is because usually you don't just load something and be done with it. You tend to have to work with it. And there was a lot of duplication and repeated calls in the way I used to write my code that goes away once I have a Model in memory.

So sometimes the DDD approach can actually be more performant. But to be sure, other times it WILL do more work and WILL retrieve more data than the "Smart UI" (as Evans would say). But its worth it for the increased consistency and peace of mind that you gain, which is what allows you to deal with greater complexity.

DDD: Always Valid Model Objects

2009-09-21T08:46:00.002-04:00

One of the rules of DDD (Domain Driven Design) is that your model objects should never be in an inconsistent state. For example, if you have a Contact object and Last Name and Social Security Number are required attributes, DDD says you can not create a Contact without specifying Last Name and SSN.

When I first heard this rule it caused me to dismiss DDD right out of hand. My thinking was, “How dumb! How am I going to bind the Contact to the UI if the Contact can’t be inconsistent? After all, the UI will be inconsistent all the time.”

It turns out the answer to my question is pretty simple, you don’t bind the Contact to the UI. If you’re following the Model View View-Model pattern, you bind a View-Model to the UI, then when the user clicks the save button you use the properties on the View-Model to build a Contact. If there is anything wrong an exception is thrown and you display a validation error through the UI.

In a way this kind of sucks. You have more hoops to jump through now.

You have to “duplicate” the properties of your Model in your View-Model
You have to “map” between the View-Model and Model properties
Constructing a Model object can become somewhat difficult

Eric Evans addresses #3 with the Factory Pattern. Unfortunately he doesn’t say anything about #1 and #2. Interestingly regarding #1 and #2 though, if you were developing for the web you would be forced into this situation regardless because your Model is on the server while your UI is on the client. There would be no way to “reuse” your Model in the UI.

This lack of “reuse” certainly will cause more code to have to be written, but I would like to take a paragraph to say I think this is really a good thing. For one thing, to data bind to your UI you may have to make certain accommodations like changing data types or the structure of your properties. You don’t want concerns like this affecting the design of your Model. Another downside to this “reuse” is you are tightly coupling your UI to the Model. If refactorings happen in the Model they will affect all your views. If you add the View-Model layer only the View-Models will have to be updated (depending on what the refactoring was of course…).

The real question is are these hoops worth it? What’s so good about making sure your Model is always consistent? Simply put, this is an effective way to manage complexity. If your rules for “consistency” are simple, this can seem overkill. But as those rules get more and more complicated, especially as they begin to apply to more than one object in your model, making sure you’re consistent becomes drastically more difficult. By requiring all objects to always be valid, you’re taking a huge load of uncertainty off the shoulders of the Application Developer and you’re making a clear and consistent rule that all Model Developers must follow.

Thus once again bringing that DOMAIN complexity into check, even if it does require more code in the Application layer.

DDD: How to tackle complexity

2009-09-14T07:30:00.000-04:00

In DDD (Domain Driven Design) you create an object model that represents your application's domain. This model contains all the relationships and logic of that domain. The purpose of this is to make the complexity of the domain manageable. DDD involves lots of patterns and concepts, but when distilled there are a couple big picture things that really serve to tackle the complexity:

Explicit representation of domain concepts
Continuous Refactoring to "Deeper Insight"

The thing about complexity is its complex. Complex things are hard to understand. If they are hard to understand, they are difficult to get right the first time. And as hard as they are the first time, they're exponentially harder the second, and the third, and you get the idea.

This is the real issue: complex software is hard to understand. That's what leads to a system which everyone is so afraid to update, they would rather start from scratch. Maybe at first you're willing to just go in and add a hack or two. But each hack raises the complexity and increases the ugliness of the code until finally its just not worth trying anymore. That's another way of saying FAIL.

So we must come up with a way to deal with this complexity. The first way DDD does this is by taking advantage of the power of Object Orientation, models, and abstraction. But that's a bit too broad. We need to figure out how to structure those objects and models. That's where DDD applies the idea of explicitly representing domain concepts.

The idea is simple. If there is an import concept in your domain, you should be able to see it in the Model. You shouldn't have to dive into the code to extract important concepts, they should be represented by an object in the Model. Say there is some action in your domain that can only be taken when certain conditions are met. If these conditions are minor, you can just add if statement guards to the method that performs the action. But if these conditions are an important part of the domain, hiding them away in the code isn't good enough. Instead of an introduce a Policy Object that represents the conditions that must be met for that action to be performed. Now the conditions are explicitly represented in your domain.

You'll see this same idea expressed in Factories, Repositories, Services, Knowledge Levels, etc. Its a huge part of what it takes to make your system understandable.

The second idea that makes DDD work is Continuously Refactoring to "Deeper Insight." The deeper insight bit means if you discover something new about your domain after you already have a model, you don't just bolt it on. You figure out if that new thing indicates something fundamentally important about your domain. If it does, you refactor the model to explicitly represent that new understanding. Sometimes these refactorings will be small. Other times they will be big and cost a lot. It doesn't matter, you do them anyway.

If you don't your model will lose its expressive character and become more and more brittle. More and more complex. Harder and harder to understand. So you have to fight to always keep your model as simple and expressive and accurate as you possibly can. If you're lucky, these refactorings can lead to what Eric Evans calls a Breakthrough, where new possibilities or insights suddenly appear that would have been impossible before. That may take some real luck. If you're not that lucky, these refactorings will at least lead to a model that is flexible in the places where the domain requires flexibility. That means it will be easier to handle future insights and refactorings.

The really really cool thing about this is these two concepts form a cycle and feed each other.
The more you Refactor the more Explicit your model becomes. The more Explicit your model, the easier it is to Refactor!

Models

2009-09-08T07:30:00.001-04:00

Dictionary.com defines Epiphany as "a sudden, intuitive perception of or insight into the reality or essential meaning of something, usually initiated by some simple, homely, or commonplace occurrence or experience." This definition does not include a feeling of "DUH!" or "DOH!" or "Damn, I'm an idiot, how did I not see that sooner?!" though all of those certainly apply to the Epiphany I had a few weeks ago.

In this case, that moment of insight revolves around the concept of a Model. MVC, MVP, MVVM: each pattern starts with Model but what is a Model? For me, it had been nothing more than a data house, a class with get/set properties and MAYBE a couple of recurring little utility methods (format date or whatever). In other words, it was a class representation of the database. It was my LINQ-to-sql or Entity Framework data classes. Or my model classes in Ruby on Rails, which represent and map to the database.

That is a model, it's a model of the database, but is that really a useful thing? Is that what your Model in MVC, MVP, MVVM is supposed to be? The Epiphany came in the form a question from one of my co-workers. He asked, "Where's the OO?"

Indeed, where is the object orientation in my code? And that's when it hit me, there is none.

I mean, I write in C# so technically everything is an object. But I don't have any classes that exist to model a real world thing or concept. They just aren't there. I have utility/framework classes that exist to support the application infrastructure itself. Classes like StoredProcedure and Parameter, etc. These weren't a big leap, I'm basically just copying the .NET Framework. But I'm not writing a Framework, that's not my job. My job is to write a business application for my client, and I don't have a single object dedicated to modeling the client's business.

So effectively, when it comes to the domain, I'm writing procedural code. I might as well be writing in C.

How could this have happened? When I look back at the projects I did in college I was modeling all over the place. I had graphs and nodes and simulations and swarm agents and they were all objects, all models. But when I entered the business programming world and was confronted with Guis and Databases, I didn't transfer any of that. The main challenges then appeared to be 1. dealing with the UI and 2. dealing with the database. So all my efforts were set about those areas, the actual domain logic that the application was really there for was just scattered around.

Enter Domain Driven Design: The key point that I simply didn't understand is that Object Oriented Programming is about modeling the world. Yes, it's about abstraction and encapsulation, but those are really tools used to create a Model.

This is obvious, and if you had said it to me I would have thought I'd always known it and wouldn't have grasped the significance. The significance is that it's a drastic change in perception. If you're writing software, your software is doing something for someone. I would have asked the question, "what is it doing?" But a more appropriate question is "what does it model?" The "doing" is then included in the answer, along with the relationships and the data elements and the rules etc...

And now that you're modeling, you're taking advantage of all the benefits of object oriented programming. There is a very good reason why every modern language is object oriented.

In Domain-Driven Design: Tackling Complexity in the Heart of Software Eric Evans talks about all the reasons why modeling the domain is so important, and also introduces all kinds of software patterns that help make modeling the domain possible. There are two quotes that I think really explain the importance of this concept of modeling and indicate why it is such a shift of perception.

Model-Driven Design has limited applicability using languages such as C, because there is no modeling paradigm that corresponds to a purely procedural language. Those languages are procedural in the sense that the programmer tells the computer a series of steps to follow. Although the programmer may be thinking about the concepts of the domain, the program itself is a series of technical manipulations of data. The result may be useful, but the program doesn't capture much of the meaning. Procedural languages often support complex data types that begin to correspond to more natural conceptions of the domain, but these complex types are only organized data, and they don't capture the active aspects of the domain. The result is that software written in procedural languages has complicated functions linked together based on anticipated paths of execution, rather than by conceptual connections in the domain model.

The last sentence is the biggie for me. I've never heard a better description than "complicated functions linked together based on anticipated paths of execution." That perfectly describes what my code has looked like.

Evans also has this to say:

In an object-oriented program, UI, database, and other support code often gets written directly into the business objects. Additional business logic is embedded in the behavior of UI widgets and database scripts. This happens because it is the easiest way to make things work, in the short run.

When the domain-related code is diffused through such a large amount of other code, it becomes extremely difficult to see and to reason about. Superficial changes to the UI can actually change business logic. To change a business rule may require meticulous tracing of UI code, database code, or other program elements. Implementing coherent, model-driven objects becomes impractical. Automated testing is awkward. With all the technologies and logic involved in each activity, a program must be kept very simple or it becomes impossible to understand.

Not only is it the easiest way, it's the way the Microsoft tools encourage you to work. I now understand why the ALT.NET community was so pissed about Entity Framework. But that's the standard Microsoft approach: catering to the least common denominator. And as long as what you're doing is simple, you'll be fine. In fact, that's probably the right way to go because you'll have less code, less overhead, and still be able to understand and change it.

But I haven't worked on an app that was that simple, in, well ever. And I've been focusing so heavily on the technology, database access and the UI framework, that I forgot the real complexity was in the domain.

Of course now that I understand this I have to figure out how to write applications with a rich Domain Model layer. How do you persist changes from the domain? How do you do change tracking in the domain? How do different elements of the domain communicate with each other? How does a rich domain oriented application work with a service layer? And most important of all, how do you start adding a rich domain to an application that was written without one? Hopefully the DDD book will have the answer to most of these questions.

Engaged Employees

2009-08-31T07:30:00.000-04:00

First, Break All the Rules is a fantastic book about management and companies and I highly recommend it for managers and non-managers alike. Toward the end it makes the case that good managers lead to engaged employees.

I was totally enthralled by that term engaged employees. It's a perfect description of one of the characteristics that can make one person so much better at their job than another. I'm sure you've seen examples of this too. One person can be ridiculously smart, and yet do bad work. Or they might be amazingly talented, and still they don't do good work. You see examples of this every day in the grocery store checkout clerk, or the person at the front desk of your hotel, or the cleaning staff at your office, or the police officer directing traffic.

Sometimes when a person isn't performing well in their work, the failure is attributed to them being lazy, or immature, or even that they're just not challenged enough and so are bored. That may all be true, but the problem is really that they are simply not engaged in their work, for reason or another.

It turns out this concept of employee engagement has a rich history dating back to at least 1993 when it was described as "an employee's involvement with, commitment to, and satisfaction with work."

This really goes to the heart of what Peopleware was all about. Every manager wants their employees to be involved and committed to their work, but they all too often forget that they also need to be satisfied with it. Peopleware talks about this all over the place. And probably the two best real world examples are Google and Fog Creek.

Companies with disengaged employees are simply bleeding money. Its like the difference between people who drive the speed limit and coast into all their stops and people who are constantly speeding up to the bumper of the car in front of them and hitting the brakes... The second dude is just throwing gas away and killing his mileage. Every time he pulls around someone and floors it he feels good, but then he has to slam on the brakes again. And when he fills up at the pump and calculates his mileage, he'll blame his car or traffic conditions for the low mileage, anything but himself. This same short sightedness is the reason why companies want to skimp on amenities for their staff or the quality of their product. But its costing them engaged employees.

So engaged employees are important, the challenge is in getting them. Some people need someone around to keep them actively engaged, but others don't. The people that don't need help are your "self-motivated" people. The kind who need to be engaged to be happy. I would argue that most people in the world fall into this category, they just might be engaged in things other than their work. Look at sports. I'm not into sports. At all. I enjoy watching a game, but I just can't get into the stats and the history and memorizing years and events and people and on and on and on... I'm simply amazed by people who can learn all that stuff and talk and argue about it endlessly. The amount of energy and dedication required for that is phenomenal and it takes some serious smarts. Imagine if you could harness just a little bit of that energy and apply it towards something slightly more productive.

And therein lies the rub. How do you get people to be engaged, and more importantly how do you get them engaged in the right things?

I've had the fortune of knowing and working with a lot of smart people. Most have been naturally engaged. But it's interesting to see what that results in. If there is a project at work that excites them, they're all in. But when it doesn't, or when some other factor is causing them to dislike it, they'll find other things to get into: side projects at home, or frameworks or "minor" bugs or related "enhancements" at work. Expending energy on these things is extremely valuable experience to the individual but maybe not so much to the company.

But on the other hand, the greatest breakthroughs can frequently come from people messing with unrelated stuff. Just look at Google's 20% time and the making of ad sense. Ad sense practically single-handedly funds Google, and it was invented by someone in their 20% time. So having some leeway is equally important.

There is a line to walk here. If you have a lot of people who are very engaged, but always in things that don't help you achieve your business goals, you're screwed. And if most of your people are not engaged at all, you're screwed.

Walking this line is going to be difficult, but I would guess that most companies aren't even aware of it. Maybe their employees are engaged anyway, because of good managers or just because of luck. But that doesn't seem too likely.

So how do you get people to be engaged? Peopleware, and First, Break All the Rules, and Joel Spolsky, and 37 Signals can answer that question better than I can, so I'll refer you there.

In the mean time I'm curious, does your company engage you?

Optimize for Success

2009-08-24T07:22:00.000-04:00

Lets pretend you are writing some code. This code does something. Occasionally, it may fail for some reason, but we intend it to succeed, and it will succeed more often than it fails.

As an example, lets say we're going to update a record in a database, but before we do we need to a do a get so we can see if any of the values changed. If some values have changed, we'll do some stuff after the update. Maybe we'll send an email or something. We're going to do a concurrency check by optimistic concurrency (see if it changed in any way using a last update timestamp column) before we do our save.

Now, this is just an example so I can make a larger point, so bear with me here. Lets go ahead and pretend we're using LINQ-to-sql to do the update, so LINQ will also do the concurrency check for us.

Our psuedo code looks kind of like this:

Get record
Update record
if record.prop changed:
send email

Now, if the update fails due to the concurrency check, this will just bomb out.

But notice that when the concurrency check fails, we still did the Get operation. This kind of sucks because we didn't need to do it. And we can't do the Update before we do the Get, that would defeat the purpose. So we're going to have to do the concurrency check manually.

Wait. What? Why am I getting all upset about this? WHO CARES if I do a Get I don't need to do in a failure scenario. Unless there is something unusual about this failure, like it's going to happen all the time, or it has some record locking implications, none of which apply here. I'm optimizing for the wrong thing. I should be optimizing for success, not for failure (While avoiding premature optimization, of course).

If I could remove the Get completely, so the method would Succeed and not need it, that might be something worth talking about. But it is totally not worth adding any code complexity just to optimize this method for a failure case.

Thus: Optimize for Success, and don't get too worked up over failure.

Another C# Fluke

2009-08-17T14:32:00.003-04:00

Here's some code:

var l = new List<int?>();
l.Add( null );
( ( System.Collections.IList)l ).Add( null );

You would expect those two Adds to be equivalent and that after executing them l.Count would be equal to 2.

Instead you get an ArgumentException on the second Add. Turns out System.Collections.Generic.List implements both Add(...) and System.Collections.IList.Add(...) and the interface specific one does some input validation not done by the other Add. This validation doesn't understand Nullable types, so you get an exception.

I guess its a probably a bug, due to the fact that Nullable<> behaves kind of strangely through reflection.

A C# Language Quiz

2009-07-29T14:18:00.004-04:00

I was writing some code today when I suddenly realized that I didn't how a certain very fundamental part of the C# language would behave.

Here's an example:

using System;

namespace TestNullCastToObject
{
class Program
{
static void Main( string[] args )
{
Test t = null;
object o = t;
if ( o is Test )
Console.WriteLine( "a Null Test is a Test" );
else
Console.WriteLine( "a Null Test is _NOT_ a Test" );

Console.ReadLine();
}
}

class Test
{
public int Id { get; set; }
}
}

If you compile and run that sample what do you think the output will be?

No really, think about it.

Ok, I'll tell you what I thought the output would be. I thought the output would be "a Null Test is a Test".

Ok, now I'm going to tell you what the output is.

"a Null Test is _NOT_ a Test"

Does that surprise you as much as it did me? I think I'm actually happy that it behaves this way, but I'm still surprised.

SQL Deadlocks: More with child data

2009-06-30T11:03:00.001-04:00

In last week's post, SQL Performance: Child data, I wandered through an issue involving caching data about a parent table's child data. In that post I talked about writing SQL that would save a foreign key on the parent table to the most recent child record. This is very simple and would look like this:

begin tran TxExample
BEGIN TRY

declare @NewChild int

insert into Child ( blah, blah, blah ) values ( @blah, @blah, @blah )

select @NewChildId = SCOPE_IDENTITY()

update Parent set CurrentChildId = @NewChildId where Parent = @ParentId

if @@TRANCOUNT > 0 commit tran TxExample

END TRY
BEGIN CATCH
if @@TRANCOUNT > 0 rollback tran
END CATCH

This is relatively straight forward. It inserts the child, then updates the parent's cached data. Those two operations are wrapped in a transaction and a try catch to ensure that if anything should fail for any reason, both statements will be rolled back. This ensures data integrity.

And now it's time to talk about deadlocks. This code is susceptible to deadlocks. As a relatively contrived but none the less possible example suppose the following SQL could also be run:

begin tran TxExample2

update Parent set blah = @blah where ParentId = @ParentId

...

select * from Child where ChildParentId = @ParentId

commit tran

If these two queries were to run at the same time, operating on the same parentId, and SQL Server were to context switch them at just the write moment, they would deadlock. Specifically, if the first query completed its insert statement (line 6) and then SQL switched to the second query, we would deadlock.

This is because when the second query tries to select from the Child table, it will wait because the first query has inserted a new row and SQL Server's default isolation level is read committed, which means dirty data will not be read, instead it will wait for the data to be committed. So it's going to sit there, waiting for the first query to commit.

This isn't a deadlock yet. The deadlock happens when SQL switches back to the first query and attempts to execute the update on the parent. When it does this, it will try to obtain an exclusive lock on that parent row, but it won't be able to because the second query already has an exclusive lock from it's update. So it will wait for the second query to commit.

The first query is now waiting for the second query which is waiting for the first query and you have yourself a deadlock.

Before we fix it, we should ask ourselves "is this a big deal?" The answer is, it depends, but in general yes. If all your SQL is small and all your transactions complete quickly and you don't have very many users banging on the system concurrently then you probably wont see any deadlocks. But unless you can guarantee that all those conditions will remain the same you have to be at least a little worried. And if those conditions don't apply to you, you definitely have to be worried.

So how do we fix it? First thing we could do is to commit the transaction in the second query before executing the select. If this is possible, then it's a good idea. You want your transaction to commit as quickly as possible and you want to touch as few objects as you can while in the transaction. That said, there are plenty of reasons why you might not be able to commit the transaction after the update. For example, maybe you're reading the child data because you need it to perform another update, and those two updates have to be in the same transaction. In that case, there is nothing you can do to fix query #2.

But even if you could fix query #2, someone could some day come along and write query #3 which would introduce the same problem again. So what we really need to do is fix query #1. The way we do that is by having query #1 obtain a shared lock on all the resources we know it will need to touch, immediately at the top of the query.

Add this code after the BEGIN TRY:

set transaction isolation level repeatableread
select ParentId from Parent where ParentId = @ParentId
set transaction isolation level readcommitted

With this code in place, query #2 will not be able to execute it's update until query #1 completes. Thus, preventing the deadlock and saving the day!

This example was simple but the deadlock was still subtle and hard to see. This problem just gets more complicated the more complicated your SQL gets. And your SQL will get more complicated in direct relation to how complicated your data schema is. So you really have to be on the look out for this issue.

Before I wrap this up, I should mention that if you need to lock more than just one row in one table at the top of your query (like we did in query #1), life can get interesting. If the tables you are locking are all related you can lock them by inner joining to them. But if they are unrelated, things get interesting. If they're unrelated, you can't join from one to the next, so you need to execute separate select statements. And if two queries need to lock the same records in two unrelated tables, but they lock them in different orders (A, B vs. B, A) you can end up with a deadlock! For these cases you have to resort to what you learned in your operating systems class: always lock all your resources in the same order. Good luck with that.

I'll leave you with some rules of thumb, which apply to most cases but, of course, not all:

Keep your transactions as small as possible by touching as few objects as possible
Keep your transactions as fast as possible: if you have a query that can execute on n records in a single transaction where n is unbounded you are likely to find yourself in a world of hurt
Obtain shared locks on everything your transaction will eventually require exclusive locks on before you acquire any other locks
If you need to do any reads that don't need to be repeatable, do them before you obtain any shared or exclusive locks (this is really just in keeping with #2)
If you set the transaction isolation level to repeatable read make sure you're setting it back to read committed (even if its the last line of your query, this will make sure triggers don't execute in repeatable read)

Now, all of this has been learned by trial and error, experimentation, and a lot of reading. If you know of other ways around these issues, or if you have a different take, I definitely want to hear about it.

SQL Performance: Child data

2009-06-29T08:43:00.000-04:00

In this post I’d like to talk about a specific issue in SQL and all the various ways you could approach it. Specifically, I’d like to talk about dealing with a parent entity’s child data. As an example lets use a very simple document database that stores documents and their versions. Obviously, a document will have many versions but each version will have only one document, as in this diagram:

This is not a complete database, clearly, but it indicates that the DocumentVersion table is a child of the Document table.

Now we get to the part where this gets at least partly interesting. Lets say we’re going to write a search that returns documents. In the results we want to display information about the current version of each document as well as the total number of versions for each document.

This is a surprisingly non trivial query…

select d.Id, d.Name, v.Id, v.FileName, v.Extension, v.FileSize,
( select count(Id) from DocumentVersion where DocumentId = d.Id ) as numVersions
from Document d
inner join DocumentVersion v on d.Id = v.DocumentId
where v.Id = ( select max(Id) from DocumentVersion where DocumentId = d.Id )

Now, there are a bunch of ways to write this, but this is a perfectly good example. Notice we have an inner select in the select clause to get the number of versions and we have another select in the where to get “latest” version. Here I’m depending on SQL Server’s Identity Specification to give me the latest row because it simplifies the query. If we didn’t want to do that, I’d have to either “select top 1” while ordering by the inserted date (which isn’t on the table in our example) or use a row number function and get the row where the row number = 1 again ordered by the inserted date. Both of these query are correlated, meaning they're run for each document in our results.

This query is ugly, but it works. We could optimize it and tweak the way its written to try to get the best possible performance out of it. But is this really the best way to do this? If we think about it, we’re going to be looking at all the versions for every document returned in our search. The more documents we return, the worse this is. But worse, we’re going to do WAY more reads than we are updates in this case. New versions simply are not going to be added that often. So it seems silly to be constantly looking up information about the versions over and over and over and over again when we know its unlikely it will have changed from the last time we looked at it.

Wouldn’t it be better to cache this information on the Document table so we don’t have to keep calculating it repeatedly, thereby simplifying the query and improving its performance?

To do this, we simply add “NumVersions” and “CurrentDocumentVersionId” columns to the Document table. But now we have to keep these columns up to date. There are a few ways to do this:

Trigger on DocumentVersion updates Document’s cached columns on Insert/Delete
Code that does inserts or deletes to DocumentVersion must update cached columns
Cached columns are calculated columns that use a function to lookup values

We'll take these in turn. #1, using triggers, has these benefits:

Ensures the columns will always be up to date, no matter how the version records are changed
Code will be in just one place and we wont have to worry about it again

However, triggers have these drawbacks:

Slow (like, REALLY slow. Batch operations like inserting or deleting many versions will slow to a crawl)
Potential for subtle and hard to track down deadlock problems
Increased code complexity because the trigger must be written to handle ALL cases, even if you only use some (ex: inserting many versions at once)

On #2, updating cached columns when updating Versions, we have these benefits:

Simplest possible code
Performant
Deadlock issues are easier to see and handle

But it comes with it's own downsides:

Same code may end up in many places (ex: Insert and Delete stored procedures, if using sps)
Potential for error if someone inserts/deletes versions from a new location and forgets to update the cached columns

#3, using calculated columns, is the same as putting the lookup logic in the query (since you can't persist the value) but has the overhead of a function.

So, between #1, #2, and #3, which is the right option?

I used to use triggers, because of fear that someone would forget to update the columns if I went with #2. But the performance and deadlocking issues with triggers has now caused me to go with the "API layer" approach of #2.

I think the answer, as always, is it depends. If the tables you're using are likely to be always accessed through an API layer, then you should go with #2. But if many people will be manipulating those tables from many different areas and there is no central API layer, you're pretty much forced to go with #1.

And the question remains, is it really worth caching the data this way, or should you just keep the lookups in the queries. Once again, my favorite theme for this blog: it depends. The big question is really performance, and that depends on how the queries will be used. Are you going to be returning thousands of results, or just hundreds? Are you going to be running this query often?

In SQL there is no one size fits all rule. And worse, SQL is so complex it has to be treated as a black box, meaning you really can't reason about it. Therefore, your only hope is to test and test and test. You pretty much have to write the query every way you can imagine and then performance test each one... And that takes a lot of time.

As Scott Hanselman would say, "Dear Reader," what do you think? Have you been faced with this issue? What did you do?

The Paradox of Quality

2009-06-22T07:50:00.000-04:00

In college I had a professor named Dr. Lang. He was a great guy, and he was full of great quotes. Some of them were just inane, but others were serious and insightful. Though, usually even those were fairly silly. One of those was, "All programmers are control freaks."

There is a lot of truth to that, but I think there is a deeper reason that causes that control freak behavior. The control freak behavior is just a side effect of a deeper motivation, namely that developers are quality freaks.

The book "First, Break All The Rules" is a really great "management" book that focuses on the concept of "talents." They use the word talent in a different way than we typically use it in conversation, and they break it down in to different kinds of talents. Of particular interest is what they call Striving talents,

"Striving talents explain the why of a person. They explain why he gets out of bed every day, why he is motivated to push and push just that little bit harder. Is he driven by his desire to stand out, or is good enough good enough for him? Is he intensely competitive or intensely altruistic or both? Does he define himself by his technical competence, or does he just want to be liked?"

Given this definition, I think we can make the case that developers have the Quality Striving Talent. This is what drives us to want our code to be perfect and our UIs to be perfect and our architecture to be perfect and our designs to be perfect and on and on. Without this talent, I think a developer would go completely insane!

But, quality is not a very straight forward concept, which brings us to:

Paradox #1: Programmers vs Management
Programmers, being so quality driven, want quality in everything they do. Like, the aforementioned, code, UI, architecture, design, etc.

On the other hand management wants a product that people will buy. Obviously, quality factors in here too: people will want quality in the product they buy, so management will want quality in the product they sell. But management doesn't need quality that goes as deep as the programmers are interested in. After all, very little of the details of programming are visible to the users or to management.

The paradox here is clear, and it explains an awful lot of the friction you can encounter between programmers and management.

Paradox #2: Quality is in the eye of the beholder
For example, one person may like a UI with bright color, gradients, shadows, reflections, slidy animations and some content mixed in where it fits. Other people might look at that UI and see distracting noise and bloat. They would prefer black on white densely packed text emphasizing content.

To each his own I guess. But, which of these UIs is higher quality?

Which of these paintings is higher quality?

Paradox #3: Defining Quality
This has been a lot of talk about quality, but what is quality? Here are some factors you might look at to define the quality of an application:

Bug count
Learnability
User Friendliness
Appearance
Feature parity

But there's a problem, some of these are competing concepts. My favorite example as always is Vim and Notepad. If you sit a normal person down in front of Vim they wont even be able to get text into it. But watch a real Vim wizard work and you'll think its magic. Vim is a powerfully usable editor, but it's very very difficult to learn. On the other hand, any moron can use Notepad, but the morons and the experts both use it the same way. It's learnable, but not terribly powerful, and therefore not nearly as usable.

What about factors for code quality:

Bug count
Simplicity
Cohesion
Readability
Lines of code
Style
Broken Windows

Again, we find competing concepts! Take simplicity and cohesion for example. To achieve cohesion you may follow the SOLID design principles, all of which involve increasing abstraction, adding interfaces, and so forth. But all of these things detract from simplicity and increase the number of lines of code. Paradox!

Realizing that quality means different things to different people at different times is important to remaining sane. Especially when you're a quality driven developer. But as we've seen, quality is a slippery contradictory customer, and this really makes things non-trivial. One thing that might help is focusing on an intended audience to help you decide just what quality matters the most to that audience. By focusing on an imaginary third party’s quality needs you can remove some of your own quality driven needs and find acceptable compromises.

An Anemic Community

2009-06-09T10:57:00.005-04:00

I am a .NET developer. I've dabbled with Ruby and Python, learned Java and C++ in college, but there's no two ways about it, I'm a .NET dev. Occasionally I think it would be nice to work at one of those dev shops that hires out bodies to companies that need them so that I'd get to work with more languages. But most of the time I like being specialized, it means I have time to really learn my language and its tools. And it should mean that I turn out higher quality product because of it.

.NET is a great platform. The language is very much at the front of the curve as far as strongly typed languages go, and with C# 4.0 coming it will soon be incorporating a number of dynamic language concepts as well. Plus, as IronRuby and IronPython mature, it may become feasible to work with dynamic languages for some things and static for others, which is cool. Add to this picture LINQ, Windows Forms, WPF, WCF, ASP.NET and MVC and you're looking at a pretty compelling platform.

Now, the platform itself has its issues. The major one being that the tooling is targeted at the "least common demoninator" developer, which can make things interesting for a dev working with non-trivial applications in a non-trivial environment. But, that's a topic for another blog post. For our purposes here, lets agree that .NET as a platform is pretty darn good.

Unfortunately, the .NET community kind of sucks. There has been some improvement here in recent years with things like ALT.NET, but for the most part, the community remains pretty anemic. Compare it with the Java community for example. The vast majority of the non-microsoft products in .NET are ports from Java: nhibernate and nunit for example. Or compare it with jquery, the number of quality plugins and code samples is just ridiculous. It's a bloody fruited plain!

Now take a look at the blog posts that turn up from Google searches for .NET stuff. Especially around WPF and WCF all you find is amature quality stuff. Look at Codeplex, all you find are tons of discontinued projects people started, and what work remains there is terrible. And don't even get me started on the half solutions that riddle Code Project.

Why is this? There are a large number of .NET developers. There are all kinds of conferences and user groups. So why would it be that there is so little happening on the web with .NET. And why is it that what is happening is so bad?

I think its because .NET is a corporate community. The people who are sharing their work or writing code sample blog posts are either amatures, college kids, or "low ranking" people just getting started in their career. The actual "professionals" who are doing good work are doing it for some company or other and are unable to share it on the web! The few who actually do get involved on the web stick with waxing philosophical on their blogs (kind of like what I do...) and don't really add much value. Worse, if there were projects available, many companies wouldn't use them simply because they weren't Microsoft's.

The corporate view point is that any ideas or tools or products that have been developed by their people represent a competitive advantage. Giving away your advantage in the interest of helping the community is just stupid. Who would do such a thing?

Because of this there is little organizing going on. People are not starting useful and interesting projects and finding contributors. They can't, for fear that their employer will find out and they'll get into some kind of trouble they don't understand.

This is a real problem for .NET. I think that's why Microsoft works so hard at planting .NET bloggers in the community, like some kind of covert operation run by the CIA. That's helpful, because those people are writing good useful stuff. But it all falls short of any kind of organizing. And it also tends to keep its sights quite low, not really trying to contribute anything impressive.

The fact that no one is sharing really lowers the averge state of the art across the board. The ridiculous strides taken by Ruby on Rails and jQuery are examples of how much an open oriented platform can do. It makes it easier for people to jump on board and do non-trivial things. It also makes it more likely that as those people gain experience they'll contribute back. If it goes well, you can get a kind of snow ball effect where the people keep getting better and better and so does the technology.

I'm unclear on how the .NET world could begin to do this, since its so corporate driven. All I know is the low quality material makes a .NET developers life much much harder. And at times, its really frustrating.

Thinking like a TDDer

2009-06-08T12:45:00.001-04:00

Welcome to my 100th post! I'm going to return to a topic that I've touched on many times in the past, namely, unit testing.

Unit testing is one of those things that is very attractive in principle but gets ugly real fast in practice. I've tried it countless times but never managed to follow through. But in principle, I'm a fan. Having tests to fall back on to help prevent regression and to make refactoring easier and less stressful is a big win. And following the TDD process is very rewarding. Every couple minutes you get another green bubble, encouraging you, telling you you're making progress and doing a good job. Plus, unit testing reinforces the benefits of following good design practices like loose coupling, DRY, and Do One Thing (which are all the same...).

For me personally, unit testing still has 2 primary stumbling blocks:

Mocking
"Maintenance" development

I've talked about many other "pitfalls" including Mocking in an earlier post called Unit Testing Pitfalls. I still think those are true, but I'm coming to terms with the ways around some of them. However, mocking still drives me crazy. I hate having to specify every method and/or property that my object under test is likely to call on the dependency. On the other hand, I hate writing Stub objects because it takes time and effort and you frequently make mistakes that cause tests to fail. Maybe I just need to learn all the features of RhinoMocks (if only the documentation was better) or learn Moq or something. If I find a way to resolve this, I'll let you know.

The second issue is "Maintenance development," by which I mean, going back to code you've written and changing it. When I'm first working on some code I find TDD very easy, but when I have to go back to it to fix bugs or add features I always go straight for the code and cause my Unit Tests to be out of date. And therefore, worthless.

A Continuous Integration server which ran the Unit Tests would force me to keep the tests up to date, but I don't like to be forced to do things. In fact, I'd probably just start commenting out the failing tests so I could get my bug fix checked in. Yeah, I know! Hideous behavior right? Should be punishable by death. But I'm lazy, and so are you.

My problem is that I'm still not thinking like a TDDer. If a TDDer needed to fix a bug, they would start by adding a failing test, then they would work until the test passed, THEN they'd try to repo the bug and make sure it was gone. That's the process they would follow, but what's really interesting in the mindset that process requires.

You've probably heard people say that tests are as good as if not better than documentation. I've even heard people says tests can replace specifications! These people are completely insane! Lock them up in a padded room, throw away the key, and please please please don't let them write any more blog posts!

Ok, I know, they aren't REALLY crazy, they're just using the words to mean something different than what the words REALLY mean. Welcome to software development, we love overrides so much we can't resist overriding words in the English language!

So if you fight through the crazy and think about the documentation argument for a moment, that unit tests are documentation, you can see where this might make sense. First, assume we're talking about API documentation and not end user documentation, obviously. I'd like to see an end user figure out which button they need to click on the screen by reading a bunch of unit tests. Don't worry, I'm sure they'll re-up your contract next year!

Anyway, you have a bunch of tests like, Apply_is_enabled_when_required_fields_are_filled() and so forth. A smart enough and patient enough person could read through tests like this and figure out what the object under test does and how to code against it. In fact, because of the volume of code in the tests, they'd be able to understand and extract much more meaning than they could from some MSDN style documentation.

But can you imagine if the only documentation Microsoft published for the .NET framework was its unit tests? The MSDN style docs are hard enough most of the time! Unit tests ARE NOT documentation! But they are incredibly documentative (made up words are fun!), and they are very very useful for someone who intends to work on the object under test. So, it makes a lot of sense to think about them this way. That is, its a useful mindset.

What about the argument that they are specifications. This one is even more off base. A specification is what you write up front to describe what the software should do and how it should work and basically what it should look like. Yeah, Agile people jump up and down and say you don't need specs. "Just write a story card!" they squeak. "Pin it to your wall!" Sorry little agile buddy, your story card is not a spec. It's not even the replacement of a spec. At some point you're still sitting down and figuring out what the software needs to do, and deciding how it should work. Maybe you're meeting with your "customer representative" and you show him your little story card and he starts rambling on about the details of that story while you furiously scribble notes on a legal pad. Then you sit down with your programming buddy (read: "pair") and you start writing code. Well, the scribbles on your note pad are the spec.

Despite all the condescending language in that paragraph, this is actually fine by me. As long you manage to capture enough detail in that meeting and are able to think through the consequences of your implementation decisions as well as the issues of integrating this part with the other parts of your software, this is a perfectly acceptable way of producing specifications. You might want to be a bit more diligent if you are writing the software that flys the space shuttle, but for most business and banking apps, I think you'll be just fine.

But, back to my point, unit tests are not Specifications. However, they do specify how the object under test should behave. In fact, they *should* specify absolutely everything about how that object should behave, if they're going to be truly effective. And this again indicates something about the mindset of the TDDer.

If unit tests are really going to work, they have to be more than regression tests. And they have to be more than a crutch you lean on during "initial development." They have to be both documentation and specification. This mind set leads you to all kinds of realizations.

For example, to be effective documentation and specification, they have to focus on behavior and they have to have good names. So, "Add_returns_false_null_person" is not a good test name. Maybe you should have gone with "Add_fails_when_person_is_missing."

If you're thinking about your tests this way and someone finds a bug, what's the best way to approach it? Not diving into the code and looking at stack traces trying to find the line of code that's in error, no sir! If there's a bug, you must have missed something in your specification of the problem. So what you're going to do is go look through your spec, find the missing piece that explains the bug, then update your spec.

What this means is that your goal when you're writing your tests shouldn't really be code coverage. And it shouldn't be to test every input/output combination. It should be to fully describe and specify all the required and expected behavior of the object you are testing. These should more or less turn out to be the same thing, but because the mindset is so completely different the details will be different. The details will be better. And you might actually stand a chance at keeping those tests up to date. Heck, you might even enjoy keeping them up to date instead of feeling like it's a chore.

Well, I don't know, that's asking a lot.

Is changing the way you approach your tests going to keep the real life details from getting ugly? Nope. You're still going to end up with test code that is ridiculously repetitive because you have to test the same method with these inputs, and those inputs, and those other inputs. And you'll still have to update all this repetitive code when you decide to change the type of one of the method parameters. And you'll still struggle with different ways to refactor your tests to cut down on the repetitiveness, which adds more abstraction and occasionally doesn't work out so well. And don't forget, you'll still be mocking out the dependencies. You know how I feel about mocking.

But now that your tests are more valueable, at least in your mind, this might all be worth it. I guess we'll just have to try it and see.

What Mercurial Can't Do: Subtree Repos

2009-05-01T07:39:00.004-04:00

On the Mercurial website, at the very bottom of this page is a section titled "What Mercurial can't do" which says it can't host related projects together in one repository. Then it links out to the Forest Extension which makes nesting repositories easier to work with (and which comes with TortoiseHg, you just have to turn it on).

Its true that Mercurial can't do this, not even with the Forest Extension. Apparently there are plans for Mercurial to build some kind of subtree repo support in the near future. However, I'm not sure what it will look like. It might not enable the behavior I'm going to discuss here. We'll have to wait and see.

To describe what this really means, here's a possible layout you might find in TFS:

$\
MathUtils\
GraphUtils\
ProjectA\
ProjectA\
MathUtils\
GraphUtils\

In this example MathUtils and GraphUtils are seperate projects which have been written to be reused by other projects. ProjectA is reusing both of them.

ProjectA could just reference $\MathUtils and $\GraphUtils directly. But if there are many projects reusing them and making changes to them that could get hairy. Every time someone in ProjectB changes MathUtils and checks in, they could potentially break ProjectA immediately. To avoid this, ProjectA wants to isolate itself from changes other people might be making to the shared utils. To do this, they create a branch of MathUtils and GraphUtils in their own project. Now they can decide when the right time to bring in changes from other people is, or when the right time to share their own changes with other people is.

To bring in changes from outside in TFS you'd right click on $\MathUtils and say merge, selecting $\ProjectA\MathUtils as the target. To push changes back in TFS you'd right click on $\ProjectA\MathUtils and say merge, selecting $\MathUtils as the target.

In TFS, if you made changes to ProjectA and MathUtils and checked in, you'd get a single checkin containing all changes. When you then merged MathUtils back, the changeset would basically be split so only the changes to $\ProjectA\MathUtils got merged to $\MathUtils (For more on how TFS does merges, check out this post What Mercurial Can't Do: Merge by Changeset). This works because TFS doesn't respect "changesets" across branches.

This does not work in Mercurial. You could setup the same directory tree structure, but you couldn't have just one repository. Instead, you'd have to have $\ProjectA\MathUtils and $\ProjectA\GraphUtils setup as their own repositories nested in the $\ProjectA repository. Mercurial is smart, and it wont try to check in MathUtils or GraphUtils to ProjectA because it recoginizes them to be distinct repos.

So this setup actually works just fine, but it's more work. In TFS, we could do one checkin. In Mercurial, we'll have to checkin changes to each repo individually. That's probably not such a big deal, but in TFS you didn't have to keep track of where you were making changes. In Mercurial you might forget you changed something in MathUtils, and then you might forget to check it in. This is because when you do an "hg status" you'll only see changes in ProjectA. This is where the Forest Extension comes in handy. It adds an "hg fstatus" command which is basically a recursive status command. That way you wont lose track of your changes, but you still have to commit them individually.

But lets think about this for a minute. Is it really a smart thing to do, having one changeset with changes that affect both ProjectA and MathUtils? What's your checkin comment likely to be? Probably something like "ProjectA's xxx feature now can whizbang." If you're looking at the change history for $\Project\MathUtils and you see "ProjectA's xxx feature now can whizbang" does that makes any sense? Does it tell you what changed in MathUtils?

So once again, this is something that Mercurial just can't do. But maybe that's a good thing.

What Mercurial Can't Do: Merge by Changeset

2009-04-30T07:22:00.005-04:00

Mercurial is a distributed source control system that works very well on Windows and has great windows shell integration with TortoiseHg.

TFS is a centralized behemoth that does source control but also integrates (usually poorly) with every product Microsoft has ever released (not including Bob).

I've used both of these, but I've used TFS much more extensively. I recently started looking into what it would take to switch from TFS to Mercurial and was rather surprised to find a couple things that TFS can do that Mercurial cannot.

The first of these is the ability to do a merge by changeset. In TFS, say you create some branches as follows:

Create a new TFS project called Project
Check in some source at $\Project\Source
Branch that source to $\Project\NewBranch
Do 3 checkins to $\Project\NewBranch

Suppose you're not completely done with whatever it is you're working on so you can't merge all three changesets back to $\Project\Source just yet. But lets say your second changeset includes a bug fix to a file that you wont be changing as part of your "main" work in the other changesets. Maybe you want to go ahead and merge this bugfix back to $\Project\Source right now.

In TFS you could do this very easily.

In Source Control Explorer, go to $\Project\NewBranch
Right click and select "merge"
Change to the "Merge by selected changeset" radio button
Make sure the target is $\Project\Source
Click next
On the next page, select only the second changeset
Click next and the merge is performed

Ta-Da! You just merged changeset #2 without merging changeset #1 or #3.

You cannot do this in Mercurial, at least, not with a "merge" operation. The only way to accomplish the same type of thing would be to create a patch out of NewBranch and apply it to Source using the hg export and hg import commands.

So the big question is, why can't you do this in Mercurial? The answer goes to the heart of what makes Mercurial so different from TFS. The first thing to realize is that Mercurial does not have "branch lines."

In TFS when you branch code TFS knows that the one is a parent of the other and you can only merge across that branch line. This means you can't do merges between two siblings. For example, in B -> A <- C, you can't merge B to C. You can only merge along the branch lines (Unless you do a baseless merge, which doesn't really count). In Mercurial, the normal way of creating a branch is to simply clone the repository, which means you have a full copy of the entire history of the repo. The image to the right shows what an hg pull would look like when bringing changes from a cloned repository into the original repository.

"A" represents the starting point. "B" represents the first change from the repository we're pulling in. What you can see here is that when you pull changes from mercurial, a temporary "branch" is created containing all the changesets from NewProject in parallel with any changesets from Source.

"C" is the only actual merge, in the tranditional way we think of merging, because it actually brings all the changes together. This is beautiful in its simplicity because until you get to C you don't have to do any work. Each changeset represents what changed from the parent, so you just import all the changesets and associate them with the correct parent. Then only at the very end do you have to do any merges.

A merge in TFS is not so clever. Basically all TFS does is figure out every file that changed on either side, and do a 3-way merge on each in turn, resulting in a new changeset. The upside of this is we can select a single changeset, ignore all the changesets around it, and do a merge.

In Mercurial, you can't pick just one changeset and merge. You have to merge all the changesets before it too because that's the definition of how a merge works in Mercurial. The upside of this is that all the changesets are preserved.

For example, say you want to know who added a certain file. In mercurial, you'll be able to figure this out regardless of what "branch" (cloned repository) it was added in. In TFS, you're screwed because the file will be added in a "merge" changeset. The merge may not have been done by the same person who added the file (in fact, it usually wont be), so to find out who added it, you have to manually follow the branches and inspect the history on each in turn. The same is true (and worse, actually) if you want to know who updated a line in a certain file.

Sadly for me, we're constantly "de-tangling" our changes by doing merges by changeset. But lets think about that for a minute. Is merging a single changeset even a sane thing to do? It turns out, not so much, because its possible for this to result in a broken state. Here's how:

Joe Bob adds a new file "hippo.cs" and updates the C# project file
Joe Smith adds a different new file "giraffe.cs" and updates the C# project file
Joe Smith merges his changeset and ONLY his changeset up
The result of the merge does not compile. The error is, "Can not find file "hippo.cs"

WTF!? Why is it looking for hippo.cs?! We didn't merge that changeset! How does it even know about hippo.cs? Its because Joe Bob and Joe Smith both changed the C# project file. When Joe Smith changed it, Joe Bob's changes were already in it. So Joe Smith's final C# project file includes Joe Bob's "hippo.cs" file. But when Joe Smith did a merge, he didn't include Joe Bob's changeset, so the hippo.cs file didn't get merged. And now you're broken.

This happens like all the freaking time with project files (which are the bane of branches and merges). But fortunately it's easy to fix. Just remove the missing files from the project file. But I think you can probably see that if this happened to any file other than the project file, like a real source file, you'd be in a world of hurt.

I'm actually STUNNED, given how much effort TFS puts into protecting the users from themselves that it allows you to merge selected changesets in this way! But it does. And its a feature that Mercurial just can't match, even if it is a feature that can lead to trouble. But maybe that's a good thing.

Merging with TFS

2009-04-28T13:18:00.004-04:00

We celebrated "TFS Sucks" day at my office last Friday, so this blog post is a little late.

I don't understand this behavior. I'm merging one branch back to another by selected changeset (so I that I can go through what's changed before I merge). I tell it to do the merge and it comes back with the "Resolve Conflicts" dialog. There was one file in the changeset, so I see one file.

According to the dialog: "Conflicting changes have been detected. To resolve conflicts, select items and click Resolve."

When I right click on it and Compare -> Source to Target... I don't see changes on both sides, I only see the changes I knew I was bringing in.

When I right click on it and Compare -> Source to Base... I see exactly the same changes.

When I right click on it and Compare -> Target to Base... my tool tells me there are no changes.

So, this means there are no changes in my "Target." If that's the case, what "Conflicting changes" have been detected? There is no conflict! Of course, when I tell it Auto Merge All, it succeeds, but why did I have to go through this step at all?

I'm using SourceGear's DiffMerge as my merge tool, maybe that has something to do with it?

Vim Update Many Files

2009-04-27T20:08:00.003-04:00

Here's a really nice Vim trick I learned today and have already used twice.

I'll demonstrate it with an example. I wanted to go through all my .cs files and fix the formatting (they all had 4 spaces per tab and I wanted to update them to 2 spaces per tab).

First, I opened Vim and set the current directory to the root of my source folder.

:cd C:\blah\blah\source\code

Next, I told Vim to pull in all my .cs projects in the entire source tree.

:args .\**\*.cs

** means to go recursively down 30 directories (you can set the max depth, default is 30. Try :help ** for more).
Finally, I told it I wanted it to format each file, from beginning to end, using the "equalsprg" and then to save it.

:argdo exe "normal gg=G" | w

To break this down:

:argdo tells it to run the specified command on all "args", or all open files
exe tells it to execute the given command
normal runs the :normal command, which allows you to execute normal mode commands like motions, etc
gg goes to the top of the file, = formats the file, G goes to the end of the file
| chains commands together
w writes the file

Pretty awesome I thought.

Another common use of :argdo is to run a search and replace across many files. Open the files you want either with the :args command or from the command line. Then do:

:argdo %s/matchon/replacewith/ge | w

If this would require you to open a ridiculous number of files, most of which wouldn't have matches, then you should use the :vimgrep command instead. This will put only files that match your regex on the quickfix list. You can then move through them with :cnext and execute any commands you want. The downside here is that you have to execute your command over and over in each file.

Word CommandBarButton Tag

2009-04-23T17:01:00.000-04:00

I'm guessing that no one cares about this, but I just spent all day tracking down this issue and blogging about it is going to make me feel better.

I was adding buttons to the toolbar in Word through Visual Studio Tools for Office (VSTO). My situation had me adding buttons to certain open documents. I ran into a problem where after adding two (or more) buttons to two (or more) documents I started getting two click (or more) click events every time I clicked (just once!) on any of the buttons.

CommandBar cmdBar; // get this from somewhere
CommandBarButton cmdBarBtn = (CommandBarButton)cmdBar.Controls.Add( blah, blah, blah );
cmdBarBtn.Caption = "Example";
cmdBarBtn.Tag = "Example";
cmdBarBtn.Click += btn_Click;

Turns out, that Tag property is really really special. You expect the CommandBarButton object to pretty much take care of differentiating one button from another, but that's not how it works in Word.

Word uses the Tag to differentiate between all the various button instances. Even though executing the code above twice DEFINITELY creates two different button instances (I verified by changing the Caption on the second one), Word can't tell them apart because they have the same Tag. So, because you registered two click events on the same Tag, you get two click events to fire.

I changed my code to:

cmdBarBtn.Tag = Guid.NewGuid().ToString();

Now my clicks are properly associated with the button that was clicked.

Ahh... I do feel better.