Anne Epstein

Why is mercurial hanging during adding file changes?

Anne Epstein — Thu, 11 Aug 2011 21:35:00 GMT

So, I was opening a new project in mercurial that had a bit of a history. I ran the normal:

hg clone ssh://hg@bitbucket.org/mycompany/somerepository

Which has worked pretty fast on the smaller repositories that I've cloned. THIS time, it got to "adding file changes" .... and stopped. The thing appeared totally hung, for at least the 10 minutes that I waited. I cancelled it, and tried again. No luck. So, after some searching, I came upon the debug command, which looks like this:

hg --debug clone ssh://hg@bitbucket.org/Mycompany/Somerepository

The result?

files: 891:4346 chunks (1.50%)
adding lib/AutoMapper/Automapper.dll revisions
files: 892:4346 chunks (1.51%)
adding lib/FluentNHibernate/Antlr3.Runtime.dll revisions
(etc... a LOT... 4346 times, actually)

Turns out that the problem was just that there were a lot of revisions of everything, and it was all just taking a while. Oops, I probably should have waited instead of restarting it, I would have been 10 minutes further along. But at least with the debug switch now, I know what it's doing and where it's up to if it's going to take a long time.

Python’s Yield

Anne Epstein — Thu, 27 Jan 2011 06:40:31 GMT

This week, I’ve been playing with implementing an HTTP client in Python. Why Python? It seemed like a straightforward language for this sort of thing, and the fact that I don’t know it at all is a bonus learning opportunity! In any case, I make no claim to be any kind of authority at all on the language. I am certain there are better ways to do all of these things, but I don’t know them-yet!

After a bit of googling and reading and copying and pasting, I ended up with the following methods.

def getLine(s):
    line = ''
    for l in iter(lambda: s.recv(1), '\r'):
        if (l != '\n'):
            line += l
    return line

def getHeader(s)
    for line in iter(lambda:getLine(s), ''):
        yield line

def getContentLength(header):    
    for line in header:
        if (re.match("Content-Length: \d+$", line)):
            return int(line[line.find(": ")+2 : len(line)])
    return -1

def fetchlines(s):
    header = getHeader(s)
    contentLength = getContentLength(header)

# other methods to get message

Assume fetchlines is the entry to the above methods. First, we are getting header lines up to the first empty line, then, we are pulling the content length from those lines. In later code (not shown), I get the body of the message based on the content length. But I had a strange problem. Here’s the header I was receiving:

HTTP/1.1 200 OK
Date: Wed, 26 Jan 2011 06:36:39 GMT
Server: Apache/2.2.4 (Unix) mod_ssl/2.2.4 OpenSSL/0.9.8a DAV/2 PHP/5.2.1
Last-Modified: Tue, 14 Dec 2010 17:23:19 GMT
ETag: "1104fb7-1ea9-4976213a4cfc0"
Accept-Ranges: bytes
Content-Length: 7849
Vary: Accept-Encoding
Content-Type: text/html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
...

I was expecting to pop everything down through the Content-Type line into header, then get the content length value in contentLength, then, socket reading being a one-way operation, the message reading code would start right on in at the DOCTYPE line. Instead, I kept having a strange problem: every time I started in on reading the message body, I ended up at the Vary line instead! I wasn’t sure why: I was specifically looking to read down to first blank line, and there was no blank line before Vary. I looked for hidden \r characters; nothing.

After some reading, I discovered the problem: yield. Though looking at a few examples, I’d been under the impression that yield was essentially a nice and terse loop-returning-collection construct; i.e. it would run through, collect up all the results, return as a collection all by itself. I did remember hearing something about yield being *weird* in C#, but I hadn’t actually ever used it, and couldn’t remember was the problem was. Besides, this was Python, and it seemed to be working, except for this little issue. How magical.

Uh... no.

Turns out this a case where it may have sorta looked like a duck, but it was something else entirely... Though I was treating the function’s return value like the list I believed it to be, getHeaders wasn’t returning a list. It was returning an iterator to a generator function. A generator function, when called through an iterator, will run through its body, return a value back to the accessor of the iterator at the yield, save state, and just hang out until you call for the next iterator item, at which point, it will resume on the line after the yield. So the loop in getHeaders wasn’t actually run when the method was initially called-instead, each run through that loop is done once for each run through getContentLength’s loop... and since getContentLenth exits once it gets the content length, it never pulls the rest of the items from the iterator, and so, the Vary line never gets pulled in getHeaders... the blank line condition is never even hit, and the Vary line is still waiting to get pulled when it was time to read the message body-not what we want!

A quick fix? take out the yield and use a regular list. It adds an extra line or two over the yield, but this way, I’m sure the full header gets read before pulling the message body.

def getHeader(s):
    header =[]
    for line in iter(lambda:getLine(s), ''):
        header.append(line)
        print line
    return header

Lessons to take from this: 1) yield is NOT generating a list, it’s an iterator pointing to a generator, which is entirely different, and on a broader note, 2) copying and pasting code that you don’t understand can result in behavior you don’t understand-or even worse, behavior that you only think you understand!

R is for Reliability

Anne Epstein — Sun, 09 Jan 2011 00:59:00 GMT

37 Signals was in the news last year for the wrong reasons: they experienced multiple outages in a short time period. From their own site, “The virtual machines had severely degraded network and CPU performance causing them to grind to a halt.” Sounds unfortunate, but these things just happen, right?

Well, there’s more to it than that. Though accidents happen, reliability isn’t luck. The severe problems they had were really due to a series of choices they’ve made over the years, one of which was to take their hosting in-house. Hosting some low-usage internal company document server is one thing-but if the core reason for their business relies on those servers, trying to minimize that hosting cost at the possible cost of reliability might be a cost-saving way to shoot themselves in the foot. But enough picking on 37 signals; they are no different from many of us doing dev work-we all tend to think of the development as the “real” work and everything else as something that’s not a big deal, something we maybe don’t need pay attention to, something we don’t need to pay so much for.

Reliability is not simple. It’s easy enough to get something working, but what if people come to rely on the application on that server being up at all times? What if getting people to depend on you always being available is part of your business strategy? People wanting high uptime throw around nines of uptime, as in 3-nines or 5-nines (i.e. 99.999% uptime)Think about that: that means you get only a few minutes a year for your service to be down, and that year includes thanksgiving, Black Friday, 4am on any random day, etc. For really high uptime, by the time you get the alert you’re already near the end of your SLA.

To get that kind of reliability, your infrastructure has got to be be ready for even the rare cases-what if the power goes out? What if it stays out for 4 days? What if internet is down? What if the internet provider’s pipe goes down? What if a server’s fan goes bad and that server’s fan type isn’t sold in the area? What if the area is flooded by a freak storm? What if the server room catches on fire? What if someone trips over a cord? What if all the support people going to lunch together get hit by a car? What if your so-far-been-reliable data supplier with the 99.999% SLA has a big failure of their own? What if there’s a bad code deploy...or a bad system update? Maybe these questions seem paranoid, perhaps these things seem silly and too unlikely. And honestly, for many of these possible problems, any one of them is a little rare to much worry about on its own. What you’re concerned about isn’t single risks on their own, it’s your entire system-level risk profile-how likely is anything of any kind going to happen. Calculating, and then bringing that risk down, ends up looking at the small things in service of understanding that big picture. There’s a pretty good chance at least one of these sorts of things will happen this year, or maybe next year-the thing is you don’t know what thing it’ll be, so for high reliability, you have to be ready for a lot.

What does “ready” mean? Well, for the internet, it may mean paying to have multiple internet providers, It may mean replacing servers while they are still running well to ensure parts are available. Generators, along with fuel and staff that know how to work it are a must. Many of these require having several complete systems ready to take over at any sign of trouble, located other places in the country, already spun up, with staff. It may mean having more duplicate systems to bring OS updates and patches online with the ability to fall back at any sign of trouble. It may mean spending much more on software development to build in the ability for the software to handle a multitude of seemingly unlikely scenarios, be it slowdowns, flaky connections, incorrect data, 20x the normal load, or dependencies failing. Certainly, there may be some risks that one might not want to prepare for-either because they are too unlikely, it's too expensive to prepare, or it's just a risk that changes the whole picture. Take a space alien invasion of Austin, TX for instance. This could be a severe risk for my company's projects-however it's a risk that we are OK not mitigating against as 1) to the best of my knowledge a space alien invasion is pretty unlikely 2) fully-office-ready alienproof bunkers are really expensive, and 3) We all might have other things to worry about that point.

How to handle each type of potential failure is a choice; a choice you may not realize you are making, but a choice nonetheless, and we’ve all made them on all of the projects we’ve worked on, though we’ve probably not thought about it for most of them. All those lines of protective code you didn’t write when playing around on a toy project? You could have written them but you didn’t (probably a good idea in that case). On the other hand, for a development shop working on a project that requires high reliability, it makes sense to build reliability into the software itself, and it makes sense for the dev shop to handle that internally as part of their core competency; however, for aspects of reliability that are outside of the dev shop’s core skillset, the company faces a choice: 1)make all of the minutia another core competency, devoting resources accordingly(i.e. you are no longer just a dev shop, you have specializations in development, disaster mitigation, recovery and server/hosting), 2)outsource those areas to someone for whom it is a core competency, or 3)accept that reliability just isn’t going to be what one might like, and set client expectations accordingly. The right choice here is going to depend on company size, resources, and desire for focus; more important than the actual choice made is to make it purposefully and with fullunderstanding-otherwise, there is a fourth possibility-promising a lot more than you’re ready to deliver.

I should mention that a really excellent book on the subject of reliability is Release It! If you’ve ever made a promise of uptime and not thought much about what you were promising, this book will give you a lot to think about.

Passive voice is killing your design documents

Anne Epstein — Thu, 12 Aug 2010 01:02:00 GMT

Recently, I was proofreading a technical design document written about a system that was unfamiliar to me. It described the flow of execution through a fairly complex system. I found it more difficult to follow the design than it should have been, but I wasn’t sure why. After some thought, I narrowed down on the culprit-like many technical documents, this one was written using a fair amount of passive voice.

“The passive voice is a grammatical construction (a "voice") in which the subject of a sentence or clause denotes the entity undergoing an action or having its state changed.” (Wikipedia) For example, an active voice sentence would be “I dyed the cat green”, while examples of passive voice would be “The cat was dyed green by me”, or even just “The cat was dyed green”. Notice that in the last of these sentences, you never learn that it was me that did the cat-dying; I’ve managed to forgo letting you know how I was involved at all. I think many of us in technical fields have learned to write this way throughout our careers. From being asked for impossibly precise estimates to being asked if certain unknown tasks were possible- then being held to those guesses, I think it’s become a defense mechanism for many of us to distance ourselves from the things we are trying to communicate.

There’s a problem with this natural instinct to fall back to the safety of passive voice: it can sabotage our attempts at clarity. Following the flow of an unfamiliar technical topic is often *hard*. The person trying to absorb knowledge is carefully trying to follow the process from step to step, somewhat like a driver trying to make his/her way to an unfamiliar destination using printed-out google map directions.   A passive voice sentence in a description of a complex technical algorithm can feel analogous to ripping out the middle steps of those google map driving directions. The result is something like this: “X performs an action, sending data to Y. Y then transforms the data. Action Q is then performed” This leaves the reader asking, "Why was Action Q performed? Was it X, Y, or something else entirely that performed action Q? How many intervening steps are invisible/implied? Is Q even related to X and Y?"

Another reason people use the passive voice is because they aren’t sure about some of the details at the time of writing, so passive voice provides a way to keep the document smooth and professional-sounding, while providing the illusion of continuity.   I would argue that if the goal is to communicate clearly, actually stating and explaining the uncertainties is going to be much more helpful to readers. Consider this revised version of the previous phrase: “X performs an action, sending data to Y. Y then transforms the data. Then, either Y or X performs Action Q -we're not sure which, it all depends on what turns out to be less resource-intensive, and we haven't finished that research yet”. This new version may feel more wishy-washy, but it is easier to follow, and it communicates some important uncertainties that the previous version glossed over.

Maybe discussing a grammatical point seems to be a bit off-topic or even pointlessly fussy, but use of the passive voice has a real effect on the readability of explanatory/design documents.  (In comparison, I doubt the lay/lie grammar issue or even use of a word like irregardless is going to cause any real readability problems if they show up in your document) Most programmers have to write this kind of document now and then, and it is in everyone’s interest that we write these documents so others can understand what we’ve written. A little thought about wording can go a long way towards making documents a means of communication instead of another reason for confusion.

Why I'm a programmer

Anne Epstein — Sat, 19 Jun 2010 05:23:00 GMT

I don't know if you can say I'm a programmer solely because of my dad(by training a physicist and by profession an engineer), but he was a big part of getting me there. I had my first experiences with a computer when I was very young, no older than 4 or 5- my dad would write small games games for me on the TRS-80: One of them was a spelling game: a word would flash for a moment on the screen, and then I'd type in the word, and the program would say whether I'd spelled the word right-this was obviously cool for both of us. He also showed me how he could tweak the game-add new words, adjust timing, etc. When I was just a bit older, my dad got me a kids' book on BASIC , and I went through most of the programs in there, running them on that TRS-80 by repeatedly listing them out to look over the code, and editing by individual line. vi would have been full-screen luxury, but the line editor was all I knew, so it was great! (Yup, one of those programs was a choose your own adventure game).

Aside from the actual programming, my dad's interest in computers kind of infiltrated the house-there were occasional cassette tapes labeled with program names and printed program listings lying around in drawers near the computer, and we tied up the tomato plants in the backyard with tape from spools of old reel-to-reel that was being thrown out at his office. (We also tied up the dog with some aircraft carrier deck emergency barricade line, but that's another story) Later on, when we finally replaced the TRS-80 with a Gateway 486 and the new computer's CPU turned out to be defective, I remember him replacing that chip while I hung out nearby. (It seemed perfectly normal to crack open and poke at the insides of a computer on the coffee table, though I'm now pretty sure Gateway would have done it for him, had he wanted to ship the machine back) Not to mention tons of hacking that autoexec.bat together once that computer was finally working...I of course helped through all these things!

By the time I was in middle school or so, my dad had made his way to management. I have many memories of the walks my dad and I would go on together, when he'd tell me about what he was learning for work at the time. He spent many walks telling me about Total Quality Management(a blatant generalization might be to call it a second cousin of Lean), and how they were trying to use it in the Navy. I remember him telling me that when one stepped back and looked at results, TQM looked like a better way to do things, but a lot of people were resistant. I asked, "but if it's better, why don't they want to use it? Why won't they at least give it a try?" (heh, a question I've asked myself about all sorts of things again many times later on.) He said something like "some people get comfortable doing things the way they've always done them, and they don't want to try things another way." This answer frustrated me-it seemed so absurd, it couldn't possibly be true. I believe it now; it still frustrates me, but I understand it better.

I can't say my dad ever *tried* to make me a programmer-he just showed me things that interested him. My experience growing up is not unlike those of many other programmers I've met-if anything, it may be notable that my dad didn't differentiate at all in showing me and my brother these things up to our individual levels of interest. I never got the idea that I should be any less interested in computers, in airplanes, or in helping to build furniture, than I was in my ballet lessons or drawing. I'm very appreciative of that-I didn't have any idea that some of my interests were perhaps "boy" things, and I don't think I really understood that my perception was maybe not the norm until I was older. By that time, well, I liked what I liked, and I'm glad things worked out as they did. Thanks, Dad!

FEELING INSPIRED?

I believe one of the best kinds of introduction you can give a kid to something you feel passionate about is to just *show* them you love it. Not necessarily by (just) teaching, but by example. Showing then that you genuinely think something is a blast is so much more inspiring than saying it would be a useful skill, or even claiming that it's fun (but not appearing to have fun yourself), regardless of the subject matter. Those things aside, if you are interested in directly teaching your kids some programming, here are a few resources I've run across, though I haven't personally checked them out:

Hello World! - A python-based kids' programming book written by a father and his 9-year-old son. Scott Hanselman has a great podcast episode discussing this book.
.NetRocks! podcast episode on teaching programs to kids with SmallBasic
Hackety Hack - a Ruby-based programming environment aimed at beginners (including kids)
Scratch - Scratch is a programming language that makes it easy to create your own interactive stories, animations, games, music, and art -- and share your creations on the web.

(Edit: Added Scratch)

Winforms (and WPF) Testing

Anne Epstein — Wed, 24 Mar 2010 19:33:00 GMT

Those of us who primarily work on web projects have a fairly good idea of what's available for automating UI testing of web applications; Selenium and WatiN , for instance. At Headspring, we take this up a level by isolating our UI testing code from the nuts and bolts of the page HTML, as well as the equally troublesome nuts and bolts of these UI testing tools. You can take a look at how we do things by taking a look at MvcContrib, which has a testing framework and Code Camp Server, which has some example usage.

A code snippet from Code Camp Server to illustrate:

                   Form<logininputproxy>("/login/login/index") // url
                       .Input(m => m.Username, "admin")
                       .Input(m => m.Password, "password")
                       .Submit();

It's great to have these kind of tools for the web, but what if we end up working on a Webforms or WPF project? It turns out there's a project on CodePlex called white, and it does for winforms and WPF what WatiN does for the web. Crank it up, and just like WatiN pops up a browser and starts interacting with your web app for you, white will click around and enter text into your winform app. However, working with white as-is means tightly coupling our testing code to the tool-which we want to avoid just as much with winforms as we did with web testing. Not to mention, if white is wrapped well enough that working with it is pretty much the same as working with WatiN, that's one less difference to remember-meaning more focus on solving problems, and less on figuring out how to work yet-another-tool. We're only starting out, but here's an example of how working with it; with any luck, it should look a bit like the code snippet above:

            Form<testform2input>("White Hello World") // form name
                .Input(x => x.PersonName, "Bart Simpson")
                .Submit();

Winforms aren't exactly like the web, and likely the metaphors age going to have to adapt to avoid strange impedance mismatches- so it's probably not going to be seamless moving from one to the other. However, if it's similar, taking into account these differences, then when switching to testing a winforms project, it might seem...about the way you'd expect.

This is very, very young, but the starter code for this wrap of white is available at my CodePlex repository.

On Learning

Anne Epstein — Mon, 28 Dec 2009 15:01:00 GMT

As programmers, we work in a discipline that is a bit different from that of many other people we know. Many other disciplines have developed to a point of reasonable maturity and stability, with a broad, long-agreed-upon base of knowledge, educational path, and perhaps even certification of some kind to show mastery. Programming, on the other hand, is comparatively an infant. The idea of programming as its own discipline is really only a few years old. Programming has no agreed-upon education of any kind- though degrees in computer science are widespread, it's not widely accepted that a computer science degree is necessary, or even particularly helpful for those going into a programming career. In contrast to a long-agreed upon base of knowledge, programming has gone through revolutionary changes every few years. Whether it's change of programming language or the changes in software development methodology, programming-related knowledge has a very short half-life.

Surrounded by a constantly shifting landscape of knowledge, we can't stay satisfied with the knowledge acquired a few years ago, or even a few months ago. Granted, it is probably a bad idea to constantly switch to the newest and coolest thing like some kid always reaching for the shiniest toy, forgetting the pile of neglected toys at his feet, but the discipline of programming is young enough that new, field-changing ideas emerge all the time. We're still finding the ground rules here. I suppose you could keep on working the way you always have, using the languages and techniques that have worked for you for years-however, if you do that without keeping an eye to the future and embracing key improvements, you will likely find yourself in a backwater of software development after some time. Not to mention, if truly improved ways of working come about, it could be considered unethical to blithely ignore them and deliver inferior products. On a different note, I feel that it's intrinsic to programming to explore-if exploration of new ideas doesn't excite you, you may find a career as a programmer a long trudge.

As a programmer, you must take responsibility for your own learning and the progression of your career. Perhaps you currently feel your company should do this for you; after all, it's work-related, right? If you feel this way, either change your mind right now, or think about reconsidering your career choice. Here's the thing-it doesn't *matter* to the company how you personally progress in your career and if you learn the new tools -all an organization needs to care about is if the staff as a whole progresses - and it can do that through new hiring if needed, saving them the work of training you. Truthfully, though you might find it unpleasant, your company might be perfectly happy if you don't learn-you can be the guy who babysits the old apps which you already know, and they can hire new, fresh guys for new projects. That's not so good for you though. If your project gets canceled, you can find yourself with antique knowledge and a lot of years of catching up to get to get yourself in a hire-able state.

From a company's perspective, keeping up with the pace of things is incredibly important - you don't want to be the company producing horse-buggies when Ford comes out with the Model-T. Though it can be "easier" to keep moving through turnover and new hiring, actively encouraging learning can really keep employees caring about your company and actively looking out for ways to get better. Though you might luck out on hiring the right visionary, creating and keeping a whole staff of loyal visionaries is going to be a lot easier and successful in the long run. (yes, it's practically a stereotype now, but think of google-many of their successful products emerged through side employee side projects actively fostered by the company)

On a final, personal note: as of early December, I joined the staff at Headspring, in Austin, Texas. One of the things that makes me happiest to work at this company is its commitment to growth and development of its staff, and its excellent balance between recognizing the things that have made the company a success, and recognizing that continuing that success means not resting on its laurels. Today, I will be spending the day at an all-day technical staff retreat, and I'm looking forward to it. Learning can be a lot of work, but it's really fulfilling. Off to learn now...

NHibernate and Composite Keys

Anne Epstein — Sat, 21 Nov 2009 01:29:00 GMT

Composite IDs are a common pain point a beginning NHibernate user runs into. Here's everything you need to get them up and running.

First, a caveat: composite keys are certainly mappable in NHibernate, but it's a little trickier than a typical single identity key would be. Compared to a normal key, there's some extra setup work, queries are a bit more painful, and they tend to be less optimized in terms of lazy loading. Because of these things, experienced NHibernate users often avoid composite keys entirely when possible. However, there are many legacy situations where multiple existing apps all hit the same db-a situation in which, if a composite key is already in place, it’s probably going to have to stay. As that's the most common use case for composite keys, I'll start from the assumption that you've got an existing database that you can't alter. (this is a *bad thing* - see THIS POST for why, but as developers, those kinds of decisions aren't always under our control)

YOUR OBJECTS

As I mentioned above, if you're considering mapping a composite key, you probably already have a database. (if not, I’d highly advise an alternative-perhaps sets, perhaps idbags, but that’s for another blog post) The NORMAL, PREFERRED direction of model design would be to work up your classes and once they work the way you want, extract the persistence structure from that (i.e. your DB). But if that were an option for you... well, you probably wouldn't be using a composite key in the first place. Anyway, let's take a scenario:

Your existing tables:

In our brand new NHibernate app, we want to have an object that corresponds to the CategoryProducts idea. This is a start:

namespace SuperShop.Domain
{
    public class CategoryProduct
    {
        public virtual Product Product { get; set; }
        public virtual Category Category { get; set; }
        public virtual string CustomizedProductDescription { get; set; }

        private DateTime _LastModifiedOn;

        public override bool Equals(object obj)
        {
            if (obj == null)
                return false;
            var t = obj as CategoryProduct;
            if (t == null)
                return false;
            if (Product == t.Product && Category == t.Category)
                return true;
            return false;
        }
        public override int GetHashCode()
        {
            return (Product.SKU + "|" + Category.Name).GetHashCode();
        }
    }
}

So, why the Equals and GetHashcode? If you try to map a composite key without them, you'll get an NHibernate error stating that they are required. Here's why: With this two part identifier, NHibernate can't do a simple single id object compare - you need to tell it how to decide equality. Implementing Equals and GetHashcode are always a good idea for anyway so your objects will be have properly in cases like multi-session scenarios where an unsaved object might really be the same as an existing object elsewhere, but in the composite key scenario, not having it is not an option- NHibernate doesn't even *have* a mostly-works technique to fall back on. (Note, this is almost certainly not the most ideal Equals and GetHashcode implementation-take a look here for more on the topic- but hopefully this gives you the general idea. )

MAPPING

A mapping:

<hibernate-mapping>   
<class table="OrderItemProductDetails" name="SuperShop.Domain.ComponentPersonalization, SuperShop.Domain">
    <composite-id>
        <key-many-to-one class="SuperShop.Domain.OrderItemComponent,SuperShop.Domain" name="OrderItemComponent" column="OrderItemProductID" />
        <key-property name="DetailType" column="DetailTypeID" type="SuperShop.Domain.DetailTypes,SuperShop.Domain" />
    </composite-id>
    <version name="LastModifiedOn" column="LastModifiedOn" type="timestamp" access="field.pascalcase-underscore" />
    <property name="DetailValue" column="DetailValue" type="String"></property>
    <property name="DetailCharge" column="DetailCharge" type="Decimal"></property>

</class>
</hibernate-mapping>

Note the <version> element, as well as the matching _LastModifiedOn in the class above. These two items combined let NHibernate know how to tell if an entity is new or not. In the usual scenario where NHibernate manages the ID, NHibernate monitors whether the id value is the original unsaved value and determines whether to Save or Update for you if you call SaveOrUpdate(), a very handy method. If NHibernate is not managing the ID, as is the case in an Assigned ID (think an SSN that you manage) or in composite (where you create the relationships or values yourself that make the id) then it doesn't know how to tell if your id is saved or not-its usual technique doesn't work. So with <version> NHibernate gets a column it has control over, and can safely monitor for an unsaved value. Without this, you'd be unable to use SaveOrUpdate with this element-you'd have to call Save or Update as appropriate-additionally, since ALL cascading functions on collections are essentially NHibernate calling SaveOrUpdate, you're not going to be able to use cascading. Alternatively, if you don't like the <version> column, you could implement IInterceptor 's IsTransient() method to get similar functionality. (see documentation at nhforge)

So, if you want to take the <version> approach, you'll need to add a new DateTime column to your CategoryProducts table:

QUERYING

An inconvenient aspect of composite ids is the need to query on all parts of the id. For instance: a GetByID query:

from CategoryProducts c where c.Products = :p and c.Category = :cat

but it's not just on the GetByID, it's *whenever* you might need to search using a particular CategoryProduct. For instance,

select distinct p from ProductImages p join p.CategoryProducts c where c.Products = :p and c.Category = :cat

ID OBJECT

Composite IDs can be problematic for lazy loading... When lazy loading, NHibernate will get just the ids of a collection, and hold off on getting the rest of the object until it's needed. An important fact- NHibernate can't partially load an object-in terms of discrete "things". If you're talking a plain integer ID, it can load up the integer, and then load up the associated object later. With our object as specified above, the smallest single discrete thing that contains the key is... the whole object- so, you've effectively killed your lazy loading. What to do if we want lazy loading? Well, make something smaller that contains the key. Let's make that ID object:

[Serializable]
public class CategoryProductIdentifier {
        public virtual int ProductId { get; set; }
        public virtual int CategoryId { get; set; }

        public override bool Equals(object obj)
        {
            if (obj == null)
                return false;
            var t = obj as CategoryProductIdentifier;
            if (t == null)
                return false;
            if (ProductId == t.ProductId && CategoryId == t.CategoryId)
                return true;
            return false;
        }
        public override int GetHashCode()
        {
            return (ProductId + "|" + CategoryId).GetHashCode();
        }
}

then, CategoryProduct becomes:

public class CategoryProduct
{
        private CategoryProductIdentifier _categoryProductIdentifier = new CategoryProductIdentifier();
        public virtual CategoryProductIdentifier CategoryProductIdentifier
        {
            get { return _categoryProductIdentifier; }
            set { _categoryProductIdentifier = value; }
        }

        private Product _Product;
        public virtual Product Product
        {
            get { return _Product; }
            set { _Product = value;
                _categoryProductIdentifier.ProductId = _Product.Id; }
        }

        private Category _Category;
        public virtual Category Category
        {
            get { return _Category; }
            set { _Category = value;
                _categoryProductIdentifier.CategoryId = _Category.Id; }
        }
        public virtual string CustomizedProductDescription { get; set; }

}

Mapping Tweaks:

 
<hibernate-mapping>
<class name="CategoryProduct" table="CategoryProducts">
    <composite-id name="CategoryProductIdentifier" class="CategoryProductIdentifier">
        <key-property name="ProductId" column="ProductID" type="Int32" />
        <key-property name="CategoryId" column="CategoryID" type="Int32" />
        <version name="LastModifiedOn" type="timestamp" column="LastModifiedOn" />
    </composite-id>
    <many-to-one name="Product" column="ProductID" class="Product" insert="false" update="false" access="field.pascalcase-underscore" />
    <many-to-one name="Category" column="CategoryID" class="Category" insert="false" update="false" access="field.pascalcase-underscore" />
    <property name="CustomizedProductDescription" column="CustomizedProductDesc" />
</class>
</hibernate-mapping>

The key thing to note here is that Product and Category are referred to twice in both the class and the mapping. The reason for this is that caching uses primitives like int or string, so we need to feed it something caching-ready.The reason for this is because to index by a custom class in the cache, this composite id class, like an ordinary id, gets serialized. This serializability comes for free if your id is a single int, but your id is your own custom object, as with this composite id class, you've got to explicitly both specify that serialization is allowed, and make sure the object is valid for serialization. So we've pulled out those ids as ints into the identifier. However, we still want to be able to traverse these relationships, so we still include the class. However, if NHibernate tried to update the same db field twice, you'd get errors. To be able to have both the product relation and the ProductId mapped separately in the same class, we mark the class reference as non-updatable. Also, note that the Equals and GetHashcode moved to the CategoryProductIdentifier class - the CategoryProduct class is for the most part free of the composite burden; the burden of composite-ness is now on the CategoryProductIdentifier class.

It’s entirely possible I’ve missed something in regards to NHibernate usage with composite keys-if so, let me know, and I’ll add it in. If I got anything factually wrong, let me know about that too so I can make it right!

EDIT: Added in serialization info that I'd forgotten. Thanks to bonskijr for letting me know!

NHibernate Caching and Assigned Ids - A Potential Pitfall

Anne Epstein — Thu, 08 Oct 2009 13:53:00 GMT

Yesterday, I spent some time setting up and optimizing SysCache2 caching. (By the way, if you're doing this kind of NHibernate tuning, NHProf - a devlicio.us friend - is crucial, it would have definitely been a much tougher slog without it) With caching turned on, one of our previously well-functioning classes was consistently giving the following error EVERY time we tried to modify and save an instance:

ReadOnlyCache: Can't write to a readonly object

Looking this up, I ran across a LOT of information about how this meant the cache line in the mapping was wrong - that it needed to be marked read-write, not read-only. The odd thing was, this is what the cache line in our mapping was like already:

So, our problem was NOT the obvious issue that our object was improperly marked as readonly. But... after a lot of flailing around, the solution emerged. Check out this definition of the ID on this entity:

Note the use of "assigned" for the id generator. It means NHibernate doesn't control the id-it's manually managed. Now if you're familiar with the assigned generator, you know that it's a somewhat troublesome thing to work with-specifically, if you call SaveOrUpdate on an entity using it, NHibernate won't know whether it's a new entity or not on its own, because NHibernate tracks the state of the entiy using the id. This affects cascade saves as well, so assigned ids aren't fun. If you take over managing that id yourself, NHibernate doesn't have anything to hang its tracking off of. Assigned ids are something you should probably avoid for new applications, but sometimes that's not an option for legacy apps.... If you NEED to use an assigned id there's an easy way to get this entity acting like a normal NHibernate entity-you can give NHibernate something to use like this:

<version name="LastModifiedOn" type="timestamp" column="LastModifiedOn" />
-with the matching column, and the matching property in the class - or maybe even just using the access="field.." option in the mapping so it's not mistakenly taken to be something to be part of the class's regular api by another programmer-this field is NOT FOR YOU, it's for NHibernate. Look at it if you want, but do not touch! Note that if you add a column like this to an existing table, you're going to have to prepopulate the existing rows for your your new column with an initial datetime or you'll get errors-any date at all, so long as it's in the past. 1/1/2009 12:00:00 is as good a value as any....

What does this have to do with my caching problem? The particular entity I was getting the erroneous readonly caching problems on didn't have the <version> property-a bad state of things for an assigned id entity. Oddly, it *was* in fact saving properly using SaveOrUpdate - perhaps it was the unsaved-value = "0" on the id allowing it to work, maybe not. Anyway, the solution ended up being adding a <version> property and column, and then it worked like a charm. I probably should have caught that this entity was missing <version> earlier, but it seemed to work ok, and the potential problem stayed hidden until the entity got thrown into the cache.

By the way, if you don't like <version>, there are alternatives, like IInterceptor.IsTransient()

You are your Institutional Knowledge

Anne Epstein — Fri, 28 Aug 2009 03:38:00 GMT

In the midst of recession, many companies are closing locations and laying off staff to cut costs. The thinking from many of these companies is that the remaining staff can pick up any projects the laid-off staff previously did, resulting in all the important work being accomplished. Secondary tasks may end up being done a bit less frequently, and any extra can get picked up by contractors, who don't entail the same hire/fire burden.

There are some flaws in this thinking for anything with real complexity, and these flaws all relate to institutional knowledge.

First, it assumes that the remaining staff knows what is important. In reality, today's workers are often so siloed they don't have any idea what the person two cubes over is doing. Lay off a few people, and entire critical infrastructure applications may be forgotten-until they stop working, to the confusion of the remaining staff. Also likely- remaining staff will know of the existence of such applications, but most likely won't know where such apps sit, or how they work. It's hard to prioritize importance when you don't have the basic facts.

Second, it assumes unfamiliar applications are not overly burdensome to learn, even when the previous maintainers are no longer present. The truth is that many existing applications were built over many years using fragile programming techniques, and then went through a maintenance period that resulted in counterintuitive, self-contradicting, hard-to-follow, even misleading code. These issues may not have been issues for the previous maintainers because they were so familiar with it that they knew all the common problems, and how to handle them. I think of the old Dodge Caravan I drove around in college-not fuel injected, so I knew to press the gas when I started. The hydraulics in the rear door were broken too, so I knew to keep my head down, and I kept a hockey stick in the back to prop it up. I also knew to use premium fuel or it would start knocking like crazy. but I knew about these issues, had workarounds, and was largely unaffected.

(This one was not mine, but mine was very similar save for being white with fake wood) However, had I lent the car out to a friend without this info, my friend may not have even been able to start it, and later probably would have ended up getting smacked in the head with the rear door, and then wondered if they broke the thing after filling up with regular gas and hearing extra knocking. These problems take only a few minutes to understand; unfortunately, learning all the subtleties of a large, complex codebase can take months or even longer, even with the guidance of someone with experience.

Sure, with time, the important applications will be discovered and eventually learned, but what happens in the meantime? What if the important app is customer-facing? What if it runs your business's billing? Can you live without this app for days while your staff takes baby steps learning the most basic elements of an app, trying to figure out why it's non-functional? Can you live with new and different critical errors happening repeatedly, as staff learns the idiosyncracies of the app the hard way? Or, can you live without any work of substance being done on a critical app for an extended period of time for fear of encountering the issues mentioned above? This is what happens with the loss of institutional knowledge. If you can't afford these kinds of problems, you might reconsider the nature of your business... regardless the stated goal of the company or any products it might sell, reliance on knowledge means your business, IS, to some level, the institutional knowledge of the people that make it up, and losing that knowledge could be tantamount to losing the business. Tread carefully.

EDIT: A related concept I really wanted to mention is the idea of the Truck Factor, i.e. "The number of people on your team who have to be hit with a truck before the project is in serious trouble" Anytime you lay off people who have specialized knowledge, you're lowering your truck factor... if you lower it down to 1, particularly on a stressful project, you're potentially in real trouble.. even if that person isn't hit by a truck, there's a good chance he/she will leave when the economy recovers.

Bottlenecks

Anne Epstein — Wed, 22 Jul 2009 05:46:00 GMT

I’ve been reading though Implementing Lean Software Development as a member of the San Antonio Tech Book Club. Bottlenecks are discussed in Chapter 7, and I had some notes/reflections on the subject…

Lean suggests that it’s important to figure out, and focus on the main problem. This is an idea we know from optimizing our programs, for instance, if a query is 55 out of a 60 second operation, the thing to optimize is the query-optimizing anything else isn’t going to make much difference. Similarly, if an app spends a week in testing, and 6 months in approval, making even huge improvements to finish testing earlier can make at most 1 week’s difference in the final delivery date-but even a minor improvement in the approval process can save several weeks. Maybe the biggest bottleneck is a tough task-but it’s a task that has the potential to make the biggest real difference.

Addressing bottlenecks isn’t easy. To correct a problem, you’ve got to accept it as true, and take ownership. Think about that-ownership doesn’t mean denying the problem exists, making excuses or the problem, or blaming the people who set up the current system initially, it means saying this problem is my problem. It doesn’t mean taking the blame, or even denying blame. Blaming doesn’t solve problems, removing the root cause solves problems.

When working on removing a bottleneck, sometimes the first approach doesn’t work. Continuously and honestly evaluating the success of new approaches is crucial-it’s equally important that though we take ownership of a problem, we don’t take the results of a particular new approach personally. If we become overly attached to a failing approach, or even a successful approach, we won’t be open to a newer, more successful approach, and worse, we might even see the newer approaches as a threat.

Perhaps the prerequisite to all of this is organization-wide commitment to eliminating bottlenecks. The biggest hindrance can be anywhere between initial request and final delivery, and without an org-wide commitment, it might be impossible to make changes needed to solve problems, and it might even be difficult to identify what the true problems are. Your team may be optimizing the heck out of itself, but if there’s a bottleneck elsewhere, those efforts might not be having a noticeable effect, and depending on the situation of the real bottleneck, those small-scale efforts might even be making things worse. Why waste that effort? Make things better, but make sure you’re really making them better.

Austin Code Camp 2009

Anne Epstein — Sun, 17 May 2009 23:57:00 GMT

I just wanted to give a heads-up to people in the central Texas area that a great event is coming up at the end of the month: Austin Code Camp 2009. It's taking place May 30th, and the lineup of likely sessions is looking fantastic. Our friends at Los Techies are going to have a substantial presence there, and I'll be there too. A lot of people have worked hard on setting up this free event, and I'm really looking forward to it. If you're in the area, you should definitely attend-I hope to see you there!

VB.Net and the Case of the Iffy Ifs

Anne Epstein — Tue, 12 May 2009 05:49:00 GMT

Previous Items in Series:
Intro to series on quality VB.Net
VB.Net and the Spaghetti Code Of Doom

In the previous item in this series, we looked at a pretty typical class, discussed some of its problems, and started in on fixing those problems. We finished with smaller, easier-to-understand pieces of code, but it still exhibits some notable problems. Most visible for me are all those branching if/else statements, some of them nested. This is a potential problem for several reasons:

Each time we add/remove/change the customer levels, we're going to need to change both the OrderTotalService and the ShippingService. That's one core change we have to make, and now we're going to have to update two different pieces of code. Hopefully, we remember to do that!
Each time we go in and mess with any customer level, we have to change these classes. Hopefully, we change the classes safely and don't accidentally destabilize other customer levels when we're in there! This is starting to feel a little risky...
Are all these tangled ifs really readable? they take some mental walking through to see what's going on.
It looks like constructing test data that matches the pattern required to drill through all these conditions could be tricky. Testing is important, but it's a lot harder to find the motivation when it seems like it's going to be an overwhelming task.

So, one way to handle the mess is to try some polymorphism. The basic idea we're going to use is that instead of switching on a property of a class, create a set of classes based on that property that share an interface, and move the switched behavior over to the classes. Let's give that a try.

The Customer class has been subclassed into three different classes to reflect the Gold, Silver, and Basic customer levels. Then, the customer level-specific product price and shipping calculations have been moved into the subclasses:

Public Interface ICustomer
    Property CompanyName() As String
    Property ContactName() As String
    Property Orders() As IList(Of Order)
    Sub AddOrder(ByVal order As Order)
    Sub RemoveOrder(ByVal order As Order)
    Function CalculateProductPrice(ByVal prod As Product) As Decimal
    Function CalculateShipping(ByVal orderTotal As Decimal) As Decimal
End Interface

Public Class GoldCustomer
    Inherits Customer
    Implements ICustomer

    Public Overrides Function CalculateProductPrice(ByVal prod As Product) As Decimal Implements ICustomer.CalculateProductPrice
        If prod.IsMegaDeal Then
            Return 11.5D
        Else
            Return prod.Price * 0.8
        End If
    End Function
    Public Overrides Function CalculateShipping(ByVal orderTotal As Decimal) As Decimal Implements ICustomer.CalculateShipping
        Return 0D
    End Function
End Class

Public Class SilverCustomer
    Inherits Customer
    Implements ICustomer

    Public Function CalculateProductPrice(ByVal prod As Product) As Decimal Implements ICustomer.CalculateProductPrice
        If prod.IsMegaDeal Then
            Return prod.Price * 0.7
        Else
            Return prod.Price * 0.9
        End If
    End Function

    Public Overrides Function CalculateShipping(ByVal orderTotal As Decimal) As Decimal Implements ICustomer.CalculateShipping
        Return 3.5D
    End Function
End Class

Public Class BasicCustomer
    Inherits Customer
    Implements ICustomer

    Public Overrides Function CalculateProductPrice(ByVal prod As Product) As Decimal Implements ICustomer.CalculateProductPrice
        If prod.IsMegaDeal Then
            Return prod.Price * 0.9
        Else
            Dim ddService = New DealDayService()
            If ddService.IsDealDays() Then
                Return (prod.Price - 3)
            End If
        End If
    End Function
    Public Overrides Function CalculateShipping(ByVal orderTotal As Decimal) As Decimal Implements ICustomer.CalculateShipping
        If orderTotal < 20 Then
            Return 1.4D
        ElseIf orderTotal < 40 Then
            Return 2.5D
        ElseIf orderTotal < 70 Then
            Return 5D
        Else
            Return 8D
        End If
    End Function
End Class

Public MustInherit Class Customer
    Implements ICustomer

    ... (original methods) ...


    Public MustOverride Function CalculateProductPrice(ByVal prod As Product) As Decimal _
        Implements ICustomer.CalculateProductPrice
    Public MustOverride Function CalculateShipping(ByVal orderTotal As Decimal) As Decimal _
        Implements ICustomer.CalculateShipping
End Class

(A note: we're seeing the effects of a VB.Net "feature" here: in VB.Net-methods implementing an interface *must* state what it's implementing from that interface. Customer shouldn't really need to implement ICustomer-and thus wouldn't need the two MustOverride methods. The problem is that if Customer didn't implement ICustomer, then when GoldCustomer, etc used Customer's methods to implement ICustomer, we'd get a compile error because those method/properties in Customer wouldn't have the "implements" mapping. We can avoid this issue by making GoldCustomer contain a Customer instead of inheriting from it, but that's a discussion for a different post)

This shrinks the OrderTotalService to just a few lines, and shrinks the ShippingService to nothing-it would be reasonable to completely remove the ShippingService class in the next refactoring pass.

Public Class OrderTotalService
    Public Function CalculateOrderTotal(ByVal ord As Order) As Decimal
        ' this first part adds up the cost of all the products
        Dim total As Decimal = 0D
        For Each prod In ord.Products
            total += ord.OrderedBy.CalculateProductPrice(prod)
        Next

        Dim shipService As New ShippingService
        Return shipService.CalculateShipping(ord.OrderedBy, total) + total
    End Function
End Class

Public Class DealDayService
    Public Function IsDealDays() As Boolean
        Return DateTime.Now > New DateTime(2009, 1, 18) AndAlso DateTime.Now < New DateTime(2009, 2, 18)
    End Function
End Class

Public Class ShippingService
    Public Function CalculateShipping(ByVal cust As ICustomer, ByVal total As Decimal) As Decimal
        ' this next section adds in the shipping
        Return cust.CalculateShipping(total)
    End Function
End Class

The last change here is in the Product class-I made a decision that though there were only two hardcoded MegaDeal products, this felt like a group that could change, and I wanted to get those hardcoded values out of there. Instead of being hardcoded, those values are now going to be stored as a flag in Product. (It very well may be that MegaDeals would be better implemented as some way other than as a property the first priority was getting out the hardcoded values-that can be revisited)

Private _isMegaDeal As Boolean
Public Property IsMegaDeal() As Boolean
     Get
        Return _isMegaDeal
     End Get
     Set(ByVal value As Boolean)
         _isMegaDeal = value
     End Set
End Property

Next, I'm going to do a simple change on those inner ifs just to make them a bit easier to visually scan:

Public Class GoldCustomer
    Inherits Customer
    Implements ICustomer 

    Public Overrides Function CalculateProductPrice(ByVal prod As Product) As Decimal Implements ICustomer.CalculateProductPrice
        If prod.IsMegaDeal Then Return 11.5D
        Return prod.Price * 0.8
    End Function 

    Public Overrides Function CalculateShipping(ByVal orderTotal As Decimal) As Decimal Implements ICustomer.CalculateShipping
        Return 0
    End Function
End Class

 
Public Class SilverCustomer
    Inherits Customer
    Implements ICustomer

    Public Overrides Function CalculateProductPrice(ByVal prod As Product) As Decimal Implements ICustomer.CalculateProductPrice
        If prod.IsMegaDeal Then Return prod.Price * 0.7
        Return prod.Price * 0.9
    End Function

    Public Overrides Function CalculateShipping(ByVal orderTotal As Decimal) As Decimal Implements ICustomer.CalculateShipping
        Return 3.5D
    End Function
End Class

 
Public Class BasicCustomer
    Inherits Customer
    Implements ICustomer

    Public Overrides Function CalculateProductPrice(ByVal prod As Product) As Decimal Implements ICustomer.CalculateProductPrice
        If prod.IsMegaDeal Then Return prod.Price * 0.9
        Dim ddService = New DealDayService()
        If ddService.IsDealDays() Then Return (prod.Price - 3)
    End Function

    Public Overrides Function CalculateShipping(ByVal orderTotal As Decimal) As Decimal Implements ICustomer.CalculateShipping
        If orderTotal < 20 Then Return 1.4D
        If orderTotal < 40 Then Return 2.5D
        If orderTotal < 70 Then Return 5D
        Return 8D
    End Function
End Class

Conclusion:

Getting rid of those ifs made a big difference.

Now, if we need to modify a customer level, we can do it in one place, in that level's class. No more hoping we got all the places we branch over customer level types! In addition, if we need to add a new level, *none* of this code needs to change-we just add a new class for that new level that implements ICustomer, and everything that's here works as-is.
Related to the item #1, when we change a customer level we don't have to change the places that level is used. We can test the heck out of what actually *had* to change, the modified or new level, and since nothing else was touched, we won't have to consider any other exiting code destabilized.
Unfortunately, there are still ifs, but we've split them out some, so they're not so nested. OrderTotalService and ShippingService are a lot cleaner and easier to look at.
Testing these classes is a little easier now. we don't have to feed as many combinations CalculateOrderTotal now-it treats all customer types the same. We still have to test that logic in the inheritors of ICustomer, but without that extra layer of ifs, it should be a little clearer what conditions need to be tested.

This code can definitely still be improved! For one thing, we'll have to keep an eye on the logic inside the customer level subtypes. Though the differing computation logic is now split off and close to what it differs on, I've got a feeling that code could get complex and overwhelm those classes over time. The DealDays logic and dates seem to be fixed to a single datespan, but that's the sort of thing clients want changed regularly, and we also don't have a way to calculate the order total without the shipping right now. We also still haven't addressed the upcoming desire to be able to handle pricing for different shipping vendors either.

The Case for Multiple DBs in Multi Tenancy situations

Anne Epstein — Fri, 24 Apr 2009 05:49:00 GMT

Multitenancy isn't easy, and one of the particularly challenging aspects is what to do with the database. In Ayende's excellent posts on multi tenancy, he's pretty clear that he feels a database per tenant is the way to go . There are two major, reasonable approaches that I know of: First, to have a database per tenant. (This is the approach discussed by Ayende in his multi tenancy series) A second common alternative is to have a tenant id column in all tables that require differentiation by client. My team was in a situation requiring multitenancy, and needed to select an approach. Wary of the idea of multiple databases, the team immediately decided to go with the tenant column approach.

Looking back, I think Ayende was right. The one advantage of the tenantid column system is not having to deal with multiple databases. There are many disadvantages. I'll discuss the advantages, and then give some thoughts on how to deal wwith the complexity of working with multiple databases.

Advantages of multiple databases:

In my mind the most important developer-centric advantage of having multiple databases is the ability to refactor the database. Many times, developers will design a system a system with certain assumptions, then realize those assumptions are wrong. Other times, a major new feature would be much easier and cleaner with certain system changes. If the database is only talking to one system, then when the developer comes upon these situations, s/he can also refactor the database to reflect that understanding, and keep the mapping between database simple, clean and easy to understand. When the code is pushed live, db update scripts also get pushed live. However, if multiple systems (in this case tenants) talk to the same database, this kind of db structure change just can't happen-pushing a db change would alter everyone's db, meaning everyone's code would also have to be updated at once. In many situations, that kind of simultaneous mass upgrade would be totally unacceptable, meaning that once this kind of multitenancy system is put in place, the single db can never again be altered.

Another advantage of multiple databases is the ability to easily load-balance by moving individual tenants to different db servers as necessary. If the databases are architected to be isolated, this isn't a problem-change a few connection strings, and you're done. In a shared system, all tenants suffer if one tenant is overloading the server.

On a related note, though nobody likes to think about it, clients sometimes leave, and when they do, they will want to be able to take their own data with them. If they have their own database, it's as easy as zipping it up and getting it to them. If all tenants are on the same db, it could be a much more difficult exercise involving picking through every table and selecting out the tenant-specific data. Hopefully you dont give them anything that isn't theirs. Which cleads to the next point.

From a business perspective, data separation is probably the most important reason to go with separate dbs. With a multitenant system, it's impossible for one tenant to end up with anyone else's data. Other tenants' data is in a completely different system, so there's no worry about messing up the tenantid and passing a client someone else's data. Depending on the situation, this could just be embarrassing; in other situations, you might be looking at a lawsuit.

Lastly, having separated databases makes many things in the application more complicated, for the life of the application. Every join, every query, every new table, every mapping has to deal with the tenant id. If the strategy is to use the tenant id to make a composite key on the tables, then you've got that much additional complexity-and if you're using a tool like NHibernate, you'll soon find out that working with composite keys is a LOT less seamless than a single autogenerated key. Eliminate the tenantid, and a whole lot of things become forever simpler, clearer and have a lot less potential for error.

How to deal with multiple databases?

The trouble with multiple databases is that now you have 20 or even 100 databases to manage instead of one. Managing this manually would be a difficult, error-prone task that could become overwhelming. In addition, any shared data can become out of sync. The solution? Don't do *anything* manually with these databases.

For versioning: each time you create a new version of your application, create a db script to update the db from the prior version from the last version to the newest one, and create a script to roll the database back from the newest version to the previous version. There are tools to help produce this kind of script... If you're programming in .Net, Visual Studio Database edition has a tool to help do this for SQL Server, and Red Gate has one as well called SQL Compare. Bundle this script in with the deploy of the code, and push it all out together, to each tenant as they get upgraded.

For shared data: 99% of the time, shared data is of the sort where it needs to be updated soon, but not necessarily real-time. An example of this is cross-client data for statistical computation. To allow this kind of computation in a multiclient situation, create a process that goes to each system and collects the appropriate data, possibly in varying ways based on the tenant's installed version, then pulls that data into a single system. The combined reporting system can be optimized and tweaked for reporting needs without affecting the designs of the individual tenants. Conversely, if there's any reference data they all need to share (dropdown data, etc) this information can be coordinated with a similar process that does a data push to the different systems, based on application version.

This kind of automation is some up-front work, but once it's done, it's done, and it allows tremendous application flexibility... and once it's set up, it's easy-there's no reason it couldn't become fairly invisible from a management perspective.. In addition, it's really a very separate task from the business application itself-it's actually so separate that the syncing application can be written by a completely different team that knows little to nothing about the main application. For that matter, I'm not sure if there's a tool out there that does all of these things already(I bet there is, come to think of it), but there's no reason such a tool couldn't exist.

Scrum-It's not about completing the sprint

Anne Epstein — Fri, 27 Mar 2009 04:18:00 GMT

This week, I was reading "Agile Principles, Patterns, and Practices in C#" by Robert Martin. In one of the early chapters, he talks a bit about Scrum, and mentions that working overtime (excepting the week of the release sprint) is a big antipattern.

Now, I've been on a team that practiced Scrum, and we worked at least some overtime fairly regularly-our single goal was to complete our chosen tasks for the sprint. We felt that we'd made a solid commitment that because we'd said we'd do those things, we'd do them. On my team, overtime was a good thing if it allowed us to keep our word and finish the sprint. In fact, finishing sprints was the whole key to our version of Scrum... the company expected us to finish our sprints, and in return, we got to have sprints.

How does the team's regular use of overtime mesh with Scrum? It doesn't. We had it wrong. Scrum's not about completing sprints, it's about learning how your own team works. When I read about overtime was pretty much forbidden in Scrum, I finally got it-if overtime is forbidden and you're behind, sometimes you just don't get everything done. If you couldn't get everything done in the time allotted, something is *wrong*-and whatever it is, you've gotta bring that out into the open so you know about it, as soon as possible. If people work overtime every sprint, you may not even know that the team is consistently unable to finish in the time allotted, and the root problem may compound over time. Whatever it is, be it an increasingly degraded/unreliable codebase, team disagreements, an underperforming team member, frequent interruptions, or tasks that just ended up being a whole lot more difficult than expected, if you don't know about it, you can't fix it. Forbidding overtime and allowing the sprint to finish uncompleted when things don't work out keeps the need for introspection on and resolution of the underlying issues at the forefront. How you fix the problems is up to you and isn't specifically addressed by Scrum-it may involve team building, examining distribuion of work, taking aside some people to find out why their velocity is down, or focused efforts to increase team velocity such as training or bringing in some XP(Extreme Programming) techniques. Whatever you try, Scrum can be a companion in making sure you know how you're improving.

In the end, consistently completing sprints is really kind of a side affect, a wonderful reward that comes from the hard work of building and understanding of the truth about your team, and then making the tough changes based on that knowledge. Sure, completed sprints is what you hope for, but overtime isn't the answer-it's just a short term solution when what you're dealing with is a long-term problem. If finishing the sprint tasks becomes more important than understanding the team's weaknesses, then Scrum becomes simply a way to organize tasks-you've lost all the introspection and growth, leaving just an alternate form of Microsoft Project.