I'm not entirely sure yet what I'm going to do with the content on this blog. There's quite a few posts I surely want to keep around, but certainly not all of them. For now, I'm going to keep the blog up so everything stays available but after a while, I'm gonna shut it down. I might keep up some kind of static archive of my favorite posts, or I might just put them on GitHub in MarkDown format. If you have any suggestions on what I should do with it, I'd be happy to hear them.

I will remain active on Twitter and I plan to be more active on GitHub from now on. But the blogging thing ends here and now. I'd like to thank everyone for reading, especially the ones who've been around since the beginning. It's been an interesting ride for me, but it's time for something else

1. The people I work with aren't good/talented/disciplined enough to make this work.

2. It can't possibly work in our organization.

The truth though, is that it's actually because of you. Whenever you say "that won't work for us" or something similar because of the people you work with, you immediately show your lack of faith in them. You might think that that lack of faith is justified, but that basically means you've given up all hope of making things better. You know it, and worst of all, they know it too. People who aren't trusted by their coworkers or superiors generally pick up on that rather quickly. Some of them will respond by putting in more effort and trying to do better. Others will become demotivated and wonder why they should even try because "it's never good enough anyway". At this point, your lack of trust in your coworkers becomes a self-fulfilling prophecy.

You might want to consider another approach. Think back on how you've responded to situations where you where challenged by coworkers or superiors but where it was clear that they trusted you and believed that you were capable of doing a better job. I'd bet that for most of you, it improved your motivation and your willingness to do better. It probably helped you bring out the best in you. You can use this to your advantage when you're trying to get better results out of your coworkers. It's quite simple really: it's all about how you approach someone. If you want somebody to do better, and you approach them in a positive manner you're much more likely to get better results. If however, they sense your distrust (and most people pick up on that quite easily), they will most likely be defensive or apprehensive. The odds are quite low that something good will come out of that.

If you want people to improve and do better, it's important to establish a culture where everybody knows that it's ok to make mistakes, and that every mistake is an opportunity for the whole team to get better. If that's not the case in your situation, what is preventing you from trying to get to that point? Even if you're not officially in a leadership position, there's no reason why you can't set the tone and be a leader. Start making an impact on one or two people and build off of that growth. Once you've got a few people convinced, it's much easier to bring the rest along with you. You don't need to shoot for the stars from the beginning either. Try to improve whatever it is you think needs improvement with small steps, in an iterative approach. Instead of thinking "that will never work for my team", think "we're not quite ready for that yet, but we will be someday".

If you believe you can't improve your way of working due to organizational reasons, it's important to also start with your team. It's easy for management to ignore the opinion of one person, but it gets progressively harder for them to ignore it as more and more people start asking for change. It's much easier to persuade management to try something new if you can show that the whole team believes it's worth a shot. Keep pressing the issue until you get the chance to try it for a short time. If you keep asking for it, management might just get tired of hearing it and who knows, they'll grant you some time to try it out. You'll never know if you stop trying.

Of course, you won't always succeed in convincing management or your coworkers. But you've gotta try. And if you keep hitting walls, you might actually be better off moving on to another job with other people instead of being frustrated in your current position. Even if you don't agree with what I'm saying in this post, you can't possibly disagree with this: not trying will get you nowhere.

The first time you do something, you just do it. The second time you do something similar, you wince at the duplication, but you do the duplicate thing anyway. The third time you do something similar, you refactor.

That's a quote from Don Roberts, found in Martin Fowler's Refactoring book (page 58). That quote didn't really make me think twice at the time I read it, but it got stuck in my head later on when I heard it from someone who's always been very influential to me. When this person told me that this was one of his favorite rules, I was still in my "no way dude, all code needs to be as clean as it can be" phase. But it's really grown on me over the last few years. Now, just to avoid any confusion: this post is about my interpretation of 'Three Strikes And You Refactor', not about what may have been originally intended or what some people think of it now.

The most important thing I remember when learning about TDD was:

1. Write a failing test

2. Do the simplest thing that could possibly work to make the test pass.

3. Refactor.

With all the literature that's out there, as well as popular opinion on how important clean code is, it's tempting to go overboard with that 'refactor' step. It's tempting to go for the solution that is crispy clean. The solution that'll have your coworkers or peers go "ohhh, now that is nice". However, you really need to ask yourself the following question: "is it really worth it?". Wanting to make sure that every piece of code is perfect takes up a lot of time, and I've learned that it certainly isn't always worth it. Besides, it's quite easy to go off on a path of introducing new concepts and abstractions to your code that end up being totally unnecessary. If this increases complexity in parts that don't really matter, it ends up being a huge waste. Not only for the person who wrote it, but for every person who has to read it and comprehend it later on.

'Three Strikes And You Refactor' is a helpful rule of thumb here. Try to keep things as simple as possible and don't be afraid of a little duplication here or there. Once you find yourself (or a teammate) doing the same thing for a third time, it's worth cleaning up because that third time very often means that it wouldn't be the last time either. This way, you only introduce concepts and abstractions that are really needed, and enables you to avoid adding unnecessary cruft to the code.

I've always disagreed with this seemingly popular point of view or opinion that every developer should spend a lot of time outside of his/her working hours blogging or publishing their code on Github or contributing to open source. Those activities can be important but aren't necessarily so and the people doing them aren't inherently more valuable or important than developers who choose not to contribute to open source software or who don't feel the need to make a name for themselves. I also disagree with the notions that developers who are only involved with software development during their 8 working hours a day must be inferior or that they can't be passionate about it or that they'll never be great at it or any of that other crap that you'll sometimes hear people say.

I know some great developers who put their code out there and/or their thoughts by blogging. I also know some bad developers who put their code out there and/or their thoughts by blogging. I know some great developers who aren't involved with software development outside their working hours. I also know some bad developers who aren't involved with software development outside their working hours. There are good and bad developers on both sides of every situation you can come up with.

I think it depends a lot on what people do during their 8 working hours. Can we really say that developers who work 8 hours a day on challenging projects with strong co-workers in teams where knowledge is passed around continuously are missing out on anything if they don't really spend any other time on software development outside of work? That just doesn't make a lot of sense to me. If I look back on my own career so far, I've typically been rather passive outside of working hours in periods where I felt like I was working on cool things, where I was challenged, where I was learning a lot on the job. Conversely, I've always been much more involved with software development outside of work when I was working on things during working hours that I didn't find challenging or interesting or where I wasn't learning anything new. I'm guessing that this holds true for a lot of people, though certainly not all.

Of course, if you're not working on interesting things or continuously learning and improving on the job, it's your own responsibility to make sure that your skills and insights stay up to date. You don't have to put in the effort to do that, but if you don't, you don't really have a reason to complain either when you're not happy with the kind of work that you're doing. Your employer is not responsible for your career and your future, you are.

And if you are working on interesting things and continuously learning and improving on the job, you don't need to pressure yourself to do more outside of your working hours because certain people in the community say you should. A lot of people get involved with blogging and open source with the hope that it'll end up leading to more interesting work in the long term. Who knows, you might just be a step ahead of them already.

First of all, let's take a look at an image that Microsoft uses on its MVP page:

"Independent Experts. Real world Answers."

This is indeed what the MVP program should be about. It is however the exact opposite of what I and many others in Belgium have experienced. Let's cover the words in those 2 sentences a bit more in-depth.

Independent

Most of the .NET MVP's in Belgium work for a handful of companies that happen to be big sponsors of every important Microsoft event in Belgium. Now, could it be possible that the best .NET developers in Belgium all work for these handful of companies? I highly doubt it. I know a lot of great developers in Belgium and none of them work for these companies. And I'm quite sure that none of those people are less valuable or knowledgeable than any of those MVP's. In my experience, there is hardly anything independent about the Belgian .NET MVP's. Most of them got their first MVP award while they were employed by sponsors or 'strategic partners' of Microsoft. Most of them tout the latest and greatest Microsoft technologies, pretty much unquestioned, even if it means contradicting their statements and positions when they were promoting the technology that Microsoft wanted them to promote 3 or 4 years ago. And they're never openly critical of anything Microsoft-related out of fear that it will negatively impact their chances of being renewed as an MVP. They may not work for Microsoft, but they are anything but independent of Microsoft.

Experts

This is a tough one. What makes one an expert? In some fields, people are considered to be experts based on the amount of research they've done. In other fields, people are considered to be experts based on what they've actually done. I don't think I'm going out on a limb here by saying that in the field of software development, what you've done should be more important than what you've researched or read about. In my opinion, an expert in a specific technology should have built at least one, and ideally more than one, non-trivial system using the technology in question and should be able to tell you about mistakes they've made, things you should avoid, and should never use examples that promote bad practices. I rarely hear MVP's talk about things you should avoid. I rarely hear MVP's talk about mistakes they've learned from. I routinely see examples from MVP's that make me cringe. I routinely hear MVP's promote technologies even though they haven't actually built anything impressive with it. I think of a lot of words when I see examples or proposed solutions by MVP's, but the word 'expert' unfortunately isn't one of them. This also applies to the Microsoft Extended Experts Team by the way. I can only question the validity of such a team if it contains people who are listed as experts of a particular technology when I happen to know for a fact that some of those people have never even used the technology they're supposed to be an expert in in a real world project. Let's be honest here: most of the MVP's and the MEET members simply aren't experts. They are people who have a strong interest in a particular (or more) technology and do their best to promote said technology. There's nothing wrong with that at all, but there's a huge difference between that and being an expert in something.

Real world answers

This one cracks me up the most because most of the Belgian MVP's hardly work on real world projects. And when they do, they typically work on "multiple projects" as they like to say it. For anyone who's ever worked on multiple projects at the same time: could you honestly say that your contributions were truly valuable to the end result? Let me put it differently: don't we all know that people generally produce the best results when they don't have to juggle too many things at once and can focus on as few things as possible? You will rarely hear about a Belgian MVP working full time on a non-trivial real-world project. A lot of them get to spend a certain number of paid hours each week on 'community work', which means blogging, writing articles for magazines, creating presentations for events, etc. When they're not working on that, you'll often find them talking on Twitter about pretty much anything but interesting or demanding work. And of course, their employers will occasionally send them to clients for a bit of consulting here or there, or to do a product demo. Unless these guys are putting in 60 to 80 hours a week (which I seriously doubt), I'm having trouble figuring out when exactly they do the kind of interesting work on real world projects that would enable them to provide answers based on real world experience.

It also doesn't help that I've frequently heard rather unflattering stories about these MVP's from people who've worked with them. It's pretty much always the same: they're not invested into the project, they spend too much time 'interacting with the community' and if you let them go near important parts of the code base, bad things usually happen. Now, I'm not the type of person who simply believes everything he hears. In fact, when it comes to matters like these, I generally only believe the stories I hear from people of whom I'm truly sure that they know what they're talking about. And if you've been reading this blog for a while, you'll know I'm rather critical and that I set the bar rather high when it comes to that.

Lastly, my personal working experience with MVP's has been abysmal as well. I once worked for a client where we had to get rid of 2 people because of how bad they performed in real world circumstances. One of them was an MVP at the time, and now works at Microsoft as an Evangelist. The other wasn't an MVP at the time, but has been one for a few years now and is even one of the Regional Directors of this region. A few years later, I had drinks with a manager of the company they both worked for and he bluntly admitted that both of them are useless on real projects but that their MVP status was a huge benefit to the company from a marketing and image perspective. Not exactly the image or branding I'd be looking to put out there, but hey, to each his own.

Why On Earth Did I Write This?

For 2 reasons. I think this whole MVP thing sets a bad example to young and impressionable developers who aren't sure what to focus on to advance their careers. To outsiders, these MVP's sure look like they've got a great thing going and it's easy to see how that could impress people who don't know better. There are already quite a few developers who suck up to these MVP's in the hopes of someday getting to 'that level' as well. I'd much rather see these people put effort into learning about concepts that will last them a long time and building skills that are transferable to multiple technologies instead of sucking up to the wrong people, and trying to network their way to more interesting work because the work that results from that is truly mediocre at best. By all means, try to make a name for yourself by blogging and speaking publicly. But do yourself a favor and stay true to yourself and favor quality over quantity when considering the audience you want to target. Hey, it worked for me

The second reason is because this whole MVP thing is, in my opinion at least, offensive to the many dedicated professionals who give their all at their jobs as they work on real projects that matter, as they try to mentor younger developers in their workplace, as they try to optimize the way they develop software not only for themselves, but for their teammates and their employers or clients. These are your true valuable professionals.

I'm sure I'm gonna get a lot of flak for this post, but honestly, it won't really make any difference to me. The people who'll be offended are people I'd rather not work with anyway. And of course, I feel quite confident that I'll rarely, if ever, have to deal with these people in a professional setting since I'm primarily interested in interesting and demanding work

I hope that this newfound appreciation for Open Source within Microsoft will lead to another huge improvement in collaborative development in the open source .NET community. While Microsoft is now open to accepting contributions from the community, it would be a tremendous step forward if Microsoft would also contribute to other prominent open source .NET projects in the future. In the past, we've seen numerous open source .NET projects become popular and widely used. And unfortunately, Microsoft responded to some of those projects by producing their own libraries and frameworks that basically do the same thing. Except that, for most of those projects, they never quite matched the quality of the open source projects they were inspired by. If only all of that effort spent on duplicating already existing libraries would've been spent on improving what was already there, the entire .NET community would've been better for it.

I'd love to see a Microsoft that works with open source developers and encourages them, instead of trying to duplicate their efforts whenever they feel they need to provide their own library or framework for something that's already covered by a superior open source alternative. These duplicated projects only alienate people that at one point were passionate enough about the .NET platform to work on improving it for free, in their spare time. These are the people that Microsoft needs to cherish and nourish instead of competing with them. Microsoft has shown some interesting signs of better understanding of open source development and collaboration in the past year or so. Here's to hoping they take that critical next step as well.

Let's talk about the overhead on development first. If you're using WCF services in your web app, you need proxies to access those services. Some people prefer to generate the proxies based on the WSDL of the services that will be used. In the worst case, this leads to regenerating proxies and all of the types that are defined in the WSDL every time you change a service contract or one of the types that are used by the services. If multiple people need to make changes to any of these concurrently, this easily leads to merging problems when people need to commit their changes. Another way is to share the same types on both sides (client & server), and implement your service proxies by inheriting from ClientBase and manually keeping the implementation of the proxies up to date with the definitions of their service contracts. This is better than regenerating a bunch of code all the time, but you're still writing a lot of redirection code for the purpose of, well, what exactly? Another possibility is to use dynamic proxies which automatically implement the service contracts but this increases the amount of infrastructure code you need to put in place and it's not always clear to everyone how exactly communication with the services happens. There's also a lot of WCF configuration for each service that you need to maintain, and it can quickly grow unwieldy.

Then there's the overhead on performance. I hope we can all agree that any operation that goes out of process is at least an order of magnitude slower than a similar operation that can be executed in process. First of all, there's the networking overhead (even if your services are hosted on the same machine as the web app) that you have to keep into account. Secondly, there is the cost of serializing and deserializing everything that is transferred between the client and the server. Even with the most efficient bindings and serializers, the cost of all of this quickly adds up on high-traffic web apps. That's not to say that WCF services are inherently slow. They can be very fast and efficient, but they'll never be as fast and efficient as executing that logic in process within the web app.

Finally, there's the extra overhead it introduces to the deployment phase:

• more endpoints to set up and transfer artifacts to
• more configuration
• more monitoring of endpoints
• more servers if you're not hosting the services and the web app on the same machine

Of course, people will argue that there a plenty of benefits to using a WCF service layer in a web app. The ones I hear about most often are the forced separation of business logic and UI logic and improved scalability and reliability. I really disagree that you need a physical separation of business and UI logic. I much prefer approaches where the separation is based on abstractions. A good example was recently posted by Ayende (here and here). And when it comes to scalability/reliability, a web app that isn't dependent on a WCF service layer is as easy (or even easier depending on your setup) to scale than one that is entirely dependent on WCF services. First of all, if you care about scalability/reliability your web app should already be prepared to run behind a load balancer. If you already have a load balancer in place, you can just add more web servers to your setup when needed. If you'd host the WCF services on the same machines that are hosting the web front end, you'd get less total throughput from one server than you would if that one server could just host a web app that fully executes in process (not including the database obviously). If you're hosting the WCF services on separate machines, you'd end up with more servers to handle the load and to achieve the reliability you need than you would with just being able to add more web servers to your setup. That also increases your licensing costs. And of course, it also means increased networking overhead on every service call, which also implies that the threads on your web servers will be blocked for longer periods while they wait for those service calls to return. Unless you're calling those services asynchronously, but most people simply don't. Also, if you have serious scalability and reliability requirements you're probably better off with asynchronous messaging solutions than with SOAP services.

WCF has its benefits (though I prefer Web API's or asynchronous messaging over SOAP services these days) and it has its use cases. I just don't think internal service layers for web apps is one of them.

What are the benefits that you think an internal WCF service layer brings to your web app? And what's your opinion on how they stack up versus the downsides?

I like to believe learning should be a hands-on activity as well. Basically, stop consuming, start producing. Don't get me wrong, I do think there is value in reading blog posts (I might be slightly biased on this one), reading books and watching videos, but I find that this value is marginal compared to what you gain by actually doing it.

Hands-on activity (producing) is certainly a very important part of any learning process, but I wouldn't go as far saying that the value of reading books/blogs (consuming) is marginal compared to that of producing. In fact, I believe their value to be pretty equal. I've seen too many people who start producing simple things, and then think they've got a pretty good grasp of the technology they're using and then move on to producing more complex or bigger things without actually knowing enough of the technology they're using to support the more complex or bigger scenarios. The results certainly aren't always pretty and I'm sure each and every one of you has seen this scenario unfold with at least one developer you know. Probably more than that even.

I think in a lot of cases, people start the producing phase perhaps a bit too early and then in their enthusiasm of seeing things working sort of skip the more boring consuming that could've benefitted them a lot. Once you've started producing, you need to keep consuming regularly. A tremendously valuable part of any learning experience is getting feedback and insight from minds that have more experience with a given subject than yours. If you're lucky, you can get this from your coworkers. If you're not that lucky, you'll need to find other sources and books, blogs, videos, user group meetings, etc can be a great way to fill that void. And even if you do get to learn a lot from your coworkers, it never hurts to learn more from the experiences of others outside of your immediate circle, if only because their situations and constraints will differ from yours as well.

The other very important part about consuming is really getting to know the technology you're trying to learn. I've always found it very important to at least get an idea of how things work internally within a technology that I'm using. You certainly don't need to know all of the implementation details but just having an idea of it can really help you avoid a lot of problems once you need to use a technology in a more advanced way than in your initial experiments. Most importantly, it should give you better insights as to whether you're using the technology properly, which unfortunately isn't always the same as getting something working. And as a bonus, you'll probably learn about features you won't immediately need but knowing that they're there can save quite a bit of time and effort later on. Just imagine the improvement of the signal-to-noise ratios that you'd see on mailinglists, forums, and StackOverflow if everyone took the time to get a better grasp of the technologies they're using.

When I start with learning new libraries or frameworks, I usually start off by reading most (and often, all) of the official documentation of the technology before I even get into building something myself. If I want to learn a new programming language I'll look for the most recommended books for that language and buy one (or more, if I wasn't satisfied with the first one). I won't even start using the language until I've gone through the book. Once I feel like I've got a pretty good theoretical grasp of the technology, I start building something with it. I also start looking for good blogs on the technology and subscribe to them. I'll also start following influential people of the technology on Twitter. And I just continuously try to soak up as much knowledge as I can from people who're doing more impressive things with the technology than I am. At first, you might not understand everything they're talking about but after a while, things just start clicking and you're getting a really good grasp of things. None of this is a substitute for learning from producing, but it certainly is an incredible addition to it. And one that makes a world of difference, IMO.

Simplicity/Complexity of the application

A common mistake is to decide to use a fashionable architectural style in a project that doesn't really have enough complexity to warrant the use of said architectural style. For instance, there's no reason whatsoever to go the DDD route if your application is mostly a forms-over-data CRUD app. You won't really get any benefit from it, and you'll only increase the total cost of the required development effort. On the other hand, an application which will feature a lot of behavior in a complex domain will certainly benefit from using a DDD approach versus a data-driven approach that implements the behavior in a more procedural, transaction-script based approach. Be honest to your customer/employer and yourself and make an objective decision based on the simplicity or the complexity of the required solution that you're being paid to build. Don't let enthusiasm for a fashionable architectural style influence your decision.

Expected lifetime of the application

Some applications live for years and years, and some are only temporary solutions that will be phased out a year or two after having been put in production. Of course, some of those temporary solutions end up being used much longer than what was originally expected due to a variety of reasons but that certainly isn't always a given. Unless you're familiar enough with how things typically go at your client or your company to predict these things, your best bet is to just go with the initially expected lifetime of the application. Does a temporary solution require the best possible architecture out there, and the resulting impact that would have on the cost to build it? I'd say it most often doesn't. In these cases, there's nothing wrong with going for the approach that is simply good enough for what is required. Striving for perfection is expensive, and generally not worth it for short-term solutions. Conversely, if the solution is expected to last a long time, a solid architecture becomes more important and is worth the extra investment to (try to) get it right if that makes it possible to keep the costs of long-term development under control.

Strategic importance of the application

Some applications are meant to improve the core business of a company, while others play a more supportive role in something that isn't actually part of the company's core business. Some applications generate revenue (directly or by improving efficiency of revenue-generating activities) while others are meant to reduce costs, typically administrative in nature. Cost reduction is important, but it's not quite as important as revenue generation and that's a factor that should be taken under consideration. A strategically important application almost always warrants putting in the effort to come up with a good architecture because it's highly likely that the application will have to evolve in whichever direction the business evolves. That's not to say that the same kind of effort wouldn't be important for cost-reducing applications. But it just might be less important than you'd like to think it is. Again, good architecture increases the cost of the project, and the return on that investment in the big picture of the company should not be ignored.

Skill-level/discipline of team

Good code and good architecture requires skill and discipline. A strong team is capable of letting good architecture grow organically and keep it at a high quality level. For a mediocre team, letting it grow organically is too often a recipe for disaster. Ideally, we'd always be working with strong teams but as we all know, that simply isn't always possible. For mediocre teams, it often makes sense to put architecture in place that is more restrictive in nature and where everyone knows how the code should be structured in advance. The downside is that this typically reduces flexibility and individual creativity, and often introduces more ceremony and indirection than what many of us would ideally like to see. But it's likely that those downsides are offset by having everyone on the same page and taking some possibly difficult decisions away from people who might not be strong enough to make the right decisions. I know that sounds harsh, but that's a reality that many of us have to deal with.

Conformity/Continuity

A lot of IT departments or software development companies prefer to use the same architecture and frameworks/libraries for most (or even all) of their projects because it makes it easier to have people work on multiple projects. In this situation, it's easier to move people between projects or have them do maintenance work on a project they weren't initially involved in. While this often prevents going with the most suitable approach for every project, it does introduce a few benefits that are hard to argue with. People will need less time to get familiar with a code base. In-house training and sharing of knowledge get easier. Bringing in new people (especially for temporary assignments) also gets somewhat easier because you have a baseline of required skills/knowledge that should go a long way within the organization. A huge downside however, is that it creates an environment that will quickly frustrate creative developers. Also, when new strong developers are brought in, they could quickly tune out once they realize they got there too 'late' to have any influence on how projects are developed. The interesting part about this driver is that it is influenced by the skill-level-discipline driver, while simultaneously influencing it towards the future.

Non-functional requirements

This is the last driver on the list, but it is certainly not the least important. Obviously, architecture is greatly influenced by some important non-functional requirements. There are many factors that can have a profound influence on what the most suitable architecture of a system could be, and it's important to think about those as quickly as possible. Do you need to respond in real time? Do you need to support mobile devices? Will there be connectivity issues? Will you need to scale massively? What are the auditing requirements? Are you dependent on third-party services? Do you need to minimize resource consumption? What kind of accessibility do you have to keep into account? The list goes on and on and many of these issues can end up becoming huge problems if you don't think about them ahead of time. Of course, you can't think of everything in advance, and many non-functional requirements can be introduced during the lifetime of the application instead of being known in advance. But the more you know in advance, the better you can prepare your software for them from the beginning.

Conclusion

There is a virtually endless variety of architectural choices and styles that can influence how you develop software. And unfortunately, there's a lot of dogma surrounding it as well. I hope this post made it clear that there is no definitive 'right' or 'wrong' when it comes to architecture, and that (as pretty much everything else in this business) it all really depends

I'm sure you can think of more architectural drivers, and I'd love to hear about any that you think I've forgotten.

Of course, it really depends on the architecture of your system. If you're using services that are only used by the front-end of your system, then I'd still advise against making those entities available outside of the service boundary for reasons that I've discussed earlier. That doesn't mean that you should go the entity-mimicking-DTO-route though. In fact, entity-mimicking-DTO's introduce a few of the same downsides you'd get from exposing entities directly through your services. For an 'internal' service (i.e. only used by your application), I think it makes a lot more sense to use DTO's per service operation which are optimized for the use case that that service operation is meant to implement.

Essentially, an internal service's operations will be either queries or commands. When it comes to queries, why not just do the simplest thing possible? In most cases, it's sufficient to just return the data in the exact same form that the data will be displayed in. Quite often, that means a denormalized set of data where the data comes from more than one type of entity/table. There's no reason to send a bunch of entity-mimicking DTO's that reference each other to the client if you're going to use the data to populate a grid. Just send a list of DTO's which are already in the most optimal form. Populating those DTO's can then simply be done through straight SQL, a view, a stored procedure, a projection through your ORM, a map/reduce operation, or whatever else that makes sense. The point is that in most cases, you should just populate those DTO's as directly as you can instead of mapping to those DTO's. In this case, the DTO's offer a clear-cut benefit and don't really introduce tedious ceremony code in your codebase.

As for the commands (inserts, updates, anything that creates something or results in something happening), why even use DTO's in the first place? People will often argue that the ability to reuse the entity-mimicking-DTO's for these operations is a benefit. Personally, I don't really see the benefit. The mere presence of entity-mimicking-DTO's only encourages people to use them for the queries as well. Instead, I prefer to go with types that encapsulate all of the data relevant to the current command (the data to be inserted/updated, or whatever else the command needs). In a very simple scenario, this could be an InsertCustomerCommand type which simply has properties for the data that needs to be inserted. Nobody will be confusing these types with any other purpose than what their name communicates.

If you're building a web app where the server-side code does everything in process (i.e. without a WCF service), then you can in many cases just use the entities directly without a real drawback, though I'd recommend keeping an eye on unexpected select N+1 problems, since those will frequently come up when you're preparing your views. But even in this case, using DTO's that are optimized for the scenarios in which they're being used can really simplify the query-side of your system a lot.

There's nothing inherently wrong with DTO's if you use them to simply transfer data. If you're using them to transfer entities, you're robbing yourself of their biggest benefit while only making things more complex than they need to be.

Of course, not everyone remembers to set those lengths, so I thought it'd be great if I could force people to do this. With a little creative use of Fluent NHibernate's conventions, it's quite easy to enforce this:

public class StringsMustHaveLengthConvention: IPropertyConvention, IPropertyConventionAcceptance
{
    public void Apply(IPropertyInstance instance)
    {
        var msg = string.Format("The string property '{0}' of type '{1}' does not have a length value specified, " +
            "which is required for performance reasons. Add something like this to your mapping override:\r\n" + 
            "\tmapping.Map(e => e.{0}).Length(50); // with an appropriate length for this property",
            instance.Property.Name, instance.EntityType.Name);

        throw new MappingException(msg);
    }

    public void Accept(IAcceptanceCriteria<IPropertyInspector> criteria)
    {
        criteria.Expect(x => x.Type == typeof(string)).Expect(x => x.Length == 0);
    }
}

With that convention in place, you won't even be able to run your code until you've specified the string lengths

The biggest issue with storing passwords is that you have to assume that it's always possible that someone can get access to your database. Yes, even if it's not directly exposed to the outside world, which it never should be. Whatever security measures you've put in place to protect your database, it's a good idea to assume that sooner or later, someone will be able to punch a hole through your security measures and be able to read the data. So obviously, you really don't want to store clear-text passwords. You also don't want to store encrypted passwords because encrypted data can always be decrypted. And if people get access to those encrypted passwords even if they weren't supposed to, it'd be wise to assume that they also know how to decrypt them, or that it won't take them long to figure it out.

A much better approach is to store a hashed representation of the password instead, using a strong one-way cryptographic algorithm and a unique salt value per password. If the cryptographic algorithm is one-way, it means you can't apply another algorithm to get the original source value again. The only way to compare passwords is to apply the cryptographic algorithm on a given password using the originally used salt value, and then compare the resulting hash with the one you've stored. If they are identical, the given password is the same as the one that was used originally. If they differ, the password is invalid.

Attackers can still employ rainbow tables to try to find password values that generate the same hashes as the ones in your database. Luckily, generating rainbow tables takes time and plenty of space as well so it makes it much harder for attackers to find the passwords. This is why it's so important to use a unique salt value per password. It effectively means that a rainbow table would have to be generated for every single salt value that you've used, making it practically infeasible to find the original password values.

Let's demonstrate this with a simple example. The example is from a Node.js application, but this technique can be applied with whatever technology stack you're using.

This is my User model:

var mongoose = require('mongoose'),
    crypto = require('crypto'),
    uuid = require('node-uuid'),
    Schema = mongoose.Schema,
    ObjectId = Schema.ObjectId;

var userSchema = new Schema({
    name: { type: String, required: true, unique: true },
    email: { type: String, required: true },
    salt: { type: String, required: true, default: uuid.v1 },
    passwdHash: { type: String, required: true }
});

var hash = function(passwd, salt) {
    return crypto.createHmac('sha256', salt).update(passwd).digest('hex');
};

userSchema.methods.setPassword = function(passwordString) {
    this.passwdHash = hash(passwordString, this.salt);
};

userSchema.methods.isValidPassword = function(passwordString) {
    return this.passwdHash === hash(passwordString, this.salt);
};

mongoose.model('User', userSchema);
module.exports = mongoose.model('User');

Notice that the salt property of my User type has its default value set to 'uuid.v1'. In this case, uuid.v1 is a function which will be invoked by Mongoose whenever a new User instance is created. Every User instance will thus have a UUID value stored in its salt property. You can also see that I'm not storing the given passwordString in the setPassword function, but that I calculate the hash value based on the passwordString and the UUID salt value.

Suppose I create a user with the following code:

var user = new User({
    name: 'test_user',
    email: 'blah'
});
user.setPassword('test');

user.save(function(err, result) {
    if (err) throw err;
}); 

Its database representation will look like this:

{ 
    "passwdHash" : "b604367796274cf64177eec345532fc6ca66c6f0501906f82bb03f7916265e9d", 
    "name" : "test_user", 
    "email" : "blah", 
    "_id" : ObjectId("4f1dbb2cfa6157b118000001"), 
    "salt" : "304a33f0-45fc-11e1-80d2-43c594a44fa0" 
}

If an attacker would get access to this, he'd have to generate a rainbow table using the salt value, which takes time, and even then he has no guarantee that the rainbow table will actually contain the correct password. Again, this is why it's so important to use a unique salt for every password. Also, you can use whatever value you want as the salt value so if you can determine it based on some other fields or by using a specific formula you don't need to store the actual salt value. It's recommended to use a long salt value though. Theoretically speaking, it's safer if the salt value isn't stored so clearly as I'm doing here, but even with the salt value clearly visible to a possible attacker, it would still be practically infeasible for him to generate all those rainbow tables.

And of course, my actual authentication function is still very simple as well:

var authenticate = function(username, password, callback) {
    User.findOne({ name: username }, function(err, user) {
        if (err) return callback(new Error('User not found'));
        if (user.isValidPassword(password)) return callback(null, user);
        return callback(new Error('Invalid password'));
    });
};

So as you can see, there's nothing hard or complicated about storing credentials in a secure manner. It's quite easy to do so and there are no downsides to doing this.

So as I was browsing through a few possible alternatives for my authentication needs, I started thinking about my thought process when evaluating libraries/frameworks to use. I generally base my decision on the following items, listed in order of importance (to me):

• How well does it work for my scenario? If a library satisfies all other items on this list, that certainly doesn't mean it's an automatic lock. How it works and the impact it has on my code is definitely the most important factor.
• Popularity. I've noticed that I let the number of watchers/forks on sites like Github influence my opinion. If a project has many watchers and many forks, odds are high that there's a relatively large group of happy users as well as people involved with the project. It also increases the odds that the project will be around for a while. Of course, inactive Open Source projects often remain available as well but if nobody's working on it, I'm not exactly tempted to take a dependency on it. Log4net is a notable exception to this, obviously. But when a project has a lot of people interested in it, or better yet, contributing to it, it's a good sign that you'll easily get help if needed, it's only going to get better in the long run and that it might get forked should the original developers stop working on it. As the author of an Open Source project that doesn't have a lot of watchers/forks (Agatha), I'm aware that my point of view on this is rather hypocritical but hey, it is what it is.
• Code quality. I don't have the time to do an in-depth review of the code as I'm sure most of us don't do either. But I do like to glance over the code to get a general feel of the quality of the code. I focus mostly on the clarity of the code and also keep an eye open for sloppiness or downright WTF's. I guess the questions I'm mostly trying to answer when doing this are: "is this code I'd like to try to improve or fix if I need to?" and "how easy would it be to debug this when I need to troubleshoot some non-obvious issues?".
• Location of code and issue tracker. A lot of people will probably take issue with this, but I consider it to be a major plus if the project is on Github. Not just because of my personal preference of Github, but because they truly encourage people to collaborate and contribute to projects and they make it very easy to do so. Also, the site is fast! I cringe when I have to look over issues of projects on Codeplex because it's just terribly slow. And the UI doesn't come close to that of Github either. I've heard that Bitbucket is pretty similar to Github, but I've never even looked for projects there. In any case: I want to be able to download the latest version of the code at any time, or of a particular branch if I need to, as easily as possible. I also prefer an issue tracker which is fast, responsive and easy to search. It doesn't have to be Github, but those 2 requirements are important to me.
• License. If it's GPL, I don't use it. Also, I check whether or not a commercial license needs to be purchased when you want to use the library/framework in production. Pay attention to dual-licensed projects because that Open Source license might not apply to commercial/production use!

I'd love to hear your thoughts on this. Did I miss any important factors? I just quickly put this post together so it's likely that I missed some good ones

First of all, you need a way to start your Node.js process automatically when your system boots, and to shut it down when the system is shut down. This will depend on the type of server you're running on. In my case, it's a Debian server so I just went with the sysv init script from Nico Kaiser. Another popular alternative is the upstart utility, which is already preinstalled if you're using Ubuntu. Once you have a start|stop|restart script in place, you'll want something to monitor the Node.js process to restart it in case it goes down. An easy to use tool for this is monit. Nico Kaiser again has a good example script available for Node.js on Github.

Once you have your sysv init or upstart script in place, as well as monit, your Node.js process can stay running on your server. Of course, you probably have it set to listen to connections on some other port than port 80 because that's what your Apache server is listening on. So now, the only thing you have to do is configure Apache to proxy all requests coming in on port 80 through the URL of your Node.js site to your local Node.js process. You'll first need to install mod_proxy and mod_proxy_http. After that, the configuration to make it work is quite easy:

<VirtualHost 109.74.199.47:80>
    ServerAdmin davy.brion@thatextramile.be
    ServerName thatextramile.be
    ServerAlias www.thatextramile.be

    ProxyRequests off

    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>

    <Location />
        ProxyPass http://localhost:3000/
        ProxyPassReverse http://localhost:3000/
    </Location>
</VirtualHost>

And that's it. Every request coming in at http://thatextramile.be or http://www.thatextramile.be will be forwarded to http://localhost:3000 where Node.js is listening. Note that the ProxyPassReverse is required to make sure that all HTTP response headers will contain the proxied URL instead of the real one (localhost).

If you need the raw throughput that Node.js offers, this solution is far from optimal. Every request that comes in through Apache will cause an Apache thread to wait/block until the response is returned from your Node.js process. This is essentially the same as when hosting PHP or Ruby through Apache, so it's not a problem, but it does take away one of the benefits of using Node.js. Again, this approach only makes sense if you're already using Apache to host other sites and you just want to add a Node.js site with minimal impact to your server.

So of course, I couldn't resist doing the site with Express/Node.js. And I'm pretty happy with the result too: about 130 lines of server-side code that allows me to write the content of the site in Markdown (including caching and automatically updating the cache if the Markdown files are updated) and my preferred solution to displaying feed items. Does a simple site like this call for Express/Node.js? Of course not. But I knew it would be simple and fun, so that's why I used it.

The hardest part was actually writing the content of the site. Apparently, writing content for a company website is a lot harder than churning out a few blog posts occasionally, and I definitely underestimated that. Not because there's a lot of content (there's very little actually), but because you want to avoid ending up with a lot of content. It has to be concise and yet still contain everything you want it to say. After many revisions, it says everything I want it to say, and unlike on this blog I actually managed to keep it short

An extra benefit of doing it with Express/Node.js is that it reminded me how much fun I have working with that stack, so I'm going to get back to work on my breakable toy which I've been neglecting for too long. I suppose this will lead to a few future posts as well

Another great part about being independent is that you're free to do extra things on the side as well. I'm very happy with the responses that I've gotten from the people who've taken my NHibernate course, and I'm going to continue to offer it in 2012. I initially thought it would be something I'd do once or twice, but I was able to do the course 4 times this year: 1 public and 3 private ones. I've got another private one scheduled in February and will organize another public course in 2012, though this time I'm taking care of the whole thing myself instead of working with another company to organize it.

The most important thing I've learned from this first year is that it's important not to try to do too much. The first 6 months of 2011 were way too busy and hectic, and I only spent one full day doing absolutely nothing work-related. I've been taking things much slower since July and I'm spending much more time enjoying other things in life outside of work. For 2012, my biggest priority is pacing myself much better and spreading things out more. I always seem to alternate between months where I'm always working on something, followed by months of doing as little as possible outside of the 40 hours a week I do for my client. Hopefully, I can finally find a maintainable balance between those two extremes in 2012

On the blogging front, 2011 was a different year than the previous ones on this blog. In the previous years, I averaged about 150 posts a year but this year I only published 71 posts. And I'm actually happy about that. In fact, I hope to get that number down to 50 next year. I still like writing posts, but I no longer feel the need to like I did in the past while I was trying to build a bit of an audience. That often meant pushing myself to keep publishing new posts. Nowadays, I'll only write a post if I really feel like it, which in the long run is much more sustainable anyway. As far as topics covered this year, things have changed quite a bit as well. Some of my views, opinions and technical preferences have changed this year, which naturally reflects in the content I've published. I'm sure I've lost some readers this year because of it, but I also seem to have gained a lot more readers than I've lost. I don't know yet what I'll be writing about in 2012, but I will continue to push people to learn new things (or old things that people seem to forget about) and do better like I always have on this blog, in my own unique way of course

I hope you all enjoyed Christmas, and I wish you all a happy 2012. If it does turn out to be our last year on this planet, we might as well make it count.

• It needs to be fast
• The fewer requests that are impacted by retrieving the feed data, the better
• If I publish a post, the links on the company website should contain the new link within 30 minutes
• The simpler the solution, the better

I came up with a very simple solution, which satisfies these requirements better than any other solution I could think of, or heard of from other people. It is extremely fast, doesn't delay any requests, and doesn't require me to deploy anything but the company website. I'm building the site with Express on Node.js, which means I can take full advantage of the asynchronous nature of Node.js to implement this.

Let's go over the code... in the script that starts the express server, I have the following code:

var express = require('express'),
    app = module.exports = express.createServer(),
    NodePie = require('nodepie'),
    request = require('request'),
    recentFeedItems = null;

app.dynamicHelpers({
    getRecentFeedItems: function() {
        return recentFeedItems;
    }
});

// ... some extra configuration of Express that isn't relevant to this post

var processFeed = function(callback) {
    request('http://feeds.feedburner.com/davybrion', function(err, response, body) {
        if (!err && response.statusCode == 200) {
            var feed = new NodePie(body);
            feed.init();
            recentFeedItems = feed.getItems(0, 5);
            if (callback) callback();
        };
    }); 
};

setInterval(processFeed, 1800000); // process feed items every 30 minutes

processFeed(function() {
    app.listen(3000);
    console.log('Express started on port 3000');    
});

I'll discuss the code in just a moment, but first I want to show the view code that renders the links:

<ul>
<% getRecentFeedItems.forEach(function(item) { %>
    <li><time class="date"><%= item.getDate().getDate() + '/' + (item.getDate().getMonth() + 1) %></time><a href="<%= item.getPermalink() %>"><%= item.getTitle() %></a></li>
<% }); %>
</ul>

And that's all. This is the solution in its entirety!

If you're new to Node, this code probably requires some explanation. Let's start with this part:

app.dynamicHelpers({
    getRecentFeedItems: function() {
        return recentFeedItems;
    }
});

Here I'm adding a dynamic helper to the Express application. It basically means that my views have access to the getRecentFeedItems function, which returns the value of the recentFeedItems variable. It's important to know that the getRecentFeedItems function creates a closure on the recentFeedItems variable created above it. That means that if the value of the recentFeedItems variable changes at any point in time, the getRecentFeedItems function will return that new value.

var processFeed = function(callback) {
    request('http://feeds.feedburner.com/davybrion', function(err, response, body) {
        if (!err && response.statusCode == 200) {
            var feed = new NodePie(body);
            feed.init();
            recentFeedItems = feed.getItems(0, 5);
            if (callback) callback();
        };
    }); 
};

This just creates a function that we can use later on. It retrieves the feed asynchronously, and when the result is retrieved, we parse the feed using the NodePie library and we get the 5 most recent items which we store in the recentFeedItems variable. Again, this creates a closure on the recentFeedItems variable which means that every time we assign a value to this variable, any subsequent call to the getRecentFeedItems function will return the value we just assigned to it because both functions point to the same memory thanks to the magic of closures. Finally, if a callback is provided as a parameter, the callback will be invoked.

setInterval(processFeed, 1800000); // process feed items every 30 minutes

processFeed(function() {
    app.listen(3000);
    console.log('Express started on port 3000');    
});

The call to setInterval makes sure that the processFeed function is called every 30 minutes. After that, we call the processFeed function manually, and we pass in a callback where we start the Express server. This guarantees that the feed items will be in memory before the server starts processing requests.

What makes this solution so great is that we take full advantage of some of Node's benefits. Whenever we retrieve the RSS feed, Node.JS will retrieve that data asynchronously. As soon as it has fired the request to get the RSS feed, it just goes to the next event in its eventloop so no request is kept waiting while we wait for the data to be downloaded. Until the data from the RSS feed is returned, each request will just use the items that are stored in the recentFeedItems variable. Once the data has been returned, our callback is executed which overwrites the value of the recentFeedItems variable. We don't need to do any locking here because the Node.JS eventloop is single-threaded: while our callback is running, no other code that has access to the recentFeedItems variable can be executed anyway. And the actual parsing of the RSS feed is done by NodePie, which uses expat behind the scenes, which is supposedly the fastest C XML parser available.

Looking back on my initial requirements, I think this solution matches very well.

Evented/Asynchronous I/O

In most technology stacks, API calls for I/O operations are synchronous. As in, the thread that executes the operation is blocked for the duration of the I/O operation until that operation has completed. Once completed, execution of your code proceeds. Of course, a lot of technology stacks have asynchronous variants of those operations available as well, but generally speaking, they aren't used as often as the synchronous variants. In Node.js, it's the other way around. All I/O operations are asynchronous and there are only a few synchronous implementations available (and you're generally discouraged from using them).

This means that whenever you do an I/O operation (file manipulation, network requests, database operations, etc…), Node.js initiates the I/O operation through a lower-level C/C++ layer which will perform the operation asynchronously. Once the operation has completed, Node.js will execute your callback function that you passed as a parameter to the I/O operation's function call. The important thing here is that while the I/O operation is being executed, Node.js doesn't have to wait for the operation to complete, and is able to focus entirely on processing other events. And those events can be anything: incoming network requests, executing callbacks from other operations that have completed, or invoking whatever function that is assigned to a particular event.

Eventloop

The Node.js eventloop is what makes Node.js so interesting and powerful. Node.js basically just keeps reading from an event queue until that queue is empty. As it loops through the events to be processed, it invokes the JavaScript functions that have been assigned for those events. If any of those functions performs an I/O operation, Node.js will initiate the operation and then immediately move to the next event in the event queue. Once the I/O operation has completed, an event will be added to the event queue with a reference to your original callback. Once all preceding events have been processed, Node.js will get to the newly added event and invoke your callback. Because all I/O operations are asynchronous, this enables Node.js to maximize its efficiency as it processes events because it doesn't need to wait for slow I/O operations to complete.

Single-threaded

One thing that people don't always realize is that the Node.js eventloop is single-threaded. This has some nice benefits but there's a huge drawback as well. The biggest benefit is that you don't need to worry about concurrent access to shared state. After all, there is never more than 1 thread executing your JavaScript functions. This means you don't have to write any locking code to protect shared state. The drawback to the single-threaded eventloop is that you need to be careful not to block the event loop. If you're planning on doing heavy synchronous processing in your JavaScript code, you need to realize that no other events can be processed by Node.js until that synchronous block of code has completed. Obviously, since there's only one thread going through the eventloop, any delay you cause in your code can be very costly to overall throughput and performance. For now, it's best to execute synchronous processing routines as a child process, possibly even in a language that is more suitable for this than JavaScript. But it seems that future Node.js versions will provide a more integrated way to deal with this.

Why JavaScript?

JavaScript's support of closures and it treating functions as first class objects means it's ideally suited for the evented programming model that Node.js offers. Many people still think of JavaScript as a joke or a toy language, but it's a lot more powerful than many people think it is. Yes, it certainly has problems as well, but it's definitely worth learning. I do hope that this post has made it clear that there's a lot more to Node.js than simply being server-side JavaScript. What makes Node.js so interesting are the principles that I've tried to explain in this post. Those principles can be implemented with other languages as well, and could be made to work just as great as, or perhaps even greater than Node.js itself. But you'd be hard-pressed to find a language that's so ubiquitous, yet completely devoid of a pre-existing synchronous I/O infrastructure.

Anyways, I hope I succeeded at making it somewhat clear how Node.js works and why it's so different from most other technology stacks.

Of course, I don't want to update those links manually whenever I publish a new post, so they need to be retrieved from my blog's RSS feed, which is published by Feedburner. I was hoping to be able to retrieve only the metadata from the posts (date, title and URL is all I need) because my feed always contains the last 20 posts and its total size is usually above 100KB. I haven't found a way to do that, so getting the information I need has to be retrieved through the full feed. Sure, 100KB isn't much but keep in mind that you need to retrieve it and parse it and that I absolutely want to minimize the time each request takes and that I'd rather not see any visual delays on the page either.

I'm interested in hearing how you would implement this. You have total freedom to pick the technologies you'd like to use and no limits on how you'd use them. My only requirements are these:

• It needs to be fast
• The fewer requests that are impacted by retrieving the feed data, the better
• If I publish a post, the links on the company website should contain the new link within 30 minutes
• The simpler the solution, the better

My solution can be found here.