There is a lot to say about the DLR, IronRuby, and dynamic languages in general, but right now I am mostly interested in exploring the transformative experience the recent exposure to this exciting new (to .NET) technology has touched off for me, and how that same experience may hypothetically play out in the larger community.

I once claimed that NHibernate, like Marijuana, was the gateway drug for .NET developers, a taste of an alternative perspective on software development that would inevitably lead to experimentation with and ultimately adoption of a different set of .NET tools and software development methodologies than they might otherwise have tried. Well, if NHibernate is Marijuana getting you in at the gate, then IronRuby is surely LSD, plunging you down the Rabbit Hole. For the uninitiated, and when taken with an open mind, the Ruby language presents an entirely different dimension of technical consciousness that is both beautiful and strange, and comes complete with its own culture and thriving community of passionate, intelligent people. An odd people; they call themselves Rubyists, speak of "The Ruby Way", and count among their sages highly enigmatic and brilliant eccentrics.

Metaphorical hyperbole aside, I do believe that Microsoft’s inclusion of a Ruby interpreter as part of its official toolkit (as well as the peculiar space made for dynamic languages in C# 4.0) will tear a hole in the sky big enough for adventurous .NET programmers to climb through and discover a fascinating new universe on the other side. As IronRuby matures over time and becomes capable of running more and more mainstream Ruby frameworks and libraries the boundaries that tie .NET programmers exclusively to the Windows platform will begin to dissolve. For example, rather than choosing between ASP.NET or ASP.NET MVC to front a service oriented .NET system, a .NET development team may be just as likely to choose Rails, or Sinatra, or some other pure Ruby framework via IronRuby, and thus Windows may begin to shift subtly away from requisite platform and towards deployment detail (with a little help from Mono) for certain applications, or application components, particularly web based user interfaces or middleware that rely on HTTP services as a backend rather than the database.

Of course, whether or not the .NET community at large will ultimately enjoy this experience depends quite a bit on the way in which IronRuby gets adopted. If use of IronRuby is relegated to being little more than a fancy way of scripting your C# application or automating routine development tasks, IronRuby’s mind altering, paradigm shifting potential will remain largely unrealized.

What Microsoft will ultimately get out of all this and how they will sell IronRuby is hard to say. On the one hand, significant exposure to the "Ruby Way" will certainly lead otherwise loyal .NET programmers to experiment with mainstream Ruby frameworks and libraries and begin to interact with the communities that support them. That means day trips to the Linux side of the river, and who knows where that will lead. Interbreeding and funny looking quad-colored penguin children probably. On the other hand, a truly first rate Ruby implementation on .NET may entice Rubyists to pack some .NET in their pipes from time to time, particularly if elegant, dynamic, convention based Ruby frameworks begin to emerge for compelling .NET technologies like WPF.

In any case, I myself am thoroughly enjoying this mind bending adventure. Apropos of nothing, for a physical demonstration of what it feels like to discover something like Ruby, or .NET, or SOA or EDA or whatever it may be, try a few 3D Stereo Cross-eye photos. I think the experience you go through when viewing these photos is a perfect, visceral representation of undergoing a technical paradigm shift: taking otherwise well known concepts and by blending them with some new, often subtle change in perspective and WHAM! suddenly realizing amazing new depth in the world along a dimension you never knew existed. Here is one of my favorites. There are hundreds of these on Flickr, but be careful they can be addicting.

Instructions on how to view these photos can be found here.

Whatever your motivations, this is a trivial feature to implement in almost any environment, and ASP.NET MVC is no exception. ASP.NET MVC’s flexible design also makes it possible to implement this kind of proxy in a very reusable, application-developer friendly way, reducing the code required on your controller to a single line. Of course, you could put your proxy in an HTTP (.axd) module and reduce the line count to zero lines of code if you wanted, but you may not want to open up a generic, free for all proxy mechanism to the client, for the same reasons that the cross domain restriction was created in the first place. I stole part of this solution from here. Here’s the code:

    public class ProxyResult : ViewResult
    {
        public ProxyResult(Uri targetUri)
        {
            TargetUri = targetUri;
        }

        public ProxyResult(string baseUrl,string queryString)
        {
            TargetUri = new Uri(baseUrl + "?" + queryString);
        }

        public Uri TargetUri
        {
            get; set;
        }

        public override void ExecuteResult(ControllerContext context)
        {
            WebRequest proxy = WebRequest.Create(TargetUri);
            using (WebResponse proxyResponse = proxy.GetResponse())
            {
                context.HttpContext.Response.ContentType = proxyResponse.ContentType;
                using (Stream proxyResponseStream = proxyResponse.GetResponseStream())
                {
                    StreamHelper.CopyStream(proxyResponseStream, context.HttpContext.Response.OutputStream);
                }
            }
        }
    }

    public static class StreamHelper
    {
        public static void CopyStream(Stream input, Stream output)
        {
            var buffer = new byte[32768];
            while (true)
            {
                int read = input.Read(buffer, 0, buffer.Length);
                if (read < = 0)
                    return;
                output.Write(buffer, 0, read);
            }

        }

    }

To provide proxied access to a specific, known url to your client side code via your controller:

    public class MapController : SecuredController
    {
        private readonly string _wmsServerUrl = ConfigurationManager.AppSettings["GeoServerUrl"] + "/wms";
        private readonly string _wfsServerUrl = ConfigurationManager.AppSettings["GeoServerUrl"] + "/wfs";

        public ActionResult Show()
        {
            return View("Map");
        }

        public ProxyResult WmsProxy()
        {
            return new ProxyResult(_wmsServerUrl, Request.QueryString.ToString());
        }

        public ProxyResult WfsProxy()
        {
            return new ProxyResult(_wfsServerUrl, Request.QueryString.ToString());
        }
    }

And in your client side javascipt code, you simply reference your local web app, and whatever query string would normally be required:

points = new OpenLayers.Layer.WMS(
            "Units",
            "/Map/WmsProxy",
            {
                layers: 'hsi:someprivatelayer',
                styles: '',
                srs: 'EPSG:4326',
                format: 'image/png',
                tiled: 'true',
                tilesOrigin: "143.60260815000004,-43.851764249999995",
                transparent: true
            },
            {
                'opacity': 1,
                'isBaseLayer': false,
                'wrapDateLine': true
            }
        );

The problem? Symmetry. The dummy data used in the test produced identical results with incorrect behavior as it did with correct behavior. 

Here’s the data that was dummied up for the suite of tests:

"SRID=32719;POLYGON((0 0,1 0,1 1,0 1,0 0))"
"SRID=32719;POLYGON((0 1,1 1,1 2,0 2,0 1))"
"SRID=32719;POLYGON((1 1,2 1,2 2,1 2,1 1))"
"SRID=32719;POLYGON((1 0,2 0,2 1,1 1,1 0))"
"SRID=32719;POLYGON((0 0,1 0,1 1,0 1,0 0))"
"SRID=32719;POLYGON((0 1,1 1,1 2,0 2,0 1))"
"SRID=32719;POLYGON((1 1,2 1,2 2,1 2,1 1))"
"SRID=32719;POLYGON((1 0,2 0,2 1,1 1,1 0))"
"SRID=32719;POLYGON((0 0,1 0,1 1,0 1,0 0))"
"SRID=32719;POLYGON((0 1,1 1,1 2,0 2,0 1))"
"SRID=32719;POLYGON((1 1,2 1,2 2,1 2,1 1))"
"SRID=32719;POLYGON((1 0,2 0,2 1,1 1,1 0))"
"SRID=32719;POLYGON((0 0,1 0,1 1,0 1,0 0))"
"SRID=32719;POLYGON((0 1,1 1,1 2,0 2,0 1))"
"SRID=32719;POLYGON((1 1,2 1,2 2,1 2,1 1))"
"SRID=32719;POLYGON((1 0,2 0,2 1,1 1,1 0))"
"SRID=32719;POLYGON((0 0,1 0,1 1,0 1,0 0))"
"SRID=32719;POLYGON((0 1,1 1,1 2,0 2,0 1))"
"SRID=32719;POLYGON((1 1,2 1,2 2,1 2,1 1))"
"SRID=32719;POLYGON((1 0,2 0,2 1,1 1,1 0))"

It isn’t hard to imagine this limited set of three numbers combined in many different ways and coming up with identical results. While it makes for easy test writing, it makes for deceptively ambiguous testing.

In case you run into the same problem, the SpatialProjections.Envelope aggregate function for the PostGIS dialect will produce an envelope where the fourth coordinate contains two Y values, a simple transposition in a string format. A less regular distribution of sample data would have caught this straightaway.

The wonderful thing about this experience, however, was that this bug, which was crippling to the feature we were trying to implement, was easy to find, easy to fix, and within an hour we were back on track. When the patch is applied to the trunk, we can sync back up with the project, and in the mean time we won’t be implementing some awkward work around likely to break in future releases, waiting for some team somewhere to find it as important as we do. Of course, this is one of the hallmarks of Open Source, nothing new – but it is rather pleasant to experience it for yourself.

I don’t think the reason for this phenomenon is any great mystery – as our culture continues to accelerate and radiate increasing quantities of noise (in the form of information and technology driven social interactions), our minds become exponentially busier in order to keep up, yet our lifestyles don’t often permit sufficient time to “decompress”  so we have trouble sleeping. Our minds simply can’t stop twitching when we turn off the lights and close our eyes.

If you have any doubt that the pace of change is accelerating in our society, watch this. Actually, watch it anyway, it is a fascinating presentation. Our natural tendency, when we reflect on our rapidly changing culture, is to judge it as either robbing us of our souls or as the righteous march of human progress. In reality, however, whether or not this change is good or bad is a meaningless question. Nothing can hinder or contribute to the phenomenon by some application of our will, such change is outside the realm of human control.

Even so, just because we can’t alter the onslaught of cultural entropy doesn’t mean we have to be utterly swept away by it, simply accepting as inevitable fact that we’ll be sleeping less and less year after year.

The video I just referenced asks this question at its conclusion: “What does it all mean?” This question, of course, is not a question, but the Question. Christian mythology holds that Man fell from grace when Eve ate from the fruit of knowledge and thus launched all of human history. The fruit of knowledge needed only to transmit to mankind that single Question to have its effect. It isn’t an opposable thumb or the ability to make tools that defines and sets our species apart from the rest of the animal kingdom, it is that Question, the relentless pursuit of which has given birth to the entirety of human culture. Furthermore, our inability to answer that Question accounts for our uniquely human suffering – the widening spiritual hole in our experience of life that our consumerist culture is scrambling harder and harder to fill, by way of technology and mass production. This is perfectly captured in this cartoon:

As a side note, I highly recommend subscribing to the Noise to Signal RSS feed – a desperately needed dose of perspective in the form of very funny cartoons. Anyway, my intention is not to be critical of our culture, but simply to consider it in order to aware of it so as to find a way to live in it without being utterly enslaved by it. I believe this is critical, because while the acceleration of our culture may be inevitable, it is accelerating towards increasingly unsatisfying ways of approaching our thirst for meaning.

My belief is that the answer to the question “What does it all mean?” is the question itself – “it” means to be aware of the utterly un-resolvable mystery that is human experience, and to explore and study this mystery as long as we live. Our culture is fundamentally unsatisfying (from a spiritual perspective) because it is oriented towards blindly trying to find a salve for some illusory spiritual wound, it does not approach the Question directly, and so it won’t ever have a chance at answering it, or of quenching our thirst. Of course as any Zen Buddhist will tell you the Question is a trick question to begin with – it is the metaphysical phrasing of the joke “Who is buried in Grant’s tomb?” – Grant of course. “What does it all mean?” – it means itself. The goal of a Zen student’s practice is to “get the joke” – not intellectually, which is trivial, but viscerally, experientially, which is not so trivial. Anyway, for those of us who are not Zen students, that doesn’t help much. Even so, I urge you to not forget about the mystery – if we lose the context of the Mystery, or the Question, then our accelerating culture will eventually eat us alive or at least wear us down, but as long as we remain aware of the Mystery and the Question, and attend to them both on a regular basis, then any pace of change or mutation of our culture will be both endurable and enjoyable. Of course, this is easier said than done, which is why I’m up at three in the morning philosophizing instead of sleeping peacefully next to my wife

Gone are the days of blissfully sailing across the open sea gleefully screaming "Fire!" one minute and hitting the deck the next as your competitor’s logo-imprinted cannon balls whiz overhead. We must continue the battle,  of course, but the open sea has been replaced by the thin edge of an epic vortex that keeps growing larger, spinning faster. When everyone is always talking about increasing "Business Agility," whether it be through implementing a well-governed SOA or by calling upon the power of Satan, what they are really talking about is finding some way to keep from sliding down the side of the whirlpool and into the gaping maw of a giant squid.

Melodrama and nautical fantasy aside, the operating environment of the modern business is changing. Businesses are becoming increasingly sensitive to their environments as "Business Intelligence" – the buzz word that refers to actually making some use of the torrents of data flowing through our IT systems – begins to permeate our organizations.

Originally Business Intelligence (transactional data processed and presented for analysis) was employed primarily at the executive level to allow decision makers to analyze the past in order to make predictions (and therefore decisions) about the future. The current trend, however, is to push this kind of processed, aggregated and integrated data deeper and deeper into the organization, sometimes right down the the line level worker. This is being called "Operational Business Intelligence." Operational Business Intelligence is not at all strategic – it is intended to provide fast, objective feedback to operational units in order to improve real time or near real time decision making. Well, this kind of business intelligence isn’t your granddaddy’s business intelligence – it’s value is derived directly from its immediacy, its freshness and its narrow relevance. It isn’t a map for the Generals to study while they sip whiskey and play with plastic figurines. Rather, it is the static filled screams of "OH MY GOD THEY’RE COMING THROUGH THE FUCKING WIRE – I NEED AIR COVER NOW!" bellowing from the field lieutenants radio. It does little good to see such data on a report a day or two (or even an hour or two in some cases) after it is generated. This kind of data shouldn’t even be called analytics; it is direct feedback from living business processes and the good old fashioned ETL based nightly batch process that does some pretty math and coarse grained aggregations and takes forever to process just doesn’t cut the mustard. That kind of business intelligence it is too general and too damned old to be useful for making in the moment operational decisions.

Business Activity Monitoring (BAM) to the rescue! The acronym and the term were invented by some highfalutin Gartner analysts awhile back ago and refer to business intelligence systems which aggregate, correlate, and report (usually via live dashboards and other more immediate alert mechanisms) real time or near real time business data as it is generated by the transactional systems that run the business. Additionally, BAM systems will often employ a rules engine to detect actionable scenarios in the incoming event feeds and either initiate an automated response to such scenarios or alert an appropriate human who can take some appropriate action.

As you can probably imagine, the architecture and design of a BAM system must be dramatically different from that of a batch based, ETL powered system. Furthermore, simply populating the data warehouse isn’t enough – BAM systems deal with real time business events and data feeds and therefore must be prepared to detect and act upon (even if simply via some kind of alert or dashboard dashboard alarm) various business conditions. This kind of task is something data warehouses are ill equipped to address.

Rather than processing data at some interval, BAM solutions monitor business events and aggregate, process and act on those events as they are occurring. This style of processing lends itself almost perfectly to an Event Driven Architecture, or EDA. In an EDA distributed systems communicate with each other indirectly by raising "events" whenever some interesting state change occurs in the domain. These events are usually just messages placed on a Message Bus, or ESB, to which any other system also connected to the message bus can subscribe. At any given time, there may be hundreds or thousands or millions of "business events" flowing through an organizations message bus, providing a perfect opportunity for a BAM system to tap in and do its groovy business.

An Event Driven Architecture or Message Bus are not strictly required for a BAM system to work. BizTalk, for example, provides a BAM API which it expects LOB systems (or BizTalk orchestrations) to call into at various stages of processing to directly inform the core BizTalk BAM database of any interesting business activity. Somewhat more robustly, some of the enterprise BAM solutions provide a wide array of adapters that can allow transactional systems to publish events directly to the BAM software. I don’t think it would be fair to call this kind of arrangement an Event Driven Architecture; even though events are being piped directly out of transactional systems and into the BAM system, the event notifications are essentially a private conversation between each transactional system and the BAM system.

Even so, just because you can have BAM without an EDA and an EDA without BAM, the two technologies (or architectures) are perfectly suited for one another. BAM provides a layer of intelligence and analysis over real time business events, and an EDA provides a conduit for transactional systems to publish their business events to the broader enterprise in blissful ignorance of who else may be watching, including the BAM system. An EDA also provides a completely natural way for the BAM system to distribute information or initiate action when it detects the need to do so – it can just raise its own derived events, to which a dashboard, human notification system, etc. can happily subscribe. Put the two together and BAM! You have a closed feedback loop that will enable your operational managers and line level decision makers to successfully keep your Ship of Commerce out of the tentacles of a giant squid, and maybe even sink a few enemy galleons along the way.

Well, all this is just fascinating stuff, but if you aren’t a big enterprisey company with deep pockets, you will be pretty disappointed when you get to the BAM store with your fist full of change and your head full of high hopes. Most of the existing BAM vendors have been swallowed up by the infrastructure giants like IBM, Progress Software (Sonic ESB) and the like. Probably top notch solutions, but I wouldn’t know because I work for an SMB. Their salespeople generally won’t even acknowledge that companies like ours exist. The one free standing BAM vendor I could find, Systar, also prefers to sell to the big fish, or, as their "business development" guy put it, "Tier 1 Companies who pay licensing fees that start at a quarter of a million dollars". I’m not 100% sure what a Tier 1 company is, but based on the fact that he tried to convince me we didn’t really need a BAM solution, I’m guessing we’re not one of them   So, as usual, that leaves us SMB’s alone at the bar sitting next to a Microsoft solution, in this case BizTalk, trying to decide if we should just head home alone and take care of business ourselves, or down a few more shots of Tequila, heave a deep, forlorn sigh of resignation, turn to BizTalk and say "How you doin’?"

I should have more to say about that soon as we’ll be implementing a BAM/EDA solution over the next few months. Will a bag-over-its-head be enough to stomach an ongoing relationship with the BizTalk BAM product, or will we be able to stitch something together from open source tools and good old fashioned grit, piss and vinegar? Only The Shadow knows…

Change is nothing new. In fact, change is all there is. We all attempt to resist it in one way or another, but it is a useless effort, like trying to stand still in a rushing river. There is no such thing as "resisting" change – the best you can do is to pretend it doesn’t exist, but you can’t stand still. Even though you may not alter your behavior, your environment will change around you and your relationship to your environment will change despite your stubbornness, it can’t be helped. Whether change is good or bad is a meaningless question, and the question of whether to accept it or ignore it isn’t really a question of "self improvement" but a question of how much you will enjoy your profession and your life.

So while the call to change coming from ALT.NET is extraordinarily valuable, the relentless emphasis on the tangible business results of some particular change, such as a guaranteed suite of regression rests resulting from a move to TDD, creates a limiting environment in which to accept it meaningfully and realize its full value. The only way to accept change is to experience it with an open mind and feel its value for yourself. An emphasis on quantitative results is absolutely necessary for arousing interest in some new technique or methodology, but it should not be the only emphasis. Once interest is aroused the emphasis should immediately shift from the expected benefits which, at this stage only reinforces preconceptions and inhibits the learning process, to remaining observant, alert and curious about the actual experience and actual results the technique or methodology provide.

TDD is a perfect example because everybody is always talking about its objective benefits, such as a safety net for major refactorings, enabling collective code ownership, cleaner API’s resulting from testable API’s, but the real value of TDD is a complex mix of all of these things and more. You can listen with your jaw agape to  someone extolling its benefits for days at a time but until you actually start to practice it you can’t really have any understanding of it at all. This is true with everything, but with TDD the effect is very dramatic, and therefore very instructive. People are always saying "it’s about design, not testing" but then they go on to describe benefits that are often mostly about testing. This is because the design aspect of TDD is something you will feel when you try it with an open mind, it is extremely difficult to articulate. The results are not so hard to articulate, such as decreased coupling or increased isolation, but an understanding of why these benefits result from TDD is more visceral than intellectual.

People talk about learning curves as if they were impediments to obtaining the bountiful rewards of some shiny new methodology, but the learning curve is exactly what produces the rewards. The learning curve IS the primary value. The moment the learning curve flattens out is the moment you will find a new methodology to replace the old one. Ian said in his post that change results from the pain of a process outweighing our natural resistance to change, but I completely disagree. I believe change is the fundamental and most basic aspect of our professional lives, the very reason we enjoy what we do as much as we do, and that methodologies evolve not at the last possible moment when we can’t stand the agony any longer but at the first possible moment, the very moment we suspect our current practices cannot evolve any further. A work environment where nothing changes, where technologies are frozen at some arbitrary point in time and previously established development practices have taken on the aura of biblical law are most developers worst nightmare.

Yet, as Udi Dahan points out, we often feel we are not at liberty to go off experimenting with change at the expense of pressing business realities. And this is true to a large extent, and I agree with his conclusions – TDD or any other new process should be introduced into an environment responsibly, ensuring an appropriate level of safety for the business as well as the team, or the results could well end in disaster. The danger, though, is to use that as an excuse to avoid accepting change altogether. While it is suicide to ram some new methodology into the gears of a running machine, a more sinister form of suicide is to neglect change entirely because no time ever feels like the right time. Consciously and constantly adapting to change should be built into the foundation of your business culture and your development culture. This is the responsibility of the management and line level developers alike. If your company is not interested in continuously evolving along with the software industry and therefore does not provide a safe (if controlled) environment for experimenting with and adopting new technologies, practices and methodologies then you should find another job. Conversely if a developer on your team is not interested in adapting to change then you should transition them off your team, either out of the company or into a support role.

Finally, all this highlights for me the need to emphasize that business results are not why we come to work every day. You may like to believe that you invest yourself as deeply as you do in your career because you want to contribute to the success of your business, or provide the best possible experience for your users, or streamline business processes, and these things mean something, but you know and I know that those reasons aren’t the real reason you show up at your keyboard. You love your job because it is mysterious and fascinating and full of creative possibilities and interesting logical puzzles which give you immense pleasure to solve and explore. You love to create something out of nothing. Fortunately, this kind of enjoyment usually aligns, or can be made to align, quite well with business objectives. But the point is motivation – yes developers want to be efficient, yes developers want to deliver maximum business value, but even more than that they want to maximize their creative possibilities and they want to enjoy the time they spend in front of their computers. So if you want to sell TDD, Agile, IoC, or whatever it is, don’t forget to sell that aspect of things as well as the objective, tangible efficiency or quality gains.

My team recently attended an Open Space conference, Open Agile 2008, which is a sort of new age, ad-hoc, self organizing meeting of passionate people. An Open Space conference has a number of principals, but the one that really caught my attention was the commandment to "Be Prepared To Be Surprised." What is meant is this: put aside your pre-conceptions and expectations and expect to learn something new at any given moment, and in unexpected ways. This applies equally well to a new technology or methodology such as TDD: "You will come away with full test suites that will dramatically improve the quality of your software, your designs will have lower coupling and a cleaner API and fewer defects. Oh, and be prepared to be surprised."

Well, those days of agony and uncertainty are gone with the introduction of "process-per-tab" browsing found in Chrome. I believe the newest version if IE will also support this model. Anyway, my computer suddenly slowed to a crawl, and task manager showed Chrome as the culprit. But not Chrome in general, just a tab in chrome. Actually, not a tab in chrome either, a window inside a tab in chrome that was hosting a flash animation, which had run amuck. Chrome actually has its own task manger, so the solution to my problem? Right click on the Chrome tool bar, choose task manager, look for excess processor usage, kill said process – and all is well. All 20 tabs breathed a collective sigh of relief, and I un-wet my pants and kissed my monitor. The damage? A portion of one of the web pages I had open showed a broken plugin graphic and a nice little notice at the top of the page said something like "A flash plug-in has crashed." A single tear rolled down my cheek for the poor, lonely flash animation I was forced to kill, but many more lives were saved. Thus is the price of progress.

I’ll be taking a closer look at this over the next few weeks, but my general feeling is that this product will add tremendous value to the community, despite the many existing CMS products already available, and I am very excited to kick the tires.

Project Links

* Main Product Page: http://www.sensenet.hu/engine.aspx

* CodePlex site: http://www.codeplex.com/sensenet

* Documentation Wiki: http://wiki.sensenet.hu/index.php?title=Main_Page

* Discussion Forum: http://forum.sensenet.hu/

Sample Sites

Additionally, they have two case studies already of full blown, production sites built on their new system. The case studies give a brief overview of the projects and link to the live sites. Both of the case study web sites are Hungarian, though, so the experience is a bit awkward (unless you can read Hungarian, of course), but still gives you a flavor for the CMS. [Update] – the Invitel site has an English version here: english.invitel.hu

For me, it was quite a treat. In addition to being a rabid technologist, I am also an armchair philosopher, so this kind of material is food for me. I will have more to say about the enormous question so fleetingly put in Rod Cottingham’s post another time, but today I want to think a little bit about the flurry of emotion regarding TDD, because while the passion its practitioners exhibit for the methodology is in itself compelling I don’t feel they are otherwise expressing its true value very effectively to those who I presume they are speaking, which is to say to me, the layman and the novice.

I myself have only just begun to experiment with Test Driven Development in particular and Agile methodologies in general, along with the rest of my small and equally inexperienced team. Speaking from such a demographic, I can confirm that TDD is a difficult discipline to become comfortable with. Speaking from a project management perspective, I can also confirm that starting from standing still getting up to cruising speed with TDD is, at least superficially, expensive from a productivity standpoint. I believe every member of my team is both talented and passionate about the mystical process of creating quality software, but taking up TDD is nevertheless quite challenging.

Even so, our limited and amateurish experience with TDD so far has deeply impressed all of us, so much so that the productivity cost of the learning curve has never once felt like anything but an investment. To wonder whether or not the cost is worth the benefit is perplexing to me, I can’t imagine having that emotion. Maybe I’m spoiled by the talent and passion of my colleagues, maybe if you find yourself alone amongst a crew of 9 – 5 clock punchers the row is much tougher to hoe, but you don’t have to scratch very deep into the surface of TDD to find its extraordinary value, even though it does require constant discipline (which often falters, by the way) to continue using it, at least at the early stages we are in.

What is this great value that outweighs the effort?

Well, there isn’t any point in my spelling out the tangible, objective benefits. That the TDD monks express quite well, and anyway I’m a layman, so I wouldn’t be able to do it effectively in any case. What I find strange, though, is how TDD seems to always be discussed outside of its Agile context – as if it were a side dish you could have with any kind of meal without it losing any of its overall value. Rather, I see TDD as the chorus to the Agile song – it may be beautiful to sing in isolation, but it grew out of, and belongs to, a larger melody, and to really understand its value an experience and understanding of the larger context is of the utmost importance.

The larger context? Iterative, outside in development, of course. Agile is a highly iterative development methodology that focuses first on the need and then on the implementation in short repetitive cycles. It has the potential to transform software development from a giant thought experiment in which we attempt to construct  tightly fitting, complicated wardrobes of clothes for our customers by asking them to describe the shapes of their bodies in perfect detail and list up front every social function they will ever need to attend, to one in which we take small, precise measurements, create one article of clothing at a time, try it on the customer, have them spin in front the mirror, and make adjustments accordingly: to the patterns, to the materials, to our tools and our processes.

The universe itself is rhythmic, cyclical and defined by nothing if not constant change, and so therefore are our lives, our work and the needs of our businesses. Software, unlike other more traditional engineering disciplines that deal with physical space and universal themes, must keep constant pace with the rhythm and change of human activity for it to be useful. Not only that, but the complex needs software aims to serve cannot simply be measured, like a plot of land or the gorge a bridge must cross, it must intersect usefully with myriad human minds rather than relatively predictable physical material, and the human mind is famous for not knowing itself very well.

So Agile attempts to adjust for this by addressing human need in an interactive way – assess the general outline of the need first, build a little something, see how it fits, make adjustments here and there, fill the general shape of the need, add the detail in layers as it is discovered. It brings to my mind an image of a potters wheel – the clay spins always, an iteration at a time, you shape a vessel first with your fingers, relying as much on tactile feedback as you do your logical understanding. Then you apply detail, little by little, and when it is finished you can fire it in the kiln and paint it, but not before. If you build the detail too soon, it will quickly get rubbed away as the true need emerges, or worse, it will remain because you’ve put the thing in the kiln too soon, but it won’t be appropriate to the piece you’re creating, so you cover it up with fresh clay and fire it again, and you end up with something structurally unsound, or ugly, or altogether useless.

What does all this melodrama have to do with TDD? TDD is your fingers in the clay. Code is clay, after all, not gears and pistons and fittings, it has constraints, of course, but is largely pliable imagination. TDD is what finds an ideal shape for the code that is elegant, appropriate and capable of realizing the emerging and actual (not theoretical) needs of your customers as they are incrementally uncovered by the larger Agile process. TDD feels out, does not deduce via thought experiment, an appropriate API. This has been my takeaway from our limited experiments with the methodology, at any rate. TDD as the smallest possible step, the smallest possible rhythm, red-green-refactor, red-green-refactor, you rely predominantly on a sense of practical aesthetic, an ability of the mind to find a natural shape for things, a process that is usually expressed in terms of “reducing friction” by the community of vocal practitioners. Of course this is a logical process too, of course, not just an artistic one, there are general rules and guidelines, it is a strange coupling of mathematics and logic with imagination and creativity and pseudo-tactile feedback. Your mind is clicking along in a scientific way, but a larger part of the brain is enlisted to participate, a part that, as engineers, we too often discount, more than just the cold eye of logic with too little information at its disposal and too high an opinion of itself. In the end its all very Taoist The Taoist butcher, as you may know, never has to sharpen his knives because he always cuts his meat precisely along the grain – he isn’t cutting at all, actually, but merely separating the material along its natural seams. TDD aims to find those seams in the seemingly solid material of a programming language as it applied to a complex human problem and the result is often of very high quality, both aesthetically and functionally, and the secret is both qualities are the same quality when approached in this way.

Anyway, that was a lot of grandiose hoo ha, and I’d be surprised if any of you left brained technologists have made it this far. Whether or not I’ve made any sense, my main point is that when practitioners are always saying TDD is about design and not about testing, I think this is in essence what they mean. Even if it isn’t, that distinction – the distinction between “design” and “testing” is tragically under-emphasized, or at least under-explained, and that distinction, in my mind, is the crux of the whole matter, is the first thing that should be said about TDD to anyone who is interested in listening. In that context, the question of “is it worth the cost” is almost nonsensical.

If by some miracle you have read this far, I’d love to hear what you think – what is your understanding of TDD? Or, do I simply need to lay off the hookah?

Replace “Toaist” with “Programmer” in that last sentence and you have the essence of our industry

In this post I’m interested in how a set of claims presented by a user when making a service call can be evaluated against specific authorization rules to determine whether or not the user has access to the requested resource.

The details of how a user is associated with such claims, how they are secured, and how claims are embedded inside calls to your service are beyond the scope of this post, but the links listed above should provide a place to put the tip of that chisel. For the purposes of this post I will assume that claims are presented via a list of IClaim objects, and an IClaim is nothing but a Name/Value pair – IList<IClaim>. This is very similar to how Micrisofts new Zermatt Identity Framework represents claims, and this was intentional as I believe that Zermatt is probably the future of Identity Management on the .NET platform.

So then, given IList<IClaim> claims, how can we make rational authorization decisions, such as “I only want users who are in the Premium Members role and whose age is older than 21 to access this (probably seedy) service method.” One obvious, simple approach would be to evaluate the claims programmatically. Some simple extension methods can make this much easier:

public static class ClaimsExtensions
{
   public static T GetClaimValue<t>(this IList<iclaim> claims, string claimType)
   {
       foreach (IClaim claim in claims)
       {
           if (String.Compare(claim.ClaimType, claimType, true) == 0)
           {
               return TypeNormalizer.EnsureType<t>(claim.Value, default(T));
           }
       }
       return default(T);
   }

   public static bool ClaimEqualsAny(this IList<iclaim> claims, string claimType, string claimValue)
   {
       foreach (IClaim claim in claims)
       {
           if (String.Compare(claim.ClaimType, claimType, true) == 0)
           {
               if (claim.Value.Equals(claimValue, StringComparison.InvariantCultureIgnoreCase))
                   return true;
           }
       }
       return false;
   }

   public static bool EvaluateClaim<t>(this IList<iclaim> claims, string claimType, Func<t , bool> eval)
   {
       var claimValue = claims.GetClaimValue</t><t>(claimType);
       return eval(claimValue);
   }
}

Then, from within a method that needs protecting, you can easily do something like this:

public void MyServiceMethod()
{
    if (CurrentUser.Claims.EvaluateClaim<int>(CustomClaimTypes.AgeInYears,age => age < 21) ||
        CurrentUser.Claims.ClaimEqualsAny(CustomClaimTypes.Role,"Premium User") == false)
    {
	//Whatever you do when someone isn't authorized. Throw a not authorized exception? Return a fault code?
    }

    //Implement your service method here, your user is old enough and has paid their money
}

If you were clever, you could probably figure out a way to use LINQ expressions to make the syntax even cleaner. This is all good and well, but it has at least one major problem – authorization rules are probably likely to change far more frequently than your application code. Even worse, you may wish your power users to be able to manage their own authorization rules for certain kinds of access or for named permissions. Using the technique described above, any change to any authorization rule will require you compile and deploy your application.

One common solution to conundrums such as these is to use a DSL (Domain Specific Language). And while you certainly could write a custom DSL to implement your authorization rules, authorization rules have an interesting characteristic: they always resolve to a discrete value, a true or a false. They are simply boolean expressions.

There are a number of Expression Evaluators available from the open source community that will fit the bill perfectly. An Expression Evaluator will allow you to specify your authorization rules as simple strings, which means they can be stored in a configuration file or a database or some other persistent store and modified and run-time as needed. They are also serializable by definition and can easily be requested by and cached at the user interface to allow the user interface to use the same authorization logic  to render itself appropriately with respect to the permissions of the currently logged in user that the server will use to authorize access to its API without (necessarily) having any code in common other than a reference to the same expression evaluator library.

In this example I will be using Simple Expression Evaluator available on Codeplex. I chose this library purely out of nepotism, because I wrote it, but you should be able to use any expression evaluator that catches your fancy. A number of options for .NET are listed here.

Using an Simple Expression Evaluator we could rewrite the above authorization rule like this:

AgeInYears >= 21 and MatchesAny(Role = "Premium User")

In the sample project I added a custom function, IsInRole, allowing this syntax:

AgeInYears >= 21 and IsInRole("Premium User")

To make this happen, Simple Expression Evaluator will need a little help in interpreting claim names. The main reason behind this is that claims are usually represented by a URI, such as http://thefreakparade.com/claims/role, but accessed through code using string constants, such as

public static class CustomClaimTypes
{
     public const string Role = "http://thefreakparade.com/claims/role";
     //...more...
}

and therefore accessed in code like this:

var claim = new Claim(CustomClaimTypes.AgeInYears,43);

The expression evaluator will interperet something like AgeInYears to be a variable, and the name AgeInYears won’t naturally correspond to anything, because it’s really just a property name, the real claim is represented by a URI that would be far too ugly to stick in an expression. The solution is to give the expression evaluator a little help when evaluating variables, like so:

public bool Authorize(IList<iclaim> claims,string authExpression)
{
    var context = new ExpressionContext(claims);

    context.ResolveUnknownVariable +=
        (sender, args) =>
            {
               args.VariableValue =
                 claims.GetClaimValue(_claimAliases[args.VariableName]);

            };

    context.ResolveMissingFunction +=
        (sender, args) =>
            {
                if (args.FunctionName.Equals("IsInRole",StringComparison.OrdinalIgnoreCase))
                {
                    args.Function = (funcArgs) => IsInRole(claims,funcArgs);
                }
            };

    return ExpressionEvaluator.EvaluateExpression(authExpression,context);    

}

This method was modified a little from the actual sample to the purpose of clarity, but the general idea is the same. Also demonstrated is the technique required to add an IsInRole method to the expression language. The actual implementation of the IsInRole functionality is implemented off screen, see the sample for the actual code. This method also relies on a hashtable called _claimAliases being pre-populated with a mapping between the URI of a claim type and the friendly name the expression language will use. In the sample app, I simply used reflection to load the alias table with a mapping between the field name of each const and the actual URI.

Now, if you simple use the expression evaluator an call Authorize from within your service methods, like this:

public void MyServiceMethod()
{
    if (AuthorizationHelper.Authorize(
           CurrentUser.Claims,"AgeInYears >= 21 and IsInRole('Premium User')" == false)
    {
	//Whatever you do when someone isn't authorized. Throw a not authorized exception? Return a fault code?
    }

    //Implement your service method here, your user is old enough and has paid their money
}

Then all we added with the dynamic expression was complexity – your rules are still defined inline in your code. What you really want is a syntax like this:

public void MyServiceMethod()
{
    if (AuthorizationManager.Authorize(CurrentUser.Claims,Permissions.CanAccessServiceMethod) == false)
    {
        //no access
    }
    //service code
}

Where Permissions.CanAccessServiceMethod will key into a persistent authorization rule store such as a config file or database, so that the rule expressions themselves can be dynamically loaded at runtime and therefore modified outside of the code. The sample application takes advantage of a very simplified implementation of the Repository pattern and introduces an IAuthorizationRuleRepository to solve that issue. If you were really, really smart you could get to a syntax like this without much extra effort. Note this sample application does not quite go this far, but it shouldn’t be a stretch for you get there yourself if you care to:

[RequiresPermission(Permissions.CanAccessServiceMethod)]
public void MyServiceMethod()
{
    //service code
}

The only caveat with an attribute based approach is that access to your protected method is all or nothing. You can’t return a set of records filtered one way when one permission level is present, and another for a lower level of permission. But nothing would prevent you from mixing an attribute based approach with an inline-code based approach as appropriate.

Finally, here is a sample of the actual authorization syntax as implemented by the Identity at Rest sample application:

public List<monkey> ListAllMonkeys()
{
    //Only administrators can list all monkeys.
    //Everyone else can just list shaved monkeys.
    if (AuthorizeFor(
            Permissions.MonkeyShavingService.CanViewUnshavedMonkeys) == false)
    {
        return ListMonkeysByStatus(MonkeyStatus.ShavedClean.ToString());
    }

    return _monkeys;
}

And that’s about it. If you’ve made it this far, your very well aware that this post is describing a general approach to claims based authorization using an expression oriented DSL – the code samples listed here aren’t detailed enough to get you from point A to point B in an actual implementation. However, the included sample application, which is an evolution of the Identity At Rest sample application provided for this post, does provide an end to end application that fully implements the concepts discussed here. The sample comes with a “smart client” Winforms application that uses WCF to communicate with a remote  REST based API. All API calls are secured using the above described techniques.

I would be the last one to suggest that this approach is a “best practice” in managing claims based authorization, but it made sense to us and seems to work pretty well so far.  As always, I’d love to hear about any other approaches that you may be using.