Overview:

Ripple is a new kind of package manager that was created out of heavy usage of the standard NuGet client. The feeds, the protocol, and the packages are the same. Ripple just embodies differing opinions and provides a new way of consuming them that is friendlier for continuous integration.

Features:

1. Staying up to date with the latest build

Ripple introduces the concept of “Fixed” vs. “Float” dependencies. For internal dependencies, it’s often beneficial to keep all of your downstream projects built against the very latest of your internal libraries. For the Fubu team, this means that changes to FubuMVC.Core “ripple” into downstream projects and help us find bugs FAST.

2. Command line friendly

Ripple is 100% command line and has no ties into Visual Studio. The usages were designed for how the Fubu team works and for integration with our build server.

3. Generating nuspec files

Keeping your packages up to date with versions can be a challenge when you have a lot of them. Ripple provides the ability to automatically generate version constraints for the dependencies in your nuspec files so that you never get out of sync.

…and lots more.

Getting Started:

Ripple is published both as a Ruby Gem (ripple-cli) and as a NuGet package (Ripple) — which you can use with Chocolatey.

You can read the “Edge” documentation here: http://darthfubumvc.github.io/ripple

Note:  The ripple docs are powered by our brand new “FubuDocs” tooling. Jeremy will likely be writing about that one soon. If he wasn’t planning on it…then I think I just volunteered him for it.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

In it, the proposal includes an “IsStale” flag on the collection, as well as on the query itself, so you could do something like:

GET /api/environments?nonstale=true

Or similar. This presents a rather weird choice to the end user of the API – consistency as a choice, not on mutating operations (PUT/POST/DELETE) but on idempotent GET operations. I presume that this will use the “WaitForNonStaleResults” behavior of RavenDB – but this isn’t really something I’d expect to directly expose to clients.

Without directly exposing our persistence mechanism to our clients (i.e., what if we switched to SQL Server? Redis as a write-through cache to MondoDB? and so on), we have a number of options of dealing with eventual consistency in our REST APIs.

Option 1: Do nothing

The easiest approach for our API is to simply not care. Non-stale results should only really appear when we’re dealing with queries and collections – if you’re dealing with stale resources from GET actions directly against a resource, you’ve really got a weird interaction model.

Looking at some other APIs, like Netflix or GitHub or Trello, you don’t really see any option for influencing the consistency choice on a read.

So we could just ignore it. This is what a lot of APIs do, accept the POST/PUT/DELETE, and make sure that my resource itself is affected correctly. If I do a DELETE /users/jbogard and then a GET /users/jbogard, I expect a 404. But if I query users or search them, then I might not expect those results to be reflected in that model. I can be explicit in this by having search be a completely separate set of resources/entry point (i.e. /search?entity=users&name=jbogard), so it’s completely explicit that search/query is different than interacting with resources.

This is similar to how a library with a card catalog might work. Do I synchronize the activity of checking out a book with checking the catalog? Or might someone have grabbed the book from when I checked the card catalog? Or do I have someone hold the card while checking out a book?

What I don’t do is allow a person looking for a book to either be disappointed OR have the choice of yelling out to everyone in the library, “DOES ANYONE HAVE THIS BOOK OR IS OTHERWISE WANTING TO CHECK THIS BOOK OUT”. I fix my interaction model and just be up front about it.

Option 2: Provide feedback on consistency results

Instead of just punting on whether or not the collection/query is stale, we might provide that as feedback. We could return dates of last update, things like “results as of mm/dd/yyyy”. We often do this on printable reports so that when people print a report (and it is now preeeeeetty much as stale as it gets), we can include a timestamp of when it was printed so that anyone looking at it is informed of how fresh the data is.

Just a tidbit of information to the end user to let them know if their results are up to date or not. If the client made a mutating operation, they can compare the date of this mutation with the date on the query results to make a simple decision to query again, or just move on as planned. Again, the power is in the client not to affect how we query, but what decisions they can make.

Option 3: Return a 202 ACCEPTED status

If we really want to model an asynchronous operation in our REST API, we can look to the HTTP status codes to communicate this explicitly to our clients. We do this in a couple of places where processing is too expensive in the context of a request, so we return a 202 to let clients know that “we got your request, but it’s not getting processed at this moment.” The description from the W3C reads:

The request has been accepted for processing, but the processing has not been completed. The request might or might not eventually be acted upon, as it might be disallowed when processing actually takes place. There is no facility for re-sending a status code from an asynchronous operation such as this.

The 202 response is intentionally non-committal. Its purpose is to allow a server to accept a request for some other process (perhaps a batch-oriented process that is only run once per day) without requiring that the user agent’s connection to the server persist until the process is completed. The entity returned with this response SHOULD include an indication of the request’s current status and either a pointer to a status monitor or some estimate of when the user can expect the request to be fulfilled.

We can return a pointer to an indication of the current status, or some estimate of completion. But we’re being 100% up front that your request is processed asynchronously.

This is similar to any long-running transaction. You place an order at a fast-food restaurant, and are given a correlation identifier that represents your order. You can come back and ask the status of your order at any time. But the cashier certainly doesn’t cook our order while we’re standing in line!

Option 4: Write-through cache

If these operations are expected to succeed (or we verify that the transactional write has succeeded), we can do a simple trick that a lot of high-volume websites do. If we don’t have an AP-system like Dynamo (choosing availability and partitionability over consistency), we might choose consistency instead. To do so, we can cache index results, and write our updated value to that store along with our regular persistent store.

It’s not the most exciting of options, but if we know that writes are much less frequent than reads (and we’re not partitioning, i.e. we pick CA of CAP), then it’s not too far-fetched to write to both our cache and our document store.

Of course, if this is all to force us to bypass our consistency model of the database we’ve chosen, it’s a lot of work. But still, it’s an option nonetheless.

Option 5: Choose a database that matches our consistency needs

If we don’t like the consistency model that our database provides, or we feel like we want to allow clients to choose a consistency model, we might view this as a case where we’ve simply chosen the wrong model. If clients NEED consistency, why not pick something that gives that to them? Or don’t allow them to choose.

This would be like, on writes, allowing clients that interact with a database that uses a relational model to also indicate the isolation level on writes. That’s something that should really be encapsulated by the operation, and made with SLAs and contracts about the behavior.

Conclusion

We have a lot of options here, and all are valid in some scenarios. In each example, we’ve chosen a consistency model that matches our needs, rather than have a compromised consistency model exposed from our database of choice. Before Amazon put out their Dynamo paper, it’s not like us as users of the website knew that this storage system existed in the back end. It was encapsulated. The most we could do was “Ctrl+F5” to force a request back to their servers, but that still had no real guarantees.

Instead of letting our database leak its consistency model to our API, it’s better to choose a model that makes this consistency model explicit, or offer no guarantees at all. But as a rule, I as a consumer should not care that you’ve chosen Raven DB instead of SQL Server for your back end. It’s just none of my business.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

• Observer pattern
• Controller pattern
• Pattern variations
• Scaling sagas
• Routing slips

NServiceBus sagas are a simple yet flexible tool to achieve a variety of end goals. Whether it’s orchestration, choreography, business activity monitoring, or just other long-running business transaction variants, the uses of an NServiceBus saga are pretty much endless.

When choosing to go with a centralized workflow, we also saw that there is a cost to centralization with the introduction of a bottleneck. With the routing slip pattern, we can include instructions with our message in a header so that each recipient only needs to reference the attached instructions. In a routing slip, flow is linear, but there’s nothing stopping us from including more advanced instructions for state machines, compensations and so on.

I tend to think of the NServiceBus saga as more of a facility, than a specific pattern, because it doesn’t force us to go in any one direction. Rather than assume a specific role or function for a saga, I like to keep things a bit more flexible, and choose one of the many conversation/messaging patterns available for each given scenario.

In the end, sagas are a useful tool (and one that can be over-used, not every workflow deserves central management), but a nice one to have. Every time I introduce NServiceBus sagas to folks that spent time with other messaging tools, whether it’s big orchestration with ESBs or bare-metal messaging tools, the simplicity and code-centric nature of NServiceBus sagas either excites or depresses, depending on the possibility of switching or introducing new tools.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Check it out: “Hello, Arduino” in CODE Magazine, an introduction to the programmable microcontroller designed to be approachable and fun. If you’ve ever had a hankering to see your code manifest your will in the corporeal world, you’ll love the Arduino. My article will take you step-by-step through a “Hello, World” project, even (especially!) if you’re a complete beginner to electronics.

(Why’d I buy the magazine? Not to worry, EPS sent me three copies of my own, as is right and proper. I just wanted an extra one to leave at TechShop. Plus, what a thrill to purchase a magazine with my words and ideas in it! “Yes, I’d like a latte and maybe you recognize the author of this article, hmm?” Regrettable narcissism, I know.)

I’ll be giving an Arduino presentation at That Conference in August in Wisconsin. Please come to my talk, say hi, show me what you’re building!

If you’re a parent or get to hang out with kids on a regular basis, introduce them to the Arduino. You will have a blast building stuff together. Here is a nine-year-old developer I pair-programmed with at CodeMash (she specified the requirements and did all the typing):

I’d love to hear about your projects. Make great stuff!

Since the queues in this picture are different resources than the database, I need to involve a third party, or transaction manager, to coordinate transactions between these three resources.

DTC, when it works, works really well. It’s much, much easier to not care about the consequences of a lack of coordination. In fact, I’d recommend not caring until you actually do care, because ditching two-phased commits does require work. Luckily for us, there are a ton of resources on how to do exactly that!

Most of the time, literature around avoiding 2PC is concerned about an entirely different situation, where I have two separate databases:

We’re doing messaging, which means that it’s typically the consumer of the message that does something against other data stores. So even though we’re avoiding communicating with two databases, it’s still two resources, and thus a need to coordinate!

But again, that coordination comes with a cost. A fairly large cost, in some recent testing we found that overall throughput dropped 80%, or to put it another way, ditching DTC saw a 5X increase in throughput. Five fold!

For some systems, that throughput doesn’t matter much, but for those that have a reasonably high volume of messages, or sensitive SLAs, it’s worth investigating alternative approaches.

General rules of thumb

Like most messaging approaches, the ways of avoiding coordination are right in front of our faces. In Gregor Hohpe’s excellent paper on Starbucks, he points out any real-world system that values throughput over absolute consistency avoids distributed transactions. The basic ideas are:

• Idempotency is king. Get this and you’re halfway home
• Strategies for dealing with downstream effects is a business decision

Idempotency is absolutely required, but it’s not that hard to apply. For some operations, we can rely on natural idempotency. If I’m asked to turn on the light, receiving the request twice means the same outcome – the light is on! For state machine-like systems, idempotency is a bit easier.

For operations that aren’t naturally idempotent (launch the nuclear missile), we’ll need to get a little creative. If we can identify some correlating information from a request (The president called at 9:15 to launch the missile) or just assign some correlation information (The president has issued request #132 to launch the missile), we can simply keep a journal on the receiving side. If it’s expensive to keep a journal around, we can recycle/trim our journals if they get too big.

Downstream effects become more interesting. If throughput is a high concern, we can rely on compensating actions (customer didn’t have enough money, cancel the order) or more journaling. Instead of sending a message immediately, shouting out messages to downstream systems, we can instead just write down in the same persistent store as our other data another journal for outgoing messages.

Once our local DB transaction is complete, it’s just a matter of sending the messages we’ve written down to send out down the line, and crossing them off our list of “sent” messages. And since downstream systems can deal with at-least-once messages through our idempotency guarantees.

How I learned to stop worrying and ditch 2PC

In some current systems, we’re deciding on a service-by-service basis whether or not we want to enlist or not enlist in distributed transactions. It’s still annoying to try and build a system-wide solution (though the event sourcing guys have this more or less in the bag), so until then, I can just use business decisions to guide me one way or the other.

But it is time to let go and stop worrying so much. Honestly, unless your services have downstream side effects, you can safely turn off DTC if your work is idempotent. If you have downstream side effects, there’s a number of paths to choose from. While I’m not saying goodbye forever (still the best solution if it were absolutely free to use), it is time to shop around.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

One thing that grounds all of this is what we consider the message as developers, and what our transport and messaging system considers to be the message. For example, the Azure REST services expose contracts as XML. XML, by itself, and JSON make for great transport formats because the underlying technology provides a fairly universally acceptable common type system. Although your language might need to bridge from your format to theirs, people standardize on ISO formats for standard primitives to maximize interoperability (and minimize serialization mistakes).

Dealing with such large XML documents from a client API can be a pain, however. XML is notoriously finicky with respect to case sensitivity, and I can’t count how many times I’ve been bitten by this when dealing with raw XML and REST APIs. We would need to assume that documentation exists, and until forms become common in REST APIs, it’s difficult to say that HATEOAS will simply solve all these problems of self-describing APIs.

Instead, we often see REST and other messaging clients, out of convenience, build DTOs as a means of representing the message. But first – what is a message? A message is just data. It’s defined by a header and body, where the header is used by the transport/messaging system and the body is ignored (picture courtesy http://www.eaipatterns.com):

However, messages aren’t types. But what about sharing something like this?

[DataContract]
public class PurchaseOrder
{
	private int poId_value;
	
	// Apply the DataMemberAttribute to the property.
	[DataMember]
	public int PurchaseOrderId
	{
	
	    get { return poId_value; }
	    set { poId_value = value; }
	}
}

That’s still not terrible, because underneath the covers our message is still just XML or JSON. We use this type as a description or blueprint of our message, because it’s simpler to describe our message in C# terms instead of a looser type like XML or JSON, which are more difficult to describe and use in C#. I’m ignoring dynamic types in .NET – those to me are a bit of a hack in this case. In WCF, proxy classes get generated on the client side, so we’re still not taking an assembly dependency. This isn’t available in REST or other messaging technologies, leaving clients reliant on documentation to “Get it right” – assuming that they don’t make mistakes translating raw XML or JSON into code building raw XML or JSON on the client side.

So what typically happens in a homogenous environment is that data contract assemblies are shared:

Both client and server share a contracts assembly, and use the contracts assembly as a description for how to construct and consume the raw messages.

We introduce coupling on the client side with a raw type shared across the server boundary, but it’s up to those building the system to determine if this sort of coupling introduces any potential risks. When we look at coupling, we must always balance risk. If coupling introduces low risk, it might be acceptable (assuming we’re more or less prescient in our future system’s design).

From my experience, as long as the messaging infrastructure doesn’t assume that the message is built from a type and therefore leak those concerns, this sort of model can be a nice compromise in ecosystems where types as blueprints for messages ensures safety in our message construction and consumption. It’s similar to building MVC applications around View Models – they’re a blueprint for building forms, and a means of accepting raw form POSTs. Side note – I found it hilarious that the Rails folks ran into that mass assignment security problem – it’s a problem I’ve never, ever had in MVC.

But that’s the real kicker – our messaging infrastructure can’t assume types, as the message is not the type. We might use a type as a convenience to build and consume, as we do in MVC, but ultimately, our messaging infrastructure can’t assume a type. MVC handles this quite well, with model metadata and its ModelState objects. The original request is always preserved in its raw form (dictionary of strings), but the model provided to the controller action is an approximation of that request.

It’s only when we assume that we’ve literally shared types that we’re going to slip into real type coupling, and everything that SOAP failed to deliver comes back again.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

… like I said … “half-jokingly”

I give you the Slap-Slides, proof of concept! (no audio)

The code is written in Johnny-Five, running in NodeJS, connected to an Arduino Nano. I bought the cheapest veggie-chopper I could find at my local grocery store, ripped the blade out and mounted a 10k button on to the plate that the blade was previously on.

I’ll be soldering this in to a proper set up housed entirely in the veggie-chopper, and stabilizing the button (right now it fails every 2 or 3 clicks because of the mounting in the veggie chopper.

UPDATE: I have it nearly complete.

I have the primary circuitry soldered on to a component board, and some holes cut in the housing in order to clean up the entrance of a USB cable in to the chopper.

Here’s a series of pictures that show the components and assembly… making code skinny, one slap at a time :D

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Back in our sandwich shop example, we had a picture of an interaction starting a process and moving down the line until completion:

Not quite clear in this picture is that if we were to model this process as a saga, we’d have a central point in which all messages must flow to for decisions to be made on the next step. But is there really any decision to be made in the picture above? In a true saga, we have a sequence of steps and a set of compensating actions (in a very simplistic case). Many times, however, there’s no need to worry about compensations in case of failures. Nor does the order in which we do things change much.

Humans have already found that assembly lines are great ways of breaking down a long process into individual steps, and performing those steps one at a time. Henry Ford’s Model T rolled off the assembly line every 3 minutes. If only one centralized worker coordinated all steps, it’s difficult to imagine how this level of throughput could be achieved.

The key differentiator is that there’s nothing really to coordinate – the process of steps is well-defined and known up front, and individual steps shouldn’t need to make decisions about what’s next. Nor is there a need for a central controller to figure out the next step – we already know the steps up front!

In our sandwich model, we need to tweak our picture to represent the reality of what’s going on. Once I place my order, the sequence of steps to fulfill my order are known up front, based on simply examining my order. The only decision to be made is to inspect the order and write the steps down. My order then flows through the system based on the pre-defined steps:

Each step doesn’t change the order, nor do they decide what the next step is (or even care who the next step is). Each step’s job is to simply perform its operation, and once completed, pass the order to the next step.

Not all orders have the same set of steps, but that’s OK. As long as the steps don’t deviate from the plan once started, we don’t need to have any more “smarts” in our steps.

It turns out this pattern is a well-known pattern in the messaging world (which, in turn, borrowed its ideas from the real world): the routing slip pattern.

Routing slips in NServiceBus

Routing slips don’t exist in NServiceBus, but it turns out it’s not too difficult to implement. A routing slip represents the list of steps (and the progress), as well as a place for us to stash any extra information needed further down the line:

    public interface IRoutingSlip
    {
        Guid Id { get; }
        IEnumerable<IProcessingStep> ProcessingSteps { get; }
        IDictionary<string, string> Attachments { get; }
    }

We can attach our routing slip to the original message, so that each step can inspect the slip for the next step. We’ll kick off the process when we first send out the message:

Bus.Route(sandwichOrder, new[]
{
	"Preparation",
	"Oven",
	"Packing",
});

Each handler handles the message, but doesn’t really need to do anything to pass it down the line, we can do this at the NServiceBus infrastructure level.

public class PackingHandler 
	: IHandleMessages<SandwichOrder>
{
	public void Handle(SandwichOrder message)
	{
	    // pack the sandwich
	}
}

The nice aspect of this model is that it eliminates any centralized control. The message flows straight through the set of queues – leaving out any potential bottleneck our saga implementation would introduce. Additionally, we don’t need to resort to things like pub-sub – since this would still force our steps to be aware of the overall order, hard-coding who is next in the chain. If a customer doesn’t toast their sandwich – it doesn’t go through the oven, but we know this up front! No need to have each step to know both what to do and what the next step is.

I put the message routing implementation up on NuGet and GitHub, you just need to enable it on each endpoint via configuration:

public class Startup : IWantCustomInitialization
{
	public void Init()
	{
	    Configure.Instance.RoutingSlips();
	}
}

If you need to process a message in a series of steps (known up front), and want to keep individual steps from knowing what’s next (or introduce a central controller), the routing slip pattern could be a good fit.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

We took a little haitus last year, but we’re coming back this year for our Third Pablo’s Fiesta Open Space conference.

What is it?

Pablo’s Fiesta is a an Open Space conference on Software Quality and Craftsmanship.  It’s a chance for us to come together, learn from each other, and share our experiences and passion for what we do in an open and inclusive atmosphere.

If you have never been to an Open Space conference, it’s a conference experience like no other.  The schedule is defined at the conference open session, so it’s up to us to make it a great conference!!  Unlike traditional, presentation centric conferences, you are encouraged to move around and find a topic or session that is the most stimulating for you.  Or just start your own.  It’s more of a conversation than presented material.

The Schedule
I’ll provide more detailed schedule information as I get more things planned, such as any parties and social events.

If you’ve been to previous Fiestas, this one is just a little different.  The previous conferences started on Friday night for the opening session  and schedule selection and sessions on Saturday and Sunday.  This year, we will only have sessions on Saturday.  We are going to plan some social activities for Sunday, October 27th, like a picnic and a hackfest, so you might want to plan on staying in Austin until late Sunday.

Friday October 25, around 6:00 PM.  Opening Session.  This is where we propose topics for the conference and define the schedule for the conference.

Saturday October 26, around 9:00 AM.   Conference Sessions begin.

Sunday October 27, TBA. Social Activities.  We’ve got some ideas were bouncing around, probably a picnic or some other activity, plus a hackfest or something.

It’s going to be a great time and I look forward to meeting you all there!

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

It was almost a year ago that I announced a book on building Backbone.js plugins. Since then… nothing. Well I finally got tired of sitting on it and over the weekend I made a bunch of edits, cleaned up a lot and released the first 6 chapters of the book via Leanpub!

https://leanpub.com/building-backbone-plugins

Learn the how and why of building plugins for Backbone.js. This book provides first hand lessons learned, best practices, and patterns for building flexible add-ons and frameworks components, based on my experience – both success and failure – from the last few years of working with Backbone and building plugins for it!

On Buidling Backbone Plugins

I’ve built more than a handful of Backbone related tools, plugins and add-ons in my time working with it. From nearly the beginning of my exploration and use of Backbone, in fact, I’ve been looking for and developing better methods of handling common tasks and solving the same problems. This has led me to create plugins and frameworks such as MarionetteJS (Backbone.Marionette), Backbone.Picky, Backbone.ModelBinding and many more – some published on my Github account, many unpublished and created for a specific client application.

In all of the plugins and add-ons that I’ve built, there have been lessons learned. The continued success of frameworks like MarionetteJS provide many useful insights in to how to properly create view layers and additional object types that support the creation of large Backbone application. Equally as valuable, though, failures such as Backbone.Modelbinding provide insight in to the patterns and practices that should be avoided when creating add-ons. Or, at least, they illustrate some of the scenarios and circumstances in which these patterns and practices fall apart.

A Work In Progress

This book is a work in progress, with 6 chapters roughly completed at this time, 3 or 4 additional chapters in rough draft, and one or two chapters planned beyond that. Expect this book to change over time, with edits and additional chapters being added. Early adopters will be able to purchase the book at a discounted price. The amount of demand I am seeing from early adopers will largely determine the schedule that I keep for additional releases, and the eventual price that is charged for the finished product.

https://leanpub.com/building-backbone-plugins 

Be sure to follow the book on twitter for updates, as well: @BackbonePlugins

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

The probe is a 100K “meat probe” (aka “thermistor”) that I took from a store bought meat thermometer. Something like this replacement probe would work great. It’s a bit of a guessing game as to wether the probe is a 10k or 100k Ohm probe. I tried 10, 100, 1k, 10k and finally 100k resistors in my thermistor setup. You can read about thermistor setup on the Arduino playground, and at Hacktronics.

The Arduino code produces a JSON document when I make an HTTP request to it. The software is built with Kendo UI Mobile, and reads the JSON document on a 1 second interval.

I’ll be blogging about this more, and hopefully soon, with plans on going through a step by step “how I learned” series – at least, that’s the plan. For now, this little teaser video should give you a good idea of what can be done with an Arduino, some simple parts from a store, and a mobile app framework like Kendo UI Mobile. :)

… I need to learn how to optimize my C code for battery life. I’ve run that 9volt battery maybe 10 or 20 minutes, total, and it’s already dead. :P

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Installing Geeknote

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

But recently I’ve been working with a team that has broadened my horizons and imparted a valuable lesson to me: “there’s nothing Git can do that TFS can’t.” Now if you’ve been following me for a while, you know this has been patently untrue. Until today.

Announcing posh-tf

Obviously the killer feature that TFS lacks as an alternative to Git is that it doesn’t integrate well with PowerShell. According to the UserVoice suggestion, at least one person thinks this needs to be remedied. And so I am pleased to present posh-tf: a PowerShell environment for Team Foundation.

posh-tf provides everything that posh-git provides, except for the things that aren’t readily available from tf.exe:

• No current branch information
• No file status information
• No tab completion for new, modified or deleted files
• No tab completion for branches, shelvesets, etc

Actually, it would probably be easier to just list what it does provide:

• Tab expansion for commands: tf ch<tab> expands to tf checkout
• Tab expansion for help: tf help ch<tab> expands to tf help checkout
• A prompt hook if someone can figure out a way to get current directory status.

Can’t wait to get started? The readme has everything you need!

I’m thrilled with what I have to release today, and am looking forward to working with the TFS community to make posh-tf even better!

Introduction

When sagas were originally conceived in MassTransit, they were inspired by an excerpt from Chapter 5 in the book SOA Patterns by Arnon Rotem-Gal Oz. Over the past few months, the community has argued discussed how the use of the word saga has led to confusion and how early implementations included in both NServiceBus and MassTransit do not actually align with the original paper published in 1987 by Princeton University and written by Hector Garcia-Molina and Kenneth Salem in which the term was coined.

With MassTransit Courier, the intent is to provide a mechanism for creating and executing distributed transactions with fault compensation that can be used alongside the existing MassTransit sagas for monitoring and recovery.

Background

Over the past few years building distributed systems using MassTransit, a pattern I consistently see repeated is the orchestration of multiple services into a single business transaction. Using the existing MassTransit saga support to manage the state of the transaction, the actual processing steps are created as autonomous services that are invoked by the saga using command messages. Command completion is observed using an event or response message by the saga, at which point the next processing step is invoked. When the saga has invoked the final service the business transaction is complete.

As the processing required within a business transaction changes with evolving business requirements, a new version of the saga is required that includes the newly created processing steps. Knowledge of the new services becomes part of the saga, as well as the logic to identify which services need to be invoked for each transaction. The saga becomes rich with knowledge, and with great knowledge comes great responsibility (after all, knowledge is power right?). Now, instead of only orchestrating the transaction, the saga is responsible for identifying which services to invoke based on the content of the transaction. Another concern was the level of database contention on the saga tables. With every service invocation being initiated by the saga, combined with the saga observing service events and responses, the saga tables gets very busy.

Beyond the complexity of increasing saga responsibilities, more recently the business has requested the ability to selectively route a message through a series of services based on the content of the message. In addition to being able to dynamically route messages, the business needs to allow new services to be created and added to the inventory of available services. And this should be possible without modifying a central control point that dispatches messages to each service.

Like most things in computer science, this problem has already been solved.

The Routing Slip Pattern

A routing slip specifies a sequence of processing steps for a message. As each processing step completes, the routing slip is forwarded to the next step. When all the processing steps have completed, the routing slip is complete.

A key advantage to using a routing slip is it allows the processing steps to vary for each message. Depending upon the content of the message, the routing slip creator can selectively add processing steps to the routing slip. This dynamic behavior is in contrast to a more explicit behavior defined by a state machine or sequential workflow that is statically defined (either through the use of code, a DSL, or something like Windows Workflow).

MassTransit Courier

MassTransit Courier is a framework that implements the routing slip pattern. Leveraging a durable messaging transport and the advanced saga features of MassTransit, MassTransit Courier provides a powerful set of components to simplify the use of routing slips in distributed applications. Combining the routing slip pattern with a state machine such as Automatonymous results in a reliable, recoverable, and supportable approach for coordinating and monitoring message processing across multiple services.

In addition to the basic routing slip pattern, MassTransit Courier also supports compensations which allow processing steps to store process-related data so that reversible operations can be undone, using either a traditional rollback mechanism or by applying an offsetting operation. For example, a processing step that holds a seat for a patron could release the held seat when compensated.

MassTransit Courier is free software and is covered by the same open source license as MassTransit (Apache 2.0). You can install MassTransit.Courier into your existing solution using NuGet.

Activities

In MassTransit Courier, an Activity refers to a processing step that can be added to a routing slip. To create an activity, create a class that implements the Activity interface.

public class DownloadImageActivity :
    Activity<DownloadImageArguments, DownloadImageLog>
{
}

The Activity interface is generic with two arguments. The first argument specifies the activity’s input type and the second argument specifies the activity’s log type. In the example shown above, DownloadImageArguments is the input type and DownloadImageLog is the log type. Both arguments must be interface types so that the implementations can be dynamically created.

Implementing an Activity

An activity must implement two interface methods, Execute and Compensate. The Execute method is called while the routing slip is executing activities and the Compensate method is called when a routing slip faults and needs to be compensated.

When the Execute method is called, an execution argument is passed containing the activity arguments, the routing slip TrackingNumber, and methods to mark the activity as completed or faulted. The actual routing slip message, as well as any details of the underlying infrastructure, are excluded from the execution argument to prevent coupling between the activity and the implementation. An example Execute method is shown below.

ExecutionResult Execute(Execution<DownloadImageArguments> execution)
{
    DownloadImageArguments args = execution.Arguments;
    string imageSavePath = Path.Combine(args.WorkPath, 
        execution.TrackingNumber.ToString());

    _httpClient.GetAndSave(args.ImageUri, imageSavePath);

    return execution.Completed(new DownloadImageLogImpl(imageSavePath));
}

Once activity processing is complete, the activity returns an ExecutionResult to the host. If the activity executes successfully, the activity can elect to store compensation data in an activity log which is passed to the Completed method on the execution argument. If the activity chooses not to store any compensation data, the activity log argument is not required. In addition to compensation data, the activity can add or modify variables stored in the routing slip for use by subsequent activities.

In the example above, the activity creates an instance of a private class that implements the DownloadImageLog interface and stores the log information in the object properties. The object is then passed to the Completed method for storage in the routing slip before sending the routing slip to the next activity.

When an activity fails, the Compensate method is called for previously executed activities in the routing slip that stored compensation data. If an activity does not store any compensation data, the Compensate method is never called. The compensation method for the example above is shown below.

CompensationResult Compensate(Compensation<DownloadImageLog> compensation)
{
    DownloadImageLog log = compensation.Log;
    File.Delete(log.ImageSavePath);

    return compensation.Compensated();
}

Using the activity log data, the activity compensates by removing the downloaded image from the work directory. Once the activity has compensated the previous execution, it returns a CompensationResult by calling the Compensated method. If the compensating actions could not be performed (either via logic or an exception) and the inability to compensate results in a failure state, the Failed method can be used instead, optionally specifying an Exception.

Building a Routing Slip

Developers are discouraged from directly implementing the RoutingSlip message type and should instead use a RoutingSlipBuilder to create a routing slip. The RoutingSlipBuilder encapsulates the creation of the routing slip and includes methods to add activities, activity logs, and variables to the routing slip. For example, to create a routing slip with two activities and an additional variable, a developer would write:

var builder = new RoutingSlipBuilder(NewId.NextGuid());
builder.AddActivity(“DownloadImage”, “rabbitmq://localhost/execute_downloadimage”, new
    {
        ImageUri = new Uri(“http://images.google.com/someImage.jpg”)
    });
builder.AddActivity(“FilterImage”, “rabbitmq://localhost/execute_filterimage”);
builder.AddVariable(“WorkPath”, “\\dfs\work”);

var routingSlip = builder.Build();

Each activity requires a name for display purposes and a URI specifying the execution address. The execution address is where the routing slip should be sent to execute the activity. For each activity, arguments can be specified that are stored and presented to the activity via the activity arguments interface type specify by the first argument of the Activity interface. The activities added to the routing slip are combined into an Itinerary, which is the list of activities to be executed, and stored in the routing slip.

Managing the inventory of available activities, as well as their names and execution addresses, is the responsibility of the application and is not part of the MassTransit Courier. Since activities are application specific, and the business logic to determine which activities to execute and in what order is part of the application domain, the details are left to the application developer.

Once built, the routing slip is executed, which sends it to the first activity’s execute URI. To make it easy and to ensure that source information is included, an extension method to IServiceBus is available, the usage of which is shown below.

bus.Execute(routingSlip); // pretty exciting, eh?

It should be pointed out that if the URI for the first activity is invalid or cannot be reached, an exception will be thrown by the Execute method.

Hosting Activities in MassTransit

To host an activity in a MassTransit service bus instance, the configuration namespace has been extended to include two additional subscription methods (thanks to the power of extension methods and a flexible configuration syntax, no changes to MassTransit were required). Shown below is the configuration used to host an activity.

var executeUri = new Uri(“rabbitmq://localhost/execute_example”);
var compensateUri = new Uri(“rabbitmq://localhost/compensate_example”);

IServiceBus compensateBus = ServiceBusFactory.New(x =>
    {
        x.ReceiveFrom(compensateUri);
        x.Subscribe(s => s.CompensateActivityHost<ExampleActivity, ExampleLog>(
            _ => new ExampleActivity());
    });

IServiceBus executeBus = ServiceBusFactory.New(x =>
    {
        x.ReceiveFrom(executeUri);
        x.Subscribe(s => s.ExecuteActivityHost<ExampleActivity, ExampleArguments>(
            compensateUri,
             _ => new ExampleActivity());
    });

In the above example two service bus instances are created, each with their own input queue. For execution, the routing slip is sent to the execution URI, and for compensation the routing slip is sent to the compensation URI. The actual URIs used are up to the application developer, the example merely shows the recommended approach so that the two addresses are easily distinguished. The URIs must be different!

Monitoring Routing Slips

During routing slip execution, events are published when the routing slip completes or faults. Every event message includes the TrackingNumber as well as a Timestamp (in UTC, of course) indicating when the event occurred:

• RoutingSlipCompleted
• RoutingSlipFaulted
• RoutingSlipCompensationFailed

Additional events are published for each activity, including:

• RoutingSlipActivityCompleted
• RoutingSlipActivityFaulted
• RoutingSlipActivityCompensated
• RoutingSlipActivityCompensationFailed

By observing these events, an application can monitor and track the state of a routing slip. To maintain the current state, an Automatonymous state machine could be created. To maintain history, events could be stored in a database and then queried using the TrackingNumber of the RoutingSlip.

Wrapping Up

MassTransit Courier is a great way to compose dynamic processing steps into a routing slip that can be executed, monitored, and compensated in the event of a fault. When used in combination with the existing saga features of MassTransit, it is possible to coordinate a distributed set of services into a reliable and supportable system.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

• Deadlocks
• Starvation

This is because of the fundamental (default) design of sagas is that:

• A single saga shares a single saga entity (causing deadlocks)
• All messages handled by a saga are delivered to the same endpoint (causing a bottleneck)

Two very different problems, with very different solutions. But, as per usual, we’ll look at how we might handle these situations in the real life to see how our virtual world should behave. Both of these problems (amongst others) are called out in the Enterprise Integration Patterns book, so it shouldn’t be a surprise if you run into these issues. Not as clear, however, is how to deal with them.

Deadlocks

Deadlocks in sagas are pretty easy to run into if we start pumping up the concurrency on our sagas. Back in our McDonald’s example, we published a message out and waited for all of the stations to report back:

In our restaurant, there’s only room for one person to examine a single tray at a time. When someone finishes a food item, they have to go look at all the trays. But what if someone is putting food on my tray? Do I push them out of the way?

Probably not, I need to wait for them to finish. It turns out that this is what happens in our sagas – we can only really allow one person at a time to affect a saga instance. The way NServiceBus can handle this by default is by adjusting the isolation level for Serializable transactions. This ensures that only one person looks at a tray at any given time.

What we don’t want to have happen is the fries person look at the tray, see a missing sandwich, and decide that the order isn’t done. The sandwich person does the same thing in reverse – look at the tray, see missing fries, and decide the order isn’t done.

It turns out that our Serializable isolation level has unintended consequences – we might lock more trays (or ALL of the trays) unintentionally. If there are more than one message that can start our saga, then effectively every employee has to look at all the trays to see if our tray is already on the counter, or we’re the first one and need to put a new one out. But if someone is also doing the same thing – multiple trays for the same order might go out!

We don’t want to block the entire counter for our order, so we can adjust our scheme slightly. If we implement a simple versioning scheme, we can simply have the employees look at a version number on the receipt, make their changes, and bump the version number with a little tally mark. If the tally mark has gone up since we last looked at the tray, we’ll just re-do our work – something has changed, so let’s reevaluate our decisions.

To get around the problem of too many trays for the same order, we can rely on a third party to ensure we don’t have too many trays. If when anyone places a tray down, the supervisor enforces this rule that only one tray per order can exist, they’ll ask us to put our tray back up and redo our work.

Two things we want to do then:

• Introduce a unique constraint on the correlation identifier to prevent duplicates
• Use optimistic concurrency to allow us to isolate just our own tray (and not affect anyone else)

This is a rather straightforward fix – and in fact, NServiceBus defaults to using optimistic concurrency in 4.0 (now in beta). This was a rather easy fix, but what about our other problem of the bottleneck?

Starvation and bottlenecks

Back in my early days of school, I worked at a fast food restaurant during breaks (surprise, surprise). I worked the graveyard shift of a 24-hour burger joint, and it was just me and one other person manning the entire place. From 10PM to 6AM, just two of us.

One thing that I found pretty early was that we had quite a few “regulars”. These were folks that every morning, came in and ordered the same thing at around the same time. Because it was quite early, they could rely on little to no contention for resources, and the queue was more or less empty.

Most of the time.

But every once in a while, we would get someone ordering food for a large group of people. We didn’t do catering, but we did sell breakfast tacos. Handling one or two at a time wasn’t a problem, but someone coming in and ordering 50 tacos – that’s a problem. Our process looked something like this:

The problem was, because there were only two of us, most messages flowed through exactly one channel. In NServiceBus Sagas, this is what happens as well – all messages for a saga are delivered to a single endpoint. We might have the situation like this:

My cook has dozens of tacos to go through before getting to that second order, and because that person packs all the food and preps all the food, all items from the first order must be both prepped and packed before order #2 is complete:

That’s not a huge problem, except we have both packing and prepping done by a single person. Another order comes in, the prepping for that one interferes with the others. Additionally, items are sitting in the saga queue waiting to get packed while prepping finishes.

In many NServiceBus sagas, the sagas themselves don’t perform the actual work. They’re instead the controllers/coordinators, making decisions about next steps. But all the messages are delivered to the exact same queue, effectively creating a bottleneck. I’m waiting for all prepping to be done before moving on to any packing. The time it takes to this entire set of orders is basically the time it takes to prep all items plus the time it takes to pack all items.

The problem is our saga is modeled rather strangely. In the real world, long-running business processes, split into multiple activities/steps, are performed many times by different people. Effectively, different channels/endpoints/queues. But our saga shoved everyone back into one queue!

What we did in our taco tragedy above was that I came off the line from taking orders and performed the prep stage. For N items, we took a total time of N+1 or so. We had two queues going for the one saga:

The total number of items in the Pack queue was always small – packing was easy and quick. By splitting the saga handling into separate queues, we ensured that an overload in one type of message didn’t choke out all the others. This isn’t the default way of managing long-running processes in NServiceBus though, we have to set this up ourselves.

When dealing with the Process Manager pattern, it’s important to keep in mind what this pattern brings to the table. We’re able to support complex message flow, but at the price of potential bottlenecks and deadlocks.

Next time – a better pattern for linear processes that doesn’t bring in deadlocks and bottlenecks, the routing slip pattern.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

I quietly released it over the weekend, and I’m glad I did. I already had a bug report and have released v1.0.1 to fix it. :D

The last few years of working on Marionette have been tremendously fun, exciting, frustrating at time, and have provided much more work and opportunity for me than I had ever imagined. I started this project out of need, and hoped that it might help a few people out – at least show them what could be done and give them ideas on how to do it. I never expected that this project would have 1/10th the number of stars it has on Github, or that this project would lead to working with dozens of amazing companies, travelling around the world, create new friendships, and land me some awesome speaking opportunities on podcasts and at conferences! 

What’s New

There’s a ton of new things that have happened since I started the 1.0 release candidates. And actually, going in to “release candidate” model was probably the worst mistake I’ve made with Marionette, so far. I had far too many breaking changes, far too many times for it to really be a release candidate series. But things are stable now, and have been for a bit. There were a few additions and breaks in this release, some of which came from dependencies.

You can read the complete change log and upgrade guide in the repository, but I want to highlight a few of the notable things quickly.

Backbone.Wreqr Updates

Backbone.Wreqr is an integral part of Marionette and helps to provide a lot of the communication infrastructure for scaling applications. 

In the latest release of Wreqr, I’ve renamed “addHandler” to “setHandler” for all Wreqr objects. There was confusion around the name “addHander” as it implied the ability to add multiple handlers. Since none of Wreqr allows multiple handlers for any one registered name, I changed this to “setHandler”. Anyone updating to Wreqr v0.2.0, along with Marionette v1.0.0, will need to account for this.

I’ve also made the commands somewhat persistent when there is no handler currently registered. Several people noted that they were running in to situations where a command needed to be executed, but due to the order of module loading being unpredictable, the handler wasn’t there. To fix this, a command can be executed whether or not the handler is available. As soon as the handler is available, the actual execution of the command will take place. Commands will persist in memory, for the life of the page. Refreshing the page or navigating to another page will destroy the not-yet-executed commands.

Marionette.RegionManager

One of the most requested features in recent times has been the ability to add and remove regions from layouts. I set out to solve this and realized that half of the code I needed was already in the Application object, and the other half was in the Layout object. So I consolidated most of this code in to a single object called RegionManager. Both the Application and Layout now use a RegionManager to manage their regions. This allows a more consistent way of managing the regions, and provides all of the add and remove functionality that is needed, on both.

STICKERS!!!!

WOOO! STICKERS!!! :D

You can order yours from DevSwag. They are providing order management and fulfillment for stickers, for Marionette and many other amazing open source projects.

Semantic Versioning: A Goal

I have a goal, moving forward, of using Semantic Versioning for the version numbers. I don’t think I can promise perfect semantic versions, though, for a few reasons. First off, this is my first project to hit v1.0. I know I’m going to make mistakes and screw up some little detail here and there. But for the most part, I plan on following semver. Secondly, and more importantly, though, Marionette relies on other libraries which have their own release cycles. Sometimes releases coincide. Other times they don’t. Sometimes they break Marionette, other times they don’t. This will make it challenging to follow semver.

What’s Next?

It’s been a whirlwind ride with near constant updates along the way. I’ve learned a ton about JavaScript and running an open source project. And this is the first time I’ve pushed an open source project to 1.0, so I’m sure there are still a ton of lessons left to learn.

There are still more than 50 tickets open in the Marionette repository, and a lot of things that I want to do to help re-stucture some of the ugly parts. I want to tear apart Modules, for example, and create separated namespacing, sub-applications and components. There are a lot of other feature requests and issues reported that need big changes, too. I’m looking forward to where Marionette will head, and hope to continue to build a strong community around it – including a more active core contributor team.).

Thank You!

Yes, it’s the grammy-award speech time! Someone start the “wrap it up” timer…

I owe an especially large thanks to Jarrod Overson, Matt Briggs, Brian Mann and Tony Abou-Assaleh. These guys have been a great core team to bounce ideas around with, to help out with issues, and provide needed support and feedback. And to Ruben Vreeken for all the work he puts in to the Google group, answering questions. :)

Marionette wouldn’t be what it is without the amazing contributors. Everyone that has contributed in any way – answering questions in the google group, sending issue reports, ideas, pull requests, writing blog posts and articles, screencasts, and talking at user groups and conferences. I’m still and continuously amazed at how much has been done in the community with this library. Thank you all!

A special thanks to Backbone, of course, which is what got me into JavaScript again, and allowed me to build Marionette. Thank you to MedienFreunde for the amazing logo and website.

And thank you to all of the awesome people building apps with Marionette. Projects like Halo Waypoint, Redbull Music Academy, and StubHub are among the most widely known systems built with Marionette – then there’s the three error reporting / analytics tools: AirBrake.io, RayGun.io and CrashLytics. But there are so many more out there, still It’s been awesome seeing Marionette in all these places, and I’m looking forward to seeing where it goes from here! :)

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

• Command-oriented (request/response) in the Controller pattern
• Event-oriented (pub/sub) in the Observer pattern

Of course, these aren’t the only ways our saga could behave. We could have any combination of these.

Publish-Gather

Looking back on our McDonald’s example, we could improve our situation a little bit. We could have a situation where we want a command to start a saga, then have the saga itself publish a message. It would then wait for events to come back (ignoring order):

The advantage here is that we only have one entry point to our saga. We don’t have to worry about our saga potentially getting started by any number of messages that were pushed out.

When you place an order at McDonald’s, it’s almost always the cashier that places the tray on the table. When stations are finished with fries, sandwiches etc., they don’t really come to the counter and make a decision “do I need a new tray?”. The tray is already there, so our saga has already started.

There’s also nothing stopping our downstream processes from spawning off additional sagas – but that’s hidden from our originator.

Reporter

Another role our saga can play is one that doesn’t make decisions, but instead merely reports status:

This is a situation where a saga might never actually complete, and goes on forever. Its role is to communicate status of a longer-running process in the back-end, not for coordination purposes, but for reporting purposes.

The reason we might want a saga for this case is we still don’t know what order we’ve received messages from downstream systems we don’t own. As we learn about downstream events, we can evaluate them based on our knowledge of the overall business process. An order can’t go backwards from “shipped” to “verified”, so receiving these out of order doesn’t change the fact that the order shipped! Note: this doesn’t necessarily imply the status is in the saga entity itself. It still could be separate.

Keeping it as a saga lets us handle messages in any order and keep some centralized logic around interpreting these messages.

Sagas/process managers are pretty flexible in how we compose the pieces together. I often get questions around “why can’t I design a saga like a workflow?” And the answer is that sagas are meant to handle cases where I don’t have a directed workflow, where I live in a world where messages can arrive out of order.

It’s a much easier world to scale – but we need to accept the complexity it brings on the other end.

There is another clear downside here – we have shared state amongst multiple messages and handlers, which can potentially lead to scaling problems. Next time, we’ll look at scaling sagas.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Your favorite open source kanban board built on top of the GitHub api has gotten a little bit more awesome. I’ve decided to release the socket.io backend that is keeping huboard.com all up to date and super fancy.

Why nodejs (express + socket.io)?

1. Simple

Nodejs (express + socket.io) was by far the easiest thing for me to get running on heroku. The original version I published back in August was only about 26 lines of javascript.

2. Reliable

I’m absolutely amazed, I pushed that code up to heroku back in August and haven’t touched it. It ran for about 6 months and I never even checked on it. As far as I can tell it never even crashed.

3. Documentation

There is a ton of great tutorials and blog posts just a google search away on getting socket.io up and running on heroku.

• Heroku – Dev Center
• Socket.io site

4. Cheap (free as in beer)

Plain vanilla install of socket.io and express configured to use xhr-polling has been running problem free on heroku with one free dyno for 6 months. I’m not exactly sure how far that will scale using the MemoryStore (eventually I’ll run out of RAM) but huboard is sitting at nearly 4000 users and haven’t had any problems yet

Other options

I did explore other options

• .NET – server sent events – (fubumvc or signalr)

  • FubuMVC.ServerSentEvents
    • Not supported by heroku (see #1 && #4)
    • SSE only – no fallback support
    • Very little documentation
  • [SignalR]
    • Not supported by heroku
    • Not as stable (at the time)

• Ruby – EventMachine

  • I originally wanted to use something simple on top of ruby. I tried several different approaches and ultimately I just couldn’t figure out how to host a HTTP POST endpoint && the socket connections inside the same app.

Why did it take you so long to release it?

Well the TL;DR; is that it wasn’t secure. It was largely experimental. Back when I published the RealTime™ support the socket.io server was only about 26 lines of nodejs. It had no security because I didn’t know how to set it up. So I secure it the best I could (with a simple correlation string) and called it a day. The code wasn’t published so hackers at least had a harder time connecting to the socket than if the code was in the open. Security by obscurity (I know shame on me, I’m sorry).

How are you going to make it secure?

Step one is open sourcing it, people who are using huboard and know more than I do will hopefully care enough to help me make it as secure as possible.

• Things I’m doing already.

All my users trust me with their OAuth token and I take that responsibility seriously. I try my best to make sure your authorization key doesn’t get into the wrong hands. Huboard never exposes your authorization key unencrypted.

The socket server now performs a handshake when making a connection. Your encrypted OAuth token is passed into socket.io via query string, which turns around and passes the encrypted key to a huboard api endpoint that decrypts your token and asks the GitHub api if its valid.

*| Disclaimer *| I’m not a security expert, I’m not happy that I’m passing the token back to huboard for verification. It doesn’t feel right. I tried for days to decrypt the token using node in order to validate the token inside the handshake code but I just ran into dead ends. I think it has something to do with the encoding of ruby vs node but I had very similar encryption code on both platforms and could figure out how to get node to decrypt a string from ruby or visa versa. Please help!

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

When it comes to the Mediator and Event Aggregator patterns, there are some times where it may look like the patterns are interchangeable due to implementation similarities. However, the semantics and intent of these patterns are very different. And even if the implementations both use some of the same core constructs, I believe there is a distinct difference between them. I also believe they should not be interchanged or confused in communication because of the differences.

It’s All About Logic

The TL;DR version of this article is this: where does the logic live? An event aggregator has no logic, other than forwarding events from a publisher to a subscriber. A mediator, on the other hand, encapsulates the potentially complex logic of coordinating multiple other objects and/or services, to accomplish a goal. A mediator contains real application / business / workflow / process logic.

This is the line I’m drawing in the ever-shifting sands of implementation details and fuzzy heuristics.

Event Aggregator

The core idea of the Event Aggregator, accoring to Martin Fowler, is to channel multiple event sources through a single object so that other objects needing to subscribe to the events don’t need to know about every event source.

Backbone’s Event Aggregator

The easiest event aggregator to show is that of Backbone.js – it’s built in to the Backbone object directly.

var View1 = Backbone.View.extend({
  // ...

  events: {
    "click .foo": "doIt"
  },

  doIt: function(){
    // trigger an event through the event aggregator
    Backbone.trigger("some:event");
  }
});

var View2 = Backbone.View.extend({
  // ...

  initialize: function(){
    // subscribe to the event aggregator's event
    Backbone.on("some:event", this.doStuff, this);
  },

  doStuff: function(){
    // ...
  }
})

In this example, the first view is triggering an event when a DOM element is clicked. The event is triggered through Backbone’s built-in event aggregator – the Backbone object. Of course, it’s trivial to create your own event aggregator in Backbone, and there are some key things that we need to keep in mind when using an event aggregator, to keep our code simple.

jQuery’s Event Aggregator

Did you know that jQuery has a built-in event aggregator? They don’t call it this, but it’s in there and it’s scoped to DOM events. It also happens to look like Backbone’s event aggregator:

$("#mainArticle").on("click", function(e){

  // handle the event that any element underneath of our #mainArticle element

});

This code sets up an event handler function that waits for an unknown number of event sources to trigger a “click” event, and it allows any number of listeners to attach to the events of those event publishers. jQuery just happens to scope this event aggregator to the DOM.

Mediator

A Mediator is an object that coordinates interactions (logic and behavior) between multiple objects. It makes decisions on when to call which objects, based on the actions (or in-action) of other objects and input.

A Mediator For Backbone

Backbone doesn’t have the idea of a mediator built in to it like a lot of other MV* frameworks do. But that doesn’t mean you can’t write one in 1 line of code:

var mediator = {};

Yes, of course this is just an object literal in JavaScript. Once again, we’re talking about semantics here. The purpose of the mediator is to control the workflow between objects and we really don’t need anything more than an object literal to do this.

var orgChart = {

  addNewEmployee: function(){

    // getEmployeeDetail provides a view that users interact with
    var employeeDetail = this.getEmployeeDetail();

    // when the employee detail is complete, the mediator (the 'orgchart' object)
    // decides what should happen next
    employeeDetail.on("complete", function(employee){

      // set up additional objects that have additional events, which are used
      // by the mediator to do additional things
      var managerSelector = this.selectManager(employee);
      managerSelector.on("save", function(employee){
        employee.save();
      });

    });
  },

  // ...
}

This example shows a very basic implementation of a mediator object with Backbone based objects that can trigger and subscribe to events. I’ve often referred to this type of object as a “workflow” object in the past, but the truth is that it is a mediator. It is an object that handles the workflow between many other objects, aggregating the responsibility of that workflow knowledge in to a single object. The result is workflow that is easier to understand and maintain.

Similarities And Differences

There are, without a doubt, similarities between the event aggregator and mediator examples that I’ve shown here. The similarities boil down to two primary items: events and third-party objects. These differences are superficial at best, though. When we dig in to the intent of the pattern and see that the implementations can be dramatically different, the nature of the patterns become more apparent.

Events

Both the event aggregator and mediator use events, in the above examples. An event aggregator obviously deals with events – it’s in the name after all. The mediator only uses events because it makes life easy when dealing with Backbone, though. There is nothing that says a mediator must be built with events. You can build a mediator with callback methods, by handing the mediator reference to the child object, or by any of a number of other means.

The difference, then, is why these two patterns are both using events. The event aggregator, as a pattern, is designed to deal with events. The mediator, though, only uses them because it’s convenient.

Third-Party Objects

Both the event aggregator and mediator, by design, use a third-party object to facilitate things. The event aggregator itself is a third-party to the event publisher and the event subscriber. It acts as a central hub for events to pass through. The mediator is also a thirdy party to other objects, though. So where is the difference? Why don’t we call an event aggregator a mediator? The answer largely comes down to where the application logic and workflow is coded.

In the case of an event aggregator, the third party object is there only to facilitate the pass-through of events from an unknown number of sources to an unknown number of handlers. All workflow and business logic that needs to be kicked off is put directly in to the the object that triggers the events and the objects that handle the events.

In the case of the mediator, though, the business logic and workflow is aggregated in to the mediator itself. The mediator decides when an object should have it’s methods called and attributes updated based on factors that the mediator knows about. It encapsulates the workflow and process, coordinating multiple objects to produce the desired system behaviour. The individual objects involved in this workflow each know how to perform their own task. But it’s the mediator that tells the objects when to perform the tasks by making decisions at a higher level than the individual objects.

An event aggregator facilitates a “fire and forget” model of communication. The object triggering the event doesn’t care if there are any subscribers. It just fires the event and moves on. A mediator, though, might use events to make decisions, but it is definitely not “fire and forget”. A mediator pays attention to a known set of input or activities so that it can facilitate and coordinate additional behavior with a known set of actors (objects).

Relationships: When To Use Which

Understanding the similarities and differences between an event aggregator and mediator is important for semantic reasons. It’s equally as important to understand when to use which pattern, though. The basic semantics and intent of the patterns does inform the question of when, but actual experience in using the patterns will help you understand the more subtle points and nuanced decisions that have to be made.

Event Aggregator Use

In general, an event aggregator is uses when you either have too many objects to listen to directly, or you have objects that are unrelated entirely.

When two objects have a direct relationship already – say, a parent view and child view – then there might be little benefit in using an event aggregator. Have the child view trigger an event and the parent view can handle the event. This is most commonly seen in Backbone’s Collection and Model, where all Model events are bubbled up to and through it’s parent Collection. A Collection often uses model events to modify the state of itself or other models. Handling “selected” items in a collection is a good example of this.

jQuery’s on method as an event aggregator is a great example of too many objects to listen to. If you have 10, 20 or 200 DOM elements that can trigger a “click” event, it might be a bad idea to set up a listener on all of them individually. This could quickly deteriorate performance of the application and user experience. Instead, using jQuery’s on method allows us to aggregate all of the events and reduce the overhead of 10, 20, or 200 event handlers down to 1.

Indirect relationships are also a great time to use event aggregators. In Backbone applications, it is very common to have multiple view objects that need to communicate, but have no direct relationship. For example, a menu system might have a view that handles the menu item clicks. But we don’t want the menu to be direcly tied to the content views that show all of the details and information when a menu item is clicked. Having the content and menu coupled together would make the code very difficult to maintain, in the long run. Instead, we can use an event aggregator to trigger “menu:click:foo” events, and have a “foo” object handle the click event to show it’s content on the screen.

Mediator Use

A mediator is best applied when two or more objects have an indirect working relationship, and business logic or workflow needs to dictate the interactions and coordination of these objects.

A wizard interface is a good example of this, as shown with the “orgChart” example, above. There are multiple views that facilitate the entire workflow of the wizard. Rather than tightly coupling the view together by having them reference each other directly, we can decouple them and more explicitly model the workflow between them by introducing a mediator.

The mediator extracts the workflow from the implementation details and creates a more natural abstraction at a higher level, showing us at a much faster glance what that workflow is. We no longer have to dig in to the details of each view in the workflow, to see what the workflow actually is.

Event Aggregator And Mediator Together

The crux of the difference between an event aggregator and a mediator, and why these pattern names should not be interchanged with each other, is illustrated best by showing how they can be used together. The menu example for an event aggregator is the perfect place to introduce a mediator as well.

Clicking a menu item may trigger a series of changes throughout an application. Some of these changes will be independent of others, and using an event aggregator for this makes sense. Some of these changes may be internally related to each other, though, and may use a mediator to enact those changes. A mediator, then, could be set up to listen to the event aggregator. It could run it’s logic and process to facilitate and coordinate many objects that are related to each other, but unrelated to the original event source.

var MenuItem = Backbone.View.extend({

  events: {
    "click .thatThing": "clickedIt"
  },

  clickedIt: function(e){
    e.preventDefault();

    // assume this triggers "menu:click:foo"
    Backbone.trigger("menu:click:" + this.model.get("name"));
  }

});

// ... somewhere else in the app

var MyWorkflow = function(){
  Backbone.on("menu:click:foo", this.doStuff, this);
};

MyWorkflow.prototype.doStuff = function(){
  // instantiate multiple objects here.
  // set up event handlers for those objects.
  // coordinate all of the objects in to a meaninful workflow.
};

In this example, when the MenuItem with the right model is clicked, the “menu:click:foo” event will be triggered. An instance of the “MyWorkflow” object, assuming one is already instantiated, will handle this specific event and will coordinate all of the objects that it knows about, to create the desired user experience and workflow.

An event aggregator and a mediator have been combined to create a much more meaningful experience in both the code and the application itself. We now have a clean separation between the menu and the workflow through an event aggregator. And we are still keeping the workflow itself clean and maintainable through the use of a mediator.

Pattern Language: Semantics

There is one overriding point to make in all of this discussion: semantics. Communicating intent and semantics through the use of named patterns is only viable and only valid when all parties in a communication medium understand the language in the same way.

If I say “apple”, what am I talking about? Am I talking about a fruit? Or am I talking about a technology and consumer products company? As Sharon Cichelli says: “semantics will continue to be important, until we learn how to communicate in something other than language”.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

The Worst Part

The worst parts of the constant rejection include not getting any feedback on why, and the constant growing self doubt. But the absolute worst rejection was from CodeMash last year, where they passed me up in favor of some other guy that wanted to do a Marionette session. I heard reports that it didn’t go well… and they chose this guy over me, the creator of Marionette. That one still stands out as the most crushing rejection.

But enough self-pitty-crap. You’re here the find out WHY and how to fix it, right?

The Submission

I ended up posting one of my recent submissions to a gist so that I could get some actual feedback. This is the basic session submission information that I’ve been using for a while now. I change it up from here, based on the specific talk that I’m going to give, but it generally looks like this:

Scaling JavaScript Apps With Backbone And Marionette

Nearly everyone understands how to build a simple JavaScript application these days. From the classic “Todo” JavaScript app, to simple forms-over-data jQuery apps, we’ve all been around that block a few times. With the recent explosion of JavaScript MV* tools and frameworks, though, many of us find ourselves in over our heads, looking at patterns and practices that work well for small applications and pages but fail when scaling to anything substantial.

In this session, Derick Bailey will give you an introduction and walk through of many of the patterns and practices that your JavaScript applications need to be scalable. You’ll learn why copying Ruby on Rails’ pattern of “Models”, “Views” and “Controllers” folder names is wrong for scaling JavaScript apps, and how to correct that . You’ll learn about the necessity of separating the various concerns of your application. You’ll learn about patterns that aggregate and coordinate functionality from other parts of the system, how and when to properly decouple disparate areas of your application through messaging patterns, and more. And all of this will be illustrated with Backbone, MarionetteJS and additional plugins and patterns that can give you an edge in creating scalable applications in JavaScript.

I asked people to tear this apart for me, and tell me what I have been doing wrong. I got exactly what I wanted, and I have a lot of great tips to move forward, now. 

The Raw Responses

There’s a lot of useful tips in the raw responses, and lessons that can be picked from the responses. Rather than just trying to summarize things, though, I’m just going to post the responses directly and add my reactions.

@derickbailey I’ll be honest, it’s the beard man. They are Jealous of your testosterone.

— Scott Koon (@lazycoder) March 15, 2013

Ok, Scott’s probably right. My beard is amazing. But it’s probably more likely:

@derickbailey Ok, I’ll be actually honest here. I have no idea WHY code on the client needs to scale past a single user?

— Scott Koon (@lazycoder) March 15, 2013

Here’s one of the problems: semantics. “Scalability” is an over-used and abused word these days. When I say “scalability” in that abstract, I mean scaling to 10′s of thousands of lines of code, dozens of feature areas, sub-applications and general code size. I’m not referring to number of users since this all runs on client-side, one-person-at-a-time browsers. 

@derickbailey True, but you don’t say that in the abstract. I’m guessing you mean tons of data on the client w/complex cardinality.

— Scott Koon (@lazycoder) March 15, 2013

Dually noted, and something I need to clarify in my abstract.

@derickbailey @jbogard @robconery Needs more excitement! Get me excited about something!

— Kelly Sommers (@kellabyte) March 15, 2013

This is more likely the largest issue, and is the most common thread of feedback that I got. I spent years learning to take the emotion and extraneous fluff out of my written communications because I worked for large corporations where CEO’s wanted logical justification for me spending $100,000 on servers and software. Hand-waving doesn’t go over well in that environment, but apparently it’s what you need in a conference submission.

Rob Conery responded to my gist, with this:

##Taking The Suck Out of Javascript with MarionetteJS

Many have heard of Single Page Applications and, even though most of us in the room might not admit it, have experimented with the idea late at night… only to face the inevitable self-loathing and vacant, soulless feeling the next day. In this talk Derick Bailey will show you a happier path for building Single Page Applications, using Marionette together with BackboneJS.

Marionette helps to structure your Javascript applications – doing the things you don’t want to do (or have tired of doing thousands of times over) – and it won’t leave your code feeling bloated and lifeless.

This sparked a small response line in twitter:

Even if it bums people out, it’s a much more exciting and entertaining abstract. I would go to this session.

@robconery @kellabyte @derickbailey just do what jon skeet did in his abstract “I’M JON SKEET. SCIENCE BITCHES!!”

— Jimmy Bogard (@jbogard) March 15, 2013 

hmm… I’m considering this, though I think Jon Skeet might have problems with me claiming to be him. :D

@derickbailey @jbogard @robconery the abstract is too long. I liked the direction Rob was heading in the comments

— Chad Myers (@chadmyers) March 15, 2013 

Also very true. Anyone that has read me blog or articles knows that I have a hard time being concise. I definitely need to work on the length of abstract, which would also help move it toward the goal of more excitement. Less to read means I have to pack more information and punch in to fewer words.

In a separate email conversation, Steve Smith (@ardalis) had this to say:

Rob’s comment is great. Short, exciting, attention-getting abstracts are the best, but they’re not easy to write. What’s at least as important to those selecting sessions for an event is the speaker themselves. If you’re well-known, especially for the topic, that matters far more than the abstract. It’s also helpful to include your speaking background, so that those who don’t know you at least know you’re not new to speaking. Including ratings can help, too, as in: “Presented a similar talk to 100 developers at XYZ code camp and was rated 9.2 out of 10”

My first pass at abstracts usually look a lot like your initial one, which is kind of like the standard Microsoft product naming convention of describe-what-it-is-in-general-terms. If I’m submitting to something I already know I’m accepted for, I usually won’t spend a lot of time trying to sex it up, either (like, for a user group, for instance). But for something like TechEd or any other larger event where I don’t necessarily already have an “in”, I’ll try to spend more time making the abstracts more exciting, and also demonstrating my past experience.

There’s a lot of great summary in here, for sure. I’m not sure I buy the “celebrity” angle at this point, though. In spite of being invited to speak about Marionette (among other things) and in spite of Marionette being the most watched Backbone framework on Github, with my name all over it, people still reject my sessions in favor of other people. 

Unless you’re Jon Skeet, Addy Osmani or Paul Irish, you’re going to have to beg your way in to most conference submissions. You’re going to have to pay attention to the detail and make the submission exciting, or you’ll be passed up in spite of your name being all over the project that they want to hear about.

Lastly, @kellabyte capped it off with a dose of reality for all of this advice:

@derickbailey @robconery @jbogard Nothing guarantees anything we say is worth a damn though :P

— Kelly Sommers (@kellabyte) March 15, 2013

While this is true, I’m grateful to have received this feedback! This is infinitely more useful than the generic, patronizing “we had too many great submissions to choose from” responses that conference organizers like to send.

What Other Advice Do You Have?

I know I have a lot of work ahead of me to improve my submissions, and I’m still looking for more advice and more feedback. What else would you add to the conversation? What other tips and tricks do you have? 

I don’t want others to have to go through this. Help us all learn from your lessons and patterns in successful session submissions. Post in the comments here, or in a response blog post. Just be sure to share your tips with the world so that others can hopefully avoid the mistakes and learn from the successful lessons.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.