Arnon Rotem-Gal-Oz's Cirrus Minor

Evolving Architectures – Part II but Design is emergent

Arnon Rotem-Gal-Oz — Thu, 11 Mar 2010 21:28:49 GMT

This is part II of a series on agile architecture. You can read part I here.

In the previous installment I provided a definition for software architecture and raised the apparent friction between the up front design implied by software architecture and the YAGNI approach and deferred requirements prompted by agile development in the large. This installment take a look at an additional angle of the problem which is the difference between design and architecture (while architecture is a type of design I cam calling design the code or close abstractions of it that don’t yet fall under “architecture” as defined in the previous post). The difference between the two, as the title suggests, is that while design can be emergent, Architecture, unfortunately needs to evolve.

Unless you’ve been living under a rock, you’ve probably already know about Test Driven Development (TDD) or its cousin BDD. If you’ve actually used TDD, you’ve probably noticed the impact it has on the actual code. Writing code only to cater for defined tests coupled with refactoring that keeps the design tight we get to a result that, more often than not, simpler than going through the more traditional design first (you can take a look at Uncle Bob Bowling Kata for a simple, albeit synthetic , sample)

In fact, a more accurate expansion of the TDD acronym is Test Driven Design. Yes the tests are there to make sure the code adheres to the specified requirements. However, the iterative process (test,code & esp. refactor) makes the design emerge . The emergent design effect, works very well with the goals and tenets of Agile and Lean methodologies as it helps eliminate wasteful future-proofing code that we don’t need; eliminate waste of extra component (due to pre-design) etc.

This sounds great, so it is very tempting to take that to the architecture level, after all Architecture is a type of design, wouldn’t it be efficient to do TDA (Test Driven Architecture) as well. I think that in theory it might be possible. However as the old adage goes “Theory and practice are the same – at least in theory”. So in practice, the problem is that architecture is global and have solution-wide consequences. Time and size (of work) constraints make us want to set the playfield for the solution as soon as possible. Architecture decisions are hard to postpone on the one hand and have extensive influence on the other. E.g. a buy vs. build decision; 3-tier vs. SOA; RDBMS vs a NoSQL solution etc. if we try to have the architecture grew the way “regular” design can the ripple effects will be devastating to the development process.

What we have to do instead is evolve the architecture. Start with something that is in the ball park and then when we know more about the problem make changes, evolve the architecture over time. This is beneficial since requirements also tend to change over time so if we find a way to evolve the architecture we can deal with that as well.

In the next part I’ll try to give a couple of examples on how this can actually be done i.e. how you can evolve an architecture

*A better name is Test Driven Design but that’s not how it stated, as far as I know anyway

xsights is hiring

Arnon Rotem-Gal-Oz — Thu, 14 Jan 2010 05:13:25 GMT

I am happy to say that we (xsights) are looking to add a few more developers to our team. If you are interested ping me or send your CV to jobs at xsights dot com.

Please note that all the openings are in Israel (in the Natanya vicinity)

Server Side Developer

Position Description and Responsibilities:

We are looking for a motivated developer, a team player, who can improve and add features
to our product, focusing on a distributed environment with technology challenges.
Improve performance of existing components by clustering, messaging, caching etc.
Integrate with external services.

Requirements:

At least 2 years experience in C#/Scala, understanding OOP principles - must
Experience in concurrent programming - must
Experience with communication protocols (TCP, UDP, WCF) - an advantage
Experience in Unit Testing - an advantage
Knowledge in Distributed Services and Technologies, such as Hadoop, ActiveMQ - an advantage

Web Developer

Position Description:

We are looking for a motivated web developer for our distributed system, to improve and add new features to the inner admin site and to the company’s public beta site.

Responsibilities:

Web site development and deployment Server side programming in a distributed, challenging environment, in C#/Python/Ruby

Requirements:

At least 1 year experience in C#/Python/Ruby - a must
At least 2 years in web development with full understanding of development and deployment process:
Client Side: HTML, CSS, JS, upload using a Flash Uploader (such as SWFUpload), Ajax, jQuery.
Server Side: IIS and ASP.Net or other web server and technology Django/Rails/ASP.Net MVC - a big advantage FTP deployment - an advantage DNS settings - an advantage.

Junior Developer

Position Description:

We are looking for a motivated developer, a team player, who can program tools and tests in C#/Python, for acceptance/system tests of our distributed system. A creative and responsible person, with broad thinking.

Responsibilities:

Create an automated testing environment
Find and pinpoint problems
Code solutions

Requirements:

Experience in C#/Python - must
Experience with HTTP protocol - must
Experience in monitoring tools (WireShark, Fiddler) - an advantage
Experience in Unit Testing - an advantage
Knowledge in Distributed Services and Technologies, such as Hadoop, ActiveMQ - an advantage

Evolving Architectures – Part I What’s Software Architecture

Arnon Rotem-Gal-Oz — Wed, 13 Jan 2010 23:11:10 GMT

I’m writing a short series of posts for MS Israel MCS blog (in Hebrew) and I’d thought I’d translate them to English, as it seems to me they are interesting enough.

In this series I am going to talk about Evolutionary Architecture or , some of the aspect of dealing with software architecture in agile projects. The topic is interesting since architecture and agile seems to have some conflicting forces at work to better understand that let’s start by defining software architecture

There are many definitions for software architecture, with the simplest one (attributed, I think, to Kent Beck) that software architecture is what software architects does. Leaving the fact that (unfortunately) sometimes software architects are very far from building software architectures, the definition doesn’t tell us much. There are many definitions around some are good and some are bad. My current definition is

“Software architecture is the collection of decisions affecting the system’s quality attributes; which have global effects and are hardest to change. Software architecture provides the frame within which the design (code) is built.”

Let’s review the components of this definition

“affecting the system’s quality attributes” – I’ve written a lot about quality attributes in the past. In a nut shell, quality attributes (often going by the name “non-functional reqs.”) includes aspects of the system like scalability, security, availability etc. Architectural decisions have a direct effect on the system’s ability to meet these types of goal;
“Global effects” – Design decisions effect the module or the class where they happen. Decisions with macro effect (e.g. choosing a technology, scaling approach) can completely alter a solution
“Hardest to change” – The most interesting part of the definition, at least in regard to evolutionary architecture.The definition mentions that the code is built within the frame and rules set by the architectural decisions. Change in these decisions can have significant consequences. As a (over simplified) example - You can’t take a standalone system developed in Access and move it to a service oriented, Hadoop based solution with out major changes in the code, data flows and what not.

The definition of Software Architecture above, seems to prescribe that for best results we need to do a lot of up front design to get the architecture right. If that’s true than we have a severe mismatch with agile and/or lean where handling requirement and design up-front is a big no-no (YAGNI- you ain’t gonna need it, comes to mind) – Is there any way to make them work together.

I think yes, and as you’ve probably guessed, the answer is evolving the architecture over time. While this may sound simple it isn’t – I’ll try to give a few strategies to make that work later in the series. Before that, the next part, we’ll examine why design can be emergent which architectures need to evolve

Keep the BIT – check system liveliness

Arnon Rotem-Gal-Oz — Mon, 21 Dec 2009 15:59:06 GMT

Moving to architectures like SOA that increase the number of overall “moving parts” or components in the system means that reliability is going down. It is simple math really – if you have 10 components each with a 0.99 reliability then the total reliability is 0.99^10 or 0.904 and that’s before we take into account messages traveling over the wire and the network’s reliability (or lack thereof). What this does is leave us trying to build reliable systems from (a growing) bunch of unreliable components. I know, I know, there’s nothing new here. We’ve been using techniques like redundancy, statelessness etc. to help mitigate this since the beginning of times. With these techniques we decrease the “Mean Time Between Failure” (MTBF) but increase the “Mean Time Between Critical Failure” (MTBCF) or the system’s overall MTBF.

Another aspect of reliability (and reliability calculations) is MTTR or “Mean Time To Repair” which in software mainly has to do with how much time does it take before we know something is wrong. The usual approach to that is monitoring which I’ve written about in the past (e.g. the blogjecting watchdog pattern). In this post I want to expand a little on another approach , which while not common in IT systems, can be useful at times.

Enter the BIT – which is short of “Built In Tests”. BIT is a technique I picked up when I worked on multi-disciplinary systems that also included embedded systems. Each and everyone of the embedded systems we developed (or integrated into the solution) supported BIT . Actually they usually supported several types of BIT at least PBIT, CBIT and IBIT

PBIT – Power-On Built In Test – usually a short test the system runs to make sure all of its components are ready to go. You actually saw this one a lot of times since this is what motherboards do as you turn them on (all the blips and lights etc.)
CBIT – Continuous Built In Test – Make sure the system is functioning, even when it isn’t really busy so we’ll know about problems before we actually try to use the system
IBIT – Initiated Built In Test – provides a way to find out exactly what’s wrong when one of the other test types failed

BIT is very understandable for embedded systems, after all these are closed boxes with limited access to their innards and inner workings. but isn’t that also true for SOAs? After all we are building a bunch of blackboxes that interact to provide some business benefit, how can we be sure that everything is working fine esp. when we don’t control fully control some of the parts?

As mentioned above, a system, especially a distributed one, is built from relatively unreliable components. A continuous test helps us make sure things are working as expected. What we are doing is taking some of the code we wrote to run integration and acceptance tests (which runs a scenario end-to-end) deploy it as a service into the system which we call “liveliness check” and have it run periodically. Every time the liveliness runs it sends a notification (twitter message) so we know the test itself works. If it fails it sends more notifications (twitter, Email, SMS etc.) to an administrator.

This liveliness or CBIT serves as an early warning system. Since the end result is known in advance we can have a pretty good idea if something went wrong. E.g. we know how much time it should take for a test Id, we know what the result of that image is etc. The fact that it works even when the system is in low utilization means we can find out about problems and deal with them before they happen to end-users. That’s a big plus for us.

The advantage over regular monitoring solutions (this is not an either/or – monitoring is also needed) is that you know the specific business scenarios are properly working, which is a higher confidence that things are ok from knowing a specific server or service is running.

On the flip side, or the downside of adding a periodic liveliness is adding complexity into the system. In our case, we have to add a process to clean the traffic data added by the test messages. Also, while we try to make the system behave as usual as much as possible, certain parts of the system will have to know about the test messages and handle them differently. Again, in our case the reporting has to know to disregard test messages and not count them. This is even more problematic in other types of systems, for instance if you simulate an order, you don’t want the purchase order to actually go out to a supplier.

To sum this up, adding a liveliness check as part of the system to create a continuous built-in-test can increase your confidence that things are working as they should. It can also help you identify problems earlier. Like everything in life, it doesn’t come without tradeoffs and you should weight your benefits vs. costs before utilizing it in your systems.

Google Chrome -The browser is the new Desktop (2)

Arnon Rotem-Gal-Oz — Fri, 20 Nov 2009 20:23:44 GMT

You’ve probably read/heard that yesterday Google held a press event on making the ChromeOS the’ve been working on an open source project. Actually they’ve announced ChromeOS back in July, but now that the source is available it is making more waves. This is a very logical move for Google, and even though I think that they will need to enable some local/offline capabilities before it would be a viable option.

As usual* for Google, they are not the first in this space (If you want to get a webbook today you can try out litl, which looks like a very nice, or the GOS cloud on the Gigabyte M912) - but Google’s size and position in the market makes this interesting for everybody.

Anyway, I am almost thinking about a career change to analyst as I’ve been says that’s what they’ve up to since they released Chrome in Sept. 2008. when I published Google Chrome -The browser is the new Desktop. Here’s an excerpt:

Now,in my opinion, Google makes a bold move to change the rules and re-define the playground - if webapss need to run on the desktop, let's make the browser the new desktop.
What makes me say that? because it is focused on application (see the comics),because the browser runs each tab in its own process, because it has a process monitor, because it is a link on the google home page...

From the chrome "OS" point of view we can look at javascript,HTML etc. as the IL (bytecode in java speak) on which the application run. This makes cross-compilers like GWT and the good side of MS Volta (vs. the bad side) the next abstraction layer. I expect these will be more significant in the future

On the same note, it is probably good time to provide a link to a presentation on the future of PCs I’ve created in 2007, which highlights some of the trends that are more apparent today in moving to the cloud

* the most obvious example is of course search (altavista, hotbot etc. have all been there before..), a more recent example is the free GPS turn-by-turn where Waze (an Israeli company by they way) is already there but Google’s announcement is considered disruptive

More on WCF oddities

Arnon Rotem-Gal-Oz — Sun, 15 Nov 2009 12:07:43 GMT

Jesse Ezell left the following comment on my previous WCF rant (Windows Trick-or-treat Foundation)

You wouldn't expect WCF to take care every TCP/IP registry setting as well would you? At some point, the things WCF exposes have to stop so transport specific settings come into play. What would really suck is if WCF completely abstracted every detail of every transport and came up with new names for things like cookies and specific http request and response headers in the name of creating a "unified" experience across every transport.
The point of WCF is to give you a unified communication API. You don't have to change your code to switch from HTTP, to MSMQ, to TCP, etc. However, the point of WCF is not to hide all the specifics of each transport from you. There are a lot of transport specific options in WCF that you won't find in a WSDL file. However... none of these settings actually require you to change any code to take advantage of them. IMO, WCF does a really good job of providing a uniform communication API and limiting transport specific details to configuration settings. Even in this case, you were able to resolve the issue by configuration settings rather than code changes.
If you had used http request manually, this is one of very few settings you would actually have control over via configuration. What about other popular competing APIs? Look at something like NServiceBus. NServiceBus has a transport model that abstracts communication to some degree... but what happens when you want to change the format of the messages on the wire? For example, maybe you want to switch from raw XML messages to messages with SOAP envelopes on an endpoint or limit the depth of XML node hierarchies to protect against XML attacks. Maybe you want to switch to a completely new transport that was provided by a vendor that has never seen your product. Maybe you want to add compression, or chunking, or certificate based security. Can you do that all via configuration files without ever touching code in any other .NET API? Maybe this is my own ignorance... but I don't know of any .NET communication API that offers even half of the flexibility of WCF.

Well, I would actually expect WCF to do one of two things either provide a complete API of all the communications need (next version of .NET communication , unified communication model and all that..) and retire the other .NET communication libraries or on the other hand provide a thin layer of abstraction that will make it clear you need to move to the specific underlying protocols.

What we’ve got now is something that isn’t quite there on the first and way away from the second – which means that when you try to do serious stuff with WCF you hit these unexpected (ok now they are) snags where you don’t know where to go – until you realize that this is a specific thing regarding TCP this or HTTP that which is not readily apparent and is not well documented or even worse you need to set it outside of WCF altogether.

In my experience , you can’t in fact, “just change the binding” and expect everything to work unless you are doing very simple stuff. For instance when if you move from HTTP binding to TCP you’d find that the channels are suddenly getting closed after periods of inactivity and you need to “keep them alive” or if you move in the other direction from TCP to HTTP you’d find that the size of messages gets larger by an order of magnitude etc.

Not to mention the “training wheels” approach to setting defaults (at least some of it is fixed for .NET 4) which I talked about a few times in the past. Also there cryptic error messages that make you scratch your head looking for the configuration item you need to set. for instance if your send a large message (>8K) you won’t see any problem in the sending side but you’d get the following

“An exception of type 'System.ServiceModel.Dispatcher.NetDispatcherFaultException' occurred in mscorlib.dll but was not handled in user code

Additional information: The formatter threw an exception while trying to deserialize the message: There was an error while trying to deserialize parameter http://tempuri.org/:ClientPrintResult. The InnerException message was 'There was an error deserializing the object of type System.String. The maximum string content length quota (8192) has been exceeded while reading XML data. This quota may be increased by changing the MaxStringContentLength property on the XmlDictionaryReaderQuotas object used when creating the XML reader. Line 2, position 40523.'. Please see InnerException for more details.”

I am sure all of you immediately understood you need to set ReaderQuotas, MaximumReceiveMessageSize and MaxBuferSize on the binding e.g.:


            var binding = new WebHttpBinding()
{ ReaderQuotas = { MaxArrayLength = 20 * 8192}, MaxReceivedMessageSize = 20 * 8192,
MaxBufferSize = 20 * 8192 };

Don’t get me wrong, WCF isn’t all bad or anything like that but it does get annoying like hell at times.

As for the reference to other frameworks - I can’t speak for NServiceBus because I didn’t write it (though I think the answer would be the same) - but if I consider the communication framework I did write for xsights (which builds on WCF by the way) it is not as presumptuous as WCF. It does not pretend to be an all-encompassing communication layer for .NET and It is build to provide a specific architectural approach – which is why it shouldn’t be judged by the same standards.

SOA – There could be only one…

Arnon Rotem-Gal-Oz — Fri, 13 Nov 2009 17:42:48 GMT

There’s no Architecture in Business Service Orientation ! There, I’ve said it, there are no two types of SOA. I am not trying to say that business-level service orientation doesn’t exist or isn’t valuable. However I am trying to say that labeling that SOA harms both Service Orientation at the business level and SOA (a.k.a. “technical SOA”)

For the record here are my definitions for both Business Service Orientation and Service Oriented Architecture (SOA)

Business Service Orientation is an IT paradigm at the enterprise-level that aims to componentize and partition the business’s software and to get composability and flexibility (and thus achieve better business and IT alignment etc.). Service Orientation can be implemented using various enterprise architecture practices (around governance, portfolio management etc) as well as various software architectures including (but not limited to ) SOA, EDA, BPM, REST and combinations of them.

Service Oriented Architecture as an architectural style for building systems based on interacting coarse grained autonomous components called services. Each service expose processes and behavior through contracts, which are composed of messages at discoverable addresses called endpoints. Services’ behavior is governed by policies which are set externally to the service itself. SOA is derived of four predating architectural styles, namely Client/Server, Layered System, Pipes and Filters and Distributed Agents.

To reiterate - calling Business Service Orientation SOA serves only to muddy the water and make both terms nebulous. If you just have to have a TLA just call them BSO & SOA

I know that in “What is SOA anyway” I also refer to two SOAs. I didn’t have enough confidence to say it bluntly when I wrote that paper, but I did emphasized Service Orientation for BSO and Architecture for SOA. Also regarding a more formal definition of SOA, that shows how it is derived from the four other styles – I started explaining that quite some time ago (Intro, Client-Server, Layered, Pipes and Filters) – I still need to explain Distributed Agents and summarize (about time I’ll do that)

Windows Trick-or-treat Foundation

Arnon Rotem-Gal-Oz — Fri, 30 Oct 2009 22:58:53 GMT

Yes, this is another WCF rant…

We were getting ready to launch an open-for-all version of our service, we were also adding more cores to the system, to make sure our computation engine will be able to handle higher request loads (We’re basically implementing the Gridable Service pattern - if it is interesting, I can expand on that in another post). We tried a few load tests, which on first look, seemed OK. However we then noticed that we get WCF timeouts on some of the calls.

WCF timeouts!? what gives? we already took care of all the annoying throttling defaults. After busting my head (and Google) for a few hours, I found that apparently, when using http bindings in WCF you only get 2 outgoing channels for each server (IP) you are connecting to. yep WCF thinks your services are novice users using a browser. So if you have a service that tries to access a remote server with more than two requests simultaneously (say, under load…) you may find, like us, that you’d get occasional timeouts.

Yes, there’s also a solution, which also took time to find – it is called ConnectionManagment element in the network configuration (I am yet to find a programmable way to do this). So now the services app.config (see below) sets the value to 16 instead of 2 and everything is well again, at least until the next default hits us..


          <
          system.net
          >
<
          connectionManagement
          >
<
          add 
          address 
          = "*" maxconnection = "16" />
</connectionManagement>
</system.net>

From trying to work with WCF in the last few years, it seems its abstractions are very leaky.They are almost leaky to the point of rendering it useless as a “unified” framework. So, in the spirit of the times, and as the title suggest – maybe they should just rename WCF to Windows Trick-or-treat Foundation – Alas, we are getting more and more of the “trick’ part rather that the “treat”, but at least the acronym fits well.

Offtopic: xsights – now open for all

Arnon Rotem-Gal-Oz — Tue, 20 Oct 2009 22:08:17 GMT

When we added our iPhone application xsights-light to the appstore about a month ago we intended it as a demo application we can use to showcase potential clients what we can do (with the aim of providing a white-label service). We activated a few images (3 on the site and a few others) and forgot about it.

We were very surprised to find out that even though we didn’t publicized it anywhere or anything like that, hundreds of people found it interesting, downloaded the app and tried it on just about anything. Here are just a few examples:

As I mentioned above, we only had a few images activated, so naturally, people weren’t very happy and we got some “rave” reviews like “would be nice if it worked as advertized”, “only identifies 3 images” and others that used less refined language :)

Anyway, we’re listening – and we decided to make xsights-light a little more useful. For one we are adding content, we started with movie posters of movies playing in the US and we intend to add more in the future. What’s more interesting, is that we’re opening up the platform for general use i.e. you can upload (almost) any image you like (such as the images above), add a URL and activate anything you like be that a Pepsi can, an wedding invitation or whatnot (if you’d take a look at @xsightspulse on twitter, you can see what others are activating)

Apparently, we’re not the only ones who think this is interesting, as we also got a nice mention on Tech-Crunch. We agreed to let Tech-Crunch readers be the first users of our service so head over to the article for an invite, we hope to open it for general use in a week or so.

SOA Patterns presentation on E-VAN (recording)

Arnon Rotem-Gal-Oz — Tue, 06 Oct 2009 21:57:06 GMT

Yesterday I gave a talk on SOA pattern on the European Virtual Alt.Net user group. You can find the recording of that talk here as well as download a pdf of the slides.

Before I’ll talk a little about the substance I want to say a few words about office-live meeting (the platform used for the presentation). To sum this in one word the experience was horrid. It took me more than 35 minutes just to upload my presentation. Then I had to switch to windows XP (VM in parallels) to speak since it has problem with Windows 7 (low sound volume). However, the worst thing is that throughout the presentation I constantly lost control of the slides progress (i.e. couldn’t move the slides forward), which was very distracting.

Anyway, if ignoring all that, I think overall the presentation is still beneficial and addresses a few interesting issues that are challenging like flexibility, reporting and management of SOA. If I am to sum the presentation I’d say that when you build a system on SOA you get a system built of (relatively) a lot of components of questionable reliability. You can reap a lot of benefits in the flexibility department, but you have to address several challenges in the performance, availability, management (etc.) departments. Additionally you need to look at the overall solution from an holistic viewpoint since different parts of the solution can push in different direction or only cover part of the picture.

Lastly thanks to Jan and Colin for organizing the event and for all the attendees for giving me an hour and half of their time