<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" media="screen" href="/~d/styles/rss2full.xsl"?><?xml-stylesheet type="text/css" media="screen" href="http://feeds.feedburner.com/~d/styles/itemcontent.css"?><rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:wfw="http://wellformedweb.org/CommentAPI/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:sy="http://purl.org/rss/1.0/modules/syndication/" xmlns:slash="http://purl.org/rss/1.0/modules/slash/" xmlns:feedburner="http://rssnamespace.org/feedburner/ext/1.0" version="2.0">

<channel>
	<title>Cirrus Minor</title>
	
	<link>http://arnon.me</link>
	<description>Musings of a holistic architect by Arnon Rotem-Gal-Oz</description>
	<lastBuildDate>Fri, 04 May 2012 03:24:58 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
			<atom10:link xmlns:atom10="http://www.w3.org/2005/Atom" rel="self" type="application/rss+xml" href="http://feeds.feedburner.com/CirrusMinor" /><feedburner:info uri="cirrusminor" /><atom10:link xmlns:atom10="http://www.w3.org/2005/Atom" rel="hub" href="http://pubsubhubbub.appspot.com/" /><item>
		<title>Utility trees and quality attributes</title>
		<link>http://feedproxy.google.com/~r/CirrusMinor/~3/EGFaG_ZiBCY/</link>
		<comments>http://arnon.me/2012/04/utility-trees-quality-attributes/#comments</comments>
		<pubDate>Fri, 27 Apr 2012 12:17:52 +0000</pubDate>
		<dc:creator>Arnon Rotem-Gal-Oz</dc:creator>
				<category><![CDATA[Blog]]></category>
		<category><![CDATA[SAF]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[ATAM]]></category>
		<category><![CDATA[Design]]></category>
		<category><![CDATA[non functional requirements]]></category>
		<category><![CDATA[quality attributes]]></category>
		<category><![CDATA[Software Architecture]]></category>
		<category><![CDATA[software quality metrics]]></category>

		<guid isPermaLink="false">http://arnon.me/?p=617</guid>
		<description><![CDATA[I recently answered this question in Stackoverflow : What is an utility tree and what is it&#8217;s purpose in case of Architecture tradeoff analysis method(ATAM)?  I did answer the question there but here&#8217;s a better explanation with lots of examples base on the initial version for chapter 1 of SOA Patterns (which didn&#8217;t make it into [...]]]></description>
			<content:encoded><![CDATA[<p>I recently answered <a href="http://stackoverflow.com/questions/10152034/">this question</a> in Stackoverflow :</p>
<blockquote><p>What is an utility tree and what is it&#8217;s purpose in case of Architecture tradeoff analysis method(ATAM)?</p></blockquote>
<div> I did answer the question there but here&#8217;s a better explanation with lots of examples base on the initial version for chapter 1 of <a href="http://www.manning.com/rotem">SOA Patterns</a> (which didn&#8217;t make it into the final version of the book).</div>
<div>
<p>There are two types of requirements for  software projects: functional and non-functional requirements. Functional requirements are the requirements for what the solution must do (which  are usually expressed as use cases or stories). The functional requirements are what the users (or systems) that interact with the system do with the system (fill in an order, update customer details, authorize a loan etc.).</p>
<p>Non-Functional requirements are attributes the system is expected to have or manifest. These usually include requirements in areas such as performance, security, availability etc. A better name for non-functional requirements is &#8220;Quality Attributes&#8221; . Below are some formal definitions from IEEE standad 1061 “Standard for a Software Quality Metrics Methodology”  for quality attributes and related terms:</p>
<blockquote>
<ul>
<li><strong>Quality attribute</strong> A characteristic of software, or a generic term applying to quality factors, quality subfactors, or metric values.</li>
<li><strong>Quality factor</strong> A management-oriented attribute of software that contributes to its quality.</li>
<li><strong>Quality subfactor</strong> A decomposition of a quality factor or quality subfactor to its technical components.</li>
<li><strong>Metric value</strong> A metric output or an element that is from the range of a metric.</li>
<li><strong>Software quality metric</strong> A function whose inputs are software data and whose output is a single numerical value that can beinterpreted as the degree to which software possesses a given attribute that affects its quality.</li>
</ul>
</blockquote>
<p>Most of the requirements that drive the design of a software architecture comes from system&#8217;s quality attributes. The reason for this is that that the effect of quality attributes is usually system-wide (e.g. you wouldn&#8217;t want your system to have good performance only in the UI – you want the system to perform well no matter what) &#8211;  which is exactly what software architecture is concerned with. Note however, that few requirements might still come from functional requirements)<a title="" href="#_ftn1">[1]</a>.  The question is how do we find out what those requirements are?</p>
<p>The answer to that is also in the software architecture definition. The source for quality attributes are the stakeholders. So what or who are these &#8220;stakeholders&#8221;? Well, a stakeholder is just about anyone who has a vested interest in the project. A typical system has a lot of stakeholders starting from the (obvious) customer, the end-users (those people in the customer organization/dept that will actually use the software) and going to the operations personnel (IT – those who will have to keep the solution running), the development team, testers, maintainers, management. In some systems the stakeholders can even be the shareholders or even the general public (Imagine for example, that you build a new dispatch system for a 911 center).</p>
<p>One of the architect&#8217;s roles is to analyze the quality attributes and define an architecture that will enable delivering all the functional requirements while supporting the quality attributes. As can be expected ,sometimes quality attributes  are in conflict with each other – the most obvious examples are performance vs. security or flexibility vs. simplicity and the architect&#8217;s role is to strike a balance between the different quality attributes (and the stakeholders) to make sure the overall quality of the system is maximized.</p>
<p>Contextual solutions (e.g. patterns) can be devised to solve specific quality attributes need. However saying that a system needs to have &#8220;good performance&#8221; or that it needs to be &#8220;testable&#8221; doesn&#8217;t really help us know what to do. In order for us to be able to discern which patterns apply to specific quality attribute ,  we need a better understanding of quality attributes besides the formal definition, something that is more concrete.</p>
<p>The way to get that concrete understanding of the effect of quality attributes is to use scenarios. Scenarios are short, &#8220;user story&#8221;-like proses that demonstrate how a quality attribute is manifested in the system using a functional situation</p>
<p>Quality attributes scenarios originated as a way to evaluate software architecture. The Software Engineering Institute developed several evaluation methodologies, like Architecture Tradeoff Analysis Method  (Clements, Kazman and Klein, 2002) that heavily build on scenarios to contrast and compare how the different quality attributes are met by candidate architectures. ATAM (and similar evaluation methods like LAAAM which is part of MSF 4.0) suggest building a &#8220;utility tree&#8221; which represent the overall usefulness of the system. The scenarios serve as the leafs of the utility tree and the architecture is evaluated by considering how the architecture makes the scenarios possible.</p>
<p>I found that using scenarios and the utility tree approach early in the design of the architecture (see writings about<a href="http://arnon.me/spammed-architecture-framework-saf/">SAF</a>) can greatly enhance the quality of the architecture that is produced. When you examine  the scenarios you can also  prioritize them and better balance conflicting attributes.</p>
<p>The scenarios can be used as an input to make sure the quality attributes are actually met.  Furthermore  you can use the scenarios to help identify the strategies or patterns  applicable to make the scenarios possible (and thus ensure the quality attributes are met) within the  system. We usually group scenarios into a &#8220;utility tree&#8221; which is a representation of the total usefulness (&#8220;utility&#8221;) of a system . As you can see in the diagram below we have the   key quality attributes (Performance, Security etc.). Each of the quality attributes has sub categories (e.g. Performance is broken into latency, data loss etc.). Each sub category is demonstrated by a scenario that we expect the system to manifest.</p>
<p><a href="http://arnon.me/wp-content/uploads/2012/04/utree.png"><img class="aligncenter size-full wp-image-620" title="utree" src="http://arnon.me/wp-content/uploads/2012/04/utree.png" alt="" width="468" height="174" /></a></p>
<div></div>
<p>The tree representation helps get the whole picture but the important bits here are the scenarios so let&#8217;s explore them some more. Scenarios are expressed as statements that have  3 parts: a <strong>stimulus</strong>, a <strong>context</strong> and a <strong>response</strong>. The  stimulus is the action taken (by the system / user/ other system / any other person); response is how the system is expected to behave when the stimulus occur, and the context specifies  the environment or conditions under which we expect the to get the response. For example in the following scenario: &#8220;When you perform a database operation , under normal condition, it should take less than 100 miliseconds.&#8221;</p>
<ul>
<li>&#8220;Under normal condition&#8221; is the <strong>context</strong></li>
<li>&#8220;When you perform a database operation&#8221; is the <strong>stimulus</strong></li>
<li>&#8220;it should take less than 100 millisecond&#8221; is the <strong>response</strong> expected from the system.</li>
</ul>
<p>&nbsp;</p>
<p>Here are a couple of  additional examples for quality attribute scenarios:</p>
<ul>
<li>Performance –&gt;Latency -&gt; under normal conditions a client consuming multiple services should have latency less than 5 seconds.</li>
<li>Security-&gt;Authentications -&gt; Under all conditions, any call to a service should be authenticated using X.509 certificate</li>
</ul>
<p>You can also check out <a href="http://arnon.me/wp-content/uploads/2012/04/Sample-Quality-Attributes.docx">this document</a> for a few more scenario examples from a system I worked on in the past</p>
<div>
<p>&nbsp;</p>
<hr align="left" size="1" width="50%" />
<div><a title="" href="#_ftnref1">[1]</a> Design has the ratios reversed  i.e. most of the requirements for design come from functional requirements and a few requirements might come from the quality attributes.</div>
</div>
<p>illustration by <a href="http://www.flickr.com/photos/epsos/3406617569/sizes/s/in/photostream/">epsos.de</a></p>
</div>
<p>&nbsp;</p>
<script src="http://feeds.feedburner.com/~s/arnonrgo?i=http://arnon.me/2012/04/utility-trees-quality-attributes/" type="text/javascript" charset="utf-8"></script><div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=EGFaG_ZiBCY:yjQ3Eng9EME:F7zBnMyn0Lo"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=EGFaG_ZiBCY:yjQ3Eng9EME:F7zBnMyn0Lo" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=EGFaG_ZiBCY:yjQ3Eng9EME:cGdyc7Q-1BI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=cGdyc7Q-1BI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=EGFaG_ZiBCY:yjQ3Eng9EME:G79ilh31hkQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=G79ilh31hkQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=EGFaG_ZiBCY:yjQ3Eng9EME:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=EGFaG_ZiBCY:yjQ3Eng9EME:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=EGFaG_ZiBCY:yjQ3Eng9EME:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=EGFaG_ZiBCY:yjQ3Eng9EME:gIN9vFwOqvQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=EGFaG_ZiBCY:yjQ3Eng9EME:I9og5sOYxJI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=I9og5sOYxJI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=EGFaG_ZiBCY:yjQ3Eng9EME:-BTjWOF_DHI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=EGFaG_ZiBCY:yjQ3Eng9EME:-BTjWOF_DHI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=EGFaG_ZiBCY:yjQ3Eng9EME:qj6IDK7rITs"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=qj6IDK7rITs" border="0"></img></a>
</div><img src="http://feeds.feedburner.com/~r/CirrusMinor/~4/EGFaG_ZiBCY" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://arnon.me/2012/04/utility-trees-quality-attributes/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		<feedburner:origLink>http://arnon.me/2012/04/utility-trees-quality-attributes/</feedburner:origLink></item>
		<item>
		<title>Develop Map/Reduce with reduced assumptions</title>
		<link>http://feedproxy.google.com/~r/CirrusMinor/~3/2DXWSm_3kjM/</link>
		<comments>http://arnon.me/2012/04/develop-mapreduce-reduced-assumptions/#comments</comments>
		<pubDate>Mon, 16 Apr 2012 20:26:34 +0000</pubDate>
		<dc:creator>Arnon Rotem-Gal-Oz</dc:creator>
				<category><![CDATA[Blog]]></category>
		<category><![CDATA[debugging]]></category>
		<category><![CDATA[Hadoop]]></category>
		<category><![CDATA[Java]]></category>
		<category><![CDATA[map/reduce]]></category>
		<category><![CDATA[partitioner]]></category>
		<category><![CDATA[pig]]></category>

		<guid isPermaLink="false">http://arnon.me/?p=601</guid>
		<description><![CDATA[It all started with this odd bug&#8230; One of our teams is writing a service, that among other things, runs map/reduce jobs built as pig scripts with Java UDFs. The scripts accepts CSV files which have a text header followed by lines of data. It performs some grouping and then calls a UDF which essentially [...]]]></description>
			<content:encoded><![CDATA[<p>It all started with this odd bug&#8230;</p>
<p>One of our teams is writing a service, that among other things, runs map/reduce jobs built as <a href="http://pig.apache.org/docs/r0.9.2/">pig</a> scripts with Java UDFs. The scripts accepts CSV files which have a text header followed by lines of data. It performs some grouping and then calls a UDF which essentially filter, enrich and transforms the data outputting another CSV with a new header followed by data &#8211; something like the following:</p>

<div class="wp_syntax"><table><tr><td class="line_numbers"><pre>1
2
3
4
</pre></td><td class="code"><pre class="pig" style="font-family:monospace;">input = load '/data/INPUT2.dat' using PigStorage(',')...
grouped = GROUP input BY key;
results = FOREACH grouped GENERATE  evaluateUDF(input) as output;
STORE output...</pre></td></tr></table></div>

<p>and all was well. The job spawns a few map tasks partitions, groups the data and runs a single reduce where the actual evaluation happens.</p>
<p>Then someone wanted it to run faster &#8211; We can do that by adding reducers we can do that by adding PARALLEL X to the group statement where X is the number of reducers we want. And then the trouble began. Everything worked well for up to 5 reducers &#8211; go any higher and few results were lost. The script is pretty basic (the actual script looks a little different but that&#8217;s basically it) the UDF is not earth shattering yet still sometimes results are lost.</p>
<p>Even though I am very busy (well, not really, but you know, some members of my team might be reading this&#8230;) this was so annoying I just had to understand what happen so I took it upon myself to debug this. The main problem was I couldn&#8217;t write a unit test to duplicate the behavior  - it only occurred when running on the large file&#8230;</p>
<p>First thing I noticed is that we had duplicate results in the output, I removed those and saw that actually we&#8217;re only losing one result (again, only with 5 or more reducers). I diffed the  correct output with faulted one and I even had the problematic key.</p>
<p>I told myself something must be wrong with the partitioner  - The default partitioner that pig (and hadoop in general) uses is fairly simple it takes the hash code of the key, ands it with max int (so we get a positive number) and divides that by the number of partitions:</p>

<div class="wp_syntax"><table><tr><td class="line_numbers"><pre>1
</pre></td><td class="code"><pre class="java" style="font-family:monospace;"> <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #009900;">&#40;</span>key.<span style="color: #006633;">hashCode</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">&amp;</span> <span style="color: #003399;">Integer</span>.<span style="color: #006633;">MAX_VALUE</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">%</span> numPartitions<span style="color: #339933;">;</span></pre></td></tr></table></div>

<p>I couldn&#8217;t really see what might be wring with this but still I wrote my own instead:</p>

<div class="wp_syntax"><table><tr><td class="line_numbers"><pre>1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
</pre></td><td class="code"><pre class="java" style="font-family:monospace;"> 
<span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">class</span> Partition <span style="color: #000000; font-weight: bold;">extends</span> Partitioner <span style="color: #009900;">&#123;</span>
    <span style="color: #666666; font-style: italic;">//@Override</span>
    <span style="color: #000000; font-weight: bold;">private</span> <span style="color: #000000; font-weight: bold;">final</span> <span style="color: #000000; font-weight: bold;">static</span> Logger logger <span style="color: #339933;">=</span> LoggerFactory.<span style="color: #006633;">getLogger</span><span style="color: #009900;">&#40;</span>Partition.<span style="color: #000000; font-weight: bold;">class</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000066; font-weight: bold;">int</span> getPartition<span style="color: #009900;">&#40;</span>PigNullableWritable key, Writable value, <span style="color: #000066; font-weight: bold;">int</span> numPartitions<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">try</span> <span style="color: #009900;">&#123;</span>
            <span style="color: #000000; font-weight: bold;">if</span><span style="color: #009900;">&#40;</span>key.<span style="color: #006633;">getValueAsPigType</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #000000; font-weight: bold;">instanceof</span> <span style="color: #003399;">String</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
                <span style="color: #003399;">String</span> skey<span style="color: #339933;">=</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#40;</span><span style="color: #003399;">String</span><span style="color: #009900;">&#41;</span>key.<span style="color: #006633;">getValueAsPigType</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span>.<span style="color: #006633;">replaceAll</span><span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;<span style="color: #000099; font-weight: bold;">\&quot;</span>&quot;</span>,<span style="color: #0000ff;">&quot;&quot;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                <span style="color: #003399;">Long</span> ikey<span style="color: #339933;">=</span> <span style="color: #003399;">Long</span>.<span style="color: #006633;">getLong</span><span style="color: #009900;">&#40;</span>skey<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">int</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#40;</span>ikey<span style="color: #339933;">&amp;</span> <span style="color: #003399;">Long</span>.<span style="color: #006633;">MAX_VALUE</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">%</span> numPartitions<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            <span style="color: #009900;">&#125;</span>
            <span style="color: #000000; font-weight: bold;">else</span> <span style="color: #009900;">&#123;</span>
                <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #009900;">&#40;</span>key.<span style="color: #006633;">hashCode</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">&amp;</span> <span style="color: #003399;">Integer</span>.<span style="color: #006633;">MAX_VALUE</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">%</span> numPartitions<span style="color: #339933;">;</span>
            <span style="color: #009900;">&#125;</span>
        <span style="color: #009900;">&#125;</span>
        <span style="color: #000000; font-weight: bold;">catch</span> <span style="color: #009900;">&#40;</span><span style="color: #003399;">Exception</span> e <span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
            <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
    <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#125;</span></pre></td></tr></table></div>

<p>Problem solved &#8211; everything works.  I guess that damn Hadoop partitioner is all flawed &#8211; not! &#8211; So now I am even more annoyed because I really don&#8217;t understand what&#8217;s going on here. but at least now I control the partitioner so I added some logging. I saw that the problematic key goes to partition 0 &#8211; so I made everything go to partition 0 and everything worked. then it finally stuck me. I logged all the keys that go into partition 0 and saw that the problematic key is the first .   Duh!  I should have noticed that earlier. Here&#8217;s what the UDF code essentially looks like:</p>

<div class="wp_syntax"><table><tr><td class="line_numbers"><pre>1
2
3
4
5
6
7
8
9
10
</pre></td><td class="code"><pre class="java" style="font-family:monospace;">	<span style="color: #000000; font-weight: bold;">public</span> DataBag exec<span style="color: #009900;">&#40;</span>Tuple input<span style="color: #009900;">&#41;</span> <span style="color: #000000; font-weight: bold;">throws</span> <span style="color: #003399;">IOException</span>
	<span style="color: #009900;">&#123;</span>
            <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>session <span style="color: #339933;">==</span> <span style="color: #000066; font-weight: bold;">null</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#123;</span>
                 session <span style="color: #339933;">=</span> ....
                 <span style="color: #000000; font-weight: bold;">return</span> addHeadersNames<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            <span style="color: #009900;">&#125;</span>
&nbsp;
            <span style="color: #000000; font-weight: bold;">return</span>  eval<span style="color: #009900;">&#40;</span>input<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
	<span style="color: #009900;">&#125;</span></pre></td></tr></table></div>

<p>The Tuple is the input we get from the pig script into our UDF, what happens here is that we throw out the first input we get and just output the headers for our result file. As it happens when we have enough partitions a significant input (the missing result) is the first we get in the partition</p>
<p>The developer who wrote this assumed that the first input line the UDF would get would be the input header file. It is alright to discard it and output the new header file instead. However Hadoop doesn&#8217;t work like that when we have code that runs in a map/reduce it might be called in different circumstances that we originally assumed and we have to pay attention to that. For instance, another problem with the code above is that when run with multiple reducers you get multiple copies of the output header&#8230;</p>
<p>So there you have, the title says it all : develop map/reduce code with reduced assumptions</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<script src="http://feeds.feedburner.com/~s/arnonrgo?i=http://arnon.me/2012/04/develop-mapreduce-reduced-assumptions/" type="text/javascript" charset="utf-8"></script><div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=2DXWSm_3kjM:rfFUp_97zhQ:F7zBnMyn0Lo"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=2DXWSm_3kjM:rfFUp_97zhQ:F7zBnMyn0Lo" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=2DXWSm_3kjM:rfFUp_97zhQ:cGdyc7Q-1BI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=cGdyc7Q-1BI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=2DXWSm_3kjM:rfFUp_97zhQ:G79ilh31hkQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=G79ilh31hkQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=2DXWSm_3kjM:rfFUp_97zhQ:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=2DXWSm_3kjM:rfFUp_97zhQ:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=2DXWSm_3kjM:rfFUp_97zhQ:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=2DXWSm_3kjM:rfFUp_97zhQ:gIN9vFwOqvQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=2DXWSm_3kjM:rfFUp_97zhQ:I9og5sOYxJI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=I9og5sOYxJI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=2DXWSm_3kjM:rfFUp_97zhQ:-BTjWOF_DHI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=2DXWSm_3kjM:rfFUp_97zhQ:-BTjWOF_DHI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=2DXWSm_3kjM:rfFUp_97zhQ:qj6IDK7rITs"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=qj6IDK7rITs" border="0"></img></a>
</div><img src="http://feeds.feedburner.com/~r/CirrusMinor/~4/2DXWSm_3kjM" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://arnon.me/2012/04/develop-mapreduce-reduced-assumptions/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		<feedburner:origLink>http://arnon.me/2012/04/develop-mapreduce-reduced-assumptions/</feedburner:origLink></item>
		<item>
		<title>SOA Patterns status update</title>
		<link>http://feedproxy.google.com/~r/CirrusMinor/~3/2ELxkv0t88o/</link>
		<comments>http://arnon.me/2012/03/soa-patterns-status-update/#comments</comments>
		<pubDate>Fri, 16 Mar 2012 20:27:12 +0000</pubDate>
		<dc:creator>Arnon Rotem-Gal-Oz</dc:creator>
				<category><![CDATA[Blog]]></category>
		<category><![CDATA[Featured Posts]]></category>
		<category><![CDATA[SOA Patterns]]></category>
		<category><![CDATA[anti-patterns]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[book]]></category>
		<category><![CDATA[Service Oriented Architecture]]></category>
		<category><![CDATA[SOA]]></category>

		<guid isPermaLink="false">http://arnon.me/?p=573</guid>
		<description><![CDATA[I want to say I am delighted but I guess relieved is the right word here  - I&#8217;ve finally submitted all the book chapters to manning and I&#8217;m now doing an editing round on all the chapters (re-adding missing images, update pattern references, fixing typos etc.). &#160; &#160; &#160; &#160; The current (and hopefully final) table of contents  is [...]]]></description>
			<content:encoded><![CDATA[<p>I want to say I am delighted but I guess relieved is the right word here  - I&#8217;ve finally submitted all the book chapters to manning and I&#8217;m now doing an editing round on all the chapters (re-adding missing images, update pattern references, fixing typos etc.).</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>The current (and hopefully final) table of contents  is as follows:</p>
<ul>
<li>Part I &#8211; SOA Patterns</li>
<ul>
<li>Chapter 1 &#8211; Solving SOA pains with patterns &#8211; an introduction</li>
<li>Chapter 2 &#8211; Basic structural patterns</li>
<li>Chapter 3 &#8211; Performance, scalability and availability patterns</li>
<li>Chapter 4 &#8211; Security and manageability patterns</li>
<li>Chapter 5 &#8211; Message exchange patterns</li>
<li>Chapter 6 &#8211; Service consumer interaction patterns</li>
<li>Chapter 7 &#8211; Service integration patterns</li>
</ul>
<li>part II &#8211; SOA in the real world</li>
<ul>
<li>Chapter 8 &#8211; SOA anti-patterns</li>
<li>Chapter 9 &#8211; Putting it all together &#8211; a case study</li>
<li>Chapter 10 &#8211; SOA vs. the world &#8211; how SOA fits with other techs:  REST, Cloud and Big data</li>
</ul>
</ul>
<div>As part of the update round I&#8217;ve added a patten map that shows some of the relations between the pattern which you can see below (click for larger image).</div>
<p><a href="http://arnon.me/wp-content/uploads/2012/03/patternmap.jpg"><img class="aligncenter size-medium wp-image-574" title="patternmap" src="http://arnon.me/wp-content/uploads/2012/03/patternmap-300x216.jpg" alt="" width="300" height="216" /></a></p>
<p>I suppose it would take some more time before the final version of the book will be available (it is getting there&#8230;) but I guess  this is also a good time to thank the thousands of you who bought this book  via early access even though it was far from finished.</p>
<script src="http://feeds.feedburner.com/~s/arnonrgo?i=http://arnon.me/2012/03/soa-patterns-status-update/" type="text/javascript" charset="utf-8"></script><div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=2ELxkv0t88o:7RPWYfWHjUw:F7zBnMyn0Lo"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=2ELxkv0t88o:7RPWYfWHjUw:F7zBnMyn0Lo" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=2ELxkv0t88o:7RPWYfWHjUw:cGdyc7Q-1BI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=cGdyc7Q-1BI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=2ELxkv0t88o:7RPWYfWHjUw:G79ilh31hkQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=G79ilh31hkQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=2ELxkv0t88o:7RPWYfWHjUw:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=2ELxkv0t88o:7RPWYfWHjUw:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=2ELxkv0t88o:7RPWYfWHjUw:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=2ELxkv0t88o:7RPWYfWHjUw:gIN9vFwOqvQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=2ELxkv0t88o:7RPWYfWHjUw:I9og5sOYxJI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=I9og5sOYxJI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=2ELxkv0t88o:7RPWYfWHjUw:-BTjWOF_DHI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=2ELxkv0t88o:7RPWYfWHjUw:-BTjWOF_DHI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=2ELxkv0t88o:7RPWYfWHjUw:qj6IDK7rITs"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=qj6IDK7rITs" border="0"></img></a>
</div><img src="http://feeds.feedburner.com/~r/CirrusMinor/~4/2ELxkv0t88o" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://arnon.me/2012/03/soa-patterns-status-update/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		<feedburner:origLink>http://arnon.me/2012/03/soa-patterns-status-update/</feedburner:origLink></item>
		<item>
		<title>REST with JAX-RS and silverlight clients</title>
		<link>http://feedproxy.google.com/~r/CirrusMinor/~3/pGLG4JMzbFU/</link>
		<comments>http://arnon.me/2011/12/rest-jaxrs-silverlight-clients/#comments</comments>
		<pubDate>Mon, 05 Dec 2011 19:37:26 +0000</pubDate>
		<dc:creator>Arnon Rotem-Gal-Oz</dc:creator>
				<category><![CDATA[Blog]]></category>
		<category><![CDATA[.NET]]></category>
		<category><![CDATA[C#]]></category>
		<category><![CDATA[Java]]></category>
		<category><![CDATA[JAX-RS]]></category>
		<category><![CDATA[REST]]></category>
		<category><![CDATA[Silverlight]]></category>

		<guid isPermaLink="false">http://arnon.me/?p=556</guid>
		<description><![CDATA[I am Working on a RESTful  service using Jersey (not my first choice, but a reasonable compromise). It is hosted inside FuseESBand all is well. Here&#8217;s an overly simplified version of the resource that reports status: 1 2 3 4 5 6 7 8 9 10 11 @Path&#40;&#34;/stat&#34;&#41; public class StatusResource &#123; @GET @Produces&#40;MediaType.TEXT_PLAIN&#41; public String [...]]]></description>
			<content:encoded><![CDATA[<p>I am Working on a RESTful  service using <a href="http://jersey.java.net/">Jersey</a> (not my first choice, but a reasonable compromise). It is hosted inside <a href="http://fusesource.com/products/enterprise-servicemix/">FuseESB</a>and all is well. Here&#8217;s an overly simplified version of the resource that reports status:</p>

<div class="wp_syntax"><table><tr><td class="line_numbers"><pre>1
2
3
4
5
6
7
8
9
10
11
</pre></td><td class="code"><pre class="java" style="font-family:monospace;">@Path<span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;/stat&quot;</span><span style="color: #009900;">&#41;</span>
<span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">class</span> StatusResource <span style="color: #009900;">&#123;</span>
    @GET
    @Produces<span style="color: #009900;">&#40;</span>MediaType.<span style="color: #006633;">TEXT_PLAIN</span><span style="color: #009900;">&#41;</span>
      <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">String</span> getStatus<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>checkStatus<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span>
            <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #0000ff;">&quot;Ok&quot;</span><span style="color: #339933;">;</span>
         <span style="color: #000000; font-weight: bold;">else</span>
            <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #0000ff;">&quot;Not so good&quot;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#125;</span></pre></td></tr></table></div>

<p>Then we tried to connect a silverlight client to this server (don&#8217;t ask me why).  The first hurdle was a security one. Silverlight doesn&#8217;t like to communicate with a server it didn&#8217;t launch from. To resolve that you need to <a href="http://msdn.microsoft.com/en-us/library/cc197955(v=vs.95).aspx">add a ClientAccessPolicy.xml to the root of your domain</a>to allow cross-domain access. Here&#8217;s a simple all-permissive version of it:</p>

<div class="wp_syntax"><table><tr><td class="line_numbers"><pre>1
2
3
4
5
6
7
8
9
10
11
</pre></td><td class="code"><pre class="java" style="font-family:monospace;">@Path<span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;clientaccesspolicy.xml&quot;</span><span style="color: #009900;">&#41;</span>
<span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">class</span> ClientAccessPolicyResource <span style="color: #009900;">&#123;</span>
    @GET
    @Produces<span style="color: #009900;">&#40;</span>MediaType.<span style="color: #006633;">APPLICATION_XML</span><span style="color: #009900;">&#41;</span>
    <span style="color: #000000; font-weight: bold;">public</span> <span style="color: #003399;">String</span> getStatus<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #000000; font-weight: bold;">return</span> <span style="color: #0000ff;">&quot;&lt;?xml version=<span style="color: #000099; font-weight: bold;">\&quot;</span>1.0<span style="color: #000099; font-weight: bold;">\&quot;</span> encoding=<span style="color: #000099; font-weight: bold;">\&quot;</span>utf-8<span style="color: #000099; font-weight: bold;">\&quot;</span>?&gt;&lt;access-policy&gt;&lt;cross-domain-access&gt;&lt;policy&gt; &lt;allow-from http-request-headers=<span style="color: #000099; font-weight: bold;">\&quot;</span>*<span style="color: #000099; font-weight: bold;">\&quot;</span>&gt;&lt;domain uri=<span style="color: #000099; font-weight: bold;">\&quot;</span>*<span style="color: #000099; font-weight: bold;">\&quot;</span>/&gt;&lt;/allow-from&gt; &lt;grant-to&gt; &lt;resource path=<span style="color: #000099; font-weight: bold;">\&quot;</span>/<span style="color: #000099; font-weight: bold;">\&quot;</span> include-subpaths=<span style="color: #000099; font-weight: bold;">\&quot;</span>true<span style="color: #000099; font-weight: bold;">\&quot;</span>/&gt;&quot;</span><span style="color: #339933;">+</span>
     <span style="color: #0000ff;">&quot; &lt;/grant-to&gt;&lt;/policy&gt;&lt;policy &gt;&lt;allow-from http-methods=<span style="color: #000099; font-weight: bold;">\&quot;</span>*<span style="color: #000099; font-weight: bold;">\&quot;</span>&gt;&lt;domain uri=<span style="color: #000099; font-weight: bold;">\&quot;</span>*<span style="color: #000099; font-weight: bold;">\&quot;</span>/&gt;&lt;/allow-from&gt;&lt;grant-to&gt;&quot;</span><span style="color: #339933;">+</span>
      <span style="color: #0000ff;">&quot;&lt;resource path=<span style="color: #000099; font-weight: bold;">\&quot;</span>/<span style="color: #000099; font-weight: bold;">\&quot;</span> include-subpaths=<span style="color: #000099; font-weight: bold;">\&quot;</span>true<span style="color: #000099; font-weight: bold;">\&quot;</span>/&gt; &lt;/grant-to&gt;&lt;/policy&gt;&lt;/cross-domain-access&gt;&lt;/access-policy&gt;&quot;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #009900;">&#125;</span></pre></td></tr></table></div>

<p>By the way &#8211; since silverlight is entirely a client technology you can also serve it from the Java web server (<a href="http://www.tomcatexpert.com/tags/tomcat-silverlight">as long as you emit the right mime-type </a>) in which case you don&#8217;t need the cross-domain policy mentioned above.</p>
<p>Another interesting phenomena we saw was that the client only got the response once. no matter how many times we polled the server. We ran fiddler and saw that no request are in fact going out to the server &#8211; it was the browser&#8217;s (IE8) cache that was holding us back. It took me some head scratching as to how to solve this &#8211; but the solution is pretty simple &#8211; return an &#8216;expires&#8221; http header from the server to make sure the browser will only cache the result for a limited time. To do that with jersey you need to return a Response object instead of the text (or JSON/HTML etc.)</p>

<div class="wp_syntax"><table><tr><td class="line_numbers"><pre>1
2
3
4
5
6
7
8
9
10
11
12
13
14
</pre></td><td class="code"><pre class="java" style="font-family:monospace;">@Path<span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;stat&quot;</span><span style="color: #009900;">&#41;</span>
<span style="color: #000000; font-weight: bold;">public</span> <span style="color: #000000; font-weight: bold;">class</span> StatusResource <span style="color: #009900;">&#123;</span>
&nbsp;
    @GET @Produces<span style="color: #009900;">&#40;</span>MediaType.<span style="color: #006633;">TEXT_PLAIN</span><span style="color: #009900;">&#41;</span>
    <span style="color: #000000; font-weight: bold;">public</span> Response getStatus<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
&nbsp;
        Response.<span style="color: #006633;">ResponseBuilder</span> response <span style="color: #339933;">=</span> Response.<span style="color: #006633;">ok</span><span style="color: #009900;">&#40;</span>checkStatus<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
    <span style="color: #003399;">Date</span> expirationDate <span style="color: #339933;">=</span> <span style="color: #000000; font-weight: bold;">new</span> <span style="color: #003399;">Date</span><span style="color: #009900;">&#40;</span><span style="color: #003399;">System</span>.<span style="color: #006633;">currentTimeMillis</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">+</span> maxCacheInterval<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    response.<span style="color: #006633;">expires</span><span style="color: #009900;">&#40;</span>expirationDate<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
     <span style="color: #000000; font-weight: bold;">return</span> response.<span style="color: #006633;">build</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #009900;">&#125;</span></pre></td></tr></table></div>

<p>While I think Silverlight is a technology on its way out. It is still around. It is nice to know that with a couple of simple moves you can get it to play nice even with a Java backend</p>
<hr />
<p>Illustration by <a href="http://www.sxc.hu/photo/1342301">arinas74</a></p>
<script src="http://feeds.feedburner.com/~s/arnonrgo?i=http://arnon.me/2011/12/rest-jaxrs-silverlight-clients/" type="text/javascript" charset="utf-8"></script><div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=pGLG4JMzbFU:JR2sMRjSg4g:F7zBnMyn0Lo"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=pGLG4JMzbFU:JR2sMRjSg4g:F7zBnMyn0Lo" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=pGLG4JMzbFU:JR2sMRjSg4g:cGdyc7Q-1BI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=cGdyc7Q-1BI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=pGLG4JMzbFU:JR2sMRjSg4g:G79ilh31hkQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=G79ilh31hkQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=pGLG4JMzbFU:JR2sMRjSg4g:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=pGLG4JMzbFU:JR2sMRjSg4g:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=pGLG4JMzbFU:JR2sMRjSg4g:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=pGLG4JMzbFU:JR2sMRjSg4g:gIN9vFwOqvQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=pGLG4JMzbFU:JR2sMRjSg4g:I9og5sOYxJI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=I9og5sOYxJI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=pGLG4JMzbFU:JR2sMRjSg4g:-BTjWOF_DHI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=pGLG4JMzbFU:JR2sMRjSg4g:-BTjWOF_DHI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=pGLG4JMzbFU:JR2sMRjSg4g:qj6IDK7rITs"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=qj6IDK7rITs" border="0"></img></a>
</div><img src="http://feeds.feedburner.com/~r/CirrusMinor/~4/pGLG4JMzbFU" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://arnon.me/2011/12/rest-jaxrs-silverlight-clients/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		<feedburner:origLink>http://arnon.me/2011/12/rest-jaxrs-silverlight-clients/</feedburner:origLink></item>
		<item>
		<title>SOA Patterns : Composite Frontend (PDF)</title>
		<link>http://feedproxy.google.com/~r/CirrusMinor/~3/be7Vk0N_0_U/</link>
		<comments>http://arnon.me/2011/10/soa-patterns-composite-frontend-pdf/#comments</comments>
		<pubDate>Tue, 18 Oct 2011 18:26:58 +0000</pubDate>
		<dc:creator>Arnon Rotem-Gal-Oz</dc:creator>
				<category><![CDATA[Blog]]></category>
		<category><![CDATA[SOA Patterns]]></category>
		<category><![CDATA[Design]]></category>
		<category><![CDATA[Patterns]]></category>
		<category><![CDATA[Service Oriented Architecture]]></category>
		<category><![CDATA[SOA]]></category>
		<category><![CDATA[Software Architecture]]></category>

		<guid isPermaLink="false">http://arnon.me/?p=544</guid>
		<description><![CDATA[I got a few request for a PDF version of the pattern so here it is :  Composite Frontend Pattern]]></description>
			<content:encoded><![CDATA[<p>I got a few request for a PDF version of the pattern so here it is :  <a href="http://arnon.me/wp-content/uploads/2011/10/Composite-Frontend.pdf">Composite Frontend</a> Pattern</p>
<script src="http://feeds.feedburner.com/~s/arnonrgo?i=http://arnon.me/2011/10/soa-patterns-composite-frontend-pdf/" type="text/javascript" charset="utf-8"></script><div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=be7Vk0N_0_U:GLU1PUs0Rwo:F7zBnMyn0Lo"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=be7Vk0N_0_U:GLU1PUs0Rwo:F7zBnMyn0Lo" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=be7Vk0N_0_U:GLU1PUs0Rwo:cGdyc7Q-1BI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=cGdyc7Q-1BI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=be7Vk0N_0_U:GLU1PUs0Rwo:G79ilh31hkQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=G79ilh31hkQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=be7Vk0N_0_U:GLU1PUs0Rwo:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=be7Vk0N_0_U:GLU1PUs0Rwo:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=be7Vk0N_0_U:GLU1PUs0Rwo:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=be7Vk0N_0_U:GLU1PUs0Rwo:gIN9vFwOqvQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=be7Vk0N_0_U:GLU1PUs0Rwo:I9og5sOYxJI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=I9og5sOYxJI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=be7Vk0N_0_U:GLU1PUs0Rwo:-BTjWOF_DHI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=be7Vk0N_0_U:GLU1PUs0Rwo:-BTjWOF_DHI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=be7Vk0N_0_U:GLU1PUs0Rwo:qj6IDK7rITs"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=qj6IDK7rITs" border="0"></img></a>
</div><img src="http://feeds.feedburner.com/~r/CirrusMinor/~4/be7Vk0N_0_U" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://arnon.me/2011/10/soa-patterns-composite-frontend-pdf/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		<feedburner:origLink>http://arnon.me/2011/10/soa-patterns-composite-frontend-pdf/</feedburner:origLink></item>
		<item>
		<title>SOA Patterns : Composite Frontend</title>
		<link>http://feedproxy.google.com/~r/CirrusMinor/~3/K6BarDmPI4Y/</link>
		<comments>http://arnon.me/2011/10/soa-patterns-composite-frontend/#comments</comments>
		<pubDate>Mon, 17 Oct 2011 06:02:40 +0000</pubDate>
		<dc:creator>Arnon Rotem-Gal-Oz</dc:creator>
				<category><![CDATA[Blog]]></category>
		<category><![CDATA[Featured Posts]]></category>
		<category><![CDATA[SOA Patterns]]></category>
		<category><![CDATA[service consumers]]></category>
		<category><![CDATA[SOA]]></category>
		<category><![CDATA[soa patterns]]></category>
		<category><![CDATA[UI]]></category>

		<guid isPermaLink="false">http://arnon.me/?p=523</guid>
		<description><![CDATA[I am not blogging much these days &#8211; most of it is due to trying to get my bloody book finished. A case study and a finished anti-pattern chapter where recently pushed to the MEAP, and here&#8217;s one additional pattern from chapter 6 (service consumer patterns): When we try to think about service consumers, the [...]]]></description>
			<content:encoded><![CDATA[<p>I am not blogging much these days &#8211; most of it is due to trying to get my bloody book finished. A case study and a finished anti-pattern chapter where recently pushed <a href="http://www.manning.com/rotem/">to the MEAP</a>, and here&#8217;s one additional pattern from chapter 6 (service consumer patterns):</p>
<p>When we try to think about service consumers, the obvious candidates are, of course, other services. Nevertheless there are other software components that interact with services e.g. legacy systems,  Non-SOA external systems or reporting databases. The Composite Frontend pattern deals with yet another type of service consumer – the User interface.</p>
<p>&nbsp;</p>
<p>First let just verify that User interfaces aren’t in fact services. One reason user interfaces are not services is that they converge several business areas e.g. if you want to enter an order you’d probably also want to lookup information about the customer, maybe you’d also want to browse the product catalog, look at open invoices etc. In addition to convergence, user interfaces deliver data rather than process it. User interfaces are data producers (actually there’s one exception to that – where the UI is the front of a “human service” see orchestrated choreography pattern (in chapter 7) for more details.</p>
<p>&nbsp;</p>
<p>Ok, so UIs aren’t services, does it matter? Well, it does and the problem is not that UIs aren’t services per se. The main challenge caused by user interfaces comes from their main difference i.e. the aggregation or convergence of several services into a cohesive and useful UI.</p>
<p>&nbsp;</p>
<p>6.1.1      The Problem</p>
<p>To better understand the challenges caused by user interface working with multiple services, let’s consider an example with just a single point of friction.</p>
<p>&nbsp;</p>
<p>In a project I worked on we’ve designed the C4ISR (Command, Control, Communications, Computers, Intelligence, Surveillance and Reconnaissance) system for an Unmanned Naval Patrol Vehicle (UNPV). One of the services in the system was dubbed “Common Operational Picture” or COP for short. The COP ‘s responsibility was to handle anything that is detected by sensors e.g. ships, planes, whatever (There’s technical jargon for all that like targets, detections, tracks etc. but that’s not important here). One of the main UI representations of the COP was a map ,such as the one in figure 6.4 below, which showed all the detections. Clicking on map icon, say a ship, presents some information the COP knows about it, such as id, nationality, course  etc.</p>
<p style="padding-left: 30px; text-align: center;" align="center"><a href="http://arnon.me/wp-content/uploads/2011/10/6.4.png"><img class="aligncenter size-full wp-image-530" title="6.4" src="http://arnon.me/wp-content/uploads/2011/10/6.4.png" alt="" width="401" height="268" /></a> <span style="text-align: -webkit-auto;">Figure 6.4 A simplified  illustration of a  front end for a “Common Operational Picture” service of a naval command and control system. We see a shore line and then using NATO symbology: 2 radars, a submarine and an a ship (the Unmanned Naval Patrol Vehicle)</span></p>
<p>&nbsp;</p>
<p>The system had a few other services in addition to the COP, amongst them there was the “UNPV service”. The UNPV service was responsible for anything related to the UNPV itself for instance, setting it on a navigation course, turning it around, etc. The UNPV service had several UI screens to allow managing and monitoring these functions. Another responsibility of the UNPV service was to send its location to the COP (locations are the COP’s responsibility, remember?) so back in the UI, one of the icons of the map is that of the UNPV.</p>
<p>&nbsp;</p>
<p>What happens when a user clicks on the UNPV icon on the map?  Remember, while the desired outcome is to display a popup with options related to controlling the UNPV, the click is on the map, a control serviced by another service (the COP). In an Object Oriented system the UNPV might be a sub-class of a detection so that it would accept the same event and respond a little differently (in a more specialized way). Here, however the COP and the UNPV are completely different services, developed by two different group and maybe even two different companies.</p>
<p>&nbsp;</p>
<p>We may be able to dismiss this specific example and just solve it with a specific solution e.g add an ‘if’ statement somewhere to call up the correct commands and to interact with the correct service(s). The problem, however, is that this example is just the tip of the iceberg. For instance: how do we handle security? – do we need to login for each service separately? How do we handle things that all the services need? SOA’s premise is that we’d get a sort of lego-like enterprise where we can compose different business processes easily. Is there any way  we can get that in the  user interface? In summary:</p>
<p>&nbsp;</p>
<blockquote><p><strong>How do you we interact with multiple services, get an integrated, cohesive user interface and still preserve SOA principles and modularity benefits?</strong></p>
<p>&nbsp;</p></blockquote>
<p>One option is, as mentioned above, write specific client code. Using this approach “an application”, is any specific composition of services. For the example above, the application would include the two service (COP and UNPV) and a UI that ties them together. The up-side of this approach is that each application delivers a consistent experience for the user. After all, a specific or tailored application can be made to be very cohesive. Additionally there are many tried and tested ways to build flexible UIs with a proper separation of concerns, e.g. using Model-View-Controller and its variations (in a multitude of rich client and web technologies) so we’d probably be able to reuse some of the UI-side logic even going  from application to application. Nevertheless, we do lose on flexibility. For one, any service change that has UI aspects needs to be redone for each of its UI instances (applications). More so, since the UI specifically ties multiple services changes in one service may cause another to mal-function within the unified UI. We also lose on Composability, or the ability to replace services and to create new business flows (relatively) easy. Overall it’s a bad option long-term but it can be made to work as a short term solution.</p>
<p>A related option is taking a similar approach of tying several services together but instead of integrating them on the client side we integrate the services together on the server side. This approach shares it pros and cons with the previous solution. However there are specific circumstances where it does make sense and you can read about them in the next pattern (section 6.5 Client/Server/Service)</p>
<p>Lastly, we have the option to have independent UI components per service. This would overcome the limitations we’ve mentioned above since each service’s corresponding UI can evolve independently and you can just cram a as many of these as you like to create an application. Unfortunately, with all its benefits this is a non even an option here as, by definition, we won’t have the mechanisms for UI components that work cross services. i.e. it won’t actually solve problems like the one in the example and we can’t get a cohesive UI.</p>
<p>&nbsp;</p>
<p>6.1.2      The Solution</p>
<p>What we need, essentially, is a way to compose services together while keeping their respective autonomy on one hand and providing mechanisms to glue them together as a cohesive whole – That’s what the Composite Frontend pattern is about:</p>
<p>&nbsp;</p>
<blockquote><p><strong>Apply the Composite Frontend pattern to aggregate services while providing them unified client-side services like layout and theming as well as coordination services for client-side service integration</strong></p></blockquote>
<p>&nbsp;</p>
<p style="padding-left: 30px; text-align: center;"><a href="http://arnon.me/wp-content/uploads/2011/10/6.5.png"><img class="aligncenter size-full wp-image-529" title="6.5" src="http://arnon.me/wp-content/uploads/2011/10/6.5.png" alt="" width="502" height="266" /></a></p>
<p style="padding-left: 30px;">Figure 6.5 The Composite Frontend pattern. Each Service has a Portlet which is a Service Agent combined with a UI logic (most likely Model in a MVC UI pattern). The UI host provides services for the different potlets to weave them together into a coherent UI.</p>
<p>The Composite Frontend pattern is about taking the ideas (and sometimes the technologies) behind web portal and applying them to SOA services. Web portals provide unified access point that aggregates multiple web pages. They also provide single sign-one and personalization. SOA interfaces need that and more.</p>
<p>The composite front-end pattern is composed of two main components: the portlet and the host. The portlets are the building blocks, “the composites” which are fused together to form the user interface. The portlets are made of at least two components. The user interface logic (views and controllers in MVC lingo). The second component, the service proxy (or agent) , is the more interesting one from an SOA perspective. The service proxy is a client-side representation of a service. The proxy serves as the model for the user interfaces components. It is usually recommended to have a single proxy per service -just like it is recommended for each service to maintain its own data store. Furthermore, from a product management perspective it can be seen as  part of the service itself</p>
<p>&nbsp;</p>
<p>The host is the “value-add” part of the composite frontend pattern. The host provides the glue that ties the different portlets into a cohesive whole. As such, the host provides  several roles. Firstly it provides the canvas or surface on which the portlets are displayed. Additionally it controls the life-cycle of the portlets and lastly it provides capabilities (avoiding the loaded term services&#8230;)  like inter-portlet communications and single-sign-on.</p>
<p>&nbsp;</p>
<p>Lets revisit the problem discussed in the previous section. We had a right-click on a ui component, which should have produced a context menu with options from two services. How would that would with the composite frontend pattern? One option is that a click would be first intercepted by the host which would then dispatch it to any registered portlet. Another option illustrated in figure 6.6 below , is for the click to  be intercepted by the first portlet the Common Operation Picture (or COP), have it notify the host and have the host ask all the iixnvolved portlets to render the right-click menu. The COP portlet should pass enough information as part of the event for the other porlets to be able to do something meaningful with.</p>
<p style="padding-left: 30px; text-align: center;"><a href="http://arnon.me/wp-content/uploads/2011/10/6.6.png"><img class="aligncenter size-full wp-image-528" title="6.6" src="http://arnon.me/wp-content/uploads/2011/10/6.6.png" alt="" width="366" height="284" /></a>Figure 6.6: Sample event flow in a Composite Frontend. Events are intercepted by the user interface components of individuals portlets , The events are transferred to the host which dispatches them to registered protlets for handling. The host can then render the results for display</p>
<p>The composite frontend pattern is a service consumer pattern so the proxy will utilize the various service interaction patterns like saga, request/reply etc. (see chapter 5) and it can benefit from the various service composition patterns like Service Registry and Servicbus (see chapter 7)</p>
<p>You’ve probably notices the use of the term portlets to describe the service agents and you might be wondering why the pattern is named Composite Frontend rather than Portal. The main reason for that is that the pattern can also be used with rich client implementations and not just web ones – let’s explore that further in the technology mapping section</p>
<p>&nbsp;</p>
<p>6.1.3      Technology Mapping</p>
<p>Normally, you’d won’t be developing your own Composite Frontend container and instead use existing products that provide the framework and usually the tooling to help build the portlets.</p>
<p>The obvious example for that are web portal frameworks. Modern enterprise web portals usually support anything from JSR 168/286 (Java Portlet specification) to  WSRP (Web Services for Remote Portlet) to open web standards like RSS, plain REST services or standards like open social.  There are a lot of products in this area both commercial like  ibm WebSphere portal server and Microsoft Sharepoint and open source options like Jboss Gatein and Lifray.  Figure 6.7 below show he layout functionality of the UI host as it is implemented in Jboss Gatein</p>
<p style="text-align: center;"><a href="http://arnon.me/wp-content/uploads/2011/10/6.7.png"><img class="aligncenter size-full wp-image-527" title="6.7" src="http://arnon.me/wp-content/uploads/2011/10/6.7.png" alt="" width="473" height="288" /></a></p>
<p style="padding-left: 30px;">Figure 6.7: The layout capability of a Composite Frontend UI host as it is implemented in Jboss Gatein portal.</p>
<p>Web portals are not the only option for implementing composite frontends. You can also implement the concept for desktop (“rich client”) applications. An example for that is the prism framework made by Microsoft’s Pattern and Practices group. Prism implements the Composite Frontend pattern for both Silverlight and WPF applications. Prism provides all the functionality of a user interface host and lets you write portlets that consume these capabilities. Code snippet 6.4 below demonstrate using an EventAggregator facility that allows inter-portlet communications (such as the one needed for the sample scenario in the map component example above):</p>
<p>Code listing 6.4 Sample use of Prism’s EventAggregator to send events between different porlets unified by the prism shell</p>
<p>&nbsp;</p>

<div class="wp_syntax"><table><tr><td class="line_numbers"><pre>1
2
3
4
5
6
7
8
9
10
11
12
13
14
</pre></td><td class="code"><pre class="csharp" style="font-family:monospace;"><span style="color: #008000;">&#91;</span>Export<span style="color: #008000;">&#40;</span><span style="color: #008000;">typeof</span><span style="color: #008000;">&#40;</span>SampleView<span style="color: #008000;">&#41;</span><span style="color: #008000;">&#41;</span><span style="color: #008000;">&#93;</span>
<span style="color: #0600FF; font-weight: bold;">public</span> <span style="color: #0600FF; font-weight: bold;">partial</span> <span style="color: #6666cc; font-weight: bold;">class</span> SampleView <span style="color: #008000;">:</span> UserControl
<span style="color: #008000;">&#123;</span>
   <span style="color: #008000;">&#91;</span>ImportingConstructor<span style="color: #008000;">&#93;</span>
   <span style="color: #0600FF; font-weight: bold;">public</span> SampleView<span style="color: #008000;">&#40;</span><span style="color: #008000;">&#91;</span>Import<span style="color: #008000;">&#93;</span> IEventAggregator eventAggregator
   <span style="color: #008000;">&#123;</span>
       InitializeComponent<span style="color: #008000;">&#40;</span><span style="color: #008000;">&#41;</span><span style="color: #008000;">;</span>
       eventAggregator<span style="color: #008000;">.</span><span style="color: #0000FF;">GetEvent</span><span style="color: #008000;">&gt;</span><span style="color: #008000;">&#40;</span><span style="color: #008000;">&#41;</span><span style="color: #008000;">.</span><span style="color: #0000FF;">Subscribe</span><span style="color: #008000;">&#40;</span>OnItemSelectedReceived<span style="color: #008000;">&#41;</span><span style="color: #008000;">;</span> <span style="color: #008080;">#1</span>
   <span style="color: #008000;">&#125;</span>
   <span style="color: #0600FF; font-weight: bold;">public</span> <span style="color: #6666cc; font-weight: bold;">void</span> ItemSelectedReceived<span style="color: #008000;">&#40;</span>ItemSelectedEvent item<span style="color: #008000;">&#41;</span>
   <span style="color: #008000;">&#123;</span>
     <span style="color: #008080; font-style: italic;">//do something with item...</span>
   <span style="color: #008000;">&#125;</span>
<span style="color: #008000;">&#125;</span></pre></td></tr></table></div>

<p>#1 subscribing to an temSelectedEvent. Another portlet can call get on the EventAggregator to get a reference to the event and raise it without knowing if thare are any subscribers</p>
<p>&nbsp;</p>
<p>Lastly, in addition to web portal frameworks and desktop framework you can roll your own implementation of Composite Frontend. however, as mentioned above it is usually better to  choose one of the available options as it quite an investment to get it right.</p>
<p>6.1.4      Quality Attributes</p>
<p>Before moving on to the next pattern lets examine some business drives (or scenarios) that can drive us to use the Composite Frontend pattern.</p>
<p>In essence, the main drives to Composite Frontend are flexibility to adding and changing services and the desire for an integrated user interface that feels as a whole Table 6.X provides examples for both quality attributes:</p>
<p>&nbsp;</p>
<table border="1" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td valign="top" width="122">
<div>
<p>Quality Attribute (level1)</p>
</div>
</td>
<td valign="top" width="126">
<div>
<p>Quality Attribute (level2)</p>
</div>
</td>
<td valign="top" width="123">
<div>
<p>Sample Scenario</p>
</div>
</td>
</tr>
<tr>
<td valign="top" width="122">Usability</td>
<td valign="top" width="126">Operability</td>
<td valign="top" width="123">Under normal system use end user wants to achieve business tasks fluently. System should reuse entered data (like personal details) between different tasks</td>
</tr>
<tr>
<td valign="top" width="122">Flexibility</td>
<td valign="top" width="126">Changability</td>
<td valign="top" width="123">Under normal conditions, changing the billing process to support a new  credit card clearance provider, should take less than one week</td>
</tr>
</tbody>
</table>
<p>Table 6.3  composite frontend  pattern quality attributes scenarios. These are the architectural scenarios that can make us think about using composite frontend pattern.</p>
<p>&nbsp;</p>
<p>Composite Frontend is probably the preferred way to provide an SOA user interface. However one problem we still have to solve with integrating UIs is UIs that are not SOA aware – for instance what happens when we have an existing UI that we want to expose to services? The next pattern will try to answer exactly that.</p>
<script src="http://feeds.feedburner.com/~s/arnonrgo?i=http://arnon.me/2011/10/soa-patterns-composite-frontend/" type="text/javascript" charset="utf-8"></script><div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=K6BarDmPI4Y:oGwgCBeUCnE:F7zBnMyn0Lo"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=K6BarDmPI4Y:oGwgCBeUCnE:F7zBnMyn0Lo" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=K6BarDmPI4Y:oGwgCBeUCnE:cGdyc7Q-1BI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=cGdyc7Q-1BI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=K6BarDmPI4Y:oGwgCBeUCnE:G79ilh31hkQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=G79ilh31hkQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=K6BarDmPI4Y:oGwgCBeUCnE:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=K6BarDmPI4Y:oGwgCBeUCnE:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=K6BarDmPI4Y:oGwgCBeUCnE:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=K6BarDmPI4Y:oGwgCBeUCnE:gIN9vFwOqvQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=K6BarDmPI4Y:oGwgCBeUCnE:I9og5sOYxJI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=I9og5sOYxJI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=K6BarDmPI4Y:oGwgCBeUCnE:-BTjWOF_DHI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=K6BarDmPI4Y:oGwgCBeUCnE:-BTjWOF_DHI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=K6BarDmPI4Y:oGwgCBeUCnE:qj6IDK7rITs"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=qj6IDK7rITs" border="0"></img></a>
</div><img src="http://feeds.feedburner.com/~r/CirrusMinor/~4/K6BarDmPI4Y" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://arnon.me/2011/10/soa-patterns-composite-frontend/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		<feedburner:origLink>http://arnon.me/2011/10/soa-patterns-composite-frontend/</feedburner:origLink></item>
		<item>
		<title>HP’s enterprise play</title>
		<link>http://feedproxy.google.com/~r/CirrusMinor/~3/zh_IccqO3ME/</link>
		<comments>http://arnon.me/2011/08/hp-bigdata/#comments</comments>
		<pubDate>Sun, 21 Aug 2011 21:03:25 +0000</pubDate>
		<dc:creator>Arnon Rotem-Gal-Oz</dc:creator>
				<category><![CDATA[Blog]]></category>
		<category><![CDATA[Analytics]]></category>
		<category><![CDATA[Big Data]]></category>

		<guid isPermaLink="false">http://arnon.me/?p=510</guid>
		<description><![CDATA[The internet is ablaze with posts and articles on HPs abandonment of WebOS and the PC business. I don&#8217;t have a lot to add to this discussion (It&#8217;s a pity, I was considering a Pre3 device, blah blah). I am personally more interested in the last bit of their announcement that talked about the 10.3bn$ [...]]]></description>
			<content:encoded><![CDATA[<p style="text-align: left;">The internet is ablaze with posts and articles on HPs abandonment of WebOS and the PC business. I don&#8217;t have a lot to add to this discussion (It&#8217;s a pity, I was considering a Pre3 device, blah blah). I am personally more interested in the last bit of their announcement that talked about the <a href="http://www.bloomberg.com/news/2011-08-18/hp-said-to-be-near-10-billion-autonomy-takeover-spinoff-of-pc-business.html">10.3bn$ acquisition </a>of <a href="http://www.autonomy.com/">Autonomy</a>. If we add this to <a href="http://www.hp.com/hpinfo/newsroom/press/2011/110214xb.html">HP&#8217;s acquisition of Vertica just 6 months ago</a> and the (supposedly )<a href="http://www.informationweek.com/articles/229401466">failed attempt to acquire Tibco</a> we can see HP is making a  play for the whole unstructured/big-data/analytics field.</p>
<p style="text-align: left;">Vertica is  a columnar database optimized for analytics. In a nut shell, columnar databases allow fast aggregation of  data as well as holding a lot of columns per row &#8211; both traits are helpful when trying to report on data (but vey wasteful when you try to do OLTP operations). Autonomy adds to that path analysis and related algorithms to provide insight from stored data. Add that to HP&#8217;s hardware capabilities and you can get a high-performance data appliances that can handle large amounts of data efficiently.</p>
<p style="text-align: left;">HP is not alone in this trend. We see similar moves by <a href="http://www.emc.com/about/news/press/2010/20100706-01.htm">EMC which acquired Greenplum</a> followed by  unified GreenPlum/Hadoop  SW-HW solution, IBM <a href="http://www-03.ibm.com/press/us/en/pressrelease/32955.wss">acquiring Netezza</a> and  rolling  its own Hadoop version, heck, even <a href="http://techcrunch.com/2011/03/03/teradata-buys-aster-data-263-million/">Teradata recently acquired Aster Data to bolster its analytics offering</a>. It&#8217;s almost like we&#8217;re back in the 1950&#8242;s with companies creating dedicated appliances instead of general purpose computers.</p>
<p style="text-align: left;">While Teradata, an NCR spinoff, pioneered this trend, I think Oracle&#8217;s introduction of Exadata popularized this trend. Oracle which is probably the leader could (and still can) be sold with any hardware. All of a sudden, with the acquisition of Sun, Oracle started to keep the high-end customers to itself. This was especially a blow to HP considering the first incarnation of exadata ran on HP&#8217;s hardware. Anyway, as we saw above, the big vendors* are rushing to build their own &#8220;complete software and hardware suits&#8221; and HP, it seems, is doing the same. By talking about dropping its PC business HP is trying to pull an IBM (and its Lenovo move) and focus on Enterprise solutions in general and the hot analytics market in particular</p>
<p style="text-align: left;">Meanwhile I am reading that <a href="http://www.zdnet.com/blog/btl/hps-touchpad-fire-sale-the-fallout/55594">HP servers are having trouble handling the demand for their tablet dumping </a>- I guess that&#8217;s also bad press to the cloud message of HP. I hope their analytics message and solutions will fare better</p>
<p style="text-align: left;">By the way, for now this is only interesting for larger corporations which are willing to spend the big bucks involved  (e.g. Vertica was priced at 100K$ per terabyte last time I checked). Smaller companies esp. SaaS provider, will probably still solve their big data problems with Hadoop, Cassandra and the like. However <a href="http://www.marketwatch.com/story/worlds-data-more-than-doubling-every-two-years-driving-big-data-opportunity-new-it-roles-2011-06-28">as data is more than doubling every two years</a> and as the top-tier market will get saturated, I am sure we&#8217;ll start seing offerings with more reasonable price-tags over the next few years.</p>
<hr />
<p style="text-align: left;">* You may have notices that Microsoft is missing from this list &#8211; MS is also making moves in this field with Parallel Data Warehouse and Dryad . Unlike the other&#8217;s they do not have hardware capabilities as well as some other challenges. Maybe I&#8217;ll dedicate a post to them in the future.</p>
<script src="http://feeds.feedburner.com/~s/arnonrgo?i=http://arnon.me/2011/08/hp-bigdata/" type="text/javascript" charset="utf-8"></script><div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=zh_IccqO3ME:KvQ79ot-EsQ:F7zBnMyn0Lo"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=zh_IccqO3ME:KvQ79ot-EsQ:F7zBnMyn0Lo" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=zh_IccqO3ME:KvQ79ot-EsQ:cGdyc7Q-1BI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=cGdyc7Q-1BI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=zh_IccqO3ME:KvQ79ot-EsQ:G79ilh31hkQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=G79ilh31hkQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=zh_IccqO3ME:KvQ79ot-EsQ:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=zh_IccqO3ME:KvQ79ot-EsQ:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=zh_IccqO3ME:KvQ79ot-EsQ:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=zh_IccqO3ME:KvQ79ot-EsQ:gIN9vFwOqvQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=zh_IccqO3ME:KvQ79ot-EsQ:I9og5sOYxJI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=I9og5sOYxJI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=zh_IccqO3ME:KvQ79ot-EsQ:-BTjWOF_DHI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=zh_IccqO3ME:KvQ79ot-EsQ:-BTjWOF_DHI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=zh_IccqO3ME:KvQ79ot-EsQ:qj6IDK7rITs"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=qj6IDK7rITs" border="0"></img></a>
</div><img src="http://feeds.feedburner.com/~r/CirrusMinor/~4/zh_IccqO3ME" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://arnon.me/2011/08/hp-bigdata/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		<feedburner:origLink>http://arnon.me/2011/08/hp-bigdata/</feedburner:origLink></item>
		<item>
		<title>RabbitMQ – part I –  AMQP</title>
		<link>http://feedproxy.google.com/~r/CirrusMinor/~3/hIFtn1igCHs/</link>
		<comments>http://arnon.me/2011/08/amqp/#comments</comments>
		<pubDate>Fri, 05 Aug 2011 18:37:30 +0000</pubDate>
		<dc:creator>Arnon Rotem-Gal-Oz</dc:creator>
				<category><![CDATA[Blog]]></category>
		<category><![CDATA[amqp]]></category>
		<category><![CDATA[message bus]]></category>
		<category><![CDATA[messaging]]></category>
		<category><![CDATA[RabbitMQ]]></category>
		<category><![CDATA[Technology]]></category>

		<guid isPermaLink="false">http://arnon.me/?p=491</guid>
		<description><![CDATA[I haven&#8217;t blogged in a few months and I guess it is about time I get back to writing. A lot has happened including me leaving CloudValue (I still think it can be interesting business-wise, but it wasn&#8217;t very challenging technically) and moving to a new company (where I&#8217;ll probably won&#8217;t get rich but I [...]]]></description>
			<content:encoded><![CDATA[<p>I haven&#8217;t blogged in a few months and I guess it is about time I get back to writing. A lot has happened including me leaving CloudValue (I still think it can be interesting business-wise, but it wasn&#8217;t very challenging technically) and moving to a new company (where I&#8217;ll probably won&#8217;t get rich but I get to work on challenging projects :) ). Anyway, I working with quite a few interesting technologies these days (Hadoop,  Greenplum, BigInsight and whatnot )and I thought it might be a good idea  to write about some of them. The first technology I am going to talk about is RabbitMQ.</p>
<p>RabbitMQ is a message bus built in Erlang. It implements the AMQP protocol, which, unlike JMS, is also a wire protocol and not just an API. (meaning you can use an AMQP client with any compliant AMQP server) AMQP has a few concepts which are different from &#8220;regular&#8221; messaging infrastructures (like MSMQ or MQSeries). The illustration below shows the main concepts of AMQP</p>
<p><a href="http://arnon.me/wp-content/uploads/2011/08/Rabbitmq.png"><img class="alignleft size-large wp-image-492" title="Rabbitmq" src="http://arnon.me/wp-content/uploads/2011/08/Rabbitmq-1024x445.png" alt="" width="491" height="214" /></a></p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>Virtual Host – Virtual host is administrative container which  provides access control boundaries. Each virtual host has its own set of exchanges. A connection can only be associated with a single virtual host</p>
<p>Exchange – An exchange is a message routing agent. Exchanges can be temporary, auto-deleted or durable. Messages are published to exchanges and then routed to queues that are bound to the exchange (if no queues are bound the message will be lost).Exchanges support one of the following routing schemes</p>
<ul>
<li>Direct – based on a routing key. Any queue (or exchange) binding to the key will get the message</li>
<li>Fan-out – no routing key , each queue or exchange bound to this type of exchange will get a copy of the message</li>
<li>Header -  Header based routing allows compound routing based on a combination of lists of identifiers (essentially key-value pairs) the routing can be done on either any or all key-values specified. The subscription then specified a set of key-values and an operator (and or any) e.g. { &#8216;x-match&#8217; =&gt; &#8216;any&#8217;, :customer =&gt; &#8217;23&#8242;, :agent =&gt; &#8217;007&#8242; }. This looked like a promising direction until we’ve found that it isn’t well supported (e.g. it isn’t even documented in the C# documentation of RabbitMQ) and doesn’t perform well</li>
<li>Topic – Topic based routing allows compound routing based on topic hierarchy. Topics is based on pattern matching (a strong core capability of Erlang, RabbitMQ’s implementation language) and allows versatile subscription based on wildcards: * a single segment placeholder or # zero or more segments  (e.g. foo.*.bar will match foo.baz.bar, foo.bag.bar but not foo.baz.bag.bar, foo.#.bar will match all of them)</li>
</ul>
<p>Both queues and exchanges can bind to an exchange. The default exchange uses the queue name as routing key (i.e. sends a message directly to a queue)</p>
<p>Queue – A queue is a FIFO buffer. It is important to note that the FIFO order is only guaranteed when a single consumer is used. Queues, like exchanges cab be temporary, auto-deleted or durable. Getting messages from exchanges to queues is done by binding the queue to an exchange.</p>
<p>So that&#8217;s a short into to AMQP itself. In the next installments I&#8217;ll expand a little about RabbitMQ itself, show some coding samples and finish it up with some pros/cons</p>
<hr />
<p>illustration by <a href="http://www.flickr.com/photos/beautyredefined/2389559961/">beautyredefined</a></p>
<script src="http://feeds.feedburner.com/~s/arnonrgo?i=http://arnon.me/2011/08/amqp/" type="text/javascript" charset="utf-8"></script><div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=hIFtn1igCHs:ha8EJNpxEAE:F7zBnMyn0Lo"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=hIFtn1igCHs:ha8EJNpxEAE:F7zBnMyn0Lo" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=hIFtn1igCHs:ha8EJNpxEAE:cGdyc7Q-1BI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=cGdyc7Q-1BI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=hIFtn1igCHs:ha8EJNpxEAE:G79ilh31hkQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=G79ilh31hkQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=hIFtn1igCHs:ha8EJNpxEAE:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=hIFtn1igCHs:ha8EJNpxEAE:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=hIFtn1igCHs:ha8EJNpxEAE:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=hIFtn1igCHs:ha8EJNpxEAE:gIN9vFwOqvQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=hIFtn1igCHs:ha8EJNpxEAE:I9og5sOYxJI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=I9og5sOYxJI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=hIFtn1igCHs:ha8EJNpxEAE:-BTjWOF_DHI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=hIFtn1igCHs:ha8EJNpxEAE:-BTjWOF_DHI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=hIFtn1igCHs:ha8EJNpxEAE:qj6IDK7rITs"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=qj6IDK7rITs" border="0"></img></a>
</div><img src="http://feeds.feedburner.com/~r/CirrusMinor/~4/hIFtn1igCHs" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://arnon.me/2011/08/amqp/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		<feedburner:origLink>http://arnon.me/2011/08/amqp/</feedburner:origLink></item>
		<item>
		<title>Amazon’s EC2 &amp; EBS outage</title>
		<link>http://feedproxy.google.com/~r/CirrusMinor/~3/j4DF-GlywYI/</link>
		<comments>http://arnon.me/2011/04/amazons-ec2-ebs-outage/#comments</comments>
		<pubDate>Sat, 23 Apr 2011 21:39:13 +0000</pubDate>
		<dc:creator>Arnon Rotem-Gal-Oz</dc:creator>
				<category><![CDATA[Blog]]></category>
		<category><![CDATA[Amazon]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[EC2]]></category>
		<category><![CDATA[When things break]]></category>

		<guid isPermaLink="false">http://arnon.me/?p=466</guid>
		<description><![CDATA[Unless you are living under a rock, you&#8217;ve probably heard, if not felt Amazon&#8217;s outage (you can read more about it all over the web, e.g.  Cade Metz at the register or Julianne Pepitone at CNN money, [edit Apr26] Todd Hoff &#8216;s list of posts on the subject) This incident is very interesting in the technical sense [...]]]></description>
			<content:encoded><![CDATA[<p>Unless you are living under a rock, you&#8217;ve probably heard, if not felt Amazon&#8217;s outage (you can read more about it all over the web, e.g.  <a href="http://www.theregister.co.uk/2011/04/21/amazon_web_services_outages_spans_zones/">Cade Metz at the register</a> or <a href="http://money.cnn.com/2011/04/21/technology/amazon_server_outage/index.htm">Julianne Pepitone at CNN money</a>, [edit Apr26] <a href="highscalability.com/blog/2011/4/25/the-big-list-of-articles-on-the-amazon-outage.html">Todd Hoff &#8216;s list of posts on the subject</a>) This incident is very interesting in the technical sense as well as very  disturbing.</p>
<p>First off it is a major event since Amazon controls almost 60% of the Infrastructure as a Service market (<a href="http://online.wsj.com/article/SB10001424052748704739504576067580949404062.html">per WSJ</a>) and an incident like this is bad publicity for the whole cloud concept. After all if Amazon is fledgling what does that mean for Rackspace and the other IaaS players &#8211; not to mention PaaS players like Google and Microsoft (since PaaS solutions require more complicated software and higher integration with end-solutions.</p>
<p style="display: inline !important;">It is also worth mentioning that this isn&#8217;t the major outages for Amazon. One notable &#8220;availability event&#8221;  occurred in 2008 where S<a href="http://status.aws.amazon.com/s3-20080720.html">3 had major problem for about half a day</a> and <a href="http://www.datacenterknowledge.com/archives/2009/07/19/outage-for-amazon-web-services/">a few minor problem with EC2 </a> in 2009. What&#8217;s stands out here is that the availability zones features that was supposed to isolate this type of breakdown to smaller areas broke and only sites that had data-center redundancy (like Netflix for example) managed to handle the interruption while sites like foursquare, reddit and even, it seems,  <a href="https://forums.aws.amazon.com/thread.jspa?threadID=65649&amp;tstart=0">a company monitoring cardiac arrests </a></p>
<p>were all harmed.</p>
<p>Another alarming behavior on the part of Amazon is the lack of transparency / bad crisis management in handling the outage. For example Keith Smith CEO of BigDoor (one of the startups affected by the outage) <a href="http://www.geekwire.com/2011/amazoncoms-real-problem-outage-communication">writes</a></p>
<blockquote>
<p style="display: inline !important;">&#8220;Starting at 1:41 a.m. PST, Amazon’s updates read as if they were written by their attorneys and accountants who were hedging against their stated SLA rather than being written by a tech guy trying to help another tech guy.&#8221;</p>
</blockquote>
<p>When a big supplier fumbles a lot of companies are affected and it is bound to get big press. Also, systems, esp. complex ones, have bugs and its understandable that things may break from time to time (and I am sure that Amazon, which evidently has top talent would find a way to prevent this from recurring). However, Cloud providers need to understand that <a href="http://en.wikipedia.org/wiki/Uncle_Ben#.22With_great_power_comes_great_responsibility.22">&#8220;with great power comes great responsibility&#8221;</a> and the technical offering need to be strengthened with great support and transparency.</p>
<p>On the technical level, it also means that, while moving to the cloud carries a lot of benefits and can save a lot on operations costs, the responsibility for our application&#8217;s up time is still our responsibility and depending on the  cost of failure (on the scale from few dollars to lives of people)  we should also architect for disaster regardless of vendor claims. When we build closed systems we may  look at the Mean-Time Between Failure (MTBF and MTBCF) advertised by hardware manufacturers but we&#8217;d also add software based reliability mechanisms &#8211; for the cloud that may mean cross data-center (region) deployments, cross cloud provider deployments, or even on-premise backup &#8211; This is the same measures you&#8217;d take when your dealing with the electric company, if it is important enough, you&#8217;d install UPSs and generators and alternate sites and whatnot, you just have to figure out how important business continuousness is</p>
<p>I guess we&#8217;ll all wait to see how all this unfolds and what will be the after-effects of this outage on Cloud-computing. I personally think that it is still a good move in many cases, however this incident, help focus that the responsibility for our application&#8217;s wellness  is still ultimately  ours</p>
<p>Edit Apr. 24th:</p>
<p>I just read a post in <a href="http://www.codinghorror.com/blog/2011/04/working-with-the-chaos-monkey.html">Coding Horror</a> which refers to a <strong>year old </strong>post in Netflix&#8217;s blog called &#8220;<a href="http://techblog.netflix.com/2010/12/5-lessons-weve-learned-using-aws.html">5 lessons we&#8217;ve learned using AWS</a> [Amazon WebSevices]&#8220;. Netflix, in case you&#8217;re wondering survived Amazon&#8217;s outage and indeed, in lesson #3 they explain that if you want to survive failures you have to plan and constantly test for it:</p>
<div id="outer-wrapper">
<div id="wrap2">
<div id="content-wrapper">
<div id="main-wrapper">
<div id="main">
<div id="Blog1">
<div>
<div>
<div>
<div>
<div>
<div id="post-body-381417170823959085">
<blockquote>
<div><strong>3. The best way to avoid failure is to fail constantly.</strong></div>
<div>We’ve sometimes referred to the Netflix software architecture in AWS as our Rambo Architecture. Each system has to be able to succeed, no matter what, even all on its own. We’re designing each distributed system to expect and tolerate failure from other systems on which it depends.</div>
<div>If our recommendations system is down, we degrade the quality of our responses to our customers, but we still respond. We’ll show popular titles instead of personalized picks. If our search system is intolerably slow, streaming should still work perfectly fine.</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p>&nbsp;</p>
<p>&nbsp;</p>
<script src="http://feeds.feedburner.com/~s/arnonrgo?i=http://arnon.me/2011/04/amazons-ec2-ebs-outage/" type="text/javascript" charset="utf-8"></script><div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=j4DF-GlywYI:eCTiOr9-gxk:F7zBnMyn0Lo"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=j4DF-GlywYI:eCTiOr9-gxk:F7zBnMyn0Lo" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=j4DF-GlywYI:eCTiOr9-gxk:cGdyc7Q-1BI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=cGdyc7Q-1BI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=j4DF-GlywYI:eCTiOr9-gxk:G79ilh31hkQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=G79ilh31hkQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=j4DF-GlywYI:eCTiOr9-gxk:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=j4DF-GlywYI:eCTiOr9-gxk:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=j4DF-GlywYI:eCTiOr9-gxk:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=j4DF-GlywYI:eCTiOr9-gxk:gIN9vFwOqvQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=j4DF-GlywYI:eCTiOr9-gxk:I9og5sOYxJI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=I9og5sOYxJI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=j4DF-GlywYI:eCTiOr9-gxk:-BTjWOF_DHI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=j4DF-GlywYI:eCTiOr9-gxk:-BTjWOF_DHI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=j4DF-GlywYI:eCTiOr9-gxk:qj6IDK7rITs"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=qj6IDK7rITs" border="0"></img></a>
</div><img src="http://feeds.feedburner.com/~r/CirrusMinor/~4/j4DF-GlywYI" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://arnon.me/2011/04/amazons-ec2-ebs-outage/feed/</wfw:commentRss>
		<slash:comments>4</slash:comments>
		<feedburner:origLink>http://arnon.me/2011/04/amazons-ec2-ebs-outage/</feedburner:origLink></item>
		<item>
		<title>Evolving Architectures – Part VII – Parallel</title>
		<link>http://feedproxy.google.com/~r/CirrusMinor/~3/QIIUOn30HAY/</link>
		<comments>http://arnon.me/2011/04/evolving-architectures-part-vii-parallel/#comments</comments>
		<pubDate>Thu, 14 Apr 2011 12:39:16 +0000</pubDate>
		<dc:creator>Arnon Rotem-Gal-Oz</dc:creator>
				<category><![CDATA[Agile Architecture]]></category>
		<category><![CDATA[Blog]]></category>
		<category><![CDATA[Featured Posts]]></category>
		<category><![CDATA[Agile]]></category>
		<category><![CDATA[architectural changes]]></category>
		<category><![CDATA[Design]]></category>
		<category><![CDATA[Software Architecture]]></category>

		<guid isPermaLink="false">http://arnon.me/?p=452</guid>
		<description><![CDATA[(This is part seven of a series. You might want to check) Parallel and Simplification are the yin and yang of architecture evolution. Simplification, as mentioned previously, is about having foresight and thus provides for,relatively, easy evolution (i.e. architectural additions and not changes). Parallel is about reacting to changes in requirements as they come (no [...]]]></description>
			<content:encoded><![CDATA[<p>(This is part seven of a series. You might want to check<a href="http://arnon.me/category/blog/featured-posts/agile-architecture/"> the previous parts first</a>)<br />
Parallel and Simplification are the yin and yang of architecture evolution. Simplification, as mentioned previously, is about having foresight and thus provides for,relatively, easy evolution (i.e. architectural additions and not changes). Parallel is about reacting to changes in requirements as they come (no foresight) and making architectural changes.<br />
But wait, architectural changes are hard by definition* &#8211; if we make a change to the architecture a lot of things are going to break &#8211; starting with the build (or the continuous deployment if you have one) and things will just deteriorate from there. An architectural change will take, relatively, a lot of time and there are a lot of risks in trying to put everything back together. Which is exactly why we need &#8220;parallel&#8221;.<br />
As the name implies this technique is about building the new by the old. Routing old stuff to use the new and then removing the old. An analogy to help visualize this is the building of a new interchange. In most cases, when a new interchange is built, the traffic isn&#8217;t completely stopped. Instead some detours with temporary assignment of old lanes is built. Then the new lanes, bridges etc. are built, then the traffic is detoured to the new infrastructure while the old one is also fixed/changed ans then the open the new interchange. Sure, things slow down while the work is in progress but it doesn&#8217;t stop. The same is true for businesses &#8211; nobody would be willing to halt the business while you get organized to make architectural changes that weren&#8217;t foreseen.<br />
Parallel is needed because no matter how good our initial designs are,  business requirement change and with them the architecture that needs to support them. Additionally in many cases it isn&#8217;t worthwhile solving the ultimate problem on the onset, it would just cost too much (time and money) to do so. Parallel eases the pain of making significant architectural changes. Remember software is there to serve the business not the other way around :)<br />
Is parallel just a technique without architectural implications? Well, yes and no.For one it can be used as a general technique regardless what the architecture is. However one thing you can do is design for evolvability, SOA, REST and EDA (or a combination of them) are all architectural approaches that support evolvability. If you make the investment to build on an architectural style that is makes integration easy it helps a lot when things change. It also helps localize changes within defined boundaries (e.g. within the scope of a single service) and lastly they help in the interim stage where you have both the old and the new in place as you keep the old integration boundaries (events, contracts etc.) intact while adding the new ones.<br />
For instance, in a project I worked on, we&#8217;ve made significant changes using Parallel. The company made a business change which had the the technical ramifications of having  to increase the scale of the database the solution had to handle from hundreds to millions of items. To do that we had to change the back-end database (from files to Cassandra), the way we distribute the work (from centralized to distributed) and what not. The product architecture was built on Event based SOA using context based routing**. Changing the database for example was a localized change within the components that actually has to work with it keeping the rest of the system intact. The distribution was handled in the infrastructure by changing the distribution policy of event again without affecting the existing event stream. Adding new events and components wasn&#8217;t a problem either. routing events to new subscribers is a built in feature in event driven architectures, and the same goes for removing old subscribers when they are no longer needed.We did halt on deploying the interim results into production &#8211; apropos the interchange analogy mentioned above, the interim product had performance problems while the work was in progress but we did have everything working while we made a significant change and we were able to make it gradually while keeping everything connected and working</p>
<p>Lastly note that in real world scenarios you probably use a mix of simplification, parallel and yes even leaps &#8211; We need to have several tools in our tool-set. We can&#8217;t afford to just have a hammer since after all, not every problem is a nail &#8211; right?</p>
<hr />
<p>* Architecture definition as it appears in the<a href="http://arnon.me/2010/05/evolving-architectures-part-whats-software-architecture/"> first post of the series</a> :</p>
<blockquote><p>“Software architecture is the collection of decisions affecting the system’s quality attributes; which have global effects and are hardest to change. Software architecture provides the frame within which the design (code) is built.”</p></blockquote>
<p>** The context based routing was added as part of a previous architectural change. I mention this here as it is an example for a simplification &#8211; an architectural element that was not part of the original architecture but was added later when the actual need arrived (when we had to support both internal operations as well as 3rd party integration)“Software architecture is the collection of decisions affecting the system’s quality attributes; which have global effects and are <strong>hardest to change</strong>. Software architecture provides the frame within which the design (code) is built.”</p>
<p>*** illustration source <a href="http://www.newson6.com/story/13134681/construction-closes-highway-75-at-interstate-44-in-tulsa-again">newson6</a></p>
<script src="http://feeds.feedburner.com/~s/arnonrgo?i=http://arnon.me/2011/04/evolving-architectures-part-vii-parallel/" type="text/javascript" charset="utf-8"></script><div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=QIIUOn30HAY:OEAnvZTu_kc:F7zBnMyn0Lo"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=QIIUOn30HAY:OEAnvZTu_kc:F7zBnMyn0Lo" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=QIIUOn30HAY:OEAnvZTu_kc:cGdyc7Q-1BI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=cGdyc7Q-1BI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=QIIUOn30HAY:OEAnvZTu_kc:G79ilh31hkQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=G79ilh31hkQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=QIIUOn30HAY:OEAnvZTu_kc:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=QIIUOn30HAY:OEAnvZTu_kc:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=QIIUOn30HAY:OEAnvZTu_kc:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=QIIUOn30HAY:OEAnvZTu_kc:gIN9vFwOqvQ" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=QIIUOn30HAY:OEAnvZTu_kc:I9og5sOYxJI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=I9og5sOYxJI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=QIIUOn30HAY:OEAnvZTu_kc:-BTjWOF_DHI"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?i=QIIUOn30HAY:OEAnvZTu_kc:-BTjWOF_DHI" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/CirrusMinor?a=QIIUOn30HAY:OEAnvZTu_kc:qj6IDK7rITs"><img src="http://feeds.feedburner.com/~ff/CirrusMinor?d=qj6IDK7rITs" border="0"></img></a>
</div><img src="http://feeds.feedburner.com/~r/CirrusMinor/~4/QIIUOn30HAY" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://arnon.me/2011/04/evolving-architectures-part-vii-parallel/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		<feedburner:origLink>http://arnon.me/2011/04/evolving-architectures-part-vii-parallel/</feedburner:origLink></item>
	</channel>
</rss><!-- Dynamic page generated in 3.154 seconds. --><!-- Cached page generated by WP-Super-Cache on 2012-05-08 19:23:39 -->

