Android 8 is like that. Not that it’s in the service of a malevolent genius, although I’m beginning to wonder, but it lulls you into a false sense of security, and then throws some significant challenges at you.

I got a new phone last week. I have loved my Sony Experia XA Ultra which I have used for the last two years, but been constantly frustrated by the miserly 16GB main memory. The Experia XA1 Ultra is an almost identical device, but with a decent amount of main storage. I had to forgo the cheerfully "bling" lime gold of the XA, replaced by a dusky metallic pink XA1, but otherwise the hardware change was straightforward.

So, initially, was the transfer. Android now has a feature to re-install the same applications as on a previous device, and, where it can, transfer the same settings. This takes a number of hours, but seems to work quite well. I had to manually transfer a few things, but a couple of hours in I worked through the list of applications, and most seemed to be in order with their settings. I could even see the same pending playlist in the music player which, after a lot of trial and error, I installed to randomly play music while I’m on the bus.

The new version of the Android alarm/clock app seems to be complete b****cks, and more trouble than it’s worth, but there’s no barrier to installing the old version which seems to work OK. My preferred app to get Tube Status updates is no longer available to download, but I could reload the old version from a backup. So that was most of the problems in the upgrade dealt with.

My instincts had abandoned me. It was too easy…

I had also forgotten Weinberg’s New Law. ("Nothing new works")

I got to the gym, and tried to play my music, using the standard Sony music player. Some of it was there, but the playlist I wanted wasn’t. I realised the app could no longer see WMA files (Windows Media format), which make up about 95% of my collection. A bit of googling, and it turned out the recommendation was to install PowerAmp, which I did, and it worked fine.

Then I got on the bus, and tried to play some randomised music. Nothing. The app had the files in its playlist, but couldn’t find them. I rapidly confirmed that the problem again was WMA files, which had suddenly become "invisible" to the app. After yet more trial and error installing, the conclusion is that it’s the Android Media Storage service which is at fault. Apps which build their own index (like PowerAmp) are fine. Apps which are built "the proper way" and use the shared index are screwed, because in the latest version of Android this just completely ignores WMA files.

Someone at Google has taken the decision to actively suppress WMA files from those added to the index. This isn’t a question of a problematic codec or similar – they had perfectly good indexing code which worked, and for some reason it has been removed or disabled. I can only think it’s some political battle between Microsoft and Google, but it’s vastly frustrating that users are caught in the crossfire.

I trust Dante reserved some special corner of Hell for those who break what works, for no good reason. If his spectre wants a bit of support designing it, I’ll be glad to help.

And I’ll resist saying "that wasn’t too bad" when I upgrade my technology…

“No, I’m implying that you’re stupid. You’re inferring it.”

– Wilt, by Tom Sharpe

My latest contract means spending some time on a bus at each end of the day. The movement of the bus means it’s not comfortable to read, so I treated myself to a nearly new pair of decent BlueTooth headphones, and rediscovered the joys of just listening to music. I set the default music player app to “random” and let it do its stuff.

That’s when the trouble started. I started thinking about the randomisation algorithm used by the music player on the Sony phone. I can’t help it. I’m a software architect – it’s what I do.

One good music randomisation algorithm would look like this:

1. Assign every song on your device a number from 1 to n
2. When you want to play a random song, generate a random number between 1 and n, and play the song with that number.

However in my experience no-one ever implements this, as it relies on maintaining an index of all the music on the device, and assigning sequential numbers to it. That’s not actually very difficult, given that every platform indexes the music anyway and a developer can usually access that data, but it’s not the path of least resistance.

Let’s also say a word about generating random numbers. In reality these are always pseudo-random, and depending on how you seed the generator the values may be predictable. That may be the case with Microsoft’s software for picking desktop backgrounds, which seems to pick the same picture simultaneously on my laptop and desktop more often than I’d expect, but that’s a topic for another blog, so for now let’s assume that we can generate an acceptably random spread of pseudo-random numbers in a given integer range.

Here’s another algorithm:

1. Start in the top directory for the music files
2. Pick an item from that directory at random. Depending on the type:
   • If it’s a music file, play it. When finished, start again at step 1
   • If it’s a directory, make it your target and redo step 2
   • If it’s anything else, just repeat step 2

This is easy to implement, runs quickly and plays nicely with independently changing media files. I’ve written something similar for displaying random pictures on a website. It doesn’t require maintaining any sort of index. It generates a good spread of chosen files, but will play albums which are alone under the first level root (usually the artist) much more than those which have multiple siblings.

My old VW Eos had a neat but very different system. Like most players it could work through the entire catalogue in order, spidering up and down the directory structure as required. In “random” mode it simply calculated a number from 1 to approximately 30 after each song, and used that as the number of songs to skip forwards in the sequence.

This was actually quite a good algorithm. As well as being easy to implement it had the side-effect of being at least partially predictable, usually playing a couple of songs by the same artist before moving on, and allowing a bit of “what’s next” guesswork which could be entertaining on a long drive.

So what about the Sony music app on my phone? At first it felt like it was doing the job well, providing a good mix of genres, but after a while I started to become suspicious. As it holds the playlist in a readable form, I could check that suspicion. These are key highlights from the playlist after about 40 songs:

• 1 from ZZ top
• 1 from “Zumba”
• 3 from Yazoo!
• 1 from Wild Cherry
• 1 from Wet Wet Wet
• Several from “Various Artists” with album titles like “The Very Best…”
• 0 from any artist filed under A-S!

I wasn’t absolutely sure about the last point. What about Acker Bilk and Louis Armstrong? Turns out they are both on an album entitled “The Very Best of Smooth Jazz”…

I can also look ahead at the list, and it doesn’t get much better. Van Morrison, Walter Trout, The Walker Brothers, and more Wet Wet Wet

So how does this algorithm work (apart from “badly”)? I have a couple of hypotheses:

• It implements a form of the “give every track a number” algorithm, but the index only remembers a fixed number of tracks numbering a few hundreds (maybe ~1000), and anything it read earlier in the indexing process is discarded.
• It implements the “give every track a number algorithm”, but the random number generator is heavily biased towards the end of the number range.
• It’s attempting a “random walk”, skipping a random number of steps forwards or backwards through the list at each play (a bit like the VW algorithm, but bidirectional). If this is correct it’s odd that it has never gone into “positive” territory (artists beginning with A-S), but that could be down to chance and not impossible. The problem is that without a definite bias a random walk tends to stay in the same place, so it’s a very poor way of scanning your music collection.

Otherwise I’m at a loss. It’s not like I have a massive number of songs and could have run into an integer size limit or similar (there are only around 11,000 files, including directories and artwork).

Ultimately it doesn’t matter that much. I can live with it for a while and I can probably resolve the issue by downloading another music player app. However you can’t help feeling that a giant of entertainment technology like Sony should probably manage better.

Regardless of that, it’s an interesting exercise in analysis, and also potentially in design. Having identified some poor models, what constitutes a “good ” random music player? I’ve seen some good concepts around grouping songs by “mood”, or machine learning from previous playlists, and I’ve got an idea forming in my head about an app being more like a radio DJ, looking for “links” between the songs in terms of their artist names, titles or genres. Maybe that’s the next development concept. Watch this space.

"I have a television / hi-fi / home cinema system which has several components from different manufacturers. I would like to control all of them with a single remote control. I would like that remote control to be configurable, so that I can decide which functions are prioritised, and so that I can control multiple devices without having to switch "modes". (For example, the primary channel controls should change the TV channel, but at the same time and without changing modes the volume controls should change the amplifier volume.) As not all of my devices are controllable via Wi-Fi, Infrared is the required primary carrier/protocol. The ideal solution would be a remote control with a configurable touch screen, probably about 6" x 3" which would suit one-handed operation."

I can’t believe I’m the first person to articulate such a use case. In fact I know I’m not, for two reasons. When I set up the first iteration of my home cinema system in about 2004, I read a lot of magazines and they said similar things.

And then I managed to buy a dedicated device which actually did this job remarkably well. It was called a Sunwave Universal Remote, and had a programmable LCD touchscreen. It had the ability to choose which device functions appeared where, and to record commands from existing remotes or define macros (sequences of commands). This provided some, limited, "mixed device" capability, although the primary approach was modal (select the target device, and then use controls for that device). A set of batteries lasted about a year.

There were only two problems. First, as successive TVs became smarter than in 2004 it became an increasing challenge to find appropriate buttons for all the functions from within the fixed option list. Then, after 13 or so years of sterling service the LCD started to die. I still own the control, but it’s now effectively unusable.

My first approach was to try and get a direct replacement. However it’s clear that these devices haven’t been manufactured for years. The few similar items on eBay are either later poor copies, with very limited functionality, or high-end solutions based on old PDAs at ridiculous prices.

But hang on. "a configurable touch screen, probably about 6" x 3"". Didn’t I see such a device quite recently? I think someone was using one to make a phone call, or surf the internet, or check Facebook, or play Angry Birds, or some such. In fact we all use smartphones for much of our technology interaction, so why not this use case?

Achtung! Rabbit hole! Dive! Dive!

Why not, indeed? Actually I knew it was theoretically possible, because my old Samsung 10" tablet which was about to go on eBay had some software called "Peel Remote" installed as standard, and I’d played with controlling hotel TVs with it. I rescued it from the eBay pile and had an experiment. The first discovery was that while there’s a lot of "universal remote" software on Google Play, most is rubbish, either with very limited functionality, or crippled by stupid amounts of highly-invasive advertising. There are a few honourable exceptions, and after a couple of false starts I settled on AnyMote developed by Color Tiger. This has good "lookup" support to get you started, a nice editing function within the app, and decent ways to backup and share remote definitions between devices. A bit of fiddling got me set up with a screen which controlled our system much better than before, and it got us through all our Christmas watching.

However picking up a 10" tablet and turning it on every time you want to pause a video is a bit clumsy, so back to the idea of using a phone…

And here’s the problem. Most phones have no infrared support. While I haven’t done any sort of scientific analysis, I’d guess that 70-80% (by model) just don’t have what’s known as an "infrared blaster", the element which actually emits the infrared signals. Given that this is very simple technology, not much more than an infrared LED in the phone’s top edge, it’s an odd omission. We build devices stuffed with every sort of wireless and radio interface, but omit this common one used by much of our other technology.

Fortunately it’s not universal, and there are some viable options. A bit of googling suggested that the LG G2 does have an IR blaster, and I tracked down one for about £50 on eBay. It turns up, the software installs…, and it just doesn’t work. That’s when I find the next problem: several of the phone manufacturers who make both TVs and phones (LG and Sony are the most obvious offenders) lock down their IR capabilities, so they are not accessible to third party software. You can use your LG phone to control your LG TV, but that’s it, and f*** all use to me.

Back on Google and eBay. The HTC One M7 and M8 do have IR and do seem to support third-party software. The M8 is a bit bigger, probably better for my use case, and there’s one on eBay in nice condition for a good price. It turns up, the software installs…, and then refuses to run properly. It can’t access the IR blaster. Back on Google and confirm the next problem. Most phones which have been upgraded from Android 5 or earlier to Android 6 have a changed software interface to the infrared which doesn’t work for a lot of third-party software. Thanks a billon, Google.

OK, last roll of the dice. The HTC One M7 still runs Android 5. I find a nice blue one, a bit more money than the M8 ironically, but still within budget. It turns up, the software installs…, and it works! I have to do a few minor adjustments on the settings copied from my tablet, but otherwise straightforward. I had to install some software to make the phone turn on automatically when it’s picked up, and I may still have to do a bit of fiddling to optimise battery life, but for now it’s looking good…

Third time lucky, but it really didn’t have to be that difficult. For reasons which are impossible to fathom, both Google and most phone manufacturers seem to somewhere between ignoring and actively obstructing this valid and common use case. Ironically, given their usual insularity, things are a bit easier in the Apple world, with good support for third party IR blasters which plug into an iPhone’s headphone socket, but that wouldn’t be a good solution given the rest of my tech portfolio. For now I have a solution, but I’m not impressed.

I often hear a statement which worries me, especially but not exclusively in agile projects, along the lines of “we’ll make sure it works when we test it later”.

Now you may think this is an odd view coming from a man who has written testing courses, presented conference papers on testing and developed testing tools, but let me explain myself.

First up, there’s the old chestnut that the objective of testing is not to prove something works, but to find errors. All you can actually do by testing is locate problems to be fixed, although obviously if problems are hard to find, that increases confidence in your product. However the much deeper issue is that testing is commonly viewed as an alternative to properly understanding and documenting the expected behaviour of a system, and reviewing in advance whether a proposed design will deliver that behaviour. That can be a recipe for failure.

Obviously in some areas this is an acknowledged and viable trade-off. If we are exploring functional alternatives, or working in a problem space where extracting documented requirements is tricky, then agile development and testing is a powerful solution, and we accept the rework that may result where we get it wrong. Having said that, even in something like UI development it may be better to develop cheap models such as wireframes, and at least attempt to explore solution fit before we commit too much to code.

The problem is that when we come to the more fundamental architectural elements and non-functional behaviour, the dynamics change dramatically. The best way to show this is a variant of the testing “V Model”:

For functional details, the gap between development and testing is small, and they can quickly be reworked and retested. However some of the key architectural and non-functional aspects can only be fully tested late in the delivery process (and frequently only late in the overall programme), if at all. The “testing gap” becomes huge, the impact of any change substantial, and the rework path lengthy.

One challenge is that many non-functional tests require an environment representative of the technology and scale of the production system. If this is provided at all, it is typically late in the project, or testing has to be shoe-horned into a short window on the production system before operations commence. If that uncovers a major issue, it is simply too late.

That’s assuming that the issue is detectable. In an agile development, it may be difficult to understand “what acceptable looks like”, if there is no adequate agreed, documented definition of the expected non-functional behaviour.

The other challenge is that good non-functional testing is hard, and limited in what it can achieve. Simulating a peak load is difficult, especially with the variety of data in a real production peak. You can simulate planned and unplanned equipment failures and restarts, but by definition only predictable events. If a problem only emerges from lengthy running or a “perfect storm” event, then testing is unlikely to uncover it. Basically resilience is testable, performance may be testable, reliability isn’t. Similar considerations also apply to other non-functional aspects like security.

The Solution

The solution is to adopt an analytical and predictive approach: trying to understand, articulate and document the expected behaviour of the solution, before you build it. Importantly this is not just thinking about the solution (although thinking is vital), but thinking with models.

Models in this context take many forms. They can be diagrams, possibly based on UML, but not necessarily: for example reliability block diagrams or fault tree analyses are powerful tools to understand resilience and reliability. They can be spreadsheets, for example profiling expected transaction mixes and their relative resource requirements. They can also be active software, whether simulations of some expected behaviour, or point implementations to quantify some aspect of the solution, but the point is that their purpose is to understand the solution before a major technical commitment, not to deliver functionality. Irrespective of form all models should lend themselves to a quantitative understanding of the solution, not just “what?”, but “how much?” and “how well?”.

For example, here’s a simple redundancy scheme modelled using RelQuest, my own Visio-based fault tree analysis tool, from which we can not only understand the various combinations of failures which lead to loss of service, but the relative probability and impact (e.g. Mean Time to Repair) for each combination.

Models and simulations provide you with an early understanding of the system behaviour, so you can understand whether something should work, or not, and if not where to focus your efforts. They can be detailed, like the example fault tree above, or doing an early first pass on a platform provider’s sizing tool, but a more approximate approach may also provide value.

Numbers are your friends. I am a great fan of Fermi estimates (see the sidebar) – quick “order of magnitude” approximations to see if you have understood the key elements in a problem, and whether the answer looks viable or not.

You can easily get viable estimates of this type for performance, capacity or reliability. If the answer is “no problem”, like we can easily accommodate millions of transactions per hour on a single server and we expect thousands, then you’re probably fine. If the answer is the other way round, like the developer who proudly presented me a solution which would take 1s CPU time to do a calculation, but we needed to do a thousand a second, then the design needs to change (I got it down to 2ms, which was acceptable). If it’s marginal, then you probably need to do a more accurate model and calculation, or build a greater degree of flexibility into the solution.

Simulations or low-volume experiments may be a valid way to understand CPU, storage and memory usage, network bandwidth requirements, threading, virtualisation, and even failover behaviour. Anything which scales linearly can be measured at low volume and extrapolated, but you need to be wary of areas such as network latency or storage throughput where that may not be valid.

Ultimately anything which builds your understanding and proves that you have thought about the problems in advance is good, even if some detail may only be confirmed at later stages. The key point is that the problems become targets for analytical thinking rather than hope and prayers, and that makes them solvable.

The Conclusion

Testing on its own is absolutely necessary, but very much not sufficient. For tests to be meaningful you have to describe the predicted behaviour in advance, and for the system to have any chance of passing those tests it has to be engineered accordingly. We increasingly seek to drive functional development from written user stories and behaviour specifications. In the same way professional development must be driven by quantitative models which forecast non-functional behaviour for testing to confirm, not discover in surprise.

The web site suggested I ring the reservation number, which I did, listened to 5 minutes of surprisingly loud rock music (you never mistake being on hold for Virgin with anyone else), and got through to a helpful chap. He said "OK, I can see the problem, I will re-issue the ticket." Two minutes of more distinctive music, and he invited me to try again. Same result. He confirmed that we were definitely booked in and had our seat reservations, and suggested that I wait until I get to the airport. "They will help you there." Fine.

Next morning, we were tackled on our way into the Virgin area by a keen young lady who asked if we had had any problems with check in. I said we had, and she led us into what can best be described as a "krall" of check-in terminals, and logged herself into one. This displayed a smart check-in agent’s application, complete with all the logos, the picture of Branson’s glamorous Mum, and so on. She quickly clicked through a set of very similar steps to the ones I had tried, and then click OK. "Oh, that’s odd", she said.

Next, she opens up a green screen application. Well, OK, it’s actually white on a Virgin red background, but I know a green screen application when I see one. She locates my ticket, checks a few things, and types in the command to issue my pass. Now I’m not an expert on Virgin’s IT solutions, but I know the word "ERR" when I see it. "Oh, that’s not right either" says the helpful young lady "I’ll get help".

Two minutes later, the young lady is joined by a somewhat older, rather larger lady. (OK, about the same age as me and she looked a lot better in her uniform than I would, but you get the idea.) "Hello Mr Johnston, let’s see if we can sort this out". She takes one look at the screen and says "We actually have two computer systems, and they don’t always talk to each other or have the same information."

… which could be the best, most succinct summary of the last 25 years of my career I have heard, but I digress…

Back to the story. The new lady looks hard at both applications, and then announces she can see the problem (remember, all this is happening on a screen I can see as well as the two Virgin employees). "Look, they’ve got your name with a ‘T’ here, and no ‘T’ here" (pointing to the "red screen" programme).

Turning to the younger lady, she says "Right, this is how to fix it." "Type DJT, then 01" (The details are wrong, but the flavour is correct…) "Put in his ticket number. Type CHG, then enter. Type in his name, make sure we’ve got the T this time. Now set that value to zero, because this isn’t a chargeable change, and we can do a one letter change without a charge. Put in zero for the luggage, we can change that in a minute. Type DJQ, enter. Type JYZ, enter. OK, that’s better. Now try and print his pass." Back to the sexy new check in app, click a few buttons, and I’m presented with two fresh boarding passes. Job done.

Now didn’t we have a series of books where a bunch of older, experienced wizards taught keen you wizards to tap things with sticks and make incantations? The solution might as well have been to tap the red screen programme with a wand and shout "ticketamus"…

The issues here are common ones. Is it right to be so dependent on what is clearly an elderly and complex legacy system? Are the knowledge transfer processes good enough, or is there a risk that the next time the more experienced lady who knows the magic incantations might not be available? Why is such a fundamental piece of information as the passenger names clearly being copy typed, not part of the automated integrations? As a result, is this a frequent enough problem that there should really be an easier way to fix it? Ultimately the solutions are traditional ones: replace the legacy system, or improve its integrations, but these are never quick or easy.

Now please note I’m not trying to get at Virgin at all. I know for a fact that every company more than a few years old has a similar situation somewhere in the depths of their IT. The Virgin staff were all cheerful, helpful and eventually resolved the problem quickly. However it is maybe a bit of a management error to publicly show the workings "behind the green screen" (to borrow another remarkably apposite magical image, from the Wizard of Oz). We expect to see the swan gliding, not the feet busily paddling. On this occasion it was interesting to get a glimpse, and I was sympathetic, but if the workings cannot be less dependent on "magic", maybe they should be less visible?

At the risk of mis-quoting the Bible, the answer is legion, for they are many…

To get the functionality I want

I have always been a believer in getting computers to automate repetitive actions, something they are supremely good at. At the same time I have a very low patience threshold for undertaking repetitive tasks myself. If I can find an existing software solution great, but if not I will seriously consider writing one, or at the very least the “scaffolding” to integrate available tools into a smooth process. What often happens is I find a partial solution first, but as I get tired of working around its limitations I get to the point where I say “to hell with this, I’ll write my own”. This is more commonly a justification for personal projects, but there have been cases where I have filled gaps in client projects on this basis.

Related to this, if I need to quickly get a result in a complex calculation or piece of data processing, I’m happy to jump into a suitable macro language (or just VB) to get it, even for a single execution. Computers are faster than people, as long as it doesn’t take too long to set the process up.

To explore complex problems

While I am a great believer in the value of analysis and modelling, I acknowledge that words and diagrams have their limits in the case of the most complicated problem domains, and may be fundamentally difficult to formulate and communicate for complex and chaotic problem domains (using all these terms in their formal sense, and as they are used in the Cynefin framework, see here).

Even a low-functionality prototype may do more to elicit an understanding of a complex requirement than a lot of words and pictures: that’s one reason why agile methods have become so popular. The challenge is to strike a balance, and make sure that an analytical understanding does genuinely emerge, rather than just being buried in the code and my head. That’s why I am always keen to generate genuine models and documentation off the back of any such prototype.

The other case in which I may jump into code is if the dynamic behaviour of a system or process is difficult to model, and a simulation may be a valid way of exploring it. This may just be the implementation of a mathematical model, for example a Monte Carlo simulation, but I have also found myself building dynamic visual models of complex interactions.

To prove my ideas

Part of the value I bring to professional engagements is experience or knowledge of a range of architectural solutions, and the willingness to invoke unusual approaches if I think they are a good fit to a challenge. However it’s not unusual to find that other architects or developers are resistant to less traditional approaches, or those outside their comfort zones. Models and PowerPoint can go only so far in such situations, and a working proof of concept can be a very persuasive tool. Conversely, if I find that it isn’t as easy or as effective as I’d hoped, then “prove” takes on its older meaning of “test” and I may be the one being persuaded. I’m a scientist, so that’s fine too.

To prove or assess a technology

Related to the last, I have found by hard-won experience that vendors consistently overstate the capabilities of their solutions, and a quick proof of concept can be very powerful in confirming or refuting a proposed solution, establishing its limitations or narrowing down options.

A variant on this is where I need to measure myself, or others, for example to calibrate what might or might not be adequate productivity in a given situation.

To prove I can

While I am sceptical of overstated claims, I am equally suspicious if I think something should be achievable, and someone else says “that’s not possible”. Many projects both professional and personal have started from the assertion that “X is impossible”, and my disbelief in that. I get a great kick from bending technology to my will. To quote Deep Purple’s famously filthy song, Knocking At Your Back Door, itself a exploration into the limits of possibility (with censorship), “It’s not the kill, it’s the thrill of the chase.”.

In the modern world of agile development processes, architect and analyst roles are becoming blurred with that of “developer”. I have always straddled that boundary, and proving my development abilities my help my credibility with development teams, allowing me to engage at a lower level of detail when necessary. My ability to program makes me a better architect, at the same time as architecture knowledge makes me a better programmer.

To make money?

Maybe. If a development activity can help to sell my skills, or advance a client’s project, then it’s just part of my professional service offering, and on the same commercial basis as the rest. That’s great, especially if I can charge a rate commensurate with the bundle of skills, not just coding. My output may be part of the overall product or solution or a enduring utility, but more often any development I do is merely the means to an end which is a design, proof of concept, or measurement.

On the other hand, quite a lot of what I do makes little or no money. The stuff I build for my own purposes costs me little, but has a substantial opportunity cost if I could use the time another way, and I will usually buy a commercial solution if one exists. The total income from all my app and plugin development over the years has been a few hundred pounds, probably less than I’ve paid out for related tools and components. This is a “hobby with benefits”, not an income stream.

Because I enjoy it

This is perhaps the nub of the case: programming is something I enjoy doing. It’s a creative act, and puts my mind into a state I enjoy, solving problems, mastering technologies and creating an artefact of value from (usually) a blank sheet. It’s good mental exercise, and like any skill, if you want to retain it you have to keep in practice. The challenge is to do it in the right cases and at the right times, and remember that sometimes I really should be doing something else!

The book should be valuable to many: If you want to be an expert software designer, this book provides an overview of the skills and knowledge you need to develop. If you want to recruit such a person, this provides a set of key indicators and interview prompts. If you are in one of those software development organisations which believes that quality architecture can somehow emerge by magic from the unguided work of undifferentiated coders, this might make you think again.

If you are, or think you are, a software architect, this book should act in the same way as a good sermon: it will remind you of what already know you should be doing, and act as a prompt against which you can measure your own performance and identify areas for improvement. It reminded me that I can sometimes be slow to listen to the views of others, or evidence which may change a design, and slow to engage with new technologies, and I will try to act on those prompts.

This book resolutely promotes the value of modelling in software design. Formal models and analysis have their place, but so do informal models, sketches, and ad-hoc notation. The key point is to externalise ideas so that they can be shared, refined and “tested” in the cheapest and most effective of ways, on paper or a whiteboard. We are reminded that all these are hallmarks of true expert software designers. Code has its place, to prove the solution or explore technicalities, but it is not the design.

The book also promotes the value of richness in these representations. Experts should explore and constantly be aware of alternatives, and model the solution at different levels of abstraction, in terms of both static and dynamic behaviours. Continuous assessment means not only testing, but simulation. If required, the expert should build his or her own tools. While solving simple problems first is a good way to get started, deep, early understanding of the problem space is essential, and experts must understand the whole context and landscape well enough to make and articulate design prioritisations and trade-offs.

I thoroughly recommend this book. It may seem slight, but it delivers a powerful reminder on the process of design, and the necessary, different thought processes to succeed with it.

A watch-based alarm doesn’t work very well for me – my hearing is just not good enough. Seeking to reduce the amount of gadgets I carry, I have therefore for many years relied on phones and their PDA predecessors to fulfil the function of alarm clock, especially when I’m travelling. It’s not a difficult role, and I have not had to complain about it. Until now.

In my normal weekly cycle I don’t have much need for a clock as I wake naturally at about the right time each day. This makes the operation of such a function even more critical, as it has to be absolutely reliable on days which are exceptions, and I don’t get much opportunity to do much advance "testing" of what I assume is something that should "just work". However, I do have the alarm set every day when I’m working away from the home, and although I couldn’t be absolutely sure I was coming to suspect that it wasn’t going off at the right time. The first couple of times I assumed "user error": incorrect settings, volume too low etc., but I had eventually eliminated those, and confirmed the behaviour: the alarm didn’t go off at the programmed time. It went off after I had woken up and clicked the button to wake up my phone’s screen.

This is about as useful as a chocolate fireguard, and about as welcome as a fart in a spacesuit.

A bit of Googling confirmed that the problem is quite widespread. I’ve read stories of people with new phones being late for work or missing important appointments. Others describe a similar problem with other programs including not getting notified promptly of night-time messages or similar: potentially quite a problem for those "on call". Fortunately I caught the problem before it caused me any trouble, but that might not have been the case, as I have an upcoming trip with about 8 flights and several other dawn starts.

The web is full of useless "solutions" like factory resetting the phone, but after eliminating those, I tracked down the cause of the problem. With Android 6 ("Marshmallow"), Google introduced something called "Doze" mode. This is a deep sleep mode which kicks in if the device is at rest, screen off, and no significant ongoing activity like an active data transfer. You know, like it tends to be at night. In this state, the system not only slows down processing, but also suspends the bulk of normal background activity. This includes, for no articulated good reason, suspending timers and related event triggers. So your alarm application doesn’t know what time it is, and doesn’t fire. Your messaging app doesn’t know when to poll for incoming events. Simple, core functions of your smartphone just cease to work.

Allegedly, if you change the code of your alarm or other app to use a "different kind" of timer, that should work, but after testing four or five I concluded that this is just not true, certainly on my phone. In any case, I usually just use the stock Android "clock" app, and surely they would have remembered to update that, wouldn’t they? You can also nominally turn off Doze for selected applications, but as far as I can see it makes bugger all difference.

It turns out that the root problem is that in at least some Android 6 implementations, Doze mode actually disables the underlying operating system events on which the other timers are based. It doesn’t matter how sexy your alarm app is, or whether Doze knows about it or not, if the underlying timers are blocked!

There’s a heap of advice on the web about how to disable Doze for individual apps (tried that, doesn’t work), but not about how to disable it completely. I’d tried all sorts of settings without success. However I finally found a useful little app called Disable Doze, which does what it says on the tin, and turns Doze off completely. Allegedly (according to Google) this would result in my phone discharging its battery at a terrifying  rate and ending up doing a Galaxy Note 7 impersonation, but I can confirm that with Doze off in light use my phone is still only consuming about 10% battery per day. The only noticeable effect so far is that alarms and notifications work again.

My worry is that until Google acknowledge their mistakes, they may come up with another "improvement" which disables this fix. I don’t know what tests Google perform in this area, but they are clearly inadequate. This really is a "0 out of 10" effort, a true "breaking change".

However for now things are looking good, and hopefully this blog will help alert others to the problem and the fix.

Call me old-fashioned, but I very much prefer using a mouse to a trackpad or its relatives, and since my earliest experiences with Windows 3.0, I’ve tended to go for Microsoft mice by preference. Over the years they gained additional buttons and a wheel, lasers replaced the ball, and wireless connections replaced wires, but the core ergonomics and functionality have been maintained and gradually improved. About 2005 this resulted in the Microsoft wireless mouse, of which we have had several, colour matched to different PCs.

However when I started using a MacBook as my primary PC, I had a couple of challenges with this strategy. Firstly, while it may be pure vanity I like to have a mouse which visually matches my laptop, and the somewhat “chunky” Microsoft options didn’t really float my boat. More importantly with the limited set of ports on a MacBook I couldn’t afford to tie up a USB port with the mouse or (worse) risk damaging one if I forgot to unplug the wireless dongle, something I have experienced on other PCs. As the MacBook runs with BlueTooth and internal wireless permanently powered on, a BlueTooth solution seemed sensible.

A visit to PC World didn’t reveal many options. Apart from the Apple mouse (the ergonomics of which I don’t particularly like) most mice seemed to be either wired, WiFi based and/or very chunky. Then I discovered the HP Z5000, an elegant thin white slab, with BlueTooth, two buttons and a wheel. Great!

Or so I thought… Time revealed two problems. One is ergonomic: the wheel is the same smooth white plastic as the body, and if your fingers are at all wet or slippery it is completely impossible to scroll accurately. The  other is electronic, with the PC and mouse periodically becoming “disconnected” and requiring some random mouse movement or, occasionally, cycling the mouse’s power to re-establish connectivity. For reasons not immediately apparent, this appears to become worse when working in bright ambient outside light, just the conditions under which you can’t afford intermittent loss of the mouse’s position.

After working with these limitations for a year, we finally gave up after our last holiday, and decided enough was enough. Research suggested a new option, in the form of the HP Z8000.

This is a piece of gorgeous industrial design: a thin black slab edged in brushed aluminium which is a very good match to the MacBook’s own finish. The top surface is a capacitative touch panel – tap to click the mouse, swipe forward and back to simulate the wheel scrolling normally, or left and right to simulate a horizontal scroll. It also allegedly has much improved power management and connectivity. Wonderful! Well worth the £40+ asking price.

Or so I thought… To start with there’s no evidence whatsoever that HP have addressed the connectivity problems. If anything, they are worse. More of an issue is that the touch panel just doesn’t work very well. If you are very careful and precise with all your movements it’s just about usable in a program like Microsoft Word. However if the software supports any form of horizontal scrolling (e.g. XnView, or Windows Explorer in “tile” mode), then you end up with a working context which jumps about constantly and randomly. With some programs, such as CaptureOne, it becomes almost unusable.

Back to Amazon, and I discover this gem:

Yes, it’s a Microsoft “Designer” BlueTooth mouse. Price about £16, although it does vary. Just a lump of black plastic, although at least it’s now thin enough to work alongside the MacBook. Textured scroll wheel and two obvious buttons, each with a definite “action”. Picks up the control points in CaptureOne without issue. And so far not a single random connectivity problem.

Function trumps form, substance beats style, in mice if not always in men!

If faced with political extremism, the predominantly liberal groups who control and shape our technology would typically be horrified and opposed. However at the same time they are forcing on us fashions and design paradigms which in their own way are just as odious, impacting the richness of our experience, and limiting rather than improving our ability to interact with technology.

I refer, of course, to the Colour Nazis. The members of this movement probably don’t think of themselves that way, and if forced to adopt a label would choose something much more neutral, but it is becoming apparent that some of their thinking is not that different.

This is not the first time I’ve complained about this. In 2012 I wrote “Tyranny of the Colour Blind, or Have Microsoft Lost Their Mojo?”. The trouble is that things are getting worse, not better. Grappling with Office 2016 I’m coming to grips with some really dramatically stupid decisions which can only be explained by a Nazi zeal to remove the colour from our technological interactions.

Here’s a quick test. Find Open, Save and the Thesaurus in Office 2003:

Now let’s try Office 2010:

Not too bad. The white background actually helps by increasing contrast, and the familiar splashes of colour still draw your eye quickly to the right icons, although the Thesaurus is a bit anonymous. Now let’s try Office 2016:

The faded grey on a grey background colour scheme has wiped out most of the contrast, and you’d be struggling to make these out if you have ageing sight in a poor working environment. The pale pastel yellow of “Open” is still just recognisable, but the “Save ” button has turned to a weird pale purple, and the Thesaurus is completely anonymous. I’d have to go hunting by hovering over each and reading the tooltip. (Before anyone shouts, I know I’ve used an add-in menu here to get a like-for-like comparison, but all this is equally true for the full-sized ribbon controls.)

Now let’s look at a really stupid example. One of Word’s great strengths is the ability to assemble and review tracked changes from multiple reviewers. In Word 2010 each will be assigned a distinctive colour, and I can very quickly see who’s who:

OK that works well. Let’s see what they’ve done in Office 2016:

WTF! One place where colour has a specific role as an information dimension, and they’ve actually taken it away. In the document the markup does use some colour, but in the form of a few pale pastel lines. Instead the screen is cluttered up with the name of the author against every single change, which makes it unreadable if multiple authors have made changes to a single page.

I am always among the first to remind designers not to rely on colour, as it doesn’t work well for about 8% of the population, or in some viewing conditions. But that’s no reason to remove it. Instead you should supplement it (e.g. make icons both distinctive colours and shapes), or allow the users a choice. Word 2016 should allow me to choose whether to use colour or explicit names in markup balloons, and I wouldn’t be having this rant.

There is apparently a name for this fad, “Complexion Reduction” (see Complexion Reduction: A New Trend In Mobile Design by Michael Horton). The problem is that its advocates seem to have lost sight of some key principles of human-computer interaction. One of these is that for normally-sighted people there’s a clear hierarchy in how we spot or identify things:

1. Colour. If we can look for a splash of colour, that’s easiest. That’s why fire extinguishers are red, or the little red coat was so poignant in Schindler’s List.
2. Shape / position. We manage a lot of interactions by recognising shapes. That’s why icons work in the first place. We even do this when the affordance supplies text as well. If you’re a native English speaker and reader you will inevitably have tried to move a door the wrong way, because “PUSH” and “PULL” have such similar shapes, and your brain tries shapes first, text second.
3. Text. When all else fails, read the instructions. That’s not a joke, it’s a real fact about how people’s brains work. If I have to go hunting in a menu or reading tooltips, then the designer has failed miserably.

Sadly I don’t know if there’s any way to influence this. These decisions are probably being made by ultra-hip youngsters with ironic beards and 20 year old eyes who don’t really get HCI. I’d just like one of them to read this blog.

Addendum — May 2019

So the hierarchy for interactions is first colour, then shape, then text.

So please could someone explain to me why the latest versions of Android have also decided to force almost all application icons into a uniform shape (circular on my Sony phone, a rounded rectangle on my Samsung tablet) with exactly the same background colour?

On my phone, all the main Google apps now have icons which are white circles with tiny splashes of the same four colours. The Sony apps (including the main phone functions) are white circles with small icons, using the same pale blue, within them. To add an extra spice, the launcher I use occasionally moves the icons around, if I add a new front-page app or the labels change.

My poor brain has no chance whatsoever. I open my phone, and then have to READ labels to make sure I’m opening the right app. Hopeless!

It’s not that way with core software, and especially Microsoft products (although they are not the only offenders). The big software providers continue to foist endless upgrades on us, but I can’t see any evidence of improvement. Instead I can actually see a lot of what is known in other trades as "de-contenting", taking away useful capabilities which were there before and not replacing them.

Windows 10 continues to reveal the loss of features which worked well under Windows 7, with unsatisfactory or no replacements. I mourn the loss of the beautiful "aero" features of Windows 7 (with its semi-transparent borders and title bars)  and a number of other stylistic elements, but there are some serious functional omissions as well. I couldn’t work out why my new laptop kept on trying to latch onto my neighbour’s Wifi, rather than use my high powered but secure internal service, and discovered that there’s now no manual mechanism to sort WiFi networks or set preferences. There is, allegedly, a brilliant new automated algorithm which just makes it automatic and no bother to the user. Yeah, right. Dear Microsoft, IT DOESN’T ***** WORK. Fortunately in the way of these things I’m not the only one to complain, and literally in the last couple of weeks a helpful Belgian developer has released a tiny utility which replaces the ability to list and manipulate the WiFi networks known to a Windows 10 machine (https://github.com/Bertware/wlan10). That’s great, and the young man will be receiving a few Euros from me, but it shouldn’t have to be this way. By all means add an automatic sequencer to the new system, but leave the manual mechanism as well.

However, my real object of hate at the moment is Microsoft Office. Since I set up the new MacBook with Windows 10 it’s never been entirely happy with the combination of versions I want to use: Office 2010, plus Skype for Business 2016. (Well actually I’d really prefer to use Office 2003, but I’m over that by now :)) I’ve had the odd problem before, having to install Visio 2016 because Visio 2010 and Skype/Lync 2016 keep breaking each other. I’m not sure how that’s even possible given the "side by side" library architecture which Microsoft introduced with Windows XP, but somehow they managed it, and they clearly don’t care enough about the old versions to fix the issue.

I could live with that, but a couple of weeks ago more serious problems set it. There was an odd "blip", and then OneNote just showed blank notebooks with the ominous statement "There are no sections open in this notebook or section group". That looked like a major disaster, as I rely on OneNote both to organise my work and to-do lists on a daily basis, and as a repository of notes going back well over 10 years. However a quick check online, and on other devices revealed that my data was fine. I lost a good chunk of a working day to trying to fix the problem, including a partial installation of Office 2016 to upgrade to OneNote 2016. That’s a lot more difficult that it should be, and something Microsoft really doesn’t want you to do. Nothing worked. By the end of the day I was so messed up I did a system restore to the previous day, hoping that would restore my system state and fix the original problem. At first glance this appeared to fix Office, although OneNote was still showing blank notebooks. However I then had a moment of inspiration and went online to OneDrive.com, and clicked the "edit in OneNote" option. This magically re-synced things, and got my notebooks re-opened on the laptop. Success?

Unfortunately not. Things seemed OK for a few days, but then I started getting odd error messages, and things associated with Outlook and the email system started breaking. Apparently even a complete "System Restore" hadn’t completely restored the registry, and my system couldn’t work out which version of Outlook was installed. An office repair did no good, and eventually I decided to bite the bullet and upgrade to Office 2016. Even that wasn’t trivial, and took a couple of goes but eventually I got there, and my system is now, fingers crossed, stable again.

And that would be fine if Office 2016 was actually a straightforward upgrade from its predecessors, maintaining operational compatibility under a stable user interface, but that’s where I came in. The look and feel, drained of colour and visual separation, is in my opinion poorer than before but I’ll probably get used to it. I’ve got an add-in (the excellent Ubit Menu) which gives me a version of the ribbon which mimics the Office 2003 menus, and which I also used with Office 2010, so I can quickly find things. But what that can’t do is fix features which Microsoft have just removed.

Take Outlook for example. I really liked the "autopreview" view on my inbox folders. Show me a few lines of unread emails, so I can both quickly identify them and, importantly, scan the content to decide whether they need to be processed urgently and if any can just be deleted, but hide the preview once I’ve read them. Brilliant. Gone. I have multiple accounts under the same Outlook profile, which is how Microsoft tell you it’s meant to work, and in previous versions I could adjust the visual properties of the folder pane at the left so I could see all the key folders at once. Great. Gone. Now I’m stuck with a stupid large font and line separation which would be great if I was working on a tablet with my fingers and a single mail account, but I’m not. Dear Microsoft, some people still use a ****** PC and a mouse…

Or take Word. Previous "upgrade" Office installations carefully preserved the styles in the "Normal" template, so that opening a document in the new version preserved its layout. Not this time. I’ve had to go through several documents with detail page layouts and check each one.

None of this is a disaster, but it is costing me time and money and it wouldn’t be necessary if either Microsoft didn’t keep forcing us to upgrade, or if they made sure to keep backwards compatibility of key features. It’s also not just a Microsoft problem: Adobe and Apple are equally guilty (witness features lost from recent versions of OSX, or the weird user interface of Acrobat XI). The problem seems to be that the big software companies don’t seem to have a business model for just keeping our core software "ticking over", and they confuse change with improvement, which is proving to not be the case now that these systems are functionally mature and already do what people need them to do.

I’m not sure what the answer is, or even if there is an answer. We can’t take these products away from the companies, and we don’t want them to become moribund and abandoned, gradually decaying as changes elsewhere render them unusable. Maybe they need to listen harder to their existing customers, and a bit less to potential "captures", but I’m not convinced that’s going to happen. Let the struggle continue…

It’s a while since I built a rule engine, and I’ve never truly designed an expert system before, so I thought it might be a good idea to do some reading and understand the state of the art. That’s when the the trouble started. My client recommended a book on analysis for knowledge based systems, which I managed to track down for 1p + postage (that should have warned me). I got through most of the introduction, but statements such as "these new-fangled 4GLs might be interesting" and "we don’t hold with this iterative development malarkey" (I paraphrase slightly, but not much) made me realise that the "state of the art" documented was at least a generation old. The book has a few sound ideas about data structure, but pretty much everything it says about technology or process is irrelevant.

Back on Amazon, and I tried searching for "expert system", "knowledge base" and "rule based system". That generates a few hits, but nothing of any substance younger than about 12 years old, nothing on Kindle, and prices varying dramatically between a few pence and the best part of £100, both indications of "this is an old, rare book" and neither tempting me to make a punt. It doesn’t help that the summaries tend to be a list of technologies I’ve never heard of, and few seem to be focused on re-usable concepts and techniques.

OK, I thought. There’s obviously just a new term and I don’t know it. Wikipedia wasn’t much help, observing that the term "expert system" has largely gone out of use, and offering two opposing views why. Either expert systems became discredited and no-one does them any longer (I don’t believe that), or they just became "business as usual" (quite possible, but a good reason why you might write a book about them, not the opposite). No indication of the "modern" term, and few recent references.

Phone a friend. I emailed a couple of friends both of whom are quite knowledgeable in a breadth of IT topics hoping that one of them might say "Oh yes, we now just call them XXX". Nope. Both suggested AI and one suggested "cognitive computing", but as I’ve already observed, that’s a fundamentally different topic. Beyond that both were just suggesting the same terms I’d already tried.

Googling a practical question such as "rule based systems in .NET" produces a few hits and suggests that the state of technology support is pretty good. For example, Microsoft put the "Windows Workflow Foundation" into .NET in about 2008, and this includes a powerful rule engine which is perfectly reusable in its own right. So the technology is there, but again there’s not much general information on how to use it.

This appears to be a case where fashion is getting in the way. If something works, but is not "in", then authors don’t want to write about it, and editors don’t actively commission material. If the "thing" is something where the technology has improved, but not in a "sexy" way, then it goes unreflected in deeper or third party literature. Maybe that explains why Oracle seem driven to rename all their technologies every couple of years, it’s their way of attracting at least a modicum of interest even if it does confuse the hell out of developers trying to work out what has changed, and what really hasn’t.

So be it. I’m going to build a rule-based expert system knowledge base, and I don’t care if that’s not the modern term. It’s just frustrating that no-one seems to have written about how to do this with 2015 technology…

128GB in 2 hours is roughly 1GB/minute, or say 160 Mbps (megabits per second). That’s about the effective transfer rate for USB 2, and is getting on for Gigabit LAN speed. It’s about 50 times faster than the best I get from BT Broadband, and probably over 100 times faster than the sustained broadband bandwidth over a week, which is about how long 128GB would take to transfer. Plus remember that that’s the download speed, and upload is another factor of ten slower…

Now I would be the first to admit that there are some limitations to the "pigeon post" architecture, especially in terms of range. The latency also precludes chatty protocols. But in terms of sheer transfer bandwidth Yankee Doodle Pigeon has "broadband" beaten hands down!

I decided to replace a large picture frame in my office with a electronic display, on which I could see a rolling presentation of my own images. This is not a new idea, but decreasing prices and improving specs brought into my budget the option of a 40"+ 4K TV, which on the experience of our main TV should be an excellent solution.

New Year’s Eve brought a trip to Richer Sounds in Guildford. As usual the staff were very helpful and we quickly narrowed down the options to equivalent models from Panasonic or Sony. The Panasonic option was essentially just a smaller version of our main TV, but the colours were slightly "off" and we preferred the picture quality of the Sony. The Panasonic’s slideshow application is OK, but limited, but the Sony’s built-app looked downright crude. It looked like a difficult choice, but then I realised that the Sony operating system is something called "AndroidTV" with Google Play support, and promised the option of a more open platform, maybe even development myself. Sold!

In practice, it’s exactly as I expected. The basic hardware is good, but the Sony’s default applications beyond the core TV are a bit crude. However a bit of browsing on Google Play revealed a couple of options, and I eventually settled on Kodi, a good open-source media player, which does about 90% of what I want for the slideshow. Getting it running was a bit fiddly, not least because a key picture-handling setting has to be set by uploading a small XML file rather than via the app’s UI, but after only a bit of juggling it’s now running well and doing most of what I want.

Beyond that, I can either develop an add-on for Kodi, or a native application for AndroidTV. However as the existing developer community has provided a 90% solution, I’m not in a great hurry.

I call that a result for platform vs product…

The problem is that my "colleagues" (I’m not quite sure what the correct collective term is here) are wrong. For different reasons. They are all thinking of the camera as a unitary product, and none of them (even Molitor, who claims to have some experience as a system architect) are thinking as they should, of the camera as a platform.

OK, one at a time, please…

There are a lot of good ideas in Ming Thein’s article. A lot of his suggestions to improve current mirrorless cameras are good ones with which I agree. The trouble is that he is trying to design "Ming Thein’s perfect camera", and I suspect that it wouldn’t be mine. For a start it would end up far too heavy, too expensive and with too many knobs!

Kirk Tuck gets, this, and his article is a sensible exploration of trade-offs and how one photographer’s ideal may be another’s nightmare. However he paints a picture of flat-lining development which is very concerning, because there are some significant deficiencies in current mainstream cameras which it would be great to address.

Andrew Molitor then picks up this strand, and tries to explain why all camera feature development is difficult, and prohibitively expensive, and why Expose to the Right (ETTR) is especially difficult. Set aside that referring to Michael Reichmann as "a pundit" is unkind and a considerable underestimation of that eminent photographer’s capabilities, there are several fallacies in Molitor’s articles. Firstly, it just would not be as difficult as claimed to implement ETTR metering, or any variant of it. It’s just another metering calculation. If you have a camera with some form of live histogram or overexposure warning, then you can already operate this semi-manually, tweaking down the exposure compensation until the level of clipping is what you want. If you can do it via a predictable process, then that enormously powerful computer you call a digital camera can easily be made to replicate the same quickly and efficiently. That’s what the metering system does. It’s even quite likely that the engineers have already done something similar, but hidden it. (Hint: if you have a scene mode called something like "candle-lit interior", you’re almost there…)

I suspect the calculations of grossed-up cost are also fallacious. If that were the case, in a market which manages US sales of only a few tens of thousands of mirrorless cameras per year (for example), we would never get any new features at all. The twin realities are that by combining multiple features into the normal streams of product or major release development, many of the extra costs are amortised, but we also know that the big Japanese electronics companies apply different accounting standards to development of their flagship products. If Molitor’s argument was correct, we would not see features in each new camera such as a scene mode for  "baby’s bottom on pink rug" (OK, I made that one up :)) or in-camera HDR, and things like that don’t seem to be a problem. I simply cannot believe that "baby’s bottom on pink rug" will generate millions of extra dollars revenue, compared with a "control highlight clipping" advanced metering mode, which would be widely celebrated by almost all equipment reviewers and advanced users.

So assuming that I’m right, and on-going feature development is both feasible and desirable, where does that leave us?

Ming Thein is not alone in expressing disappointment with the provision of improved features focused for the advanced photographer, and I agree with him that the slow progress is really very annoying. In my most recent review, I identified several relatively simple features which would be of significant value to the advanced photographer, and which could easily be implemented in the software of any good mirrorless camera without hardware changes, including:

1. Expose to the right or other "automatically control highlight clipping" metering
2. Optimisation for RAW Capture (e.g. histogram from RAW, not JPG)
3. Proper RAW-based support for HDR, panoramas, focus stacking and other multishot techniques
4. Focal distance read-out and hyperfocal focus
5. Note taking and other content enrichment

All of these have been identified requirements/opportunities since the early era of digital photography. Many of them are successfully implemented in a few, perhaps more unusual models. For example the Phase One cameras implement a lot of the focus-related features, the Olympus OM-D E5-II does a form of image stacking for resolution enhancement, and Panasonic have just introduced a very clever implementation of focus bracketing in the GX8 based on a short 4K burst. However by and large the mainstream manufacturers have not made any significant progress towards them.  Even if Molitor’s analysis is correct, and this is all much more difficult than I expect (despite my strong software development experience) you would think that over time there would be at least some perhaps limited visible progress, but no. If the concepts were really "on the product backlog" (to use the iterative development term), then some would by now have "made the cut", but instead we get yet more features for registering babies’ faces…

My guess is that some combination of the following is going on:

• The "advanced photographer" market is relatively small, and quite saturated. Camera manufacturers are therefore trying to make their mid-range products attractive to users who would previously have bought a cheaper device, and who may well consider just using a phone as an option. To do this, the device needs to offer lots of "ease of use" features.
• Marketing and product management groups are focused on the output of "focus groups", which inevitably generate lowest-common denominator requirements which look a lot like current capabilities.
• Manufacturers are fixated on a particular set of use cases and can’t conceive that anyone would use their products in a different way.

The trouble is that this leaves the more experienced photographers very frustrated. The answer is flexibility. By all means offer an in-camera, JPG-only HDR for the novice user, but don’t fob me off with it – offer me flexible RAW-based multishot support as well. Re-assignable buttons are a good step in the right direction, but they are not where flexibility begins and ends. The challenge, of course, is to find a way to provide this within fixed product cycles and limited budgets.

I think the answer lies with software architecture, and in particular how we view the digital camera. It’s time for us all, manufacturers and advanced users alike, to stop thinking of the camera as a "product", and start thinking of it as a "platform", for more open development. In this model the manufacturer still sells the hardware, complete with basic functionality. Others extend the platform, with "add-ins" or "apps", which exploit the hardware by providing new ways to drive and exploit its capabilities.

We’ve been here before. In the early noughties, mobile phone hardware had evolved beyond all recognition (my first mobile phone was a Vodafone prototype which filled one seat and the boot of my Golf GTI, and needed a six-foot whip antenna!) However, you bought your phone from Nokia, for example, and it did what it did. If you didn’t like the contact management functionality, you were stuck with it.

Then Microsoft, followed more visibly by Apple and eventually Google, broke this model, by delivering a platform, a device which made phone calls, sure, but which also supported a development ecosystem so that some people could develop "apps", and others could install and use those which met their needs. Contact management functionality is now limited only by the imagination of the developer community. Despite my criticism of some early attempts, the model is now pretty much universal, and I don’t think I could go back to a model where my phone was a locked-down, single-purpose device.

The digital camera needs to go the same way, and quickly before it is over-run by the phone coming at the same challenge from the other side. Camera manufacturers need to stop thinking about "what other features should we develop for the next camera", and instead direct themselves to two questions, one familiar and one not. The familiar one is, of course, "how can we make the hardware even better"? The unfamiliar one is "how can we open up this platform so that developers can exploit it, and deliver all that stuff the advanced users keep going on about"?

Ironically, for many manufacturers many of the concepts are in place, just not joined up. The big manufacturers all offer open lens mounts, so that anyone can develop lenses for their bodies. In the case of Panasonic, Olympus and the other micro-four thirds partners it’s even an open multi-party standard. Panasonic certainly now deliver "platform" televisions with the concept of third party apps. There’s a healthy community of "hackers" developing modified firmware for Canon and Panasonic cameras, albeit at arms length from and with a slightly ambivalent relationship to the manufacturers. I’m sure many of those would very much prefer to be working as partners, within an open development model.

So what should such a "platform for extensibility" look like? Assuming we have a high-end mirrorless camera (something broadly equivalent to a Panasonic GX8) to work with as base platform, here are some ideas:

1. A software development kit, API and "app store" or similar for the development and delivery of in-camera "apps". For example, it should be possible to develop an ETTR metering module, which the user can choose as an optional metering mode (instead of standard matrix metering). This would be activated in place of the standard metering routine, take in current exposure, and return required exposure settings and perhaps some correction metadata. Obviously the manufacturer would have to make sure that any such module returned "safe" values, but in a mirrorless camera it should be very easy to check that the exposure settings are "reasonable" and revert to a default if not. Other add-ins could tap into events such as the completion of an exposure, or could activate functions such as setting focal distance. The API should either be development language-agnostic, or should support a well-known language such as Java, C++ or VB. That would also make it easier to develop an IDE (exploiting Visual Studio or Eclipse as a base), emulators and the like. There’s no reason why the camera needs an "open" operating system.
2. An SDK for phone apps. This might be an even easier starting point, albeit with limitations. Currently manufacturers such as Panasonic provide some extended functions (e.g. geotagging) via a companion app for the user’s phone, but these apps are "closed", and if they don’t do what you want, that’s an end of it. It would be very easy for these manufacturers to open up this API, by providing libraries which other developers can access. My note taking concept could easily be delivered this way. The beauty of this approach is that it has few or no security issues for the camera, and the application management infrastructure is delivered by Google, Apple and Microsoft.
3. An open way to share, extend and move metadata. Panasonic support some content enrichment, but in an absolutely nonsensical way, as those features only work for JPEG files. What Panasonic appear to be doing is writing to the JPEG EXIF data, but not even copying to the RAW files. The right solution is support for XMP companion files. These can then accompany the RAW file through the development process, being progressively enhanced by different tools, and relevant data will be permanently written to the output JPEG. This doesn’t have to be restricted to static, human-readable information. If, for example, the ETTR metering module can record the difference between its exposure and the one set by the default matrix method, then this can be used by the RAW processing to automatically "normalise" back to standard exposure during processing. XMP files have the great advantages that they are already an open standard, designed to be extensible and shared between multiple applications, and it’s pretty trivial to write code to manipulate them, so this route would be much better than opening up the proprietary EXIF metadata structures.
4. A controllable camera. What I mean by this is that the features of the camera which might be within the scope of the new "apps" must be set via buttons, menus and "continuous" controls (e.g. wheels with no specific set positions), so that they can be over-ridden or adjusted by software. They must not be set by fixed manual switches, which may or may not be set where the software requires. The Nikon DF or the Fuji XT1 may suit the working style of some photographers – that’s fine – but they are unsuited to the more flexible software environment I’m envisaging. While I prefer the ergonomics of "soft" controls, in this instance they are also a solution which promotes flexibility, which is what we’re seeking to achieve here.

This doesn’t have to be done in one fell swoop, and it might not be achieved (or even appropriate) 100% for every camera. That’s fine. Panasonic, for example, could make a great start by opening up the "Image App" library, which wouldn’t require any immediate changes to the cameras at all.

So how about it?

Microsoft SharePoint can’t do this. It has never been able to do this, and it still can’t. Microsoft have worked out that it’s a requirement, they just seem completely incapable of implementing a usable solution to achieve it, despite the fact that doing so would instantly bridge a significant gap between their online DM solution and their desktop products.

For the first 10 years, they had no solution at all. Then Office 2010 introduced "Microsoft SharePoint Workspace 2010". This promises, but under-delivers. It can cache all documents in a site into a hidden folder on your PC, and allows access to them through an application which looks a little bit like Windows Explorer, but isn’t. It’s very fiddly, and breaks all the rules about how you expect Office apps to work. It’s also slow and unreliable. Google it, and you find bloggers who usually praise Microsoft products to the skies using words like "excrable". Despite at least three office releases since 2010, Microsoft don’t appear to have made any attempt to fix it.

There’s now an alternative option, in the form of OneDrive for Business. This has a different balance of behaviours. On the upside, you can control where it syncs files so that they do appear in Explorer in a controlled fashion. On the downside, you can only link to a single SharePoint site (not much use if you have a client with multiple sites for different groups), and it still insists on synching all files in bulk, which is not what you want at all. On top of that I couldn’t get it to authenticate reliably, and was seeing a lot of failed synchronisations leaving my copy in an indeterminate state. There’s supposed to be a major rewrite in progress, bringing it more inline with the personal version of OneDrive, which works quite well, but no sign of anything useful yet…

Having wasted enough time on a Microsoft-only solution, I reverted to a solution which does work fairly well, using the excellent Syncback Pro. You have to log in using  Internet Explorer and the "keep me signed in" setting before it will work, but after that it delivers exactly what I want, allowing the selection of an exact subset of files, and the location of the copy on your PC, with intelligent two-way synchronisation. Perfect.

Perfect? Well, sort of. Syncback works very well, but even it can’t work around some fundamental limitations of SharePoint. The biggest problem is that when SharePoint ingests a file, at resets both the file modified date and the file created date to be the date and time of ingestion! When you export or check the file, it therefore appears to be a changed, later version than the one you uploaded. Proper professional DM systems just don’t do this, and the Syncback guys haven’t found a solution. Worse, I discovered that SharePoint process was marking some files as checked in, and therefore visible to other users, and some as still checked out to me, and therefore invisible to others.

The latter is a real problem, since the point of uploading the files is to share them with others. It’s also very fiddly to fix as SharePoint doesn’t seem to provide any list of files checked out, and there’s no mechanism to check files in in bulk – you have to click on each file individually and go through the manual check-in process.

Aha, I thought. Surely Microsoft’s excellent development tools will allow me to quickly knock up a little utility to search through a site, find the files checked out to me, and programmatically check them in. Unfortunately not. the first red flag was the fact that on a PC with full installations of Office and a couple of versions of Visual Studio, there’s no installed object model for SharePoint. After a lot of Googling I found a download called the "Office Developer Tools for VS 2013". I didn’t think I needed this, given what I already had installed, but ran the installer anyway. This took longer to complete than a full installation of Office or Visual Studio would, and in the process silently closed all my open office apps, losing some work. When it finished I still couldn’t see the SharePoint objects immediately, but adding a couple of references to my project manually finally worked. Right up to the point where I tried to test run the project, at which point the execution failed on the first line. It appears that these objects are designed to only support development but the code must execute on a server running SharePoint – there’s no concept of developing a desktop tool remotely interrogating a library.

OK, I thought. What about web services? I remember in the early days of SharePoint I was able to use SOAP web services to access and interrogate it, and I thought the same should still be true. To cut a long story short, that’s wrong. There’s no simple listing of the API, and attempting to interrogate services using Visual Studio’s usually excellent tools failed at the first post, with unresolveable authentication errors. In addition they seem to have moved to a REST API which is fundamentally much more difficult to drive if you don’t have a clear API listing. A lot of developers seem to be complaining about similar issues. I did find a couple of articles with sample code, but it all seems to be very complicated compared with what I remembered of the original SOAP API.

After wasting a couple of hours on "quickly knocking up a little utility" I gave up, at least for now. Back to the manual check-in method…

I’ve never been a fan of SharePoint, but it appears to be betting worse, not better. At least the first versions were simply cr@p. The new versions are very complicated cr@p.

While this correctly plots the pattern of adoption of a new technology, it hides a nasty truth, that the “plateau of productivity” is not a picture of nice, gentle, continuous, enduring improvement. Eventually all good things must come to an end. Now sometimes what happens is that an older technology is replaced outright by a newer one, and the old one continues in obsolescence for a while, and then withers away. We understand that pattern quite well as well. However, I think we are now beginning to experience another behaviour, especially in the software world.

Welcome to the Software Utility Curve:

We’re all familiar with the first couple of points on this curve. Someone has a great idea for a piece of software (the “outcrop of ideas”). V1 works, just about, and drums up interest, but it’s not unusual for there to be a number of obvious missing features, or for the number of initial bugs and incomplete implementations to almost outweigh the usefulness of the new concept. Hopefully suitably encouraged and funded, the developers get cracking moving up the “Escarpment of Error Removal”. At the same time the product grows new, major features. V2 is better, and V3 is traditionally stable, usefully and widely-acclaimed (the “Little peak of Usefulness”).

I give you, for example, Windows 3.1, or MS Office 4.0.

What happens next is interesting. It seems to be not uncommon that at this point the product is either acquired, or re-aligned by its parent company, or the developers realise that they’ve done a great job, but at the cost of some architectural dead-ends. Whatever the cause, this is the point of the “Great Architectural Rewrite Chasm”. The new version is maybe on a stronger foundation, maybe better integrated with other software, but in the process things have changed or broken. This can, of course, happen more than once…

MS Office 95? Certainly almost every alternative version of Windows (see my musings on the history and future of Microsoft Windows).

The problems can usually be fixed, and the next version is back to the stability and utility of the one at the previous “Little Peak of Usefulness”, maybe better.

Subsequent versions may further enhance the product, but there may be emerging evidence of diminishing returns. The challenge for the providers is that they have to change enough to make people pay for upgrades or subscriptions, rather than just soldiering on with an old version, but if the product is now a pretty much perfect fit to its niche there may be nowhere to go. Somewhere around Version 7 or 8, you get a product which is represents a high point: stable, powerful, popular. I call this the “Peak of Productivity”.

Windows 7. Office 2003. Acrobat 9.

Then the rot sets in, as the diminishing returns finally turn negative. The developers get increasingly desperate to find incremental improvements, and start thinking about change for its own sake. Pretty soon they come up with something which may have sounded great in a product strategy meeting, but which breaks compatibility, or the established user experience model, and we’re into negative territory. The problems may be so significant that the product is tipped into another chasm, not just a gentle downhill trundle.

Ladies and Gentlemen, I proudly present to you Microsoft Office 2007. With its ribbon interface which no-one likes, and incompatible file formats. We also proudly announce the Microsoft Chair of Studies into the  working of the list indentation feature…

I’m not sure where this story ends, but I feel increasing frustration with many of the core software products we all spend much of the day with. MS Office 2010+ is just not as easy to use as in the 2003 version. OK, youngsters who never used anything else may be comfortable with the ribbon, but I’m not convinced. I’m not sure I ever asked for the “improvements” we have received, but it annoys intensely that we still can’t easily set the indents in a list hierarchy, save the style, and it stays set. That  said, I have to credit Microsoft with a decent multi-platform solution in Office 365, so maybe there’s hope. Acrobat still doesn’t have the ability to cut/paste pages from one document to another, although you can do a (very, very fiddly) drag and drop to achieve the same thing… And this morning I watched an experienced IT architect struggling with settings in Windows 8, and eventually helped him solve the problem by going to Explorer and doing a right click, Manage, which fortunately still works like it did in Windows NT.

There’s an old engineering saying: “If it ain’t broke, don’t fix it”. Sadly the big software companies seem to be incapable of following that sound advice.

So far so good. However although the Microsoft Azure information may come in handy for my next role, what I really need is some good quality, easy to read guidance on how current generic guidance relates to the Oracle SOA/Fusion Suite. I identified four candidates, but none of them seem worth completing:

• Thomas Erl’s SOA Design Patterns. This is very expensive (more than £40 even in Kindle format), gets a lot of relatively poor reviews, and I didn’t much like the last book I read by the same author.
• Sergey Popov’s Applied SOA Patterns on the Oracle Platform. This is another expensive book, but at least you can read a decent-length Kindle sample. However doing so has somewhat put me off. There are pages upon pages upon pages of front-matter. Do I really want to read about reviewers thanking their mothers for having them before I get to the first real content? Fortunately even with that issue the sample gets as far as an introductory chapter, but this makes two things apparent. Firstly, the author has quite a wordy and academic style, but more importantly he has re-defined the well-established term "pattern" to mean either "design rule" or "Oracle example", neither of which works for me. However I really parted company when I got to a section which states "… security … is nothing more than pure money, as almost no one these days seeks fun in simple informational vandalism", and then went off into a discussion of development costs. If this "expert" has such a poor understanding of cyber-security it doesn’t bode well…
• Harish Gaur’s Oracle Fusion Middleware Patterns. Again, this appears to have redefined "pattern" as "Opportunity to show a good Oracle example", but that might be valid in my current position. Unfortunately I can’t tell you much more as the Kindle sample finished in the middle of "about the co-authors", before we get to any substantive content at all. As it’s another relatively expensive book with quite a few poor reviews I’m not sure whether it’s worth proceeding.
• Kathiravan Udayakumar’s Oracle SOA Patterns. Although only published in 2012, this appears to already be out of print. It has two reviews on Amazon, one at one-star (from someone who did try and read it) and one at three stars (from someone who didn’t!).

In the meantime I’ve started what looks like a much more promising book, David Chappell’s Enterprise Service Bus. This appears to be well-written, well-reviewed and reasonably priced. What really attracts me is that he’s attempted to extend the "Gregorgram" visual design language invented for Enterprise Integration Patterns to service bus architectures, which was in many ways the missing piece from the Service Design Patterns book. Unfortunately the book may be a bit out of date and Java-focused to give me an up-to-date technical briefing, but as it’s fairly short that’s not an issue.

After that it’s back to trying to find a decent book which links all this to the Oracle platform. If anyone would like to recommend one please let me know.

The book starts with a good introduction and list of the patterns and supporting "guidance" sections, and is then followed first by the patterns, and then the guidance sections (useful technology primers). This is where things break down a bit, as the patterns are presented in alphabetical order, which means a somewhat random mix of topics, followed by the same again for the guidance sections. I attempted to read the book cover to cover over about a week and I found the constant jumping about between topics extremely confusing, and the constant repetition of common content very wearing. In addition by presenting the guidance material at the end it is arguably of less value as most of the concepts have already been covered in related patterns. Ultimately the differentiation between the two is very arbitrary and not helpful. For example is "throttling" really a pattern or a core concept? If "throttling" is a pattern why is "autoscaling" not described as a pattern?

The book would be about 10 times better if it were re-organised into half a dozen "topics" (for example data management, compute resource management, integration, security…), with the relevant guidance and overviews first in each topic, followed by the related patterns which could then be stripped of a lot of repetitive content, and topped off with common cross-reference and further reading material.

This is not just a book about cloud specifics. A lot of the material reflects general good practice building and integrating large systems, even for on-premise deployment, and reinforces my view that "Cloud" is just a special case of this established body of practice. As a result there’s quite a lot of overlap with older pattern books especially Enterprise Integration Patterns, which is also directly referenced. The surprisingly substantial content related to message-based integration, confirms my view that this is still the best model for loosely coupled extended portfolios, but I would have appreciated more on the overlap with service technologies.

The overlap with other standard pattern books might have been managed just by referencing them, but this would play against Microsoft’s objective of making this material readily available to all readers at low cost.

The book is spectacularly good value for money, especially as you can download it free from Microsoft if you are prepared to do a bit of juggling with document formats. That it forms part of a series also available under similar options is even better. This perpetuates Microsoft’s tradition of providing cheap, high-quality guidance to developers and sits in sharp contrast with the high costs of comparable works from not only independent publishers (which may be understandable) but other technology vendors.

The book does assume some familiarity with Windows Server concepts, for example worker roles vs machine or application instances, and doesn’t always explain these terms. A glossary or an clear reference to a suitable external source would have been useful.

At a practical level I’m pleased to see that the Kindle version works well, with internal links hyperlinked and clear diagrams, plus access to each pattern directly from the menu in the Android Kindle app. Offset against this are a few cases of poor proofreading related to problems with document format conversions, in particular with characters like apostrophes turned into garbage character strings.

Overall I found this a useful book, and I’m sure it will become a valuable reference work, but I just wish the authors and editors had paid more attention to the high-level structure for those trying to read it like a traditional book.

Changing Font Size

The current behaviour of the font selection option is completely brain-dead, especially when switching between documents. Suppose I open one book which has been composed using a large base font. The text comes up very large and I set my font size to 2. I then open a second book, which has been composed using a smaller base font, and I have to change the font setting to 4 to get back to a size I’m comfortable with. Open the first document and the text is now enormous!

The application should actually work as follows. I would set a preferred font face and size and that would just be used automatically for all the bulk text in all documents. Anything styled with style tags like normal,  body text,  list,  should just use my selected font and size. Automatically. Paragraphs with heading styles would use progressively larger fonts, and the style might change to an author preference, although I should be able to over-ride that.

If that’s not possible, although I really don’t understand why not, then any change I make to my settings should apply only for a single document, and my settings for each document should be remembered if I switch between them. If I have to set size 2 in one document and size 4 in another to get a consistent reading experience the app should remember that.

Have the developers ever actually used the devices and apps with real eBooks?

Collections and Tagging

When,  early on, you have half a dozen books in your Kindle account, the lack of effective library management tools is not too much of an issue. When, like us, that library has grown to several hundred titles this starts to be a major problem.

Amazon allege that the solution is to use collections. That might help, if it weren’t for another brain-dead implementation. Collections on the physical Kindle are a local data structure, effectively invisible to other devices. In the Android app they are quite a usable feature, and sync with other Android devices, but not other platforms. On the PC you can create local collections, and allegedly import collections from physical Kindles (although I haven’t got that to work) but the collections are then completely independent of all other devices.

Is this really the best that can be achieved by one of the leading cloud services companies? Surely it’s not rocket science to come up with an architecture for collections / lists and tags, which is synchronised with the cloud account from and to all devices on the account? (And I note that there can’t possibly be any real technical issue, because notes and highlights synchronise perfectly across all my devices…)

Again, this looks like the developers are either stupid, or lazy, or completely indifferent to the implications of their substandard work.

Book Descriptions

If you are reading a book on the Kindle, you can quickly pop up some key descriptive details. Relatively recently Amazon have supported the same feature in the Android app, although it doesn’t work for books which aren’t open. On the PC it’s not supported at all.

There are three sets of books for which I would like to be able to quickly access descriptive details, ideally on- and off-line:

• Books I have downloaded to my device, but which I’m not currently reading
• Books in my archive, to remember which is which
• Books which are being recommended by Amazon within my mobile reading experience, e.g. the recommendations panel on the home page of the Kindle app.

No, I do NOT want to "view in store", especially if it’s a book I’ve already downloaded and I’m just not 100% which is which from the cover image, and I’m offline. And I don’t really want to have to open up a book to see it’s description. Surely it wouldn’t be rocket science (again) to download the key descriptive details for all the books in the above categories at every sync, and have those details available via a long press from the overview pages just like they would be from within an open book?

Position References

Some books insist on referring internally by using a page number from the printed edition. If you’re referring to a specific position in a book in the outside world, this is also still a common practice (and probably the only viable one unless the book has quite a fine-grained and well-numbered heading structure). Kindle insists on referring to and navigating locations using an internal "position" reference, which not only has zero relationship to the outside world, but can change from time to time depending on font choice and other settings. Therefore unless you have access to the physical edition as well as the eBook, you’re stuffed. It’s not even easy if you have a relative reference (e.g. page 200 of 300), because you have to get the calculator out to work out that this is equivalent to "position 3400 of 5393".

It would undoubtedly be better if authors creating Kindle versions of technical and reference books made sure all internal references were simply hyperlinks to the right point in the document. However I’m sure Amazon could help as well. How about, for example, holding the page count of the physical edition(s) against the Kindle version, and modifying the "Go To" dialog so that I can specify the target position as a percentage, or as a page number relative to the page count for the physical edition?

The Back Button

The physical Kindle and all Android devices have a "back" button, which should take you back steadily through your work contexts, like the back button on a browser. On the Kindle, or the PC app, this behaves as you’d expect. If you follow a link within a book, then it takes you to a new page, but the back button takes you back to the page you were previously reading. Only when you get back to your first context does it take you right out to the menu. Not on Android. Click on a link to an external source, and the back button takes you back into Kindle at the right point. So far so good. Click on an internal link, and the back button takes you right out of the book. To make matters worse it has now remembered the location you navigated to as your "current" location, so to get back to where you were previously you have to navigate manually. Completely useless, and presumably about 1 line of code to fix properly.

Conclusions

I don’t think I’m being unreasonable here. Amazon make a vast amount of money out of the Kindle platform, and could make more if it is a sound platform for reference books as well as novels and the like. None of these issues would take a vast amount of effort to fix, just the will to be bothered and do a professional job. Amazon’s persistent indifference on these points reveals an attitude which bugs me even more than the issues themselves.

The Service Design Patterns is in the same series as the EIP book (and the closely related Patterns of Enterprise Application Architecture), and overtly takes the earlier books as a baseline to build an additional set of patterns more directly related to Service-oriented integration. Where the earlier books’ content is relevant, it is just referred to. This helps to build a strong library of patterns, but also actively reinforces the important message that designers of newer integration architectures will do well to heed the lessons of previous generations.

The pattern structure is very similar to the one used in the EIP book, which is helpful. The "Headline" context description is occasionally a bit cryptic, but is usually followed by a very comprehensive section which describes the problem in sufficient detail, with an explanation of why and when alternative approaches may or may not work, and the role of other patterns in the solution. The text can be a little repetitive, especially as the authors try to deliver the specifics of each pattern explicitly for each of three key web service styles, but it’s well written and easily readable.

This is not a very graphical book. Each pattern usually has one or two explanatory diagrams, but they vary in style and usefulness. I was rather sad that the book didn’t try to extend the original EIP concept and try to show the more complex patterns as assemblies of icons representing the simpler ones. I think there may be value in exploring this in later work.

One complaint is the difficulty of navigating within the Kindle edition, or in future using it as a reference work. Internal references to patterns are identified by their page number in the physical book, which is of precisely zero use in the Kindle context. In addition the contents structure which is directly accessible via the Kindle menu only goes to chapter level, not to individual patterns. If you can remember which chapter a pattern is in you can get there via the contents section of index, but this is much more difficult than it should be. In other pattern books any internal references in the Kindle edition are hyperlinked, and I don’t understand why this has not been done here.

To add a further annoyance, the only summary listings of the patterns are presented as multiple small bitmapped graphics, so not easily searchable or extractable for external reference. An early hyperlinked text listing with a summary would be much more useful. Please could the publishers have a look at the Kindle versions of recent pattern books from Microsoft Press to see how this should be done?

A final moan is that the book is quite expensive! I want to get all three books in the series in Kindle format (as well as having the hardcover versions of the two earlier books, purchased before ebooks were a practical reality), and it will cost over £70. This may put less pecunious readers off, especially as there’s so much front matter that the Kindle sample ends before you get to the first real pattern. That would be a  shame, as the industry needs less experienced designers to read and absorb these messages.

These practical niggles aside, this is a very good book, and I can recommend it.

The authors and editors have clearly set themselves the admirable aim of producing a short and concise overview of the field. Unfortunately in the quest for brevity they have ended up with a terse, dry and dense writing style which is very difficult to read. At times it feels almost like a game of "buzzword bingo". I frequently had to re-read sentences several times to understand the authors’ intended relationships between the elements, and I’m a very experienced integration architect.

At the same time, for a book on architecture there are very few explanatory diagrams, wordy descriptions being used instead. To add insult to injury a few low-value diagrams such as one depicting the cycle of interaction between business and IT change drivers are used repeatedly, when once would be enough.

The first chapter provide a overview of service orientation and its key principles, characteristics, goals and organisational implications.  This is followed by a chapter on service definition and composition. Ironically this part of the book is is quite repetitive, but manages to omit some key concepts. There’s no real concrete explanation of what a service is or does – maybe that’s taken as read, but a formal definition and some examples would go a long way. Likewise there’s nothing at this point on basic concepts such as service contracts and self-description, synchronous vs asynchronous operation or security. The second chapter goes into some detail on the idea of service composition but only really deals with the ideal green-field case where functionality can be developed new aligned exactly to business functions.

The following chapter on the SOA manifesto is better, but again doesn’t recognise the realities of real enterprise portfolios, with legacy systems, package solutions and external elements which must be maintained and exploited, and non-functional priorities which must be met.

Chapter 5 deals with service-related technologies and their potential interactions. This is good, and for me represented the core value of the book, but is crying out for some diagrams to supplement the lengthy text. There are good notes on service definition under Model Driven Service Design, but this key topic should really have been a major section in Chapter 3 in its own right. The statements about technical architecture are rather simplistic, with an overall position of "this is expensive and difficult, or just use the cloud" which is not necessarily right for all organisations.

The next chapter, on business models, is very prescriptive. It is also slightly misleading in some places about the role of IT in transactional services – such services are delivered by a business unit, possibly but not necessarily enabled by and carried through an IT service. It would be perfectly viable in some cases for specific services to have a manual implementation. This is well explained in the case study, but not here or in the Business Process Management section of the previous chapter.

The final chapter of the main text is a "case study" describing the wholesale transformation of a car rental company through adoption of service, agile and cloud approaches. It feels slightly contrived, especially in terms of its timeline, the preponderance of successes, and the surprising lack of resistance to CIO-led business change. However it fills a useful gap by explaining much better than the technologies chapter how the different technologies and approaches fit together and build on one another.

Appendix A is a taster for the other books in the series. Unfortunately the content is presented as small images which cannot be resized and are almost unreadable in the Kindle version. It has also been "summarized", with the result that it appears to add very little meaningful detail to what has already been said.

Appendix B is a useful expansion of the main text regarding organisational preparation, maturity levels and governance for SOA. I would personally have been tempted to merge the first two parts to the main text rather than positioning them as an appendix, where they are necessarily repetitive of some material which has already been read.

Appendix C is another taster for one of the other books in the series, this time with an overview of cloud computing. While this is at a fairly high level, it’s a useful and well-written overview for those unfamiliar with the concepts.

Overall this is a frustrating book. There is some good material, but missing key "reality checks" and presented in a terse, text-heavy style which makes it harder to read than it should be.

Hopefully you should see everything working properly now, but let me know if not.

AgileArchitect.org has had the most significant makeover, and is now fully responsive and mobile-friendly, just like our other sites.

Happy browsing!

If you did have an email bounced, please feel free to re-send. And if you do meet someone who hacks other people’s servers, feel free to give him a kick for me.

I’m building a system in which I have to process fault data. Sometimes this comes with a standard fault code (hallelujah!), but quite often it comes with the manufacturer’s own fault code and a description which may (or may not) be quite close to the description against one of the standard faults. If I can match the description up, I can treat the fault as standard.

The problem is that the description matching is not exact. Variations in punctuation are common, but the wording can also change so that, for example, “Evaporative emission system incorrect purge flow” in one system is “Evaporative emission control system incorrect purge flow” in another. To a human reader this is fine, but eliminates simplistic exact matching.

I spent some time Googling fuzzy matching, but most of the available literature focuses on character or even bit-level matching and looks both complex and compute-intensive. However finally I found the Jaccard similarity coefficient. This is designed for establishing the “similarity” between two objects with similar lists of attributes, and I had a “lights on” moment and realised I could apply a similar algorithm, but to the set of words used in the pair of descriptions.

The algorithm to calculate the coefficient for a given pair is actually very simple:

1. Convert Text1 to a list of words/tokens, excluding spaces and punctuation. In VB.NET the string.split() function does this very neatly and you can specify exactly what counts as punctuation or white space. For simplicity it’s a good idea to convert both strings to uppercase to eliminate capitalisation variations.
2. Convert Text2 to a list of tokens on the same basis.
3. For each token from Text1, see if it appears in the list of tokens from Text2. If so, increment a counter M
4. For each token from Text2, see if it appears in the list of tokens from Text1. If so, increment M
5. Calculate the coefficient as M / (total number of tokens from both lists)

This produces a very intuitive result: 1 if the token sets are an exact match, 0 if they are completely disjoint, and a linearly varying value between. The process does, however, ignore transpositions, so that “Fuel rail pressure low” equates to “Fuel rail low pressure”. In my context this matches what a human assessor would do.

Now I simply have to repeat steps 2-5 above for each standard error description, and pick the one which produces the highest coefficient. If the value is below about 80% I treat the string as “matched”, and I can quote the coefficient to give a feel for “how good” the match is.

Hopefully that’s useful.

No, the law to which I refer is Moore’s Law, which states effectively, if you need reminding, that computing power doubles roughly every eighteen months.

Recently I’ve been doing some work to model a system in which two sub-systems collaborate by exchanging a very large number of relatively fine-grained web services. (I know, I wouldn’t have designed it that way…) The two partners disagree about how the system will scale, so it fell to me to do some modelling of the behaviour. I decided to back my analysis up with a practical simulation.

Working in my preferred environment (VB.Net) it didn’t take long to knock up a web service simulating the server, and a client which could load it up with either synchronous or asynchronous calls on various threading and bundling models. To make the simulation more realistic I decided that the service should wait, with the processing thread under load, for a given period before returning, to simulate the back-end processing which will occur in reality. The implementation should be simple: note the time when the service starts processing, set up the return structures and data required by my simulation, check the time, and then if necessary sit in a continuous loop until the desired total time has elapsed.

It didn’t work! I couldn’t get the system to recognise the time taken by the internal processing I had done, which threw out the logic for the loop. Effectively the system was telling me this was taking zero time. The problem turned out to be that I had assumed all processing times should be measured in ms. 5ms is our estimate of the average internal processing time. 6ms is our estimate of the round trip time for the web services. It seemed reasonable to allow a few ms for the processing in my simulation. Wrong!

It turns out that VB.Net now measures time in Ticks, which are units of 100ns, or one tenth of a microsecond. So I rewrote the timing logic to use this timing granularity, but still couldn’t quite believe the results. My internal processing was completing in approximately 1 Tick, or roughly 10,000 times faster than I expected.

Part of this is down to the fact that my simulation doesn’t require access to external resources, such as a database, which the real system does. But much of the difference is down to Moore’s Law. The last time I did something similar was around 10 years ago, and my current laptop must at least 100 times faster.

The moral of the story: beware your assumptions – they may need a refresh!

The latest centre of incompetence seems to be Google. Hot on the heels of my last moan on the same topic, I’ve just wasted some more effort because of a major Google c**k-up in Android 4.4.X, AKA KitKat. My new app, Stash-It!, includes a web browser based on the “Webkit” component widely used for that purpose across the Android, OSX and Linux worlds. On versions of Android up to 4.3, it works. However when I released it out into the wild I started getting complaints from users running KitKat that the browser had either frozen altogether, or was running unusably slowly.

It took a bit of effort to get a test platform running. In the end I went for a VM on my PC running the very useful Androidx86 distribution (as the Google SDK emulator is almost unusable even when it’s working), and after a bit of fiddling reproduced the problem. Sometimes web pages would load, sometimes they would just stop, with no code-level indication why.

After various fruitless attempts to fix the problem, I discovered (Google.com still has some uses) that this is a common problem. In their “wisdom” Google have replaced the browser component in KitKat with one which is a close relative of the Chrome browser, but seem to have done so without adequate testing or attention to compatibility. There are wide reports of deadlocks when applications attempt any logic during the process of loading a web page, with the application just sticking somewhere inside the web view code. That’s what was happening to me.

The fix eventually turned out to be relatively simple: Stash-It feeds back progress on the loading of a web page to the user. I have simply disabled this feedback when the app is running under KitKat, which is a slight reduction in functionality but a reasonable swap for getting the app working… However it’s cost a lot of time and aggro I could well have done without.

Can anyone arrange a plague of frogs and boils for Google, please?

Stash-It! responds to an odd side-effect of the difference between the iOS and Android security models. On the iPad, there are a large number of applications which offer an “all in one” approach to managing a group of related content. These are a bit frustrating if you want to share files transparently and seamlessly between applications, but there are times when you want to manage a group of files securely, and then the iOS approach is great.

Android is the original way around. The more open file system and component model encourages the use of specialist applications which do one job well, but it can be a challenge to keep related files of different types together, and hide them if you don’t want private client files or the like turning up un-announced in your gallery of family photos!

Stash-It! tries to plug this gap, by providing an “all in one” private file manager, tabbed browser and downloader for Android. You can get all these functions independently in other apps, but Stash-It! is the only one which brings them together in one place. It’s the ideal place to keep content you want safe from prying eyes: financial and banking records, health research, client documents. I suspect a few will even use it for a porn stash, but that’s not its only use!

There are built in viewers for most common image and movie formats, plus PDF and web files, so you don’t have to move these outside the application to view them. However when you do need to use an external application Stash-It! has a full suite of import and export functions to move your files or open them with other applications.

It took a while to design the security model. Stash-It! encrypts the names of files so that they can’t be read, and won’t be visible to the tablet’s gallery and similar applications, but the content of your files is untouched, so there’s little risk of losing data. Hopefully this strikes a sensible balance between privacy and risk.

Even if you’re not too worried about privacy Stash-It! is a great place to collect material related to as particular project, with all your different file types and web research in one place. You can bookmark web links, but also positions in video files or PDF documents. Web pages can be saved intact for reference or offline reading. Again you can do a lot of these things in separate apps, but I believe Stash-It! is the first one to bring all these functions together where you might want them.

I’ve got a lot of ideas in the pipeline to improve it further, but its now time to test the market and see whether I’ve spotted a gap which needed plugging or not.

Take a look and let me know what you think!

Here’s the Google Play Page. You can also read the helpfile.

As part of my preparation, I decided to do a literature scan to see how this topic may have moved on since the last time I did some significant work on it, a couple of years ago. I have to say that based on my initial research I’m not that impressed… I don’t know whether to be flattered or slightly perturbed that AgileArchitect.org comes up squarely at the top of a Google search. There are a few decent web articles around, although most are several years old and I’d seen them before. The Google search also turns up several dead links.

Amazon turned up a couple of loosely-related books, and the most obvious candidate appeared to be "Lean Architecture: for Agile Software Development" by James O. Coplien and Gertrud Bj�rnvig. I’ve now read a couple of chapters, but my first impression is not very favourable. I may be rushing to judgement, in which case I’ll apologise later, but the book seems to somehow equate "architecture" with "code structure" with "project structure", which isn’t right at all, missing a number of the most important dimensions of any true architecture.

This led me to ask myself a very basic question. "What do I mean by ‘Agile Architecture’?". In Coplien and Bj�rnvig’s book they seem to answer "an architecture which facilitates agile development". That may be one definition, but it isn’t mine.

I think the confusion arises from the difference between "agile" applied to a process (e.g. software development), and applied to a product. In the former case, the Agile Manifesto undoubtedly applies. In the latter, I’m not so sure. I think that for a product, and especially its architecture, the primary meaning of "agile" must be "able to respond to change". The larger the change which can be handled quickly and cheaply, the more agile the architecture. An architecture which has been built in a beautifully run agile project but which needs new code the first time a business rule changes is fragile, not agile. The system which can absorb major changes in the business rules without a single line of code is genuinely agile. The integration architecture which allows multi-million pound system A to be upgraded with no impact on adjacent multi-million pound system B, or which allows the company to be restructured just by re-configuring its services, is the most agile of all.

I’m slightly worried that "agile" may have become a "reserved word", and this "architecture in the large" definition may run counter to accepted practice. Is that right, or am I reading too much into a few examples?

I don’t want to say too much about the application itself until I have something ready to put on the market place. Suffice to say I think I’ve spotted an odd gap in the market where the weaknesses of iOS force a number of good solutions to one problem of information management, whereas Android’s more flexible architecture ironically mean the problem goes unsolved. Watch this space.

I was initially a bit worried that the learning curve for Android development might be very steep, especially when I started working through the standard Java-based examples in the official Google development toolkit. Like all Java development that approach seems to require a vast amount of “scaffolding” code, which must be constructed with very little environmental help, to achieve very simple results. This didn’t look good.

Then, thankfully, I discovered Basic4android. This is a remarkable toolkit developed by a small group in Israel which allows the development of Android software using a powerful but very accessible language and IDE based on Visual Basic. Behind the scenes, this is compiled into standard Android Java code, so ongoing delivery of applications is standard, but the coding and design process is close to “pure VB”.

The development environment has all the features you could reasonably ask for, including code completion, syntax highlighting, background compilation and the like. Remote debugging extends to devices connected over the Internet as well as via cable or local networks, and has a cunning feature where you can “hot swap” the code behind a running application allowing a range of changes to a running test application without restarting it. These are very impressive abilities for a product from a relatively small company.

Just as with the original VB, Basic4android has a model which allows developers to supplement the platform capabilities with shareable components, libraries and code snippets, and a very active community has rapidly built a library of “donationware” which provides easy access to the majority of Android features. I’ve had to be a bit ingenious in a few cases, but even as a newbie on my first project I haven’t yet found a requirement which can’t be met with a few lines of code.

On a slightly more negative note, Basic4android doesn’t seem to provide a good solution to the problem of supporting multiple screen sizes and orientations, except by writing multiple hard-coded scripts for the various options. This problem has been solved for websites with the concept of the “responsive grid”, and it ought to be possible to arrange the UI of an Android app with similar logic (e.g. “arrange these two controls side by side with the label taking 75% of the width, unless the screen is narrower than X, in which case arrange them vertically”). If this can be done in Basic4android I haven’t yet worked out how.

Debugging on a physical device connected directly to the PC is very straightforward, but of course limited to the devices you own, and a bit clumsy if you fancy doing a spot of work when travelling. While the Android development kit includes an emulator for the PC, it runs so slowly as to be completely unusable, even on a high-spec machine like my AlienWare M17x. I may have discovered a better compromise, in Android-x86, a port of Android which runs happily in a VMWare virtual machine. Installation was easy, but there are a few foibles I haven’t yet conquered. Again, watch this space.

Overall my adventure into Android development is shaping up well. More news later.

You should now be able to get the full experience of “Thoughts on the World” on all devices

Please bear with me if there’s the odd foible as I complete the changes. Also please let me know if you spot any problem – but don’t forget to let me know what device you’re using.

My blog now supports multiple views, with the default being an “abstract”, giving an experience a bit like Google Currents. Use the icons at the top of the page, just below my title bar, to switch between them. Please let me know what you think.

Although there’s no obvious reason why not,  Panasonic and Olympus have not published the specification for this data.  That leaves those of us who want to use a RAW processor other than LightRoom or SilkyPix struggling to get decent results with our MFT images.

Building on some excellent work done by “Matze”  (thinkfat.blogspot.co.uk/2009/02/dissecting-panasonic-rw2-files.html)and Raphael Rigo (syscall.eu/#pana) I decided to have a go at implementing a parser in my CAQuest plug-in for Bibble/AfterShotPro. However although getting the raw data is fairly straightforward I have discovered that the algorithm is more complex than we thought,  and seems to vary from lens to lens.

I have therefore decided to open up the exercise to a “crowd-sourcing” model to try and get several eyes on the problem. As we uncover algorithms which work well for one lens or another I’ll publish them here,  and also build them into CAQuest.  Over time we may come to completely understand the complete MFT algorithm,  and our work will then be done.  Of course,  if one of the MFT partners wants to help by publishing the algorithm,  that would also be perfectly acceptable :).

The project pages are here: www.andrewj.com/mft/mftproject.asp, with a discussion hosted at the Corel AfterShotPro forum.

Sometime around 2007-8, maybe by coincidence just when Bill Gates retired, this all went to hell in a handcart, and since then I’ve been cursing new Microsoft software versions as much as praising them. Each release has brought frustrations, and in many cases they have been sufficiently severe to drive me to adopt a competitor’s product, or at least a third party add-on.

XP SP 2 broke WMA format so it is incompatible with most third party players. My car was new in 2008, but I have to rip CDs using an XP SP1 virtual machine. Vista broke the reliable and flexible ntbackup. It took a bit of effort to get it working again, and it’s still part of my (more complex) backup strategy, but the “heavy lifting” is now done by Acronis rather than Windows.

The disruptive user interface and file format changes of Office 2007 have been widely discussed elsewhere. Suffice to say that I never used Office 2007, and run Office 2010 only with a third party add-on which restores the old menus. The compatibility-breaking changes to follow up flags in Outlook 2010 are extremely annoying, but as yet insufficient to drive me to an alternative product.

The same is not true of the changes to Virtual Machine support in Windows 7. Before that move, I used Mirosoft’s own Virtual PC extensively. However, the loss of compatibility, features and reliability were so severe that I now only use and recommend VMWare WorkStation/Player for this purpose. You can read about my experiences here.

The latest problem, and what has prompted this blog, is the appalling state of Internet Explorer 9. I have been a faithful user of IE since V1, and have lived, fairly happily, with its limitations through to IE8. However, since “upgrading” to IE9 I have become completely disillusioned, because it just isn’t reliable enough. Here are a sample of the things which just don’t work properly:

• Downloading dynamically-generated PDF files, such as bills from BT,
• MasterCard SecureCard authentication. This one’s a real pain if you’re at the end of a long online purchase, and you find your main credit card won’t work,
• The combined address / Google search bar. If I type in a valid www…. address, I expect the browser to at least attempt to use it , not do a search!
• Printing. Some long text pages, especially from typepad blogs, get mashed with the main font/character set replaced by something unreadable,
• Rendering some web sites readably at all. Some of the worst offenders, ironically, are Microsoft’s own “support” forums.

By direct contrast, Google Chrome seems to do a decent job of all the above. I am hereby announcing my intention to make it my primary browser whenever I have a choice.

I’m now really scared about Windows 8, with it’s so far half-hearted changes to the desktop. What will that wreck?

Now in fairness, Microsoft are not the only, or maybe even the worst offenders in this space. For example Bibble/Corel have just pushed through a change to their AfterShot Pro software which no-one wanted and which breaks a plugin I’ve written, and I suspect in that community I have some influence to say “the new version is broken, don’t use it.”

I really don’t understand Microsoft’s behaviour here. Are all these compatibility wrecks conscious decisions? If so, do the conquest sales related to cool new features really outweigh the loss of loyalty from existing users? If not, have they just got lazy and complacent? Who knows?

Like most myths this may have started from an original truth, but is now a lie. I am it’s latest, but I suspect far from only, victim.

Let me explain. For over a year now I have been developing a plugin for the RAW developer Bibble and it’s recent successor, Corel AfterShot. These plugins are developed using c++ and the Nokia QT framework, which theoretically allows the same code and user interface design to compile and run on Windows, Linux and Mac.

As a died in the wool Windows developer, that’s where I started. There’s a QT add-in to Visual Studio, so with a bit of juggling I managed to get one of the examples to load into VS, build, and run using Bibble as the target executable, and I was off. I was on a fairly steep learning curve in respect of the programming model, but I had very few problems compiling and running things.

When it got to the stage that I had something to share with the Bibble community I published the Windows version, and another member of the community kindly cross- compiled for the other platforms. There was another learning curve to make sure my code compiled cleanly on the other platforms, but nothing too drastic. For over a year I sent code updates to Jonathan, and got compiled Linux and Mac libraries back.

Although Jonathan still provides a very helpful service, it became apparent that if I wanted to have full control over the application versions I support, and be able to verify my plugin’s portability, I needed the ability to compile and run each version myself. I wasn’t prepared to buy and carry extra hardware around, but maybe VM technology would work.

I started with Linux. I had a couple of false starts but quickly found a site which has pre-built VMs for most Linux distributions (http://www.trendsigma.net/vmware/), and homed in on Lubuntu – based on Ubuntu but with a quite Windows-like shell. I downloaded and installed AfterShot and QT Creator, loaded up a copy of my code, and clicked “build”. And it worked first time! Getting a completely slick solution took a bit more effort, but it works so well I don’t now even copy the Windows code, I just open the same directory from my Linux VM and run the Linux builds in place.

So far so good. Now for the Mac. What could go wrong?

Apple’s antipathy to virtualisation is well known (see https://www.andrewj.com/blog/2012/mac-osxa-third-class-os/), and strictly speaking you’re not allowed to run OSX on non-Apple hardware. However, I got a legal copy of Snow Leopard, and found advice on the web on how to run up an OSX VM. You have to be patient, especially as it can take a couple of goes to boot the VM cleanly or install software, but generally it works pretty well.

First challenge: installing software. Unlike Windows or Linux, where non-trivial software has to be installed by a program which does useful things like error checking, on the Mac you often just simply drag a package of files to “Applications”. When it works, this is nice and easy. But if it fails, and I’ve now experienced this several times, it does so silently leaving you with a broken installation, which may not be obvious until things don’t work.

Eventually I manage to install Bibble, ASP, QT and, adding 30% to the size of the OS, XCode, the Mac development environment. I open QT Creator, and try to open my project. The first problem is that QT puts platform-specific settings in a small file alongside the main project files, and this has a fixed name, so you can store either the Linux or the Mac settings, but not both. Sigh, and copy the files from Windows to a local directory on the Mac.

After a bit of juggling, I got the plugin to compile. I then had to set up the deployment steps, renaming and copying files. Cue the next annoyance. Deployment configuration requires copying and adjusting complex filenames and paths. On Windows, you right-click any file or folder and there’s an option to copy the path to the clipboard. I was delighted to find that exactly the same trick works on Linux. OSX has no such simple solution. In the end I found a page of instructions on the web, which involved setting up macros as the like, but it wasn’t “easy”.

So I’ve compiled and deployed my plugin. Start AfterShot. Half the plugins and tools are missing, including mine. Re-install AfterShot, reboot the vMac, and now the standard tools are there, but still no sign of my plugin. The files are there, but the log has a very cryptic message about how my plugin “does not contain any recognisable symbols”.

Cue about 5 days of testing, trying different things, emails and posts on the Bibble forums. To cut a long story to a tolerable length…

1. No one seems to know how to compile successfully using QT Creator on the Mac, you have to convert to XCode,
2. XCode can’t cope with spaces in your folder or filenames. Another reason why the code can’t be shared directly with Windows and Linux,
3. When you’ve finished compiling, you have to do an extra step called “fixing the paths”. The QT/XCode combo manages to embed hard-coded pathnames in the executable files, which fail immediately you rename or move it. This doesn’t happen on the other platforms. Fortunately there are scripts available to perform the fix, but it’s certainly not obvious,
4. It’s all very sensitive to precise choices of SDK versions and the like,
5. Plugins for Bibble have to be compiled using a different QT version to those for ASP, and you can’t have more than one version installed on a Mac (ironically this restriction doesn’t apply on Windows or Linux, where there’s no problem just using the latest libraries for both). I had to use the “clone” feature of VMWare to create a “branched” VM with a different installation – on a physical machine I’d be sunk.

That’s not quite the end of it. One of the SDK features I use in AfterShot relies on an asynchronous processing model. On the other platforms and on a physical Mac my code runs fine, but on the “vMac” it returns invalid data. This is clearly a Bibble/Corel problem, but one for which no-one has yet offered an explanation or fix.

Now I accept that some of this may be down to my use of a VM, and some is definitely weaknesses in the Mac versions of the QT and Corel software. But setting those aside, the fact remains that the Linux exercise was genuinely quite easy, the Mac version far from it.

So there are lies, damn’ lies and statements about how “Macs are easy, they just work”. The next person to repeat that lie to me may suffer a painful injury to a soft part of their anatomy…

I’m a Windows user, spending much of the working week away from home. I get a vast amount of value from virtualisation. It allows me to carry just one PC with multiple “client specific” images, and enables me to keep running legacy software almost indefinitely. My main client uses the same technology to provide legacy support for essential software, which in long-cycle engineering businesses can easily be 20-30 years old, as physical assets in such businesses age many times more slowly than the computing equipment around them.

I also develop plugins for the Bibble RAW processor. The same code should work on Windows, Mac and Linux, but you have to compile and test on each platform to confirm this. I’ve recently added a Linux Virtual Machine to my kit. This was remarkably painless, just a few hours work, and I can now rapidly cross-compile and test my Windows-based developments under Linux. If there’s an issue which means having to support more than one flavour or version of Linux adding it would be trivial.

I just can’t do this for the Mac. I don’t want to buy and carry another laptop (which would be useless for any other purpose), and you can’t get virtualised OSX, either as a VM or as a service, through any legal and “safe” route. The result: as far as I am concerned OSX is a “third-class” OS, almost a “technical ghetto”, and I have to rely on the good offices of other developers to deliver my plugins for it.

People will put up with a lot in the name of love. Maybe Mac users “love” their computers enough to tolerate this behaviour. But looking in from outside I find Apple’s attitude perplexing and very annoying.

It is perfectly possible to make the iPad play nicely as part of a professional Windows-based environment, but you do have to be prepared to grab the bull by the horns, dump most of the built-in apps (which are almost all pretty useless), and take control of both file management and communications via partner applications on the PC. This article presents some of my hard-won tips and recommendations on how to do this and get productive work out of the iPad’s great hardware.

This morning’s keynote was excellent, it’s just a shame that I had to leave early to set up for my own talk. It could have been subtitled “why ‘cloud’ means people trying to sell you stuff”, and was the most balanced discussion I have yet heard on cloud computing. The most interesting observation is that individual component reliability is very much subservient to scalability and “elasticity”, which has major implications for more critical applications.

The rest of the day’s presentations were a mixed bunch. Some were too academic, others very light on real content. The one exception was Mike Rosen talking about SOA case studies, which included both real successes and failures, and should be the yardstick for anyone looking to move to SOA.

One thing I have learned from this conference is a (arguably the) real purpose for Twitter. It’s a great way for a group engaged in a joint activity like this to have a shared background conversation. In many ways it’s the electronic reincarnation of the DeMarco/Lister red and green voting card system, but with wider and longer reach. It’s not without problems: it can be a distraction, some users can dominate with high volume, low value tweets and retweets, and Twitter’s search and the available clients (certainly on the iPad) are not optimised for hashtag-based operation. However, these are minor complaints.

The iPad makes a superb conference tool, and I was amazed by the number of them in use, for making notes, reviewing slides, and tweeting. Interestingly I think this trend will drive a move to standardise on PDF-format material: slides published this way worked very well, but some available only in PowerPoint format weren’t viewable.

My congratulations and thanks to the conference chairs and the IRM team for an excellent event. Time to start thinking about a topic for the next one…

– Posted using BlogPress from my iPad

Location:Falcon Rd,Wandsworth,United Kingdom

Take, for example, South West Trains, who run commuter services to the South West of London. There are two main lines out from Waterloo via Guildford and Woking, but also a number of parallel minor lines, like the secondary line to Guildford which runs past my house.

When North Surrey got a foot of snow for the first time in 30 years in February 2009, it was clear that no trains were going to run on any of these lines for a couple of days, but only a relatively short stretch of the lines was blocked. It was still possible, for example, to get from Surbiton (about 10 miles nearer to London than my home) to Waterloo.

I had to attend a course in London, and the roads were becoming passable, so I dug the car out and drove to Surbiton. It rapidly became clear that everyone else had had the same idea. How had SWT reacted? By running the same four commuter services an hour from Surbiton. These were, of course, enormously overcrowded and slow. What about the other trains which would, for example, have usually been running the express services carrying the rest of the traffic? These were nowhere to be seen, presumably sat in a siding near Waterloo. Would it have been beyond the wit of man to press some of these into use as additional shuttle services to carry the excess traffic from those stations which were accessible? Apparently so.

Last night, I got caught again. I got to Waterloo at 10:30 pm to see a blank indicator board. The cause of the trouble was signalling problems in turn due to cable theft at Woking. Now I don’t blame the rail companies for that, and I hope the perpetrators are found, hung, drawn and transported to South Georgia, but I do think the train companies’ response is inadequate.

True to form, they had reverted to “what’s left of plan A”, running a tiny number of overcrowded and delayed services under manual signalling procedures. Now theoretically my line should not have been affected. Not only should I have been able to get home, but my line is perfectly capable of carrying some additional “relief” traffic, as it does when there is planned engineering work on the main lines. (About once a month the 8 commuter services per hour are joined by about 20 express and freight services, and when planned that seems to work fine.) With a bit of ingenuity you could even alert taxi drivers at the intermediate stops to the sudden need for their services, at profitable late night rates.

Is that what happened? I should coco. Instead not even the regular services to my home station appeared to be running. I ended up on one of the overcrowded trains to Surbiton, and finished my day with a £40 cab ride.

Why is this so difficult for the train companies to get right? In both of these cases there was no fundamental problem with the remaining infrastructure or rolling stock. In both cases they even have a model for the alternative schedule. For last night it’s in a file marked “Saturday service with engineering work at Woking”. Staff flexibility might be the problem, but that must be resolvable, maybe via higher overtime rates?

There’s also an architectural lesson here. I design computer systems and networks. My clients run national power networks. In both cases the customers expect those systems and networks to be resilient, and to cope with growing demand without wholesale replacement. It’s not always possible to justify dedicated “DR” capacity, so you have to get inventive with alternative configurations of the capacity you do have, and then run tests and introduce clever asset monitoring and management practices to make sure those configurations can be used safely.

If we can do it, why can’t the transport operators?

– Posted using BlogPress from my iPad

Location:Cobham,United Kingdom

I was umming and aahing about whether to attend yesterday’s seminar sessions, and couldn’t make up my mind which to join. In the end I made up my mind about the morning session while having a cup of coffee on the way, when I recognised one of the speakers, Lawrence Helm, as having given an excellent presentation a couple of years ago on NASA’s knowledge management problems. This time he and his colleague Robert Stauffer were talking about NASA’s adoption of Capability Modelling, and how they have put it to use supporting some very high level decisions about NASA’s future shape.

This was another stimulating session, and really benefitted from the extra space from making it a half-day session. Lawrence and Robert actually ran out of time, which was probably a testament to the depth of the material and the discussions it engendered.

The principle of relating capabilities to strategic objectives was not new to me, although the NASA examples certainly were. What did surprise me was the level of detail required for capability definitions in that environment. For example, the launch capabilities relate specifically to certain target longitudes and temperature ranges, and could not be moved to a location outside those ranges (for example Korou or Baikonur) without re-engineering the rocket platforms.

The afternoon session was also a bit random, as I got confused between Mike Rosen’s half-day seminar and his separate one hour talk for which I had the slides. Not a problem, the half day session on case study methods was very educational. The example, of how Wells Fargo created a federated model to integrate their various systems under a common customer model was interesting, and plays nicely into my EAI talk tomorrow. Like a good sermon, I didn’t learn much new, but I felt thoroughly validated that Wells Fargo did what I would have recommended, and succeeded with it. We had a very robust discussion on the importance of stable service interfaces, so hopefully that will drum up some support for my talk.

You get a very good class of attendee at these sessions. Alec Sharp joined the NASA session, and John Zachman joined the afternoon session, although he didn’t participate much.

Thursday’s highlights have probably been the two keynotes: this morning on how different companies have developed different strategies to come through and out of the recession, and this afternoon on “how to think like a CEO” and get your messages across to senior managers. However, there was also an excellent talk this morning by David Tollow on how EA feeds management and planning of long term outsourcing deals, from the supplier’s viewpoint. Very relevant to many of us in the current day and age.

Just to make things interesting, Sally has asked me to swap slots with someone else tomorrow, so my talk which was carefully trimmed to the constraints of the last slot on Friday will now be at 10 am. This may or may not be a good thing.

Wish me luck!

– Posted using BlogPress from my iPad

Location:Portman Towers,Paddington,United Kingdom

I’ve just uploaded the slides to my website. You can download them here.

Unfortunately, I’ve been put into a real “graveyard slot”, right at the end of business on the Friday afternoon. And I thought Sally, the conference chair, was my friend! If you are attending, I would very much appreciate your support. If not, I’d appreciate your thoughts on this topic, as always.

Update 13th May: the restructuring is complete, and you should see new reviews more fully integrated than previously. The process was non-trivial, and may spawn a blog post at some stage.
Thanks
Andrew

Oddly, although the wonderful “Waterfall 2006” web site still exists, I couldn’t find DDD on it. So I checked with Google and found a couple of references to non-ironic (as far as I can tell) papers on the subject. Yes, some people seem to think that document-driven development is a good idea! Now I might be prepared to concede this for applications where documents are themselves the key business objects (some legal processes, for example), but as far as I can see this isn’t what those papers were referring to. If that’s the case, they really haven’t understood…

What I did find, however, was a wonderful blog post from a few years ago with the excellent title “Asshole Driven Development“, in which Scott Berkun has collected a wide variety of development and project management anti-patterns. It takes a while to read through all the comments, but doing so is quite rewarding, if mainly as a form of therapy. At least you know you’re not alone.

The list is pretty comprehensive, but despite over 300 contributions, I couldn’t see my own b�te noir. A lot of large corporate organisations now seem to follow a governance methodology I term IAKOM (the “It’s A Knock Out Method”), known on the continent as la Methode Jeux Sans Frontieres (MJSF). Those of a certain age will remember a series of hilarious television games in which relatively simple tasks (such as carrying a bucket of water) were rendered impossible by the imposition of progressive handicaps and obstacles (such as carrying the bucket up a greased slope against a rubber bungy while wearing clown shoes and being pelted with wet sponges).

Some IT governance is like that. Just when you think you might have a fair run at doing something, a new governance hurdle or document check is inserted into the process. It wouldn’t be so bad if it all made sense, but sometimes it feels almost capricious. Some organisations are more enlightened than others, but as a general industry trend it’s inescapable.

I don’t know what the answer is. If you do, let me know!

The trouble is that for every St. Paul’s there must be a Bletchley Park – an architectural mish-mash which while possibly important, successful or even revered is inherently inelegant, or even downright ugly.
My analysis is that behind the glossy facade, the iPad software architecture has to be the best current example of “Ugly Architecture”.

In many ways it’s strongly reminiscent of PCs in the days of DOS, or maybe Windows 3.0, before the emergence of strong component-based architectures and unifying design standards in Windows 95 and NT.
The fundamental problem is the application-centric model, in which each application is a stand-alone combination of code and data, with very few shared services or components. This naturally leads each application developer to “do their own thing”, implementing separate, widely varying solutions for communications, document storage, printing support and so on. Apart from a token “open in another app…” supported by some applications, there’s effectively no cross-application linking, leading to massive duplication of functionality and data, and some significant functional limitations, for example the inability to directly open a URL embedded in a document.

Each application has its own data area, which may or may not interact with iTunes, web sites or a PC via FTP, websites via WebDAV or various different cloud storage services. Data which should arguably be general visible just isn’t – you can upload video files to the photos area, but they won’t be visible in the videos list. To test a variety of editors with a document you need to deliver a different copy of the document to each app.

Each application supports different models for document exchange, and different cloud stores, so a user potentially has to have multiple separate cloud accounts. While “public” cloud storage may be fine for individuals’ personal data (although individuals may still have valid security and privacy concerns), it is a real concern if used for corporate information. In corporate contexts, connectivity, security, copyright, access rights, service levels, data protection and privacy obligations, regulatory and legal constraints may all be compromised or complicated by cloud use, and become significant issues.

There’s also an interesting security implication to this which you don’t often see discussed. Because there’s no accessible file system, and no extensibility model for the application filing model, there’s nowhere for anti-virus solutions to run, and as of today iPhones and iPads are effectively unprotected devices. There are probably numerous iPads in the wild acting as festering reservoirs of infected documents. Those who are security conscious can’t be happy about this, and I know that many corporate security departments are making moves to ban connectivity to corporate services for that reason.

Even if an application interacts with the host PC more directly, you get multiple copies of documents, typically the original, a copy in iTunes and one on the device, with no mechanism to synchronise them or compare version information. Apple’s own applications such as Pages are even worse, with a completely separate iTunes space from their own “My Documents” spaces, and an additional copy step in each direction. This is a version control and management nightmare!

Why could the iPad not support a simple shared filing area with proper two-way synchronisation to the host PC, as the Pocket PC has had from day 1?

The communications architecture is a similar mess. The only application which can communicate with the host PC over USB is iTunes, but iTunes can’t use WiFi. All other apps have to use WiFi, but there’s no real shared comms application infrastructure, so the result is another diverse and fragmented “roll your own” free for all. The most obvious way for a companion device to talk to its host PC, BlueTooth, isn’t supported at all!

The WiFi only design works fine in the confines of, say, a small home office. Elsewhere it’s problematic at best. Paid WiFi (e.g. in a hotel) is typically limited to a single device, so you’ll end up paying twice if you want to connect both devices. Corporate WiFi systems are typically similar, and you may not be allowed to connect the iPad directly. Even if you do get connectivity, these networks are often set up to prevent routing between devices, as a security measure, so that’s that, then.

The alternative is to set up either the PC or iPad as a hot-spot itself. On the iPad, this is only possible on jailbroken devices. On the PC, it can be complicated and opens up potential security issues. Neither is ideal.

Apple’s policies effectively put software development back in the Stone Age, in the particular sense that “monolithic” means “single lump of rock”. Each application has to be “stand alone”, implementing many things which should arguably be shared. For example, each file management application implements its own storage management dialogs, its own comms model, its own browser, its own PDF and Word file viewers, each with their own subset of functionality, dialogs and gesture support, and so forth. There simply doesn’t seem to be any real concept of shared components or companion applications. Let’s be clear: I’m not criticising the application developers for trying their best to provide a comprehensive solution – my criticism is directed squarely at the crass architecture through which Apple force such an approach.

Even those applications which implement the “open in another app…” capability to open documents in other viewers suffer two common problems: you frequently have to open the document natively before you can send it elsewhere, and the act of doing so usually creates yet another copy of the document to manage separately!

Ironically, where there are shared components they impose significant constraints and limitations. The most obvious is the keyboard. Essentially there’s only one way to get text directly into any application, and that’s to use the on-screen keyboard configured exactly as the application developer decides. It’s “my way or the highway”. This is a dramatic contrast with the Microsoft world, where even a humble 2003-era Pocket PC supports not only a variety of built-in and third-party on-screen keyboards, but also handwriting recognition, character recognition (like the Palm Pilot), Swype, and even limited voice recognition. Importantly, these are all user-selectable at any time text input is required. On the iPad you can buy an app with a different keyboard layout, or dictation capability, but you have to cut and paste the raw text into your target application and typically reformat it to suit. This is simply primitive.

What makes all this worse is that the iPad application approval/delivery model makes it unlikely that anyone will innovate a better solution. No approved application can have legal access to another app’s or central iTunes data. Without approval, you won’t appear in the App Store or run on non-jailbroken devices, so Apple simply impose their will, whether good or bad.

OK. I am starting to love my iPad, but the software architect within me is incredibly frustrated. This great hardware is hamstrung by a clumsy, unimaginative, software architecture and oppressive centralist control by those who worship according to The Book of Jobs. It could be so much better.

</rant>

Interestingly, a very similar submission last year didn’t make the cut. Whether the change is due to an increase in the quality of my submission, or a decrease in that of the competition, only time will tell…

To find out more, visit www.andrewj.com/plugins.

In the words of the Tao of Lao-Tsu, “If you want to control something, you must first let it go free”. Big organisations just don’t get this, because too many people want to hold onto their favourite control mechanism. Agility is a culture, and trying to do it inside a culture of formal control is very difficult. Occasionally you can succeed in spite of the controllers, and that’s a real achievement.

Now I’m not sure whether his guidance on decision making process is generic, or whether that process depends on your organisational position and role in determining “the right thing”.  I don’t know whether the key difference in our perspectives is between leadership as a manager versus leadership as an influencer, or the difference between managerial and technical leadership, or the difference between synthesising solutions and deciding which to adopt, but Mike’s article portrays a very different perspective to mine.

Mike portrays as key the ability to focus on key issues, and exclude those which are “not relevant”.

In my experience as an architect and technical leader, I spend a lot of time understanding and analysing the different forces on a problem. These design forces may be technical, or human: financial, commercial or political. The challenge is to find a solution which best balances all the design forces, which if possible satisfies the requirements of all stakeholders. It is usually wrong and ultimately counter-productive to simply ignore some of the stakeholders or requirements as “less important” – any stakeholder (and by stakeholders I mean all those involved, not just senior managers) can derail a project if not happy.

Where design forces are either aligned or orthogonal, there is usually a “sweet spot” which strikes an acceptable balance. The problem effectively becomes one of performing a multi-dimensional linear analysis, and then articulating the solution.

However, sometimes the forces act in direct opposition. A good example, currently personally relevant, is system security, where requirements for broad, easy access directly conflict with those for high security. In these cases the architect has to invest heavily in his skills in diplomacy – to invest a lot of time understanding stakeholder positions. One common problem is “requirements” expressed as solutions, which usually hide an underlying concern which can be met many ways, once understood.

In cases of diametrically opposed requirements, there are usually three options:

1. Compromise – find an intermediate position acceptable to both. This may work, but it may be unacceptable to both, or it may fatally compromise the architecture.
2. Allow one requirement to dominate. This has to be a senior level business decision. As an architect, you then have to be sensitive to whether the outcome is genuinely accepted and viable, or whether suppressing the other requirements will cause the solution to fail.
3. Reformulate the problem to remove or reduce the conflict. In the security example the architect may come up with a cunning partitioning of the system which allows access to different elements under different security rules.

Of course, you can’t resolve all the problems at once – that way lies madness. An architect uses techniques like layered or modular structures, and multiple views of the architecture to “separate concerns”. These are powerful tools to manage the problem’s complexity.

It’s also important to remember that the architecture, and its resolution of the various design forces (i.e. how it meets various stakeholder needs) have to be communicated to many who are not technical experts. The technical leader must take much of this responsibility. I have had great success with single-topic briefing papers, which describe aspects like security in business terms, and which are short and focused enough to encourage the readers to also consider their concerns separately.

One area where I do agree with Mike is the need to listen to the voice inside, and carry decisions through with integrity. For an architect, the question is whether the architecture is elegant, and will deliver an adequately efficient, reliable and flexible solution. If your internal answer to this is not an honest “yes”, you need to understand why not, and decide whether you and your users can live with the compromises.

And finally, the architect must protect the integrity of the solution against the slings and arrows of outrageous projects. Monitor in particular those design aspects which reflect compromises between design forces, because they will inevitably come under renewed pressure over time. You have to not only do the right thing, but ensure it is done right.

Non-Sequiteur

About the weird title: Mike is attempting to create his blog based largely on 1970s Prog Rock references. As a tribute to such an excellent idea, I feel compelled to join in (at least on this occasion)!

I recently converted my blog from an old bespoke (= “custom”, for my American friends) solution to one based on WordPress. However, this created a problem, in that the WordPress model is to hold all content in the database, and that wasn’t the right model for me.

Firstly, I have a number of articles which are very long for a blog post, and I had no interest in restructuring them. I also didn’t want to break external links to the existing articles.

Next, I decided that I wanted the freedom to continue to write in that style. Some of my writing takes several weeks, and it works for me to draft it as separate HTML pages. I also sometimes want to include active content or multiple images, and I don’t want to create a large and unwieldy WordPress database full of such stuff.

Finally, my online photo galleries are managed and generated using Jalbum, and I wanted to find a way of neatly integrating single images into my blog, complete with the watermarks and metadata extraction which Jalbum manages so well, without duplicating that functionality in WordPress.

This is probably typical of many older web sites, but WordPress doesn’t really embrace the integration of external content. This article describes how I solved this problem, and a WordPress plugin I have developed to make my solution reusable.

I have therefore now created a number of new “views” of my blog, and a number of specialist feeds, as follows:

• Thoughts on the World. This is the original and complete blog and feed, and will give you access to all my content, in chronological order. Expect a mix of articles including my observations of the world of software development and architecture, humorous observations, photography and personal announcements.
• The blog address is www.andrewj.com/blog
• The feed is http://feeds.feedburner.com/ThoughtsFromAndrewJohnston
• The Professional Blog. This will focus on professional articles based around my observations of the world of software development and architecture. Articles for AgileArchitect.org will appear here, as will updates on my products and publications, and I may cross-post relevant reviews. You’ll also see any practical announcements relating to my websites and feeds.
• The blog address is www.andrewj.com/blog/pro
• The feed is http://feeds.feedburner.com/StructureOfTheWorld
• The Photo Blog. This will focus on my photos, and photography-related articles and reviews. You’ll also see any practical announcements relating to my websites and feeds. There’s also a separate feed for updates to my gallery.
• The blog address is www.andrewj.com/blog/photo
• The feed is http://feeds.feedburner.com/ImagesOfTheWorld
• The feed for my gallery is http://feeds.feedburner.com/GalleryOfTheWorld
• The Review Pages. This presents my reviews of books, films and music. You’ll also see any practical announcements relating to my websites and feeds.
• The blog address is www.andrewj.com/blog/reviews
• The feed is http://feeds.feedburner.com/ReviewsOfTheWorld

For more details, visit my Blog Views and Feeds page.

My new blog is now live. This allows a lot more flexibility, including commenting etc.

Please can you make sure you are taking your RSS feed from one of the following:

http://feeds.feedburner.com/ThoughtsFromAndrewJohnston (preferred)

https://www.andrewj.com/blog/index.php/feed/ (straight from my site)

I still need to update some of the links to longer old articles, so please bear with me if these are a bit odd over the next few days. Also you may see some duplicate items in the Feedburner feed – if so, please ignore these, and they should clear shortly.

Thanks, and enjoy the new blog.

Andrew

Migration of our web sites and email to the new hosting server is almost complete. See previous article/rant for some of my findings. After some initial frustration with my hosting provider (WebFusion) when I discovered the
limitations of their new Linux shared hosting services compared with the old one, I have to say a big “Thank You” for their efficiency in finding a better solution for me, and providing me with effective technical support to get it up and running.

If you do have any problems with our sites or email, let me know…

I’m currently developing a new blog, based on WordPress. This will make it easier to post “on the fly” than with the current solution, hand-carved from XML and ASP (now PHP).

When the new blog is running, I’m going to have a regular post for fans of my photography, so you can see what I’ve been working on. In the meantime, I’ve updated my gallery pages so they are a bit easier to navigate, bookmark and search.

And talking about my photography, I’ve recently been accepted by the Alamy stock agency. So please all rush at once and spend lots of money licensing my pictures for all those uses you’ve dreamed of but were too polite to mention

See you soon,

Andrew

I’ve just spent a week-end migrating my website to a new hosting server. As part of that process, I had to rewrite all my old ASP code using PHP. Here’s what I learned:

1. The Apache/Linux community have misleadingly changed the meaning of “ASP”. If you bought a Linux-based hosting service 5+ years ago with “ASP”, it meant a *nix port of Active Server Pages. That worked for me, as I could develop it on Windows. Now, if you buy a Linux hosting service with “ASP” it means “Apache Server Pages”, and the embedded language is Perl. Useless!
2. PHP has positively the worst combination of features for a language:
   • A c-based language’s sensitivity to case, ending semicolons and curly bracket counts,
   • None of the protections against errors in the latter that a C++/Java (or VB) language gives you, like strong typing and forced variable declaration,
   • No single-step debugging. Now I accept that this may not be 100% true, so don’t all write in with the names of all the debuggers I didn’t find in a quick search for tools on Sunday morning, but certainly I don’t have one at the moment,
   • It runs differently on Windows and Linux, and in a way I haven’t yet understood 100%, so I can only test by uploading to my live website.

That said, I’ve still got it! I’ve managed to convert my blog and my book reviews, and I’ve actually improved on my old code for the latter. Just please let me have VB.NET back for my next major project.

OK. </rant>

Although mainly about photography, I think there are lessons for anyone contemplating a disruptive, rewrite from scratch change to a software product – beware!

My photography zone has been updated with images from my trip to Maine, Vermont and New York last autumn, and from the dramatic Winter weather in the UK. Photos from my trip to the Venice carnivale should start to follow shortly.

Overall there was much less focus on technology (read SOA and modelling tools) this year, and a vigorous and renewed focus on business alignment and business architecture, which, if we can deliver, potentially places architecture where it should be, as the business’s agent.

But there’s a problem. Good business analysis is fundamental to this, yet several delegates bemoaned the current lack of good business analysts. User organisations often struggle to articulate and abstract their needs, and this feeds into all downstream processes. Modelled requirements are an increasing rarity, poorly substituted by imprecise verbal statements in Word or PowerPoint.

The problem is, of course, not unique to analysts, and may have common cause with the equal lack of architects. Senior architects and analysts both tend to have several big birthdays under the belt, and many learned their trade as developers, gaining both practical method skills and the experience of turning ideas into working code. (The majority of exceptions have other “making it work” experience, such as building networks or running data centres.)

But in the current world of ERP packages and large-scale outsourcing, many organisations no longer build anything themselves. The live classroom has been thrown away.

I have worked with a number of good, keen young analysts, but most work for large supplier companies who still have both well-funded training programmes and the breadth of work to build experience and a broad skill set. These guys and girls can do a good job, but at the risk of higher costs and potential conflicts of interest.

We already know that this may reduce organisations’ ability to ensure the right solution to their needs, or assure its quality. Recent observations suggest that organisations who forgoe getting their hands dirty in IT will also suffer an increasing difficulty in creating a clear, concise and structured statement of those needs themselves.

I’ve also always had great admiration for those photographers whose work can really move the emotions, particularly the great landscape photographers like Ansel Adams. Taking good representational photographs is hard enough, but establishing a composition which can both emotionally inspire the viewer and satisfy the critics is even harder.

Over the last year or so I’ve been rediscovering photography with a Digital SLR, and on my new photography pages you’ll find some of my efforts, plus (in coming weeks) some hints, tips and tools for the D-SLR photographer.

I explore two very different examples: the value of understanding RS232 in this 3G+ world, and some recent discussions on service reliability, both of which can be resolved using some quite old knowledge…. (Read More…)

I presented a paper on Agile Architecture. If you regularly read my work you’ll recognise many of the ideas, but I’ve managed to bring them all together for the first time. You can download my slides and script here.

What was very interesting was how the thrust of the material has changed from a few years ago. No-one was claiming that a given framework, process or toolset can solve EA problems. At the risk of being uncharitable I thought John Zachman’s ideas sounded very tired, and there was almost no mention of alternative frameworks such as TOGAF. I may have self-selected by not attending any vendor sessions, but there was also no promotion of tools or technology. A common view was that EA, SOA and many supporting concepts are currently entering the trough of the hype cycle.

Instead the focus was largely on people-related problems and approaches. The labels varied, but several speakers introduced ideas familiar to agile architects. Maybe we’re doing something right after all.

Why you need architects, in song and dance.

Enjoy!

As a first step, I am trying to develop a comprehensive list of test automation “best practices”, grouped roughly by life-cycle (or iteration) stage. I’m trying to find practices which are broadly independent of specific methods and technologies, although obviously tool support may vary depending on the chosen technology.

This article is my first draft of such a list.

I’d welcome suggestions from my readers if you think there are any omissions (or if you substantially disagree with anything I’ve included).

Thanks

Andrew

I’m presenting a paper entitled “The Agile Architect”. This focuses on both how agile projects can have a strong architecture, and how architects can learn and benefit from agile approaches. I take a rather different approach to some recent papers with a similar title (e.g. at this week’s otherwise excellent Microsoft Architecture Insight conference), which suggest that agile projects can “do away with the architect”.

I look forwards to seeing you there.

Andrew

But I’m not sure that’s the key question. I think the right question is “Who are the architects?”

Somehow, a number of tasks must be discharged, but how varies from project to project. In the last year I’ve had a modest building project which tells an interesting story about how different people contribute to “the architecture”. Read more here…

This is a story of a project I worked upon which followed an informal, agile process, but delivered a successful architecture. Hopefully it serves to support my assertion that agile can have an architecture, but needs an agile architect.

However, UML doesn’t help when it comes to the transformations themselves, and typically the detail has to be captured either in code, or a proprietary format. Most good integration tools provide some sort of “visual mapping tool”, where the developer drags and drops to create links between representations of the two structures, usually imported directly from their physical schemas. Here’s an example using SeeBeyond. Altova provide a good stand-alone data mapping tool called Mapforce – here’s an example showing it in use. The problem is that these tools work directly with the physical structure, and don’t export the mapping information in a reusable format, so that information is completely disconnected from the UML analysis or design models.

I have experimented with trying to represent mapping information in a UML model, but so far without much success. The best solution I’ve found so far is to use some sort of “pseudo code” (it could be OCL, pseudo-Java, pseudo-VB or anything similar). For example, we could easily annotate the model with code fragments such as:

Database.order_table.order_no = Message.Header.OrderNo

(where each element refers to a UML Package.Class.Attribute combination).

The problems are that it’s not clear where to put this annotation, most UML modelling tools won’t help generate it, and there’s no graphical representation. Ultimately, writing pseudo-code like this is probably not much better than abandoning the model and moving straight to using your integration tool.

My question is: does anybody know a better way? Has anybody found a good way of representing mapping information in UML? And if so, is there any good tool support?

If you know, please send me a message.

While the metaphor might still be valid, some people are beginning to question how far it should be taken. Helland’s article, like others before it, implies that “good” EA looks rather like a medium-sized modern American town, complete with relatively standard services, civic buildings and commercial venues. In an answer to the original “Metropolis” article Richard Veryard and Philip Boxer have published “Metropolis and SOA Governance” which challenges several of Helland’s assumptions.

I think that maybe we should extend the metaphor by thinking about cities, or Enterprise Architectures, as very diverse entities. What sort of “city” do you live in? To what extent is it planned? What is the vision, and do the citizens share in it? Does the EA resemble a nice neat midwest town, a dark, brooding Gotham City, a glass and steel Utopia, a federation of small towns with lots of empty space between them, a medieaval walled town, or a wartime mid-european ghetto?

And the metaphor can be taken further. Do you want to promote “infill development”, closing up functional gaps, or do you want to keep clear separation between the various zones? Do you want the shared services to be clearly visible, as they are in modern, purpose-built towns or hidden beneath a facade which looks much older or simpler? Do you expect to eventually knock down and rebuild older “legacy” zones, or do you want to preserve them with the minimum of change (a common requirement for our valuable historic buildings)? Do you want to accomodate the small hardware shop (read small the bespoke system) as well as the new DIY superstore (the ERP package)?

Finally, remember that it is extremely rare for a city to be truly planned and designed from scratch. You usually start with something established. Even if the city has been flattened by a bomb, you’ll have to observe land rights (this is what stopped Christopher Wren and Charles II realising their grand design after the Fire of London). This is equally true of Enterprise Architectures.

The city planning metaphor is a powerful one, but its true power may come if we use it to explore problems as well as utopian ideals.

Three short quotes from Helland’s conclusions bear repeating directly. On heterogeneity he says:

Remember that heterogeneity happens. Unless you have a very simple application portfolio, shared services will not be achieved by trying to put all
of your applications on one version of one platform. Even if you could, the next
merger would change that! Rather, you have to design for interoperability and
integration across platforms. This is the force that is driving the industry
wide work in service-oriented architectures.

He extends the popular “city planning” metaphor to IT investment:

IT investment is a balance of funding the sacred, protecting historic monuments, and allocating spending between infrastructure and business opportunity. Striking this balance is a key facet in effective governance, and in realizing the potential of IT in your organization.

And finally, those who seek to maintain control of their enterprise
architecture through heavy governance would be well advised to note:

You have to maintain a light hand. It is counterproductive to try to dictate
what happens in every structure in town, what color shirts are made, and how much is charged for soap. You have to embrace the semi-autonomous approach to governance that is characteristic of our cities, and allow the process owners to optimize and achieve efficiencies with as few constraints as
possible.

However, Martin’s essay goes beyond common programming and data techniques to look at the development of specialist tools which he calls “Language Workbenches”. I’m not completely convinced that we need these in the world of XML and XSD. If you have a defined schema for you XML-based DSL (and aren’t all the many *ML langauges just different DLSs?) then any schema-sensitive editor will provide you with good design and editing support. The leading IDEs (e.g. Visual Studio) all have such a tool built into their core capabilities. Surely we now have a sufficiently sophisticated set of XML-based tools and standards that we have an opportunity to exploit synergies rather than re-inventing the wheel?