iWayneo

The Architechture behind highstreet.ly

2021-05-21T04:15:00.009-07:00

I recently read a blog post (https://anthonynsimon.com/blog/one-man-saas-architecture/) by a guy called Anthony Simon, discussing the architecture behind his one man startup. I found it encouraging to see someone elses setup and how it's been architected.

I've worked for a lot of larger (and therefor big budget) clients who utilise cloud, on-prem or hybrid aproaches. Some of these clients spend £10,000 a week JUST on logging infrastructure! Clearly that kind of budget is way out of reach for a startup, and so when you're a self funded startup you need to look at ways of running things the best you know how- at budget! And so this is why Anthny's post struck a cord with me. We all know how to build highly scalable systems with budget, but when it's just you, trying to figure it all out without the huge infrastructure, budget, teams etc - it's a little isolating!

And so, because I found Anthony's post so useful, I thought I should share what we have @ highstreet.ly.

First up a little background.

Our small team is comprised of 4:

Myself - I do all of the infrastructure, hosting, backend coding and 50% of the front end
Chris - Front end
Michael - marketing and business
Jon - Design

We currently run it from our homes - 2 of us in the south of England, 1 in the Midlands and 1 in Mexico!

We started off as a ticketing platform, and before that a native app for festivals. But we never really got too far with that and things changed last year...

Highstreet.ly was a pandemic baby. We were aware of the struggle that lots of highstreet shops were managing pre-covid - but during covid things clearly got a lot worse. We knew that alot of bricks and mortar businesses were struggling to compete with other services offering either similar products with fast delivery and competetive pricing or utilising the highstreet businesses and wrapping them in a last-mile delivery solution and charging them HUGE fees for the pleasure. We wanted to give these businesses an option to compete where they aren't being cheated - we wanted to help empower the highstreet.

And so highstreet.ly was born. We are a subscriptions based SaaS offering, providing tools that we hope will allow highstreet businesses to offer services they maybe wouldn't otherwise be able to at a cost which is far, far less than the other offerings. Our service has three main components:

Shop / widget (for self hosting)
Dashboard
Operator app

In order to keep the costs down but still pack a punch we host our own servers in a datacenter in Brighton(FastNet). This cuts our hosting fees into a fraction of what they would be in the cloud.

I'm going to echo what Anthony said in his post: Our architecture is the way it is because it's what suits us, it was developed over trial and error and reflects my own personal experience and expertise - your solution may / will be vastly different. There is no right or wrong way to archteour system - I'm certainly not able to advice on that. Our system is architected the way it is so that we can streamline our infrastructure, keep our costs down and focus on building the software that makes our product.

I also want to repeat what Anthony said is his post: You can tell by the image below that we use a huge amount of amazing open source software - we really do stand on the shoulders of giants. Making a SaaS product is truly a challenge but it would be pretty much impossible without the amazing work put in by the people behind these projects.

Here's a birds eye view of our architecture:

What we got:

We have 3 x DL380 servers hosted in the datacenter
We have 1 x SonicWall firewall
The 3 servers each run Esxi (Open Source version) to host the nodes that make up our hosting
1 of the VMs runs Postgres
1 of the VMs runs RabbitMQ
We then have 12 VMs which make up our Kubernetes cluster

On all of this we run 3 environments (LIVE, TEST, DEV) - each in their own namespaces with their own databases and vhosts in RMQ.

To provision the cluster I use the amazing KubeSpray (https://github.com/kubernetes-sigs/kubespray)

I've also added to that our own ansible project to install things after the cluster is initialised - stuff like CertManager etc. We used to host Postgres and RMQ inside the cluster using Rook / Ceph but I found the performance lacking and recovering from a failure was very involved.

Once the cluster is up and running we use Pulumi (https://www.pulumi.com/) to deploy our services. However, this is done via a github action - with a remote runner which lives in our cluster. So pushing our pulumi code with a git tag will trigger a build of our IaC project - which sends the build context to the runner living inside our cluster which then installs all of the services for the namespace being targetted. It sounds complicated but it's very, very easy once it's set up and it means I never have to SSH onto any servers to deploy anything.

We have 3 types of containers we need to build to make up our system:

DotNetCore API services
Ember / Svelte NGinx containers
Migrations jobs

The migrations jobs use FlyWay (https://flywaydb.org/) and work by convention - any new script in the relevant folder is picked up and the job runs. The dotnet core images are built using GitHub actions and stored in Docker Hub and the same for the Ember / Svelte images - except these are compiled in the action and the /release folder is copied into the NGinx image.

We have 5 bounded contexts and 4 of them have a message receiver process so we have 9 services that run as microservices inside each of our cluster namespaces. We use the CQRS/ES pattern where applicable.

For authentication we use IdentityServer and our microservices run in a zero trust architecture (https://www.paloaltonetworks.com/cyberpedia/what-is-a-zero-trust-architecture). Our UIs use OAuth and JWT tokens to authenticate against our servers.

Because we are using a microservice architecture we also use a BFF setup (https://tsh.io/blog/design-patterns-in-microservices-api-gateway-bff-and-more/#:~:text=BFF%20is%20essentially%20a%20variant,multiple%20gateways%20for%20each%20client.) - this allows us to present a specialised API for each of our clients which only presents what is necessary. It also allows us to do things like cahcing and rate limiting. For the BFF we use Ocelot: https://github.com/ThreeMammals/Ocelot

When we specify a service with an ingress (api's UI's etc) - cert-manager automatically provisions the TLS for it - this hangs off of the Pulumi infrastrusture where by Pulumi will create the DNS at Digital Ocean automatically for us based on the configuration of the environment being deployed. We use the cert-manager DNS01 challenge with DO to make this happen: https://cert-manager.io/docs/configuration/acme/dns01/digitalocean/

Putting all of this together, we have a fully automated CI/CD pipeline with versioning at every level - even the infrastructure can be rolled back with a single command. This alone is something I see "big budget" projects fail at all of the time. The code and wiring to acheieve this is simple, uses standard. open source tooling - and that matters because as we grow, new developers joining the team will be able to "get" all of this really quickly.

For local development we use Kind-Kubernetes https://kind.sigs.k8s.io/docs/user/quick-start/ - the good thing about this is that, to spin up a cluster locally just requires a new Pulumi stack definition. My local cluster uses exaclty the same code to initialise as the deployed, publicly visible environments.

For logging we use NewRelic - this has a log scraper installed inside our cluster which automatically ships our telemetrics to NR for analysis and alerts.

Since we are a subscriptions based service, we currently use ChargeBee (https://www.chargebee.com/) to host our plans and add-ons. We have a webhook set up on our end which recieves events from ChargeBee and allows us to mirror the configuration and react to changes in user subscriptions.

The good thing about ChargeBee is that it enables us to set up the subscriptions infrastructure without needing to implement 100% of it - we can settle that tech debt when we need to. It also integrates with Stripe.

For our shop widget we use Stripe to handle card payments. The good thing about Stripe - for us - is that we can route payments for products purchased to our B2C customers Stripe accounts and our platform fee to our Stripe account. We also plan to allow our B2C customers, customers (!) to subscribe to their products - and Stripe allows this in a very easy and fluid manner.

That's a kind of whirl wind tour of the systems behind highstreet.ly! Techies with a keen eye will see there's one or two elements missing from the picture above (CDN etc) - we're working towards these things and we will get there. It's all a part of the challenge! I hope it helps reassure someone else out there in the same way reading Anhony's post reassured me!

It's been a long time baby!

2020-06-12T08:36:00.010-07:00

It's been a while (5 years!) so I thought I'd start up the old blog again - see where it goes!

To kick things off I thought I'd do a couple of videos showing off the sona* stack as it stands today. YES we are STILL working in our spare time on the sona* dream!

It's currently being built as a ticketing / delivery platform. Depending on the options selected during deployment it will run as either a ticketing platform or a delivery / online shop app. The main focus atm is the delivery side - allowing local shops a cost effective way to sell products and deliver them to customers amid the pandemic.

This video kicks things off by showing off some of our devops stuff. It shows our use of things like Kubernetes (full blown and developer minikube installations), Ansible, Pulumi, a home made CLI written in NodeJs, Consul and some other goodies.

The system is still written with a CQRS/ES backend written in C# and uses a mixture of EmberJs, Svelte and Xamarin for the front-ends. And they say full stack doesn't exist? ;)

Jwt Authentication with Ember + SailsJs / Waterlock Part 2 - Google oAuth2

2015-09-23T06:54:00.002-07:00

Leading on from part 1 Jwt Authentication with Ember + SailsJs / Waterlock I decided to continue with the theme by adding the ability to authenticate with a 3rd party authentication service - in this example I'm using Google oAuth2.

Now this wasn't simple for me to figure out initially! Partly because, in my mind, the confusing docs surrounding Google oAuth2 where they state:

When a user grants access to your app for a particular scope, the user is looking at the user consent screen, which includes project-level product branding that you set up in the Google Developers Console. (For information about setting up the consent screen, see Setting up OAuth 2.0 in the Developers Console help.) Therefore, Google considers that when a user has granted access to a particular scope to any client ID in a project, the grant indicates the user's trust in the whole application for that scope.

Now, I read that as: when you collect an authentication code from one medium within your applications "product" suite, provided other parts of that suite use a client ID that share the same "project" at Google's end - then you can use that client and authentication code to gain an access token - and thus validate the user. Apparently not so! I spent ages going round in hoops over this and settled on a simple setup which means I can authenticate the user on both the client (Ember) and server (Sails).

To make things clear - the idea of all this is to provide a sensible UX for a browser application to authenticate using 3rd party auth services. This includes having the auth flow inside the application and as we are an SPA we don't want to have any nasty page refreshes or redirects (in the main browser). I am pulling in a few other libraries to achieve this and have had to modify the Waterlock library slightly to enable this to work. I'll be cleaning that up and pushing back to Waterlock soon but for now I have included the deps as submodules inside our repo.

The desired flow is:

User clicks "authenticate with Google" button
Popup appears with Google's auth flow
User provides access to our app
Popup closes (main app never refreshes or redirects)
Our app receives the authorization code from Google
Sends that code to our API
Our Api then validates that code by exchanging it for an access token
If the code is valid we find or create a user and set up the JWT for the client app
Our client app receives the JWT and uses that for future requests to the server

Quite long winded but the net result is a great UX for our users and a reliable authentication flow that's secure. Now - one major change that I have had to make here to make this work - baring in mind that I have no idea how to get "Cross Client Authorization" to work - was to change the way the app access the API. To enable this I am proxying the API through ember. You can achieve this by running the following command (instead of just running "ember server"):

ember server --proxy http://localhost:1337

Simple!

In terms of changes - I have included a modified version of waterlock-google-auth in the API - this has a simple setup in /config/waterlock.js - you can see the required bits in my repo and by reading the readme on the main repo. The changes I had to implement were to ommit the requirement of the client to provide a CSRF - I may add a step where the client can retreieve one before starting the auth flow - I may omit it by using a separate endpoint for SPA auth - not sure of the implications yet. That's all for the server!

The client has a few new pieces:

I've included the ember-cli-simple-auth-torii package - this handles auth flows requiring a popup
I've also included the ember-cli-simple-auth-oauth2 package
And finally the torii package.

Because we are customizing the standard flow a tad (sending the auth code to the API) I had to implement a custom authenticator: https://github.com/wayne-o/ember-waterlock-example/blob/master/sonatribe-ui/app/authenticators/authenticate-google.js

The interesting part here is where I exchange the auth code for a validated user with an access token provided by our server - this makes up our JWT! Nice.

I'll likely move on to add Facebook authentication and possibly create our own Spotify add-on to Waterlock and wire that in. Finally I'm planning on adding web sockets to provide "live" UI updates.

Jwt Authentication with Ember + SailsJs / Waterlock

2015-09-18T08:29:00.000-07:00

Thought I'd knock together a little example of using Waterlock/SailsJs with EmberJs and using JWT as the mechanism.

JSON Web Token (JWT) is a JSON-based open standard (RFC 7519) for passing claims between parties in web application environment. The tokens are designed to be compact, URL-safe and usable especially in web browser single sign-on (SSO) context.

JWT is a means of providing a simple claims based authentication between a client and a server. The token is an encrypted piece of data which can be sent as part of a payload to allow access to restricted services. The full RFC is here https://tools.ietf.org/html/rfc7519

The basic gist of JWT is this:

User authenticates with server (identity/password, oAuth etc)
Server generates JWT using secret key and some payload (usually a JSON string that provides some basic identification information, a subset of the User object for instance)
Server responds with JWT access_token
Client stores access_token in cookie or local storage (we will be using local storage)
Client provides access_token in future requests.

While this may look a lot like the standard cookies session ID flow there is a major difference in that a JWT is a small encrypted package which defines the access claim and identifies the user. They are inherently much more secure than cookies as well as allowing statelessness - and this is the big one. Yeah sure we can manage session state across servers in a cluster using a central cache like Redis. Go ahead, hope you have a nice day breaking the fundamental design of the web.

Waterlock is a nifty little user authentication//JSON web token management tool built for Sails:

Waterlock - http://waterlock.ninja/
SailsJs - http://sailsjs.org/

Using these two projects in tandem makes it ridiculously easy to get up and running with JWT and a decent REST API. Couple that with EmberJs - "a framework for ambitious web applications" and you have a pretty niffty set-up.

The best way for handling auth in an Ember app is through the use of Ember Simple Auth:

Ember Js - http://emberjs.com/
Ember Simple Auth - http://ember-simple-auth.com/

While all of these parts are really simple to use they aren't necessarily that simple to get up and running with. This whole area of software development (JavaScript in particular) is evolving at a rate of knots, JWT is a relatively new technology and server side frameworks such as SailsJs and client side frameworks like EmberJs are moving so fast it's really, really hard to keep up unless you're lucky enough to be able to dedicate your full attention! This is both good an bad news, there's a lot of well meaning blog posts out there explaining how to do XYZ with these tools but as soon as they are published they're out of date. I expect the same will unfortunately be true of this post. But for now at least the repo is using the very latest of each of the dependancies and (I hope) provides some clear guidance in getting this set up.

So that's the server side and UI components. I'm not going to go through setting up each of these components - you can see how that's all done on each of their repos. I'm also asuming that if you're interested in this little cocktail of code you're probably aware of each of them but a little confused as to how you go about pinning it all together.

So down to the code! The repo is here: https://github.com/wayne-o/ember-waterlock-example go ahead and clone it if that's what flicks your switch - I expect it does else you wouldn't have read this far.

There's 2 parts to this repo:

The Server
The Client

I'll walk us through the interesting parts of the server initially - there's not much to it!

I've added a couple of extras here that help make things simple. Ember-Data is a brilliant little library - it doesn't do everything and I don't use it everywhere but where it is used it's very clever and ahead of it's time. One of those areas is being a pretty complete implementation of the JSON-API spec: http://jsonapi.org/. And to get SailsJs dancing nicely with this spec we need to update the blueprints. To do this I use the brilliant repo by mphasize: https://github.com/mphasize/sails-ember-blueprints. What this does is turn the standard JSON output from sails into nice neat JSON API compatible JSON. Neat.

I've also included waterlock and waterlock-local-auth. To get all these you just need to cd into the /sonatribe-api directory and run

npm install

Now - that's pretty much it - if you're doing these from fresh you'll need to run the generate blueprints command but that's all documented on the repo's readme.md - RTFM ;)

The noteworthy config changes include in the /config/models.js file:

https://github.com/wayne-o/ember-waterlock-example/blob/master/sonatribe-api/config/models.js

You can see that I've added the "associations" and "validations" nodes to the config. It just defines explicitly how you want these types of nodes to appear in the JSON output.

Next up is waterlock.js:

https://github.com/wayne-o/ember-waterlock-example/blob/master/sonatribe-api/config/waterlock.js

You're going to need to tell it to pluralizeEndpoints and also you need to rename the token name - ember-simple-auth expects "access_token" - don't do this and you're into a world of pain. You also want to set "stateless" to false - we don't want any nasty sessions on the server thank you please. Sessions are grotty at the best of time and mess with your head when you try and step up in the scalability realm. JWT is stateless authentication. It keeps things clean.

That's it for the API! Simple yeah.

So now onto the client - this is where things can get a little... involved. But considering what you're getting - I'd say not too involved. You'll want to cd into the /sonatribe-ui folder and do the usual npm install && bower install stuff.

I've included ember-cli-simple-auth and ember-cli-simple-auth-token which are set up in the /config/environment.js file: https://github.com/wayne-o/ember-waterlock-example/blob/master/sonatribe-ui/config/environment.js

Go with my defaults for now - you can mess with these later if you want.

Things to note:

I've used pod structure - it's great for organising an app small or large.
I've used the new computed decorators: https://github.com/rwjblue/ember-computed-decorators meaning we get Java style attribute decoration for things like @observe and @property and even @whateverYouWant

All this works out of the box when you npm/bower install my repo. The login and register components can be found here https://github.com/wayne-o/ember-waterlock-example/tree/master/sonatribe-ui/app/pods/components - we let the framework do the work so these components are ridiculously simple:

The only noteworthy thing being the slightly different flow for registering as opposed to loging in - waterlock (currently) uses the same workflow and if an un-registered user attempts to login and "createOnNotFound" (https://github.com/wayne-o/ember-waterlock-example/blob/master/sonatribe-api/config/waterlock.js#L48) is true, waterlock assumes you want to create a new users - and provided the dupe checking passes (no duplicate username/email validations) - a new user is created. In future versions of Waterlock there will be an explicit /register endpoint. For now it's best to query the /users endpoint for email / username availability. I'm doing this here: https://github.com/wayne-o/ember-waterlock-example/blob/master/sonatribe-ui/app/pods/components/register/register-panel/component.js#L10 (not wired up atm - I'll likely do that in the week to finish up).

So the process is to call /login if the validations for username / email pass - get the user registered and logged in and then, using the response with our authentication data we can populate the users User model and save. Finally we push the user off to the main app by sending an action to the parent route ("userAuthenticated").

I'm aware that because I am accessing Ember-Data and JQ.Ajax within this component I am completely side stepping DDAU (http://www.samselikoff.com/blog/data-down-actions-up/) - and this is a bit cringe but I'm going to leave it as is as I want the example to be clear. The simplicity of using these libraries side by side should be unhindered!

C# Threading Primers

2015-09-16T05:59:00.000-07:00

I've been refreshing my brain on threading in C# recently and have compiled a list of resources I found excellent on the subject.

I'll leave them here for future me and also for anyone interested in the topic - save some google bandwidth :)

Joe Albahari - Threading in C# http://www.albahari.com/threading/
Async Programming : Unit Testing Asynchronous Code: Three Solutions for Better Tests - https://msdn.microsoft.com/en-gb/magazine/dn818494.aspx?f=255&MSPPError=-2147217396
Async/Await - Best Practices in Asynchronous Programming - https://msdn.microsoft.com/en-us/magazine/jj991977.aspx
Talk: Async best practices - http://blogs.msdn.com/b/lucian/archive/2013/11/23/talk-mvp-summit-async-best-practices.aspx

Working with .NET on ubuntu

2014-10-20T10:17:00.000-07:00

If you're a .NET dev interested in finding out what it's like to develop .NET apps on Ubuntu the cross over can be quite painful. I've created a vagrant box which comes loaded with the bleeding edge MonoDevelop (actually compiled from the source).

To get started you need vagrant installed. You can run this on any OS: OSX, Windows or Ubuntu. You'll also need VirtualBox - both free and OS :)
Then just pull my vagrant config from the repo.

All you need to do then is open a terminal in that directory and type:

vagrant up

BOOM!

It takes a while to get loaded 1st time as it builds MonoDevelop from source but it's all set up and ready to go when it's finished.

Vagrant is a great tool, lots of NET developers might not have heard of it but it makes creating dev (and prod) environments super easy.

Enjoy :)

Hosting an ember-cli app inside a ServiceStack (or any) MVC app

2014-10-04T01:37:00.001-07:00

The resulting code for this post can be found at: https://github.com/wayne-o/bower-link-test

We are looking into refactoring sonatribe to be an ember-cli SPA. Using the Discourse code base as inspiration I've been spiking various parts to see exactly how feasible this would be.

This all started with us wanting to mimic the discourse header (ignoring the messy sub header):

As you can see the header is a really nice, dock-able UI and has a lot of features. My initial thoughts after looking at the code to isolate the header were that this would be near on impossible. We wanted to do this because we want the transition from sonatribe to our discourse installation to be seamless - it's important that it feels like one app. Ignoring the obvious integration points outside of UI (things like updating sonatribe or discourse when the user receives a PM - this will come later hopefully in the form of plugins). I'd never used Ember or Ruby before and so diving into the Discourse source code was a little daunting!

Reverse engineering the code I managed to stub a lot of the code for things like preload store and bound-avatar and basically get a functioning version of the header isolated into it's own ember-cli project:

Not bad! Wiring this up to our own search API I actually managed to get the grouped search results working. You can see this working here:

The search API is a really poor knocked-together ServiceStack API which has zero optimizations so ignore the poor perf on the results!

All well and good but I was still stubbing the preload store. This is where Discourse have done the clever thing of loading the app with all of the data it needs to render itself on boot. Looking at the Discourse HTML you can see there is a lot of JS/JSON injected in. The problem here is that ember-cli doesn't really work this way. Ember-cli gives you the folder structure to develop in and uses a filesystem watcher to recompile your code (ES6, SCSS etc) into a distributable form. It even serves the index.html inside a HTTP server for you.

Discourse works this way because the SPA knows you're logged in through the presence of a user object injected into the preload store. They use omniauth which loads a popped browser window to handle the auth mechanism - once the user has authenticated it calls into a hook (window.opener.Discourse.authenticationComplete) to notify discourse that authentication has completed and sends the result. If the user successfully authenticated the browser is reloaded and because the user is authenticated the users JSON is loaded into the preload store.

So, serving the index.html as a dynamic (is that still the right term? It sounds so ASP 3.0!!!) page is pretty much a must have.

To do this we are going to need to drop the HTTP server provided by ember-cli and work out a way to push the /dist folder to another folder which can be part of a hosted project. I'm a big fan of ServiceStack but the hosted project could literally be any language/framework.

The idea here is to have 2 projects:

ember-cli app where we can develop the JS and CSS
an integration project where we can host index.html and inject session state and preloaded data

To keep the integration project up-to-date when we make changes in the ember-cli project we can use Bower link (http://bower.io/docs/api/#link) - this will replicate changes to the integration project live using symlinks. You can link the /dist folder to the target project - you can see this here: https://github.com/wayne-o/bower-link-test/blob/master/BowerLinked/BowerLinked/bower_components/assets

Rebuilding the project on changes can be done using the ember build --watch command.

What this will do is - every time you edit some code in the ember-cli project - ember will notice the files have changed and re-build the code - it then copies the resulting build to the /dist folder. Bower's symlink will then copy the code to the integration project.

We can then host our own contents of index.html as a view inside our hosted environment: https://github.com/wayne-o/bower-link-test/blob/master/BowerLinked/BowerLinked/Views/Home/Index.cshtml

So now we get the mixture of serverside code inside the ember-cli index.html and we can inject in session state and preloaded data!

This means I can go back to my header spike project and replace all of the stubbed code with real user data and start to really figure out the auth aspects of this refactoring.

While this is a ridiculously simple example - the project in github does show this all working:

The "this is a test" lines are coming from the cshtml and the ember app is booted below it! All I need to do now is get the authed user from ServiceStack and serialize whatever data we need in the UI into the preload store.

w://

Running ServiceStack / RavenDb API on Ubuntu

2014-09-07T07:12:00.002-07:00

I've been playing about with Docker and Vagrant a lot recently, I find them to be really amazing tools. Obviously with the amount of OS envy streaming out of my little foray with Docker in particular I am trying to move the Sonatribe API over to run on Mono inside a docker container.

ServiceStack runs great self hosted on Mono but I have been getting issues with RavenDb: async calls seem to fail and things like bulk inserts seem to fail with this: http://stackoverflow.com/questions/13859467/ravendb-client-onlinux-connecting-to-windows-server-using-mono-http/13866929#comment40187983_13866929

If anyone has any help or pointers on these i'd be massively grateful!

Sonatribe conversations platform - CQRS knockout - part 2: GetEventStore.com

2014-07-30T06:43:00.001-07:00

Following on from part 1 (CQRS Journey) this is my implementation spike using GetEventStore. Assesing which CQRS setup we should use at sonatribe for the new conversations feature.

The code can be found here.

This implementation is quite a bit different so I'll discuss where they branch off from each other and maybe try and figure out the pros and cons.

Using EventStore as the persistence mechanism for storing event streams takes a lot of faff out of the process. For instance if you don't want to you don't need to have any service bus sitting between the API -> domain service and read model. However - EventStore is a more lower level technology and because of that you don't get so much out of the box as you do with CQRS Journey.

With this implementation I decided to keep the domain service in a separate process at the other end of a RabbitMQ queue. So - looking at the API you won't see a huge amount of difference in terms of complexity (except maybe RMQ is heaps simpler to get up and running).

Moving on from there I have a console app (as opposed to a worker role) running with TopShelf (for easy promotion to Windows Service when required). The code for this is here: https://github.com/wayne-o/ges-tryout/blob/master/src/Conversations.DomainService/Program.cs. Really simple way of wiring up the command handlers using Castle Windsor.

Looking at the one and only command handler: https://github.com/wayne-o/ges-tryout/blob/master/src/Conversations/Commands/CreateNewConversationCommandHandler.cs you can see how easy it is to save an aggregate root into the event store using the GetEventStoreRepository. This was one area that the developer is left to figure out - finding this repository was a great help.

You can see in the AR that when I create a new conversation it fires a new ConversationStarted event. This event is listened to in the read model. The wiring up for the read model you can see here: https://github.com/wayne-o/ges-tryout/blob/master/src/Ges.ReadModel/Program.cs. Again - there is some work to do here to turn the received events into something that can be worked with. There is also some (overly simple) code here to dispatch the events to the event handlers.

Inside the event handlers I am denormalizing the data to a RavenDb read model.

Aside from the added work of figuring out the event dispatcher I find the GES architecture much much simpler than the Azure based CQRS Journey code. There seems to be much less ceremony - even after I reduced it drastically in my refactoring of CQRS Journey. I haven't got any perf details for either implementation but Greg claims GES will natively handle 20,000 messages a second on a single node so I doubt perf is an issue for either implementation.

Next up I'll be adding some real features using each of these infrastructures. I like to work from an emergent design BDD style approach so will be looking at which of these works best under a test 1st dev cycle.

Sonatribe conversations platform - CQRS knockout - part 1: CQRS Journey

2014-07-28T08:47:00.001-07:00

I'm about to start hacking away at a new part of the sonatribe API - a feature we've been calling Conversations. I've been a fan of CQRS/ES for a while now and think that the natural way that a conversation is a stream makes it a very good fit for some event store goodness.

Where as I used JOlivers EventStore in the past itteration of sonatribe (we attempted to make the whole site using CQRS/ES at one point but struggled with infrastructure and complexity) - things have moved on since then. Most notably Microsoft have done a whole Patterns & Practices feature on CQRS/ES and there is now code available that can be used in projects like this. And perhaps more exciting is the fact that GetEventStore.com is out in the wild and ready for use!

Before I go running in and developing the platform with one or the other I decided to take them both for a spin. Starting with the MS P&P Cqrs Journey code I made a pretend set up so I could asses which would be better suited to our needs.

I guess the main concerns for me are:

Simple dev cycle- as I mentioned above - complexity seemed to be a major time burner for us before - I want this implementation to be drastically simpler.
Testable - this is a blatant must - previously this was quite hard - there was very little guidance here also.
It needs to be able to run on my laptop on a train! I do a lot of this work during my commute!

So - taking the CQRS Journey code (http://msdn.microsoft.com/en-us/library/jj554200.aspx) I hacked away and took out the bits I didn't like, the bits I didn't need and added some bits I felt I wanted. Mainly I don't like Unity IOC - I feel like you have to hold it's hand far too much. I initially tried swapping in Funq but it was too simple in features so ended up with Castle Windsor - my old time favourite but might be a bit bloaty for some.

I only managed to port over the SQL bits - the Azure ServiceBus part is not working yet and probably won't - for reasons I'll specify in my next post.

I also decided to use RavenDb as my read model - Table Storage is OK but RavenDb is absolutely nuts in terms of speed and simplicity.

Anyways - the code is here: https://github.com/wayne-o/cqrs-journey-tryout

My opinion after all this is: It's nice to have all the code - esp for the event store, the publishers and recievers etc. There is a _lot_ of config and messing about with the original code - I tried to simplify this by registering the command and event handlers by convention but it still feels very "ceremony"

Next up I'll be taking geteventstore.com for a spin - a very quick and simple dive into getting a command through an API and eventually handling the command to maintain a read model - exactly the same scenario as the CQRS Journey try out - just using geteventstore.com as my event store...

Stop Checking in the Settings!!

2014-07-19T07:27:00.001-07:00

Ideally configuration hangs off of the build setting or (better still) the deployment process takes care of the configuration. If you're using Puppet or Octopus to deploy your code then this is super easy. Having this sort of infrastructure isn't always feasible especially on smaller projects with smaller teams. The project I'm working on at the moment is one such codebase. We share settings.xml files, there are many of them and each developer needs to configure these files as per their own environment as well as the deployed environments - per feature. This is a problem we are very aware of and we have scheduled tickets to take care of it but in the mean time I'm going to look into how we can use Git to limit the pain of this.

I am a massive culprit for checking in settings files - everybody does it - especially the nub on the team (me)... It's really annoying to the poor person who does an update after the point and then has to rework the settings files. So here are a couple of ways we can fix this (this is very basic git stuff so if you're well versed on Git you'll be well aware of the following!):

Open up a command prompt and navigate to where you keep your git repos. Mine are normally in d:\git

Lets start with a fresh plate, run the following:

We've just created a brand new git repo for us to play with. very straight forward. We don't want to work in master so lets create a new branch:

Now that we have the dev branch we can make our settings file.

Add some dummy dev environment config to the settings.xml file. Now we can add that to the dev branch and commit it.

Now that that's safe and sound we can create our feature branch where we will want to do some work and manage our own config:
Put some code in test.cs so we can commit that into the wayne branch

Now lets customize the settings.xml file:

Make the dev specific settings feature specific so you know if this works or not when we come to doing the switcheroo. Now we want to commit the feature specific settings into the feature branch:
Now we'll continue to do some more work:
And commit that into the feature branch
OK so now we have a dev branch with some settings in, we branched from dev to create a feature branch, did some work in that branch, configured the settings file to make it specific to that branch and then did even more work, big day in the office. What we want to do now is to merge the work in the feature branch to the dev branch - but we definitely do not want the settings from the feature branch to show in dev, that's just going to wind everyone up.
To do this we can revert the commit we made to push the settings into the git branch. We can check the log to see the commit and revert it. To do this run the following:

This will yield something like the following (type q to exit the log):
As you can see the commit with ID "965f41809225abbe2ed2b74d6e162e27c9c18a72" so to revert this commit we can issue the following command:
And then we can safely merge our feature branch into dev:
Open up settings.xml and verify it's as expected - dev settings, not feature settings.
So that was one way of making sure we don't merge stuff we don't want to merge. If we wanted to do some more work in the feature branch and get the settings back we can use the same approach above to undo the commit made by git revert - git simply creates a compensating action to revert the commit by and this creates another commit - so we can just revert the commit created by the compensating action (which will create another compensating - git is just event sourcing after all!). Another approach is cherry pick commits from the feature branch. This will apply a single commit to the branch you are merging into. Here is the general gist of it:

Git is powerful stuff - cherry picking is basically the converse to the 1st approach. It shows how useful doing small, discreet commits can be. In general there are only a small handful of git commands that I use frequently - you can do so much with a small amount of git commands it's ridiculous. But there are some extremely useful, more "advanced" features which really bring Git into it's own. I'm still learning Git but one thing I have found is that using it properly really pays off and there is always a sane solution to whatever insane mess I find myself in!

Regression testing ServiceStack services with RavenDB embeded

2014-05-25T06:15:00.001-07:00

One of the great things about using RavenDB and ServiceStack on Sonatribe is the ability to spin the whole system up inside a unit test. ServiceStack offers AppHostHttpListenerBase which can run in process and RavenDB offers the embeded database which can run in-memory. The fact that these two run in-process and in-memory means that regression testing Sonatribe REST services is simple and fast. Almost as fast as unit testing!

To be able to run tests like the following regression test:

I use the AppHostHttpListenerBase in my BaseTest class. Which is roughly the same as the following:

Now, When I inherit from SelfHostedBaseTest each test spins up a new host and a new in-memory RavenDB instance. This means I have a clean slate for each test, ideal for running everything in isolation. Not only this but I am also testing my services in the exact same stack as they run in production!

CQRS without CQRS and RavenDB

2014-05-24T09:37:00.002-07:00

Following on from my previous post about sonatribe, one of the hard decisions I had to make on this iteration was refactoring out the CQRS code to simplify the dev cycle and improve velocity. I'm a big proponent of CQRS - it answers a lot of questions in traditional dev and while there is some upfront complexity it is far outweighed by the lack of accidental complexity apparent in traditional software projects.

One of my favorite features of a CQRS system is the denomalized rad layer - where the data stored in the read store maps directly to the DTO's coming out of the UI. When I refactored the code, I really wanted to be able to still get this sort of functionality. Denormalized read models have been around a long long time - SQL Server has "Views" where the developer can specify some SQL - normally making use of JOIN and UNION to present a readily available denormalized view over the data.

In RavenDb we can achieve this using TransformResults. In a document database you model your data differently - rather than joins and lookup tables you can store reference data inside the main document (aggregate root). A good rule of thumb being that if you ever need to load that part of the data alone, independent of other data, then that data is a document of it's own, with it's own ID. Documents can reference other documents however using the Denormalized Reference pattern (http://ravendb.net/docs/2.5/faq/denormalized-references). Denormalized references can become a chore when the referenced data gets updated - in this case you will need to PATCH the documents to reflect any changes needed.

One of the cases where we have used TransformResults is to present the event profile (a festivals main page on the website) - in order to cut down requests to gather the data to present on this view (tags, images, users who are going, campsites, lineup info etc) we can aggregate all of that information in one TransformResult:

While this looks quite the beast - it's actually quite simple - we're pre-compiling all of the associated data for each event. This means we can pull all of the JSON needed for an event profile using a single call!

It's ridiculously simple and without cache turned on we can return from the REST API an event profile in less than 100 milliseconds - with cache turned on and tuned we can smash that right down - that kind of optimization comes much later however.

Now this isn't _really_ anything like CQRS - but it gives me a good-enough-for-now alternative. At some point I will be bringing CQRS back into sonatribe, this time round most likely using Greg Youngs GetEventStore (http://geteventstore.com/) - previously I used JOlivers EventStore and CommonDomain (which is used in Greg Young's implementation to my understanding). But for now I'm happy with this setup and it's simplicity.

Sonatribe technical bits and pieces

2014-05-24T09:10:00.000-07:00

The sonatribe project makes use of a broad range of technology from PHP to Microsoft Azure and a lot in between. We host the main site on Ubuntu but the backend is hosted on Azure and makes use of ServiceStack to build the REST API. The main components of this project are:

PHP UI - running on an Azure Ubuntu VM
ServiceStack REST API - running on an Azure Windows VM
RavenDB database
EXTJS admin application
2 x Azure worker roles used to process event and artist information
Azure ServiceBus
Azure Cache
Azure CDN
SignalR backed chat server
Xamarin iOS app
Xamarin Android app
Facebook canvas

Sonatribe has always been developed in our (stu, chris and myself) spare time - so no wonder it's taken ~3 years to develop and get the alpha out the door! The initial build of sonatribe (the site is currently on it's third rebuild!) used a CQRS backend and although I maintain this is the best architecture for a project like sonatribe, the complexity in the development/test cycle slowed progress too much, for such a small team trying to get a working "something" out the door, I had to make the hard decision to refactor the code and simplify it.

Both the main website (PHP UI) and the Android & iOS apps use the same API for authentication and data. This is a great design as it means all of the business logic stays in one place. And when you're building a system that relies on multi-axis permissions/roles with a range of business logic and views over the data - this is a big advantage.

The Extjs admin app allows us to import data from clashfinder. The data coming from that JSON is very basic and only specifies the act name, stage and time of the set. In order to build upon that data to be able to provide spotify playlists, artist bio's and images etc we have an import pipeline which also allows us to update that data (quite complex as the clashfinder data has no Id information). Importing information for each act is quite a long running process as we gather as much information as we can about each artist, we download images relevant to each artist and generate renditions for use in the site. Because of this long running process the import process follows the following flow (ignoring clashfinder updates - that's another story!):

Initial import fired from admin app
The REST service picks up the import ID and pulls the data from clashfinder
The JSON is deserialized into POCO
A RavenDb bulk insert operation is started
The locations (stages) are pulled out of the data and are added to our database
For each of the artist sets we create a new listing event - a simple document which specifies the act name, stage, start and end
The bulk insert is committed to the DB - this process can take ~ 1 sec for a large festival with ~2000 acts and ~190 stages
For each listing event that was created we push a message onto the Azure ServiceBus
The worker process at the end of this queue picks up each message and processes it asynchronously.
We check to see if we can find any information about the act by using their name - we check spotify, last.fm, music brainz, wikipedia etc
If we can find the artist we download their spotify track Ids, artist bio, images etc and create a new artist in our DB linking all of these assets.
The artist is added to the listing event and saved
If we can't find the artist by the act name alone we try to break the name down - searching for substrings - perhaps the act is a collaboration.
For each artist we find in the act name we add them to the listing event and save it away

The admin site has it's own schedule designer using bryntum scheduler which we can use to create and edit lineups. The preferred method atm is clashfinder imports due to the communal collaboration that goes into creating them.

Using Xamarin to develop the Android and iOS apps means we can share a lot of the code base between the apps as well as make use of SignalR to provide chat functionality. Servicestack also provides a great PCL client - it's a no brainer. The apps are currently in the early stages of development - at the moment you can log in, view festivals and their lineups. We aim to have very basic implementations out in the app stores before Glasto. The apps will be free :)

We've recently released the 1st alpha of sonatribe (http://alpha.sonatribe.com) and while there are a few rough edges we're very pleased with the result. The platform differs from anything out there at the moment because of the social aspect of the site as well as the fact that we will be offering native Android and iOS apps. While the alpha is a very bare bones (and in some places quite clumsy) implementation, we have a great platform to build on. This year our aim is to stabilize the platform, introduce the mobile apps and react to feedback. Our next big undertaking is tackling conversations, to build on the social aspect of the site. We're planning on swerving the traditional "forum software" implementation and intend to build a conversation platform from the ground up based on the usage of our demographic - something we can grow and improve upon and will be suitable for the next 10 years - not extinct 10 years ago. We've always found forums to be too 1990's and we've always said Facebook waters down the semantics too much for something like this (the whole reason sonatribe exists).

We've got a great set of features and a lot of work has gone into getting to where we are, we're listening to user feedback and looking forward to the next set of great features. There's a hell of a lot left to do though!

w://

Sonatribe Alpha

2014-04-03T00:19:00.000-07:00

Alpha = release to friends and family!

Sonatribe has been a long time coming! We started the project as FestivalStar around 2009 in a kind of accidental, off the cuff development. Back then it was an iPhone app that supported multiple festivals which did quite well on the AppStore. Since then we've scrapped the iPhone app (for now) had a change of direction, a complete redesign and after about 4 iterations (and 5 years!) later we're about ready for an alpha release! Sonatribe is built by Chris, Stu and myself. We've been hacking away at this in our spare time so it's been a long and windy road.

The alpha that we're releasing is by no means the end product. We've got the project to a state where we have a small, manageable set of features we hope will be useful to festival goers. Sonatribe development is a constant ongoing thing and we'd like to generate a feedback loop from our users where by we shape the site and it's features based on this feedback loop. At the moment we have the skeleton to be able to provide you a small set of tools to socialize and plan your festival experience. By using the site we can get an idea of your likes in terms of music genres and can provide you recommendations for bands/DJs and events you might not have heard of before.

We've teamed up with clashfinder.com to provide lineup data (we have our own lineup designer in the backend too which we feed back to clashfinder). We've also integrated with Last.fm to provide artist information.

So what do you get? Like I said, it's early days but at the moment you can browse festivals, see the lineups for the festivals, create your own schedule, view act information (where they've played, where they're playing, bio, images etc), create a "campsite" and invite friends to it. When you have a campsite you get a private place to chat and a group schedule. You also get a (at the moment - very simple) profile page like any social network with private messaging.

We've built Sonatribe from scratch to be exactly what it is - a social network for festival lovers. We've not shoehorned any blogging platform to create it. We have complete flexibility in it's future direction and a metric ton of ideas for features and goodies. There's no advertising and we have no intention of splattering the site with any. We're not sure how the site will pay for itself as yet but we want to keep it as clean and user friendly as possible.

At this stage sonatribe.com is an alpha release and so you shouldn't expect it to be a complete and finished product. There will be some workflows that could be better, some of the site may seem a little slower than we'd like and somethings might just fail! There might be some features you'd like to see and some you'd like to tweak. All of your feedback is valuable and will help shape the project as it matures.

How to provide feedback: On the right hand side of the site there is a tab you can click to provide feedback. If you want to be notified of progress via email you can join the sonatribe-users email list: https://groups.google.com/forum/#!forum/sonatribe-users
How to access the site: You can get to the site at: [You'll be provided the super secret URL!] - you will need a Twitter, Google or Facebook account to login.

Rules of the road:

You can invite some friends, we'd like to get some real world usage.
We might have to cap new members at some point while we work out the inevitable creases but for now we'd just like to get the site working through it's paces.
If you're going to a festival that isn't listed just pop us an email, get in touch using the "Feedback and support" tab.
This is an alpha, things might go wrong for you, if anything dodgy happens please let us know and we'll be able to make the site more stable using your feedback.

And finally, we aren't affiliated with anyone, nor do we claim to. If you're a festival organizer we'd love to hear from you. Our intention is to provide a means for users to discover your events and help build your community. We're building tools to help you manage your events and interact with your community which we aim to provide for free.

Looking forward to receiving your feedback!

Wayne, Chris, Stu

Xamarin Android 3G helper

2013-10-28T08:58:00.002-07:00

Managing 3G connectivity in Xamarin Android is a nause - you have to mess about with JNI and a host of other gruff. I've created this little helper to make it easy:

RavenDb tip – deploying unstable builds regularly [REPOST]

2013-10-28T03:33:00.000-07:00

When significant updated builds to RavenDb are posted to the Hibernating Rhinos build page (which is fairly frequently – a good thing) it means I need to deploy the updated DLLs to the servers. We currenty manage 8 instances of RavenDb – 1 for each service that supports sonatribe. That can be an aruous task – especially as we host 4 isntances on 1 server and 4 on another.

I’ve been playing about with PowerShell recently and thought this would be a good opportunity to get my teeth into it. I was initially trying to get the WebDeploy stuff to work scripted but that was a massive nause.

I ended up just going with a script i can run on each server. Here are the steps I needed to accomplish:

stop iis
remove the old files from the db folder including the bin and all it’s contents but excluding the Data directory and the web.config and log4net.config files
copy the contents of the deplyment folder (which i set up to include the plugins needed) to the database directory
restart iis
Simples.

Here’s the script:

Nice and simple. Just create an array of strings containing the paths to the folders where ravendb is hosted. Loop through them deleting and copying. Restart IIS.

This saves me about an hour a week and also guarantees that I don’t accidentally miss something. It’s a shame the web-deploy stuff in msdeploy didn’t work – we host the databases in IIS so it would have been a suitable solution and would have meant i could deploy from my dev machine without even needing to log onto the server. I expect I’ll try and tackle that at some other point.

w://

Taking a stab at generating strongly typed Facebook Graph C# classes for .Net/Asp.Net (My attempt) [REPOST]

2013-10-28T03:29:00.002-07:00

The facebook API is flakey to work with at the best of times. However as Deap mentioned on his blog post: facebook documentation requires login (showstopper) it’s gotten even worse if you want to automate the process by scraping the docs. I’ve managed to take a stab at this using a different method to retrieve the docs – this method allows us to 1st login to the FB docs and then scrape away – building up the strongly typed objects on the way. Here’s my branch: FacebookSdk-contrib on github. This is using HtmlUnit – a Java project – so we’re also using a wrapper to use it under the IKVM runtime. It’s nearly there. Just some final bits to iron out. w:/

Replication predicates in RavenDb [REPOST]

2013-10-28T03:28:00.001-07:00

Replication in RavenDb is pretty fekin sweet. I helped Oren by submitting bugs and testing it out in sonatribe dev by pushing it in a real world usage scenario. One of the features I asked about was the ability to filter out parts of the namespace for replication – instead of the ‘all or nothing’ approach used by default. Oren (as you’d expect) simply replied with a request for a patch. So here is my patch.

My fork is here: https://github.com/wayne-o/ravendb
Basically – my aim is to allow the developer to set replication filters – so only a part of a namespace is replicated – rather than the whole db. To do this I’ve added a ReplicationPredicate class in the Raven.Bundles.Replication.Data namespace. this is used when initialising the db:

it’s a work in progress but i’m finding it hard to get the time to commit to it due to the new baby. if anyone fancies lending a hand – there’s a test in the SimpleReplication.cs file Can_filter_replication_using_replication_predicates
w://

Asp.Net Mvc action selector – with list support [REPOST]

2013-10-28T03:25:00.000-07:00

Here is my updated AcceptParameterAttribute class with support for data bound lists:

some sample code to use this updated version:

decorate the action with something like:

and then the buttons in the list can have the following format for the name:

hth

Installing Solr in Apache on Windows

2013-10-24T09:24:00.002-07:00

I've been messing about with Solr a lot recently and although the finished setup is great at what it does, getting it setup in the first place is a complete nause - especially on Windows. Anything to do with Java is going to be an XML configuration nightmare so to help myself in the future and whoever else stumbles into the same issues I had, I'll document the process here.

First up you want to ensure you have the latest install of the Java SDK installed. Grab that from Oracle and do a standard install. http://www.oracle.com/technetwork/java/javase/downloads/index.html

Next up you want to get Apache installed as a Windows Service. I installed the latest to date (Tomcat 8.0.0-RC5 (alpha)) - grab that from here: http://tomcat.apache.org/download-80.cgi and follow the instructions.

I'm not going to go into the details of installing the packages above as they are standard installations and you should accept the defaults in most cases.

Now that we have Tomcat installed and running we can use it as the container to host the Solr instance. Grab the latest build of Solr from their website (http://lucene.apache.org/solr/). This is where the install gets a little interesting. Getting Solr running for your requirements will mean altering a lot of config XML files. It's obviously very handy to have those XML files under source control so keeping the config files inside the default locations withing Solr is not going to work. In this setup we'll want to point Solr to a home directory other than the default so we can keep the config files under Gits control.

Once Solr has downloaded unzip it and grab Solr***.war (where *** is the version number) file from the solr-4.5.0\solr-4.5.0\dist folder and place it inside the lib (C:\Program Files\Apache Software Foundation\Tomcat 8.0\lib) folder in Apache - rename it so that it is called Solr.war. Now grab all of the jar files from solr-4.5.0\solr-4.5.0\dist\solrj-lib and solr-4.5.0\solr-4.5.0\dist folders and place them inside the same lib folder in Apache. It might not be necessary to install all of the jar files but incrementally adding them by deciphering Solrs exceptions takes an age and I doubt it has any negative impact.

As I mentioned before, we want to store the config outside of the normal Solr installation so that it can live in the main repo folder along with whatever other supporting code you may have. To do this we need to tell Apache where to find our Solr home. To do this we need to place some XML inside Apache's conf directory (mine is at C:\Program Files\Apache Software Foundation\Tomcat 8.0\conf\Catalina\localhost). Here is an example which points Solr to c:\solr

Now restart the Apache service. If everything has gone well, navigating to http://localhost:8080 should bring up the Apache homepage. Navigating to http://localhost:6161/solr/ will bring up an error complaining about collection1 not being available. This is to be expected as we haven't given the core config yet.

At this point, if you're seeing other isuses you're going to need to start using the logs directory in Apache to debug whatever issues you may have. If you have reached the same point as me then you can carry on the installation as follows.

Inside the Solr download folder there is an example installation. Grab the solr folder from inside the examples: (\solr-4.5.0\solr-4.5.0\example\) and copy all of the files to c:\solr - or where ever you have decided your config home directory is. You'll need to open solr.xml and replace the contents with this:

Rename the collection1 folder to test-solr as that is what is specified in the solr.xml. Restart Apache and reload http://loalhost:8080/solr. You should now see Solr's homepage (all be it with a warning like: Error Instantiating SearchComponent, solr.clustering.ClusteringComponent failed to instantiate)!!

SearchComponent is a contrib to the Solr project. For this example I have no need to go into getting that working so I'll just remove it form the config. Edit the solrconfig.xml file to look like this: https://gist.github.com/wayne-o/7140242 (linked not previewed as it's huge).

Now restart Apache and refresh the solr homepage. You should now be able to select the new test-solr core in the UI!

As a final note - Unless you want to keep the solr index (you don't) I would add the data folder to the gitignore file - or svn ignore it - whatever SCC you use.

Testing Android apps using Calabash

2013-07-12T05:04:00.000-07:00

BDD testing ASP.NET Mvc websites is a well beaten path with the likes of Nunit, Specflow, Selenium WebDriver etc but testing the features of an iPhone or Android app with an automated UI test suite is not so well covered. Xamarin have recently bought out LessPainful (https://www.lesspainful.com/); the guys who make Calabash (http://calaba.sh/). Calabash is an open source project which aims to deliver BDD driven UI tests to the mobile device. LessPainful allows developers to run those tests in the cloud on a huge number of devices. Xamarin are taking LessPainful and adding a C# API (currently Calabash is driven through Ruby). Tests are written using the Cucumber DSL and custom steps are implemented using Ruby. With a C# API, developers will be able to define steps using Nunit which means SpecFlow could be used to add cucumber support and automatic step skeleton creation. Potentially sharing feature files across Android / iPhone as well as HTML 5 mobile app tests.

For now though there is no public access to Xamarin's TestCloud (http://xamarin.com/test-cloud), the new name for LessPainful. The Calabash source is still open and available on GitHub, so anyone can set up and run BDD style tests using Calabash - just no TestCloud access for a while.

At Psonar we wanted to get end to end coverage on UI features and though a lot of the code is covered by unit tests, there's nothing like getting feedback from automated UI tests. Setting up Calabash was not a straight forward "install this, run this; BOOM" type of process. So I've written up my findings for future devs to get their heads around the process, it is a well worthy investment of time. I'm no Ruby/Java dev. The exceptions and error reporting as well as versioning hell that I've just been through hasn't persuaded me to jump ship yet either! Some understanding of the two are very helpful though.

First of all you need to make sure you have Ant (http://ant.apache.org/) installed. I've not had any problems installing this; it's very straightforward "unzip to c:\ant, add ANT_HOME env var". Simples.

Next up you want to download ruby 1.9 + devkit (version 1.9 is important as 2.0 has no compatible ghurkin lex parser - this tripped me up big time):

http://dl.bintray.com/oneclick/rubyinstaller/rubyinstaller-1.9.3-p448.exe?direct https://github.com/downloads/oneclick/rubyinstaller/DevKit-tdm-32-4.5.2-20111229-1559-sfx.exe

Installing Ruby is simple, just double click and run. Installing the DevKit is similar to Ant in that you need to unzip it to c:\DevKit, add that path to your PATH env var. Open a cmd and navigate to c:\DevKit and run the following two commands:

This configures the DevKit in your Ruby installations. Next up you need to use the following command to install the calabash-android gem:

If you are getting errors at this point about not being able to compile native gems, you probably installed the wrong versions of Ruby and DevKit.

All being well you should now be able to create a new folder for your Calabash tests and navigate to it and run:

This will install a default skeleton Calabash directory structure with some example cucumber code. Adding your app to the directory involves (in visual studio):
Select Release build (Calabash only supports release build APK's atm). Ensure you have no SDK linking in the projects properties view. Select Tools/Deploy Android Project. You need to specify the Android debug key located in "AppData\Local\Xamarin\Mono for Android". Specify the following when prompted:

password: android
alias: androiddebugkey
key password: android

Open a new cmd and run:

This fires Android ADB - the debug bridge that effectively outs all of the banter Android has with itself; very handy for debugging problems. Once the app is published and you've followed these steps copy the APK to the Calabash test folder and in another cmd window run:

This will tell calabash to copy the app to the device or emulator and the Calabash server (installed next to your app on the device) will start receiving delegated commands issued from the calabash process running on your computer (which interprets the Cucumber test scripts and turns them into commands sent to the device as JSON). To get you started there are a bunch of pre-canned test steps on the Calabash GitHub Wiki:

https://github.com/calabash/calabash-android/blob/master/ruby-gem/lib/calabash-android/canned_steps.md

My advice (if you are a Xamarin developer) would be to stick rigidly with the pre-canned steps for the moment. Stepping outside of this safety wall will mean you will more than likely face porting them over to C# at some point in the future. While I am sure Xamarin will maintain the ruby implementation, no one really wants 50/50 Ruby/C# definitions to maintain. The pre-canned steps seem to be fairly comprehensive and allow for a lot of expression so only advanced scenarios would absolutely require custom steps definitions. Although only using pre-canned steps are a lot more verbose (you can't aggregate steps for "Given" style steps for instance) - it's a price worth putting up with for now.

I will try and add a follow up to this post explaining the process of adding iPhone tests - sharing the feature files across both OS's. Very tasty indeed!

w

Building the www.brightonsausageco.com website - 30/10/2012

2012-10-30T14:22:00.002-07:00

This is a follow up to the 1st in this series: http://iwayneo.blogspot.co.uk/2012/10/building-www.html

And the second http://iwayneo.blogspot.co.uk/2012/10/building-www_16.html

And the third: http://iwayneo.blogspot.co.uk/2012/10/building-www_26.html

And the fourth: http://iwayneo.blogspot.co.uk/2012/10/building-wwwbrightonsausagecocom.html

Battery died on the train to Nottingham yesterday so I shall be starting back on it tonight! Looking back at my final sentence before the battery karked it I see: "Let's go and add the IRepository.GetAll() method then"... I best look at the tests and make sure that's where I left off...

After having a ganders I can see that I have added a MongoInstaller and MongoCollectionComponentLoader (wholesale robbed from https://gist.github.com/2427676 barr a couple of minor mods) which allows me to inject a lazy loaded MongoCollection into my repositories. My repository implementation now looks like:

And a test:

So, now we have the repository working with MongoDB (!!) and it's being injected into the controller so we can now mock and test and actually use MongoDb in our project. Now we need to go and modify the AdminControllerBehaviors test which checks the result of calling the Categories action so that we check to see if there is a view model returned which contains a list of categories. That test now looks like :

Now if we run the specflow file now we're moving ahead again except we fail at:

This is because even though I know we are grabbing the categories from the DB we never actually saved them because we never had a test that demanded it. Now we do so lets extend the repository to add some persistence:

Now we just need to go back and change the CategoriesAdd POST test so that we make sure we save the category into the DB:

I've also whizzed through and updated the test for the loading of the categories action so that we are asserting that we are returning some categories:

And the updated CategoriesController now looks like (I'm not suggesting this code is production ready but it passes this test!):

I also added the following to the Categories view:

Now that we have persistence using MongoDb (which I have never used before) and we have all of the IoC, MVC infrastructure stuff all set up and we can save categories I feel pretty good about the progress. Picking off the categories feature was going easy on myself because it was the easiest feature to drop into. But I also knew that this feature would also bare the brunt of the infrastructure bootstrapping side of stuff. Although this feature isn't 100% complete and the code that is there isn't exactly tight I am happy that it's a good grounding to build on. I'm going to leave it there tonight. Next session I need to complete the feature and make the specflow test pass. I'll also set up a public deployment for the staging code - at some point I should hook this into TeamCity / Octopus deploy but that's further down the line.

Building the www.brightonsausageco.com website - 29/10/2012

2012-10-30T12:44:00.001-07:00

This is a follow up to the 1st in this series: http://iwayneo.blogspot.co.uk/2012/10/building-www.html

And the second http://iwayneo.blogspot.co.uk/2012/10/building-www_16.html

And the third: http://iwayneo.blogspot.co.uk/2012/10/building-www_26.html

Monday morning, 6:00am!! I left the site on Friday with a passing step which basically meant that the add category form wouldn't b0rk on submission. I know there's some blank spots to fill here (persistence) but I don't have a failing step yet so I'm being ignorant to all these things - I'll tackle them when the tests require me to.

From the feature file I can see that we are failing on the following step: "Then I should see "category1" show up in the list of existing categories" because when we click the Save button we get a 404 POSTing to /admin/categories/add. This was a simple coding error with inconsistent action names: the GET action was called CategoriesAdd and the POST was called Categories - so the routing wouldn't work POSTing to that URL - I need to get the routing tests to be more detailed.

The good thing about allowing my development to be guided by small steps in a feature file is that I can easilly gain continuity and remain focused during disparate, micro sprints: TBD (Train Based Development!). To get this step passing I just need to add a redirect in the POST action so that we're headed back to the main category page where we can add the list of existing categories. Before I can add the redirect to the action I need to make sure I have a failing test:

Here is what the action now looks like in order to make that test pass:

We're still not out of the woods however - we still have a failing step and in fact - I think the woods just got darker! The reason we're still failing is because we aren't showing a list of existing categories. Existing categories is the first mention of real persistence - we hinted at the fact that we were storing the categories somewhere when we submitted the form but there was not concrete spec suggesting that was the case. Now we have the requirement to pluck categories out of somewhere. This means a decision on what storage mechanism must be made. Because I don't want the unecessary complication of RDBMS plus ORM I have already opted out of SQL. This leaves me with the 3 good doc based storage engines that I would choose from:

RavenDb
Mongo
Riak

And without much adoooo I can tell you that I have opted for MongoDb on the bases that it is free and has a large community. [installs mongo + NuGet for driver in both MVC and Behaviors projects]. So I'm happy with the idea that I'm going to be fetching a set of Categories from MongoDb. To do this I need to pass in a repository to the Admin controller which I can use to talk to the DB. This means going back to the first test and making the AdminController instantiation work with an IRepository:

And the code for the AdminController now looks like:

That gives us the ability but we're not doing so let's sort that. To display the categories we need to be able to get all of them from the repository. Let's go and add the IRepository.GetAll() method then.

Building the www.brightonsausageco.com website - 26/10/2012

2012-10-26T09:58:00.001-07:00

This is a follow up to the 1st in this series: http://iwayneo.blogspot.co.uk/2012/10/building-www.html

And the second http://iwayneo.blogspot.co.uk/2012/10/building-www_16.html

Third day; Third blast at the site and it's been 11 days since I last looked at the code. I've done a lot of client work and a lot of work on www.sonatribe.com in between then and now so i don't really have a clue where I left it! Looking at the selenium tests I can see that the site spins up and it tries to reach the admin/categories/add URL - there's no link so I guess that's as good a place as any to start.
Without getting anal about testing the UI by precompiling it and blah I'm going to dive right in and add the link to the /admin page and get that step passing. It's as simple as:

One step further! Now the selenium test moves the browser to the admin/categories/add url but again there's no code to handle this. Again - all URLs start with a route so I look at the routing test and find there's no route test defined for this URL so I go ahead and add that:

This test obviously fails as there's no route actually defined - tests don't make code durrrrrr... I'll add that to the BrightonSausageCoEcomms.RouteConfig class:

Chuck in a cheeky little view in /Views/Admin/CategoriesAdd.cshtml with it's corresponding empty view method returning View() in the controler (test first obviously) and we're done! Spinning up the selenium test again lets us go another step further. Next line in the feature is: "And I enter "category1" into the "CategoryName" textbox". This is again going to fail because we have no CategoryName text box so I'll add that

Spinning up the selenium tests again I can see the browser is taken to the add category page and the CategoryName is populated! It's almost too much to handle... There's also a checkbox to add so I'll do the same rinse and repeat until I get past that line. Once that's all good I fumble on the next big line: "When I click the "Save" button". I'll add the save button then shall I?! Now the problem isn't really here - now that I added it - that line goes through ok... The problem is the next line: "Then I should see "category1" show up in the list of existing categories"
That's a pretty big statement! First up I added the submit button but since there's no form nothing happens. Even if there were it's not going anywhere as the controller has no clue what to do with such a request. Even if it did - what the hell does it do to make that sort of magic actually happen? Curiouser and curiouser... I'll add the form tag and see wha gwarn... The code now looks like:

Perfecto.. except an HTTP POST to that url goes nowhere - although we have the routing set up we only know how to accept the HTTP GET verb - lets make the controller able to deal with the POST verb by adding another action.
This action is going to be slightly different because we're going to be recieving data from the forms post. The name of the category and the enabled value will be sent to the action in the form of a view model. A view model is just that - it's the model specific to the view. A model is just a data structure. So our test will need to pass this viewmodel into the action. And the action will need to be decorated with the HttpPost attribute and it will return a redirect to the admin/categories view but that's pretty much all we know for now. Here's the test (I'm leaving out the attribute part for now as I'm on the train and the reception is awful!):

Notice we now have a CategoriesAddViewModel - I used resharper to drive the definition of that so by the time I had the test written I had the overload for Categories(CategoriesAddViewModel) created and the CategoriesAddViewModel inside the mvc project - all be it a blank class - it's enough to get this test passing for now.
That's it for now - I'm done for today. Not a lot achieved but I'm fried!!