Hacking Thought Blog

Notes from my first real Dart Web App

2013-05-01T11:19:00.003-04:00

I mostly build interfaces for IE. Well they have to be compatible for IE7+ so like most web developers I build them on Chrome and then fix the IE issues. Since Dart or dart2js only works with IE9+ I didn't have any projects that I could use for it. Finally I have a contact database I needed to build that I only needed to support HTML5 browsers! I figured I would give Dart a real shake down. I didn't want to build everything from scratch which I was luckily that DWT (Dart Web Toolkit) existed.

Dart

The language type system is a little Java ishy but that makes it really trivial to learn if you know Java. I find this mainly annoying with Generics but it is just one of those things that is here to stay I suppose. It is dynamic enough to do things like for(var x in [foo, bar, bat]) {x.show();}. After writing Python for almost a decade I have a hard time living without a level of flexibility. I like the Javascript bits since it make for first pass coding really fast and then it can be statically typed on future passes. I love being able to organize the code in files that make sense and not worry about some complex build process since dart2js will optimize it for me (dependency management is so much better done by machines).

DWT

I spent a couple years writing GWT code and it has its ups and downs of which I have written about extensively. I have also written a good mountain of Javascript and more recently Coffeescript. DWT give that rich API from GWT, I have found the static typing makes it easy to read libraries and consistant coding. Fast development combined with a great widget library and code that is not a complete mess to look at. Honestly I have been amazed at how easy it is to write DWT. I have found it faster to write DWT that it does GWT mainly because Dart is not the pig Java is. It has been relatively easy to style with the existing application using bootstrap.

var's, typedef's and dynamic's oh my!

Oh how I love var. Even though I write static typed code all the time I love being able to just type var when I am in the zone. Save from having to look up a type or to create it and loose the flow. Many times I am not sure exactly what the type is going to be so better to just have a place holder and then during a code cleanup pass fix the var types.
dynamic is the type that json.parse returns. I have found this extremely useful for two purposes. I like being able to configure using it like I do with the DataTable in dwt_lhj. JSON from the backend is another thing I often don't think needs a formally design data structure.
typedef is how to define a function. Say I have a function that take to int and I need to call it once the server responds. For example the Paginator that I wrote in the dwt_lhj tables code:
typedef SelectedPage(int page, int start);
When the user selects a different page then the paginator will call this function. Compile time failure if it doesn't work woot!

dwt_lhj

This are a handful of things I wrote that seem to be missing. I am sure eventually there will be better implementations however I figured I would share them in case someone else had simular needs.
UL and LI elements in HTML tags are missing in GWT as well in DWT. Since I use them a lot for navigation (bootstrap) and for making list of things! I created pretty simple wrappers. It is very self explanatory and easy to use with very little code.
Displaying tabular data is another thing that is currently not implemented in DWT. GWT calls them cell widgets. These are pretty large undertaking from a code perspective to implement for DWT. I have used Cell Widgets a lot in GWT. I find them very tedious to code. I have been using Datatables.net which is pretty nice Javascript library but I have found the API to be pretty complex for the functionality I am looking for (maybe that is just the docs). I was hoping to build something in the middle that I could quickly build but would have a lot of future flexibility that I could change the behavior in. So I built one. I also built a simple paginator that I use with the bootstrap paginator css classes. I am hot real happy with the code but it works and I look forward to improving it in the future.
Wrote a regex routing library because I couldn't get the existing https://github.com/dart-lang/route library to work. Based on the tickets they seem to exclude the features I needed. I decided to write a simple one that uses the DWT History package however I think it could quickly be improved and migrated to be a more general purpose library. When I get some better docs and examples up and someone finds it useful I will publish it in the pub.dartlang.org repository.

Mopping up

So far development in dart has been pretty fantastic. There are a couple bumps in the road with bugs in DWT or old examples to dart code online. I think in a year most of this will get sorted out. There seems to be some amount of effort going into coding Dart for the backend. In general I don't find much advantage to this. I find languages like Python and Go to be much better language for solving those types of problems. As a "unicorn" (I code both front end and backend) I find certain languages have strengths that appeal to the types of front end or backend problems (Dart craptastic support for concurrency IMO is one reason I wouldn't want to use it for backend coding). For building large code base of widgets Dart seems to be fitting the bill nicely. This app will hopefully go into production in June and I will follow up with another post then.

My first live Dart + Go web app (lhj.me)

2013-03-26T20:43:00.001-04:00

Last year I wrote a url shortener using Go and Riak. Trying to eat my own dog food since I wrote a REST API for Riak in Go. It used jQuery and just so I could get this up and running as quickly as possible. I put this up on lhj.me. Partly to test my libraries goriak and httphacks that I wrote in Go. It was a fun project! I was amazed how much I used goroutines in such a small app.I used this app just for personal use and experimentation. Basically just a dev instance and it crashed and died a week ago or so. So by by database and all.
This motivated me to bring her back up again instead of ec2 I thought I would try rackspace. Replace the javascript with Dart! Well technically javascript generated by Dart. I have been writing a lot of Coffeescript lately which I don't mind but I really want to find something with DEPENDENCY MANAGEMENT! So Dart it was to see what it is like to code in.
First I made some Golang code updates to make hosting the static content a bit easier in the future. Then it was writing Dart. I tried the Chromium that is distributed with Dart to make the coding faster. I wasn't able to get it working quickly. Since it was a simple ajax request I just compiled Dart to Javascript from the command line using Dart2js tool. The Dart code: http://lhj.me/dart/ushort.dart
I have already written about some of my complaints about Dart for a intro presentation I did last summer. The presentation and code are here. The presentation was more of an intro to the language as a general purpose language. Major complaints I had was about concurrency and some confusing OO features but mostly I found it an improvement on Javascript. Since I complain about... well ... everything it was a pretty short list of complaints.
A language that requires or is a pain to use without an IDE like Java I tend to shy away from. I mean it is hard enough to write code but to be fighting complex builds systems or tools that make coding extremely slow makes me want to run away. Vim (my editor of choice) has a bundle for syntax highlights for Dart which I installed. The indention is not so good but I assume it will get better as Dart becomes more popular. If I build anything more complex than a page of code I plan to use the IDE. The most surprising thing to me was when I went to compile it with type checking it failed to import the html package! ERROR: "Do not know how to load 'dart:html'". I guess this makes sense but I was pretty shocked thinking the types I was using would get checked. The type checks are a bit sad (dart --enable_type_checks) since it is a long command line option you pass to the dart vm. Takes digging for in the documentation to find (it will not show up if you type dart --help).
Since I know both Java and Javascript the syntax seems very natural. The API could use some help. I found the HttpRequest documentation much to be desired. It took me a lot of trial and error to figure out how to do a POST with JSON. They added some useful things like a HttpRequest.get static method. I was surprised to find out there was no HttpRequest.post static method. Giving an intro screencast on how to build a Dart web app in 10 minutes 'get' save a lot of code however for the web apps I build I use POST as much if not more than GET.
From languages like Python, Ruby, Go or Coffeescript there seems to be a lot of typing. Compared to Java on the flip side it is a very terse language. About the same as Javascript maybe a small amount more if I want to compile with types.

Coffeescript + requirejs keeping web app development bearable

2013-03-14T10:29:00.003-04:00

I have been building web applications since the turn of the century (and I have the grey hair in my beard to prove it). Back in my college days M$ was basically the only UI. Legacy Unix apps had X support and Linux seemed like it might be give no Windows GUI's a comeback until 2000 tech crash right when I graduated. Luckily my network programming and database programming allowed me to write web applications aka the only other interface game other than Windows. At the time Javascript was a dirty word. It was a gimmicky Netscape (pre Mozilla pre Firefox) feature that didn't work on IE (90% of the market). IE6 came out and Firefox which had good Javascript support but maintaining 2 code bases for each browser to handle the inconsistencies seemed expensive. More importantly I developed on Linux and IE was 90% of the platform so I would have to develop on gasp XP?
Sea change game when Gmail was released combined with some snazzy demos with sprinkling of Javascript and AJAX! Autocomplete was what converted me over. Autocomplete user experience was worth the extra work. I was dipping my toe in for a while from 2004 to 2008 when I realized the change in Javascript AJAX request was going to also mean a change in backend server. I blogged back in 2008 about this: "The new web seems to be lot of little request which means more small requests and it is going to require a lot more static resources." from then on I would start spending 50% of my development in Javascript GUI land.
I basically went though three phases before settling on what I am coding in now.

Phase 1

Generated Javascript by the backend was my first approach. All the apps I was building in the early days where based on web pages with some limited interactivity. Wanting to minimize browser computability by building backend widgets that would generate the HTML and Javascript so once a widget was made compatible with all the browsers it could be added to any web page with some level of confidence. This worked great for legacy apps that could be improved greatly with some simple show/hide or single ajax widget like autcomplete. This fell apart as the Javascript became the majority of the code being written.

Phase 2

The second generation was using jquery, YUI and other Javascript frameworks. Writing mountains of Javascript (10-100K) for a page. Selenium test made browser compatibility possible. The first advantage was being able to decouple the Javascript code base from the backend (Python/Java). Quickly found that Javascript lack of basic tooling like dependency management, packaging and performance optimization really stood out. Reinventing the wheels was tedious and ineffective.

Phase 3

GWT solved all of these problems! The biggest problem was learning Java which I had been coding since the 90's in, so no big deal right. Wrong! Swapped out one build mess for another. A lot of developer seem to love Maven however I found it as time consuming if not more so than Makefiles or ant. Development was slow! Java may be fast at number crunching but it has always been a resource hog. 8Gigs of ram dual core 64bit solid state drive. Sit back and wait for you changes to get rebuilt before you can view them. Another trade of writing software is slow but managing it and running it is quick.

Current Phase

Enter Requirejs and Coffeescript! All tool choices are compromises. Making code changes and seeing those changes immediately makes me more productive, keeps the momentum (critical in any project) and is more fun (look forward to return back to coding after a day off)! Coffeescript allows for immediate generation of Javascript so I don't have to wait for a compile step. It provides a sane for loops and other syntactical sugar greatly reducing lines of code. Requirejs make dependency management in Javascript bearable. As a backend developer I find manual dependency management feels like the dark ages (bring out your dead) of coding. Especially after the automagic of dependency management of a language like Go. I still waste a lot of time thinking about what file I should use to make sure the page loads faster and trying to manually consolidate code. There is a way to generate single Javascript files however after GWT all these techniques are pretty lame. Now that I embed my html templates in the Coffeescript source files and I render things in a way that lazy loads I am not loosing to much performance (famos last words about unmeasured performance).

Future Phase?

Dart seems to be a pretty good future option that solves some of the problems. Still has manual dependency management but it won't compile if that is wrong. Since I need to support 3-4 versions of IE back I am hoping I can fully test it out once IE12 comes out (not holding my breath on that release date). Until then I hope they fix the dependency management stuff using some type of url import statement. Until then I am just thankful for the great work of the Requirejs and Coffeescript teams.

Presentation / Discussion of Dart

2012-09-06T12:12:00.000-04:00

This blog post is about two months late since I gave the presentation / hosted a discussion about Dart over two months ago. This is the first talk I did which was all code. I figured it was an intro to a language so everything should be in that language. I organized it via a github repo: https://github.com/lateefj/dart-intro I had all the slides opened in a vim tab and just when though each one demoing the code and discussing it.

There where three types of developers at the meeting (technically it was a Python/Django group but other came). These where some observations I had from the different groups.
Python/Django - They generally write backend software that uses mostly static web pages. Or generators like Rails, Django admin stuff. By and large they enjoy backend programming (Python / Ruby) and find Javascript for some snazzy UI stuff annoying to write in. Dart is not very appealing to them because it is a more taxing syntax "Javaish".
Java/GWT - By and large they see that Google is moving its mindshare to Dart from GWT and came wondering what all the hoopla is about. As a long time Java developer it is frustrating that the Java community has turned into a monolanguage culture. Since there is a library somewhere to do just about anything in Java it make most Java programmers look at everything with blinder. Dart looks like Java, smells like Java and specifically for GWT folks it generate Javascript so what is the point?
Javascript - The latest rage (or has it already died out). Anyway Javascript developers don't really see why an alternative is necessary.

In general I think everyone in the discussion found it interesting even if most couldn't figure out why? My motivations was to learn mostly by discussing it with other developers. Bottom line for the types of large web applications that I have been developing I have tried man approaches (GWT, Coffeescript ect) but it all comes down to Javascript was design and built in the 90's and it is very difficult to over come its limitations. Will leave my thoughts about Dart for the next post....

Part 2 Dart vs Go vs Python (and PyPy) Performance

2012-04-11T17:40:00.000-04:00

UPDATE: In the comments Will pointed out I had a bug in the Python code so updated 22:00 EST sorry.
UPDATE 2: Anonymous posted a more comparable version of the Go (http://pastie.org/3772555) so I updated the numbers 04-12-2012 09:00 EST (sorry). Please let me know if there are any other bug fixes!

The previous post I wrote Surprising performance from Dart inspired me to do some more test runs with Dart, Go, Python and PyPy. For round two of some simple performance comparisons between Dart, Go, Python and PyPy I decided to do some basic file reading and processing. Since the vast majority of data that I process is JSON I decided to generate some random JSON data and store it into a file. Using each language read the entire file and for each line parse the JSON and aggregate it. Assuming I wrote the code right (big assumption) then I can get an idea which language is best for processing large sums of JSON.

The code basically is broken down into a module (package) that contains an Aggregator class (type) and the main file which reads from the data file. The hosted repository is on bitbucket.
Dart code:
I was able to code this using async style programming which in theory would maximize for IO performance if Dart supported async IO. I was a bit shaky at first with the terse syntax of the onLine callback but I am getting a bit ok with it. The duplicate use of StringInputStream seems a bit wordy, probably could have just said "var stream" instead of "StringInputStream stream" to increase readability (Go got this right with the := syntax). Otherwise the code is very straight forward.

Since Dart is design for the browser and it is a preview the IO would be very slow but I expected it to be able to process JSON very fast since that is a large part of what a modern web application does. This also means that the StringInputStream should be pretty fast since it will be the main source of data for the application. Go would probably outperform Python and not really sure about Python vs PyPy. I figured PyPy would use more memory but be faster for doing aggregation.

Runtime comparison:

This is only 10,000 JSON documents in a single file. I have no idea how it does it but Python just kills it. Not that surprising that it is 18x faster than Dart but 2x (roughly) faster than Go is pretty amazing I knew the Python 2.x IO was fast but I didn't realize it was this fast. I think there could be a faster way to process the JSON in Go it just didn't jump out at me to do it without using the Unamarshal.

Memory comparison:

I expected Go to have used the least memory but didn't expect it to be so tiny (2,076, after update Python is 4,692). Dart was unusable but that is expected in a preview release. PyPy is not much better I guess I need to send them a note and ask them what they think. Python again rock star here.

I have chosen to kick the tires at a pretty high level but I think it is a realistic situation since web apps these days are processing a lot of JSON. Dart faired as expected for a preview release. Python rock star status as usual. PyPy we will see what happens when I contact them about it. Finally Go was very disappointing how much runtime it took to process the JSON, however it did recover some dignity by it exceptional use of memory.

Surprising performance from Dart

2012-04-05T22:00:00.000-04:00

About every two years I try to learn a new programming language. I find the new language opens up a new set of applications and I can use it to improve existing applications. Dart this year mainly because I have been writing a lot of GWT (Java -> Javascript) and Javascript over the last couple years. They both have a lot going for them and against them. From what I have read about Dart it takes the best parts from both.
GWT Complaints:

Costly project setup and maintenance
Java development is slow
Language and library incompatibilities

Trying to setup and maintain a GWT application is a project in itself. From what I have used of Maven it seems to save little time if any. Then there are a lot of choices of design vs performance and cross browser support that are pretty loose / loose.
Java development is slow. Yep I said it and many say it isn't but it is. The JVM is extremely fast at doing a lot of things unfortunately it seems all development type things like IDE (GUI), compilation and building use excessive amounts of CPU and memory in the JVM. It is jaw dropping to see a hello world REST service take up the better part of 100MB (jar dependencies mostly from maven). Reloading a page using GWT (any servlet for that matter) requires a recompile step that is slow and bottom line a momentum killer.
GWT has support for calling into other Javascript libraries but it is not trivial. A lot of productivity is lost not being able to add a jquery plugin to a page in 10 seconds.

Where GWT kills it is deployment. The end product (assuming you can get it built) is a fully optimized cross platform wonderfulness that would take massive amounts of time to get equivalent Javascript app built.

Javascript Complaints:

Hard to manage large code bases
Browser incompatibilities

Trying to manage a large Javascript code base extremely painful. After a couple files it gets messy and slow. Keeping related code in separate files is really nice however it has an impact that those files have to get loaded into the browser unless you are using some build system. Then there is a lot of decorating of pages that only need X amount of code vs another page which needs a different set. There are ways but namespace seems like something that should be built into the language like almost every other programming language.

Debugging IE is what my generation of developers will be telling war stories over someday I suppose. However I feel like it is time to move on. Over a decade of doing it is time to innovate beyond these kind of shenanigans.

The great part about Javascript is it is extremely fast to develop. I don't even have to download the framework I can just add a script tag to the page and reload the page. This is really great for getting products built quickly or adding features quickly.

Dart:

Static type safe compilation as an option for those really large code bases (libraries anyone). However also the ability to just reload the browser for fast development. New language but with very simular syntax as Javascript that compiles to Javascript to support legacy browsers? Maybe the best of both worlds plus a bit more ontop?

With most programming languages I play with I start with Fibonacci number sequence as the first program. I am a fan of Fibonacci and it is a very simple algorithm to get the basics of a language. So lets take a look at the code:

Dart fib code:

fib(int n) {
  if (n == 1) {
    return 1;
  } else if (n == 0) {
    return 0;
  } else {
    return fib(n-1) + fib(n-2);
  }
}

  main() {
    var length = 35;
    for (int i = 0; i < length; i++) {
      print(fib(i));
    }
  }

I decided to benchmark compare Dart with Python, PyPy and Go run (go run fib.go) which compiles and runs the the program and then compiled it to see it fully optimized.

Then I compared memory utilization:

Fat disclaimer about the benchmark is that this is basically a number crunching thing that I ran a couple times to take these numbers for what they are (primitive). PyPy being way faster than Python is not a surprise at all. Lets take Python out because it is is skewing the chart.

The first really big surprise was the Dart was faster than PyPy. Another big surprise was how much memory PyPy was using like 5X (JIT maybe that is why Java is such a memory hog). Dart had amazing performance and memory utilization for a preview release of a language. The compiled version of Go had amazingly low memory utilization which lines up with my experience running a web app written in Go. This is very primitive experiment so I think I will try something next time with a little more complexity to see if CPU and memory hold up with a little bit of pounding. This is exciting news because I think Dart can be not only a front end web language but also a backend language. Gives me ideas!

Update: Part 2

Added Secondary Index support to goriak

2012-03-09T22:16:00.002-05:00

Have been pretty heads down on writing business software and was getting a little burnt. So to revive I hacked a little on the driver I have been playing with for Go (golang) to use Riak. There are basically 3 motivations other than I need to do a little fun hacking to recharge the batteries.

Go weekly is getting close to version 1 so they added packaging using the "go command" which I cover in a previous blog.
Riak add secondary index which is so much more developer friendly than before.
The types of problems I really enjoy solving are ones that Go and Riak would be good tools for.

Quick recap is that I had the basic support for key/value store with a hint to the type to structure to marshal it would do the conversion of struct to json and back. Get, Put and Delete basically was all that was implemented at the time.

Secondary Indexes provide a binary (string) and integer types for indexing. I implemented exact match and range queries. So method signatures:

SearchBin(bucketName, index, value string) ([]string, error)

SearchBinRange(bucketName, index string, start, end string) ([]string, error)

SearchInt(bucketName, index string, value int) ([]string, error)

SearchIntRange(bucketName, index string, start, end int) ([]string, error)

Now there is a complete example included with the source code:

Time to billed that kill web service! Source code is on bitbucket: https://bitbucket.org/lateefj/goriak/

Presentation on Error Handling with event sourcing

2012-03-02T15:34:00.001-05:00

Slides:https://docs.google.com/presentation/d/1Xs-Q1ZkiDW9wolk3vYqqyq9aBpdImv_xzYxlJYfKxzQ/edit#slide=id.p
In my ever growing desire to be able to automate capturing all errors and being able to validate fixing them I put together a demo and some simple code https://github.com/lateefj/cltpy_chat. Basically it captures all events and marks session that have a strack trace error in them so that they can be later found and rerun on the system. The idea being some day to be able to capture all events and then rerun them agains the code that has fixed them to prove that the fix will work. I am hoping to come out with some tools for Javascript and Python to be able to easily automate this.

golang go command & dependency management

2012-02-21T09:17:00.002-05:00

Finally decent dependency management!
The elephant in the room has always been that the code depends on specific versions of software to build and or run. Once you accept this putting dependencies in the source code is very obvious. A quick comparison of the languages I use most:

Javascript:
Bottom line javascript doesn't really do any dependency management that is left for the most part up to the developer to make sure that all the required packages are included on the page or using some framework. There seems to be some magic possible with coffeescript, GWT or other web frameworks that provide dependency management however are not interoperable. This is nasty as web apps get really large. I am looking to Dart in the future to solve this.

Java:
The only real game in town is Maven. The XML file configuration is horrendous IMO especially compared to Ant. The enormous amount of jars and memory needed to compile a simple web service is surprising. Finally the support for maven repository is far from universal.

Python:
Probably the best with a combination of virtualenvwrapper, pip and running pip freeze this is by far the fastest and easiest to use until go get command came along. There is still manual tracking of dependencies where even if it is simple "pip freeze > deps.txt".

Go:
This is covered pretty completely here (caution for the sensitive the blog is pretty graphic). Bottom line if I have added dependency I run "go get" and they are automatically downloaded from the import statements in my source code. I can run "go build" without any Makefile or build script. Once I has pushed up my library into the interwebs and another library is added to the go world.

This forces the master/default/trunk of the repo to be production. New code must be developed in a branch but isn't that how it should be anyway? The dependency (pom.xml, pip freeze, ect) configuration is commit into the source tree anyway (usually with some additional instructions). Lets hope other languages start supporting this!

Event Sourcing

2012-02-09T13:24:00.000-05:00

Event Sourcing: Bingo! I have started to experiment by building applications. Now that NoSQL and cloud storage solution allow for basically unlimited storage capacity this is very accesible option. Event Sourcing could be added an application that is using an RDBMS. Storing the current state in and RDBMS or any indexed data store (in my case MongoDB with some tuned indexes). One of the main goals I have it to track all the UI (Javascript) events so that any error I can replay everything that happens to debug it. I have done this before in a game but never hooked in the error replay so can't wait to see how it works.

InfoQ: Events Are Not Just for Notifications

2011-12-06T17:23:00.000-05:00

InfoQ: Events Are Not Just for Notifications:

It is nice to see a fantastic explanation of what I think of as Event Programming. I have blog about a way to implement a simple Javascript dispatcher and implementing a cross platform event system. And have been chomping at the bit to implement a fully audited application that stores all the events in one of these new document databases (MongoDB, Riak). To be able to mark any user session with an error (or even allow them to report an error) and then to "replay" that users session would be pretty hot. The concept is difficult but implementation is so much easier especially testing. Now to just evangelize until it is considered a "Best Practice".

Presentation on Cloud Deployment & Mongodb @ Charlotte Python Group

2011-10-20T08:56:00.002-04:00

A few weeks back I have a presentation on using Fabric to deploy Python WSGI application in the cloud. The slides are:https://docs.google.com/present/view?id=dg3v9j47_210cxt77g7h
The code from the demo is here: https://github.com/lateefj/RTC-example
In the presentation I displayed how taking down one node in the cluster would not interrupt the other two nodes. Using fabric this was not only feasible but relatively easy once you understand the architecture. I also review the tools that I used like supervisor to restart WSGI (Flask), fabric for deployment and servers setup and finally mongodb for data storage.

I also quickly introduced mongodb: https://docs.google.com/present/view?id=dg3v9j47_212g2kvq2tv
The main point I was trying to make was that in early development RDBMS strict schema just get in the way of fast development. It is a great tool for prototyping. I use it anytime at the beginning of a project because I don't know the final data model. Also my past experience is that every mature application that has an RDBMS is that the schema is totally crazy to work with. So why not start with a very flexible schema and give it some room to evolve as the application matures.

Startup Cloud Cluster on a Budget

2011-06-12T09:40:00.003-04:00

Recently I have been working on a project that needed a highly available service. Since I have been pitching “the three pillars of a service: robust, fast and accessible”. With all the marketing hype around cloud computing and choices from something like Google AppEngine that provides a very restricted environment to something like Amazon AWS which just provides virtual hardware. The problem is that creating a service from scratch but I can’t live with the restrictions of something like AppEngine or Heruku nor do I want to spend a ton of time setting up something like AWS and Cassandra. This is a two part blog on setting up a robust, fast and accessible service in a few hours and for a very affordable price.

This blog will cover the concepts and overview of the solution and the technologies used for the implementation. The second blog will cover the configuration and deployment.

3-legged Table

I have come to the belief that for any service to compete it has to be supported by a three legged table where the legs are Robust, Fast and Accessible. Without any one of these the table is only valuable to produce heat in the fireplace.

Robust

Web systems should not have “scheduled downtime”. I think that the convergence of two things have force us web developers to architect for 0 scheduled downtime. The first is to provide a 24/7 business platform for our users. The second is after launch the software is alive with continues updates until it gets turned off. Traditional business systems ran during business hours and then could be “maintained” from 5pm pacific time to 8am eastern time. There is a general move to a continuous updates of software where the more frequently the updates the more stable the system is and faster the technology can adjust to the business need (I am a fan).

Fast

Usage can be directly correlated to the performance of a service. In a competitive market slow services will not be able to compete. Premature optimization has been the road to ruin of many a project but the ability to quickly optimize the service after launch will decide the success of the product.

Accessible

The point of a service is for it to be access by users. One of the reasons I think that JSON became so popular is it simplicity. Again in competitive markets the barrier to entry approaches zero. This mean well document services that support 80% of the platforms (mobile, desktop ect) is a requirement to compete in the marketplace.

Tools

Linux

This is a given for they types of service that I build. I could see a specific service that was for M$ that required using libraries that only work with Windows however I don’t image easy would be a part of that setup. Specifically I use Ubuntu however the configuration files / deploy script could very quickly be changed to use any Linux / BSD distribution.

Fabric

Fabric is a Python tool for doing system administrative tasks. There are a lot of tools that do this most are pretty specific I find this tool provides the best balance between easy of use and flexibility. The problem I have with shell scripts is they tend to be great to start with but quickly find the limits when needing to do some string manipulation.

Nginx

This is my web server of choice at the moment. I don’t think this matters so much but it provides very simple and quick configuration which is key the the easy part of setting up a cloud. Ultimately most web service is going to run proxied by something like this that will do all the gzip compressing and static file serving ect.

MongoDB

Mongo is one of those new fangled database under the NoSQL (not only SQL). I think it provides the most general purpose and easiest to setup. The console is very easy to manage data. However for this exercise the critical part was the two line configuration change difference between master and slave. This made failover very simple. I think that redis would have also been as easy and the configuration probably as simple but I liked having some of mongo’s mixtures of traditional and modern features like having indexes and map/reduce support.

Architecture

In this case I am using a load balancer (http://www.rackspace.com/cloud/cloud_hosting_products/loadbalancers/technology/) which I am not exact sure the specifics but they suggest only need to configure one load balancer it is automagically high availability. We also assume that the client that is talking to mongo has the proper settings that it will automatically roll over to the slave if the master goes down. There is not panacea we still could have a data center failure (as happend recently with Amazon cloud) so we might want to locate a slave replica in another data center ect. However I wanted to maintain a balance between highly available, easy to maintain, fast to setup and low cost.

Summary

My goal to create a starter cluster or cloud infustructure that focused on highly available, setup in an hour, easy to maintain and most critically low cost solution. I have published a basically template code (https://bitbucket.org/lateefj/easy_cloud) which I will highlight the configuration in my next blog. This example setup that I have uses two virtual servers plus you would have to manually configure the load balancer (took me 2 minutes) but the base price is around 35$ a month which is mind blowingly low cost in time and money for what you get (YMMV this is the minimum it can cost with basically 0 bandwidth usage). I will be hosting all my projects soon on this type of setup soon so I can sleep better at night.https://bitbucket.org/lateefj/easy_cloud

Testing in Python with a bit of Django

2011-01-26T09:54:00.001-05:00

Gave a presentation last night (January 25, 2011) to the Charlotte Django and Python group. Base don the feedback I got from the last presentation I tried to change things up. Mainly by using only 4 slides (including title slide) and lots of code. Which I basically just went though writing a test on a django model.

Slides: https://docs.google.com/present/edit?id=0Af1-cNaYamXFZGczdjlqNDdfMTk4ZGN3dmZod24&hl=en
Code: https://bitbucket.org/lateefj/py_testing/overview (still have to fix views.py)

The best part was the discussion (as always). I would love to do a full day on this which would include setting up buildbot and selenium.

Erlang First Impressions

2011-01-24T09:07:00.000-05:00

I attended an Erlang training Saturday January 22, 2011 taught by Kevin Smith (twitter) of much Erlang fame. I first met him last year when he did an Riak presentation to the Charlotte Ruby group. Anyway he is an excellent instructor (maybe a couple to many emacs plugs ;) and it was an amazing class. It was so good I feel like my brain is a little hung over from the experience (or maybe the beer we had after). In an effort to try to medicate I figured I would write up my thoughts.

The good:

Concurrency #1 implementation detail
Runtime reload of code
Shell
Atoms
Pattern matching parameter over loading / conditional methods
Message passing as core part of language is easy and wonderful
Monitor / Subervisor to restart services that fail

The bad:

Strings (this is pretty bad since it is the most used datatype)
Contextual termination ",", ";", "." can all terminate a line / function (I almost like it but don't)
Syntax is 30 years old and has not aged well
Immutable types

Erlang make concurrency very easy in some ways by making a lot of other programming very difficult. Writing simple service oriented things like key/values stores or simple, reliable CRUD task it is perfect for. Where as for algorithms, string processing and other complex operations it is a poor solution for. Unfortunately I don't have many concurrent distributed problems where the difficult part is getting the concurrency right. However I now understand why databases like CouchDB and Riak are written in erlang. At the moment I am not writing a lot of simple services however I can see how it would be helpful in the future for simple services.

Shelving Java and OS X

2010-11-02T23:35:00.000-04:00

History.

The nature of the relation I have as a developer is changing with Mac the operating system that I use for development and Java which is currently the language I used second most (could have been #1 but lets not get ahead). I wrote my first Java back in 1998 where I locked myself in a basement to teach myself a programming languge. I started professionally using OS X back in 2003 or so. At the time I was developing J2EE deployed in a JBoss container. So pretty much I have been developing on Mac the last 7 years and Java for 12.

Why Java?

I taught myself Java in college for a couple reasons:

* The compiler was free

* Easier and faster development than C/C++

* Cross platform

All in all I am really happy it gave me a good 10 years of employment and I learned a lot more about software development than if I was focusing on keeping track of seg falts, memory and pointers. Sun slapped an @depricated on Java when it decided to never open source it. Sun and friends controlled the language basically killing the evolution of the language. Even though it did make Java very popular in the enterprise market for new products as well as replacing mainframes. As I saw the companies that control Java (mainly Sun) could send an army of typing monkeys to build their products on. My developer interests were for the language to continue to evolve so that I could maximize productivity and have fun. I think if Java was open sourced a couple years back, then it would meet more of my developers interests. On the plus side my experience with writing Java software means a lifetime of work even though the mindshare of developers is waving good bye (hopefully that is including me).

Why Mac?

* Unix, a decent terminal and all the wonderful developer tools

* Hardware support so plugin monitor and it works, or usb drive or projector

* Pretty does matter even as a developer I would rather work on something that is not hurting my eyes

The last couple of years with the Mac has been a lot of fun. Even when I decided to get a Apple laptop I always thought the move was somewhat temporary and that in a couple years the Linux desktop would catch up and I would switch back. OS X was a great transition not only because it solved a lot of the hardware issues like plugging in second monitor just works, a unix shell and the list of Windows issues (most of which still exist on Windows 7) to many to mention. Back in the early 2000’s Apple was providing an attractive option. Even though OS X has stagnated since Leopard was release for obvious reasons of the devices that start with an ‘i’ Apple has been changing its attitudes toward developers. My current running theory is that it has something to do with the fact that they have vendor locked a lot of developers via the iPhone. I do write some mobile apps but I do it on Android (it was not rocket science to predict a much larger market share to Android years ago) and I am mostly interested in web applications. I can’t blame Apple for not caring about the small group of developers who they do not make money off of via iTunes app or will not be making money off of. We ultimately are just a small niche user group with really loud mouths ;). Apple got the boost it needed to get back into the game with FreeBSD and focused on a good product and the succeeded. If Apple sees the future as selling consumer devices and making a cut on the software it is distributing to the consumer what can I say I am envious.

Java on Apple doesn’t matter.

Apple’s future is on devices and software righten in a platform they control which is currently Objective-C. The more they control the market place and the platform the more value they see they are adding for their customer and I guess they can rationalize charging for it? I am not high and mighty I purchased a Wing IDE license because it save me time and ultimately makes me more valuable. I am a believer paying for software (it is nice to eat). Even I think the small amount for the OS X platform was a good deal with all the software it came with. As a developer the more open the platform / language / environment is the more creative and possibilities there are for me to create value. That is why the web is so awesome. Creative freedom seem to have pass by both Java and Apple these days and that is why Java or Apple or Java on Apple doesn’t matter.

What operating system then?

Short answer: I am working on it.

Long Answer:

Windows being the master of the closed platform that is not an option. From what I have seen and played with on Ubuntu the platform seem solid and the hardware issues are no worse than on any of the other Linux distros. They have a really great package management system. The Unity UI looks pretty enough for my needs.

What about a programming language?

Programming language is less clear. I can use Python for most of the backend programming I need to do. Since web development is mainly IO bound with rare exception I can use Python. I use hand coded Javascript only for trivial things where I only need a couple lines. GWT has pretty much converted me into thinking that it is the way to go for web application so I need something that fast enough to be able to render optimized Javascript. As Java currently also supports Android it would be nice if I could learn one language to do all the frontend development.

Erlang?

Erlang sound fantastic for distributed server side coding. I don’t have much need for writing distributed server things like databases. Sometimes I need to hack a little Python web service and maybe a bit more but I really don’t have a use case unless I start hacking databases.

Foo on JVM?

JVM derived language. Really, I mean really? If any of those languages get ported to another VM that is _OPEN_ we can talk until then GOTO @depricated Java.

C# (mono)?

C# (mono) might not be a bad option for the near term. There is an Android port for mono already http://monodroid.net/ even if the tutorial talks about using it in visual studio this could be a contender. As far as GWT replacement http://projects.nikhilk.net/ScriptSharp S# or script# is the only thing that comes up when I google it. Will this be an open solution for the long term? Will Microsoft decide to do what Oracle and Apple are doing now? I don’t see this as a long term solution because of the risk of it ending up in the same place.

Golang?

Golang (Go) is the the language I like the most however it is in such a primitive stage. On the plus side the language was created by sponsored by Google and both Android and GWT are basically Google open source projects. As much as I would rather not look to another company academia doesn’t seem to be interested in creating programming language (or significantly contributing to open source projects) for developers like myself who are on the ground.

Filling the vacuums.

The Mac vacuum is a relatively small gap to fill since we (open source community) will probably just need to tweak an existing solution. The Java vacuum is much larger one. Java was successful as a backend tool where I think golang and I get the impression erlang could fill. On the frontend I have noticed that Wing IDE written in Python is so much faster than Eclipse or Netbeans (not that this is news). http://pyjs.org/ is a nice port of GWT to Python which means that Python could be a good replacement for both frontend and backend. I think there is still a need for a statically compiled GUI language.

Enlightenment though eventlet

2010-10-18T18:43:00.001-04:00

I did a presentation on Tornado Web to the local Python user group a couple weeks ago. There is one annoying thing about Tornado and one show stopper for most development. Tornado depends on pycurl for doing client request which is pretty strange since the core module can do client and server sockets. For any nonblocking to work with Tornado it has to have an implementation using the core Tornado library. So out goes all legacy clients that use Python sockets (very few use nonblocking sockets).

Blinded By Idealism
My first impression was that monkey patching the standard library was ugly. In reality this makes eventlet usable for legacy projects as well as very simple to write backward libraries that support both blocking and nonblocking. I was caught in the idealism trap where I wanted a clean paradigm to exist though the entire stack. Re-implementing all socket libraries is to large a barrier to change mindshare that nonblocking is a better programming paradigm. Especially if the new implementation is dependent on a non standard library like Tornado, Twisted ect.

Eventlet
The PyCon 2010 presentation is excellent overview. Bottom line eventlet provide a synchronous interface to nonblocking io or can just write nonblocking io using coroutines. Eventlet is mature, stable, fast and documented. This makes it very usable right now using existing libraries and the benefit of the improved paradigm/performance can be taken advantage of without much code change. Also as I have had more time to use Go and it concurrency model I am realizing the power of coroutines in any language.

In Practice
This all came about because I needed to write something that would traverse a site and validate the responses based on content type and regexes. Things like correct encoding to making sure the proper data is being displayed on a specific url. So a spider with validators that can be hooked in based on a regex or content type. Writing the test code I was able to create a eventlet server and have it output data so that I could test that the full stack of the spider code.

Yet again I learn though my mistakes.

Lawyers trade marked me!

2010-09-24T18:19:00.000-04:00

Well first time for everything I suppose. I built this widget which new name will is HT Walkability Score since the old name is trade marked but I can link to the website walkscore.com. Since this was just a hobby project and something I enjoyed doing and just kicking around to continue to improve my Android skills it is no big deal. I suppose they are about to build there own or something because other wise they wouldn't really care that much I am guessing. Hoping they will ring me up to work on it!

There was life before G1 and now there is life before Nexus One

2010-09-10T21:11:00.002-04:00

My Nexus One arrive today. Since I wrote weruat.com for the first Android Challenge of which I ended up just paying a chunk load of time. I got a G1 when they first came out and started to play around with some fun mobile applications. Last January I bought my wife a Nexus One (just love the idea of Freedom!). So I kind had an idea of what I was getting myself into.

I didn't think I took for granted the Android Navigation but I think I was not using my phone because it had gotten so slow. It reminds me how important speed is for any software. I was starting to think I would use my phone less now that my Google Voice number ring in gmail so I just answer most calls with my headset. Then I realized how much dead time I have waiting for a meeting to start or end as the case may be.

Just made me realize building a better mouse trap doesn't have to be innovative or more feature filled it could just be FASTER!

NonBlocking (Async) Tornado Presentation for CharPy

2010-08-31T21:38:00.000-04:00

NonBlocking (Async) Tornado Presentation for Charlotte Python Group

There are slides and example code. I had a lot of fun making and giving this presentation. Got some good feedback and would like to give it again so I can continue to improve it and my presentation skill. Please give me feedback to improve the presentation or let me know if any other groups would be interested in it.

goriak API tweaks

2010-08-12T07:44:00.000-04:00

My initial attempt at coding goriak was a mess (as expected). As I started adding read,write and links to the API I started to realize the failing of my modeling. The new API is based on a Document struct:

// Define a basic document to reduce API calls

type Document struct {

Bucket string

Key string

Value []byte

Links []*Link

JSONObject interface{}

}

This encapsulates most of the values that I was passing as parameters to the functions. Since Riak is a key/value it is pretty straight forward that a Put(doc) will store the for the string Key the bytes in Value. However if JSONObject != nil then it will marshal it. This works in reverse for Get(doc) if the JSONObject != nil then it will unmarshal.

// Definition of a link from one document to another

type Link struct {

Bucket string

Key string

Tag string

}

Link between documents is such a simple feature but is also very powerful. Using links can quickly turn a key/value database into a graph. Specifically with the linkwalking that will allow traversal of the graph. To support it I just had to parse and set the header "Link:". Both Put and Get now support storing or retrieving the list of Links associated with a document.

The API now:

PutWDWReturn(doc *Document, w, dw int, returnObject bool)

Put(doc *Document)

DeleteRW(bucketName, key string, rw int)

Delete(bucketName, key string)

GetR(doc *Document, r int)

Get(doc *Document)

The methods with r,w,dw and rw are all wrappers around the Riak REST API. The default is to use whatever the bucket is configured for node persistence control. Since the expectation is the exception that to override the bucket default configuration I provide the wrapper like Get that wraps GetR to save typing.

Next on my list to implement is LinkWalking!

Goriak uploaded code to the public

2010-07-27T00:07:00.000-04:00

Goriak a Go (golang) library for the Riak database.

Got enough time to learn enough Go to code up a basic key/value interface to Riak. It is still missing a couple of the REST API calls which I plan to implement over the next couple weeks. I figured it had enough work on it to share with the world and hopefully some one else can review my code and contribute to it. I originally looked to just contribute to another project but the only project I could find was riak.go and it didn't have as complete an implementation as I had. I was able to use a couple lines of riak.go and it really helped me implementing the Put function so I am really glad it is there. Also the author is really nice he is just going off to work on something else so I decided to create a new repository for the project.

The current API has three functions:

GetBucket - Retrieves a Bucket type which contains all kinds of information about the bucket

Get - Marshals the JSON that is returned by the request into a the interface that is passed in.

GetCAP - Same as get but support an option for the number of nodes that need to agree before returning

Put - Unmashals the type passed in and uploads it to the server

PutCAP - Same as Put but can specify the number of nodes that need to write the json

Delete - Removes the object from the cluster

DeleteCAP - Same as above but can specify number of nodes to delete it from

I still need to implement MapReduce, LinkWalk, Ping and ServerStatus. These should come over the next couple weeks as I get time to spend on them. MapReduce being the only non trivial one.

I did start to experiment with some concurrent access patterns. The first I call BackgroundWait which can make Get or Put which return a request id. The responses are stuffed in a channel until the code calls Wait(requestId). Instead of waiting for every single request it can do many requests at the same time and just wait when required. This reduces the number of choke points. Assuming web applications do something like this:

Check Authentication
Application configuration / setup that where user is not required
Based on user / permissions additional data is requested
Possible a second round of data based on application data
Response returned
Logging

As the use of key/value and document databases increase I think this will be more important to maximize backend performance. Riak ability to scale up both for reading and writing allows for this type of access pattern similarly to memcached. I don't know of any RDMBS (maybe VoltDB?) that could handle a large number of concurrent requests since they have very constrained IO or process (thread) resources. It is a lot of fun to think about ways that increase IO distribution could mean potential new access patterns for low latency data access.

Would be happy for any help, code review and contribution much appreciated.

Concurrently golang programming

2010-07-20T16:48:00.001-04:00

I would guess like most developer my first exposure to concurrency was in my parallel programming class in college. MPI and PVM where rather painful libraries to C/C++ experiences to cute my teeth on but it was a very exciting and rewarding experience. When I started writing Java thread programs there was an uneasy sense that parallel programming couldn't be this easy (after all the locks and synchronous calls there wasn't many operations running in parallel) but it sure was a lot of fun. In Python I found threads to be crazy simple but only useful for IO wait and loving miltiprocess programming. Parallel Python was the only thing that actually provided the same true parallelism as the original MPI and PVM but was also amazingly simple. Goroutines (coroutines) make me never want to see thread programming again. The killer combination of channels (kinda like a queue in the thread world) and coroutines are really enjoyable to code compared to threads and queues.

Currently I am working on creating a library for golang to talk to riak. If multiple request need to be made then they can be done using goroutines. Thus reducing round trip wait time. Depending on IO limitations this could increase the speed of web pages that need multiple resource for a specific url. For things like memecache and very fast datastores this would work great. The downside is that potential overloaded IO situations would be made much worse by the increase number of connections. This would probably limit to nonblocking IO datasources which most new datasources can handle a large number of connections.

First Impressions of golang

2010-06-30T18:19:00.000-04:00

Another language?
Yep golang or just go depending if you want to find anything from a search result. For me I write mainly in Python, Java, Javascript. Python as my GOTO (pun intended) language because I can write thing extremely quickly and sometimes even cleanly. Java for embedded platforms and GWT. Javascript the language of the web so that mean almost all developers use it (or should be). It is inevitable that a new language will replace older ones but I think there are two things that might have delayed the creation of golang and ultimately what I intend to use it for.
The first and largest is if Java continued to evolve not just its libraries but significant syntactical changes. Java from a completeness of implementation is great and the toolset is wonderful. Java requires way to much code volume. Code generators help however the code that is not generated has to access a majority of the generated code and no matter what there is a feeling of bloat. All the performance improvements in Java have yet to make development feel faster.
The second thing is Objective-C lack of popularity. Maybe because like Darwin it has a kinda of open source aspect to it but really the only significant implementation is proprietary implementation by Apple. Even though the IPhone has help its popularity it has reduced it openness and reduced its ability to become a general purpose language.
Golang fate could be similar because Google could be the Sun of Java (even more bad punning) or the Apple of Objective-C. Golang is not currently on that projection but I would not rule it out as a possibility.
Golang's Niche
I would like golang to replace everything I use Java and Objective-C. There is a lot of Python code that I think should be in a statically typed language but for various reasons mainly time I will note rewrite but golang would be great. Even though golang is very productive to develop in for a static language, on occasion my face slams into STATIC TYPE. For real stable API's that could have a real a large amount of dependent code I can see it will fit nicely. Until I get proficient with it I am going to try to keep working with it on simple projects.
Tasty Features
Terse Syntax
With exceptions of languages like Perl less typing is better. Golang shines in this manner. It is the most terse static typed language I have ever programmed in. Refreshing from a Java perspective and surprisingly close to Python.
Type System
I must say I really loved C when I first learned about stucts and header files to define functions. There is something really solid about the type system that I haven't felt since a C header file except more terse. Make me never want to see another Java interface in my life.
Channel
I am a bit of a server performance geek and well what can I say channels are just awesome. I have yet to write any code with channels but it is one of the things I am most excited about. I plan to use it when I write a golang driver for Riak. I would like to write a WSGI inspired web server in golang that I could access from Python and that is where I look forward to testing the mettle of channels.
Testing
When it come to code it is all about "hitting the ground testing". The development tool gotest makes this fast and easy.

Walkscore Widget with Ads!

2010-06-11T17:24:00.000-04:00

So my adventure into mobile development continues. The new version of Walkscore Widgets has ads!

The widget didn't change much. The app got a lot of features besides Ads (going to need to support my habit of writing code somehow). Progress bar and better messaging to explain why a lookup failed. The progress bar was difficult until I figured out that threads are really needed for good UI development. Over all I am pretty happy about it. As always I figured out more features I would like to add and I had a bug fix 24 hours after the release but in general I am much happier. Still could use some graphic design help but maybe the next app...