Hacking Thought Blog

InfoQ: Events Are Not Just for Notifications

2011-12-06T17:23:00.000-05:00

InfoQ: Events Are Not Just for Notifications:

It is nice to see a fantastic explanation of what I think of as Event Programming. I have blog about a way to implement a simple Javascript dispatcher and implementing a cross platform event system. And have been chomping at the bit to implement a fully audited application that stores all the events in one of these new document databases (MongoDB, Riak). To be able to mark any user session with an error (or even allow them to report an error) and then to "replay" that users session would be pretty hot. The concept is difficult but implementation is so much easier especially testing. Now to just evangelize until it is considered a "Best Practice".

Presentation on Cloud Deployment & Mongodb @ Charlotte Python Group

2011-10-20T08:56:00.002-04:00

A few weeks back I have a presentation on using Fabric to deploy Python WSGI application in the cloud. The slides are:https://docs.google.com/present/view?id=dg3v9j47_210cxt77g7h
The code from the demo is here: https://github.com/lateefj/RTC-example
In the presentation I displayed how taking down one node in the cluster would not interrupt the other two nodes. Using fabric this was not only feasible but relatively easy once you understand the architecture. I also review the tools that I used like supervisor to restart WSGI (Flask), fabric for deployment and servers setup and finally mongodb for data storage.

I also quickly introduced mongodb: https://docs.google.com/present/view?id=dg3v9j47_212g2kvq2tv
The main point I was trying to make was that in early development RDBMS strict schema just get in the way of fast development. It is a great tool for prototyping. I use it anytime at the beginning of a project because I don't know the final data model. Also my past experience is that every mature application that has an RDBMS is that the schema is totally crazy to work with. So why not start with a very flexible schema and give it some room to evolve as the application matures.

Startup Cloud Cluster on a Budget

2011-06-12T09:40:00.003-04:00

Recently I have been working on a project that needed a highly available service. Since I have been pitching “the three pillars of a service: robust, fast and accessible”. With all the marketing hype around cloud computing and choices from something like Google AppEngine that provides a very restricted environment to something like Amazon AWS which just provides virtual hardware. The problem is that creating a service from scratch but I can’t live with the restrictions of something like AppEngine or Heruku nor do I want to spend a ton of time setting up something like AWS and Cassandra. This is a two part blog on setting up a robust, fast and accessible service in a few hours and for a very affordable price.

This blog will cover the concepts and overview of the solution and the technologies used for the implementation. The second blog will cover the configuration and deployment.

3-legged Table

I have come to the belief that for any service to compete it has to be supported by a three legged table where the legs are Robust, Fast and Accessible. Without any one of these the table is only valuable to produce heat in the fireplace.

Robust

Web systems should not have “scheduled downtime”. I think that the convergence of two things have force us web developers to architect for 0 scheduled downtime. The first is to provide a 24/7 business platform for our users. The second is after launch the software is alive with continues updates until it gets turned off. Traditional business systems ran during business hours and then could be “maintained” from 5pm pacific time to 8am eastern time. There is a general move to a continuous updates of software where the more frequently the updates the more stable the system is and faster the technology can adjust to the business need (I am a fan).

Fast

Usage can be directly correlated to the performance of a service. In a competitive market slow services will not be able to compete. Premature optimization has been the road to ruin of many a project but the ability to quickly optimize the service after launch will decide the success of the product.

Accessible

The point of a service is for it to be access by users. One of the reasons I think that JSON became so popular is it simplicity. Again in competitive markets the barrier to entry approaches zero. This mean well document services that support 80% of the platforms (mobile, desktop ect) is a requirement to compete in the marketplace.

Tools

Linux

This is a given for they types of service that I build. I could see a specific service that was for M$ that required using libraries that only work with Windows however I don’t image easy would be a part of that setup. Specifically I use Ubuntu however the configuration files / deploy script could very quickly be changed to use any Linux / BSD distribution.

Fabric

Fabric is a Python tool for doing system administrative tasks. There are a lot of tools that do this most are pretty specific I find this tool provides the best balance between easy of use and flexibility. The problem I have with shell scripts is they tend to be great to start with but quickly find the limits when needing to do some string manipulation.

Nginx

This is my web server of choice at the moment. I don’t think this matters so much but it provides very simple and quick configuration which is key the the easy part of setting up a cloud. Ultimately most web service is going to run proxied by something like this that will do all the gzip compressing and static file serving ect.

MongoDB

Mongo is one of those new fangled database under the NoSQL (not only SQL). I think it provides the most general purpose and easiest to setup. The console is very easy to manage data. However for this exercise the critical part was the two line configuration change difference between master and slave. This made failover very simple. I think that redis would have also been as easy and the configuration probably as simple but I liked having some of mongo’s mixtures of traditional and modern features like having indexes and map/reduce support.

Architecture

In this case I am using a load balancer (http://www.rackspace.com/cloud/cloud_hosting_products/loadbalancers/technology/) which I am not exact sure the specifics but they suggest only need to configure one load balancer it is automagically high availability. We also assume that the client that is talking to mongo has the proper settings that it will automatically roll over to the slave if the master goes down. There is not panacea we still could have a data center failure (as happend recently with Amazon cloud) so we might want to locate a slave replica in another data center ect. However I wanted to maintain a balance between highly available, easy to maintain, fast to setup and low cost.

Summary

My goal to create a starter cluster or cloud infustructure that focused on highly available, setup in an hour, easy to maintain and most critically low cost solution. I have published a basically template code (https://bitbucket.org/lateefj/easy_cloud) which I will highlight the configuration in my next blog. This example setup that I have uses two virtual servers plus you would have to manually configure the load balancer (took me 2 minutes) but the base price is around 35$ a month which is mind blowingly low cost in time and money for what you get (YMMV this is the minimum it can cost with basically 0 bandwidth usage). I will be hosting all my projects soon on this type of setup soon so I can sleep better at night.https://bitbucket.org/lateefj/easy_cloud

Testing in Python with a bit of Django

2011-01-26T09:54:00.001-05:00

Gave a presentation last night (January 25, 2011) to the Charlotte Django and Python group. Base don the feedback I got from the last presentation I tried to change things up. Mainly by using only 4 slides (including title slide) and lots of code. Which I basically just went though writing a test on a django model.

Slides: https://docs.google.com/present/edit?id=0Af1-cNaYamXFZGczdjlqNDdfMTk4ZGN3dmZod24&hl=en
Code: https://bitbucket.org/lateefj/py_testing/overview (still have to fix views.py)

The best part was the discussion (as always). I would love to do a full day on this which would include setting up buildbot and selenium.

Erlang First Impressions

2011-01-24T09:07:00.000-05:00

I attended an Erlang training Saturday January 22, 2011 taught by Kevin Smith (twitter) of much Erlang fame. I first met him last year when he did an Riak presentation to the Charlotte Ruby group. Anyway he is an excellent instructor (maybe a couple to many emacs plugs ;) and it was an amazing class. It was so good I feel like my brain is a little hung over from the experience (or maybe the beer we had after). In an effort to try to medicate I figured I would write up my thoughts.

The good:

Concurrency #1 implementation detail
Runtime reload of code
Shell
Atoms
Pattern matching parameter over loading / conditional methods
Message passing as core part of language is easy and wonderful
Monitor / Subervisor to restart services that fail

The bad:

Strings (this is pretty bad since it is the most used datatype)
Contextual termination ",", ";", "." can all terminate a line / function (I almost like it but don't)
Syntax is 30 years old and has not aged well
Immutable types

Erlang make concurrency very easy in some ways by making a lot of other programming very difficult. Writing simple service oriented things like key/values stores or simple, reliable CRUD task it is perfect for. Where as for algorithms, string processing and other complex operations it is a poor solution for. Unfortunately I don't have many concurrent distributed problems where the difficult part is getting the concurrency right. However I now understand why databases like CouchDB and Riak are written in erlang. At the moment I am not writing a lot of simple services however I can see how it would be helpful in the future for simple services.

Shelving Java and OS X

2010-11-02T23:35:00.000-04:00

History.

The nature of the relation I have as a developer is changing with Mac the operating system that I use for development and Java which is currently the language I used second most (could have been #1 but lets not get ahead). I wrote my first Java back in 1998 where I locked myself in a basement to teach myself a programming languge. I started professionally using OS X back in 2003 or so. At the time I was developing J2EE deployed in a JBoss container. So pretty much I have been developing on Mac the last 7 years and Java for 12.

Why Java?

I taught myself Java in college for a couple reasons:

* The compiler was free

* Easier and faster development than C/C++

* Cross platform

All in all I am really happy it gave me a good 10 years of employment and I learned a lot more about software development than if I was focusing on keeping track of seg falts, memory and pointers. Sun slapped an @depricated on Java when it decided to never open source it. Sun and friends controlled the language basically killing the evolution of the language. Even though it did make Java very popular in the enterprise market for new products as well as replacing mainframes. As I saw the companies that control Java (mainly Sun) could send an army of typing monkeys to build their products on. My developer interests were for the language to continue to evolve so that I could maximize productivity and have fun. I think if Java was open sourced a couple years back, then it would meet more of my developers interests. On the plus side my experience with writing Java software means a lifetime of work even though the mindshare of developers is waving good bye (hopefully that is including me).

Why Mac?

* Unix, a decent terminal and all the wonderful developer tools

* Hardware support so plugin monitor and it works, or usb drive or projector

* Pretty does matter even as a developer I would rather work on something that is not hurting my eyes

The last couple of years with the Mac has been a lot of fun. Even when I decided to get a Apple laptop I always thought the move was somewhat temporary and that in a couple years the Linux desktop would catch up and I would switch back. OS X was a great transition not only because it solved a lot of the hardware issues like plugging in second monitor just works, a unix shell and the list of Windows issues (most of which still exist on Windows 7) to many to mention. Back in the early 2000’s Apple was providing an attractive option. Even though OS X has stagnated since Leopard was release for obvious reasons of the devices that start with an ‘i’ Apple has been changing its attitudes toward developers. My current running theory is that it has something to do with the fact that they have vendor locked a lot of developers via the iPhone. I do write some mobile apps but I do it on Android (it was not rocket science to predict a much larger market share to Android years ago) and I am mostly interested in web applications. I can’t blame Apple for not caring about the small group of developers who they do not make money off of via iTunes app or will not be making money off of. We ultimately are just a small niche user group with really loud mouths ;). Apple got the boost it needed to get back into the game with FreeBSD and focused on a good product and the succeeded. If Apple sees the future as selling consumer devices and making a cut on the software it is distributing to the consumer what can I say I am envious.

Java on Apple doesn’t matter.

Apple’s future is on devices and software righten in a platform they control which is currently Objective-C. The more they control the market place and the platform the more value they see they are adding for their customer and I guess they can rationalize charging for it? I am not high and mighty I purchased a Wing IDE license because it save me time and ultimately makes me more valuable. I am a believer paying for software (it is nice to eat). Even I think the small amount for the OS X platform was a good deal with all the software it came with. As a developer the more open the platform / language / environment is the more creative and possibilities there are for me to create value. That is why the web is so awesome. Creative freedom seem to have pass by both Java and Apple these days and that is why Java or Apple or Java on Apple doesn’t matter.

What operating system then?

Short answer: I am working on it.

Long Answer:

Windows being the master of the closed platform that is not an option. From what I have seen and played with on Ubuntu the platform seem solid and the hardware issues are no worse than on any of the other Linux distros. They have a really great package management system. The Unity UI looks pretty enough for my needs.

What about a programming language?

Programming language is less clear. I can use Python for most of the backend programming I need to do. Since web development is mainly IO bound with rare exception I can use Python. I use hand coded Javascript only for trivial things where I only need a couple lines. GWT has pretty much converted me into thinking that it is the way to go for web application so I need something that fast enough to be able to render optimized Javascript. As Java currently also supports Android it would be nice if I could learn one language to do all the frontend development.

Erlang?

Erlang sound fantastic for distributed server side coding. I don’t have much need for writing distributed server things like databases. Sometimes I need to hack a little Python web service and maybe a bit more but I really don’t have a use case unless I start hacking databases.

Foo on JVM?

JVM derived language. Really, I mean really? If any of those languages get ported to another VM that is _OPEN_ we can talk until then GOTO @depricated Java.

C# (mono)?

C# (mono) might not be a bad option for the near term. There is an Android port for mono already http://monodroid.net/ even if the tutorial talks about using it in visual studio this could be a contender. As far as GWT replacement http://projects.nikhilk.net/ScriptSharp S# or script# is the only thing that comes up when I google it. Will this be an open solution for the long term? Will Microsoft decide to do what Oracle and Apple are doing now? I don’t see this as a long term solution because of the risk of it ending up in the same place.

Golang?

Golang (Go) is the the language I like the most however it is in such a primitive stage. On the plus side the language was created by sponsored by Google and both Android and GWT are basically Google open source projects. As much as I would rather not look to another company academia doesn’t seem to be interested in creating programming language (or significantly contributing to open source projects) for developers like myself who are on the ground.

Filling the vacuums.

The Mac vacuum is a relatively small gap to fill since we (open source community) will probably just need to tweak an existing solution. The Java vacuum is much larger one. Java was successful as a backend tool where I think golang and I get the impression erlang could fill. On the frontend I have noticed that Wing IDE written in Python is so much faster than Eclipse or Netbeans (not that this is news). http://pyjs.org/ is a nice port of GWT to Python which means that Python could be a good replacement for both frontend and backend. I think there is still a need for a statically compiled GUI language.

Enlightenment though eventlet

2010-10-18T18:43:00.001-04:00

I did a presentation on Tornado Web to the local Python user group a couple weeks ago. There is one annoying thing about Tornado and one show stopper for most development. Tornado depends on pycurl for doing client request which is pretty strange since the core module can do client and server sockets. For any nonblocking to work with Tornado it has to have an implementation using the core Tornado library. So out goes all legacy clients that use Python sockets (very few use nonblocking sockets).

Blinded By Idealism
My first impression was that monkey patching the standard library was ugly. In reality this makes eventlet usable for legacy projects as well as very simple to write backward libraries that support both blocking and nonblocking. I was caught in the idealism trap where I wanted a clean paradigm to exist though the entire stack. Re-implementing all socket libraries is to large a barrier to change mindshare that nonblocking is a better programming paradigm. Especially if the new implementation is dependent on a non standard library like Tornado, Twisted ect.

Eventlet
The PyCon 2010 presentation is excellent overview. Bottom line eventlet provide a synchronous interface to nonblocking io or can just write nonblocking io using coroutines. Eventlet is mature, stable, fast and documented. This makes it very usable right now using existing libraries and the benefit of the improved paradigm/performance can be taken advantage of without much code change. Also as I have had more time to use Go and it concurrency model I am realizing the power of coroutines in any language.

In Practice
This all came about because I needed to write something that would traverse a site and validate the responses based on content type and regexes. Things like correct encoding to making sure the proper data is being displayed on a specific url. So a spider with validators that can be hooked in based on a regex or content type. Writing the test code I was able to create a eventlet server and have it output data so that I could test that the full stack of the spider code.

Yet again I learn though my mistakes.

Lawyers trade marked me!

2010-09-24T18:19:00.000-04:00

Well first time for everything I suppose. I built this widget which new name will is HT Walkability Score since the old name is trade marked but I can link to the website walkscore.com. Since this was just a hobby project and something I enjoyed doing and just kicking around to continue to improve my Android skills it is no big deal. I suppose they are about to build there own or something because other wise they wouldn't really care that much I am guessing. Hoping they will ring me up to work on it!

There was life before G1 and now there is life before Nexus One

2010-09-10T21:11:00.002-04:00

My Nexus One arrive today. Since I wrote weruat.com for the first Android Challenge of which I ended up just paying a chunk load of time. I got a G1 when they first came out and started to play around with some fun mobile applications. Last January I bought my wife a Nexus One (just love the idea of Freedom!). So I kind had an idea of what I was getting myself into.

I didn't think I took for granted the Android Navigation but I think I was not using my phone because it had gotten so slow. It reminds me how important speed is for any software. I was starting to think I would use my phone less now that my Google Voice number ring in gmail so I just answer most calls with my headset. Then I realized how much dead time I have waiting for a meeting to start or end as the case may be.

Just made me realize building a better mouse trap doesn't have to be innovative or more feature filled it could just be FASTER!

NonBlocking (Async) Tornado Presentation for CharPy

2010-08-31T21:38:00.000-04:00

NonBlocking (Async) Tornado Presentation for Charlotte Python Group

There are slides and example code. I had a lot of fun making and giving this presentation. Got some good feedback and would like to give it again so I can continue to improve it and my presentation skill. Please give me feedback to improve the presentation or let me know if any other groups would be interested in it.

goriak API tweaks

2010-08-12T07:44:00.000-04:00

My initial attempt at coding goriak was a mess (as expected). As I started adding read,write and links to the API I started to realize the failing of my modeling. The new API is based on a Document struct:

// Define a basic document to reduce API calls

type Document struct {

Bucket string

Key string

Value []byte

Links []*Link

JSONObject interface{}

}

This encapsulates most of the values that I was passing as parameters to the functions. Since Riak is a key/value it is pretty straight forward that a Put(doc) will store the for the string Key the bytes in Value. However if JSONObject != nil then it will marshal it. This works in reverse for Get(doc) if the JSONObject != nil then it will unmarshal.

// Definition of a link from one document to another

type Link struct {

Bucket string

Key string

Tag string

}

Link between documents is such a simple feature but is also very powerful. Using links can quickly turn a key/value database into a graph. Specifically with the linkwalking that will allow traversal of the graph. To support it I just had to parse and set the header "Link:". Both Put and Get now support storing or retrieving the list of Links associated with a document.

The API now:

PutWDWReturn(doc *Document, w, dw int, returnObject bool)

Put(doc *Document)

DeleteRW(bucketName, key string, rw int)

Delete(bucketName, key string)

GetR(doc *Document, r int)

Get(doc *Document)

The methods with r,w,dw and rw are all wrappers around the Riak REST API. The default is to use whatever the bucket is configured for node persistence control. Since the expectation is the exception that to override the bucket default configuration I provide the wrapper like Get that wraps GetR to save typing.

Next on my list to implement is LinkWalking!

Goriak uploaded code to the public

2010-07-27T00:07:00.000-04:00

Goriak a Go (golang) library for the Riak database.

Got enough time to learn enough Go to code up a basic key/value interface to Riak. It is still missing a couple of the REST API calls which I plan to implement over the next couple weeks. I figured it had enough work on it to share with the world and hopefully some one else can review my code and contribute to it. I originally looked to just contribute to another project but the only project I could find was riak.go and it didn't have as complete an implementation as I had. I was able to use a couple lines of riak.go and it really helped me implementing the Put function so I am really glad it is there. Also the author is really nice he is just going off to work on something else so I decided to create a new repository for the project.

The current API has three functions:

GetBucket - Retrieves a Bucket type which contains all kinds of information about the bucket

Get - Marshals the JSON that is returned by the request into a the interface that is passed in.

GetCAP - Same as get but support an option for the number of nodes that need to agree before returning

Put - Unmashals the type passed in and uploads it to the server

PutCAP - Same as Put but can specify the number of nodes that need to write the json

Delete - Removes the object from the cluster

DeleteCAP - Same as above but can specify number of nodes to delete it from

I still need to implement MapReduce, LinkWalk, Ping and ServerStatus. These should come over the next couple weeks as I get time to spend on them. MapReduce being the only non trivial one.

I did start to experiment with some concurrent access patterns. The first I call BackgroundWait which can make Get or Put which return a request id. The responses are stuffed in a channel until the code calls Wait(requestId). Instead of waiting for every single request it can do many requests at the same time and just wait when required. This reduces the number of choke points. Assuming web applications do something like this:

Check Authentication
Application configuration / setup that where user is not required
Based on user / permissions additional data is requested
Possible a second round of data based on application data
Response returned
Logging

As the use of key/value and document databases increase I think this will be more important to maximize backend performance. Riak ability to scale up both for reading and writing allows for this type of access pattern similarly to memcached. I don't know of any RDMBS (maybe VoltDB?) that could handle a large number of concurrent requests since they have very constrained IO or process (thread) resources. It is a lot of fun to think about ways that increase IO distribution could mean potential new access patterns for low latency data access.

Would be happy for any help, code review and contribution much appreciated.

Concurrently golang programming

2010-07-20T16:48:00.001-04:00

I would guess like most developer my first exposure to concurrency was in my parallel programming class in college. MPI and PVM where rather painful libraries to C/C++ experiences to cute my teeth on but it was a very exciting and rewarding experience. When I started writing Java thread programs there was an uneasy sense that parallel programming couldn't be this easy (after all the locks and synchronous calls there wasn't many operations running in parallel) but it sure was a lot of fun. In Python I found threads to be crazy simple but only useful for IO wait and loving miltiprocess programming. Parallel Python was the only thing that actually provided the same true parallelism as the original MPI and PVM but was also amazingly simple. Goroutines (coroutines) make me never want to see thread programming again. The killer combination of channels (kinda like a queue in the thread world) and coroutines are really enjoyable to code compared to threads and queues.

Currently I am working on creating a library for golang to talk to riak. If multiple request need to be made then they can be done using goroutines. Thus reducing round trip wait time. Depending on IO limitations this could increase the speed of web pages that need multiple resource for a specific url. For things like memecache and very fast datastores this would work great. The downside is that potential overloaded IO situations would be made much worse by the increase number of connections. This would probably limit to nonblocking IO datasources which most new datasources can handle a large number of connections.

First Impressions of golang

2010-06-30T18:19:00.000-04:00

Another language?
Yep golang or just go depending if you want to find anything from a search result. For me I write mainly in Python, Java, Javascript. Python as my GOTO (pun intended) language because I can write thing extremely quickly and sometimes even cleanly. Java for embedded platforms and GWT. Javascript the language of the web so that mean almost all developers use it (or should be). It is inevitable that a new language will replace older ones but I think there are two things that might have delayed the creation of golang and ultimately what I intend to use it for.
The first and largest is if Java continued to evolve not just its libraries but significant syntactical changes. Java from a completeness of implementation is great and the toolset is wonderful. Java requires way to much code volume. Code generators help however the code that is not generated has to access a majority of the generated code and no matter what there is a feeling of bloat. All the performance improvements in Java have yet to make development feel faster.
The second thing is Objective-C lack of popularity. Maybe because like Darwin it has a kinda of open source aspect to it but really the only significant implementation is proprietary implementation by Apple. Even though the IPhone has help its popularity it has reduced it openness and reduced its ability to become a general purpose language.
Golang fate could be similar because Google could be the Sun of Java (even more bad punning) or the Apple of Objective-C. Golang is not currently on that projection but I would not rule it out as a possibility.
Golang's Niche
I would like golang to replace everything I use Java and Objective-C. There is a lot of Python code that I think should be in a statically typed language but for various reasons mainly time I will note rewrite but golang would be great. Even though golang is very productive to develop in for a static language, on occasion my face slams into STATIC TYPE. For real stable API's that could have a real a large amount of dependent code I can see it will fit nicely. Until I get proficient with it I am going to try to keep working with it on simple projects.
Tasty Features
Terse Syntax
With exceptions of languages like Perl less typing is better. Golang shines in this manner. It is the most terse static typed language I have ever programmed in. Refreshing from a Java perspective and surprisingly close to Python.
Type System
I must say I really loved C when I first learned about stucts and header files to define functions. There is something really solid about the type system that I haven't felt since a C header file except more terse. Make me never want to see another Java interface in my life.
Channel
I am a bit of a server performance geek and well what can I say channels are just awesome. I have yet to write any code with channels but it is one of the things I am most excited about. I plan to use it when I write a golang driver for Riak. I would like to write a WSGI inspired web server in golang that I could access from Python and that is where I look forward to testing the mettle of channels.
Testing
When it come to code it is all about "hitting the ground testing". The development tool gotest makes this fast and easy.

Walkscore Widget with Ads!

2010-06-11T17:24:00.000-04:00

So my adventure into mobile development continues. The new version of Walkscore Widgets has ads!

The widget didn't change much. The app got a lot of features besides Ads (going to need to support my habit of writing code somehow). Progress bar and better messaging to explain why a lookup failed. The progress bar was difficult until I figured out that threads are really needed for good UI development. Over all I am pretty happy about it. As always I figured out more features I would like to add and I had a bug fix 24 hours after the release but in general I am much happier. Still could use some graphic design help but maybe the next app...

It is good to be a geek (me)!

2010-06-10T16:39:00.000-04:00

As I sit in a franchise restaurant waiting for an Android developer group meeting I realize that there are people envious of my coding skills. I can't believe I take for granted the decade I have spent learning Java, Python, C, SQL, Javascript, Linux, OS X, Android ect. Makes my current frustration trying to get an Android progress bar seem ridiculous. Going to Google I/O and spending time in coffee shops where everyone is using modern technology has made me complacent. I think I just saw someone using OUTLOOK! I mean who uses Outlook never mind exchange any more? Ok back to progress bar land with a lot less frustration and complacency.

GWT is great except for that pesky little Java problem.

2010-05-20T15:45:00.000-04:00

First Java is really the only language option in the current language options. During the Google I/O keynote GWT actually mentions a spring tool that generates boiler plate code. This is ultimately my problem with Java it is not expressive enough. With some exceptions (low level languages like C) this is a FAIL mode for any language. Just can't beat the GWT UX though from performance to maintainability.

Cloud application (AppEngine) or server (Rackspace, amazon)?

2010-04-16T09:21:00.002-04:00

Flexibility vs Productivity vs Creativity

Starting out building an application the thing we don't know is the path the application is going to go. Like the CAP theorom that states pick two of the three options (Constancy, Availability, Partition Tolerance) we have to pick two of Flexibility, Productivity and Creativity.

Take Hardware Off The Table

We have enough to worry about with all the laptops, monitors and cell phones but add rack of servers and we are really not happy. These type of resources are also very cost prohibitive especially for a startup or poorly funded project. Physical servers, switches, battery backups, facility (with special sprinkler system), security systems on physical access to equipment, power redundancy, network redundancy and capacity are all things that are enjoyable engineering tasks but a complete distraction from focusing on the project.

Hosted Server Overhead

Amazon user experience for managing servers is horrible so I am only going to talk about what I think the viable option is something like Rackspace. Come down to 3 things:

Setup - still need to manage configuration and setup of all the services that are running database, web servers ect. Even a small deployments can get complex quickly and require lots of resources.
Deployment - getting code actually onto the servers (this is harder than it sounds especial with web servers, databases, message queues ect)
Scaling - If your application is at all useful then people will want to use it and so you will need to figure out how to add more web servers, databases (RDBMS big piles of pain and expensive consultants licking there lips)

Hosted Application Constraints

AppEngine since that is the most popular and I have and am currently using I will point out some of the issues.

Development and testing can only use a very limited sample data set because of the way the development environment is.
Simple and easy crojob that taskqueue is not. (probably going to end up using cron hitting a url anyway)
It is very expensive when the traffic hits!
Limited on which software libraries can use!

How about Hybrid

What constraints that AppEngine places will limit creativity and flexibility however is very productive. Rackspace allows for maximizing flexibility and creativity on tools to solve the problem but puts a large burden on productivity. A hybrid approach might be exactly what the doctor ordered. Starting out with AppEngine and migrating to a solution like Rackspace as the needs arise to be more flexible and creative.

Everyone codes!

2010-04-07T09:06:00.001-04:00

At Pycon 2010 the last day Keynote (video here) was given by Antonio Rodriguez and I really dug his presentation. Since then I have been thinking about how to enable everyone in an organization to write code. It seem obvious that us coders / managers would want to enable everyone in the organization to have the tools to solve thier own problems.

So like how?
Well there are some technologies/methods that I think are converging that in combination could make it much easier to develop features, reports ect in.

Bespin
First thing we need is an editor and environment for all these new developers in our organizations. Reasons I think Bespin is the right solution for this:

Browser based: so don't have to set up client(s) on X machine
Extensible: Seem like it is pretty easy to extend to support what ever custom needs of setup there might be
Hosted: If it is running in the cloud and need more instances to support more users easy. Development environment can be managed centrally. Maybe even on demand?
Sharing: Code review, training and help can be giving though the tools provided.

Simple Data Model
Normalization is not the friend of a simple data model. This came to me because I designed an address book many years ago and recently I was trying to explain this to someone who could solve their own problems. So a table of user that has a contact where a contact has a list of email address. So to get a users email address need to write a join with 3 tables. Sure could create a view for this, but then when they find an issue how will they modify the record?
Document databases with map/reduce I think lends itself to creating simpler data models. Because:

Most computer users think of databases as documents. There is probably more data in spreadsheets than in RDBMS. So this will be comfortable for them.
Document database model tend to be less normalized and thus not having to traverse many documents to find things like email addresses. (Side note that Riak links are a darn good way to traverse documents)
Map/Reduce is simple and easy to understand conceptually. SQL is a mind bending to switch from a modern programming language like Javascript or Python to one from the 70's.

Better API's

There is a big difference when writing an API for internal development team vs for a third party. My memory is not very good so a couple months down the road I have to read way to much code to use my own API's. It has come to me because I was writing the original code just for core developers.

A question of language?

I would like to say that Python is the best choice. I know it would have the fastest learning curve but there are some downsides. I think right now Javascript has a leg up for this type of implementation.

Javascript

Upside:

Web is the unifying platform and Javascript is the language of that platform.
Map/Reduce javascript is supported in Mongo, CouchDB, Riak which are the document databases that come to mind for me.

Downside:

Is not an easy language to quickly learn how to use and play with. Would need to use something like Rhino.
Small community of backend development.

Python

Upside:

Very easy to learn and experiment with in terminal.
Has lots of web backend supported libraries.

Downside:

Map/Reduce with Python would need to be something like ZODB which has a lot of other pitfalls.
Would need to use pyjamas or something like it for front end. As I have been writing a lot of GWT (in Java) this is not a short learning curve.

Maybe I am underestimating these new coders and they can handle a mixture of Javascript and Python just fine.

Dreaming
So my pie in the sky for this solution would be a very scalable database like Riak with a Python map/reduce plus a WSGI Python and finally Javascript with jQuery. Python would cover most of the use cases (reporting, data management) and for the intern or advanced non developer could write Javascript to create a better end user experience. Now just need the time and money to create such a solution.

Riak Presentation at Charlotte Ruby meeting

2010-04-06T23:24:00.000-04:00

Saw an great introduction to Riak tonight. It is really an amazing product that is moving very fast. There where some surprises that I didn't realize that came along with the decentralized model. One of which is the inability to do averages. It is also very impressive how robust and potentially scalable Riak is. After the presentation I asked some of the more common questions and found out some things.

Pre and post commit hooks
Sounds like they are close to actually adding a pre and post commit hook. In RDBMS we would call these triggers. This came up when I was asking about making indexes easier to manage.

Distributed Indexes
I had read somewhere that they had been working on full index searches that worked just like all the other data stored in Riak. So I asked a bit more if that would also support regular indexes on that data so that performance could be greatly improved. The answer was it is in development but it will be a while before a production release is ready.

Very exciting stuff. I am hoping to continue learning Riak and using it with my Python and Javascript projects.

Out with Flextable and in with FlowPanel for the designer

2010-03-26T07:24:00.000-04:00

Thanks to a good friend and awesome web designer Wylie I have been cleaning up my GWT to make it more designer friendly. At first I was thinking maybe all this generated HTML is a really bad idea but I am starting to come around that it is much cleaner if the design is limited to just CSS changes. Wylie is great about pointing out poorly generated code:

When I was originally trying to lay things out I was using GWT layouts that where creating lots of tables. FlowPanel to the rescue that is now using div tags. One of the glaring missing layout types in GWT are list type. Since HTML supports lists it is strange that GWT does not support them natively. I created a wrapper for a list in about 15 minutes. For my current project I am glad I picked GWT and for a full blown web app still think it is better than the alternatives. The big test will be how easy it integrates into the backend.

Upgrade to Walkscore Widget

2010-03-10T18:33:00.001-05:00

As with most things a good does of FAIL makes things better. So why not make a list of all the things that I did wrong.

Ugly is not OK. Sure in development land ugly works but users don't use ugly crap with maybe a few exceptions like myspace.
Small is good. Especially on a mobile phone's home screen. Yep my widget was taking up twice the space it needed. I ended up having to create a full activity to do this but such is life.
Noise. Basically I had some additional text to help explain that they could not get walkscore if they had disabled the phones ability to locate them. Now that I added the feature to lookup any address they can have all phone location disabled and still use walkscore from the phone.

The only user comment I got also said they wanted more options. Unfortunately there are not many options to give. Either you have a walkscore for the current location or the address entered or you don't.

A couple features I would like to add are first an address suggestion. If the user enters an address in that has a list of suggestions then I need to let them choose the address currently I just pick the first one. Another feature I need to add is to track the number of times the score is retrieved because I am currently limited on how many request I can make to the walkscore servers a day. Mega thanks for the icons goes to the great Wylie Fisther!

GWT FlexTable is nice

2010-03-03T08:00:00.001-05:00

UPDATE: More recent post Out with Flextable and in with FlowPanel for the designer where I point out the downside of trying to design with FlexTable.
Compared to YUI's Datatable GWT FlexTable is so much easier to work with.
I spent a couple months developing in YUI2 which I generally had a great experience developing it in. At times I felt it was a bit slow but batteries are included and could get a prototype built fast! Also I love that it is just Javascript code so it seems very potable and easy for other developers. Since I have been writing Android apps and Java is one of my first loves (all those twice bitten) GWT has not been difficult for me. Most recently working with FlexTable I have learned where GWT has a great advantage over YUI2.

I was trying to find out how to hook up mouse events on a table of data when I ran into the exact example I needed on stackoverflow. This was a cleaner and easier implementation for mouse events in GWT than in YUI. I have found this to be generally true that GWT much easier to organize the code base in for large projects. I still feel simple user interfaces are better off with something like jQuery and that GWT is on the other end of the spectrum with really large complex user interfaces.

Rendering and entire page seems slow to me now. AJAX interfaces are raising the bar for us web developers. No longer can we just cache pages in memcache and expect them to be fast. Render time is like 70% of the user waiting for the page to load. That is the new true measurement of performance.

Code tools

2010-02-22T22:11:00.000-05:00

Yes that is a dual meaning. I was reading this blog about text editors. The truth is that I am an editor slut. I use different editors for the different things that I need them for. I am not just a one trick pony nor am I one language coder or a one editor kinda guy.
Eclipse is my Java editor. I would use something less bloated but the Java code I develop in have extremely awesome integrations into Eclipse. Android apps that I develop are so much easier to develop, test and deploy using eclipse. I can deploy straight to my phone plugged into the USB port which is just such a time saver. I have become a big fan of GWT because of its performance for the end user. Not to mention it works fast and great on mobile browsers. GWT is much faster to develop using Eclipse just be saving the file and refreshing the browser.
WingIDE I just started trying out since PyCon. I am feeling kind a hooked for hacking hard core in Python. I was using Komodo Edit which also had good Javascript support but it is way lacking in Python support. The downside being I am going to have to pay for WingIDE and I don't think I can afford the professional version but it is probably worth it since I spend like 6 hours a day hacking Python.
Emacs I use for basically commit messages but I am thinking I might use it entirely instead of TextMate which I purchase years back and it is nice but not all that for making my editing life easier.

Since I wear many hats as a developer entrepreneur then I need to use whatever is the most productive tool.

PyCon 2010 was awesome!

2010-02-22T21:44:00.000-05:00

This years PyCon 2010 was pretty much awesome! Really enjoyed the entire experience. My favorite talk was the last keynote Antonio Rodriguez the slides are online but I could not find the video yet. I enjoyed the async non-blocking talk by Peter A Portante. I took the time out to speak with him about it more and he pointed out that we need better access to kernel non-blocking to create a standard Python API that would be uber fast. I agree that is long term goal I am just focused on advocacy right now. Mark Ramm is one of my favorites folks out there. He is a real hands dirty funny kind of person. We discussed NoSQL which was mostly him teaching me stuff about his actual deployment and us agreeing on everything else. Right now I am leading NoSQL as time savings not so much as performance because I think people don't realize the are getting shafted by the performance issues with RDBMS. Anyway I tried to push Riak on Mark by sending him and email and pointing out a bug on his blog (I am sure that didn't help). I am planning on going again next year! Excited that I am going to go to Google IO also this year :)