Hacking Thought Blog

Goriak uploaded code to the public

2010-07-27T00:07:00.000-04:00

Goriak a Go (golang) library for the Riak database.

Got enough time to learn enough Go to code up a basic key/value interface to Riak. It is still missing a couple of the REST API calls which I plan to implement over the next couple weeks. I figured it had enough work on it to share with the world and hopefully some one else can review my code and contribute to it. I originally looked to just contribute to another project but the only project I could find was riak.go and it didn't have as complete an implementation as I had. I was able to use a couple lines of riak.go and it really helped me implementing the Put function so I am really glad it is there. Also the author is really nice he is just going off to work on something else so I decided to create a new repository for the project.

The current API has three functions:

GetBucket - Retrieves a Bucket type which contains all kinds of information about the bucket

Get - Marshals the JSON that is returned by the request into a the interface that is passed in.

GetCAP - Same as get but support an option for the number of nodes that need to agree before returning

Put - Unmashals the type passed in and uploads it to the server

PutCAP - Same as Put but can specify the number of nodes that need to write the json

Delete - Removes the object from the cluster

DeleteCAP - Same as above but can specify number of nodes to delete it from

I still need to implement MapReduce, LinkWalk, Ping and ServerStatus. These should come over the next couple weeks as I get time to spend on them. MapReduce being the only non trivial one.

I did start to experiment with some concurrent access patterns. The first I call BackgroundWait which can make Get or Put which return a request id. The responses are stuffed in a channel until the code calls Wait(requestId). Instead of waiting for every single request it can do many requests at the same time and just wait when required. This reduces the number of choke points. Assuming web applications do something like this:

Check Authentication
Application configuration / setup that where user is not required
Based on user / permissions additional data is requested
Possible a second round of data based on application data
Response returned
Logging

As the use of key/value and document databases increase I think this will be more important to maximize backend performance. Riak ability to scale up both for reading and writing allows for this type of access pattern similarly to memcached. I don't know of any RDMBS (maybe VoltDB?) that could handle a large number of concurrent requests since they have very constrained IO or process (thread) resources. It is a lot of fun to think about ways that increase IO distribution could mean potential new access patterns for low latency data access.

Would be happy for any help, code review and contribution much appreciated.

Concurrently golang programming

2010-07-20T16:48:00.000-04:00

I would guess like most developer my first exposure to concurrency was in my parallel programming class in college. MPI and PVM where rather painful libraries to C/C++ experiences to cute my teeth on but it was a very exciting and rewarding experience. When I started writing Java thread programs there was an uneasy sense that parallel programming couldn't be this easy (after all the locks and synchronous calls there wasn't many operations running in parallel) but it sure was a lot of fun. In Python I found threads to be crazy simple but only useful for IO wait and loving miltiprocess programming. Parallel Python was the only thing that actually provided the same true parallelism as the original MPI and PVM but was also amazingly simple. Goroutines (coroutines) make me never want to see thread programming again. The killer combination of channels (kinda like a queue in the thread world) and coroutines are really enjoyable to code compared to threads and queues.

Currently I am working on creating a library for golang to talk to riak. If multiple request need to be made then they can be done using goroutines. Thus reducing round trip wait time. Depending on IO limitations this could increase the speed of web pages that need multiple resource for a specific url. For things like memecache and very fast datastores this would work great. The downside is that potential overloaded IO situations would be made much worse by the increase number of connections. This would probably limit to nonblocking IO datasources which most new datasources can handle a large number of connections.

First Impressions of golang

2010-06-30T18:19:00.000-04:00

Another language?
Yep golang or just go depending if you want to find anything from a search result. For me I write mainly in Python, Java, Javascript. Python as my GOTO (pun intended) language because I can write thing extremely quickly and sometimes even cleanly. Java for embedded platforms and GWT. Javascript the language of the web so that mean almost all developers use it (or should be). It is inevitable that a new language will replace older ones but I think there are two things that might have delayed the creation of golang and ultimately what I intend to use it for.
The first and largest is if Java continued to evolve not just its libraries but significant syntactical changes. Java from a completeness of implementation is great and the toolset is wonderful. Java requires way to much code volume. Code generators help however the code that is not generated has to access a majority of the generated code and no matter what there is a feeling of bloat. All the performance improvements in Java have yet to make development feel faster.
The second thing is Objective-C lack of popularity. Maybe because like Darwin it has a kinda of open source aspect to it but really the only significant implementation is proprietary implementation by Apple. Even though the IPhone has help its popularity it has reduced it openness and reduced its ability to become a general purpose language.
Golang fate could be similar because Google could be the Sun of Java (even more bad punning) or the Apple of Objective-C. Golang is not currently on that projection but I would not rule it out as a possibility.
Golang's Niche
I would like golang to replace everything I use Java and Objective-C. There is a lot of Python code that I think should be in a statically typed language but for various reasons mainly time I will note rewrite but golang would be great. Even though golang is very productive to develop in for a static language, on occasion my face slams into STATIC TYPE. For real stable API's that could have a real a large amount of dependent code I can see it will fit nicely. Until I get proficient with it I am going to try to keep working with it on simple projects.
Tasty Features
Terse Syntax
With exceptions of languages like Perl less typing is better. Golang shines in this manner. It is the most terse static typed language I have ever programmed in. Refreshing from a Java perspective and surprisingly close to Python.
Type System
I must say I really loved C when I first learned about stucts and header files to define functions. There is something really solid about the type system that I haven't felt since a C header file except more terse. Make me never want to see another Java interface in my life.
Channel
I am a bit of a server performance geek and well what can I say channels are just awesome. I have yet to write any code with channels but it is one of the things I am most excited about. I plan to use it when I write a golang driver for Riak. I would like to write a WSGI inspired web server in golang that I could access from Python and that is where I look forward to testing the mettle of channels.
Testing
When it come to code it is all about "hitting the ground testing". The development tool gotest makes this fast and easy.

Walkscore Widget with Ads!

2010-06-11T17:24:00.000-04:00

So my adventure into mobile development continues. The new version of Walkscore Widgets has ads!

The widget didn't change much. The app got a lot of features besides Ads (going to need to support my habit of writing code somehow). Progress bar and better messaging to explain why a lookup failed. The progress bar was difficult until I figured out that threads are really needed for good UI development. Over all I am pretty happy about it. As always I figured out more features I would like to add and I had a bug fix 24 hours after the release but in general I am much happier. Still could use some graphic design help but maybe the next app...

It is good to be a geek (me)!

2010-06-10T16:39:00.000-04:00

As I sit in a franchise restaurant waiting for an Android developer group meeting I realize that there are people envious of my coding skills. I can't believe I take for granted the decade I have spent learning Java, Python, C, SQL, Javascript, Linux, OS X, Android ect. Makes my current frustration trying to get an Android progress bar seem ridiculous. Going to Google I/O and spending time in coffee shops where everyone is using modern technology has made me complacent. I think I just saw someone using OUTLOOK! I mean who uses Outlook never mind exchange any more? Ok back to progress bar land with a lot less frustration and complacency.

GWT is great except for that pesky little Java problem.

2010-05-20T15:45:00.000-04:00

First Java is really the only language option in the current language options. During the Google I/O keynote GWT actually mentions a spring tool that generates boiler plate code. This is ultimately my problem with Java it is not expressive enough. With some exceptions (low level languages like C) this is a FAIL mode for any language. Just can't beat the GWT UX though from performance to maintainability.

Cloud application (AppEngine) or server (Rackspace, amazon)?

2010-04-16T09:21:00.002-04:00

Flexibility vs Productivity vs Creativity

Starting out building an application the thing we don't know is the path the application is going to go. Like the CAP theorom that states pick two of the three options (Constancy, Availability, Partition Tolerance) we have to pick two of Flexibility, Productivity and Creativity.

Take Hardware Off The Table

We have enough to worry about with all the laptops, monitors and cell phones but add rack of servers and we are really not happy. These type of resources are also very cost prohibitive especially for a startup or poorly funded project. Physical servers, switches, battery backups, facility (with special sprinkler system), security systems on physical access to equipment, power redundancy, network redundancy and capacity are all things that are enjoyable engineering tasks but a complete distraction from focusing on the project.

Hosted Server Overhead

Amazon user experience for managing servers is horrible so I am only going to talk about what I think the viable option is something like Rackspace. Come down to 3 things:

Setup - still need to manage configuration and setup of all the services that are running database, web servers ect. Even a small deployments can get complex quickly and require lots of resources.
Deployment - getting code actually onto the servers (this is harder than it sounds especial with web servers, databases, message queues ect)
Scaling - If your application is at all useful then people will want to use it and so you will need to figure out how to add more web servers, databases (RDBMS big piles of pain and expensive consultants licking there lips)

Hosted Application Constraints

AppEngine since that is the most popular and I have and am currently using I will point out some of the issues.

Development and testing can only use a very limited sample data set because of the way the development environment is.
Simple and easy crojob that taskqueue is not. (probably going to end up using cron hitting a url anyway)
It is very expensive when the traffic hits!
Limited on which software libraries can use!

How about Hybrid

What constraints that AppEngine places will limit creativity and flexibility however is very productive. Rackspace allows for maximizing flexibility and creativity on tools to solve the problem but puts a large burden on productivity. A hybrid approach might be exactly what the doctor ordered. Starting out with AppEngine and migrating to a solution like Rackspace as the needs arise to be more flexible and creative.

Everyone codes!

2010-04-07T09:06:00.001-04:00

At Pycon 2010 the last day Keynote (video here) was given by Antonio Rodriguez and I really dug his presentation. Since then I have been thinking about how to enable everyone in an organization to write code. It seem obvious that us coders / managers would want to enable everyone in the organization to have the tools to solve thier own problems.

So like how?
Well there are some technologies/methods that I think are converging that in combination could make it much easier to develop features, reports ect in.

Bespin
First thing we need is an editor and environment for all these new developers in our organizations. Reasons I think Bespin is the right solution for this:

Browser based: so don't have to set up client(s) on X machine
Extensible: Seem like it is pretty easy to extend to support what ever custom needs of setup there might be
Hosted: If it is running in the cloud and need more instances to support more users easy. Development environment can be managed centrally. Maybe even on demand?
Sharing: Code review, training and help can be giving though the tools provided.

Simple Data Model
Normalization is not the friend of a simple data model. This came to me because I designed an address book many years ago and recently I was trying to explain this to someone who could solve their own problems. So a table of user that has a contact where a contact has a list of email address. So to get a users email address need to write a join with 3 tables. Sure could create a view for this, but then when they find an issue how will they modify the record?
Document databases with map/reduce I think lends itself to creating simpler data models. Because:

Most computer users think of databases as documents. There is probably more data in spreadsheets than in RDBMS. So this will be comfortable for them.
Document database model tend to be less normalized and thus not having to traverse many documents to find things like email addresses. (Side note that Riak links are a darn good way to traverse documents)
Map/Reduce is simple and easy to understand conceptually. SQL is a mind bending to switch from a modern programming language like Javascript or Python to one from the 70's.

Better API's

There is a big difference when writing an API for internal development team vs for a third party. My memory is not very good so a couple months down the road I have to read way to much code to use my own API's. It has come to me because I was writing the original code just for core developers.

A question of language?

I would like to say that Python is the best choice. I know it would have the fastest learning curve but there are some downsides. I think right now Javascript has a leg up for this type of implementation.

Javascript

Upside:

Web is the unifying platform and Javascript is the language of that platform.
Map/Reduce javascript is supported in Mongo, CouchDB, Riak which are the document databases that come to mind for me.

Downside:

Is not an easy language to quickly learn how to use and play with. Would need to use something like Rhino.
Small community of backend development.

Python

Upside:

Very easy to learn and experiment with in terminal.
Has lots of web backend supported libraries.

Downside:

Map/Reduce with Python would need to be something like ZODB which has a lot of other pitfalls.
Would need to use pyjamas or something like it for front end. As I have been writing a lot of GWT (in Java) this is not a short learning curve.

Maybe I am underestimating these new coders and they can handle a mixture of Javascript and Python just fine.

Dreaming
So my pie in the sky for this solution would be a very scalable database like Riak with a Python map/reduce plus a WSGI Python and finally Javascript with jQuery. Python would cover most of the use cases (reporting, data management) and for the intern or advanced non developer could write Javascript to create a better end user experience. Now just need the time and money to create such a solution.

Riak Presentation at Charlotte Ruby meeting

2010-04-06T23:24:00.000-04:00

Saw an great introduction to Riak tonight. It is really an amazing product that is moving very fast. There where some surprises that I didn't realize that came along with the decentralized model. One of which is the inability to do averages. It is also very impressive how robust and potentially scalable Riak is. After the presentation I asked some of the more common questions and found out some things.

Pre and post commit hooks
Sounds like they are close to actually adding a pre and post commit hook. In RDBMS we would call these triggers. This came up when I was asking about making indexes easier to manage.

Distributed Indexes
I had read somewhere that they had been working on full index searches that worked just like all the other data stored in Riak. So I asked a bit more if that would also support regular indexes on that data so that performance could be greatly improved. The answer was it is in development but it will be a while before a production release is ready.

Very exciting stuff. I am hoping to continue learning Riak and using it with my Python and Javascript projects.

Out with Flextable and in with FlowPanel for the designer

2010-03-26T07:24:00.000-04:00

Thanks to a good friend and awesome web designer Wylie I have been cleaning up my GWT to make it more designer friendly. At first I was thinking maybe all this generated HTML is a really bad idea but I am starting to come around that it is much cleaner if the design is limited to just CSS changes. Wylie is great about pointing out poorly generated code:

When I was originally trying to lay things out I was using GWT layouts that where creating lots of tables. FlowPanel to the rescue that is now using div tags. One of the glaring missing layout types in GWT are list type. Since HTML supports lists it is strange that GWT does not support them natively. I created a wrapper for a list in about 15 minutes. For my current project I am glad I picked GWT and for a full blown web app still think it is better than the alternatives. The big test will be how easy it integrates into the backend.

Upgrade to Walkscore Widget

2010-03-10T18:33:00.001-05:00

As with most things a good does of FAIL makes things better. So why not make a list of all the things that I did wrong.

Ugly is not OK. Sure in development land ugly works but users don't use ugly crap with maybe a few exceptions like myspace.
Small is good. Especially on a mobile phone's home screen. Yep my widget was taking up twice the space it needed. I ended up having to create a full activity to do this but such is life.
Noise. Basically I had some additional text to help explain that they could not get walkscore if they had disabled the phones ability to locate them. Now that I added the feature to lookup any address they can have all phone location disabled and still use walkscore from the phone.

The only user comment I got also said they wanted more options. Unfortunately there are not many options to give. Either you have a walkscore for the current location or the address entered or you don't.

A couple features I would like to add are first an address suggestion. If the user enters an address in that has a list of suggestions then I need to let them choose the address currently I just pick the first one. Another feature I need to add is to track the number of times the score is retrieved because I am currently limited on how many request I can make to the walkscore servers a day. Mega thanks for the icons goes to the great Wylie Fisther!

GWT FlexTable is nice

2010-03-03T08:00:00.001-05:00

UPDATE: More recent post Out with Flextable and in with FlowPanel for the designer where I point out the downside of trying to design with FlexTable.
Compared to YUI's Datatable GWT FlexTable is so much easier to work with.
I spent a couple months developing in YUI2 which I generally had a great experience developing it in. At times I felt it was a bit slow but batteries are included and could get a prototype built fast! Also I love that it is just Javascript code so it seems very potable and easy for other developers. Since I have been writing Android apps and Java is one of my first loves (all those twice bitten) GWT has not been difficult for me. Most recently working with FlexTable I have learned where GWT has a great advantage over YUI2.

I was trying to find out how to hook up mouse events on a table of data when I ran into the exact example I needed on stackoverflow. This was a cleaner and easier implementation for mouse events in GWT than in YUI. I have found this to be generally true that GWT much easier to organize the code base in for large projects. I still feel simple user interfaces are better off with something like jQuery and that GWT is on the other end of the spectrum with really large complex user interfaces.

Rendering and entire page seems slow to me now. AJAX interfaces are raising the bar for us web developers. No longer can we just cache pages in memcache and expect them to be fast. Render time is like 70% of the user waiting for the page to load. That is the new true measurement of performance.

Code tools

2010-02-22T22:11:00.000-05:00

Yes that is a dual meaning. I was reading this blog about text editors. The truth is that I am an editor slut. I use different editors for the different things that I need them for. I am not just a one trick pony nor am I one language coder or a one editor kinda guy.
Eclipse is my Java editor. I would use something less bloated but the Java code I develop in have extremely awesome integrations into Eclipse. Android apps that I develop are so much easier to develop, test and deploy using eclipse. I can deploy straight to my phone plugged into the USB port which is just such a time saver. I have become a big fan of GWT because of its performance for the end user. Not to mention it works fast and great on mobile browsers. GWT is much faster to develop using Eclipse just be saving the file and refreshing the browser.
WingIDE I just started trying out since PyCon. I am feeling kind a hooked for hacking hard core in Python. I was using Komodo Edit which also had good Javascript support but it is way lacking in Python support. The downside being I am going to have to pay for WingIDE and I don't think I can afford the professional version but it is probably worth it since I spend like 6 hours a day hacking Python.
Emacs I use for basically commit messages but I am thinking I might use it entirely instead of TextMate which I purchase years back and it is nice but not all that for making my editing life easier.

Since I wear many hats as a developer entrepreneur then I need to use whatever is the most productive tool.

PyCon 2010 was awesome!

2010-02-22T21:44:00.000-05:00

This years PyCon 2010 was pretty much awesome! Really enjoyed the entire experience. My favorite talk was the last keynote Antonio Rodriguez the slides are online but I could not find the video yet. I enjoyed the async non-blocking talk by Peter A Portante. I took the time out to speak with him about it more and he pointed out that we need better access to kernel non-blocking to create a standard Python API that would be uber fast. I agree that is long term goal I am just focused on advocacy right now. Mark Ramm is one of my favorites folks out there. He is a real hands dirty funny kind of person. We discussed NoSQL which was mostly him teaching me stuff about his actual deployment and us agreeing on everything else. Right now I am leading NoSQL as time savings not so much as performance because I think people don't realize the are getting shafted by the performance issues with RDBMS. Anyway I tried to push Riak on Mark by sending him and email and pointing out a bug on his blog (I am sure that didn't help). I am planning on going again next year! Excited that I am going to go to Google IO also this year :)

Walkscore Widget: my first Android Application in the Market

2010-02-15T08:20:00.000-05:00

Sunday February 14 my Valentine to the Android community was free software! One of my favorite sites is walkscore.com and they have ported it to the iPhone. Well I have an a G1 which runs Android. They do not have there entire API accessible just the score. So I create an Android widget instead of a full map application like they have on the iPhone.

Android Widgets are applications that can run the desktop of the phone. Popular ones are Facebook feed, Pandora and stuff like that. They are a bit challenging because they need to be optimized to minimize utilization because of battery issues and stuff like that.

There are basically two actionable items. The first is the green dot and the logo will get the phones current GPS location and update the score on the right hand side. The second is the score which if tapped will open to walkscore websites full page. The website page is a bit slow but it is functional.

The walkscore API was easy to use the tricky parts have been the widget lifecycle and making sure not to use to much battery. There seems to be bugs I am bumping into a lot over the last 18 hours but I have made a couple updates but that is somewhat expected.

During the time that the GPS and score are being retrieved then the display changes to a retrieving icons. This lets the user know that the application is working. For a while I was thinking about doing an autoupdate. First the battery usage is a bit excessive. Second this isn't that useful because the user will need to be looking at the phone and a majority of the time the user will not even have there phone on. Finally I have a limited quota to how many walking scores I can do in a day and the autoupdate one phone could consume almost all of them.

If you have an Android phone and can do me a favor and test it out that would be great just search for "walkscore" in the Market and it should come up. Hopefully I will eventually get access to the full API and can build a cool mapping layered one.

Off to work on my next app.

These are a few of my favorite things..

2010-02-12T09:03:00.000-05:00

GWT + Tornado Web + Riak = hackingthought.com

Why?

The high performance of frontend combined with high performance backend make for an awesome user experience.
Non-blocking web servers are just cool! Actual I have blogged about this a lot but it really comes down to efficient resource utilization and simplified programming model.
Scalable data storage system. This I have also blogged about how RDBMS are quickly flawed for scaling even a small volume of web traffic.
As these are my favorite tools that I call myself an "expert" (and expect to get paid to consult for them) I should put them together to see what the issues are.

GWT

I wrote up my opinions on javascript frameworks before. For a project where I am choosing the tools I am going to pick whatever creates the best user experience. As the web has provided a lot of platform independent development at the end of the day there are a lot of compromises we must make.

The current GWT compiles Java to Javascript (bytecode if you will) for 6 different browsers. The performance is noticeable not only at load time but using the application. The major down side I can see if there is a bug in the compiled version it could be very difficult to find.

Tornado Web

I wrote a non-blocking web server and I am not even using it (shame shame shame). The truth is I was looking to try and us a non-blocking (async) library for Python and I found it here. The code better utilizes the operating systems non-blocking apis like epoll on Linux and kqueue on BSD. The problem is that I quickly figured out the code base was just so much better than what I was had. The only real problem is that they didn't support some of the features and WSGI in a way that I would have liked. I will probably create a little project to do this but that is for another blog. I already have made some improvements in my clone. So not only do that have a simple non-blocking API for the servers side but they have a good one for using as a client (will talk about this later).

Riak

So I have covered how I think RDBMS that have any amount of traffic fail to provide constancy and force developers into writing lots of performance code. Any content management system more than likely is going to need to scale for large volumes of data and large volumes of traffic. Riak rules! They just release 0.8 which included a Javascript map/reduce plugin for writing map reduce. I actually didn't need to use the map/reduce but for the future features I plan on having I will soon. The main bottleneck I have is that there is no non-blocking Python API for Riak. I plan to use Tornado http non-blocking client to create a Riak client during PyCon ATL 2010 (If anyone can join me on a sprint let me know). This will allow this platform to scale up quickly.

In summary I am very excited about this stack. It really feels like a best of bread tools for creating a rich user experience that can scale for the masses (once non-blocking Riak client is fixed). Please kick the tires and report bugs hackingthought.com.

Android is not iPhone killer, the other 95% of the market should be scared though

2010-01-05T15:14:00.001-05:00

There is a lot of hype about Google phone release (Nexus One?) which sound like it is going to be an awesome phone. However I don't really see it as an IPhone killer. The other 2 billion cell phones on the planet should be worried though.

Phones?
Because there are more phones than computers and as the data networks grow we should see the services that we build for our applications will start to get heavily used by not just the web applications but also by mobile applications.

Android soon to be the 800 pound gorilla.
This year we will see 50 Android phones which will dominate the market place as far as diversity of hardware. I think within 18 months we will see a $50 Android phone with a contract and within 24 months $0 with a contract. This will be a major boon to application distribution and since Android is an open platform anyone individual or company can distribute there software on it. This low cost smart phone what is open for hardware and software evolution will allow for many killer apps.

What Palm & Blackberry of the world?
I read a rumor that Blackberry was going to get a decent browser in the summer maybe? Well it is a bit late to be playing catch up. Palm WebOS is great technical innovation. If Palm and Blackberry merged then they might have a competitive advantage with the user base and excellent technology. I don't think it would be to difficult to constrain WebOS to whatever corporate environments love to lock out there users. However I am not holding my breath.

But the iPhone is "cooler"
The iPhone was a game changer technically which is not unusual for Apple. Nor is it unusual for Apple not to be able to dominate the market with the exception of the iPod. Personal computer to the digital camera Apple has innovated a lot of products however have not been able to maintain market share. I think iPhone will continue to grow and mature but will not capture the extremely large market that exists for cell phones.

As I have been hacking with Android I have noticed that these little applications tend to communicate with lots of little requests. GPS, messaging ect. The wireless data networks are about to take a pounding and so are the web applications/services that these mobile platforms are using. The one thing is for sure is that things will be different than I expect.

Is RDBMS pain boil down to premature optimization?

2009-12-30T18:19:00.000-05:00

As I extract my self from stored procedure pondering how at the door step of 2010 we are still using such an inflexible, complex and surprising language to manage data? The stored procedure with its PERFORM syntax instead of SELECT when you are calling a stored procedure inside of a stored procedure and the stored procedure you are calling does not return anything. There are so many of these type of rules that 30 years ago would have felt natural. Today this is a waste of developer time. Postgresql provide many different languages but this sounds much better than it is in practice. What is really bothering me why is this so painful? Why have RDBMS not matured to make our tasks easier and why hasn't a better data storage technology come about?

In this blog I am going to explore some of the issue that I have seen with RDBMS's. All of them seem to come back to premature optimization.

Data consistency is all that matters!
This is a common argument why all data store must be ACID compliant. I have heard my co-workers laugh about how NoSQL databases store inconsistent data. This seem ironic in a way because I have read most of their code and writing a lot of code that uses ACID RDBMS and that data can easily be placed in an inconsistent state. This is for performance reasons that we do this. When I was writing EJB's and each data record locked for reading and writing this was a very consistent way to access data. The problem with this is no data can be read or writing in parallel. It is amazing how few users it takes to bring a web system to a halt that is truly consistent. In the web applications that I have written the make heavy use of caches like memcache which I am sure has created many inconstant user views of data. I can think of off the top of my head over my 10 years of coding web applications, only 3 instances where I have writing been involved or seen code that actually was consistent. It might just be me but I would guess most web app developers have had a similar experience.

Disk speed is slow?
Disk have only recently started to increase in speed with SSD coming out that are multiple times faster then the spindles. Has this been the reason we have lived with RDBMS for so long. We have not had fast enough disk to use something else? We have been able to add more disks and use raid to increase performance however with RDBMS this has come down to writing to two nodes. First writing to the master and it replicating to a minimum of one slave. But wait we can scale RDBMS simply but sharding! Except we have broken rule #1 ACID if we shard data. So if data access performance will ever exceed disk array or size exceeds a disk array then RDBMS.

Object are not relations but they store sooooo close until they don't.
As a developer this seem to be something that is constantly driving me crazy. The relational data model never exactly maps to objects of the applications so it is 75% of the time consistent between tables and objects. So we design the tables first and our objects end up looking like are schema until performance suffers really bad and then we end up changing the table schema and the objects don't make any sense. Only the most trivial applications applications does this work out but I don't tend to spend much time writing or maintaining trivial applications.

RDBMS Vendor + DBA all your problems are solved
I can agree that for specific problems specific venders have design good solutions for them however this is very rare . As for the DBA since I sell myself as a knowledgeable person I agree more knowledgeable people to help solve difficult problems is a very good approach however the problem is the data access is slow! No matter how many people you have there is only so fast a single node can write to a disk array. No matter what if the system grows though more users, more data or more features the system will get slower if it can not do parallel writes.

So maybe I am just "ranting about writing software" but I think it is time to start making the transition to better data storage systems. My future productivity depends on using data stores that get out of the way and let me use modern languages and concept to manage the data. For the situations that I need to have consistent data I will use ACID RDBMS or find a way to create consistent data in NoSQL data stores.

In love with Riak (NoSQL) design

2009-11-19T09:04:00.000-05:00

I have read the documentation on Riak a couple of times. It is hard to believe the system exists. It seems to solve 90% of the problems I have with developing web applications. After I spent some time writing map/reduce javascript functions with CouchDB I find myself in love with it. As much as it is a great database for doing specific types of document storage systems it had some side effects that I really didn't like.

Riak quick list of things that are amazing to me:

No single points of failure
Every node in the system is structurally equivalent (lots of cheep hardware)
Nodes can be added at any time
Scales to large cluster or small (developer box)
Map/Reduce

Right now the map reduce is only in Erlang and I think that is the biggest issue. I don't really want to learn erlang to do map reduce however it is better than SQL over relations. Building a real application with CouchDB made me find out some real issues it had. So that is my next step.

Frisky is merging into magnum

2009-10-29T08:19:00.000-04:00

I created a web server to experiment with some concepts about a WSGI web server. The other day read about magnum which is the exact same concepts but with a much cleaner implementations. I was planning on spending the holiday rewriting Frisky to something usable. It is nice to know that there are some like minded people out there. I will now be taking the features that I experimented and ended up being good and rewrite them and submit my patches to magnum.
Once I can get my existing software working on magnum I will sunset Frisky. Based on the conversations I have had on the magnum mailing list magnum will be faster and better implementation of the same concepts soon enough. The collaboration will do me good I am happy to get out of my cabin in the woods.

Picking Javascript Web Framework Poison (YUI and GWT)

2009-10-16T13:30:00.005-04:00

This is the second blog I wanted to write about javascript web framework. The first post I wrote covered what I called "lightweight" frameworks and this I plan to cover "heavyweight" frameworks.The heavyweight frameworks that I have been working with are YUI and GWT. The heavyweight frameworks longer learning curve and a problem that requires more complex UI.

YUI
For the last six months I have been working on a couple projects using YUI. The set of UI widgets that YUI supports are very impressive. YUI is very fast to develop in. One of the projects I am working on has tons of dialog boxes, data tables and other thick client features which YUI makes dead simple and quick to use. Yahoo using this for there main site has made it very easy to support multiple browsers. For me this is perfect for a project that has lots of features.

GWT
Recently I have started to learn Google Web Toolkit (GWT). The amazing part of GWT is that is is optimized produced per browser optimizations. Since you generally don't write any javascript just Java which is compiled into Javascript the code will be cross browser compatible. The largest downside is learning curve because if you don't know Java (first language I taught myself) it is going to be painful. However the UI widgets are nice and it is blazing fast. This is on the high end of the framework however the effort is worth it if you are dealing with high volume of users or you need as fast an application as possible.

To GWT or to YUI?
So it comes back to combination of the needs of the project. My guess is that most of the time either having the skill set of Java or getting developers to learn Java to use GWT will not be worth it. If the project is significant volume of users (IE browsers) because I think performance is alway a critical aspect to user experience then GWT is the way to go. Most application start small and don't have the resources to use GWT so I would think YUI is a better choice.

Tool for the Job (IMO):

jQuery - small simple quick application that does not need a lot of AJAX
Dojo - I don't think there is really a need for something between jQuery and YUI but there might be if a project developers already have experience with Dojo and the project is pretty small
YUI - A good size application and I think the majority of projects fall under. Probably the most useful framework in the group to start a project on.
GWT - end all be all (currently) I think the most ideal solution if the resources allow for it because the end user gets the maximum performance.

Picking Javascript Web Framework Poison (jQuery and Dojo)

2009-10-14T09:47:00.002-04:00

This is the first in a two part serious that I wanted to document my experiences building a web application with different frameworks. The first part I will talk about my experience with two of what I call "lightweight frameworks" and then the second blog I will discuss the two "heavyweight frameworks". I will not be covering any of the why use a framework in this or discussion about user experience however if you are not concerned about user experience and using a framework to support the X number of browsers out there then dust off you resume you are going to be looking for work soon.

Lightweight vs Heavyweight
All though all of the javascript frameworks I am going to write about you "can" do the same thing some make things easier than others. In fact I believe that there is a spectrum based on the user interface of the application I am trying to build or said better "the right tool for the job". Also some other consideration are prototyping and one offs.

jQuery
jQuery was one of the first javascript frameworks I used. Mainly because of how simple it is to use and how much I was trying to avoid writing javascript. My favorite feature with jQuery is how terse the syntax is. This is probably why I like Python so much over Java just the speed of getting simple easy things done. jQuery is great for doing all kinds of simple DOM traversal and manipulation. There are a lot of plugins for doing all kinds of other things however I have found the jQuery UI and a lot of the plugins much to be desired in ease of setup and functional (cross browser ect).
Dojo
Dojo was another framework that I followed and managed building a rather large embedded project with it. Mainly because at the time the support for UI windows and things was much better. The syntax is kinda bloated and it takes some time getting things setup and coding but once you have the compile process working well it was very stable and functioned extremely well.

Picking Poison
As I have already mentioned it depends on the project. I would choose jQuery for any project that is mainly content but needs some simple quick DOM manipulation and maybe use of a plugin or two. The full UI is usable and I built a prototype with it but there was more typing and cross platform issues than I wanted. Dojo is kinda the next step in the spectrum if you have a product you need just a small amount of UI features like the occasional dialog box or small amounts of embedded pieces. Dojo I found to be a bit slow at times even with all the compression tools they have. I generally don't use Dojo any more because I use jQuery for simple things and use YUI because ... well that is the next blog .

Adventures in Event Everywhere

2009-10-06T07:36:00.002-04:00

So yeah I am smoking the event system though and though. Javascript based on around event system that is directly sending and receiving events to and from the backend. Which is also sending and receiving events from/to the front end. Of which it just has a bunch of event handlers which then create more events that are sent back to the front end, handled by the backend or stuffed on a message bus....

STOP!!!!

Why would anyone in there right mind every do anything like this? One of the problems I am reminded about a lot is that large software gets complex. And trying to keep everything in my head at once doesn't work. I need to break things down. More over I need to have a clean way of slicing up the functionality. The second part of this is that modern applications have server side code and client side code of which they are usually two different languages if not more because of the many mobile platforms ect. So I also would like a clean way to communicate between languages. So both client and server side can agree a protocol of events and data that is going to be passed for those event. Then both client side and server side will write event handlers for all the communication code. This is easy to quickly write communication wrappers that just call function / methods that have registered for event(s).

Pros:

Simple (conceptually and implementation)
Cross platform
Design works well with both front end and backend development
Protocol is quick and easy to be agreed on by developers

Cons:

RESTful urls are bit strange they can be generated based on event names but won't be as clean
If you don't like event programming then you will hate this
Can be difficult to debug with many listeners to a single event

At the moment I am thinking about pushing this system into a repository on bitbucket.org since I have Javascript, Flash, Python and Java bindings for it. I think I even can create a C which would cover C, C++ and Objective-C (Iphone).

Integrated Beaker Cache with Frisky

2009-08-20T10:18:00.006-04:00

Last night I spent enjoying writing some performance code on Frisky. Beaker Cache is a very nice tool that provides an overview. Key features that I wanted to use it for are:

Dog piling (cache race condition)
Multiple cache storage systems
Cache regions (different cache configuration based on namespace)

Basically at the moment with the existing system the fastest I could get my laptop to serve just the string "hello" was around 500 req/sec. If I only run the code that does the WSGI setup and have the server serve the "hello" string is 3,000 req/sec which I think is the theoretical maximum (which is brain busting fast). With Beaker (with memcached) I was able to get around 1,100 req/sec which over 50% increase in performance. This is amazingly fast however still 1/3 of what it theoretically possible so for now this is OK I can move on to other features. I think one issue is that the dog piling requires some locking so not sure how much more I will be able to do using Beaker.

The next feature I wanted to get working was autoreload of python code. I decided that the best way to do this since I have processes is to just create new processes after files have been modified. Then I thought I would in development mode just set a limit to number of request a process can handle to 1. This would mean every request would be a new process and thus reload the python. I wanted to create a request limit on processes anyway.

What does Agile Development mean to you?

2009-07-29T15:37:00.007-04:00

A coworker asked me the other day "What does Agile Development mean to you?". I could not articulate an answer. After a bit of thinking I am going to try and articulate what I want it to mean.

Agile Development Productivity

All the software that I have built is to solve some problem or need. Of witch all problems and needs over time change. As time goes on our software tends to break down because its original problem has changed and it no longer solves that problem anymore. I added productivity because it is the entire reason I think we decide to use some method of madness for developing software. This encompasses some techniques that are used to try make writing software better, cheeper and faster. At this stage of the game some of the techniques work some of the times other I am still experimenting with.

Agile

Agile to me means that it has a level of dynamic flexibility. Agile is a spectrum from one end where there is no development to the other end which is infinite development. So a complete agile software would be flexible for all possibilities known or unknown. Each project needs to decide where it needs to be on the agile spectrum.

Productivity

Sometimes it seems trying to apply the techniques we get lost from solving the problem to just trying to make the software more agile. Thus I think it is important to quantify everything as how it saves time.

Tools

Iterative Milestones

Most problems are complex and tend to change during the initial development stage of a project. First keeping the milestones short moral and motivation seems to stay higher because at the beginning of the day developers are not looking at a seemingly endless list of tasks. Second these iterations are based on deliverable that could be internal reviews or external. This keeps the project from going to far off into tangents and features that are not part of the requirements. This also allow the users to change the requirements as they understand more how the solution to the problem is going to work. This saves time be getting to the changes done early before lots of code (time) is wasted on building feature that are not needed or that are going to change.

Testing the Planet

Save time now and in the future. There have been many things said on how this save time and I don't need to cover them. However it is key aspect of maintaining flexibility so that future changes to the software as required by the dynamic world that all problems exist in will not break all existing features.

At the moment this is all I have come up with that I feel defines an Agile Development Productivity process. As other tools and techniques are flushed out I hope to blog about them.