skim

a new chapter

2010-12-03T15:00:00-08:00

Today, I gave my final presentation on Smalltalk-80, and received great feedback from my fellow craftsmen regarding my apprenticeship challenges. The feedback was very positive and rewarding. In the end, Micah offered a Software Craftsman position at 8th Light! I immediately accepted and will be coming into the office next week as a craftsman. I'm super excited and look forward to continuing to challenge myself and build quality software. I'm also happy to be part of a team who cares about the craft and quality in the work they do.

Since my apprenticeship has officially ended, this will be my last blog post. I will continue to post over on my other blog.

a scenario where tests help facilitate compatibility and communication

2010-12-01T14:00:00-08:00

Short History Lesson

Let's start off with some backstory: there was a time when only one Ruby implementation existed in the world - Matz's own Ruby Interpreter (MRI) written purely in C. Some time later, JRuby arrived onto the scene and, while it was still in its infancy, an announcement was made by Evan Phoenix for a project called Rubinius. Rubinius was not just an ordinary Ruby implementation; it was an implementation with a goal to be written in Ruby as much as possible, inspired and modeled off of Smalltalk-80. Soon thereafter, Brian Ford joined forces with Phoenix to continue Rubinius development under Engine Yard and to work on a specialized framework called MSpec. This simplistic framework allowed the Rubinius team to create tests of "correct behaviors" found in MRI and to mimic those results in Rubinius. It was a basic idea designed to measure the progress of the project, and it reaped rewards far more than originally anticipated. RubySpec became the name of the collection of "correct behavior" tests and the executable specification for all implementations in the Ruby ecosystem. By having all implementations follow the standard definition of Ruby (which in this case were MRI and KRI aka YARV), programmers would be able to move around to different implementations with ease and confidence.

RubySpec on Communication

As it stands, all Ruby implementations adhere to RubySpec. We now know how RubySpec facilitates compatibility among each implementation, but how does it facilitate communication? RubySpec tests are not only intended to be executed against implementations, but like any other open source project, it fosters contributions. Implementers and even developers are encouraged to add tests to RubySpec to enhance and increase the value and quality of the test suite. By adding tests, it creates an opportunity for implementers of different implementations to talk to each other. This is what I would call a symbiotic relationship, one where all implementers have a common goal to make their implementations look and feel the same as MRI/KRI.

The Case of Other Languages

Unfortunately, this symbiotic relationship does not appear in other languages such as JavaScript and Smalltalk (at least not yet). JavaScript implementers took the specification from ECMA (ECMA-262, Edition 3), complied with most of what was in the specification, and added their own language feature set on top. Over time, after revisions and rewrites of their JavaScript engines, it became difficult, from a developer standpoint, to distinguish between what was originally part of ECMAScript and what was part of the implementation. Perhaps that was never the goal of ECMAScript, but the language slowly became fragmented, if ever so slightly, and developers found it challenging to switch between platforms (in this case, browsers).

Smalltalk implementers also took the specification from ParcPlace (Smalltalk ANSI standard) and proceeded in different directions, never to look back. If you search the words, "Smalltalk" and "Balkanization", you will come up with some interesting articles on why Smalltalk did not really take off.

While some will argue that both JavaScript and Smalltalk are backed by proprietary vendors and, therefore, not in their best interests, Ruby has proprietary vendors as well. Fortunately, though, nearly all of the implementations are open sourced and all of them use RubySpec. Companies like Engine Yard, Microsoft, Apple, and Gemstone have created their own implementations and, while only two of them are fully implemented, developers are excited about using them and have confidence that the language they have grown to love will work the same in the new implementation.

What Can Be Done (Or Is It Too Late)?

Thanks to JavaScript developers like Juriy Zaytsev, there is a set of tests that implementers can use to find common ground in the JavaScript language. There is also the Sputnik test suite which determines how well an implementation adheres to ECMA-262, 3rd Edition. Of course, it is up to the implementers to consider trying to find common ground, but having a set of tests will reveal incompatibilities across JavaScript implementations.

For Smalltalk, there has been apparently more converging than diverging in the past decade, but we may never see the light at the end of the tunnel. Now reaching its thirties, finding a common ground may prove to be difficult. As of yet, there is no standard test suite that all Smalltalk implementers use and, as a result, communication through tests does not occur. It may be too late because of the complexity involved with each implementation. The only thing Smalltalk implementers have going for them is whether or not Seaside runs on their implementation, similar to how Ruby implementations use Rails ("The Rails Singularity") to test their own implementations.

Final Thoughts

As you can see, tests can play a major role in software development. Not only does it give confidence about the code and design, but also allows communication, similar to acceptance tests. This is one particular case where tests can help a language ecosystem thrive and sustain compatibility and communication. I propose this is one of the reasons why we love Ruby so much. Having common ground allows developers to fiddle with different implementations without worrying about syntactical issues, but at the same time, having the confidence that their code will work much the same way as it did from one implementation to another.

References

touring ends

2010-11-19T09:00:00-08:00

Yesterday was my last day of the four week tour. I enjoyed learning about each project and getting a chance to pair with everyone. I've noticed each craftsman has a different way of developing and different sets of tools that they use to craft software. If I had to compile a list, this is what it would look like:

Editor/IDE

vim
Textmate
IntelliJ

SCM

Unix commands

ack
grep
ssh
scp
screen
ps
nohup

Acceptance Testing

FitNesse
Cucumber (Ruby)

CI

Hudson

Communication

Skype
iChat
Campfire

I think knowing how to use all of these tools is beneficial, so I plan to learn how to use them very well. When pairing with another developer, you can easily adapt to his/her environment and be ready to add value to the work at hand. One thing that was nice to see is how vim is slowly becoming every craftsman's favorite editor. Even if they're starting off with the high learning curve editor, they're using it daily to quickly pick up new commands and make it part of their development environment.

Overall, the tour was a great way to learn about every project and a challenge to see if I was able to come into a project and contribute, either by developing code, pairing, or acting as a rubber duck. I think I did contribute to all projects and add value to them.

The next couple of weeks, I'll be busy on some challenges, so I may be on and off blogging about the experience. The end is near.

preliminary estimation

2010-11-18T04:00:00-08:00

For the early part of the day yesterday, I sat down with another 8th Lighter to go over the potential client's list of actionable items that need to be completed at a certain time (it sounds vague I know, but it's intentional). We went over each item on the list discussing what the item means and what work is involved to implement the item into the application. We used the PERT model to calculate each item based on the points we gave in the following categories: optimistic, realistic, pessimistic. I've used the PERT model before and it worked out much better with estimates. In addition, I leaned on the other 8th Lighter as he has more experience in Rails than I do. I look forward to what the client has to say.

Colin and I worked on some more tickets after lunch as we learned more about the application. I've noticed in addition to the Rails DSL, this particular application has DSL of its own that I have not quite grasped. Not having a local copy of the source makes it even harder. Unfortunately (for me at least), the security checks are more troubling than what it's worth, but it would have been beneficial to learn more the application.

Today, we'll be working again at the client's office. I've always been wanting to go to the Chicago Geek breakfast and finally found an opportunity, so I took an earlier train to nerd out while having breakfast. It's going to be a fun day in Chicago.

trair trogramming

2010-11-16T20:59:00-08:00

Note: I thought I'd name this post the exact same as my apprentice mate, Steven's post because we have nearly identical names, we both "pair" programmed with Colin, and we played a game of Go. These events all occurred today. It wouldn't make sense to name it otherwise.

So today, Colin, Steven, and I paired to tackle some support tickets for the client. I noticed this was a great way to learn more about the Rails application - by working on tickets and researching the problem in different areas of the application. I've read somewhere that tri-pair programming can actually lessen the experience, but it was not the case with our session. As Steven mentioned in his post, I believe he and I helped Colin. I did ask a lot of questions that probably slowed the process down, but Colin was very patient and answered all of my questions. It helped me understand the application more and the root of the problem we were solving.

Meeting with another client

I briefly left Colin and Steven to attend a meeting with other 8th Lighters and a potential client. We went through the details of what the client wanted and we are to come back with estimates. Fortunately, the client gave us the source code, so it will give me a better idea how to estimate each actionable item.

pairing on site

2010-11-15T20:59:00-08:00

Today, Colin and I took the train down to Chicago to go to the client's office to meet up with Justin where we partook in a meeting about email subscriptions. I sat in to get the gist of the goals the client has in place for the service they offer. It was a pretty interesting conversation. Usually, I'm the end-user where subscription-based email is important to me. I want to be able to unsubscribe at any time without the need to log in. The meeting was partly to take priority of the issue and make sure it's easy for the customers to unsubscribe and comply with the CAN-SPAM Act. Colin and I briefly pairing in the morning and later in the day, I ended up pairing with Justin to implement a Rails view which makes it easier to navigate than the original view in place. There was quite a few yak shaving Rails DSL we had to weed through. I'm still somewhat weak in understanding Rails routing, as well as views, so I'm not sure if I was of much help. It was fun pairing though and trying to understand the context of the problems both Colin and Justin were solving. Throughout the week, I plan to pick up Rails again and learn more about its magic.

I'll be meeting with a couple of 8th Lighters tomorrow for a potential client which is exciting in it of itself. I look forward to the meeting and seeing what is in store for this opportunity.

final stop on the tour

2010-11-12T07:00:00-08:00

Final Thoughts for the Week

I had a lot of fun working on this week's project. This was my second time working in a multi-language project, but the first time where one language called the other directly (e.g, JRuby). It was clear to me that having multi-languages in projects have its benefits, but can be cumbersome. However, by creating rake tasks and having a set of tests, I felt confident that any changes made to the code could be done easily and the tests will reveal any existing functionality that broke during the process or reveal any smells. I learned some new technologies along the way and put several design patterns to use. I remember reading Corey Haines' tweets about simple design and how design patterns might not be good for beginner developers like myself. However, I agree with Dave and others that design patterns help with communication. By throwing around design pattern keywords, it helps developers communicate with each other. I don't think design patterns should be something you should always try to follow, but use it to reveal a design that fits the context of the problem.

Next Client

I'll be working with Colin and Justin starting today and pretty much all next week. I'm really excited to be working on this project and learning more about Rails and new technologies. More details to follow.

refactoring the pdf feature

2010-11-10T05:00:00-08:00

I continued working on the pdf library from Monday and finally completed it, creating a decent set of tests. With the tests now in place, I was able to refactor the code and have confidence that the changes I made didn't break existing functionality. I then proceeded to implement the pdf template into the Rails project, replacing old code with the new. I fiddled around with the ant build.xml to create a task to run the unit tests. Then I created a rake task to run the ant task as part of the all_tests task. Now when I run all_tests, it will run the unit tests for the Java library first, then specs, and finally cukes. This will give me confidence that changes I've made on the Java side won't affect tests written on the Ruby side.

I did run into a problem where a particular hash value passed to the Java side was nil. Unfortunately, I was not expecting null value (Java newbie?) and therefore did not have a test for it. So I ended up creating tests and a wrapper class that always returns an empty string if the value is null. This will get rid of the NullException errors and make the library more resilient to nil values.

My next task is to remove the noise in the controller and move it to some models. Currently, the pdf code is all in the controller and it's a bit noisy. I've run into this kind of situation several times before and moving it to the model (or even lib) helps cut down the noise. Controllers should not know too much on how to perform jobs. The models should know the details and controllers usually delegate to the models or views. After making the code more DRY, I'll be adding more pdf templates on the Java side.

8th Light Hack Night #2

Last night was the second hack night and Craig and I worked on the Twitter feature from the last time I worked on it. The way I implemented the feature was a bit hacky and I knew that from the start, so I asked Craig to help me. To be honest, I basically watched him do his magic via iChat and tried to follow his way of tackling view problems such as this. In the end, he got it working (took longer than expected) and I went ahead and made the changes to the rest of the bio pages. While the hacky way took no time at all, I really wanted to learn how to do it the right way. In this case, diving in to inline styles, HTML source and understand the root of the problem. I appreciate Craig taking the time to do all of that.

Other news

I'm trying to learn more Vim commands and use vimperator (on Firefox) and vimium (on Chrome). While I still reach for the mouse and IDE to help solve bigger problems, I will keep working on using more keyboard shortcuts, using vim exclusively for Ruby/Rails/insert dynamic language projects. Don't get me wrong, I like using all sorts of editors. I want to be better at all of them, but right now I want to focus on Vim. It's a great, powerful editor, and every developer, tester should learn how to use it.

java and pdf

2010-11-08T20:00:00-08:00

Today was a busy day at the Chicago office. I was creating a PDF template for one of the features in the project. The PDF is generated from a custom library in Java that depends on a pretty well known PDF library called iText. I was constantly at the javadoc looking for clues to create the template. The other templates Paul made were purely text-based, but this template required the use of tables, so I had to do a bit more research. I was also thinking about how to create tests around the template. I ended up going back to the abstract class (which the template extends on) and refactored a bit to make it DRY. I think I may go back some more and write more tests. I get an itch on the back of my neck whenever there are little to no tests. The discipline I learned the past five months taught me to write tests first. The LOCs in the tests should be about the same as the LOCs in the source.

Unfortunately, I did not finish by the end of the day, so I'm going to finish it up tonight and I'll be working on more templates tomorrow and get through the stories for this week. Tomorrow is 8th Light's second hack night, so I look forward to pairing with someone and learn more about Rails and get through some of the stories for the 8th Light website.

new client and git

2010-11-06T12:00:00-07:00

Note: This post should've probably been a tweet.

New Client

I worked with Paul yesterday to integrate a pdf generator in a Rails project. The code that generates the pdfs is written in Java and called within Rails via JRuby. On the Java side, we implemented the template pattern to make the code as DRY as possible. Paul gave me access to the GitHub repo and access to the staging server. I plan to fiddle around with it over the weekend and pick up Rails again to familiarize myself with the dsl.

Lunch 'n' Learn - Git Session

Craig presented his final session on Git, talking in detail about branching, merging, rebasing, and tagging. It was nice to see in detail why some merges are fast forwards, recursive, and conflicted. I've been merging lately, but will start rebasing for future commits.

Vim

Ever since I saw Eric Smith's vim skills, I decided to pick up new vim commands and even install vimium and vimperator on Chrome and Firefox respectively. These extensions allow you to use vim (like) commands on the browser without having to use the mouse.

on-site iteration meeting

2010-11-04T21:59:00-07:00

I drove down to the client's office today to sit in for the weekly iteration meeting with Jim and the Erics. The meeting consisted of the stories that were due for this iteration and what's to come for the next iteration. Of course, they did not know what was involved in the next iteration until the stories were estimated right then and there. So they had to estimate them...using their hands. I like this approach. On the count of three, each person throws out a number. Of course, not everyone will estimate the same number of points, so there's a short discussion to convince one another how it's this many points or that many points. Usually, in a situation where there's three developers estimating stories, the favor is on the two that has nearly identical estimates. At least, that was the pattern I noticed today. New stories, bugs, and production support were all part of the new iteration. They also talked about automating deployments, new development features, and specific bugs.

After the meeting, followed by lunch, Jim and I sat down and started working on a couple of customer-facing bugs that were JavaScript related. We were able to identify the problem right away and fix them. Something I'm aware of as a tester, but have not experienced as a developer is the inevitable differences between each browser's ability to render pages. Not all browsers were created equal and each of them have their own standard for what's regarded as an error. One particular bug was only noticeable in Internet Explorer 7, while WebKit based browsers did not show the error at all, not even IE8. This was a rare moment where I sided with IE7's feature to display the error because it was not so great JavaScript code. I won't go into details, but it shouldn't have worked and IE7 barked up the right tree while others did not. Of course, the end user doesn't care for errors, but as developers, it's nice to get some visual feedback during testing to avoid end users from seeing them.

Next stop on the tour

This concludes my working with Jim and the Erics. I enjoyed working with them and learning about the projects they work on as well as their technical skills which I'd like to pick their brains another day. Time was well spent on the deployment, staging, IT side which I'm still not too familiar with and would like to learn more. Also the projects are fairly large and I'd like to learn more about them so I can contribute to the iteration. Anyway, I had fun and look forward to working on the project again.

Starting tomorrow, I will be working with Paul at the Chicago office and will be working with him on a Rails project which I'm familiar with since the last time I worked on it. It should be fun working on the project again and look forward to the stories.

xml responses and decorators

2010-11-03T17:00:00-07:00

Today, I spent the day pairing with Eric Smith, attempting to grasp the context of the problem he was working on. Basically, the story asked for generating response xmls for every set of answers to survey questions. Some questions have follow-up questions which also need to be in the response xml. At first, I had a bit of trouble understanding the context of the problem, but later understood the basics and tried the best I could to be a really good rubber ducky, a spell checker, and bouncing ideas around. I hope I did not slow him down.

When working with pairs, I always try to observe the other person's workflow. Specifically during the session, I try to learn about the way the person writes tests, the tools that were used, the diagrams on the whiteboard, the breaks in between coding, etc. Eric is a VIM power user and he knows his TDD, so I was able to pick up on a few VIM commands here and there, learn a bit of Rails DSL, and RSpec tricks:

:null_object on a mock object basically says any method called on the mock object will not throw an error.
using anything in the argument allows any parameter to be passed in
be_blank is only available in Rails, but allows you to test values that are blank (instead of using \== "")

I was happy to see the way I usually write RSpec tests are somewhat similar to the way Eric writes his. He liked spiking code to understand the context of the problem and wrote tests for the particular area we were working on. At the end of the day, we did not finish the story, but I hope I had contributed to the story, rather than be a hindrance.

Tomorrow, I'll be on site with Jim and the Erics and I will continue working with them to help with stories, on call tech support, etc.

indexes save lives

2010-11-02T18:00:00-07:00

Eric Meyer and I were able to identify the problem of the long running process from yesterday. By creating a composite index, in one of the tables, it looks like we solved the problem! We checked the logs and saw each SQL query took anywhere from 1-5 seconds before the index and 0.0002-0.0007 seconds after the index. A dramatic improvement and significant time reduction. Eric and Jim will be working on it further tomorrow to see if it truly fixed the problem.

The next task Eric and I started working on was seeing if Selenium tests were able to run on a Solaris environment. I quickly checked the Selenium website and saw it was supported. Eric took the lead and tried to make it work, but we didn't have time to finish the task. He will continue working on it tomorrow.

Tomorrow, I plan to work with Eric Smith.

Chicago Ruby Meetup

Justin Love presented his talk on AMP, a cross-repo version source control in Ruby. AMP is basically a one-stop shop Ruby front end solution to two major distributed VCS: Mercurial and git. At least, that's the idea for the upcoming project. The original John Locke version only supports Mercurial because the developers ran into some roadblocks with git. Since Mercurial is GPL licensed, the developers ported Mercurial to AMP and also licensed it as GPL. The next version of AMP called AMP Redux (pronounced 'redo') will separate interface and implementation and support both Mercurial and git. They've learned a lot from the process of creating the John Locke version and currently creating AMP Redux.

Why use AMP (or why another VCS)?

Justin Love listed the major VCS out there (e.g., git written in C, Mercurial written Python) and explained how there are no VCS solutions written in Ruby. Why not? There seems to be a general concensus that Ruby is not viable for large-scale applications, just web frameworks. AMP wants to prove that Ruby is viable as written in AMP's manifesto. Here's what else it says:

Is free of religious devotion to one VCS
Lets us customize how we interact with it
Lets us create extensions to add features for all supported VCS
Lets us customize default commands as well as new
Discover the limits of Ruby
Be an example of proper documentation

What can AMP do?

The John Locke version currently supports Mercurial and can access Mercurial repos like the hg command. git, bazaar, svn, and cvs are hopeful supported VCS in AMP Redux. Justin made an analogous comparison of AMP to ActiveRecord in Rails. AMP wants to be like ActiveRecord.

How can I use AMP?

gem install amp
amp --help

How can you help out?

needs documentation
website is in place, but doc out of date
read TODO list

Justin will have this talk at the upcoming RubyConf in New Orleans.

long running process

2010-11-02T06:00:00-07:00

Today, Jim and I paired for the day working on an issue involving a long running process. We ended up placing a bunch of log messages around code on the server and client side as well as using the logs generated by the Rails app. After some time, we were able to narrow it down to a server side process. Fortunately, it was not an issue with the webservice, but a long running process occurring on the server side. The process took longer than the client's timeout and the client disconnected. Here are some things I've learned from the troubleshooting process:

Trace logs don't tell you the whole truth

While this may be obvious, the details of the log did not exactly spell out the exact problem. It did a great job of pointing out the vicinity of the problem though. By placing a bunch of log messages around the area in the code, we were able to pinpoint the problem.

Communication is important

Another obvious one, but wanted to note the importance of communication. With the help of both Erics, we were able to pinpoint the problem. We all worked on it together and found the problem. It was great to have other developers next to you to help you out when needed.

Rinda Server/Client and Tuples

I heard Jim and the Erics use these terms a lot in conversations, but did not know what they were. Now I understand the terms and context used in this project. It's a neat way of managing tasks and/or background jobs. I'm not sure where the statemachine fits in to the scheme of things, but Rinda server/client reminds me of the socket service I implemented in my HTTP server and tuples are like messages that represent things that need to get done.

Testing Tasks are Hard

While I'm getting more familiar with unit (or micro) testing and TDD, I've not tested tasks before. In a previous Rails project, I created a Rake task, but did not create a test for it. There's something difficult about creating tests around a single command that does a lot of things. Of course, the individual methods have been unit tested, but integration like tests is a different story. I followed Jim's lead and learned his approach to the problem. While we did not finish by the end of the day, it was interesting to see how it was created.

Next Up

I'll be pairing with Eric Meyer today. We are taking the train down to the Chicago office and will attend the Ruby meetup afterwards. I look forward to learning more about the project.

next stop on the tour

2010-10-30T13:00:00-07:00

My next stop on the apprenticeship tour, I'll be working with Jim and the two Erics. There are multiple projects for this particular client and they're using Rails among other technologies. I spent most of yesterday getting the development environment set up, making sure all tests ran successfully. I've not worked on a project with so many unit tests, so it will be pretty interesting. While it took me about half a day to get set up, it should only take a couple of hours. I ended up creating several wiki pages on FitNesse for instructions on how to set up subversion and the development environment for Snow Leopard using rvm and homebrew. The next person taking on these projects should be able to easily get the environment up and going in less time than me.

8LU - Git Session by Craig

Craig continued his git series by talking in depth about a lot of git commands. Most I knew about, but there were some that I was not aware of, so it was nice to learn about them. One particular thing I learned from Craig and try to follow on projects is to create a local branch when making changes, then rebase master on top when ready to push changes to GitHub. Here are the steps:

pull latest from GitHub repo to local master
checkout new branch (e.g., git checkout -b new_branch)
work on stuff
commit stuff on new_branch
checkout master and pull changes
checkout new_branch and rebase master on top of new_branch
checkout master and rebase new_branch on top of master
push changes
delete new_branch

I look forward to his next git session next week. It should hopefully cover some good topics on branching and merging.

8th Light Hack Night

This is the first 8th Light hack night I participated. We're going to make some improvments to the 8th Light website and one of the things I'm doing is to add the Twitter profile links to each craftsman's page. I already have what I need and will be getting it done today.

client work retrospective

2010-10-28T17:00:00-07:00

Today is my last day working with Doug and Li-Hsuan. In case you haven't noticed, I'm on my apprenticeship tour where I tour around each client project, helping out the 8th Lighters, learning about the project, "stand-in" for stand up meetings, and experience real world scenarios. I really enjoyed working with them and look forward to working them in the future.

Ruby Gem Creation

While this was a small task, I didn't know how to create a Ruby gem before. Fortunately, Doug already had the library ready to go by encapsulating the library in a module (not sure if there is a name for this convention). I just needed to use Jeweler and it created the Gemspec for me. Doug simply upped the version and recreated the gem when pulling in changes. It worked out well.

EventMachine

I was pretty much using EventMachine the entire time I was on the project. Li-Hsuan and I kept adding features as Doug asked for them. We added features like multiple callback and errback handling and learning a ton about EventMachine during the process. Testing EventMachine was a bit of a headache. I can't tell if the tests were not designed properly or if it was the fact that EventMachine was sometimes difficult to understand, but we were able to test everything we've implemented. I'd sometimes forget how certain features worked in the library and had to read the code again to understand what was going on. Also, I don't know how many times we put puts or p statements in the code and tests to figure out what was going on. Again, I'm not sure if it's a result of code smells or something else.

Client Communication

While our stand up meetings were short, it was nice to introduce myself to the customer and talk to them briefly about the features we've implemented. For the past eight years as a software tester, I've never actually talked to a customer. My connection with the customer were business analysts, developers and perhaps the QA manager. So working with the client this week shined a different kind of light. By removing the middle persons in between you and the client, you can provide personal service to the client and have better communication. This is also my first time working in an agile environment, so the the frequent feedback (e.g., daily stand ups) help bridge the communication between us and the client.

Touring Continues

Starting tomorrow, I will be working with other 8th Lighters on a different project. I look forward to learning a lot and getting my hands dirty. It's going to be fun.

client work on a windy day

2010-10-28T07:30:00-07:00

We continued working on the EventMachine library for our client, adding more features Doug needs to make the implementation work. One big feature we nearly finished yesterday (just needs integration testing) is storing more than one callback on both the server and client side. Before, both sides were only storing a single callback. Doug ran into a problem where the second callback overwrote the first one because the second remote call finished before the first. Now we're storing the callbacks in a hash with a unique key which is passed as part of the message that gets sent through the remote call process. This broke a lot of tests and we spent more time fixing the tests and making them pass than writing actual code. I'm wondering if our design has smells which makes it difficult to change the code and the ever changing requirements. I think we should look at the design as a whole and see there are any SOLID principle violations, and make changes necessary.

Our next task is to work on filtering out "insecure" methods on remote calls. The library currently allows both sides to make remote method calls (e.g., call the remote API's methods). Since all Ruby objects come with a set of default methods, we don't want the client to call methods on the server that could potentially do harm (e.g., instance_eval, instance_variable_set, methods).

client work on a rainy day

2010-10-27T06:30:00-07:00

Yesterday, I had an early start in the morning. We had a bi-weekly iteration meeting with the client where we demoed stories we've been working on the past couple of weeks. It was nice to see the story Li-Hsuan and I worked on demoed to the client albeit how little the feature was.

Li-Hsuan and I continued to work on several EventMachine related tasks. I have to say, there's something to be said about EventMachine's method calling convention. Usually, when I see a constant after a constant with two colons (e.g., EventMachine::), I usually see a class or module. But EventMachine likes to use '::' to call module methods. You could simply run the same method using the dot notation, but the documentation and all example code uses '::'. Perhaps, I've not read enough Ruby code, but it seems a little odd.

EventMachine::run {
}

EventMachine.run {
}

So when attempting to mock EventMachine, at first I didn't know how to do it. But after trial and error, it's the same way as any other type of mocking:

EventMachine.should_receive(:stop_server).once

During the entire time working on EventMachine, we were using Ruby MRI 1.8.7, but this library will be used on JRuby. So we switched to JRuby, installed the gem, and saw a lot of our tests fail. We found out from Doug, the Java implementation is not quite the same as its C extension sister. We ended up building the gem from the GitHub repo.

[~/local/git/em-gem] rake java:gem

Some tests still failed after installing the edge version, but we were able to make them pass and checked in the code. There's still some more work to be done, so we will be sitting with Doug today to go over some of the new tasks.

eventmachine

2010-10-25T21:30:00-07:00

Today, Li-Hsuan and I finished up an EventMachine task we started on Friday. We were testing and implementing a way for the server to send an errorback (a callback for errors) if the desired port for the server is occupied. We thought TCPServer occupied a port, but in fact it did not. I'm not sure what it was doing. We ended up creating EventMachine servers back to back and was able to catch an exception which then surfaced up to the calling code. After completing the task, we were given another task where Doug wanted us to handle exceptions and have them surface up via errorback for method calls that do not exist. To make sense in all of this, there are two "connections". We can call them server and client, but both sides can send/receive messages. Similar to DRb, Doug implemented a way for both server and client to call each other's methods. If the client calls a method that does not exist locally, it automatically send the message to the server and attempt to run that message on the server side. There are two ways of which this can happen:

Client truly wants to call Server's method and calls it.
Client accidentally mispelled local method.

At the end of today, we were able to finish pretty much the entire implementation except when server makes calls to client's methods. We'll need to talk to Doug about this.

A little introduction to EventMachine

This is Ruby's evented I/O, similar to Node.js. It's designed around the Reactor pattern and reminds me of the Socket Service and Executor Service I implemented for the HTTP server in Java, except way faster and C10k aware (although not a counter solution to it). It encapsulates the socket level, so you can focus on the application logic and is designed to be used to create both server and client.

EventMachine::run is used to run the event loop. It does block, so a callback is needed to terminate the event loop. EventMachine::stop_event_loop is used to stop the event loop. All open connections will cease and close.

EM is an alias of EventMachine and can be used interchangeably. There are other aliases in EventMachine.

class Echo
  def post_init
    puts "Hello"
  end
    
  def receive_data(data)
    p data
  end
end

EventMachine::run do
  EventMachine::start_server '127.0.0.1', 8081, Echo do |connection|
    # connection is EventMachine::Connection + Echo module
  end
end

The code above is an example of a simple server started in EventMachine. Echo is a handler, a key component in EventMachine to handle particular network protocols. This module magically mixes in with an instance of EventMachine::Connection and is returned in the start_server block.

# code from EventMachine wiki

module Echo
  def receive_data(data)
    p data
  end
end

EventMachine::connect '127.0.0.1', 22, Echo

For connections, handlers can be blocks/procs, modules or classes. In the example above, the module Echo is used to handle connections on localhost port 22.

I'm not sure if I made any sense in this introduction, but there's a lot to say about EventMachine and what I briefly wrote here is just touching the surface. It has deferables, periodic timer, different protocol support, error handling, etc.

References

EventMachine GitHub repo
EventMachine rdoc

client work, ppp

2010-10-24T21:30:00-07:00

Client Work

On Friday, Li-Hsuan and I continued working on the story we started Thursday and completed it by the end of the day. We showed Doug our work and he was happy with it and we checked it in. During the day, he asked us to create a local gem for one of his libraries. I took over the task and created the gem. There's a nice gem out there called jeweler that pretty much does all the work for you. I first created a blank project using jeweler:

[~/local/git] jeweler test-gem

I only did this so I can grab the Rakefile it auto-generates. The Rakefile is pretty much all you need. I opened up the Rakefile and modified the values in there:

Jeweler::Tasks.new do |gem|
  gem.name = "test-gem"
  gem.summary = %Q{Some summary}
  gem.description = %Q{Some description}
  gem.email = "skim@email.com"
  gem.homepage = "http://github.com/sl4m/test-gem"
  gem.authors = ["skim"]
  gem.add_development_dependency "rspec", ">= 1.3.0"
  gem.add_dependency "eventmachine", ">= 0.12.10"
  # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
end

Then I ran the following command:

[~/local/git/test-gem] rake install

This will build the gem and install it. You can also use rake build to build the gem. rake -T will show you other rake tasks you can execute.

After all of that, Li-Hsuan and I were assigned another task: to work on an EventMachine related problem. I've not worked on EventMachine, but I heard it's similar to Node.js. We weren't able to complete the task in time, but I took some time over the weekend to read up on EventMachine.

PPP Book

I read a bit more from the PPP book. It went through the seven design smells and the SOLID principles, both of which I blogged about. I quickly read through them again and found some gems.

Requirements Changes Are Not The Fault of Customer

When I worked as a software tester, I helped develop some tools for our team and remember wanting the entire design up front (BDUF). I also remember having the same conversation as the book, where I talked to the manager and told him that the requirements can't change. Of course they can! This should be expected when working in software. Customers will make changes. The design should be resilient to requirement changes, able to meet them without showing signs of design smells.

Anticipating Change

While there's an advantage to add protection to your design to requirement changes, if you don't expect change in a particular area, adding protection might be wasted effort and make the design more complex. As long as the design makes it easy to add protection, then it's best to add it later when the change calls for it.

client work at the chicago office

2010-10-22T06:30:00-07:00

Client Work

Today, I went down to the Chicago office to work with Doug and Li-Hsuan. It looks like I'll be working with them for about a week or so. We had a quick stand-up meeting where I was introduced to the client. Li-Hsuan and I were assigned a couple of stories while Doug worked on a different story. Right off the bat, we had a bit of trouble configuring IntelliJ, but were able to resolve them. We started by creating tests and do some refactoring. I tried to mimic what the Bobs did in the bowling session. I really wanted the tests (and code) to be readable and clear. I think in the end, we ended up with tests that read nicely. We also started applying the Strategy pattern for a particular story we were working on. This was a pattern I recently learned about! It was so exciting to be applying the knowledge I've learned this week. I look forward to working more on the story tomorrow.

Clojure Koans

During lunch, the 8th Light "city" office likes to have hands-on Lunch n' Learn sessions (everyday?). This week they're doing Clojure Koans. It was fun going through the koans again. We would each take turns completing a koan in a counter-clockwise fashion. Fortunatley I did not forget Clojure's syntax and worked my through them all until lunch was over.

Note: I apologize for the short post. I will have more material when I read more from the PPP book and Design Patterns book.

principles, patterns, and practices

2010-10-21T08:00:00-07:00

I picked up the Agile Software Development - Principles, Patterns, and Practices (otherwise known as PPP) book again. It made for good use when learning about the five patterns. I regret not reading this book earlier. Colin wrote in his blog how he could have taken advantage of some of the refactorings for his Tic Tac Toe Java implementation after reading the Refactoring book. I feel the PPP book could have helped me not only with refactoring, but with pattern usage, avoiding SOLID principle violations and with the Bowling Kata. But at the same time, after getting comfortable in Java, the PPP book seems easier to read. I can understand and read the code.

Bowling Game with the Bobs

I really enjoyed reading the dialogue between the two Bobs tackling the bowling problem. They started off with a simple UML design on a napkin that had three classes: Game, Frame, Throw. In the end, there were two, of which one were not in the original design: Game and Scorer. Scorer was created to remove an SRP violation in the Game class. What I found interesting was the consistent refactorings throughout the session, the simplicity of the design, and scratching their own itch when it came to code smells. The resulting code works, can easily be changed (easy to add new functionality), and is readable - the three functions every software module should have.

Here are some notes during their session:

After creating the simple UML diagram, they started at the end of the dependency chain and worked backwards
They moved up the dependency chain until they were working on an object that had more than just getter/setter methods
They pointed out uglies without hesitation
They asked if this Bowling module will be used by thousands of people other just them when adding conditions to protect user
They pointed out things they didn't like without hesitation
They avoided circular references
They wanted their tests to reveal design (e.g., introducing LinkedList for Frames)
They avoided code that did not make sense or was not readable (e.g., Magic Number 21)
They changed the method names numerous times
They avoided writing code that could have different results when using different compilers (e.g., order dependency)
They removed tests that were not useful
At the end, most of the methods contained one line of code

What is Software Design?

One part in Uncle Bob's book talked about Jack Reeves' article titled, "What is Software Design?" In it, Reeves describes how source code is the documentation for software. He compares and contrasts hardware design and software design, hardware engineering and software engineering. Programming, testing, and debugging are all part of software design activities and should not be shortchanged. He believes software development should not mimic engineering as it's more of a craft. Unlike hardware engineering, the software development cycle does not have a step where the "designers" hand off the design to the manufacturers where they go ahead and create the product from the design. The designers create the product at the same time as creating the design.

I tweeted yesterday about wanting to read Jack Reeves' afterword to his afterword to his afterword to his article if he should ever write one. I want to know if functional languages is considered an advance in programming in his POV and if auxilliary documentation still needs more emphasis.

Well that's I have so far. I'll write more when I make progress in the PPP book.

five design patterns

2010-10-20T09:00:00-07:00

"When we consider using a pattern, we must decide whether the pattern solves the problem, whether it fits the context, and whether the benefits outweigh the costs." - Uncle Bob Martin

I will go over the five design patterns I learned:

Abstract Server
Abstract Factory
Observer
Strategy
Template Method

Abstract Server

This pattern is considered one of the simplest patterns of all OO design patterns. It is used to break dependency of client and server and prevents DIP violation, and therefore prevents OCP violation. Although, it has the word "server" in it, it has nothing to do with a server. In Uncle Bob's article, he uses a button/light example to show how this pattern can be used. DIP says:

"Depend on abstractions, not on concretions"

The button depends on light. The solution is to introduce an interface called Switchable in between the two concrete classes. The concrete class and derivative class now point to the new interface. The name of the interface is pretty interesting. It's not named similarly to the derivative, but with the client. This is because the interface belongs to the client and will be packaged together. When thinking about static languages, it is important to package classes/interfaces together to avoid the need to re-compile. As the PPP (Principles, Patterns, and Practices) book says, "the logical binding between the client and the interface is stronger than the logical binding between the interface and its derivatives."

Abstract Factory

Textbook definition: Provide an interface for creating families of related or dependent objects without specifying their concrete classes.

This pattern basically takes away concrete class references from the client and moves them to a concrete factory class. An abstract factory class is then used to prevent DIP violation and sits in between the client and the concrete factory class. In the concrete factory class, you'll have a method like "make" that will take a string argument of the name of the concrete class you want to instantiate. I asked Micah about the if/else statement in the factory class. He said it's ok to use since it's used to instantiate classes.

Abstract Factory uses Strategy - abstract factory class with concrete factory classes.

Observer

Textbook definition: Define one-to-many dependency between objects so that when one object changes, all its dependents are notified and updated automatically.

This is a subset of the pub/sub pattern. A publisher allows subscribers to subscribe/unsubscribe notifications. Micah gave an example about a Clock client. If another class wants to get the time for every millisecond, or nanosecond, etc., it's better to tell the class that the time has updated instead of the class asking if the time updated. This is especially true when there are more than one class asking. Will each class have a while loop and ask forever? An observable class will notify all of its subscribers (or observers) and each observer will then get information from the observable or execute other commands specific to its needs.

Strategy

Textbook definition: Define a family of algorithms, encapsulate each one, and make them interchangeable. Strategy lets the algorithm vary independently from clients that use it.

The Strategy pattern is about taking advantage of polymorphism. It allows you to easily swap different algorithms. Uncle Bob illustrated a Logger class that allows for different types of formats and different types of recording the logs. Strategy says create an interface called Recorder and have all derivatives define the "algorithm". So the RamRecorder derivative class will have an algorithm to write to memory, FileRecorder will have an algorithm to write to file, and so on.

Template Method

Textbook definition: Define the skeleton of an algorithm in an operation, deferring some steps to subclasses. Template Method lets subclasses redefine certain steps of an algorithm without changing the algorithm's structure.

Template Method is similar to Strategy except it has some implementation to the algorithm and leaves it up to the derivatives to finish the implementation. There might be situations where two derivatives will perform the same kinds of the steps in the beginning or the end, but in the middle is where it's actually different. By using Template Method, you can move the duplicate code up to the base class and only have different code in the derivatives (keeps it DRY).

Implementing design changes to Tic Tac Toe

I went ahead and made changes to my Tic Tac Toe implementation in Java. I introduced Abstract Server, Abstract Factory. I created interfaces and abstract class for Board, UI, and Player. I then ended up with several factories to create the Board, UI, and Player. I thought I could introduce Template Method to each player's makeMove method, but I noticed Game actually places a move on the board after getting the move from Player. This is because it checks to make sure the move is valid.

This was a great exercise. I'm glad to have learned about these five design patterns and look forward to learning all of them (from the Design Patterns book).

References

Design Patterns by GOF
Agile Software Development PPP
Uncle Bob articles

revisiting solid principles

2010-10-19T07:30:00-07:00

Design Patterns

According to Gamma in the book, Design Patterns: Elements of Reusable Object-Oriented Software, in order for a solution to qualify as a pattern, it must be used in at least three unrelated projects. This was called the "rule of three." Design Patterns are important, not only for their solutions to problems, but for their high-level names that can be used as vocabulary for communication. Identifying design patterns and knowing which one to use is just as important as other any other development practice.

Design patterns and SOLID principles complement each other. The patterns help prevent violating SOLID principles and help create a solution to a problem in context (James O. Coplien).

Micah and I went over the SOLID principles yesterday using my Tic Tac Toe Java implementation as an example. We identified my design violated the following:

When violating one principle, sometimes it violates another. For example, when violating DIP, in order to fix DIP, you violate OCP. Micah asked me to draw a UML class diagram that shows a Tic Tac Toe design that does not violate any SOLID principles. He also gave me a list of patterns to look up and see if it can be used for the design:

I will be presenting the new UML diagram and explain the five patterns later today. I will describe them in detail in the next post.

scna 2010 and code retreat

2010-10-17T21:30:00-07:00

SCNA 2010

I had a really great time at the conference. I finally had a chance to meet people I follow on Twitter for the first time, had a chance to meet new people, enjoyed the talks, and fun activities. Did I tell you how much I enjoy single track conferences? In terms of the talks, I really enjoyed the apprenticeship panel, where former apprentices from both 8th Light and Obtiva talked about their experience. It was interesting to see the similarities and differences between the two apprenticeship programs as well as hearing the apprentices' experiences, hardships, and responses to some great questions asked from the audience. I also enjoyed the SCNA speaker panel where the top four voted speakers answered questions generated from the audience via poll. Most of the answers were entertaining and educational. Here's some of them:

"There is more to software craftsmanship than writing code" - Michael Feathers

"Learning the interesting subtle techniques that make my craft better and practicing them in hideous detail" - Uncle Bob Martin

"The world will be better if we just practice the craft and have fun. We need not evangelize. Welcome those who come" - Chad Fowler

"Don't convince others. Convince yourself and behave in accord. Others will wonder what the magic is" - Uncle Bob Martin

"I would much rather work with someone less experienced who knows how to communicate than a genius who does not" - Enrique Comba Riepenhausen

"I will not insult my employer by working 2 in the morning" - Uncle Bob Martin

The Chicago dine around tour was an excellent idea where each attendee had the opportunity to meet other attendees. What perfect network opportunities. I also enjoyed the pre-conference reception on Thursday night as well as the lunches in the Nikko, Japanese style buffet area. In short, SCNA was an excellent conference. Like Chad Fowler said, "if you've never been, highly recommend going next year."

Thanks to Angelique Martin for organizing this awesome conference. I look forward to next year's.

Code Retreat with Corey Haines

Today, we had the Code Retreat at the 8th Light office, where 20 some people paired up and TDD'd on Conway's Game of Life. If you recall, someone at RailsConf earlier thsi year, ran similar sessions at BohConf and Corey Haines, a couple of months ago, came by the office to give us a couple of sessions. It was great to work on the sessions again. Similar to a kata, I think the Code Retreat is a great to practice TDD, experiment with different ways of tackling a problem, ways in which you normally wouldn't do on a client project, and allow you to pair with another person to experience other people's style - their way of thinking, coding style, abstracton, refactoring techniques. Overall, I really enjoyed the sessions. I'm glad to have tried different languages (e.g., Ruby, JavaScript, Java) and try to attack the problem from the lowest level to the highest level to somewhere in the "middle". I look forward to attending Corey's Code Retreat next time he has one in Chicago.

solari project on limelight

2010-10-13T16:00:00-07:00

The last tidbit of info in my previous blog post talked about a small Limelight project for SCNA. I'm going to talk about it now in detail. Yesterday, Micah brought up the idea of creating a program to select names out of the list of SCNA attendees for the book giveaways. He wanted it to look like a game or something entertaining, so for whatever reason, I thought about the old departure boards. You know, the ones that flap and create that unique sound. After a bit of googling, I found out an Italian manufacturer, Solari, created most of these boards. People usually refer to it as a Solari departure board or split flap display.

Micah and I paired for most of the day, first creating the randomizer and making sure winners won't be pulled again. Then I looked into finding a recording of the flap sound each character on the board makes. I could not find an isolated recording, but did find a field recording of the sound effect. I loaded it up on Audacity, cut a piece of it so it only plays back a single flap, and we ran it for almost all letters of the winner's name. I say almost because there were some collisions in the sound if a good number of them were being played at the same time. After fiddling around with the sound effects, we found it to be pretty satisfying. I don't know about you, but I think the split flap displays not only are visually pleasing to the eye, but also sound cool. Micah worked on the rest of the program, putting in nice visual touches and making the button look really cool. If you're attending SCNA, you'll obviously get to see it. I'll post a picture of it after the conference.

Book Reading

For the rest of the day, I'm going to pack for the conference and catch up on some reading. More on that later.

http server in java demoed, faking out web calls

2010-10-12T21:30:00-07:00

Challenge Demo

I demoed the HTTP server to Micah today and it went well. We looked at the code and he pointed out some smelly areas. For instance, my Response class was doing most of the work. The Request class would massage the data coming in from the client's socket and the Response will do everything else. Micah pointed out that my construct200Response, construct404Response methods should be calling Response class to "construct" a response based on the status code, but everything was all in the Response class. Oops. Overall, the server worked as intended and the requirements were met. The last thing he asked was what would I do if I were to extend the HTTP server. Here's a list of what I do:

support all request methods (e.g., HEAD, POST, PUT, DELETE, TRACE, CONNECT)
support all status codes
support all current MIME types
add more header fields to the Response (e.g., Server, Content-Encoding, Date)
make the Response class more modular

One final thing that I didn't quite understand was how to read a GET request properly. My current implementation looked at the first 1024 bytes and that's it. I tried to create a while loop and wait for -1. However, it does not work that way. -1 is returned from an InputStream read method is there's a end of string/line situation, but not for a socket InputStream. The only time it'll return -1 is if the socket is closed. So after talking through with Micah, the correct way to read a GET request is when you reach two carriage return line feeds (\r\n). You might ask, what about the entity body? If you recall the entity body is separated by two carriage return line feeds from the request headers. Fortunately, the entity body does not appear for GET requests. If an entity body appears, the client's socket should provide the Content-Length.

So here's the breakdown:

A GET request will not have an entity body, so look for Content-Length of zero or two carriage return line feeds (whichever comes first).
On other requests that are allowed to have entity bodies (e.g., POST), look for Content-Length and use that value to count the entity body coming in.

This should be a pretty sure guarantee, you have the entire request.

Faking out web calls in specs

Following collective ownership, I'm glad to complete changes to specs I've written with Matt when working on a Rails project. What we've naively done is make direct calls to an actual server. This is a big no no and we should have faked those calls. For instance, if I were on the train and did not have internet and wanted to run the specs, I was out of luck. All of those tests relying on connecting to the server were going to fail. So I basically swapped out the Net::HTTP instance for a fake HTTP and when the post method was called, it will return a fake response. This worked for the majority of the tests, but I ran into a problem trying to fake out Net::HTTP for a class method. In order for this fake to work, I need a hold of the instance of class I'm testing. Unfortunately, the class method was creating an instance of this class and storing it in a local variable which I have no access to. Fortunately (thanks to Colin), there's a way around this. Having learned about stubs just recently (vs. mocks), this adds a whole new cool to stubs.

it "does something" do
  some_class = SomeClass.new
  some_class.https = FakeHTTPModule
  SomeClass.stub!(:new => some_class)
  SomeClass.class_method
end

As you can see in the code above, what I'm trying to do is create an instance of the class I want to test (e.g., SomeClass) and set the FakeHTTPModule to https attribute. This FakeHTTPModule has the fake post method which some_class will call. Now in the class_method, it instantiates SomeClass and stores it in a local variable. I want to get a hold of this local variable to set https to the fake module. With this stub technique, it's possible to do this.

Small Limelight Project for SCNA

Up next, Micah and I will be working on a simple Limelight project for SCNA conference. I look forward to doing some awesome pair programming and working in Ruby again (although I really enjoyed Java). We already have some ideas for what we want to do for this project. To give you a hint, I'm looking for a sound effect for the Solari departure board. More on this later.

scna conference

2010-10-11T21:30:00-07:00

This Friday, I will be attending the SCNA (Software Craftsmanship North America) Conference with other 8th Lighters and software craftsmen from all over the world. I look forward to the talks (love the fact that it's single track) and look forward to the fun activities like the SCNA 5k, Chicago Dine Around event, SCNA after party, and code retreat with Corey Haines. Tickets are still available, so I highly recommend checking it out! This will be my first SCNA attendance. I look forward to meeting other craftsmen.

http server in java - request and response

2010-10-11T15:00:00-07:00

I'm happy to say the HTTP server is complete. There were some gotchas as Micah said late last week, but when seeing an index page load up for the first time last night, I was super excited. When was the last time you were excited when a web page loaded successfully?

HTTP Request and Response

I've implemented the HTTP Request and Response classes. The Request class was straightforward. It reads the request from the socket's InputStream. Fortunately for GET requests, the request byte size is small (roughly 500 bytes), so I do not need to do some sort of checking. If it were a POST request, my max BUFFER_SIZE of 1024 will not be sufficient.

The Response class was where all the work was done. Once the Request class has parse the request and identified the URI path and parameters, it's the Response class job to write a response to the client. First, it checks to see what the client is asking for. If you recall, my HTTP Server was to do the following:

serve static pages
when visiting /hello off the root, it will display any parameters if found
display 404 Not Found for anything else

The URI path coming from the Request does a nice job of identifying if the URI path is without a file name. If so, it attempts to add /index.html and check to see if the file exists. If it does, then it's added as part of the URI path, otherwise left alone. The Response class will use this information to determine if the file exists. If it does, it will serve that static page by reading the file contents via FileInputStream. FileInputStream instance will stream chunks of bytes to the socket's OutputStream along with the Response Status Line and Header Fields.

If the file does not exist, it checks if you're attempting to visit the hello page. If you were, it will display the parameters in a simple HTML table, separated in key and value columns. It makes sure not to display parameters that only have keys listed.

Lastly, if neither the file exists nor you're visiting the hello page, it will send a 404 Not Found response.

HTTY aka Heck To The Yeah aka Happy Trails To You

I found a tweet yesterday morning from Corey Haines about HTTY. It's a "console application for interacting with HTTP servers. It's something of a cross between curl and the Lynx browser." Perfect! I tried it out on my HTTP Server and sure enough, when I executed the get command on htty, it displayed the GET request on the server side. This was a handy tool when testing out my HTTP server. Of course, I also used all different kinds of browsers on Mac and Windows to see what the request protocol looks like.

Poor man's load testing

Just to have fun and test out the SocketService, I kicked off the HTTP server and had a bunch of tabs opened on different parts of the local site and had them refresh every second. I also held down refresh (Command + R) for a minute and regularly checked the Activity Monitor. I think the highest I've seen it use threads was 20. The CPU utilization was low. I haven't tried looking for it on Google, but I'd like to see how it does on a network performance tool. Anyone know of any?

File serving

I ran a simple test to load a pdf and mp3 file and it worked out a hitch. I've seen a consistent SocketException error on Chrome 7 where it tries to get the mp3 file three times. Fortunately, the SocketException error does not effect the mp3 streaming and I was able to play the song on every try.

Conclusion

This was a fun project. I've learned so much about HTTP servers and how they work. I've also learned a heck of a deal on multi-threading, although I can't say I'm an expert now. This was a rewarding experience. I hope the demo tomorrow will be smooth.

http server in java - concurrency with executor service

2010-10-08T21:30:00-07:00

public void listen()
{
  requestExecutorStart();
  while (!requestExecutor.isShutdown() && executorRunning)
  {
    final Socket socket = acceptConnection();
    if (socket != null)
    {
      processConnection(socket);
    }
  }
}

SocketException

This Java exception bit me in the dust today. According to the Javadoc, when ServerSocket's close method is called while any thread is currently blocked by the accept method, it will throw the SocketException. When I was testing SocketService, I had this exception thrown, but could not figure out what was throwing it. Javadoc is your friend. I caught the exception and moved on. Fortunately, Micah had some time to pair with me. We were able to finish the implementation of the SocketService. I went over what I needed to finish and he seemed to agree with everything I said.

HTTPHandler: Request and Response

The SocketService will handle all client requests and have its ExecutorService create and run the tasks passing the socket to the handler. The handler will read the socket's request using its getInputStream to read the request in full, parse it, and create a response to write to the socket's getOutputStream. I believe at this point, each task is self-contained and will not need multi-threading. Unfortunately, there are some gotchas that I'm not aware of, sounds funny I know. Micah said there will be some things I will encounter. I hope I won't see too many.

Lunch n' Learn - git

The new Lunch n' Learn series is from Craig. We went over the basics of git - installation, links to different references, gitconfig. I learned several things new in today's session like gitconfig color ui and bash autocompletion. I look forward to the next session which will be after SCNA. I could use some more git skills.

http server in java - concurrency

2010-10-07T19:30:00-07:00

An HTTP server handles more than one request at a time. Since it's very much possible that more than one request can come in at the same time, the "listener" needs to run concurrently and create more tasks to process every client's request. This is the heart of the topic for today's blog post.

References used

Chapter 10: Concurrency in Effective Java 2nd Edition (thanks Colin!)
Craftsman Articles #6-10 (thanks Justin!)
LinkedList, ExecutorService in Java Doc (thanks Sun?)
Decorator Pattern in Head First : Design Patterns

The synchronized keyword

"Synchronization is required for reliable communication between threads as well as for mutual exclusion."

In Java, synchronized is used to "ensure that only a single thread can execute a method or block at one time. This allows multiple threads to observe objects in a consistent state as well as ensure each thread knows about the modifications other threads have made when entering in the synchronized method.

"...synchronization has no effect unless both read and write operations are synchronized."

One way for threads to communicate with each other is the use of a boolean (or any data type). In the main thread, the boolean is created and initially set to false, and the child thread is started and will continue to run until it sees the boolean set to true. Both reading and writing to this boolean needs to be synchronized, otherwise, it's possible for the child thread to never see the new value for the boolean set by the main thread.

"...when multiple threads share mutable data, each thread that reads or writes the data must perform synchronization."

The author explains the best way to avoid inconsistent state is to "confine mutable data to a single thread." Unfortunately, the way things are looking, I will need mutable data to be shared across threads, so I will most likely be using a form of synchronization.

Prefer executors and tasks to threads

Starting with Java 1.5, java.util.concurrent was added to the Java platform and offers an Executor Framework, a flexible interface-based task execution facility. This has many advantages over creating and managing threads on your own. For example, it can:

wait for a particular task to complete
wait for a collection of tasks to complete
wait for the executor service's graceful termination to complete
retrieve the results of each task

The Effective Java author states that you should not write your own work queues nor should you work directly with threads. It sounds to me, like working with threads is a thing of the past. Thread served as both the unit of work and mechanism for executing it.. Now they are separate with the Executor Framework. task is the unit of work and executor service is the mechanism for executing tasks (Item 68, p.272).

Fortunately, Colin will bring a book called Java Concurrency in Practice which goes into more detail about the Executor Framework. I look forward to using this as a way to create and manage tasks (e.g., responding to client requests, reading static files).

Decorator Pattern

The Decorator pattern is used in implementations of the List interface. Collections.synchronizedList method wraps a list in a decorator to prevent accidental, unsynchronized access to the list when making structural modifications (e.g., adding/removing elements).

I read the Decorator Pattern in Head First and didn't know the pattern was used for concrete components of InputStream class (e.g., FileInputStream, StringBufferInputStream, BufferedInputStream), but it makes sense now. I recommend reading the example that they show in the book if you want to know more about the pattern.

Observer Pattern

Note: While this pattern does not tie into the concurrency topic, it was mentioned in the Concurrency chapter of Effective Java, so I thought I'd mention it.

The Observer pattern is a loose coupling relationship between a publisher and one or more subscribers. The publisher (or subject) maintains a list of the subscribers (or observers) and notifies them when there are changes within the publisher. The subscriber has the ability to subscribe to a publisher to receive notifications and unsubscribe if it no longer wants notifications. While Java has a built-in Observable class, it's a class and therefore an existing class that extends another class cannot use Observable. To make things worse, it's impossible to utilize the built-in Observer because there is no Observable interface. In the end, if the built-ins do not work for you, you can easily roll your own implementation.

http server in java

2010-10-06T21:59:00-07:00

To describe in detail, I am to write an HTTP server that does the following:

uses only standard Java libraries
no porting of other HTTP server libraries (e.g., WEBrick, Micah's own HTTP server)
cannot view the source of said HTTP server libraries
must be written TDD style
serves multiple clients simultaneously
serves static files and all default MIME types
when navigating to a specific path, it will display POST params on page

I came up with a list of common MIME types and HTTP status codes here and here. I may store these in HashMaps.

Here's what I believe needs to happen in order to create this server:

create a simple HTTP client to test server
create a service that handles client socket connection requests (multi-threading will be involved)
create classes to handle request and response
support GET/POST request methods
possibly create a route manager to handle paths

Resources that will be helpful:

It's all about Sockets

During today's research and a bit of spiking, I discovered that the lowest level component used to communicate between two computers is a socket. Recalling a story I worked on a couple of weeks ago, my time server listened for a client's connection request. Once the connection has been made, it used the socket's getOutputStream to write a message that included the current time, and closed the socket connection. This message during the telnet demo displayed on the console.

A socket is considered to be a reliable connection for transmission of data between two hosts. Fortunately, Sockets and ServerSocket are part of the standard Java library. I avoid the need to worry about packet encoding, lost and retransmitted packets, and unordered packets which Socket class handles for me.

I wrote a test to see what the difference was between the socket on the server side and the socket on the client side. Sure enough this is the difference:

// client socket
<Socket[addr=localhost/127.0.0.1,port=7546,localport=60989]>

// server socket
<Socket[addr=/127.0.0.1,port=60989,localport=7546]>

I tried connecting to the time server using the browser and to my surprise, it displayed the time message in plain text. Bingo, this has to be the way to send responses. I looked at the javadoc for Socket class and sure enough, it supports these methods: getInputStream() and getOutputStream(). These two methods should allow the server to read the client's request and send the appropriate response to the client. During trial and error, I found out the request doesn't come packaged in a pretty format like it does in Rails or any other web framework. The request has to be read in bytes, parsed, and the response written out "by hand" and sent to the client. Fortunately though, both protocols are standardized and defined by RFC 2616 (aka HTTP/1.1):

        Request       = Request-Line              ; Section 5.1
                        *(( general-header        ; Section 4.5
                         | request-header         ; Section 5.3
                         | entity-header ) CRLF)  ; Section 7.1
                        CRLF
                        [ message-body ]          ; Section 4.3

       Response      = Status-Line               ; Section 6.1
                       *(( general-header        ; Section 4.5
                        | response-header        ; Section 6.2
                        | entity-header ) CRLF)  ; Section 7.1
                       CRLF
                       [ message-body ]          ; Section 7.2

Request Protocol

It has three components:

method-URI-Protocol/Version (e.g., POST /index.html HTTP/1.1)
request headers (pairs of name:value)
entity body (e.g., POST params)

An odd, but important part of the request is the blank line after the request headers and before the entity body. This blank line separates the two. If you find a blank line before let's say the method-URI-Protocol/Version or request headers, it should be ignored.

Response Protocol

It also has three components:

Protocol-Status Code-Description (e.g., HTTP/1.1 200 OK)
response headers (pairs of name:value)
entity body (e.g., raw html)

Same important blank line applies here as well. It must be in between response headers and entity body. Any respectable browser will do that for you.

To be continued

Now I need to start spiking and get some request/response actions going between client and server.

end of iteration recap and apprenticeship challenge

2010-10-05T15:00:00-07:00

End of Iteration 11

I am happy to say I've finished all stories for Iteration #11 and managed to catch and finish the Tic Tac Toe project in Rails 3. Micah and I had our meeting today to talk about each story and my first apprenticeship challenge.

Fixed bug in statemachine gem

Earlier in Iteration #11, I fixed a small bug in the statemachine gem and submitted a pull request. Micah applied the fix and pushed out a new version. I demoed the Java generator and it correctly generated the states.

Limelight Styles Parser

This zero point story took a bit more time than I initially thought. While most of the features were implemented from Iteration #10, it still needed to support comments. As described in my earlier posts, when the parser finds a '#' (and not in a string), it transitions into a purgatory state where all characters are discarded. It waits until it sees new line to get back to the previous state from purgatory state and resume parsing the file. When I picked up Justin from the train station, this morning and told him about the solution, he helped me discover a problem I had. In order to determine that the '#' was not in a string, I stored the previous character. This worked great to resolve style attributes that have '#' in them (e.g., #FFFFFF), but does not fare well when the '#' is not first character of a string. Fortunately, Micah let it slide since it's not a common situation you'll find in a styles.rb file.

FitNesse/SliM

This also took longer than I expected. I needed to figure out the DSL that is SliM. Fortunately, from the SliM Overview video and some trial and error and I was able to get through and create tests. Micah pointed out during the demo, that the acceptance tests should be easy to understand and anyone should be able to know what it's testing. I had stored the expected output in files, but I could have easily stored them in the SliM table. Using the literal text markup, I could have stored multi-line text.

Tic Tac Toe in Java

This story was easier than I thought and using my Ruby version as reference, I was able to easily write the game test first. I explained to Micah about the mocks I created for both stdin and stdout and ran a quick demo via console.

Tic Tac Toe in Rails 3

Finally, Tic Tac Toe in Rails 3 was a nice story to work on. I skimmed (no pun intended) through the Edge Rails Guide for Rails 3 and learned more about Rails 3, much more than I did the first time around. I ended up eliminating the game engine duplication as described in this post, learned how to test controllers, understood more about models, and learned more about the new RSpec (e.g., executing specs via bundler). I demoed the game and it worked like my WEBrick implementation.

Apprenticeship Challenge

My apprenticeship challenge is to write an HTTP server that will serve default MIME types as well as parse GET request params. More on that this week.

return of tic tac toe on rails 3

2010-10-04T21:59:00-07:00

From The Craftsman 62, The Dark Path:

"It's not so much choosing the right tests, Alphonse; it's about realizing that you are trying to solve the wrong test."

If you recall from a previous blog post, I did not write tests for my Rails 3 version of Tic Tac Toe, so I'm re-writing it TDD style. I should have known better to write tests. Most of the code was a straight copy from my WEBrick version, so I thought it did not need to be tested, but there were some things that were modified to work in Rails that should have been tested. For instance, I had ditched my own config class for Rails config and moved all UI methods into the controller. Instead of moving the methods to the controller (it should not belong in there), I should have "moved" them to the model and test drive from there.

Game Engine - No longer duplicated

For the longest time, I was trying to figure out how to make the same game engine that worked for all UIs, work for Rails 3. I thought about creating a gem and eventually went with the idea of just copying the game engine into the lib directory, but that in it of itself is a smell. Today, I'm happy to say the game engine is one again. This probably goes against Rails convention, but the Rails 3 project points to the game engine that is not within the project. The game engine sits happily outside on the same level as the Rails 3 project. I created a namespace, TicTacToeEngine and wrapped the module name around all classes. So the directory structure looks more gem-like (e.g., tic_tac_toe_engine.rb and tic_tac_toe_engine directory). Through this process, I discovered I was able to keep the config class and used it instead of Rails 3 config which is simply a global constant anyway.

Re-visiting MVC architecture

Michael Hines puts it nicely by saying the following in his blog post:

In MVC the role of a model is to maintain the domain logic. The role of the view is to make the model easy for the user to understand and match his mental state. The role of the controller is to respond to events by telling the view and/or the model what to do with that event. It’s not the controller’s job to know how to do it, though.

My Game controller knew too much by knowing how to do the job of running Tic Tac Toe. It really should be the model that knows how to do the job. Colin mentioned this as well that he usually places domain logic inside models.

CSV is evil, YAML to the rescue

I had an issue running the console version of the game via Ruby 1.9.2. It had something to do with the changes made to CSV class. I'm not sure why I didn't think of this sooner, but I scrapped CSV for the simple-to-use YAML. Having since used YAML in Rails projects, I knew this was alternative to the evil CSV.

To be continued

I still need to test drive the model, but I should be good to go for the deadline. More on that tomorrow.

fitnesse with slim and tic tac toe in java

2010-10-03T21:59:00-07:00

My FitNesse with SliM story is complete. Albeit, not creating many acceptance tests for the Limelight styles parser, I found the FitNesse's DSL to be a tad difficult. Fortunately, Uncle Bob posted the SliM overview video, where I repeatedly watched the video to get every little detail. I had problems defining the classpath, figuring out where fixtures are supposed to go, figuring out how to implement the style table, and figuring out how to compare expected and actual.

Fixtures

I ended up putting the fixture in a separate package under the main package. The fixture class name ended up being the same as the class name that contains the main method. I'm not sure what's the convention for this sort of thing, but it works, so I'm happy.

Expected and Actual Comparison

I initially tried to approach the problem by doing a file comparison, but could not find a standard Java library that does file comparison. I ended up with a solution where the expected Java code are stored in files. Upon comparing expected and actual, the appropriate file is read and stored into a string, then a string comparison is made. This worked out to be the best way.

Tic Tac Toe in Java

My Tic Tac Toe in Java implementation is complete. The time I lost working more on FitNesse, I gained back by completing this implementation earlier than my estimation. Since I've already went through the trials and tribulations of Tic Tac Toe, it was not difficult to write the game in another language. Nothing I've written in Ruby for my Tic Tac Toe, could I not port into Java. I was rather surprised at how easy it was to port. One thing I ran into trouble was mocking standard input/output. I ended up with a solution that I'm somewhat pleased. The standard input is still a little icky, but I managed to create a workaround solution. Take a look:

StdIn mocking

public ByteArrayInputStream createInputStream(String string)
{
  return new ByteArrayInputStream(string.getBytes());
}

public StdUI createStdUI(String string)
{
  ByteArrayInputStream bs = createInputStream(string);
  return new StdUI(bs, new MockPrintStream());
}

StdOut mocking

// MockPrintStream.java

import java.io.PrintStream;

public class MockPrintStream extends PrintStream
{
  public MockPrintStream()
  {
    super(new MockOutputStream());
  }

  public void write(int some)
  {
  }

  public void println(String message)
  {
  }
}

// MockOutputStream.java

import java.io.OutputStream;

public class MockOutputStream extends OutputStream
{
  public void write(int something)
  {
  }
}

The code is available on my GitHub repo.

fitnesse and acceptance tests

2010-10-01T10:00:00-07:00

One of my stories calls for creating a FitNesse/Slim tests for my Limelight styles converter. I went to the FitNesse website to read about FIT (Framework for Integrated Testing), what FitNesse provides to the user, and how to get it set up on my machine. The installation process is super simple. Place the jar file in an empty directory and run:

[~/local/fitnesse] java -jar fitnesse.jar -p 34863
FitNesse (v20100303) Started...
    port:              34863
    root page:         fitnesse.wiki.FileSystemPage at ./FitNesseRoot
    logger:            none
    authenticator:     fitnesse.authentication.PromiscuousAuthenticator
    html page factory: fitnesse.html.HtmlPageFactory
    page version expiration set to 14 days.

It creates a FitNesseRoot folder and launches the webserver. Navigating to http://localhost:34863 opens up the FitNesse homepage.

In the diagram above, you'll notice two different types of test systems: Slim and Fit. Slim (Simple List Invocation Method) is what I'll be using with FitNesse. Fit is the older test system (and not really used anymore?).

The basic idea is to create a fixture that connects the test tables with actual code that does the real work. In my case, I'll need to create a fixture that reads from the test table and call from my LLStyles class to do the real work, that is, parse the Limelight styles.rb and display Java code. Since the LLStyles class currently displays the code to stdout, I'll need to add support of returning the Java code from the method. More on this later.

Agile Software Development, Principles, Patterns, and Practices

I did a bit of reading a couple of nights ago and went through the list of XP practices:

Customer Team Member
User Stories
Short Cycles
Acceptance Tests
Pair Programming
TDD
Collective Ownership
Continuous Integration
Sustainable Pace
Open Workspace
The Planning Game
Simple Design
Refactoring
Metaphor

I like how I see all of these practices in place at 8th Light. One interesting tidbit I read regarding "open workspace" was how a "war room" like environment doubles productivity. You'd think it would be distracting to hear other people's conversation, but it's actually nice to hear what's going on with the other projects and people can chime in if someone needs help or a rubber ducky.

working on stories

2010-09-29T16:00:00-07:00

This morning, I started off with a quick fix to my Tic Tac Toe project. When I updated JRuby to 1.5.3, I wanted to run my Limelight Tic Tac Toe to make sure it was working. To my surprise, the Negamax player kept erring out. Upon looking at the code I found out I got away with using Object#returning, a Rails method, inside my NegamaxPlayer class. I'm not sure how that was possible. When I ran the specs, all Negamax player tests were failing. I then discovered Object#returning is a Rails method via Google search. Fortunately, this was an easy fix. Just get rid of Object#returning.

def indexes_of_max(array)
  max = array.max
  indexes = []
  array.each_with_index do |e, i|
    indexes << i if e == max
  end
  indexes
end

Specs are happy and the change is now pushed to the GitHub repo.

Pull Request accepted for Statemachine gem

I'm happy to say this is probably my first official open source contribution. It wasn't a significant fix or improvement to the library, but fixed a very minor issue when generating statemachine code in Java. Check it out.

Comment support in Limelight Styles Parser

I thought about adding a superstate, but decided to handle it on my own through custom code. Comments are different than the other characters because once you enter in a comment, every character is part of the comment until new line. So my transition list is a bit longer. I trap comments for all states into a comment_mode state where I don't act on any character except new line.

def generate_parser_sm
  sm = Statemachine.build do
    trans :nothing, :comment, :comment_mode, :enable_comment_flag
    trans :nothing, :whitespace, :nothing
    trans :nothing, :new_line, :nothing
    trans :nothing, :open_curly_brace, :nothing, :error
    trans :nothing, :close_curly_brace, :nothing, :error
    trans :nothing, :alphanumeric, :name, :add_to_name
    trans :name, :comment, :comment_mode, :enable_comment_flag
    trans :name, :alphanumeric, :name, :add_to_name
    trans :name, :whitespace, :finding_curly
    trans :name, :new_line, :finding_curly
    trans :name, :open_curly_brace, :finding_attributes, :create_style
    trans :finding_curly, :comment, :comment_mode, :enable_comment_flag
    trans :finding_curly, :whitespace, :finding_curly
    trans :finding_curly, :new_line, :finding_curly
    trans :finding_curly, :open_curly_brace, :finding_attributes, :create_style
    trans :finding_attributes, :comment, :comment_mode, :enable_comment_flag
    trans :finding_attributes, :whitespace, :finding_attributes
    trans :finding_attributes, :new_line, :finding_attributes
    trans :finding_attributes, :close_curly_brace, :nothing
    trans :finding_attributes, :alphanumeric, :attr_name, :add_to_attr_name
    trans :attr_name, :comment, :comment_mode, :enable_comment_flag
    trans :attr_name, :alphanumeric, :attr_name, :add_to_attr_name
    trans :attr_name, :whitespace, :finding_attr_value
    trans :finding_attr_value, :comment, :comment_mode, :enable_comment_flag
    trans :finding_attr_value, :whitespace, :finding_attr_value
    trans :finding_attr_value, :alphanumeric, :attr_value, :add_to_attr_value
    trans :attr_value, :comment, :comment_mode, :enable_comment_flag
    trans :attr_value, :alphanumeric, :attr_value, :add_to_attr_value
    trans :attr_value, :close_curly_brace, :nothing
    trans :attr_value, :whitespace, :attr_value
    trans :attr_value, :new_line, :finding_attributes, :add_attribute
    trans :comment_mode, :new_line, :comment_mode, :disable_comment_flag
    trans :comment_mode, :alphanumeric, :comment_mode
    trans :comment_mode, :whitespace, :comment_mode
    trans :comment_mode, :open_curly_brace, :comment_mode
    trans :comment_mode, :close_curly_brace, :comment_mode
    trans :comment_mode, :comment, :comment_mode
    startstate :nothing
  end
  sm.to_java(:output => SRC_DIR, :name => "LLParser", :package => "limelight.parser")
end

Two things are involved. In the context class, I created a privatized variable that stores the comment flag. This flag determines whether or not the parser is in comment_mode or not. Right before it sets the flag to true when it enters the comment_mode, I set the state to a privatized variable in the main file (LLStyles.java). This is the state right before it enters comment_mode state, so I can go back to the state when I'm out of comment_mode state.

I'm sure there's a better solution, probably by using superstate. But it works and I'm pretty happy about it. It handles this styles.rb file just fine.

Tic Tac Toe Java

I'm now working on the Java implementation of Tic Tac Toe. Fortunately, I don't have to implement 4x4, therefore I do not need to support MongoDB. I also do not need to support additional computer players other than the Negamax player. This is the list of classes I'll need to implement:

Board
Game
HashCache
HumanPlayer
NegamaxPlayer
NilCache
Player
UI

new iteration, new stories

2010-09-28T21:59:00-07:00

End of Iteration Meeting

Micah and I had our end of iteration meeting. I'm happy to say I've completed all stories for this iteration with one minor exception: I still need to complete the Tic Tac Toe Rails 3 app. It was marked as zero points since it's unfair to bill the customer when it was clearly my mishap. So I will work on it this iteration. Also, I mentioned in my previous post, I forgot to support comments in the Limelight styles.rb converter, so I'll have to handle that as well. I demoed the other stories to Micah and he seemed happy with all of them. As I get in the habit of thinking in story points, my story estimates and completion should improve.

Iteration #11 Stories

Here's what in store for Iteration #11:

Fix a generator bug in statemachine gem
Write FitNesse/slim tests for the styles.rb translator
Write a Tic Tac Toe implementation in Java

The last two will be the most challenging for this iteration. I don't think the Tic Tac Toe story will be too challenging as I've written it already in two other languages, rather it will just take a bit more time writing in Java. I think I'm more comfortable writing in Java, but there are some things that force me to look in the Java API docs.

Pull Request for statemachine gem

Later in the day, I actually fixed the generator bug in the statemachine gem and sent Micah a pull request. It was a rather easy fix. You can see the pull request here.

Comment Support for Limelight styles.rb converter

This feature is difficult to implement than I previously thought. Since the styles.rb file is being read byte by byte, it needs to know when it enters the comment state. Once it finds the # (Ruby's comment), it'll need to skip all characters before the new line. I'm not sure whether or not to use superstates with history state or if I get away with just adding more transitions. Regardless, I'll need to keep track of when the fileInputStream goes into comment mode and when it leaves comment mode (upon detecting new line). I'll look into this more tomorrow.

using state machine for parsing - part 2

2010-09-28T13:00:00-07:00

On Sunday, I covered the very basics of a statemachine to create a parser. Yesterday and today, I worked on the code to create that parser. Using Micah's statemachine gem, I wrote a Ruby script that generates the statemachine code by listing all transitions.

begin
  require 'statemachine'
  require 'statemachine/generate/java/java_statemachine'
rescue LoadError
  require 'rubygems'
  require 'statemachine'
  require 'statemachine/generate/java/java_statemachine'
end

SRC_DIR = File.expand_path(File.dirname(__FILE__) + "/src/")

def generate_parser_sm
  sm = Statemachine.build do
    trans :nothing,            :whitespace,        :nothing
    trans :nothing,            :new_line,          :nothing
    trans :nothing,            :open_curly_brace,  :nothing,            :error
    trans :nothing,            :close_curly_brace, :nothing,            :error
    trans :nothing,            :alphanumeric,      :name,               :add_to_name
    trans :name,               :alphanumeric,      :name,               :add_to_name
    trans :name,               :whitespace,        :finding_curly
    trans :name,               :new_line,          :finding_curly
    trans :name,               :open_curly_brace,  :finding_attributes, :create_style
    trans :finding_curly,      :whitespace,        :finding_curly
    trans :finding_curly,      :new_line,          :finding_curly
    trans :finding_curly,      :open_curly_brace,  :finding_attributes, :create_style
    trans :finding_attributes, :whitespace,        :finding_attributes
    trans :finding_attributes, :new_line,          :finding_attributes
    trans :finding_attributes, :close_curly_brace, :nothing
    trans :finding_attributes, :alphanumeric,      :attr_name,          :add_to_attr_name
    trans :attr_name,          :alphanumeric,      :attr_name,          :add_to_attr_name
    trans :attr_name,          :whitespace,        :finding_attr_value
    trans :finding_attr_value, :whitespace,        :finding_attr_value
    trans :finding_attr_value, :alphanumeric,      :attr_value,         :add_to_attr_value
    trans :attr_value,         :alphanumeric,      :attr_value,         :add_to_attr_value
    trans :attr_value,         :close_curly_brace, :nothing
    trans :attr_value,         :whitespace,        :attr_value
    trans :attr_value,         :new_line,          :finding_attributes, :add_attribute
    startstate :nothing
  end
  sm.to_java(:output => SRC_DIR, :name => "LLParser", :package => "limelight.parser")
end

if $0 == __FILE__
  generate_parser_sm
end

As said previously, calling the to_java method will generate the statemachine code in Java. Had I've been writing this parser in Ruby, I would have not needed to generate the code. I could have left the build method intact and call it in another Ruby class, and have the code generated in memory, on the fly. According to Eric Smith, this is how it's done for one of their client projects.

Typically how it's done when working in Ruby

require 'rubygems'
require 'statemachine'


module ParserStatemachine
  def self.create_with(parser_context)    
    return Statemachine.build do
      trans :code, :other, :code, :print
      trans :code, :pigpen, :code, :print
      trans :code, :endl, :new_line, :print
      trans :new_line, :other, :code, :print
      trans :new_line, :pigpen, :in_comment
      trans :new_line, :endl, :new_line, :print
      trans :in_comment, :other, :in_comment
      trans :in_comment, :pigpen, :in_comment
      trans :in_comment, :endl, :new_line
      startstate :new_line
      context parser_context
    end
  end
end

The important piece of information I needed to figure out was what to implement after the Java code was generated from the statemachine gem. After reading through the code and help from Eric, I found out it was the context interface class that needed to be implemented.

public interface LLParserContext
{
  // Actions
  void addToAttrName();
  void addToAttrValue();
  void addToName();
  void error();
  void findAttrValue();
  void findAttributes();
  void findAlphanumeric();
}

I created a class that implemented LLParserContext and inside this class, it wrote style names and their attributes to the output string. When this all said and done, the idea is for the jar file to spit out Java code equivalent of a Limelight styles.rb file in the console. Fortunately, with Micah's help today, I was able to able to get through this story and complete it with tests. It's amazing how much you can get done when pairing with someone else, even better when pairing with an experience Java programmer. The result of this story is shown in my GitHub repo. Unfortunately, I forgot to take into account comments, so I'm working on that issue now, but for any styles.rb file without comments, it should convert it nicely to Java code.

import limelight.styles.RichStyle;
import java.util.HashMap;

HashMap<String, RichStyle> styles = new HashMap<String, RichStyle>();
RichStyle style = null;

style = new RichStyle();
styles.put("title", style);
style.setWidth("100%");
style.setFontSize("30");
style.setFontFace("ArialRoundedMTBold");
style.setTextColor("white");
style.setHorizontalAlignment("center");
style.setBottomPadding("10");

style = new RichStyle();
styles.put("label", style);
style.setWidth("50%");
style.setFontSize("12");
style.setFontFace("arial");
style.setFontStyle("plain");
style.setTextColor("#363636");
style.setHorizontalAlignment("right");
style.setRightPadding("5");

style = new RichStyle();
styles.put("bold_label", style);
style.setWidth("50%");
style.setMinHeight("1");
style.setFontSize("12");
style.setFontFace("arial");
style.setFontStyle("bold");
style.setTextColor("#363636");
style.setHorizontalAlignment("left");
style.setLeftPadding("5");

style = new RichStyle();
styles.put("advise", style);
style.setWidth("100%");
style.setFontSize("12");
style.setFontFace("arial");
style.setFontStyle("plain");
style.setTextColor("#363636");
style.setHorizontalAlignment("left");
style.setTopPadding("10");

style = new RichStyle();
styles.put("buttons", style);
style.setWidth("100%");
style.setPadding("5");
style.setHorizontalAlignment("center");
style.setTopPadding("20");

style = new RichStyle();
styles.put("button", style);

While I did pull some hair out initially, this was a fun challenge. I really enjoyed learning about the state machine and how it can be used to represent a system. I could have used regular expressions instead of the state machine, but I felt this approach made more sense and visually I was able to understand what state, transition, event I was in. If I were working in regular expressions, I would be in regex hell, plus it wouldn't have been fun.

using state machine for parsing

2010-09-26T21:00:00-07:00

Finite-state automaton is

"a mathematical abstraction sometimes used to design digital logic or computer programs. It is a behavior model composed of a finite number of states, transitions between those states, and actions, similar to a flow graph in which one can inspect the way logic runs when certain conditions are met."

I'm working on the story that asks to create a jar file that will read Ruby code (Limelight styles.rb) and convert it into Java code. My initial approach was to use regular expressions to parse the Ruby file in several different stages. When I approached the problem to Micah this past Friday, he came up with a better solution, using a state machine to create a parser.

He gave me a nice lecture, briefly describing the Turing machine and how state machines are said to be "Turing complete", then drew a UML state machine of a turnstile system and a table listing all possible state, event, action combinations. Here is his presentation I found on the 8th Light blog.

He took it one step further and did the same thing for the parser in addition to drawing the UML diagram of the Java classes involved. The UML state machine for the parser is a little bit more involved as there are more transitions (roughly 18) but it should not be too difficult to implement since the state machine code is auto-generated.

Fortunately, Micah wrote a Ruby gem called statemachine, which I can use to generate the state machine code. It generates it in both Ruby and Java. All you need to do is provide a list of transitions (e.g., old state, event, new state, action). I looked at the tests in the GitHub repo to see how Micah generates the statemachine code in Java. I've also taken a look at the sample exercise to get an idea of how it was done in Ruby. In short, he calls the to_java method to generate Java code from Ruby. It takes three arguments: output directory, class name, and package and returns the auto-generated code in output_directory/package_name.

In my next post, I will explain what was involved after auto-generating the Java code.

thinking in story points

2010-09-23T21:00:00-07:00

In an effort to prevent missed iteration deadlines, I'm now thinking about points everyday. How many points do I have left on the board? How many points left in the week? Am I ahead or am I falling behind? Each craftsman and apprentice are to complete 2 points per day and are assigned 10 points worth of stories in a given week. Of course, my inexperience at estimating stories burdens me as I mostly underestimate stories. There are other times though where my estimate was pretty close, but unnecessary yak shaving ensued. Some stories weren't meant to be difficult, but I placed them in that category and read too much about the story topic before actually spiking out the solution. I'm hoping with enough experience and time, I will be better at estimating stories.

With that in mind, I continued working on my stories, today with the Bowling kata in Java story. Fortunately, Uncle Bob wrote a powerpoint presentation which walks you through the kata. This kata is a lot longer than the Prime Factors kata and therefore slightly more difficult. I clocked in at 15 minutes to complete the kata. At the end of the day, I picked music, and recorded the kata using Snapz Pro X and uploaded it on Vimeo. I made some typing mistakes here and there, but nothing to cry about. I'm pretty happy with it. I think I'll record the Prime Factors kata as well in the near future.

My next story is to write a jar file to will convert a Limelight styles.rb into a Java equivalent. I don't even know where to begin, so I'm going to ask Micah a lot of questions. Hopefully, he'll have time to pair and I can get an understanding of what the Java equivalent looks like, so I know where I need to go in terms of the code.

Tomorrow is going to be a busy day, with a new apprentice and a continuation of the 8LU session with Doug on DSP (digital signal processing).

socket server in java

2010-09-22T19:00:00-07:00

Note: Here's the two day recap while I dive deeper into the Java programming language.

I'm reading the source code Micah written for Socket Server in Java. It looks similar to the code for the Craftsman articles (6-9) on Object Mentor. As I'm reading the articles, I'm understanding how to code is being driven with tests and the communication involved between the two programmers as they ping pong pair program. Here are some observations:

Anonymous Inner Class to Implement an Interface

connectionCounter = new SocketServer()
{
  public void serve(Socket s)
  {
    connections += 1;
  }
};

This is probably the first time I'm seeing this. It looks similar to passing a block to a Ruby method, but not the same idea. SocketServer is an interface which require a serve method to be implemented. This block of code defines the serve method required for the SocketServer interface and creates an instance of the anonymous inner class. Pretty interesting stuff.

Importing Java Libraries

I've noticed sometimes, developers will use an '*' at the end of an import statement to indicate that they would like all libraries. I wonder when to properly use the '*' (e.g., more than 3-4 packages?) and if it's considered lazy to immediately use '*' when only using one package.

import java.io.*
import java.net.Socket;

Intentional Programming

I was unaware of the name of the concept used when calling code that does not yet exist. This is commonly used in TDD. You write code that calls a method that does not yet exist, run the test to make it fail (red), write the new method, run the test to make it pass (green), and refactor if necessary.

Race Condition

I kept hearing this term being thrown around in the office, but didn't know what it was. It basically occurs in multi-threaded environments. Asychronous events need are sensitive to the order of execution.

Running tests multiple times

While it might not be necessary to run single-threaded tests multiple times, multi-threaded tests should probably run be multiple times. In the case of Alphonse in the Craftsman articles, he saw a test pass 7/10 times. He witnessed the race condition.

Mock Object Pattern

This is a pattern I'm pretty comfortable with, after using it many times via RSpec. The act of mimicking real objects with simulated ones to test behavior.

Creating methods to make testing easier

This is an interesting technique which I learned in the beginning of my apprenticeship. Similarly, in Ruby, changing an attribute from a read-only variable to a read-write variable to make testing easier is allowed. As developers, we trust each other that we will not be changing the attribute's value directly.

Implementing the Socket Server with Micah

I missed this story in my last iteration, but made it a priority to get it done first.

As I was reading the articles and the source code to understand how socket servers were created, Micah stopped by and paired with me briefly. In what seemed like a matter of minutes, we created a simple socket server that displays a message with the current time. I'm both sad and embarassed how easy this was. Sad because I spent so much time reading material after material trying to understand how to implement a socket server. I was also trying to play catch up and re-learn how to write in Java and what methods to use. Embarrassed because the customer paid 6 points to have this story complete, where it should have really taken 1-2 points. Micah mentioned that I'm probably reading too much. This is true. I will try to focus more on spiking and then test drive the stories.

I finished the socket server at the end of the day by test driving it. I had several issues mimicking a client socket connection to the server. Fortunately, the material I learned from the Craftsman articles gave me an idea to create a thread and loop attempts to create a client socket connection until the connection is made.

Goodbye Iteration #9, Hello Iteration #10

As mentioned above, I missed my socket server story, so it's now in Iteration #10. Other stories include performing the Bowling Kata in Java and creating an executable jar file that will read a Limelight styles.rb file and output equivalent Java code. The former story does not sound too difficult. I have experience performing Prime Factors kata in front of an audience and I've seen Bowling Kata on the web, so I'll do some research. The latter story, however, sounds a bit tough. The Limelight styles.rb code looks similar to CSS or object literals in JavaScript. This will require some parsing and understanding what the Java code equivalent looks like. I'm going to have to do a lot of spiking. I wonder if this code will be used in the Limelight source?

learning java

2010-09-20T21:00:00-07:00

Note: I took some time off, and now I'm back in action.

I tried reading Effective Java, but quickly realized I needed to step back and re-read the basics. So it was a good thing Colin mentioned another book, Agile Java. When I briefly read the first chapter, I knew this was the book to start out. Here are some notes thus far.

Java is included in Mac OS X by default

Apple makes it super easy to start developing in Java and provides the latest JDK 1.6 in Snow Leopard. I thought when I was having a JUnit issue, it had something to do with the default installation of Java, but it was not.

Classpaths

The author talks about the classpath as being "one of the more confusing aspects of working with Java". I couldn't agree more. When calling javac (java compile command) or java, you may need to provide classpaths. When having more than one classpath, a delimiter is used to separate each path. For whatever reason, the delimiters are different when working in Windows and Unix systems. Windows requires you to separate paths using a semi-colon, whereas Unix requires a colon. The book examples default to Windows which I overlooked. This created some frustrating moments last week. It's also important to place appropriate quotes around paths with spaces. This is more of a Windows dilemma (e.g., Program Files).

Windows

javac -cp .;c:\some\path\to\junit.jar ClassTest

Unix

javac -cp .:~/some/path/to/junit.jar ClassTest

Specifically for Mac OS X, if you place the jar files in /Library/Java/Extensions or ~/Library/Java/Extensions, you do not need to explicitly provide the path.

A bit of Java History

Charles Nutter recently posted a great article on Oracle vs Google lawsuit and gives great detail about the history of Java. I did not know Java is an open and closed platform and how Sun unfortunately did not open source the test kit to allow developers to create fully compatible open-source Java. Charles further piqued my interest in Java and I look forward to learning more about it.

Java REPL

There's a Java REPL called Beanshell. I can't tell if it's still in active development. The notes say it works for 1.5, so I'm assuming, it's not, but seems like a nice tool to have when in need to interpret code.

Uncle Bob's Craftsman Articles

Uncle Bob's Craftsman articles has a lot of information about Java and TDD. I like the way he story tells about an apprentice's experience. It only seem natural for an apprentice to read about an apprentice.

Socket Server in Java

Fortunately, I found an article on writing Socket Server in Java (from Uncle Bob as well), thanks to Justin's blog post. I'll be up all night reading it and try to get this four point story done.

Java Online Resources

rails 3 woes - part 2

2010-09-15T21:00:00-07:00

I finished the Tic Tac Toe on Rails 3 today. There are still some unconventional code that should be translated to Rails. Things like generating HTML from helper methods where I can use the tag helpers instead. Here are some of the gotchas I experienced.

Rails 3 lib directory does not autoload

This is an intentional feature in Rails 3 and was introduced to make Rails "behave more closely" to Ruby projects. If you want your application to behave like it did in Rails 2.x, just add the config.autoload_paths in the application.rb file.

module RailsApp
  class Application < Rails::Application
    config.autoload_paths += %W(#{config.root}/lib)
  end
end

Rails autoload convention - unfamiliar territory

I guess I was unfamiliar with the way Rails loaded up the game engine module in my lib directory (once I got the autoload working). I had my own config class that was part of the game engine to support configurations. The problem was that the classes under the game engine module needed to load up in a certain order (I think that might be a smell). There is a constant in one of the files that assumed another file instantiated a class and set it to that constant. So I scrapped that configuration set up and use Rails instead. I didn't quite understand where to put constants as things have slightly shifted in Rails 3 config, so I placed it in the environment.rb file:

TTT_CONFIG = {
  :boards  => { :'3x3' => { :active => true, :cache => :hash, :on => "3x3.jpg", :off => "3x3_dim.jpg" },
                :'4x4' => { :active => true, :cache => :mongo, :on => "4x4.jpg", :off => "4x4_dim.jpg" } },
  :pieces  => { :o => 'O',
                :x => 'X' },
  :players => { :human => { :value => "human",       :on => "player_human.jpg",    :off => "player_human_dim.jpg" },
                :easy  => { :value => "easy",        :on => "player_easy_cpu.jpg", :off => "player_easy_cpu_dim.jpg"},
                :med   => { :value => "med",         :on => "player_med_cpu.jpg",  :off => "player_med_cpu_dim.jpg"},
                :hard  => { :value => "unbeatable" , :on => "player_hard_cpu.jpg", :off => "player_hard_cpu_dim.jpg"} },
  :cache => {}
}

XSS Protection is on by default

This was a brand new problem for me in Rails. As I've said in the beginning of this post, I'm generating HTML (code borrowed from the WEBrick version of Tic Tac Toe) and of course, when the HTML was handed back to the view, it converted the HTML to be safe. In order to resolve this problem, I called html_safe after each method call in erb.

<%= options_for_board.html_safe %>

What I should really do is take advantage of Rails tag helpers and some CSS. As Craig pointed out, the mouse events that I've implemented in JavaScript can be done in CSS. If only I had some more time, I could try it with CSS...

Protect from Forgery

I'm not sure if this was not on by default in Rails 2.x, but it is for Rails 3. In the ApplicationController, it's the very first thing in the class. Since this is just a game, I commented it out, but if I were working on a real app, I would not do this and find another way. By commenting this out, I'm probably doing something wrong.

I had fun with Rails 3. I hope to work on it more and I plan to read Diapora project soure code which happens to be Rails 3.

rails 3 and new iteration

2010-09-14T21:00:00-07:00

New Iteration

Micah and I went down to the Chicago office because he had a meeting with a client. We had the end of iteration meeting in the afternoon, and unfortunately, I did not complete the Rails 3 Tic Tac Toe in time. I was stuck trying to figure out why the game engine library was not loading automagically (Rails convention) when copied to the lib directory. It kept giving me uninitialized constant error messages. Once I have that figured out, I just need to change the erb files from the WEBrick project, tweak things here and there and it should be ready to go. I'm really hoping for a 2-point story. It should optimistically take half a day to do.

Since I missed this iteration and some in the past, Micah sent me a spreadsheet to better estimate my stories. It uses PERT estimate calculation. PERT stands for Program Evaluation and Review Technique and was used by the U.S. Navy in the 1950's. Instead of giving one estimate for a story, you give three:

Optimistic Estimate - if absolutely everything went according to plan
Realistic Estimate - real world, greatest chance of success
Pessimistic Estimate - wildly pessimistic, all wild obstacles included

With these estimates, it'll calculate the true estimate based on this formula:

(O + 4R + P) / 6

This will calculate the mean and it rounds to the nearest point. For example, I said one particular story has an optimistic estimate of 2 points, realistic estimate of 3 points, and pessimistic estimate of 4 points. The mean for these estimates come out to 3. This is my estimate for that story.

Hopefully, this will aid me in providing accurate estimates for the stories.

What's New in the Next Iteration

Last week, I said I will be working in the Java language. Here are the new stories for this iteration:

write a telnet server in Java
write the Sieve of Eratosthenes in Java

The Rails 3 Tic Tac Toe story is part of this iteration as well. I'm excited to be learning a new language. I took a Java class a while back and familiarized myself with the language. Hopefully, I'll be able to recall most of the information. I'm also borrowing Effective Java from the 8th Light library, so I'll need to do some reading this week.

rails 3 woes

2010-09-14T13:00:00-07:00

Rails 3 Woes

It was an honorable day of yak shaving yesterday. I read bits and pieces of Jeremy McAnally's Rails 3 Upgrade Handbook, official Rails 3 Guide, Rails 3 Tutorial, Rspec-Rails, watched some of the official Rails 3 screencasts, and Google searched aplenty. I ran into some issues which I'd like to share with you.

RSpec 2

RSpec 2 is required for Rails 3. The latest version as of this writing is RSpec 2 Beta 22 which was released a couple of days ago. Unfortunately, I could not get it to generate specs when generating controllers. The instructions are well written, but it doesn't want to generate specs for me. This was also after trying to figure out how to make RSpec work when not using Bundler. I thought since Rails 3 is more modular and agnostic, I could generate a Rails app without Bundler and without Active Record. This way, I can focus on the other newer features and learn Bundler later.

Note: If you decide to not use Bundler in your Rails app, you'll have to require 'rspec-rails' somewhere for the development and test environments. Apparently, when including it in the Gemfile, it automatically requires it.

I decided to delete the Rails project and generate a new Rails app with default options. While it still didn't generate the specs for controllers, I didn't want to run into potential issues that I have not run into.

Action Dispatch - nothing new for cookies

It was not easy to find information about Action Dispatch and to see if there were any changes to the way you access in cookies in Rails 3. None of the freely available guides talked much about it in detail, so I used Rails 2.x information in the third edition of Rails from PragProg. Similar to WEBrick, all cookies are stored in a collection: WEBrick uses an array, but Rails uses a hash. The reading and writing to the cookies is as easy as:

class CookiesController < ApplicationController
  def action_one
    cookies[:the_time] = Time.now.to_s
    redirect_to :action => "action_two"
  end
  def action_two
    cookie_value = cookies[:the_time]
    render(:text => "The cookie says it is #{cookie_value}")
  end
end

Including TTT Game Engine without duplicating code

I still need to figure out how to include the TTT Game Engine without duplicating the code. It currently resides in the game_engine folder as well as inside the Rails 3 app. For now this will work, but I need to figure out a better way.

I'll talk more woes later today.

rails and effective study habits

2010-09-12T21:59:00-07:00

Rails 3

Last Friday, I was having some trouble figuring out how to add the Tic Tac Toe game engine library into my Rails 3 project without duplicating the files. A couple of ideas came to mind. I can create a symlink and point it to the game engine folder. Another idea was to create a local gem and create a rake task that vendors the game engine when running the Rails application. That way, it can have the latest version of the game engine library. I went for the latter as I didn't think symlink would work, so I started researching on how to create a gem.

I looked at one particular gem project, Specjour, to see how Sandro created the gem folder structure layout. He uses a Ruby gem called jeweler, which is used to create and manage gems. It seemed like an easy way to create a gem. There's a bin folder which contains a file named after the gem and includes the command line options. I'm sure I'm missing the intricate details, but for the most part, everything else looks the same as my game engine library. The only thing I'll have to do is create a namespace for the game engine and include all classes under the new namespace. This shouldn't hard to do, as I've done it before in a Rails project.

With regards to session handling, which I'll need for multiplayer support, it looks like I'll need to read up on Action Dispatch. This is a new component in Action Pack in Rails 3 that handles the following:

Request handling and parameter parsing
Sessions, Rails' flash, and cookie storage
File uploads
Routing, URL matching, and rescuing errors
HTTP conditional GETs
Client response and HTTP status code

Taken from Rails 3 Upgrade Handbook

I will into these in more depth tomorrow.

Effective Study Habits

I read an article on the New York Times referenced by Mark Needham and Michael Feathers that talks about the common wisdom about good study habits and how they contradict with new, simple techniques. Here are some quotes that I find interesting:

Varying the type of material studied in a single sitting - alternating, for example, among vocabulary, reading and speaking in a new language - seems to leave a deeper impression on the brain than does concentrating on just one skill at a time. Musicians have known this for years, and their practice sessions often include a mix of scales, musical pieces and rhythmic work. Many athletes, too, routinely mix their workouts with strength, speed and skill drills.

I always thought my inability to study a single piece of material in a single sitting was a flaw in my study habits. Like Mark, I can't sit down and study something for more than a couple of hours. I think my mind wants to wander and study something else on my long list of interesting topics.

"The idea is that forgetting is the friend of learning," said Dr. Kornell. "When you forget something, it allows you to relearn, and do so effectively, the next time you see it."

It may be that the brain, when it revisits material at a later time, has to relearn some of what it has absorbed before adding new stuff - and that that process is itself self-reinforcing.

The harder it is to remember something, the harder it is to later forget. This effect, which researchers call "desirable difficulty," is evident in daily life.

So when you forget something you've learned, you're essentially creating a "desirable difficulty." When you relearn the material, it will be more difficult to forget. I wonder how pomodoros and varying materials per session can help absorb more of the material.

prime factors kata recital, dsp

2010-09-11T08:00:00-07:00

Prime Factors Kata and DSP Session

I was the opening band for today's Lunch 'n Learn, right before Doug presented his first session in Digital Signal Processing. I didn't talk during the kata which Micah pointed out at the end. It wasn't necessarily a bad thing; he mentioned other people like to talk while they walkthrough, while others play music. I chose the third option. Anyway, I started off on the right foot, showing off some cool keyboard shortcuts on RubyMine, but on the test, "it factors 9", I missed a crucial step and had a test hanging. It wasn't suppose to hang. A couple of people were hinting that I was missing something and I was able to quickly recover. It's funny, you practice so many times alone, but when you're performing in front a live audience, you choke. And I choked. It was fun though. I received several compliments and I felt pretty good about towards the end.

We went right into Doug's session on DSP. He began with the basics: what is sound? sample rate? bits? It was an interesting topic for me since I'm interested programming in audio (e.g., Max/MSP) and have some experience recording live music with a DAT recorder. Our homework is to write an audio filter similar to what karaoke machines do to filter out the vocals. He gave us a sample XCode project and told us we should not have to write more than one line of code. I'm not sure what that means right now. I'll have to take a look at the code in depth (C++) to figure out what he's talking about.

Putting WEBrick TTT to the Test

I started up my WEBrick version of Tic Tac Toe and asked Brian to try out the game, while I also played several games. It worked without problems except Brian found a bug where clicking the empty spaces after the game was over, it displayed "Internal Error". I looked into the issue and it's the fact that the rhtml when converted tries to use the current player's piece. Since the current player was not being set it was trying to access the piece attribute on a nil object. I fixed the problem and added a test to make sure this problem won't happen again.

I'll be focusing on the Rails 3 version of Tic Tac Toe this weekend. I'm reading the edge version of the Rails Guide to guide me along the way.

practicing prime factors kata

2010-09-09T21:59:00-07:00

I spent most of today practicing the Prime Factors kata in preparation for tomorrow's "recital". I will be performing the kata in front of the other 8th Lighters during the Lunch 'n Learn. I feel pretty confident about it after working through many times over. I lost count and can't tell how many times I've done it, but it feels good to know that I can now outpace Uncle Bob ;-). Of course, katas aren't about speed, but about the practice and form. I think it helps me get into the TDD groove by starting off with nothing, but a test. This was a fun exercise and I look forward to practicing more katas like the Bowling kata and continually improve my TDD skills.

For the rest of the day, I worked on finishing up the multiplayer support for WEBrick Tic Tac Toe. Unfortunately, it was not smooth sailing as there was a small hiccup along the way. For some reason, WEBrick cookies (or cookies in general) do not like cookie values as , , , , , , , ,. Google Chrome showing me the cookie value is stored correctly, but when retrieving the cookie value on the GET request, it returns as nil. To resolve the issue, I temporarily replace empty spaces with dashes and revert it back on the GET.

# when saved to a cookie
@board.serialize.gsub(/\s/, '-')

# when retrieved from the request
board_state = cookie_value(request, "board_state").gsub(/-/, ' ')

Now everyone's happy and I ran a simple test - ran an instance of the game on Google Chrome and Firefox and it seems to play independently. Awesome. I refactored the servlet as I found some duplicate code and refactored the tests. This looks ready to be delivered to the customer. I'll take a look at the code again this weekend just to double check I'm not missing anything while I work on the Rails 3 version. That's going to be a lot of fun.

webrick sessions for ttt multiplayer support

2010-09-08T21:59:00-07:00

Pairing with Micah

Micah and I paired towards the end of the day working on the story to allow multiple players play Tic Tac Toe games on the same web server. We first did a bit of refactoring on the Board class to remove a smelly argument list for initialize method and added some class methods to be called in the story. Then Micah created a nice, lengthy RSpec test which mocks cookies and params coming in from the request and verifies that the game is resumed (WEBrick is now creating new servlet instances), makes a move (using the params value), and stores the new board state, board, and players into the request.cookies array. Of course, since we ping ponged, I wrote the feature, but it was a really nice exercise. Pair programming really helps me be more engaged than trying to figure out the problem alone. It was really nice to be able to bounce ideas around, get assistance for a good number of errors I wrote in, and mostly learn from the other person. Now that Micah gave me a great boost to this story, I should be on my way to complete this by tomorrow.

Web Heresies

I read a great article about Avi Bryant's talk on the need to re-evaluate design decisions that were unconsciously adopted into modern web frameworks. The slides go through an example to create a simple web application using recipes from Seaside:

Justification
Inspiration
Implementation
Re-implementation
Omissions

Although, the talk is several years old, it was an interesting exercise to go through and to see how WEBrick can be used to serve custom procs rather than the servlet templates WEBrick offers. While I didn't have time to re-implement my WEBrick piece for Tic Tac Toe, it'll be interesting designing it the "Seaside" way.

Rails 3 on Windows is Hard

As said previously, I coined the idea to develop the Rails 3 app on the Windows platform and I tried to get started tonight. Unfortunately, I ran into issues with Bundler and the IDE (RubyMine), so I gave up for the night. I'll try again and see if I can passed them. Another issue, not platform related is the issue of including the Tic Tac Toe game engine library without duplicating it. I may finally throw it in a gem include it in the Rails project. I'll have to see...

new stories, new iteration

2010-09-07T21:59:00-07:00

It's great to be back after a long, much needed weekend. Today, Micah (customer) and I had our end of iteration meeting where I presented a couple of stories that I worked on last week. The first story was persisting scores for player O and X in my Tic Tac Toe project. It works without problems in both console and Limelight and is stored via CSV. Micah noted that while CSV may be suitable for a project like Tic Tac Toe, in real world situations, it's best to store in a database where the possibility of file corruption is minimized. The second story was about supporting multiple players in a WEBrick environment. Unfortunately, I was unable to finish this, but had a spiked version of the game working in a single thread. Now I need to add session support via cookies, then throw it all away, and write it TDD style.

Micah created new stories for this new iteration. I'm pretty excited to be working on them. Here's the short list:

Build Tic Tac Toe on Rails 3
Practice Prime Factors Kata in Ruby

I wrote a post on how to setup Rails 3 for Windows on my other blog. Now I have a reason to build a Rails 3 app! The points for this story is not as high because I'm somewhat familiar with Rails from the past couple of weeks of work. There are changes to Rails convention in version 3, but not enough to slow me down. Maybe I should write the app on my Windows box? There shouldn't be anything different from a Unix environment. The only external library I use is mongo gem. We'll see...

I never expected Micah to give me a story to practice Prime Factors Kata. When Jim presented the kata during his Node.js/Vows.js presentation, I really enjoyed writing it. Now I need to practice it and perform the kata in front of other 8th Lighters on Friday. Yikes. I haven't presented anything yet to any of the 8th Lighters (except Micah and Paul), so this will be interesting. I hope no one shows up. I'm kidding.

I made sure to ask Micah all questions about each story before we finished our meeting. Story descriptions can sometimes be generalized, so I'm always confirming what he wants out of each story I'm not sure about. I've learned my lesson from previous communication hiccups where Micah wanted one thing and I created something else. Communication is key.

Another story not ready for this new iteration is writing some Java code. More on that next week.

Ruby Meetup

Our new intern, Brian, Li-Hsuan, and I went to the Ruby meetup. My last meetup was probably a couple of months ago, so it was exciting to be back and meet new faces and old. There were a couple of talks that night. One was on Mongoid, an ODM for MongoDB. One of the neat features it supported was geospatial 2D querying which reminded me of fuzzy searching. I don't have a reason for using Mongoid right now, but it seems to be a winner among the Mongo ORM libraries out there. The second talk was about consumer-driven contracts. I was only able to get bits and pieces of the talk. The way consumers and providers work together are similar to how customers and developers collaborate in an agile environment. Consumers write tests in RSpec or any language and push it to git submodule where the providers run the tests for consumer to make sure features that the providers are providing to the consumers aren't broken in future releases. If that didn't make any sense, that's pretty much what I got out of the talk, I apologize. Unfortunately, we had to leave immediately after the meetup to catch the train.

game change

2010-09-04T15:00:00-07:00

Yesterday, I paired with Micah to go over the problem I was having about handling WEBrick servlet instances and sessions. He asked me great questions when I told him about the problem in detail. One obvious problem, when removing the Singleton pattern in the servlet class, is the fact that the web server will instantiate a new servlet for every GET/POST request, therefore state was no longer persisting. The game engine requires an instance of the UI and I used the servlet as the acting UI. Instead of using the servlet, I could abstract the UI methods in the servlet into a new UI class and create an instance of that. I may not be thinking this through clearly, but this might work.

The real underlying problem is the game engine is running in a separate thread. Threads can be complicated and hard to work with, especially if you don't know what you're doing. I introduced a thread in the Limelight version of Tic Tac Toe and for the most part, it's been working without problems. It's a whole different game (no pun intended) when introducing threads in a web environment. It doesn't quite work the same. The game engine is unable to magically report back that the computer made its move and update the web page. I'm currently using what I call a hack where the meta tag refreshes the page, which calls the GET request every two seconds. Micah quickly spiked a method called "nonblockingplay" which only runs when it's the computer player's turn. I'll have to somehow implement the human player's side and have it work with both Limelight and WEBrick implementations. Micah also showed a neat way of using the cookie. I could store the board state inside a cookie and have the servlet resume the game. I don't currently have a way for the game to resume, but I don't think it'll be too difficult to support. This way the server does not have to hold instances of the game for every player (big performance hit when managing thousands of users) and I can lose the Singleton pattern.

TODO List

lose the Singleton pattern => servlet should not be persisting state
lose the thread implementation
create a Session class to hold all sessions on server side (for now)
later implement a way for game to resume and store board state in cookie

back to the drawing board

2010-09-02T21:59:00-07:00

WEBrick woes

It looks like my idea of combining the instances of the servlets paired with cookie sessions failed. Unfortunately, I could not figure out how to get the request prior to the instantiation of the servlet. The request has the array of cookies stored for the browser. I was unable to access the cookies therefore I was unable to return the instance tied to a cookie id. Drats.

So I tried a different approach. I tried storing the sessions in the servlet, remove the Singleton pattern, and retrieve the session by cookie id during a do_GET and a do_POST. The do_POST method is called when the user submits the game options. The do_GET method will be called during the rest of the game to get user's move as well as the computer's move via meta tag refresh hack. Since these methods are the only entry points to the servlet, I thought having the servlet handle the sessions would work. Except I forgot about one minor detail. The game object has an instance of the servlet (ui) object and sends messages to it on a different thread. Unfortunately, since I removed the Singleton pattern, each servlet triggered by GETs and POSTs will create a new instance of the servlet. This is just the way WEBrick works. Since it's creating a new instance, the game object no longer has the correct servlet object and it blows up on runtime. At this point, I'm not sure what to do next. This is my first time reading/writing to cookies. I might have the whole concept of cookies wrong, so I plan to research it more tomorrow.

As for the other story, "Persist player records", I thought about using MongoDB to persist the scoreboard scores, but decided against it to avoid writing a failsafe when MongoDB does not exist on the machine. CSV was the simple answer and I was able to implement it right into the Scoreboard class. Before, the Scoreboard class created a new hash for both players ('O', 'X'). Now it will read from the csv and write to the csv when Scoreboard#add_score is called.

webrick cookies

2010-09-01T21:59:00-07:00

While searching for some sample code on WEBrick::Cookie, I found a gist of a simple library that wraps around WEBrick to make it easier to write WEBrick code. Reading through the code, I found out how WEBrick::Cookie was instantiated and read back from the response and request, respectively:

def add_cookie(name,value)
  @res.cookies.push WEBrick::Cookie.new(name,value)
end
def read_cookie(name)
  @req.cookies.each { |c| return c.value if c.name == name }
  return false
end

The invaluable WEBrick documentation at Eric Hodel's website explained how cookies are retrieved and set:

Cookies are read in by WEBrick::HTTPRequest automatically, and are available as an Array from HTTPRequest#cookies. Cookies may be appended to the HTTPResponse#cookies Array when creating a WEBrick::HTTPResponse.

The key to implementing the story "allow multiple players to play TTT at the same time against the computer," is to create cookies for each new session and return back the servlet instance tied to that cookie. I'll need to store cookies/instances in a hash probably as a class variable (something I'm not fond of using, but apparently it's used for WEBrick servlet instantiations).

I've implemented the home page in a hokey way where I passed the DocumentRoot as part of the config hash to the new method of WEBrick::HTTPServer:

@server = WEBrick::HTTPServer.new({:Port => @port, :DocumentRoot => @document_root})

However, looking through Hodel's docs, there's another way to do the same thing:

@server.mount("/", HTTPServlet::FileHandler, @document_root)

Fortunately, the latter way is better because it allows me to use a FileHandler config called :HandlerCallback which by definition:

is a callback which is invoked before the handler for the request.

I plan to create a callback method to create the cookie when the user is on the home page. This is all still an idea in my head, so I'll need to work on the implementation tomorrow. Will post updates then.

@server.mount("/", HTTPServlet::FileHandler, @document_root, :HandlerCallback => callback_method)

rails and tic tac toe webrick

2010-08-31T21:59:00-07:00

Matt and I managed to resolve the issue I discovered last night. The solution was easier than I thought. There's yet more Rails conventions that I'm don't know about and Active Record Observer is one of them. We found the methods that sent emails to users upon registration and activation and we added a conditional logic to prevent emails from being sent. Paul wanted a couple of other minor visual fixes, so we found the erb file responsible for the view and fixed them. I pushed the changes to the repo and pulled it down from the staging server to execute a practice run and it ran without a hitch. Last I heard it's being deployed to production. I'm pretty excited to see a feature we implemented go live. This is probably first major feature I've implemented. I look forward to working on more features in the future.

During the day I spent some time reading some Ruby blog posts and picking up Well Grounded Rubyist again. Blog posts by John Nunemaker showed examples about building an object mapper and his previous post on how Ruby's method lookups work. It's interesting when including a module in a class and overriding the module's method, you're able to call the module's method by calling super. Building an object mapper is slightly more complicated and really neat at the same time. I can't think of a use case for it other than John's mention of MongoMapper. Like Paolo Perrota said in his talk though, I think it's good to understand how to do the cool tricks.

Tic Tac Toe WEBrick

I'm going to work on a story for my Tic Tac Toe project to allow multiple players each play their own game on WEBrick. This will require exchanging cookie information and possibly changing the way the servlet works. By default, WEBrick creates new instances of every servlet mounted on the server. This unfortunately doesn't work in my situation because I need to keep the same game instance. I'm going to see if I can somehow use cookies while preserving the code that I have to keep the same instance.

  def self.get_instance config, *options
    @@instance_creation_mutex.synchronize do
      @@instance = @@instance || self.new(config, *options)
    end
  end

rails demo to customer

2010-08-30T21:59:00-07:00

Today was the day to demo our Rails features to the customer. Paul and Craig were onsite with the customer and showed them the new features. While some features worked, the rake task feature had some problems. I ssh'd into one of their staging servers to look into the issue. They were running Ruby version 1.8.6 and I tested in an environment that had 1.8.7, so I installed rvm to install 1.8.7 (Note: there may be rvm issues in different Linux distros. I highly recommend checking out the rvm documentation index under Operating Systems and Packages). After installing the required gems, I ran the rake task and saw the exact same error Paul showed me earlier in the day. I opened the task using vi and added a bunch of debug statements, only to find out during the User#register call, it sends out an email. However, some users in the database do not have emails, so this is where it's blowing up (net/smtp blows up). Unfortunately, I could not replicate this error in the local environment, so I'm looking into figuring out how to register a user that does have an email. Hopefully, Paul will shed some light on what he wants me to do.

I forgot to mention last night that I watched Paolo Perrotta's A Metaprogramming Spell Book talk from Ruby Kaigi. I really enjoyed his talk as he went through the Ruby Object Model (including eigenclass) and explaining on some level how Rails framework was designed. He included snippets or "spells" from his Metaprogramming Ruby (same guy) book on his blog and explained most of them in the talk. At the end, he explained while metaprogramming should not necessarily be used freely, it's important to understand the power of what it can do. Don't be afraid to look under the hood.

node.js knockout aftermath

2010-08-29T21:59:00-07:00

Node.js Knockout was this weekend and it was a great learning experience. I highly recommend everyone to experience a high adrenaline, 48 hour coding spree at least once to really understand team dynamics, importance of communication, the advantages of pair programming, and dealing with head-on challenges.

I think everyone in the team felt the same way in the beginning of the knockout, whether or not we'll produce something out of the 48 hours. Unfortunately, we were a little unprepared. Personally, I felt very unprepared and should have read about HTML5's features like websocket, canvas, and learned how to use third party libraries like Faye, PusherApp, jQuery, etc. Unfortunately, like all of my past excuses, time was limited and I could not get to the reading. What was great though was that we didn't give up and we trekked on, learning new technologies on-the-fly, communicating with each other to come up with stories, with what events the server needs to fire, what events the client needs to fire, what bugs do we see, and so on. Although, it was a bit rough in the first six hours, we used the next twenty-seven to build something out of thin air.

What the heck did you make?

We created a multiplayer tank game using new technologies: HTML5 websocket, canvas, and node.js. This is the Node.js Knockout after all. Before I take you down the yellow brick road, here is the link to our game: Nōdo Tanku. It requires any of the following supported browsers:

Chrome 5 or greater
Safari 5 or greater
Firefox 4 Beta 1 or greater

It basically relies on a browser that supports websocket draft 75 or 76 because that is what Faye client/server relies on. Fortunately, Faye supports both.

Although, our game comes with a good number of bugs, I think we all agreed as a team that we pulled the almost impossible. We're all very happy with what we deployed to Joyent. I think the sound effects added another element to the overall game experience. The hand drawn tanks, some jQuery jazz, and almost hard-to-read TypeKit font made the game super fun.

What I need to improve

Fortunately, we all watched PeepCode's Node.js screencast. This was our foundation of how we structured both the server side and client side JavaScript. The thing I regret the most is not TDD'ing the client side JavaScript code. Craig and I were pretty much the client side team, while Jim and Justin were the server side team. I'm happy they used Hudson CI and Jasmine Node to test their server side JavaScript, but because I was not familiar with client side testing frameworks or libraries, I did not test drive our client side libraries. This introduced some bugs that were created when adding new features. I'm pretty sure there were several occasions where I implemented a new feature, and later another team member reported a problem with existing functionality. I need to test first!

Other things I need to improve on is my CSS and overall understanding of the HTML5 features. I think probably half of the time, I was doing a lot of reading to understand how to implement things here and there. That time could have been spent creating tests first, implementing more features, fixing bugs.

Vote for us!

This post cannot be said without saying, you should vote for us! Well...vote for us if you like our game. I had so much fun working with the other 8th Lighters. I think we all had a lot of fun and enjoyed playing our own game. We hope you do too.

Vote Here!

rails and node.js - part 6

2010-08-27T15:00:00-07:00

We finished the Rake task. With the help of Eric Smith, Matt and I were able to get past a roadblock from yesterday and do a bit of refactoring I couldn't quite figure out. A little Google search yielded a nice little tutorial on Rake. To load the Rails environment in Rake, we basically needed to put this in our code:

namespace :api_example do
  desc "Update user records by calling API Example"
  task :update_users => [:environment] do
    # task code here
  end
end

The [:environment] piece loads up the Rails environment. Then in the console, we run the rake task like this:

[some_branch][~/local/git/some_repo] rake api_example:update_users

We demoed the work we've done in the past couple of weeks to Paul and he seemed impressed with it. It will be demoed to the customer next Monday, but before then, Paul will walk us through the deployment process. I'm glad to have worked with Matt and Paul on this Rails project. I've definitely learned a lot. I feel like my RSpec and Ruby skills have improved. I've gained more experience on pair programming and learning more about the backend - MySQL, ssh, running rake tasks, learning about environment variables (e.g., RAILS_ENV), script/commands, etc.

Node.js Knockout

Node.js Knockout starts today and it's pretty exciting, at the same time, nerve wrecking. I would love for us to complete the project we want to work on, so I hope to contribute as much as I can to get this working. I know we will learn a lot of things out of this challenge. This will be an exciting experience working with three other people I have not worked before and plus learning how they develop and be the worst (but try to contribute). I will give details about the weekend experience...after the weekend.

rails and node.js - part 5

2010-08-26T21:59:00-07:00

We finished the API features for the Rails project and did a bit of refactoring. As the feature set grew, it made more sense to rename refactor the namespace, methods, etc. We also used extract method refactoring to move methods into a new module where it made more sense. The final piece of this is to write a script that will use the API methods to retrieve customer information and update the database with that information. We tried creating a script that was placed in the script folder, but unfortunately could not require the API module we created. We may look into creating a Rake task and have Rake load the Rails environment. I'll have to look into how Rake tasks work. This will go live next Monday, so I'm a bit excited and nervous. I look forward to putting on the finishing touch tomorrow and run a quick demo with Paul.

I forgot to mention yesterday Matt and I worked on some XML data and used Ruby's standard library, rexml, to do the job. It's actually fairly easy library to use. The entire xml is stored in an elements hash and by providing it the XPath, it will return the element(s).

Node.js

Unfortunately, I didn't make head way in the Node.js, HTML5 reading as I hoped. I guess I'm a little burned out from the week, but I look forward to working on our Node.js Knockout project this weekend. Apologies for the short blog post, but I will have lots to talk about after this weekend.

rails and node.js - part 4

2010-08-25T21:59:00-07:00

Another feature has been completed on the Rails project. Fortunately, today we were given the new URLs for the APIs that we needed to implement. By testing it out in an IRB session, we were able to make sure the APIs worked (as intended) and see what we needed to do to create the code. Yesterday, I talked about using DelayedJob or Resque to handle our background jobs, but after talking to Colin in the morning, we both agreed those libraries might be overkill for what we need. Matt and I briefly spoke with Paul and I basically told him we would use simple cron jobs to handle the background jobs using whenever. Whenever allows you to create cron jobs with clean Ruby syntax.

So as said above, we finished one API and began working on another to pull customer data. This API will be the most time-consuming to implement since we need to update the User table and create a hash map of new fields to old. We also migrated a bunch of new fields to the table that need to be created in order for this API to work properly. A neat little tip from Colin when creating mass fields is to just create one and edit the migration file to add the others:

[some_branch][~/local/git/some_repo] script/generate migration AddNewFieldToTable new_field:string
      exists  db/migrate
      create  db/migrate/20100823165501_add_new_field_to_table.rb

Open migrate/20100823165501_add_new_field_to_table.rb to add other fields, rename the class, and rename the file name to make it more meaningful about what you're doing. Then rake db:migrate.

These new API calls require credentials, so instead of storing the credentials in the class, we stored them in a yaml file and created a layout similar to the database.yml. Of course, as with any credentials, it's best not to commit the yaml file to the git repository, so another great tip from Colin was to create another folder under config (e.g., local), store the yaml file there, and add that folder to .gitignore. This way, you can still use the credentials for development/test, but it won't be checked in to the git repository. I noticed we were also storing the URLs to the APIs in the classes, so I moved them appropriately in the new yaml file.

I'm always learning something in this Rails project and it's a great testament how important it is to have a breakable toy when learning new languages/technologies. I look forward to learning as much as I can in this project.

Node.js Peepcode Screencast

I watched the Node.js Peepcode screencast and enjoyed learning more about Node.js. With the recent projects I've been working on (e.g., TicTacToe on WEBrick, Node.js, Rails), I'm learning a lot about web development, GET/POST, request/response, and more. I think I'm comfortable with them now and I understood most of everything from the screencast. Callbacks are still strange little features to me, but it makes sense how they create asynchronicity. Unfortunately, the demo Geoffrey used in the screencast did not work for me. I think the external service is down. As much as I'd like to agree how Node.js provides clean errors to understand what you did wrong, it sometimes spits out errors that will have you run around for a bit. I guess as I get more comfortable using the library, I will have a better feel for what each error means.

Rails offers a development, test, and production environment automatically and the ability to easily switch between them. Sadly, Node.js does not. But fortunately, Geoffrey provided a nice server script that you run that watches for any changes in the project and restarts the server for you. Also, he structured the project layout very similar to Rails to modularize the code for readabiliy and accessibility. I think I will use this setup for the Node.js Knockout. That being said, I recommend this screencast as it does contain a lot of good information to get started on Node.js (my unbias review).

rails and node.js - part 3

2010-08-24T21:00:00-07:00

Matt and I finished the Rails feature that we started yesterday. What I thought would be a difficult task to log a user without a password, ended up being fairly simple. The authentication call was the gatekeeper and determined if the user was allowed to log in. If the authentication method was never called, you can just log the user in. Now of course, we're not going to do that. We created a simple class that returns a user if the user is coming from the third party site with valid params. Then the log in method takes the user object and logs her in.

Our next feature involves working with several APIs that depend on each other. We basically need to call these APIs in a background process to synchronize a table in our database with the third-party database. I've not worked with background processes (e.g., cron jobs) before, but this will be an exciting challenge. I asked Colin about it and he suggested we look into Resque. I read defunkt's reasons behind creating Resque after experiencing with many other background job systems for GitHub. I like the fact that Resque has a manager aka monit/god who can watch the workers and make sure they're doing the job in a timely manner. It's also heavily inspired by DelayedJob which I was also looking into prior to Colin's suggestion. defunkt "recommends DelayedJob to anyone whose site is not 50% background work." I'll look into both and see which will be easier to implement. Fortunately, both of them are still in active development.

Reading

I spent some time reading a bit on the Rails conventions. Database table names are pluralized, models are singular, file names are lowercased and underscored, while the classes are camelcased (e.g., IAmAClass). Now I know Rails, behind the scenes, converts the names automatically to conventionally call things naturally. Another interesting bit was how Rails automatically loads everything up. This is why you rarely see require commands in the code. Even libraries in the lib folder automatically load up. We ended up storing our api library in a sub-folder of lib and since the classes were under a common module, it was automatically loaded up in the environment. Conventions rule.

Node.js Knockout Preparation

Craig just gave a heads up that Heroku does not support websockets. Drats. I believe PusherApp should still work since it works via REST, so we may end up using that. I actually do not see another option at this point unless we build our REST + websocket server system (a replica of PusherApp). I digress. PeepCode Screencasts just came out with a Node.js screencast that I may end up buying and watching before the knockout. While I enjoyed reading Dave Hoover's GroupDASHpon code, I guess it won't hurt to see how Geoffrey Grosenbach implements Node.js.

I've noticed a lot of Rubyists have similar "coding styles" a lot more than any other languages I've used. JavaScript is no exception. Actually, I think a lot of JavaScript developers/hackers just write code that either follows the convention of the js framework/library or just makes things up, like add a bit of Ruby flavor. I know Douglas Crockford doesn't like the idea of using $ and _ by itself to represent a namespace.

rails and refactoring

2010-08-23T21:00:00-07:00

"Read, Learn! Never stop learning. Learn many languages. Learn many disciplines. Practice TDD and never let it go. When you write code, write the best code you can. Never succumb to the temptation to rush. The only way to go fast, is to go well!" - @unclebobmartin, interview

Matt and I continued working on the Rails project adding another feature to send and receive post form data. Both features share some methods, so I thought now was a good time to experiment with the "pattern" I've been seeing in ruby gem source code. I took our class that contained the first feature, moved it under a parent module and abstracted shared methods into a child module of the same parent module. By creating the parent module, I have created a namespace to prevent any name conflicts and such. The new feature is also part of the same namespace in its own class and includes the child module to call shared methods. I showed Matt and Paul of the new change, and they seem to be okay with it. We made sure our existing tests passed and moved on to the new feature. I think the experiment was a success.

The feature called creating new column to an existing table. Luckily with Rails conventions, it's a rather simple process:

[some_branch][~/local/git/some_repo] script/generate migration AddNewColumnToTable new_column:string
      exists  db/migrate
      create  db/migrate/20100823165501_add_new_column_to_table.rb

AddNewColumnToTable is actually broken down conventionally: Add NewColumn To Table. After, we ran rake db:migrate for both development and test RAILS_ENV. A solid 15 minutes or so, we were stuck at a problem where the new column was not accessible via accessor. We found out later we needed to load the database schema for the test environment (I'm taking a guess - it's because the test database is always empty, so the migration rolled back):

[some_branch][~/local/git/some_repo] rake db:schema:load RAILS_ENV=test

We do hit some roadblocks every now and then due to our limited knowledge of Rails and experience, but I'm actually quite surprised at the rate we're progressing. We probably owe it to Rails' conventions - once you get the hang of it, it's actually really easy to get around. In an effort to move things along, I'm going to pick up some Rails knowledge on the side. Luckily, I have a couple of Rails books I can use as resources.

Refactoring

I noticed the last time I talked about refactoring from Martin Fowler's book was probably back in late June. I shall pick up where I left off here with a couple of refactorings. Last night, I remembered Doug's favorite refactoring pattern was "Replace Conditional with Polymorphism." I read about it, and it's a refactoring pattern similar to one of the SOLID principles: Dependency Inversion Principle. It talks about avoiding writing explicit conditionals and to look for switch or if/else statements that check type. These are places where this refactoring pattern will become handy.

I use the move method refactoring a lot and it's nice to know the name of what I'm using. When classes have too much behavior or are collaborating too much and are too highly coupled, move method should be used.

node.js, html5, websocket

2010-08-22T20:00:00-07:00

I spent some time this weekend learning about HTML5, websocket, Faye, PusherApp, and more in preparation for the Node.js Knockout next weekend. There's a lot of content to cover. HTML5 is the new standard of HTML superseding HTML 4.01, XHTML 1.0, and XHTML 1.1. Websocket is a feature in HTML5 that allows for bi-directional, full-duplex communication channels. I'm sure you know about all of this already, so I'll link some references I discovered over the weekend:

Dive into HTML5 by Mark Pilgrim
Firefox 4 (pre-beta 1), Chrome 5, and Safari 5 use websocket draft 75
Firefox 4 (beta 1+) and Chrome 6 use websocket draft 76
PusherApp - client/server messaging system
Faye - client/server messaging system (supports websocket draft 75/76)
Node.js third party library compatibility list
Node.js Knockout blog post series on how to use Node.js
Node.js prizes (are you motivated?)

There's an interesting relationship between implementation and specification and how one can influence the other. When I think about Google, Mozilla, Apple, and even Microsoft implement the latest draft into their "edge" version browsers, I can see how their implementation styles can drive the specification of HTML5 in their favor when it's finalized. Mark Pilgrim, author of HTML5 Diving Up and Running, is right about the fact that "the ones that win are the ones that ship." It's basically how the img tag became standard.

Throughout this week, I plan to learn more on HTML5 and websocket.

rails and node.js - part 2

2010-08-20T20:00:00-07:00

I was up pretty late last night working on an issue on the first Rails project I worked on last week. The flash file implementation worked for all browsers, except Firefox. Apparently, Gecko based browsers are very strict. After fiddling with it for many hours on end, the swf_tag just needed an additional parameter and it was happy.

= swf_tag "swf_file_name", :div_id => "div_id_name"

I worked on this problem for a long time and all I got was a :div_id. I still feel like I learned a lot from this. The Rails project is using Haml, so I needed to learn how to write some Haml. The docs are well written and for the most part, I found my answers from the doc.

Lunch 'n Learn - Node.js and MongoDB

Jim gave his final session on Node.js and introduced MongoDB into the mix. It was nice to see MongoDB again after having worked on it to store 14 million documents for all possible 4x4 board states. He presented a JavaScript library called Mongoose which seems to be inspired by ActiveRecord and we used it to interact with a MongoDB database in Node.js.

npm install mongoose

Our homework is to write something for our Node.js project that persists the data. For my project, I will be storing the list of repos for each user. The only problem I see with persisting the data is when the repository list changes on GitHub (e.g., removing/adding repositories). I'll need to figure out how to make sure the list that is returned reflects what is on GitHub.

New tasks for Rails project

Matt and I sat in on a meeting with Paul and the customer to give updates on the project. I finally understand what we need to do to implement the next feature for the Rails project. This will be slightly more challenging since we need to authenticate a user coming from another site. Currently there is no method in the project that allows for this, so we'll need to create it from scratch. I also noticed the workflow is similar to the previous task, so it should be able to use the class we created after some data abstraction. I'll have to do some reading to see which pattern might be useful for this situation. Perhaps, just a simple Module will do the trick.

Node.js Knockout

The Node.js Knockout is next weekend and our team had a meeting of what we need to learn and tackle in order to create a stellar project and win the prizes. I can't tell you what our project idea is yet, but after the weekend, I'll let you know. Right now, I'll need to read up on HTML5, CSS3, websockets, and Node.js. It's going to be fun times. I hope to learn a lot from this challenge.

rails and node.js

2010-08-19T20:00:00-07:00

Today, Matt and I briefly worked on the Rails project. We added a configuration option that allows the feature we implemented to be turned on and off. I think Paul mainly wanted this for the demo tomorrow. I committed the changes, but didn't push it to the GitHub repo. For some reason, I had trouble creating a remote branch, so I did some reading on the net. I'm not sure why I didn't have problems before when creating branches on my Tic Tac Toe project, but I had to follow a different path to create a remote on a GitHub repo I do not own.

I updated my post on git branching and merging. Basically, I needed to execute this command:

git push origin origin:refs/heads/new_branch

Once I created the new remote branch, I was able to create a local branch, have it tracked with the remote branch, and check it out, all in a one-liner command:

git checkout --track -b new_branch origin/new_branch

Bingo! New branch is tracked, and I'm able to push and pull without stating the additional parameters: repository and refspec.

While we waited for the next assignment, I read the entire page on vows.js to learn what it's about and to help create tests for my node.js example. The examples on vows.js weren't that particularly helpful when it came to testing out the node.js server and the methods that handle GET/POST requests. Maybe it assumes you already know how to test them? If so, I need to do some catching up because I'm having a little trouble. I also found out it doesn't have any mocking features and some parts feel a bit unintuitive. When I sat down with Colin late in the afternoon, we ran into a problem where the topic wouldn't accept a function directly. I think vows.js attempts to invoke the function, which Colin didn't want. So by wrapping an anonymous function around the original function and have it returned, Colin was able to the use the original function. It felt like a needless step, but I guess that's just the way it works.

Node.js Knockout is next week, and I don't feel fully prepared. I should really know my Node.js if I want to make contributions to our project. I will be focusing on Node.js everyday to brush up on my JavaScript/web skills.

rails and sleep

2010-08-19T07:00:00-07:00

My, my, a 15 minute cat nap turned into a full fledged sleep thanks to my Android timer not going off. I had planned to work on the Node.js homework, but it looks like I'll be working on it tonight.

Yesterday, Matt and I continued working on the Rails project. We actually finished the implementation by the end of the day, working through some spiking to see the working solution. Many thanks to Craig for being super helpful and knowledgeable. We asked him many questions and he was very detailed in his answers.

I'm learning so much on Rails, I have an itch to continue to learn more. I'm really glad to have learned how to use Limelight because it shares similar conventions with Rails and I'm glad to have learned how to work with WEBrick because it's as low level as you can get when developing the web - talking directly to the web server and have it serve your pages.

Today, we will work on another task similar to the last one. I look forward to the challenges and the learning.

Reading

On Tuesday night, I began reading parts of the Design Patterns in Ruby and came across Russ Olsen's ideas on the set of meta-design patterns:

Separate out the things that change from those that stay the same.
Program to an interface, not an implementation.
Prefer composition over inheritance.
Delegate, delegate, delegate
You ain't gonna need it

The first one sounds like the layman's terms for saying manage your dependencies, understand the symptoms of rotting design, and apply the SOLID principles.

Program to an interface, not an implementation is Dependency Inversion principle which says to "depend on abstractions, not on concretions."

I knew about composition and applied it to my Tic Tac Toe project, but did not know it was the word to describe it. Delegation is something new to me. I didn't know this was allowed per se. It's where you have one class, say Car and an Engine class. You create an instance of the Engine class inside the Car class and the methods start_engine and stop_engine simply call the Engine instance methods.

From delegation, I read about Law of Demeter which Craig threw out yesterday in one of our conversations. Then I quickly read about the Presenter pattern which uses Forwardable to delegate and not violate the Law of Demeter.

I plan to go back to my Tic Tac Toe project and apply these design patterns.

rails and reading code

2010-08-17T21:00:00-07:00

Today Matt and I met at the Libertyville office to pair on the Rails project. Last night, while on the train, I tried working through sample code that would send a POST request automatically using HTTParty, but for some reason I could not get it to work. We asked Micah for some help and within minutes, had a working solution for the project. Mentor help availability cannot be taken for granted. He basically ran a simple test using curl, then switched over to Ruby's net/http standard library to find a solution. I think I'm finally going to give HTTParty a rest and use the net/http solution. Here's sample code with the help of this article:

require 'net/http'
require 'net/https'

Class Sample

  def post
    http = Net:HTTP.new('http://url', 443)
    http.use_ssl = true
    response, data = http.post('path/to/request', 'name1:value1&name2=value2')
    puts data
  end
end

Now that we spiked the code, we cleaned out the project and began writing specs against a class that handles the post requests. With the help of the 8th Lighters in the office, we were able to ask questions and get some guidance on how to implement the feature into the project. Matt and I will be working on this throughout the week and I hope we can finished it sooner than expected.

Reading Other People's Code

I'm trying to do a lot of reading at home to learn more about Rails conventions, Ruby, RSpec, and web development in general. I just picked up Ruby Design Patterns to understand ways to design code (don't worry I still plan to read the other books). I'm also planning to take some time to read other people's code. A tweet from Dave Hoover today sparked an idea about learning from other people's code. Last week, I read some of the source for the WEBrick library and while reading code in Specjour, noticed some similarities in design and structure. Things like wrapping all classes in the same module to create a shared namespace, creating a main file that lives on the root outside of lib that runs all the requires, autoloads, versioning, logging features - reading the code sparked ideas in my head about how I can refactor my Tic Tac Toe game engine and turn it into a library that other UIs can easily implement.

Hopefully, I'm not overwhelming myself by doing too much all at once. This is exciting time for me to learn all kinds of things to help become a better developer. If only I had all the time in the world, I could maybe finish reading the things I want to read...when squids fly...oh wait.

rails conventions

2010-08-16T16:00:00-07:00

Today was the first day working on the Rails app. Matt and I pair programmed using iChat screen share to take turns moving around the application code to learn more about it. My lack of using certain tools prevented us from rapidly progressing forward today. We needed to ssh into the client's staging server to pull down a backed up database so we can use it locally. Then we needed to figure out how to create a user so we can log in to the test application. Fortunately, Craig and Paul were helpful and guided us in the right direction. There was an issue we discovered that appended post form data parameters to the end of the response URL without a question mark which caused a Rails routing error. Fortunately, it was not a problem on our side and as a workaround, we'll add the question mark to the URL.

I must say, I enjoy the Rails conventions, but it does have a small learning curve when you're rapidly learning and developing Rails. I'm sure as I continue, the conventions will be second nature, as they should.

I plan to look at the application code some more tonight and read the RailsGuides. We're having a stand-up tomorrow, so I should have an estimate of how long this implementation task is going to take.

ChicagoDB Meetup

This will my first attendance at the ChicagoDB meetup. It's going to be at the Hashrocket office and I look forward to meeting Chicago Rubyists I haven't met yet. The topics include Google's MapReduce and CouchDB. Look forward to learning about those NoSQL databases and my only experience thus far has been with MongoDB.

webrick - part 6

2010-08-16T07:00:00-07:00

If you recall, I was having a problem on my WEBrick implementation of Tic Tac Toe where the meta tag hack was showing the board states when Negamax was running. Negamax was actually causing all sorts of problems because WEBrick servlet was relying on the board's state. I was able to fix this by modifying the Negamax call to create a duplicate Board class instance when running the Negamax algorithm. This way it will have no effect on the state of the board in any way. Other improvements I've made was to prevent hovering of the piece when it was not a human's turn. This was a simple JavaScript fix that required an additional privatized variable to store the state if the current player was allowed to make click moves.

The try again and quit buttons are now properly displaying on the HTML and work as intended. I had a problem with the try again button because I had forgotten to reset state for a variable that was being used in session. The @status ivar was used to identify which image message was going to be displayed (e.g., Player X make a move, Player O is making a move). I made sure @status reset itself and it seems to be working fine now. A different kind of problem for the quit button was regarding the thread state which ran the game engine. I'm still not all too familiar with threading, but when the quit button returned the user to the options page, the game engine thread was still running. When the user created a new game, a new game engine thread was created. Somehow this was creating some weird problems I could not identify. I looked at the rdocs for the Thread class and used the exit method on the game engine thread when the user plays a new game. I also made sure it only exits if the game engine thread exists. This seems to have solved the threading issue and I no longer see the anomalies.

Overall, this WEBrick project, while probably the most difficult, was also the most educational. I learned a lot about how a webserver works (e.g., WEBrick), how much of a headache it can be when dealing with a game engine thread and the webserver, understanding all elements of creating a web application (e.g., HTML, CSS, JavaScript), and the many things you can take for granted when using an awesome web framework like Rails and Sinatra.

I look forward to working on the Rails project. This will be a fun week.

working on my third rails app

2010-08-14T21:59:00-07:00

Yesterday was a steady day at the office. Our Lunch 'n Learn was postponed to next week since most of the craftsmen are out at Agile 2010. I re-read parts of the Principles and Patterns and re-watched parts of Jim Weirich's talk so I could write about it.

As the post title indicates, I will be working on my third Rails app. I'm pretty excited as this takes on a slightly bigger task. I'm teaming up with Matt Hinger and will be implementing the piece of functionality that I wrote for the second Rails app. There will be some new things I need to learn how to do. One is using Cucumber for acceptance tests, other is login authentication. I'm not sure what else is in store, so I'll need to study the Rails project for that. I'm really excited to be working on client projects. This should be a great experience with Rails and the Agile process.

End of Iteration 6

I had a chance to talk to Micah while he was at Agile. During the meeting, I expressed my concerns about the meta tag hack I implemented and he welcomed me to the world of web development. I then realized my concern was legitimate. This was not an easy thing to do on the web where you're having to deal with GET/POST requests vs. a GUI event loop where it's a bit more flexible and forgiving. I discovered some issues with the meta tag hack and will need to fix the issues for the next iteration. So I was not able to complete all ten points, but most of them were completed and moved over to the archive. Since I will be working on the third Rails app, my focus next week will be on that project. I will be fixing the meta tag issues this weekend.

principles and patterns - part 2

2010-08-13T14:00:00-07:00

I finished reading Uncle Bob's Principles and Patterns, except for the part on Principles of Package Architecture. After talking about the four primary symptoms that indicate rotting designs, Uncle Bob goes through the SOLID principles which are primarily about dependency management. I was fortunate enough to find Jim Weirich's talk on SOLID principles and how it applies to dynamic languages like Ruby. I will be using his references to describe some of the SOLID principles.

Single Responsibility Principle

"A class should have one, and only one, reason to change"

A class should have one responsibility. Having two responsibilities makes it less flexible for change and potentially harder to test. An easy to way to tell if you're violating this principle is to write down the purpose of the class. If there is an AND or OR, the class is probably doing more than one thing.

Open Closed Principle

"You should be able to extend class behavior, without modifying it"

Uncle Bob states this is one of the most important principles in object oriented design. Abstraction is the key to this principle. If you see yourself adding if/else or switch statements, this is probably an indication that it needs to be abstracted.

In Ruby, if you wanted to use a third party library, but need different behavior, instead of modifying the library, extend it.

Liskov Substitution Principle

"Derived classes must be substitutable for their base classes"

This is something I observed when reading about this principle. Since Ruby supports duck typing, it's natural to substitute derived classes for their base classes. There are two conditions that must be met for derived classes:

it must require no more
it must promise no less

Jim walked through an example of a square root method which requires a number greater or equal to zero and provides an accuracy of three decimal places. The design by contract therefore requires a number greater or equal to zero and an accuracy of three decimal places. All derived classes must meet this requirement, otherwise will violate the Liskov Substitution Principle.

Interface Segregation Principle

"Make fine grained interfaces that are client specific"

Having many interfaces is more flexible than having one big interface that handles all. Having many will not impact other interfaces when changing code for another.

Dependency Inversion Principle

"Depend on abstractions, not on concretions"

If you depend on concrete classes, you create explicit references and make it difficult to use other classes that like the ones you originally depended on. By introducing an interface, you allow other classes to be used that conform to the interface.

In Ruby, interfaces do not exist. Fortunately, there is a style of dynamic typing called duck typing.

"when I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck."

Due to the dynamic nature of the language, explicit references are hard to create in Ruby. There are no explicit interfaces, but it has something similar to what Jim calls "protocol". The protocols, like interfaces, have a list of methods with semantics.

Conclusion

I really recommend watching Jim Weirich's talk. There's a lot of really good information that was not covered here. He ends the video with this question:

SOLID principles were designed with static languages in mind. Are there SOLID-like principles that are specific to dynamic languages like Ruby?

He doesn't have an answer, but welcomes discussion. I look forward to hearing about the principles.

committed my second rails app

2010-08-12T09:59:00-07:00

I worked on my second Rails app today. This time starting from scratch. Paul wanted me to a build a small Rails app that sends POST form data to a remote server. Unfortunately, this could not be tested locally since the remote server only supports the staging server with the proper credentials. So, I had to do what I could to make the app workable. I didn't know much about sending POST form data to another server, so again I googled around to see if there were any related topics. Nothing grabbed my attention so I used my lifeline and asked a Rubyist friend to see what he thought. He answered HTTParty. Long story short, I implemented what I thought would work, but it failed when Paul deployed it to the staging server. Then I saw him modify the form attributes:

<form action="https://remote-server-url" method="post">

Sigh. That was too easy. It bypassed my simple HTTParty call. Sometimes the answer is not always a third-party gem. Anyway, we got the form to work, but it looks like the remote server isn't responding so Paul is going to deal with the issue.

WEBrick Tic Tac Toe

I learned a lot from this Rails app. From changing the routes, to understanding the Rails conventions, to introducing Bundler and failing because the staging server didn't have Bundler. I enjoyed learning about Rails.

I worked more on the WEBrick Tic Tac Toe. The design is looking more like the Limelight version and I've modified the way I introduce the refresh meta tag into the HTML. I still feel like it's hack and I wish I knew a better way to do this, but since it works, I'm not complaining. I'm going to try working on it more tonight and continuing reading my books.

Reading

I spent some time last night reading on The Well Grounded Rubyist and the paper, "Principles and Patterns" by Uncle Bob. This is the first time I'm understanding one of the SOLID principles, the Liskov Substitution, which states that "derived classes should be subsitutable for their base classes...Derived methods should expect no more and provide no less." I'm pretty sure I'm violating this principle in my project. When I read this principle, I thought about polymorphism, and of course there is relationship. According to this article, "The Liskov Substitution helps to guarantee inclusion polymorphism, which is a good thing because inclusion polymorphism improves reuse." I'm going to have to read this article in full when I get home.

committed my first rails app

2010-08-11T09:59:00-07:00

Today was a pretty fun, exciting day. I took the train down in the morning to go to the Chicago office where I worked on my first Rails app! The idea was to come down for the day to assist Paul on a simple task - replace JavaScript rendering banners with a flash-based banner per client's request. Paul gave me write permission to commit to the GitHub repo and I went ahead and cloned the repository to discover what this Rails app was made of.

After a bit of trouble setting up the required gems (this project could use Bundler) to run the tests, I needed to figure out how to embed a flash file. Now I could have done a bit more research to find what's the best way to embed the flash file, but I chose something that immediately came out of a Google search. It's a Rails plugin called swf_fu which makes it super easy to embed flash files using its own tag method:

<%= swf_tag "flash_file" %>

After including the swfobject.js file, it does its magic. I made sure the tests continued to pass and made sure the flash file visually rendered correctly. I proceeded to remove references of the JavaScript banners, but had a bit of trouble with the database. Paul stepped in and helped me solve the problem. You just don't mess with the schema.rb file. D'oh.

I made the tests pass again after removing the old banner references and pushed my changes to GitHub. That felt good. It might've been a small task, but I enjoyed contributing. Paul asked me to do another small task that sounds a bit more challenging. I'm basically creating a new Rails app that can send a post form. I'm constantly reading the Rails Guides and googling around for some ideas. I may end up falling back on a gem (no pun intended).

WEBrick Tic Tac Toe

Micah and I were supposed to have our end of iteration meeting today, but missed each other via Skype. He's at the Agile 2010 conference, so it was a bit tough coordinating the time. We will try tomorrow. Basically, an update from yesterday's news on the WEBrick project, I was able to fake GET requests by polling a refresh every two seconds using the meta tag. It still produces weird board glitches and the usability is kind of hokey, but it does work. Hopefully, Micah will give advice on how to resolve the situation. I heard he's built many web servers, most of which eat WEBrick alive.

principles and patterns

2010-08-10T20:00:00-07:00

Note: I haven't blogged twice in a day before, but I thought this was important as part of my personal, mini-retrospective.

I'm beginning to notice a trend in the types of stories Micah assigns each iteration. Every story seems to challenge the architecture and design of my Tic Tac Toe project. Conveniently, I came across Uncle Bob's article on Principles and Patterns that I'd like to share in this blog post.

Uncle Bob lists four primary symptoms that indicate rotting designs. Unfortunately, this is probably why some stories take longer than they should.

Rigidity

This is a symptom where software is difficult to change. "What begins as a simple two day change to one module grows into a multi-week marathon of change..."

In my project, I'm noticing rigidity. Implementing new features to the project, estimated as 2 points, took longer than expected. I'm noticing as the project grows, making changes become more difficult. More time is consumed chasing after needed changes in dependent modules.

Fragility

This is closely related to rigidity and is the symptom where software breaks when changes are made. Sometimes causes breaks in areas where code was not changed.

In a previous company, I remember seeing this exact symptom in one of the projecs. As the project grew, it became more fragile and changes that were suspected to not affect other areas ended up affecting those areas.

Immobility

The symptom where software or parts of the software cannot be reused in other projects due to excessive baggage, not modular.

To me, this sounds like a good ol' set of refactoring should resolve the problem. In my Tic Tac Toe project, I know it can be refactored six ways from Sunday. From spending time reading some libraries on GitHub and the WEBrick library, I have some good ideas on what I can do to further modularize my code.

Viscosity

This symptom comes in two forms: design and environment. It's the dilemma where the developer is faced between making a change that preserves the design, but is more difficult to employ or making a change that breaks design (e.g., hacks), but easier to employ. In this situation, "viscosity of the design is high. It is easy to do the wrong thing, but hard to do the right thing.

Viscosity of environment comes about when the development environment is slow and inefficient." Developers are faced with compromising design for quicker compile times, check-ins to source control, etc.

In my project, I don't really notice viscosity of environment - Ruby is an interpreted language (I'm using JRuby, but it's not that slow) and Git is awesome. However, I believe I run into viscosity of design, consciously and sub-consciously. With my lack of design pattern knowledge, I'm most likely creating hacks than finding ways to preserve design.

End of Session

There's more to this article than what I mentioned so far. Uncle Bob goes into ways of making our designs resilient and protect them from rotting. I will go into this in another post.

Simply put, however, I really need to complete these books:

Refactoring
Working Effectively with Legacy Code
Agile Development : Principles, Practices, Patterns

I read a bit of Justin's blog and noticed how he listed design patterns everyday. I also remember reading Colin's blog on how he could have taken advantage of some of the refactorings for his Tic Tac Toe project had he read it earlier.

webrick - part 5

2010-08-10T18:00:00-07:00

I made some good headway on the WEBrick stories today. I dove into a bit of JavaScript to create a mouse over/out effect - it will show the user's piece over an empty square. There is some passing of information between the servlet and JavaScript functions. Seeing as I've never done this before, I basically created a closure in JavaScript to hold a privatized variable where I can set the current player. The mouseOver and mouseOut functions use this variable to display the proper image of the piece. There's probably a better way, but this seems to work.

I continued working on the game engine side of the project, transferring a lot of similar code found in my Limelight project. I'm going to have to come back and extract these duplicates and create a shared module of some sort, but for the sake of completing these stories, it'll have to do. Most of everything is in place, except I'm having a bit of trouble getting the computer to communicate to the WEBrick UI that it's made its move. If you recall, the game engine is running in a separate thread so it can run independently from the UI. Through the Limelight Hangman example, I was able to work through a solution for the Tic Tac Toe on Limelight. However, in WEBrick, I'm finding it difficult to send GET requests programatically. Maybe it's my lack of GET/POST knowledge and lack of web development overall, but this is where I'm stuck on. I could temporarily have the computer make its move during the human player's GET process, but in a situation where two computers goes head to head, the page will never be refreshed. I may look into Ajax for a solution. Anyway, I plan to work on this more tonight. Hopefully, I'll see the light on the other end.

Here's what I have so far:

webrick - part 4

2010-08-09T18:00:00-07:00

I made some small breakthroughs today on WEBrick. First, I figured out how to make moves using the image based links. Simple enough, the links return back to the BoardServlet via GET. Now I just need to figure out a way to block the user when it's the computer's turn. I guess I won't know until I get to that point, but it's a good starting point to the story, "Allow the user to click on a square to make a move."

I scoured the interwebs again to find information on WEBrick and I ended up finding some good articles.

From these articles, I keep noticing the Mutex class being used to handle multithreading situations. I put the synchronize code block in my code, but I'm pretty I won't run into the problem. Inside the synchronize block, however, I'm going to need. See here (since pygments is giving me a hard time with this code).

What I found out through trial and error is that on every refresh, a new servlet instance is created. Since a new one is created, the old one is lost and I'm no longer able to keep track of the game state. I've not ever used class instance variables before and I've not so good things about it. However, I can't think of a better way to resolve the problem (and without using global variables).

As you may know, WEBrick documentation is a bit scarce. WEBrick's own website seems to be down and the latest documents on the web date back to 2004. I realize WEBrick development is dead and most libraries that utilize WEBrick provide the convenience of not having to know the details of how WEBrick works, but I wish there was good documentation. Then it hit me. Why haven't I thought of this before? The WEBrick library is just a Ruby library. I can just look in there. So that's where I am now.

There are several roadblocks which are preventing me from working through the story "Web design should conform to Limelight design. It should look almost identical." One of the roadblocks is trying to serve an ERB with references to CSS and the background image. For some reason, it cannot find the files. I tried providing it a full path, path relative to the current working directory, return the path via method call in ERB - I feel like I've done everything to no avail. What seems to work as of now is to read an rhtml file, run ERB, and spit it back out to an html file. Unfortunately, after that I have no control since the DefaultFileHandler servlet takes over. I'll have to do more research on this...

vows.js, ttt stories

2010-08-08T21:00:00-07:00

Note: Apologies for the delay again, but I was a bit under the weather on Saturday. This post is a combination of last Friday and the weekend.

Lunch n' Learn - Vows.js

Jim continued the Node.js session, introducing a BDD framework, vows.js. We used the Prime Factors Kata to learn how to write vows.js tests. I've heard about the katas from Uncle Bob, but haven't tried them until then. I really enjoyed going through the kata and it's a really good way to practice TDD. I'd like to work on the Bowling Kata in the near future.

Our homework assignment for next coming week involves writing tests for the Node.js server we built. Jim suggested it's best to start over and write tests first to let the design come out of the tests. I look forward to working on this assignment this week.

Tic Tac Toe stories

Aside from my loss of Saturday, I worked on my stories today and actually finished the Scoreboard story. This calls for keeping track of wins/losses/draws for both the console UI and Limelight UI. This is what it looks like:

Console UI Scoreboard

Limelight UI Board with View Scoreboard button

Limelight UI Scoreboard

I also managed to fix the transparency issue with some of the images, something Micah pointed out last iteration. As you can see above, the pieces as well as the messages no longer have a white background. The trick to solving this was exporting the images as *.png and not *.jpg. Apparently, and I did not know this, but png files support transparency whereas jpg files do not. Well I guess that's why I notice people using png in GUI development.

There's still a lot to do in this iteration. I'm still somewhat stuck on the UI design for WEBrick. I'm not HTML/CSS savvy (yet), so I'm reading up on it, and at the same time figuring out how WEBrick works. Of course, the Node.js homework helped my understanding of how web servers work. More on that later.

Friday Dinner with 8th Lighters

Colin made a wonderful suggestion to have dinner at a nearby restaurant. A good number of us went and some invited family members. It was really great to hang out with the 8th Lighters outside of work. I really had a great time. Uncle Bob and his wife made a guest appearance which made the night even better. Although I didn't get the chance to talk to him, it was a great feeling knowing I'm part of a great team.

node.js homework

2010-08-06T10:00:00-07:00

I worked on my Node.js homework throughout last night for 8th Light University, so I apologize for writing this post now. As you may have read last week, Jim is currently leading the 8LU session on Node.js. He assigned us homework to create a Node.js server that will talk to services like Flickr, GitHub, Twitter, Facebook, or locally like the file system. I decided to have my server talk to GitHub using their API and return back a list of repositories for a specific user. There were some painful moments along the way, but I learned a lot from this assignment that will definitely help get through a roadblock I have now with WEBrick. It's my lack of GET/POST knowledge and fully understanding how the server communicates with the client. So fortunately, coming out of finishing the homework, I have better understanding and will be able to press forward with the WEBrick stories.

Here is the code for the Node.js homework. It's spiked, so it looks a bit messy, but it does work.

Note: This was developed using Node.js version 0.1.103

var sys,
    url,
    querystring,
    http,
    searchUser,
    listRepos,
    postHandler;
sys = require('sys');
url = require('url');
querystring = require('querystring');
http = require("http");

searchUser = function(req, res) {
  res.writeHead(200, {'Content-Type': 'text/html'});
  res.write('<form action="/result" method="post">' +
      'GitHub user: <input type="text" name="github_user">' +
      '<input type="submit" value="Search"></form>');
  res.end();
};

listRepos = function(req_data, res) {
  var user,
      github,
      headers,
      request;

  user = req_data.github_user;
  github = http.createClient(80, "github.com");
  headers = {
    'host':'github.com',
    "User-Agent": "NodeJS HTTP Client",
    "Content-Length": "0"
  };
  request = github.request("GET", "/api/v2/json/repos/show/" + user, headers);

  request.addListener('response', function(response) {
    if (response.statusCode > 200) {
      res.writeHead(response.statusCode, {'Content-Type': 'text/plain'});
      res.write("No user found.")
      res.end();
      return;
    }
    response.setEncoding('utf8');
    var responseBody = [];

    response.addListener("data", function(chunk) {
      responseBody.push(chunk);
    });

    response.addListener("end", function() {
      res.writeHead(200, {'Content-Type': 'text/html'});
      var repos,
          repoNames,
          i,
          max,
          repoObj;

      repoNames = '';
      repos = JSON.parse(responseBody.join(""));
      //repos = querystring.parse(responseBody.join(""));
      //console.log(sys.inspect(repos));
      max = repos["repositories"].length;
      for (i=0; i<max; i+=1) {
        repoObj = repos["repositories"][i];
        repoNames += '<strong><a href=' + repoObj["url"] + '>' + repoObj["name"] + '</a></strong><br />'
      }

      res.write("<h1>" + user + ' has ' + repos["repositories"].length + 
        " repositories:</h1><br /><hr>" + repoNames);
      res.end();
    });
  });
  request.end();
};

postHandler = function(req, res, callback) {
  var _request,
      _content;
  _request = {};
  _content = '';

  if (req.method == "POST") {
    req.addListener('data', function(chunk) {
      _content += chunk;
    });
    req.addListener('end', function() {
      _request = querystring.parse(_content);
      return callback(_request, res);
    });
  }
};

http.createServer(function (req, res) {
  switch (url.parse(req.url)["pathname"]) {
  case '/':
    searchUser(req, res);
    break;
  case '/result':
    postHandler(req, res, function(req_data, res) {
      listRepos(req_data, res);
    });
    break;
  }
}).listen(6633)

console.log('Server running at http://localhost:6633');

I couldn't figure out how to handle form data. Luckily, someone posted on how to do this. Kudos for his work. Also, I had a bit of trouble understanding how to GET data from GitHub using the API, so I looked at this node-github library to get some ideas on how to do it. It looks like a nice library and I definitely got the information I needed, but I didn't want to use it for the sake of understanding Node.js.

I look forward to today's session with Jim. We're going to learn about vows.js, an asynchronous BDD framework for Node.js. Four of the 8th Lighters including myself will be participating in this month's Node.js knockout. I look forward to learning a lot of Node.

Book Reading

Unfortunately I have been putting my book reading to the side to finish up the stories in the past couple of iterations, but I'm going to dedicate reading every night. The current books I'm reading are The Well Grounded Rubyist, The RSpec Book, Refactoring. I've used Ruby for a couple of months straight now, so when I picked up The Well Grounded Rubyist again, it was a good feeling when I already knew some of the content I was reading.

In the months to come I plan to read these books (stolen ideas from Colin's wonderful blog):

Happy reading!

end of iteration 5, changing project structure

2010-08-04T21:59:00-07:00

End of Iteration 5

Today was the end of Iteration 5 meeting where Micah and I discussed the stories that were assigned for the iteration. Finally, I'm back to completing a full iteration. Overall, he seemed happy with the stories and I moved each one to the archive. He did note the cosmetic issue with the Limelight Tic Tac Toe program. I need to figure out how to make the background of the pieces transparent and not white as they are now. I'm adding that to my todo list. Another thing on the list is to re-organize the project structure which I talk about below.

For iteration #6, one of the stories calls for making my WEBrick Tic Tac Toe look exactly like my Limelight Tic Tac Toe. I still need to read up on how to ping pong actions between user and servlet once the board is displayed on the page. I think I have an idea though.

Changing Tic Tic Toe Project Structure

I spent some time re-organizing the project structure. I thought if I organize it in the beginning of the new iteration, it will help improve my workflow as I'm constantly looking up different classes, specs, and such. It'll also allow me to run certain tests that are specific to what I'm adding/changing. Of course, I will run the entire suite once I'm ready to commit the code.

This is what the new project structure looks like now:

[master][~/local/git/tic_tac_toe_ruby] ls -l 
total 24
-rw-r--r--  1 skim  staff   758 Aug  4 19:12 README.md
-rw-r--r--  1 skim  staff   239 Jul  8 12:06 Rakefile
drwxr-xr-x  4 skim  staff   136 Aug  4 16:52 assets/
drwxr-xr-x  4 skim  staff   136 Aug  4 16:43 console/
drwxr-xr-x  4 skim  staff   136 Aug  4 16:45 game_engine/
drwxr-xr-x  4 skim  staff   136 Aug  4 14:27 limelight/
-rw-r--r--  1 skim  staff  1791 Jul 26 13:55 ttt.iml
drwxr-xr-x  4 skim  staff   136 Aug  4 16:40 webrick/

StdUI is now under console, Limelight under limelight, and WEBrick under webrick. The Tic Tac Toe engine sits in game_engine, and assets contains the images that currently drive Limelight. Under each UI, there is a lib and spec folder. I'm liking the new project structure and hopefully it'll meet my needs.

After working through the project structure changes, I noticed a few good things that came out of it (other than the fact that it's neatly organized):

Discovering cruft
Learning more about $LOAD_PATH
Learning more about Limelight's handle on image paths

I found several cruft that were not removed when I extracted classes about a month ago. These came up when running the game engine specs. I'm glad to be removing the cruft, because really who needs 'em! There were some headaches when trying figuring some $LOADPATH issues, but through the small headaches, I have a better understanding about what I need to do for a future project. Finally, since my Limelight UI utilizes images, I had trouble figuring out the cwd (current working directory) for backgroundimage and hover.backgroundimage. I had initially set them to "images/background/...", and tried adding the new images path to the $LOADPATH, but it doesn't read from there at all. I'm not sure what I was thinking, but I thought it would look in the $LOAD_PATH. I quickly fired up Limelight Playbills to load up the Limelight docs and found out in the Background tutorial that "the filename of the image should be relative to the root directory of the Production." Bingo. I quickly converted all paths by using an instance variable I created from Production, but ran into a little problem. Styles.rb does not seem to understand the production object, so I had to hardcode in the new paths

This is probably the biggest commit I've ever made to GitHub, albeit its non-feature change; it's nice to having a clean project again.

"Spicy Tuna" Hand Roll Mocks

I probably should've stopped blogging after the project structure topic, but I wanted to talk a bit on hand rolling mocks. Some of my RSpec tests needed a mock of an instance of WEBrick::HTTPRequest which is created on the fly when running WEBrick::HTTPServer. I ended up hand rolling my own mock because it was much easier than using RSpec's mock. Let's take a look at why it was easier. The mock was created in the before(:each) block for setup convenience. Here is the RSpec way:

@request = mock("request")
@request.should_receive(:query).and_return(hash = mock("hash"))
hash.should_receive(:[]).with('board').and_return('3x3')
hash.should_receive(:[]).with('player_o').and_return('human')
hash.should_receive(:[]).with('player_x').and_return('unbeatable')

Something doesn't taste right about the mock code above. First, I'm actually creating two mocks: one for the request and another for the query which contains a hash. Second, it's expecting hash to receive :[] message with three different arguments. Some of my test won't use all three, so it'll fail on "should receive". Let's take a look at when you hand roll your own mock:

Note: Thanks to Colin for pointing me in the right direction to create a class instead of creating an object and using instancevariableset to create the query ivar.

class MockRequest
  attr_accessor :query
  def initialize
    @query = {}
  end
end

describe BoardServlet do
  before(:each) do
    @request = MockRequest.new
    @request.query['board'] = '3x3'
    @request.query['player_o'] = 'human'
    @request.query['player_x'] = 'unbeatable'
  end
end

Doesn't that look much cleaner? It flows better than using RSpec mock. I think there's a special place for hand rolling your own mocks, but I don't think it should be done all the time. RSpec mocks do a great job by itself.

webrick - part 3

2010-08-03T21:59:00-07:00

I worked on my last two stories for this iteration, polishing one, and creating tests/code for the other. I'm a little worried my code is becoming more smelly as more features are being added to the project. Currently, all three UIs (console, Limelight, WEBrick) are in the same project. All specs live in the same spec folder, all console and shared classes live in the same lib folder, and Limelight is structured in its own unique way. This next iteration I plan to have a better layout of my source and tests.

Another thing I'm noticing is that there are some shared methods that need to be extracted and put into a shared library. I'm having a problem naming the class/module that would best fit the collection of these shared methods. Hopefully Micah can shed some light on this.

Colin gave some pointers how to test ERB stuff. I ended up extracting a heredoc string and putting it into its own file. Plus Ruby code in the ERB heredoc string has been extracted into its own methods so I can test them easily.

I'm constantly reading Gnome's Guide to WEBrick to understand WEBrick a bit more and it's actually a great guide to learn about WEBrick. Most of the content is making sense to me now and I think I have a good understanding on how things work in WEBrick. One thing I'll need to figure out is how to have a hyperlink mimick a human player move on the Tic Tac Toe. I've been playing with the idea of using query strings in the URL or somehow tagging value to a particular link clicked inside a Tic Tac Toe square. I'll need to do some more research.

webrick - part 2

2010-08-02T21:59:00-07:00

Today I researched some more on WEBrick, learning about servlets, get/post, response/request, and ERB. I was able to get a simple home page going for my Tic Tac Toe program. Similar to the other UIs I've worked on, this WEBrick version will also make sure MongoDB is properly installed before enabling the 4x4 option. The select inputs are retrieving their "options" from the config.rb I created for a recent story. Here is what I have so far.

The only problem I'm facing now is figuring out how to test WEBrick. And by that I mean, how to properly test that the options a user picks actually creates the proper board and players. I googled around a bit and found out Sermoa created a wiki called licky and created her own testing framework. I'll probably end up using net/http library inspired from her project.

ERB

To retrieve the data from config.rb to HTML, I googled around to find out how to run blocks in ERB. This Stack Overflow question helped me solve the problem. Here is how I pull the data:

<select name="board">
  <% TTT::CONFIG.boards.active.each do |board| %>
    <option><%= board %></option>
  <% end %>
</select>

Tomorrow, I plan to work on the story which calls for displaying the Tic Tac Toe board on WEBrick and start by creating tests first.

webrick

2010-08-01T21:59:00-07:00

Today, I did a bit of research on WEBrick, an HTTP server library that is part of the Ruby standard library. While it's no longer used in production (superseded by Mongrel) for Rails apps, it's still great for kicking off local development/testing environments or trying out new things. A couple of stories for this iteration involves having WEBrick serve my Tic Tac Toe program. I'll be using these resources I found on the web:

WEBrick Resources

TTT Configuration Story

I finished the configuration story which allows the user to configure how the board will cache the moves - in a MongoDB, Ruby hash, or nothing. I also added a nice feature that detects if a MongoDB is properly installed on the machine. If it isn't, it automatically disables the board which is configured to use Mongo.

RSpec

Instead of reading The Well Grounded Rubyist, I fell into reading the latest beta of RSpec (b15). There are some cool features I didn't know about that existed in RSpec. The first cool feature is how you can use pending in three different ways. I knew how to use one way:

it "should return 0"

Simple enough, since I did not provide a do block, it acts as a pending test. Here are the other ways to create pending tests.

it "should return 0" do
  pending "not yet implemented"
end

When RSpec reaches the pending method, it will stop execution and skip the test.

it "should return 0" do
  pending("bug report 123") do
    @documents.should == 0
  end
end

This will actually run the code in the block. What's great about this feature is that if the code actually passes, RSpec will raise a PendingExampleFixedError letting the user know that the pending is no longer needed because it's fixed.

I also needed to know how a mock can raise an exception.

it "returns false if MongoDB is not installed" do
  Mongo::Connection.should_receive(:new).and_raise(Mongo::ConnectionFailure)

  MongoCache.db_installed?.should == false
end

I needed to use this to mock an environment where MongoDB is not installed. I think my RSpec tests are looking better. I know some people are against mocking/stubbing and I'd like to find out why. Perhaps, excessive use of it can be bad or if you default to using mocks/stubs where a test can easily be written without them. I'll have to do some more reading.

ttt configuration

2010-07-31T11:00:00-07:00

Configuration file for Tic Tac Toe

My next story involves creating some sort of configuration file for my Tic Tac Toe program to centralize data and disable options like 4x4 board if MongoDB is not present. I have zero experience creating configuration files in Ruby. I've created config files for other little programs I've built, but they were in XML (yuck). I asked Micah how I should go about creating one. He said I can use a simple Ruby file that contains a module with constants. I took a step further and found this class someone created for creating a config object. It wraps a hash underneath, and automatically creates methods to access the hash by its keys using dot notation. I thought that was pretty cool. I'll see if it'll work for my needs. I ended up adding several methods to the class to access the hash's keys, values, and index.

Lunch 'n Learn - Node.js

Jim Suchy presented his 8LU session on Node.js. It was a good introduction to Node.js, an evented I/O technology for V8 JavaScript. V8 is Google's fast JavaScript engine, written in C++, currently used in Google Chrome, and Android browser (2.2+). I didn't quite understand what evented I/O truly meant, but after the session, I had a better idea. The slides demonstrating bunnies, hamsters, and the squid helped out. Near the end of the talk, Jim said Node.js might be overkill for some applications which do not have request bottleneck problems. It's also an overkill to create pure Node.js applications. He sees Node.js to complement web frameworks like Rails to handle long running processes and such to reduce request time. Our homework assignment due next Friday is to create a local Node.js server that makes HTTP requests to Flickr, GitHub, Twitter, Facebook, etc.

Pairing with David

David Putnam came in to the office today for the Lunch 'n Learn and I sat down with him briefly in the afternoon to fix a small RSpec problem he had on his Windows platform. He also uses IntelliJ for Ruby learning, so I showed him a bit of RSpec stuff I learned from the apprenticeship. I also gave him some pointers about applying RubyInstaller's DevKit to be able to install certain gems that weren't meant for the Windows platform. As a person coming from Windows, I know how difficult it can be to set up a Ruby/Rails environment in Windows. Unix-based users have it so easy!

The Weekend

I plan to work on the stories assigned in Iteration #5 to play a little catch up game. Also, from my apprenticeship retrospective, I plan to read chapters from The Well Grounded Rubyist (which should have been completed months ago), and work on a hobby project.

ui story complete

2010-07-29T19:00:00-07:00

The UI story is finally complete. Unfortunately, I had underestimated the story as one point; it actually took three points. I sent an email to the customer asking if he likes the new UI layout and mentioned the story took longer than expected, but I'm going to hold off on potentially warning the customer that not all stories will be completed for Iteration 5. Even I'm slightly behind, I'd like to wait until after the weekend. I'll hopefully be able to catch up then.

Here is the new Options scene:

Here is the new Board scene:

Most of the props you see now are image based using a font called Party LET on the Mac. Since the Windows platform does not support this font, I decided to use image based props. Personally, I think the new layout looks more playable, at the least, more pleasing to the eye than the awful gradient look. Hopefully the customer will enjoy it much more than the work involved. Conveniently, I packed the program as an *.llp file and it's available in the Downloads section of my GitHub repo. Unfortunately, the mongoDB needs to re-memoized since it was too big to store on the repo (~1.04GB). You should be able to easily play the 3x3.

The next story I'm going to work on involves creating a configuration file that enables/disables the 4x4 board. Until then...

ttt stories and clojure koans

2010-07-28T21:59:00-07:00

End of Iteration 4

Unfortunately, I was not able to deliver all stories (again). I completed the 4x4 Negamax story, but still had a couple of UI stories that I clearly underestimated (in points). Instead of warning the customer early on, I waited until the last minute which was a big mistake. I asked Micah when I should warn the customer, and he said immediately when you know. I think every day during an iteration, I will evaluate the number of points left and determine if I'll be able to complete all stories by the end of iteration. A simple warning to the customer is better than telling the customer at the last minute. Bad surprises are the worst thing to bring to the table at the end of iteration meeting. I've learned my lesson.

With that said, I continued working on the UI story that was supposed to be delivered for Iteration 4. I've got pretty far ahead and I have a better understanding of how Limelight works. Now it's a matter of churning through the UI bits to get it completed. I focused most of my effort today to work on the Options scene. I showed you yesterday what the new Board scene looks like. Here is the new Options scene:

I still need to implement some UI methods to give the user an indication that he/she is clicking on each option. The following items also still need to be implemented:

Quit button in Board scene
Try Again prop with Yes/No child props

I hope to get this completed by early tomorrow and start working on the WEBrick stuff to put my Tic Tac Toe on the web.

Clojure Koans at Chicago Clojure

Li-Hsuan and I took the train down to attend the Chicago Clojure meetup. This was my first Chicago Clojure meetup and 8th Light's Colin Jones led the Clojure Koans session for the evening. I wrote detailed instructions on how-to get started here. I really enjoyed learning my first functional programming language. I recently up an ebook version of Joy in Clojure, but haven't read it yet. This session was a great kickstart to learning the language. I highly recommend learning Clojure this way. It immediately gets you hands-on and you'll soon understand the syntax and language fundamentals.

pairing and ttt stories

2010-07-27T20:00:00-07:00

Paul asked me if I wanted to pair with him and I gladly accepted. The last time I paired with Paul was when I was in the 8th Light office for the first time before the apprenticeship offer. It was great pairing with him because I get more exposure to the Rails framework and other tools associated with it. For a couple of hours, we worked on a story and created acceptance tests using Cucumber and generated a controller, model, view with its respective specs. It was great getting some exposure to the variety of tools used in Rails projects. I had the opportunity to ask Paul many questions and he was kind enough to answer them all and elaborately.

After lunch, I went back to working on my UI stories. I had finally fixed those nasty failures in the specs after moving Board Type, Player O, Player X, Start Game, Exit to the new Options scene. There was a bit of trouble when I was fixing a method that removes the existing board on a new game. scene.children was not responding to the way I wanted it to when deleting child props. The code example below does not work:

scene.children.each do |p|
  scene.remove(p) if p.name == "something"
end

What ends up happening is when the first child is deleted, the children collection shifts up one, so it skips deleting the second one. During my debugging session, I saw it deleted every other prop. I talked to Eric Meyer about it and he seemed to have experienced the issue himself. He showed me a workaround solution - basically run through the children collection to find the props you want to delete and store them into an array. Then loop through the new array to delete them.

children_to_remove = []
scene.children.each do |p|
  children_to_remove << p if p.name == "something"
end

children_to_remove.each do |p|
  scene.remove(p)
end

My specs now pass and the game runs as great as it did before. I think I might miss my full 10 points for this iteration which I'm not too happy about. As you can see from the image above, the layout looks completely different, but it's not quite there to be delivered to the customer. I still need to add the Try Again message and add the Quit button to the Board scene. Also another story calls for keeping track of the scores. I'm going to push through the first half of tomorrow to get a lot of it done (hopefully). Iteration #4 ends tomorrow afternoon.

future stories for tic tac toe

2010-07-26T19:00:00-07:00

New Stories for Iteration #5 and #6

Iteration #4 is not near its end, but Micah and I sat down today and created more stories for my Tic Tac Toe project. The stories are going to be focusing on...drum roll please...throwing my Tic Tac Toe engine on a web app! Cool! I asked Micah a lot of questions regarding the stories. For one, I'm not allowed to use any external gems, just standard Ruby libraries. So goodbye, Rails, Sinatra, Rack, yadda yadda. Fortunately, I have erb on my side. I'm not sure how valuable it is to have erb at this point, but I'm glad I have the option to use it. I'll also be able to use WEBrick, which is that HTTP web server people use to test their applications in a dev environment. I'm going to have to do some reading about cookies and really understand how web applications used to work.

UI Story

I continued working on the UI story. It's coming along and so far I was able to get the options scene working. I still need to add the custom listboxes for the options. One problem I encountered in Limelight is that it couldn't find the mongo gem.

[master][~/local/git/tic_tac_toe_ruby] limelight open .
["/Users/skim/.rvm/gems/jruby-1.5.1/gems/limelight-0.5.5-java/lib", "/Users/skim/.rvm/gems/jruby-1.5.1/gems/jruby-openssl-0.7/bin", "/Users/skim/.rvm/gems/jruby-1.5.1/gems/jruby-openssl-0.7/lib", "/Users/skim/.rvm/rubies/jruby-1.5.1/lib/ruby/site_ruby/1.8", "/Users/skim/.rvm/rubies/jruby-1.5.1/lib/ruby/site_ruby/shared", "/Users/skim/.rvm/rubies/jruby-1.5.1/lib/ruby/1.8", ".", "/Users/skim/local/git/tic_tac_toe_ruby/lib", "/Users/skim/local/git/tic_tac_toe_ruby/lib"]
Exception in thread "AWT-EventQueue-0" /Users/skim/.rvm/rubies/jruby-1.5.1/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require': no such file to load -- mongo (LoadError)
    from /Users/skim/.rvm/rubies/jruby-1.5.1/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
    from /Users/skim/local/git/tic_tac_toe_ruby/lib/negamax_player.rb:9
    from /Users/skim/local/git/tic_tac_toe_ruby/lib/negamax_player.rb:31:in `require'
    from /Users/skim/.rvm/rubies/jruby-1.5.1/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
    from /Users/skim/local/git/tic_tac_toe_ruby/lib/player.rb:21:in `create'
    from /Users/skim/local/git/tic_tac_toe_ruby/default_scene/players/default_scene.rb:63:in `create_players'
    from /Users/skim/local/git/tic_tac_toe_ruby/default_scene/players/default_scene.rb:14:in `scene_opened'
    from /Users/skim/.rvm/gems/jruby-1.5.1/gems/limelight-0.5.5-java/lib/limelight/stage.rb:195:in `open'
    from /Users/skim/.rvm/gems/jruby-1.5.1/gems/limelight-0.5.5-java/lib/limelight/producer.rb:128:in `open_scene'
    from /Users/skim/local/git/tic_tac_toe_ruby/options_scene/players/options_scene.rb:23:in `play_new_game'
    from (eval):1:in `instance_eval'
    from /Users/skim/local/git/tic_tac_toe_ruby/options_scene/players/button.rb:12:in `mouse_clicked'
    from :1
    ...internal jruby stack elided...

I had Micah come over to troubleshoot the problem and I found out that Limelight does not look at the system gems. Instead you'll need to pull in the external gems you want to use into your project similar to how Bundler works. By running this command, it will automagically pull in the gem:

[master][~/local/git/tic_tac_toe_ruby] limelight freeze mongo
    creating directory:  ./__resources
    creating directory:  ./__resources/gems
    creating directory:  ./__resources/gems/gems
    creating directory:  ./__resources/gems/gems/mongo-1.0.5
    unpacking gem:       ./__resources/gems/gems/mongo-1.0.5
    creating directory:  ./__resources/gems/specifications
    creating file:       ./__resources/gems/gems/mongo-1.0.5/limelight_init.rb

You'll need to do this for dependencies as well:

[master][~/local/git/tic_tac_toe_ruby] limelight freeze bson
    creating directory:  ./__resources/gems/gems/bson-1.0.4
    unpacking gem:       ./__resources/gems/gems/bson-1.0.4
    creating file:       ./__resources/gems/gems/bson-1.0.4/limelight_init.rb

I think in the future, Limelight will incorporate Bundler to handle external gems.

Apprenticeship Retrospective

It's amazing how fast time flies. I've been here for almost two months now, and it feels like I just started a couple of weeks ago. Micah put two columns on the whiteboard, "Good" and "Improve On". He and I both took turns writing things down on both columns. Overall, I believe I'm on the right track. To stress on what needs to improve, here is the list we wrote down:

TDD
story estimation
continually read books
focus/time box/pomodoro
more pairing time
wear all kinds of hats
in-depth geekiness on current topics
0 point iteration
meet commitments
hobby project
sustainable pace
attend more user groups

As you may recall, one particular week, I came out of an iteration completing zero points. How can anyone do that? I need to practice estimating the stories better and give them points that allow me to work at a sustainable pace. I'm going to have to figure out how to work on 10 points a week, work on a hobby project (which I can easily pick from my long list), read a book, attend more user groups, and play the ukukele with my feet. Okay sans the last part. Did I tell you I love this apprenticeship program? No seriously, I'm learning so much, it's rewarding to learn from a group of passionate craftsmen. :)

a working solution to 4x4 negamax

2010-07-25T21:00:00-07:00

I have a working solution for the 4x4 Tic Tac Toe Negamax player. After a week of yak shaving, I found a way to store the 4x4 Negamax moves into a database (MongoDB) and have it play against any player including itself.

It required the following:

Patience
Negamax algorithm with memoization with a dash of hash buffering
MongoDB indexing (super easy to create)
Patience
A supercomputer (or decent dual-core)

Initially, I was only going to grab the positions 0, 1, 4, 5 and get the other positions by rotating/flipping the board. I found out during the process, however, that it was not needed and memoization pretty much took care of the rest. Once I had 0, 1, 4, 5, it was very easy to get the other 12 positions.

 0 | 1 | 2 | 3    3 | 2 | 1 | 0
---+---+---+---  ---+---+---+---
 4 | 5 | 6 | 7    7 | 6 | 5 | 4
---+---+---+---  ---+---+---+---
 8 | 9 | 10| 11   11| 10| 9 | 8
---+---+---+---  ---+---+---+---
 12| 13| 14| 15   15| 14| 13| 12

 12| 13| 14| 15   15| 14| 13| 12
---+---+---+---  ---+---+---+---
 8 | 9 | 10| 11   11| 10| 9 | 8
---+---+---+---  ---+---+---+---
 4 | 5 | 6 | 7    7 | 6 | 5 | 4
---+---+---+---  ---+---+---+---
 0 | 1 | 2 | 3    3 | 2 | 1 | 0

 3 | 7 | 11| 15   15| 11| 7 | 3
---+---+---+---  ---+---+---+---
 2 | 6 | 10| 14   14| 10| 6 | 2
---+---+---+---  ---+---+---+---
 1 | 5 | 9 | 13   13| 9 | 5 | 1
---+---+---+---  ---+---+---+---
 0 | 4 | 8 | 12   12| 8 | 4 | 0

 0 | 4 | 8 | 12   12| 8 | 4 | 0
---+---+---+---  ---+---+---+---
 1 | 5 | 9 | 13   13| 9 | 5 | 1
---+---+---+---  ---+---+---+---
 2 | 6 | 10| 14   14| 10| 6 | 2
---+---+---+---  ---+---+---+---
 3 | 7 | 11| 15   15| 11| 7 | 3

I have 14,080,239 documents in my MongoDB. This includes 3x3 and 4x4 board states and roughly equates to about 1.04GB. It took an estimate of 5-6 hours to populate the database, running two different games at the same time on a blank board. It pretty much maxed out my CPU. I was getting a little worried the mongod process was going to max out my memory, but fortunately it did not. I think it went all up to 2.5GB in terms of memory usage. I'm storing three fields to each document:

document = {"board" => "OXOXOXOXO", "piece" => "O", "score" => 1}

Initially when running the process, it was storing the documents at a slow pace of about 100 documents/sec. I read on mongodb-user Google Group that it was not recommended to create the index before a mass data load. I didn't think I was going to have 350 million documents so I created it by using a simple command in an IRB session:

ruby-1.9.2-preview3 > coll.create_index("board")
 => "board_1"

Wow. It significantly boosted the performance. It was storing at a rate of about 2000+ documents/sec and it kept increasing. Now since I created the index early on, when new documents are written to the database, it also indexes them at the same time. I still think creating the index before will be faster than creating it after. I needed the performance right away since writing 14 million documents can take some time.

My RSpec test suite passed and I checked in the latest code. I'm happy to finally move this story from Working phase to Complete. Now on to the UI story...

continuous yak shaving - lessons learned

2010-07-24T10:00:00-07:00

This was a wonderful week of yak shaving. Experimenting with MongoDB, memoization, memory allocation optimization, score randomization, all while in a ~~recursive~~, ~~recursive~~ recursion, led to some important lessons learned.

Continuously run your tests

Your tests are there to help you understand if the changes you've made in the code broke existing functionality. When introducing memoization, memory allocation optimization, score randomization to my NegamaxPlayer class, I failed to add and test them individually. The course of action I should have taken:

Write the test for new method
Run the test, see it fail
Add code for new method
Run the test, see it pass
Run the entire test suite
Commit changes
Repeat Step 1

Start with a clean slate, if your tests are failing

Sometimes, when you're knee deep in your newly written code, the tests aren't passing, and you're getting super frustrated, it's best to start over. After a long day of attempting to fix tests and frustration ensued, I ran the wonderful command git checkout -- file and started over. I added the new methods individually and made sure each method passed all tests before moving on to the next. Seeing green once again put me in a really good mood.

Truly understand how algorithms work

I know this sounds obvious, but while working on the Negamax algorithm code, I sometimes get lost inside the recursion (I think I need to read The Little Schemer. On any given day I'll understand how it works, but there are times when I'll forget and won't quite understand it. Without truly understanding the algorithm, I'm at a disadvantage and solving problems for it can be difficult. I know since last month when I started working on it, I have a much better understanding of the algorithm. I'm not quite sure how to improve my understanding of it, except by reading the code everyday. I guess it's just a matter of time.

Use your math skills, estimate completion time

When working on storing all possible 4x4 board states in MongoDB, I failed to calculate how long it will take for me to store the moves to the database and if the generated board states were redundant. After running the code for about a day, I calculated the completion time to be about 150 years which in no way was going to be finished...in my lifetime. Had I calculated before running the code, I could have saved A LOT of time. I think I've lost a total of 4 days by not using simple math skills to understand the root of the problem. This was probably one of the hardest lessons learned.

Work on one story at a time, perform commits frequently

This was part of my retrospective for Iteration #1, but it needs to be said again. I tend to work on a story, leave it uncomplete, and work another story. When I attempt to go back to the first story, I don't remember what I did, worse, I don't remember how to fix the failures that were introduced from before. Of course, everyone works differently, and some will be able to leave unfinished stories behind and work on them later. As I learn the ropes, I think the best way to manage is to work on them one at a time. If I'm working on a story, I'll break it down to smaller tasks and commit frequently. Or if the story can be broken down into multiple stories, I can work on one, break it down to smaller tasks and commit frequently. One of the 8th Lighters told me he likes to have a story anywhere from 1-3 points. Anything higher and he breaks them into smaller stories. I think when stories are kept small, they're easier to manage and rewarding when you completed them. Having one big story staring at you in its incomplete state can be daunting and frustrating.

Conclusion

There were some good lessons learned this week. I can't say I'll ever stop yak shaving, but with time comes discipline and hopefully less yaks will need to be shaved. Happy weekend!

storing 4x4 on mongodb - part 3

2010-07-23T09:00:00-07:00

I'm back to storing 4x4 possible board states to MongoDB. If you recall from the previous posts, I spent days working on what I thought was the solution, when in fact I was attempting to store trillions of board states which contained a lot of redundancy. I removed alpha-beta pruning and added memoization using MongoDB to read/write board states and its respective best scores. The best thing about using memoization and MongoDB is that if I were to stop the method, I won't lose the work it has previously done. I can stop and resume at any time. I'm in the process of optimizing the Negamax algorithm; Micah pointed out that my code was creating new arrays at each recursive call which allocates memory for each array. Instead of creating new arrays for every board, I plan to "undo" the board move when it's completed the score for that move. This should speed things along. It's been running almost half a day and has only stored ~310,000 board states vs. the ~130 million it used to store. It's either running a lot slower than the previous algorithm or there's a lot of redundancy.

I'm also in the process of making my RSpec tests pass again. When I was working on the UI story and changing the GUI around, my tests that were looking for those GUI fields started to fail. I have a bad habit of leaving a story and working on another and not remember what I did in the previous story. I plan to focus my time today to fix these failing RSpec tests.

end of iteration 3

2010-07-22T08:00:00-07:00

"Accept that there will be some things that you’re not good at, or that would require a disproportionate investment of time and effort in order to make a small improvement." - Apprenticeship Patterns, Learn How You Fail

Stories Not Delivered

Unfortunately, I was unable to deliver the two stories in Iteration 3. I assumed I was able to write all possible board states to MongoDB. If there are 16! possible boards, then the time to write them all to MongoDB would take approximately 150 years. I will not complete this story in my lifetime. I should have thought about the problem a bit more than just relying on the little information I already knew. Since the MongoDB problem took up most of the time, I was unable to complete UI story. All in all, there were some lessons learned and I hope to better my understanding of the problem.

These stories were pushed to Iteration 4, points were re-evaluated, and another story was assigned. The new story is about keeping track of wins and losses and displaying them to the user. This will require another Limelight stage and possibly a new collection in the same MongoDB.

Internalization (i18n)

The other apprentice, Li-Hsuan, gave his presentation on internalization. It's important that all software development utilize UTF-8 as a standard, especially in this day and age, where more and more countries worldwide, connect to the internet and utilize more applications daily. He presented several challenges on improper character encoding, improper encoded data in the database, failed multilingual FitNesse tests, and autocompletion feature in a multilingual application. It was interesting to see the solutions to the problem and to see how some tools do not support internationalization.

Pairing with Micah

Later in the day I paired with Micah and I showed him a potential solution to the 4x4 problem. The board can be rotated and flipped seven different ways and could possibly look the same as the original board. If I were build the trees for positions 0, 1, 4, 5, then I should be able to get the trees for all board positions. See below.

 O | O |   |
---+---+---+---
 O | O |   |    
---+---+---+---
   |   |   |   
---+---+---+---
   |   |   |

Position 0 should be the same as 3, 12, 15, position 1 should be the same as 7, 8, 14, position 4 should be the same as 2, 11, 13 and position 5 should be the same as 6, 9, 10. While I only need to build 1/4 of the board, it will still take a long time. Micah then introduced me to memoization. I read about memoization in Douglas Crockford's The Good Parts, but I didn't understand a good use case for it. The Negamax problem is a great example for understanding the power of memoization. When implemented, it was nearly identical to alpha-beta pruning. Memoization takes the redundancy out of the equation. In the 3x3 problem, alpha-beta pruning made 18,296 moves in the tree on a blank board, then 843 moves in the next, but memoization made 16,167 moves and 7 moves in the next! Clearly memoization is a better optimization than alpha-beta pruning. I think I will use the rotating/flipping idea with memoization so I will only have to search 4 positions of a blank board.

I'm glad to have paired with Micah. Memoization was a great eye-opener and I will blog about the level of optimization it brings to the 4x4 board.

storing 4x4 on mongodb - part 2

2010-07-20T19:00:00-07:00

"If you count something interesting, you will learn something interesting" - Apprenticeship Patterns, Create Feedback Loops

Writing to MongoDB Fail

Note to self: If you're going to generate 16! board states for a 4x4, make sure the associated values are correct after the first thousand generated.

I failed to analyze the generated data stored in my local MongoDB and found out after writing 122 million documents that the best moves were stored as NULL. It was because of an unfamiliar territory called block scope. In JavaScript, there is no block scope which I'm used to. Functions are pretty much the only scope. However, in Ruby, block scope exists. So what I failed to do in the method is create an initial value outside the block. D'oh.

I was also displaying each board state in a puts method and used JRuby to execute the method. Since JRuby does not support C-extensions at this time, it means it does not support a gem called bson_ext which accelerates the Ruby BSON serialization for MongoDB. Kyle Banker is working on a Java version of bson_ext, but I'm not sure when it'll be ready. Reading yesterday's Ruby shootout revealed Ruby 1.9.2 to be among the top performers, so I installed 1.9.2-preview3 using rvm and installed bson_ext gem. I'm now executing the method again. One thing to note, I fixed the block scope issue above by setting the best_move variable to -1. However, I noticed some board states still had a best_move == -1 when it should have changed to a value 0-15. After some time investigating, I found out the reason why it was -1 was because the board was in a state where it was not Negamax's turn. Meaning the recursive call was for other player's move. Since I only care about board states when it's Negamax's turn to move, I don't need to store these board states to MongoDB. This should shave off time and storage space.

Limelight UI

To experiment placing images on my Limelight UI default scene, I quickly created several background images to replace my ugly gradient background for my Tic Tac Toe program. It's pretty easy to set the background image. In the styles.rb file, I just added this snippet:

default_scene {
  background_image "images/background_nuclei.jpg"
}

I'm going to experiment with adding the board lines and pieces to the UI and attempt to create new stages for the options and "try again" dialog.

storing 4x4 on mongodb

2010-07-19T19:00:00-07:00

I worked on a method to write the best moves for each board state starting from an empty board. If you think about it, there are 16! possibilities. As of this writing, there are 28,782,679 documents stored on my local mongodb. Holy cow. Li-Hsuan and I did some quick math and it's about 0.02GB now, soon to reach a hypothetical 25GB. I still need to index the database and test the performance. I'm hoping it's not going to be slow.

Here is the added code to the NegamaxPlayer class:

# ...
require 'mongo'

class NegamaxPlayer < Player

  attr_accessor :best_move

  def initialize(piece)
    super(piece)
    @conn = Mongo::Connection.new
    @db = @conn.db('test')
    @coll = @db.collection('coll')
  end

  # ...

  def store_moves(board, piece, depth, alpha, beta)
    # ...
        if score > alpha
          alpha = score
          best_move = s
          @best_move = s if depth == 1
        end
    # ...
    @coll.insert({"board" => board.to_a, "best_move" => best_move})
    # ...
  end

I haven't started on the other story which involves creating new Limelight scenes and adding a new theme. I'm going to work on the themes sketches using Pixelmator and see how I can add it as a prop to my Limelight UI.

day 2 of ruby midwest

2010-07-19T10:00:00-07:00

Here are my favorite talks for Day 2 of Ruby Midwest. Yehuda Katz's keynote was similar to Chris Wanstrath's, touching on the importance of the Ruby community. He called the Ruby community, "a blessing and a curse" because it :

enjoys experimenting with new projects
does not put up with crap
expects basic commands to run without reading the manual
expects a high level of UX
expects support for all platforms, unique environments (e.g., Windows, JRuby, Passenger, rvm, Unicorn, gemcutter)
expects the project to work with other gems
expects maturity of a project
expects proper maintenance and frequent releases

He listed four phases of his life as a Ruby developer:

Phase One : Noob
Phase Two : Aspiring Plugin Author
Phase Three : OSS Contributor
Phase Four : Full-time OSS at Engine Yard

Like everyone else, he started off as a noob, using Windows as his primary OS, building small Rails apps for a non-proft. He called it a time of confusion, because he did not know things like methods, etc. But of course, that didn't stop him from becoming an aspiring plugin author, building plugins such as autodb and yhtml. As an OSS contributor, he contributed to projects like DataMapper and Merb. Merb had a following and Rubyists were switching to Merb from Rails because they love trying out new projects, even though the project was not stable and contained bugs. While disrupting the Ruby ecosystem (as Merb did) can be a good thing to bring constructive competition, it can also breed arrogance and can leave stable, working projects unnoticed (Rails). In the case of Merb though, features that were great in Merb, integrated into Rails after the Rails-Merb merge.

Yehuda strongly encourages OSS contributors to

give high-level UX to the users
make sure basic commands work and does not require a manual
make sure all platforms, special environmental factors are supported
be excited if the first project takes your ideas (e.g., Rails vs Merb)
continue contributing to the project for the long haul if you're the primary maintainer
continue contributing to the project even after the "hype" dies
make sure the next person is in it for the long haul if they decide to jump ship
announce the project is no longer maintained and in need of a new maintainer

Yehuda strongly encourages the Ruby community to

reward long-term maintainers
encourage up-and-coming projects to mature
push mature projects to improve UX
become a maintainer
expect reading the manual for complex commands

Wesley Beary also had a similar talk on UX. Here are some bullet points:

do not worry about competition (it's a good thing)
"tutorialize" - as a maintainer of the project, help out as much as possible (e.g., IRC)
encourage contributors

Aman Gupta talked about memprofiler. I listened to his talk with Joe Damato at LA Ruby earlier this year, but most of it was over my head. Listening to it again months later, I mostly understand the power of memprofiler and how it can help you find bottlenecks in your code. Here are my notes:

Ruby's GC (garbage collector) does not scale as object count increases
More objects == longer GC time, fewer objects == better performance
When GC is running, nothing else runs which pads response time.
useful tools
1. ObjectSpace.each_object
2. ObjectSpace.count_objects (in Ruby 1.9)
3. gdb.rb: gdb hooks for REE (http://github.com/tmm1/gdb.rb)
  - linux only
4. bleak_house
  - http://github.com/fauna/bleak_house
  - installs a custom patched version of ruby
  - tells you what is leaking, and also where
5. heap dumping patches
  - simple patch to ruby VM
  - http://gist.github.com/73674
  - simple text based output format
6. memprof
  - easy to use
  - no patching the VM
  - just require the gem
  - includes file/line, object contents, info about references between objects
  - simple analysis
  - allow processing via various languages and databases using simple JSON data format
  - http://github.com/ice799/memprof
  - gem install memprof
  - use ruby 1.8.x
  - under active development
  - how it works
  - rewrites your ruby binary in memory
  - injects short trampolines for all calls to internal VM functionals to do tracking
  - more info - http://timetobleed.com
  - http://bit.ly/memprof-abi
  - memprof features
  - memprof.track
    - Memprof::Middleware
  - memprof.dump
    - detailed info about block in JSON
  - memprof.dump_all
    - dumps out every single live object as json
    - one per line to specifed file
  - http://memprof.com
    - web-based heap visualizer and leak analyzer
    - built using rails 3 app

Unfortunately, I didn't stick around for Cory Flanigan's Zero to Hero talk which I heard was really good. Overall, the conference was great. I'm glad I attended and look forward to next year's. Hopefully by then, I'll give a talk on something. Kansas City is a great city. I look forward to coming back again soon for a nice weekend vacation. If you ever decide to visit to KC, be sure to check out Fresher than Fresh Snow Cones.

day 1 of ruby midwest

2010-07-17T10:00:00-07:00

Here are my favorite talks of the day. Starting with Chris Wanstrath (@defunkt)'s keynote, he listed ways to improve OSS projects and ways to improve the overall experience to the user. He gave project examples of what worked and what didn't work. Here is a partial list of ways to improve OSS experience:

be a lazy maintainer - write a road map of the features you want in the project
convert README to man pages
create a maintainer/contribution guide
use semantic versioning
(too few and too many releases) == bad
it's much harder to remove a feature than to accept it
non-maintained projects should SAY SO!
documentation is a good thing
make installation process super simple
make the API human-friendly
always have a license file for projects
google your project name before you make it official
small projects should not have a domain especially if you do not plan to renew the domain fee

Projects that work

Rails
jQuery
Homebrew
DJango
Unicorn

JRuby by Charles Nutter

Lately I've been using JRuby for my Tic Tac Toe project, so this talk was useful. Here are my notes from the talk:

supports native threads with concurrency
trinidad is a thread-safe, one-process (gem)
run in thread safe mode
scaling is awesome with one process!
-J-Xmx128M
Use any Javal library like it’s a Ruby library
No-compile single-file deployments
Cross-platform GUI libraries
tooling and maturity
- debugging tools
- profiling, memory tools
- commercial jvm environments
tools
- jruby -J-Xrunhprof:format=b,depth=15 -J-Djruby.reify.classes=true memory_example.rb
- jhat -J-Xmx200M java.hprof
- go to http://localhost:7000/
- jvisualvm
new frontiers
- google app engine
- java, python, jruby
- android
- java mostly (or jruby)
- anywhere with a java server
- deploy like any other application
ruboto
- gem install ruboto
- ruboto generate app
- ruboto release
three full-time devs at engine yard

There are two Ruby Summer of Code projects that are contributing to JRuby. One of them is support for C extensions in JRuby. As of now, a good number of tests passing for Thin. The other project is for the Ruboto project which allows you to write Ruby code to create Android apps. Soon enough, you'll be able to install the ruboto gem, generate the app, write Ruby code, and release it to the Android Market.

Blog posts on jmap/jhat, VisualVM here: [1][2]

Unobtrusive CSS by John Hwang

I'm a CSS noob, so this was a nice opener to CSS and understanding the underlying problems people experience with plain CSS. There are CSS frameworks, semantic/meta frameworks that try to resolve the problems in CSS:

lack of abstractions
repetitive selectors (not DRY)
difficult to control layouts
efficiency/organizational issues
code re-use is non-existent
code organization is non-existent
file size just keeps getting better
difficult to maintain or change

CSS frameworks...

promote grid layouts
best practices
sensible defaults
reset CSS
typography

but problems are present:

nothing more than a collection of CSS snippets
tight coupling of presentation and content
almost as bad as using table layout

In John's (@tavon) point of view, CSS meta frameworks are the way to go, especially Sass.

Sass...

has nested rules
has variables
has mixins
compiles and outputs CSS
fully CSS3 compliant (SCSS syntax)
Turin complete language
Firebug integration
selector inheritance

With meta frameworks like Sass, you can build unobtrusive stylesheets which allows:

faster implementation
faster change design
flexibility for theming and alternate stylesheets
easier redesigns

Lightning Talks

During the OMGWTF BBQ, there were 7 or 8 lightning talks.

Mirah by Charles Nutter

Mirah (formerly Duby) is a new programming language that allows you write code that looks syntactically like Ruby, but compiles down to fast JVM bytecode. Charlie (@headius)gave a couple of performance examples. Mirah was pretty fast compared to Ruby MRI. I look forward to seeing this project grow and plan to try it out in the near future.

Fog by Wesley Beary

Wesley (@geemus) presented his cloud computing gem that helps interact with cloud services easily. It currently supports the following cloud-based services:

Amazon Elastic Computer Cloud (EC2)
Amazon Simple Storage Service (S3)
Amazon SimpleDB
Rackspace Files
Rackspace Server
Slicehost
Terremark vCloud Express

I will post about day 2 tomorrow.

mongodb

2010-07-15T19:00:00-07:00

I spent all day today looking at the MongoDB documentation, playing around with the commands in Try MongoDB and irb sessions. I said I was going to try MongoDB Koans, but thought the documentation was good enough to get started. You can read about the MongoDB experience on my other blog. I have a clear head for what I want to do to write all of the possible moves for the 4x4 board. I'm going to try without create an index to see if there's a dramatic performance increase when indexing. I also plan to write the moves to the database and export the Mongo collection as a *.bson file which you can easily do with the mongodump --db dbname --collection collname command. This command can be run even if the database is running and active. Then, I'll somehow use mongorestore to restore the collection.

I'm headed to Ruby Midwest early tomorrow morning. I will blog about the talks and hackery.

end of iteration 2

2010-07-14T14:00:00-07:00

NoSQL spike

Today I looked over some of the NoSQL data stores out there, specifically, Redis, CouchDB, and MongoDB. In the end, I chose MongoDB to store the possible moves for the 4x4 board. Why MongoDB? Just personal preference. I said yesterday I wouldn't use it because I didn't have any available 8th Lighters to help me, but honestly the docs out there are well written and give you all the information you need to get started.

Speaking of which, I'll be using these links to guide me to Mongo heaven:

I read the MongoDB Quickstart guide to learn how to install MongoDB to my MacBook Pro via homebrew:

brew install mongodb

Be sure to have the latest brew formulas or you might get an older version of MongoDB. To do so, run these commands:

[/usr/local] git init
Initialized empty Git repository in /usr/local/.git/
[master][/usr/local] git remote add origin http://github.com/mxcl/homebrew.git   
[master][/usr/local] git pull origin master
From http://github.com/mxcl/homebrew
 * branch            master     -> FETCH_HEAD

Install mongo gem:

gem install mongo

Create a folder and kickoff an instance of MongoDB:

[~] mkdir -p /data/db
[~] mongod
mongod --help for help and startup options
Thu Jul 15 00:31:03 Mongo DB : starting : pid = 82628 port = 27017 dbpath = /data/db/ master = 0 slave = 0  64-bit 
Thu Jul 15 00:31:03 db version v1.4.4, pdfile version 4.5
Thu Jul 15 00:31:03 git version: 9dcac11bd72c7cb34ca278313f033efcdd4908eb
Thu Jul 15 00:31:03 sys info: Darwin erh2.10gen.cc 9.6.0 Darwin Kernel Version 9.6.0: Mon Nov 24 17:37:00 PST 2008; root:xnu-1228.9.59~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_37
Thu Jul 15 00:31:03 waiting for connections on port 27017
Thu Jul 15 00:31:03 web admin interface listening on port 28017

In another Terminal instance, run this simple example to test out the MongoDB installation and mongo ruby driver:

require 'rubygems'
require 'mongo'

include Mongo

db = Connection.new.db('ruby-mongo-examples')
coll = db.collection('test')

# Erase all records from collection, if any
coll.remove

# Insert 3 records
3.times { |i| coll.insert({'a' => i+1}) }

puts "There are #{coll.count()} records in the test collection. Here they are:"
coll.find().each { |doc| puts doc.inspect }

# Destroy the collection
coll.drop

See the results

[~/local/ruby] ruby mongo_simple.rb

**Notice: C extension not loaded. This is required for optimum MongoDB Ruby driver performance.
  You can install the extension as follows:
  gem install bson_ext
  If you continue to receive this message after installing, make sure that the
  bson_ext gem is in your load path and that the bson_ext and mongo gems are of the same version.
There are 3 records in the test collection. Here they are:
{"_id"=>BSON::ObjectID('4c3e9de8cc9ed64326000001'), "a"=>1}
{"_id"=>BSON::ObjectID('4c3e9de8cc9ed64326000002'), "a"=>2}
{"_id"=>BSON::ObjectID('4c3e9de8cc9ed64326000003'), "a"=>3}

I didn't get much further than that. I will be working on the MongoDB Ruby Koans tomorrow to get a better understanding of MongoDB and usage patterns.

End of Iteration 2 Meeting

I sat down with Micah (customer) to go over the two stories I was assigned for Iteration 2:

Implement 4x4 board
Create UI mockups
Improve performance of Negamax player in 4x4 board (spike)

I think overall he was satisfied with what I've done to the 4x4 board and UI mockups. I estimated the new stories to be about five points each:

Implement UI mockups
Improve performance of Negamax player in 4x4 board

There are still a couple of stories in the Backlog which I'll work on in Iteration 4. But for this new iteration, I will be focusing on the two stories.

ui mockups, nosql, code retreat

2010-07-13T20:45:00-07:00

UI Mockups

I met with the customer today to go over the UI mockups and overall he seemed happy with them. I need to add a Try Again yes/no overlay on the board and add tasteful images to the board and pieces. He still wants an unprecedented, eye candy look to the game, so if I can't find any Creative Commons images on the web, I may have to create them from scratch using an image editor. I wonder if I can make the board and pieces look 3D. I'll have to do some research.

Breaking Open/Closed Principle

When I showed Micah (mentor) the 4x4 board option, I explained to him about the 4x4 complications of the regex pattern matching for the CpuPlayer class. He indicated that the EasyCpuPlayer and NegamaxPlayer can be extensible (open/closed principle) when changing things like supporting a 4x4 board, but the CpuPlayer cannot. I'm not sure if there is a way to make it extensible (e.g., throwing the patterns into a module and including it in CpuPlayer) or if pattern matching is just not the way to go. Either case, it still tries to win/block when it sees the pattern and is code-wise, better than the EasyCpuPlayer. I have spare time to put effort into the performance of the Negamax against the 4x4.

NoSQL for Negamax

I told Micah (mentor) that it takes the Negamax player approximately 68 minutes to make its first move. He asked me if I knew ways of improving the performance and hinted that the NegamaxPlayer is repeating itself every game. I thought storing it in a database didn't work when Colin recently talked about his solution. I got it all wrong, however. Building the index and writing the possible moves to the database took the longest. When playing the game, however, the lookup was actually really quick. Since NoSQL is the rage these days, Micah wanted me to spike and look into using a NoSQL data store. Awesome! What a great way to experiment with non-relational data stores! I have some experience reading/writing to relational databases, so it'll be interesting how non-relational data stores work in comparison. Of the NoSQL data stores out there, Redis), CouchDB, MongoDB, and Cassandra) were mentioned by some 8th Lighters who have used them. I think I'm going to use CouchDB or Redis. I hear MongoDB an awful lot in Twitterverse, but the 8th Lighter who has experience using it is on vacation. Rats. From the spike, I should have an understanding of what data store to use. More on that later.

Code Retreat with Corey Haines

Corey stopped by the 8th Light office and guided us on a session of his code retreat. It's the original code retreat that inspired the code retreat sessions I participated at BohConf (at RailsConf) this past month. We all paired up and TDD'd to our heart's content in an attempt to create Conway's Game of Life. Here are some notes I wrote down from the session:

don't immediately jump to abstractions
create smaller scopes
test like breathing
reveal intent
create smallest test/code possible

The code retreats are about minimizing the gap between what you wish you could code and what you actually can code within a given time. The gap is how much you suck.

At the end of the session, we threw away our code in a habitual effort to detach ourselves from the code. I'm going to try these code retreat sessions for the Game of Life challenge and see where it leads me.

ui mockups

2010-07-12T17:30:00-07:00

Tic Tac Toe - 4x4 board

The 4x4 board implementation on Limelight was fairly straightforward. I learned about the build command in Limelight to build props when a player is triggered. This allows the user to select the board (3x3 or 4x4) and upon clicking the New Game button, it calls this method:

  def build_squares
    @board.ranges.each do |r|
      scene.build do
        row do
          r.each do |s|
            square :id => "square_#{s}"
          end
        end
      end
    end
  end

CpuPlayer class required a bit more effort because of the regular expression pattern sets. I created a basic version of the pattern sets for the 4x4 board. It currently only knows when to win and block, but nothing in between (e.g., building itself a win). The 8th Lighters around me suggested to create a true rules-based logic. If I have more time, I'll look into this, but as of now, this should suffice. Here's what it looks like now:

UI mockups using Balsamiq

I created a couple of mockups on what I wanted the Tic Tac Toe UI to look like. To be honest, it looks pretty much the same with an exception to the options (e.g., board size, players, start game button) which were moved to another scene. Here are the mockups thus far. I sent them to the customer so that he may review them and see if it's what he wanted. I have a feeling he'll want something completely different. I'll let you know.

Options

Board

Ruby Midwest

I'm attending the first Ruby Midwest conference in Kansas City later this week. It's a two-day, single track conference, packed with some great talks. I don't know about you, but I love single track conferences. It's rewarding to be able to listen to every single talk. I look forward to meeting more Rubyists and experience the Kansas City BBQ.

4 x 4 board

2010-07-11T21:30:00-07:00

The 4x4 board option is almost complete with exception to updating the rules-based patterns for CpuPlayer class and adding the Limelight UI elements. It seems to be working fine in stdin/stdout. The only problem I'm running into is testing the Negamax player on a 4x4 board. The first move just takes way too long! A 3x3 board has about 9! or 362,880 possible moves. A 4x4 board, however, has 16! or 2.09227899 × 10^13 possible moves. I helped it out by moving first as a human, so it really has 1,307,674,368,000 possible moves which is about 3,603,600 times more possible moves than the 3x3 board.

I'm running the Negamax player on a 4x4 board and for the first time since I upgraded to this MacBook Pro, I'm hearing the fans spin. This computer is working hard to compute those trillion moves. It's been about 5 minutes so far and it still hasn't made its first move. I'm going run it over night just to find out how long it takes to make the first move.

Update: it took Negamax player 4106.039 seconds or roughly 68 minutes to make the first move.

Choose player type for 'O' (enter 'h' for human or 'e' for easy cpu, 'm' for  medium cpu or 'u' for unbeatable cpu) h

Choose player type for 'X' (enter 'h' for human or 'e' for easy cpu, 'm' for  medium cpu or 'u' for unbeatable cpu) u

Choose board type (enter '3' for 3x3 or '4' for 4x4) 4


   |   |   |   
---+---+---+---
   |   |   |   
---+---+---+---
   |   |   |   
---+---+---+---
   |   |   |   


Enter your move, player 'O': 0


 O |   |   |   
---+---+---+---
   |   |   |   
---+---+---+---
   |   |   |   
---+---+---+---
   |   |   |   

Player 'X' is making a move

Tomorrow, I plan to finish up the story and work on the mock-up story afterwards. I'll post an update on how long it took Negamax player to make its first move.