Tapestry Central

Caught between Two IDEs

2009-07-02T13:10:00.000-07:00

I seem to be caught between two IDEs: Eclipse and IntelliJ. I abandoned Eclipse a couple of years back, partly based on wide spread recommendations from many different people, and partly because Eclipse just stopped working for me (it crashed out).

After I got started with IntelliJ I started to appreciate its merits, despite a generally clunky interface (with lots of modal windows), truly awful documentation. Many things are streamlined and only a ctrl-alt-shift-coke-bottle-touch-your-nose away.

However, over time, using IntelliJ got slower and slower and slower. It also started running the Tapestry test suite horrifically slowly: 40 minutes and up (it should be about five). It would often go away, even when memory wasn't tight. Indexing? Checking Repositories? Computing primes? No way to tell.

Meanwhile, Eclipse has been moving forward, with Eclipse Galileo being a Cocoa (not a Carbon) application. Critical plugins such as M2Eclipse have gotten nice, and the Clojure plugin is mostly better than the IntelliJ one (though both are very early).

For a while I was using IntelliJ when teaching Tapestry (as part of the VMWare image I use when training) ... and I got a lot of resistance. People were much happier with Eclipse on the last couple of go-rounds, and I'm sticking with it.

Overall, I'm feeling that most of what I've grown used to in IntelliJ is present in Eclipse, just handled a bit differently. The Clojure plugins are a wash; IntelliJ has the edge on the Git plugin. I think Subversion inside Eclipse is actually better.

I've even cranked up NetBeans but didn't find anything there compelling enough to switch.

It seems like all my major tools (Firefox, Firebug, Eclipse, IntelliJ) are in the habit of growing too complex, and doing too much stuff in the background that I don't care about. All those intentions in IntelliJ that you have to turn off (for performance reasons), and all those extra plugins for Eclipse that you need to not download in the first place ... they're all getting in my way.

I think a lot of this falls into the general category of accidental complexity ... to address the limitations of the Java programming language, all this extra stuff is coming into play: tools and wizards and plugins and indexes and whatnot. I find it pretty pleasant to work with Clojure instead, where the accidental complexity of Java is managed and isolated and the IDE doesn't feel the need to be overly ambitious. That's the Clojure concept right there ... grow the language to your needs, rather than building up tools. I think that's the Tapestry ethic as well.

Public Tapestry Training in London: Aug 10 - 12

2009-06-30T10:33:00.000-07:00

Formos is partnering with SkillsMatter to provide open enrollment Tapestry training. The training will take place at the SkillsMatter offices in downtown London.

This is the same course materials that I use for my on-site Tapestry training ... and I'll be debuting a new lab on Ajax techniques.

Speaking on Clojure and Cappuccino as NFJS Seattle

2009-06-22T15:11:00.000-07:00

I'll be talking about Clojure and Cappuccino at NoFluffJustStuff Pacific Northwest Software Symposium, which runs from Sept. 18th to the 20th in Redmond, WA.

I seem to have the 'C's covered, but how about the 'T's? If you are attending NFJS or you absolutely, positively, 100% would attend if I was speaking on Tapestry there ... provide some feedback to Jay!

Apache Tapestry 5: State of the Union

2009-06-19T09:32:00.000-07:00

I've just uploaded my slide deck from JavaOne this year:

Tapestry: State of the Union

View more documents from Howard Lewis Ship.

As usual, embedded animations and screencasts don't work, but there's a lot of value here regardless.

This presentation was a slightly extended version of the Webinar.

Clojure talk from Open Source Bridge

2009-06-18T12:04:00.001-07:00

Here's the slides for my Clojure talk at Open Source Bridge:

Clojure: Functional Concurrency for the JVM (presented at Open Source Bridge)

View more PDF documents from Howard M. Lewis Ship.

The upload to SlideShare was confused by parts of the slide deck, but you can follow the gist of it.

Presenting Clojure in 45 minutes is a challenge, and I'll need to be even briefer at OSCON next month.

Why choose Tapestry?

2009-06-16T15:45:00.000-07:00

I recently had an e-mail exchange with a Tapestry user; after congratulating me on creating Tapestry, he went on with the following observation on his organization:

The company I work at unfortunately chose JSF for their big app. The reason was that Tapestry was "brittle" in the sense that, if one developer breaks something, on a page or a service, very often the whole site won't come up because the initial registry startup will fail. Or for example, if page A has a pagelink to page B, and page B is broken, then page A won't render. While I agree that we shouldn't ship unless the whole app is working, this is a thousands of pages big app with hundreds of mediocre (as in likely to break things) developers. They'd rather have 80% of the thing working than nothing at all. I never thought of this for my own projects, and haven't had the time to examine the truth of their claims. What's your take?

I provided the following response:

Early failures are absolutely, 100%, the only path towards code quality. You may have heard the phrase "no broken windows" (see "The Tipping Point" by Malcom Gladwell for more details) but the short form is that when errors go uncorrected (whether they are broken windows in an abandoned building, or broken code in an application) they tend to multiply quite rapidly.

The things that will "break" a link from page A to page B are substantial problems such as invalid templates, references to unknown properties or components, or compile errors in the page B class ... things that no other developer should ever see when page B's developer is working and checking in code. That is, problems that should never be checked into trunk, but instead kept in a local workspace or a private branch.

An organization that thinks that fail early is a problem is an organization that isn't prepared to develop a large application in any technology. The image I'm getting is one where there is no build server, no continuous integration, at best CVS for source code management (or possibly one of those "shared directory" monstrosities) .... i.e., a chaotic environment where errors are allowed to be checked in to the trunk and can go unnoticed for some time.

The solution to coding errors in pages or components is not to wait until your testers (or end users) find the bugs, but to identify and fix the bugs early. That's called "engineering discipline" and the reality is that even self-professed "mediocre" developers can do it. Tapestry helps because it fails early and has great exception reporting to guide you right the problem so that you can fix it.

Another factor here is enforced helplessness. If only Fred understands page B and he's out when it's broken, then all development stops waiting for Fred to get back. I hit this problem myself, years ago working on a large Struts application (those words give me the heebie jeebies now!). We had lots of code, a fragile and slow build process, and many little code "fiefdoms". I spent too much wasted time twiddling my thumbs.

Nobody should "own the code"; if page B is is broken, Julie (who normally develops page A) should be free to fix it. Julie will need to understand the page B code well enough to fix it, but also you need an overall environment with shared source, no repository locks (that is, nothing that says "Only Fred can change this file"), and no management PHB's getting in the way. Pair programming is the best way for Fred and Julie to share knowledge so that they can understand each other's code. Even if pairing occurs only part time, it's very effective at knowledge transfer as well as ordinary coding.

The idea that "mediocre" developers should use JSF as it is more tolerant of errors is absurd! Tapestry 5 is designed to improve productivity for all developers, by streamlining, simplifying, being smart and being concise ... not to mention live class reloading and best-of-breed exception reporting, which makes it fast to identify and fix those errors.

If your doctor tells you to eat less red meat, that doesn't mean you should switch to a diet of fried chicken three meals a day! Likewise, if you have concerns with code quality from your developers, you should not switch to a less agile, more code-intensive, less supportive development model and hope to catch all the bugs in QA. Sweeping problems under the rug is never a winning strategy.

Coming down off my soap box, I should also add that Tapestry 5.1 works a little bit differently than 5.0 in this respect, so it does (in fact) defer more of the page loading and validation until a link is actually clicked. This is more for performance reasons than to shield developers from application problems. Even in 5.0, the loading and validation was the "reach" from page A to pages explicitly referenced (usually via PageLink during the rendering of page A), so it's a highly unlikely case that a single error in a 1000 page application will keep the application from starting up, unless the start page of the application links to all 999 other pages.

Re-reading the above post I can't emphasize enough: you can't ignore quality problems. Quality problems lead to development failures, schedule slips, missing functionality, low morale and high turnover. Saying "we don't have time to fix the quality problem first" is to ignore the second law of Thermodynamics. You are expecting a miracle, literally writing it into your project plan.

Formos addresses this issue two ways: First, we use Scrum and deliver on (typically) 4 week cycles. Thus we set real deadlines and have a constant check on quality (we're providing working code constantly). We don't even try to predict what we'll be doing six months or two years from now, we just deliver a steady, manageable stream of software.

Secondly, Formos uses Tapestry precisely for all the reasons that the anonymous developer's organization rejected it, and for many, many more reasons besides.

Back from Tapestry 5 Training / Madison, WI

2009-06-12T12:56:00.001-07:00

Just got back last night from training in Madison WI. This was primarily an intro to Tapestry 5 for Tapestry 4 developers (and a couple of new-to-web-programming developers) and a lot of hands on pair programming to get them going with their conversion work. It was a lot of fun, and as usual, I learned a bit.

I have a couple of weeks respite now before OSCON which will be followed with training in York, UK, then Estonia, then London.

I'm continuing to hold off on code changes to Tapestry 5, though I hit enough minor annoyances that I'm considering a 5.1 bug fix release before starting 5.2 in earnest.

I have a backlog of work related to Tapestry360 and then my main emphasis is going to be Documentation, Screencasts, Articles, Documentation and Documentation, plus some additional Documentation. It may be a while before I start with new features for 5.2.

Upcoming Clojure Talks

2009-06-07T08:24:00.001-07:00

I'll be speaking about Clojure at Open Source Bridge on June 17th here in Portland.

In addition, I'll be speaking on Clojure at OSCON, on July 23rd. OSCON has left its traditional home, Portland, in favor of San Jose, alas. If you are thinking of attending, use my friends, co-workers and family discount code, os09fos, and save 20%.

Tapestry Road Show: Madison, Wisconsin

2009-06-07T07:49:00.001-07:00

This week, I'm in Madison, Wisconsin doing some on-site Tapestry training and mentoring. I've been here before and remember it as a pretty cool little college town. In any case, if you want to get together one night this week and chat Tapestry, Java, Open Source, Clojure or anything like that, drop a line!

Vendor Client Relationship Video

2009-06-01T09:39:00.000-07:00

This has been making the rounds on YouTube:

As a vendor, I can identify with this ... but I think a lot of clients out there (those not using Formos, for example) are ordering and charged for the Filet and receive the Tacos. Seriously ... multi-year development efforts are a different beast than purchasing a DVD, which is why short iterations and agile development keep everyone's expectations (of both cost and functionality) inline and satisified.

Back from Training and Portland Code Camp

2009-05-31T12:22:00.001-07:00

It's been a rough week ... I still had my sore throat (noticeable during the webinar) when I arrived to do four days of accelerated Tapestry training in Michigan. Returning after midnight on Friday, I had morning and afternoon slots at Portland Code Camp on Saturday to talk about Clojure and Tapestry. I think it was a good little conference, and 75 minute time slots are just barely enough time to say something meaningful.

I attended a nice introduction to jQuery (once again confirming that I backed the wrong horse when selecting Prototype over jQuery for Tapestry), and another session on coding for iPhone.

The only other session I attended was iPhone Development from an ex Softie by Rory Blyth. It was entertaining in an unusual way, since Rory is very glib in a stream of consciousness kind of way, but he spent all but five minutes of his time ranting against Objective-C and iPhone toolkits. Literally he had five minutes for the core of his talk!

I was one of a few people in the audience who knew Objective-C (though it's more than ten years since I coded in it) and found many of his objections quite unreasonable. Basically, he wants Objective-C to look like every other language derived from C, which is missing the point of what Objective-C actually is: a melding of concepts from C and Smalltalk designed to operate on the very constrained hardware available, even for desktops, in the late 80's. It obligates developers to do something unreasonable by today's standards (a cumbersome retain count mechanism, rather than garbage collection), and the (optional) type syntax (such as (NSString *)) reveals its C heritage (as Smalltalk doesn't deal with declared types).

I even made this point to him; that Objective-C may be a natural fit for the constrained devices such as mobile platforms. His response to any challenge from any audience member was that we were afflicted with "Stockholm syndrome".

Strangely, a few minutes after I pointed out the "constrained device" theory (which he dismissed, disjointedly citing Windows smart phones as a "success") he then talked about ... the constraints of the iPhone in terms of memory, battery and CPU utilization.

Basically, Rory is unable to wrap his head around anything unfamiliar or to understand how a difference in philosophy can inform how a language syntax evolves, as well as the terminology (i.e., Objective-C's "receivers", "messages" and "selectors") used to describe that language.

There's a quote from the book Freakonomics, roughly (from memory):

Morality is how we think we should live our lives. Economics reveals how we actually do.

Rory has a kind of "language morality" that states the objects should be listed first, with periods separating member access, such as method invocation, and that languages that deviate from this are failed and broken. Unfortunately for that argument, the explosive success of the iPhone and the iPhone app market indicates that Objective-C is a tremendous development platform for the kind of intuitive, focused, responsive applications that dominate the market.

It was a shame, because his style was entertaining, if very "slacker" styled, and if he organized his thoughts a bit and kept track of the clock, his valid criticisms of the iPhone development environment would hold a bit more weight and reach a wider, more receptive audience.

Tapestry 5 Support in IDEA Maia

2009-05-26T14:11:00.000-07:00

The next generation of IDEA, Maia will include Tapestry 5 support; this is according to Hugo Palma, who wrote the initial version of the IDEA Tapestry plugin that has been expanded for inclusion in Maia. Cool!

Tapestry & Clojure @ Portland Code Camp

2009-05-25T07:48:00.000-07:00

Portland Code Camp is a local (in Portland) free conference on development tools and languages ... and it's coming up fast (this Saturday). I'll most likely be talking about Tapestry and Clojure ... I actually won't know until they make their last minute session selection tomorrow (it's based on how many attendees express interest). If you're in the Portland area and want to learn a bit more about Tapestry or Clojure, sign up and vote!

Tapestry Road Show: Ann Arbor, Michigan

2009-05-24T19:05:00.000-07:00

This week, I'll be in Ann Arbor, MI to do a bit (aka, four grueling days) of on-site Tapestry training. If you're in Ann Arbor and would like to get together for a drink and a chat about Tapestry, Open Source, or anything along those lines, please drop a line!

Last call: Tapestry Webinar (Thu May 21)

2009-05-20T08:49:00.001-07:00

Formos is sponsoring a live Tapestry Webinar on May 21st. I'll be presenting a 35 - 40 minute introduction to Tapestry 5, followed by live Q&A. I'll be focusing on the big-picture issues about what makes Tapestry useful, unique and fun. If you've been curious about what Tapestry is, or want a chance to ask a question about Tapestry directly, this is a great chance to do it!

Click a link below to register for the Webinar:

Tapestry 5 Performance Study

2009-05-14T11:27:00.001-07:00

Ben Gidley has done a detailed analysis of Tapestry 5 performance He used some quite reasonable assumptions, including running tests for over an hour to gauge memory use over time. His conclusion is that Tapestry performance is excellent, which is a relief if not a surprise.

What I'm really pleased with is that he's demonstrated that Tapestry will perform well under load; more importantly, if a Tapestry application gets "Slashdotted", it may slow down, but it recovers once the wave of requests dies down.

Ben compared a Tapestry application to a Struts application with similar behavior and found that T5 out-performed Struts pretty handily, despite the fact that T5 doubles the number of requests (due to its automatic redirect-after-post behavior).

Thanks for the great work Ben!

Clojure @ JavaWorld

2009-05-12T13:26:00.000-07:00

Another nice introduction to Clojure is now online: Clojure: Challenge your Java Assumptions at JavaWorld. An introductory Tapestry article is due at JavaWorld shortly as well.

Apache Tapestry Community Map

2009-05-09T08:47:00.000-07:00

Inspired by a map for Clojure, I created a map for Tapestry users:

View Apache Tapestry Users Worldwide in a larger map

Want to add yourself? The map is wide open, something I'll probably regret later.

Update: People have been adding themselves over the weekend, and trends are starting to appear. It seems like adoption is heavier in Europe than in the US (at least, among the narrow set of people who use Tapestry, know about the map, and have actually added themselves). Still, if that sample is true, it begs the question: why? Is it more to do with the European developers, or due to features of Tapestry (such as localization support).

I'm tending towards the former: talking with many Europeans shows that managers have a more "hands off" attitude towards project leaders, who are given more free reign to use tools of choice as long as they deliver. But that's also a horrible generalization from limited data. In fact, a contrary opinion was offered by some developers in Oslo last september ... in Norway, a small number of developers make strong vocal choices (lately, Spring and SpringMVC) and the whole community follows along. But even Europe is a big place which room for many different developer cultures.

Tapestry 5 Webinar: May 21st

2009-05-08T11:10:00.000-07:00

Click a link below to register for the Webinar:

Apache Tapestry 5.1 Final Release - 5.1.0.5

2009-05-06T15:53:00.000-07:00

The stable release for Tapestry 5.1, release 5.1.0.5, is now available for download.

Tapestry 5.1 includes a large number of improvements to the speed and scalability of Tapestry without sacrificing backwards compatibility. However, you should still consult the upgrade notes that identify a couple of minor "gotchas".

Major improvements from Tapestry 5.0 to Tapestry 5.1:

Performance optimizations that speed rendering of very complex pages, and reduce memory usage
Static JavaScript libraries will now be dynamically combined into a single request
Automatic GZIP compression of static and dynamic content, for clients that support it
Significant improvements to Tapestry's property expression language
Partial page updates (via Ajax) may now update multiple client-side Zones
Many new features and improvements to Tapestry template files
Improved client- and server-side validation of numeric input values
User's preferred locale is now encoded into the URL, rather than stored in a cookie
A new ProgressiveDisplay component for incremental page loading
Vastly improved Quickstart Archetype
New Tapestry IoC features to make it easier to override services and contributions to service configurations
Many new localizations of validation messages
Component reference documentation now identifies events triggered by components
Really slick new client-side logging facility (based on Blackbird)
New API for URL rewriting (to provide improved control over Tapestry URLs)

... and many, many more bug fixes and improvements; full details are available in the release notes.

Download Tapestry 5.1.0.5

Clojure: Turns out, parsing is somewhat iterative, not functional

2009-05-05T16:43:00.000-07:00

... at least for me. I'm working on some more real-world code in Clojure, and I'm struggling with turning a stream of tokens (representing XML content parsed from a stream) into a DOM-like hierarchy. The tokens represent open tags, attributes, close tags, and text content.

I've written dozens of recursive decent compilers in the past, in Pascal, C, PL/1 and Java. The problem is, that style of compiler works by consuming a stream (of characters or tokens) ... a destructive operation.

Logically, I want code such as:

List parseTemplate(tokens)
{
  result = []

  while (tokens not empty)
  {
    token = pop(tokens)
    if (token.type == text)
    {
      add text token to result;
      continue;
    }

    if (token.type == element)
    {
      e = parse_element (token, tokens)
      add e to result;
      continue;
  }
}

Now, in Clojure I can write a (parse-element) function that takes a bunch of tokens, and it can work its way through the tokens using Clojure's (loop) and (recur) forms ... but to the calling function, tokens will not have changed.

One approach might be to pass along some pretty complex state; literally a state machine (inside a map) so that each function can keep push and popping values into and out of the state before invoking each. This would involve recursing on each token. Anyway, I have a rough idea of what that would look like and it seems complicated.

Instead, I've been pounding my head, looking at Clojure's support for monads. Monad are a way to combine certain types of functions together in a way that appears as if there were mutable state. You provide some base types of monadic functions and, within the context of a monad definition, it streamlines the way those basic monadic functions can be strung together to form more complex monadic functions.

Still trying to wrap my head around the concept however. It gets really confusing because the real power of monads appears when the monadic values, the things returned from a monadic function, are themselves functions. Yep, that's where I get lost too.

Update: passed the initial Monad hurdle; I'm finally beginning to understand them, perhaps even enough to explain them to other people.

Excellent Clojure Introduction

2009-05-03T09:40:00.000-07:00

R. Mark Volkmann has put together a terrific Clojure Introduction. It starts with basic syntax and quickly works through all the core functions, broken out by categories such as "Collections", "Iteration" or "Concurrency".

Very brisk, but full of functions (from the core) that I didn't know of.

For example, here's a function I never noticed before, (get-in), as defined in the Clojure API

(get-in m ks)
returns the value in a nested associative structure, where ks is a sequence of keys

That's accurate, but really encourages you to read the source or experiment in the REPL to see exactly how it works. Worse, its mixed in with hundreds of other functions and really easy to miss (I never noticed it before!).

Here's what the Clojure Introduction says:

To demonstrate this we'll create a map that describes a person. It has a key whose value describes their address using a map. It also has a key whose value describes their employer which has its own address map.
(def person {
  :name "Mark Volkmann"
  :address {
    :street "644 Glen Summit"
    :city "St. Charles"
    :state "Missouri"
    :zip 63304}
  :employer {
    :name "Object Computing, Inc."
    :address {
      :street "12140 Woodcrest Executive Drive, Suite 250"
      :city "Creve Coeur"
      :state "Missouri"
      :zip 63141}}})
The get-in function takes a map and a key sequence. It returns the value of the nested map key at the end of the sequence. The -> macro and the reduce function can also be used for this purpose. All of these are demonstrated below to retrieve the employer city which is "Creve Coeur".
(get-in person [:employer :address :city])
(-> person :employer :address :city) ; explained below
(reduce get person [:employer :address :city]) ; explained below

This is a document to keep linked and to print out and study. Thanks Mr. Volkmann!

Ruby and CouchDB

2009-04-27T10:47:00.002-07:00

I've started looking into CouchDB of late; I think it may be a great solution to some of our common issues around collecting "survey" style data from users. Our current approach uses a big nasty hierarchy of Hibernate entities.

I wanted to have some real, and real-world, data. Since I'm a big board gamer, I thought it would be nice to have a local copy of the BoardGameGeek database. I wrote a program that scrapes data from the BGG site and loads it into CouchDB.

#!/usr/bin/ruby

require 'rubygems'
require 'hpricot'
require 'open-uri'
require 'thread'
require 'couchrest'


# Extension to Hpricot

class Hpricot::Elem
  # content at
  # if block given, return value comes from yielding
  # actual content to the block
  def cat(expr)
    result = self.at(expr)

    return nil unless result

    content = result.inner_html.fixup

    return (yield content) if block_given?

    return content
  end

  # content-at converted to int (via to_i)

  def cati(expr)
    self.cat(expr) { |content| content.to_i }
  end

  def search_to_text(expr)
    self.search(expr).map { |e| e.inner_text.fixup}
  end
end

$BGG = "http://boardgamegeek.com"


$catalog_pages = 0
$game_pages = 0
$games_added = 0

# Access to couch db.  From what I can tell, the Databsae is multithreaded.

$DB = CouchRest.database!("http://127.0.0.1:5984/board-game-geek")

# The thread pool.

$POOL_SIZE = 10

$QUEUE = Queue.new

workers = (1..$POOL_SIZE).map do |i|
  Thread.new("worker #{i}") do
    begin
      proc = $QUEUE.deq
      proc.call()
    end until $QUEUE.empty?
  end
end

def enqueue &action
  $QUEUE << action
end


def parse_browser_page(url)
end

class String
  def fixup()
    self.gsub("&#039;", "'").gsub("&amp;", "&")
  end
end

def parse_and_load_game(game_id)

  $game_pages += 1

  page = Hpricot.XML(open("#$BGG/xmlapi/boardgame/#{game_id}?comments=1&stats=1"))

  bg = page.at("//boardgame")

  ratings = bg.at("//ratings")

  comments = (bg/"comment").map do |e|
    {
            "user" => e[:username].fixup,
            "comment" => e.inner_text.fixup
    }
  end

  doc = {
          "_id" => game_id,
          "title" => bg.cat("name[@primary='true']"),
          "description" => bg.cat("description"),
          "designers" => bg.search_to_text("boardgamedesigner"),
          "artists" => bg.search_to_text("boardgameartist"),
          "publishers" => bg.search_to_text("boardgamepublisher"),
          "published" => bg.cati("yearpublished"),
          "categories" => bg.search_to_text("boardgamecategory"),
          "mechanics" => bg.search_to_text("boardgamemechanic"),
          "images" =>{
                  "url", bg.cat("image"),
                  "thumbnailUrl", bg.cat("thumbnail"),
                  },
          "players" => {
                  "min" => bg.cati("minplayers"),
                  "max" => bg.cati("maxplayers"),
                  "age" => bg.cati("age")
          },
          "stats" => {
                  "rank" => ratings.cati("rank"),
                  "averageRating" => ratings.cat("average") { |content| content.to_f },
                  "ownedCount" => ratings.cati("owned")
          },
          "comments" => comments
  }

  enqueue do
    $games_added += 1

    $DB.save_doc(doc)
  end

end


def process_game(game_id)

  begin
    doc = $DB.get(game_id)

    # Found, do nothing

  rescue RestClient::ResourceNotFound

    # Not in the database yet, so fire off a request to parse its page.

    enqueue { parse_and_load_game game_id }

  end
end

def parse_catalog_page(url)

  puts("[%24s] %4d catalog pages, %4d/%4d games parsed/added (%4d actions queued)" % [Time.now.ctime, $catalog_pages,
       $game_pages, $games_added, $QUEUE.length])

  doc = Hpricot(open(url))

  $catalog_pages += 1


  doc.search("//table[@id='collectionitems']/tr/td[3]//a") do |elem|
    href = elem[:href]
    game_id = href.split('/').last()
    enqueue { process_game game_id }
  end

  next_page_link = doc.at("//a[@title='next page']")

  return unless next_page_link

  next_page = next_page_link[:href]

  # Add this last, to give the other actions a chance to operate.
  # A better mechanism would be a priority-based queue, where the catalog
  # page parse action is lower priority than the other actions.

  enqueue { parse_catalog_page($BGG + next_page) }

end


# Kick it off

start_page = $ARGV[0] || ($BGG + "/browse/boardgame")

enqueue { parse_catalog_page start_page }

# Wait for all the workers to complete

workers.each { |th| th.join }

puts "BoardGameGeek loader complete."

puts "Queue not empty!" unless $QUEUE.empty?

Key pieces of this is Hpricot to parse HTML and XML, and CouchRest to get the data into CouchDB.

Did I go overboard? I don't think so ... this runs in about two hours, and created a database of nearly 40,000 documents (about 340MB). Using a thread pool just seemed to make sense, since (outside of the XML parsing), every aspect of this is I/O bound: pulling data from BGG or pushing data to CouchDB.

The code is naive about some threading issues and simply crashes if there's an error. Oh well.

Next up: learning how to build views against this data and deciding how to use it all. I'm thinking Cappuccino.

Wither Derby?

2009-04-27T09:32:00.001-07:00

Over lunch last week with co-workers, we were discussing the Oracle buy out of Sun and how that would affect MySQL and what other options there would be. Several people had some experience with PostgreSQL (including myself), but then I brought up Apache Derby ... and no one had used it or event downloaded it, and only a few had even heard of it. I just checked their web site and the last update was in November 2008. When I last played around with it, it seemed to have all the major features of a real database and the advantage of being pure Java (I can't speak to performance).

How many people are using Derby and if not, why not?

Tapestry 5.1 is Beta!

2009-04-20T11:20:00.000-07:00

We just had a vote, and Tapestry 5.1 (that is, release 5.1.0.3) is a beta release. At this point, everything is stable except for minor changes to the brandy-new URL rewriting APIs. I've been chasing down bugs and improvements for the 5.1.0.4 release, which will probably be created and voted on later this week. My guess is that 5.1.0.4 will be the final/stable release for 5.1.

I'm already thinking for 5.2. Ditch Maven for Buildr? Strong possibility. Start using Groovy for tests? Why not? Finally get Spring Web Flow integration working ... you betcha! Portlets? Shouldn't be a problem. Release date? I think before the end of the year.