VanDev

MongoDB Fancy Ass Grouping

2009-10-16T00:00:00+00:00

So lately I’ve been playing with Mongodb for CrowdFlower. We need to calculate statistics from the information that we gather. Mongo provides support for this but has little documentation. I thought it would be appropriate to describe how I got some of the fancier group statements working. Let’s take this document format as an example:

    {
     "state": "finished",
     "data": {
       "everythings_good": "yes"
     }
   }

The key property alone of the group function can not be used because our key is in an embedded member. We have to use the keyf property of the group command. I’m using the Mongo ruby library and it currently doesn’t support the use of this property. Instead I issue the raw db_command:

    db = Mongo::Connection.new.db('docs')
    db.db_command({
      "group" => {
        "ns" => "ratings",
        "$reduce" => Mongo::Code.new(<<-JS),
          function(obj,prev){ prev.count++; }
        JS
        "$keyf" => Mongo::Code.new(<<-JS),
          function(obj){
            var g = {}
            g[obj.data.everythings_good] = true
            return g
          }
        JS
        "cond" => {},
        "initial" => {:count => 0}
      }
    })

This command returns the following value:

    {
      "retval"=>[
        {"yes"=>1.0, "count"=>17355.0}, 
        {"no"=>1.0, "count"=>6395.0}
      ], 
      "count"=>23750.0, 
      "keys"=>2, 
      "ok"=>1.0
    }

Note: the keyf property was not working properly until version 1.1.0 of mongodb

Fancy Zoom Mootools time

2008-09-16T00:00:00+00:00

So I stumbled upon this project on github.com… If you browse the archives of this blog you might see I have an affinity for Mootools over Prototype or JQuery (Although JQuery is hot, and the new FX stuff is tempting me to stop being a rebel). Regardless, Fancy Zoom for Mootools is born. It’s a pretty straight forward port. The main performance gain is that the effects are only initialized once no matter how many FancyZoom instances there are on a page. Click on the thumbnail below to see it in action.

In unrelated news the image you just clicked on is of me speaking at the local San Francisco JavaScript meetup a couple weeks ago. I attempted to give a tutorial of Mootools called Instimage. If you do a search for “sf js” my slides are the first result for some reason… Pagerank is the end all. Peace.

JSAwesome

2008-04-22T00:00:00+00:00

Here is a sample call to JSAwesome:

    new JSAwesome('an_id',[
      ['text_field', 'default value'],
      ['#text_area', ''],
      ['_hidden_input','i_am_invisible'],
      ['*radios', ['so|nice', 'many', 'radio|boxes', 'yeah']],
      [['^checkbox', true], '^not'],
      ['^choices', ['Choice 1|one', 'Choice 2|two', ['Choice 3|three', true]]],
      ['single_select', ['boo', '~yah']],
      ['nested_select', 
	 {'rad': ['cool', 'neat'], 
	  'awesome':
	    {'crazy': ['Indeed|~shit', 'man'],
	     'way': ['oh', 'yeah']}
	 }]]).to_html()

This generates this form (_choose awesome, than crazy, than Indeed from the set of select boxes at the bottom of the form to have your mind blown_)

With this markup:

    <div id="an_id">
  <div class="error text_field">
    <label for="an_id_text_field">Text field</label> <input id="an_id_text_field" class="text_field" type="text" name="an_id[text_field]" />
  </div>
  <div class="error text_area">
    <label for="an_id_text_area">Text area</label> <textarea id="an_id_text_area" class="text_area" name="an_id[text_area]"></textarea>
  </div>
  <div class="error hidden_input">
    <input id="an_id_hidden_input" class="hidden_input" type="hidden" name="an_id[hidden_input]" value="i_am_invisible" />
  </div>
  <div class="error radios">
    <fieldset>
      <legend>Radios</legend> 
      <label for="an_id_radios">
          <input id="" class="radios" type="radio" name="an_id[radios]" value="nice"> So</label> 
       <label for="an_id_radios">
          <input id="" class="radios" type="radio" name="an_id[radios]" value="many"> Many</label> 
       <label for="an_id_radios">
          <input id="" class="radios" type="radio" name="an_id[radios]" value="boxes"> Radio</label> 
       <label for="an_id_radios">
          <input id="" class="radios" type="radio" name="an_id[radios]" value="yeah"> Yeah</label>
    </fieldset>
  </div>
   <div class="error row_5">
      <div style="float: left; margin-right: 5px;">
        <label for="an_id_checkbox">
           <span>
              <input id="an_id_real_cool" class="real_cool" type="checkbox" name="an_id[checkbox]" value="true"> 
              <input id="an_id_real_cool" class="real_cool" type="hidden" name="an_id[checkbox]" value="false">
           </span> Checkbox</label>
      </div>
      <div style="float: left; margin-right: 5px;">
        <label for="an_id_not">
            <span>
                <input id="an_id_not" class="not" type="checkbox" name="an_id[not]" value="true"> 
                <input id="an_id_not" class="not" type="hidden" name="an_id[not]" value="false">
             </span> Not</label>
      </div><br style="clear: left;">
    </div>
    <div class="error choices">
      <fieldset>
        <legend>Choices</legend> 
            <label for="an_id_one">
               <span>
                  <input id="an_id_one" class="one" type="checkbox" name="an_id[one]" value="true"> 
                  <input id="an_id_one" class="one" type="hidden" name="an_id[one]" value="false">
               </span> Choice 1</label> 
             <label for="an_id_two">
               <span>
                  <input id="an_id_two" class="two" type="checkbox" name="an_id[two]" value="true"> 
                  <input id="an_id_two" class="two" type="hidden" name="an_id[two]" value="false">
               </span> Choice 2</label> 
            <label for="an_id_three">
                <span>
                    <input id="an_id_three" class="three" type="checkbox" name="an_id[three]" value="true">
                    <input id="an_id_three" class="three" type="hidden" name="an_id[three]" value="false">
                </span> Choice 3</label>
      </fieldset>
    </div>
   <div class="error single_select">
      <label for="an_id_single_select">Single select</label>
          <select id="an_id_single_select" class="single_select" name="an_id[single_select]">
              <option value="">Choose Category</option>
              <option value="boo">boo</option>
              <option value="yah">yah</option>
           </select>
    </div>
  <div class="error nested_select">
    <label for="an_id_nested_select">Sub cats</label> <select id="an_id_nested_select" class="nested_select" name="an_id[nested_select]">
      <option value="">
        Choose Category
      </option>
      <option value="rad">
        rad
      </option>
      <option value="awesome">
        awesome
      </option>
    </select> <select id="an_id_nested_select_1" class="nested_select_1" name="an_id[nested_select_1]">
      <option value="">
        Choose Subcategory
      </option>
    </select>
  </div>
</div>

Wow, lot’s of HTML from a little JSON. Let’s do an overview of the parameters.

The first parameter to JSAwesome is an id of an element already in the DOM. It also acts as a namespace for the names and id’s of form elements to allow for multiple JSAwesome forms on a page. Looking at the html you can see you have plenty of classes added as well to style your generated form to your hearts content.
The second parameter is an array of tuples specifying the form elements in order. The tuples can create the following elements using these rules: * textfield - the first element is the name with no special character and the second element is a string with the default value * textarea - the first element is the name beginning with a # and the second element is a string with the default value * hidden - the first element is the name beginning with a _ and the second element is the value * radio - the first element is the legend of a fieldset / name of the radio buttons beginning with a * and the second element is an array of names. Optionally you can specify label / name pairs separated by the | character. * checkbox - the first element is the name and label of a checkbox beginning with a ^ character. If a second element is present, the checkbox will be checked. The example above also demonstrates the ability to group fields together on the same line. If you use an array of tuples, or in this case checkboxes can be specfied with a single string if the you don’t want it checked, those elements will be wrapped in div’s with the style set to ‘float:left’. Optionally label / name pairs work the same as radio’s. * checkbox group - the first element is the legend of a fieldset and the second is an array of checkboxes as described above * single select - the first element is the name of the select and the second is an array of options. Optionally the options can specify name / value pairs separated by the | character. You can also use the ’~’ character to specify a custom input. When this option is selected a textfield will appear to allow the user to enter a custom value. * nested select - the first element is the name of the select. The second element is a hash with the keys being the options and the values being hashes or arrays of the child selects. Optionally name / value pairs and custom inputs work the same as single selects.

But there’s more. JSAwesome takes a third parameter which gives you custom labels and validations. This use case might look like this:

    var test = new JSAwesome('validated',[
      ['text_field', 'default value'],
      ['#text_area', ''],
      ['single_select', ['boo', '~yah']]
      ], 
      {'text_field': "JSAwesome",
       'text_area': {
          'required':true,
          'validation': ["\\d+", "Field must contain atleast one digit"],
          'label': "Input some text with atleast one digit"
       },
       'single_select': {
          'required':true
       },
       '~': 'Custom instructions',
       '{}': ['Select something']
   })
   test.to_html()
   test.addValidation()

Which generates this form:

Bam, is your mind completely blown yet? Here’s an overview of the 3rd parameter:

The keys to the hash are either names of elements in the form (_without the special characters_) or one of two special global names ~ or {}.
The values of the hash are either a string or another hash. If it’s a string, it represents the label for the element specifed. If it’s a hash you can specify all or one of the following:
- required - a boolean specifying whether the field is required. if it is used for a radio or checkbox group it will require atleast one radio button to be clicked. It can also be used alongside of validation to make a text input field both required and validated
- validation - a tuple with the first element being a javascript RegEx string. The second is a message to be displayed on failure
- label - the label, just as if you had given only a string, but can be used alongside required and validation
If the key is ~ this specifies the value to be used in custom fields as opposed to the default ‘Custom…’ (_choose ‘yah’ from the select box above_)
If the key is {} the value is a tuple with the first element being the default value for the root select element, and the optional second parameter being the default for any sub-selects.

So there you have it. JSAwesome in a nutshell. We use this library over at Dolores Labs for easily creating dynamic forms for all of our experiments. It beats the crap out of writing HTML forms manually and makes our tasks load a lot faster. We also get validation for free. It currently requires mootools 1.2b, which is packed with the project. Fork away!

6.5 months... not bad

2007-10-30T00:00:00+00:00

Hmmm, well I don’t know what to say. This will have to do until I come up with something…

Inside the Croppr frontend

2007-04-16T00:00:00+00:00

I learned allot creating croppr. It’s pretty intersting what you can do with the DOM and some CSS. Let’s take a peek under the hood.

The first thing croppr does upon instantiation is create a bunch of DOM elements. Here’s what that looks like.

Toggle Styles

    <div style="background: white;
            position: absolute;
            top: 0px; left: 0px;
            width: 100%; height: 545px;
            opacity: 0.8;"/> 
<div id="croppr" style="overflow: hidden;
                        position: absolute;
                        top: 0px; left: 0px;
                        width: 100%; height: 545px;">                  
  <div style="border: 1px solid #000000;
              overflow: hidden;
              position: absolute;
              top: 50%; left: 50%;
              width: 118px; height: 118px;
              margin-top: -60px; margin-left: -60px;">
    <img src="/croppr/images/1" style="cursor: move;
                                       position: absolute;
                                       top: 50%; left: 50%;
                                       margin-top: -272px; margin-left: -357px;
                                       width: 714px; height: 544px;
                                       opacity: 1;"/>
  </div>
  <img src="/croppr/images/1" style="cursor: move;
                                     position: absolute;
                                     top: 50%; left: 50%;
                                     margin-top: -272px; margin-left: -357px;
                                     width: 714px; height: 544px;
                                     opacity: 0.3;"/>
  <a class="exit">
    <span>EXIT</span>
  </a>
  <a class="crop">
    <span>CROP</span>
  </a>
  <div class="track">
    <div class="handle" style="cursor: move;
                               position: relative;
                               left: 154.312px;"/>
  </div>
</div>

Toggle

The next element is our container for all the fun goodies we’re about to add (_container_). It serves three purposes.

Makes cleanup easy. By deleting this element, all the others will disappear.
Namespaces the rest of our elements. You can target #croppr .exit in your CSS without effecting other elements with the class of exit.
Get’s rid of vertical and horizontal scroll bars. By setting overflow:hidden when the image we are cropping goes outside of the window, the browser doesn’t add scrollbars and screw up our math.

The transparent box is not placed in this div because we want a nice fade back into the original environment after we have deleted all of the cropping tools. More on this in a little bit.

Next we have the mask. The width and height of the mask is set to the width and height of the desired crop - 2. We subtract two because it has a 1 pixel border and the box model includes the border in the width and height. A copy of the main image (_ghost_) is put inside of the mask. overflow:hidden is set on the mask as well, making the parts outside of it invisible. Both the mask and the ghost are positioned absolutely and have top and left attributes set to 50%. We then give them negative top and left margins. The mask’s margin’s will always be half of its width and height, while the ghost’s margins are based on it’s current position. This puts the work of keeping the items centered if the window resizes in the hands of the browser. All of this is done rather elegantly with mootools:

    this.mask = new Element("div").setStyles({
      border: "1px solid #000",
      overflow: "hidden",
      position: "absolute",
      width: this.options.crop[0]-2+"px",
      height: this.options.crop[1]-2+"px",
      top: "50%",
      left: "50%", 
      marginTop: -this.options.crop[1]/2+"px", 
      marginLeft: -this.options.crop[0]/2+"px" 
}).injectInside(this.container);
this.ghost = this.image.clone(false).injectInside(this.mask);

The image itself is next up. She is infront of the mask so we can drag her around without having the mask conflict. We set the opacity to .3 making the ghosting effect. The positioning of this image is identical to the one inside the mask. Infact, as you drag the image, it’s css is copied straight into the css of the image inside of the mask. The only difference is the opacity.

    draw: function() {
  this.ghost.style.cssText = this.image.style.cssText;
  this.ghost.setOpacity(1);
}

this.drag = new Drag.Base(this.image, {
  limit: {
    x: [-this.size[0] + this.options.crop[0] / 2, -this.options.crop[0] / 2], 
    y: [-this.size[1] + this.options.crop[1] / 2, -this.options.crop[1] / 2]
  },
  modifiers: {
    x: "margin-left", 
    y: "margin-top"
  },
  onDrag: this.draw.bind(this)
});

Mootool’s awesome Drag class provides a way to constrain the object being drug via the limit parameter in the constructor. This logic is repeated in the scaling function (_updateSlide_) to insure the image is not scaled outside of the mask. This function is the hairiest of them all… maybe we’ll dig into her another time.

Last but not least we have the CROP and EXIT buttons along with the slider itself. These need to be infront of everything else, which is why they are last. There is no styling done explicitly by the croppr javascript library. Instead, it gives you the freedom to style them however you would like.

When we’re done using croppr, cleanup is as easy as:

    this.container.remove();
new Fx.Style(this.trans, 'opacity', {duration: 1000, onComplete: this.trans.remove}).start(0.8, 0);

And that’s that, bending the DOM to do your dirty work turns out not to be so bad…

Croppr, ready for primetime

2007-04-11T00:00:00+00:00

It’s been a long time coming… Croppr has grown out of it’s infancy. Sparked recently by the release of Gravatar 2.0 and the Ruby on Rails podcast Camping II, I dove back into the code and cleaned it up. Croppr now adheres to the minimalistic principles of Camping. RMagick was thrown out in favor of ImageScience. The javascript was tweaked for performance and readability using the ever so light and powerfull MooTools. The magic is really in the javascript, I encourage you to check it out. All the pieces come together as a living tutorial of how to implement Croppr on your own. Go play with my vain demo! If you want to get sneaky and upload your own image tag /new to the end of the URL, but keep it clean… Here’s an overview on how to get her up and running on your own machine:

    sudo gem install mongrel
sudo gem install camping
sudo gem install json
sudo gem install sqlite3-ruby (you need to have sqlite3 installed) 
sudo gem install image_science (you need to have freeimage installed)

svn co http://svn.vandev.com/croppr/trunk croppr
cd croppr

ruby croppr.rb

It’s that easy. You should be able to follow the documentation in the javascript and the implementation example in the camping app to get her working yourself. Goodluck, and let me know of any crazy browser issues. I half-assed tested it in IE 6 and 7… Safari and Firefox are golden.

Mootools and my sidebar, doing it

2007-03-11T00:00:00+00:00

This afternoon I was perusing the web and ended up mining Yelp for something to do in this new and foreign city. I noticed a slick effect that was used to keep the map in the view port as you scrolled. I had done this before with the position:fixed css directive, but it’s rather limited especially in a centered layout. You also can’t choose your easing equation or delay the effect… Thanks Moo, let’s do it.

I added a set of links which link to targets on this page in my sidebar entitled “On this page…”. I want the links to scroll the page to the targeted element, and I also want the links to follow me there so I can click on another one if I need to. 8 lines of javascript with the aid of Mootools gets her done.

    var slideEffect = new Fx.Style('sidenav', 'margin-top', {wait:false, duration:800, transition:Fx.Transitions.circOut});
  window.addEvent('load', function() {
    var top = $('sidenav').getPosition().y - 10;
    window.addEvent('scroll', function(){
      slideEffect.start.delay(600, slideEffect, Math.max(0, document.documentElement.scrollTop - top));
    });
    new SmoothScroll({duration:700, transition:Fx.Transitions.circOut});
  });

Let’s analyze the crap out of this code. I first create a new Fx.Style specifying wait:false (_so If I start scrolling down the page and than suddenly switch direction, the effect will simply start over in that new direction_). I’m modifying the margin-top property of the sidenav element. I also specify a custom transition circOut which starts fast and than slows to a stop. Using window.addEvent('load', I wait for the document to load because the position of my div is relative to the image of my face. You could use the domready event if the position of the div you will be scrolling is not relative to an image… Next I get the position of the top of my div and subtract 10 from it to give it some padding.

Now for the magic. I listen for the scroll event and when it happens I fire the recently created slideEffect. However, because the delay is cool and if I don’t delay the effect will look really jumpy, I delay the start of the effect. Moo provides a delay method which can be called on any function. You have to call it on the function, not on the thing the function returns: e.g. myFunction.delay(100) or function(){return 'rad'}.delay(300) instead of myFunction('awesome').delay(30). The second attribute to delay is an object that will become this in the function delay is being called on. In this case we want this to be the effect itself. The third attribute is the attribute(s) that will be passed into the function. For us, this is an integer representing the new margin. We calculate this as the maximum of either 0 or the difference of the current viewport offset and the original offset of the image.

The last thing to do is make a call to SmoothScroll, making all links that point to a target scroll to the element with that targets id (this was detailed in my last post). So there it is, a detailed description of how my sidebar and mootools did it.

Mootools, a silly Fx tutorial

2007-02-28T00:00:00+00:00

I’ve been loving mootools lately so I figured I would demonstrate some of it’s awesomeness by explaining a few of the effects generated on this very page.

Awesome bouncy nav tabs (_upper right, funny creative projects_)

    $$('#header ul li a').each(function(el) {
  var bounce = new Fx.Style(el, 'top', {duration:700, transition: Fx.Transitions.elasticOut});
  var colors = new Fx.Styles(el, {wait:false});
  
  el.addEvent('mouseover', function(){
	el.setStyle('background-color', '#d5e88f');
	bounce.start(15,0);
  	colors.start({
		color: '35342e'
  	});
  });
  el.addEvent('mouseout', function(){
  	colors.start({
		'background-color': '35342e',
		color: 'd5e88f'
  	});
  });
});

Allright, so moo gives us a DOM selector much like prototypes. I use it to grab all the a tags in my nav list, and then I iterate over them. Moo has three generic Fx classes, Style, Styles, and Elements. I don’t use Fx.Elements here, but you can read about it via moos excellent documentation. If you are familiar with the scriptaculous Morph effect, these will come natuarlly. They let you tween a list of css attributes, leaving the door of possibilities for amazing custom effects wide open.

First I create two effects. One to deal with the motion, bounce, and the other to deal with font and background color, colors. Moo supplies an optional set of robust transitions based on Robert Penner’s easing equations (_the easing demo is extremely helpful_). I specify the elasticOut transition upon instantiation of the bounce effect to make it, well bouncy. Setting wait to false in the Fx.Styles call tells the effect to go ahead and run even if there is another effect just like it already running. This let’s the tab fade back to normal if I mouseout before it’s finished.

After the effects are created, all that’s left is attaching them to an event. Voila, crazy cool bouncy tabs.

A silly unobtrusive smooth scrolling effect (_click “it’s not what you think” next to the VD logo_)

    new SmoothScroll({duration:1000});

Another optional effect (thank you kickass building tool), is SmoothScroll. Upon instantiation she finds all of the targets (aka anything with an id) and adds an event to all the a tags referencing that target (aka href=”#name_of_target”), which scrolls the window to that element… just click it.

Opaque image rollover (_what the flick_)

    $$('div#flickr img').each(function(e){
  var fade = new Fx.Style(e, 'opacity', {wait:false});
  fade.set(.5);
  e.addEvent('mouseover', function(){
	fade.start(1);
  });
  e.addEvent('mouseout', function(){
  	fade.start(.5);
  });
});

This uses the same techniques as the first effect. The set method allows you to set the initial position of the effect. In this case it makes all of the images 50% opaque when the page loads.

So there you have it. Mootools is silly powerful and fun. It puts the control in your hands, and does so in a lightweight and extremely sexy manner.

Really awesome, I mean really awesome

2007-02-12T00:00:00+00:00

So back on the 15th of December friend and fellow rubyist, Kevin Clark, forwarded me a job description from Powerset. Well, I applied to said job and long story short I’m starting tomorrow! I drove a Budget rental truck up to San Francisco from San Diego on Friday. I don’t have a place yet… a slight problem which will hopefully be remedied very soon. Anyway, we had an awesome party last night to celebrate the series A funding, and I’m excited to get started and settle into San Francisco.

A mephisto tag cloud

2007-02-07T00:00:00+00:00

As I continue to customize my new mephisto installation, I wanted a tag cloud for the Folksonomy section at the bottom of the page. At first I thought I would use a mephisto plugin… after further consideration I decided that was a little overkill. JavaScript kicks ass, especially with the recent release of mootools version 1.0, so I decided to let it do the work. I created a small class called Overcast which makes tag clouds easy peasy japaneasy. So, just how easy is it to get a tag cloud in mephisto? Here’s a snipet from my layout.liquid

    mootools
overcast
</head>

<div id="overcast">
  
</div>
<script type="text/javascript">
/* <![CDATA[ */
  over = new Overcast({min: 1, fuzz: 2, overlay: "/images/gloss.png"});
/* ]]> */
</script>

A couple sexy things to note are the cool transparent overlays ontop of each tag, and the ability to fuzz tags when your weights aren’t very diverse. Also, if you’ve got Firebug you can type over.update(“amazing”,4) in the console to add a new tag to the cloud with that weight, or over.update(“javascript”, 3) to change the weight of an existing tag. All of the functionality is documented in the source.

Grab the source from the include in this page. I’ll put it under version control soon.

Mephisto is finally up and running

2007-01-12T00:00:00+00:00

Dang, procrastination is a terrible terrible thing. I had been doing allot of traveling prior to the Holidays and just didn’t have the motivation needed to put the time into setting up Mephisto. I finnaly got around to doing it, and it went pretty well. The Typo conversion didn’t happen as smoothly as I had hoped and I had to hack the Flickr plugin a bit to get her up and going, but she’s finally here. This time around I shouldn’t rank as #1 on Google for a series of absurdly explicit terms describing grotesque sexual actions… Thank you Akismet. I also decided to drop the blog from blog.vandev.com. From now on, this is vandev.com.

I’ve got a couple new little projects that I’ll release in the near future. Merb has entertained me as of late, as well as MooTools. Anyway, I’ll get them up and I’ll finish polishing the site. Polishing it until it shines.

The title says it all

2007-01-06T00:00:00+00:00

The title’s kinda funny right? I’m working on it, something real funny will be here soon!

MaxMind GeoLite on Rails with PostGIS

2006-10-15T00:00:00+00:00

Ughhhh, many hours of frustration have been spent trying to push the MaxMind GeoLite City CSV files into a database configured with PostGIS. Anyway, I figured I would document the process that I went through incase someone else wants some free IP based geocoding in there Rails app.

MaxMind gives you two CSV files for the city database. One has an assload (2,783,434) of IP blocks that map to the id of a row in the other table containing location data. The location data is pretty rich. It contains a postal code, area code, dma code, country, region (state in US), and a latitude and longitude. Because I wanted to be cool and more efficient, I decided to store the lat / lon as a point in PostGIS format. Using PostGIS along with Guilhem Vellut’s Spatial Adaptor plugin for Rails, makes doing geo-spatial operations sexier and easier than ever. You can use the Spatial Adaptor with MySQL, but you’ll never be as cool as the guy using it with PostgreSQL.

First off I had to create my tables.

    create_table "geo_ip_locations" do |t|
  t.column "country", :string, :limit => 2
  t.column "region", :string, :limit => 2
  t.column "city", :string
  t.column "postal_code", :string, :limit => 7
  t.column "dma_code", :integer, :limit => 3
  t.column "area_code", :integer, :limit => 3
  t.column "geom", :point, :null => false, :srid => 4269, :with_z => false
end

add_index :geo_ip_locations, :geom, :spatial => true

create_table "geo_ips" do |t|
  t.column "start_ip", :integer, :limit => 10
  t.column "end_ip", :integer, :limit => 10
  t.column "geo_ip_location_id", :integer
end

add_index :gep_ips, :start_ip
add_index :geo_ips, :end_ip
add_index :geo_ips, :geo_ip_location_id

A few things to note:

The geom column has the srid set to 4269, this threw me for a loop for way too long. Make sure you have the spatial_ref_sys table populated. I’m also assuming that the MaxMind data is mapped to the WGS 84 standard.
The limit’s on the start and end ip’s are set to 10. This creates bigint columns in my DB. I’m not sure what version of Rails started doing this (I’m on Edge), but the columns have to be bigints for the IP block information to be imported.
Remember to create a couple models for the tables
You may want to add an index on postal_code or anything else you’ll be using in a SQL WHERE clause

Now because the CSV file with the blocks of IP addresses in it is so large I went with an importing tool to get it into the table. I was going to use PostgreSQL’s COPY command, but it doesn’t seem to support CSV’s with double quotes. I went with Navicat and took advantage of my 30 day trial. The CSV with the locations in it is another story. We need to convert the lat / lon pairs into points for PostGIS. Not only that, but if we want to be really cool we need to convert the city names into UTF-8. The file that MaxMind gives us has ISO-8859-1 encoding. Iconv comes to the rescue. You can either convert the entire file from the command line, or just use the ruby library to do it in the import script below.

    require "#{File.dirname(__FILE__)}/../../config/environment"
require 'fastercsv'
require 'iconv'
ICONV = Iconv.new( 'UTF-8', 'ISO-8859-1' )

FasterCSV::HeaderConverters[:underscore] = lambda { |h| h.underscore }

FasterCSV.foreach('geo_ip_locations.csv', {:headers => :first_row, :col_sep => ",", :header_converters => :underscore}) do |row|
  begin
    row['id'] = row.delete('loc_id')[1]
    row['geom'] = Point.from_x_y(row.delete('longitude')[1].to_f, row.delete('latitude')[1].to_f, 4326)
    row['city'] = ICONV.iconv(row['city'])
    cool = GeoIpLocation.new(row.to_hash)
    cool.id = row['id']
    cool.save!
    $stdout.print '.'
  rescue
    puts $!.message
    $stdout.print 'f'
  end
  $stdout.flush
end

Things to note:

You need FasterCSV for this to work gem install fastercsv
You need to delete the first line (_with the copyright info in it_) in the CSV for this script to work.
If you converted the file from the command line, comment out the three lines referencing iconv to improve performance.
I created a directory called transform in my db directory, renamed and moved the location CSV there, and created a file with the above code in it.
Always remember to put the srid in the Point.from_x_y call, otherwise you’ll be very very frustrated.
You’ll need to specify encoding: unicode Postgres, encoding: utf8 MySQL in database.yml to use UTF-8 with the DB. See the Rails wiki
The id is set this way to preserve it.

That’s it! There you have it, the possibilities are now endless. As I said before, I set this up on PostgreSQL, but this should work the same on MySQL. The one main difference will be the srid’s. MySQL doesn’t support them, so you don’t need to specify them. Goodluck!

QuickPresent

2006-10-06T00:00:00+00:00

I threw together a quick and simple way to make a presentation on a mac using Quicksilver and Camping a while back. Well, I finally got around to refining it and putting it into a both tiny and awesome package. This puppy is made of two files, and makes presenting dead simple. You just outline your thoughts in a text file, and QuickPresent smacks it onto your screen using Quicksilver’s largetype feature. Not only that, but it also gives you a slick little web interface to show you what you’ve said, and what you’re about to say. Let me show you…

A few other slick features include automatically opening URL’s and Files from your hard drive. All you need is Camping and Quicksilver (with the command line tool (qs) plugin installed). Just type camping quickpresent.rb from the command line and your up running. If you want to try something cheaper than Keynote or Powerpoint, play with QuickPresent. Let me know what you think.

Get it from my SVN Repo: http://svn.vandev.com/presenter

Croppr podcast... a little late

2006-09-22T00:00:00+00:00

The SDRuby podcast site is filling up with great episodes. One of which being my little talk on camping and Croppr (Episode 002 on the bottom). The thing has been up for a couple of monthes now, but my posting consistency has been miserable lately. I’ve had more pressing issues, but today is a new day. My next project will be pushing this thing over to Mephisto. I should also spend some time fixing the Rails date_select helper, but that’s another post all together. The next time I speak, the entire back end of this puppy will be different… totally awesome.

Croppr is born... perhaps pre-mature

2006-07-05T00:00:00+00:00

So, I finally got around to testing TextMate’s new blogging bundle. It’s dead simple… Anywho, I figured I would use it to give a sneak peek of my newest invention Croppr, which I will be revealing in it’s full glory tomorrow night at SDruby.

All right, so Croppr was born out of a desire to make a better way to crop images online. Specifically the need came from my buddy Tom Werner’s pet project, Gravatar. The inspiration came from iChat’s avatar cropper:

After days of arduous JavaScript hacking with the help of Prototype and eventually Prototype Lite with moo.fx I had a lightweight browser equivalent. Have a peek and play around.

What good is a lightweight JavaScript cropper without a lightweight back end to actually do the cropping? Enter camping. In just 225 lines of sexy Ruby code I have a fully functional image cropping machine. I hope to have a running demo available in the near future. As far as licensing goes, I’m still working on it so hold your shorts. Until than, come to the SDruby meetup (start driving now if you have to) and see all the awesomeness that is Croppr!

EDGE and a UFO Satellite

2006-06-27T00:00:00+00:00

I have just witnessed quite the spectacle. About 15 minutes ago, I saw something strange outside of the Encinitas Ca Starbucks on the 101. In the twilight sky over the vast Pacific a UFO appeared. Screaming up from the horizon the strange object slowed abruptly and than continued on it’s path South-Southeast. At the place where it slowed a large smoke plume appeared which seemed to get brighter as the sky darkened. Fortunately I had my new W600i camera phone. I snapped 8 pics and used the phones wireless modem feature to link up to the EDGE network and send them to Flickr.

REXML could not parse this XML/HTML: 
<typo:flickr img="176781709" size="medium"/>

I watched as the object seemed to emit some type of vapor sporadically as it trailed off. This was definitely not a comet and had to have been something engineered… upon further review it must have been a satellite being launched from somewhere in the Pacific I’m very curious to see if there were any other sightings. As far as the validity of these photos go, I know the quality is poor but check out the upload timestamps on Flickr. If there were other sightings, this could easily be verified. Crazy stuff.

Update __________________________________

It is now confirmed that this was no UFO, nay it was a Delta IV rocket launching a NROL-22 reconnaissance satellite into orbit. Here’s the press release.

It was launched from Vandenburg AFB just north of Santa Barbara. I’m glad it wasn’t aliens or the Koreans…

RMagick on Mac OS X 10.4 Tiger

2006-06-06T00:00:00+00:00

Recently I needed Rmagick installed on my Mac OS X 10.4 system. This turned out to be nearly impossible, until I got down and dirty and just went for a fresh compile from source. Actually it became more impossible until I found enlightenment on the long and arduous journey. The following is a pretty bare-bones install. It leaves out excess functionality like exporting to TIFF, working with PostScript or PDF files, color management, EXIF headers, SVG images, or MSWord documents. All of this functionality can be added by downloading the source from the links above, and compiling them before you run ./configure for ImageMagick. You can also install GraphicsMagick if you like, it shouldn’t make a difference which one you use.

The following instructions will have you up and running in no time (actually about 30 minutes depending on the speed of your system ImageMagick takes forever). These instructions assume you have installed Ruby according to the instructions at HiveLogic. If you have not, make sure you atleast follow the steps up to the point of installing ruby. This makes sure your path is correct and your build environment is in place.

X11:

4 Tiger install DVD or http://www.apple.com/downloads/macosx/apple/x11formacosx.html

Freetype2:

    curl -O http://umn.dl.sourceforge.net/sourceforge/freetype/freetype-2.2.1.tar.gz
tar zxvf freetype-2.2.1.tar.gz
cd freetype-2.2.1
./configure && make && sudo make install
cd ..

Jpeg:

    curl -O ftp://ftp.uu.net/graphics/jpeg/jpegsrc.v6b.tar.gz
tar zxvf jpegsrc.v6b.tar.gz
cd jpeg-6b
ln -s `which glibtool` ./libtool
export MACOSX_DEPLOYMENT_TARGET=10.4
./configure --enable-shared && make && sudo make install
cd ..

PNG:

    curl -O http://umn.dl.sourceforge.net/sourceforge/libpng/libpng-1.2.10-no-config.tar.gz
tar zxvf libpng-1.2.10-no-config.tar.gz
cd libpng-1.2.10
cp scripts/makefile.darwin Makefile
make && sudo make install
cd ..

ImageMagick:

    curl -O http://umn.dl.sourceforge.net/sourceforge/imagemagick/ImageMagick-6.2.8-0.tar.gz
tar zxvf ImageMagick-6.2.8-0.tar.gz
cd ImageMagick-6.2.8
./configure && make && sudo make install
cd ..

RMagick (finally and thankfully):

    sudo gem install rmagick

That’s it, goto RMagick’s web site for information on using RMagick in your apps. Be on the look out for an incredible little Camping app that uses the power of RMagick to appear on VD soon!

Why do online banking websites blow?

2006-05-13T00:00:00+00:00

Recently my health insurance changed from a high deductible plan to a more reasonable PPO. Anyway I still have an HSA from my previous plan. I thought it might be a good idea to transfer my HSA money into my IRA… wow more acronyms than the w3c. After finding the website of my HSA provider, I decide to sign up for an account. This is what I got:

First off, the logo and the layout are simply stunning. Secondly, if you haven’t already, take a closer look at the requirements for both the user id and the password. Why the H are they requiring such a random user id? It’s requirements are way higher than that of the passords. Why only 4 characters of headway? How are you supposed to create anything even remotely meaningful? After 3 - 5 minutes of deep thought I was able to come up with something amusing, but I’m a power user, I doubt there demographic is even capable of figuring out how to make a user id… If they really need this crap to be that random, why not just hand out a random id?

As if this weren’t enough, I was prompted to use there advanced forgotten password system before I could continue. Take a look:

Boy, they were right on the money with this one. For starters, they opened up a new window for me, thanks Leesport, can’t get enough windows. And the questions… “my favorite president?”, who has a favorite president? You’ve gotta love the assumption that all of Leesport’s customers have pets and love sports, why wouldn’t they?

Finally after getting into this valuable service that Leesport is offering it’s customers, I found it to be utterly worthless. There was a transfers tab, but of course this did nothing. How can such a high profile site blow so hardcore? The friggin thing looks like a highschool project.

I can honestly say that Leesport’s service is by far the worst experience I have had with online banking. HSBC comes in at a close second. Fortunately, I found someone that cares about design and functionality in Bank of America.

Introducing AjaxSpy

2006-05-04T00:00:00+00:00

While playing with the new RJS templates in rails, I found it difficult to know what was actually being returned from my requests. Thus, AjaxSpy was born. It’s a simple JavaScript and CSS file. All you need to do is include the js file in the header of your document, the css is automagically included (must be in the stylesheets directory). The script relies on Prototype 1.5.0rc0, and for it to be sexy you need the Scriptaculous effects library. You can include it conditionaly based on params in the query string, or do some fancy session stuff. That’s all up to you. This is what I’ve been doing

    <%= javascript_include_tag "prototype", "effects" %>
<%= javascript_include_tag "debugger" if params["debug"] %>

The script is currently only working with Firefox and Safari, I’ll work on the bastard child that is IE down the road. Here’s what it looks like, syntax highlighting and all:

I present to you, AjaxSpy:

Again, to install simply include the js in the layout, it automagically includes the css (as long as it’s in a directory named “stylesheets” relative to the site route). Let me know if you have any problems.

A new beginning

2006-05-04T00:00:00+00:00

So, a little over a year ago, I started www.vanblog.net. It was my first experience blogging, and I didn’t really know what I wanted. Well, I have a better idea now. The problem with VanBlog was that it was either too personal or too tech for different audiences. Enter, VanDev. This is my effort to take the tech out of VanBlog. I also needed a place to put projects that I’m working on a place to thrive… I don’t even know what that means.

I’ll take this time to address questions:

Why VanDev?

Last name = Van Pelt, it’s both dutch and awesome.

What’s VanBlog?

That would be my personal blog now, where I talk about things like my feelings and changes occuring in my body.

VanDev, it’s not what you think.

What’s with these old entries?

They were brought over from VanBlog.

Do you where size 12 shoes?

Yes, as a matter of fact I do.

So, there it is. I’ll post something awesome soon about the design of this blog. Oh, you just wait.

Adelphia, getting a taste of its own medicine

2005-07-25T00:00:00+00:00

Recently I moved to a new apartment. We decided to cancel service and get service with a new provider. Adelphia SUCKS. They questioned me for “illegal” server activity. I made an FTP server to back up one of my servers back home, and the freaked out because they saw activity on port 21 and 22. Anyway, it turns out that Adelphia is the only cable company that offers service in my area. A month ago I made an appointment with Adelphia to “set up” my Internet. It just so happened that during the 2 hour window they gave me to be home, I had other arrangements. I asked them to come towards the end of the window, instead they came at the beginning and all hope was lost. The next available slot was a week from the following Saturday from 1-5. I wasn’t going to commit a four hour time slot on a Saturday afternoon, so I said ‘F’ it and bought an external WIFI antenna for my laptop. That’s when the fun started.

Things were going swimmingly until my neighbor caught on. One day the connection was WEP’ed. Fortunately I was able to hop on to another neighbors. It didn’t last long until they WAP’ed theirs (I have no idea how they knew I was on, probably bandwidth). Anyway, that was my last option. I ground my teeth, and called Adelphia to set up an appointment. That was when the unbelievable happened. I came home from work today, desperate for Internet, and I decided to try and plug my cable modem into the wall. I had thought about it before, but wrote off the possibility of the modem just hopping onto the network. Well, I’ve waisted the past month without Internet, when I could have had it for free. I’m cruising now, will Adelphia find out?

Your own personal Geocoder... for FREE

2005-07-18T00:00:00+00:00

After the release of the google maps api, I began working on custom applications. Because google doesn’t allow address lookups in its api, you have to generate a latitude and longitude yourself for a given address. The immediate problem was the need for geocoding, preferably geocoding within php. This is where the US Census comes in. The easiest solution is to download a mysql dump of all the zips with there corresponding lat/longs. That mysql dump can be found here

Well, this is great, but it’s not detailed enough. I needed to geocode to the street level. geocoder.us offers a free web service for doing this but you can only make one request every 15 seconds. I have a database of 3500 addresses, so that wasn’t going to work. geocoder.us uses the US census’s Tiger Data, and the perl Geo::Coder::US module. The problem with this solution is the amount of time it would take compile. You have to download every zip file from every state, and than compile it. I’m not patient enough. Thats when I found Dan Egnor. Dan won the 2002 Google Programing contest with a program that does just what I needed it to. Read on to do it yourself.

Ok, so the biggest issue is skipping out of the compile process. Dan offers a free download of a bz2’ed version (~300mb) of the compiled address location data (by the way, the README on his site is good reading material). Dans program does more than I needed it to, so I simplified his source code (Thanks Dan) to do simply what needs to be down. It can be downloaded here. Simply download my source, bunzip2 the map data from Dans site (assuming Linux here), and tar -jxvf the source, run make on the source and issue a command like ./geo-coder (location of the map data ex. ../../all-map) '(the address you want to find)'. This returns either SUCCESS or FAILURE. Some simple php code will execute and return the address within your web app:

REXML could not parse this XML/HTML: 
<typo:code lang="php">
$progress = array();
exec("./geo-coder ../all-map '$address'", $progress);
if(substr($progress[0],0,7) == "SUCCESS"){
 //The below returns an array with, $ll[0] = longitude and $ll[1] = latitude from Dan's program
 $ll = explode(", ",trim(substr(strstr($progress[3], '@'),1)));
}else{
 //use a zipcode (from the query or DB table in my case) to find the lat/long in the mysql zip database
 $zips =& $db->query('SELECT latitude, longitude FROM zipcodes WHERE zip = '.$row['zip']);
}
</typo:code>

So there it is. A geocoder in no time at all. The geocoder is about 80% effective (according to Dan). If enough interest is created with this post, I will create a torrent of the most up to date Tiger address information, which should improve the accuracy. Don’t hesitate to email me at vanpelt@gmail.com if you have questions or requests.

Google maps continues to amaze me

2005-06-08T00:00:00+00:00

I was browsing around today, and I found an interesting article on slashdot. It had to do with third party uses of google maps, and google cutting down on them. There turns out to be some amazing stuff being done google maps and third party software. As mentioned earlier, there was the gps in the car use but these applications are just too cool. My favorite was a tool that took craigslist rental listings and displayed them in your area dependent upon price. There is also a site that points out sites to see from the satellite imagery.

In order of coolnees:

Who knows what will pop up next???

WiFi in your car?

2005-04-20T00:00:00+00:00

I found this article in slashdot about wifi in your car. In all actuality I’m writing this article on the 28th, but I’ve been thinking about this idea for a while and it needs to be written down. I was un-aware of the ability to supply WiFi over the cell network at high speeds and a reasonable price until this was brought to my attention. Tor Amundson has a site detailing the process of creating a WiFi access point for your car using an EVDO pcmcia card. EVDO can deliver around 300kbs in it’s major supported areas (big cities), 100kbs in normal digital service areas (smaller cities), and 5kbs in remote areas (the sticks). The best part is, it is a flat fee of $80 a month for unlimited access.

Tor uses his device for some pretty cool stuff. He hooked up a GPS unit to the router and it syncs with Google Maps Standalone on a server else where. This allows for a web application that displays the exact location of a given vehicle as long as it has Internet. The cellphone networks are key here. There is Internet offered via satellite, but the satellites are too small a target to hit in motion. The first application of this that came to mind was in the RV market. Because my father deals allot with RV’s, and I generally deal with the technical end, I saw this as a great fit. Imagine the possibilities. Anyone in the coach could hook up to the WiFi and be surfing the net, going down the road. The server could be synced with the coach systems. In particular the engine heater and thermometers so you could check the temp of the coach from your house, or Japan for that matter. Security is also a big plus in this area. You could have web cams and the works, right there at your finger tips.

Beyond the RV market, my next thought was Emergency Vehicles. The possibilities are endless. You have the Internet in your vehicle now, out of this world. I hope to be thinking about this idea further and hopefully applying it soon. I should be moving back to Iowa mid summer. I will soon be immersed once again in the RV world. Look out world, here comes the Internet!

Mi Casa... Google Maps

2005-04-11T00:00:00+00:00

Google maps Try

Bad things have happened

2005-04-02T00:00:00+00:00

The blog was down for over a day. With all of the traffic I’ve been getting this is a huge deal… Actually, I had a few other larger projects on the server that were down as well. After around 3 hours on the phone with my father, it’s all up and running again.

The downtime was due to a faulty cat5 cable, we believe. It was ridiculous. At first we figured it was the router so we bought a new one. After playing with cat5 configurations we discovered it wasn’t the router. Our next step was to reboot the server. I was biting my teeth. The server had been running for 216 consecutive days (Debian!). It booted smoothly and we’re up and running now!

The experience will definitely force me to make more backups more often. It’s really difficult living in California and having your personal server in Iowa. I’m blessed to have a father with time and patience.

Seperating presentation and application logic

2005-03-23T00:00:00+00:00

I was browsing around for different web application development solutions. I’m well acquainted with PHP and I have done some separation of logic with Smarty. Smarty only does so much, and basically is just a cleaner way of presenting data. Almost a hybrid PHP. After a few articles dealing with J2EE and struts I found some php solutions for making a Model View Controller based web app. I also learned a little about Java based web design, which is interesting as well.

As far as the Java solution goes there are really two decent ways of doing it from what I understand. You could use Struts or Tapestry. Struts functions as a controller object for the web application realm. If you want to know more about the Model View Controller paradigm check out wikipedia’s answer. Basically through the use of an XML configuration file you tell struts what to do with actions from the UI. The view and model stay the same as any web application would. In this case the view is generally written in JSP and the model done with servlets. I am by know means an expert so if you want to know more check out there website.

After surfing around for a little while I found allot of complaint about the learning curve for Struts. I also read some complaints about the functionality. There was an interesting explanation of Tapestry on slashdot. Tapestry gives a level of abstraction to the web application. You know longer refer to URI’s or add query parameters, you think of everything in terms of objects, methods, and parameters. This seems like a very helpful way of programming a large project. It is just one more step in making a web application more like a desktop app.

Well, I’m a PHP guy, and I don’t see my self migrating to Java anytime soon. There are two good solutions out there that I know of currently: WACT and PHPTAL. Both seem like good solutions. WACT looks a little more feature rich, but PHPTAL is based on Zope. Zope a very mature application server. It too is an option for a large scale project. I’ll be sure to experiment with the PHP implementations in all of my free time. I’m just looking for the cleanest and most extensible way to create a web application. At the very least looking at all of these techniques gets the mind into learning mode.