In the Trenches

Cloud Storage - What's a Guy to Do?

2012-04-25T09:53:00.000-04:00

I have a lot of photos; but I'm no photographer. In fact, most of the photos are the sucky kind that only members of my family could ever like. But we do like them - they represent various important moments in our lives. Thus, it is important to me that I preserve them.

A few years ago a friend of mine had his photo harddrive crash on him. He didn't have a backup. He lost a lot of photos that were incredibly important to him. That hit home pretty hard so I made sure to keep a physically synced external harddrive backup of all my photos.

A year later another friend came to me looking for help. His laptop and his external harddrive had both died at the same time; probably some kind of power surge or something. Either way, he had lost all of his photos that he had religiously backed up to his external harddrive. That hit home again so I started syncing my photos to the cloud.

I signed up with TextDrive along time ago. They are Joyent now; but when I signed up as part of their "Mixed Lunch" plan or something like that the account came with a few bonuses. One was a "StrongSpace" account with 25GB of space. This was way before things like Dropbox existed. The tools provided by TextDrive weren't nill but I found some good windows tools that would SSH to my StrongSpace and do an rsync backup of my photos.

Then StrongSpace was sold and my space was bumped to 50GB! Perfect, I have about 40GB of photos now. But, StrongSpace kind of sucks for viewing your files online. And, because I basically just backup photos, I wanted a better solution so I paid for the 80GB plan with picasaweb at Google. Then I setup Picasa to sync all my photos there. This had the added benefit of making all of my photos available in the photo gallery of my Android phone. I was pretty happy with my setup; two remote backups and the local external drive.

Then I changed jobs. I took my external laptop with me and hooked it up to my Mac (a new computing experience for me), installed Picasa and told it to sync. It re-uploaded all of my photos and my 80GB was suddenly full. Not good. I ended up installing the Google command line tools and deleting all my photos from picasaweb. I haven't re-synced to that service. Thus my 80GB of space has been laying fallow.

I've used DropBox since about when it came on to the scene; I'm a cloud storage hipster. I love the service but I've been unwilling to spend the $200/year to get 100GB of space. I'm kind of cheap. I have a SkyDrive account through my new job but I never used it so I missed out on getting 25G for free; so I'm stuck at the 7GB entry point announced this week. Plus, I'm not sure Microsoft is really a better option for me than Google is - plus I already have the grandfathered 80GB for $20/year plan on Google so why not use Google Drive?

Well, Raganwald is why. He seems like a pretty smart guy and he has some real concerns with Google and his concerns keep me thinking and second guessing my decisions to be so tightly tied into the Google Cloud. I use them for Email, Calendaring, Blogging, Google+, Maps, heck just about everything. Do I have anything to really worry about with Google? I don't know but Reg's concerns keep on niggling in the back of my mind.

Google Drive came out yesterday and my 80GB was bounced to 85GB. That's plenty of room for me to sync all of my photos to it; plus because it works like Dropbox, all the photos will also sync to my home computer - and I can setup a local rsync on it to mirror the images to yet another external harddrive. That would give me four local copies and the two cloud copies. If I lost my photos at that point - well fuck it.

Google Drive has some cool features. I like the way it provides a web-client viewer for so many different file types (35 I think). That in itself is pretty damn handy. But it is one more way to tie myself to Google. They are currently one huge single point of failure for my online life. I don't like single points of failure so I'm tempted to use a different service for my photo syncing. And, if I'm going to switch, maybe I should take Reg's concerns to heart as I pick an alternative; thus I probably want to avoid MS as well. Sure, Microsoft isn't exactly an advertising business like Google is - but it would be naive to think they aren't also in advertising.

So where does that leave me? Here are the options I am considering: Dropbox and SugarSync. Why? Well they are the only two I've either used or have had wholeheartedly recommended to me by other Cloud Storage Hipsters of course. In either case I think I would get the 100GB plan so SugarSync is a full $50 a year cheaper. The question I then have to answer is "Is Dropbox worth an additional $50/year?" And, remember, I'm kind of cheap. However, I have to remind myself not to compare the pricing of either with that offered by Google since Google augments my low rate by mining my data.

Here is an article that does a brief rundown of the differences between a variety of different cloud storage services. Again, I'm just focusing on SugarSync and Dropbox and, honestly, based on this article I can't see a reason to not go with SugarSync for my purpose of keeping all of my photos backed up to the cloud and between my home and work computers.

So what do you suggest? SugarSync, Dropbox, or is there some other service that just kicks ass, cares about privacy, and will do just what I need at a reasonable cost? Let me know because I need to make a choice!

Remove Ubunutu Dual Boot with Windows 7

2012-03-14T10:24:00.002-04:00

My linux luck continued with a failure in getting Ubuntu 11.10 to work properly right out of the box so I decided to uninstall it. However, removing it isn't nearly as simple as installing it was. Here are the steps you need:

You need to remove the linux partition(s) - I used the free tool EASUS Partition Master, removed all of the non windows and non-recovery partitions, and then stretched out my windows partition to reclaim that space.
After you setup the new partition rules your computer has to reboot. After the bios screen the unix boot loader Grub will still appear offering to start linux or windows. Start windows but then hit f8 to get to the recovery menu for windows.
Pick to start windows in recovery mode; a graphical user interface will open. Eventually you'll have the chance to pick from a small menu of about eight options, pick the "Command Line" option (or console).
At the command line enter "bootrec /fixmbr" without the quotes. This will get rid of grub. It's almost instantaneous.
Restart your computer; EASUS Partition Master will then do it's partition resizing foo. This isn't instantaneous so just have some patience.
You're done.

Install PHP PEAR on Windows 7

2012-02-29T19:23:00.003-05:00

This seems like it should be obvious but it turns out getting PEAR isn't straight forward. The documentation at pear.php.net is inaccurate for a windows install where you use the installer at php.iis.net to install PHP on IIS7. It turns out there is no go-pear.bat file in your php directory after installing this way. However, after some trial and error and a bit of luck I figured out this seven steps to getting PEAR installed.

Download go-pear.phar at http://pear.php.net/go-pear.phar
put that file in your php home directory.
open a command window as administrator; move to your php home directory
enter "php go-pear.phar" without quotes
accept the default value for everything it asks about; system wide, path options, updating your php.ini etc
once the script is done pear is installed; you just have one last step
in windows explorer go to your php home directory and double click on PEAR_ENV.reg to update your registry

PHP array_map in practice

2012-02-29T16:32:00.000-05:00

Last week I spent some time making sure I understood how map and reduce actually work and what their purposes are. This week I needed to use map while fixing a silly problem. Some of this background info will be pure cruft to most of you but hopefully it will be enlightening in it's own way.

There is an application somewhere in our organization that utilizes an access database to store a catalog of information. This catalog is then made available for consumption within our web properties via a webservice that is written in php and which returns the data contained within the database in either json or xml format.

Here is the basic way the json was being generated previously:

 $rs=odbc_exec($conn,$sql);

 //declare an array to hold the query results
 $data = array();
 
 //populate the jobs array with the query result set
 while($curRow = odbc_fetch_array($rs)){
  $data[] = $curRow;
 } 
 echo json_encode(array('jobs'=>$jobs));

Very straight forward and, it appeared to work well until today. Someone copied and pasted a descriptive field of one of the catalog items from a word document into the access database and a single smart quote (apostrophe really) worked its way into the data. When this happened that field was rendered in the json as null even though there was clearly data contained within.
It took me a little while to see that it was a smart quote buried in the middle of the paragraph of text but once I did I knew right away how to clean it out - using MAP; or in php paralance, array_map. Combined with this nice function to convert smart quotes I had a winning solution on my hand without having to add much code at all. Here is the final version of that same code:

 $rs=odbc_exec($conn,$sql);

 //declare an array to hold the query results
 $data = array();
 
 //populate the jobs array with the query result set
 while($curRow = odbc_fetch_array($rs)){
  $data[] = array_map("convert_smart_quotes",$curRow);
 } 
 echo json_encode(array('jobs'=>$jobs));

Just updating that one line in the middle of my while loop let me make sure that every element in every row was clean of nasty smart quotes by telling array_map to call the convert_smart_quotes function on each element. I have to say I really do like the power and simplicity of map (and reduce).

Rollback from Mountain Lion to Lion

2012-02-27T13:24:00.001-05:00

Disclaimer: Before you do this; if it doesn't work you can't blame me. It worked OK for me but maybe I was just lucky.

Ok, with that out of the way here's the lowdown. I installed the developer preview of Mac OS X Mountain Lion and immediately started to notice some things that I really needed wouldn't work anymore. No parallels, macports had issues and couldn't compile things with XCode 4.4 (what is required for Mt Lion), and quite a bit more dealing with things that depended on macports or the compiler within xcode.

So what did I do? I tried to be creative and everything failed.

Oh, and I didn't have a time machine snapshot of my system before the upgrade. Anyway, I eventually went to google to find some solutions and found this posting explaining how to install uninstall Mountain Lion except it doesn't show you how to uninstall Mountain Lion at all; it just shows you how to install Lion on another partition. It does, however, have the key to the castle so to speak and I'll get into that in a little bit.

Before you can install Lion you need a copy of Lion. Makes sense right? Well I didn't have a copy but the aforementioned post told me I could get it using my apple dev center account but then it said I couldn't because I was running mountain lion and that I'd need to get it another way. Well damn.

Later in the article he said you have to fool the system to let you install lion on the other partition by changing the version number of Mountain Lion. I thought, well, what if I just do that now so I can get a copy of Lion and BAM! it worked. So here are the basic steps.

I use sublime as my text editor and I have a command alias to launch sublime using the word sublime from the terminal..
$ sublime /System/Library/CoreServices/SystemVersion.plist
It will ask you for your password; enter it then change the two version instances from 10.8 to 10.7
Save the file; you'll be asked for your password again; enter it.
Now download Lion. If you don't have an dev center account I guess you can pay the $30.
wait a while. Lion is big and it takes a long time to download. It kind of freaked me out because it showed it going into my "Applications" at first but then it felt like nothing was happening. Have some patience; go read reddit for a while.
Eventually the download will finish and the Lion installer will open automatically. Notice, nowhere in this list did I tell you to make a new partition. Screw that step!
Install right over mountain lion. Just pick the partition you normally have things on and GO.
WAIT WAIT.. finish reading this before you go...IT'S REALLY IMPORTANT
When the install is done your machine will reboot and prompt you to login but you won't be able to. Lion hates your old passwords and gets rid of them all - you can't login yet. Just reboot again and wait for the screen to go completely black then hold down cmd+r
HOLD THEM DOWN; don't just press and release. HOLD THEM BOTH until a window opens called "Utilities"
In the main menu bar there is a services option, click on it and find terminal. If it isn't services just look in each menu until you find terminal; open terminal
in the terminal window type "resetpassword" without the quotes and hit enter
A new windows opens and it shows you all of your old accounts. Find yours and reset your password.
While you're at it reset the administrator password.
Once your down with that close all the windows then in the system menu pick restart
This time you don't have to hold down any keys.
Eventually the login screen will pop up; login as yourself
Now go to the system menu and install any updates so you're nice and secure.

That worked for me (I also reinstalled xcode 4.3 from the app store after that).

Good luck

-------
Some additional notes:

After you re-install XCode 4.3; go to preferences, downloads, and install the command line tools
I also reinstalled macports to be on the safe side
parallels still doesn't work for me after the rollback.. I will reinstall parallels and hope for the best UPDATE: uninstall and reinstall fixed it.
because of some changes to mail; rolling back left me with a broken mail app; not a deal breaker as I can just use outlook as we get our mail from an exchange server at the office.
UPDATE: you can't use the systems preferences panel for setting desktop backgrounds after doing this. You can still set your primary monitor background using the "right click, services menu" on the image you wan to make your wallpaper.

Handbrake, Applescript, Automator, and iTunes Together as a Folder Action

2012-02-09T23:33:00.001-05:00

Yesterday I posted about my initial attempt at automatically converting .avi files with handbrake to m4v files and having them automatically added to iTunes. That plan worked fine if the items dropped into my watched folder were all of the type .avi AND they weren't in a subdirectory.

Tonight I decided to try and address these two deficincies. First I can now handle the case where a folder is the "input" parameter; though my knowledge of applescript is so deficient I can only deal with one layer of directory heirarchy - sorry, no recursion. I also now test for the extension of the file being processed to make sure it is an .avi before proceeding (otherwise I just skip the file).

Yesterday I also made the script look a little more generic by hiding some of the elements of my directory structure by using words like "me" - today I decided to just post it as I wrote it. You will have to change some variable values to make this work on your system.

on run {input, parameters}
 repeat with x in input

  -- need to determine if input is a file or a directory..
  set details to (info for x without size)
  set allFiles to {}

  if kind of details is "folder" then
   -- if it is a folder, we need to grab all the contents as a list
   set tempFiles to list folder (x) without invisibles
   repeat with y in tempFiles
    set the end of allFiles to ((x as string) & y) as alias
   end repeat
  else
   -- if it is a file then we make a one item list..
   set the end of allFiles to x as alias
  end if
  
  -- now loop over the list we just made.
  repeat with i in allFiles
   
   -- only process avi files.
   if name extension of (info for i) is "avi" then
    try
     set text item delimiters to ":"
     set file_name to last text item of (i as text)
     set text item delimiters to ""
    on error
     set text item delimiters to ""
    end try
    
    set origFilepath to quoted form of POSIX path of (i as alias)
    set newFileName to "" & (characters 1 thru -5 of file_name as string) & ".m4v"
    set newFilepath to "/Users/bill/temp_movies/mp4/" & newFileName & ""
    
    
    --apple uses colons as delimiters while the  Untitled=drive; Users=directory; me=username; mp4=temp directory..
    set finalPath to ("Untitled:Users:bill:temp_movies:mp4:" & newFileName)
    
    
    --start the conversion; shell command uses the forward slash as the delimiter; hence two finalpath defined.
    set shellCommand to "nice /Applications/HandBrakeCLI -i " & origFilepath & " -o " & newFilepath & " --preset=\"AppleTV 2\";"
    do shell script shellCommand
    
    --I've told itunes to copy files on import; this way I don't have m4v's laying all over the place
    tell application "iTunes"
     add finalPath
    end tell
    
    --after the import is done I delete the m4v file I just created (from the temp directory)
    tell application "Finder"
     delete finalPath
    end tell
    
    -- prepare the movie for later deletion..
    set trashPath to "Untitled:Users:bill:temp_movies:to_trash:"
    tell application "Finder"
     move i to folder trashPath
    end tell
   end if
  end repeat
 end repeat
 return input
end run

Automatically convert .avi to .m4v and add to iTunes using Automator

2012-02-08T23:44:00.000-05:00

That is a long title but not nearly as long and arduous as my process of figuring out how to do what it says. Here is the scenario...

Let's say you and your mom are sharing a dropbox folder and she plops in an .avi of her recent performance in the community theater where she lives. You don't really like the avi format and would prefer it in the m4v format and inside iTunes so you can watch it on your TV using your apple tv. What is one way to do that?

Well, I'm going to tell you. Now, trust me, this probably isn't the best way; but it works and that's all I wanted. Initially I tried the export movies task in automator but the resultant video wasn't working in iTunes - I'm not sure why. Therefore I resulted to using the Handbrake CLI from within an apple script via automator. Here is the script:

on run {input, parameters}
 repeat with i in input
  try
   set text item delimiters to ":"
   set file_name to last text item of (i as text)
   set text item delimiters to ""
  on error
   set text item delimiters to ""
  end try
  
  set origFilepath to quoted form of POSIX path of (i as alias)
  set newFileName to "" & (characters 1 thru -5 of file_name as string) & ".m4v"
  set newFilepath to "/Users/me/mp4/" & newFileName & ""
  

  --apple uses colons as delimiters while the  Untitled=drive; Users=directory; me=username; mp4=temp directory..
  set finalPath to ("Untitled:Users:me:mp4:" & newFileName)
  
  
  --start the conversion; shell command uses the forward slash as the delimiter; hence two finalpath defined.
  set shellCommand to "nice /Applications/HandBrakeCLI -i " & origFilepath & " -o " & newFilepath & " --preset=\"AppleTV 2\";"
  do shell script shellCommand
  
  --I've told itunes to copy files on import; this way I don't have m4v's laying all over the place
  tell application "iTunes"
   add finalPath
  end tell
  
  --after the import is done I delete the m4v file I just created (from the temp directory)
  tell application "Finder"
   delete finalPath
  end tell
  
 end repeat
 
 return input
end run

Basically to use this you create a new automator "Folder Action" using an "Apple Script" paste this entire thing into the script (replacing anything apple puts there by default). Then pick the folder you want to watch. I noticed that the task usually fires off about 10 seconds after an .avi file shows up in the directory.

This still needs something to confirm the file that just plopped into the watch directory is an actual .avi file. I didn't worry about that though it is probably not too hard to add that check (though with apple script and its' crazy syntax I wouldn't bet on it).

2 Months of Mac : The Good and the Bad

2012-02-07T16:01:00.001-05:00

I started using a mac two months ago after over twenty years of windows use. However, rather than reiterate my opinion on both of my blogs (personal and technical) I've posted the full story over on my personal blog and I'm just putting a link to it here:

http://blog.rawlinson.us/two-months-with-a-mac-the-good-and-the-bad

I don't like cross linking like this but I hope you'll understand that this is a topic that might be interesting to both my technical friends/followers and my non-technical friends/family. I hope you'll click through and read it and, maybe even comment with some helpful tips and tricks.

Enable/Disable an Enrollment in Blackboard Learn using IMS XML

2012-01-27T09:07:00.002-05:00

Honestly, this seemed like it would be a straight forward process. Blackboard certainly hints at it being simple and straightforward. But it turns out disabling (and re-enabling) an enrollment isn't as obvious as you might hope.

XML

There is a bit of conflicting documentation but this is how we ended up getting it to work using Blackboard Learn 9.1 SP 8.

login to blackboard learn
go to the system admin tab
go to data integration
select student information system integrations
select your xml integration option (I'm not going into how to create one)
in the little arrow drop down pick "Edit"
make sure on the Enrollments row that "deletes" is set to "Disable"
go back to the Student information system integrations page
using the arrow drop down pick "Advanced Configuration"
use the arrow drop down on enrollment and pick "Field Mapping"
CHECK the "change or update?" box for "Row Status"

Tell it to use a "Custom Script" and paste in this script:

var rs = true;
if ( data.rolesToAddOrUpdate.get(0).status = 0 ) {
rs = false;
}
rs;

WARNING: If you don't setup blackboard as indicated then including recstatus=3 as an attribute on the ROLE node will cause the enrollment to be DELETED from blackboard. This will cause all artifacts created by that user within that class to be disassociated from them; thus if they are re-enrolled in the class they won't be tied to any of those artifacts. Many of the artifacts will still be there; they'll just be associated with Anonymous.

By disabling the enrollment the artifacts remain associated with the user so that when they are reenrolled in the class they don't have to re-upload all of their assignments etc.

This process is useful for dealing with drops/holds on students as well so that their enrollment in the class isn't deleted (though it does disappear from the students interface) until their enrollment is reinstated.

Flat Files

If you are using the flat file (pipe delimited) process between banner and blackboard things are more straightforward. You just have to include the column "ROW_STATUS" with a value of "disabled" in order to disable an enrollment and "enabled" to enable the enrollment.

A disabled enrollment using either of these methods will appear in to an administrator (and perhaps in some other scenarios) with a small red circle to the left of the enrollment record.

You can learn about more of the process and the difficulties that were experienced in our transition from Vista to Learn over at John Cummings blog

Force Wordpress to be Case-Insensitive

2012-01-27T08:46:00.001-05:00

We recently had a minor annoyance happen when some marketing material was sent out that had a url with a capital "B" in it; such as http://www.mysite.com/Bill instead of http://www.mysite.com/bill. Our webserver runs IIS so it is case-insensitive but wordpress, which is being used to host the hypothetical "bill" website is written in PHP which is case sensitive.

Fortunately it is fairly easy to change this default behavior without using URL rewrite rules (which are limited to suckness on IIS 6 anyway). Wordpress Plugins to the rescue!

Our wordpress install uses a multi-site setup with over 150 sites so clearly this is a plugin you'd want to network activate. Sadly, for some reason, network activate didn't do the trick. Instead the plugin had to go in the mu-plugins directory (mu-plugins = must use plugins; they are forced onto every site and loaded before anything else basically).

I found the prefect plugin code over at unfocus.com's website but I renamed it to something that made a little more sense for us (from a maintenance perspective) in the future. Here is the code of the plugin in case that site goes away someday.

 <?php
  /*
  Plugin Name: case-insensitive-url
  Plugin URI: http://www.unfocus.com/projects/
  Description: A plugin to make wordpress case insensitive in regards to urls.
  Version: 1.0a
  Author: Kevin Newman
  Author URI: http://www.unfocus.com/projects/
  */
  function case_insensitive_url() {
   if (preg_match('/[A-Z]/', $_SERVER['REQUEST_URI'])) {
    $_SERVER['REQUEST_URI'] = strtolower($_SERVER['REQUEST_URI']);
    $_SERVER['PATH_INFO']   = strtolower($_SERVER['PATH_INFO']);
   }
  }
  add_action('init', 'case_insensitive_url');
 ?>

That's it. This now runs automatically at the beginning of every page load and insures that each request is handled in a case-insensitive manner. I'm not sure what, if any real, performance impact this has, but it is worth the cost for the reliability of knowing all incoming urls are treated the same regardless of case. It's too expensive in terms of reputation and marketing dollars to have pamphlets, postcards, and flyers printed that point to a 404 error if we don't do it.

Create App Shortcut on Mac for MacPorts installed program

2012-01-20T16:52:00.001-05:00

This is here just so I remember how to do this:

create a new automator task
type = application
add a node: "Run Applescript"

in the body of the method add

do shell script "/opt/local/bin/{executable_name} &"

change {executable_name} to the right value; for example if launching pidgin change it to pidgin
save the task to your Applications directory
profit?

Thanks go to this StackOverflow/SuperUser post that answered this for me.

I was a failure at using Platypus which was the other suggestion.

Parsing Large XML Documents in Ruby

2012-01-20T14:44:00.000-05:00

Today I was given a fairly large XML file (roughly 32 Mb) that I needed to parse in order to see if there was a match between a concatenation between two node values and the contents of a separate file that had comma separated values in it. Always looking for an excuse to use a different language than I normally do I decided to attack this in Ruby because I remember it being particularly good at handling File IO.

I'm going to present two solutions to this problem here. I'm sure they can both be improved. However, the second solution was the one I settled on because it is substantially more performant.

Example 1 - Using SimpleXML
Look, I don't know Ruby well at all so when I saw a library called SimpleXML I thought - "sweet that's right up my alley!" It really was easy to use but, as you'll see, it wasn't the right tool for this particular job.

require 'rubygems'
 require 'xmlsimple'

 duplicates = Array.new()
 enrollments = Array.new()


 # read in the xml of enrollments
 puts "reading in enrollment file; this could take a while."
 data = XmlSimple.xml_in('students.xml')

 puts "enrollment file loaded; now to parse it!"
 cnt = 0
 data['membership'].each do |course|
  courseId = course["sourcedid"][0]["id"][0].strip
  course["member"].each do |student|
   studentId = student["sourcedid"][0]["id"][0].strip
   enrollmentId = studentId << "," << courseId
   enrollments.push(enrollmentId)
  end
  cnt++
  if cnt > 9 then
   p "."
   cnt = 0
  end
 end


 #read in the comma delimited list..
 puts "reading in the drop file"
 drops = File.readlines("term_drops.lis");
 drops.delete_at(drops.length-1)


 drops = drops.collect{|x| x.strip}

 puts "comparing..."

 duplicates = enrollments & drops

 if duplicates.size > 0 then
  puts "found the following duplicate enrollments"
  puts duplicates
 else
  puts "YES! no duplicates found"
 end

By far the slowest part of this process was the line data = XmlSimple.xml_in('students.xml') - in fact with my large XML file I got bored and gave up waiting for it to load. What I needed was a parser that would pull in one node of the document at a time.

Nokogiri

It turns out there are a few solutions but I settled on Nokogiri and I'm glad I did. Honestly, the documentation is a little confusing but the library itself works great.

require 'rubygems'
 require 'nokogiri'

 duplicates = Array.new()
 enrollments = Array.new()


 puts "reading in drops file"
 drops = File.readlines("term_drops.lis");
 drops.delete_at(drops.length-1)
 drops = drops.collect{|x| x.strip}


 puts "reading in enrollment file; this could take a while."
 data = Nokogiri::XML::Reader(File.open("students.xml"))

 data.each do |node|
  if(node.name == "membership" && node.node_type == Nokogiri::XML::Reader::TYPE_ELEMENT)
   doc = Nokogiri::XML(node.outer_xml)
   classId = doc.xpath("//sourcedid/id")[0].inner_text.strip
   studentId = doc.xpath("//member/sourcedid/id")[0].inner_text.strip
   enrollmentId = studentId << "," << classId

   if drops.include?(enrollmentId) then
    puts  enrollmentId
   end
  end
 end

As you can see both scripts are pretty short - ruby really is nice for this kind of stuff. I also do a few weird things in both concerning the way I pull in my comma delimited flat file (I drop off the last line for instance but I don't really have to).

Their were a couple little gotcha's I ran into using Nokogiri. First is that when I do the data.each do |node| block it literally will iterate over every node in the document down to the most distant child. Thus I needed to filter things out by using the node.name check. Secondly, when it iterates it actually deals with every node twice; once on entering the node and once on exiting it. That is why I am checking for the node_type as well as the node.name.

Also, and this was just stupidity on my part, in order for me to do the xpath query on the node (after creating a small xml document using Nokogiri) I had to use outer_xml and not inner_xml. Using inner_xml worked ok when I was just searching for the classId because when I created the document using inner_xml it actually ignored all sibling nodes to the sourcedid node.

I also don't really know xpath all that well thus there is probably a better way for me to make sure I'm only getting the first sourcedid when I ask for classId instead of ALL sourcedid values in the document when I fire of the xpath query. I'm not really sure; if there is feel free to let me know in the comments.

Overall this code is pretty rough and specialized for this one off task I was asked to do but hopefully the Nokogiri code will help others as they parse an xml document while looking for values that might be spread out over various nodes.

Mount Samba Share on MAC to a Location of Your Choice - at Login

2012-01-19T10:25:00.000-05:00

One thing I've discovered in my move from Windows to MACs is that it isn't as easy to consistently mount a windows share to a consistent location in your file system. Initially I wasn't even sure if it was possible. The built in GUI doesn't really give you an option. For instance if you are mounting the following two shares : smb://server1/d$ and smb://server2/d$ it is possible that one will end up at /Volumes/d$ and the other will end up at /Volumes/d-1$ or something like that.

The problem is it isn't easy to tell by looking which one is server1 and which one is server 2's mount point. Thus, if you mount them in an inconsistent order server1 could end up at /Volumes/d$ once and then /Volumes/d-1$ the next time. It's not an ideal situation.

Now you may already be asking - who cares? If you setup an automator script with you login things will always be mounted in the same order and be at the same volume location and you'll just get used to knowing where things are. True. However, I work on a team and one of the things I also want to be able to do is have an ant script that can reliably run on each person's machine that can access these various samba shares (imagine one click deployment to a samba share?) Thus I need a way to consistently have all this stuff in a place that does two things:

Be a nice human readable location on the file system so you instantly know what should be there
Is the same on everyones machine and exists in an easy to recreate way for new members of the team.

Thus I am using an automator task that uses both an applescript and a bash script to achieve my result. The apple script part is used to request the users login password for the windows domain which is then used in the bash script when mounting the samba shares.

So here is the apple script part:

on run {input, parameters}
 tell application "System Events"
  set the_password to "password"
  display dialog "Please enter your Windows password: " default answer "" buttons {"OK", "Cancel"} default button "OK" with icon 2 with title "Password" with hidden answer
  set the_password to text returned of the result
 end tell
 return {the_password}
end run

I stole that bit from Aric Friesen. I removed a bunch of his script but the basic core part of it remains unchanged. In my automator task the second step is to "Run Shell Script" were you Pass Input as arguments. Here is my bash script:

 #!/bin/bash

 echo $1
 if [ ! -d ~/windows ]; then
 mkdir ~/windows
 fi
 if [ ! -d ~/windows/server1-d ]; then
  mkdir ~/windows/server1-d
 else
  umount ~/windows/server1-d
 fi
 if [ ! -d ~/windows/server2-d ]; then
  mkdir ~/windows/server2-d
 else
  umount ~/windows/server2-d
 fi


 mount_smbfs "//DOMAIN;USERNAME:$1@SERVER1/d$" ~/windows/server1-d
 mount_smbfs "//DOMAIN;USERNAME:$1@SERVER2/d$" ~/windows/server2-d

There may seem like some extraneous fluff in there but I want to make sure it works no matter if the drives have ever been mounted before. It turns out to mount a drive to a predetermined location you have to have a directory created at that location already. The mount_smbfs command will then change that directory into a mount point. Thus I have some checks to make sure my directories already exist and that the mounts haven't already happened before I try to mount to them. I would prefer if I could tell that the directory were actually already being used as a mount point (so I could skip mounting it) but I couldn't find a reliable way to identify if a directory was a mount point and what it was actually pointing to.

Once I had this automator process setup I saved it as an application called "MountNetworkDrives". From there you can make it execute on login by following these simple steps:

Open System Preferences
Click on Users & Groups
Using the "Current User" select "Login Items"
Click on the small "+" at the bottom of the right panel
Find your new application "MountNetworkDrives" and add it

And Voila! You have created a consistent way to setup mounts on your mac. Now you can share that automator script with your teammates. Just have them change the username value in the bash script step and it is good to go.

MySQL CAST and CONCAT

2012-01-11T14:13:00.000-05:00

CAST
This is just here to help me remember. In MySQL if you want to CAST an int into a string you use:

CAST(x AS CHAR)

I had assumed it would be VARCHAR but I was wrong. As a further reference here is info on all of the TYPES supported by MySQL. What I find interesting is the fact that MySQL does have a VARCHAR type but you can't cast an int to it. Why Not? In the end it doesn't matter too much; the CHAR option worked.

CONCAT

A similarly odd behavior is how you concatenate strings in MySQL. I had thought I would use & or || but both of those return a 0 if you use them to concatenate. Instead you have to wrap all of the strings you want to concatenate up into a call to the CONCAT function like so:

CONCAT('http://www.rawlinson.us/blog', page_name, '/datafiles/?file_id', CAST(file_id AS VARCHAR)) as url

How many Days|Weeks|Months Ago? [php]

2012-01-11T14:05:00.000-05:00

Yesterday I wrote a simple wordpress plugin that checks an rss feed and finds posts in that feed that contain specific keywords. Any posts found that are also younger than n number of days are then displayed at the top of a page just as if they were posts in the wordpress blog.

When I was display the data though I thought it would be nice if the pubdate sent with the feed actually showed up as "2 days ago..." or "4 months ago.." or whatever was applicable. Thus, I did what any good lazy coder does I googled to see if anyone had written this up in PHP already and sure enough someone had.

I found this function on the drupal support community message boards posted by someone named "drenei"

   <?php
   function ago($timestamp){
   $difference = time() - $timestamp;
   $periods = array("second", "minute", "hour", "day", "week", "month", "years", "decade");
   $lengths = array("60","60","24","7","4.35","12","10");
   for($j = 0; $difference >= $lengths[$j]; $j++)
   $difference /= $lengths[$j];
   $difference = round($difference);
   if($difference != 1) $periods[$j].= "s";
   $text = "$difference $periods[$j] ago";
   return $text;
  }
   ?>

Installing Ruby on a Mac fails with an error while running configure

2011-12-12T10:25:00.003-05:00

Today I tried to upgrade my Ruby version to the latest (1.9.3-p0) using RVM (Ruby Version Manager). When I tried the install it failed with an error:

ERROR: Error running ' ./configure --prefix=/Users/bill/.rvm/rubies/ruby-1.9.3-p0 --enable-shared --disable-install-doc --with-libyaml-dir=/Users/bill/.rvm/usr ', please read /Users/bill/.rvm/log/ruby-1.9.3-p0/configure.log
ERROR: There has been an error while running configure. Halting the installation.

I wasn't sure what to do but after a bit of digging on google and some forums I found the solution. But, before I give you that here is the reasoning I found for the issue:

it works with 1.9.2 - BUT I was also not able to get RVM going on Lion. I have xcode 4.2.The problem is that xcode uses LLVM GCC and not (standard? I forget the term) GCC.

Finally, here is how to get it to install without any issues:

rvm install 1.9.3 --with-gcc=clang

This solution was found via dcinzona at github. He found the answer on stackoverflow

Make F5 Refresh the page in Chrome on a Mac

2011-12-07T13:19:00.001-05:00

It turns out the F5 key doesn't do a page refresh on a mac. That is a habit I just don't think I'll be able to break. Fortunately, macs have a built in way to define application specific keyboard mappings. Here are the instructions for mapping the F5 key to refresh in Chrome.

Launch "System Preferences"

Click the "Keyboard" icon

Select "Keyboard Shortcuts" tab

Hit the little "+" button under the main white area

In the little pup-up window select "Google Chrome" as the Application

Type in "Reload This Page" in the "Menu Title" field exactly as it appears in the Chrome's View menu (no quotes of course)

Click inside the "Keyboard Shortcut" field and hit F5 key (or any desired combination)

Restart Chrome to make it work

If you use safari this same process will work for that application as well

Thanks to "paulcik" and his comment on the google support forums.

My New Job

2011-11-18T10:34:00.001-05:00

For the past 12 years I have worked for a wonderful small company named Strictly Business Computer Systems. During that time I've had the priviledge of working alongside some amazingly bright and talented people. I will miss both the workplace and my coworkers enormously as I move forward towards a new opportunity at Marshall University. I'll still be developing software using a variety of languages and tools and I'll still be blogging my solutions to weird little problems; I'll just be doing it with a little less focus on ColdFusion and a little more on PHP, .Net, and SharePoint.

Can't Establish Security Trust Relationship with US Govt

2011-10-17T13:28:00.004-04:00

If you are ever working on a project for the US Govt and are accessing some site within their network (via a VPN) and you start to get security and trust relationship errors then you need to install the DoD certificates. I found them at this US Army Corps of Engineers Website. I downloaded the zip file, ran the installer, and all of my trust problems with the government were resolved.

Create a C# Model From a SQL Server Table

2011-09-29T13:52:00.000-04:00

If you aren't using a framework of some sort that helps make this even easier here is a sql script that will help you generate a set of getters/setters for a SQL Server table in C#.

I haven't tested it much (nil) but it looks like it will work ok. Basically you can just cut and paste the records returned into your class definition and you'll be good to go. You may, however, wish to fix the case of the property names to match the proper convention. I'm not sure how to fix the case using just SQL.

SELECT 'public '  +
		CASE data_type
			WHEN 'bigint'		THEN 'int'
--			WHEN 'binary'		THEN 'raw'
			WHEN 'bit'		THEN 'int'
			WHEN 'char'		THEN 'char'
			WHEN 'datetime'		THEN 'DateTime'
			WHEN 'decimal'		THEN 'decimal'
			WHEN 'float'		THEN 'float'
--			WHEN 'image'		THEN 'long raw'
			WHEN 'int'		THEN 'int'
			WHEN 'money'		THEN 'double'
			WHEN 'nchar'		THEN 'string'
			WHEN 'ntext'		THEN 'string'
			WHEN 'nvarchar'		THEN 'string'
			WHEN 'numeric'		THEN 'int'
			WHEN 'real'		THEN 'decimal'
			WHEN 'smalldatetime'	THEN 'DateTime'
			WHEN 'smallmoney'	THEN 'double'
			WHEN 'smallint'		THEN 'int'
			WHEN 'text'		THEN 'string'
			WHEN 'timestamp'	THEN 'Date'
			WHEN 'uniqueidentifier' THEN 'string'
			--WHEN 'varbinary'	THEN 'raw'
			WHEN 'varchar'		THEN 'string'
			ELSE 'CHECK THIS.. ' + data_type + ' DataType'
		END + ' ' + COLUMN_NAME + ' { get; set; }'
	
FROM information_schema.columns
WHERE table_name = 'class'

There are also a few data types in here I'm not sure I converted properly; this was just a quick one-off thing I put together for a friend. So, if you have any suggestions or corrections please leave them in the comments and I'll update it as I go.

How to Get Group Consensus

2011-09-15T16:44:00.000-04:00

This is a post I started writing a few years ago (March 2008) but never published for some reason...

I don't have any first hand experience with the KJ Technique but it sounds pretty promising. I can't explain it as well as this linked article does so I'll just send you there. However, if you have already used this technique (or something like it) and have some feedback I'd love it if you left a comment.

The KJ-Technique: A Group Process for Establishing Priorities
I actually came across this article via Assaf Arkin's blog Labnotes. I just started reading his blog last week so it's already paid off in introducing me to this new concept. Thanks Assaf.

As I started reading the article on the KJ Technique I couldn't help but continually think about Assaf's idea of building this as a web service. Well, actually, it kept focusing on potential UI designs for really bringing the power of the technique to a web-based application. My conclusion? I'm not very good at UI design! My focus was more on how to bring this concept to a distributed team without need of some kind of teleconference. How can you simulate the sticky-notes that can be grouped - or regrouped by different non-directly-communicating team members? If you can come up with a good UI that clearly communicates the shift of a sticky from one group to another (or a total merging of groups) by a remote user to all other users in the group it would be fantastic. In fact, if it was really clear what was going on all the time and everyone could keep messing with the virtual sticky-notes I almost think a distributed version of the KJ-Technique would be better than the one described in the article because it would really help enforce the "no discussion" rule for the first 2/3's of the process.

console.log in IE

2011-09-15T16:29:00.001-04:00

If you're like me you've probably been happily working on a web project in chrome or firefox; using console.log to debug and then tried the site out in IE and had it fail due to a javascript error. The error? console is undefined of course. Here is my simple and very limited solution to the problem. At the beginning of my site, before any other javascript is defined I have this little snippet:

var console = console || {
    log : function(text){
        /* 
        do nothing.. this prevents us from having errors
        in ie when we accidentally leave console.log calls in the js code
        on deployment:
        
        if we want a cheap console in ie just add alert(text); to this outside the comments
        */
       
    }
}

Basically what this does is it checks to see if console is defined, if so it returns the already defined console object. If the console object isn't defined, then it returns a new object that contains a method of log. The log method takes in a value and then does nothing with it. The comment block tells you how you can make it do a bit more. There are plenty of more feature rich solutions to this problem out there but, in general, this works for me. It's lightweight and easy to use. I hope it saves you a few tiny headaches in the future.

Quickly Script a SQL Server View to Oracle Table

2011-08-19T17:06:00.001-04:00

I recently had a need to mirror a SQL server view in an Oracle database. Without getting to detailed the basic premise is that a job runs once daily that copies data from the SQL Server via hetergenous data services into the Oracle table. This allows some useful reporting queries to run that are joined across multiple Oracle schemas without having the slowdown involved with communicating directly over the heterogenous link on each query.

Anyway, some of these views are stupid huge and I did not want to type each and every column name and column type. Thankfully there is a great resource inside SQL server that made my life quite a bit easier - the INFORMATION_SCHEMA.COLUMNS table. It basically describes all the bits and pieces involved in all of the tables and views in your database.

It's important to note that not all SQL server columns fit exactly into Oracle columns so I also needed to include a bit of logic to make the translation between datatypes.

Here is the query:

SELECT '"' + column_name + '"     ' +
		CASE data_type
			WHEN 'bigint'		THEN 'number(19)'
			WHEN 'binary'		THEN 'raw'
			WHEN 'bit'		THEN 'number(3)'
			WHEN 'char'		THEN 'char'
			WHEN 'datetime'		THEN 'timestamp'
			WHEN 'decimal'		THEN 'number(' + CAST(numeric_precision as varchar) + ',' + CAST(numeric_scale as varchar) + ')'
			WHEN 'float'		THEN 'float(49)'
			WHEN 'image'		THEN 'long raw'
			WHEN 'int'		THEN 'number(10)'
			WHEN 'money'		THEN 'number(19,4)'
			WHEN 'nchar'		THEN 'nchar'
			WHEN 'ntext'		THEN 'long'
			WHEN 'nvarchar'		THEN 'nchar'
			WHEN 'numeric'		THEN 'number(' + CAST(numeric_precision as varchar) + ',' + CAST(numeric_scale as varchar) + ')'
			WHEN 'real'		THEN 'float(23)'
			WHEN 'smalldatetime'	THEN 'date'
			WHEN 'smallmoney'	THEN 'number(10,4)'
			WHEN 'smallint'		THEN 'number(5)'
			WHEN 'text'		THEN 'long'
			WHEN 'timestamp'	THEN 'raw'
			WHEN 'uniqueidentifier' THEN 'char(36)'
			WHEN 'varbinary'	THEN 'raw'
			WHEN 'varchar'		THEN 'varchar2( ' + CAST(CHARACTER_MAXIMUM_LENGTH AS varchar) + ')'
			ELSE 'CHECK THIS.. ' + data_type + ' DataType'
		END + ', '
	

FROM information_schema.columns
WHERE table_name = 'your_table_or_view_name'

This will basically give you backa bunch of records that have the column name surrounded by quotes and then the type of the column followed by a comma.

Once you have that you just need to prepend the "CREATE TABLE tablename (" bit at the beginning and remove the final comma and append a closing ");" and you've got a nice script to create a table.

If you have a small table this probably isn't too helpful. However, some of the views I have to mirror have over 250 columns!

It is possible that at times you will see a -1 in the datatype field. If that happens it is because the length was set to "MAX" however I didn't really need to update my query to handle that so I'll leave that as an exercise for you if you need to handle that special case.

Oracle Error: ORA-12541 - TNS: no listener

2011-07-26T11:28:00.001-04:00

I was on vacation last week and when I returned, I could no longer connect to my local Oracle instance. My C# apps and PL/SQL developer both reported a ORA-12541 TNS: no listener error.

I hadn't changed anything before I left and I certainly hadn't changed anything as soon as I got back so I was a little perplexed. Fortunately, there was a pretty easy solution which I extracted from The Oracle FAQ.

Check your listeners status by going to the command line and entering the Listener Control Program using command LSNRCTL. Once in the program enter the command: STATUS

The tool suggests you also start and stop the listener but that had no effect for me. My listener just didn't want to work.
Recreate your listener. Again, go to a command prompt but this time open up the Network Configuration Assistant using the command: netca
Inside this tool you first need to delete your listener. Mine was actually called "listener" and I just accepted all of the defaults when deleting it. Once it is deleted, recreate it, again I used all of the defaults.
After it was created I went back to the Listener Control Program (LSNRCTL) and ran the STATUS commaned. This time it looked pretty good and, sure enough, my apps and PL/SQL Developer were able to connect to the database.

Google+ : My Favorite Feature So Far

2011-07-04T11:54:00.000-04:00

When I first heard about Google+ I thought that circles would be my favorite feature. I thought having an easy way to filter who I published things to would be great. And, so far, it is. However, there is another aspect of circles that I like even more. They are helping me find people who share my interests and who have interesting things to talk about.

For example, with this Blog I talk about development stuff leaning mostly towards Cold Fusion development. Thus when I was defining my circles I created a ColdFusion circle and put some folks like Sean Corfield and Marc Esher in my CF circle. They both usually have interesting stuff to talk about in regards to CF and so I'm happy they both also put me into a circle that also seems to deal with development topics.

These relationships were previously defined though. We've all communicated in some fashion or another online either via twitter, a forum, or facebook. Because of these relationships I trust that the people they are talking to about development stuff actually care about it and probably have something to add to the conversation at some point. Thus, when one of them posts about a development topic to a limited audience I can look at the other members of the limited audience and instantly find other people who share that particular interest. I can't, necessarily, hear what they have to say - but I can begin to form a new relationship with these people by putting them in one of my circles.

Once they are in my circle they will start to see some of my content on that topic. If they like what I have to say they will most likely add me to their development circle and then I'll start hearing what they have to say and we will begin an online relationship around our shared interest in development.. If they don't like what I have to say they will just ignore me and the relationship will be terminated - which is fine.

This works for any topic I'm interested in. I might have a C# circle or a Disc Golf or a Mountain Bike circle and, in each, I might find; via the extended circles, others who share my interest and who I might want to have a lasting conversation with.

Circles, for me, aren't as much about filtering for who hears what I have to say but are about finding people who have something to say I want to hear.