A New Begining

Ruby, Mechanize, Nokogiri, Xpath and Firebug

noreply@blogger.com (Shawn) — Sat, 16 Oct 2010 19:08:00 +0000

I have done a fair amount of extremely painful screen scraping to get data for a web page in the past using tools like WWW::Mechanize or LWP for perl. It can be disgustingly ugly to parse HTML to get at the data you need. When it came it recently all the bad experiences I have had came rushing back and I was dreading starting. Luckily the world is a nicer place and this had become trivial with ruby. The once painful process became:

Open page I want to scrape in firefox
load firebug
rick click on the html element I want to scrape and select copy xpath
fire up textmate and create a simple ruby mechanize script
past in the xpath to script.

The code simply looks like this:

require 'rubygems'
require 'mechanize'
require 'mysql'

agent = Mechanize.new

page = agent.get('http://url.to.scrape.com')

upString = page.parser.xpath("/html/body/form/table[2]/tr/td/table/tr[5]/td/table/tr[6]/td[3]/a/img")[0]['title'].to_s

Life is good.

Creating an image of a running EC2 instance

noreply@blogger.com (Shawn) — Sun, 26 Sep 2010 02:34:00 +0000

Prerequisites

AWS User ID
AWS Key ID
AWS Secret Key
x.509 Key pair (cert and private key)

All of this can be obtained from the AWS account info page here. Note AWS does not store the private key of your x.509 key pair, if you do not have you will need to create a new key pair.

Creating the bundle

Upload your x.509 cert and private key to your running ec2 instance.

scp PATH_TO_KEYS/{cert,pk}-*.pem root@AWS_INSTANCE:/mnt

Log into your ec2 instance

ssh -i YOURKEY.pem root@AWS_INSTANCE

Set up some environment variables to make the processes a little easier. Set arch to either i386 or x86_64 depending if you have a 64 bit or 23 bit instance. If your not sure which to choose you can check here

# export AWS_USER_ID=YOUR_AWS_USER_ID
# export AWS_ACCESS_KEY_ID=YOUR_KEY_ID
# export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY
# export arch=i386

Create bundle

ec2-bundle-vol -r $arch -d /mnt/ -p $prefix -u $AWS_USER_ID -k /mnt/pk-*.pem -c /mnt/cert-*.pem -s 10240 -e /mnt,/root/.ssh

Upload bundle

ec2-upload-bundle -b $bucket -m /mnt/$prefix.manifest.xml -a $AWS_ACCESS_KEY_ID -s $AWS_SECRET_ACCESS_KEY

ec2-register --name "$bucket/$prefix" $bucket/$prefix.manifest.xml

Compiling GCC 4.4.3 on Solaris

noreply@blogger.com (Shawn) — Sat, 27 Feb 2010 22:32:00 +0000

First I have to say the compiling on solaris can be a major pain in the ass. I found my-self wanting gnu find on one of our solaris 10 machines but had issues compiling it, there was a known bug in gcc that can be fixed by upgrading gcc. That is when the fun began. I download gcc from gnu.org. The first problem I ran into was that I didn't read the dependency list, my bad. I needed to grab gmp and mpfr. I downloaded and compiled both and passed the --with-gmp and --with-mpfr flags and I though I was all set. Turns out that I was wrong I ran into a problem finding the mpfr lib. That sucked seeing as I just downloaded and compiled them. After looking at the config.log it was pretty obvious, I was building against 64 bit libraries. Adding the following got me to the next error:

env CC="gcc -m64"

The next problem was similar, apparently Solaris includes the 32 bit binaries in the search path but not 64 bit. The error I got was:

configure: error: cannot compute suffix of object files

When I checked the config.log the specific error I found was:

libgcc_s.so.1: wrong ELF class: ELFCLASS32

I was able to fix it with the following:

export LD_LIBRARY_PATH=/usr/sfw/lib/64/

Ok so now things looked to be on the right path. The compile went on for an hour and half and died with a new error, again the error was wrong ELF class. At this point I was about to pull my hair out. The problem was the CFLAGS were not being passed down to the cross coplier, note the following line:

/users/srb55/gcc-4.4.3/host-sparc-sun-solaris2.10/prev-gcc/xgcc -B/users/srb55/gcc-4.4.3/host-sparc-sun-solaris2.10/prev-gcc/ -B/global/inf/sys/software/gcc/gcc-4.4.2/sparc-sun-solaris2.10/bin/ -c -g -O2 -DIN_GCC
--- snip ---

So the fix to this wasnt so bad I just disbabled the bootstrap compile with --disable-bootstrap. This time the compile completed (after like 3 hours). The configure line that worked in the end was:

./configure --prefix=/global/inf/sys/software/gcc/gcc-4.4.3 --with-mpfr=/global/inf/sys/software/mpfr/mpfr-2.4.0 --with-gmp=/global/inf/sys/software/gmp/gmp-5.0.1/ --enable-shared --disable-nls --disable-bootstrap --disable-multilib -enable-languages=c,c++

Victory!

Creating a TextMate Bundle

noreply@blogger.com (Shawn) — Mon, 16 Mar 2009 00:31:00 +0000

I spend a fair amount of time dealing with code written in SQR. Since this isn't a very popular language there isn't a good editor with syntax highlighting for it. I use TextMate a good deal and so it seemed natural to write a bundle for SQR. Step one is to open TextMate and click Bundle then Bundle Editor and then Show Bundle Editor. Once there click the plus at the bottom of the screen and add new bundle.

Now name the bundle something appropriate, in my case SQR. Now you need to add a new language to your bundle, the language is what controls syntax highlighting.

This will generate the following code.


{ scopeName = 'source.untitled';
fileTypes = ( );
foldingStartMarker = '/\*\*|\{\s*$';
foldingStopMarker = '\*\*/|^\s*\}';
patterns = (
 { name = 'keyword.control.untitled';
  match = '\b(if|while|for|return)\b';
 },
 { name = 'string.quoted.double.untitled';
  begin = '"';
  end = '"';
  patterns = (
   { name = 'constant.character.escape.untitled';
    match = '\\.';
   },
  );
 },
);
}

Lets go through this in detail.

scopeName (line 1) — this should be a unique name for the language. The convention is for these to be dot separated where the left most piece is the most specific. In my case I used source.SQR. (TextMate 1)
fileTypes (line 2) — this is an array of file type extensions that the language should (by default) be used with. This is referenced when TextMate does not know what grammar to use for a file the user opens. If however the user selects a grammar from the language pop-up in the status bar, TextMate will remember that choice. (TextMate 1)
foldingStartMarker / foldingStopMarker (line 3-4) — these are regular expressions that lines (in the document) are matched against. If a line matches one of the patterns (but not both), it becomes a folding marker. The means you will get the arrow in the left margin that will allow you to expand and collapse large sections of code (TextMate 1)
patterns (line 5-18) — this is an array with the actual rules used to parse the document. In this example there are two rules (line 6-8 and 9-17). (TextMate 1)

For example this is the beginning of my definition for the SQR language.


{ scopeName = 'source.SQR';
fileTypes = ( 'sqr', 'sqc', 'SQR', 'SQC' );
foldingStartMarker = '(?i)^\s*+(Begin-Procedure|If|Begin-Report|Begin-Heading|begin-select|begin-sql|#ifdef).*';
foldingStopMarker = '(?i)^\s*+(End-Procedure|End-If|End-Report|End-Heading|end-select|end-sql|#end-if).*';
patterns = (
 { name = 'comment.line.double-slash.SQR';
  match = '!.*\n';
 },
              ...
);
}

The name for the scope is source.SQR and it operates by default on files with the following extensions sqr, sqc, SQR, and SQC. Lets look at just one of the folding markers, Begin-Procedure/End-Procedure. The regex match I set up does a case insensitive match looking for a line that starts with any number of spaces then Begin-Procedure and basically the same for End-Procedure. The result:

Yeah for little arrow thingys!

So the other you might note is the lines with '!' are grayed out, this is because they are set to be comments by our pattern. The name of the pattern is a special name that matches a TextMate property, for a full list see here under 12.4 Naming Conventions. In this case I choose one that is for comments, "comment.line.double-slash" then added ".SQR" for the right scope. The second part of the pattern is the match in this case "!.*\n." This will watch and '!' and any characters after it.

You can go on to add more patterns for reserved words, variables, strings, etc. In not too long you can have syntax highlighting all set. Then you can go on to add snippets and other commands, but we while have to save that for another time.

You can download what I have so far here.

(TextMate 1, http://manual.macromates.com/en/language_grammars#naming_conventions.html)

My Next Mac?

noreply@blogger.com (Shawn) — Sat, 27 Sep 2008 11:14:00 +0000

Bricked Mac - Time-Lapse from RP Cuenco on Vimeo.

Installing Oracle Calendar on Fedora

noreply@blogger.com (Shawn) — Sun, 14 Sep 2008 17:34:00 +0000

This will be the first post in a series chronically my "Can I use Fedora for my work OS?" project. My first task was to see if I could install Oracle Calendar which is our enterprise calendaring system.

In theory this should not be a hard install you can get the installer from oracle.com and unzip that tarball and run gui_install.sh. Of course things never go that easy, the first error I got was this:

Preparing to install...
Extracting the JRE from the installer archive...
Unpacking the JRE...
Extracting the installation resources from the installer archive...
Configuring the installer for this system's environment...
awk: error while loading shared libraries: libdl.so.2: cannot open shared object file: No such file or directory
dirname: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
/bin/ls: error while loading shared libraries: librt.so.1: cannot open shared object file: No such file or directory
basename: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
dirname: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
basename: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
hostname: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
Launching installer...
grep: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
/tmp/install.dir.7150/Linux/resource/jre/bin/java: error while loading shared libraries: libpthread.so.0: cannot open shared object file: No such file or directory

After doing some research it seems that the LD_ASSUME_KERNEL environment var can cause problems, it is old var used during the time of kernel threading switch. Some apps, mostly java, still have this set and it cuases problems in fedora. The soluction is to turn it off:


perl -pi -e 's/export LD_ASSUME_KERNEL/#xport LD_ASSUME_KERNEL/' cal_linux

After that I got a missing object error:

libXp.so.6: cannot open shared object file: No such file or directory

This was an easy fix, I just installed the missing lib.


yum install libXp.so.6

The Final error I had was:

libstdc++.so.5: cannot open shared object file: No such file or directory

I checked the lib version and found that I had so.6. My first though was to link this to the so.5 and hope for the best. I decided to first look for a compat library and I found it. The solution:


yum install compat-libstdc++-33

Finally I had a working calendar! Woo Hoo!

Techie Haiku

noreply@blogger.com (Shawn) — Wed, 03 Sep 2008 11:15:00 +0000

Your razor-sharp wit
Can never stand up to my
Adamantium

My Macbook Lives

noreply@blogger.com (Shawn) — Tue, 02 Sep 2008 19:29:00 +0000

My macbook experienced some serious issues last Friday. When I boot I get the login screen because I have enabled the root account for various things. When I logged in as my normal user I got stuck at a blue screen that would flash intermittently. It was like the mac couldn't log in. After some painful trouble shooting here is what I did to fix it:

login into the computer using the root account
created a new user called xyz
renamed the home folder (/Users/xyz) to xyz_old
renamed my home folder to xyz
rebooted
logged in as xyz

I was able to login as the new user and all my settings were just as I left them. I did have to type in my password for the keychain but that worked fine. The best part is I still had my time machine back up just in case.

I need this for my mac

noreply@blogger.com (Shawn) — Tue, 02 Sep 2008 19:05:00 +0000

Garcias is rocking

noreply@blogger.com (Shawn) — Mon, 25 Aug 2008 23:19:00 +0000

This is the first time we have got my brother to eat at Garcias in
like five years! Mallory is way excited!!!

This scares the crap out of me

noreply@blogger.com (Shawn) — Tue, 12 Aug 2008 15:20:00 +0000

When to reinvent the wheel

noreply@blogger.com (Shawn) — Mon, 11 Aug 2008 01:30:00 +0000

One of the cardinal rules of software development is to never reinvent the wheel. I have in my ten years of software development strictly adhered to this rule, primarily because programmers are lazy. For the most part it is in your best interest to used tested code the is already written, however sometimes it can be a detriment.

Recently I was working on a project where I wanted to do an auto suggest drop down for a text box. I started off, as usual, slogging through the net looking for something that work. Without too much work I was able to find about three contenders. I downloaded, installed and tested each. None of the three was really what I wanted so I adventured back to the Internet in search of the perfect auto suggest JavaScript and before I knew it I had spent a whole work day (8 hours) just scouring the internet.

I took a step back and wrote:


  function suggest() {
   var searchBox = document.getElementById('search-form-query');
   var suggestions = document.getElementById('suggestions');
   
   var url = 'searchData.cfm';
   var pars = 'field=' + searchBox.value;

   var ajax = new Ajax.Updater(
   {success: 'suggestions'},
   url,
   {method: 'get', parameters: pars, onFailure: reportError});
   
   suggestions.style.display = 'inline'
  }
  
  function hideSuggestions() {
   var suggestions = document.getElementById('suggestions');
   suggestions.style.display = 'none'
  }

With a tiny bit of HTML to help:


<div id="suggestions" style="overflow: auto; position: absolute; background-color: white; width: 150px; height: 250px; display: none;" onclick="hideSuggestions()">This is where the auto suggestions will go</div>

This code took me maybe ten minutes to write and test and it was exactly what I wanted in the end. In the end it was far more efficient for me just to write the code and move on. It's important for programmers to strike a good balance between writing their own stuff and using packages from the Internet.

How to Pick a Lock with a Bump Key [How To]

noreply@blogger.com (Shawn) — Sun, 10 Aug 2008 03:55:00 +0000

I made one :)

Sent to you by Shawn via Google Reader:

How to Pick a Lock with a Bump Key [How To]

via Lifehacker by Gina Trapani on 8/2/08

From the "use this for good, not evil" files comes a fascinating instructional video on how to pick a lock with a "bump key"—a key modified to fit and open any lock. Like the instructor in the video says, thieves have been using this technique for a long time now—so it's interesting (in the "knowledge is power" kind of way) to see how it's done. Before you start making evil plans, keep in mind that most states in the U.S. consider a bump key a burglary tool so don't go getting yourself arrested.

Bump a Lock [How-To Wiki]

The Guild

noreply@blogger.com (Shawn) — Thu, 07 Aug 2008 16:37:00 +0000

The Shinnning

noreply@blogger.com (Shawn) — Sun, 27 Jul 2008 07:51:00 +0000

Dog Barks for six years

noreply@blogger.com (Shawn) — Wed, 23 Jul 2008 15:17:00 +0000

Tiny Dog Has Been Barking Nonstop For 6 Years

The iPhone has arrived.

noreply@blogger.com (Shawn) — Tue, 15 Jul 2008 00:06:00 +0000

http://www.geekculture.com/joyoftech/joyarchives/1128.html

Log rotate final solution

noreply@blogger.com (Shawn) — Thu, 10 Jul 2008 03:07:00 +0000


#!/usr/bin/perl -w
use strict;

my $numDays = 7;        #Number of days to retain Output Directories
my $numLongDays = 30;   #Number of days to reatain archived tars

#get the current date time and break up
my ($sec,$min,$hour,$mday,$mon,$year,$wday,
$yday,$isdst)=localtime(time);

#create date stamp
$year = $year + 1900;
my $dateStamp = "$mon-$mday-$year";

#find out who is running this process
my $who = `whoami`;
chomp $who;

#Get the list of directories
my @baseDirs = `find /usr/peoplesoft/$who/appserv/prcs/* \\( ! -name dir -prune \\) -type d -print`;

#for each dir arcvhive and delete folder from log_ouput
foreach my $baseDir (@baseDirs) {
  chomp $baseDir;

  #set the path to the directory that has the logs and the archive directory
  my $logDir = "$baseDir/log_output";
  my $logArchiveDir ="$baseDir/log_output_archive";

  unless(-d $logArchiveDir){
      mkdir $logArchiveDir or die "Couldn't create dir: [$logArchiveDir] ($!)";;
  }
 
  #archive then delete out put folders
  system("/usr/bin/tar cvf - `/usr/bin/find $logDir/* -mtime +$numDays` | gzip > $logArchiveDir/log_output_$dateStamp.tar.gz");
  system("/usr/bin/find $logDir/* -mtime +$numDays -exec rm -rf {} \\;");
  system("/usr/bin/find $logArchiveDir/* -mtime +$numLongDays -exec rm -rf {} \\;");
}

Why its important to rotate logs.

noreply@blogger.com (Shawn) — Wed, 09 Jul 2008 01:13:00 +0000

Any of you who have been in the sysadmin business for more then a few days know how critical it is to rotate log files. When logs get to big your system will stop working. For example each file system has a maximum file size (http://www.novell.com/documentation/suse91/suselinux-adminguide/html/apas04.html#tab:maxsize) which if reached will cause problems. In the old days linux had a 2 gig ceiling and when your apache access log reached it apache would crash and not come back up. Some times worse.

Today I ran into a new one, in Solaris there is a limit to the number of sub directories that can be created. The magic number is none other than 32,767. We had a process that was storing log files for in there own sub directories. It had been doing this for multiple years and finally it had the lmit. We got this rather unhelp error, mkdir: Failed to make directory "test3"; Too many links.

The solution was rather simple. Tar up and archive the folders and then delete the originals. I set up a quick script to do this monthly and hopefully we will be all set. Going forward I would like to get a more robust solution in place like the standard logrotate. Dare to dream.

Siesta

noreply@blogger.com (Shawn) — Thu, 19 Jun 2008 17:57:00 +0000

Good on article on how to nap: http://www.boston.com/bostonglobe/ideas/naps/.

Arent they just precious

noreply@blogger.com (Shawn) — Wed, 18 Jun 2008 18:37:00 +0000

Uncle Shawn

noreply@blogger.com (Shawn) — Wed, 18 Jun 2008 02:50:00 +0000

I just found out that I'm going to be an Uncle, how cool is that? It is totally awesome and scary all at the same time, I cant imagine how my brother feels! It really gets you thinking about life and the whole cycle of things. I'm way excited for this little sea monkey! I cant wait to teach him/her how to calculate pot odds and bluff on the river. I'm already thinking about taking them to scouts and camping and the other nine million other things that will be fun to do.

Congratulations Chad and Mal!!!!!

Firefox 3

noreply@blogger.com (Shawn) — Wed, 18 Jun 2008 02:36:00 +0000

I'm very impressed with the latest release of Firefox. I have for a long time switched between safari and Firefox mostly because I love the way safari looks. The native look and feel of safari is fantastic! Not to mention safari is a pretty darn speedy. Firefox has come a long way, the original version looked awful and was very clunky. I would have given up on it if it wasn't for the amazing plugins. Version 3 is just as functional as ever and finally feels like it belongs on the mac. The history looks great and is search able too.

Way to go Mozilla!

Interviewing

noreply@blogger.com (Shawn) — Fri, 13 Jun 2008 01:52:00 +0000

Interviewing is hard work. Its hard to be the interviewee and its hard to be the interviewer. Fortunately I haven't had to be on the interviewer side of the table in quite some time. I'm a fan of Joel Spolsky who writes a fantastic tech blog, his suggestion is that you should look for people who are smart and get things done. That is it. Simple. When I'm on a interview committee that is my Mantra. The problem I have seen lately is that others on the committee have very defined set of things they are looking for that blurs their judgment of the canidate at hand. I believe that at the end of the day you can teach someone to use a tool or set of tools but you can not teach them to think in a different way. Personality traits are hard if not impossible to change and further more we should not want to change a person to fit our job and environment.

Today we did a phone screen for two candidates. Both were technical savvy however one had a lot more programming experience then the other. The one with programming experience certainly could do the technical aspect of the job however the canidate had a hard time communicate at a level that would engage a functional user. This to me is a red flag. We are not going to be able to change this candidates communication style, they have operated this why for years. Trying to shape away years of learned behavior is almost impossible. The other canidate was an effective communicate that understood process and methodology. This canidate was not a strong programmer. In my opinion we can teach programming and we can teach the tool. The good news is we are starting with a clean slate!

My goal when looking at some one to hire is to find someone smart that gets things done. This is the corner stone! After we narrow down the candidates I'm looking for some one that fits the job, the organization and has the innate skills that can't be trained.

Scripting SCP

noreply@blogger.com (Shawn) — Sun, 08 Jun 2008 19:23:00 +0000

I love scp, its easily one of the most used unix commands in my tool bag. That said it is a real pain inf the butt to try and script scp. I was in the midst of writing a little script that would create ssh key pairs and distribute them to a number of servers. On paper this seemed to be almost trivial and that my friends is the problem wit paper. In order to make scp more secure the developers had it communicate using PAM instead stdin, sterr and stdout. Bummer, This meant there was no easy way to supply scp with the users password. So there is no command line option to supply the password and I can't write to it over stidn, I was a little stuck. After searching that Internet, and I mean really scouring, I found a command called expect. Using this I wrote a quick and dirty shell script to do the scp part:


#!/usr/bin/env expect -f 
set password [lindex $argv 0 ]
set file     [lindex $argv 1 ]
set command  [lindex $argv 2 ]

# trick to pass in command-line args to spawn
eval spawn scp -r $file $command

expect "password: $"
send "$password\n"

# wait for regular shell prompt before quitting
# probably a better way using 'wait'
expect "$ $"

With this the whole scripting experience became trivial again, rock on!