Brandon McCaig's Blog

"-//CS//Lyrics 1.0//EN" AKA Lyrics DTD/XML/XSL/XHTML/CSS

2011-09-01T20:56:00.023-04:00

I know I hinted towards a technical blog post last time. I expected my next post to be about Java or Perl Web programming. Well, that one is coming, I suppose. I am making ground and eventually would like to write about it. I don't think I'm quite satisfied with my progress yet though (nor am I so dissatisfied that I need plea for help yet). This post is somewhat technical, but it's also somewhat personal.

I love to sing. I sing all the time. In the shower, in my car, at the computer, etc. Everywhere, basically. I love music and singing is one of the things that I like most about it. Or perhaps, more accurately, the lyrics are. I love lyrics. For me to like a song the lyrics had better be clever. You get bonus points if they're emotional or humorous and I can relate to them. I guess then it shouldn't be surprising that I also love to rap.

Indeed, it wouldn't be an exceptional day for somebody to notice me driving by them in Sault Ste. Marie with my stereo cranked, singing or rapping along. Whenever I hear a new song that I like I'll look up the lyrics online (in my younger days I would do it from the CD cover, but a wide screen monitor is a lot bigger and easier to read from in the dark). I'll then practice reciting them while listening to the song(s) on repeat. This could mean hearing a new or leaked song on YouTube and just starting the same video over and over again (shout out to YouTube Repeat!!). More often, though, it means that I just bought a new CD from one of my many preferred artists and am listening to it on repeat.

The Amarok media player is great for this because it supports a lyrics module that allows you to view the lyrics for the current song right there in the media player automatically! It's great for people like me that enjoy analyzing and learning the lyrics of a song, or even just singing them after 6 or 8 months have gone by and they aren't fresh in your mind anymore.

Often I find that the various lyrics sites online don't have 100% correct lyrics. Some people do a really bad job of transcribing them. I don't know if it's laziness or incompetence, but I personally find it really annoying when I'm trying to sing or rap along to a new song and I'm stumbling over the wrong lyrics on a page.

It seems that most user maintained lyrics sites online make it difficult to contribute though. You generally have to register with some random site, which means a whole new set of credentials to remember, and a whole new server with your personal information just waiting to be hacked and compromised. They also don't do a very good job of describing the lyrics or presenting them to the user. There's usually a mix of slang and unslang and it's never really clear when the transcriber is sure of a particular word or just making an [un]educated guess. You may be surprised just how often a word or two is a mystery in very popular songs.

This combined with my thirst for XML knowledge lead me to develop a simple XML document type for lyrics. It's a work in progress so nothing is feature-frozen yet. Currently I have a document type definition (DTD) to validate against, an XSL stylesheet to transform the lyric data into standards compliant XHTML 1.0 Strict, and a CSS stylesheet to accompany the XHTML output to prettify the output a bit (~~no guarantees that the CSS is standards compliant, unfortunately~~ and the W3C's validator seems to think it is valid CSS 2.1 also!).

Currently I only have two songs transcribed. The first is We As Americans, by Eminem, from the Encore bonus CD. That was basically my initial test drive, if you will, for the format and stylesheets.

Two evenings ago I got to thinking about one of my favorite bands, Eisley, and wondering if they had released any new albums since I last checked. It turns out that indeed they had! The Valley was released back in March. I made it a plan to stop by the music store yesterday to look for it. Now Eisley is somehow still somewhat of a below-the-radar band, which is puzzling because they're so amazingly talented and creative and beautiful. In any case, a side effect of this is that music stores often don't stock them (some of them don't even have a listing for them!). I could rant all day about music stores and the music industry and how stupid things have gotten (see my previous post about the movie industry for a hint of what to expect), but I'll try to avoid doing that now and save it for another post.

Long story short, I went out to buy one CD and came home with seven...and none of them was the one that I went out for! "But, Brandon," I hear you wondering. "What about Eisley?"

Oh, I ordered it. Don't you worry. As expected, the music store didn't have it in stock. Unfortunately, it cost me $20 to order through the store. The night before I had seen that the online retailer linked from the band's Web site was only asking $10. Without knowing the shipping fees though it's hard to say if I'm any worse off. Not that I'd mind buying an Eisley CD for $20. In fact, I think I probably paid $20 for each of Room Noises and Combinations too. What bothers me is that Eisley will only see a tiny percentage of that, if any, and the crappy music stores and record companies that I'm avoiding ranting about will take most of the profit.

Moving on, the CDs that I actually came home with were as follows in no particular order: Theory of a Deadman (Theory of a Deadman), The Truth Is... (Theory of a Deadman), Taylor Swift (Taylor Swift), Greatest Hits (Guns N' Roses), 22 More Hits (George Strait), American Saturday Night (Brad Paisley), and This Is Country Music (Brad Paisley). I was also trying to buy or order The Spirit Room (Michelle Branch), but not only did the music store (the only one left in my city to my knowledge) not have any Michelle Branch CDs in stock, but her CDs weren't even available from their distributor peoples! Ugh.

So anyway, having bought all of these CDs, I then had to rip them into my computer so I could sync them with my iPod (I could also rant about how much Apple sucks, but there's a different time and post for that too). In doing so, I found myself listening to old and new music and singing along and by the time 2 AM rolled around I wasn't really tired. In fact, I couldn't seem to stop singing or rapping. I tried going to bed, but eventually just started to rap the lyrics to one of the songs I had written back in high school, and then decided that I should transcribe it and publish the lyrics on my site. That way I'd have a digital copy (it was quite a challenge to remember the words after so many years), since I had destroyed, lost, or abandoned all others copies. I eventually gave in and decided to do this immediately while it was fresh in my mind (both to do it and most of the lyrics).

So the second song I have transcribed is actually my own! It's called Wanna Be God. I separated it into a separate file to initially keep it off the radar and also to keep it separate from "real" songs. :) There are a few other songs that I wrote back in high school. One or two of them I might feel are worth publishing online also. We'll see if I ever get around to that.

In the meantime, please check out my XML document type, and accompanying DTD, XSLT, and CSS. And, of course, check out the lyrics to both songs:

http://lyrics.castopulence.org/
- http://lyrics.castopulence.org/lyrics.xml (This is the core document that I'll currently put all lyrics in, though eventually I might split them each into their own file)
- http://lyrics.castopulence.org/lyrics.xsl
- http://lyrics.castopulence.org/lyrics.dtd (https://www.castopulence.org/2011/09/lyrics)
- http://lyrics.castopulence.org/lyrics.xhtml
- http://lyrics.castopulence.org/lyrics.html
- http://lyrics.castopulence.org/lyrics.css
http://lyrics.castopulence.org/wanna_be_god

Eventually I'd like to implement a server-side backend to manage them (once I get either a Java or Perl Web environment setup to my satisfaction, of course ;), at which point I could potentially open the system up to open ID users and let the world help me to fix lyrics. I'm not yet certain where I stand with regards to copyright infringement though. Technically I am violating Eminem's (or his record company's) copyright by publishing the lyrics to his song without permission. I do have a disclaimer on the page to highlight the fact that I don't own the copyright... It seems harmless to me, but nevertheless, there is always the possibility that somebody will feel they are losing money as a result and try to convince a judge to make me reimburse them... Hopefully that doesn't happen... Honestly, I can't even figure out how I would contact Eminem or Interscope to ask for permission. Their Web sites are terrible. :P

If you are or represent a copyright owner and wish for me to have a work removed from my site then please kindly address an E-mail to me (bamccaig at castopulence dot org) detailing who you are and what relationship you have to the copyrighted work, and the copyright owner's wishes, and I will promptly add any disclaimers you want or, if necessary, lock that work up behind a secured connection for personal use only.

What Happened To The Film Industry?

2011-08-24T01:18:00.002-04:00

I've had this tab open for a week or more because I've wanted to write another technical blog post about something, but a topic to write about just never really came to mind. [Many more days later I now realize that my next technical blog post should be about my attempts to learn Java and Perl Web programming and my struggles with that.] So instead, I'm going to rant about something non-technical: movies.

I'm sort of out of the loop on movies lately. I've only seen a small fraction of new releases in the past few years. Which is a shame because I do rather enjoy movies. I also enjoy criticizing them.

On Wednesday I decided to rent some movies out of the blue. There was only really one that I had in mind when I went to the video store: The Adjustment Bureau. I wasn't sure exactly what it was about, but I knew that Matt Damon stars in it and his movies are usually pretty good (e.g., the Bourne series), so that was enough for me to invest in it. But here I find my first critisicm of the movie industry as of late: either they fail horribly to advertise in this Internet age or else there's actually very little good to advertise.

I think it's actually a lot of both. I do watch TV relatively often, but I rarely see movie previews anymore, and almost never see good ones. Online, where I spend the vast majority of my time, I never see movie ads. In fairness, that could be attributed to the AdBlock Plus Firefox extension, which does an AWESOME job of hiding pretty much every ad on the Web. Especially ads that don't really belong on a particular page (i.e., third party ads). However, I don't actually mind clever, non-intrusive advertisements. In fact, I rather enjoy them. Still, I block all advertisements on the Web because every schmuck and his grandmother have their Web sites filled with intrusive and annoying ads about completely retarded "products" and "services" that nobody in their right mind would invest in. That, or the ads are incredibly intrusive and slow you down. So I just filter them out of my Web, see none of them ever, forget all about their existance, and guarantee that basically no site will ever get any ad revenue from me.

So I'm left wondering, do the movies made today suck or am I just not getting the previews for the good ones due to ad blocking? I can't even really remember watching any truly good movies in the past few YEARS. Perhaps I'm just spoiled by video games and as a consumer of art have become superior to mere movie audiences, expecting more? Perhaps not. Still, it seems that all of the movies that I watch nowadays just suck. The characters are lame and unintersting. The story doesn't really make any fucking sense at all. The action sequences defy all understanding of the laws of physics and logic, both in the real world and the movie universe too. Basically the entire movie is either predictable, ridciulous, or both.

On this particular visit to the video store I did what I usually do: I walked from one end of the store to the other, browsing all of the new releases to find which ones seemed most interesting. Which in the past few years has actually turned into interesting at all. For the most part, I wasn't really excited about anything other than The Adjustment Bureau. In fact, had I been interested only in getting good movies, I probably would have only even rented The Adjustment Bureau, since it's the only one that actually looked good to me. For some reason though I felt like getting movies that I thought would probably suck, hoping to be proven wrong and perhaps restore some faith in the movie industry. The movies I rented were as follows (in the approximate order that I found them in the video store): For Colored Girls, Battle: Los Angeles, Drive Crazy, Just Go With It, Gun, and The Adjustment Bureau. I was tempted to also get Source Code (being a computer programmer those words have a special meaning to me), but from what I could recall from previews and what I could gather from the back of the case it didn't seem even related to computer programming so I expected I would be disappointed. [I happened to see a preview shortly after that made me think I might enjoy it, but nevertheless I didn't think I would when I rented these movies.]

I'll now discuss the movies in the order that I watched them. I watched the movies in what I predicted to be worst-to-best order, with the exception of Gun and For Colored Girls since at random and on the spot I had rented them for an extended period of time to avoid late fees. I'll try to flag spoilers, but no promises.

Day One

I couldn't really remember any previews for Battle: Los Angeles in the video store, but I had a feeling that it would suck. Nevertheless, it was a Sci-Fi/"war" movie (both things that I like under the right circumstances) and it was one of the movies that the video store has a ton of copies of so it was obviously supposed to be a popular title (of course, popular certainly doesn't mean good, and often means quite the contrary). Again, against my better judgement, I decided to get it anyway. It was indeed terrible. Primarily because of lazy writing (perhaps Hollywood fired all of the talented writers?). (SPOILER ALERT; skip to the next paragraph if you care) For example, the aliens that attacked the Earth came from space inside of what were identifed as meteorites, IIRC. The American forces mobilized shortly after realizing the potential threat, but naively identified that the invading forces were grounded, assuming that they had no air forces and that the air would belong to the humans. Of course, it doesn't take long for advanced enemy aircraft to appear, wreaking havoc on the U.S. forces. That's just a very simple example. I could write an entire blog post about all of the things wrong with this movie if I cared. Maybe even turn it into a series of posts. I digress... I did like some of the characters (familiar actors and actresses helped with that), but ultimately it was just a disappointment.

Day Two

Next I watched Drive Angry. I had low expectations for this one as well. Again, I couldn't remember any previews, at least not in detail, but for some reason the movie reminded me of Ghost Rider, another Nicolas Cage movie that I found terribly disappointing. IIRC, Ghost Rider had a certain "cheap" quality to it, and I got the same first impression from Drive Angry. Again, I rented it just in case I was wrong, but unfortunately I was not. The movie made basically no sense, and while it was clear from the special features that the crew enjoyed making the movie (which is all well and good for them), some even citing it as some of their best work, I personally thought that it sucked horribly. I'm not criticizing the actors or actresses so much as the overall story and script. (SPOILER ALERT; skip to the next paragraph if you care) For example, Cage's character, who is an escapee from Hell (the fiery one from Christian fiction, which in this movie is portrayed as a more traditional prison with a supernatural flare), walks into a church to rescue his infant granddaughter from a Satanic cult planning to murder her as a sacrifice for Satan. His granddaughter is laying down in a basket or something at the front of the church where a church leader would normally be preaching from, but foolishly Cage's character just walks in silently as if to avoid disturbing the service, and eventually gets disarmed when the entire church basically pulls a gun on him. Sure, he's "undead", so practically speaking he can't really be hurt, but his granddaughter is not so undead so I consider it rather risky to allow himself to be so easily captured. Personally, it frustrates the Hell out of me whenever a protagonist in a movie makes a foolish mistake like that. It's like they aren't even trying to ensure success. Again, there are many more examples of stupidity that I could cite from this movie. That wasn't even a very good example, but it has been a few days since I saw the movie and it wasn't all that memorable, so that's the best that I've got. Even the characters seemed ridiculous in this movie. Their actions do not make sense. During the special features it was explained that they didn't really go through a whole lot of revisions with this film's script. They basically decided to do a movie of this style, wrote up a quick draft in a couple of weeks, and submitted the first version. Apparently it was praised for it's originality (it wasn't original at all) and creativity. I guess people thought it had a pure, untainted artistic quality to it. I disagree. It didn't fit together well. It lacked polish. It wasn't well thought out, or perhaps thought out at all. I guess they figured they could subsitute an interesting script with guns, tits, and cars. While these three things are all good, they alone don't make for an interesting movie.

Next I watched Just Go For It, an Adam Sandler film. Adam Sandler is awesome and I like pretty much all of his movies. For some reason though I didn't have high hopes for this one. Perhaps because I had already been let down by Grown Ups. Perhaps I had seen previews that tipped me off. As usual, I was right. This movie was pretty terrible. Let me also say that I love Jennifer Aniston: Friends is one of my favorite all time shows (I have the entire series on DVD). I think in just about every movie I've seen Anison in though (at least since Friends ended) I've been disappointed. Her acting seems blatantly poor. To give her the benefit of the doubt, I choose to believe that it's because she played Rachel Green for so long that it became a part of who she is, making it difficult to portray other characters. Just Go With It lacked scructure and polish. Watching the film I got the distinct feeling that the movie was designed to incorporate improvisation by the actors and actresses into their roles. Given the title and Adam Sandler's roots, this wasn't really a surprise. However, it was clear that many of the cast members were terrible at improv, and as a result I think the performances suffered. I noted a few moments where Jennifer Aniston appeared to screw up her lines. I guess since the movie was somewhat improvised they decided not to bother with too many (any?) retakes, but in my opinion this hurt the quality of the film. Perhaps worst of all was the improv of Nick Swardson, which often seemed to be of a somewhat vulgar nature (which seemed out of character for the movie). I remember a particular scene where the two young children were present that particularly struck the wrong nerve with me; he even tried to involve them in the dialogue when the adults withdrew. I also got the distinct impression that his "improv" on numerous occasions bordered on sexual harrassment towards the female cast members, and in particular Jennifer Aniston. While I can commend them for doing something different and risky, especially so for those that were clearly outside of their comfort zone with the improv, I have to criticize the film for poor performances. Adam Sandler did a decent job of it, which is no surprise, but even he seemed to struggle with it at times. I might have been more open to the idea if it was made blatantly clear that it was improvized so I wasn't blindsided by it. All of that out of the way, I did like parts of the movie, and enjoyed some of the characters. (SPOILER ALERT; skip to the next paragraph if you care) I liked the way Sandler's and Aniston's characters fell for each other in the end; that sort of made up for the amateur improv in the end.

Day Three

I finally got around to watching The Adjustment Bureau. I loved it. I was a little bit afraid of it going in a direction that I wouldn't like, but instead it remained interesting and gripping and well put together. There were a few unexpected twists that I really enjoyed. I loved the characters and I loved the story. It was actually so emotionally jarring (?) that I was hurting myself with the rate at which I was drinking bottled water (the water being a substitute for beer). I love to feel emotions through art, especially strong ones, and this movie delivered for me. I really enjoyed Matt Damon's and Emily Blunt's characters. (SPOILER ALERT; skip to the next paragraph if you care) I initially expected the Adjustment Bureau to be a purely political force. Something relating to corrupt government and potentially secret societies and things of that nature. This was interesting to a point, but it would have ultimately frustrated and bored me. I found the change of direction that the movie takes unexpected and very welcome. It was timed very well too. The "twists and turns" were well executed. I really liked the unique and original take on the creator concept (I am, however, an atheist). It was an interesting concept the way the Adjustment Bureau was run formally and like a company hierarchy. I particularly liked how it was pretty much completely independent of the religious beliefs that I imagine inpired it. I thought it was just brilliant.

Day Five (?)

I hadn't heard of For Colored Girls, but it seemed like a movie that I would be interested in so I read the synopsis on the back and decided to get it just in case I'd like it. I watched it next. It wasn't at all what I expected. I have mixed feelings about it. I didn't really enjoy it, but perhaps that was the point. I will say this though: it's a Tyler Perry movie. I wouldn't even know who that is if it weren't for one thing: I have a brother that loves the "Madea" movies. I personally think they're highly overrated and as a result feel the same way about Tyler Perry, since he seems to be the mind behind them all. In a way, I guess For Colored Girls was more or less just another Madea movie without Madea. In fact, some of the cast seemed to be the same, and though I never pay much attention when I get pressured into watching a Madea movie, I think some of the shared cast members even played a similar role. The title sounded to me like it was supposed to send a positive message, but I didn't really get one at all from this movie. I will say this though: it was nice to see some actresses that I haven't seen in a while, such as Whoopi Goldberg and Phylicia Rashād.

Day Seven (?)

Gun is a 50 Cent movie, so that's pretty much the only reason that I rented it. I watched it last for no good reason at all. It was undoubtedly the worst movie of all six, and with this selection that is saying something. It looked pretty low budget (indeed, Wikipedia says $10 million) so I wasn't really expecting much. Nevertheless, I like a lot of 50 Cent's music and I liked the Get Rich Or Die Tryin' movie. I suspect that Before I Self Destruct (the movie) was also low budget so I expected something on par with that, which itself wasn't very good. Indeed, Gun had a similar low-quality feel to it. I think probably what I found the worst was the dialogue, which didn't really feel natural, and even seemed improvised at times. I don't know if the script was just poorly written or if it was intentionally left with holes in it for the actors to fill, but next time I think I'd rather see a fully thought-out script. :-X (SPOILER ALERT; ... but you probably shouldn't care here) The movie reveals that Kilmer's character is a rat sometime near the middle of the movie, but towards the end I actually expected a twist where it turned out that he had been loyal and another character had been the rat. Instead, the movie remained blatantly predictable with Kilmer's character actually being the rat. Perhaps the worst part was Jackson's character not figuring this out. It's painful to watch most of the dialogue in the film, but in particular there is a moment where Jackson's character invites Kilmer's character to a warehouse to talk in person, and then invites him to take a walk through the warehouse, etc. The dialogue is very hestitant and awkward here, and you're almost sure that Jackson's character MUST know that Kilmer's character is the rat. Indeed, he begins to question him about it, but then he foolishly just takes Kilmer's character's awkward word for it and instead makes him his "number one" (I forget exactly how they worded it in the movie, but the idea was that he was basically considered the most trusted). It's quite literally painful to watch how bad it is...

Summary

Overall, I rented 6 movies, and 5 of them were pretty much terrible. The Adjustment Bureau was awesome! There were a few redeeming qualities in Just Go For It, but not enough to make it enjoyable. The other 4 just completely sucked. Perhaps the reason I haven't been watching many movies is because most of Hollywood stopped even trying to make decent movies anymore and instead relies on marketing and pop culture to make money...

UNglobal State Programming Environment

2011-06-09T16:58:00.001-04:00

A year or two ago I stumbled across a very useful Google Tech Talk on YouTube about global state and Singletons and why they're bad:

It makes a lot of sense and after letting it sink in and experimenting with dependency injection myself I can say that it results in much better designs, whether or not you're writing tests (the video is Java-centric and test-centric, but neither is necessary for it to apply). Thinking about it though popular programming environments do impose a lot of global state that is in my experience regularly accessed without a second thought. When you collaborate with other people it's hard to make them understand how global state and Singletons are bad when to them it's the easy and fast way to get their job done. It's my opinion that a large part of the problem is that programmers are just used to global state from their programming environments. What we need is for languages to be updated and newer languages to be done right in the first place. No more global state in the programming environment. Pass that state into the program's entry point. For example:

// C
int main(const system_t * const system, const int argc, const char * argv[])
{
    fprintf(system->stderr, "This was a triumph.\n");

    return 0;
}

// C++
int main(const System & system, const int argc, const char * const argv[])
{
    system.cerr << "This was a triumph." << std::endl;

    return 0;
}

// C#
private static int Main(System system, string[] args)
{
    system.Error.WriteLine("This was a triumph.");

    return 0;
}

/* Java (albeit, while they're at it, main should return an int in Java too) */
public static void main(System system, String [] args)
{
    system.err.println("This was a triumph.");

    system.exit(0);
}

Etc. Wouldn't that be a wonderful platform to develop on? I certainly think so.

Kernel-space Politics

2011-05-23T16:46:00.003-04:00

I thought of an interesting analogy the other day during an IRC conversation that I thought I'd share with you. :) It was probably almost a week ago (Blogger was down for maintenance at the time) and I forget exactly what the discussion was about, but more or less just how the members of government exploit the people for their own personal gain. In defense of privatizing as much as possible, I argued that the government is a little bit like the kernel of an operating system. If you want the system to be secure and reliable then you need to move as much as possible into userland (i.e., outside of the kernel). This way the individual processes (services, businesses, etc.) can attempt to do wrong, but ultimately the authority (kernel AKA government) has the power to stop them. On the other hand, if these processes are occurring directly within the kernel then they have unrestricted access to the entire system and can do whatever they want, even if they happen to corrupt or crash the system in doing so.

Then again, with the world's major currencies under private banking monopoly control you have to ask yourself whether the government is actually the kernel or only the system drivers. :-X The government can only hope to keep the system secure and reliable if it is actually ring 0 and free of corruption itself.

.NET P/Invoke With Managed Callbacks

2011-05-17T09:44:00.004-04:00

Update: See append at the bottom for an update.
Update2: Open sourced the bindings. See append2 at the bottom for an update.

I've finally thought of another topic to write about. Or, rather, the topic fell on my desk. Again. It is about using managed methods as callbacks in native code. I first learned about this months (a year?) ago when I was working on a ZBar.Processor proxy class for zbar-sharp (.NET bindings for the zbar bar code scanning library). This was my first real experience using P/Invoke.

The zbar_processor is effectively a high-level object to scan frames from a Web cam for bar codes and return the encoded data back to the calling application. The data is returned via a callback function. This seemed simple enough once I figured out how P/Invoke basically is done. I just wrote up a compatible method and passed it into the zbar API. While it did seem to work, the native library would eventually throw memory access violations.

It turns out that the problem was me (SHOCK). Or rather, my code. Or rather, missing code. See in .NET the equivalent of a function pointer is basically a delegate instance. This is effectively a method object (I imagine it to be similar to a functor in C++). In any case, calling the delegate instance is equivalent to calling the method. Even the this reference is preserved.

#digress Delegates In C#

A delegate type is declared like so:

namespace Application
{
    class Example
    {
        public delegate void foo(int bar);
    }
}

What this does is basically declare a delegate type named "foo" with no return value and a single int parameter. You can then create an instance of the delegate like so:

foo f = new foo(MyMethod);

Where MyMethod is a static or non-static method that matches the signature. The instantiation can also be implicit:

foo f = MyMethod;

For a more complete example:

using System;

namespace Application
{
    class Program
    {
        static void Main()
        {
            new Program().Run();
        }

        public void Run()
        {
            // Using a method, explicit instantiation.
            Example.foo f1 = new Example.foo(this.Foo);

            // Using a method, implicit instantiation.
            Example.foo f2 = this.Foo;

            // Using a lambda, explicit instantiation.
            Example.foo f3 = new Example.foo(
                    (int bar) => Console.WriteLine(bar));

            // Using a lambda, implicit instantiation.
            Example.foo f4 = (int bar) => Console.WriteLine(bar);

            f1(1);
            f2(2);
            f3(3);
            f4(4);
        }

        void Foo(int bar)
        {
            Console.WriteLine(bar);
        }
    }

    class Example
    {
        public delegate void foo(int bar);
    }
}

#enddigress

As I was saying before I so rudely interrupted me, I was missing a tiny little detail. In order to use one of these delegates as a callback in native code you'd first have to instantiate one. OK, first we actually need a native method to call, and a callback type.

namespace Application
{
    class Example
    {
        public class NativeMethods
        {
            public delegate void zbar_image_data_handler_t(
                    IntPtr image,
                    IntPtr userdata);

            [DllImport("libzbar0")]
            public static extern IntPtr zbar_processor_set_data_handler(
                    IntPtr processor,
                    zbar_image_data_handler_t handler,
                    IntPtr userdata);
        }
    }
}

Now in order to use it we simply do as above:

using System;

namespace Application
{
    class Program
    {
        static void Main()
        {
            new Program().Run();
        }

        public void Run()
        {
            IntPtr processor = IntPtr.Zero;

            /*
             * Use your imagination here. We would have had to call native
             * methods to create and initialize a zbar_processor struct.
             */
            
            var handler =
                    new Example.NativeMethods.zbar_image_data_handler_t(
                    this.ZBarProcessorImageDataHandler);

            Example.NativeMethods.zbar_processor_set_data_handler(
                    processor,
                    handler,
                    null);
        }

        void ZBarProcessorImageDataHandler(IntPtr image, IntPtr userdata)
        {
            Console.WriteLine("Bar code successfully scanned!");
        }
    }
}

Pretty straightforward, really, but there is a problem! Can you see it?! Not likely because it isn't even in the code. Remember that .NET is managed and garbage collected and delegates are basically objects. What this means is that when the garbage collector finds a delegate that is no longer referenced by anything it will eventually garbage collect it. This was something that certainly hadn't occurred to me when I was writing this code. It wasn't until I sought help online that somebody (it was many months ago, but I'm guessing in #csharp on freenode) pointed out that the delegate object could be destroyed by the time the zbar library attempted to call it. So what is the solution? We need to keep a reference to it around so it isn't destroyed. We can do that by simply creating a field to hold it.

...
    class Program
    {
        Example.NativeMethods.zbar_image_data_handler_t
                imageDataHandler_;

        ...

        public void Run()
        {
            IntPtr processor = IntPtr.Zero;

            /*
             * Use your imagination here. We would have had to call native
             * methods to create and initialize a zbar_processor struct.
             */
            
            this.imageDataHandler_ =
                    new Example.NativeMethods.zbar_image_data_handler_t(
                    this.ZBarProcessorImageDataHandler);

            Example.NativeMethods.zbar_processor_set_data_handler(
                    processor,
                    this.imageDataHandler_,
                    null);
        }

        ...
    }
...

At this point, after fixing a few other subtle bugs, the program ran fine without access violations! Unfortunately, the zbar_processor (at least, at the time) wasn't very reliable and we ultimately ended up implementing the Web cam interface ourselves and passing the scanned frames to zbar.

In any case, I've recently been writing .NET bindings for wkhtmltox, another native library. This one is for converting HTML/CSS Web pages to PDF documents. It works really well because it's based on the WebKit framework (hence, the "wk" in the name). So I forgot all about the lifetime of the callback delegates and ran into the same problem. Today I finally remembered and looked back at the zbar bindings to see what was wrong.

I've now fixed that by adding delegate fields to the appropriate proxy class, but unfortunately I'm STILL getting access violation exceptions. It's probably going to be some other subtle bug as a result of .NET speaking to native code. I haven't figured it out yet though.

If you have any ideas about what else would cause memory access violations in P/Invoke programs or have any questions about what I've shown then please leave a comment. :)

Append:

Seems there is more to the story than I thought. See the following thread for details:

http://www.gamedev.net/topic/270670-pinvoke-callback-access-violation/

The gist of it is that there is a System.Diagnostics.InteropServices.UnmanagedFunctionPointerAttribute attribute that can be used to mark up your callback delegates with P/Invoke attributes (for example, calling convention). I'm not sure why, but specifying CallingConvention.CDecl for my callback delegates seems to postpone or avoid the access violations. I don't really understand why because the functions of the library are being called with the default calling convention, which I think is CallingConvention.StdCall. Attempting to specify CallingConvention.CDecl for the native functions immediately causes the stack to become imbalanced/corrupted (according to the .NET runtime; I think I trust it in this case ;). It doesn't really make sense to me that the callbacks would use a different calling convention so I suspect that the different calling convention for the callbacks just coincidentally changes the way the memory is accessed to avoid violations... I don't know. I'm lost. :(

Append2:

I've turned the .NET bindings for wkhtmltox (wkhtmltopdf only) into an open source class library. :) You can find the code at its GitHub repository. It doesn't work properly yet (in fact, I've given up on it for now and am working on invoking the wkhtmltopdf.exe process instead), but I see no technical reason why it can't so I (or you) just need to figure out what's currently wrong. :)

Fedora Package Contributor for Allegro

2011-04-15T14:33:00.002-04:00

It's been a long time since I've posted anything on here and I guess it's long overdue. I've either lacked a good topic to type about or was just too drained to do so. Well no longer!

I figured I would take a moment to announce that I've become a package contributor for the Fedora project. What does that mean? It just means that I'm currently contributing to the development or maintenance of one or more of the RPM packages for the project. In particular, I'm maintaining the Allegro 5 packages. What this means is that I took the Allegro 5 source code and turned it into RPM packages. The main repository that I'm using to track Allegro 5 package development is available on GitHub: http://github.com/bamccaig/allegro5-rpm. Don't confuse me though with the actual Allegro developers who deserve the real praise for the Allegro library!

There are plans for me to help out with or take over the Allegro 4 packages as well (which I think will include games and other related projects). I've just recently gotten Allegro 5 adopted into Fedora 13 and 14 ~~(15 AKA Alpha is pending)~~ and 15. AFAIK, Fedora is the first distribution to get Allegro 5 packages! So yay for that! I'll now be working on updating the Allegro 4 packages (allegro) to the latest stable version, 4.4.x.y.

I guess that makes me an "official" contributor to Fedora. So that's pretty exciting. I think so, at least. What are you waiting for? Go put my packages in action! You can learn how to here if you don't already know.

Since there isn't much else to say about that, I'll take a few moments to discuss a few of the things I learned about working with RPMs. RPMs are based on a .spec file that basically just describes the package in a text file: name, description, version, release, where to get the source code, the software license, build instructions, which files to install, what their permissions should be, etc. To turn a .spec file into an SRPM[1] or RPM you use the rpmbuild tool. Once installed you can easily query the package system for information about a package using rpm or yum. For example:

$ # Lets query the basic information about the core allegro5 package.
$ yum info allegro5
*snip*
Name        : allegro5
Arch        : i686
Version     : 5.0.0
Release     : 3.fc13
Size        : 994 k
Repo        : installed
From repo   : updates
Summary     : A game programming library
URL         : http://liballeg.org/
License     : zlib
Description : Allegro is a cross-platform library intended for use in computer games
            : and other types of multimedia programming. Allegro 5 is the latest major
            : revision of the library, designed to take advantage of modern hardware
            : (e.g. hardware acceleration using 3D cards) and operating systems.
            : Although it is not backwards compatible with earlier versions, it still
            : occupies the same niche and retains a familiar style.

$ # Lets query the changelog of the core allegro5 package.
$ rpm -q --changelog allegro5
* Wed Mar 09 2011 Brandon McCaig  5.0.0-3
- Adding file permissions to subpackages.
- Moving devel files (namely .so symlinks) to devel packages.
- Added %doc section proper; readmes, changes, license, etc.
- Fixed SF.net URI.
- Modified BuildRequires.
- Added main devel dependency to subpackage devels.
- Replaced many al_*.3* manpage files with a glob.
- Replaced many header files with directory and %exclude macros.
- Added allegro5.cfg file under /etc/allegro5rc.

* Fri Mar 04 2011 Brandon McCaig  5.0.0-2
- Merged primitives addon packages into core packages.
- Merged memfile addon packages into core packages.
- Merged "main" addon packages into core packages.
- Merged font packages into core packages.
- Merged color packages into core packages.
- Merged doc package into the devel package.
- Fixed spelling mistakes.
- Removed explicit library dependencies.

* Fri Feb 25 2011 Brandon McCaig  5.0.0-1
- Initial version.

$

I was actually quite happy to learn just how easy it is to create RPMs that work. RPM seems to be very smart and figures out a lot of the dependencies itself. Contributing simple packages really isn't very difficult if you already know a thing or two about how to build them from source (programming experience helps too). So if you're a programmer that uses Fedora (or any other Linux distribution) and wants to help out by becoming a packager then I suggest you look into it. It's not as difficult as you might think.

Thanks to my sponsor[2] for guiding me through the process and helping me out when I got stuck.

References

[1] An SRPM is a source RPM. It basically is just an RPM that contains the .spec file, source tarball, and any patches that might be bundled. The SRPM is an intermediary stage between the source code and .spec and the fully-fledged RPM.
[2] I'll refrain from naming him without his permission to do so, just in case, but you can probably find out who it is easily enough by checking out the fedora bugzilla system if you care. :) He knows who he is. :)

SQshelL - Building and Installing sqsh In Linux (Fedora 13) Without The Sybase CT-Library

2010-07-06T16:31:00.024-04:00

If you're like me then you generally prefer fast, efficient command-line interfaces to expensive (CPU- and RAM-wise) and slow GUI interfaces. Recently, my Windows 7 (Seven) box got really slow, and in particular Visual Studio and SQL Server began to crawl. I couldn't really explain it because the Windows Task Manager said that the System Idle Process was using all of the CPU and the system was only using about half of my 3 GB of RAM, but nevertheless, scrolling a single line of code took seconds! Something had to be done.

I had finally had enough and insisted to j0rb that I was going to install Linux. Now, I currently still require Windows because most of our projects are .NET-based (and bleeding edge, .NET 4.0 stuff), and it seems Mono just isn't quite there yet. I was very pleased to hear that they do support many (or most?) .NET 4.0 features, but apparently not LINQ to SQL, which we make extensive use of. To solve this contradiction, I would install Windows in a virtual machine. I ended up installing Fedora 13 (x86) as the host, with Windows 7 (Seven) Ultimate in a VirtualBox guest. I was quite nervous about whether or not it would actually perform adequately (especially considering how poorly Visual Studio and SQL Server were performing on the raw metal), but so far so good.

On to the topic at hand: building and installing sqsh in Fedora 13. sqsh is basically a shell interface for Sybase ASE (I guess?) or SQL Server. It allows you to do powerful things that you expect from your system shell, like redirecting streams, and environment variables, etc.; as well as interact with a database in SQL. I'm not overly familiar yet with what it can or can't do (I just heard about it and went ahead and installed it), but my hopes are high. Hopefully it will at least reduce the need for me to wait for SQL Server to startup and close, or ugh connect.

Naturally, when I first heard of it I instinctively asked YUM for it. Sadly, I was disappointed to learn that it is not in the default repositories. "No matter! I will just install from source," I told myself.

The first step to installing from source is obviously to get the source. So head on over to SourceForge, where it's currently being hosted, and download the latest version (I'm using 2.1.7 at the time of writing).

http://sourceforge.net/projects/sqsh/

Assuming your results are similar to mine, you should now have a sqsh-<version>.tar.gz file somewhere on your file system. Firefox downloaded it straight to my Downloads folder (dl for short). I keep all source trees in ~/src for organization, so lets go ahead and extract it to there now (you can extract it where ever you like).

[bamccaig@krypton ~]$ tar -xzf dl/sqsh-2.1.7.tar.gz -C src
[bamccaig@krypton ~]$ cd src/sqsh-2.1.7
[bamccaig@krypton sqsh-2.1.7]$

Great. So far so good. Now lets take a look for "readme" files that we should read before proceeding (which are typically named in all uppercase):

[bamccaig@krypton sqsh-2.1.7]$ ls -1 | egrep '^[[:upper:]]+$'
AUTHORS
COPYING
INSTALL
README
[bamccaig@krypton sqsh-2.1.7]$

Excellent! There is a README and most importantly an INSTALL! Go ahead and read through both of those files now. I'll wait.

... ... ...

OK, now that we've read those two files, we should all be aware of the dependencies. Basically you need Sybase CT-Library OR you need FreeTDS. This threw me for a loop at first because when I reached the Sybase CT-Library dependency it was worded in such a way that made it sound absolutely required. If you go on to read the next dependency, you'll see that you can alternatively use FreeTDS. It sounds like there is actually a couple of freely available Sybase downloads for Linux (free as in beer, not as in speech), but I'm not entirely sure which, if any, contain the Sybase CT-Library and worse you're required to register with Sybase to download them (ugh). I was pleased to discover that doing so is unnecessary.

Instead, I say again, you can use FreeTDS; and so that's what we're going to do. Lets see if we can find it.

[bamccaig@krypton sqsh-2.1.7]$ yum search freetds | grep '^freetds'
freetds-devel.i686 : Header files and development libraries for freetds
freetds-doc.i686 : Development documentation for freetds
freetds.i686 : Implementation of the TDS (Tabular DataStream) protocol
[bamccaig@krypton sqsh-2.1.7]$

That is in the default repositories so lets go ahead and install it (including the development files; and WTH, the documentation too).

[bamccaig@krypton sqsh-2.1.7]$ su -c 'yum install freetds freetds-devel freetds-doc'
Password:
[Hopefully YUM spits out happy here]
[bamccaig@krypton sqsh-2.1.7]$

With that out of the way, we're almost ready to build. Taking another look at the INSTALL file, we need to specify the base SYBASE directory in the SYBASE environment variable. I know what you're thinking. I know I was. I don't have Sybase! This doesn't apply to me! Not so fast. It turns out that it does apply to you (and I).[1] We instead need to point the SYBASE environment variable at the base directory for FreeTDS.

What this means is that $SYBASE/include and $SYBASE/lib should tell our build process where to find the necessary header and library files. So where are those files installed for FreeTDS? You can make an educated guess or you can ask rpm itself. I actually didn't know about this until now. It's hopefully going to make things a lot easier in the future.

[bamccaig@krypton sqsh-2.1.7]$ rpm -ql freetds freetds-devel | \
        egrep '/(include|lib)' | sort
/usr/include/bkpublic.h
/usr/include/cspublic.h
/usr/include/cstypes.h
/usr/include/ctpublic.h
/usr/include/sqldb.h
/usr/include/sqlfront.h
/usr/include/sybdb.h
/usr/include/syberror.h
/usr/include/sybfront.h
/usr/include/tdsconvert.h
/usr/include/tds.h
/usr/include/tds_sysdep_public_32.h
/usr/include/tds_sysdep_public.h
/usr/include/tdsver.h
/usr/lib/libct.so
/usr/lib/libct.so.4
/usr/lib/libct.so.4.0.0
/usr/lib/libsybdb.so
/usr/lib/libsybdb.so.5
/usr/lib/libsybdb.so.5.0.0
/usr/lib/libtds-0.82.so
/usr/lib/libtdsodbc.so
/usr/lib/libtdsodbc.so.0
/usr/lib/libtdsodbc.so.0.0.0
/usr/lib/libtds.so
[bamccaig@krypton sqsh-2.1.7]$

You can check the man or info page(s) for details, but essentially what we did is query the package manager to list the files that were installed; then we filtered out the results we weren't interested in (since we're only interested in files within an include or lib directory, I greped for that within the path). For good measure, I also sorted the results.

As you can see, FreeTDS is installed in /usr. No real surprise there. OK, so we need to set SYBASE to /usr.

[bamccaig@krypton sqsh-2.1.7]$ export SYBASE=/usr
[bamccaig@krypton sqsh-2.1.7]$ echo $SYBASE
/usr
[bamccaig@krypton sqsh-2.1.7]$

And now we're finally ready to start the build process. sqsh uses the "GNU auto-configuration package" as so many open source projects seem to do, so it uses the rather standard ./configure, make, make install build instructions. If you want readline support (and most people probably do) then you'll probably want to add the --with-readline option while configuring (you'll obviously need to have readline installed to do this though). Review the INSTALL file for more options.

[bamccaig@krypton sqsh-2.1.7]$ ./configure --with-readline
[Lots of output... Should be happy if your environment is complete...]
[bamccaig@krypton sqsh-2.1.7]$ make
[Everything is going good, the project compiles, but then all of a sudden...the linker is angry!]
gcc    -L/usr/lib  cmd_alias.o cmd_bcp.o cmd_buf.o cmd_connect.o cmd_do.o cmd_echo.o cmd_exit.o cmd_for.o cmd_func.o cmd_go.o cmd_help.o cmd_history.o cmd_if.o cmd_input.o cmd_jobs.o cmd_kill.o cmd_lock.o cmd_loop.o cmd_misc.o cmd_read.o cmd_reconnect.o cmd_redraw.o cmd_reset.o cmd_return.o cmd_rpc.o cmd_set.o cmd_shell.o cmd_show.o cmd_sleep.o cmd_wait.o cmd_warranty.o cmd_while.o var_ctlib.o var_date.o var_debug.o var_dsp.o var_hist.o var_misc.o var_passwd.o var_readline.o var_thresh.o dsp.o dsp_bcp.o dsp_csv.o dsp_conv.o dsp_desc.o dsp_horiz.o dsp_html.o dsp_meta.o dsp_none.o dsp_out.o dsp_pretty.o dsp_vert.o dsp_x.o sqsh_alias.o sqsh_args.o sqsh_avl.o sqsh_buf.o sqsh_cmd.o sqsh_compat.o sqsh_ctx.o sqsh_debug.o sqsh_env.o sqsh_error.o sqsh_expand.o sqsh_fd.o sqsh_filter.o sqsh_fork.o sqsh_func.o sqsh_getopt.o sqsh_global.o sqsh_history.o sqsh_init.o sqsh_job.o sqsh_readline.o sqsh_sig.o sqsh_sigcld.o sqsh_stdin.o sqsh_strchr.o sqsh_tok.o sqsh_varbuf.o sqsh_main.o -ldl -lm   -lreadline -lcurses  -o sqsh
cmd_bcp.o: In function `bcp_signal':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:1164: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:1166: undefined reference to `ct_cancel'
cmd_bcp.o: In function `cmd_bcp':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:337: undefined reference to `ct_cmd_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:877: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:894: undefined reference to `blk_done'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:898: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:901: undefined reference to `ct_close'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:904: undefined reference to `ct_con_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:910: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:924: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:942: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:947: undefined reference to `blk_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:952: undefined reference to `ct_close'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:953: undefined reference to `ct_con_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:957: undefined reference to `cs_loc_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:395: undefined reference to `ct_command'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:406: undefined reference to `ct_send'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:865: undefined reference to `blk_done'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:866: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:867: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:422: undefined reference to `ct_con_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:429: undefined reference to `ct_callback'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:438: undefined reference to `ct_callback'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:447: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:460: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:475: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:488: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:521: undefined reference to `ct_con_props'
cmd_bcp.o:/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:503: more undefined references to `ct_con_props' follow
cmd_bcp.o: In function `cmd_bcp':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:558: undefined reference to `cs_loc_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:539: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:566: undefined reference to `cs_locale'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:600: undefined reference to `cs_locale'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:582: undefined reference to `cs_locale'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:617: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:630: undefined reference to `ct_connect'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:672: undefined reference to `blk_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:703: undefined reference to `blk_init'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:685: undefined reference to `blk_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:722: undefined reference to `ct_results'
cmd_bcp.o: In function `bcp_data_bind':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:973: undefined reference to `ct_res_info'
cmd_bcp.o: In function `cmd_bcp':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:818: undefined reference to `blk_done'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:837: undefined reference to `blk_done'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:797: undefined reference to `ct_fetch'
cmd_bcp.o: In function `bcp_data_xfer':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:1060: undefined reference to `ct_fetch'
cmd_bcp.o: In function `cmd_bcp':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:769: undefined reference to `blk_done'
cmd_bcp.o: In function `bcp_data_bind':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:1013: undefined reference to `ct_describe'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:1031: undefined reference to `ct_bind'
cmd_bcp.o: In function `bcp_data_xfer':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:1096: undefined reference to `blk_bind'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:1112: undefined reference to `blk_rowxfer'
cmd_connect.o: In function `cmd_connect':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:719: undefined reference to `ct_con_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1172: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1187: undefined reference to `ct_close'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1196: undefined reference to `ct_con_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1202: undefined reference to `cs_loc_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1208: undefined reference to `ct_exit'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1209: undefined reference to `cs_ctx_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:769: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:814: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:865: undefined reference to `cs_loc_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:869: undefined reference to `cs_locale'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:908: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:925: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:949: undefined reference to `ct_connect'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:978: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:567: undefined reference to `cs_ctx_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:597: undefined reference to `cs_config'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:725: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:573: undefined reference to `cs_ctx_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:579: undefined reference to `cs_ctx_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:585: undefined reference to `cs_ctx_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:591: undefined reference to `cs_ctx_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:782: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:610: undefined reference to `ct_init'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:613: undefined reference to `ct_callback'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:621: undefined reference to `ct_callback'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:635: undefined reference to `ct_config'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:702: undefined reference to `ct_config'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:651: undefined reference to `ct_config'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:668: undefined reference to `ct_config'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:822: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:836: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:850: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:882: undefined reference to `cs_locale'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1129: undefined reference to `cs_loc_drop'
cmd_connect.o: In function `check_opt_capability':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1251: undefined reference to `ct_capability'
cmd_connect.o: In function `cmd_connect':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1137: undefined reference to `ct_options'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:896: undefined reference to `cs_locale'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1037: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1089: undefined reference to `ct_cmd_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1094: undefined reference to `ct_command'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1115: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1105: undefined reference to `ct_send'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1111: undefined reference to `ct_results'
cmd_connect.o: In function `syb_client_cb':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1571: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1580: undefined reference to `ct_cancel'
cmd_do.o: In function `cmd_do_sigint_cancel':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:677: undefined reference to `ct_cancel'
cmd_do.o: In function `cmd_do_exec':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:304: undefined reference to `ct_cmd_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:312: undefined reference to `ct_command'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:327: undefined reference to `ct_cmd_drop'
cmd_do.o: In function `cmd_do':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:260: undefined reference to `ct_close'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:261: undefined reference to `ct_con_drop'
cmd_do.o: In function `cmd_do_exec':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:322: undefined reference to `ct_send'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:334: undefined reference to `ct_results'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:434: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:435: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:351: undefined reference to `ct_fetch'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:446: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:447: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:423: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:424: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:464: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:465: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:381: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:382: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:342: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:343: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:469: undefined reference to `ct_cmd_drop'
cmd_go.o: In function `cmd_go':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_go.c:468: undefined reference to `ct_cmd_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_go.c:475: undefined reference to `ct_command'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_go.c:511: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_go.c:481: undefined reference to `ct_cmd_drop'
cmd_reconnect.o: In function `cmd_reconnect':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_reconnect.c:58: undefined reference to `ct_close'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_reconnect.c:59: undefined reference to `ct_con_drop'
cmd_rpc.o: In function `rpc_param':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_rpc.c:559: undefined reference to `cs_convert'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_rpc.c:573: undefined reference to `ct_param'
cmd_rpc.o: In function `cmd_rpc':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_rpc.c:228: undefined reference to `ct_cmd_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_rpc.c:441: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_rpc.c:476: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_rpc.c:279: undefined reference to `ct_command'
var_ctlib.o: In function `var_set_interfaces':
/home/bamccaig/src/sqsh-2.1.7/src/var_ctlib.c:82: undefined reference to `ct_config'
dsp.o: In function `dsp_cmd':
/home/bamccaig/src/sqsh-2.1.7/src/dsp.c:140: undefined reference to `ct_cmd_props'
/home/bamccaig/src/sqsh-2.1.7/src/dsp.c:181: undefined reference to `ct_send'
/home/bamccaig/src/sqsh-2.1.7/src/dsp.c:248: undefined reference to `ct_cancel'
dsp.o: In function `dsp_signal':
/home/bamccaig/src/sqsh-2.1.7/src/dsp.c:770: undefined reference to `ct_cancel'
dsp_bcp.o: In function `dsp_bcp':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_bcp.c:65: undefined reference to `ct_results'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_bcp.c:94: undefined reference to `ct_fetch'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_bcp.c:76: undefined reference to `ct_fetch'
dsp_csv.o: In function `dsp_csv':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_csv.c:66: undefined reference to `ct_results'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_csv.c:95: undefined reference to `ct_fetch'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_csv.c:77: undefined reference to `ct_fetch'
dsp_conv.o: In function `dsp_type_len':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_conv.c:505: undefined reference to `cs_convert'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_conv.c:520: undefined reference to `cs_convert'
dsp_conv.o: In function `dsp_datetime_conv':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_conv.c:361: undefined reference to `cs_dt_crack'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_conv.c:341: undefined reference to `cs_convert'
dsp_desc.o: In function `dsp_desc_fetch':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:434: undefined reference to `ct_fetch'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:553: undefined reference to `cs_convert'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:498: undefined reference to `cs_convert'
dsp_desc.o: In function `dsp_desc_bind':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:86: undefined reference to `ct_res_info'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:186: undefined reference to `ct_describe'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:369: undefined reference to `ct_bind'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:296: undefined reference to `ct_bind'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:392: undefined reference to `ct_compute_info'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:406: undefined reference to `ct_compute_info'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:136: undefined reference to `ct_compute_info'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:163: undefined reference to `ct_compute_info'
dsp_horiz.o: In function `dsp_horiz':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_horiz.c:97: undefined reference to `ct_results'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_horiz.c:140: undefined reference to `ct_res_info'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_horiz.c:154: undefined reference to `ct_fetch'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_horiz.c:163: undefined reference to `ct_get_data'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_horiz.c:217: undefined reference to `ct_fetch'
dsp_html.o: In function `dsp_html':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_html.c:84: undefined reference to `ct_results'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_html.c:115: undefined reference to `ct_res_info'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_html.c:135: undefined reference to `ct_fetch'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_html.c:144: undefined reference to `ct_get_data'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_html.c:187: undefined reference to `ct_fetch'
dsp_meta.o: In function `dsp_meta_int_prop':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_meta.c:613: undefined reference to `ct_res_info'
dsp_meta.o: In function `dsp_meta_transtate':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_meta.c:640: undefined reference to `ct_res_info'
dsp_meta.o: In function `dsp_meta_desc':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_meta.c:206: undefined reference to `ct_res_info'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_meta.c:216: undefined reference to `ct_describe'
dsp_meta.o: In function `dsp_meta_fetch':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_meta.c:695: undefined reference to `ct_fetch'
dsp_meta.o: In function `dsp_meta':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_meta.c:65: undefined reference to `ct_results'
dsp_meta.o: In function `dsp_meta_bool_prop':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_meta.c:584: undefined reference to `ct_res_info'
dsp_none.o: In function `dsp_none':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_none.c:54: undefined reference to `ct_results'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_none.c:65: undefined reference to `ct_fetch'
dsp_pretty.o: In function `dsp_pretty':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_pretty.c:87: undefined reference to `ct_results'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_pretty.c:127: undefined reference to `ct_res_info'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_pretty.c:141: undefined reference to `ct_fetch'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_pretty.c:154: undefined reference to `ct_get_data'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_pretty.c:208: undefined reference to `ct_fetch'
dsp_vert.o: In function `dsp_vert':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_vert.c:79: undefined reference to `ct_results'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_vert.c:106: undefined reference to `ct_res_info'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_vert.c:121: undefined reference to `ct_fetch'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_vert.c:127: undefined reference to `ct_get_data'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_vert.c:305: undefined reference to `ct_fetch'
sqsh_ctx.o: In function `sqsh_ctx_pop':
/home/bamccaig/src/sqsh-2.1.7/src/sqsh_ctx.c:104: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/sqsh_ctx.c:113: undefined reference to `ct_close'
/home/bamccaig/src/sqsh-2.1.7/src/sqsh_ctx.c:116: undefined reference to `ct_con_drop'
sqsh_init.o: In function `sqsh_exit':
/home/bamccaig/src/sqsh-2.1.7/src/sqsh_init.c:315: undefined reference to `ct_close'
/home/bamccaig/src/sqsh-2.1.7/src/sqsh_init.c:316: undefined reference to `ct_con_drop'
/home/bamccaig/src/sqsh-2.1.7/src/sqsh_init.c:322: undefined reference to `ct_exit'
/home/bamccaig/src/sqsh-2.1.7/src/sqsh_init.c:337: undefined reference to `cs_ctx_drop'
collect2: ld returned 1 exit status
make[1]: *** [sqsh] Error 1
make[1]: Leaving directory `/home/bamccaig/src/sqsh-2.1.7/src'
make: *** [build-subdirs] Error 2
[bamccaig@krypton sqsh-2.1.7]$

VERY angry. Obviously, we're missing a library. ct_blah suggested to me that it was one or more Sybase CT-Library libraries that it couldn't find. Since we're using FreeTDS in place of it, I went back to our friend rpm and with the help of grep and sed got the linking options that I was looking for.

[bamccaig@krypton sqsh-2.1.7]$ rpm -ql freetds freetds-devel | \
        grep '^/usr/lib/' | \
        sed -re 's#/usr/lib/lib##' -e 's/([^\.-]*).*/\1/' -e 's/(.*)/-l\1/' | \
        sort | xargs
-lct -lct -lct -lsybdb -lsybdb -lsybdb -ltds -ltds -ltdsodbc -ltdsodbc -ltdsodbc
[bamccaig@krypton sqsh-2.1.7]$

For those of you that don't speak UNIX so well yet, that basically means:

List all the files that were installed by the freetds and freetds-devel packages (it almost certainly would suffice to just check freetds-devel, but it doesn't hurt to include freetds in the search).
Filter out all options that don't begin with /usr/lib (since we're looking for libraries and we know they are installed in /usr/lib).
Remove the directory (/usr/lib) and prefix (lib) from each library (most shared libraries are named something like libfoo.so).
Remove any trailing characters (characters that are after a dot or dash, since most shared libraries have version information appended to their name, resulting in something like libfoo.so.1.2.3, for example; those are often symlinked by friendlier names, however, and the linker will do the work of figuring out what our pretty names mean).
Prefix each library with -l, since that is the option that GCC uses to specify libraries.
Finally, sort them for easy reading and join them onto a single line.

Reviewing the command line in the above error message, we can see that sqsh isn't being linked with any of them! That has to be problem! The h4x solution then is to edit the Makefile in question and add those to the command.

Upon opening that file, I found a handy SYBASE_LIBS variable that seemed rather relevant. Above it is even a handy comment suggesting that users of some platforms might need to manually edit that line. Open that file in your favorite editor...

[bamccaig@krypton sqsh-2.1.7]$ vim src/Makefile

...and replace that line with the results from our earlier query:

SYBASE_LIBS = -lct -lct -lct -lsybdb -lsybdb -lsybdb -ltds -ltds -ltdsodbc -ltdsodbc -ltdsodbc

Save the file and try to build again.

[bamccaig@krypton sqsh-2.1.7]$ make
[Now it should almost certainly be happy unless the world just hates you...]
[bamccaig@krypton sqsh-2.1.7]$ su -c 'make install && make install.man'
Password:
[Installation should go happily too]
[bamccaig@krypton sqsh-2.1.7]$

If you can run sqsh now (for example, sqsh --help to test) then the build and installation should have been successful! Congratulations! Now you (and I) just have to learn to use it. :-X

To get you started, the following allows me to connect to the SQL Server located on our LAN, running on a Windows server (**shudder**).

[bamccaig@krypton ~]$ sqsh -S <server_ip> -U <username>
[Program information]
Password: 
[I am now inside of a sqsh shell!]
> quit
[bamccaig@krypton ~]$

Obviously, not the most convenient way to connect, typing the server, username, and password (!) each time. Surely there must be handy configuration options to make things smoother. For now, I have created shortcut scripts (readable/writeable/executable only by me) that do the connecting for me to save me the trouble.

References

1. Thanks to the maintainer for clearing this up via E-mail.

Wasabi - Apparently ASP/VBScript Isn't Evil Enough

2010-05-31T11:26:00.003-04:00

I've been looking for "project management" software (I know, yuck) for some time. Mainly because I want to incorporate bug/issue tracking, discussion boards, and wikis into j0rb (and in personal projects, etc.). Firstly, it's very difficult to keep track of bugs by hand and often you forget about them before you fix them. When it comes to software development there's just too many issues, whether bugs, design decisions, or potential hiccups; to keep track of. I also want some persistent medium to discuss software design with colleagues. At j0rb we currently discuss things either through instant messaging or we talk in person.

The problem with instant messaging is that there's no organization to it. If you forget what was discussed or what conclusions everyone came to then you have to search through chat logs and hope to find it. If you do find it, you have to sort through intertwined discussions to find the relevant material. It's also hard to have meaningful discussions because instant messengers are generally designed to send single sentences or short paragraphs. It's hard to communicate code ideas or complex ideas that require more space to type. The problem with talking in person is that there's no log at all. There's no searching it to refresh your memory, etc.

Open source projects have gotten by with mailing lists for a long time and it seems the most experienced still prefer them to discussion boards. I figured there must be good reason for this so I've personally taken a liking to them. However, I don't think mailing lists would work well at j0rb. It's a Windows-based shop and I generally hate Windows-based mail clients for good reason. I generally use Gmail's Web interface for personal mail, which works quite well for most E-mail needs, but I wouldn't be allowed to discuss company topics through a remotely hosted service. At least, not a free one with no guarantees about security or privacy.

I'd be happy to try mutt from Cygwin against a locally hosted server, but odds are that my colleagues would be sending HTML E-mails and wouldn't understand why that was a problem (nor why I'd choose to use a plain-text client). For these reasons, I don't think a mailing list would work particularly well at j0rb. Colleague incompatibility. :P A discussion board seems to be the next best thing (with the advantage of edits to correct mistakes, etc.).

Wikis are great for keeping track of a growing and dynamic knowledge base of information. I think they'd work well for documenting certain gotchas discovered in languages, APIs, and platforms, etc.; as well as our own software's gotchas.

Anyway, I've recently been reading Joel on Software. There are a lot of really good articles that make a lot of sense. It seems Joel Spolsky is very experienced in project management and the like. The allure of fully defined specifications and feasible schedules got me interested in taking a look at FogBugz, some project management software developed by Joel's company, Fog Creek Software. I generally avoid commercial software, since as a general rule it's garbage, but I am willing to appreciate some commercial offerings (everything from Microsoft not being among them *cough*).

Anyway, it sounded like FogBugz was a pretty complete solution and I had hoped that it was better than what we have now (which can't do any of the things I mentioned above). I suggested it as an option to management, but the price seems kind of steep. Fortunately, they offer the hosted service for free to individuals so I decided to sign up for personal projects so I could get a look at just what it offered. I was pretty disappointed. At least at first glance, the UI seems rather bloated and disorganized. I'm having a hard time trying to figure out where different types of information are organized. It seems all it really tracks are "cases", which can be bugs, issues, or what have you. Then they use "filters" to determine which cases you see, by tons of criteria. This seems to be the way to show cases for a particular project. It seems awkward that way, but maybe that's just because I'm not used to it.

I decided to ask the Interwebz what it thought of FogBugz. I started where I usually start: at Wikipedia. It was there that I discovered, much to my dismay, Thistle and Wasabi.

It seems that FogBugz was originally written in Classic ASP/VBScript. I can sympathize because our main software project is mostly written in the same. What appears like an OK language from the surface is surely not. Visual Basic is bad enough, but VBScript is like a stripped down Visual Basic with half of the features missing or mutilated. Those that have worked with VBScript for any considerable period of time know that it is the decay-er of sanity. It has numerous limitations and has a weak "standard library" and most non-trivial functionality comes from server "components" that do not appear to be native VBScript at all (I've always assumed they were DLLs written in C or C++, but I suppose it's possible that they're actually written in pure evil). VBScript can't do much without these components. You apparently need these components for things like handling file uploads or connecting to databases. Things that other languages, like PHP, Python, and Perl can do "themselves". Probably because those languages are "open".

Apparently, Fog Creek Software wanted FogBugz to run on Linux servers, but ASP/VBScript is (officially) an abomination of IIS and the Windows operating system. I think at this point most would realize their mistake (developing software with ASP in the first place) and work towards correcting it by rewriting the application in a better language (and one that was cross-platform), but it seems that Fog Creek decided instead to develop a "compiler" (converter, or what ever you want to call it) that could convert ASP into PHP so that they could run their application on Linux boxes without having to completely rewrite it.

Now I don't know how many lines of code FogBugz was at the time or how complicated this "compiler", dubbed Thistle, was to write (it's over my head, I know that); but I know that it only took a few days of maintaining an ASP/VBScript application for me to begin begging almost daily to rewrite the application in something more sane. They didn't stop there, however. They apparently realized that VBScript combined with Thistle was too limited in what it could do (light bulb, anyone?). They missed their second opportunity to change platforms and instead wrote another "compiler" that extends VBScript's functionality, adding modern features that probably should have been there in the beginning, and spits out either PHP or .NET. They call this monstrosity Wasabi. All of that trouble to extend evil that never should have existed in the first place. I can't imagine that developing Thistle and Wasabi was faster than just rewriting their application in a cross-platform language, preferably an open one that will be around for a while, and leaving it at that.

Apparently they keep both of these tools internal. So not only did they create such evils, encouraging the persistence of VBScript-ness, but they didn't even release it to the world for them to benefit (free or otherwise). WTF.

I'm somehow less enthusiastic about FogBugz now..

Incompetent Drivers - Tailgating

2010-05-18T19:24:00.049-04:00

I live 20 - 30 minutes outside of the city I work in so I commute each day. I have a heavy foot and like to drive fast, but I do it safely. I never exceed the speed that I feel is safe for the road conditions and I'm not afraid to drive well under the speed limit if conditions require it. I aced the Young Drivers of Canada driving school program when I was 17 and very much appreciated the things they taught me (to my dismay, most of my classmates joked about how stupid the lessons were...). Driving just came naturally to me, but some of the things I learned from Young Drivers of Canada really opened my eyes. They made the more tedious tasks of driving become routine and logical (things like where your mirrors should be pointed, how to effectively see all around your vehicle, and how to easily back into a parking space) and really just gave me the confidence to do the safe thing instead of the easy thing. It's a real load off of your shoulders to hear someone in a position of "authority" say that it's OK to hold up traffic et. al. It's common sense, but "peer pressure" has a way of clouding your judgment. I think we're all victims of this at times.

Anyway, I love to drive and I think that I'm a very good driver. I do like to go fast though. When I was ~20 I got two speeding tickets within a month of each other. Fortunately, the police officers were kind both times, but my insurance company still hit the fan when it found out. They had me signing documents agreeing to never drive my parents' cars and forced me to insure myself under my own name instead of under my father's (the latter is much cheaper than the former for those that don't know). Anyway, that didn't last too long and I eventually broke through the threshold period, but it was enough of an eye opener to see just how expensive it could be if you get caught. I would like to argue unfairly so since I'm such a safe driver, but there's obviously no way for the police nor insurance companies to know who is competent vs. who is reckless.

After that I always stay below the speed limit (the actual speed limit though, not necessarily the posted one), at least when there's a chance that police could be ahead (or following me; or in the air!). For my second speeding ticket I was caught on radar from a few kilometers (and miles) away. Think of a straight stretch of highway so long that you can't even see the corner. The police must have been sitting on the shoulder right at the corner. It blew me away just how far they could see. There was absolutely no chance anyone could have seen them, at least not without illegal scanning technology. I was impressed, albeit beaten. In my defense, I was passing slow drivers on a completely open highway so I was somewhat justified and it was completely safe. I digress...

Anyway, speeding or not, I always focus on safety. I never drive faster than is safe and always try to drive defensively. I'm not going to get anywhere faster being two car lengths ahead and still trapped behind traffic so there's no benefit to tailgating anyway. It's just stupid. Not everyone is so enlightened as I am though.

Usually when I encounter tailgaters its on the highway, which is silly because for the past 4 years or so I've been driving on a 4-lane highway. If I'm not going fast enough for you then pass me. There's an open lane for just such a purpose! SURPRISE. This doesn't seem to occur to people. I can only assume then that they think they're saving fuel by "drafting" me or something. Not enough fuel, if any, to outweigh the risks, I promise you that.

Defensive Driving

My usual defense against tailgaters is to let off the accelerator and just coast. It isn't so drastic that they don't have time to react, like they would if I braked (since they're following too goddamn close), but it's enough to get a reaction out of them. The typical reaction is for them to jump into the left lane to pass me. Little do they know that I'm not slowing down to stop or turn. I'm slowing down to get them off my ass. Once they are, I pedal-to-the-metal back up to the speed limit. If they want to pass me then by all means pass me, but I'm not going to reward their incompetence by passing me for them. My accelerating is basically "body" language to communicate that I know they're tailgating me and I won't allow it. GET. OFF. MY. ASS. There really isn't much they can do about it. They can either comply, get repeatedly slowed down because I keep doing it until they stop (wasting fuel in the process to get back up to speed), or hit me (in which case either the police will be called and they'll be charged or I'll follow them home and [censored]). There's also the possibility that they could follow me home, but I feel I'm not doing anything worthy of such vengeful acts and if they're that upset by it then sooner or later they're going to snap and it might as well be on just me than on an entire family or something. I also tend to watch for people following me (some people say I can be paranoid; pffft) so I'd hopefully notice them and just not go home. I'd just call the police from my cell phone and let them catch up to us. Of course, they could try to run me off the road too, but this still fits under the it-might-as-well-be-me category. Besides, as I said I'm fairly confident driving and love to push the limits so I'd say I'm better prepared for such maniacs than the average driver on the road would be. If they kill me, shucks. The world will not stop turning. Hopefully the police catch them and lock them up.

The Event

Anyway, on to today. I'm driving home as I usually do. I'm still within city limits driving on Trunk Road in Sault Ste. Marie, Ontario, Canada.

View Larger Map
The speed limit is 70 km/h, though you can generally get away with about 85 km/h. If you do get ticketed, and you technically can, you shouldn't lose any demerit points so it's not too much of a risk. Not all traffic speeds though and I respect that. As long as they're doing the speed limit (or have a valid excuse not to) I'm fine with it. I'll go around them if it's safe to (Trunk Road is a 5-lane) or else I'll wait until we get to the highway just ahead where things usually open up more and then I'll pass them. Today there was a small line of slow moving traffic in the left lane so I pulled into the right lane to pass them (not something I like to do on the highway unless absolutely warranted[1], but within city limits traffic is much more disorganized). As I checked my mirrors and blind spot, I noticed there was a silver Volkswagen Jetta (a relatively new model) coming up on me quickly. Now we'd been cruising at the same speed for a few kilometers so it wasn't like he was blind-sided. He could have changed lanes before catching up to us and moved right on past. Instead, he came right up behind me. At this point, the right lane was clear so I pulled into it to pass the slower moving traffic.

I accelerated to 85 km/h and set my cruise control. I watched in my rear view mirror as the Volkswagen eventually pulled around the slower moving traffic and pulled in behind me. Albeit, it was a delayed reaction. Someone in such a hurry shouldn't have been so slow unless they're incompetent (and therefore shouldn't be speeding), but I digress. This Volkswagen accelerates right up on my ass to the point where I can't even see his bumper in my rear-view mirror. I would guess maybe a car length or so behind me, but it's obviously hard to judge. A car length is not enough at 85 km/h. I'm driving a '95 Pontiac Grand Prix so it's not like it's the greatest stopping vehicle in the world, but a car length isn't even enough time for a human to react at 85 km/h if I happen to need to lock up my brakes. He was too goddamn close. As usual, I let off my accelerator (actually disengaged my cruise control) and started coasting. The Volkswagen hesitated to slow down, but then aggressively switched lanes moving in behind the slower moving traffic to our left. As usual, the threat was diverted so I accelerated hard back up to the "real" speed limit of 85 km/h. I saw in my rear view mirror that the Volkswagen jumped back in behind me, which is to be expected since the other lane is blocked, but what pisses me off is that he goes right back to tailgating me. He didn't take the hint or is just too much of an asshole to care. I let off my accelerator again (again technically disengaging my cruise control), but saw that he was wise (I use that term loosely) to me now and wasn't phased by it. I flipped him the bird to let him know bluntly that I was displeased with him and decided to just gun it past the slow moving cars so I could get him off my ass.[2] Speeding ticket be damned. I would have loved to see a cop right then. This asshole hammers down too (he's in a newer car in better condition so he has no trouble keeping up) and rides my ass all the way past the slower line of cars. Up ahead I was making a left turn onto the highway so as I came around the slower moving cars I smoothly signaled and began to lane change. As I did, I saw in my rear-view mirror that the Volkswagen driver changed lanes with me. I did it as soon as it was safe to in front of the slower moving traffic so he must have been cutting them off very closely. Once he saw that I had occupied the other lane he changed lanes and aggressively passed me. I held up the bird as he passed, trying to get a look at the driver, but he had tinted windows and I couldn't see anything.

The Volkswagen driver eventually pulled into the left lane a ways ahead of me, but at literally the last second jumped back into the right lane, taking the old highway out of town instead of the new one. Now he was nowhere near me so he couldn't have been toying with me. And if he had planned to take the old highway all along why would he have switched lanes? My best guess is that he realized that I was taking the new highway and didn't want to be himself followed home. Smart move because with him in front of me I would very likely have gotten his license plate and reported him to the police (it was far too tense to do so earlier). It's also possible that he spotted a "cop?" just up ahead down the new stretch of highway and got scared, but it wasn't your typical cop. I think it was the kind of cop that pulls over transports and inspects their loads and stuff. They have a different car anyway.

Conclusion

The Volkswagen driver and I parted ways there, but I kept an eye out for him ahead hoping we would cross paths again since both highways essentially lead to the same place (so I could call the police; somebody that aggressive should be taken off the streets). Unfortunately, I didn't see him again. Maybe my bald head and baggy clothes scared him (I know it's enough for the police to profile me... :P). I can only hope.

I was pretty shaken up after the fact, as you get when you're both angry and afraid of what might happen. It occurred to me when I got home and had calmed down that having a camera mounted in the back of my car would be really useful. Snap a photo of these idiots' license plates without taking my eyes off the road and calmly pull over to report them. That would be a wonderful luxury.

Now I know that I've been referring to the Volkswagen driver as a "he", but that's just out of habit since I didn't get a look at the driver (tinted windows) and guys tend to be the more aggressive drivers. However, I don't want to eliminate the possibility that this was indeed a woman. Men tend to be dangerous because of testosterone. They're chemically charged for it. Women tend to be dangerous drivers because they're incompetent. I don't mean any offense to the women, but it's nothing new that women are far more likely to consider a car nothing more than a means of transportation whereas men often consider them fun and dedicate time to learning all about them. I regularly see women speeding on the highway (and I mean speeding because they're blowing past me and I'm already "speeding") and the impression I get is that they aren't really aware of how the vehicle's characteristics change as velocity changes. Just this morning a woman passed me on the highway in a shallow downhill S-curve. She came around me in the middle of it and on the way out of it she drifted a couple of feet over the center line. You may have thought she was changing lanes having overtaken me (albeit she started drifting a little early for that), but no, she wasn't. She continued on her path a couple of feet over the center line and then moved back in the left lane before finally changing lanes another 200 yards down the highway (without signaling). Sigh.

I can't wait until we have computers capable of driving cars in traffic so that everyone else can surrender their license. >:( I, however, love to drive, so I'll continue to do so myself, thank you very much.

References

1. I've seen a pattern of incompetent drivers driving in the left lane. The highway is never so busy that you need two lanes of traffic. The left lane is basically just a passing lane. Somebody that drives in it constantly appears incredibly incompetent to me. Occasionally you encounter slow moving traffic that is driving in the left lane, forcing you to pass them on the right (or awkwardly follow behind them).

2. I regretted this decision almost instantaneously. I essentially let him win, but I felt like I had little choice. If I slowed down enough the slower moving traffic would pass me and he'd move between us. If I stayed with the slower moving traffic he would stay on my ass which just wasn't safe. If this was Los Angeles I would have put a few shots through his windshield, but again I digress.

Source Code Management -- A Minor Success Story

2010-05-18T16:08:00.023-04:00

In my quest to become an open source [UNIX-y] programmer (or hacker :-o), I try to learn about development tools and practices that make development easier. One such category of tool is source code management (SCM) tools (also called version control software). Essentially they keep track of the history of changes that you make to your code[1], as well as who made each change, when they made the change, and even comments from the author of the change explaining the change. This greatly helps developers manage a project because it allows us to keep track of what we've done and even what we're doing right now. It allows you to undo changes that you've made easily and share changes easily between developers. There are plenty of benefits so don't consider this a complete list.

For the record I've found Git to be the best SCM. At j0rb, however, we use Subversion. I myself learned about Subversion a year or so after college and taught myself to use it. Then I introduced j0rb to it and eventually managed to get it adopted. More recently (~past six months) I started using Git after watching Google Tech Talks on YouTube of Linus Torvalds and Randal Schwartz explaining why Git is the superior SCM and why everything else sucks. I didn't want to be "stupid and ugly" so I naturally adopted Git. Now I'd like to switch j0rb over to Git, but I mostly work with Windows-y, GUI-y programmers that are afraid of something like Git. Needless to say, they are refusing to change for now. We all know what that makes them.

Anyway, I've been working on something at j0rb for the past ~week. With Git, I could easily branch and/or commit locally as I go to separate changes, but with Subversion branching and merging is expensive and painful and there's no local repository to commit to. That's because everything is centralized so committing would put my changes in the central repository that everyone uses, which would mean that the application that my colleagues and I are working on would be broken until I'm done, preventing others from doing any work of their own.

Branching (Side Tracked)

One of the nicest things about Git's branching mechanism is that I don't need to go anywhere in the file system. When I change branches in Git, Git automatically makes my working directory that I'm already in the branch that I'm switching to (checking out, technically). With Subversion, a branch is really just a copy of some tree; it's a duplicate of a subtree. In order to work on the new branch, I need to check it out somewhere else on my file system (or I could remove my working directory and overwrite it with the new branch). To demonstrate:

# With Git, it's simple. Create and checkout a new branch named 'newbranch'
# based on the master branch at the current HEAD of the branch (last commit).
bamccaig@castopulence:~/src/example$ git status
# On branch master
nothing to commit (working directory clean)
bamccaig@castopulence:~/src/example$ git checkout -b newbranch master
Switched to a new branch "newbranch"
bamccaig@castopulence:~/src/example$ git status
# On branch newbranch
nothing to commit (working directory clean)
bamccaig@castopulence:~/src/example$

As can be seen above, with one simple command Git has created a branch and I'm already in it! I didn't have to do anything else. I can just start working. And Git is fast when it comes to branching so I didn't have to wait for anything. The new branch just points to the master branch so there was no need for expensive duplication of data. Subversion tells a different story, however:

# With Subversion, it's quite painful and it's also pretty slow. Create
# and checkout a new branch named 'newbranch' based on the repository
# trunk in the HEAD revision. Note that in my experience it's best to do
# branching in Subversion server-side. At least if you ever intend to
# merge back into the original branch...
bamccaig@castopulence:~/src/example/trunk$ svn cp -m 'Example...' \
        file:///home/bamccaig/src/example.repo/trunk \
        file:///home/bamccaig/src/example.repo/branches/newbranch

Committed revision 2.
bamccaig@castopulence:~/src/example/trunk$ svn up .. && \
        cd ../branches/newbranch
A    ../branches/newbranch
A    ../branches/newbranch/foo
A    ../branches/newbranch/bar
A    ../branches/newbranch/baz
Updated to revision 2.
bamccaig@castopulence:~/src/example/branches/newbranch$

Notice that Subversion basically requires me to not only type out a semi-lengthy URL (twice!) and download another complete copy of the original branch (in this case branches/newbranch, which is a copy of trunk), but also requires me to move around in the file system. In this simple example, I had the entire tree in my working copy (from the root of the repository, including branches, tags, and trunk). Often though you aren't interested in all of the many branches and tags that exist so you'll only checkout the branch(es) that you're interested in. In that case, you have to type out a checkout command with what's probably a semi-lengthy URL and then type out a command to change to the new branch's working directory. In short, branching in Subversion is just no fun. And don't get me started on merging... :'(

Back To The Story

OK, so here I am with a lot of changes to my working copy (give or take, 15 added or modified files). I come back in to work on Monday after the weekend and start working on a separate, though related project. When I finally get that done in mid-afternoon I get back to my original project, rebuild it and run it (something I generally do to get an idea of what state things are in and remind myself what I was working on last; again, Subversion doesn't help much when there's 15 added or modified files). To my dismay, this ASP.NET project throws a StackOverflowException immediately upon launching in Visual Studio's development Web server, which subsequently "crashes" the server since there's really no recovering from that. Unfortunately, with a StackOverflowException, there is apparently no stack trace (something I discovered right then) because the stack[2] itself is in an unholy state. On to tracking down what was causing the problem. But how? A stack overflow usually means you're either calling too many nested functions (often a result of recursion) or you've allocated too much memory on the stack.

Here's where the SCM comes in handy (albeit, this particular SCM still comes up short). I tried looking at all of the changes I had made since my last commit to see if I could spot anything suspicious.

[bamccaig@j0rb:foo]$ svn diff | less -S

Unfortunately, nothing stood out. I had added some new LINQ to SQL entities to the project and added some code to work with them. Much of the new code was generated for me by Visual Studio. The code I had worked on didn't stand out as a culprit.

This is where having Cygwin installed, a UNIX-like environment for Windows, comes in handy[3]. I decided the most efficient way for me to find the problem was to undo the changes I had made, confirm that it worked, and then redo the changes bit by bit until I encountered the StackOverflowException. This way I would know where to look for problems: the last applied changes. UNIX and UNIX-like operating systems (and Cygwin, as mentioned above) have tools that make this easy. First, I generate patches[4] with the SCM, Subversion, and a little shell scripting.

[bamccaig@j0rb:foo]$ for f in `svn st | grep '^M' | \
        sed -r 's/M *(.*)/\1/'`;
do
    svn diff "$f" 1> "$f.patch" && svn revert "$f";
done

For every file, foo/bar, that was modified since the last commit, I get a file foo/bar.patch that stores the changes made. Then I undo those changes (svn revert). For added files, the changes are irrelevant because they're basically the entire file anyway so instead I just temporarily remove them from the Visual Studio project. I can easily get a list of which files to remove though using Subversion and the shell again.

[bamccaig@j0rb:foo]$ svn st | grep '^A'

With all the changes undone (there were no deleted files in my working copy) I was able to retest the code. Lo and behold, it runs fine now. This confirmed that it was indeed me that broke it (damn). That came as no surprise though because I had been working on it for close to a week without problems and without pulling changes from the central repository. I was the only one making changes.

Now comes the fun part. Applying each patch one at a time and testing. It might sound tedious, but imagine how much more tedious it would be without the SCM or UNIX tools. To apply the patches from before, we use (surprise) the patch program.

[bamccaig@j0rb:foo]$ patch -p0 -ui path/to/the.patch

The -p0 option is required to leave paths in the patches alone. The default behavior for patch is to strip off the directory part, leaving only the filename. That only works if the file you're patching is in the current working directory. Mine are all over the working tree. The paths just happen to be correct from where I'm working though so the 0 says to strip nothing from them. The -u option tells patch that the patch file is in unified format, which is what Subversion's diff sub-command outputs by default. patch will likely figure this out on its own, but why waste the resources? :P The -i option specifies the patch file to use (which is followed by the path to the file).

Since I'm using Cygwin, with UNIX newlines[5] on a Windows-based project, however, patch is going to somewhat mangle my source code by filling my files with the wrong newline type. It's not a problem. The code will still work, but Subversion will see every line as a change, which will make reviewing the changes later on rather difficult. To fix this, we can use the unix2dos tool to convert them back. To save myself a lot of tedious typing, I created a bash function for all of this.

p() {
    patch="$1"
    file="`dirname $patch`/`basename $patch .patch`";
    patch -p0 -ui "$patch" && unix2dos "$file" && rm -i "$patch";
}

This way instead of typing out that long patch ... unix2dos command line, I can just say `p path/to/file.patch'. After each patch is applied, I'd confirm that it applied properly and remove the patch file to mark which ones I had done. Then I'd refresh the Visual Studio solution, rebuild it, and run it. If there was no StackOverflowException then I'd move on to the next patch. Once again, the SCM and UNIX tools allow me to easily track my progress. The following function listed which patches I had yet to apply:

[bamccaig@j0rb:foo]$ c() {
    svn st | grep patch;
}

I used that list to try to apply patches in order of dependencies to avoid unrelated problems.

The Suspense Is Killing Me!!!11

So what was the bug in the code causing the StackOverflowException? I have no clue... :( After going through the above, I seemingly have applied all patches and added all new files back to the project and it works fine. The only bug I encountered in the code had to do with a data-layer interface that I recently tried (again, I now realize) to get fancy with. Essentially, LINQ to SQL is handled through a DataContext class that is generated for you by Visual Studio. When you query for a set of entities, they are linked to the DataContext that retrieves them. When you make changes to them, the DataContext knows and uses those changes to generate SQL that ultimately updates the database. However, often times the changes are coming from the client, or take place over a few layers of the application. It's hard in some of these instances to maintain the original DataContext and the entities that are attached to it. It's particularly difficult when using serialization to communicate with a user agent. Fortunately, there is an interface to attach detached entities. Unfortunately, it requires both the modified object and the original unmodified object to know what record it's dealing with. This means that something as simple as saving an entity can require an entity-specific query and it just generally results in code bloat. To get around this, I created an interface that returns a LINQ Expression that identifies the entity and implement it for each entity. This way, the framework that I've developed can automatically fetch the original object, reducing the bloat to a simple call:

1
2
3

new LinqManager().Save(entity); // INSERT or UPDATE.
new LinqManager().Delete(entity); // DELETE.

Anyway, getting even more lazy, I also added an interface that returns the record identifier (all tables of this database have INT record identifiers). This allowed me to generate a typed LINQ Expression using generics and that interface.

public Expression<Func<IEntity, bool>> DefaultIdPredicate(
        IEntity e)
{
    return o => o.GetRecordId() == e.GetRecordId();
}

It sounds good and compiles happy, but it fails hard at run-time because LINQ to SQL can't translate it into SQL. During the process of applying the above patches I eventually ran into an exception whose message spoke of this. It was then that I remembered trying it previously (which is why the above method existed already), but it failed and I reverted, leaving the method intact for a future revelation. Instead I'm stuck resorting to ugly type-casting and explicit property access, which I manually re-coded throughout the project. That is the only fix that I made to the code as I applied the changes.

I can only hope that was the problem, though I'm not sure how that could cause a StackOverflowException, and be thankful that I had an SCM, even a poor excuse for one, and a UNIX-like environment to help me through this mess..

References

1. Source code management (version control software) tools aren't limited to tracking source code. They can actually track changes to any set of files (though it may depend on the particular tool), but as a general rule they don't work as well with binary files as they do with text files.

2. If you're unfamiliar with the call stack or stack vs. heap then ask Wikipedia. I'm normally happy to explain it, but I feel exceptionally lazy right now. I'll give you some hints though.

3. Though not as handy as running a UNIX-like operating system, such as Linux, would be. Unfortunately, I'm stuck with Windows at j0rb, but I digress..

4. Patches are essentially instructions for how to change a file automatically. They show the difference between two files, which can be used to automatically modify the original and produce the new one. I feel like I'm doing a horrible job explaining this today so I'm trying to refer to material that will do a better job explaining than I can right now. See here.

5. http://en.wikipedia.org/wiki/Newline.

.NET + XPath + Namespaces (Conclusion)

2010-05-05T13:46:00.019-04:00

In my last post, I discussed how XML namespaces were interfering with XPath expressions that were being used by an application to map XML data. Thanks to kind people on #xml on irc.freenode.net I finally made sense of it.

Within an XSLT document, which you of course know is an XML document, XPath expressions can apparently use the namespaces defined in the XSLT document (I assume then that they can't use namespaces defined in the transforming XML; confirmed). I think that means that if I have the following XML document:

<?xml version="1.0" encoding="utf-8"?>
<root>
    <level1 xmlns:foo="http://www.bamccaig.com/foo">
        <level2>Text</level2>
    </level1>
</root>

Then I could use an XSLT document like this to transform it:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0"
        xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
        xmlns:bar="http://www.bamccaig.com/foo">
    <xsl:template match="//bar:level2">
        <xsl:value-of select="text()" />
    </xsl:template>
</xsl:stylesheet>

Note that the XPath expression used for the <xsl:template> element's match attribute...

//bar:level2

...uses the namespace prefix defined in the XSLT document (i.e., bar), not the namespace prefix defined in the XML document (i.e., foo).

What this helped me to realize is that it doesn't matter that the XML source document that I'm querying for data with XPath has child elements with default namespaces (i.e, no prefixes) because the prefixes can come from elsewhere. It was then that the .NET API began to make sense. In a way, my .NET is taking the place of the XSLT document, and it then makes sense to define namespaces in .NET. I tried to avoid this originally because it felt like hard-coding what should be soft-coded, but now it makes sense.

In my last post, I mentioned that the XPathNavigator.Evaluate method accepted an IXmlNamespaceResolver. While the XPathNavigator is itself an IXmlNamespaceResolver, it doesn't make sense that it could resolve our namespace (since the namespace in the given example is a default namespace, with no prefix).

So just like in the XSLT, we're going to define a namespace prefix for the namespace used in the XML. We can do that with an XmlNamespaceManager (which implements IXmlNamespaceResolver). Unfortunately, and it escapes me why, we need to pass an XmlNameTable to the XmlNamespaceManager's constructor. Both XmlDocument and XPathNavigator have NameTable properties of that type, but we've already concluded that those won't help us here. Why is it required? I don't know, but it is. With that information, we can finally resolve our namespace:

var nav = ...; // Our XPathNavigator.
var nsMan = new XmlNamespaceManager(nav.NameTable);
 
nsMan.AddNamespace("bar", "http://www.bamccaig.com/foo");
 
...
 
var it = nav.Evaluate("/root/bar:level1/bar:level2/text()",
        nsMan) as XPathNodeIterator;

There is one other thing you might be interested in. The data contract API that we're working with (from the post, LINQ to SQL + Serialization) adds a default namespace to the serialization based on the .NET namespace of the serialized type. This happens to be of the form[1],...

http://schemas.datacontract.org/2004/07/Clr.Namespace

...where the namespace is Clr.Namespace. If you don't like that then there are various ways to control it. You can pass a named parameter into the DataContract attribute, however, since in my case the types (and attributes) are generated for me, I don't have control of what parameters are passed to the attribute. The alternative solution that I preferred (because it was easy) was to set an assembly attribute.[1]

Essentially, I opened up the Properties/AssemblyInfo.cs file that Visual Studio had generated for me automatically and added the following line(s):

// Setting the data contract namespace for the assembly...
// See http://msdn.microsoft.com/en-us/library/ms731045.aspx.
[assembly: ContractNamespace(MyNamespace.ContractNamespace.Uri,
        ClrNamespace = "MyNamespace")]

Note: You'll also need a using directive for the System.Runtime.Serialization namespace, but that should be a given. ;) The latest Visual Studio should resolve that for you if you ask it to politely. Alternatively, you could fully-qualify the attribute.

Note that instead of using a string literal, which I could have done, I elsewhere declared a constant field that could be referenced (both here, and earlier, when creating the XmlNamespaceManager), which makes it a lot easier to change the namespace URI in the future.

And that's all there is to it. Let me know if this does or doesn't work for you, or if you have any questions, concerns, or advice. :P

References

http://msdn.microsoft.com/en-us/library/ms731045.aspx

.NET + XPath + Namespaces

2010-04-30T16:25:00.024-04:00

Solved in a subsequent post.

This post is somewhat of a continuation to my previous post. I managed to get LINQ to SQL entities with custom properties serialized fine, however, the output contains XML namespaces which leads me to my next brick wall.

The application that I'm working on uses XPath expressions to match data within the serialization XML. XPath is namespace aware, which means that if you have a structure like this:

<?xml version="1.0" encoding="utf-8"?>
<root>
    <level1 xmlns="http://www.bamccaig.com/foo">
        <level2>Useful data!</level2>
    </level1>
</root>

Then the following XPath expression will NOT grab level2's text node:

/root/level1/level2/text()

The reason? level1 and level2 are in a different namespace than root. I have thus far failed to figure out how to work with this in .NET. In a past experience with this very problem not long ago, I was fortunate enough to be suggested a hack from the kind people on #xml on irc.freenode.net:

/*[local-name()='root']/*[local-name()='level1']/*[local-name()='level2']/text()

The asterisk (*) says to match all nodes (irrespective of namespaces) and the local-name() function returns the tag name (i.e., "root" for <root />). This works, or at least it did in the past, however, as you can imagine it's the Wrong Way(tm) and it's not particularly pretty either. The application I'm working on has a series of XPath expressions to maintain and I don't care to let this hack infect those.

Unfortunately, my attempts to solve this problem have so far failed. The XPathNavigator.Evaluate method accepts an IXmlNamespaceResolver, presumably for this very situation, but I'm unable to coerce it into resolving the namespace. :(

The most frustrating part of this is that the XML that it's [supposed to be] "navigating" is from the serialization process documented in my previous post. The XML was loaded with XmlDocument and the XmlNavigator was created with the XmlDocument's CreateNavigator method. The XmlDocument and XPathNavigator both have a NameTable property, which presumably maps out the namespaces within (it can be used to source an XmlNamespaceManager right in the constructor!), but somehow the XPathNavigator still fails to match to XPath expressions I'm feeding it. And get this: XPathNavigator implements the IXmlNamespaceResolver interface (and no, passing the navigator to the navigator does not work, I tried)!

I'm obviously missing something, but it's going to be nightmare to sort through all of the documentation, especially when I don't even know where to start...

Why, Microsoft?! Why?

Solved in a subsequent post.

LINQ to SQL + Serialization

2010-04-30T11:08:00.007-04:00

So here I am for my first /real/ post. I recently faced off with serializing LINQ to SQL generated types with custom properties. "I" have done this before using WebMethods by simply returning them:

[WebMethod]
public EntityA GetEntityA(int id)
{
    using(var db = new MyDataContext())
    {
        return db.EntityAs.SingleOrDefault(
                o => o.id == id);
    }
}

Custom properties serialize fine that way and show up on the client-side (in JavaScript). The serialization is all handled for me by ASP.NET though. The way I originally was doing serialization is like this:

var sb = new StringBuilder();
var xs = new XmlSerializerFactory(
        ).CreateSerializer(
        entityToBeSerialized.GetType());
 
using(var xw = XmlWriter.Create(sb))
{
    xs.Serialize(xw, entityToBeSerialized);
}
 
var serialized = sb.ToString();

This was working sufficiently for what I was using it for because the entities had no custom properties that were required in the serialization. Now I have entities that /do/ require their custom properties to be serialized. It took a little bit of Googling to find the solution (who has time to R all TFMs?). I found a solution that works (on MSDN, no less).

http://msdn.microsoft.com/en-us/library/bb546184.aspx

In short, you enable serialization for the LINQ to SQL model by opening it (the .dbml) in Visual Studio and setting the Serialization Mode (the only options I see are None and Unidirectional; guess which one to set it to). This tells LINQ to SQL to markup the generated types with serialization attributes. Next, I needed to markup my own custom properties with these same attributes.

[DataMember]
public int SomeCustomProperty
{
    get
    {
        ...
    }
 
    set
    {
        ...
    }
}

Finally, I had to change the way I was serializing (my original way isn't aware of, or at least doesn't seem to care about, the attributes we've applied).

var dcs = new DataContractSerializer(
        entityToBeSerialized.GetType());
var sb = new StringBuilder();
 
using(var xw = XmlWriter.Create(sb))
{
    dcs.WriteObject(xw, entityToBeSerialized);
}
 
var serialized = sb.ToString();

This is what is currently working for me, at least. ;) Best practices and "'Ur Doin' It Wrong, n00b!" are welcomed. :P

** EDIT **

Note that the output is slightly different from the output of the original serialization classes, so code that you've written based on the old XML structure may need to be modified. I now have to remap a set of XPath expressions... Sigh. In truth, it's resulting in slightly better code though, IMHO.

Introduction

2010-04-29T13:24:00.001-04:00

This is my first blog post evar (well, outside of stupid social networking sites)! About a week ago I was trying to solve a code related problem at j0rb and realized that I could be writing blog posts about the problems and solutions I encounter. It seems more and more I'm finding the answers that I'm looking for in blogs. Often, however, I stumble across poorly written blogs that only partially answer questions or contain errors. I generally enjoy writing and try to be meticulous so hopefully I'll be able to bring some value to the "blogosphere" (I use that term loosely).

To introduce myself, I'm a computer programmer from Ontario, Canada. I live very near to the Michigan, USA border. I predominantly work with Microsoft-based Web technologies at j0rb, but I prefer free/libre/open source and text-based user interfaces! In my spare time, I play lots of video games and experiment with various software development concepts (platforms, languages, APIs, etc.).

I have self-diagnosed obsessive-compulsive personality disorder; which is just how I like to say that I try to be a perfectionist. ;) I'm obviously not infallible so I won't claim to never make mistakes. "Grammar Nazis" are welcome here. If you spot a mistake in my spelling or grammar I would appreciate if you'd point it out (try not to go overboard though). Also, I tend to edit posts a lot on forums (to correct mistakes) so expect the same behavior here.

:wq