Gog Blog

Log Parsing

2014-03-26T20:51:00.000+00:00

I love log files, they have so much value.
I work with IIS sites a lot and IIS log files are wonderful things. I also enjoy looking at Squid, Apache, and ELB logs. Really, I enjoy it.

I have two tools I especially love for IIS log file parsing.
This Log Parser is amazing; it gives you SQL syntax for quering text files including IIS log files or more generally CSV files. I think it does more but those two are the ones I'm most often dealing with. It's really fast at processing too.
The other tool I love for taking a peek at IIS log files is Microsoft Excel. Excel is a wonderful bit of software and doesn't mind chewing on large log files. I import them as data from a text file, space delimited, and then format the whole thing as a table. This gives me filtering and sorting on the columns which allows me to answer questions like 'which request with a response code in the 200s takes the longest', 'which requests are 404-ing most often'.

Alongside graphing solutions (I'm a fan of Graphite) parsing the log files is powerful; the graph will show some weirdness such as an increase in average response time, or a large number of errors and the log files will give you the details. Hopefully you'll find an answer to the deviant behaviour there.
I've found this tutorial to be an excellent introduction to Graphite and it has some tutorials on setting it up and configuring it.

In general I find that graphing solutions will give you a question, the logs will give you the answer.

Management Studio Feature Request

2014-03-20T18:48:00.000+00:00

I spend quite a bit of time in Sql Server Management studio in bursts, at the moment I'm there quite a lot because I'm creating an SSIS package.
I find that I create tons of tabs which roughly group together a bunch of related, useful queries. They're nothing I want to keep long-term so I seldom save them but after a while I forget which bit of sql is in which tab. Things get worse when I have more tabs than can be displayed and the one I want is scrolled off the screen somewhere; "Hmm, was that query in SQLQuery20? *click* Nope, not that one. What is this stuff?".

I'd love to have two features here:

Rename a tab without having to save it. I really should save it but sometimes I really only need them for an hour or two, I don't want the clean-up afterwards;
Save a whole session for later. So say I'm working on the SSIS project, I have lots of tab open which are useful for now. But then I need to do something else, or management studio is running slowly so I want to close it and re-open it later. Wouldn't it be cool if I could save the whole session as it is and close management studio. Then when I get back to it I can re-open my SSIS project and have it reconnect to the databases I had, re-open all the tabs back to how they were including their names and positions;

That'd make me happy.

2013-01-31T19:12:00.000+00:00

Oh look, it's been a whole year since I added an entry. Its apt then then I found a new fad to start playing with.

I've always liked the idea of a diary; I like the thought of having written hind-sight on seemingly insignificant days. Problem is I hate *writing* diaries, especially 'real' ones on dead tree.

Here's the proposed solution for me: Oh Life.
I joined two days ago and I've added two entries. I'm two for two, how could it go wrong?

A feature I like about this is reading a random entry and it even emails you a random entry from time to time. Hopefully the site will keep my entries private...

Ever wondered what a local variable is?

2012-01-17T18:04:00.000+00:00

I managed to mumble an answer about a variable with a very limited scope, which wasn't far from the truth.
In any case, here's the answer to a question no-one else really though to ask but is quite interesting when you get to read the answer:

What is the defining characteristic of a local variable?

On the importance of good, readable code

2011-11-17T15:18:00.001+00:00

Browsing through a solution I'm involved with I found a line that looks like this:

Utils.Reshape(xsltFileStream, out xslt);

It caught my eye because I have no idea what that method does based on that bit of code. It reshapes something (perhaps it reshapes the Util?), it receives a Stream and out's an XslCompiledTransform.
I could guess that it runs some xml through the xsl transform but it doesn't have any xml as a parameter and Utils is a class name, not an object (therefore making the method a static method, it doesn't use some xml from an instance variable).

So what does it do? Here's the method (I really should get a C# syntax highlighter...):

public static bool Reshape(Stream source, out XslCompiledTransform target)

{

XmlTextReader xr = new XmlTextReader(source);

target = new XslCompiledTransform();

target.Load((XmlReader)xr);

return (true);

}

So, it creates an XslCompiledTransform from the Stream you provide. Then it returns 'true'.
To be fair, this is very old code (I checked the source control history) and it's probably been changed (I didn't say refactored) a million times by different people to get into this state. There aren't any automated tests in this solution so it's scary to change code in this situation; countering that though is the fact that this code isn't in a library that's shared with anyone else and it's only called in one place in this solution.

What would I change?

That's a silly method name, 'LoadXslTransformerFromStream' is better but I bet there are even better names still. Descriptive names are great, descriptive and succinct get extra points but really long method names don't slow your code down so don't be afraid of them.
Returning a hard-coded value (that isn't used anyway) = FAIL. Make the method return an XslCompiledTransform, lovely.
Are utility classes really a good idea? I'm undecided but shouldn't they be encapsulated in a relevant object rather than floating as a static method in a helper class?
That cast to XmlReader could be improved - XmlTextReader is an XmlReader.

Anything else?

EDIT: Never mind a syntax highlighter, I'll try .NET Pad for a bit. Here's the Reshape method with some formatting.

Recruitment email gone wrong

2011-11-17T14:03:00.001+00:00

If you sign up to a recruitment site/agency you get called for ever. Even if they found you a job they'll call you in a year or so to make sure you're still happy or if not then they can help you find another role (and make some money for themselves). Anyway, you're meant to be friends with them so that you go back to them; I got an email this morning that didn't really warm me to the recruitment site:

Are you still working as NULL? Your profile was last updated 22/03/2004 and there are currently positions open that you may be missing out on.

Perhaps they need some fresh programmers to work on their mass mailer?

Flickr, Mendhak and the full screen option

2011-11-17T08:05:00.001+00:00

What's a blog for if not for self-promotion. I'm feeling altruistic too so I'm going to promote my friend Mendhak!
I've been a Flickr user for years; I love taking photos although I consider that I'm only a mediocre photographer. I love looking at portrait photos and Flickr was the first site I found that let me indulge in other people's photos in a rich way and with an abundance of photos. I've recently found 500px which has a nicer interface but you always prefer that first site, it'll take a while to port over to anything else.

Anyway, my friend Mendhak also shares a love of photography and of coding and has produced a signature generator and a full screen show-off-your-photo generator.
That's enough promotion for Mendhak, here's the self-promotion. Behold, leaves in autumn what I took (best viewed full screen on the biggest screen you own).

Thanks Mendhak : )

IEqualityComparer with Linq

2011-11-11T15:47:00.001+00:00

I was completely thrown by a simple Linq issue today.

I was reading in two lists of strings (user names as it happens) and I wanted to know which ones from the second list weren't in the first list. Simple problem right?
First stab was to iterate over the second list; if FirstList.Contains(string) then I wasn't interested in that entry but if it didn't contain that entry I was interested in it and would squirrel it away.
Multiple runs of this code provided a different number of results despite the source data being the same. Confusion reigned for a bit, scratched head and poked the code a few times until I realised I wasn't telling the .Contains call how to figure out if one string was the same as the other.
That sounds simple but C# wasn't making any assumptions, I don't know what the default equality comparer was doing but providing my own implementation of IEqualityComparer<T> (T was a string in my case) solved this problem.

I love the satisfaction of solving a problem like this but good grief I felt a bit silly to be stumped by something so simple : )
For full closure I really should have a go at finding out what the default comparer actually does, no time at the moment though.

EDIT (12th November, 2011): Spent some more time on it. I haven't found what the default comparer does yet however Microsoft have already created StringComparers for me.
This is an abstract class that has a couple of static properties that expose the concrete examples. In my case I would have used StringComparer.OrdinalIgnoreCase as my comparer instead of writing my own.

Thanks Microsoft : )

First Post w00t w00t

2011-11-11T15:41:00.001+00:00

A beginning is a very delicate time.
Know then that it is the year 2011 and this is the first post. Who can say how long I'll be posting, I may get bored and stop.

In any case, this is slightly more interesting than most first posts and I shall probably be the only one to read it whilst it's fresh.

See you later?