LinLog

Have I mentioned I hate computers?

pageer@skepticats.com (Peter Geer)

In case there was any doubt, I'm not a hardware guy. Yeah, I can do the basics - put a PC together from parts, swap out components if something goes bad - but it's not an area where I have a lot of experience or confidence.

So what should happen to me last week? Hardware problems galore! It started with one relatively small thing and just snowballed to both the PCs in my house being more or less out of commission.

The first problem was with Sarah's PC. It would occasionally soft-lock (meaning it wasn't completely unresponsive, but wasn't operational either) and need to be rebooted. A quick inspection revealed a bunch of I/O errors. My first instinct was "bad drive", but just to be safe, I figured I'd put the drive in my PC, just to make sure the SATA controller wasn't going bad.

Good thing I did, because after some testing, it looked like it was the controller! On Sarah's PC, heavy disk activity (copying a few dozen gigs of data) always resulted in an error sooner or later. But on my PC, I never saw any problem. I even tried swapping out the SATA cable with one from my box and it didn't make any difference. So I concluded that the problem was the SATA controller.

Meanwhile, some time during this testing, one of the drives on my PC went bad! I'm assuming the problem was in the drive's controller card, because the disk was still spinning up and not making any funny noises, but the system BIOS just wasn't seeing it. In fact, when the drive was connected, the BIOS would hang on detecting it! The worst part was that it was the new 750GB drive I bought just over a month ago! On the up side, I didn't have anything irreplacable on it, and was eventually able to revive it long enough to copy most of the data off (unfortunately I didn't have enough working storage for everything, but I got the stuff I really cared about). Of course I was just past NewEgg's return window, so I had to put in an RMA to the manufacturer.

So fast-forward to this past weekend. The new $20 PCI SATA/RAID controller for Sarah's box came in. I've never dealt with an add-on SATA controller before, but it was dead simple to install - put it in the PCI slot, connect the drive, and you're done. It worked great - for one evening. Then I started seeing more disk errors. This time they were failures from ext3_find_entry() rather than generic I/O errors, but that's still not too helpful. So now I'm back to square one on that.

On top of that, I've now been seeing lock-ups on my PC. Hard locks, too, as in the system goes totally unresponsive and the capslock light starts blinking. That, for the Linux people in the crowd, indicates a kernel panic. This one was a bitch to track down, too, because it was happening randomly as I did things from within X11, but didn't seem to happen on the command line. And since kernel panic output gets dumped to the console, I couldn't see what was going on (because you can't switch from an X display to the console when the system is crashed). So I ended up leaving the system at the console and eventually got a kernel panic again. Turns out it was coming from the USB subsystem somewhere. I still don't know if it's one of my peripherals, my hub, or the USB controller, so I'll have to do some experimenting.

It's weeks like this when I really appreciate the philosophy of technical people who say, "Forget building it, just buy from a system vendor so that you can have it repaired under warranty." Fixing problems like this is sickeningly time consuming and generally not much fun. Especially if you're not a hardware guy.

More laptop fixups

pageer@skepticats.com (Peter Geer)

As I mentioned the other day, I recently acquired a hand-me-down laptop. My sister-in-law bought herself a shiny new Dell and gave me her old Compaq Presario R4000 with the dead battery.

So I'm currently in the process of fixing this system up. So far I've just wiped the drive, laid down a fresh install of Windows XP SP3, and installed the drivers and some standard applications. It's actually not a bad little system - Athlon XP-64 processor, 80GB hard drive, DVD burner. Hardly state of the art, but it beats the pants off my old Dell Inspiron B120.

Aside from the battery (which is gonna cost me about $75 to replace), there's only one problem: memory. The system comes with 512MB of RAM, which was fine for the time, but is a little small now. The problem is that the system has two 256MB RAM modules - one of which is under the keyboard, which is really annoying. And they're not especcially cheap modules, either - it takes DDR PC2700 sticks, which go for about $40 per gig on NewEgg. By way of contrast, I recently got another 1GB stick of DDR2 PC5300 to max out the RAM on my Dell, and it only cost me $12 with free shipping.

So I'm debating how much to get now. In better times, I'd just spend the $80 and max out the memory. But times are tough, and it's an old laptop, and I'm already spending $75 to get a new battery. So maybe I'll just stick to 1GB, or perhaps even half a gig. It wouldn't be quite so annoying if they'd just used 1 stick in the first place. As it is, I'm going to lose half the existing RAM if I upgrade either DIMM slot, so I'm not going to get out of this as cheap as I'd hoped.

Either way, it'll still be nice to have another laptop around, so that Sarah and I aren't fighting over the one. And even if I max out the RAM, it's still way cheaper than getting a new system, so it's still a win-win situation.

No more dollar store USB cables

pageer@skepticats.com (Peter Geer)

Note to self: no more buying cables at the dollar store.

Last week I was in the local dollar store looking for pocket-sized packages of tissues and lint rollers (because there's no point spending a lot of money on that sort of thing), when I came across some of those retractable USB cables. You know, the kind that have the spring-loaded roll so that they wind themselves back up - kind of like cheap blinds. I probably should have known better, but come on - it was only a dollar!

Anyway, this afternoon I tried using that cable and it didn't go so well. I attempted to use it to connect my 320GB external USB drive to the hand-me-down laptop that my sister-in-law just gave us after buying a new system. I was using SystemRescueCD and PartImage to take a drive image of the original installation - you know, just in case - and the performance was...let's just say "disappointing". More specifically, the estimated time to completion, on a 75GB drive, with about 40GB used and no compression on the image, was around 11 hours. For those with no point of reference, using the same external drive to image my other laptop, which has about 30GB of used disk, takes maybe half an hour.

And what's worse, partimage kept hanging every 5 or 10 seconds, so even that estimate wasn't accurate. I left the laptop to image while we went on a wine tour of Seneca lake this afternoon, and after being gone for about 7 hours, partimage still said it had 6 hours of work left.

So, just to check, I tried it with my regular USB cable and, sure enough, it's running along at 667MiB/min, estimated completion time 55 minutes. It's still running a little choppier than my other laptop, but not too bad.

So I guess this discredits my old opinion that cables are cables and they either work or they don't. Turns out there are different levels of "not working". Like I said, I should have known better.

Let the experiment begin

pageer@skepticats.com (Peter Geer)

Well, the new year is upon us, we have a new president, and the economy is in the toilet. So I figure this is as good a time as any to try something new. That's why I'm switching to Windows.

Of course, I'm not switching switching. My home desktop is still running Kubuntu 8.04 and will be for the foreseeable future. However, a couple of weekends ago I burned a System Rescue CD, shrank the Kubuntu partition on my laptop, and set it up to dual-boot Windows XP. I also resurrected the long-dormant Windows partition on my PC at work. I'm still keeping Kubuntu around on both systems (at least for now), but I'm making an effort to actively use Windows more often than not.

There are a number of reasons for this. For one, I've been getting a little fed up with Linux lately, especially after the last Kubuntu upgrade. I'm thoroughly unimpressed with KDE 4, I'm getting tired of things crashing randomly under KDE 3, and I find that I just don't use or want to use much Linux-specific software anymore. It's also been a while since I felt really connected to the community and I was never really a "true believer" in Free Software or the power of Open Source in the first place. So I just feel like I don't have as much of an anchor in the Linux world anymore, despite using it every day.

The Windows world, on the other hand, seems to have a lot to recommend it these days. For instance:

It's the native platform for .NET. I did a little .NET work at my last job and quite liked it. I've been wanting to get deeper into .NET and C# for a while now and using Windows will allow me to do that more easily.
Yes, I know Linux has Mono, and that's all well and good. But let's face it - it's just not the same. For one thing, Mono isn't as complete as Microsoft's runtime. For another, the tooling just isn't there - MonoDevelop is a fine IDE, but it can't hold a candle to SharpDevelop, let alone Visual Studio. And, of course, Mono isn't as marketable, as most of the companies looking for .NET developers are working on Windows.
Powershell. I've been wanting to try out PowerShell for a while. I keep hearing about it on .NET Rocks and RunAs Radio and it seems really cool. Again, there is a Mono port (called PASH) in the works, but it appears to be only about half done, whereas PowerShell on Windows is pretty much mainstream now.
It has everything I need and more. I don't remember where or when, but within the last few months, I came to a realization: pertty much all the software I care about runs on Windows. Granted, there are a many programs I use on a regular basis that are UNIX-only, but all the stuff I really need is cross-platform: Opera, Firefox, Komodo Edit, Vim, PHP, Python, Apache, MySQL, PostgreSQL, etc. There are even Windows ports of most of the common UNIX utilities (awk, sed, grep, etc.), not to mention Cygwin. As for the things that don't run on Windows...well, I found I'm just not all that attached to those programs, and there are Windows programs that do the equivalent anyway. So really, even with switching to Windows, my toolset doesn't change that much.
It's just more marketable. Let's face it - the economy is bad and I haven't the money or the inclination to move to where the jobs are. In other words, if I wanted to live in New York City or California, I wouldn't be working in Rochester right now. So it makes sense for me to develop my skills in a direction that's in demand in my area.

From the online research I've done, Java and .NET seem to be the most in demand in the Rochester area. Java may be bigger, but the fact is that I know jack about Java, whereas I have some experience with VB.NET and C#, so that seems like a more viable path. By contrast, there's usually only a handful of PHP jobs available, and apparently PHP developers are a dime a dozen anyway. And despite all the hype, I've only ever seen 1 Ruby job available within 50 miles of my house.

So off I go on this new adventure. Hopefully it will all go well and I'll emerge a happy and productive .NET developer. The only problem is I might have to change the name of this blog.

PowerShell highlighting in Vim

pageer@skepticats.com (Peter Geer)

Well, I feel stupid. I just finally got syntax highlighting for PowerShell to work in Vim. I did a little Googling and found my solution in the PDF document linked form this article. Why you need an 8 page PDF to explain this is a completely different issue.

Turns out I was just missing some lines in my filetype.vim file. I already had the syntax, indent, and file type plugins, I just didn't think to add the code to activate them (for some reason I thought that stuff was loaded dynamically). However, page 6 of that PDF gave the answer:
" Powershell au BufNewFile,BufRead *.ps1,*.psc1 setf ps1
I just added those lines to my filetype.vim, below the corresponding entries for Povray files, and voilà! Syntax highlighting now works.

Turning paper into a DVD jacket

pageer@skepticats.com (Peter Geer)

Just on a random link propagation note, today I found a very handy guide on making your own CD jackets out of paper. I was burning a couple of DVDs and didn't have any empty cases laying around, so I figured there must be a good way to fold them up in a piece of paper. A little Googling turned up this handy guide. It has a PDF of instructions and even measurements for use with different sizes of paper (though you can pretty easily eyeball it, especially with US letter).

Just thought that was a useful little piece of information. I'm going to have to hold on to that one for those frequent occasions where I have a disk and nothing to put it in.

Apparently we were "hacked"

pageer@skepticats.com (Peter Geer)

You know that weird jump in auto-incremented IDs we saw in our database at work the other day? Well, we discovered the cause: apparently we were "hacked".

I use the work "hacked" in the loosest sense of the word. The culprit contacted us the next day, making a number of demands and issuing veiled threats. I may get into the details in another post (it's a long story), but the point is that he appeared to have, at best, a script kiddie level understanding of security. He pointed out a number of flaws in the site, but the ID incrementing was the only one that could actually have caused us any real trouble - and he didn't even mention it! We assume that he actually did that by accident and didn't realize the potential implications if we ran out of ID numbers.

The flaw allowed the ID incrementing was actually quite simple. No SQL injection or cross-site scripting. It was a simple case of passing too much data into a function.

Here's how it works. Our site, which runs on PHP and MySQL, is built on a custom MVC framework. It uses a custom Active Record-style ORM, which is where the flaw lies. The ORM performs database updates and inserts by validating the object's fields against the database schema. Basically, it reads the schema from the database, compares that with the data to be updated, and constructs the SQL accordingly. So when you execute the save() method on an object, it will save the values for the fields that appear in the table schema, but ignore any other fields in the object. The ORM also has static update() and insert() methods that take an associative array, mapping the indexes to field names and performing this same validation. So if you have an array of data, only some of which maps to actual columns in the underlying table, you can just pass the whole thing and not have to go through and separate out the fields you need to save.

That last point is where we got in trouble. We have a method that adds items to our media table. It takes an array of data, does some sanitizing and validation, calls the insert() method to add it to the media table, and adds appropriate records to other tables. The problem was that, in the place where this was called most frequently, we were passing in $_POST as the data array. And while this method did sanitize the fields that we wanted to add to the database, it didn't check for extra fields that just happen to be valid fields in the media table. So, to make a long story short, if you were to put an "id" field in the POST and assign it an integer value, our ORM would happily add that field and value to the INSERT statement it sent to the database.

Of course, this was easily fixed. In fact, it wasn't even hard to find. I did a fair amount of work on our ORM at the beginning of the year, so once I made the connection between the "hacker" and the ID number jump, the source of the bug was immediately obvious. It's just one of those things that nobody ever thought to check until it became a problem.

So the moral of the story is: security is all about attention to detail. Following the "rules" is all well and good, but it's not enough. In our case, we were sanitizing data to protect against XSS attacks and using PDO prepared statements to protect against SQL injection attacks, but it wasn't enough. By forgetting to check for unexpected additional input, we left ourselves open to a completely different type of attack. Of course, it's a significantly less serious class of attack - maxing out our auto-incrementing IDs is recoverable, if annoying - but it's still an issue.

With any luck some good will come out of this. I think we've all learned to be a little more mindful of such issues. And perhaps this will act as a cue to management that maybe - just maybe - it would be better to do some actual testing and review of new code before it's released, rather than just pushing things into production and hoping they work.

Up 79 million in 20 seconds

pageer@skepticats.com (Peter Geer)

We had one of those weird experiences at work today. The kind where something strange happens, and nobody knows how or why, and when you look into it, it doesn't even make sense.

Basically, the auto-numbered ID field on the table we use for media items (videos, pictures, etc.) jump up noticably this morning. And by "noticably" I mean it went from about 1 million to about 80 million. And when we looked at the timestamps, the jump happened in about 20 seconds. The IDs created before 8:38:12 AM were in teh 1 million range, and the ones createdafter 8:38:32 AM were in the 80 million range.

So the obvious question is: how did this happen? It doesn't look like it was caused by actually adding 79 million records to the table. There were only about 800,000 records total, and no indication in our moderation logs of any mass deletions. We didn't get any indication of increased server load either. In fact, the only reason we even noticed it is because the media IDs are in our URLs. I kind of doubt a bot could have created 79 million new media items in 20 seconds without at least generating a Nagios warning. In fact, I doubt our master database server could handle 79 million writes in 20 seconds.

So what does that leave? User error? Nobody with access to the database server was even working at 8:30 in the morning. Random MySQL screw up? Maybe, though that's a really wierd random error. Something else? Who knows...?

After sniffing around the server and tossing out ideas for 30 or 40 minutes, we ultimately gave up. It's a little disquieting that we don't know what happened, but we really can't justify spending all that much time on this. It's a very weird problem, but nothing is broken and we all have more important things to worry about at the moment.

Edit: Found the cause. Apparently we were hacked.

Kubuntu Intrepid: Another failed upgrade

pageer@skepticats.com (Peter Geer)

Well, that sucked.

I upgraded my Kubuntu box at work from 8.04 to 8.10 on Monday morning. It did not go well. Not only did the experience waste several hours of my time getting my system back to a state where I could actually do some work, it left me feeling bitter and fed-up.

Not that the upgrade failed or anything - on the contrary. The upgrade process itself was relatiely fast and painless. So, in contrast to some of my previous upgrade experiences - which have left systems completely inoperable - this wasn't that bad. It's just that, once the upgrade was done, nearly every customization I'd made to my desktop was broken.

Broken Stuff

As for the breakages, they were legion - at least it felt that way. The 2 most annoying were the scrolling on my Logitech Marble Mouse trackball and KHotKeys. It turns out the mouse scrolling was fixable by adding a line to my xorg.conf to disable some new half-working auto-configuration feature.

KHotKeys, on the other hand, was a lost cause. From what I've read, it just plain doesn't work right in KDE 4. So, since key bindings are an absolute must-have feature for me, I worked around it by installing xbindkeys. This works well enough, but it's a huge pain in the neck. Now, not only do I have to recreate all my key bindings, but I have to look up the DBUS commands for all those built-in KDE functions rather than just picking them from a list.

Another annoying one was that the upgrade somehow broke the init scripts for my MySQL server. I don't know how the heck that happened. I tried uninstalling it, wiping the broken init scripts, and reinstalling, but they weren't recreated, which seemed odd to me. I eventually ended up just doing a dpgk --extract on the MySQL package and manually copying the scripts into place.

On another weird note, KDE and/or X11 has been randomly killing the buttons on my mouse. I'll be working along fine and suddenly clicking a mouse button will no longer do anything. It still moves, and the keyboard still responds, but clicking does nothing. Restarting the X server resolves the problem, but that's cold comfort. It seems to happen randomly - except for when I try to run Virtual Box, in which case it happens every time the VM loses focus. Fortunately I'm more of a VMware person, so that's not a big deal, but it's still disquieting.

KDE4 In General

The other big pain-point is KDE 4. To be perfectly blunt, I don't like it. It has a few neat new features, but so far it doesn't seem worth the effort to upgrade.

The good parts that I've noticed so far seem to be small. For instance, Dolphin has a couple of nice enhancements. The one that sticks out is the graphical item-by-item highlighting. It allows you to click a little plus/minus icon to select/deselect an item, so that you no longer need to hold the control key to do arbitrary muliple selects. The media manager panel applet is nice too. It pops up a list of inserted storage devices and allows you to mount and eject them. I have to admit that I also really like the new "run" dialog. It does program searching much like Katapult, but makes it easier to run arbitrary commands and select commands with similar names. While it doesn't have some of the cool features supplied by Katapult's plugins, it's still quite good.

On the other hand, there are a lot of things I don't like (not counting the breakage). For one, I think the new version of Konsole is a huge step backward. I can't access the menus with keyboard shortcuts, the "new tab from bookmark" feature is MIA, the session close buttons are gone, and generally everything I had gotten used to is missing.

And then there's the new "kickoff" application menu. I'm getting slightly more used to it, but I still don't like it. It just feels a lot slower to access items using it. This is only made worse by the "back" button for browsing sub-menus, which is extremely hard to click when you're in a hurry (hint: Fitt's law doesn't apply on multi-monitor setups).

As for the "cool" new look of KDE 4...I'm not a fan. Maybe it's just because I don't have any of the fancy desktop effects turned on on my system (a side-effect of the crappy integrated video card that's part of my tri-monitor setup), but I just don't think it looks good. Yeah, the bare desktop itself is kind of nice looking, but the window theme is ugly as sin. It's one of those "brushed metal" sort of looks, which I find even more depressing than Windows 95 gray. It's too dark for my taste and far too monochromatic. I also find the active window highlighting to be way too subtle to be helpful. The icons also leave something to be desired. They look nice, but they don't look distinct - even after a week, it takes me a second to figure out what some of them are supposed to represent. It kind of defeats the entire point of icons.

As for the much touted Plasma, I'll grant them this - it is pretty. The panel and desktop plasmoids do pretty much all look nice. Not that it matters to me, though, because I never see my desktop - it's always covered with work. And while the various applets and widgets may look pretty, approximately 90% of them are completely useless. That's the problem with all desktop widgets for any platform. I find that if a desktop widget actually provides enough valuable functionality to justify leaving a space open for it on the desktop, it's job is probably better served by a full-fledged applicaiton. And if it's not important enough to make constantly visible, then why bother to put it on the desktop at all? I'm never going to see it, so I might as well save the RAM and CPU cycles.

Conclusion

Overall, I guess Kubuntu 8.10 and KDE 4 aren't bad systems. But to be honest, I'm not impressed. For the first time, I think that the new Kubuntu is not an improvement. In fact, I have no plans to upgrade the 3 Kubuntu boxes I have at home any time in the forseeable future.

The thing that's most disappointing to me about the upgrade to KDE 4 is that it totally defeats my purpose in switching to KDE in the first place. When I switched from the ROX desktop to KDE back in 2005, my main reason was that I was tired of having to build my own desktop. ROX was great, but it was a small community and just didn't have the range of applications and degree of integration that KDE had. You see, I always had this crazy idea that I could just use all KDE applications and everything would be tightly integrated and work well together and there would be harmony throughout my desktop.

However, more and more I've been finding that that just isn't true. Part of the problem is that lots of KDE applications just aren't that good - many of them are missing functionality and have stability problems. I find myself using fewer KDE applications all the time. I dropped Quanta+ for Komodo Edit; I tried to like Konqueror, but it just doesn't hold a candle to Firefox or Opera; I recently tried to become a KPilot user, but was almost immediately forced to switch to JPilot; I finally got fed-up with Akregator and am just using the RSS reader in Opera's M2 mail client; I still use KMail, but not because I particularly like it - I just dislike it less than M2 or Thunderbird. In fact, I think the only KDE app I would actually miss is Amarok. (K3B is very good too, but I don't burn enough disks to care what program I use, just so long as it works.)

So now I'm starting to wonder: What's the point of using KDE? If I'm not using many KDE applications, and most of the ones I am using could be easily swapped out, it seems like there's nothing keeping me with it. Maybe I should just switch to GNOME. Or maybe Windows. I have been wanting to get more into .NET development, and my tollerance for things not working has been falling over the years, so Windows is sounding better all the time.

I think next weeek I'm going to have to reinstall my work machine. Maybe a fresh install and a fresh KDE profile will give me a better experience. Or perhaps I'll ditch Kubuntu and go for straight Ubuntu with GNOME. Or perhaps I could take another look at ROX. I don't know. And while I'm at it, I think I might reinstall that old Windows partition I still have on that machine. Maybe some time playing with a nice clean install of XP, or even Vista, if we have a spare copy, will give me a little perspective.

Weird index breakage

pageer@skepticats.com (Peter Geer)

Here's an odd one. So I was trying to run a simple query for a data import I'm working on. I've got about 11,000 rows in the media table with a site_id of 2 and I wanted to see which media types were represented. So I ran this:

SELECT DISTINCT media_type FROM media WHERE site_id = 2;

To my surprise, this returned...nothing! No rows at all. So I took off the DISTINCT, just to check, and got back all 11,000 rows. What the heck?

Something was obviously wrong. This should never happen - if there are rows with the correct site_id, that query should always return something. So what now? Might as well look at the query plan:

mysql> explain extended select distinct media_type from media where site_id = 2 \G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: media type: range possible_keys: idx_media_site_media_type,idx_frontpage key: idx_typesiteid key_len: 2 ref: NULL rows: 15 Extra: Using where; Using index for group-by 1 row in set, 1 warning (0.00 sec)

mysql> show warnings \G *************************** 1. row *************************** Level: Note Code: 1003 Message: select distinct `cms`.`media`.`media_type` AS `media_type` from `cms`.`media` where (`cms`.`media`.`site_id` = 2) 1 row in set (0.00 sec)

Nothing out of the ordinary there. It's using the idx_typesiteid index, which is correct. Maybe that index is broken. Let's try another one:

select distinct media_type from media use index (idx_media_site_media_type) where site_id = 2;

Yup, it worked that time - I got 5 rows back, which is about what I was expecting. I get the same results if I use
IGNORE INDEX (idx_typesiteid) as the index hint. So idx_typesiteid must just be broken. I should probably fix that, but it was getting into peak hours for our site and I didn't want to run an OPTIMIZE TABLE or anything that would lock the media table.

But wait, there's one other weird thing. Just for the heck of it, I tried running the query with a few different index hints. And guess what: the query still works if I use FORCE INDEX (idx_typesiteid). Does that make sense to anybody? So if I ignore that index it works, and if I force the same index, it works, but if I give no hint, it claims to use that index and the query returns no data. I got nothin' on that one.