See Sharp C#

On dealing with EC2 Micro's bursting

Thu, 15 Sep 2011 21:20:00 -0400

My girlfriend recently decided to go on a long trip, taking lots of pictures along the way. Storing them was getting to be a problem, so I figured that hosting them on one of my EC2 instances would be a decent way to go about things. This, in theory works well, in the sense that she can easily push files up, and making a gallery out of those files will be relatively simple.

So I did just that. And the upload went fine, and the gallery creation went fine. For the most part. You see, the software that I'm using to make galleries out of the images creates thumbnails at runtime (caching the results on disk), and with huge folders of images, this becomes quickly taxing on a micro instance, especially in realtime.

I needed a solution that would let the thumbnails be generated, but at the same time, I needed to be able to not have my micro instance be CPU-throttled all the time during the generation of the thumbnails.

Easy way out? I made a quick C# program that constantly scans the upload folder for any files that haven't been thumbnailed by the gallery web application, and adds work to its queue if it finds new pictures. Between each thumbnail process, the thread sleeps ten seconds before starting on the next thumbnail; if the queue is empty, it'll scan the directory every five minutes. It'd be better if I could make this a service, instead of a program, but for a quick-and-dirty solution, this works just fine.

Twilio Transcription

Sun, 11 Sep 2011 22:27:00 -0400

Today I had some free time to play around with Twilio, and decided to check out the transcription service that is offered. It's relatively simple to work with, you write some 'TwiML' (Twilio XML, essentially) either directly or through one of their helper libraries, and the service records what the user is saying for up to two minutes. After the recording is done, Twilio will transcribe the text (this is not done in realtime) and will post to the TwiML that you specify it to a link to the recording MP3, the transcripted text, and a link to the transcripted text.

This is pretty neat stuff. I really love the idea. However, there are some kinks they need to work out of the system. One of my attempts, which was along the lines of 'This is another transcription attempt. Hopefully this will be posted to the database. Ending transcription now.' was transcribed as 'Mother****er call hopefully this will be able to do that at this point in time'. Not exactly what I had in mind (and Twilio helpfully had the full curse text). To be fair, the saved call sounds pretty terrible (it's a 16bit .wav file at 8kHz, not the best and there probably was some artifacting from being on a cellphone) and would have been reasonably difficult for me to interpret if I didn't already know what it was.

Moral of the story: Check to see how services you interact with deal with curse words. If you're trying to develop a family-friendly or otherwise somewhat professional service, ensure that you have proper filtering in place to deal with potential issues.

It's a cool service, but you'd have to put some effort into determining how to monetize around it. The transcription service goes for $0.05/minute, and with an additional $0.01/minute charge already in there for the call itself, you'll have to figure out how to pay for the service.

More Amazon EC2 Bits

Wed, 31 Aug 2011 11:45:00 -0400

Amazon recently sent me an alert regarding a worm that's going around, called Morto. The worm basically works by attempting to connect to the default Remote Desktop port (3389), and then attempts a few simple username/password combinations to login. If successful, your machine will need to be cleaned.

I was a bit worried, because I realized my EC2 instance was still running Remote Desktop against the default port. I changed it, and updated the EC2 security group settings to allow inbound traffic to the newly selected port, disconnected, and rebooted (to make the security group settings come into effect).

Only after I rebooted did I remember that there was yet another firewall that I forgot to configure, the Windows firewall. I was completely locked out of my machine.

Doing a bit of searching on the net revealed a few other people that had run into the same issue. The list of steps that you should follow is thus:

Shut down the instance that you need to restore access to
Detach the volume that is attached to root
Create a new instance (in the same zone as the first instance, otherwise the new instance cannot attach the volume)
Attach the detached volume to the new instance
Get into the new instance, set the detached volume to 'Online' within Windows
Use Regedit in offline mode to edit the registry values associated with the Firewall (make sure to get Domain, Private, and Public on all ControlSets), ensure that you unload the hive after completion to save the registry.
Turn the drive to 'Offline' within Windows
Shutdown the new instance
Detach the old volume from the new instance
Attach the old volume to the first instance, ensuring that you set the device to /dev/sda1
Start the old instance
Connect with Remote Desktop, add new rules to the Windows Firewall to allow for the port that you need
Ensure that Windows Firewall is running, you may need to edit the registry again and reboot to start it properly

This should get you back into the machine without having to go through AWS premium support.

Now, the other tip I have: The micro instance I use is extremely weak, CPU-wise. I noticed that configuring items while connected via RDP was unbearably slow, and then I realized that the RDP window was far larger and used far more colors than I really needed. I found a site which contains many RDP settings which you can set within the text-editable RDP file, and so I created a new RDP file with the server address, new port number, window height, window width, and color depth (session bpp), which makes my RDP sessions on my micro instance very snappy. Silly that I didn't realize this before, but now I know.

Sudoku Solver in Silverlight

Fri, 08 Jul 2011 10:02:00 -0400

I wrote up a quick Sudoku solver in Silverlight, using the backtracking technique. There's probably some more optimization to do, and I really need to figure out how to properly get the UI thread updated in realtime while it's solving it, but I'm pretty happy that I was able to at least get this going.

Quick bit on Amazon EC2

Wed, 06 Jul 2011 20:56:00 -0400

This blog is hosted on a micro instance at EC2. I figured this would be enough, and so far, with the tiny [zero] amount of traffic I've been getting, this has proved perfectly fine.

However, I was also hoping I could do some experimentation with Silverlight or general .Net development on this instance. On attempting to install VS2010 Express on this instance, I realized things were going extremely slowly. I realized that it just wasn't going to happen when I looked at this documentation, which really showed me that the CPU (aside from the RAM) of my micro instance just wasn't going to cut it to run a heavy IDE like Visual Studio.

My new plan: I'm currently setting up a large instance, which I'm going to run any development on, and then deploy to this machine. Hopefully, this will help me keep costs down; instead of having a gigantic constant cost to run a large instance indefinitely, I'll just kick up the large instance when I want to do some experimentation ($0.48/hr, not bad for a pretty hefty machine), and keep the micro instance constantly up to deal with hosting whatever I put together.

Next up: I'm going to try and create a Sudoku solver in silverlight (probably done before) just to see how difficult it is to put together.

On Diagnosing Infinite Loops with WinDBG

Tue, 05 Jul 2011 23:17:00 -0400

Imagine this scenario: With not much traffic hitting your site, the entire server has ground to a halt and the CPU is pegging at 100%. No requests are being serviced, despite IIS taking all available CPU time. What's going on? One potential issue is that you have a poorly written page or method with an infinite loop.

I ran across this very issue in a production version of a page a few months ago. After a seemingly random certain amount of time serving pages normally, our servers would start to individually stop responding to requests, with a CPU that was completely saturated. Our own internal analysis ability was woefully inadequate at the time, so we used an MSDN support ticket to ask for help in diagnosing the issue.

Once we had a server in the 'bad' state, we pulled a DebugDiag dump as per Microsoft's instructions, and shipped it off. About a week later, they told us that they thought they had spotted the issue: An infinite loop. They even were able to pinpoint the general code and give us a stacktrace of the problematic issue.

After fixing the issue, I got to thinking: How were they able to spot the issue? Was there something that we could do the next time and try and shave some of that turnaround time down? Thankfully, one of my teammates was thinking the same thing and sent an email to Microsoft. They provided a link to WinDBG, and gave us a few quick pointers on how to use it.

After receiving the pointers, I played with the dump and WinDBG quite a bit and came up with a process that my teammates can use if there is a similar situation in the future:

Get the server into the 'bad' state
Pull a DebugDiag dump from the server
Open up WinDBG, and open the dump from there
If necessary, run this command in WinDBG to setup the downloading of Microsoft Debugging symbols: .sympath SRV*c:\localsymbols*http://msdl.microsoft.com/download/symbols
Run this command to setup SOS: .loadby sos mscorwks
Run this command to see long-running threads: !runaway
From the top down, see what each thread is doing by issuing these commands in order:
1. ~[threadnumber]s - Switches the WinDBG context to the threadnumber (for example, to switch to thread 3, type ~3s)
2. !clrstack - Shows you the stacktrace of the current thread, as long as current thread is a managed thread
If more information is needed on a particular stacktrace, try running !clrstack -a, which will give you memory locations for objects that you can examine to determine more information. For example, relatively early on in the output of that, there should be a Page or Context variable with some numeric memory address assigned to it, which can be successively dug through with !dumpobject [numericMemoryAddress] (again, remove the square brackets) to find the request URL, form variables, session variables, etc.

Hopefully, with the process above, you'll be able to diagnose at least very simple infinite loops and save yourself the ticket and the time to wait for MSDN support to get back to you. There is another extension similar to SOS called PSSCOR2 that I will be going over in a future post. Hit me with a comment if this helped you, or if there's anything you think is wrong!

On simplistic performance analysis

Tue, 05 Jul 2011 19:42:00 -0400

Recently, I was asked to take a look at a poorly performing ASP.Net application. I'd like to document the process I used to get quick performance gains to grab some time to further analyze the root cause.

The performance tests that were being run prior to me getting on the case were showing a big amount of CPU time being spent on the webserver, but the DB server having around half as much utilization. Given my understanding of the application, which was that the webserver shouldn't be doing near as much work as the DB server, this was a bit troubling. Knowing that another performance test run would just show me similar numbers, without any idea of root cause, I asked my teammates to pull some DebugDiag dumps while running another performance test.

When I got my hands on the dumps, I found that many threads were busy processing XML files. Doing a deeper analysis into what XML files were being read, and why, I found that there was a bit of code that was looking through a single XML file (from disk, no less!) multiple times per page load. Obviously, this was not helping performance at all, and helped account for why the web server was struggling. The code was seeking through the XMLDocument, breaking the loop and disposing of the file handle when finding the entry it was looking for. This wasn't the most efficient way of doing things, but thankfully, this lent itself well to an easy fix.

The first quick idea I had was to just cache the XMLDocument object in memory (application variable) to avoid having to constantly hit the disk, and then allow the previous code to loop through the nodes. This helped reduce the amount of IO on each page load, and this would have made a pretty good improvement in and of itself. Looking more closely at what the code was doing, however, I realized that the code itself was basically using the XML file as a dictionary of sorts.

The final bit of code now reads the XML file on application startup, and stores the entries in a Hashtable object in an application variable. The old code has been modified to query the Hashtable object, thus saving even more time (why bother looping through nodes when we can get an instant response?)

Doing this fix resulted in much more throughput on our next performance test. The webserver and DB's CPU consumption started to even out a bit, which was a great indication that we were making a lot of progress with a quick fix.

Some key takeaways:

DebugDiag is a great tool for getting a snapshot of what exactly is going on in your application at any given time. If there are many threads working on something, you have empirical evidence that your application spends a long time doing whatever it is those threads are doing, which can help to diagnose where your bottlenecks are.
Without load, you won't get much out of a DebugDiag dump, as it records just whatever is going on the instant you take a dump. Thus, you'll just get a bunch of threads doing a whole lot of nothing.
Once you've found a potential issue, and have an initial quickfix in mind, take a step back and determine what the code itself is supposed to be doing. Is there a better way to rewrite your fix?
Don't blindly assume that your new code is going to improve performance. Always run another performance test after your proposed fix to ensure that performance doesn't stay the same, or that it even becomes worse