Sriram Krishnan

Moved to http://www.sriramkrishnan.com/blog

2006-12-02T17:24:00Z

I've stopped posting here and have moved to http://www.sriramkrishnan.com/blog - please point your aggregators to http://www.sriramkrishnan.com/blog/atom.xml

P0

2006-07-20T00:38:00Z

I recently completed a year at Microsoft. I thought I’d take the chance to write about some of the fun anecdotes from my one year here. Blood, sweat, screams tears and joy – I’ve lived through them all in my short stint here.

Nothing in Microsoft is feared as much as a P0 bug. P0, which stands for Priority 0, is the highest importance a bug can be given. You’ll usually see a P0 bug if there is a build break i.e. for some reason, the build didn’t complete or the smoke tests on the new build didn’t pass.

For a Microsoft developer, life stops if you get a P0 bug assigned to you. To quote a colleague, “a P0 means that you drop whatever you’re doing, forget about the outside world, and stop breathing even. The only thing that matters in your life now is that P0 bug”.

As a Program Manager, dealing with coding bugs is usually not a part of my day-to-day work– therefore I’ve never had to deal with P0 bugs for quite a while. Until one late night in August.

August 2005 was a stressful period. We were in the final stages of shipping Visual Studio 2005. This meant that we spent all our time fixing late bugs and cleaning up the product to get it out the door for it’s late October release.

At such a late point in the product’s cycle, there was always the fear that a showstopper bug might come at you from nowhere. Of course, there was no logical reasoning for this fear – the number of bugs being found was dropping drastically. But still, there was Murphy’s Law and the gnawing fear that something might dramatically fail at the last moment.

This particular night, I was up late at office looking at some bug. Finally done with that bug, I packed up my stuff and prepared to head home when a developer on my team asked me if I was free. He had a problem – he was investigating a couple of P0 issues and all of a sudden, a third P0 issue had been assigned to him.

Assigning the bug to myself, I unpacked my bags again and pulled up to my computer. I wasn’t worried – a large number of P0 issues are false alarms and are caused due to some weirdness in the machine running the test or in the automated test code. I’ve rarely seen P0 bugs due to actual issues in the code.

This bug sounded relatively benign. The toolbox was failing to load its controls when you opened a device project. My guess was that it was due to an unclean machine where someone hadn’t uninstalled the previous raw build cleanly.

Let me explain some terminology here. When we did Whidbey, we had two builds of Visual Studio. One was the layout build which we (and you) use with the nice graphical installer and everything. The other build was the ‘raw’ build which just registered the binaries, some debug tools and registry keys. With raw builds, you could build with debug information and with optimizations turned off and debug your way through easily.

In this particular case, our internal testing tools had just installed the raw build on this machine and were running through their usually battery of tests. Everything was hunky-dory until they tried to create a device project, at which point everything came down in flames since there was nothing in the toolbox.

I poked around in the debugger in a bit. After spending 30 minutes or so in the debugger, I knew why the toolbox wasn’t getting populated. For the simple reason that .Net Compact Framework wasn’t installed on the machine.

This was weird – NetCF gets installed along with the rest of Visual Studio and there was no reason for it not to be installed.

I broke into a cold sweat. Not finding NetCF was a pretty serious problem – it would render useless most of the device stuff. Had somebody checked in code which broke the setup? This was a ship stopper bug if I had ever seen one.

I poked through the logs of the raw build’s setup (essentially spew from a huge batch file) and the plot thickened. The log file indicated that the NetCF setup had been run just like the setup for every other part of Visual Studio. But for some weird reason, the files weren’t there. It was as if the setup had never happened. I looked through the event logs and they were a mystery – they told me that NetCF’s installation was kicked off. But for some reason, they never recorded the successful completion or any error.

I tried rerunning setup and voila! – NetCF installed perfectly and all tests passed successfully. I was confused – what had happened that first time? I tried rerunning setup a few times but couldn’t spot the problem. Beaten, I told the tester that I couldn’t ‘repro’ the bug and went home – but not before leaving a note in the bug asking people to assign any similar bugs to me. If this happened again, I wanted to know about it.

I didn’t have to wait long. 24 hours later, there was a mail in my inbox from another tester. A new build had exhibited the same issue. The toolbox wasn’t loading controls.

I plonked myself down and logged in remotely into the test machine. The scene of the crime was identical to the last time. No NetCF. Logs indicating the successful installation of NetCF.

I started poking around in the event logs of the machine. What was happening in the machine around the time of the incident? Was there a hardware failure? Was there a network outage?

The event logs didn’t reveal much. However, one entry caught my eye – just before the entry documenting the NetCF setup, there was an entry about a Systems Management Server operation on the machine. SMS is how we deploy updates to Windows and other internal tools inside Microsoft and they’re very routine around here. However, the reason this caught my eye was due to the fact that I had seen a similar entry in the previous machine.

On both machines, SMS was doing something a few minutes before NetCF’s setup was kicked off.

This was the clincher – I now had a good hunch as to what was happening. I checked in a piece of code to add logging of NetCF’s setup (basically the verbose logging option of msiexec). My code also checked now to see whether the setup failed and if so, it bailed out.

Hardly a day later was my guess vindicated. NetCF installation failed on another machine – but this time, my logs had documented the sequence of events.

For those of you who know the innards of msiexec, you would know that only one setup can happen at any point of time. Msiexec.exe tries to grab a global mutex – if another instance is already holding the mutex, it fails. The reasoning is simple – you don’t want two setup programs to be making changes to the system at the same time as they could trample all over each other.

Visual Studio 2005 setup is made up a few monolithic MSIs and smaller MSIs such as NetCF which are installed as part of the installation process. There is no one huge Visual Studio 2005 .msi file.

Some readers might have spotted the ending now. What was happening was a freak occurrence. In the infinitesimally small time gap between the end of the previous msi’s setup and the beginning of NetCF’s setup, the machine was unlucky enough to encounter a setup request from Systems Management Server. This was causing the NetCF installation to fail and things went rapidly downhill from thereon. The difference this time was that my log files had faithfully recorded all this.

Whew!

I shot off a mail off to the folks running the build machines to make sure the SMS updates happened before and we thankfully never had to deal with this again. We never did find out why this bug had never occurred earlier with other parts of setup. My guess is that something in SMS was changed

Moral of the story – I really don’t have a moral actually. Except maybe, that shipping a product is an incredibly fun and stressful experience all at once.

[Please post comments at http://www.sriramkrishnan.com/blog/2006/07/p0.html ]

The Microsoft Device Emulator goes Shared Source

2006-07-17T17:39:00Z

At BarCamp yesterday, someone asked about the ability to see the source code for the emulator. I was *so* tempted to talk about this but restrained myself.

Well - no more restraining myself.

The Shared Source Device Emulator 1.0 is now available for download under this license

Check out Barry's blog post on the emulator at http://blogs.msdn.com/barrybo/archive/2006/07/17/668492.aspx

Note: You need to download libpng and zlib separately and extract it into the right folders before you can build. Read the 'How to Build' file in the docs folder

[Cross posted at http://www.sriramkrishnan.com/blog/2006/07/microsoft-device-emulator-goes-shared.html . Please post comments there. You might want to subscribe too :-)]

Feed mess

2006-07-15T20:14:00Z

Something I've been working on at work deals with feeds - I have to read, parse and derive meaning out of RSS and Atom feeds in the wild.

And it's not been fun.

The Universal Feed Parser is nice and everything but I'm still being forced to debug through weird Unicode issues. Or issues like some Feedburner feeds having the original permalink in feedburner:origLink

Well, atleast I'm not the only one facing these issues. Sam Ruby maintains a list here and the Google Reader team talks of the problems they've run into (and I've run into a lot of them as well)

[Cross posted at my new blog at http://www.sriramkrishnan.com/blog/2006/07/feed-mess.html. Post comments over there. Subscribe if you feel like it too :-)]

Moving to http://www.sriramkrishnan.com/blog

2006-06-24T22:45:00Z

After several years, I've finally found the time to get off my lazy backside and get myself a domain and some webspace. Henceforth, I'm going to be blogging at www.sriramkrishnan.com/blog

Does this mean I'm going to stop blogging here? Yes and no. I'm going to post all Microsoft related items and technical items on both blogs. However, everything else (non-technical content, non-Microsoft content,etc) will be at the new blog alone.So if you're feeling lazy and don't want to hear about my experiments with random software, stay right here :-)

You can see the first ever post at my new blog

So long Robert

2006-06-12T00:20:00Z

Good luck to you in your new gig. You'll be missed around here.

Sheer beauty - how do you poll?

2006-06-08T04:38:00Z

From time to time, you see an algorithm or a technique that makes you marvel at the sheer beauty of it.

Barry Bond(who is our in-house emulation God, architect and chief troublemaker all rolled into one :-)) posted about the improvements that he and his team made to the emulator to speed it up by a whopping 40% from its first version to the second.

As a part of the team, I've been privileged enough to see check-in after check-in and to see the emulator team chip away at performance hotspots. For anyone interested in profiling or just in making their code faster, definitely go read that post.

There's a little innocuous section where Barry talks about how they poll for interrupts so much faster now.

The problem is simple - you have some stuff that can happen asynchronously. How do you poll(assume that polling is the only option for a variety of other reasons) and see whether anything has happened or not? What is the fastest way to poll? How frequently do you poll?

The typical implementation is what Barry and his team used in V1 of the emulator. The mainline code runs this code at a pre-determined granularity (at ARM instruction boundaries in the emulator's case) and the asynchronous event(the interrupt firing in the emulator case) is in charge of toggling fSomethingHappened (which is usually volatile since this is typically a multi-threaded scenario). I've listed the assembly I got after compiling this with full optimization. I've stopped the DoSomething function from getting inlined for the purpose of making this clearer.

    volatile bool fSomethingHappened=0;


int _tmain(int argc, _TCHAR* argv[])
{
	
	if(fSomethingHappened)
	{
		DoSomething();
	}
	
	return 0;
}


  mov         al,byte ptr [fSomethingHappened (40336Ch)] 
  test        al,al 
  je          wmain+0Eh (40174Eh) 
  call        DoSomething (401730h) 
  xor         eax,eax

You have a compare and a jump. Now, jumps are expensive and generally a bad thing. How do we get rid of the jump? In fact, how do we poll..without using *any* instructions at all?

Let's say I gave you a guarantee - that the stuff you need to do inside DoSomething when something happens is going to take a lot of time - so you don't need to optimize for that case. You only need to optimize for the common case - that is nothing happening and fSomethingHappened being false.

In this case, we can do something really interesting - we can make the code in the asynchronous event firing responsible for *adding* the code to handle it as well.

How? See below


__declspec(naked) void DoSomething()
{
	__asm
	{
	 ret;
	}
}



int _tmain(int argc, _TCHAR* argv[])
{
	
	DoSomething();
	
	return 0;
}

DoSomething:
 ret

wmain:
 call        DoSomething (401730h) 
 xor         eax,eax

This seems really weird at first glance - you have empty functions that are not doing anything. That's because the event patches the function with the code required to handle the interrupt

Here's the flow

Thread 1 calls DoSomething. Returns instantly.
Event fires on Thread 2 - so the event firing code patches DoSomething with a jump to the right code/the right code itself.
Thread 1 merrily tries to poll again - calls DoSomething and handles the event. Life is good.

Check out how low cost the polling is in the normal case -when no event is being fired, we just do a call and a ret. But I hear some of you screaming - but you're still using 2 instructions.And why a call and a ret when you are not doing anything on the stack? You said you'll use no instructions! Well, you see, we actually are using 0 instructions.

It is due to the way processors speculate and look-ahead. All modern processors will look at the 'call' and see that it hits a 'ret' directly. They'll be smart enough to figure out that a call followed by a ret is effectively a nop - and therefore, turn the 2 instructions into a nop.

So in the normal case, the body of wmain for us is effectively this

wmain:
 nop

When the event fires and patches the DoSomething function, processors realize that they cannot speculate anymore - and so they do the right thing.

Awesome, isn't it? Polling without actually executing the code to poll. Just sheer beauty - something that totally blew me away when Barry explained how this works.

Notes:

You shouldn't be worrying about optimizations like this unless you are writing an emulator :-)
The code for actually patching the function with the right code is left as an exercise to the reader :-)
This code doesn't deal with issues like memory fences, atomic patching,etc
Barry explained this to me when we were walking the Strip at Las Vegas. I know what you are thinking :-)

Done with my talk

2006-05-10T21:42:00Z

I'm finally done with my MEDC talk. It went really well though it was the last talk of the day. I'll probably blog about it in detail later

MEDC Attendees Only - If you came to this blog after watching my session to find the code, you'll have to wait a bit. I'll get it up here in a day or two. The slides should already be up on Commnet (accessible from http://content.medc2006.com )

In case you want to mail me, shoot it to sriramk [at] microsoft.com

Finally at Redmond

2006-05-01T18:01:00Z

3 long flights and an equal number of adventures later, I'm finally at Redmond at Microsoft campus. If anyone wants to meet up, shoot me a mail at sriramk@youknowwhere.com. Right now, I'm working out of this place

Some observations about Seattle

This place is cold! For someone like me coming from 40+ degrees Celsius, getting used to 40 degrees is Fahrenheit is proving a bit hard.
India and the US are different on so many levels - I get caught up in the most unexpected of places. This morning, I had real trouble crossing a road. The process is *very different* from what we have back in India!

A Haskell programmer becomes a fan of...Visual Basic

2006-04-11T22:30:00Z

Even though I'm a hardcore VB fan, I never thought I'd see the day when I'd see this happen. Go, Erik, go!
Confessions of a used programming language salesman (PDF Link)
Who's going to be the first Lisper to move to VB? :-)

Location in the Regional and Languages options control panel

2006-04-11T20:20:00Z

One of the free perks of working for Microsoft - when you want to know why a feature in a Microsoft product was done a particular way, you can always chase down the right people. Not that this is not possible outside Microsoft but it sure is a lot easier inside. In fact, a lot of times you see threads internally of the form "Why was foo in Windows done in that way?" and there'll instantly be a couple of response from the developer on the feature enumerating all the reasons why.

Ever wondered what this part of the 'Regional and Language Options' part of the Control Panel applet did? I sure did! For a long time, it wasn't obvious to me what that combo box actually did. I didn't recall seeing any app giving me any extra services. And almost no app actually recognizes which country I'm in. I was also curious as to how I can get at that information through a documented API instead of poking through the registry.

A couple of emails and a few of Michael Kaplan's blog posts later, I was set straight - that combo box sets the GeoId for your system. Here's a small snippet of code which prints out what your current GeoId is

#include <stdio.h>
#include <windows.h>


int wmain(int argc, wchar_t* argv[])
{
	GEOID currentGeoId = GetUserGeoID(GEOCLASS_NATION);

	if(GEOID_NOT_AVAILABLE != currentGeoId)
	{
		wchar_t wszBuffer[100];

		if(0!= GetGeoInfo(currentGeoId, GEO_FRIENDLYNAME, wszBuffer,
			sizeof(wszBuffer)/sizeof(wchar_t), 0))
		{
			
			wprintf(wszBuffer);			
		}
	}

	return 0;
}

Michael Kaplan has blogged about this extensively here, here and here.

Random Things

2006-04-11T19:19:00Z

Hacked together a prototype for my Orcas feature over a weekend. Not bad for a PM,eh? :-). Sorry, I can't tell you what it is...yet.
Spent a week at MEDC India at Bangalore and had an absolute blast. I did a data demo as part of the keynote which went well - though some of my jokes fell flat. Here's a photo of me and the rest of the Visual Studio for Devices gang at MEDC. Quite an attractive bunch, aren't we?
I'm travelling to Redmond and Las Vegas the end of the month. So if you're anywhere near the Seattle area, drop me a mail and we can meet up. I'm going to Las Vegas to do a talk at MEDC 2006. I'm doing APP320 - Web 2.0 and Mobile Devices. But more on that in later posts.
Hyderabad is getting really, really hot. Literally. When I stepped out of the flight from Bangalore, my body took a while to get used to heat. Who expects the temperature to be 38 degrees Celsius at 9 in the night?
I'm finally getting to contribute to some virtual machine development in Microsoft. Not in any official capacity, but it's really cool all the same.
Sql Everywhere is wicked. As in, really wicked.
Software engineering. At Microsoft, you really do understand what the word 'engineering' stands for. Example problem I'm faced with - you have a platform underneath you and you have to build something on top of that platform. How do you build both at the same time? And do so in a month?
My body clock is seriously out of whack. Too many late nights on the computer in the last year or so have resulted in my being unable to sleep till 4 AM in the morning. I've tried everything under the sun to get myself to sleep on time. But it really doesn't help counting sheep when you start thinking of how a search engine can use blog links and track conversations on the 5th sheep.
I need to come up with an excuse to meet this guy while I'm in Redmond. Though he doesn't know me :-)

The MySpace generation

2006-02-19T15:41:00Z

I really have to blame Russell Beattie for all this.

First he posted this entry called "Why MySpace scares the cr*p out of me" where he says

"Why? Because I just don’t get it. I wish I did, but I don’t. Those crazy kids today, and all that. I mean, I get the components of it all - community, pictures, messages, music, social networking, yada-yada, but I don’t *get* why MySpace in particular is popular. I don’t grok it down deep. Or Facebook for that matter (not that I’ve ever seen Facebook - what a competitive advantage, hey? If you’re out of school you can’t even see what the hell the site does!). Why did these sites get such traction when others didn’t?"

The scary thing is - I feel the same way. Though I'm half of Russell's age and the same age as most of MySpace's users.

Today, I went and did the unthinkable - I created a MySpace profile for myself. You can find it here (stole some of the CSS off Russell's profile to make mine look...less chaotic than the other profiles out there :-) . The sheer freedom offered is incredible- pretty much any html and css goes. And boy, do the users exploit that!

I spent some time exploring MySpace - the sheer chaos there is incredible. Like I told someone "MySpace makes Orkut seem like an old-fashioned British tea party". I'm badly out of touch with the users of MySpace - though I'm probably the same age or a couple of years older than most of them.

Russell sums it up very well in this post

This disconnect scares me - for I have to create great software for these people and I don't understand them one bit.

Like Russell says, "I'm just a geek".

Amen

Ship-It!

2006-02-14T15:46:00Z

I finally picked up my Ship-It plaque for VS 2005 today (it had been languishing in my mail for sometime). You have to be a Microsoft employee to know how awesome it feels. :-)

It's raining tennis balls

2006-02-06T07:44:00Z

Microsoft has weird people. When you stroll through the corridors here, you’ll find people’s offices decked up in the weirdest of ways – posters, weird little gadgets,etc. You’ll also find lots of sports equipment – cricket bats, tennis balls and smelly basketball sneakers which haven’t been cleaned in weeks.

If you’re visiting the Microsoft office, be prepared to run into some of this weirdness as well. Like one poor soul found out today…

After intensely pounding away at my keyboard for hours, I went for a walk on my floor with this person. The ground floor here is slightly staggered from the rest of the floors – if you drop something from the 3^rd floor, it’ll fall into the visitor’s waiting area.

Which is exactly what happened today.

She was playing around with a tennis ball and it ‘accidentally’ rolled off the 3^rd floor, underneath the railing. It fell 3 floors…right onto an unsuspecting visitor.

Not a pleasant thing to do to your visitors.

I wish I could tell you what happened next – but in a fit of cowardice, we ran away from the scene of the crime – like a couple of kids who had broken a window.

To the poor soul who had a close encounter with a tennis ball today. We are sorry. We really are. But can we have our ball back now?