IndyNDA Bloggers

Wow… simply wow.

2009-06-30T22:01:48Z

I attended Ken’s ScrumMaster (tm) course over the last two days. Possibly more on that later, but for not I want to focus on something entirely different. People are just very incredibly decent when it comes right down to it.

Ken has an absolution jar in the center of the room to catch the $1 owed for being late back to your table or for cussing in the course. Each day a charity is chosen that has a connection to somebody in the room for the money to go to afterward. On Monday, I mentioned that I ran Dan Shi Martial Arts club, and it was selected as Monday’s charity. By the end of the day the jar held $21, and I took $20 of it to deposit as an anonymous donation. Awesome idea, and thank you everybody!

Tuesday morning I thanked everybody during the morning agenda for sharing their sins with the needy, and commented that they’d sponsored a single student for a single month, which is hugely appreciated. At some point during the day, people must have talked it around, because around 3pm Ken tapped me on the shoulder and told me that the group had collectively decided to send Tuesday’s money Dan Shi’s way as well… THANK YOU!

Around 4:45 I straightened up the money because I was having problems putting my dollar in for being late back from break. I counted ~$30 at that point. At 5:15 I collected the money, and found EIGHTY THREE DOLLARS in the mug. Everybody, thank you! That is hugely significant to an unfunded organization, and we thank you very very much.

You have collectively allowed us to sponsor a child for almost half a year. Thank you. You are all awesome, and it just reminds again that no matter how much we may disagree, argue, or bicker professionally, every single individual is first and foremost a person doing the best they can do.

New Screencast: Working with Property Grids

2009-06-29T16:56:23Z

I have a new 8 min 7 sec training webcast up over working with the PropertyGrid control in Windows Forms applications.

The video is available through JupiterMedia and can be viewed at http://www.internet.com/video/. Just look for the "Developer Video" titled "Working with Property Grids".

Direct links are:
http://www.internet.com/video/?bcpid=1534611832&bclid=1433966034&bctid=27475045001

The First Few Milliseconds of an HTTPS Connection

2009-06-10T12:57:00Z

Convinced from spending hours reading rave reviews, Bob eagerly clicked "Proceed to Checkout" for his gallon of Tuscan Whole Milk and...

Whoa! What just happened?

In the 220 milliseconds that flew by, a lot of interesting stuff happened to make Firefox change the address bar color and put a lock in the lower right corner. With the help of Wireshark, my favorite network tool, and a slightly modified debug build of Firefox, we can see exactly what's going on.

By agreement of RFC 2818, Firefox knew that "https" meant it should connect to port 443 at Amazon.com:

Most people associate HTTPS with SSL (Secure Sockets Layer) which was created by Netscape in the mid 90's. This is becoming less true over time. As Netscape lost market share, SSL's maintenance moved to the Internet Engineering Task Force (IETF). The first post-Netscape version was re-branded as Transport Layer Security (TLS) 1.0 which was released in January 1999. It's rare to see true "SSL" traffic given that TLS has been around for 10 years.

Client Hello

TLS wraps all traffic in "records" of different types. We see that the first byte out of our browser is the hex byte 0x16 = 22 which means that this is a "handshake" record:

The next two bytes are 0x0301 which indicate that this is a version 3.1 record which shows that TLS 1.0 is essentially SSL 3.1.

The handshake record is broken out into several messages. The first is our "Client Hello" message (0x01). There are a few important things here:

Random:

There are four bytes representing the current Coordinated Universal Time (UTC) in the Unix epoch format, which is the number of seconds since January 1, 1970. In this case, 0x4a2f07ca. It's followed by 28 random bytes. This will be used later on.
Session ID:

Here it's empty/null. If we had previously connected to Amazon.com a few seconds ago, we could potentially resume a session and avoid a full handshake.
Cipher Suites:

This is a list of all of the encryption algorithms that the browser is willing to support. Its top pick is a very strong choice of "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA" followed by 33 others that it's willing to accept. Don't worry if none of that makes sense. We'll find out later that Amazon doesn't pick our first choice anyway.
server_name extension:

This is a way to tell Amazon.com that our browser is trying to reach https://www.amazon.com/. This is really convenient because our TLS handshake occurs long before any HTTP traffic. HTTP has a "Host" header which allows a cost-cutting Internet hosting companies to pile hundreds of websites onto a single IP address. SSL has traditionally required a different IP for each site, but this extension allows the server to respond with the appropriate certificate that the browser is looking for. If nothing else, this extension should allow an extra week or so of IPv4 addresses.

Server Hello

Amazon.com replies with a handshake record that's a massive two packets in size (2,551 bytes). The record has version bytes of 0x0301 meaning that Amazon agreed to our request to use TLS 1.0. This record has three sub-messages with some interesting data:

"Server Hello" Message (2):
- We get the server's four byte time Unix epoch time representation and its 28 random bytes that will be used later.
- A 32 byte session ID in case we want to reconnect without a big handshake.
- Of the 34 cipher suites we offered, Amazon picked "TLS_RSA_WITH_RC4_128_MD5" (0x0004). This means that it will use the "RSA" public key algorithm to verify certificate signatures and exchange keys, the RC4 encryption algorithm to encrypt data, and the MD5 hash function to verify the contents of messages. We'll cover these in depth later on. I personally think Amazon had selfish reasons for choosing this cipher suite. Of the ones on the list, it was the one that was least CPU intensive to use so that Amazon could crowd more connections onto each of their servers. A much less likely possibility is that they wanted to pay special tribute to Ron Rivest, who created all three of these algorithms.
Certificate Message (11):
- This message takes a whopping 2,464 bytes and is the certificate that the client can use to validate Amazon's. It isn't anything fancy. You can view most of its contents in your browser:
"Server Hello Done" Message (14):
- This is a zero byte message that tells the client that it's done with the "Hello" process and indicate that the server won't be asking the client for a certificate.

Checking out the Certificate

The browser has to figure out if it should trust Amazon.com. In this case, it's using certificates. It looks at Amazon's certificate and sees that the current time is between the "not before" time of August 26th, 2008 and before the "not after" time of August 27, 2009. It also checks to make sure that the certificate's public key is authorized for exchanging secret keys.

Why should we trust this certificate?

Attached to the certificate is a "signature" that is just a really long number in big-endian format:

Anyone could have sent us these bytes. Why should we trust this signature? To answer that question, need to make a speedy detour into mathemagic land:

Interlude: A Short, Not Too Scary, Guide to RSA

People sometimes wonder if math has any relevance to programming. Certificates give a very practical example of applied math. Amazon's certificate tells us that we should use the RSA algorithm to check the signature. RSA was created in the 1970's by MIT professors Ron *R*ivest, Adi *S*hamir, and Len *A*dleman who found a clever way to combine ideas spanning 2000 years of math development to come up with a beautifully simple algorithm:

You pick two huge prime numbers "p" and "q." Multiply them to get "n = p*q." Next, you pick a small public exponent "e" which is the "encryption exponent" and a specially crafted inverse of "e" called "d" as the "decryption exponent." You then make "n" and "e" public and keep "d" as secret as you possibly can and then throw away "p" and "q" (or keep them as secret as "d"). It's really important to remember that "e" and "d" are inverses of each other.

Now, if you have some message, you just need to interpret its bytes as a number "M." If you want to "encrypt" a message to create a "ciphertext", you'd calculate:

C ≡ M^e (mod n)

This means that you multiply "M" by itself "e" times. The "mod n" means that we only take the remainder (e.g. "modulus") when dividing by "n." For example, 11 AM + 3 hours ≡ 2 (PM) (mod 12 hours). The other recipient knows "d" which allows them to invert the message to recover the original message:

C^d ≡ (M^e)^d ≡ M^e*d ≡ M¹ ≡ M (mod n)

Just as interesting is that the person with "d" can "sign" a document by raising a message "M" to the "d" exponent:

M^d ≡ S (mod n)

This works because "signer" makes public "S", "M", "e", and "n." Anyone can verify the signature "S" with a simple calculation:

S^e ≡ (M^d)^e ≡ M^d*e ≡ M^e*d ≡ M¹ ≡ M (mod n)

Public key cryptography algorithms like RSA are often called "asymmetric" algorithms because the encryption key (in our case, "e") is not equal to (e.g. "symmetric" with) the decryption key "d". Reducing everything "mod n" makes it impossible to use the easy techniques that we're used to such as normal logarithms. The magic of RSA works because you can calculate/encrypt C ≡ M^e (mod n) very quickly, but it is really hard to calculate/decrypt C^d ≡ M (mod n) without knowing "d." As we saw earlier, "d" is derived from factoring "n" back to its "p" and "q", which is a tough problem.

Verifying Signatures

The big thing to keep in mind with RSA in the real world is that all of the numbers involved have to be big to make things really hard to break using the best algorithms that we have. How big? Amazon.com's certificate was "signed" by "VeriSign Class 3 Secure Server CA." From the certificate, we see that this VeriSign modulus "n" is 2048 bits long which has this 617 digit base-10 representation:

1890572922 9464742433 9498401781 6528521078 8629616064 3051642608 4317020197 7241822595 6075980039 8371048211 4887504542 4200635317 0422636532 2091550579 0341204005 1169453804 7325464426 0479594122 4167270607 6731441028 3698615569 9947933786 3789783838 5829991518 1037601365 0218058341 7944190228 0926880299 3425241541 4300090021 1055372661 2125414429 9349272172 5333752665 6605550620 5558450610 3253786958 8361121949 2417723618 5199653627 5260212221 0847786057 9342235500 9443918198 9038906234 1550747726 8041766919 1500918876 1961879460 3091993360 6376719337 6644159792 1249204891 7079005527 7689341573 9395596650 5484628101 0469658502 1566385762 0175231997 6268718746 7514321

(Good luck trying to find "p" and "q" from this "n" - if you could, you could generate real-looking VeriSign certificates.)

VeriSign's "e" is 2^16 + 1 = 65537. Of course, they keep their "d" value secret, probably on a safe hardware device protected by retinal scanners and armed guards. Before signing, VeriSign checked the validity of the contents that Amazon.com claimed on its certificate using a real-world "handshake" that involved looking at several of their business documents. Once VeriSign was satisfied with the documents, they used the SHA-1 hash algorithm to get a hash value of the certificate that had all the claims. In Wireshark, the full certificate shows up as the "signedCertificate" part:

It's sort of a misnomer since it actually means that those are the bytes that the signer is going to sign and not the bytes that already include a signature.

The actual signature, "S", is simply called "encrypted" in Wireshark. If we raise "S" to VeriSign's public "e" exponent of 65537 and then take the remainder when divided by the modulus "n", we get this "decrypted" signature hex value:

0001FFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFF00302130 0906052B0E03021A 05000414C19F8786 871775C60EFE0542 E4C2167C830539DB

Per the PKCS #1 v1.5 standard, the first byte is "00" and it "ensures that the encryption block, [when] converted to an integer, is less than the modulus." The second byte of "01" indicates that this is a private key operation (e.g. it's a signature). This is followed by a lot of "FF" bytes that are used to pad the result to make sure that it's big enough. The padding is terminated by a "00" byte. It's followed by "30 21 30 09 06 05 2B 0E 03 02 1A 05 00 04 14" which is the PKCS #1 v2.1 way of specifying the SHA-1 hash algorithm. The last 20 bytes are SHA-1 hash digest of the bytes in "signedCertificate."

Since the decrypted value is properly formatted and the last bytes are the same hash value that we can calculate independently, we can assume that whoever knew "VeriSign Class 3 Secure Server CA"'s private key "signed" it. We implicitly trust that only VeriSign knows the private key "d."

We can repeat the process to verify that "VeriSign Class 3 Secure Server CA"'s certificate was signed by VeriSign's "Class 3 Public Primary Certification Authority."

But why should we trust that? There are no more levels on the trust chain.

The top "VeriSign Class 3 Public Primary Certification Authority" was signed by itself. This certificate has been built into Mozilla products as an implicitly trusted good certificate since version 1.4 of certdata.txt in the Network Security Services (NSS) library. It was checked-in on September 6, 2000 by Netscape's Robert Relyea with the following comment:

"Make the framework compile with the rest of NSS. Include a 'live' certdata.txt with those certs we have permission to push to open source (additional certs will be added as we get permission from the owners)."

This decision has had a relatively long impact since the certificate has a validity range of January 28, 1996 - August 1, 2028.

As Ken Thompson explained so well in his "Reflections on Trusting Trust", you ultimately have to implicitly trust somebody. There is no way around this problem. In this case, we're implicitly trusting that Robert Relyea made a good choice. We also hope that Mozilla's built-in certificate policy is reasonable for the other built-in certificates.

One thing to keep in mind here is that all these certificates and signatures were simply used to form a trust chain. On the public Internet, VeriSign's root certificate is implicitly trusted by Firefox long before you go to any website. In a company, you can create your own root certificate authority (CA) that you can install on everyone's machine.

Alternatively, you can get around having to pay companies like VeriSign and avoid certificate trust chains altogether. Certificates are used to establish trust by using a trusted third-party (in this case, VeriSign). If you have a secure means of sharing a secret "key", such as whispering a long password into someone's ear, then you can use that pre-shared key (PSK) to establish trust. There are extensions to TLS to allow this, such as TLS-PSK, and my personal favorite, TLS with Secure Remote Password (SRP) extensions. Unfortunately, these extensions aren't nearly as widely deployed and supported, so they're usually not practical. Additionally, these alternatives impose a burden that we have to have some other secure means of communicating the secret that's more cumbersome than what we're trying to establish with TLS (otherwise, why wouldn't we use that for everything?).

One final check that we need to do is to verify that the host name on the certificate is what we expected. Nelson Bolyard's comment in the SSL_AuthCertificate function explains why:

/* cert is OK. This is the client side of an SSL connection.
 * Now check the name field in the cert against the desired hostname.
 * NB: This is our only defense against Man-In-The-Middle (MITM) attacks! */

This check helps prevent against a man-in-the-middle attack because we are implicitly trusting that the people on the certificate trust chain wouldn't do something bad, like sign a certificate claiming to be from Amazon.com unless it actually was Amazon.com. If an attacker is able to modify your DNS server by using a technique like DNS cache poisoning, you might be fooled into thinking you're at a trusted site (like Amazon.com) because the address bar will look normal. This last check implicitly trusts certificate authorities to stop these bad things from happening.

Pre-Master Secret

We've verified some claims about Amazon.com and know its public encryption exponent "e" and modulus "n." Anyone listening in on the traffic can know this as well (as evidenced because we are using Wireshark captures). Now we need to create a random secret key that an eavesdropper/attacker can't figure out. This isn't as easy as it sounds. In 1996, researchers figured out that Netscape Navigator 1.1 was using only three sources to seed their pseudo-random number generator (PRNG). The sources were: the time of day, the process id, and the parent process id. As the researchers showed, these "random" sources aren't that random and were relatively easy to figure out.

Since everything else was derived from these three "random" sources, it was possible to "break" the SSL "security" in 25 seconds on a 1996 era machine. If you still don't believe that finding randomness is hard, just ask the Debian OpenSSL maintainers. If you mess it up, all the security built on top of it is suspect.

On Windows, random numbers used for cryptographic purposes are generated by calling the CryptGenRandom function that hashes bits sampled from over 125 sources. Firefox uses this function along with some bits derived from its own function to seed its pseudo-random number generator.

The 48 byte "pre-master secret" random value that's generated isn't used directly, but it's very important to keep it secret since a lot of things are derived from it. Not surprisingly, Firefox makes it hard to find out this value. I had to compile a debug version and set the SSLDEBUGFILE and SSLTRACE environment variables to see it.

In this particular session, the pre-master secret showed up in the SSLDEBUGFILE as:

4456: SSL[131491792]: Pre-Master Secret [Len: 48]
03 01 bb 7b 08 98 a7 49 de e8 e9 b8 91 52 ec 81 ...{...I.....R..
4c c2 39 7b f6 ba 1c 0a b1 95 50 29 be 02 ad e6 L.9{......P)....
ad 6e 11 3f 20 c4 66 f0 64 22 57 7e e1 06 7a 3b .n.? .f.d"W~..z;

Note that it's not completely random. The first two bytes are, by convention, the TLS version (03 01).

Trading Secrets

We now need to get this secret value over to Amazon.com. By Amazon's wishes of "TLS_RSA_WITH_RC4_128_MD5", we will use RSA to do this. You could make your input message equal to just the 48 byte pre-master secret, but the Public Key Cryptography Standard (PKCS) #1, version 1.5 RFC tells us that we should pad these bytes with random data to make the input equal to exactly the size of the modulus (1024 bits/128 bytes). This makes it harder for an attacker to determine our pre-master secret. It also gives us one last chance to protect ourselves in case we did something really bone-headed, like reusing the same secret. If we reused the key, the eavesdropper would likely see a different value placed on the network due to the random padding.

Again, Firefox makes it hard to see these random values. I had to insert debugging statements into the padding function to see what was going on:

wrapperHandle = fopen("plaintextpadding.txt", "a");
fprintf(wrapperHandle, "PLAINTEXT = ");
for(i = 0; i < modulusLen; i++)
{
    fprintf(wrapperHandle, "%02X ", block[i]);
}
fprintf(wrapperHandle, "\r\n");
fclose(wrapperHandle);

In this session, the full padded value was:

00 02 12 A3 EA B1 65 D6 81 6C 13 14 13 62 10 53 23 B3 96 85 FF 24 FA CC 46 11 21 24 A4 81 EA 30 63 95 D4 DC BF 9C CC D0 2E DD 5A A6 41 6A 4E 82 65 7D 70 7D 50 09 17 CD 10 55 97 B9 C1 A1 84 F2 A9 AB EA 7D F4 CC 54 E4 64 6E 3A E5 91 A0 06 00 03 01 BB 7B 08 98 A7 49 DE E8 E9 B8 91 52 EC 81 4C C2 39 7B F6 BA 1C 0A B1 95 50 29 BE 02 AD E6 AD 6E 11 3F 20 C4 66 F0 64 22 57 7E E1 06 7A 3B

Firefox took this value and calculated "C ≡ M^e (mod n)" to get the value we see in the "Client Key Exchange" record:

Finally, Firefox sent out one last unencrypted message, a "Change Cipher Spec" record:

This is Firefox's way of telling Amazon that it's going to start using the agreed upon secret to encrypt its next message.

Deriving the Master Secret

If we've done everything correctly, both sides (and only those sides) now know the 48 byte (256 bit) pre-master secret. There's a slight trust issue here from Amazon's perspective: the pre-master secret just has bits that were generated by the client, they don't take anything into account from the server or anything we said earlier. We'll fix that be computing the "master secret." Per the spec, this is done by calculating:

master_secret = PRF(pre_master_secret, "master secret", ClientHello.random + ServerHello.random)

The "pre_master_secret" is the secret value we sent earlier. The "master secret" is simply a string whose ASCII bytes (e.g. "6d 61 73 74 65 72 ...") are used. We then concatenate the random values that were sent in the ClientHello and ServerHello (from Amazon) messages that we saw at the beginning.

The PRF is the "Pseudo-Random Function" that's also defined in the spec and is quite clever. It combines the secret, the ASCII label, and the seed data we give it by using the keyed-Hash Message Authentication Code (HMAC) versions of both MD5 and SHA-1 hash functions. Half of the input is sent to each hash function. It's clever because it is quite resistant to attack, even in the face of weaknesses in MD5 and SHA-1. This process can feedback on itself and iterate forever to generate as many bytes as we need.

Following this procedure, we obtain a 48 byte "master secret" of

4C AF 20 30 8F 4C AA C5 66 4A 02 90 F2 AC 10 00 39 DB 1D E0 1F CB E0 E0 9D D7 E6 BE 62 A4 6C 18 06 AD 79 21 DB 82 1D 53 84 DB 35 A7 1F C1 01 19

Generating Lots of Keys

Now that both sides have a "master secrets", the spec shows us how we can derive all the needed session keys we need using the PRF to create a "key block" where we will pull data from:

key_block = PRF(SecurityParameters.master_secret, "key expansion", SecurityParameters.server_random + SecurityParameters.client_random);

The bytes from "key_block" are used to populate the following:

client_write_MAC_secret[SecurityParameters.hash_size]
server_write_MAC_secret[SecurityParameters.hash_size]
client_write_key[SecurityParameters.key_material_length]
server_write_key[SecurityParameters.key_material_length]
client_write_IV[SecurityParameters.IV_size]
server_write_IV[SecurityParameters.IV_size]

Since we're using a stream cipher instead of a block cipher like the Advanced Encryption Standard (AES), we don't need the Initialization Vectors (IVs). Therefore, we just need two Message Authentication Code (MAC) keys for each side that are 16 bytes (128 bits) each since the specified MD5 hash digest size is 16 bytes. In addition, the RC4 cipher uses a 16 byte (128 bit) key that both sides will need as well. All told, we need 2*16 + 2*16 = 64 bytes from the key block.

Running the PRF, we get these values:

client_write_MAC_secret = 80 B8 F6 09 51 74 EA DB 29 28 EF 6F 9A B8 81 B0
server_write_MAC_secret = 67 7C 96 7B 70 C5 BC 62 9D 1D 1F 4A A6 79 81 61
client_write_key = 32 13 2C DD 1B 39 36 40 84 4A DE E5 6C 52 46 72
server_write_key = 58 36 C4 0D 8C 7C 74 DA 6D B7 34 0A 91 B6 8F A7

Prepare to be Encrypted!

The last handshake message the client sends out is the "Finished message." This is a clever message that proves that no one tampered with the handshake and it proves that we know the key. The client takes all bytes from all handshake messages and puts them into a "handshake_messages" buffer. We then calculate 12 bytes of "verify_data" using the pseudo-random function (PRF) with our master key, the label "client finished", and an MD5 and SHA-1 hash of "handshake_messages":

verify_data = PRF(master_secret, "client finished", MD5(handshake_messages) + SHA-1(handshake_messages)) [12]

We take the result and add a record header byte "0x14" to indicate "finished" and length bytes "00 00 0c" to indicate that we're sending 12 bytes of verify data. Then, like all future encrypted messages, we need to make sure the decrypted contents haven't been tampered with. Since our cipher suite in use is TLS_RSA_WITH_RC4_128_MD5, this means we use the MD5 hash function.

Some people get paranoid when they hear MD5 because it has some weaknesses. I certainly don't advocate using it as-is. However, TLS is smart in that it doesn't use MD5 directly, but rather the HMAC version of it. This means that instead of using MD5(m) directly, we calculate:

HMAC_MD5(Key, m) = MD5((Key ⊕ opad) ++ MD5((Key ⊕ ipad) ++ m)

(The ⊕ means XOR, ++ means concatenate, "opad" is the bytes "5c 5c ... 5c", and "ipad" is the bytes "36 36 ... 36").

In particular, we calculate:

HMAC_MD5(client_write_MAC_secret, seq_num + TLSCompressed.type + TLSCompressed.version + TLSCompressed.length + TLSCompressed.fragment));

As you can see, we include a sequence number ("seq_num") along with attributes of the plaintext message (here it's called "TLSCompressed"). The sequence number foils attackers who might try to take a previously encrypted message and insert it midstream. If this occurred, the sequence numbers would definitely be different than what we expected. This also protects us from an attacker dropping a message.

All that's left is to encrypt these bytes.

RC4 Encryption

Our negotiated cipher suite was TLS_RSA_WITH_RC4_128_MD5. This tells us that we need to use Ron's Code #4 (RC4) to encrypt the traffic. Ron Rivest developed the RC4 algorithm to generate random bytes based on a 256 byte key. The algorithm is so simple you can actually memorize it in a few minutes.

RC4 begins by creating a 256-byte "S" byte array and populating it with 0 to 255. You then iterate over the array by mixing in bytes from the key. You do this to create a state machine that is used to generate "random" bytes. To generate a random byte, we shuffle around the "S" array.

Put graphically, it looks like this:

To encrypt a byte, we xor this pseudo-random byte with the byte we want to encrypt. Remember that xor'ing a bit with 1 causes it to flip. Since we're generating random numbers, on average the xor will flip half of the bits. This random bit flipping is effectively how we encrypt data. As you can see, it's not very complicated and thus it runs quickly. I think that's why Amazon chose it.

Recall that we have a "client_write_key" and a "server_write_key." The means we need to create two RC4 instances: one to encrypt what our browser sends and the other to decrypt what the server sent us.

The first few random bytes out of the "client_write" RC4 instance are "7E 20 7A 4D FE FB 78 A7 33 ..." If we xor these bytes with the unencrypted header and verify message bytes of "14 00 00 0C 98 F0 AE CB C4 ...", we'll get what appears in the encrypted portion that we can see in Wireshark:

The server does almost the same thing. It sends out a "Change Cipher Spec" and then a "Finished Message" that includes all handshake messages, including the decrypted version of the client's "Finished Message." Consequently, this proves to the client that the server was able to successfully decrypt our message.

Welcome to the Application Layer!

Now, 220 milliseconds after we started, we're finally ready for the application layer. We can now send normal HTTP traffic that'll be encrypted by the TLS layer with the RC4 write instance and decrypt traffic with the server RC4 write instance. In addition, the TLS layer will check each record for tampering by computing the HMAC_MD5 hash of the contents.

At this point, the handshake is over. Our TLS record's content type is now 23 (0x17). Encrypted traffic begins with "17 03 01" which indicate the record type and TLS version. These bytes are followed by our encrypted size, which includes the HMAC hash.

Encrypting the plaintext of:

GET /gp/cart/view.html/ref=pd_luc_mri HTTP/1.1
Host: www.amazon.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.10) Gecko/2009060911 Minefield/3.0.10 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
...

will give us the bytes we see on the wire:

The only other interesting fact is that the sequence number increases on each record, it's now 1 (and the next record will be 2, etc).

The server does the same type of thing on its side using the server_write_key. We see its response, including the tell-tale application data header:

Decrypting this gives us:

HTTP/1.1 200 OK
Date: Wed, 10 Jun 2009 01:09:30 GMT
Server: Server
...
Cneonction: close
Transfer-Encoding: chunked

which is a normal HTTP reply that includes a non-descriptive "Server: Server" header and a misspelled "Cneonction: close" header coming from Amazon's load balancers.

TLS is just below the application layer. The HTTP server software can act as if it's sending unencrypted traffic. The only change is that it writes to a library that does all the encryption. OpenSSL is a popular open-source library for TLS.

The connection will stay open while both sides send and receive encrypted data until either side sends out a "closure alert" message and then closes the connection. If we reconnect shortly after disconnecting, we can re-use the negotiated keys (if the server still has them cached) without using public key operations, otherwise we do a completely new full handshake.

It's important to realize that application data records can be anything. The only reason "HTTPS" is special is because the web is so popular. There are lots of other TCP/IP based protocols that ride on top of TLS. For example, TLS is used by FTPS and secure extensions to SMTP. It's certainly better to use TLS than inventing your own solution. Additionally, you'll benefit from a protocol that has withstood careful security analysis.

... And We're Done!

The very readable TLS RFC covers many more details that were missed here. We covered just one single path in our observation of the 220 millisecond dance between Firefox and Amazon's server. Quite a bit of the process was affected by the TLS_RSA_WITH_RC4_128_MD5 Cipher Suite selection that Amazon made with its ServerHello message. It's a reasonable choice that slightly favors speed over security.

As we saw, if someone could secretly factor Amazon's "n" modulus into its respective "p" and "q", they could effectively decrypt all "secure" traffic until Amazon changes their certificate. Amazon counter-balances this concern this with a short one year duration certificate:

One of the cipher suites that was offered was "TLS_DHE_RSA_WITH_AES_256_CBC_SHA" which uses the Diffie-Hellman key exchange that has a nice property of "forward secrecy." This means that if someone cracked the mathematics of the key exchange, they'd be no better off to decrypt another session. One downside to this algorithm is that it requires more math with big numbers, and thus is a little more computationally taxing on a busy server. The "Advanced Encryption Standard" (AES) algorithm was present in many of the suites that we offered. It's different than RC4 in that it works on 16 byte "blocks" at a time rather than a single byte. Since its key can be up to 256 bits, many consider this to be more secure than RC4.

In just 220 milliseconds, two endpoints on the Internet came together, provided enough credentials to trust each other, set up encryption algorithms, and started to send encrypted traffic.

And to think, all of this just so Bob can buy milk.

Feature Deduction

2009-06-10T08:21:26Z

The problem

You have what Chris Matts calls an “Order Taker” as your business analyst. This individual is occasionally your only access to the people that actually understand what’s needed, yet they’ve been positioned as the gatekeepers of knowledge.

Consider this tactic

Combine “The 5 Why’s” and “Feature Injection” and see if you can deduce the real information that led to a specific requirement or change.

The basis

The thing is, the person the BA is talking to likely does Feature Injection, even if they don’t know they’re doing it. They’ve created their mental model (or “understanding”) of what problems this software they’ve asked for will solve. Something comes along and breaks the model (or “Doesn’t quite match what we’re doing”), and they change the model (or “submit a change request”).

If you look at the outputs of this (the delta, or the specific change), you’ll observe that they probably share a common theme or two. If you keep asking “why” on the changes, you’ll accomplish two things. You’ll be able to understand what sort of learning occurred, and the further “why’s” will either lead to better answers (less “telephone” effect) or more questions for the actual business owner. Second, you’ll probably frustrate the hell out of the order taking BA, and there’s a tiny chance they’ll learn to act differently.

Either way, you probably understand the intent of the changes better, which means you can do a better job providing value.

btw, please please please try and fix the organizational structure first, but if you can’t do so, then you can begin finding the right ways to work around the impediments.

Using IIS7 Failure Tracing

2009-06-09T12:11:03Z

In the process of debugging an issue with a sample web application, I was forced to learn a bit more about IIS7's management model, and I found some very valuable information.

The problem: the user my request was running as did not have ACL-level access to the .svc file for that web service.

My old solution: Try random accounts before giving up and just giving access to "Everyone" for the entire web tree. (Yes, bad, but it's a sample)

My hope: There is a way to find out via logs, console, etc what the actual user is based on the actual failure rather than looking at all security configurations and randomly trying the referenced accounts.

What I found:

Open IIS manager, click at the web site level, and choose "Failed Request Tracing Rules"

On the right, choose "Edit Site Tracing…"

Check "Enable", and note the location of the log files (note, there are implications on production sites. I don't know what they are. Be warned.)

Now, click "Add…" on the right, and choose your settings:

When you finish, you'll see something like this:

At this point, you can go to your logs folder and see the XML results of any errors that match your filters:

Opening one of those applies a nice transform (freb.xsl, I assume), and we have our data:

Including my answer:

And lots of other cool information:

And a summary view:

Including some really useful information that I would usually try to get out of Fiddler2 because of the challenge in collecting it in some configurations:

Oslo May 2009 CTP: Intellipad Changes

2009-05-27T18:47:40Z

The May 2009 CTP of “Oslo” introduces a lot of new features and enhancements to the existing toolset. As a developer, my primary “Oslo” tool has been Intellipad. It does seem much faster and more stable after a little over an hour of use. Previous CTPs were memory hogs and would occasionally crash. I will be upgrading all of my demos and work projects over to the latest CTP over the next week, so I will get to see how much more stable it really is. Here is an overview of some of the UI Change:

Shell

For the most part we have the same familiar interface. You have the same basic layout shell which is a mixture of a standard application frame and menu bar in one. Everything else in Intellipad is a workspace window which can be docked any number of ways inside this shell. The shell itself has a couple of new features:

A new icon (same icon used by Quadrant)
The build number is now displayed next to the control box
Menu items now appear to use much less white space, display proper “…” labels on items that open new windows, and a few other UI adjustments

Workspace Window

Each workspace window has its own title bar as always. The title bar appears to be the same. We still have the document name, zoom level, a drop down to change “Modes” and a close button. However, now the the active workspace window is highlighted in “Oslo Green”. This is a minor change, but really comes in hand when working with multiple documents.

The “Edit” menu item for windows has a number of new features. Check out the list of commands:

We have our standard Undo/Redo/Cut/Copy/Paste/Delete/Find/Replace/GotoLine options which we had before, but now we have:

Disable External Changes:Behaviors
1. Selection Highlight: This is really neat! With this option as you select text in the document, it will be highlighted in “Oslo Green” and other uses of that same text in the same document will also be highlighted in blue.
2. Error Squiggles: Whether or not to show the red squiggle under errors. if you turn this off, you will still see errors in the error window, but you will have no visual indicator on the document, nor will you have the error tooltips
3. Column Indicator:
4. Link Navigation:
5. Work Wrap: Wraps the text at the end of the line
6. Line Numbers: Provides line numbers to the left of the document

This screenshot shows the “error squiggle” and “selection highlighting” in action:

There is new a View menu item which generously gives us support for quickly launching the three most important developer windows: Minibuffer, Errors, and Notifications.

There is also a new Help menu item for displaying an “Intellipad Primer” document in Rich Text mode:

Mini-Buffer

The mini buffer has change dramatically. It use to pop up a kind of modal WPF dialog which allowed you to enter in commands. This would occasionally cause Intellipad to crash on me. In May CTP, it is now a standard document window where you can enter in commands and see previous commands much like if it were a command line window.

Some people may dislike this little change, but personally I like this change as it is more consistent with the rest of the UI and user friendly.

DSL Authoring Mode

The “DSL Authoring Mode” is a 3-Windows view inside of Intellipad which contains:

The document containing the DSL
The input: some document were you could enter in data to test against the DSL
The output: The AST tree that was generated from the input via the DSL

This mode is the preferred way to write a DSL as you can see how changes affect the schema in real time.

In previous CTPs you had to launch a Intellipad with special command line options to be able to work in what was known as “DSL Authoring Mode”. This usually meant selecting “Intellipad (Samples Enabled)” from the start menu instead of just “Intellipad”. This was annoying at best. In the May CTP however, you can now enter “DSL Authoring Mode” from the standard launch of Intellipad. Just open a document in “DSL Grammar Mode”, and there is a new DSL menu item called “Split New Input and Output Views” which launches the 3-Window view.

Oslo May 2009 CTP: Released

2009-05-27T16:49:00Z

It’s been many long months since the last drop of Oslo (January 2009). Last night, Chris Sells and others announced the May 2009 CTP of Oslo (version 0.4.2). Each new CTP brings more features, more documentation, and more stability. The May CTP holds true with many new features including:

The first public release of Quadrant
Major updates to Intellipad (including a built in authoring mode)
A unified tool set for all “M” languages
A new installer (themed like Intellipad and Quadrant)
UML Domain support (Use Case, Activity, Class, Sequence, Component diagrams, etc) and a loader to import UML diagrams in XMI format into the repository
CLR Domain support and a loader to import .NET assemblies into the repository

I was told that we would have support for most of the command line tools in Intellipad for this release such as being able to compile and deploy to the repository. However, after having played with Intellipad for 30 minutes this morning, I can’t find any of these commands. They should be some simple customizations via python scripts and modifying some xml files. Hopefully the team will release these additions as well.

Download the CTP here: http://www.microsoft.com/downloads/details.aspx?FamilyID=827122a5-3ca0-4389-a79e-87af37cbf60d&displaylang=en

Install Experience

The install experience has been greatly improved. (Be sure to install any previous versions first) Here are some screenshots:

Visual Studio 2010: Extension Manager

2009-05-21T23:51:53Z

Since PDC 2008 we have been hearing about the Managed Extensibility Framework (MEF) support in Visual Studio 2010. Now that the Beta has been released, we can start to see how it is coming together. To access the MEF extensions, just open the Extension Manager from the menu bar: Tools –> Extension Manager.

It appears that we will have a host of categories for different types extensions. You can choose from “Installed Extensions” or browse the “Online Gallery”. The major categories are:

Controls
1. ASP.Net Controls
2. Framework & Libraries
3. Sharepoint Controls
4. Silverlight Controls
5. Windows Forms Controls
6. WPF Controls
Templates
1. Windows Forms
2. WPF
Tools
1. Build
2. Coding
3. Data
4. Documentation
5. Modeling
6. Other
7. Performance
8. Programming & Languages
9. Reporting
10. Setup & Deployment
11. Source Control
12. Team Development
13. Testing
14. Web

Currently there are only 12 extensions available, and none appear to be new UI controls. Some of them are pretty interesting, but I am excited about what this could lead to. You seem to be able to extend quite a few areas of Visual Studio now!

Some of the more interesting existing extensions are:

Intellisense Presenter A new way to view your intellisense in a WPF fashion.
RegEx Editor Regular expression syntax highlighting, intellisense, in place testing, and more right from your code.
Image Insertion This allows you to insert images into code files. Great for adding developer documentation, but not sure what else it would be useful for.

I haven’t been able to find the WPF xml document visualizer that they showed off at PDC 2008. Hopefully this will show up in the gallery soon along with other useful nuggets.

I wonder if this will turn into a platform for pushing out new controls to use in our applications in between releases. It was disappointing to not find any new controls in the toolbox when working with WinForms or WPF. It is also unclear about how a developer can submit new extensions that be browsed through the gallery.

Entity Framework 4.0: Scalar and Void Functions

2009-05-20T22:52:10Z

One feature that was sorely missing in the first version of Entity Framework was the automated ability to work with stored procedures that didn’t return back entities. This could have been sprocs that returned void, a scalar value, or some other custom construct.

Entity Framework v1

In Entity Framework v1 you could create a “Function Import” from these stored procedures types, but there would be no generated code that would give you access to them directly in the ObjectContext object. You could only access them via pure Entity SQL.

To add a “Function Import” you would just add the sproc to your entity model as your normally would, then right-click on the “Function Imports” in the “Model Browser”, and select “Add Function Import…”. This would bring up the following screen which would allow you to add your function.

If you selected that the function returned an Entity, then you would be able to access that entity in the ObjectContext. However, if you selected scalar or void, then you would have to manually write some kind of Entity SQL. It would be nice if these “Function Imports” appeared as operations in the ObjectContext as well.

Entity Framework 4.0 Solution

Microsoft finally completed the story for “Function Imports” in Entity Framework 4.0. As you can see in the image below, the “Add Function Import” dialog is virtually the same except for the new option to return a complex type.

You can select the None and Scalar return types as you could before. However, when the “Function Import” is created, some new code is injected into the Model code behind file that materializes the stored procedure into an operation on the ObjectContext itself:

public ObjectResult<Nullable<global::System.Int32>> GetProductCountByCategoryId(
    Nullable<global::System.Int32> categoryID)
{
 
    ObjectParameter categoryIDParameter;
    if (categoryID.HasValue)
    {
        categoryIDParameter = new ObjectParameter("CategoryID", categoryID);
    }
    else
    {
        categoryIDParameter = new ObjectParameter(
            "CategoryID", 
            typeof(global::System.Int32));
    }
    return base.ExecuteFunction<Nullable<global::System.Int32>>(
        "GetProductCountByCategoryId", 
        categoryIDParameter);
}
 

A couple of nice things about this are:

No matter what we select as the return type, it wraps it as a Nullable type since a database value can always be null
The results are wrapped in an ObjectResult which makes it consistent with all other queries.

And as we expect, we can directly access these operations in our code:

using (var context = new NorthwindEntities())
{
    ObjectResult<int?> result =
        context.GetProductCountByCategoryId(1);
    
    MessageBox.Show(
        result.FirstOrDefault().Value.ToString());
}
 

New “Execute” operations

If you look at the last line of the code generated by “Add Function Import”, you might notice is making a call to a new method called ExecuteFunction<T>. This method isn’t marked public, but there are two new methods on ObjectContext that we can use to execute functions:

int ExecuteStoreCommand
This method seems to work like “ExecuteCommand” and returns in int representing the result
T ExecuteStoreQuery<T>

This appears to be more of a combination of “ExecuteScalar”, “ExecuteNonQuery”, and “ExecuteReader”.

These methods give us the ability to run any imported stored procedure in a variety of ways. Most users, will want to just stick with the generated operations, but its always good to have options!

New Screencast: Custom Sprocs in Entity Framework

2009-05-20T14:11:00Z

I have a new 8 min 35sec training webcast up over using custom stored procedures in Entity Framework. This video introduces how to use custom sprocs in your Entity Framework model for inserts, updates, deletes, and custom sprocs for searching. The video briefly talks about how to work with void and scalar sprocs, but doesn’t get far into these as this is a feature that is getting more attention in the next version of Entity Framework.

The video is available through JupiterMedia and can be viewed at http://www.internet.com/video/. Just look for the "Developer Video" titled "Custom Sprocs in Entity Framework".

Direct links are:
http://www.internet.com/player/index.php?bcpid=1534611832&bclid=1433966034&bctid=22915014001

New Webcast: ADO.Net Data Services

2009-05-20T01:19:23Z

I have a new 7 min 55 sec training webcast up over developing ADO.Net DataServices. This video demonstrates how to create and consume a simple ADO.Net Data service.

The video is available through JupiterMedia and can be viewed at http://www.internet.com/video/. Just look for the "Developer Video" titled "ADO.Net Data Services".

Direct links are:
http://www.internet.com/video/?bcpid=1534611832&bclid=1433966034&bctid=13662214001

Entity Framework 4.0: Pluralization

2009-05-20T01:00:15Z

One of the new features in Entity Framework 4.0 (EF v2) is a service call “Pluralization”. This service is used to convert names of objects form singular and/or plural forms. For instance, if you have a table in the database that has a plural name such as “Customers”, then Entity Framework will automatically generate both object Name and Entity Set Name to “Customers” as well. The result is that in your code, the name will look like this:

Customers c = new Customers();

Ideally, it would have been nice for EF to see that the table was a plural name and set the object Name to a singular form.

Now, you may be thinking that this is pretty trivial since you can manually change the generated name, but when when you have dozens or hundreds of tables this can be tedious at best.

To use the PluralizationService when generating an Entity Framework model, you just select the option"Pluralize or singularize generated object names”:

Or if you are using the EDMGen.exe tool, you can opt in using the /pluralize option.

Example

In the Northwind database, all the tables have plural names. (Note: It is typically recommended to name your tables in the singular form).

When this database is generated using the PluralizationService, the names are set correctly.

Manually using the PluralizationService

The PluralizationService is in the System.Data.Entity.Design.dll under the System.Data.Entity.Design.PluralizationServices namespace.

It is an abstract class and hence, can’t be directly instantiated. However, there is a static CreateService method on the class to access instances of the service for various cultures. At this time, the only culture implemented is “en-us”.

It would have been very nice if this were implemented at a more core level of .Net since this could potentially be useful for a lot more than just for Entity Framework design. Here are some examples using the service:

// Create an instance of the service for
// a particular culture (only 'en-us' atm)
PluralizationService ps =
    PluralizationService.CreateService(
         CultureInfo.GetCultureInfo("en-us"));
 
// Convert strings to plural or singular
string pluralName = ps.Pluralize("Goose");
string singularName = ps.Singularize("Territories");
// Note: Correctly returns Geese and Territory
 
// Check if strings are plural or singular
bool isPluralName = ps.IsPlural("Sheep");
bool isSingularName = ps.IsSingular("Sheep");
// Note: Correctly sees both of these as true

Adding your own pluralization mapping

There is an interface using by the PluralizationService called ICustomPluralizationMapping that isn’t publicly exposed. However, since it is implemented, you can cast the service to this interface and make use of the AddWord method to add your own mapping.

In this example, we add a new word called “Foo” with the plural version being “Foos”:

// Create an instance of the service for
// a particular culture (only 'en-us' atm)
PluralizationService ps =
    PluralizationService.CreateService(
        CultureInfo.GetCultureInfo("en-us"));
 
// Cast to the mapping interface
ICustomPluralizationMapping mapping =
    ps as ICustomPluralizationMapping;
 
// Add a new word mapping
mapping.AddWord("Foo", "Foos");

You could then use the service to generate the metadata or use it in some other context:

EntityModelSchemaGenerator generator =
    new EntityModelSchemaGenerator(
        entityContainer,
        ps);
 
var edm = generator.GenerateMetadata(); 

What’s next, and why?

2009-05-12T14:29:00Z

The time has come to take another big step towards my dreams and move into the next phase of my career. Understanding how this move is a natural extension of my current position requires an understanding of where I have been in the past and an appreciation of the context of each of those positions. I can divide my life to this point into five major phases of learning and development: childhood, college, and career phases one through three. I spent my childhood learning about the world around me, about whom I am, and about what core values I hold. I do not believe these core values or understandings will ever truly change, meaning I will for better or worse be a small-town Midwestern guy for my entire life. I then studied for four years at Rose-Hulman Institute of Technology. I acquired a great deal of knowledge in my time there, but more importantly, I became aware of how to learn. I gained the perspectives that allow me to compare what I see at any given moment against my experiences, the experiences of others, a varied set of perspectives, and an extensive body of knowledge. This process granted me the ability to quickly and accurately categorize information and identify inconsistencies within that information.

Finally, I moved into my professional career. I invested seven years into SEP, Inc. on the north side of Indianapolis. I believe this time consisted of two different phases with a fuzzy boundary between them. First, I learned my craft. My first four years focused heavily on direct delivery of value as a developer. I observed leadership behaviors during this time, and I learned about more than technology, but my primary focus was becoming an expert developer and coder. Second, I began to make the first steps into technical leadership roles. I became aware of the value of learning techniques from outside my role, and later from outside my industry. I started studying and evaluating different approaches towards accomplishing software delivery. I took responsibility for the design, flow, and implementation of software projects. I explored the “rules” imposed by various regulatory frameworks, and became inspired by recognition of the essence of flow buried deep inside. I examined the structures of people and projects around me, better understanding their behaviors. In short, I became aware of the non-coding aspects of the software engineering industry. More importantly, I began reading a number of business and economics books, histories of capital and finance, current events in business and industry. I launched “Rediscovering the Obvious”.

After a time, the yearning to do something different took over, and I accepted an offer to try my hand at product development. My first short gig in that direction did not work out, but I learned a great deal and moved to my current role at Inkubook.com. I came in as a senior software developer, but James, then director of Inkubook, promoted me to a software architecture role a short while later. This time was incredibly exciting to me professionally. This opportunity allowed me to spend nearly two years deeply focused on product development in a role allowing me a great deal of input into every aspect of the product and the context in which we built it. The environment at Inkubook is incredibly empowering in a number of ways. The team’s leadership offers individuals a great deal of flexibility and empowerment in how they approach their work and development. My management has been incredibly supportive of my speaking engagements, allowing me to tell the story of Inkubook publicly and providing time in which to do so. They have supported the team’s desire to learn with a very steady book budget and a ready environment in which we discuss what we learn. The architects regularly exchange sources and ideas about any number of technology topics. Ideas are encouraged and respected regardless of who originates them, and each idea is given a chance to prove itself before a decision is made. I have not worked in any environment so supportive of learning, and the effects are visible in the obvious engagement shown by each of my peers. This passion to improve amplifies itself within the team, leading us to a team culture of continuous self-improvement by each member of the team.

Why am I leaving Inkubook? I can honestly say there is no place in Indianapolis I would rather write software, it is a great place to work and the people are incredible. However, I have long been very passionate about how people go about writing software and delivering systems. EMC has offered me a position allowing me a great deal of freedom in my personal life. I will share more about the nature of this position when I understand which aspects are confidential strategy and which are more public. However, I will happily share my excitement on what this means for my life. First, I will be spending a month in London getting to know my team and delivering on some initial contracts over there. Then, I will be free of geographical constraints with the single primary restriction of needing to be near an airport. This frees Marian, Elle, and I to explore the world and try living different places. Equally importantly, this nicely aligns my daytime responsibilities with my learning passions, making work feel less like work. Finally, I love to teach, and much of this role will be directly helping people learn in various forms. Overall, this feels like the next natural evolution of my career, and I look forward to embracing the opportunity.

Using Obscure Windows COM APIs in .NET

2009-04-24T12:37:00Z

Most native Windows APIs are simple to call from .NET. For example, if you need to do something special when showing a window, you can use the ShowWindow API using Platform Invocation Services (P/Invoke) like this:

[DllImport("user32.dll")]
static extern bool ShowWindow(IntPtr hWnd, int nCmdShow);

When you call this function, here's roughly what happens:

The CLR calls LoadLibrary on the file (e.g. "user32.dll")
The CLR then calls GetProcAddress on the function name (e.g. "ShowWindow") to get the address of where the function is located.

For the most part, it just magically works. If we had used a function like "MessageBox", the CLR would notice that it doesn't exist and would then pick between the ANSI version (e.g. "MessageBoxA") or the Unicode version (e.g. "MessageBoxW").

With the address in hand, it's easy to jump to it and you're all set. Simple and easy.

I was expecting a simple API like this when I was investigating how to register my program as the default handler for ".wav" files on Vista. In the pre-Vista days, most programs would write directly into a registry key for the file extension (e.g. "HKEY_CLASSES_ROOT\.wav") and move on. Problems come when your program wants to register itself as a handler for a "popular" extension like .MP3 or .HTM. Some programs go into an all out arms race with other programs in a fight of wills to make sure they keep the extension.

In Windows Vista and later, Microsoft wants us to use the new "Default Programs" feature. The idea is that you register what file extensions your program supports in the registry and then a nice UI allows people to easily pick which of those extensions they want to associate with your program. Digging around the documentation led me to discover that the bulk of the functionality was exposed via the IApplicationAssociationRegistration COM interface.

Ah, COM.

Over the years, I've tried to keep my distance from it. This irrational fear came from wizards that "next, next, finish"'d your way into thousands of lines of inscrutable code. It took me years of passing glances to finally understand its basics. Even then, when I needed to use it from .NET, I'd right click on my project references and click "Add Reference":

I'd pick the library I needed and then somehow I could use the types as if they were .NET objects. I didn't ask further questions and moved on.

Unfortunately, IApplicationAssociationRegistration was nowhere to be found on the "Add Reference" list since it doesn't seem to have a registered type library associated with it. Using my basic COM knowledge, I knew that if I wanted to use it I would need to know the interface identifier (IID) as well as a class identifier (CLSID) that pointed to a concrete implementation.

Following the MSDN documentation, I knew I'd probably find success in shobjidl.idl:

Sure enough, shobjidl.idl was sitting in my "C:\Program Files\Microsoft SDKs\Windows\v6.1\Include" directory and had this interface definition:

[
 object,
 uuid(4e530b0a-e611-4c77-a3ac-9031d022281b),
 pointer_default(unique),
 helpstring("Protocol URL and Extension File Application")
]
interface IApplicationAssociationRegistration : IUnknown
{
 HRESULT QueryCurrentDefault(
     [in, string] LPCWSTR pszQuery,
     [in] ASSOCIATIONTYPE atQueryType,
     [in] ASSOCIATIONLEVEL alQueryLevel,
     [out, string] LPWSTR* ppszAssociation);

...
}

A little further down was the declaration for the concrete class (coclass) and its associated class id (CLSID):

// CLSID_ApplicationAssociationRegistration
[ uuid(591209c7-767b-42b2-9fba-44ee4615f2c7) ] coclass ApplicationAssociationRegistration
{
 interface IApplicationAssociationRegistration;
}

In the IDL, we also see the definitions for the enums that the functions use:

typedef [v1_enum] enum tagASSOCIATIONLEVEL
{
 AL_MACHINE,
 AL_EFFECTIVE,
 AL_USER,
} ASSOCIATIONLEVEL;

typedef [v1_enum] enum tagASSOCIATIONTYPE
{
 AT_FILEEXTENSION,
 AT_URLPROTOCOL,
 AT_STARTMENUCLIENT,
 AT_MIMETYPE,
} ASSOCIATIONTYPE;

Getting this to work in .NET was surprisingly easy. The basic idea is that the CLR has to have just enough information to find the types:

The "ComImportAttribute" is almost as simple to use as DllImportAttribute. In addition, you need to use the GuidAttribute to specify the gigantic GUIDs.
You use the "InterfaceTypeAttribute" to specify the basic interface(s) that the interface you're importing uses. In COM, all interfaces derive from IUnknown. If the interface supports scripting then it implements IDispatch. If you provide a speedy C++ way of accessing your interface (e.g. vtable definition) and the scripting IDispatch interface, you've got a "dual" interface.
You need to translate the parameter types to their .NET equivalents. This is an incredibly mechanical process that's straightforward. If there is a chance that the underlying bits are different between COM and .NET (e.g. they're not blittable) then you need to use the MarshalAsAttribute to tell the CLR how to convert the types as necessary.
You need to remember that COM handles errors by returning HRESULTs instead of natively using exceptions like .NET uses. By default, the CLR will make the last parameter that is an OUT parameter in the IDL to be the return value (it helps if it's marked by "retval"). Therefore, you can act as if the function really returns its last parameter and the CLR will automatically check the HRESULT and throw a corresponding .NET exception as needed.
Optionally, and perhaps most controversially, you're free de-Hungarianize the parameter names and PascalCase the enum names to make them much more friendly looking to people in .NET. It's optional since it might confuse people that use MSDN documentation and expecting the original names.

In a minute or so, I translated the definitions and gladly got rid of the Hungarian prefixes by converting parameter names of "pszQuery" to just "query." I also converted all the enums and removed their unnecessary prefixes. The end result was this:

[ComImport]
[Guid("4e530b0a-e611-4c77-a3ac-9031d022281b")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
internal interface IApplicationAssociationRegistration
{   
 [return: MarshalAs(UnmanagedType.LPWStr)]
 string QueryCurrentDefault( [MarshalAs(UnmanagedType.LPWStr)] string query,
                           AssociationType queryType,
                           AssociationLevel queryLevel);
 [return: MarshalAs(UnmanagedType.Bool)]
 bool QueryAppIsDefault([MarshalAs(UnmanagedType.LPWStr)] string query,
                        AssociationType queryType,
                        AssociationLevel queryLevel,
                        [MarshalAs(UnmanagedType.LPWStr)] string appRegistryName);
 [return: MarshalAs(UnmanagedType.Bool)]
 bool QueryAppIsDefaultAll(AssociationLevel queryLevel,
                           [MarshalAs(UnmanagedType.LPWStr)] string appRegistryName);
 void SetAppAsDefault([MarshalAs(UnmanagedType.LPWStr)] string appRegistryName,
                      [MarshalAs(UnmanagedType.LPWStr)] string set,
                      AssociationType setType);
 void SetAppAsDefaultAll([MarshalAs(UnmanagedType.LPWStr)] string appRegistryName);
 void ClearUserAssociations();
}

Importing the concrete class that implements the interface was just a matter of specifying its CLSID:

[ComImport]
[Guid("591209c7-767b-42b2-9fba-44ee4615f2c7")]
internal class ApplicationAssociationRegistration
{
 // coclass is implemented by the runtime callable wrapper
}

With all of that goo out of the way, you can use the interface like a normal .NET type:

var aa = new ApplicationAssociationRegistration();
var iaar = (IApplicationAssociationRegistration)aa;
string myCurrentMp3Player = iaar.QueryCurrentDefault(".mp3", AssociationType.FileExtension, AssociationLevel.Effective);

Behind the scenes, the runtime callable wrapper has to do something like this:

Load in ole32.dll where COM functions reside.
Call CoInitialize to initialize COM.
Look up your CLSID and IID in the registry under HKEY_CLASSES_ROOT and find their associated DLL (in our case, "shell32.dll")
Create a factory for your class.
Use the factory to create an instance.
Call QueryInterface to get the specific interface we want (e.g. IApplicationAssociationRegistration)
Get a pointer to the function we want using the vtable.

After all that, we finally have a place to jump to like we did with P/Invoke.

Why bother with all of this? One reason is that Microsoft has a huge legacy investment in C and C++ in Windows. There's no compelling reason for them to rewrite things in .NET. A natural consequence is that the C++ code that implements their latest APIs will be exposed using COM for the foreseeable future. Recently, Microsoft has gone ahead and published .NET COM wrappers for some of the popular new APIs like the Libraries feature in Windows 7. With just a little work, you don't have to wait on Microsoft to do this for you.

Given that .NET was designed as a successor to COM, it's no surprise that Microsoft has made interoperability with it very seamless. The runtime callable wrapper does a good job of hiding most of the messier details. The garbage collector handles much of the bookkeeping involved with memory management that used to be the bane of COM programming. The runtime is very aware of typical COM semantics of when to allocate and free memory. It's not always perfect. Sometimes you can be pre-emptive and force your COM object to be cleaned up via Marshal.ReleaseComObject so you don't have to wait on the garbage collector, but you should be careful.

I just presented the basics of what I learned to get my job done. There's a lot more out there for more advanced scenarios. I've found the free book "COM and .NET Interop" by Andrew Troelsen to be helpful.

There's plenty of obscure Windows APIs out there for the taking. Enjoy!

Migration or new features?

2009-04-15T21:04:25Z

Migration is evil. It’s a pain in the butt. It’s difficult to test. It’s time consuming. And it only gets run by each user only once. Is it really worth it?

As soon as phrases such as “mapping the data,” “data transfer,” “data migration,” or “convert the legacy data” appear in the context of a project, a wise systems engineer or project manager should see red flags waving…

[Legacy Data and Software Project Planning]

All that time and effort put forth into migration could be used instead to implement new features instead. Users are always clammering for your application to do this or that, fix this bug, add that feature. Do they really care about migrating their old data?

But looking at the total user experience of an application, is lost data really ever ok? Especially when it can be prevented? On the face of it, I want to say no, but if users really want the ability to install your application on a flash drive, they might not care that the upgrade forgot which views they had open. Maybe it’s all a matter of impact.

But then again, some users will really care either way. So maybe you’re just screwed…

Linq OfType expression

2009-04-15T18:08:00Z

I just discovered a function in Linq that will save me a bit of typing and allow me to better express myself.

For a long while, I've written this:

ColorWrapper x = combo.Items.Where( obj => obj is ColorWrapper ).Cast< ColorWrapper >().FirstOrDefault(cw => cw.Color == value);

Today, I realized that the framework designers were ahead of me, and provided OfType<T>, which does the same thing as follows;

ColorWrapper y = combo.Items.OfType<ColorWrapper>().FirstOrDefault(cw => cw.Color == value);

Lesson Learned: Pay more attention to ALL of the aspects offered by a new technology, they all have their appropriate uses.

New Webcast: Developing Windows Services

2009-04-14T23:21:28Z

I have a new 6 min 31 sec training webcast up over developing windows services. This video looks at how to build widows services as well as an approach for easily developing them with “console application” support to help ease development. It can be a pain to develop a windows service, if you have to constantly keep deploying it just to test it out.

The video is available through JupiterMedia and can be viewed at http://www.internet.com/video/. Just look for the "Developer Video" titled "Developing Windows Services".

Direct links are:
http://link.brightcove.com/services/link/bcpid1431564240/bclid1433966034/bctid10347017001

#3!

2009-04-03T00:58:58Z

No, that’s not a bash command or a regular expression.

#3 will be joining #2 and #1, and we are pumped.
Well, I’m pumped, my wife is just trying not to feel too sick. She’ll be pumped later, like in 5 years.

Design Pattern Resources

2009-04-03T00:51:10Z

I’ve had a few people ask me for a recommendation of some resources to learn design patterns. There’s a lot of good stuff “out there”, of course, but my response usually says just three things:

Read pretty much anything this guy writes,
Learn SOLID, and lern it güd. I recommend starting here.
and
Let me know if you have any questions and let’s chat.

If you’re anything like me, you’ll find yourself up until 4 AM reading, reading, and reading as you link from post to post and article to article. I swear one of these days I will find the end of the internet.

Incredibly generic procress description

2009-04-03T00:12:00Z

Excerpted from the first draft of my experience report for leankanbanconference.com:

Somebody has an idea to add value. Somebody approves the idea has merit, requests that it be built. Somebody transforms the idea into a set of specific changes that need to be made to the application from a user’s perspective. Somebody identifies how those changes need to relate to the existing technical implementation. Somebody causes chose changes to occur to the system. Somebody confirms that those changes were indeed made. Finally, somebody agrees that the changes did indeed provide the desired value.

It may not stick, but for the moment the incredible level of generic writing has a purpose in the document.