Mark Gamache's Random Blog

Demystifying Certificate Requirements in Mutual TLS

2013-05-20T22:27:00.000-07:00

Understanding Certificates and SSL/TLS long ago became an IT fundamental.

Somehow, the industry seems to have not noticed. In my quest to take fewer calls on this stuff, here is my attempt to help demystify all the certificates involved in Client SSL/Mutual TLS. I seem to be spending 2+ hours a day on the phone talking web and server admins through this stuff.

The reason I am doing this is because Google failed me. There are a lot of docs that focus on all the other aspects of the SSL/TLS negotiation, but none of them focuses on the certificates and what they do for us. Microsoft has one of the better papers on SSL/TLS, but there is a lot lacking around the certificates and the SSL/TLS handshakes are overly summarized.

Here is the handshake that MS shows us.

While this may be a four message handshake, there are a few different conversations taking place, simultaneously, in the handshake. The conversations and their certificate requirements are more easily understood when they are pulled out of the stick context of the message format.

I will say that I am doing a bit of summarizing. SSL/TLS has four major components: authentication, message integrity, key negotiation and encryption. This post focuses heavily on the authentication aspect. I am lumping SSL and TLS together as the basic functions of the negotiations are the same, only the implementation details vary a bit. I am also leaving out a bit about session key generation.

I break the handshake down into 4 different conversations.

A. Hellos

a. The client tells the server that it wants to go secure and offers the suite of ciphers it supports.

b. The server selects a cipher from the list.

c. No certificates are involved here.

B. Server Certificate presentations

a. The server sends over the SSL certificate AND the chain.

b. The server must have the private key for the SSL certificate.

c. If the chain, (CA hierarchy) is not sent, negotiation MUST fail.

d. The client SHOULD perform full cert and chain validation against these certs, making sure they chain to a Root CA that is trusted explicitly by the client.

C. Key Exchanges

a. The client and server exchange data that allows the negotiation of session encryption keys.

b. If EDH is used, then the RSA/DSS key is used to block MITM of the key negotiation.

c. If RSA key negotiation is used, the client encrypts its initial key material with the public key on the server SSL certificates, assuming cert and chain validation succeeded.

D. Client Certificate presentations

a. The server issues a Certificate Request. This lists the key types allowed on the certificate and a list of Distinguished Names (DNs). These DNs are used as a hint to help the client determine which of its many possible certificates is the “right” one.

b. The client selects a certificate, for which it MUST have the private key, and sends the certificate to the server.

c. The client sends data that is signed using its private key, proving it holds the certificate in question.

d. The server checks the signature. If the signature passes, the server may be configured to pass/fail the client based on data in the certificate. Often the certificate serial number or Subject DN must match a static list, or the certificate must be one that is on file. The decision to pass/fail the client cert is not specified in any RFC.

Certificates Needed

	Client	Server
A - Hellos	None	None
B - Server Certificate	Root CA Cert for Trust	SSL Cert and Chain to send
C - Key Exchange	Public key from Server Cert	Private key for Server Cert
D - Client Auth	Client Cert and Private key	Client cert for explicit trust or CA cert to implicit trust. Cert values MAY be checked.

Client and Server Perspective

One often forgotten part of this is that when there are two parties, the role of client and server may change. For instance, let's say A pushes a full DB copy every night to B, and then B pulls deltas hourly during the day. In this scheme, at night, B is the server, but during the day, B is the client. This means that both parties need a cert and a server cert with chain, and they need to know how to implicitly or explicitly trust other clients, and they need Root CA certs so they can trust the server cert when they are the client.

Summary

Take the time to understand how your connections are established and it is fairly easy to figure out which certificates are used when. When it comes to figuring out what is wrong, you can usually figure out the issue by where the handshake fails. The handshake failures usually can only be seen via packet captures.

Hello IT Person, Welcome to the Security Organization

2013-03-29T18:24:00.000-07:00

This is a post that is long overdue. The IT industry went through a revolution and most people in IT missed it and are still missing it.

If you are in any form of IT related job, you are in the information security field.

You may say, “No, I’m just an IT Project Manager (analyst, whatever), security is another team”. You are wrong and your career is heading towards a cliff.

It only takes a tiny bit of Googling to realize that everyone is getting hacked. Even the biggies, RSA, Microsoft, Facebook, Symantec, and Apple aren’t immune. There are too many actors with too many motivations. If it is connected to the web, there is either money in hacking it, or it can be used as a foothold to hack for money. If it’s connected to the web, there is probably someone with a social or political agenda that makes it a target and if not, it is a platform for the hacktivists to leverage. On top of the myriad of highly skilled and motivated attackers, there are thousands off wannabe hackers simply looking for low hanging fruit to test their skills or to get a thrill. Even if you don’t have a penny to your name, your computing power is a commodity if it can be added to a botnet. Hopefully, you already know that it’s a given that everyone is a target. If you still need persuading on this point--don’t worry, there are plenty of Wal-Marts that need greeters. Get your app in early. (That’s “app” as in “application”, which is a paper form you will fill out with a pen. You won’t need LinkedIn for this.—The Editor)

So, why is security your job? Every day (week if you are a slacker?) you make decisions that impact security. It doesn’t matter if you specialize in a niche, like UI or UX; or something broad, like program or project management. If you are working with data in any way, that data has value to your organization. There will certainly be negative impact if the data is compromised or corrupted. Even if you run or maintain a static website in which the content is public and can easily be restored if lost, you don’t want your system to be a foothold into your important systems or take part in a DDoS attack.

Your Role

Backup Operators – Don’t lose the data and don’t lose those unencrypted backup files. This role should be a no brainer. This role has Confidentiality, Integrity, and Availability components.

Systems Administrators – Once again, this is a no brainer. SAs have all the access themselves and they configure access to ALL of your data. Remember, all of your applications sit on a host that an SA “owns”.

DBAs and DBOs – Please, for the love of all that is good, you guys MUST know you have to protect that data. Do you??

Project and Program Managers – I know lumping you guys onto one line will get me all sorts of hate mail. You guys decide things like what use cases exist, for what users, accessing what data. You decide whether to engage formal security teams for assistance. You decide to cut product or project scope, and everyone knows security is the first to get cut.

Developers – You guys are the worst. Yeah, I’m saying it. Inventing your own “cryptography”, passwords in log files, backdoors, assuming users will use your apps the way you want them to, and on and on. Seriously devs, get your acts together.

Network Engineers – I think this is the only role that most folks actually think of as a security role. In fact, this role is the least interesting in terms of security roles. Read those manuals and change those default passwords. And stop using self-signed certs. Can you really not remember how a MITM attack works long enough to say to a manager: “Wait, we need a real cert on that appliance admin portal page”? You know what, if you can’t explain to a manager what a MITM attack is and why your choice of cert matters, you’re part of the problem.

Your day to day job may be focused on something else, but it is only a matter of time until IT folks start getting fired for massively bad security lapses. If it is on the OWASP Top 10 or the SANS Top 25 and you don’t know a bit about it, you may want to pick up that Wal-Mart job app. No one is asking you to figure everything out about security, but you need to at least understand some basics and keep your eyes out. If you do run a system, you’d better become a security expert in the context of that system.

Senators Marco Rubio, Orrin Hatch, Amy Klobuchar hate America

2013-02-08T19:24:00.000-08:00

I posted a snarky blog on H1B Visas last year, but it appears that based on current events, it is time to post again. I doubt I will say anything that hasn’t been said before, but this needs to be said, over and over.

H1Bs are a tool of “The Man” to keep America’s record breaking income disparity on track for even bigger numbers. Yeah! Maybe we can break a new record next year!

Proponents of the H1B, or increasing the allowed number of them, will tell you heartwarming patriotic tales of the U.S. needing these so we can keep companies on shore, in business, and competing against the world. Capitalism is the best thing ever, but when you run out of local resources, you have to import them to keep your business from failing. It’s just supply and demand. We have demand. If these companies can’t get smart employees, they will go out of business and then America loses. Limit the H1Bs and Microsoft will have to fold up shop, then the Seattle area dies with it, then Silicon Valley and LA aren’t far behind. Plus, hey, we are forced to pay all these folks prevailing wage.

All of this is horseshit.

(Sorry, parents who are reading my blog to your kids before bed. I’ll try not to use that language again…)

Why is it horseshit? (Okay. I’m not trying very hard.)

Here’s why:

The H1B is simply access to a pool of lesser-paid employees. In my previous post, I referred to them as indentured servants. (Work at Microsoft for a few months, and you will see it. Comments welcome.)

Further, when the onshore talent is forced to compete against the indentured servants, they have to put in the same kind of hours the indentured do. When it comes to supply and demand, the prevailing wage rule doesn’t let the market come into play. The extra supply of labor hours drives down demand. It’s just that simple.

H1Bs are anti-American and chip away at the American Dream every day. One could make an argument that the visa is just opening a bigger market, thus very capitalist. But seriously, I kind of want America to be around and to have jobs when my kid grows up.

Here’s how the H1Bs are ruining America. If we stop or cut the H1B numbers, the demand for smart people goes up. With demand up, wages go up. Wages don’t just go up for the top tier professionals; they go up for every other tier as well. Crap, there goes our chance at another Guinness Book Income Disparity Record. If the wages go up, the jobs get much more attractive to America’s youth. If demand for those jobs goes up, we need more training institutes for whatever technical jobs are in more demand. Kids will go to college and get smart if there is a path to riches. Right now, average wages for many of these jobs are comfortably nice, but they don’t scream, “Learn me, you can vacation in Park City every year and drive an Escalade!”

The H1B keeps kids from seeing any American Dream, so the ones lucky enough to afford college just get yet another Art History Degree and fill out that McDonald’s job application.

I have met a few folks who have left tech for trade jobs, because the pay is similar and the stress is lower.

One might make a claim that there is a short term need for the smart folk, as American kids aren’t up to speed yet. I concede the possibility, but how do we fix the cycle? On our current path, it seems that we don’t, we just continue to import cheap labor.

The market only works for us if you let it.

If that were the end of it, one might be able to let go. As it turns out, and I won’t pretend to know everything about these companies, many of the big consumers of H1Bs are foreign owned companies who rarely hire Americans and specialize in filling headcount for companies who can’t get H1Bs or have more than their fair share. These companies are funneling their revenue right out of America. These aren’t even American companies in need of talent, hoping they can compete!

Don’t get me wrong, I don’t hate foreigners; I worked with some great ones at MS and many great ones at T-Mobile. What I want is to protect America, its future, and its market for high paying jobs. Keeping wages low in this manner hurts America. We need to fix this for our children.

H1B visas are nothing more than a way to keep money out of the hands of workers and in the hands of execs and larger shareholders.

Taking a look at some of the top H1B sponsors, I can’t help but notice three distinct categories. We have offshore consultancies, household name tech companies, and banks/wall street firms. None of these companies are lacking in cash to bump up the pay of their employees or to fund training and education for the lesser employees with potential. This is pure greed.

Here a look at H1B data for 2001 – 2012

Household Names

Employer	Denied	Requested
Microsoft Corporation	206	38001
IBM Corporation	416	25067
Oracle Corporation	177	12587
Intel Corporation	86	11686
Qualcomm Inc	79	8492
Accenture Ltd	57	6637
Cisco Systems, Inc	125	6273
Google Inc	30	6241
Hewlett-Packard Company	98	5907
CVS Rx Services, Inc. dba CVS Pharmacy	47	5804
Rite Aid Corporation	89	5385

Wall Street

Employer	Denied	Requested
CitiGroup	187	8147
JPMorgan Chase & Co	75	6564
Goldman, Sachs & CO	51	6527
Bank of America	79	4304
Merrill Lynch & Co., Inc	65	4140
Credit Suisse First Boston	55	3760
UBS AG	61	3659
Capital One Financial Corp	65	3648

Offshore Consultancies

Employer	Denied	Requested
Satyam Computer Services, Ltd	163	29824
Tata Consultancy Services	155	17996
Infosys Limited	222	16013
Larsen & Toubro Infotech Limited	83	14105
Wipro Limited	67	13420
Infosys Technologies Limited	55	10139
Cognizant Technology Solutions	368	9375
Patni Americas, Inc	80	8146

Well, at least Steve Ballmer gets to keep his fleet of Ferraris.

…And Larry Ellison gets to keep tropical islands.

…and our children will get to keep…what?

NTLM hasn’t been relevant for like 12 years... and other lies.

2013-01-15T14:32:00.000-08:00

A surprising number of foolish Slashdotters have pointed out that my latest work, breaking the NTLM and LM handshakes and phishing for users’ NT hashes, is totally irrelevant and has been for 12ish years.

As a fan of debate, I’ll start with points that are interesting but have no real bearing on the topic.

Slashdotters are clearly not qualified to make this assessment. Their Appeal to Authority fails.
Microsoft wouldn’t issue an advisory and Fix-It if it weren’t relevant. My Appeal to Authority is better than theirs. ;-)
The existence of one a newer protocol, Kerberos, does not make NTLM simply disappear.
Wikipedia actually doesn’t say NTLM is long dead. Wikipedia as an appeal to authority is a joke. I link to it regularly, not for its completeness, but because it is written for a layman audience. It is a great place to start if you don’t know something.
I’m not a Linux fanboy looking to disgrace MS. I’m a long time MCSE and even gave MS some props in my post.
Finally, my work is what it is, probably the last nail, of hundreds, in the coffin. I make no claim to it being inspired by God.

Now to the relevant details... When MS introduced Active Directory in Windows 2000, they implemented Kerberos 5 as the default authentication protocol FOR DOMAIN ACCOUNTS. This is a pretty important requirement. If a machine is not domain joined, or the account is not a domain account, Kerberos is not an option. The upside here is that when machines are in workgroups, it is much less likely that the accounts will have any sort of value off of the host. However, this does not stop the SSPI from trying to authenticate using your cached account credentials when accessing resources that are not on the host. This means a workgroup host could still be vulnerable to my “Send the Hash” attack if not properly configured.

Even for machines that are domain joined, while Kerberos is the default, NTLM is used in several situations:

If the service is not Kerberos enabled (Kerberized). Maybe it runs on an NT server?
The service/server does not have a Service Principal Name (SPN) registered.
The service/server has duplicate SPNs registered.
When accessing the system by IP rather than name.
Improperly built clusters.
3rd party system implemented incorrectly.
When accessing data across forests, using an older domain type trust.
When the client can’t access a KDC/DC, such as when it is outside the firewall.
When the KDC/DC is behind NAT.

Before getting in depth with a couple of these cases, I’ll make a generalization, “Kerberos is very hard to get right, except under simple conditions.”

Outside the Firewall or Behind NAT
MS’s implementation of Kerberos requires that the clients, servers, and KDC/DCs all be on the same routed network with AD integrated DNS or DDNS that allows the DCs to register SRV records. The clients must be able to find and access the KDCs to get Kerberos tickets. I am not going to cover all the details of Kerberos, but this is a key difference. With NTLM, the server you want to access does the job of finding a DC and getting the DC to validate the challenge/response after your client has done its handshake. The resource server passes the challenge and response to the DC over RPC using packet privacy and gets back a pass/fail and a list of group memberships which it uses to build the user’s access token. This is super simple and easy when the client is outside your firewall. You only need to open one port, the application's port.

If you intend to make Windows Kerberos work across NAT or behind a firewall, prepare for pain. Each Windows client has a component called the dcLocator. Its exact operations vary slightly from version to version of Windows. You might think you just need to open up TCP88 to a KDC and you’re set. You might get a pony in the mail too.

I’ll blog on the exact details at some point, but the dcLocator first needs to find the KDC DNS SRV records in the _msdcs.domainname.org zone. Right off the bat, this means that you need split DNS, as the answers inside your firewall will not be the same IP as outside. Once you have your external DNS zone and main DC records, the client will ping all the DCs and select the fastest to respond. If ICMP is blocked, nothing proceeds. The client sends a CLDAP query to the fastest DC. This is connectionless LDAP over 389 UDP. This query is to ask which AD site the client is a member of. This query is not answered in a traditional way, based on the filter. Instead, AD uses the source IP to map the IP to an AD subnet which maps to an AD site, which the LDAP search response will contain. If your client is behind NAT, then the source IP will likely be a SNAT address. From this response, the dcLocator then does a second DNS SRV query to get the DCs that are in the AD site returned from the CLDAP query. The dcLocator then pings each of those DCs and the first to respond is queried and if the response is satisfactory, then this becomes the default DC for a period of time. This time can vary by OS version. Now we are ready to do Kerberos. Some versions of Windows try UDP88 first and then when they get back the “response too large” they try TCP88 route. If UDP is blocked, these versions may not try TCP88 even if it is open. I will not be swearing to this in court as it has been over a year since I configured this type of scenario and I am writing this without a net.

This means, that for Kerberos to work outside the firewall or behind NAT, you need to:

Setup Split DNS
Create at least one domain level SRV pointing to the external IP address
Create a site DNS SRV record for EVERY DC in the default site, pointing to the external IP address.
Open port 389 UDP
Open port 88 UDP
Open Port 88 TCP
Open ICMP

Kerberos Diagram

NTLM is a lot easier to use in both NAT and outside the firewall scenarios.

NTLM Diagram

Messed up SPN Scenarios

A Service Principal Name (SPN) maps an instance of a service running on a server to the account that it runs under. Kerberos uses shared secrets, between each party and the KDC/DC, to allow for authentication and key exchange. When a client wants to use a service, it asks the KDC for a ticket to the specific service/server combination. The ticket is encrypted by the KDC so that only the service can open it. If there is no SPN registered for the service, a ticket can’t be granted. If two accounts have the SPN registered, the ticket cannot be granted. If the service is running under the context of a different account, the ticket cannot be decrypted.

Clusters

Many web admins just allow their appPools to run as local system or network service. This means they are running under the machine account, which is a domain account and has an SPN or at least can have one. In order to make your cluster work with Kerberos, all appPools must run under the same domain account and usually an SPN must be manually created by an admin. MS has made some changes to IIS to make this easier, but in many cases it is out of the frying pan...

SQL

SQL server tries to register its SPN whenever it comes online. If you are following best practices, and using a domain account to run SQL, then SQL will try to register the SPN attribute on that account. This is a proper configuration, however most accounts so not have the SELF write SPN permission (ACE), and fail. This means that Kerberos won't work and NTLM is negotiated.

Connection by IP Address

This fails as there is not an SPN set that uses the IP, such as HTTP/10.0.0.5. Rather the SPN is HTTP/www.domain.com. This can be overcome by registering the IP SPN, but does not scale well. I’ve thought about building a simple tool to do this, but my list of to-dos is long.

In the end, the threat and attack models are very different for the enterprise and stand alone users.

Enterprise Exposure

The enterprise is likely to block many outgoing ports, making it less likely that “send the hash” attacks will succeed. This however doesn’t mean there is no risk from NTLMv1. With any type of foothold, the attack is VERY effective. All it takes is one compromised host, one badge not checked, a network port activated outside of your secure areas, etc.

Enterprise users spend time in hotels, airports, hospitals, etc. These types of places are ripe for the picking.

Workgroup Exposure

For workgroup members, the threats vary considerably. The user may spend much of their time behind NAT, which may make it hard for an attacker who steals a hash to use it. The user may spend time in coffee shops, etc, making them ripe for attack.

Summary

While Kerberos is more secure than NTLMv2, it is not really fair to say it is better. They both have pros and cons. NTLM is VERY commonly used today both by design and due to it being a fallback for failed Kerberos. Depending on your systems’ settings, you may be sending LM and NTLMv1. ALL VERSIONS of Windows still accept LM and NTLM by default, so you may still be allowing the issue. You may not be initiating dirty handshakes, but you will accept them if offered.

Finally, the fix is very simple. There is almost no downside to making the change. If due to some crazy twist of fate you have a system that only works with NTLMv1 or LM, that system will break.

NTLM Challenge Response is 100% Broken (Yes, this is still relevant)

2013-01-08T10:00:00.000-08:00

Updated as it seems a lot of folks don't read. The reason this attack is interesting is that I steal or phish the hash in the attack WITHOUT physical or administrative access. Yes, I am aware of the 30 amazing tools that "already did that" using local admin access. Also here for why this is still matters.

First off, I can’t say I broke the NTLM handshake; the march of time did it. Apparently I am just the one who bothered to put it together.

There have been numerous whitepapers, hacker conference sessions, and blog posts dedicated to the weaknesses of NTLM (and LM) authentication. However, the weaknesses described in previously published works were theoretical, or required stealing hashes using admin rights. This means the host was already compromised, thus the exploits themselves are a bit boring and redundant. One couldn’t just phish for the hashes or MITM the hashes; due to the challenge response mechanism. The best case for getting to the hash or password from outside the host was to do a MITM attack (or a phish) and substitute a chosen challenge. This only worked if the victim was willing to negotiate NTLM without the Session Security Flag. This would then allow an attacker to build rainbow tables to get the hash or password. Rather, the attacker probably already had tables built for the chosen challenge. This scenario is a pretty high bar to reach.

To frame the conversation, there are actually 4 handshakes in the NTLM suite that move up in security and complexity. They are LM, NTLM, NTLM with session security, and NTLMv2. As of now, only NTLMv2 stands as secure. There is no way, other than encrypting your link with say IPSec, to secure the 3 weak handshakes. The good news is, all Microsoft OSs already have a registry key that can control the handshakes options. MS will release a KB and Advisory on 1/8 on this and the information can be found below.

Now, thanks to Moxie Marlinspike’s Cloudcracker, an attacker can skip the pre-chosen challenge and brute force the challenge response to get the NTLM hash. If the victim is running XP, the situation is even worse, as Cloudcracker will return the LM hash which can always be broken overnight to derive the user’s password .

Why Does this Matter Now?

Before I get to the exploit details, let me clarify how relevant these technologies still are in today’s world. You might think that with all the papers and presentations, no one would be using NTLM...or, God forbid, LM. NTLMv2 has been around for quite some time. Surely, everyone is using it. Right?

Wrong!

According to the last data from the W3 Schools, 21% of computers are running XP, while NetMarketShare claims it is 39%. Unless someone has hardened these machines (no MS patches do this), these machines are sending LM and NTLM responses! While these lists leave out server OSs, 2003 Server still sends NTLM responses by default. Yes, every MS OS since NT 4.0 SP4 has supported NTLMv2, but NTLM and LM were not excluded by default until Vista.

But wait, there’s more! It is also very common for companies that have heterogeneous environments to use Active Directory Group Policy to keep the settings weak, usually out of fear of breaking Samba connectivity. Sure, Samba has supported NTLMv2 for a long time, but most IT folks tend to think “Why beef up security if you might break something? No one is claiming to have broken NTLM.”

Well, here it is: I’VE BROKEN NTLM.

Now, get to fixin’.

More on fixin’ later. (I can’t take credit for breaking NTLM. I’m no math whiz. I just happen to specialize in applied crypto, and I looked in the right place at the right time.)

The Attack

When I read a summary of Moxie’s MS-CHAPv2 crack, I saw that the big deal was not that the implementation had some crazy flaw, it was that Moxie had affordably built a system that can brute force the DES keys that make up the heart of the challenge response mechanism. In less than 24 hours, given a known 64 bit plaintext (challenge) and a ciphertext (response), Cloudcracker can return the key to you.

This made me wonder what else was broken, given affordable DES brute forcing now exists.

I didn’t dig a lot deeper into the attack at the time, as I was researching NTLM so I could write a blog post on Pass the Hash Attacks. I know this is well covered territory, but I never found a paper that covered all my questions, so I figured I’d do it myself. Much of my research was done by reading the protocol details on Eric Glass’s exhaustive page on the topic. I’ve found no better source for understanding the protocols.

That’s then I stumbled across this: (Bold added for emphasis.)

The NTLM response is calculated as follows :

The MD4 message-digest algorithm (described in RFC 1320) is applied to the Unicode mixed-case password. This results in a 16-byte value - the NTLM hash.

The 16-byte NTLM hash is null-padded to 21 bytes.

This value is split into three 7-byte thirds.

These values are used to create three DES keys (one from each 7-byte third).

Each of these keys is used to DES-encrypt the challenge from the Type 2 message (resulting in three 8-byte ciphertext values).

These three ciphertext values are concatenated to form a 24-byte value. This is the NTLM response.

Note that only the calculation of the hash value differs from the LM scheme; the response calculation is the same.

That’s right, to reverse the challenge response, I just need to brute force two DES iterations that have 56 bit keys and one that only uses 2 bytes of the 56 bit key space for the last crack. Moxie shows it like this, for those who are more visual.

I immediately went to Moxie’s posts on the topic to figure out how to use Cloudcracker to brute force out my hashes. That’s when I discovered that MS-CHAPv2 uses the EXACT same math as the LM and NTLM challenge response. Moxie was kind enough to point me to the line in his code that demonstrates how one can submit the challenge and response to Cloudcracker to get back the LM or NTLM hash.

print "CloudCracker Submission = $99$%s" % base64.b64encode("%s%s%s%s" % (plaintext, c1, c2, k3[0:2]))

Just concatenate your challenge, and the first two thirds of your response, and k3 as base64 and Bob’s your uncle. The only thing that threw me was, why is k3 only 2 bytes? To save on processing power, I assume, you must brute force the final key before sending it to Cloudcraker, which just appends it to the first two recovered keys and sends it back. As the last key is the one that has 5 bytes that are always all 00, this is easy.

Using Eric Glasses example, I sent in $99$ASNFZ4mrze8lqYwcMegYR0ZrKbLfRoDz2Ag= to Cloudcracker. This is the challenge 0123456789ABCDEF, low response 25A98C1C31E81847, mid response 466B29B2DF4680F3, and D808, the first two bytes of the final third of the hash, which I brute forced locally. If you decide to follow along with me, make sure to look at Glass’s parity adjusting of the keys. DES keys are really 64 bit, not 56, but as the right bit of each byte is a parity bit, it can’t count toward the total entropy or key space. Or, you can just use his provided Java code.

Then I just had to wait until I got an email with this:

CloudCracker has successfully completed its attack against your CHAPv2 handshake. The NT hash for the handshake is included below, and can be plugged back into the 'chapcrack' tool to decrypt a packet capture, or to authenticate to the server:

cd06ca7c7e10c99b1d33b7485a2ed808

This run took 68799 seconds. Thank you for using cloudcracker.com, this concludes your job.

For those who missed it, I just got the NT hash from the challenge and response, which are easily observed on the wire. I just got the hash WITHOUT compromising the host first. To use this in an attack I just need the right MITM foothold or a phishing email.

To: All Employees

From: HR Communications

Subject: Updates to the Employee Handbook

Body: Human Resources has completed a significant rewrite and update to the Employee Handbook. While some of the changes are minor, it is worth a look for all employees. Employees with aging parents will likely be excited to see the increase in paid time off for emergency care of elder dependents. The guidelines for company events where alcoholic beverages are provided have also been updated.

Finally, with the passing of Washington Initiative 502, we are publishing the new guidelines for Marijuana in the work place.

The handbook can be found here:

\\hrFiles.ru\HRFiles\EmployeeManualv3.docx

Best Regards,

Human Resources

I call this attack “Request the Hash”. Sorry for the pun, but who wouldn’t immediately click the link to find out their companies new pot policy???

The Good News

First of all, I commend MS for changing the default to NLTMv2, as of Vista, and for leaving us the option if we just can’t live without LM or NTLM. At this point though, I think it is incumbent on MS to push the issue at the local security policy level, letting domain administrators override common sense as needed.

There might be some testing and some interoperability failures for enterprises, but the good news is the settings for LM, NTLM, and NTLMv2 have been around a long time. The reg key HKLM\SYSTEM\CurrentControlSet\Control\Lsa\LmCompatibilityLevel allows you to choose the level of fallback your system will allow. Set this to 3 or greater for workstations and you are probably set. This situation is a bit more complex as almost all windows computers are not just clients, but also servers, offering you the admin$ share and c$, etc. The same setting has slightly different bearing on which versions the machine will accept when acting as a server, say when doing remote administration of your desktops. This great article, by Jesper Johansson, covers both the client and server aspect. Personally, I am setting all my systems to 5. If you allow systems to be set below 5, you may be masking the fact that clients are still performing NTLM or LM handshakes.

Level	Group Policy Name	Sends	Accepts	Prohibits Sending
0	Send LM and NTLM Responses	LM, NTLM, NTLMv2 Session Security is negotiated	LM, NTLM, NTLMv2	NTLMv2 Session Security (on Windows 2000 below SRP1, Windows NT 4.0, and Windows 9x)
1	Send LM and NTLM—use NTLMv2 session security if negotiated	LM, NTLM NTLMv2 Session Security is negotiated	LM, NTLM, NTLMv2	NTLMv2
2	Send NTLM response only	NTLM, NTLMv2 Session Security is negotiated	LM, NTLM, NTLMv2	LM and NTLMv2
3	Send NTLMv2 response only	NTLMv2 Session Security is always used	LM, NTLM, NTLMv2	LM and NTLM
4	Send NTLMv2 response only/refuse LM	NTLMv2 Session Security	NTLM, NTLMv2	LM
5	Send NTLMv2 response only/refuse LM and NTLM	NTLMv2, Session Security	NTLMv2	LM and NTLM

You may ask, “Who cares if my server accepts LM or NTLM?” If not all your clients are managed, your server could be unwittingly used to compromise an account used by a client with weak settings.

I did a fresh Ubuntu install and verified that its SMB/CIFS client only sent NTLMv2 response, even when I attempted to downgrade using Cain. I have not yet had a chance to test web browsers that support NTLM auth via SPNEGO. That’s right; most web browsers perform NTLM auth in HTTP Headers when a trusted site requests it.

A Reminder About the Downside of Doing Nothing

While web browsers only perform the NTLM auth for trusted sites by default, generally, Windows OSs seem to make no distinction between trusted and un-trusted zones for other protocols, such as CIFS/SMB, SQL/TDS, and RPC. This means that a phishing attack with a file://server//share link could yield great results for an attacker. Based on the rapid succession of tries and retries, it is safe to assume that the first 2 – 5 attempts by a machine are the OS, via SSPI, trying to auth with the victim’s cached creds. This means the attacker does not need to see a successful auth to trust that the credentials are valid. Maybe you took my advice on Pass the Hash, and blocked all the protocols that use NTLM, at your perimeter. This doesn’t stop one of your clients, say a laptop at home outside your perimeter from trying to do something that won’t try an NTLM handshake. There are just too many ways to fail, if you allow LM or NTLM in any context. OK, so the attacker has the hash, but they are outside the firewall… You’re safe, right? No! At this point, the attacker has a valid hash and is ready to use the account to get your data. Now the attacker has two options:

· If your system is sending LM responses, getting that hash makes for super light work in cracking the password. If you are sending the better NTLM hash, you are still in hot water. Cloudcracker also has a huge set of rainbow tables. For another $20 I can submit the hash and likely get the password of an average user. There are surely plenty of ways inside your perimeter with just username and password. Moxie, if you are reading this, how about a one step operation (from the customer perspective) that pipes the recovered hash into the rainbow tables after it’s been cracked. Maybe a price a tad under $40 for the two cracks?

· BUT WAIT!! Attackers don’t need your stinking password! If I have the hash, it is password equivalent! The first step of every auth involving NTLM and NTLMv2 is to convert the password to the NT hash. Your weak settings just saved the attacker a step. Once the hash has been stolen, the only ways to render it useless are to change the password or remove all of the channels by which it can be passed. (The latter feels infeasible).

Protecting Yourself and Your Company

If you are a lone user, make sure all of your systems are set to LmCompatibilityLevel 3 or higher via local security policy, group policy, or via the registry key.

If you are a corporate type, make sure the GPO is set to force 3 or higher. You may notice that levels 4 and 5 talk about domain controllers refusing different levels. This is not a great protection and may even mask a problem. While the DC blocking the LM auth passed by a client won’t let the user access the system, the damage is already done: The LM response was sent, and the attacker may have it. Sure, it is nice for the attacker to see the successful auth so they know the crack is worth the time, but the rapid fails and retries will tell the attacker that the client OS had the creds cached and the hash that was used is almost certainly good, at a minimum on that client machine, if not the domain.

If you are running a Samba client, enter this line ‘client ntlmv2 auth’ into your smb.conf file.

Final Paranoid Thought

If you have any of the weak settings, CHANGE YOUR PASSWORD after fixing them!

I made this discovery when I realized DES was brute-forcible on a budget. I immediately started thinking of a list of all of the processes that broke along with DES. I’m sure this is exactly the list that every nations’ spy services started making when they acquired the computing power to brute force DES. I assume that it has been affordable for most of them for 5 – 10 years now.

Sorry, spies, if this closes a door for you. I read a book that says another one will open. ;-)

Appendix A: Proof of Concept Code

In order to validate my work, I built a simple tool proof the crypto and create the CloudCracker token. First off, I am not a great or professional coder, so don’t laugh when you look at the code. Second, Eric Glass’s code was a great help. His is in Java, while mine is c#. Last, I used Bouncy Castle’s c# crypto library, for two reasons. The .NET libraries do not support MD4, because for hashing it is VERY weak and the .NET libraries block the use of weak DES keys. NTLM does not account for weak keys.

My code is a fair bit redundant, checking and rechecking itself. This is because each CloudCracker summation costs $20 and I didn’t want to have to send/spend several failed submissions during debugging.

The code can be found here.

The challenge and response data should be in hex, while the password should be ASCII.

Rehashing Pass the Hash

2013-01-04T21:55:00.001-08:00

The first question one might ask is, “Really… Why are you writing about this old news now?” The answer is simple; even with the release of Window 8 and Server 2012, Pass the Hash (PTH) attacks are still incredibly simple and effective. While from an academic standpoint passing the hash is simple to understand, it is a bit more complex from an attacker or defender’s stand point. The reason for this post is that having read a huge number of papers and post on pass the hash, none of them gave me a clear picture.

The reason pass the hash is so powerful is due to NTLM’s initial design flaw. NTLM was essentially broken from the day it was released. While UNIX started saving salted hashes in the 1970s, the designers of NTLM chose to use an unsalted hash. Salting a hash is probably the most important design criterion when designing an authentication system.

A hash is a one way transformation of a variable input to a fixed length output. A hash function must be deterministic, meaning that the same input will always generate the same output, no matter what platform it might be run on. Another key property of a good hash function is that even a small change in the input creates a significant change in the output. The final vital property of a hash function is that it must be truly one way. There must be no method allowing the input to be derived using only the output.

With these properties, and the benefit of hindsight, a hash is the perfect way to for one party to prove they know a password without the other party actually knowing it. Storing reversible or god forbid plaintext passwords introduces a significant number or risks. Only the ignorant do this.

If the hash can’t be reversed, then you might think there are no more risks and you can just hash those passwords and be on your merry way. If we didn’t have a security model to think about, this might make sense. The reason we bother to hash the passwords and not store them as plaintext on our authentication servers is that, on the off chance that the auth server is compromised, we want to minimize the value of the stolen identities. Assuming you kick out the attacker and restore from backup, it would be nice to not have to force all of your users to change their passwords. It would be nice to be sure that the breach wasn’t just a foothold to a larger attack. Additionally, as users tend to reuse passwords, we don’t want to put the users’ accounts at risk on other systems or sites. This brings us to the risk created by just storing the password hash with no salt. If one has access to the hash and wants to derive the input (password), all that needs to be done is test passwords until the outputs (hashes) match. This Brute force method is always a theoretical option when trying to defeat a cryptographic function. Trying all possible passwords is time consuming, which works against the attacker. Unfortunately, with super fast processors and huge amounts of cheap storage, it is now possible to pre-compute and store huge numbers of hashes and the corresponding passwords. In the space of less than 1 TB, you can store tables that cover 96% of 1 to 9 character passwords containing upper case, lower case, and numbers. By storing just the password hash we make it completely worth an attacker’s time to create and save the pre-computes values. Even better, you can just pay when you need to use someone else’s’ tables.

Simply salting the hash wastes all that work the attacker did to pre-compute the hashes. Salting a hash is simply adding random or pseudo-random data to the input before computing the hash. The salt is saved and later used when verifying the password. When verifying the password, the salt is looked up, re-applied to the password that is being tested, and the data hashed. If the result on file and the real-time computed result match, the password was correct. By applying the random data, we have completely wasted the attackers’ pre-computation effort. Ideally, every account uses a different salt, so that even if your user database is taken, every password must be brute forced separately.

Salted Hash Example

Password Salt SHA-1 Hash

V3ryStrong 1 88 93 EB 7A 28 DA A4 95 89 B1 B5 A1 E0 C6 A0 83 9D 38 A3 39

V3ryStrong 2 59 86 41 B2 62 B7 C3 C7 54 27 78 94 FB D6 59 24 5F 77 74 40

The example is purposely weak to make for easy understanding. Usually the salt would be 8 – 16 bytes. It shows the amplification provided by SHA-1. With a single bit change the data is completely different and we see that there is no way to pre-compute all passwords, unless we pre-compute all passwords with all possible salt values, which in not feasible.

While this is fun background data, it really has little to do with passing the hash, as PTH lets an attacker who has the hash assume the user’s identity without ever knowing the password. This is because windows stores the user’s NT hash in Active Directory DIT or the SAM. The first step of NTLM and NTLMv2 handshakes is that the system converts the user’s password to the NT hash, and the handshake does some fun handshake math that proves the user knew the hash, without the hash ever crossing the network.

This system makes the hash “password equivalent”. If an attacker has the user’s hash, they just skip the “convert password to NT hash” and go straight to using it in the handshake. There are many tools for doing this, depending on where NTLM is being used. NTLM authentication occurs in-band with the protocol that is being used to access data or access a system. NTLM auth occurs inside remote file access via CIFS/SMB, TDS for accessing SQL server, RPC for all sorts or remote administration and access, HTTP for web auth, and I’m sure a few more things.

In order for an attacker to pass the hash, they need four things:

1. The valid NT hash (and username)

2. A channel to pass the hash through

3. A server hosting a service that utilizes/allows NTLM auth

4. A client (or tool) that allows the attacker to input the hash rather than the password

In a world where all four elements are available to an attacker, there is no stopping PTH. Keeping in mind the four requirements, defenders can limit, and more importantly, and understand their level of exposure.

Securing the Hash

The most important thing a defender can do is keep the hash from falling into the wrong hands. The hash resides on a computer in two main areas; the SAM (Security Account Manager) db and in RAM, for users that are logged in. There is a common misconception that the NT hash is also stored in the registry for cached logon of domain user accounts on domain joined computers. This is just that, a misconception. Depending on system settings, data is stored that allows a domain user to logon to a machine when no domain controller is available; however this data is an MD4 of the NT hash + username. It might be worth an attackers’ time to pre-compute the hashes for common user names, such as ‘Administrator’.

Looking at the threat model, one can conclude that the horse is out of the barn if the hash is stolen. In order to steal the hash, the attacker had to gain full control of the system. The attacker must either execute hash stealing code as a member of local administrators, or gain physical access to the poorly protected computer. While the horse is indeed out of the barn on that host, the attacker should be able to go no further. At least that is the defenders hope. This is, of course, not the case. If the user has the other three required elements, she can act as the user at will.

Stealing the Hash from RAM

In order for an attacker to steal a user’s NT hash from RAM, the attacker must run code with administrator rights. As with all the advice that will be offered, all the controls are fairly standard and map to basic security concepts.

· Reduce surface area

o Don’t leave sessions logged in unnecessarily, as another administrator’s failure can compromise your account.

o Avoid logging in with administrator rights. If you screw up and run attackers code, they can’t steal the hash if you aren’t an admin.

· Use up to date AV software

Stealing the Hash via Physical Access

We all know that if an attacker has physical access to a host then they own the host. The last thing we want to do is allow the attacker to use that host as a foothold to even more systems and data. In the case of PTH attacks, we probably have good news here. The NT hashes are stored in the local SAM. If the computers are not domain joined, the username and password usually don’t cross hosts, unless you took the time to sync up your usernames and passwords across them all. As I mentioned earlier, the cached domain credentials DO NOT contain the NT hash. This means the SAM from one computer is of no value on anther.

The one place where there IS a great deal of risk is in an enterprise, where standardization probably means that the local admin password is the same across every domain member. This means that the compromise of any one server is equivalent to the compromise of every server. This means that that dev server that you left up running PHP 1.2 can be a great foothold to the rest of your servers and data.

To reduce this risk:

· Use full disk encryption that requires a token or Password Based Key Derivation Function

· Enable syskey protection > 1.

o This requires a password or floppy to boot and decrypt the goodies in the SAM.

o This scales poorly for an enterprise.

· Make sure all local admin passwords are different.

Stealing the Hash from the Network

The NTLM and NTLMv2 handshakes assure that the NT hash itself can be verified, without it ever appearing on the network. There is nothing to see here.

If an attacker were to figure out a way to get the hash from the network, it would be an impressive feat.

Securing the Channels

If the attacker steals the hash, but has no path to pass it over, then the hash is of little value. One nice thing about NTLM auth is that it occurs in band with the protocol utilizing it, so as a defender, you can better understand and limit your exposure. If you block ports 80, 138,139, and 445, but open 1433, you know that the stolen hash can only be used to access SQL data but not remote file access or websites.

Controls to consider:

· Reduce Surface Area. Only open the ports you need. Duh…

· Authenticate end points with VPN or IPSec.

Stop using NTLM

This one is not so simple. One can “kerberize” all sorts of servers. That is, allow them to utilize Kerberos authentication rather than NTLM. Microsoft has done a pretty good job kerberizing applications, but in almost all cases, if Kerberos fails, the app will allow fallback to NTLM.

I know of no options for a defender here. This section is really a call to MS, and others, to allow for more granular options related to NTLM auth, Kerberos, and fallback options.

I suppose this is an opportunity for firewall vendors as well. It would be fairly easy to block NTLM auth and allow Kerberos on a per protocol basis. I suspect an F5 ninja could write a crazy iRule to do it too.

Stop Clients from Allowing the use of the NT Hash

Assuming you have dealt with the firewall rules, most attackers will have no channel to pass hashes over. The last option would be to pass from hosts inside your network. This seems plausible, as the attacker got enough access to steal the hash.

There are two classes of tools that let you work from the hash rather than the password. The first are tools like WCE, which run on windows and allow an administrator to inject arbitrary usernames and hashes into a user’s session. This allows all subsequent uses of any tool in that session to authenticate as the injected user. This means that any Windows tool that speaks NTLM will work. The second class are custom tools that may run on any OS and speak the protocol in question, be it SMB, TDS/SQL, RPC, or whatever, AND they speak NTLM taking a hash rather than password.

For a good list of current tools and tons of great info on PTH, see Still Passing the Hash 15 Years Later.

A good list, granted probably dated, of AV test results for detection of the tools, can be found here.

Controls to consider:

· Make sure EVERY host inside your firewall has up to date AV.

o Enforce this with NAC or NAP.

o Make sure hosts with AV match up to your CMDB, such that all hosts are accounted for.

· Consider AppLocker and Software Restriction Polices.

o Make sure to block known tools by hash so the attacker can’t just rename at tool

o Make sure to block known tools by name, in case the attacker makes code changes but chooses to use the same executable name.

o Consider white listing known good apps rather than trying to blacklist known bad ones.

· Minimize the number of local administrators. A local admin can stop AV or add exclusions prior to copying over and loading up tools.

· Keep up on the latest version of the tools and test your AV yourself.

Understanding Why Pass the Hash Still Works

The best way to learn this stuff is to go to Eric Glass’s NTLM page and do the math along with him. I am borrowing heavily here to summarize for lazy. ;-) Well, there is a lot more great stuff on his page that you don’t need to know to understand PTH.

There are three different types of NTLM responses:

1. NTLM Response

2. NTLM2 Response, not to be confused with NTLMv2 Response

3. NTLMv2 Response

NTLM Response

1. Convert password to Unicode and apply MD4 hash. This is the NT hash.

2. Pad the NT hash with 0s to make it 21 bytes.

3. Break the 21 bytes in to 3 keys of 7 bytes each. This is 56 bits.

4. Apply odd parity to each key. This makes each key a valid 64 bit DES key.

5. Encrypt the NTLM challenge (from the server) with each of the three keys.

6. Concatenate the three encrypted values, creating the response.

7. Send it.

As you can see, the challenge is provided by the server, and could be substituted by an attacker with network access. This allows an attacker to create pre-computed responses (rainbow tables) for that challenge. Obviously this is scary.

NTLM2 Response

Once again, not to be confused with the NTLMv2 Response. This one uses nearly the same math as the NTLM Response, but let’s the client add a “client challenge” thus negating the value of an attacker substituting their own challenge. This is used when the 0x00080000- NTLM 2 session security flag is set.

1. Convert password to Unicode and apply MD4 hash. This is the NT hash.

2. Pad the NT hash with 0s to make it 21 bytes.

3. Break the 21 bytes in to 3 keys of 7 bytes each. This is 56 bits.

4. Apply odd parity to each key. This makes each key a valid 64 bit DES key.

5. The client creates a random 8 byte client challenge.

6. The server challenge and client challenge are concatenated and MD5 is applied, creating the challenge data.

7. Encrypt the challenge (MD5 output) with each of the three keys.

8. Concatenate the three encrypted values, creating the response.

9. Send it.

NTLMv2 Response

This response is next level. It is totally different that the first two and includes a timestamp, the domain the user belongs to, as well as target information.

1. Convert password to Unicode and apply MD4 hash. This is the NT hash.

2. Concatenate the upper case Unicode username with the Unicode authentication target.

3. Perform HMAC-MD5 on the result of step 2, using the NT hash.

4. Construct blob, containing the Blob signature, 4 bytes of all 0s, the time in NT time format, the client challenge, another 4 bytes of all 0s, and the target information from the server Type 2 (challenge) message, and another 4 bytes of all 0s.

5. The server challenge form the Type 2 message is concatenated to the blob from step 4.

6. The blob from step 4 is then HMAC-MD5’d using the result of the HMAC-MD5 in step 3.

7. The HMAC response from step 6 and the blob from 4 are concatenated and constitute the response.

8. Send it.

Conclusion

Pass the hash attacks are alive and well and are not going away anytime soon. Protect your systems.

Microsoft Announces Plan to Further Drive Down Worker Wages

2012-09-30T10:14:00.000-07:00

Microsoft's latest bid to avoid paying employees market rates is genius. Everyone knows, if you want to get public and government support, you have to "Do it for the kids". Offering to put up money for science education is incredibly philanthropic, I mean self serving. I'm amused by this very small bribe attempt. It is in extreme contrast to former MS CEO Bill Gate's extreme giving with no ask for compensation.

Make no mistake, the US is full of brilliant tech people. There are more than enough to go around. The reason there are 1000s of tech startups that you've never heard of is because these people are everywhere. Why hasn't MS retained them? First, MS did employ a huge number of them and drove them away screaming. Second, MS hasn't shared the wealth for years. In the 80s and 90s they were still churning out millionaires, but now the best you can hope for is a very comfortable living. As long as startups offer the chance to get a little rich there will be an appeal. As long as MS crushes innovation internally and denies employees a chance to make a difference, startups will be attractive.

There is no doubt in my mind that if MS brought back the opportunity of the 80s and 90s, both for pay and to make a difference, they would have no trouble attracting the talent they need.

I do agree that the US needs to put more focus and money into science education, but really, what kid is going to get excited about a career in software when MS has succeeded in pulling in all the low paid foreign developers and keeping the market rate for devs low? My short stint at MS very closely matched what was described in the VF article linked above.

You know how you encourage people to learn?? Put high paying jobs out there that will allow the people to make money and have a life. Who wants to work 60+ hour weeks and never see their kids, all for a rate that is barely or below market rate?

My final point, this $10k bribe per head isn't even a market rate bribe. The Visa makes the employee an indentured servant. Work 60 hour weeks or we will give you that dreaded bad review and send you back to the developing world where you came from. If "We the People" are going to take a bribe to keep wages low, we need a bigger cut. Let's start thinking about $50k a year. I bet if we add that $50k to a base MS salary, they could attract all sorts of on shore talent, but alas, those workers might want to spend time with their kids.

I found this fictitious quote from fake MS Talent Acquisition VP Walter Van Hert: "We at MS are dedicated to acquiring the best and brightest minds, at well below market rate, because it is an honor to work for Microsoft".

Microsoft, STOP WITH THE SOCIAL!

2012-09-16T12:46:00.000-07:00

Researching some nuanced Microsoft systems information has caused me to put this string in my always open cut and pasted text file: site:microsoft.com -site:social.technet.microsoft.com -site:social.msdn.microsoft.com

I have searched for hundreds of bits of MS info and looked for hundreds of MS solutions to issues via Bing and Google. I HAVE NEVER FOUND AN ANSWER in any of the MS social sites. I have found plenty of people with the same problems as me, but so far no solutions. Occasionally I have found a band-aide or reallllly bad advice that hides the problem, but never a meaningful answer.

Please join me in not even getting MS social sites back in your search results.

Microsoft, YOU MAKE BILLIONS a year. Please stop asking your users and customers to provide support for each other. If you insist on using social, spend some money and pay really smart people who are subject matter experts to get involved. Moderate the crap out of social.

STOP WASTING MY TIME! If I wanted to rely on other users for support, I'd have gone open source.

Proper PKI Configuration for SSL/TLS Servers

2012-03-15T19:56:00.000-07:00

Don’t Teach Bad Habits

A large number of servers and applications are built with SSL configured incorrectly. If your SSL is misconfigured, it is as good as no SSL. In reality, it may be worse than no SSL, as it gives the feeling of safety where none exists. This is most common when developers and administrators leverage a framework that they don’t fully understand.

Any time a user sees one of the following errors or warnings, the server admin has failed. A user, no matter how technical, should NEVER be forced to answer these questions. Every time a user is trained to hit Yes, or OK to one of these errors, the user is being trained to fail.

Figure 1 IE SSL Error

Figure 2 Firefox SSL Error

Figure 3 Opera SSL Error

Figure 4 Chrome SSL Error

Different browsers may have slightly different text for different failures, but most users will think of them as the same thing, “The place where I have to do an extra click to get to my site”.

Who is Protecting Whom

This document only covers standard, run of the mill SSL, where the server has a certificate and the client does not. Two way or Mutual TLS is not commonly used and not covered here. One of the things that confuses people about SSL is that it is designed to allow the client to protect itself and provides on protection or authentication of the client for the benefit of the server. The server operator or application owner is supposed to be giving the client everything it needs to make sure the connection is secure. Presumably the user is going to submit some data that he or she does not want disclosed. If you are running an application where the liability is not on the client, standard one way SSL is not enough for you. You are assuming a risk that you have no control over and you need additional controls. This is the case with most online banking in the US. If the customer loses his password to an attacker, and the attacker drains the account, in many cases, the bank is liable. There have been a few cases recently where banks have been able to successfully put the burden on the customer.

It is safe to assume that if your application is misconfigured such that the client cannot accurately verify that the connection is secure, your end user will be able to make a successful case as to why the risk should be shifted to you, despite any Terms of Service or other contract that may state otherwise.

Help Them Help Themselves

There are many criteria that the client can use to determine if it should trust a connection to a server. While some clients may chose to ignore some of the criteria, it is wise to do everything right, and avoid any issues.

Subject Common Name

The Subject of the certificate is a description of the entity that offers it to a client. To those familiar with LDAP, it may look like and x.500 Distinguished Name (DN), and it is in fact referred to as a DN. For example:

subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com

However, most of the DN is arbitrary and with the inclusion of E=, for email address, it has no hierarchical convention like an LDAP DN. While some lament this inclusion, unless there really is an LDAP to look at, who cares if the data is actually hierarchical…

There are two exceptions to the “arbitrary” nature of the subject. First, parties may choose to check the subject for a specific value. Cisco VPN concentrators, for instance, can check that a certain OU or L is included.

By far the most important, for SSL/TLS is the CN, or common name. Most clients check to make sure that the CN matches the name that the client is attempting to connect to. This may be the most common configuration error. The reason for this is that as a web developer, you can’t control what the user enters in their address bar. Maybe the user will type google.com or maybe they will type www.google.com, or they might try and access some named service like maps.google.com. They may even type in an IP address. Often two or more of the sites may be hosted on the same server at the same IP address. To keep things nice and consistent, many web developers may redirect the connection to the pretty named site. This is how google does it if you enter http://google.com into the address bar:

HTTP/1.1 301 Moved Permanently

Location: http://www.google.com/

While this works great with non-SSL sites, this will fail with HTTPS unless you plan correctly. The reason that this fails is that the SSL negotiation happens BEFORE HTTP data can be passed. You can’t have an HTTP redirect if the SSL negotiation fails. This is the reason that many think that you cannot use host header based virtual sites with SSL. They are wrong, you just have to know what you are doing and get the right certificate.

We have two different options, if we want a certificate to work with multiple names.

Wild Cards Certificates

One answer to the problem of name matching is to just get one certificate that can answer for all hosts in your domain. RFC 5425 defines a wildcard that can match any host in a domain, so the subject common name *.google.com would work for www.google.com, google.com, and mail.google.com. While some clients seem to allow the wildcard to match domains as well as hosts, the RFC forbids this, so avoid it. For example, *.google.com should not work for www2.maps.google.com, as only the leftmost name is the host portion. Everything to the right is domain, and the * is not supposed to figure into domain matching. The domain here is maps.google.com.

Subject Alternate Name

The other option, especially if you are using host headers for sites that do not share the same domain name, is to use the subject alternate name or SAN. The SAN is a list of names that can also be checked by a client. One certificate could then be used for www.google.com and www.gmail.com. While clients are supposed to be able to parse both the subject common name and the SAN list, some do this poorly, therefore it is wise to add the subject common name to the list of SANs as well.

Picking the Right Name

As stated above, the client will match the name it thinks it is connecting to to the names on the certificate. In the case of a web browser, this is the name in the address or URL bar. This means that if your site is behind a load balance, then the common name must match the DNS name that resolves to the load balancer. The client has no idea there may be a cluster and load balancer.

Figure 5 Typical Web Cluster

In Figure 5, the proper common name for the certificate would be www.google.com.

In some corporate environments, users may bypass the load balancer in certain circumstances. In this case, the certificate may need SANs or to use a wild card.

Time Validity

Certificates contain a Valid From and Valid To date. The client will check to verify that the date of the connection attempt is between the two dates. Don’t let your certificates expire.

Certificate Revocation List Distribution Points

Generally it is thought that only PKI operators need to be concerned with CRLs and CDPs (CRL Distribution Points). This is not the case. A CRL is a list of certificates that the issuing CA has determined are not to be trusted. Not all certificates list a CRL, obviously if one does not exist, you don’t care about them. If the client determines that the certificate it is inspecting is listed on the CRL, it should reject the connection. The reason you should be concerned with CDPs, as an application owner, is that your clients will attempt to read them. Different clients react differently when they fail to access the CRL. Some clients will fail open and allow the connection and some will deny it. If you know your clients are behind highly restrictive firewalls, you may need to make sure that they have access not just to your site, but also to the CDP. This is a common place to fail in a B2B or corporate environment.

Basic Constraints, Key Usage, and Enhanced Key Usage

A good PKI book or web site would never lump these all together, but this is neither of those. Each of these is a field that can describe how the certificate or the associated private key can be used. Some clients may choose to make decisions based on these. IE for instance will allow and SSL connection to a server with no EKUs listed, but if the EKU section is used and the server authentication usages is not listed, it will deny the connection.

If the Basic Constrains listed in the certificate of the CA that signed your SSL certificate doesn’t list it as a CA, the client should abort the connection. This keeps someone from using a certificate and private key as if it were a CA. In the past, vendors forgot to check this, and anyone who bought a legitimate SSL certificate could use it to sign other certificates that were otherwise cryptographically sound.

Key Usage should list Digital Signature and Key Encipherment; if Key Usage is defined without Key Encipherment the client should abort the connection. If Key Usage is not listed, all uses are allowed.

The Chain

The server’s SSL certificate does not exist in a vacuum. It is at the bottom of a PKI hierarchy. Without the context of the PKI the SSL certificate is useless. The beauty of using PKI to secure (authenticate, negotiate keys and provide integrity checks) an SSL connection is that this can be done with a relatively small number of trusted third parties. One of the big reasons to leverage PKI is to avoid the risks and overhead of increasing the number of parties you, or your clients, have to trust.

A quick reminder about key management; there is NO WAY to exchange key material securely, over an un-trusted network, without an out of band trust being established. Diffie–Hellman does not provide authentication, so there is no way to know which party you are negotiating with. RSA key pairs are useless as well, as the public key must be distributed securely in order for it to be trusted. This is why OSs, Java, web browsers, and such have keystores as part of their distributions. Ponder this; did the browser or java package you downloaded come over via SSL or could an attacker have poisoned the keystore in the package?

The root CA is the most important element in a PKI. The trust it conveys is significant and makes it easy to extend trust to a huge number of parties. In order to extend this trust, subordinate CAs can propagate that trust. It is common for a well built PKI to be two or three tiers.

Figure 6 Typical 3 Tier PKI

To increase security and usability, one should never ask a client to trust a subordinate CA or SSL certificate explicitly. Trust should always extend implicitly from the Root CA. Any time you ask an end user to make the decision to trust a certificate, YOU HAVE FAILED.

It’s alright to configure dev and test wrong, right? NO!! None of your testing is valid if you don’t configure your dev and test correctly. Additionally, dev and test are your chance to make sure you fully understand how to configure your systems correctly. In the parlance of sports, “You play how you practice”. So don’t practice failing. It is well worth the extra 10 minutes to get this right.

This hierarchy is built so that a security failure at a lower level can be eliminated by simply revoking a certificate and effectively pruning an entire branch of the PKI. Figure 6 shows the chain belonging to webmail.T-Mobile.com, whereas Figure 7 shows the entire PKI. If the A1 CA, in Figure 7, were compromised, the root could simply revoke the A1’s certificate and A1 and everything below it would no longer be trusted. This has no negative effect on L1C or L1B or anything below them.

Figure 7 Big PKI

This type of design reduces the overall damage that can be done by a successful attacker. Usually the root is offline in a vault, so it can’t be hacked.

If the client only trusts the Root, but the SSL certificate is signed by an issuing CA, like L1C in Figure 7, how does the client bridge the gap and get the trust extended to it? The chain MUST be sent by the server as part of the SSL negotiation.

This is another area of common misconfiguration.

Figure 8 Wireshark sees 3 certificates

We see in Figure 8, a properly configured server sending the SSL certificate, the Issuer CA certificate, and the intermediate CA certificate. If we look at the Intermediate CA certificate, it will list its issuer as the Root, Entrust, which the clients hopefully trust. This is enough data for the client to “build” the chain and cryptographically verify that no monkey business has occurred.

As part of building the chain, the client will perform all the checks above, time validity, not revoked, etc, on each certificate in the chain. If any check fails, the client should abort the connection. If the chain successfully builds to the root, then the client can allow the session and security is assured.

Forcing SSL

Another common mistake is not forcing SSL and simply enabling it. Some site owners and designers wrongly assume that the only time they need SSL is during authentication. If the user’s authenticated session has any value at all, then the entire session must be SSL protected. HTTP sessions are trivial for an attacker to hijack. Additionally, configurations on a site that mix HTTP and HTTPS are prone to human error induced problems. Rather than trusting that links include HTTPS or that only certain pages are set to require SSL, instead set the entire site to require SSL. All the arguments about the extra processor load of SSL haven’t been relevant since 2003 or so.

Why All the Failures?

It would be nice if there was an easy to identify, universally hated, body like congress that could be blamed for SSL failures. Vendors meet this requirement well. SSL certificate vendors point their fingers at the application and OS vendors and the application and OS vendors blame the SSL certificate vendors and the open source community. The overall problem is that there is a knowledge gap between the vendors and the admins who setup the servers and applications. Very few admins know what the chain is, how to send it, or how to verify that it is being sent.

Verifying the Chain is Sent

Getting the chain sent, in the first place varies by application and platform, so it is best covered later. Checking that the chain is sent is easy. Simply install openSSL and run this command:

openssl s_client -connect host:port

This returns a very handy output full of useful information, including:

Certificate chain

0 s:/C=US/ST=Texas/L=Frisco/O=T-Mobile USA, Inc./OU=USA/CN=webmail.T-Mobile.com

---

0 is the SSL certificate and the numbers go up. The subject and issuer DNs are listed to make an easy visual verification of the chain. You can see that this application does not send the root. If you wish to verify the entire chain cryptographically, you can use the –CAfile argument and reference your Root CA certificate, in PEM format.

This would include the following line, assuming the chain checked out:

Verify return code: 0 (ok)

Sending the Chain

This section is the real reason this document was written. There is very little guidance on the web telling admins to send the chain or how to send the chain, and often vendor documents brush over this and or don’t even cover it.

Windows

This section may be a bit of a generalization, as many applications can run on windows that use their own SSL code. If the application uses the OS’s SSL, CAPI or CNG, then the OS will attempt to build and send chain based on what it finds in the machine store, or the store of the account running the server or application. This will apply to all MS server applications, such as IIS, Exchange, Active Directory, etc. Unless the developer overrides the default behaviors, this will apply to .NET apps as well. To make sure that the OS can find the certificates in the chain, make sure that they are all in the same “My” or “Personal” store that holds the SSL certificate. Windows can find the certificates in the other stores as well, but it is nice to know there they are and be clear which certificates are which.

Figure 9 Windows Machine Stores

Java

No matter which OS your JVM is running on, or the version of JVM you are running, there is one keystore that is part of the build. The cacerts file found under lib/security holds the list of trusted CAs. Java keystore files use passwords. The default password for your cacerts file is ‘changeit’. This is the list of CAs that you JVM will trust by default. If a client side Java application will use SSL, this is the store that the trust will be based on, unless the developer made a decision to use a separate store. The Java keystore format is proprietary, and is sometimes referred to as JKS. Sun provides the executable ‘keytool’ to create, edit, and inspect keystore files. Newer versions of keytool and Java seem to make it harder to mess up creating the keystore, but in older versions mistakes are very common and allowed by keytool. The JKS file format is a password protected wrapper that should contain all the CA certificates in your chain. Inside that wrapper is another password protected wrapper that holds the private key corresponding to your SSL certificate. For best results, when creating your keystore file, import the CA certificates starting from the root and working down, installing the signed SSL certificate last.

The process goes like this:

Italicizes text is entered by you. Non-italics is command output.

Create the Keystore and Key Pair

From command prompt type the following to create keystore.

keytool -genkey -alias sslcert -keysize 2048 -keyalg RSA -keystore keystore.jks

The following attributes must be specified. Remember what the password is since it will be needed when setting up server and importing signed certificates.

Password: password

First and Last Name: (must be the FQDN of the server or the F5)

Organizational Unit: (ex. used Operations in particular example)

Organization: (ex. used Company USA.)

City or Locality: (ex. used Seattle)

State or Province: (ex. used Washington)

Two Letter Country Code: (ex. used US for United States)

Once attributes have been inputted, user is asked to verify information.

Select "y" and hit

The user is then asked for key password for < sslcert >, use same password type above and hit

Keystore has now been created.

Create the Certificate Signing Request

From the command line enter

keytool -certreq -alias sslcert -keystore keystore.jks -file ssl.req

Get the file ssl.req to the CA admin or vendor for signing.

Once the Certificate has been signed, the admin or vendor will return the certificate file. In the example, this is certnew.cer.

Import the CA Hierarchy Certificates

Before importing the signed certificate into your keystore you will need to establish the trust by importing the CA Certificates. Always start with the root CA and work your way down. This example will show a three tier PKI.

keytool -import -trustcacerts -alias MYRoot -keystore keystore.jks -file "Company USA Root CA.cer"

Enter keystore password: password

Owner: CN=Company USA Root CA

Issuer: CN=Company USA Root CA

Serial number: 311a0088059a3abe4afb6520a6f749d8

Valid from: Thu Jan 04 12:54:35 PST 2007 until: Mon Jan 04 13:00:47 PST 2027

Certificate fingerprints:

MD5: 82:0D:0C:3C:24:09:D6:9C:84:A7:6D:C3:CA:44:F8:C0

SHA1: 5F:62:6A:91:5D:0D:26:F6:FB:AF:76:92:8C:2A:F1:C6:CD:D2:66:0F

Trust this certificate? [no]: yes

Certificate was added to keystore

keytool -import -trustcacerts -alias MYInt -keystore keystore.jks -file "Company USA Intermediate CA 01.cer"

Enter keystore password: password

Certificate was added to keystore

keytool -import -trustcacerts -alias MYIssuer -keystore keystore.jks -file "Company USA Issuer CA 02.cer"

Enter keystore password: password

Certificate was added to keystore

The alias creates a friendly name to list the CA in the keystore. Each alias must be unique, so make up useful, but unique names.

Import the Signed Request File

The signed request file must be imported back into the original file.

keytool -import -trustcacerts -alias sslcert -keystore keystore.jks -file certnew.cer

Enter keystore password: password

Certificate reply was installed in keystore

You now have a properly built Java keystore.

openSSL

openSSL is a fantastic tool and set of libraries that is leveraged by many applications. OpenSSL’s native format for dealing with certificates, certificate signing request, and keys is the PEM format. OpenSSL is the most versatile tool for dealing with certificates and keys, as it can convert to and from almost any format, except the Java keystore.

A typical PEM formatted certificate looks like this:

-----BEGIN CERTIFICATE-----

MIIFuzCCBKOgAwIBAgIETBsbdDANBgkqhkiG9w0BAQUFADCBsTELMAkGA1UEBhMC

VVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xOTA3BgNVBAsTMHd3dy5lbnRydXN0

…

FPagpeXsGSr0ipNqRy6m1T2mAO3Ok2B4hK6pBnhtf2n7oY1LB/e1rSIHTivJ9eaN

rOCwFjRf+bcELIi2manBx9zI0Qy8JES3vzXBG3xMkJ4Vv3QI93Uk4qn3jodtCPNj

DZZTUPbnHqRshXE6zSB7S6BvRbsmaYps9JF1+j5e3Q==

-----END CERTIFICATE-----

The chain file is just the individual CA certificates in one file, end to end, starting from the issuing CA, moving up to the root, like so:

-----BEGIN CERTIFICATE-----

MIIFuzCCBKOgAwIBAgIETBsbdDANBgkqhkiG9w0BAQUFADCBsTELMAkGA1UEBhMC

VVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xOTA3BgNVBAsTMHd3dy5lbnRydXN0

…

rOCwFjRf+bcELIi2manBx9zI0Qy8JES3vzXBG3xMkJ4Vv3QI93Uk4qn3jodtCPNj

DZZTUPbnHqRshXE6zSB7S6BvRbsmaYps9JF1+j5e3Q==

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

MIIE8jCCA9qgAwIBAgIEOGPp/DANBgkqhkiG9w0BAQUFADCBtDEUMBIGA1UEChML

RW50cnVzdC5uZXQxQDA+BgNVBAsUN3d3dy5lbnRydXN0Lm5ldC9DUFNfMjA0OCBp

…

PFS+/i/qaZ0cHimbltjI/lGQ8SSmkAaz8Cmi/3gud1xFIdlEADHzvjJP9QoyDfz8

uhZ2VrLWSJLyi6Y+t6xcaeoLP2ZFuQ==

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

MIIEnzCCBAigAwIBAgIERp6RGjANBgkqhkiG9w0BAQUFADCBwzELMAkGA1UEBhMC

VVMxFDASBgNVBAoTC0VudHJ1c3QubmV0MTswOQYDVQQLEzJ3d3cuZW50cnVzdC5u

…

RfPB+JbFi1WkzGuDFiAy2r77r5u3n+F+hJ+ePFCnP1zCvouGuAiS7vhCKw0T43aF

SApKv9ClOwqwVLht4wj5NI0LjosSzBcaM4eVyJ4K3FBTF3s=

-----END CERTIFICATE-----

Note that the begin and end certificate statements are left in place.

To create a key pair and certificate signing request do the following:

openssl req -new -newkey rsa:2048 -nodes -out file.csr -keyout file.key

This will output your key files and CSR. The vendor or PKI admin should sign the CSR and return your SSL certificate as well as the CA certificate in your chain.

With the returned files and openSSL, the sky is the limit.

Apache

First off, there are a couple of ways to configure HTTPS on apache. The main methods are via mod_ssl and Apache-SSL. Apache-SSL has not been updated in a while and should be avoided.

Mod_ssl uses openSSL libraries, so you will need openSSL to do anything fun. The native file format for openSSL is called PEM or base64, which are text files, rather than the DER format, that is binary.

The file that will contain your SSL settings is the httpd.conf file. There are three settings that you will care about.

SSLCertificateFile - This is the actual SSL certificate

SSLCertificateKeyFile - this is the associated private key file. If there is a pass phrase on the key, it must be entered at startup. This may not be appropriate for some environments.

SSLCertificateChainFile - This is chain file as described above.

F5 Local Traffic Manager – BigIP

First of all, you probably should not be terminating SSL on your F5. “Need to know” or “reduced surface area” says that you don’t want the F5 techs to be able to see your decrypted traffic. There are few good cases for decrypting SSL at the F5. If you need to inspect host headers or URLs, this may be necessary.

F5 uses openSSL so the section above will help get the needed key and cert material.

In the SSL Client Profile, there is an option to set the Chain file. Simply copy over the chain file in the openSSL format and apply the setting.

See http://support.f5.com/kb/en-us/solutions/public/6000/400/sol6401.html

And http://support.f5.com/kb/en-us/solutions/public/10000/100/sol10167.html

Forcing SSL

SSL should be forced at the web site level, to eliminate human error.

IIS

In IIS you can define the SSL settings at the site level, virtual directory level, or the page/object level. The best place to do this is at the site level.

Figure 10 Requiring SSL in IIS6

Figure 11 Requiring SSL in IIS7

In order to make a better experience, consider changing the default 403.4 HTTP error page to a redirect page, so the user does not get an ugly error.

http://raoulpop.com/2007/08/07/automatic-redirect-from-http-to-https/

Apache

In the httpd.conf file, use the directive SSLRequireSSL to require SSL all the time. http://httpd.apache.org/docs/2.0/mod/mod_ssl.html#sslrequiressl

In order to make the end user experience better, consider implementing a redirect from HTTP to HTTPS, rather than just blocking HTTP. http://www.besthostratings.com/articles/force-ssl-htaccess.html

F5 LTM

The LTM does not have extra ports setup by default. If you do not setup the HTTP port, the user will get a server not found error. Consider creating a VIP and iRule that redirects the user to your HTTPS site.

The iRule would look like this:

when HTTP_REQUEST {

HTTP::redirect "https://[HTTP::host][HTTP::uri]"

}

This simply redirects the browser, rewriting the URL with and HTTPS replacing the HTTP.

Not Covered

A few things may seem to be missing here. This document focuses on PKI related configuration. Different apps and OSs also have other SSL/TLS settings that are not directly related to PKI.

Cipher Suites

As part of the SSL negotiation, the client and server agree on a ciphersuite that is common to them both, hopefully the strongest common suite. As attacks and computing power change, the “best” ciphers change. It is called a suites, as it defines ciphers users in four areas, authentication, key exchange, messaged integrity check (MAC), and the bulk (symmetric) cipher used, including mode, for block ciphers. There is a wide array of combinations of these parameters. As the suite is only as strong as its weakest link, most ciphersuites should be eliminated from your configuration. As of December 2011, you should eliminate any suite that sets any of the four areas to null. You should eliminate any suite that relies on MD5 for its MAC. You should eliminate any ciphersuite that uses a block cipher in CBC mode. One should also eliminate suites that utilize a symmetric cipher with a key shorter than 128 bit.

For more on CipherSuites, see http://tools.ietf.org/html/rfc4346

Windows

Microsoft allows you to pick the ciphersuites you will allow, as well as prioritize how they will be negotiated. A good place to start is here http://msdn.microsoft.com/en-us/library/windows/desktop/aa374757(v=vs.85).aspx .

We aware that XP/2003 and Win7/2008 have different settings and available suites.

Java

The Java Class SSLSocket allows the developer to adjust the suites used. http://docs.oracle.com/javase/1.4.2/docs/api/javax/net/ssl/SSLSocket.html

F5

The LTM/BigIP settings for ciphersuite can be found here http://devcentral.f5.com/Tutorials/TechTips/tabid/63/articleType/ArticleView/articleId/1086418/SSL-Profiles-Part-4-Cipher-Suites.aspx

Apache

Apache support for changing your ciphersuite can be found here http://httpd.apache.org/docs/2.0/ssl/ssl_howto.html

Mixed Content

I won't go into how to fix it, or configure your systems, cause you don't pay me... but, creating a page that uses SSL, but that has elements in the page that are not over SSL is called creating mixed content. This is common for those who use CDNs, as their images are all served up via HTTP.

Mixed content gives the user fun symbols that indicate badness, but how bad?? We can't expect users to make the call on this kind of thing.

Figure 12 Google Reader Throwing Mixed Content Error in Chrome

What's a user supposed to think when they encounter Figure 12? Can your mom or grandfather get meaning out of this explanation? https://support.google.com/chrome/bin/answer.py?hl=en&answer=95617

Summary

If you want to make the customer experience good, and avoid litigation, make sure to configure SSL on your web servers correctly. Any failure in your configuration may put the client at risk, but it may be a risk that is assumed by you, due to contracts.

Make sure you:

Set the right subject common name
Send the chain
Have time valid certificates
Include the right usage information
Force SSL
Have verified that the CRLs can be reached
CHECK YOUR WORK

Enrique Salem, president and CEO, Symantec, I WANT MY $2

2012-03-02T23:30:00.000-08:00

As my readers know, in May of 2011, I coined and began protecting the terms "Advanced Persistent Protection", "Advance Persistent Controls", and "Advanced Persistent Control Suite".

This week, my brilliant reader Sterling pointed out that I now have one peer who agrees with me, Enrique Salem, president and CEO of Symantec. In a post for SC Magazine, he makes four bullet points about Advanced Persistent Protection. I'm thinking three is the limit for reasonable use. If Symantec is going to start pimping my copyrighted phrase, they need to pay!

I am currently in talks with Michael “Let’s Get Ready to Rumble” Buffer's legal team to get my cash out of these guys.

Symantec, I have no desire to litigate as long as you give me what I am due. I'm glad to take a per use fee or we can negotiate an exclusive long term deal if you'd prefer. You'd better hurry though, RSA has already started talks. I'd hate for this to sink your upcoming marketing campaign.

Cheers,

Mark

Advanced Persistent Threats - Thank You Wikipedia

2011-12-15T23:12:00.000-08:00

My head is near exploding, as I'm sure yours is, from all the APT news. It's everywhere, I swear I saw it on the cover of the Weekly World News.

What annoys me is that the first several times I heard about an APT, "they" were basically describing any other virus or malware. The only difference was that the writers did a better job hiding their command and control, and they used more, and varying, ways to hide from AV and stay resident. This is a lame term as there is no line where quality achieves the level of Advanced!!

After hearing about the nature of the RSA breach, I have decided to only give credence to those who refer to an APT as an actor. APTs are not code. An APT is someone, some organization, or a nation state who is well funded, highly sophisticated, and persistent in their goal to compromise something.

I went to Wikipedia to see what the masses were saying, assuming the worst. Wikipedia agrees with me? I may have to turn in my security spurs. :-P

Proper use, "I am an advanced persistent threat".

Improper user, "I created and advanced persistent threat".

You can kill or jail an advanced persistent threat, but you can't delete it.

LDAP Tool of the Day - getrootDSE

2011-11-22T19:33:00.000-08:00

I'm an LDAP guy. I'm not even sure what that means, but I am one. I spend a lot of my work time looking at LDAPs. For the purists, I look at directories. LDAP is just an interface to the directory. If I look at the protocol with Wireshark, does that mean I am looking at LDAPs, 'cuase I do that too. Can you really look at a directory? I've never been to our data centers. Where was I?

There are a lot of great tools for working with LDAP, but there is always room for one more, right? A common task for me is to need to look at the contents of the Root DSE and verify the SSL certificate being used, if SSL is used.

For those not familiar with the root DSE, it is an entry offered by all LDAP servers. Its DN is null or empty, depending on how you interpret the RFCs. It almost always accepts un-authenticated connections and lists information about the contents and capabilities of the LDAP server. It will usually list the supported LDAP controls, authentication types offered, and often the naming contexts is holds. Different vendors list different data, and it is this data that I am often interested in.

Here are a few typical entries:
Active Directory Domain Controller

dn:
currentTime: 20111123000138.0Z
subschemaSubentry: CN=Aggregate,CN=Schema,CN=Configuration,DC=example,DC=org
dsServiceName: CN=NTDS Settings,CN=mydomcontr08,CN=Servers,CN=Food,CN=Sites,CN=Co
nfiguration,DC=example,DC=org
namingContexts: DC=example,DC=org
namingContexts: CN=Configuration,DC=example,DC=org
namingContexts: CN=Schema,CN=Configuration,DC=example,DC=org
defaultNamingContext: DC=example,DC=org
schemaNamingContext: CN=Schema,CN=Configuration,DC=example,DC=org
configurationNamingContext: CN=Configuration,DC=example,DC=org
rootDomainNamingContext: DC=example,DC=org
supportedControl: 1.2.840.113556.1.4.319
supportedControl: 1.2.840.113556.1.4.801
supportedControl: 1.2.840.113556.1.4.473
supportedControl: 1.2.840.113556.1.4.528
supportedControl: 1.2.840.113556.1.4.417
supportedControl: 1.2.840.113556.1.4.619
supportedControl: 1.2.840.113556.1.4.841
supportedControl: 1.2.840.113556.1.4.529
supportedControl: 1.2.840.113556.1.4.805
supportedControl: 1.2.840.113556.1.4.521
supportedControl: 1.2.840.113556.1.4.970
supportedControl: 1.2.840.113556.1.4.1338
supportedControl: 1.2.840.113556.1.4.474
supportedControl: 1.2.840.113556.1.4.1339
supportedControl: 1.2.840.113556.1.4.1340
supportedControl: 1.2.840.113556.1.4.1413
supportedControl: 2.16.840.1.113730.3.4.9
supportedControl: 2.16.840.1.113730.3.4.10
supportedControl: 1.2.840.113556.1.4.1504
supportedControl: 1.2.840.113556.1.4.1852
supportedControl: 1.2.840.113556.1.4.802
supportedControl: 1.2.840.113556.1.4.1907
supportedControl: 1.2.840.113556.1.4.1948
supportedControl: 1.2.840.113556.1.4.1974
supportedControl: 1.2.840.113556.1.4.1341
supportedControl: 1.2.840.113556.1.4.2026
supportedLDAPVersion: 3
supportedLDAPVersion: 2
supportedLDAPPolicies: MaxPoolThreads
supportedLDAPPolicies: MaxDatagramRecv
supportedLDAPPolicies: MaxReceiveBuffer
supportedLDAPPolicies: InitRecvTimeout
supportedLDAPPolicies: MaxConnections
supportedLDAPPolicies: MaxConnIdleTime
supportedLDAPPolicies: MaxPageSize
supportedLDAPPolicies: MaxQueryDuration
supportedLDAPPolicies: MaxTempTableSize
supportedLDAPPolicies: MaxResultSetSize
supportedLDAPPolicies: MaxNotificationPerConn
supportedLDAPPolicies: MaxValRange
highestCommittedUSN: 124867805
supportedSASLMechanisms: GSSAPI
supportedSASLMechanisms: GSS-SPNEGO
supportedSASLMechanisms: EXTERNAL
supportedSASLMechanisms: DIGEST-MD5
dnsHostName: mydomcontr08.example.ORG
ldapServiceName: example.ORG:mydomcontr08$@example.ORG
serverName: CN=mydomcontr08,CN=Servers,CN=Food,CN=Sites,CN=Configuration,DC=examp
le,DC=org
supportedCapabilities: 1.2.840.113556.1.4.800
supportedCapabilities: 1.2.840.113556.1.4.1670
supportedCapabilities: 1.2.840.113556.1.4.1791
supportedCapabilities: 1.2.840.113556.1.4.1935
isSynchronized: TRUE
isGlobalCatalogReady: TRUE
domainFunctionality: 2
forestFunctionality: 2
domainControllerFunctionality: 3

Oracle Virtual Directory
dn:
namingContexts: ou=Groups,dc=example,dc=com
namingContexts: ou=admins,dc=example,dc=com
namingContexts: ou=employees,dc=example,dc=com
namingContexts: ou=IDMUsers,dc=idm.example,dc=com
namingContexts: ou=partners,dc=example,dc=com
namingContexts: OU=portal users,dc=example,dc=com
namingContexts: dc=example,dc=com
namingContexts: ou=OIDUsers,dc=idm.example,dc=com
objectClass: top
subschemaSubEntry: cn=schema
supportedSASLMechanisms: EXTERNAL
supportedSASLMechanisms: CRAM-MD5
supportedLDAPVersion: 2
supportedLDAPVersion: 3
supportedExtension: 1.3.6.1.4.1.1466.20037

Oracle Internet Directory
dn:
supportedsaslmechanisms: DIGEST-MD5
supportedldapversion: 2
supportedldapversion: 3
supportedextension: 2.16.840.1.113894.1.9.1
supportedextension: 1.3.6.1.4.1.1466.20037
supportedcontrol: 2.16.840.1.113730.3.4.2
supportedcontrol: 2.16.840.1.113894.1.8.1
supportedcontrol: 2.16.840.1.113894.1.8.2
supportedcontrol: 2.16.840.1.113894.1.8.3
supportedcontrol: 2.16.840.1.113894.1.8.4
supportedcontrol: 2.16.840.1.113894.1.8.5
supportedcontrol: 2.16.840.1.113894.1.8.6
supportedcontrol: 2.16.840.1.113894.1.8.7
supportedcontrol: 1.2.840.113556.1.4.473
supportedcontrol: 1.2.840.113556.1.4.319
supportedcontrol: 2.16.840.1.113894.1.8.14
supportedcontrol: 2.16.840.1.113894.1.8.16
supportedcontrol: 2.16.840.1.113894.1.8.23
supportedcontrol: 2.16.840.1.113894.1.8.29
subschemasubentry: cn=subschemasubentry
subregistrysubentry: cn=subregistrysubentry
subconfigsubentry: cn=subconfigsubentry
pwdpolicysubentry: cn=default,cn=pwdPolicies,cn=Common,cn=Products,cn=OracleCont
ext
orclupgradeinprogress: FALSE
orcltimelimit: 3600
orclstatsperiodicity: 60
orclstatslevel: 0
orclstatsflag: 0
orclsizelimit: 100000
orclsimplemodchglogattributes: uniquemember
orclsimplemodchglogattributes: member
orclsimplemodchglogattributes: orcluserapplnprovstatus
orclsimplemodchglogattributes: orcluserapplnprovstatusdesc
orclsimplemodchglogattributes: orcluserprovfailurecount
orclservermode: rw
orclreplicaid: prdoidx401_poid1
orclreplagreements: cn=replication configuration
orcloptcontainsquery: 0
orclnormdn:: IA==
orclmaxtcpidleconntime: 120
orclmatchdnenabled: 0
orcllegacyoidsyncagent: cn=odisrv+orclhostname=prdoidx001,cn=Registered Instance
s,cn=Directory Integration Platform,cn=Products,cn=OracleContext
orcllegacyoidsyncagent: cn=odisrv+orclhostname=prdoidx401,cn=Registered Instance
s,cn=Directory Integration Platform,cn=Products,cn=OracleContext
orcllegacyoidsyncagent: cn=odisrv+orclhostname=prdoidx002,cn=Registered Instance
s,cn=Directory Integration Platform,cn=Products,cn=OracleContext
orcllegacyoidsyncagent: cn=odisrv+orclhostname=prdoidx402,cn=Registered Instance
s,cn=Directory Integration Platform,cn=Products,cn=OracleContext
orcleventlevel: 0
orclentrylevelaci: access to entry by * (browse, noadd, nodelete)
orclentrylevelaci: access to attr=(orclaci,orclguname,orclgupassword,orclprname,
orclprpassword,orclcryptoscheme,orclsuname,orclsupassword) by * (none)
orclentrylevelaci: access to attr=(*) by * (search, read, nowrite, nocompare)
orclentrylevelaci: access to attr=(*) AppendToAll by group="cn=directoryadmingro
up,cn=oracle internet directory" (search,read,write,compare)
orclentrylevelaci: access to entry AppendToAll by group="cn=directoryadmingroup,
cn=oracle internet directory" (browse,add,delete)
orclentrylevelaci: access to attr=(orclstatsflag, orclstatsperiodicity,orclevent
level) by dn="cn=emd admin,cn=oracle internet directory" (search,read,write,com
pare) by * (search,read)
orclenablegroupcache: 1
orclecachemaxsize: 10000000
orclecachemaxentries: 25000
orclecacheenabled: 1
orcldirectoryversion: OID 10.1.4.3.0
orcldiprepository: FALSE
orcldebugop: 511
orcldebugflag: 0
orclcatalogentrydn: cn=catalogs
orclauditlevel: 0
orclanonymousbindsflag: 1
matchingrules: distinguishedNameMatch
matchingrules: caseIgnoreMatch
matchingrules: caseExactMatch
matchingrules: numericStringMatch
matchingrules: telephoneNumberMatch
changestatus: cn=changestatus
changelog: cn=changelog
authpassword;oid: {SASL/MD5}sHex432oGONWYembe52eKA==
authpassword;oid: {SASL/MD5-DN}UpdstrkdNdL5mxyQ8wFP5iQ==
authpassword;oid: {SASL/MD5-U}m0/awjpasdf346gaKaIHs9UQ==

One can get this all via the command line, with ldapsearch. For windows, I use the OpenDS version.

>ldapsearch -h host.name.org -p 389 -w "" -b "" -s base objectclass=*

I often forget the command, and if you need SSL, then you need to add -Z -X. Really, the -X is something that I'd complain about in most contexts, as it accepts any SSL certs. In this case, I am meaning to investigate the cert as well.

This gets me the LDAP info, but, then I'd need to use openssl to get the SSL and cert info.

Using
>openssl s_client -connect ovd.internal.example.com:636
I get the connect info:

Loading 'screen' into random state - done
CONNECTED(0000017C)
depth=3 CN = Example USA Root CA
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
0 s:/C=US/ST=Washington/L=Bothell/O=Example USA, Inc./OU=Internal Systems/CN=ovd.internal.Example.com
i:/DC=org/DC=Example/CN=Example USA Issuer CA 02
1 s:/DC=org/DC=Example/CN=Example USA Issuer CA 02
i:/CN=Example USA Intermediate CA 01
2 s:/CN=Example USA Intermediate CA 01
i:/CN=Example USA Root CA
3 s:/CN=Example USA Root CA
i:/CN=Example USA Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGFTCCBP2gAwIBAgIKN3rWsgABAAWq7TANBgkqhkiG9w0BAQUFADBSMRMwEQYK
CZImiZPyLGQBGRYDb3JnMRcwFQYKCZImiZPyLGQBGRYHZ3NtMTkwMDEiMCAGA1UE
AxMZVC1Nb2JpbGUgVVNBIElzc3VlciBDQSAwMjAeFw0xMTAxMzAwMTU2MjlaFw0x
MjAxMzAwMTU2MjlaMIGQMQswCQYDVQQGEwJVUzETMBEGA1UECBMKV2FzaGluZ3Rv
.
.
hpw6x12bNa5bqIzyCm70ENEWSZkVAIiYPgNlJEs4AjmzgHo9ixPqsRmzSbmryaAg
8Pw6PUYagF+4soVfKRTZm62m+6qHlmnsDeGjKh6YR7QSwySTH+LV0VXuPBecM7T9
tf90c4mUD/jKnk9o0up0yTxXDf/WKNQ9SHKkzxvqJ8+7DcVw5kjBe+x8H1/HK420
skqyrngmSrL/3xjN/9JdXJp3WRVa+2dYAg==
-----END CERTIFICATE-----
subject=/C=US/ST=Washington/L=Bothell/O=Example USA, Inc./OU=Internal Systems/CN=ovd.internal.Example.com
issuer=/DC=org/DC=Example/CN=Example USA Issuer CA 02
---
No client certificate CA names sent
---
SSL handshake has read 6038 bytes and written 368 bytes
---
New, TLSv1/SSLv3, Cipher is EDH-RSA-DES-CBC3-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : EDH-RSA-DES-CBC3-SHA
Session-ID: 4ECC3B5303F3EAB7AFDD9452D7671A08CA4E345DF07F7FF3A76A3B9C62B2DA10
Session-ID-ctx:
Master-Key: B074BCE20BBF51B4EF420994309A4CC3DD85DB48F9CB6C5305F984A936FD6B659588C942B63FBC0228EF570D7E05777F
Key-Arg : None
PSK identity: None
PSK identity hint: None
Start Time: 1322007377
Timeout : 300 (sec)
Verify return code: 19 (self signed certificate in certificate chain)
---

Last, I have to cut and paste the certificate into a file and use openssl to read it.
>openssl asn1parse -in "cert.pem"

0:d=0 hl=4 l=1557 cons: SEQUENCE
4:d=1 hl=4 l=1277 cons: SEQUENCE
8:d=2 hl=2 l= 3 cons: cont [ 0 ]
10:d=3 hl=2 l= 1 prim: INTEGER :02
13:d=2 hl=2 l= 10 prim: INTEGER :377AD6B200010005AAED
25:d=2 hl=2 l= 13 cons: SEQUENCE
27:d=3 hl=2 l= 9 prim: OBJECT :sha1WithRSAEncryption
38:d=3 hl=2 l= 0 prim: NULL
40:d=2 hl=2 l= 82 cons: SEQUENCE
42:d=3 hl=2 l= 19 cons: SET
44:d=4 hl=2 l= 17 cons: SEQUENCE
46:d=5 hl=2 l= 10 prim: OBJECT :domainComponent
58:d=5 hl=2 l= 3 prim: IA5STRING :org
63:d=3 hl=2 l= 23 cons: SET
65:d=4 hl=2 l= 21 cons: SEQUENCE
67:d=5 hl=2 l= 10 prim: OBJECT :domainComponent
79:d=5 hl=2 l= 7 prim: IA5STRING :gsm1900
88:d=3 hl=2 l= 34 cons: SET
90:d=4 hl=2 l= 32 cons: SEQUENCE
92:d=5 hl=2 l= 3 prim: OBJECT :commonName
97:d=5 hl=2 l= 25 prim: PRINTABLESTRING :Example USA Issuer CA 02
124:d=2 hl=2 l= 30 cons: SEQUENCE
126:d=3 hl=2 l= 13 prim: UTCTIME :110130015629Z
141:d=3 hl=2 l= 13 prim: UTCTIME :120130015629Z
156:d=2 hl=3 l= 144 cons: SEQUENCE
159:d=3 hl=2 l= 11 cons: SET
161:d=4 hl=2 l= 9 cons: SEQUENCE
163:d=5 hl=2 l= 3 prim: OBJECT :countryName
168:d=5 hl=2 l= 2 prim: PRINTABLESTRING :US
172:d=3 hl=2 l= 19 cons: SET
174:d=4 hl=2 l= 17 cons: SEQUENCE
176:d=5 hl=2 l= 3 prim: OBJECT :stateOrProvinceName
181:d=5 hl=2 l= 10 prim: PRINTABLESTRING :Washington
193:d=3 hl=2 l= 16 cons: SET
195:d=4 hl=2 l= 14 cons: SEQUENCE
197:d=5 hl=2 l= 3 prim: OBJECT :localityName
202:d=5 hl=2 l= 7 prim: PRINTABLESTRING :Bothell
211:d=3 hl=2 l= 27 cons: SET
213:d=4 hl=2 l= 25 cons: SEQUENCE
215:d=5 hl=2 l= 3 prim: OBJECT :organizationName
220:d=5 hl=2 l= 18 prim: PRINTABLESTRING :Example USA, Inc.
240:d=3 hl=2 l= 25 cons: SET
242:d=4 hl=2 l= 23 cons: SEQUENCE
244:d=5 hl=2 l= 3 prim: OBJECT :organizationalUnitName
249:d=5 hl=2 l= 16 prim: PRINTABLESTRING :Internal Systems
267:d=3 hl=2 l= 34 cons: SET
269:d=4 hl=2 l= 32 cons: SEQUENCE
271:d=5 hl=2 l= 3 prim: OBJECT :commonName
276:d=5 hl=2 l= 25 prim: PRINTABLESTRING :ovd.internal.Example.com
303:d=2 hl=3 l= 159 cons: SEQUENCE
306:d=3 hl=2 l= 13 cons: SEQUENCE
308:d=4 hl=2 l= 9 prim: OBJECT :rsaEncryption
319:d=4 hl=2 l= 0 prim: NULL
321:d=3 hl=3 l= 141 prim: BIT STRING
465:d=2 hl=4 l= 816 cons: cont [ 3 ]
469:d=3 hl=4 l= 812 cons: SEQUENCE
473:d=4 hl=2 l= 29 cons: SEQUENCE
475:d=5 hl=2 l= 3 prim: OBJECT :X509v3 Subject Key Identifier
480:d=5 hl=2 l= 22 prim: OCTET STRING [HEX DUMP]:0414D5FBEBB564FC0855035A02C36F05D3BE6AB6D990
504:d=4 hl=2 l= 31 cons: SEQUENCE
506:d=5 hl=2 l= 3 prim: OBJECT :X509v3 Authority Key Identifier
511:d=5 hl=2 l= 24 prim: OCTET STRING [HEX DUMP]:30168014688A27CD6281B170FAC4A241E1F84927278B3A00
537:d=4 hl=4 l= 305 cons: SEQUENCE
541:d=5 hl=2 l= 3 prim: OBJECT :X509v3 CRL Distribution Points
546:d=5 hl=4 l= 296 prim: OCTET STRING [HEX DUMP]:
846:d=4 hl=4 l= 294 cons: SEQUENCE
850:d=5 hl=2 l= 8 prim: OBJECT :Authority Information Access
860:d=5 hl=4 l= 280 prim: OCTET STRING [HEX DUMP]:
1144:d=4 hl=2 l= 12 cons: SEQUENCE
1146:d=5 hl=2 l= 3 prim: OBJECT :X509v3 Basic Constraints
1151:d=5 hl=2 l= 1 prim: BOOLEAN :255
1154:d=5 hl=2 l= 2 prim: OCTET STRING [HEX DUMP]:3000
1158:d=4 hl=2 l= 11 cons: SEQUENCE
1160:d=5 hl=2 l= 3 prim: OBJECT :X509v3 Key Usage
1165:d=5 hl=2 l= 4 prim: OCTET STRING [HEX DUMP]:030205A0
1171:d=4 hl=2 l= 62 cons: SEQUENCE
1173:d=5 hl=2 l= 9 prim: OBJECT :1.3.6.1.4.1.311.21.7
1184:d=5 hl=2 l= 49 prim: OCTET STRING [HEX DUMP]:302F06272B0601040182371508AFAB1B85DD9D4F82E999398785C52C83F1EE
23778AC8DE812D86D8DA8E14020164020106
1235:d=4 hl=2 l= 19 cons: SEQUENCE
1237:d=5 hl=2 l= 3 prim: OBJECT :X509v3 Extended Key Usage
1242:d=5 hl=2 l= 12 prim: OCTET STRING [HEX DUMP]:300A06082B06010505070301
1256:d=4 hl=2 l= 27 cons: SEQUENCE
1258:d=5 hl=2 l= 9 prim: OBJECT :1.3.6.1.4.1.311.21.10
1269:d=5 hl=2 l= 14 prim: OCTET STRING [HEX DUMP]:300C300A06082B06010505070301
1285:d=1 hl=2 l= 13 cons: SEQUENCE
1287:d=2 hl=2 l= 9 prim: OBJECT :sha1WithRSAEncryption
1298:d=2 hl=2 l= 0 prim: NULL
1300:d=1 hl=4 l= 257 prim: BIT STRING

I can get 98% of what I need in one command, with my new tool

>getrootdse myORG.org 636 ssl

Performing a RootDSE search ...
supportedSASLMechanisms is GSSAPI
supportedSASLMechanisms is GSS-SPNEGO
supportedSASLMechanisms is EXTERNAL
supportedSASLMechanisms is DIGEST-MD5
defaultNamingContext is DC=myORG,DC=org
domainControllerFunctionality is 3
ldapServiceName is myORG.org:myDomcontr01$@myORG.ORG
supportedLDAPVersion is 3
supportedLDAPVersion is 2
dsServiceName is CN=NTDS Settings,CN=myDomcontr01,CN=Servers,CN=myORG-West,CN=Sites,CN=Configurati
on,DC=myORG,DC=org
subschemaSubentry is CN=Aggregate,CN=Schema,CN=Configuration,DC=myORG,DC=org
supportedLDAPPolicies is MaxPoolThreads
supportedLDAPPolicies is MaxDatagramRecv
supportedLDAPPolicies is MaxReceiveBuffer
supportedLDAPPolicies is InitRecvTimeout
supportedLDAPPolicies is MaxConnections
supportedLDAPPolicies is MaxConnIdleTime
supportedLDAPPolicies is MaxPageSize
supportedLDAPPolicies is MaxQueryDuration
supportedLDAPPolicies is MaxTempTableSize
supportedLDAPPolicies is MaxResultSetSize
supportedLDAPPolicies is MaxNotificationPerConn
supportedLDAPPolicies is MaxValRange
isSynchronized is TRUE
dnsHostName is myDomcontr01.myORG.org
supportedControl is 1.2.840.113556.1.4.319
supportedControl is 1.2.840.113556.1.4.801
supportedControl is 1.2.840.113556.1.4.473
supportedControl is 1.2.840.113556.1.4.528
supportedControl is 1.2.840.113556.1.4.417
supportedControl is 1.2.840.113556.1.4.619
supportedControl is 1.2.840.113556.1.4.841
supportedControl is 1.2.840.113556.1.4.529
supportedControl is 1.2.840.113556.1.4.805
supportedControl is 1.2.840.113556.1.4.521
supportedControl is 1.2.840.113556.1.4.970
supportedControl is 1.2.840.113556.1.4.1338
supportedControl is 1.2.840.113556.1.4.474
supportedControl is 1.2.840.113556.1.4.1339
supportedControl is 1.2.840.113556.1.4.1340
supportedControl is 1.2.840.113556.1.4.1413
supportedControl is 2.16.840.1.113730.3.4.9
supportedControl is 2.16.840.1.113730.3.4.10
supportedControl is 1.2.840.113556.1.4.1504
supportedControl is 1.2.840.113556.1.4.1852
supportedControl is 1.2.840.113556.1.4.802
supportedControl is 1.2.840.113556.1.4.1907
supportedControl is 1.2.840.113556.1.4.1948
supportedControl is 1.2.840.113556.1.4.1974
supportedControl is 1.2.840.113556.1.4.1341
supportedControl is 1.2.840.113556.1.4.2026
isGlobalCatalogReady is TRUE
forestFunctionality is 2
supportedCapabilities is 1.2.840.113556.1.4.800
supportedCapabilities is 1.2.840.113556.1.4.1670
supportedCapabilities is 1.2.840.113556.1.4.1791
supportedCapabilities is 1.2.840.113556.1.4.1935
highestCommittedUSN is 376377966
rootDomainNamingContext is DC=myORG,DC=org
schemaNamingContext is CN=Schema,CN=Configuration,DC=myORG,DC=org
namingContexts is DC=myORG,DC=org
namingContexts is CN=Configuration,DC=myORG,DC=org
namingContexts is CN=Schema,CN=Configuration,DC=myORG,DC=org
configurationNamingContext is CN=Configuration,DC=myORG,DC=org
serverName is CN=myDomcontr01,CN=Servers,CN=myORG-West,CN=Sites,CN=Configuration,DC=myORG,DC=org

currentTime is 20111123004547.0Z
domainFunctionality is 2

RootDSE search completed.

SSL for encryption is enabled
SSL information:
cipher strength: 128
exchange strength: 1024
protocol: Tls1Client
hash strength: 160
algorithm: Aes128
KeyExAlgo: 41984

The certificate did NOT validate correctly
The cert information is:
Subject: CN=myDomcontr01.myORG.org
Issuer: CN=myORG USA Issuer CA 06 v1, DC=myORG, DC=org
Expires: 8/21/2012 6:46:46 AM
Hash: 6D8F0501B7881A0DCCC84E1DCF4E1DF0646A4479
Public Key: 30818902818100C9D8ADE08D8CC893934C95AFF45DCFAB317B83CD0A93D659B181B8AB476D49954F94E2EE148C9A095C86592DCA458
B488DB3D5BDE5F14EAD3FBBB0D15A6DB1B48B587EB13984B15D27B2BEF4AF421BE8861B4A0C704A5510C5A2D431202675D65F9455573BDA2083D1DCD
6A2541FDA6CD6205FFACE670467366F9FC763B5C8B50203010001
Serial: 1BC1C68D000000005EC9

Here's the c# code
https://s3.amazonaws.com/markgamache/doWork.cs

the .NET 2.0 assembly.
https://s3.amazonaws.com/markgamache/GetRootDSE.exe

Enjoy.

Learn From My Mistakes - Toddlers

2011-11-21T20:25:00.000-08:00

I was recently reminded, twice, that kids don't see the world or understand language the way big people do. Here are two examples.

Be Careful
A few weeks ago, I was working out on our deck with a hammer. My 2 year old daughter came out and kept getting in the way. Being a wonderful and loving father, I warned my daughter. Using a tone that I knew was caring and not threatening, I said, "Sweetie, watch out or daddy might smash your toes with the hammer". My daughter immediately broke out into hysterical tears, screaming, "No daddy, don't smash my toes". I guess one should think before they speak.

Dirty Apple
The other day my daughter was eating an apple out in the yard of a house we were looking at. She was eating an apple, which she dropped into the dirt. As the apple was covered in dirt, I told my daughter it was dirty and we'd get another one. Then I threw it over the house into the green belt behind in the back. Once again, my daughter broke out into tears, yelling, "daddy, apple disappear into sky"!!

Don't be like me...

Why We Use Hardware Security Modules

2011-05-24T20:38:00.000-07:00

Hardware Security Modules (HSMs) are a security device that adds a lot of expense, man hours and complexity to a data processing system. As security and usability are always a trade-off, let’s look at when you want to make the trade. First off, what do HSMs to at a basic level? An HSM is a device used for key management and encryption and decryption of data. The HSM holds the key material on the device and there is no way to export the keys in a usable format. This keeps and attacker from copying your encrypted database and then taking the key and decrypting the data offsite, on his own time, where he is less likely to be caught. Used correctly, this is a big security gain.

There are a few guiding principles to keep in mind when looking at the threats to your data and the protections that the proper use on an HSM can bring.

Encryption is easy, key management is very hard.
A computer is only as secure as the administrator is trustworthy.
Encrypted data is only as secure as the decryption key.
If a bad guy has unrestricted physical access to your computer, it's not your computer anymore.
A data processing system, in terms of security, is like a chain. It is only as strong as the weakest link.
There are some risks that are too big to be accepted by any one person or department. Requiring multiple people for some operations raises the security bar.

When considering how to implement key management, the other option we look at is software based keys. Let’s take a quick look at the trade-offs of each. As you will see, HSMs are not a silver bullet.

	HSM	Software Keys
Someone with physical access to your server can take your keys*	Maybe	Yes
An attacker who can execute code on your servers can copy your keys	No	Yes
A rogue administrator can copy your keys	No	Yes
Use adds considerable extra expense	Yes	No
Use adds considerable complexity	Yes	No
Someone with root level access to your app server can see data before it is encrypted	Yes	Yes

*Properly configured, the keys are unusable even if the HSM is stolen.

While there are hundreds of specific threats and attack vectors against your data processing systems, it is important to align your controls to specific threats. In the case of using hardware encryption modules, there are four basics threats that might make us consider the use of an HSM over software based keys.

The rogue administrator.
Attackers with physical access to servers.
Attackers who have root access on servers.

While there may be some other benefits offered by an HSM, they are ancillary and redundant to other controls that should be in place.

We need to understand a couple of basic HSM concepts before we can see what they do for us.

The Security World – This is a logical concept that can span more than one HSM. It is a group of HSMs that all share a common master key. Members of a security world can share application keys from other members in the world. Keys can be copied to security world members without possibility of comprise.
Smart Cards – The security world is run by and protected with smart cards. These cards are actually small computers that that can create keys, store them and perform operations using them. The cards can be protected with a PIN for additional security. The cards are designed to make it very difficult to copy a card in a short time and without destroying it.
k of n card sets – Many operations on an HSM require a high level of assurance, so the HSM can be setup to require more than one smartcard be used to perform an operation. The card set is sized and distributed so that it is unlikely that cardholders will be able to collaborate on subversive actions. The number of cards required to perform actions is called a quorum. A large n ensures that if cards are lost or destroyed enough can still be found to maintain uptime. k of n is based on Shamir’s Secret Sharing algorithm.
Administrator Card Set - The ACS cards run the security world, in conjunction with the HSM. ACS cards are used to backup and restore the security world. This includes adding new HSMs to the security world. The key that decrypts the backup file for disaster recovery and adding devices to the security world is base on the ACS and protected with the secret sharing algorithm.
Operator Cards – The OCS cards are used to access application keys. These are the keys that actually encrypt and decrypt the data. The OCS k of n can be different from the ACS k of n.
HSM Soft keys – The HSM soft keys these are application keys that can be used with no action from an OCS card set. All you need to do is boot the server holding the HSM, or the netHSM. If an attacker takes the HSM, they have unrestricted use of the key. They can’t export the key, but they can use the key at will to decrypt any stolen encrypted data.

When properly configured, the HSM keeps and attacker with privileged access or a rogue administrator from taking a copy of your encrypted data and copy of your key and decrypting your data at their leisure. The best the attacker can do is to grab data before it is encrypted or send encrypted data to the HSM for decryption. When looked at in conjunction with proper audit logs and some sort of IDS or IPS solution, this should significantly limit the time that the attacker has access to gather plaintext data.

If an attacker manages to steal an HSM they cannot copy your security worlds as they lack the ACS quorum. They cannot use OCS protected keys without the OCS quorum. HOWEVER, anything protected by HSM soft keys can be decrypted by simply putting the stolen HSM online in a new system.

The bottom line is this:

If you use an HSM with HSM Soft keys, you probably are wasting your money as you are not reducing risk when attackers have physical access. You can get the same logical access value with keys stored in the file system of the appropriate application tier.
If the attacker has root access at the right tier in your application, at a minimum, they can copy off the data before it is encrypted and worse case, they can call APIs to decrypt data at will.

HSMs are awesome, but don’t assume just they will solve your problems. In my experience, the biggest holes are created by the applications and APIs that rely on the HSMs.

The Next Big Thing in Information Security

2011-05-05T23:25:00.000-07:00

Many of us in the industry are bothered by the use of security vendors selling based on Fear, Uncertainty, and Doubt (FUD). FUD gets in the way of rational thought and often has a negative impact on proper prioritization and budgeting. Specific vulnerabilities may change, but the basics have not; Threats can come from the host, application, or network.

The latest “new” scary thing to fear is the Advanced Persistent Threat. This is terrifying!!! What is this new threat? The threat is just that virus and malware authors are getting better at their jobs. Now that there is a market for cyber-crime, hostage-ware and the like, the bad guys are getting more advanced. The free market is at work and the money attracts talent. All the APT does is makes itself really hard to remove. The bad guys are just finding more ways and have better logic behind their execution. In windows for example, they are placing the their code in many location, in multiple forms, running in multiple processes that reinforce each other, and that startup in many locations and for many reasons. Their code starts early and when you kill one of their processes, another re-instantiates it. They bad guys are just getting more thorough.

Now that the security marketing guys have invented this new scary threat, there is only one solution, my solution. We need to map a very specific control to a specific threat or vulnerability. There is only one control that can meet this threat head on; the “Advanced Persistent Control” or the “Advanced Persistent Control Suite”. These are really enterprise solutions. To address the consumer space; we need a product with “Advanced Persistent Protection”. No other compensating control maps so perfectly to the threat.

I have Googled, Binged, Yahooed, Patent and Trademark searched the heck out of these things* and get no hits, so I am in the process of filing the appropriate trademark, service mark, provisional patents and copyright paperwork to protect these names and technologies. Boy do I love how the patent office allows such insanely broad patents. I own these names, but will be willing to license them to security vendors if their products and bids qualify.

That’s right, Symantec, McAfee, Kaspersky, or an up and comer, line up and start bidding. For the right price, I will sell my rights completely; otherwise I may just license limited use of the names. I must warn you, don’t think you can just take these names. They are not in common use and they are mine. Like Michael “Let’s Get Ready to Rumble” Buffer, I plan to carefully and jealously guard my property.

Consumers beware, if you do not have Advance Persistent Protection, you are asking to be a victim of cyber-crime.

Corporations, without an Advanced Persistent Control Suite, you are not taking due diligence to protect your customers’ data and intellectual property. I smell grounds for gross negligence. Don't be a victim, like RSA.

Bidders, you can contact me at mark.gamache@gmail.com

*Ok, there was on unrelated hit on one of the terms. One!

The IT Revolution is Coming Soon, I Hope

2011-03-07T22:33:00.001-08:00

I've been very concerned about the IT staffing for some time. Anyone who's been in the industry for more than 5 years should be too; if not, you may not be paying attention.

The problem is, this that there are WAY too many people in IT jobs, who aren't qualified to hold the job. These people range from bumbling, but harmless, mopes who just don't seem to get it, to those who seem to get everything wrong and cost their employers thousands or even millions of dollars.

This problem is hard to quantify and even harder to combat. Here's why...

There is strong demand for qualified IT staff and the jobs pay very well. There will always be people who are willing to lie or stretch the truth and hope they don't get caught, or they "catch on".
The Dunning-Kruger Effect. Dumb people don't know they are dumb, so their resume is likely to sound good, as it is a reflection of their perception of their skills.
IT recruiters usually just look for keywords on resumes and match a few phrases given to them by hiring managers.
Hiring Managers are managers... They often aren't qualified to evaluate competency, they just know if a candidate uses acronyms and terminology in the right context.
Except for a few IT Operations jobs, there is almost no accountability for poor performance and poor decision making. No one assigns blame to an architect or developer when something fails three or six months later.
It's not "nice" to call someone dumb and apparently "not qualified" is a bit too close to dumb for some people.
Poor IT training abounds. Universities are poorly equipped to keep up with the pace of new technologies, and IT training chains specialize in selling dreams to hopeless career changers. Yes, I generalize here and expect abuse... but I am right. ;-)
Those damn 10 Best Jobs/Careers articles. Magazines, and our parents have been telling us to get in to IT since the early 90's. The money entices people who don't have the natural curiosity and other skills, to excel in IT. Seriously, take the bad advice from your high school career councilor and find a way to do what you love, or at least something that keeps you out of my way.
The myth of the 10 (12, 15, whatever) year old IT whiz. Just because your kid, sisters kid, neighbor's kid, can install windows or use Facebook and Twitter does not mean he/she is a prodigy. If a kid likes to "play doctor", it doesn't mean he/she is ready for a career in medicine.
There is too much focus on how to operate applications/systems, rather than fundamentals of operations.
IT moves fast. Training budgets, when they exist, are rarely able to keep up with the next big thing.
Time. Who the hell has time to train????

The good news is there are signs that some employers are starting catch on. This article from InfoWorld indicates that IT shops are working harder to retain skilled staff and other IT shops are looking to pick off real talent.

In the mean time, here are a few thoughts on what the industry needs to do, to save itself from the under-qualified people it's already hired or about to hire.

Start making people accountable. Create or adopt existing PDCA (Demming) feedback systems. Don't just look to improve your systems when you get to the check phase, also track who did what and and make corrections to staffing as needed.
Create career paths for performers. Find ways to identify staff who have the capability to do better work or whose talent is being wasted.
Stop caring about degrees. At least half of the truly talented folks I know in IT don't have degrees, and those that do, don't have them in CS or anything related to IT. Four years in college is a long time to side track an IT career. :-P
Start firing the worst performers! This may feel harsh, but it helps the industry as a whole. These pole should NOT be in IT. If you manage them out, re-org them out, lay them off, etc, they can easily spin the departure to a new employer. If you've fired them, when the prospective employer calls and learns they are "not eligible for re-hire", they will understand the code.
Pay even more to top performers, but tie it to real performance. Don't just give it to the guy in the seat...
Make time and find budget to train those who can absorb and use the new skills.
Find ways to figure out who the skilled folks are treat them like kings!
Employers, remember, the only leverage you have here is benefits; be it cash, flex time, free cars... When it comes to supply and demand, there is a tiny supply of skilled IT folks and a serious demand. Don't make the mistake of thinking you can replace someone.

IT folks, well the smart ones... Here's what you should be doing to enhance your awesome:

Keep your resume, especially skills, up to date and online in Monster, or your favorite jobs site.
Update your LinkedIn, and connect with other smart people.
Do not endorse people on LinkedIn you would not hire. It may be "nice" to give someone a glowing recommendation, and they may give you one back, but if they aren't actually skilled, it may reflect poorly on you.
Even if you are happy in your, actively look for the "right" job and keep your mind open. Going on interviews is good practice, even if you aren't interested in the job.

The End.

Tuning F5 Big-IP Performance to Ruin Your Performance

2010-12-06T22:56:00.000-08:00

Or Configuring the F5 BigIP to shoot you in the foot.

Ever have bizarre connection problems when a client hits a Big-IP, but not when it hits the pool members directly? Is it happening to you right now and you don’t even know it? If your application is not stateless, this post may help you a lot. If nothing else, it covers some fundamentals that we've all let rust. Also, just because you aren't seeing the problem doesn't mean it is not there…

For years, we have seen a few LDAP connection problems that could not be explained, but the volume was so low that no one was willing to take the time to work with us to find out what was going on. With the addition of some new high volume systems, the problem has become more frequent and had greater impact, probably due to poor LDAP exception handling. Our service is an LDAP with approximately 20,000,000 million objects in it, each with around 28 attributes. We service 17 to 20 million LDAP requests a day. This an average of 231 LDAP request per second, if you do the math, but as most hours are slow, our peak is closer to 600 per second. This service is actually 3 LDAP servers fronted by a single Big-IP HA pair.

Before we dig in, the F5 Big-IP or Local Traffic Manager (LTM) is a modern marvel. The people who build these things and make the “magic” they do are amazing. They can solve some really complex availability and scaling challenges. Like most marvels, when applied wrong, they may not help, or even do harm. Try taking 20,000 grams of Amoxicillin for a headache. :-P

The Big-IP was originally created to load balance traffic to web sites that take very high volumes of traffic. If you have gotten this far, you probably already know that the Big-IP is optimized to spread the traffic to a pool of servers so that they all get a portion of the traffic. The tricky part is that there are hundreds of levers, knobs and switches, most of which default to the most common use case, HTTP traffic.

While HTTP uses TCP as its transport, HTTP IS STATELESS. HTTP does not care if you have already opened a TCP socket or not. It doesn’t care if the socket the browser uses is dead. That is, your web browser is coded to deal with every possible TCP use case fast and efficiently. On the other hand, some Application Layer Protocols, such as LDAP, are stateful and use the connection oriented nature of TCP to maintain state. This is at the heart of the problems I saw last week (and ongoing for years…).

LDAP, in the most common use case, requires a client connection to bind to the server using a username and password. The entire session is authenticated in that user’s context and all reads and writes are filtered through an access control list in the directory. Just a reminded, an LDAP server is a TCP listener that translates the LDAP protocol to an x.500 directory. An LDAP server is not a directory in and of itself, it is just a port and a set of communications rules optimized for access to an x.500 directory.

Here is a typical LDAP session with 192.168.0.15 as the server:

1 14:48:40.484934 192.168.0.20 192.168.0.15 TCP 12403 > ldap [SYN] Seq=4167455526 Win=65535 Len=0 MSS=1260

2 14:48:40.523621 192.168.0.15 192.168.0.20 TCP ldap > 12403 [SYN, ACK] Seq=4126863585 Ack=4167455527 Win=3780 Len=0

3 14:48:40.523683 192.168.0.20 192.168.0.15 TCP 12403 > ldap [ACK] Seq=4167455527 Ack=4126863586 Win=65535 Len=0

4 14:48:40.603744 192.168.0.20 192.168.0.15 LDAP bindRequest(1) "mgamach@testldap.org" simple

5 14:48:40.642792 192.168.0.15 192.168.0.20 LDAP bindResponse(1) success

6 14:48:40.649008 192.168.0.20 192.168.0.15 LDAP searchRequest(2) "DC=testldap,DC=org" baseObject

7 14:48:40.675407 192.168.0.15 192.168.0.20 LDAP searchResEntry(2) "DC=testldap,DC=org" | searchResDone(2) success

8 14:49:12.076343 192.168.0.20 192.168.0.15 LDAP unbindRequest(8)

9 14:49:12.076718 192.168.0.20 192.168.0.15 TCP 12403 > ldap [FIN, ACK] Seq=4167456246 Ack=4127490647 Len=0

10 14:49:12.100687 192.168.0.15 192.168.0.20 TCP ldap > 12403 [FIN, ACK] Seq=4127490647 Ack=4167456246 Len=0

11 14:49:12.100768 192.168.0.20 192.168.0.15 TCP 12403 > ldap [ACK] Seq=4167456247 Ack=4127490648 Len=0

12 14:49:12.105203 192.168.0.15 192.168.0.20 TCP ldap > 12403 [FIN, ACK] Seq=4127490647 Ack=4167456247 Len=0

The things worth noting are that A) the bind can’t occur until after the TCP 3-way handshake, B) when the client sends the unbind the server responds with a FIN, starting the TCP session tear-down, and C) this is one TCP session for the whole LDAP session. An LDAP client can run multiple LDAP session at once, in which case, each would have its own ephemeral port. While some LDAP server and client code can perform multiple asynchronous calls inside one TCP session, it is not very common and often not done well, but he client code developers. By far, the most common method is to treat each LDAP session as a synchronous session, request, then reply then request, then reply and so on until the unbind.

If the TCP session fails for any reason, the LDAP session is gone, there is no way to revive it or reconnect to it. Hopefully the client app and underlying systems are written well enough to deal with a failure and retry in a well considered manner. This is often not the case; consider the Java bug that does not inform the LDAP stack of a TCP reset for 15 seconds. That is one very painful timeout if a customer is waiting.

Back to the problem that we were seeing… At an application level our internal customers were seeing various connection errors in their logs. Different OSs and JVM versions seemed to throw slightly different errors, adding a bit to the confusion. It took quite some time to get the data we needed to diagnose the issue. We needed a packet capture on both sides of the BigIP as well as time stamped failure logs and IPs from the apps. With such high volume traffic, trying to find an error without somewhere to start is like trying to find one special needle in a needle stack, while wearing someone else’s bifocals.

Having the data in hand and using several WireShark tricks that I will blog on later, I found the root cause. The Big-IP config we were given was not vetted for awesome. We assumed that the networking team that creates the VIP and configures the pool analyzed our app and set up a config to optimize our performance. In fact, the config seemed a bit random.

We had already visited the config once as the F5 was introducing a 13x delay in response times. This turned out to be a TCP Profile setting that was set to WAN optimized. The settings in question optimized for bandwidth savings, over speed. Ooops! The culprit that we also fixed, prior to this, was that the TPC Idle timeout on the F5 did not match that of our LDAP servers. The F5 was set to close idle threads after 5 minutes, while the LDAP servers allow 2 hours. Finally, there isn’t just one TCP profile; there is one server (S-NAT) side and one client (VIP) side. Ours had different idle timeouts, leaving the F5 stuck between two parties who expected the session to be held open for different amounts of time. With those finds and fixes in place, we still see about a .5% failure rates, which is totally unacceptable with such a high volume service. That is still an average or more that once per second.

Here’s the last bit of fundamentals needed to understand why we were still seeing failures. There was only one IP in our F5 S-NAT pool, this means that it could, theoretically, have 65536 simulations connections to our pool members. However, the default setting for the use of those ports does not force the use of them all. The F5 setting TM.PreserveClientPort, by default has the S-NAT IP use the same ephemeral port as the upstream side of the conversation; meaning the client to VIP. The laws of TCP/IP say that the S-NAT IP can only make one connection on one port. This means if two of our clients make calls using the same ephemeral port, at the same time, the one with the established connection stays live and the other gets a nice TCP reset in its Christmas stocking. This is a bit more confusing for the app because the F5 accepts the VIP side TCP handshake while it is still getting ready to try the S-NAT side connection. Only then does it realize that the port is taken and it can’t connect. Once that happens, the F5 sends and unexpected TCP reset to the client when it is expecting to be able to do an LDAP bind.

At this point, you are thinking, “but Mark, what are the odds of two of your clients using the same ephemeral port at the same time? They must be crazy low”. Well, there’s more. Different OSs limit the number of possible ephemeral ports that can be used. BSD for instance used to only use 1024 to 3976. In our case most of our clients are on Solaris 10, which allows anything above 32768. We have around 20 clients that pseudo-randomly pick a port between 32768 and 65536. This is actually a probability problem exactly like the Birthday Problem. We just have 20 people (servers) with 32768 days (ports) to choose from instead of 365. We are looking to know what the probability is of any 2 of the servers using the same port. I will ignore the complexity of adding time as a factor. That is, some clients may hold a port open for longer than others, depending on the operations being performed. If we just assume that all the machines connect once using a random port, then our odds come up as .6% which is right next to our observed impact. As we will see in a minute, this number may or may not make sense. It is really just to make the point that it is a lot more likely to have a collision, than might seem intuitive.

While there are still a couple more factors to look at, let’s look at a failure in action.

1 0.000000 192.168.113.59 192.168.0.10 TCP 61325 > ldap [SYN] Seq=2258030186 Win=32768 Len=0 MSS=1460 WS=0

2 0.000005 192.168.0.10 192.168.113.59 TCP ldap > 61325 [SYN, ACK] Seq=2422958290 Ack=2258030187 Win=4380 Len=0 MSS=1460 WS=0

3 0.042001 192.168.113.59 192.168.0.10 LDAP bindRequest(1) "uid=appBindAcct,ou=applications,c=us,dc=testLDAP,dc=com" simple

4 0.042010 10.99.248.20 10.99.26.25 TCP 61325 > ovsessionmgr [SYN] Seq=4287903691 Win=4380 Len=0 MSS=1460 WS=0

5 0.042013 192.168.0.10 192.168.113.59 TCP ldap > 61325 [ACK] Seq=2422958291 Ack=2258030284 Win=4477 Len=0

6 0.042734 10.99.26.25 10.99.248.20 TCP ovsessionmgr > 61325 [SYN, ACK] Seq=1298286822 Ack=4287903692 Win=49640 Len=0 MSS=1460 WS=0

7 0.042738 10.99.248.20 10.99.26.25 TCP 61325 > ovsessionmgr [ACK] Seq=4287903692 Ack=1298286823 Win=4380 Len=0

8 0.042741 10.99.248.20 10.99.26.25 LDAP bindRequest(1) "uid=appBindAcct,ou=applications,c=us,dc=testLDAP,dc=com" simple

9 0.043455 10.99.26.25 10.99.248.20 TCP ovsessionmgr > 61325 [ACK] Seq=1298286823 Ack=4287903789 Win=49543 Len=0

10 0.043699 10.99.26.25 10.99.248.20 LDAP bindResponse(1) success

11 0.043703 192.168.0.10 192.168.113.59 LDAP bindResponse(1) success

12 0.043706 10.99.248.20 10.99.26.25 TCP 61325 > ovsessionmgr [ACK] Seq=4287903789 Ack=1298286837 Win=4394 Len=0

13 0.088130 192.168.113.59 192.168.0.10 LDAP searchRequest(2) "mobile=2065551234,ou=subscribers,c=us,dc=testLDAP,dc=com" baseObject

14 0.088135 10.99.248.20 10.99.26.25 LDAP searchRequest(2) "mobile=2065551234,ou=subscribers,c=us,dc=testLDAP,dc=com" baseObject

15 0.088138 192.168.0.10 192.168.113.59 TCP ldap > 61325 [ACK] Seq=2422958305 Ack=2258030416 Win=4609 Len=0

16 0.088866 10.99.26.25 10.99.248.20 TCP ovsessionmgr > 61325 [ACK] Seq=1298286837 Ack=4287903921 Win=49640 Len=0

17 0.089591 10.99.26.25 10.99.248.20 LDAP searchResEntry(2) "mobile=2065551234,ou=subscribers,c=us,dc=testLDAP,dc=com"

18 0.089595 192.168.0.10 192.168.113.59 LDAP searchResEntry(2) "mobile=2065551234,ou=subscribers,c=us,dc=testLDAP,dc=com"

19 0.089598 10.99.248.20 10.99.26.25 TCP 61325 > ovsessionmgr [ACK] Seq=4287903921 Ack=1298286926 Win=4483 Len=0

20 0.089600 10.99.26.25 10.99.248.20 LDAP searchResDone(2) success

21 0.089602 192.168.0.10 192.168.113.59 LDAP searchResDone(2) success

22 0.089605 10.99.248.20 10.99.26.25 TCP 61325 > ovsessionmgr [ACK] Seq=4287903921 Ack=1298286940 Win=4497 Len=0

23 0.130614 192.168.113.59 192.168.0.10 TCP 61325 > ldap [ACK] Seq=2258030416 Ack=2422958408 Win=32768 Len=0

24 0.130861 192.168.113.59 192.168.0.10 LDAP unbindRequest(3)

25 0.130865 10.99.248.20 10.99.26.25 LDAP unbindRequest(3)

26 0.130867 192.168.0.10 192.168.113.59 TCP ldap > 61325 [ACK] Seq=2422958408 Ack=2258030452 Win=4645 Len=0

27 0.131347 192.168.113.59 192.168.0.10 TCP 61325 > ldap [FIN, ACK] Seq=2258030452 Ack=2422958408 Win=32768 Len=0

28 0.131351 192.168.0.10 192.168.113.59 TCP ldap > 61325 [ACK] Seq=2422958408 Ack=2258030453 Win=4645 Len=0

29 0.131354 10.99.248.20 10.99.26.25 TCP 61325 > ovsessionmgr [FIN, ACK] Seq=4287903957 Ack=1298286940 Win=4497 Len=0

30 0.131356 10.99.26.25 10.99.248.20 TCP ovsessionmgr > 61325 [FIN, ACK] Seq=1298286940 Ack=4287903957 Win=49640 Len=0

31 0.131359 10.99.248.20 10.99.26.25 TCP 61325 > ovsessionmgr [FIN, ACK] Seq=4287903957 Ack=1298286941 Win=4497 Len=0

32 0.131362 192.168.0.10 192.168.113.59 TCP ldap > 61325 [FIN, ACK] Seq=2422958408 Ack=2258030453 Win=4645 Len=0

33 0.132083 10.99.26.25 10.99.248.20 TCP ovsessionmgr > 61325 [ACK] Seq=1298286941 Ack=4287903958 Win=49640 Len=0

34 0.132088 10.99.26.25 10.99.248.20 TCP [TCP Dup ACK 33#1] ovsessionmgr > 61325 [ACK] Seq=1298286941 Ack=4287903958 Win=49640 Len=0

35 0.172359 192.168.113.59 192.168.0.10 TCP 61325 > ldap [ACK] Seq=2258030453 Ack=2422958409 Win=32768 Len=0

36 56.717368 192.168.113.58 192.168.0.10 TCP 61325 > ldap [SYN] Seq=3620104483 Win=32768 Len=0 MSS=1460 WS=0

37 56.717374 192.168.0.10 192.168.113.58 TCP ldap > 61325 [SYN, ACK] Seq=3332499193 Ack=3620104484 Win=4380 Len=0 MSS=1460 WS=0

38 56.758628 192.168.113.58 192.168.0.10 LDAP bindRequest(1) "uid=appBindAcct,ou=applications,c=us,dc=testLDAP,dc=com" simple

39 56.758638 10.99.248.20 10.99.26.22 TCP 61325 > ovsessionmgr [SYN] Seq=258910960 Win=4380 Len=0 MSS=1460 WS=0

40 56.758640 192.168.0.10 192.168.113.58 TCP ldap > 61325 [ACK] Seq=3332499194 Ack=3620104581 Win=4477 Len=0

41 56.759124 10.99.26.22 10.99.248.20 TCP ovsessionmgr > 61325 [SYN, ACK] Seq=3696686467 Ack=258910961 Win=49640 Len=0 MSS=1460 WS=0

42 56.759128 10.99.248.20 10.99.26.22 TCP 61325 > ovsessionmgr [ACK] Seq=258910961 Ack=3696686468 Win=4380 Len=0

43 56.759131 10.99.248.20 10.99.26.22 LDAP bindRequest(1) "uid=appBindAcct,ou=applications,c=us,dc=testLDAP,dc=com" simple

44 56.759605 10.99.26.22 10.99.248.20 TCP ovsessionmgr > 61325 [ACK] Seq=3696686468 Ack=258911058 Win=49543 Len=0

45 56.760583 10.99.26.22 10.99.248.20 LDAP bindResponse(1) success

46 56.760588 192.168.0.10 192.168.113.58 LDAP bindResponse(1) success

47 56.760591 10.99.248.20 10.99.26.22 TCP 61325 > ovsessionmgr [ACK] Seq=258911058 Ack=3696686482 Win=4394 Len=0

48 56.802823 192.168.113.58 192.168.0.10 LDAP modifyRequest(2) "mobile=4255552345,ou=subscribers,c=us,dc=testLDAP,dc=com"

49 56.802830 10.99.248.20 10.99.26.22 LDAP modifyRequest(2) "mobile=4255552345,ou=subscribers,c=us,dc=testLDAP,dc=com"

50 56.802833 192.168.0.10 192.168.113.58 TCP ldap > 61325 [ACK] Seq=3332499208 Ack=3620104727 Win=4623 Len=0

51 56.803309 10.99.26.22 10.99.248.20 TCP ovsessionmgr > 61325 [ACK] Seq=3696686482 Ack=258911204 Win=49640 Len=0

52 56.814303 10.99.26.22 10.99.248.20 LDAP modifyResponse(2) success

53 56.814309 192.168.0.10 192.168.113.58 LDAP modifyResponse(2) success

54 56.814312 10.99.248.20 10.99.26.22 TCP 61325 > ovsessionmgr [ACK] Seq=258911204 Ack=3696686496 Win=4408 Len=0

55 56.856552 192.168.113.58 192.168.0.10 LDAP unbindRequest(3)

56 56.856557 10.99.248.20 10.99.26.22 LDAP unbindRequest(3)

57 56.856560 192.168.0.10 192.168.113.58 TCP ldap > 61325 [ACK] Seq=3332499222 Ack=3620104763 Win=4659 Len=0

58 56.857061 10.99.26.22 10.99.248.20 TCP ovsessionmgr > 61325 [FIN, ACK] Seq=3696686496 Ack=258911240 Win=49640 Len=0

59 56.857066 10.99.248.20 10.99.26.22 TCP 61325 > ovsessionmgr [ACK] Seq=258911240 Ack=3696686497 Win=4408 Len=0

60 56.857068 192.168.0.10 192.168.113.58 TCP ldap > 61325 [FIN, ACK] Seq=3332499222 Ack=3620104763 Win=4659 Len=0

61 56.857269 192.168.113.58 192.168.0.10 TCP 61325 > ldap [FIN, ACK] Seq=3620104763 Ack=3332499222 Win=32768 Len=0

62 56.898034 192.168.113.58 192.168.0.10 TCP 61325 > ldap [ACK] Seq=3620104764 Ack=3332499223 Win=32768 Len=0

63 57.381196 192.168.113.58 192.168.0.10 TCP 61325 > ldap [FIN, ACK] Seq=3620104763 Ack=3332499223 Win=32768 Len=0

64 57.381200 192.168.0.10 192.168.113.58 TCP ldap > 61325 [ACK] Seq=3332499223 Ack=3620104764 Win=4659 Len=0

65 57.381204 10.99.248.20 10.99.26.22 TCP 61325 > ovsessionmgr [FIN, ACK] Seq=258911240 Ack=3696686497 Win=4408 Len=0

66 57.381931 10.99.26.22 10.99.248.20 TCP ovsessionmgr > 61325 [ACK] Seq=3696686497 Ack=258911241 Win=49640 Len=0

67 70.622841 192.168.113.38 192.168.0.10 TCP 61325 > ldap [SYN] Seq=2324271576 Win=32768 Len=0 MSS=1460 WS=0

68 70.622846 192.168.0.10 192.168.113.38 TCP ldap > 61325 [SYN, ACK] Seq=151461930 Ack=2324271577 Win=4380 Len=0 MSS=1460 WS=0

69 70.664112 192.168.113.38 192.168.0.10 LDAP bindRequest(1) "uid=appBindAcct,ou=applications,c=us,dc=testLDAP,dc=com" simple

70 70.664121 10.99.248.20 10.99.26.22 TCP [TCP Port numbers reused] 61325 > ovsessionmgr [SYN] Seq=273484448 Win=4380 Len=0 MSS=1460 WS=0

71 70.664124 192.168.0.10 192.168.113.38 TCP ldap > 61325 [ACK] Seq=151461931 Ack=2324271674 Win=4477 Len=0

72 71.664114 10.99.248.20 10.99.26.22 TCP 61325 > ovsessionmgr [SYN] Seq=273484448 Win=4380 Len=0 MSS=1460 WS=0

73 72.864329 10.99.248.20 10.99.26.22 TCP 61325 > ovsessionmgr [SYN] Seq=273484448 Win=4380 Len=0 MSS=1460 WS=0

74 74.063807 10.99.248.20 10.99.26.22 TCP 61325 > ovsessionmgr [SYN] Seq=273484448 Win=4380 Len=0 MSS=1460

75 75.264023 192.168.0.10 192.168.113.38 TCP ldap > 61325 [RST, ACK] Seq=151461931 Ack=2324271674 Win=4477 Len=0

76 95.794344 192.168.113.37 192.168.0.10 TCP 61325 > ldap [SYN] Seq=136650233 Win=32768 Len=0 MSS=1460 WS=0

77 95.794349 192.168.0.10 192.168.113.37 TCP ldap > 61325 [SYN, ACK] Seq=3257078415 Ack=136650234 Win=4380 Len=0 MSS=1460 WS=0

78 95.835606 192.168.113.37 192.168.0.10 LDAP bindRequest(1) "uid=appBindAcct,ou=applications,c=us,dc=testLDAP,dc=com" simple

79 95.835616 10.99.248.20 10.99.26.28 TCP 61325 > ovsessionmgr [SYN] Seq=3796674135 Win=4380 Len=0 MSS=1460 WS=0

80 95.835619 192.168.0.10 192.168.113.37 TCP ldap > 61325 [ACK] Seq=3257078416 Ack=136650331 Win=4477 Len=0

81 95.836114 10.99.26.28 10.99.248.20 TCP ovsessionmgr > 61325 [SYN, ACK] Seq=3190364148 Ack=3796674136 Win=49640 Len=0 MSS=1460 WS=0

82 95.836118 10.99.248.20 10.99.26.28 TCP 61325 > ovsessionmgr [ACK] Seq=3796674136 Ack=3190364149 Win=4380 Len=0

83 95.836121 10.99.248.20 10.99.26.28 LDAP bindRequest(1) "uid=appBindAcct,ou=applications,c=us,dc=testLDAP,dc=com" simple

84 95.836578 10.99.26.28 10.99.248.20 TCP ovsessionmgr > 61325 [ACK] Seq=3190364149 Ack=3796674233 Win=49543 Len=0

85 95.837317 10.99.26.28 10.99.248.20 LDAP bindResponse(1) success

86 95.837320 192.168.0.10 192.168.113.37 LDAP bindResponse(1) success

87 95.837323 10.99.248.20 10.99.26.28 TCP 61325 > ovsessionmgr [ACK] Seq=3796674233 Ack=3190364163 Win=4394 Len=0

88 95.879556 192.168.113.37 192.168.0.10 LDAP searchRequest(2) "mobile=3605551234,ou=subscribers,c=us,dc=testLDAP,dc=com" baseObject

89 95.879562 10.99.248.20 10.99.26.28 LDAP searchRequest(2) "mobile=3605551234,ou=subscribers,c=us,dc=testLDAP,dc=com" baseObject

90 95.879565 192.168.0.10 192.168.113.37 TCP ldap > 61325 [ACK] Seq=3257078430 Ack=136650861 Win=5007 Len=0

91 95.880280 10.99.26.28 10.99.248.20 TCP ovsessionmgr > 61325 [ACK] Seq=3190364163 Ack=3796674763 Win=49640 Len=0

92 95.881746 10.99.26.28 10.99.248.20 LDAP searchResEntry(2) "mobile=3605551234,ou=subscribers,c=us,dc=testLDAP,dc=com"

93 95.881751 192.168.0.10 192.168.113.37 LDAP searchResEntry(2) "mobile=3605551234,ou=subscribers,c=us,dc=testLDAP,dc=com"

94 95.881754 10.99.248.20 10.99.26.28 TCP 61325 > ovsessionmgr [ACK] Seq=3796674763 Ack=3190364902 Win=5133 Len=0

95 95.881756 10.99.26.28 10.99.248.20 LDAP searchResDone(2) success

96 95.881758 192.168.0.10 192.168.113.37 LDAP searchResDone(2) success

97 95.881761 10.99.248.20 10.99.26.28 TCP 61325 > ovsessionmgr [ACK] Seq=3796674763 Ack=3190364916 Win=5147 Len=0

98 95.922763 192.168.113.37 192.168.0.10 TCP 61325 > ldap [ACK] Seq=136650861 Ack=3257079183 Win=32768 Len=0

99 95.923035 192.168.113.37 192.168.0.10 LDAP unbindRequest(3)

100 95.923038 10.99.248.20 10.99.26.28 LDAP unbindRequest(3)

101 95.923040 192.168.0.10 192.168.113.37 TCP ldap > 61325 [ACK] Seq=3257079183 Ack=136650897 Win=5043 Len=0

102 95.923504 192.168.113.37 192.168.0.10 TCP 61325 > ldap [FIN, ACK] Seq=136650897 Ack=3257079183 Win=32768 Len=0

103 95.923507 192.168.0.10 192.168.113.37 TCP ldap > 61325 [ACK] Seq=3257079183 Ack=136650898 Win=5043 Len=0

104 95.923509 10.99.248.20 10.99.26.28 TCP 61325 > ovsessionmgr [FIN, ACK] Seq=3796674799 Ack=3190364916 Win=5147 Len=0

105 95.923511 10.99.26.28 10.99.248.20 TCP ovsessionmgr > 61325 [FIN, ACK] Seq=3190364916 Ack=3796674799 Win=49640 Len=0

106 95.923515 10.99.248.20 10.99.26.28 TCP 61325 > ovsessionmgr [FIN, ACK] Seq=3796674799 Ack=3190364917 Win=5147 Len=0

107 95.923517 192.168.0.10 192.168.113.37 TCP ldap > 61325 [FIN, ACK] Seq=3257079183 Ack=136650898 Win=5043 Len=0

108 95.924223 10.99.26.28 10.99.248.20 TCP ovsessionmgr > 61325 [ACK] Seq=3190364917 Ack=3796674800 Win=49640 Len=0

109 95.924228 10.99.26.28 10.99.248.20 TCP [TCP Dup ACK 108#1] ovsessionmgr > 61325 [ACK] Seq=3190364917 Ack=3796674800 Win=49640 Len=0

110 95.964510 192.168.113.37 192.168.0.10 TCP 61325 > ldap [ACK] Seq=136650898 Ack=3257079184 Win=32768 Len=0

First off, all of this traffic is from ephemeral port 61325, whether it is from the client or the S-NAT side of the F5. All the traffic to and from the ldap port is on the VIP side of the F5. All of the traffic to and from the ovsessionmgr port is on the S-NAT side of the F5, to the LDAP servers.

LDAP VIP: 192.168.0.10

S-NAT IP: 10.99.248.20

LDAP Servers: 10.99.26.22, 10.99.26.25, 10.99.26.28

The rest of the IPs are clients.

First off, this capture only lasts 95 seconds, yet we see 4 different clients use the same ephemeral port in that short time. This 13 KB capture is, of course, massively filtered. The actual 95 second window is about 250 MB.

During this time, there is only one TCP reset sent (Frame 75), so we will analyze what happens there first, then go back to why there aren’t more failures.

In frames 55 and 56, we see a client send an unbind to the F5 and the F5 sends the unbind to the LDAP server. This starts the end of the two TCP sessions.

58 to 66 are the proper tear downs of the TCP sessions on both sides of the F5. This means that Client, F5 VIP side, F5 S-NAT side and server all agree the TCP sessions are over.

67 is a different client staring a new TCP handshake with the VIP. It is using the same ephemeral port, 61325. As TCP Segments can arrive out of order, we can see that 68 and 71 complete the TCP handshake, then the client attempts to bind in 69. Keep in mind that the data will be delivered from the IP stack to the app in order.

70 is the S-NAT trying to establish a TCP session with a pool member, 10.99.26.22. There is no SYN/ACK segment, so the S-NAT IP tries again in 72, 73 and 74. After the four failures to establish the S-NAT side TCP session, the F5 gives up and has to do something with the established connection on the VIP side, so it sends the TCP reset in frame 75.

“But wait”, you cry, “In Frame 66 we saw the final ACK of the session close with 10.99.26.22”. WAIT indeed, TIME_WAIT to be more exact. RFC 793 says that the TCP session, while closed in a data transmission sense, is still left locked in a state called TIME_WAIT. This is designed to make sure that any segments that were out of order make it to their destination before the socket is completely closed and refuses the data. It also makes sure that there is enough time for the final ACK of the close process has arrived. After all you can’t ACK that ACK or it’s not the final ACK.

Here’s the rub, TIME_WAIT was defined as 2 times the Maximum Segment Lifetime (MSL) in RFC 793. This value was, as the FRC puts it, is “arbitrary”. To be more exact, it made sense based on the networking technologies when the RFC was written in 1981. This value, which is 4 minutes (2 X an MSL of 2 min), was appropriate for the slow networks of the day. Since then, some vendors have decided this is too long. There is good reason for this. If a server needs to make a large number of outbound connections, this 4 minute penalty before the port can be reused can lead to port exhaustion. This value is tunable on an OS level. The default for windows is still 4 minutes. Our LDAP servers have 60 second TIME_WAIT, as part of the Solaris settings. TIME_WAIT means different things to the client and server. The client cannot make any calls out on that ephemeral port, not to any server, until the wait is over. The server cannot accept traffic from the source host on that same source port until the wait is over. The server could accept new traffic from the same source host, if it were on a new ephemeral port.

Between frames 66 and 74, the final ACK and the final SYN attempt, we only see about 12 seconds. The LDAP server is clearly in its 60 second TIME_WAIT and properly refusing the connection. Now you’re confused and so was I. Well, there are a couple more F5 TCP Profile settings that come into play. First, the TIME_WAIT is settable in the TCP profile and defaults to 2 seconds!! The client and server don’t agree, by way of negotiation, on a TIME_WAIT, they both have their own. This mismatch makes collisions possible. This setting is great to have, but it must be set for your use case. The default is not right for our use case. Second, and even more interesting/scary, is the TIME_WAIT Recycle option, which is defaulted to on. This behavior is described as TIME-WAIT Truncation, in RFC 1644. This allows the S-NAT side to cut the TIME_WAIT short if it receives a packet wanting the port, just like we see in frame 72. In this case, the setting doesn’t matter, as the F5 2 second timer is long over. This setting makes a lot of sense, performance wise, IF the S-NAT does not try to hit a pool member that is still in TIME_WAIT. In our case, we see that all 3 pool members were hit in that 95 seconds window. As we only have one S-NAT IP to work with and the F5 can’t guess at the TIME_WAIT on the pool members, we are setup for failure.

The moral of the story is, all the F5's performance settings have tradeoffs. By keeping TIME_WAIT low, the F5 can save on memory, by dumping the session early and avoid port exhaustion, but unless you have the proper number of S-NAT IPs, you are shooting yourself in the foot. By setting the F5 TIME_WAIT to match the time wait of your severs, you avoid the risk of collisions, but you tie up the ports and RAM on the F5. In essence, Adding IPs to the S-NAT IP pool lets us use probability to avoid most collisions.

The bottom line is, make sure your F5 setting match your use case. You may also want to read this article from F5 on things that may cause the F5 to send a TCP reset.

Lessons learned:

Configuring a load balancer well requires great knowledge about not just networking, but the nature of the application being load balanced.

F5 has done such a great job optimizing the defaults for HTTP, that most customers may never even see problems, if they are there at all.

Fundamentals like TCP session teardown and TCP state are worth knowing more about. There is no better reference than TCP/IP Illustrated by Stevens.

IT is All Ones and Zeros, heavy on the Zeros...

2010-11-10T14:24:00.000-08:00

In a rather technical discussion, yesterday, I was accused of "splitting hairs". Call me crazy, but you can't split most IT hairs... There is no splitting a binary digit.

IT and IT types are all about accuracy and terminology. Misperception in a technical setting costs time and wastes money. When a tech is "splitting hairs", there is usually a good reason, or they are just being a dick Granted, I was doing both. ;-) ...or maybe its Asperger's...

Some Things That IT Hates About Executives

2010-05-25T19:44:00.000-07:00

I recently read Eight Things Executives Hate About IT, by Susan Cramm, and it rubbed me the wrong way. The article tries to use language that softens the hate, but it's pretty transparent.

Here are my rebuttals to a few points...

(IT) Consists of condescending techies who don’t listen.

We're not condescending, we're just smarter than the execs. There is a serious imbalance of brain power. How that makes an exec feel is there problem, granted, we might be able to rub it in less. Self esteem comes from the self, if they feel condescended to, that is a personal problem. I recommend therapy. The fat paycheck seems to compensate well for the feelings of inadequacy that the job brings up. Many of us don't "lack the expertise to advise senior executives" and many do. Who's being condescending now! It feels it's time for couples counseling.

Seriously though, the Dilbert Principle tells us, (yes, Scott Adams is s-m-r-t) that dumb people with the right political skills are regularly promoted in order to keep them out of the way of real work. I'm not saying all execs are dumb, just that there is a mixed bag. Execs are like a box of chocolates...

I'll quote the whole point here...

"(IT) Doesn't understand the true needs of the business

IT nags you for requirements and complains that you always change your mind about what you want from your systems. Why doesn’t IT understand that change should be expected in a dynamic business environment where nothing is static? "

Seriously, have you take a single programming class? One? A computer system cannot be dynamic. It can't change on its own. It cannot anticipate needs. The code (if you’re lucky) only does exactly what you tell it too. it takes inputs and outputs and does "stuff". Every different type of input or output must be fully understood, by way of a use case, and then code and error handling written for each one. If the use case wasn't thought of, how does one write code?

Proposes “deluxe” when “good enough” will do

My experience has been just the opposite.

IT projects never end

The writer complains about the projects being "perennially 90% done" . Maybe she should have read her own point above about the business being dynamic. If you keep adding 10% to the scope...

Is reactive rather than proactive

"When you need help, you feel like a technology pauper, going door-to-door begging for help from functional specialists who complain that you didn’t get them involved early enough."

Hmmmmm... If only I could see a solution here. Maybe get the specialists involved earlier? No, that isn't it. We'll go out and get crystal balls so we can know what you will need. Please hold your breath until we get back to you.

Doesn't support innovation

"When you try to brainstorm with IT about new technologies you could use to innovate – like 2.0 tools, for instance – they patronize you by dismissing your questions and noting that your people aren’t properly using the systems already in place."

Does she even know what 2.0 tools are? Web 2.0? Let me make her point about condescending IT for her? WHAT!! You want user generated content in your billing system or CRM? Maybe you want photos in your activation system? Maybe she means AJAX? Yeah, spicing up the UI a tad bit, now that is innovation.

You cannot just call every crazy, one eighth baked idea innovation. Wait, I've got it, let's keep all the zeros, but replace the ones with Folgers crystals.

Trust us, if you have even a spark of a good idea, we will help bring it to glorious fruition.

IT never has good news

"No matter how much you spend or how hard you work, the promise of technology seems perpetually beyond your reach. Even the “successful” launch of new systems is accompanied with the inevitable onslaught of bugs, crashes, and change requests"

This is what you get when your project is date driven. Your project options are: Right, Fast, and Cheap. You can pick any two.

I'd recommend cutting scope, or hiring more good developers. And while we are on that topic. Not all IT staff are equal or interchangeable. Nothing spells fail like using sub-par IT staff and PMs.

I do agree with the writer, the situation has gotten out of hand. IT and the execs do need to come to an agreement. The first step should be honest communications about capabilities and expectations. Mush of IT has messed up. We work too many hours, delivering the near impossible time and again. We've allowed the bar to be set to high. Execs, wanting to push the envelope, try to push even further. It's time to reset expectations and force execs to understand the trade-offs involved with the scope of projects.

Top Rejected Names for "I Can't Believe it's Not Butter"

2010-05-16T09:10:00.000-07:00

5. This is like butter, only made of chemicals that somehow make it more healthy?

4. This tub was mislabeled.

3. Based on hours of testing, I am sure this is Butter, yet the package claims otherwise.

2. Mmmmmm, butter! What? Lying whore!!!!!!!!!

1. I am unable to conceive of a universe, wherein this is not butter.

How I Learned to Stop Hating Java and Start Loving s_client

2010-02-23T16:00:00.000-08:00

Got SSL problems? Ask me how s_client can help!

Have you ever had one of those days when a customer tells you that they can connect to your server using SSL, but not to the F5 load balancer in front of it? Here's my tale.

I provide LDAP over SSL access, to a group of my domain controllers, by way of an F5 load balancer. None of the apps that use this are AD aware, so we just have one virtual IP (VIP) on an F5 on port 636. I don't want to add Subject Alternate Names (SANs) to my DC certificates, so we terminate SSL at the F5 and then create a new SSL session from F5 to the appropriate DCs.

As the PKI that issues our SSL/TLS certificates is internal, it is common to have to help our customers provision their local CA trust stores to trust our Root CA. Nearly all of the systems that use the F5 are Java based. Since setting up the F5, we've noticed that the systems required our customers to install all three tiers of our PKI in to their local trust store. This seemed odd, as my experience has always been that importing the root alone was good enough to allow proper chaining. We always chalked it up to weak chaining code in the Java stack. How foolish of me/us!

Last week we had a fresh windows machine try and use the F5 and they got the same result. That is, he failed to connect to the load balancer, getting a failed trust error. The customer had only installed our Root CA in his trusted roots store. Instead of just adding all the tiers of our PKI, the customer decided to point directly at one of our DCs. To my surprise, he connected with no chaining failures. Apparently the Root CA was good enough when connecting to the DC, but no for the F5.

At that point, I stopped blaming Java and demanded answers. Enter my new trusty friend openssl s_client. s_client establishes an SSL/TLS session with a server and then waits for you to send commands. Think of it as getting a socket, like telnet, only after establishing an SSL session first. But wait, there's more... s_client can also tell you all about the SSL session in question.

openssl s_client -connect server:port -showcerts will also show you what is being negotiated by your server and client.

When connecting to my DC directly I got:

CONNECTED(00000730)

---

Certificate chain

0 s:/CN=DCserver.domain.org

i:/DC=org/DC=domain/CN=Fun USA Issuer CA 02

-----BEGIN CERTIFICATE-----

MIIF5TCCBM2gAwIBAgIKS5ibh

-----END CERTIFICATE-----

1 s:/DC=org/DC=domain/CN=Fun USA Issuer CA 02

i:/CN=Fun USA Intermediate CA 01

-----BEGIN CERTIFICATE-----

MIIF/zCCBOegAwIBA

-----END CERTIFICATE-----

2 s:/CN=Fun USA Intermediate CA 01

i:/CN=Fun USA Root CA

-----BEGIN CERTIFICATE-----

MIIF5jCCBM6g

-----END CERTIFICATE-----

---

Server certificate

subject=/CN=DCserver.domain.org

issuer=/DC=org/DC=domain/CN=Fun USA Issuer CA 02

---

Acceptable client certificate CA names

/C=US/O=First Data Digital Certificates Inc./CN=First Data Digital Certificates Inc. Certification Authority

/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting/OU=Certification Services Division/CN=Thawte Personal Basic CA/emailAddress=personal-basic@thawte.com

/C=US/O=VeriSign, Inc./OU=Class 1 Public Primary Certification Authority

/C=US/O=GTE Corporation/OU=GTE CyberTrust Solutions, Inc./CN=GTE CyberTrust Root

/DC=com/DC=microsoft/CN=Microsoft Root Certificate Authority

---

SSL handshake has read 7832 bytes and written 318 bytes

---

New, TLSv1/SSLv3, Cipher is RC4-MD5

Server public key is 1024 bit

Compression: NONE

Expansion: NONE

SSL-Session:

Protocol : TLSv1

Cipher : RC4-MD5

Session-ID: 80050000AB7EA7AC223A31DAE93EFE553CFFEB093D224D59C6C3DD169AFEA1D3

Session-ID-ctx:

Master-Key: 382218AB63FFC1A874DAD739E48630D8410F05FFE24BAEF54DF7E88578DE2720AF111FF4E7C052A42A357AB476C205B5

Key-Arg : None

Start Time: 1266967263

Timeout : 300 (sec)

Verify return code: 20 (unable to get local issuer certificate)

---

You can see that the server and client negotiated a TLSv1 session using RC4-MD5. More importantly, for me, we see what certificates the server offered up for the SSL connection. I've truncated the output to remove most of the base64 encoded certificate data, as you don't need it. It can be useful if you know what to do with it, but that is out of scope. You can see that my DC will allow client certificates, but is not requiring them. You will see that the DCserver cert was offered up, along with the Issuer 02, the Intermediate, and the Root. This gives a client all the hierarchy to allow chaining to the Root, which was trusted.

When I ran the s_client connect against the F5, I got this:

CONNECTED(00000730)

---

Certificate chain

0 s:/C=US/ST=WA/L=Renton/O=Fun USA, Inc/OU=Business Systems/CN=f5.domain.org

i:/DC=org/DC=domain/CN=Fun USA Issuer CA 02

-----BEGIN CERTIFICATE-----

MIIGATCCBOmgAwIB

-----END CERTIFICATE-----

---

Server certificate

subject=/C=US/ST=WA/L=Renton/O=Fun USA, Inc/OU=Business Systems/CN=f5.domain.org

issuer=/DC=org/DC=domain/CN=Fun USA Issuer CA 02

---

No client certificate CA names sent

---

SSL handshake has read 1687 bytes and written 306 bytes

---

New, TLSv1/SSLv3, Cipher is RC4-MD5

Server public key is 1024 bit

Compression: NONE

Expansion: NONE

SSL-Session:

Protocol : TLSv1

Cipher : RC4-MD5

Session-ID: B53C2347A4947080D27A65929C3E2ADCFE6DCCFC70F57456BA8155E036347B4E

Session-ID-ctx:

Master-Key: 24E43E3324F07EDCA3C2F0DC790FFE0134B5B0FC027E08B46988A02A26A402780D548C1DAC8C8F3F1EFA0F071A098F8A

Key-Arg : None

Start Time: 1266967301

Timeout : 300 (sec)

Verify return code: 21 (unable to verify the first certificate)

---

As you can see, this is much shorter. The F5 was offering up only the server certificate and none of the chain. As the client only has the Root trusted, it cannot succeed in processing the chain. The only way a session like this can be trusted is if the client explicitly trusts the server certificate or the CA that issued the certificate.

I love s_client...

As it turns out, the F5 team forgot to load up the chain. Some systems won't even offer up SSL without a valid chain, but F5 is way better!

Why Software Certificates Aren't a Second Authentication Factor

2009-07-13T11:26:00.000-07:00

Two Factor authentication is better than one. It is a requirement for some systems that need to be more secure, such as PCI covered systems. What does this actually mean and why is it more secure?

Two-factor authentication means that you have authenticators from to different classes of authenticator:

What you know, like a password or pass phrase.
What you have, like a OTP Token or a smart card.
What you are, like a fingerprint, voice print, keystroke rhythm.

These general definitions are generally accepted by all. Let's go a step further and look at what they actually mean, and what the intent is. What are the kinds of attacks against a specific authenticator.

Why would you even want a second factor and why from a different class?
We don't want more security! If we use a send factor, an attacker has something else to get in order to impersonate the legitimate user. However, if the user can get the first factor, then you surely don't want to use one just like it. Different classes of authenticators have different threat models, attack vectors and vulnerabilities. If the attacker can steal one password, they can surely steal a second. They just do what they did, for a bit longer.

If an attacker has to somehow get my password and steal my physical token without me knowing, that's a tall order. You can get my password remotely, but you have top travel and sneak to get my token. On top of that, I carry a Spyderco Delica. That's how I spell compensating control.

The intent of the "what you have" factor, in my opinion, is that the authenticator be a physical object that can not be easily duplicated. This maintains the threat model and the Spyderco control. The "what you have" factor should be hard to copy even if the legitimate owner is colluding. "Here, have a copy of my certificate and private key", should not be an option. If the what you have can be transmitted electronically, it's not a true what you have.

Software certificates and the corresponding private keys, that are stored in some sort of non-volatile storage, i.e. your hard drive. I'll focus on windows boxes, as I can speak very accurately about them, but the following is true with any OS. If the users have physical access to the computers, then they can get admin rights, which means that they can get the private key for the certificate. Don't believe it... Remember, if you have physical access to the box, you own the box. Did you think that your whole disk encryption product could protect you? Take a look at cold boot attacks.

Once you have admin rights, there are two methods to get the private key. The private key is stored in the registry and file system. It's obfuscated a bit and protected with DPAPI, in the machine context. This means that an admin can export the private key. Wait, you've marked the private key as not exportable! Remember, the admin owns the box. The key is there, a piece of code is just telling the system to not hand it over. Let's look at C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys and HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\My\Certificates and HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\My\Keys
These are all linked together to protect you from getting the key. If we take a look in one of the key files, we see an Export Flag. In this case, local software is reading that flag and choosing whether or not to respect it. Once could just use a little magic and pull out the key. Enter the ninjas at ISEC Partners and their Jailbreak. Give it a shot, and you will see that you can export the key!

So, if the user colludes, copying the software certificate is trivial. If the user is just negligent, the attacker could ghost (clone) the drive and then get access later. If the user is negligent, but uses a good full drive encryption product, the attacker will need some time with the computer to get the key.

Protections of Certificates and Keys That Work
My examples above are related to machine certificates. Those are the only ones that can be used for IPSec, in most cases. If your user does not collude with the attacker and cant' be tricked into running their attack code, (that seems unlikely, users will click on anything) then user certificates are secure. Why? User certificates are protected by DPAPI in the user context. This means that they private key can only be decrypted if the user has typed in their password. The system does not escrow the key for the user, the user's private keys are protected with a key that is derived directly form the password using PKCS 7. If the actual password is not available, their is no way to recover the key. In a workgroup, this is absolute. If the machine is domain joined, an administrator of the domain can reset the user's password and reconnect the machine to the domain and the key can be recovered. Collusion by your domain admins is a whole lot more worrisome than your user.

Whole disk encryption that requires keys stored on a smart card or token can stop most attacks. The problem with most whole disk encryption implementations is that they are not actually secure. They just keep the dumb and honest honest. If you can boot the PC without entering a pass phrase or insert a token, then your disk encryption keys are stored locally and can be accessed by your boot loader. This means that it's just a matter of understanding how that works to get the key, or one can let the machine boot and then use the cold boot method to get the key from RAM. Use real encryption, get a token or smartcard!

One could also use SYSKEY in mode 2 or 3. This encrypts some of the SAM using a key that is derive from a password or on a floppy disk. Neither of these is particularly manageable on an enterprise scale. Most users will leave the floppy in the drive and the password has no rotation policy. Either you have to track it per PC or have a universal password for your org, which will surely get out.

So, if you have a software certificate and private key, they are electronic assets with many avenues that can allow an attacker to copy and electronically copy and transmit them. While the attack vectors for getting the private key differ from getting the user's password, the spirit of "something you have" is not met by a software certificate. This is not to discount the value of software certificates in many situations, they are great. I simply am making the point that they are what they are and one should never forget.

Sniffing the Unsniffable

2009-06-25T15:45:00.000-07:00

In a previous post I covered how to export a private key to decrypt SSL traffic in Wireshark. This is dandy, as long as you have access to the private key.

What do you do if you don't have access to the private key??? Perhaps you are using a windows box and the certificate policy is set to not allow the export of the private key. Yes, a super ninja can take it, if they know enough about the file locations and DPAPI, but it's hard. Maybe I'll cover that soon. What if you are the client and the server is some commercial service? You'll never get that key.

If you are on a windows host, you are in luck! Enter my new best friend, STrace, from Microsoft. STrace uses MS Research's amazing tool, detours. I'll avoid a huge digression and just say that detours is crazy awesome. It intercepts win32 functions in memory and allows the code to be altered on the fly. That's right "re-writing the in-memory code for target functions". Yikes!

STrace was written for something other than our purposes. It's meant to be used in conjunction with HTTPREPLAY. This allows of debugging a web browser session over and over, without the server. Booooorrrriiinnnggg. Actually it's very cool, but not for us.

One of the things that STrace does for us, to support its original goes, is to inject itself into processes that are using SSL. You simply find the PID of the process and inject the DLL and you get your data out, unencrypted. The data is in a format designed for HTTPREPLAY, which is only a bit of a pain.

Going for the Gold!

If you've ever dealt with LDAPS failures in AD, you know how tricky they can be. You can only watch the SSL negotiation, then you lose out. If the issue involves a lying crappy vendor of a developer who just read about LDAP when the project started, you always want to look at the data they are sending. LDAP on the wire never lies! Too bad that by default, the Domain Controller Certificate Template is set to not allow export of your private key. If you're like me, you may have the party turn off SSL and suddenly it all works. Hmmmm... We must see what is really being sent!

First off, the strace readme.txt is full of awesome and it's short. It's worth a once over.

The first step in the process is to figure out which process is our LDAP SSL server. We know that SSL for LDAP is on port 636, so we run netstat -b -n -a to get the list of ports and their process names and the PID

That gets us this:

TCP 0.0.0.0:636 0.0.0.0:0 LISTENING 428

[lsass.exe]

Now we know that LSASS is the process and the PID is 428. LSASS!! Oh no, that is super protected certainly we can't touch it. This may be the Holy Grail.

If you follow the readme file, your first attempt will fail. Luckily the error is mentioned right in the file.

Note that you may get some permission issue if you want to "inject" the STRACE DLL in "protected" processes (Injib may fail with error 5 (ACCESS_DENIED) on OpenProcess()). One way to workaround this problem is to run INJLIB from a CMD running under the system account.

That sound great, but how?

We will use some pstools to get the rest of what we need. If you don't know pstools and the Sysinternals Suite, you may not actually be a windows admin.

psexec -s will let us run a process as system!

let's try

psexec -s "C:\Program Files\STRACE\INJDLL.EXE" /p:428 /d:"C:\Program Files\STRACE\STRACE.DLL"

Now, if you run something like Sysinternals Process Explorer and do a find for "strace" you will see it is now attached to LSASS. No errors, no LSASS detects evil, shutting down...

At this point, you should see a strace log file on your desktop, but you don't. Sysinternals Filemon will show us why. The default location is the desktop, but where is system's desktop, it's not under documents and settings. Filemon finds it here.

C:\WINDOWS\system32\config\systemprofile\Desktop\STRACE_LSASS_PID_428_24062009_223405.LOG

3:35:26 PM lsass.exe:400 WRITE C:\WINDOWS\system32\config\systemprofile\Desktop\STRACE_LSASS_PID_428_24062009_223405.LOG SUCCESS Offset: 9630 Length: 26

Now, we can use some LDAP Tools to connect to the server over SSL and watch what happens.

When I bind to the DC, I get a whole lot, but I'll pull out the good stuff:

06/25/2009 23:55:41:078 - SecBuffer #1 BufferType:0x00000000 cbBuffer:47906816

06/25/2009 23:55:41:078 - SecBuffer #2 BufferType:0x00000000 cbBuffer:54197772

06/25/2009 23:55:41:078 - SecBuffer #3 BufferType:0x00000000 cbBuffer:-1

06/25/2009 23:55:41:078 - SecBuffer #0 BufferType:0x00000007 cbBuffer:5

06/25/2009 23:55:41:078 - SecBuffer #1 BufferType:0x00000001 cbBuffer:45

06/25/2009 23:55:41:078 - SECBUFFER_DATA - 45 byte(s) / DecryptMessage - OUTPUT

=====================================================

00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 0123456789abcdef

0000: 30 2b 02 01 01 60 26 02 01 03 04 17 61 64 6d 69 0+...`&.....admi

0010: 6e 69 73 74 72 61 74 6f 72 40 62 6c 69 6d 2e 62 nistrator@blim.b

0020: 6c 61 6d 80 08 70 61 73 73 77 6f 72 64 lam..password

=====================================================

06/25/2009 23:55:41:078 - SecBuffer #2 BufferType:0x00000006 cbBuffer:16

06/25/2009 23:55:41:078 - SecBuffer #3 BufferType:0x00000000 cbBuffer:-1

06/25/2009 23:55:41:093 - SecBuffer #0 BufferType:0x00000002 cbBuffer:5

06/25/2009 23:55:41:093 - SecBuffer #1 BufferType:0x00000001 cbBuffer:22

06/25/2009 23:55:41:093 - SECBUFFER_DATA - 22 byte(s) / EncryptMessage - INPUT

=====================================================

00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 0123456789abcdef

0000: 30 84 00 00 00 10 02 01 01 61 84 00 00 00 07 0a 0........a......

0010: 01 00 04 00 04 00 ......

=====================================================

We see here that it gets some buffers and decrypted messages. I see my username and password and then some sort of response.

To verify our suspicions, we make the same conneciton, this time without SSL and capture it via wireshark.

If we look for the same data, we get these two frames:

0000 00 0c 29 bb 00 2e 00 1a 4b 79 8d 34 08 00 45 00 ..).....Ky.4..E.

0010 00 55 c1 14 40 00 80 06 5c 60 0a 01 64 97 0a 01 .U..@...\`..d...

0020 64 95 0a 87 01 85 b6 56 6c 8e 73 0a 76 db 50 18 d......Vl.s.v.P.

0030 fc 00 d0 48 00 00 30 2b 02 01 01 60 26 02 01 03 ...H..0+...`&...

0040 04 17 61 64 6d 69 6e 69 73 74 72 61 74 6f 72 40 ..administrator@

0050 62 6c 69 6d 2e 62 6c 61 6d 80 08 70 61 73 73 77 blim.blam..passw

0060 6f 72 64 ord

0000 00 1a 4b 79 8d 34 00 0c 29 bb 00 2e 08 00 45 00 ..Ky.4..).....E.

0010 00 3e 26 b4 40 00 80 06 f6 d7 0a 01 64 95 0a 01 .>&.@.......d...

0020 64 97 01 85 0a 87 73 0a 76 db b6 56 6c bb 50 18 d.....s.v..Vl.P.

0030 fa c3 f6 bf 00 00 30 84 00 00 00 10 02 01 01 61 ......0........a

0040 84 00 00 00 07 0a 01 00 04 00 04 00 ............

If you notice, we have a byte for byte match for the bottom part of the packet. If you break out your TCPIP Illustraed, you will find that the top of each frame is the Ethernet, IP and TCP header data, we are getting just the LDAP protocol payload!

If we look closer, we see that all our LDAP traffic is there. The downside being, that we don't get information on the host sending the data and we don't have libcap format, so tools like wireshark can't dissect the protocols for us in a nice way. If you understand libcap, it looks trivial to create fake Ethernet, IP and TCP data and drop in the payload to make a libcap file, or use existing parsers to dissect the messages. Its beyond me though.

The last down side is that I can't find out how to un-inject the strace dll. So far, only rebooting has fixed this.

And that is that... If you can detour the SSL from LSASS, you can do it anywhere!

Analysis of Oracle Hyperion Web Analysis Studio 9.3 Logon Security

2009-05-27T13:30:00.000-07:00

Before I start I'll point out that Oracle does not make any claims that the Hyperion Logins are secure without SSL. This analysis was performed to clear up misconceptions on the part of others.

Analysis of Oracle Hyperion Web Analysis Studio 9.3 Logon Security

Abstract

A quick vetting of how Hyperion handles usernames and passwords shows that their controls offer little protection of credentials while crossing the network, in the absence of SSL. While passwords are not sent across the network as clear text, they are only lightly obfuscated and subject to multiple attacks.
Hyperion should not be used unless the login pages are encrypted using SSL derived from a certificate that chains to a trusted Root Certificate Authority. This is most important when using the LDAP connector as it then puts credentials at risk that might grant access to other systems.

Analysis
There are multiple ways to take a look at Hyperion yielding varying results. Hyperion sends data via Java code to the app server via an unknown application protocol. It uses Java serialized objects, which may be meaningful to some else. At the TCP level, the usernames can be seen on the wire in clear text, along with what appears to be a base64 encoded password.
A message snippet looks like this:
------------------314159265358979323846
Content-Disposition: form-data; name="param type 1"
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 8bit

com.hyperion.atf.security.authentication.IAuthenticationHelper

------------------314159265358979323846
Content-Disposition: form-data; name="param type 2"
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 8bit

com.hyperion.atf.security.users.IUserManagerCreateTask

------------------314159265358979323846
Content-Disposition: form-data; name="param value 0"
Content-Type: application/x-java-serialized-object
Content-Transfer-Encoding: binary

....sr.
csPasswordt..Ljava/lang/String;L.
csUserNameq.~..xpt..DmFVYnsCImM=t..markgamache

------------------314159265358979323846

The first thing we see is that the protocol breaks up messages with Pi. Who knows… The second thing we see is that the helpful words, Password and UserName can be found. These are followed by DmFVYnsCImM=, presumably my password and then markgamache, my username. By simply trying a few attempts with different usernames and passwords, it becomes clear that we are correct in our assumptions. Third, but for later, we also get a java class name, hyperion.atf.security.authentication.SimplePasswordToken.

Replay Attacks
Before we look at any technical attacks, we’ll look at that messaging format. The TCP data sent along with the username and password contain no sequence numbers, nonces, or timestamps. As you will see in the next section, a username and encrypted password for an account do not change across login sessions. This makes it highly likely that a simple replay would allow access to a session as another user. This does not put the password at risk, only data within the application. If the application data is not encrypted, it is likely that the session could be hijacked as well. This is a smaller concern generally.
The next step is to try and make sense of the password data. The system lets us send as many usernames and passwords as we like.

Chosen Plaintexts
By sending data we select through the password encryptor, we can then analyze the results.

Username: markgamache
Username Hex: 6d 61 72 6b 67 61 6d 61 63 68 65

Password : 1q2w3e4r
Plaintext hex: 31 71 32 77 33 65 34 72
Cyphertext Hex: 0e 61 55 62 7b 02 22 63

Password: 11111111
Plaintext hex : 31 31 31 31 31 31 31 31
Cyphertext Hex : 0e 21 56 24 79 56 27 20

Password: 111111111
Plaintext hex : 31 31 31 31 31 31 31 31 31
Cyphertext Hex 0e 21 56 24 79 56 27 20 e9

Password: 22222222
Password hex: 32 32 32 32 32 32 32 32
Cyphertext Hex : 0d 22 55 27 7a 55 24 23

Password: 33333333
Password hex : 33 33 33 33 33 33 33 33
Cyphertext Hex : 0c 23 54 26 7b 54 25 22

Password: 44444444
Password hex : 34 34 34 34 34 34 34 34
Cyphertext Hex : 0b 24 53 21 7c 53 22 25

Password: 55555555
Password hex : 35 35 35 35 35 35 35 35
Cyphertext Hex : 0a 25 52 20 7d 52 23 24

Password: 12345678
Password hex : 31 32 33 34 35 36 37 38
Cyphertext Hex : 0e 22 54 21 7d 51 21 29

Username: ted
Username Hex: 74 65 64

Password: 11111111
Password hex : 31 31 31 31 31 31 31 31
Cyphertext Hex : ab 28 71 f4 fe cb 1d 46

Username: 1111
Password hex : 31 31 31 31 31 31 31 31
Cyphertext Hex : 51 2b f3 5c a6 76 db 43

Username : 2111
Password hex : 31 31 31 31 31 31 31 31
Cyphertext Hex : af a8 8d c5 48 db 27 7c

From the small sample, we can see a few things:
• If the username is unchanged, then the password data seems to be consistent
o For markgamache, as long as the password starts with 1, the cyphertext starts with 0e
o The transform is not s simple substitution, as the 0e only corresponds to 1 as the first character
o Incrementing the value of a character in a particular position produces predictable results. In pos 1, 0e = 1, 0d = 2, 0c = 3
o The pattern is not completely predictable. In Pos 4, 24 = 1, 27 = 2, 26 = 3
o The value of a position is not varied by the character before or after. The encrypted value of Pos 3, when the plaintext is 2, is 55, both when the password is 1q2w3e4r and 2222222
• If the username changes, the cyphertext value no longer match for a position, with the password unchanged. See users 1111 and 1112 above.
• The values at a position do not vary based on the overall password lenghth. See password 11111111 vs 111111111

This tells us that the encrypted version of the password is derived in part from the username. It also shows us that for any username, we can perform a simple position by position chosen plaintext attack to find the corresponding encrypted values. This means in about 100 attempts, a user’s password can be derived. As the application allows unlimited retries, this cannot be stopped. The account would likely be locked out due to password policy, but in 10 – 30 minutes, it is likely to be unlocked by policy.

Code Attacks
As the password is “encrypted” using the client side Java code, we can take a look at that code. While the app runs automatically when you click on a web link, there is nothing stopping us from getting the code to look at. Remember, we can’t execute the code if we can’t get to it.
A simple curl command allows us to get a copy of the jar file, AnalyzerClient.jar. From there, it is a simple task to extract the jar and decompile the classes. There are lots of them, but we are saved time by the data from the wire, remember hyperion.atf.security.authentication.SimplePasswordToken.
Sure enough, there is a class named hyperion.atf.security.authentication.SimplePasswordToken. The class has both encrypt and decrypt methods referencing com.hyperion.atf.utils.encryption.EncryptionManager.
Upon investigation of EncryptionManager, we see that the username is used as the key and RC4 is used to encrypt the password. This is bad, but is validated by the chosen plaintext operations.
Following the bouncing ball and stripping the error handling for prettiness…
1. We get csPassword and a value from the wire.
2. We get csPassword = EncryptionManager.encrypt(asUserName, asPassword); from SimplePasswordToken
3. From EncryptionManager we get
public static String encrypt(String asEncryptionKey, String asClearText)
{
return encryptRC4(asEncryptionKey, asClearText);
}

And
protected static String encryptRC4(String asEncryptionKey, String asClearText)
{
byte myCypher[];
Rc4Cipher mRc4 = new Rc4Cipher(asEncryptionKey.getBytes("UTF8"));
byte myClear[] = asClearText.getBytes("UTF8");
myCypher = new byte[myClear.length];
mRc4.encrypt(myClear, myCypher);
return Base64Codec.encode(myCypher);
}

Indeed, the encryption key is the username, which is sent in the same packet as the encrypted password. The key is shipped in the lock.
All of our assumptions are confirmed.
Lastly, even thought the client should never need to decrypt the password, that’s the server’s job, the decrypt code is in the classes. This makes creating your own exploit code extra easy.

Character Positions Explained
Now that we know that RC4 is being used with the username as the key, we can see why there is a predictable shift in the cyphertext at any position if the username does not change. If you recall, for username markgamache, a 1 as the first character always yields 0e as the first byte of cyphertext. This is because RC4 is a stream cipher. It acts on a byte stream, one byte at a time. One byte in the stream had no mathematical effect on the previous or next byte. This is why block ciphers are considered better, as a rule of thumb. As our passwords are made up of characters that are, unluckily, one byte each, each character is essentially encrypted by its self. To defeat this, some sort of bit shift or IV (initialization Vector) should be used. It’s all moot though, as the key is shipped with the lock, as it were.

Summary
Hyperion passwords, and any linked SSO directory passwords, are put at considerable risk by simple code inspection or by chosen plain text attacks. While it has not been tested, a session with the application can probably be established via a TCP replay. Without additional controls, it is likely that sessions can also be hijacked. While the last two issues do not put passwords at risk, they do put the application and its data at risk.
Hyperion should not be used without SSL via a certificate that chains to a trusted root certificate authority. This will provide for secure key exchange, secure authentication, stop replay attacks and make the world a better place.

This just reinforces the basic security principal; Trust well vetted cryptosystems (like SSL) over custom solutions.

My First Vendor Security Flaw Submission

2009-05-25T22:28:00.000-07:00

I sent in my first security analysis/vulnerability to Oracle on Friday. It was for the product, Hyperion Web Analysis Studio 9.3. I've heard horror stories about vendors abusing security professionals, rather than being thankful. I hit send with a bit of trepidation. I got a fairly quick reply (12 hours or so) that was very pleasant, thanking me for the analysis and asking for a few clarifications. It was clear that my analysis was thoroughly read and understood. I must say, they are friendly folks. I'm waiting to hear back as to when I can blog on my analysis, so something will be forthcoming in the 3 4 weeks.

One thing that was annoying was that it took me quite a while to figure out where to send the information.

On a side note, the flaw is not serious, as long as you use SSL, which you should be doing anyway.