Paul Hite

Weathering the Cloud: Moving your service company ahead in the age of SaaS

2011-10-22T14:38:00.000-05:00

With phrases like "cloud computing" making their way deeper in to the everyday dialect of non-technical business managers, it no longer comes as a surprise when decision makers approach me about cutting costs by using "the cloud". They may not know exactly (or even vaguely) what that means, but they know it might save them money, so the door is wide open for a discussion about _aaS ("Something" as a Service - Software, Infrastructure, Security, and so on).

This would appear to be a huge sales opportunity for I.T. service companies, but my observations of several have shown that adoption can be slow for a number of reasons:

Lower Perceived Profits - With clients spreading the TCO of software/hardware over the course of their commitment, as opposed to making large upfront purchases, the profit from a single sale takes a long time to realize for a service company - not always an acceptable situation for a business just getting in to reselling the _aaS space. Selling a client a $10,000 Exchange 2010 server plus labor vs a measly $200 a month for an easy-to-deploy hosted solution seems counter-intuitive at first glance.
Resistance to Change - Change is not always welcomed, even by the I.T. veterans who typically operate small service companies. Old-school suspicions about the security, reliability, and longevity of hosted services are deeply ingrained in many people. Others feel threatened by the technical and financial aspects of operating a cloud-focused business, which are a drastic departure from the comfortable install-break-fix model many companies have been built on, and which also require a very different skill set to maintain.
Analysis Paralysis - A rapid explosion of cloud systems, vendors, and services have resulted in a huge selection of products to sift through. Researching, finding, and implementing the services that will be most profitable is a daunting task that can leave your decision making skills comatose.
Limited Investment Funds - The cloud is all about moving CAPEX to OPEX for the end-user, but someone has to make the initial investment to build it. For custom IaaS solutions, this expenditure could fall on the service provider. After all, the cloud has to physically exist somewhere, and someone will have to pay for the physical hardware that powers the virtual.

With these challenges, how does an I.T. company that thrives on the instant capital of hardware, software, and labor sales become a lean, agile provider of pay-as-you-go (and pay-as-you-grow) cloud services? Here's my two cents on the subject:

Take stock of your employees - The entire company has to embrace the cloud and be excited about its potential at every level, from sales to operations to management. Bring in fresh faces who are passionate and eager to work with cutting edge virtualization solutions. Make sure your financial officers understand the model and are on board with changing your income generators from large one-time projects to recurring monthly revenues.
Start with reseller opportunities - Jumping head first in to building your own private cloud could spell disaster. Work with existing providers who have strong channel partner programs and can provide the infrastructure (and the some of the liability) for your product.
Use what you sell - The world of cloud computing touches on so many aspects of technology, it can be daunting to understand. The best way to learn is to use the service you plan on selling within your own firm first. Not only is it good for your staff to become familiar with it, but it is a great way to reassure clients that your cloud product is mature and reliable.
Streamline Operations - Providing product as a service and at a distributed monthly rate means your company must be laser-focused on efficiency. When your clients were simply billed directly for time and material, you were passing the cost of inefficiency along to them. Now that they pay you a flat-rate each month for a particular service, every dollar you spend maintaining that SLA comes straight out of the bottom line. The more servers you can maintain or accounts you can manage without adding staff will keep you in the green and allow you to stay competitively priced.
Understand your clients - Not everyone is a good fit for the cloud computing model, but many are and just don't realize it yet. Learn to recognize the pain points in your customers that can be relieved with a move to cloud services. Then when you here someone say "Our e-mail just isn't reliable enough" or "We are spending too much on server maintenance", your inner sales person will scream out to sell a cloud product.

But most importantly be excited. This should be easy: There is so much potential and rapid growth in this area, it's hard not to indulge your inner geek in the possibilities this model offers.

Newest Hite Member

2011-07-29T09:46:00.000-05:00

Blog updates dragging down to zero on account of my new boss:

BPOS Offline .... again

2011-05-12T12:09:00.000-05:00

*** UPDATE 12:12PM CST ***

From Microsoft Health Dashboard:

The BPOS Operations team is working to resolve service degradation for Exchange Online mail flow for organizations served from this region. Users in affected organizations will experiencing ~40 minute delays when trying to send or receive e-mail using Outlook, OWA, or mobile devices. The BPOS Operations team is actively working to restore service. Next update will be within one hour or when new information is available.

Not sure about anyone else, but ~40 minutes is off by about infinity for me.

----------------------------------------------

Two days in a row now, and if you were watching Twitter it would appear to be affecting a lot of people in the NOAM region. Oddly, the Health dashboard still doesn't show a problem:

Try calling MS support though, it will just hang up on you.

Microsoft Exchange Online Outage

2011-05-10T13:35:00.005-05:00

**2:20PM CST UPDATE**: Microsoft posted another update to the NOAM Health board:

The BPOS Operations team continues to investigate service degradation issues with Exchange Online mail flow for organizations served from this region. The next service update will be provided within 2 hours if the issue is not resolved.

Hopefully we here something else soon.

------------------------------------------------------------------

Microsoft Exchange Online (and BPOS by extension) are currently having issues with mail flow (read: there isn't any).

From Microsoft as of 11:40am:

The BPOS Operations team is investigating alerts indicating service degradation for Exchange Online mail flow for organizations served from this region. Users in affected organizations may be experiencing delays when trying to send or receive e-mail using Outlook, OWA, or mobile devices. The BPOS Operations team is actively working to determine the root cause and restore service.

When I called in initially, I could hear the service manager yelling in the background as the call queue exploded from a few users to over 60 in a matter of seconds.

Who else is seeing this issue? What have been your impressions of BPOS recently? Anyone on Office 360 having the same problem?

Don't Quit I.T.

2011-03-05T14:48:00.000-06:00

A post from Rick Bauer on CompTIA's website caught my attention today. Rick does internal R&D on future certifications and I really agree with the comments he made in response to a TechRepublic article. IT is frustrating, underappreciated, stressful, and nearly impossible to maintain proficiency in. And I highly recommend it. Embrace the madness!

http://blog.comptia.org/2011/03/03/dont-quit-it-because-of-your-it-job/

Shutting down an SMTP AUTH Relay attack

2011-02-22T18:52:00.001-06:00

We recently had a client whose Exchange Server (configured by another I.T. company, mind you) was continously being blacklisted with no immediately identifiable cause. Our first responders checked the normal stuff and verified that the server was not an open relay in any obvious way. At the time I jumped in on the support ticket, the server did not have the proper logging enabled, so we couldn't see exactly what was occuring.

The first step was to determine if anyone was actively spamming. There are a couple logs and tools that are useful for checking current Exchange connections, but we ran Exchange User Monitor overnight on the assumption that it was a client machine sending mail via the Exchange server. However, we didn't spot any MAPI or OWA connections being made in the off-hours, so we moved on to enabling some additional logging in Windows.

First, we enabled SMTP logging in IIS to give us a bit more information as to what connections were being made. Our logs (truncated with x.x.x.x to preserve privacy) showed the following:

#Fields: date time c-ip cs-username s-ip cs-method cs-host cs(User-Agent)
2010-11-18 00:00:34 x.x.x.x localhost 192.168.15.250 EHLO - -
2010-11-18 00:00:35 x.x.x.x localhost 192.168.15.250 MAIL - -
2010-11-18 00:00:35 x.x.x.x localhost 192.168.15.250 RCPT - -
2010-11-18 00:00:37 x.x.x.x localhost 192.168.15.250 DATA - -
2010-11-18 00:00:37 x.x.x.x localhost 192.168.15.250 QUIT - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionResponse - - - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionCommand - EHLO - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionResponse - - - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionCommand - MAIL - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionResponse - - - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionCommand - RCPT - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionResponse - - - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionCommand - DATA - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionResponse - - - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionResponse - - - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionCommand - QUIT - -

Thousands of these lines logged while the client's business was closed (11:00pm to 3:00am) indicated not only massive amounts of spam being sent out, but that the attacker was external and was authenticating to the network. The first 5 lines show an EHLO connection, slightly different than the HELO in that it supports authentication, coming from an external source (which we subsequently traced to the UK and notified the business owner). The next set of lines is a spam message being sent from the local Exchange server out to the next victim.

While we had initially checked and made sure the server was not an open relay, allowing anyone to send to external domains without authenticating, most servers still allow authenticated relaying by default. This means that if you can authenticate by SMTP to the Exchange server with a valid username/password, you can relay to external domains. Not a big deal, as long as none of your users have a compromised account or weak passwords.

In order to spot which account was compromised, we turned to the Windows Event Logs. These showed a series of 1708 informational messages in the Application log as well as Success Audit messages in the Security log that pointed to an account that had been created called "test". I won't go in to why this is a bad idea for a username, but the password we found was even worse...

My colleague, Jon Jeffels (http://blog.jeffels.net/), did a little extra work and ran THC-HYDRA to crack the user account via SMTP authentication, probably the same way it was broken to begin with. Within a few seconds (with transforms applied) we had a password of "test1" to go with our username of "test" ... not very secure, folks!

Preventing SMTP AUTH Relay Attacks

There were a myriad of things that could have prevented this problem:

The best solution, in my opinion, would be to use an external mail filter of some sort. There are a lot of other reasons for this, and hosted solutions are too cheap to have any excuse for not using one. Use Smart Host settings in combination with this (and secure inbound connectors while you are at it).
Simply enforcing password security would have helped this temp user account stay secure
Or how about just not forgetting to delete TEMPORARY accounts?
Setting something besides the default lockout GPO of 50 attempts and 10 minute lockout would have been nice also. Perhaps 3 attempts and a 30 minute lockout?
Disable external relaying all together. Unless you have external users utilizing Exchange via POP3 (*shudder*), you shouldn't really need it. Check out instructions here for Exchange 2000/2003: http://www.amset.info/exchange/smtp-relaysecure.asp

While we have a ton of fun catching and shutting down attacks, the "real" attack was the password compromise that probably had occured weeks earlier. This was a totally preventable problem that was probably caused by a tech creating a "test" account, so shame on all of you I.T. companies out there who are using weak passwords and should know better!

SMB Cloud Champions Club

2011-02-22T18:12:00.000-06:00

We recently joined the SMB Cloud Champions Club, read the post here: http://infotech.us/company-blog/81-smb-cloud-champions.

Attack of the Local Failures

2010-11-20T11:38:00.000-06:00

You read that correctly. It is a 82 GB mailbox filled with Local Failure messages. Jon Jeffels spotted this when we couldn't figure out why his Outlook client kept crashing and hanging. Read his full post here:

http://blog.jeffels.net/2010/10/outlook-issues/#more-61

Trend Micro WFBS UPX Packed Updates

2010-11-15T20:04:00.000-06:00

I had a client with a myriad of issues related to some infected machines on the network. While digging through the firewall logs (SonicWALL TZ-170 or 190 I believe), I found this entry:

Typically we wouldn't want to see packed executables attempting to be retrieved from the WAN, but these occured at regular 15 minute intervals and the source IP was owned by Trend Micro. Digging in the WFBS console I found:

A whole series of failed updates matching up to the UPX packed executables being blocked in the SonicWALL.

Anyone have experience with UPX compression? Is this standard practice for AV definitions to come as a packed executable? Or is the burden on SonicWALL here to get a little more detailed?

Case-sensitive DNS? Believe it.

2010-11-04T20:41:00.000-05:00

DNS is not case-sensitive, that's what you thought, right? Me too, but apparently somebody disagrees.

We have a client who uses a Cisco ASA 5510 to provide remote access to several road warriors and small satellite offices. We recently started putting Windows 7 64-bit machines in the network, which has traditionally only had 32-bit XP laptops. The Cisco VPN client we usually utilize doesn't work on Windows 7 64-bit (routes don't get created properly). I realize that there is a compatible beta version out and the latest SSL VPN works, but unfortunately due an unbelievable snafu between Tech Data and Cisco, we were left high dry without a SMARTnet agreement. So it came down to using an excellent open-source VPN from Shrew Soft, which is compliant.

I tested the Shrew Soft product and verified it worked great on my local Windows 7 32-bit machine. I also tested it successfully on a 64-bit box, no problems.

Problem:

We use split-tunneling/split-dns to ensure that all traffic for "company.local" is routed over the VPN. Our configuration also specifies that unqualified names, like "server1", get appended with "company.local". So if I ping "server1", it should actually ping server1.company.local automatically and thus get sent across the VPN. Easy.

We went to deploy it to the client and quickly found out that name resolution was not working over the VPN. Attempting to ping "server1" just timed out while the packets shot out the local network instead of getting passed through the VPN. Since everything worked fine on my machines, I was baffled for a minute as to what the issue could be.

Cause:

The clients laptops were all connected to the domain "company.local". So they should be appending company.local to the end of every unqualified request to form a FQDN anyways, making this even more confusing.

However, a little digging showed that they were actually appending a capitalized version of the FQDN like so: server1.COMPANY.LOCAL. No big deal, right? Wrong! Our Cisco config only specified a lower-case version of the FQDN in the split-dns configuration. For some reason, the Shrew Soft VPN client (or perhaps some component used in it) interprets company.local and COMPANY.LOCAL as different domain names.

Resolution:
Add the upper-case version of company.local (COMPANY.LOCAL) to the split-dns configuration like so:

split-dns value company.local COMPANY.LOCAL

Both values are passed to the client which then allows the proper DNS requests to be intercepted and sent over the VPN.

I'm pretty sure this is a Shrew Soft issue, and perhaps one that only exists in Windows 7 or 64-bit machines. This shouldn't happen (and didn't) with the Cisco VPN client. According to Cisco:

The client receives a comma-delimited list of split DNS names from the Concentrator via modeconfig. When the Software Client receives a DNS query packet, the domain name is compared and equentially checked against the split DNS names. Case-insensitive domain name comparison starts at the end of each domain name string and continues toward the beginning of each string, resulting in a match, or no match.

Other than this little quirk, Shrew Soft has produced a great and much-needed product, so I'm certainly not bashing them at all. Maybe something to throw in the next bug release though? Or maybe I'm completely off-base here; the inner-workings of VPN client architecture is not something I'm familiar with.

Not sure what a "comupter" is ...

2010-10-15T08:37:00.000-05:00

I used to work there, nice to have reminders of why I left:

End Users are Funny Pt II

2010-09-08T17:10:00.000-05:00

Here's another conversation during a remote support session today:

Client: "So Paul, have you seen the internet today?"

Me: "Well, I've been on the internet today, does that count?"

Client: "Take a look at the front page of the internet when you can. There is this crazy looking guy on the front page."

5 Things I Learned from Air Force Networks

2010-08-20T23:46:00.003-05:00

After completing what may very well be my last tour as a Cyber Systems Operator with the Air Force Reserves, I'd like to offer this (unclassified and possibly incorrect) insight into the wonderful system that is the Air Force Global Information Grid (AF-GIG):

The more complicated the password requirement, the more likely it is to be found written on a sticky note next to a monitor.
The scream-test is the most effective method of determining whether a system is critical or a link is live. Unplug it and if someone screams at you, it was important.
Never complain about having to use outdated technology to the person who has to try and maintain that outdated technology.
The more classified the network, the older the equipment. I'm confident that at the center of the most secretive AF networks, there is just an abacus.
You don't need to know the acronyms meaning, you just need to know where the acronym is at in the rack so you can reboot said acronym when your CC complains that his acronym is not working.

Secunia Half-Year Report 2010

2010-07-26T22:35:00.000-05:00

I usually try to avoid reposting information from others, but Secunia has a great security report that has some down to earth information relevant to every computer user:
https://docs.google.com/viewer?url=http://secunia.com/gfx/pdf/Secunia_Half_Year_Report_2010.pdf

Among the interesting statistics?

A group of ten vendors, including Microsoft, Apple, Oracle, IBM, Adobe, and Cisco, account on average for 38 percent of all vulnerabilities disclosed per year.
In the two years from 2007 to 2009, the number of vulnerabilities affecting a typical end-user PC almost doubled from 220 to 420, and based on the data of the first six months of 2010, the number is expected to almost double again in 2010 to 760.
During the first six months of 2010, 380 vulnerabilities or 89% of the figures for all of 2009 has already been reached.
A typical end-user PC with 50 programs installed had 3.5 times more vulnerabilities in the 24 3rd party programs installed than in the 26 Microsoft programs installed. It is expected that this ratio will increase to 4.4 in 2010.
An Windows end-user can patch 35% of vulnerabilities with one tool from Microsoft, but requires another 13 to patch most of the rest.

Also, in terms of the raw number of vulnerabilities, Apple has shot past Microsoft and Oracle to claim the #1 spot. However, this doesn't take into account how severe the vulnerabilities are or how quickly a patch is released.

Symantec Installation Failure

2010-06-10T12:21:00.000-05:00

We've been looking for alternatives to Trend Micro WFBS lately, and I thought I'd give Symantec Endpoint Protection another shot. I spent several hours fighting with the installation package and Symantec Support, but continually had issues with a VBS file that is supposed to run in the setup. A week and several pots of coffee later, I found the problem.

During the installation, Symantec executes iisconfig.vbs, a script designed to setup all of the IIS elements for the management portal. However, the installation rolls back and SEPM_INST.LOG showed the following "return value 3" message everytime.

SESM CA: Failure in IIsConfig.vbs script - See the Windows Event Viewer application log for the failure event.

I went through every article on Symantecs website concerning the issue with no luck. Symantec insisted this was an IIS problem, but even reinstalling IIS did not resolve it. Luckily the issue jumped out at me before it came down to a wipe and reload.

I ran Process Monitor while the install was running and noticed the following line:

MsiExec.exe IRP_MJ_CREATE C:\WINDOWS\system32\cscript.exe ACCESS DENIED Desired Access: Read Data/List Directory, Execute/Traverse, Read Attributes, Synchronize, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, ShareMode: Read, Delete, AllocationSize: n/a NT AUTHORITY\SYSTEM

IRP_MJ_CREATE is the function used to open a file system object (or create a new one), so I looked at cscript.exe and sure enough, the SYSTEM account was set to deny all on the security permissions. I don't see this on any of my other 2003 servers, so I'm assuming it was a result of some hardening at one point in time before I inherited this client. As it turns out, the problem had nothing to do with IIS technically.

If anyone can tell me what sort of automated hardening (Security template perhaps?) causes this I would love to know so I can delve into other issues that may creep up on this server.

End Users are Funny

2010-05-25T19:28:00.001-05:00

I have a client who submits tickets and e-mails to me in haiku.

Paul I have a problem,

our e-mail is not working ...

can you call me please?

I have not decided if this is intentional or not.

Can you please help me?

Susan cannot print today ...

and reports are due.

But it is certainly relaxing to read.

New company blog

2010-05-19T12:19:00.000-05:00

My employer recently let me setup a new blog, so posts may slow down here while I work on getting it filled with content!

Facebook Accounts Are Almost Worthless

2010-04-28T17:59:00.001-05:00

How much would your Facebook account bring you if you sold it? According to VeriSign, just a couple of pennies. I spotted an article on Dark Reading the other day detailing a recent black market advertisement for over a million Facebook accounts, sold in 1,000 unit increments for $25-$45. About 2 to 4 cents per account, which highlights how easy it has become for these accounts to get compromised and used to spread malicious software.

This little fact just shows you how prolific Facebook has become as the new medium for malware delivery, replacing traditional e-mail. It also is an ominous indicator to the fact that people are much more trustworthy of links coming from their favorite social networking site than they are about the fishy e-mail in their inbox.

Most users are now aware of the basic "Don't open attachments or links from suspicious emails" mantra, but we have a ways to go before that education extends to social networking. The key is getting our users to understand that Facebook, Myspace, YouTube, a website, and e-mail are all simply modes of transportation for a malicious link or attachment to be delivered, and the same due diligence should be paid to each.

Secure the Edge: Protecting Exchange 2007 Connectors

2010-04-20T13:26:00.000-05:00

Many small companies employ a service such as AppRiver to provide affordable messaging security, but not many take the time to properly secure their Exchange connectors afterwards.

Introduction
A very common configuration for SMBs, particularly those running Microsoft Small Business Server, is to throw all the Exchange services on one box, forward ports 25, 80, and 443 to it, plug in your MX records, and call it a day. However, any experienced messaging administrator will tell you that it's a security concern to have your mailboxes sitting on the same server that has a publicly accessible SMTP service. Basically, you are inviting anyone and everyone to come put files on your server. Couple this with the fact that many businesses are running their servers with no messaging security component, such as Trend Micro's "Worry-Free Business Security Advanced" or Microsoft's Forefront for Exchange, and you have a recipe for quickly piling up unwanted mail, or worse.

Although a large enterprise might normally setup an edge server in a DMZ to receive mail, that's not a realistic option for the small business. So, in swoops a service such as AppRiver to save the day, providing you affordable hosted messaging security by acting as the middle-man for incoming (and usually outgoing) mail between you and the rest of the internet. Problem solved, right? Wrong. Normal, RFC-abiding mail servers may send all your mail to the secure hosted system you specified in those MX records starting with the lowest number, but not the wise spammer. He'll find port 25 open on your network and start spamming away, completely bypassing the filter. Here's how to avoid that and ensure your Exchange server only communicates with your filtering service.

Step 1: Lockdown Receive Connector
Fire up your Exchange console and drill down to Server Configuration > Hub Transport. You should already have one "Internet" connector there. Although Microsoft recommends you simply change this one, I like to leave it in place as a quick way to "re-enable" inbound mail from all sources should you ever need to.

Start by creating a new Receive Connector (Action > Server > New Receive Connector ...). Give it a logical name and specify that this will be a "Custom" connector. You should be able to keep the defaults where they are unless this is a multi-homed server and you only wish to receive mail on one interface. Don't forget to set your HELO response banner to the external FQDN (e.g. mail.domain.tld). This would also be a good time to mention that proper PTR and MX records are important.

Now that you have your connector in place, right-click on it and let's change the properties. First, you'll want to remove the entry for 0.0.0.0-255.255.255.255 in the "Receive mail from these IP addresses" box. Enter all of the IP addresses provided by your hosted filtering service (AppRiver's can be found here).

Leave all the authentication mechanisms off and what we have is an external connector that will only listen to mail coming from AppRiver. Now you can disable your internet connector and test the configuraion.

Step 2: Lockdown Send Connector

Outbound filtering is a great feature that most hosted filtering providers offer. Sometimes this feature has to be requested, but it will help prevent your users from forwarding spam or sending viruses to other networks.

Create a new Send Connector (Action > New Send Connector). Assign a name and use the "Custom" option for the type of connector.

Add a new SMTP address space, using an asterisk (*) for the address. This will tell Exchange to use this connector for all destinations. In the network settings, select the option to "Route mail through the following smart hosts". Use the FQDN or IP of the smart host you were given by your filtering provider (server###.appriver.com, for example) after clicking "Add".

Your provider will tell you if they require authentication (AppRiver does not), which will need to be specified in the "Configure smart host authentication settings" screen. After your send connector is setup, just disable the existing Internet connector and enable your new custom connector.

Of course, you can do all of this from the command line in EMS if you like. If you are curious about those commands, you can view the shell command output from any of these actions when looking at the completion screen in EMC.

Reflections on Google Apps

2010-03-20T10:27:00.001-05:00

I just completed our first deployment to Google Apps Premier for a client and a had a great time with it. I've already moved my domain (paulhite.com) over to it and I highly recommend anyone who needs a mail system for their domain to consider it. Here are a couple thoughts on the overall process and results:

Carefully consider the clients technical prowess. Although Google has taken great strides to integrate the service with Outlook (And soon the rest of the office suite), there will be a learning curve involved. Our client was very tech-savvy and thoroughly enjoyed learning about all the new features, but that will not hold true for everyone.
Plan your deployment out ahead of time. Google makes it pretty painless for even complex configurations (dual-delivery, mixed environments, etc.), but read over the entire deployment guide before you get going. There are a lot of "gotchas" that popup. For instance, support for a GAL is not nearly as straightforward as Google insinuates on their website.
A smooth migration is the key to happy adoption. Lost emails or contacts will not be received well, obviously. Google makes moving IMAP and POP3 accounts easy, but be sure you understand how it works first. Two additional thoughts on POP3:

If the client plans to use the Gmail interface and currently uses Outlook, go with the Email Uploader utility and then install the desktop access points.
If the client plans to continue using Outlook, just install the Sync tool, import their data, and it will do the rest.

For staged deployments, get your account running on Google Apps first, create your users, and then use the mail flow rules to slowly move the stream of email for individual users to Google Apps. This way you can continue to operate on both the old and new messaging infrastructure, giving you plenty of time to migrate.
Google Apps is not Microsoft Exchange. It is still inferior in some areas, but far ahead in others. Carefully evaluate the clients needs and current usage, particularly if they are already on Exchange, before migrating to GA.
Support sucks. In comparison to Appriver and other hosting services, being required to wait several hours for an email response is atrocious. Be a Google Apps expert before you attempt to deploy, or you may end up with the short stick when it comes time to blame someone for poor service.

Additionally, you'll almost certainly want to set these features as soon as the domain is online:

Create CNAME records for all the major components (mail.domain.com, calendar.domain.com, etc.)
Create SPF records to help prevent forged mail from your domain.
Enable contact sharing (Users And Groups > Settings > Enable contact sharing)
Change the user support contact information to your company (Domain settings > General > User support)
Enforce SSL on all Google Apps sites (Domain settings > General > SSL > Enable SSL)
Hide ads for all pages (Domain settings > General > Advertisement option > Hide all ads for this domain)
Change the default time zone (Domain settings > General > Time zone)
Upload a custom logo (Domain settings > Appearance > Header logos)
Get Google on your Desktop

Download the Google Apps desktop access points (Advanced tools > Google Apps desktop features)
Download Google Apps Sync for Microsoft Outlook (Service settings > Email > Outlook & BlackBerry Support)

Enable Email Migration API (Advanced tools > User email uploads)
Enabled Google Apps Sync, Offline Gmail, Gmail Labs, and Voice/Video Chat (Services Settings > Email)
Enable Postini mail filtering (Dashboard > Add Services)

There are a lot of other features you may or may not want, but the ones I just listed are nearly universal. Google Apps is not for everyone, and it still has some maturing to do, but the price point and feature set are excellent. If Google continues to add features as quickly as they have in the past and improve support, the Apps service will make a fine addition to your solutions arsenal.

Procrastinator's Recipe for a Great Project Paper

2010-03-02T20:47:00.000-06:00

I'm currently at step 6, and so was born this post.

1. Remove one great idea from brain. Put aside for several weeks.
2. Try really hard to remember what that great idea was. Decide on a different mediocre idea instead and write it down this time.
3. Let the idea simmer until one week before the project is due. Victory will taste sweeter this way.
4. Start preparing project paper. Don't research anything, it just slows you down. Stream of consciousness is most effective. Continue until you run out of thoughts, or approximately 500 words into the paper.
5. Ctrl+A, Delete. Repeat Step 4, that introduction page sucked anyways.
6. Mix 1 part great idea with 9 parts alcohol. Maintain the Balmer Peak for 72 hours and write 30 pages of really awesome sounding techno-babble.
7. Suddenly remember that great idea when you sober up. Repeat step 5 & 6.
8. Remove project paper from hard drive. Serve to reviewers at room temperature and really hope that none of them actually knows what "deep packet inspection" means.
9. Celebrate your success by repeating step 6. Great idea is optional at this point.

That's it. Good luck out there, WGU grads!

Notification: WSB Can Do Notifications

2010-02-22T10:28:00.005-06:00

I had a client who, much to my dismay, purchased a new server from Dell and opted not to go with Backup Exec but rather the free Yosemite Backup application that ships with the RD1000 drives. Not a huge deal until I discovered after a good deal of troubleshooting that Yosemite does not work with Server 2008 R2. Long story short, we were left with no option but to use Windows Server Backup and find a way to configure e-mail notifications.

According to Microsoft and the forums I looked at (including Expert's Exchange), the only information I found was either a big fat "not supported" or some convoluted and surely unreliable scripted solution. Several folks were on the right track with using Task Scheduler. However, everyone I spoke with lamented that WSB does not appear to log anything useful in the System or Application logs except this event:

Which tells us nothing except that the backup may be starting and is not at all useful to us. Take a look in the Applications and Services Logs > Microsoft > Windows > Backup > Operational log and we find something much more suitable:

So now that we see where our event logs appear for WSB, we can use Task Scheduler to create some email alerts. First, you'll need a few prerequisites on the server:

Windows Server Backup obviously has to be installed. Unlike NTBackup in 2003, it is not included in the default installation in Server 2008.
Simple Mail Transport Protocol will be needed if you don't already have an SMTP server you can relay off of (such as an ISP server). You will need to configure it to allow relaying from your WSB server with no authentication. WSB is not capable of authenticating in any way to an SMTP server.
IIS 7.0 needs to be up and running to house your SMTP virtual server.
IIS 6.0 Manager also needs to be installed to manage the legacy SMTP stuff.

To get started, open up Server Manager and drill down to Configuration > Task Scheduler > Task Scheduler Library > Event Viewer Tasks.

1. Select Create Task... from the task pane.

Enter a name and description for your new task. Be sure to select the option to "Run whether user is logged on or not" to ensure the task still runs after you log out.

2. Click on the Triggers tab and click on New.

Select "On an Event" from the drop down. Hit Custom and then open a "New Event Filter..."

3. Select all of the event levels (realistically, failures should only ever use Critical, Error, or Warning, but why take the chance?). Select "By log" and choose the Microsoft-Windows-Backup/Operational log to monitor.

Enter all of the event ID's for WSB errors and warnings and then click OK. What, you don't know these? Luckily TechNet lists them for us. Here is the string you can input for the the ID's:

5,8,9,17-22,49,50,52,100,517,518,521,527,528,544,545,546,561,564,612

You could also use Keywords such as "fail" and "warning", but this just seems cleaner and more reliable to me.

4. Head over to the Actions tab and select "New...". Most of this should be self-explanatory:

I'm sure you could script some way of exporting the logs and attaching them, but this should be enough for a simple e-mail notification just to say "Hey, go look at the server cause your backup failed". If you are using another server to relay SMTP messages, enter its IP address in the SMTP Server field. You can also specify more than one recipient by either using a distribution group or separating the email addresses with a comma.

5. If you want to test your email notifications, be sure to check the "Allow task to be run on demand" option under the Settings tab.

That's it for failure notifications! I also recommend you repeat these steps for success notifications (just change the events to event ID 4 on step 3). That way if the server just drops off the network completely or has another issue preventing it from sending mail, the lack of email message will help indicate a problem. Be sure to schedule a "backup failure test" using a bad target of some sort to double-check that your failure messages trigger appropriately.

AOL Can Break Rules

2010-02-11T10:04:00.004-06:00

While testing an SMTP connection to AOL's mail servers, I got this awesome syntax error response when I forgot to put a space between "MAIL" and "FROM":

220-mtain-mi12.r1000.mx.aol.com ESMTP Internet Inbound

220-America Online (AOL) and its affiliated companies do not
220-authorize the use of its proprietary computers and compuer
220-networks to accept, transmit, or distribute unsolicited bulk
220-e-mail sent from the internet
220-
220-Effective immediately:
220-AOL may no longer accept connections from IP addresses
220 which no do not have reverse-DNS (PTR records) assigned.
HELO paulhite.com
250 mtain-mi12.r1000.mx.aol.com
MAILFROM:test@paulhite.com
221 2.7.0 Error: I can break rules, too. Goodbye.
Connection to host lost.

Yes you can.

Study Time!

2010-02-09T19:52:00.000-06:00

Updates to the blog will be more sporadic than usual as I concentrate on my last few months of college, my final design exam for the MCSE, and the upcoming Cisco course. Phew!

Does your network stink? Make a sniffer!

2010-01-26T22:06:00.005-06:00

Sometimes there is just no substitute for looking at raw packet capture data when infiltrating troubleshooting a network. In normal situations, I'm content with loading Wireshark or Network Monitor up on a system that sits in-line with the data I require (e.g. sniffing packets from a DHCP server when you have clients that cannot get an IP address). However, there are many times when you cannot load sniffing software on the device in question, or don't want to. Perhaps you want to monitor inbound traffic to your ISP to a border router, or watch for packets leaving a device you don't manage. In these situations, you need a passive network tap to capture the traffic without touching the two end points to be monitored.

So here is how you can make your own professional-looking passive network monitoring device for your toolkit. You'll need the following:

4ct Cat5e Jacks (or inserts/modules, they have several names)
1ct 4-Port Surface Mount Box
~6" of Cat5e Cabling
Punchdown Tool with 110-Block attachment

Our passive network tap will consist of the following:

2 ports for connecting our devices to be monitored (Labeled A & B here)
1 port for monitoring the first pair (1 & 2)
1 port for monitoring the second pair (3 & 6)

Without some actual circuitry, we won't be able to monitor both upstream and downstream information at the same time. However, if you have two network ports on whatever device you are using to sniff the wire, you could bridge them to monitor both pairs at once.

Instructions

First, cut off about 6" of Cat5e cabling and strip the insulating wrap off.

Terminate port "A" with all 4 pairs using the TIA-568B standard. You can use TIA-568A if you are weird like that, but you'll have to reverse the labeling later on.

Next, untwist a small section of the orange pair (corresponding to pins 1 & 2). Terminate this pair, WITHOUT cutting it (i.e. use the blunt 110-block tool), to the receiving pins on your first monitoring port. If you are using Cat5e modular jacks that are labeled, this will be the Green & Green/White pins if you are looking at the 568B standard.

Repeat this for the green pair (corresponding to pins 3 & 6), connecting them to the green & green/white pins of the second monitoring port. Be sure to leave some slack in between the jacks. Don't cut the cable yet! We want that signal to continue on to its destination, port "B".

Now simply terminate the "B" jack according the same standard you used for the "A" jack (Should be 568B if you listened to me!). The final product should resemble this (ignore the A & B labels, I swapped those around):

Throw the cover on and label the ports as appropriate:

Now just pick your port, plug in your favorite sniffing application, and start monitoring! If you connect a PC in port A and switch to port B, then the "1,2" port will give you the PC->Switch traffic and "3,6" will give you Switch->PC. You'll have to think ahead to figure out which port to use when.

Dispelling Common Myths

One of the reasons I chose to post this is that there is a wealth if misinformation out there concerning network monitoring. Here are the most common comments I see about DIY network monitoring:

Q: Can't you just connect to the same switch as the device you wish to monitor?

A: No, switches do not forward all traffic to all ports, like hubs did (past tense, they are hard to find now). Switches maintain forwarding tables that keep track of which MAC addresses are connected to which port, and will only forward traffic to you if the traffic is intended to reach your port, or if the traffic has been sent to a broadcast address. Notice I said MAC address (Layer 2), not IP address (Layer 3). With a few exceptions, switches only operate at Layer 1 & 2 of the OSI model. Some switches are capable of port mirroring, essentially making a copy of all port traffic from one port to another. However, if you are looking for problems caused by Ethernet frame errors, these will be discarded by most switches and you will never see them with port mirroring.

Q: So then I can use a hub to monitor network traffic?

A: Yes, you certainly can. This is a common use for hubs nowadays. But keep in mind that most hubs you'll find are much slower than their switching counterparts. In addition, that bandwidth (usually 10mbps) is shared between all ports and you will be placing your monitoring station on the same collision domain as the two devices you are monitoring. This is NOT recommended, particularly for production environments. It introduces errors that you'll have to weed out from whatever errors you might be trying to troubleshoot. It is also not truly passive without a special cable, which leads to the next question...

Q: Why can't I just connect both the transmit and receive pairs (1&2, 3&6) to my monitoring port?

A: I see this suggestion a lot, it's even included as part of the instructions in a horribly inaccurate Instructables article (Which, strangely, is written by a self-proclaimed flow analysis expert). Although it seems logical to do this at first glance, the truth is that connecting a Cat5 pair that is transmitting traffic to your transmit pins will not allow you to monitor the traffic. The transmit pins can ONLY transmit, not receive. What it will allow you to do is transmit on that wire. Not something you want happening if this is a critical line, or one where you wish to be silent during observation. Even setting your card to promiscuous mode does not necessarily preclude it from throwing random information onto the wire, which is even more haphazard when you are actually interrupting a transmission between two hosts who think they are dedicated.

Q: Is this reliable? Can I leave this tap in place permanently?

A: You can. I wouldn't. It's a recipe for all sorts of EMI issues. The twists in a Cat5 cable are there for a reason and we have just removed several. Additionally, the monitoring ports have sliced into your line and we essentially terminated it three times in each direction. This is going to increase attenuation slightly. Add this to the fact that we are not regenerating the signal in any way. Long story short, this little contraption gets the job done, but don't look at me if it breaks.

Hope this helps out someone else! Post your thoughts below. Did I miss anything?