Paul Hite

Red Flags and the Value of Experience

2022-03-18T12:45:00.002-05:00

One of the things I hear often said, and something I subscribe to as well, is the idea that a lot of technical knowledge in the world of IT has a very short shelf life. When interviewing candidates, we really tend to focus on what they've done in the last few years when considering their technical skill sets. As a practitioner, I start feeling a little stale on my knowledge if I don't touch an area of IT for awhile. That's not to say that anything beyond 3 years of experience is worthless though, because the value of good experience isn't really in technical skill. What you should be paying those IT greybeards for is their presumed ability to see patterns and discern bad decisions before they are made. That's a skill of it's own, and a muscle that needs to get some exercise by being allowed the opportunity to voice an opinion and have it feel valued.

A recent real-life example that recently occurred illustrates this well:

A vendor is working on upgrading an application that is delivered in a traditional 3-tier (client > app > db) architecture. After troubleshooting an issue for several hours, one of their senior developers reached out to one of our network administrators asking for half a dozen changes to their service account and computer accounts for the DB servers in Active Directory. Buried in their (mostly) reasonable requests was this:

On a domain controller, please set the Kerberos Delegation to "Trust this computer/user for delegation to any service (Kerberos only)".

Now, back in the Windows 2000 days, this option was just called "Account is trusted for delegation" or "Trust computer for delegation" and it was considered just fine and dandy to click it if the situation required it. An admin whose technical skill is dated to that era will know exactly how to perform the requested action. A knowledgeable one from that era might even know what this action does with regards to ST and TGT tokens. But an experienced admin, regardless of their technical skill or knowledge of Kerberos, will see a red flag in the options that are now presented:

Three options for delegation

Even without an understanding of what those three options do, or even what delegation or Kerberos are, an experienced admin should be able to easily tell you which is most secure and which is least secure., because they've seen the words "trust" and "any" used in a thousand other contexts. An experienced admin should care about security, because they've had to clean up the mess when someone else didn't. An experienced admin should not defer these types of decisions to a vendor, because they've been burned by them before. An experienced admin should know better than to take the path of least resistance just to make something work.

All it takes is some experience to see the red flags in this request. With about 10 minutes of research and reading, you can quickly find out that unconstrained delegation is bad and no longer something Microsoft wants you doing. The lessons you've internalized from your experience, regardless of your skill, will help lead you to the right technical decision.

In this case, we simply said "No" to the vendor: "If delegation is a requirement, please tell us which published services it should be constrained to".

After about 30 minutes, the vendor replied that they had figured out the issue and it turned out to have nothing to do with delegation at all - an experience I think I've had a few times before ...

Converting Citrix PVS Image from XenServer to vSphere

2018-10-13T20:37:00.001-05:00

Having repeated this nightmarish migration several times now, here's the steps I've found to be most efficient:

Import your XenServer-optimized PVS image (as a VHD) in to XenCenter as a new VM.
Snapshot and boot the VM (just in case you mess up the next step you won't need to import again).
Uninstall the Citrix PVS and Citrix Guest Tools / Xen Tools bits.
Delete xen*.sys from c:\windows\system32 and c:\windows\system32\drivers
Reboot and make sure everything still comes back up. It should revert to a generic Realtek network driver.
Run VMware Converter on the VM. Alternatively you can export the VM from XenCenter as an OVA and then import it to vSphere.
Be sure you are using a VMXNet3 NIC on the vSphere VM, not an E1000.
Boot the resulting vSphere VM and install VMware tools.
Delete the ghost NIC that is left from the Realtek drivers (https://support.citrix.com/article/CTX221733), otherwise you will get the BNIstack error.
Install PVS target device software and run the imaging wizard again.
Follow all your normal steps for capturing a new image

If you run in to an IRQL_NOT_EQUAL_OR_LESS BSOD, you may be like me and have some piece of software set to redirect writes to the vDisk cache disk which no longer exists. Make sure you fix that prior to attempting a migration.

The most commonly recommended solutions for BNIstack errors during your first boot after capturing the image:

Make sure no ghost NICs are still present
Try uninstalling any antivirus and disabling IPv6
Install the hotfix for KB 2550978 (https://support.microsoft.com/en-us/help/2550978/0x0000007b-stop-error-after-you-replace-an-identical-iscsi-network-ada)
Consider changing the default open retry limits/interval for BNIstack (https://discussions.citrix.com/topic/377414-bsod-with-bnistack-and-cvhdmpsys/)

Happy migrations!

DHCP Fails after uninstalling Citrix PVS Agent

2017-12-01T20:42:00.000-06:00

While attempting to migrate a Citrix PVS base image to a new hypervisor, I uninstalled the PVS bits from the VDA and quickly found that DHCP had been broken. I have slamming my head against the wall a bit, I remembered something I had done years ago to this image to avoid an issue where PVS was failing in an older split-scope environment we had - set the PVS service "BNDevice" as a dependency of the DHCP service so that it would request the correct IP address during the hand-off to the OS (Thanks to Syxin https://www.syxin.com/tag/bndevice/).

This obviously was preventing DHCP from starting since the BNDevice service no longer existed after removing PVS tools. Simply needed to reverse that change:

HKLM\System\CurrentControlSet\Services\dhcp\DependsOnService

Remove BNDevice from the list of dependencies.

Upgrading R730 with NVIDIA K1 GRID Card

2016-01-12T13:11:00.000-06:00

I recently ordered a couple of Dell R730 servers and then got a subsequent request to add a little graphics horsepower for our VDI environment in the form of some K1 GRID cards. Turns out, the process to add these suckers in to an existing server that wasn't specifically built out for them has a few catches - you can't just drop them in to the server and take off. It's easy to do, but there isn't much official documentation from Dell on it, so here's a quick guide:

First, the requirements:

Compatible R720 or R730 series server
1100W PSUs
Compatible processor - there's a specific list of procs that are certified for use depending on your chassis, which relates to power consumption with the GPUs.

Start by removing the shroud cover:

The first problem we have to fix is the airflow issue - the factory setup for an R730 uses heat sinks that capture all of the air under the shroud and will restrict the amount of fresh air going to the GPUs.:

If you ordered for K1 cards from Dell, you should also have received an "upgrade kit" consisting of a new low-profile heat sink and a 6-pin PCIe power cable. Loosen the four screws on the CPU heat sink (I'm installing two cards here and I have two CPUs, so I'll be replacing both), wiggle them a bit to get them to detach, and carefully clean all of the grey heat sink compound off the CPU:

Make sure you also replace any empty DIMM slots with blanks. Install your new low-profile heat sinks, replace the shroud, and you'll have an unrestricted path for airflow back to the PCIe risers (I removed a fan here):

Before you put in the card(s), connect the 6-pin power cable to the appropriate slot on the riser. If you are installing two cards, connect each to their respective riser - do not try to use one cable/riser to power both, even though they do have dual connectors on the end. Route the power cables through the notch in the shroud (marked in red below). Now remove your expansion slot blanks and carefully drop in your GRID cards. They should just barely clear any obstructions to get in there - it's a tight fit.

Install your second card if applicable and then make sure you press the buttons marked in blue to deploy the supports for the card(s):

If you already have 1100W power supplies equipped, you are all done. If you don't, you'll want to upgrade before the systems go in to production. I've used the K1 cards with 750W, but the system will complain about power usage and may not have enough juice at peak consumption or during a failure of one PSU.

Now just power on your server and follow the appropriate guide from NVIDIA for installing drivers: http://www.nvidia.com/object/virtual-gpus.html. Good luck!

SCEP Policy Update Troubleshooting

2015-10-23T11:42:00.001-05:00

Because I'm a glutton for punishment, I recently started rolling out System Center Configuration Manager 2012 R2 SP1 and System Center Endpoint Protection across our VDI environment. There are always some considerations to be made in a pooled desktop / gold image type environment when loading software that uniquely identifies devices, but lucky for me SCCM/SCEP handled this just fine without any tweaking. However, there were some nuances to how SCEP policies are applied that caused some serious hair-pulling before I spotted the issues.

Antimalware Policy Basics

I should clarify I few points to ensure your policies even stand a chance of being applied in the first place:

Default policies will apply if you have not created any custom policies.
Custom policies are created under Assets and Compliance > Endpoint Protection > Antimalware Policies.
Custom policy settings will always override default policies (unless you have one of the issues described in the next sections).
Policies must be applied to a collection containing the device(s) you want to apply it to before they do anything.
In SCCM 2012 R2 SP1, policies are now cumulative and will do an automatic "client side merge" by default, obeying the order of precedence you specify in the Antimalware Policies section. This also means if two different policies apply to a workstation based on collection membership and each policy has unique exclusions, all exclusions will be added to the client. In previous versions of SCCM, this was not the case (only one policy was applied to each client and a manual server-side merge was required to combine multiple policies).
Refresh machine/computer policy from the client to force it to process new policy changes. For faster policy adjustments, Adjust the Client policy polling interval in the applied Client Settings to increase the refresh interval.

So let's assume you have some custom policies, applied to a collection, and machines in that collection aren't getting the policy:

Verify Policy is Actually Applied

The first thing to do is check and make sure your device has actually decided to acquire the policy. The easiest way to do this is to open SCEP on the device and check Help > About. You should see all the custom and default policies here:

If you aren't seeing policies here, then there is a problem between SCEP and SCCM. Your client is unaware, either due to an error or misconfiguration, that it should even try to apply any policies.

In SCCM, select the device from the Assets and Compliance screen and move to the Antimalware Policies tab. Check the Policy Application State - it should show your policies there and they should show "Succeeded" with a recent update date/time.
If the policy isn't there, then you probably didn't deploy it correctly in SCCM.
If the policy appears, but doesn't show anything for the application state, then the client hasn't yet received it. Force the device to update machine policy and wait a few minutes.
Check the CcmExec.log and EndpointProtectionAgent.log files under c:\windows\CCM\Logs for errors.

Assuming that all looks good ...

Verify Policies Are Applied in the Right Order

If you some of your custom policies are applying, but not the ones you expect, check and ensure that they are applying in the correct order. You can check which policies are being obeyed by querying the following registry key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\CCM\EPAgent\LastAppliedPolicy.

Policies listed there with higher values take precedence over settings with lower values, as I understand it. "Exclusion" policies will have the same value usually, indicating they will be merged together.

If that still doesn't help ...

Verify Local GPOs Can Be Applied

If you find that the policies are appearing the SCEP client correctly under "About", but you aren't actually getting the exclusions and settings you expect, the issue might be with local GPO processing. SCCM applies policies by using the client to inject registry settings via the local GPO, which SCEP then reads. If those policies are received by SCCM, but never make it in to the registry, then you'll get this behavior.

Run RSoP and check under Computer Configuration\Administrative Templates\Extra Registry Settings - You should see all of your SCEP policy settings here.

If they don't show up (or you don't have the "Extra Registry Settings" area), check the RSoP report for a setting called Turn off Local Group Policy objects processing under Computer Configuration/Administrative Templates/System/Group Policy. If it is applied, RSoP will tell you you which GPO is enforcing it. Change it to Disabled/Not Configured, because this setting will effectively prevent the local GPO that SCCM has injected with your SCEP settings from actually applying:

Setting Path:

Computer Configuration/Administrative Templates/System/Group Policy

Setting: Turn off Local Group Policy objects processing

And finally, if you still get inconsistent policy application ...

Verify GPO Registry Settings Can Be Refreshed After Startup

There is another GPO setting that can trip you up, and that is the Registry Policy Processing option called Do not apply during periodic background processing. This prevents registry changes requested via GPO updates from being applied during the periodic background processing of GPOs, effectively only allowing them to be applied when the computer configuration policy is applied during boot. Having this option on will stop SCEP from receiving updated registry settings from the local GPO after it boots. If you find that you are required to reboot to get policy updates, just check RSoP for this setting and if you find it, remove it:

Setting Path:

Computer Configuration/Administrative Templates/System/Group Policy

Setting: Registry Policy Processing

Option: Do not apply during periodic background processing

Hopefully these tips will save you some of the headache of getting SCEP policies reliably deployed!

Outlook Credential Prompt When Opening Exchange 2013 Public Folder

2015-07-17T17:00:00.000-05:00

After completing an Exchange 2007 > 2013 migration recently, I was left with one issue that was preventing us from stamping the project as a roaring success and moving on:

Outlook 2013 users were sometimes receiving a single pop-up prompt for credentials whenever they opened the Public Folder (we have only one). One. Single. Prompt.

Google was frustratingly unhelpful because searching for "outlook prompts for username and password when opening public folders" or something similar just resulted in a lot of folks who were always getting a pop-up that wouldn't go away. It was usually caused by an authentication failure of some sort.

However, we were in a different boat - Users got the prompt once when they first launched Outlook and opened their public folders, but after entering it they could continue - authentication worked. Next time they logged in to their PC, it would happen again. Not a show stopper, but it definitely generated its share of support calls.

After I finally stopped "troubleblasting" to think about it a bit more critically, the key pieces of this fell in to place:

A credential prompt means that either NTLM failed and/or Basic Authentication is being used (and your password hasn't been previously saved, or isn't permitted to be).
Outlook (and Office in general) were not 100% up-to-date with the latest non-critical/non-security patches due incompatibilities with the "old" environment.
We use a non-persistent desktop image in our VDI environment - Credential Manager doesn't preserve saved passwords between sessions.

A closer inspection of the Outlook Connection Status (available from a Ctrl+Right-Click menu on the Outlook tray icon) showed that we were indeed hitting the Public Folders with basic authentication instead of NTLM:

Because this was all internal communication, Outlook should have respected our selected internal authentication method for Outlook Anywhere: NTLM. It seemed to be ignoring it, but in actuality, it was incorrectly looking at our external authentication method: Basic.

Note: Running Get-OutlookAnywhere | fl server,*external*,*internal* should show your internal and external settings for all servers - essentially "NTLM" means integrated Windows authentication on a domain-joined PC, "Basic" means we just prompt the user with a standard pop-up for their password.

Apparently this is (or was) a bug in Outlook 2007, 2010, and 2013 (See KB2839517) where the client would correctly look at the first EXHTTP block during the autodiscover response (the internal setting) when connecting to a primary mailbox, but would then arbitrarily skip to the second EXHTTP block (the external setting) for any alternate mailboxes. Since we had migrated our public folder to Exchange 2013, it was now technically a shared (alternate) mailbox. Since we hadn't patched Outlook, the bug impacted us and thus basic authentication was used!

The final slice of the mystery (and the reason most environments would never be bothered by this) was that our desktop images in the VDI environment did not persist between login sessions - as soon as a user logs off, the machine is shut down and the image is reprovisioned during the next boot-up. User data is preserved (Documents, desktop items, etc) but not the Credential Manager store, which is where your basic authentication username/password is kept when you hit the "Remember password" option. For a normal environment, the user would have just clicked that button and never seen the prompt again, but our users were prompted every day after initial log in.

There were three possible fixes:

Patch Outlook to ensure it is correctly looking at the internal setting.
Configure our gold desktop image to preserve the credential store between sessions
Adjust the external authentication method to NTLM or Negotiate

Although all three would eventually happen, I went with the third option initially since it was the easiest to accomplish and didn't require an update to our desktop image. A single line of Powershell got us up and running:

Get-OutlookAnywhere | Set-OutlookAnywhere -ExternalClientAuthenticationMethod Negotiate

I could have also set it all to NTLM, but I wanted to ensure the highest compatibility with non-domain joined external computers running Outlook. An 'iisreset' later and we were in business - no more prompt and the Outlook Connection Status dialog showed NTLM across the board.

Incidentally, this also solved a problem with Lync 2013 throwing a credential prompt during login as well - I suspect the issue was similar and directly related, but I'm not completely sure why Lync was suffering the same problem.

Something else blew up before I could think on it more.

Running vCenter 5.x with SQL 2012 AlwaysOn Availability Groups

2015-04-08T23:01:00.000-05:00

After proudly starting the listener on our shiny new SQL 2012 AlwaysOn cluster, I was very eager to get vCenter moved off the brave little single-point-of-failure that is our current SQL server (a 2008 VM sitting in the virtual environment itself). I had done some research ahead of time and thought that AlwaysOn was at least sort-of supported by VMWare for protecting vCenter workloads. However, in my haste to play with a fancy new toy, I must have missed the plethora of blog posts indicating that either a) It's not actually supported at all, or b) Only Failover Clustering (shared storage) - not Availability Groups (non-shared storage) - are supported. And if you are about to do what many have done on the forums and suggest KB1037959 as evidence that they ARE supported, think again - that article is referencing support for running various clustered workloads on vSphere, not running your vCenter DB on clustered systems. Outside of a vague mention of AlwaysOn as a possible third party clustering solution to replace vCenter Heartbeat (e.g. "Best effort support"), I haven't been able to find anything official one way or the other.

But the AlwaysOn cluster was ready to go and if no one is going to tell me explicitly that I can't do it - well, that's basically an open invitation.

Here's the setup I had to work with:

ServerA - Win2012R2 - vCenter Server 5.1 U3 with SSO and Update Manager
ServerB - Win2008R2 - SQL 2008 Enterprise
DB1 - Win2012R2 - SQL 2012 (Primary)
DB2 - Win2012R2 - SQL 2012 (Secondary)

Some basic prerequisites to have in place:

AlwaysOn Availability Group setup
SQL Server in Mixed Mode
Documentation on the SQL passwords for your SSO account (rsa_user), vCenter user, and SSO master password. As long as you know the master you can reset the rsa_user and vCenter User.
Download and install the SQL 2012 Native Client on your vCenter server.

The steps I followed are relatively straightforward if you are comfortable working in SQL Management Studio. Most of these are detailed in KB7960893, but I found a few extra steps were necessary. Obviously your mileage may vary - follow along at your environment's own risk:

Back Up Databases

Stop all VMWare services on ServerA including SSO if used.
Perform a backup of all VMWare databases on ServerB using SQL Management Studio. There are usually three - RSA (Single Sign On), upmgr (Update Manager), and VCDB (vCenter).

Restore Databases and Logins

The database restore is pretty straightforward, except that we will need to recreate the logins that will be used for vCenter ODBC connections and SSO and attach them to their orphaned users in the restored database.

Restore the databases to DB1 - your primary SQL AG member.
Referencing ServerB, recreate the SQL login for your vCenter user. Map the user to the default msdb database with db_owner permissions.
On the vCenter database, associate the orphaned vCenter user with the login you just created using the command: ALTER USER username WITH LOGIN = username
Replace 'username' with your vCenter DB username.
Referencing ServerB, recereate the SQL login for the SSO users (rsa_user and rsa_admin or rsa_dba). Do not map the user to any database.
On the SSO database (RSA), associate the orphaned users with the logins you just created using the command from step 5.
Repeat the steps to create the users on the secondary SQL server, but don't bother trying to associate the orphaned accounts - secondary servers are read-only.

Add Availability Database

Next, add the three databases as Availability Databases. They will need to be in full recovery mode - if you have to switch them over, you'll also need to make a full backup before adding them to your Availability Group.

Restore the SQL Agent Rollup Jobs

I'll just reference KB1004382 here, which gives a good step-by-step walk through of how to get this done. There are some sql query files located on the vCenter server that create these jobs which perform rollups and clean the database. These queries are normally run automatically when vCenter is installed, but since we are just moving the DB, we will have to recreate several jobs manually. The important thing to note is the first step which explicitly states that you must be connected as the vCenter user in SQL Management Studio when running the queries. This way the jobs are created with the vCenter user as the owner. You'll also need to have the SQL Agent service running for these to do any good. Don't repeat this step on the secondary SQL server (see my final note below).

Update the ODBC Connections & vcdb.properties File

Back on ServerA, remove and recreate your ODBC connections using the new Availability Group listener information. Make it easy on yourself and use the same name for the ODBC connection. Be sure to create the new connections using the right version of SQL Native Client for SQL 2012 (11.0).

Next, modify the vcdb.properties file located at C:\ProgramData\VMware\VMware VirtualCenter for Server 2008/2012.

Update SSO Settings

Assuming the host name and/or port changed for SSO also (the RSA database), you'll need to follow KB2033516 to update the settings.

Completion

Start your services back up (starting with SSO) and you should be all set!

One final note: The orphaned accounts on the secondary server and lack of any rollup jobs there mean that achieving an unattended failover may be unlikely. I'm not familiar enough with AGs yet to be able to say whether the jobs can be safely created on the secondary and allowed to sit and accomplish nothing while it is inactive, or if the orphaned accounts will actually cause any issues, but I'm erring on the side of caution and assuming that a little manual intervention may be necessary.

Post in the comments if you have a suggestion/correction!

UPDATE: Our Veeam backups started to fail after this change. Turns out Veeam can't backup the vCenter database and will cause VSS errors (VSSControl: Failed to freeze guest, wait timeout). Typically Veeam detects the DB automatically and doesn't include it in the backup, but this wasn't working for us. You'll need to create a manual DB exclusion to get the backups working again. Instructions can be found here: http://www.veeam.com/kb1051

Pay Me Now, or Pay Me Later: DNS Edition

2014-08-01T08:23:00.000-05:00

I hate most low-cost hosting providers. I've rarely have a great experience with one, but Web and DNS hosting providers are usually already established when I start working with a client, and migrating to another provider is typically not marked as a high priority project. I might change my mind on that after working with a client who used iPower and suddenly lost all public DNS resolution one day.

We called up iPower twice and spoke to two oblivious techs who spouted off nonsense and promised to call back after looking in to it further. Finally, on the third call a person finally told us why the client's SOA record wasn’t propagating (causing their domain name not to resolve anywhere, so no website, no e-mail, etc. for two days). Turns out they had failed to respond to their “domain verification” e-mail, which with most providers means nothing – they are just required by ICANN to prompt you to update it once a year, but most just don’t do anything if you ignore the e-mail. iPower took it a step further and deactivated their domain when there was no response.

Paul: So, if I understand you correctly, you all sent an e-mail to the registrant e-mail address to verify the registrant details. The e-mail address was wrong, so they didn't see it. So the domain was deactivated?
Paul: So you checked to see if you had the right e-mail address by ... sending an e-mail? And if the address was not correct, you just assumed we didn't need the domain?
Deepika B: Yes, you are right.
Paul: Fantastic, well done.
Deepika B: Thank you!

Bravo, iPower. Bravo.

Enabling Office 365 Message Encryption

2014-01-15T21:35:00.002-06:00

Back in November of 2013, Microsoft announced Office 365 Message Encryption (OME) as a way to protect the contents of outbound messages to people in other organizations. This excellent update helps fence in it's predecessor, Exchange Hosted Encryption (EHE), to the Office 365 branding and management. Users of EHE had to work through more cumbersome configuration steps and were required to purchase it open volume licensing, whereas OME comes included as part of Azure Active Directory Rights Management (AADRM) in the E3 and E4 packages, or purchased as an add-on to other enterprise SKUs with the normal subscription licensing.

Unfortunately, the scheduled release date is a broad "Q1 2014" and there are still some small missing details, like "How do I friggin' enable this stuff?". Search results for OME are flooded with news articles announcing the service and irrelevant posts about EHE, and the only TechNet articles are for EHE customers who are currently being migrated to OME. However, it is possible to enable the service if you already have access to AADRM and are prepared to do a little PowerShell work.

Enable Rights Management in the Portal

The very first thing we need to do is activate your Azure Rights Management functionality.

Log in to https://portal.microsoftonline.com (you'll need to be a Global Admin, obviously).
Go to service settings on the left-hand menu
Click on rights management on the top menu
Click the manage link (Direct Link)
Click the big activate button (it's hard to miss).

Wait a minute or two and it should show a nice green check mark indicating "Rights management is activated":

Activate Internal Licensing for Information Rights Management (IRM)

If you are like me, you probably thought that previous step would be sufficient. Maybe it will be eventually, but as of January 13th it takes a bit more than that to get going. We need to whip out PowerShell and do a little tinkering with IRM first. If you haven't used PowerShell to manage Office 365 before, you'll first want to download the Sign In Assistant and then head over to get the latest Azure AD Module for Powershell x64 (the 32-bit version is available here). Then just fire up PowerShell as an administrator and run the following commands:

import-module AADRM

connect-aadrmservice

Next, we need to check and make sure AADRM was enabled properly by running the get-aadrmconfiguration command and checking for "FunctionalState: Enabled". Assuming that looks good, then we can continue and connect to Exchange Online:

import-module msonline

$cred=get-credential

$session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.outlook.com/powershell/ -Credential $Cred -Authentication Basic -AllowRedirection

Import-PSSession $session

Doing great! Now let's enable customization of our org, set the RMS key sharing location, import the RMS domain, and turn up our licensing. We are going to use the North America URL in the next step, but here are the other regions if you need them:

North America - https://sp-rms.na.aadrm.com/TenantManagement/ServicePartner.svc
EMEA - https://sp-rms.eu.aadrm.com/TenantManagement/ServicePartner.svc
Asia Pacific - https://sp-rms.ap.aadrm.com/TenantManagement/ServicePartner.svc

Enable-OrganizationCustomization

Set-IRMConfiguration -RMSOnlineKeySharingLocation "https://sp-rms.na.aadrm.com/TenantManagement/ServicePartner.svc"

Import-RMSTrustedPublishingDomain -RMSOnline -name "RMS Online"

Set-IRMConfiguration -InternalLicensingEnabled $true

Now run the Get-IRMConfiguration command and check the output.You should get back something like the following:

You can test your IRM configuration with the following command now (replacing test@tenant.onmicrosoft.com with your own admin account):

Test-IRMConfiguration -sender test@tenant.onmicrosoft.com

Assuming you get back an "OVERALL RESULT" of "PASS" from that command, then you are good to move on to the next step. However, It may take 12-24 hours for the licensing to start working in Exchange admin center.

Add OME Rules in the Exchange Admin Center (EAC)

NOW we are ready to create some encryption rules! This can be done in PowerShell, but it is much easier from EAC. OME works based on rulesets that an admin creates and you can specify what criteria you want to use for messages. Or you can simply encrypt everything leaving the organization. Make sure you take a look at Microsoft's TechNet article on the user experience before you do this though - you could very well wind up with users at your door with pitchforks otherwise.

We typically configure the rule to only fire when sending external mail where the word "secure" is in the subject line, so that's how I'll demonstrate this step. You may prefer a different trigger, and there's lots to choose from.

Log back in to the Office 365 portal (https://portal.microsoftonline.com)
Click the Admin button in the top-right next to your name and select Exchange from the drop-down. If you are accessing a tenant as a delegated admin (e.g. Microsoft Partner), then you'll need to navigate to the EAC by hitting service settings on the left-hand menu and selecting Manage additional settings in the Exchange admin center.
Once in the EAC, click mail flow on the left-hand menu and you'll be taken to the rules page by default.

Now we need to create our OME rules. Our first rule will encrypt outbound messages with the 'secure' keyword in the subject:

Click the 'plus' sign on the rules page.
Select Create new rule and give it a descriptive name (e.g. "Outbound Encryption")
Set the rule to apply if the sender is located inside the organization, the recipient is located outside the organization, and the subject includes 'secure' (or whatever keyword you wish to use)
Set it to perform the Encrypt the message with Office 365 Message Encryption action ("Apply Office 365 Message Encryption" under "Modify the message security")
Enable auditing if desired and save the rule

Encryption should be working now! Pay special attention to the order of your rules (if you have others) and make sure your encryption rule won't be cancelled out by anything else. I'd also recommend creating a second rule to decrypt inbound messages that are sent as replies. This will keep your users happy since they won't have to leave Outlook to read encrypted message replies from external users:

Click the 'plus' sign on the rules page.
Select Create new rule and give it a descriptive name (e.g. "Inbound Decryption")
Set the rule to apply if the sender is located outside the organization and the recipient is located inside the organization. Technically this will apply the rule to all inbound mail, but it won't do anything unless that e-mail was encrypted anyways.
Set it to perform the Remove Office 365 Message Encryption from the message action ("Remove Office 365 Message Encryption" under "Modify the message security")
Enable auditing if desired and save the rule

Enjoy your Office 365 Message Encryption!

That's all folks, you can now send out secure messages with Office 365 just by adding the keyword to your e-mail. It should work with any mail client, including mobile phones and OWA. I'm open to suggestions on better ways to write the rules and I'll also note here that we did have a few issues where some of our users are getting NDRs specifying they are not licensed (which appears to be untrue). We've opened a Microsoft ticket and I'll update this post with the results of that case. (UPDATE: The 550-5.7.1 error we were receiving indicating encryption was disabled on the transport server has disappeared on it's own this week. I'm guessing we jumped the gun on deployment)

Feedback welcome! Special thanks from across the pond to ahandyblog for the write up on IRM that helped me fill in the gap there.

SharePoint Downloads Interrupted for Large Files

2013-06-19T12:31:00.000-05:00

While we primarily use SharePoint Online (2013) for fairly small documents - the largest are still less than 5MB - I recently decided to start uploaded recorded team meetings from Lync 2013. The videos are about 30 minutes in length and end up being around 40-60MB in size. Although the upload runs fine, users were reporting issues when trying to stream the files or attempting to download it. The stream would simply stop and the downloads fail with a message that the download was "Interrupted". We replicated this behavior across 4 different locations with the same result - at some random point in the download, it would fail.

The only location where it seemed to work just fine was at our datacenter colo, which enjoys a snappy 1Gbps connection. The slower the internet connection at our local office though, the more likely the download was the get interrupted.

While Microsoft continually referred to this as "network issues" on our end, that just wasn't the case. From any other site (even SkyDrive), large downloads work flawlessly, and these downloads from SharePoint consistently fail.

The workaround was to disable TCP Auto Tuning on our machines by opening an administrator command prompt and entering the following:

netsh int tcp set global autotuninglevel=disabled

Auto Tuning is a feature that scales the TCP window dynamically. You can read more at this KB article: http://support.microsoft.com/kb/947239

The takeaway from that article is the following paragraph:

"If you enable Receive Window Auto-Tuning for WinHTTP traffic, data transfers over the network may be more efficient. However, in some cases you might experience slower data transfers or loss of connectivity if your network uses an older router and firewall that does not support this feature. For example, when you use Windows Internet Explorer to access applications that are hosted in Microsoft Office SharePoint Server, the HTTP traffic may slow down. This occurs because certain routers do not support the Receive Window Auto-Tuning feature."

Hopefully this is something that will be addressed in a future release/update of SharePoint Online, but as of 6/19/2013 it is still an issue for us.

Repairing Mailbox Corruption in Exchange 2010

2013-05-13T22:03:00.005-05:00

I recently got through recovering an SBS 2011 server after Active Directory face-planted in the middle of a workday. When I say recover, I mean I repeated the entire migration, using a cleaned up secondary DC - it was a fun weekend (expect another post about that experience). Although I thought we were in the clear, I got a call from the client about 24 hours after we had verified everything was working. He indicated that his iPhone had suddenly stopped receiving mail in the inbox (calendar, contacts, sent items were still fine) and throws up an error after spinning in circles for a few minutes that it "cannot connect to mail server".

I suspected leftover corruption, but we had already run through the normal database repair tools for Exchange and everything else was working fine. I checked the event logs and found:

Event ID 1008

An exception occurred and was handled by Exchange ActiveSync. This may have been caused by an outdated or corrupted Exchange ActiveSync device partnership. This can occur if a user tries to modify the same item from multiple computers. If this is the case, Exchange ActiveSync will re-create the partnership with the device. Items will be updated at the next synchronization.

Exception message: ICS synchronization failed.

There is a string that appears in that error which ties it to a specific user and ActiveSync device. I ran through the normal troubleshooting, but I had the same issue on a different phone.
The first solution I tried is pretty standard for repairing a corrupt mailbox: Just move it to another database in Exchange. If there are corrupt messages, folders, calendar items, etc. these will be logged and you can specify a threshold for how many of these failures the move should tolerate before giving up (see bad item limits: http://www.petri.co.il/baditemlimit-override-exchange-2010.htm)

However, that move presented a new problem:

Event ID: 1100

Request '(request name removed)' failed.

Error code: -2147221233

MapiExceptionNotFound: Unable to synchronize manifest. (hr=0x8004010f, ec=-2147221233)

Awesome. This mailbox is too corrupt to even move. I thought the next step would likely be an export-to-pst and import back to a clean mailbox (which causes it's own issues), but I found a friend in the new-mailboxrepairrequest cmdlet:

New-MailboxRepairRequest -Mailbox user@domain.com -CorruptionType SearchFolder, AggregateCounts, ProvisionedFolder, FolderView

After running that command, my user immediately was able to access his inbox and the event log errors disappeared. The command will disrupt access to the mailbox temporarily and you won't be able to see the progress. Instead, watch the application event logs for event ID 10048:

Online integrity check for request completed successfully

The event logs will also show you what was corrupted (it was several views in my case).

Failure of vShield Edge NAT/VPN Traffic Post-5.1 Upgrade

2013-02-25T13:57:00.001-06:00

UPDATE: Turns out this is a known issue during the 1.5 > 5.1 VSM upgrade and a fix should be released in an upcoming patch.

That's about the shortest title I could think of to be descriptive of this issue. TLDR is that NAT rules on vShield Edge appliances appear to be causing unexpected behavior on VPN traffic after a vCloud upgrade from 1.5 to 5.1.

Background: We recently upgraded from 1.5 to 5.1. For most of our vDCs, we simply have a single vSE/Routed network that connects a private subnet to a "WAN" network and pulls a public IP from a pool. We forward (NAT) and allow (firewall) selected ports (e.g. 3389 for RDP) to virtual machines. Most of these networks also have a site-to-site VPN tunnel with a physical firewall across the internet. After the upgrade, we went and converted our rules to match on original IP and then enabled "multiple interfaces" - effectively taking them out of compatibility mode. Everything looked good (even for the vSE devices still in compatibility mode)

Issue: We first noticed this when a client reported that they could not access a virtual machine via RDP using it's internal (VSE protected) IP across a VPN tunnel, but could access the VM via RDP using it's public hostname/IP address. We allow all traffic across the VPN (firewall has an any:any rule for VPN traffic). When we logged in to troubleshoot (simply thinking the VPN was down), we found that we could connect to any port on the remote VM across the VPN tunnel except 3389. I could ping from the local subnet to the troubled VM on the vApp network with no problem. I could connect to other ports that were open on the remote VM with no problem. I could not connect to 3389 across the VPN.

We thought it might be isolated, but found the issue on every VSE we have: If there existed a DNAT rule to translate inbound traffic for a particular port, that port would be unresponsive when traffic traversed the VPN tunnel destined for the target of the DNAT rule.

While vCloud Director doesn't show anything strange in the firewall section of vSE configuration, if you log in to vShield Manager and look at the firewall rules there, a "Deny" rule with the private/internal/translated IP is added for any NAT rule that exists:

This, I'm assuming, is for security reasons during the upgrade but it does not show up in vCloud Director (thus our confusion). After taking our appliances out of compatibility mode post-upgrade, the rules were still there.

Solution: After the vSE is out of compatibility mode (see pg. 49 of the vCD 5.1 Install Guide), re-apply the service configuration (Right-Click vShield Edge Appliance in vCloud Director and select "Re-Apply Service Configuration"). You can also re-deploy the appliance or add an arbitrary rule to the firewall list - both appear to have the same effect.

Weathering the Cloud: Moving your service company ahead in the age of SaaS

2011-10-22T14:38:00.000-05:00

With phrases like "cloud computing" making their way deeper in to the everyday dialect of non-technical business managers, it no longer comes as a surprise when decision makers approach me about cutting costs by using "the cloud". They may not know exactly (or even vaguely) what that means, but they know it might save them money, so the door is wide open for a discussion about _aaS ("Something" as a Service - Software, Infrastructure, Security, and so on).

This would appear to be a huge sales opportunity for I.T. service companies, but my observations of several have shown that adoption can be slow for a number of reasons:

Lower Perceived Profits - With clients spreading the TCO of software/hardware over the course of their commitment, as opposed to making large upfront purchases, the profit from a single sale takes a long time to realize for a service company - not always an acceptable situation for a business just getting in to reselling the _aaS space. Selling a client a $10,000 Exchange 2010 server plus labor vs a measly $200 a month for an easy-to-deploy hosted solution seems counter-intuitive at first glance.
Resistance to Change - Change is not always welcomed, even by the I.T. veterans who typically operate small service companies. Old-school suspicions about the security, reliability, and longevity of hosted services are deeply ingrained in many people. Others feel threatened by the technical and financial aspects of operating a cloud-focused business, which are a drastic departure from the comfortable install-break-fix model many companies have been built on, and which also require a very different skill set to maintain.
Analysis Paralysis - A rapid explosion of cloud systems, vendors, and services have resulted in a huge selection of products to sift through. Researching, finding, and implementing the services that will be most profitable is a daunting task that can leave your decision making skills comatose.
Limited Investment Funds - The cloud is all about moving CAPEX to OPEX for the end-user, but someone has to make the initial investment to build it. For custom IaaS solutions, this expenditure could fall on the service provider. After all, the cloud has to physically exist somewhere, and someone will have to pay for the physical hardware that powers the virtual.

With these challenges, how does an I.T. company that thrives on the instant capital of hardware, software, and labor sales become a lean, agile provider of pay-as-you-go (and pay-as-you-grow) cloud services? Here's my two cents on the subject:

Take stock of your employees - The entire company has to embrace the cloud and be excited about its potential at every level, from sales to operations to management. Bring in fresh faces who are passionate and eager to work with cutting edge virtualization solutions. Make sure your financial officers understand the model and are on board with changing your income generators from large one-time projects to recurring monthly revenues.
Start with reseller opportunities - Jumping head first in to building your own private cloud could spell disaster. Work with existing providers who have strong channel partner programs and can provide the infrastructure (and the some of the liability) for your product.
Use what you sell - The world of cloud computing touches on so many aspects of technology, it can be daunting to understand. The best way to learn is to use the service you plan on selling within your own firm first. Not only is it good for your staff to become familiar with it, but it is a great way to reassure clients that your cloud product is mature and reliable.
Streamline Operations - Providing product as a service and at a distributed monthly rate means your company must be laser-focused on efficiency. When your clients were simply billed directly for time and material, you were passing the cost of inefficiency along to them. Now that they pay you a flat-rate each month for a particular service, every dollar you spend maintaining that SLA comes straight out of the bottom line. The more servers you can maintain or accounts you can manage without adding staff will keep you in the green and allow you to stay competitively priced.
Understand your clients - Not everyone is a good fit for the cloud computing model, but many are and just don't realize it yet. Learn to recognize the pain points in your customers that can be relieved with a move to cloud services. Then when you here someone say "Our e-mail just isn't reliable enough" or "We are spending too much on server maintenance", your inner sales person will scream out to sell a cloud product.

But most importantly be excited. This should be easy: There is so much potential and rapid growth in this area, it's hard not to indulge your inner geek in the possibilities this model offers.

BPOS Offline .... again

2011-05-12T12:09:00.000-05:00

*** UPDATE 12:12PM CST ***

From Microsoft Health Dashboard:

The BPOS Operations team is working to resolve service degradation for Exchange Online mail flow for organizations served from this region. Users in affected organizations will experiencing ~40 minute delays when trying to send or receive e-mail using Outlook, OWA, or mobile devices. The BPOS Operations team is actively working to restore service. Next update will be within one hour or when new information is available.

Not sure about anyone else, but ~40 minutes is off by about infinity for me.

----------------------------------------------

Two days in a row now, and if you were watching Twitter it would appear to be affecting a lot of people in the NOAM region. Oddly, the Health dashboard still doesn't show a problem:

Try calling MS support though, it will just hang up on you.

Microsoft Exchange Online Outage

2011-05-10T13:35:00.005-05:00

**2:20PM CST UPDATE**: Microsoft posted another update to the NOAM Health board:

The BPOS Operations team continues to investigate service degradation issues with Exchange Online mail flow for organizations served from this region. The next service update will be provided within 2 hours if the issue is not resolved.

Hopefully we here something else soon.

------------------------------------------------------------------

Microsoft Exchange Online (and BPOS by extension) are currently having issues with mail flow (read: there isn't any).

From Microsoft as of 11:40am:

The BPOS Operations team is investigating alerts indicating service degradation for Exchange Online mail flow for organizations served from this region. Users in affected organizations may be experiencing delays when trying to send or receive e-mail using Outlook, OWA, or mobile devices. The BPOS Operations team is actively working to determine the root cause and restore service.

When I called in initially, I could hear the service manager yelling in the background as the call queue exploded from a few users to over 60 in a matter of seconds.

Who else is seeing this issue? What have been your impressions of BPOS recently? Anyone on Office 360 having the same problem?

Shutting down an SMTP AUTH Relay attack

2011-02-22T18:52:00.001-06:00

We recently had a client whose Exchange Server (configured by another I.T. company, mind you) was continously being blacklisted with no immediately identifiable cause. Our first responders checked the normal stuff and verified that the server was not an open relay in any obvious way. At the time I jumped in on the support ticket, the server did not have the proper logging enabled, so we couldn't see exactly what was occuring.

The first step was to determine if anyone was actively spamming. There are a couple logs and tools that are useful for checking current Exchange connections, but we ran Exchange User Monitor overnight on the assumption that it was a client machine sending mail via the Exchange server. However, we didn't spot any MAPI or OWA connections being made in the off-hours, so we moved on to enabling some additional logging in Windows.

First, we enabled SMTP logging in IIS to give us a bit more information as to what connections were being made. Our logs (truncated with x.x.x.x to preserve privacy) showed the following:

#Fields: date time c-ip cs-username s-ip cs-method cs-host cs(User-Agent)
2010-11-18 00:00:34 x.x.x.x localhost 192.168.15.250 EHLO - -
2010-11-18 00:00:35 x.x.x.x localhost 192.168.15.250 MAIL - -
2010-11-18 00:00:35 x.x.x.x localhost 192.168.15.250 RCPT - -
2010-11-18 00:00:37 x.x.x.x localhost 192.168.15.250 DATA - -
2010-11-18 00:00:37 x.x.x.x localhost 192.168.15.250 QUIT - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionResponse - - - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionCommand - EHLO - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionResponse - - - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionCommand - MAIL - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionResponse - - - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionCommand - RCPT - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionResponse - - - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionCommand - DATA - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionResponse - - - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionResponse - - - -
2010-11-18 00:00:44 x.x.x.x OutboundConnectionCommand - QUIT - -

Thousands of these lines logged while the client's business was closed (11:00pm to 3:00am) indicated not only massive amounts of spam being sent out, but that the attacker was external and was authenticating to the network. The first 5 lines show an EHLO connection, slightly different than the HELO in that it supports authentication, coming from an external source (which we subsequently traced to the UK and notified the business owner). The next set of lines is a spam message being sent from the local Exchange server out to the next victim.

While we had initially checked and made sure the server was not an open relay, allowing anyone to send to external domains without authenticating, most servers still allow authenticated relaying by default. This means that if you can authenticate by SMTP to the Exchange server with a valid username/password, you can relay to external domains. Not a big deal, as long as none of your users have a compromised account or weak passwords.

In order to spot which account was compromised, we turned to the Windows Event Logs. These showed a series of 1708 informational messages in the Application log as well as Success Audit messages in the Security log that pointed to an account that had been created called "test". I won't go in to why this is a bad idea for a username, but the password we found was even worse...

My colleague, Jon Jeffels (http://blog.jeffels.net/), did a little extra work and ran THC-HYDRA to crack the user account via SMTP authentication, probably the same way it was broken to begin with. Within a few seconds (with transforms applied) we had a password of "test1" to go with our username of "test" ... not very secure, folks!

Preventing SMTP AUTH Relay Attacks

There were a myriad of things that could have prevented this problem:

The best solution, in my opinion, would be to use an external mail filter of some sort. There are a lot of other reasons for this, and hosted solutions are too cheap to have any excuse for not using one. Use Smart Host settings in combination with this (and secure inbound connectors while you are at it).
Simply enforcing password security would have helped this temp user account stay secure
Or how about just not forgetting to delete TEMPORARY accounts?
Setting something besides the default lockout GPO of 50 attempts and 10 minute lockout would have been nice also. Perhaps 3 attempts and a 30 minute lockout?
Disable external relaying all together. Unless you have external users utilizing Exchange via POP3 (*shudder*), you shouldn't really need it. Check out instructions here for Exchange 2000/2003: http://www.amset.info/exchange/smtp-relaysecure.asp

While we have a ton of fun catching and shutting down attacks, the "real" attack was the password compromise that probably had occured weeks earlier. This was a totally preventable problem that was probably caused by a tech creating a "test" account, so shame on all of you I.T. companies out there who are using weak passwords and should know better!

Attack of the Local Failures

2010-11-20T11:38:00.000-06:00

You read that correctly. It is a 82 GB mailbox filled with Local Failure messages. Jon Jeffels spotted this when we couldn't figure out why his Outlook client kept crashing and hanging. Read his full post here:

http://blog.jeffels.net/2010/10/outlook-issues/#more-61

Trend Micro WFBS UPX Packed Updates

2010-11-15T20:04:00.000-06:00

I had a client with a myriad of issues related to some infected machines on the network. While digging through the firewall logs (SonicWALL TZ-170 or 190 I believe), I found this entry:

Typically we wouldn't want to see packed executables attempting to be retrieved from the WAN, but these occured at regular 15 minute intervals and the source IP was owned by Trend Micro. Digging in the WFBS console I found:

A whole series of failed updates matching up to the UPX packed executables being blocked in the SonicWALL.

Anyone have experience with UPX compression? Is this standard practice for AV definitions to come as a packed executable? Or is the burden on SonicWALL here to get a little more detailed?

Case-sensitive DNS? Believe it.

2010-11-04T20:41:00.000-05:00

DNS is not case-sensitive, that's what you thought, right? Me too, but apparently somebody disagrees.

We have a client who uses a Cisco ASA 5510 to provide remote access to several road warriors and small satellite offices. We recently started putting Windows 7 64-bit machines in the network, which has traditionally only had 32-bit XP laptops. The Cisco VPN client we usually utilize doesn't work on Windows 7 64-bit (routes don't get created properly). I realize that there is a compatible beta version out and the latest SSL VPN works, but unfortunately due an unbelievable snafu between Tech Data and Cisco, we were left high dry without a SMARTnet agreement. So it came down to using an excellent open-source VPN from Shrew Soft, which is compliant.

I tested the Shrew Soft product and verified it worked great on my local Windows 7 32-bit machine. I also tested it successfully on a 64-bit box, no problems.

Problem:

We use split-tunneling/split-dns to ensure that all traffic for "company.local" is routed over the VPN. Our configuration also specifies that unqualified names, like "server1", get appended with "company.local". So if I ping "server1", it should actually ping server1.company.local automatically and thus get sent across the VPN. Easy.

We went to deploy it to the client and quickly found out that name resolution was not working over the VPN. Attempting to ping "server1" just timed out while the packets shot out the local network instead of getting passed through the VPN. Since everything worked fine on my machines, I was baffled for a minute as to what the issue could be.

Cause:

The clients laptops were all connected to the domain "company.local". So they should be appending company.local to the end of every unqualified request to form a FQDN anyways, making this even more confusing.

However, a little digging showed that they were actually appending a capitalized version of the FQDN like so: server1.COMPANY.LOCAL. No big deal, right? Wrong! Our Cisco config only specified a lower-case version of the FQDN in the split-dns configuration. For some reason, the Shrew Soft VPN client (or perhaps some component used in it) interprets company.local and COMPANY.LOCAL as different domain names.

Resolution:
Add the upper-case version of company.local (COMPANY.LOCAL) to the split-dns configuration like so:

split-dns value company.local COMPANY.LOCAL

Both values are passed to the client which then allows the proper DNS requests to be intercepted and sent over the VPN.

I'm pretty sure this is a Shrew Soft issue, and perhaps one that only exists in Windows 7 or 64-bit machines. This shouldn't happen (and didn't) with the Cisco VPN client. According to Cisco:

The client receives a comma-delimited list of split DNS names from the Concentrator via modeconfig. When the Software Client receives a DNS query packet, the domain name is compared and equentially checked against the split DNS names. Case-insensitive domain name comparison starts at the end of each domain name string and continues toward the beginning of each string, resulting in a match, or no match.

Other than this little quirk, Shrew Soft has produced a great and much-needed product, so I'm certainly not bashing them at all. Maybe something to throw in the next bug release though? Or maybe I'm completely off-base here; the inner-workings of VPN client architecture is not something I'm familiar with.

End Users are Funny Pt II

2010-09-08T17:10:00.000-05:00

Here's another conversation during a remote support session today:

Client: "So Paul, have you seen the internet today?"

Me: "Well, I've been on the internet today, does that count?"

Client: "Take a look at the front page of the internet when you can. There is this crazy looking guy on the front page."

5 Things I Learned from Air Force Networks

2010-08-20T23:46:00.003-05:00

After completing what may very well be my last tour as a Cyber Systems Operator with the Air Force Reserves, I'd like to offer this (unclassified and possibly incorrect) insight into the wonderful system that is the Air Force Global Information Grid (AF-GIG):

The more complicated the password requirement, the more likely it is to be found written on a sticky note next to a monitor.
The scream-test is the most effective method of determining whether a system is critical or a link is live. Unplug it and if someone screams at you, it was important.
Never complain about having to use outdated technology to the person who has to try and maintain that outdated technology.
The more classified the network, the older the equipment. I'm confident that at the center of the most secretive AF networks, there is just an abacus.
You don't need to know the acronyms meaning, you just need to know where the acronym is at in the rack so you can reboot said acronym when your CC complains that his acronym is not working.

Secunia Half-Year Report 2010

2010-07-26T22:35:00.000-05:00

I usually try to avoid reposting information from others, but Secunia has a great security report that has some down to earth information relevant to every computer user:
https://docs.google.com/viewer?url=http://secunia.com/gfx/pdf/Secunia_Half_Year_Report_2010.pdf

Among the interesting statistics?

A group of ten vendors, including Microsoft, Apple, Oracle, IBM, Adobe, and Cisco, account on average for 38 percent of all vulnerabilities disclosed per year.
In the two years from 2007 to 2009, the number of vulnerabilities affecting a typical end-user PC almost doubled from 220 to 420, and based on the data of the first six months of 2010, the number is expected to almost double again in 2010 to 760.
During the first six months of 2010, 380 vulnerabilities or 89% of the figures for all of 2009 has already been reached.
A typical end-user PC with 50 programs installed had 3.5 times more vulnerabilities in the 24 3rd party programs installed than in the 26 Microsoft programs installed. It is expected that this ratio will increase to 4.4 in 2010.
An Windows end-user can patch 35% of vulnerabilities with one tool from Microsoft, but requires another 13 to patch most of the rest.

Also, in terms of the raw number of vulnerabilities, Apple has shot past Microsoft and Oracle to claim the #1 spot. However, this doesn't take into account how severe the vulnerabilities are or how quickly a patch is released.

Symantec Installation Failure

2010-06-10T12:21:00.000-05:00

We've been looking for alternatives to Trend Micro WFBS lately, and I thought I'd give Symantec Endpoint Protection another shot. I spent several hours fighting with the installation package and Symantec Support, but continually had issues with a VBS file that is supposed to run in the setup. A week and several pots of coffee later, I found the problem.

During the installation, Symantec executes iisconfig.vbs, a script designed to setup all of the IIS elements for the management portal. However, the installation rolls back and SEPM_INST.LOG showed the following "return value 3" message everytime.

SESM CA: Failure in IIsConfig.vbs script - See the Windows Event Viewer application log for the failure event.

I went through every article on Symantecs website concerning the issue with no luck. Symantec insisted this was an IIS problem, but even reinstalling IIS did not resolve it. Luckily the issue jumped out at me before it came down to a wipe and reload.

I ran Process Monitor while the install was running and noticed the following line:

MsiExec.exe IRP_MJ_CREATE C:\WINDOWS\system32\cscript.exe ACCESS DENIED Desired Access: Read Data/List Directory, Execute/Traverse, Read Attributes, Synchronize, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, ShareMode: Read, Delete, AllocationSize: n/a NT AUTHORITY\SYSTEM

IRP_MJ_CREATE is the function used to open a file system object (or create a new one), so I looked at cscript.exe and sure enough, the SYSTEM account was set to deny all on the security permissions. I don't see this on any of my other 2003 servers, so I'm assuming it was a result of some hardening at one point in time before I inherited this client. As it turns out, the problem had nothing to do with IIS technically.

If anyone can tell me what sort of automated hardening (Security template perhaps?) causes this I would love to know so I can delve into other issues that may creep up on this server.

End Users are Funny

2010-05-25T19:28:00.001-05:00

I have a client who submits tickets and e-mails to me in haiku.

Paul I have a problem,

our e-mail is not working ...

can you call me please?

I have not decided if this is intentional or not.

Can you please help me?

Susan cannot print today ...

and reports are due.

But it is certainly relaxing to read.

Facebook Accounts Are Almost Worthless

2010-04-28T17:59:00.001-05:00

How much would your Facebook account bring you if you sold it? According to VeriSign, just a couple of pennies. I spotted an article on Dark Reading the other day detailing a recent black market advertisement for over a million Facebook accounts, sold in 1,000 unit increments for $25-$45. About 2 to 4 cents per account, which highlights how easy it has become for these accounts to get compromised and used to spread malicious software.

This little fact just shows you how prolific Facebook has become as the new medium for malware delivery, replacing traditional e-mail. It also is an ominous indicator to the fact that people are much more trustworthy of links coming from their favorite social networking site than they are about the fishy e-mail in their inbox.

Most users are now aware of the basic "Don't open attachments or links from suspicious emails" mantra, but we have a ways to go before that education extends to social networking. The key is getting our users to understand that Facebook, Myspace, YouTube, a website, and e-mail are all simply modes of transportation for a malicious link or attachment to be delivered, and the same due diligence should be paid to each.