Grinding-IT-Out

Long time, no blog…

2012-01-26T13:33:00.001-07:00

So, it has been quite a while since I have posted anything here. Did you miss me? Heh… as if anyone is here.

Anyway, here’s a quick synopsis of the current goings-on around here:

1) Moving forward on our Exchange migration project. This is a biggie. We are currently running to forests/domain side by side. One domain (Win08) hosts everything EXCEPT email. The second domain (Win03) hosts our email. This has not be ideal, to say the least, from a usability and management point. But, we are going to be rectifying that shortly. We are going to be putting in an Exchange 2010 infrastructure in our new domain and migrating everything to that. So, we will finally be able to decommission our old domain and be rid of this multi-domain structure. Looking forward to that!

2) Continuing my ‘developing development’. I finished version 1.0 of my Friendly URL Management App. It is an Arena module and works pretty well, though it is a bit slow form a performance standpoint. I haven’t done any work on trying to optimize it yet. But, it is functioning. Users can visit the page to see which Friendly URLs (fURLs! Just thought of this… and like it!) exists, check their destinations, create requests to add new or edit fURLs, etc. Users can also get a QR Code for the fURL, even choosing the size of the QR Code. The app also has a management component so I can add/edit/remove fURLs from the web interface. All in all, it is a great app and I learned a ton writing it! As always, I have to give huge shout-outs to Nick and Jason for their guidance, input, and mentoring. Thanks guys!!

3) Along the lines of Point 2 above, I have been working on a new Arena App recently. This app is for our Prayer Ministry and is intended to replace an existing app with the same functionality. The existing app is hosted off-site (at a cost to us) and does not leverage any of the functionality/data available to us in Arena. So, I have been working on creating an Arena’ed version of the app. Basically, this is an app to collect prayer requests for various contributors and then create a book/compilation that is distributed to prayer team members. This has been a fun project to work on and, again, I have learned so much. I am excited by the prospect of actually rolling this out for use by our Prayer Ministry.

4) In addition to all of this, I still have my day-to-day work on network management, Windows server admin, backups, etc. etc. etc.

IIS7 Virtual Directory Reporting–Part II

2011-08-17T12:53:00.001-07:00

My last post was on the topic of IIS Virtual Directory reporting in Powershell. The script worked well and gave me a nice little report that I could then send off to others in our Communications Department, for example, when requested. It wasn’t long after this that it was suggested that we build a webpage that generated this report. Then, our Communications Department could just go view the report for themselves.

Coincidentally, I have been working on improving my web development skills. So, I was asked if I wanted to try and tackle this task. I did and we came up with the following short list of features:

Report a list of current Virtual Directories and their httpRedirect – or “Friendly URL” – attributes.
Provide a QR Code for each Friendly URL.
Link to our Arena assignment page to Add/Change a Friendly URL.
FOR ME: A page to create/modify Friendly URLs.

My initial thoughts on how I would accomplish this included parsing the XML in web.config files, traversing physical folders, reading other .config files for the relevant data, etc. It seemed like it was going to be very complicated. Then, as I was researching this project further, I came across the magic that is the Microsoft.Web.Administration Namespace. As you can imagine (or already know if you are familiar with it), this made things MUCH simpler.

Now, I’m no developer (yet), but I am learning and hope to become somewhat proficient someday. I have two great friends, resources, and mentors in Nick and Jason here at Central. With their help, I am actually getting this project built. The goal is to implement it as an Arena module to leverage various Arena features.

I hope to have more to come in the near future as this project develops. It has been fun so far and I have already learned a ton… Mostly, I am learning just how much I DON’T know!

IIS7 Virtual Directory Reporting

2011-06-22T08:58:00.001-07:00

WARNING: This post is bred out of ignorance. I don’t know much at all about IIS7 administration (it’s new to us) or the IIS7 Powershell cmdlets. Everything below is a result of web searches and trial-and-error. I am completely confident that there is a better and easier way to accomplish this stuff. I would love to hear about it! Thanks!

***

I got a request for a list of our virtual directories and their ‘HTTP Redirect’ settings on our web server. We are running IIS7. I knew that there is a Powershell module for IIS7 management – named ‘WebAdministration’ – so I figured that this would be an easy one-liner… something like:

- Get-WebVirtualDirectory | Select Name, HTTPRedirectDestination

Or something similar…

But, this did not work. After looking around for a while, it looks like the WebVirtualDirectory cmdlets don’t have any capacity to report on, or modify, these settings. Further, some of my vDirs had this information stored in a web.config file in their physical path location and some did not (was it in the IIS metabase?!). I would look at the web.config files of two vDirs that looked the same in the GUI and the web.config files would have totally different settings. I was getting very confused and frustrated.

But, I need this report so I needed a plan of attack. I settled on the following:

Get all vDir configurations to be consistent, using a web.config file.
- Make sure each vDir has a unique physical path.
- Make sure each vDir has a web.config file with the proper configuration items.
Write a Powershell script to list vDirs and pull the redirect destination information out of the web.config file.

To that end, I ended up with the following few lines of Powershell -

$hashTable = @{}

foreach ($virtualDirectory in Get-WebVirtualDirectory)
{
    $vdName = $virtualDirectory.Path.Trim('/')
    $vdPath = $virtualDirectory.physicalPath
    $webConfig = $vdPath + "\web.config"
    $webConfigXML = [xml](Get-Content $webConfig)
    $redirectDestination = $webConfigXML.configuration.FirstChild.httpRedirect.destination

    $hashTable[$vdName] = $redirectDestination
}

$hashTable.GetEnumerator() | Sort Name

From here, I just Out-File the script results to a text file and send the report off. Functional, I guess.

*******************

UPDATE: I updated this script a bit to add a little more information to the output. I also changed the output object from a hashtable.

$vDirs = @()

foreach ($virtualDirectory in Get-WebVirtualDirectory)
{
    $vdName = $virtualDirectory.Path.Trim('/')
    $vdPath = $virtualDirectory.physicalPath
    $webConfig = $vdPath + "\web.config"
    $webConfigXML = [xml](Get-Content $webConfig)
    $redirectDestination = $webConfigXML.configuration.FirstChild.httpRedirect.destination
    $vdCreationDate = (Get-ChildItem $vdPath).CreationTime #.ToShortDateString()

    $obj = New-Object object
    Add-Member -InputObject $obj -MemberType NoteProperty -Name "Name" -Value $vdName
    Add-Member -InputObject $obj -MemberType NoteProperty -Name "RedirectDestination" -Value $redirectDestination
    Add-Member -InputObject $obj -MemberType NoteProperty -Name "CreationDate" -Value $vdCreationDate

    $vDirs += $obj
}

$vDirs

Test Lab!

2011-05-09T15:52:00.001-07:00

So, this is something I have been wanting to do for a long time. Being able to ‘play’ with new technology is a huge benefit in my industry and job.

With our virtualization environment being as robust as it is now, we have some headroom and the ability to put a small lab environment in place. So, I have used this guideline to create a base lab environment.

Looking forward to using this to explore new technologies.

Managing IIS

2011-04-26T09:49:00.001-07:00

If you have to manage IIS servers and environments, and you are not watching this video series, you are missing out!

Thank you, Scott Forsyth!

Hyper-V Cluster Is IN!

2011-02-15T14:10:00.001-07:00

So, the work is complete… Well, this sort of work is never really complete, per se. But, our Hyper-V cluster project is complete enough to call it complete. Make sense?

I have blogged about this saga before, so I’m not going to go into every nitty-gritty detail. But, here is an overview of what we did.

BEFORE:

2 x Hyper-V hosts in production.
MD3000i iSCSI SAN providing storage to Hyper-V hosts.
Virtual Machine Manager used (primarily) for management of Hyper-V hosts.

AFTER:

3 x Clustered Hyper-V hosts
MD3000i iSCSI SAN providing CSV storage to cluster.
Virtual Machine Manager still being used to manage the environment.
1 x 5TB DAS to be re-deployed as storage for our Worship Arts department.

PROCESS:

Brought up a third Hyper-V Host server and attached DAS and SAN.
Migrated ALL of our VMs onto this one host (scary!!).
Reinstalled Win2008R2 on original two hosts.
Created a two-node cluster with these hosts, using the SAN to provide CSVs.
Migrated all VMs off the interim Hyper-V host and into the cluster.
Added interim host as a third node in the cluster.
Tested fail-over, fail-back, Live Migration, etc. etc. etc.
Sat quietly for a moment, very pleased with the outcome.

SOME TAKEAWAYS:

This was my first experience with clustering. It could not have been easier.
Getting a Hyper-V pass-through disk into a cluster was not as daunting as I was expecting. Just make sure you know of the limitations, caveats, and oddities.
I like VMM, though I still find I have to go into Hyper-V Manager and Cluster Manager for some things. It’s rare though, which is nice.
I wish fail-back in the cluster would use Live Migration, rather than Quick Migration. Maybe I’m missing something here.
I love being able to do maintenance on my Hyper-V hosts without having to ‘down’ my VMs. This is huge!
Now that our environment has it, I want to maintain an N+1 type environment.
I sure could use a 4th node in my cluster! And a 5th…
BackupExec 2010 works really well with the Hyper-V cluster. Yes, the agents aren’t cheap, but what is your data/environment worth?!

During this project, I hit a few snags here and there. Luckily, someone invented the Internets a while ago, so every answer I needed was only one Google search away!

Lastly, I am looking forward to SP1 that should, from what I have read, give me memory-oversubscribe and a few other cool updates.

Clustered Hyper-V Environment

2011-01-28T10:57:00.001-07:00

I got to work on this project this week. We had three discreet Hyper-V hosts. I moved all of our VMs onto one of the hosts, turning off non-essential VMs and minimizing resource usage as I could to make them all fit. Once the other tow VM host machines were empty, I re-installed the OS (Windows 2008 R2 DCE), created a failover cluster, configured CSVs, etc. and got these machines ready to host HA VMs.

I have been testing failover, live migration, etc. Things are working great! Once I have all of the VMs on the cluster, I will add the third host into the cluster, giving me a great VM hosting environment.

This project has been on my front-burner for quite a while. It is great being in the last stages of it now!

One last note, if you aren’t using SCVMM to manage your Hyper-V virtualization environment… well… you’re doing it wrong!

Here is my ‘plan’ that I followed for this project.

----------------------

RESOURCES:

How To

Pass-through Disk in a cluster

Hyper-V Cluster and VMM08

http://technet.microsoft.com/en-us/library/cc764274.aspx

Error With Live Migration

http://social.technet.microsoft.com/Forums/en-US/virtualmachinemgrclustering/thread/f60a4311-03a3-453f-891f-a1fa3d3f10d5

VIRTUAL DISKS

1Gb – Cluster Quorum Disk
3 x big - CSV volumes

TASKS

Move all VMs onto temporary VM host – VMH00
Change Host Type of VMH04 and VMH05 from non-clustered to clustered
Create 1GB virtual disk to use as quorum disk for cluster
Create Host Group in MDSM (added VH04 and VMH05 to host group)
Disconnected VM storage virtual disk from VMH05 and renamed to ‘CSV01’
Upgrade OS on VMH05 to Win2008R2DCE

Remove Hyper-V role
Add “Desktop Experience” feature to get to disk cleanup tool

Required to free up space to do the OS upgrade

Can’t upgrade OS. Need to do a new install. Can it be done from the R2 disk? YES!

Install OS into new partition – blow out old partition
IP address
Name and domain (VMH01, was VMH05)
Add features – SNMP, Failover Clustering, Desktop Experience, MPIO
Add roles - File Services, Hyper-V

Intel VT 1000 Quad Port NIC Drivers
Install Dell MD software
Configure iSCSI Initiator

Configure Hyper-V networking identical on both VMH machines
Make virtual disks available to both hosts for VM storage

On VMH01

Use iSCSI Initiator to connect to Virt Disks
User Server Manager|Disk Management to configure Virtual disks for access (initialize, partition, format, etc.)
Take disks offline (not sure if needed)

On VMH02

Use iSCSI Initiator to connect to both Virt Disks
ONLY USE Disk Management to rescan for disks

Use Server Manager to install Failover Cluster Feature

On both hosts!

Open Failover Cluster Manager (in Administrative Tools on VMH01)

Validate configuration, using both cluster node names, run all tests
Remediate issues as needed
Create the Cluster

Select “Create a Cluster” from action menu
Add both cluster nodes
Cluster name
Cluster IP address – fixed IP
Select cluster in navigation pane
“Enable Cluster Shared Volumes” in action pane
Read notice and click OK

Select ‘Cluster Shared Volumes’ in the navigation pane

“Add Storage” from action pane
Select the big disks used for VM storage

Open Hyper-V Manager on both Cluster nodes

Select ‘Hyper-V Settings’ in the action pane
Enter new directory locations –

Examples:

C:\ClusterStorage\Volume1\Hyper-V\Virtual Hard Disk (VHDs)
C:\ClusterStorage\Volume1\Hyper-V (VMs)

Add HyperVCluster to VMM08
Configure storage locations in VMM to point to all CSV resources
If not using VMM

Migrate a non-essential VM from VMH00 onto a cluster host, using cluster storage for the VM
Shut down VM
Open Failover Cluster Manager on the VM host that holds the VM

Under Cluster Name, select “Services and Applications” node
“Configure a service or application” in the action pane

Select Virtual Machine – NEXT
Select VM that was imported – NEXT – NEXT – FINISH

Right-click the VM and select START
To live-migrate –

Select the VM
“Live Migrate…” from action pane
Select available cluster node (VMH05)

Test migration, etc. in VMM.

Was getting an error with Live Migration. Had to disable cluster communication on the iSCSI networks. Cluster communication only allowed on the LAN network.
Set VM processor for ‘allow migration to VM host with a different processor’
Make sure “Enable virtual network optimizations’ is NOT checked

Move other VMs from VMH00 into the cluster.
Once VMH00 is empty of VMs, add VMH00 to cluster.

Authentication For Cisco ASA5510 VPN Clients

2010-12-21T10:38:00.001-07:00

We have two connectivity methods that require user authentication, VPN and wireless. Specifically, regarding wireless, we have two main SSIDs, one for guests and one for network users. Guest Access only allows users connected to get to the Internet. Access lists on our routers prevent devices on the Guest VLAN from accessing our network resources. Anyone can connect to this SSID.

Network LAN access is made available by a second SSID. Our Cisco Wireless LAN Controller allows only authenticated users onto this WLAN. Currently, it is configured to use one of our old DCs as a RADIUS server for this purpose. User who connect have two requirements; 1)valid AD account, and 2)membership in a specific AD group. I am hoping to migrate this to our new domain very soon.

Our Cisco ASA5510 was also configured to use our old DC as a RADIUS server to authenticate VPN clients. Like wireless, users had to have a valid AD account and be a member of a ‘Valid VPN Users’ AD group to be granted access. Yesterday, we did some work to move VPN authentication to our new AD Domain. The biggest win, I think, was switching from RADIUS to LDAP as the mechanism for this. With LDAP, there is no need to make any changes, or install anything additional, on our DCs. This is big, as it helps keep my DCs clean and uncomplicated, minimizing the roles required to run on these machines.

I am going to try and share some screencaps of the config… we will see how much I need to ‘blur out’. Hopefully, they will still be helpful and informative.

The big change we made was in the AAA Server Groups area on the ASA5510.

- We created a new AAA Server Group, just calling it ‘LDAP’.

- Under ‘Servers’, I added one of our DCs. The config is as follows:

Server Name or IP Address – Just the IP of my DC
Base DN – The DN of the container for my user accounts

The ‘Scope’ setting allows for more than one level beneath the Base DN, so if you need to search a larger section of AD, you easily can.

Login DN and Login Password: Credentials for a domain account used for AD lookups. This account doesn’t need access to anything, just needs to be able to authenticate in AD.
Group Base DN: This is the DN to the location of the group that users much be a member of to gain access.

With this set up, I then needed to modify the Connection Profile used for VPN, as well as the Group Policy. Group Policy first:

I copied the existing Group Policy into a new one and changed the required settings.

The only change I made was on the “Servers” menu item, changing the DNS Servers, WINS Servers, and Default Domain to all point to our new AD domain.

After the new Group Policy was created, I updated my Connection Profiles as needed:

On the ‘Basic’ menu, I changed the AAA Server Group to my new LDAP server group.
I changed the Group Policy to my new group policy.

The last step was getting the ‘group membership’ requirement activated. This is done under ‘Dynamic Access Policies’ (DAP). We originally only had one DAP, the default, named DfltAccessPolicy. This is a catch-all policy that is applied if no other policy matches the authenticating user.

I created a new DAP:

Selection Criteria

AAA Attribute Type: LDAP
Attribute ID: memberOf
Value: = <name of AD group user must be a member of>

There is a “Get AD Groups” button that should access your AD and pull a list of groups, if your AAA Server settings are correct.

Under ‘Access/Authorization Policy Attributes’

Action: Continue

Changed the action on the ‘DfltAccessPolicy’ to “Terminate”

VOILA! Users authenticating, via LDAP, against my AD with group membership requirements. Obviously, things can get much more complicated and granular. But, for us, this is working perfectly.

* Got to give another shout-out to BriComp Computers for their help with this. He helped me with everything but the ‘group membership’ part of this. Gave me a huge head-start!

Now, on to figuring out the WLC…

Active Directory Migration–2003 to 2008 R2

2010-12-21T09:21:00.001-07:00

It is exciting to be able to say that our Active Directory migration is complete. Sure, there are some straggling issues, but the main migration is finished. Over all, this project was very successful.

A quick recap of what we did…

I posted a brief update on things earlier. We have since migrated all of the resources in our old domain to the new Windows 2008 R2 domain… all resources EXCEPT our Exchange 2003 environment and our SharePoint Services 3.0 environment. From what I understand, neither of those products are very friendly to AD Migration.

Leaving these resources in our old domain create some interesting usage issues for our users. Regarding SharePoint, I am looking to put a new SharePoint Foundation 2010 setup in our new domain and migrate resources to that. My hope is to just duplicate functionality and deprecate our old SS3.0 service without too much trouble. We don’t use SharePoint for too much, so this isn’t a huge deal. E-mail, on the other hand…

As you can probably imagine, we are eager to migrate our mail system to Exchange 2010. I am hoping this will happen in Q1 2011. We will see. In the meantime, email seems to work best when our users authenticate against Exchange (in Outlook or OWA) with their old domain credentials. So, users are logging in to their computers, accessing file shares, etc. using their new domain account, but have to remember to use their old domain account for email. Getting users to remember to juggle domains like this is “interesting”. I mean, I sometimes forget which resources needs which domain credentials. Then there’s the issue of creating new mail-enabled accounts. Interesting stuff.

I can say that the Active Directory Migration Tool is pretty awesome! It has worked very well for us and made migration a relative breeze!

I do want to give another hat tip to Brian Ricks at BriComp Computers, LLC. He has been a great resources and has been fun to work with as well.

P2V’d My Laptop

2010-11-10T09:38:00.001-07:00

So, I am finally moving from XP to Win7 on my work machine. I had wanted to wait until getting a new laptop, as my current machine is almost 4 years old and only 32bit. But, the time has come to bite the bullet.

Now, the thought of just wiping my laptop and re-installing makes me ill. I mean, I have a lot installed and everything has a custom configuration. What I really want is to be able to run two machines, side by side, during my transition. That way, I can check settings, etc.

Hyper-V and VMM to the rescue!

I started a P2V operation in VMM last night before heading home. When I got in this morning, the job was complete.

Everything looks great and the VM is working perfectly. The only hiccup was that Windows had to be re-activated. Other than that, everything is working as expected.

Now it’s time to wipe the HD on that laptop and start the Win7 install.

Fun stuff!

Big Project: AD Upgrade via Migration and Domain Rename

2010-11-08T11:19:00.001-07:00

So, we are in the middle of our first REALLY BIG project in a while. We are moving from our current Windows 2003 domain to a new Windows 2008 R2 domain. Included in this move is a domain rename. Whew!

We got two new servers to be our Win2008 DCs. They are all configured with the new domain and we are now running test migrations using ADMT 3.2.

We are using BriComp Computers, LLC as a consultant on this project that they have been great! Brian Ricks knows his stuff, has been helpful, available, and flexible with us. I am very excited to have him as part of our team for this (and likely future) projects.

I know the actual migration of our AD resources is going to be a big part of this, probably with hiccups along the way. But, I am excited about this and confident that we will squash any bugs that come along.

E-mail Recipient Policy Changes In Bulk With Powershell

2010-11-04T16:23:00.001-07:00

We are changing our Internet domain that we are using for email, web, etc. Part of this is setting up our exchange accounts to process the new SMTP addresses.

I’m not going to go into the detail of how to roll out new SMTP addresses to all of your accounts.What I wanted to share was a Powershell one-liner that I ended up using to find accounts that were not configured to accept automatic updates to their SMTP addresses.

In ADUC, open up Properties on an account and check the ‘E-mail Addresses’ tab. At the bottom is a check-box labeled “Automatically update e-mail addresses based on recipient policy”. If this checkbox is UNchecked, then policy updates are blocked.

For some reason, some of our accounts had this unchecked (most were checked). So, I wanted a way to find these AD objects and CHECK that box, without having to actually open properties on every object in AD.

Enter Powershell (with the Quest AD Cmdlets, of course)!

The magic one-liner is:

get-qaduser -IncludedProperties msExchPoliciesExcluded | where {$_.msExchPoliciesExcluded -ne $null} | foreach-object {set-qaduser $_ -ObjectAttributes @{msExchPoliciesExcluded=''} -whatif}

The ‘-whatif’ at the end just tells the command to do a test run. Remove the –whatif to actually make the change (check the checkbox).

I ran the one-liner a second time (minus the ‘foreach-object’ block), changing ‘get-qaduser’ at the beginning to ‘get-qadgroup’ to see if I had any mail-enabled groups that needed to be updated also. I didn’t. If you do, just change ‘qaduser’ to ‘qadgroup’ as needed.

Thanks To:

http://social.technet.microsoft.com/Forums/en-US/ITCG/thread/1dacb47d-e513-478f-be97-4cb791e467dd

Network-In-A-Box Solution For Our New Campus

2010-09-09T11:25:00.001-07:00

So, we are opening a third campus this week. Exciting! This new campus provided challenges for us that created great opportunities to grow and use new technology. We need to be able to provide Check-In services to our children’s ministries at this new location. So, we need to deploy check-in kiosks that communicate with our servers back on our Mesa campus. There are some key realities this launch presented that created the need for a unique (to us) solution.

Church-In-A-Box: We will be renting a space for this campus, so our ‘church’ will need to be set up and torn down each week. We don’t have the ability to build in a permanent infrastructure like we have on our two current campuses. So, we need to have a solution that is flexible and mobile.
Repeatable: As we look to the years ahead, expansion and additional campuses will more likely include rented space. This model will likely be more feasible than buying land and putting up buildings. So, we need a solution that is repeatable and portable.
Ease Of Use: Our solution needs to be plug-and-play. We will be using servant ministers to set up and tear down each week. Now, I love servant ministers! They serve out of a wonderful heart for God and His mission! But, the reality is that gear just tends to get beat up, especially gear that needs to be moved around, packed and unpacked, etc. Also, servant ministers aren’t always the most technically-proficient people. So we need something that was going to be rugged and super-easy to set up and get working.
Self-Contained: We also wanted a solution that would not rely on the any provided technical infrastructure. This would give us much more freedom as we looked for venues. We didn’t want to have to rely on a provided network or computing infrastructure. We needed a network-in-a-box. All we need from the facility is power.

We explored the option of WiFi or 3G and VPN clients on our kiosks with printers directly attached. After some testing, it turned out that printing would not be responsive enough in this configuration. Printer performance is MUCH better if they are networked, rather than attached to the kiosks. But, printers can’t run VPN clients. We experimented with multiple NICs in the computers and network connection sharing to the printers. This did not work as we thought it might.

So, we realized that the best solution will require a router that can establish a VPN tunnel and then provide network service to our kiosks and printers. We toyed with the idea of using a computer with multiple NICs for this, using the local network as our Internet connection, and other ideas. But, after discussing this among ourselves, and running the project with our integrator/consultant (Sentinel Technologies), we came up with a set of requirements that we felt were optimal.

We wanted an all-in-one unit that would:

Provide 3G access for Internet connectivity, allowing us to not rely on local networking infrastructure.
Have Ethernet ports on the ‘inside’ that we could connect our gear to.
Support VPN tunneling over the 3G network to provide secure communication back to the Mothership.
Allow for communication initiated from either side of the VPN tunnel (two-way tunneling). This proves to be interesting, given the fact that one side of the connection is not ‘fixed’. Using 3G, our IP address will change from use to use. So, configuring fixed VPN tunnels is not possible. (HINT: dynamic route injection FTW!)
Optionally, we would like to have VLAN support and WiFi support for wireless clients on-site.

Enter the Cisco 800 Series!

Due to time constraints and shipping delays from Cisco, we ended up purchasing a used router, Model 881G. It has everything we need and want, except for the WiFi built-in. It supports 3G and has four Ethernet ports. We are using a card from Sprint for 3G access. If we need to expand services on the back-end, we can add a switch for additional Ethernet connections, and a WiFi router for wireless connectivity. But, this model should serve our needs for this current installation just fine.

Once the unit arrived, we worked with an engineer from Sentinel to get the unit up and running. By the end of the day, we had our gear ready for testing. After a few fits and starts to work out some bugs, we connected the printers and kiosks to the router and powered everything on. After a minute or so, the router had established Internet access through the Sprint 3G card, had created the VPN tunnel back to our Mesa campus, and our ASA5510 in Mesa had dynamically established the required routing rules for two-way communication back through the VPN tunnel.

Our kiosks brought up the check-in site and we ran some tests. The check-in app was responsive and printing was surprisingly quick. It was smiles all the way around!

So, our final installation will look something like this:

Simple, compact, repeatable, mobile, responsive. Everything we were looking for in a solution.

Of course, the REAL test will be this Sunday morning. Wish us luck!

ON A SIDE NOTE: I want to say a HUGE ‘Great job and Thank You’ to the other guys on our team! David did an awesome job getting the Sprint card activated on short notice and prepping the kiosks and printers. Nick and Jason, from my limited understanding, did some MAJOR work on the check-in system to support this unique configuration. These guys are amazing devs and a pleasure to work with. Phil did a great job running the project, working with our integrator and finding alternatives when our “Plan A’s” didn’t quite work out. For a while, we weren’t sure if we were going to get a router on site in time! Great job to all!

Upgrading to Exchange 2010

2010-05-10T09:21:00.001-07:00

Well, it looks like this task is finally upon us. We are currently planning to upgrade our Exchange 2003 environment to Exchange 2010. I have heard mixed reviews on the upgrade process. Most have said it is not too bad. Others have had… umm…. difficulties.

I have been reading everything I can on the subject and, at this point, feel fairly confident that I can get us migrated to a new system. My primary concern is the design of the new environment. Our current Exchange setup, while functional, isn’t exactly ‘best practice’. We don’t have the resources to truly do things ‘best practice’ (full redundancy, role/box ratio, etc.) but I am hoping to make some improvements to our implementation.

If anyone out there has any suggestions, resources, or tips… I would love to hear them!

Third Time’s a Charm

2010-02-16T20:15:00.001-07:00

Quick Tip: If the install procedure for your software requires the shutting down of services on the server, it’s a good idea to turn off any monitoring systems that may auto-restart services it sees shutdown.

So, I was installing Backup Exec 2010 on my backup server, upgrading my in-place Backup Exec 12.5 installation. The first time I ran the install, it failed. Looking at the log, it said that it failed while shutting down the Backup Exec services. I checked and the services were running. So, I shut them down myself and re-ran the install. It failed again, with the same error. I looked again and the services were again running. Slightly confused, I stopped the services again. Then, just to double-check, I refreshed the Services screen. To my momentary amazement, the services were running again!

It didn’t take but a second to realize that the “problem” was What’s Up Gold (WUG). I have WUG monitoring my backup server and it is set to restart the Backup Exec services if they go down. So, I would switch the lights off and WUG, at the other end of the hall, would switch the lights back on! OFF-ON-BACKUP FAIL… repeat!

Putting my backup server into maintenance mode in WUG solved my problem.

Good to see that WUG is doing its job. Now, I just have to remember what jobs I have given it to do.

Two Projects…

2010-02-16T15:25:00.001-07:00

One for sure and one a ‘probably’.

First, I will be upgrading our Backup Exec 12.5 system to Backup Exec 2010. I am excited about this, especially the new pass-through granular backup feature for Hyper-V and virtualized workloads. As I understand it, you can do granular restores from virtualized workloads (Exchange, SQL Server, etc.) without having to do a discreet backup of the VM itself. You just have to do a backup on the VM host of the environment. Of course, you still need the Backup Exec agents everywhere – a Hyper-V agent on the VM Host server as well as Exchange and SQL Server agents on the VMs running those respective workloads. Then, once the agents are in place, you just need to run a backup on the VM host machine, grabbing all of the VMs in the process. Form there, you can then do granular restores (individual mail items, SQL DBs, etc.) I am reading more about it and am excited to try it out.

The second project is a third VM Host server for our environment. We currently have two Dell 2950s running Win2008/Hyper-V. We are looking to add an R710 to the mix. We don’t have everything worked out, but I am hoping that we will be able to explore Hyper-V R2, clustering, CSVs, Live Migration, etc. We will see.

Fun stuff!

Odd TCP/IP Behavior in Hyper-V Virtual Machines

2009-12-17T20:31:00.001-07:00

I have two Hyper-V hosts running a total of around 20 VMs. I recently came across some odd behavior that ended up in a call to Microsoft Support, as I couldn’t figure it out on my own and we didn’t want to spend any more time on it ourselves. Basically, I was seeing the following:

As you can see, ping times were all over the place. We found a solution in a combination of KB articles and blog posts.

RESOURCES:

KB938448

KB895980

http://fawzi.wordpress.com/2009/10/28/hyper-v-domain-controller-negative-ping-results/

http://joystickjunkie.blogspot.com/2009/04/erratic-or-negative-ping-times-on-hyper.html

http://blogs.msdn.com/tvoellm/archive/2009/02/18/why-does-my-avg-disk-write-sec-counter-keep-climbing.aspx

I have three VMs that are multi-proc and all three of them were doing this. All three VMs are running a flavor of Windows Server 2003. I don’t know if this happens with other OSs on multi-proc VMs… I am guessing not. With the /usepmtimer switch added in the boot.ini file, all three are now working as expected. I hope that the Hyper-V team is working on a solution to this so that boot.ini file manipulation is not required in the future.

List of Accounts in Local Administrators Group

2009-12-10T16:55:00.001-07:00

Not all of this code is original. Thank you to the many many people in the Powershell community who freely share their code, expertise, and talent with the rest of us. In that spirit, here’s my script for reporting accounts in the local Administrators group on domain workstations. I hope it helps others.

NOTE: This script requires the Quest AD Cmdlets

------------------------------------------------------------------------------

$ErrorActionPreference = "SilentlyContinue"

$a = New-Object -comobject Excel.Application
$a.visible = $True

$b = $a.Workbooks.Add()

$c = $b.Worksheets.Item(3)
$c.Name = "Un-Pingable Machines"
$c.Cells.Item(1,1) = "Machine Name"
$c.Cells.Item(1,2) = "Logon Account"
$c.Cells.Item(1,3) = "Report Time Stamp"
$d = $c.UsedRange
$d.Interior.ColorIndex = 19
$d.Font.ColorIndex = 11
$d.Font.Bold = $True

$c = $b.Worksheets.Item(2)
$c.Name = "Good Machines"
$c.Cells.Item(1,1) = "Machine Name"
$c.Cells.Item(1,2) = "Logon Account"
$c.Cells.Item(1,3) = "Report Time Stamp"
$d = $c.UsedRange
$d.Interior.ColorIndex = 19
$d.Font.ColorIndex = 11
$d.Font.Bold = $True

$c = $b.Worksheets.Item(1)
$c.Name = "Violators"
$c.Cells.Item(1,1) = "Machine Name"
$c.Cells.Item(1,2) = "Logon Account"
$c.Cells.Item(1,3) = "Report Time Stamp"
$d = $c.UsedRange
$d.Interior.ColorIndex = 19
$d.Font.ColorIndex = 11
$d.Font.Bold = $True

$worksheetOneRow = 1
$worksheetTwoRow = 1
$worksheetThreeRow = 1

$filter = "Administrator",
    "Domain Admins",
    "Enterprise Admins",
    "crmadmin",
    "EXService",
    "RTCDomainServerAdmins",
    "SymBEServices",
    "Backup",
    "BackupExec"

$computers = Get-QADComputer | Where-Object {$_.OSName -notmatch "server"} | %{$_.Name}

$group = "Administrators"

foreach ($computer in $computers)
{
    $ping = new-object System.Net.NetworkInformation.Ping
    
    $Reply = $ping.send($computer)
    
    if($Reply.status -eq "success")
    {
        $users = $false
        $needHeader = $true
        
        $g = [ADSI]("WinNT://$computer/$group,group")
        $userList = $g.psbase.invoke("Members")
        foreach ($user in $userList)
        {
            $entry = $user.GetType().InvokeMember("AdsPath","GetProperty",$null,$user,$null)
            $match = $false
            foreach ($i in $filter)
            {
                if ($entry -match $i)
                {
                    $match = $true
                }
            }
            if ($match -eq $false)
            {
                if ($needHeader)
                {
                    $worksheetOneRow = $worksheetOneRow + 1
                    $c = $b.Worksheets.Item(1)
                    $c.Cells.Item($worksheetOneRow,1) = $computer.ToUpper()
                    $c.Cells.Item($worksheetOneRow,3) = Get-Date
                    $needHeader = $false
                }
                $c.Cells.Item($worksheetOneRow,2) = $entry
                $worksheetOneRow = $worksheetOneRow + 1
                $users = $true
            }
        }
        
        if (-not $users)
        {
            $worksheetTwoRow = $worksheetTwoRow + 1
            $c = $b.Worksheets.Item(2)
            $c.Cells.Item($worksheetTwoRow,1) = $computer.ToUpper()
            $c.Cells.Item($worksheetTwoRow,3) = Get-Date
            $c.Cells.Item($worksheetTwoRow,2).Interior.ColorIndex = 4
            $c.Cells.Item($worksheetTwoRow,2) = "No Invalid Users"
        }
        
        $users = $false
        $g = ""
        $userList = ""
        $Reply = ""
    }
    else
    {
        $worksheetThreeRow = $worksheetThreeRow + 1
        $c = $b.Worksheets.Item(3)
        $c.Cells.Item($worksheetThreeRow,1) = $computer.ToUpper()
        $c.Cells.Item($worksheetThreeRow,3) = Get-Date        
        $c.Cells.Item($worksheetThreeRow,2).Interior.ColorIndex = 3
        $c.Cells.Item($worksheetThreeRow,2) = "Not Pingable"
    }
}

$c = $b.Worksheets.Item(1)
$d = $c.UsedRange
$d.EntireColumn.AutoFit()
$c = $b.Worksheets.Item(2)
$d = $c.UsedRange
$d.EntireColumn.AutoFit()
$c = $b.Worksheets.Item(3)
$d = $c.UsedRange
$d.EntireColumn.AutoFit()

Oddity with Hyper-V and Virtual Machine Manager (VMM)

2009-12-02T22:28:00.001-07:00

Every once in a while, one of my Hyper-V hosts will show up in VMM as needing attention. Specifically, the status will show “Needs Attention”, rather than OK. Attempting to refresh the host gives me an “Error (2912)” and/or an “Error (2927)”. In the past, I would attempt to fix this by restarting the WS-Management (WinRM) service. This would almost always result in the service hanging, stuck on ‘Stopping’. From there, my only solution has been a host reboot. Not exactly what I would like. Well, today, I found a solution that did not involve me shutting down ten VMs and rebooting my Hyper-V host box.

I got to the same point as in the past. But, while researching for a better solution, I ran across this blog post about killing a service hung on ‘stopping’.

After reading through it, I found the PID of my service and ran the ‘taskkill /PID xxxx /F’ command, using the PID of my WinRM service. (UPDATE: To get the PID, run ‘sc queryex WinRM’) It looked like it worked, because my RDP connection to the server instantly went dead. But, in a few seconds I got my RDP session back (not sure what happened there…)

I was then able to start WinRM and refresh my host in VMM.

Not exactly elegant, but I didn’t have to reboot my VM host… and that’s something!

Sprint and Palm – Don’t Shoot Yourself in the Foot!

2009-11-22T15:39:00.001-07:00

I want to start by saying that I love my Palm Pre! And, I really like Sprint as a carrier. I always have great coverage everywhere I travel. The Pre is a really nice phone. Palm’s WebOS is great and will only get better (it is only at version 1.3.1). Give it a few major versions to really grow.

That being said, Sprint and Palm are facing a major uphill battle. First, Sprint is a little dog in a big fight, when compared to Verizon, AT&T, etc. Whenever I see smartphone coverage, Sprint is never referred to in terms of market leadership. Likewise, Palm has been slipping in prominence for years and is, when you look at the numbers, just a bit-player in all of this. Android 2.0 and the Droid phone have just made a crowded field even more crowded.

So, how is Sprint and Palm supposed to compete against Verizon, AT&T, Android, iPhone, WinMo, etc. etc. etc.?!

Fortunately, I think there is an answer, and it is staring them right in the face… if only they will be brave enough to see it and act (I know they see it, but will they act?!).

Openness… the answer is openness!

Sprint and Palm need to embrace the rich ‘hacker’ user-based community that is developing around WebOS. There are some awesome things happening that are really extending the functionality of this platform. Homebrew apps outnumber ‘official’ App Catalog apps. There are over 100 patches for WebOS available that do all sorts of cool things! People are building themes for WebOS that allow users to personalize their phone in great ways. In short, the users themselves are passionate about this platform and are doing some amazing work at growing and extending it.

In short, people are passionate about, and hungry for, this platform! I am not the only one who loves my Palm Pre!

I believe that Palm’s WebOS (and Sprint) will be a success to the degree that Palm and Sprint embrace this environment. They need to create an environment and promotes and encourages this sort of development. Remove restrictions, publish resources for developers, and DON’T HINDER APPS THAT GENERATE THE MOST EXCITEMENT!

I am specifically talking about tethering!!

It seems like ‘tethering’ is a bad word in the smartphone business. Everyone wants to tell you how great their phone is, how many apps are available, how great the network coverage is…. they tout their ‘unlimited data plans’… Then tell you about all the limitations! Unlimited data should mean that I should have no limitations on my data use! That’s a service I would pay for.

Tethering is that ‘killer app’ that Sprint and Palm needs. It appeals to casual users, geeks, and working professionals. They should have no limits on tethering and should be promoting that they have truly unlimited data, including tethering. Work with your app developers. Incorporate their best ideas into the OS. Give them the freedom to develop apps that your users want (and that other platforms don’t have). You want to grow, DO SOMETHING DIFFERENT THAN THE OTHER GUYS!

I am very happy being a Sprint/Pre owner. I only hope that, when my contract comes up, that I will still be. But, that it not in my hands… that is up to Sprint and Palm.

OpenDNS – A Great Tool at a Great Price

2009-11-12T08:50:00.001-07:00

So, this week I have sold out to OpenDNS! I have known about them for some time, but have never really dug in to their services. But, with the recent release of their premium services (they still have a free version, which I HIGHLY recommend!), their buzz has gone way up.

This week, I decided to create an account and put OpenDNS on my network at home. It took all of five minutes and the system works great. Account creation took just a minute and configuring my home router (an old Linksys) took just another minute. With that done, and filtering set up and stats turned on, I was ready to go! Category-based site filtering began working immediately. But, that is not all this service offers. I am still learning about the other features.

What prompted this, you may ask? Several factors converged on me this week to urge action.

First, I read a few items, like this one and this one. It just became time to really consider adding some protection to my network for my family. Second, my son, despite my protestations, continues to grow up! As is more normal that it probably should be, kids are using computers more and more and at an earlier age. The last thing I need is for my 6-year-old to stumble upon material he doesn’t need to be seeing.

This topic of discussion continued at work. We had been looking at Google Web Security services, to add to our Postini e-mail management services already in place. Other services in this arena include tools like Websense. However, the more I read and researched, the more I kept gravitating to OpenDNS. First of all, the entry price-point can’t be beat (FREE!!). And, as you move into their newly-offered premiere services, they are still extremely price-competitive.

I would highly suggest and recommend that you look in to OpenDNS and their offerings. Check it out for both your home and enterprise! I think you will be glad you did!

Backup-To-Disk Problems with BackupExec 12.5 to a Virtual Disk on a MD3000i

2009-08-31T09:03:00.001-07:00

…A long title for a weeks-long problem!

Well, it has been a long time since I have updated this blog. But, that doesn’t mean nothing has been going on! :-)

For the past couple of weeks, I have been troubleshooting a problem with my backups. I use BackupExec, so that shouldn’t really surprise anyone! But, in this case, the problem (as best as I can tell) turned out to lie elsewhere.

Here’s the skinny…

I do all of my backups to tape, except for my Exchange backups. They go to disk. I have a 3TB disk on my MD3000i that I use for this. That way, I can make the most efficient use of BackupExec’s GRT technology. It was working fine for a while until (as is often the case in Windows environments) it just stopped working.

My backup-to-disk jobs started failing with the error code: E00084AF

Symantec’s KB had a number of articles that spoke to the issue, but nothing seemed to work. I spent about a week on my own trying to solve the problem, running updates, tweaking the registry, deleting/recreating jobs… Nothing worked, so it was then time to call Symantec Tech Support.

Now, like most people, I DO NOT like calling tech support, especially for large companies. This has nothing to do with my ego and everything to do with the fact that, in most cases, the first-level support is likely a guy just like me… someone who kinda-knows the product, is sitting in front of a computer either reading from a ‘tech-support script’ or just searching their own KB as you describe your problem to them. I know they are trying to be helpful, but you end up spending most of your time re-hashing everything you have already tried! </rant>

I will say this, however… the Symantec guys were willing to ‘spend the time’ with me on this. I never felt rushed by them or brushed aside. I appreciated that.

Anyway, none of this troubleshooting helped and we all went into the weekend scratching our heads, wondering what we were going to try/look at next. Then, over the weekend, I had an idea…

As a Windows guy I have learned that, sometimes, you just need to start over. For example, if a distribution list in Outlook isn't working right, you may have to just delete it and re-create it (or add then remove someone). I have come across similar situations many, many times... Situations where 'touching' an object somehow resets things and gets it working again. Sometimes it's just a matter of changing a setting, saving, and then changing the setting back.

This is essentially what I did with my virtual disk on my MD3000i. I went into the management console of my MD3000i and changed the 'ownership/preferred path' of the virtual disk from one controller module to the other. Then, after a server reboot I ran a test job and it worked. The backups have been running fine ever since.

I have no idea what initiated this issue, or where it originated. That is the most frustrating part. I am just glad that things are working again!

Powershell and E-mail

2009-07-13T10:21:00.001-07:00

There are times when I need to notify a group of people of a change made on our network file system. Perhaps the contents of the folder has changed and I need to let everyone who has access to that folder know. Perhaps permissions to a folder has changed (someone has been added or removed) and I want to notify everyone with rights to the folder.

This is normally an annoyingly manual process. Cull names from the security tab and generate a list of people, then paste them in to a mail message, etc… you get the idea.

So, I decided I would see if I could write a Powershell script to do the heavy lifting for me. Specifically, I want my script to:

Gather the e-mail addresses of everyone with access to a shared folder
Create an e-mail message and address to these people
Save this message in my Drafts folder for further processing

Really a simple task, but, if automated, will save me tons of time.

The script is not yet written, but I have the basics down. Of course, it is ridiculously simple with PowerShell (and the Quest AD Cmdlets)

Here is the basic framework I have thus far…

# Get Email address of group members
$addrs = Get-QADGroup <GroupName> | Get-QADGroupMember | select email

$ol = New-Object -comObject Outlook.Application

$mail = $ol.CreateItem(0)

#Address mail
foreach ($addr in $addrs)
{
    $mail.Recipients.Add($addr.email)
}

$mail.Subject = "Some Subject"
$mail.Body = "Some Body"

#Save to drafts
$mail.Save()

As you can tell, there is a lot of work yet to do. Input, validation, etc., etc. But, in just a few lines of code, this script is already performing tasks that would take me minutes to do. I love how easy it is to access AD objects and COM objects and pass data back and forth.

Perspective

2009-06-21T12:00:00.001-07:00

I invest a lot of time (measured in actual minutes and hours) on computers. My job is in IT, managing dozens of Windows servers, dozens of Dell and Cisco switches/routers/WAPs/etc., over a hundred Cisco IP phones and their users, multiple software packages and all the other ‘trimming’s that come with a typical SMB systems installation. I spend many more hours reading and learning about technology, trying to keep up on trends, learn about what’s on the horizon, develop my skills on solutions we have in place. Much of my free time is spent on the computer, playing games, watching TED Talks, Stumbling, etc. All this to say, I’m no different than most of you, I am guessing…

I spend a lot of time on computers.

But, today is Father’s Day. For me, this is a day of perspective. Because, when I look into the eyes of my two sons, when my 5 year old runs up to me and gives me the longest hug I’ve had in a long time and tells me, “I’m so glad you are my father.”, well I am reminded of what is really important.

I just want to say to all you fathers out there, Happy Father’s Day. I hope and pray that this is a day of joy and happiness for you.

BAD_ADDRESS = bad!

2009-06-18T14:51:00.001-07:00

I was working to deploy some new IP phones on our Gilbert campus, and kept getting DHCP address assignment errors. The phones would sit there ‘configuring IP’… Just sitting there. Meanwhile, my DHCP scopes was filling up with leases to “BAD_ADDRESS”. Do a web search for “DHCP BAD_ADDRESS” and you will get a good idea of the problem.

While some reported this problem being associated with Mac clients or other IPv6 clients on the network, this was not my problem at all. My problem was simple duplicate IP addresses on the network. The tough part of this was that there were no DNS entries for the offending IP address and no valid DHCP leases for these IPs. Yet, I was able to ping the addresses, so something out there was using these addresses.

I tried using ping/arp to find the devices on the network, but did not have any success until a network engineer I was talking to suggested that go to my core router/switch on the network and do my ARP lookups on that device. I had been doing them from my workstation and a couple of edge switches. This was the key and I had struck gold. My core switch (managing all of my VLANs) had all of these IP/MAC entries in its ARP table.

From there, I was able to find the actual devices that has these BAD_ADDRESSes. This exposed the root problem that turned out to be an interesting residual from a previous issue I had worked on. It turns out that there were a number of phones on my network that were still configured to use the now-defunct IP address from our old multi-homed configuration. So, essentially, their DHCP server no longer existed. Thus, they had little choice but to hold on to their assigned IP address for dear life, hoping and praying that, someday, their long-lost DHCP server would return. Little did they know that the server was sitting right next to them, just with a new IP address. I quickly generated a list of these devices and rebooted them. They immediately found the DHCP server and got an IP address.

But, back to the BAD_ADDRESS issue… My DHCP scope had no record (no active leases) for these residual IP addresses being held by these orphaned devices. So, when I plugged a new phone in, my DHCP server was more than happy to attempt to hand those IP address out. From what I have gathered, the basic steps in DHCP go something like this (super-simplified and possibly not even right):

Client makes request
Server pulls an unused address from the appropriate scope
Server responds to client with this IP address and associated network configuration
Client verifies that IP address is actually available (not currently on the network)

SUCCESS! Client keeps the network configuration and is happily on the network
FAILURE! Client reports back to DHCP server that IP is already in use

DHCP adds entry in its DHCP lease DB for this IP address, assigning it to ‘BAD_ADDRESS’
Start process over with next available IP address

Once all devices were talking to the correct DHCP server, this problem simply went away. My new phones were immediately configured and working.