Geekworldinc

The Story of the 30 seconds freeze on Netapp filer every five hours at 9am, 2pm, 7pm and 12am

2014-04-29T14:14:00.000-07:00

We have two netapp filers in active/active HA. One filer is for the SAS drives mostly storing the OS disks for the vmware VMs and the second filer filer02 hosting the data virtual disks for our VMware vms, CIFS and exchange database and logs.

A few weeks ago, we started noticing about 30-45 seconds freeze in outlook at 9:00am and 2:00pm. We only noticed the freeze at these times because this was during hours. We thought it was something to do with exchange servers. There were events related to delayed writes on the database and logs LUNs. However, the issue was occurring on all the exchange mailbox servers. We changed the backup schedules on all the servers. We take a backup every 4 hours. However, the issue persisted.

We then thought it could be a network issue. We use dedicated switches for storage traffic. However, our VMs ran fine and we did not notice the freeze on the VMs. When we checked the performance logs for the SAN we noticed a spike in NFS latency and CIFS latency around 9:01am, 2:01pm, 7:01pm and 12:00am. None of the de-dup schedules were running at all these four times. In the syslogs of the filer we saw the following for each of the exchange server’s iscsi initiators at the times mentioned above.

We still couldn’t understand what was causing the LUNS to reset the connection.

Event:

iscsi.notice

Severity:

notice

Message:	ISCSI: Initiator (iqn.1991-05.com.microsoft:server-mbx-05.domain.com) sent LUN Reset request, aborting all SCSI commands on lun 7
Triggered:	Sun Apr 27 00:01:32 PDT

iscsi.notice

Severity:	notice
Message:		ISCSI: New session from initiator iqn.1991-05.com.microsoft:srv-mbx-07.domain.com at IP addr 172.20.1.64

Triggered:

Fri Apr 11 09:01:19 PDT

We changed the schedule for all our VMware backups and any other backups that happened on the hour. There was still no change in this behavior.

Then I learned about the bug mentioned in

http://cormachogan.com/2013/02/08/heads-up-netapp-nfs-disconnects/

about the Netapp and NFS issue. We applied the workaround by changing the NFS.maxqueuedepth to 64 on all our hosts. Because the issue had started appearing after we had added a new host.

It still did not make a difference. Something was off and didn’t make sense. Because our VMs were not showing the freeze condition.

We went through every netapp log, every vmware log on each host but the only thing that pertained to this situation was the following in the \var\log\vmkernel.log for each host at the mentioned time.

2014-04-28T21:01:42.787Z cpu9:10653)NFSLock: 608: Stop accessing fd 0x410013247ce8 3

2014-04-28T21:01:42.787Z cpu9:10653)NFSLock: 608: Stop accessing fd 0x4100132692a8 3

2014-04-28T21:01:42.787Z cpu9:10653)NFSLock: 608: Stop accessing fd 0x41001219b328 3

2014-04-28T21:01:42.787Z cpu9:10653)NFSLock: 608: Stop accessing fd 0x41001218a228 3

2014-04-28T21:01:42.787Z cpu9:10653)NFSLock: 608: Stop accessing fd 0x410013242c68 3

2014-04-28T21:01:42.787Z cpu9:10653)NFSLock: 608: Stop accessing fd 0x41001219fae8 3

2014-04-28T21:01:42.787Z cpu9:10653)NFSLock: 608: Stop accessing fd 0x41001219e5e8 3

2014-04-28T21:01:42.787Z cpu9:10653)NFSLock: 608: Stop accessing fd 0x41001219fe68 3

2014-04-28T21:01:42.787Z cpu9:10653)NFSLock: 608: Stop accessing fd 0x410012190268 3

2014-04-28T21:01:42.787Z cpu9:10653)NFSLock: 608: Stop accessing fd 0x410013234fe8 3

2014-04-28T21:01:42.787Z cpu9:10653)NFSLock: 608: Stop accessing fd 0x410012190968 3

2014-04-28T21:01:51.783Z cpu6:8198)NFSLock: 568: Start accessing fd 0x410013246468 again

2014-04-28T21:01:51.801Z cpu6:8198)NFSLock: 568: Start accessing fd 0x41001219b328 again

2014-04-28T21:01:51.801Z cpu6:8198)NFSLock: 568: Start accessing fd 0x410012190268 again

2014-04-28T21:01:51.801Z cpu6:8198)NFSLock: 568: Start accessing fd 0x410013247ce8 again

2014-04-28T21:01:51.801Z cpu6:8198)NFSLock: 568: Start accessing fd 0x41001219fe68 again

2014-04-28T21:01:51.801Z cpu6:8198)NFSLock: 568: Start accessing fd 0x410013242c68 again

2014-04-28T21:01:51.801Z cpu6:8198)NFSLock: 568: Start accessing fd 0x410012190968 again

2014-04-28T21:01:51.801Z cpu6:8198)NFSLock: 568: Start accessing fd 0x41001219fae8 again

2014-04-28T21:01:51.801Z cpu6:8198)NFSLock: 568: Start accessing fd 0x41001218a228 again

2014-04-28T21:01:51.801Z cpu6:8198)NFSLock: 568: Start accessing fd 0x4100132692a8 again

2014-04-28T21:01:51.801Z cpu6:8198)NFSLock: 568: Start accessing fd 0x410013234fe8 again

2014-04-28T21:01:51.801Z cpu6:8198)NFSLock: 568: Start accessing fd 0x41001323dda8 again

2014-04-28T21:01:51.801Z cpu6:8198)NFSLock: 568: Start accessing fd 0x41001219e5e8 again

We were about to give up and call support for vmware and netapp, when I thought of aggregate snapshots. When I checked the aggregate snapshots….there it was..each aggregate snapshot was taking place at the mentioned times. Changed the schedule to off-peak hours, and saw the iscsi.notice at the times i changed the schedule to. I would have never thought aggregate snapshots could cause a freeze.

Script to monitor Exchange 2010 queue and send email alerts if threshold is reached

2014-02-22T15:40:00.000-08:00

This is a script that can be scheduled as a scheduled task and run every few minutes on a server and exchange management tools installed. This script monitors the exchange queues on your CAS servers and send email alerts (make sure you specify an external email address as well) if the queue threshold that you specify is reached.

# Script: Exch2010QueueMonitor.ps1
# Purpose: This script can be set as a scheduled task to run every 30minutes and will monitor all exchange
#2010 queue's. If a threshold of 30 is met an
# output file with the queue details will be e-mailed to all intended admins listed in the e-mail settings

# Comments: Lines 27, 31-35 should be populated with your own e-mail settings
# Notes:
# - tested with Exchange 2010 SP1 - SP3
# - The log report output file will be created under "C:\Support\Scripts\queue.txt"

$snapins = Get-PSSnapin
$snapins | foreach-Object {

if ($_.name -match "Exchange")
{
$exchloaded = $TRUE
}
}
if ($exchloaded -eq $TRUE)
{
if ($showgui)
{
Write-Host -ForegroundColor Green "Exchange 2010 Snapin already loaded."
}
}
else
{
Add-PSSnapin *Exchange*
if ( $showgui ) { Write-Host -ForegroundColor Green "Exchange 2010 Snapin had to be loaded." }
}

$filename = “C:\Support\Scripts\queue.txt”
Start-Sleep -s 10
if (Get-ExchangeServer | Where { $_.isHubTransportServer -eq $true } | get-queue | Where-Object { $_.MessageCount -gt 30 })

{

Get-ExchangeServer | Where { $_.isHubTransportServer -eq $true } | get-queue | Where-Object { $_.MessageCount -gt 30 } | Format-Table -Wrap -AutoSize | out-file -filepath C:\Support\Scripts\queue.txt
Start-Sleep -s 10

$smtpServer = “smtprelay.domain.com” #your smtp server
$msg = new-object Net.Mail.MailMessage
$att = new-object Net.Mail.Attachment($filename)
$smtp = new-object Net.Mail.SmtpClient($smtpServer)
$msg.From = “noreply_exchange@domain.com” #send as address
$msg.To.Add("admin1@domain.com") #change this address for admin address
$msg.To.Add("admin2@externaldomain.com") # add external email address like gmail etc
$msg.Subject = “CAS SERVER QUEUE THRESHOLD REACHED - PLEASE CHECK EXCHANGE QUEUES”
$msg.Body = “Please see attached queue log file for queue information”
$msg.Attachments.Add($att)
$smtp.Send($msg)

}

Citrix Xenapp 6 - Published Desktop disconnects when published application is launched from within a published desktop

2014-02-22T15:29:00.001-08:00

When you attempt to launch a published application from within a published desktop session, the published desktop session will disconnect. This is because Citrix receiver by default tries to reconnect to all your open sessions on launch. So when you attempt to start the published application, it disconnects the session of the published desktop. You can modify this behavior by changing the following registry key.

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\Dazzle]
"WSCReconnectMode"="0"

Change this key on your published desktop. Once this is modified, relaunch the published application and your desktop session should remain intact.

SOLVED - Citrix Xenapp 6.0 hotfix rollup fails to install - Error 1904 Module C:\program files (x86)\citrix\system32\rpm.dll failed to register

2014-02-17T20:51:00.000-08:00

When you try to install hotfix rollup on a Citrix Xenapp 6.0, you receive the following error.

To resolve this, leave this window open. And browse to C:\windows\system32\ and rename the file cutildll64.dll to cutildll64.dll.old. Now hit retry and the hotfix should install successfully. after the install completes, reboot the server and then copy the new file from C:\program files (x86)\citrix\system32\cutildll64.dll to C:\windows\system32\.

Restore items that had last modified date of a particular day from a Netapp snapshot

2014-02-17T19:29:00.002-08:00

We ran into a problem where we got hit by a cryptolocker on one of our cifs and that ended up encrypting a bunch of files. Now this particular cifs share had around 500GB of data. we managed to restore the share from a snapshot but the users had modified a large number of files just before the virus hit. So the dilemma was how to just restore the files that were changed on a particular day.

Well, I love powershell for a reason. Some of the things to make sure are.
1. .snapshot directory should be visible on the netapp cifs share.
2. get the snapshot name of the day (or hour) you want to restore from. In this example, it is nightly.4

The command is

Get-ChildItem -path Y:\~snapshot\nightly.4\share \operations *.* -Recurse | where-object {$_.lastwritetime.day -eq 5 -AND $_.LastWriteTime.Month -eq 9 -AND $_.lastwritetime.year -eq 2013} | copy-item -destination Y:\share \temp\restored

you can change the following to reflect the last modified date.

1. $_.lastwritetime.day to day of the month

2. $_.LastWriteTime.Month to the month of the year

3. $_.lastwritetime.year to the year

How to resolve Citrix desktop delivery service console discovery process errors

2014-02-03T14:52:00.001-08:00

The discovery process might fail with the following error

“Errors occurred when using servername in the discovery process”

- If the local computer is member of the farm, start the discovery process again and add local computer to the list of the servers and run the discovery again.

- If the discovery still fails, check if the server(the datacollector) is up. And MFCOM service and IMAservice is running on the server and also on local computer.

- If MFCOM service is not running, the server will need to be rebooted

- Run the command qfarm /load to check if the local server, and the servers are in the list. If they are not, run the following command on the server that is not in the list

Net stop imaservice

Net start imaservice

Rerun discovery

- If the process still fails, check if the licensing serveris up and does not have any errors in the eventlog related to citrix

- Also check if datastore (SQl server) is up and instance that the datastore is residing on is running on it.

Once we are sure that the datastore is up, open command prompt on any of the XenApp servers and run the following command. (make sure the command prompt is run as administrator)

Dscheck

See if any inconsistencies show up. If there are inconsistencies, run the following command to clear the inconsistencies

Dscheck /clean

Run the discovery again. This should fix the problem

Powershell script to remove smtp addresses with a domain from mailboxes in Exchange 2010

2014-01-30T17:24:00.003-08:00

This script is to remove a smtp domain from a client’s mailboxes. Email address policy is how these domains get added to the mailboxes. However email address policies are additive only and cannot be used to remove the domain that was added using email address policy. They have to removed manually from each mailbox.

When will we need this?

A good scenario is when a client company ABC has changed their company smtp domain from abc.com to xyz.com and no longer want to receive any email on the old smtp abc.com that is they want that if somebody sends an email to abc.com they should get a bounceback. The old domain abc.com has been removed from the accepted domains and the MX records for abc.com no longer point to your exchange server. Externally this will work correctly because obviously the DNS has been modified and abc.com has been removed from accepted domains in our exchange. But internally on the same exchange server, users will still be able to send and receive an email to abc.com address. The email addresses with abc.com either needs to be removed manually from each users exchange properties or you can use powershell to do it for you.

How will this work?

1 I am using a custom attribute for filtering the get-mailbox command, but you can use –scope to use OU for filtering or select all the users. Modify the customattribute1 value to the of the client and domain name (domain.com) to the target smtp domain in the script and copy the script

2 Open the exchange management shell and paste the script in the shell window (press enter once)

All the email address for client with clientcode ‘clientcode’ that contain email addresses with smtp domain ‘domain.com’ will be removed.

#Script to remove email address for a particular domain as EAP is additive Only

# BEFORE USING - please change the domain name and custom attribute as mentioned in comments

# IMPORTANT: DOMAIN IS NOT YOUR AD DOMAIN BUT THE SMTP DOMAIN YOU WANT TO REMOVE

#Gets the client mailboxes for the users with customattribute1 set as 'abc'

foreach($Clientmailbox in Get-mailbox -ResultSize Unlimited | where{$_.CustomAttribute1 -eq 'abc'})

{

#for each mailbox grabs the email addresses and filters the addresslist

#for the smtp domain the needs to be removed

#and then removes the email address

#CHANGE THE DOMAIN from domain.com to the corresponding domain

$Clientmailbox.EmailAddresses |

　　　 ?{$_.AddressString -like '*@domain.com'} | %{

　　　　　 Set-Mailbox $clientmailbox -EmailAddresses @{remove=$_}

　　　 }

Powershell script to create a report for whether or not a Citrix Xenapp Hotfix is installed on Citrix Xenapp farm servers or not

2014-01-29T09:33:00.001-08:00

One of the second biggest pain for me as a farm administrator has always been to figure out if the XenApp Servers in the farm do have the latest rollup (or hotfix installed. And which are the servers where the latest roll up is not installed. The biggest pain of course is to install it remotely on all the XenApp servers ( I am still working in that one).

So I have this script that creates a report for a particular rollup pack or hotfix telling which servers have the hotfix installed and which don’t. It also creates a list of servers where the hotfix is not installed so I can install the hotfix on those servers.

I have tested this script on XenApp 6.0 farm. I ran the script from a XenApp server in the farm.

# Script to create a report on whether the hotfix is installed or not on the

# XenApp servers

# Created by: Ruby Nahal

# Register Addins if missing

$snapin = Get-PSSnapin | where {$_.name -eq 'Citrix.Common.Commands'}

if($snapin -eq $null){ Add-PSSnapin Citrix.Common.Commands }

$snapin = Get-PSSnapin | where {$_.name -eq 'Citrix.XenApp.Commands'}

if($snapin -eq $null){ Add-PSSnapin Citrix.XenApp.Commands }

$farm = Get-XAfarm #get farm info

$servers = Get-XAServer # get all the xenapp servers in the farm

[string]$CXAName # string object to store the Xenapp server name

$logfile = "C:\Support\XAreport.txt" # log file to store CXA name and if the hotfix is installed or not

$CXAList = "C:\Support\CXAlist.txt" #list of servers that do not have the hotfix installed

Foreach($CXA in $servers) #run for each CXA or Xenapp server in the farm

{

[Boolean]$RUisInstalled = $false #boolean variable to check the status of hotfix on the server

$CXAName = $CXA.Servername #Get the server name for the CXA in the current loop

Write-Output $CXAName # write the cxa name

#function XenappHotfixRU2Report([string]$HotfixName) #can also use this script as a function for multiple hotfixes

# {

# Get current computername and XenServer object

[String]$HotfixName = 'XA600W2K8R2X64R02' #hotfix in question - will be commented out if used as the function

foreach($hotfix in (Get-XAServerHotFix -ServerName $CXAName)) #run for each hotfix installed on the server

{

if($hotfix.HotfixName -eq $HotfixName) #if hotfix name is equal to the $hotfixname set the boolean variable to true

{

$RUisInstalled = $true

}

else

{

echo ".." # else, basically do nothing

}

echo "'n****** $CXAName *******" | Out-File $logfile -Append -Width 240 # write the CXA name to the log file

if ($RUisInstalled) # tell us if hotfix is installed or not by writing it to the log file

{

echo "Rollup Installed" | Out-File $logfile -Append -Width 240

}

Else

{

echo "Rollup was Not Installed" | Out-File $logfile -Append -Width 240

echo $CXAName | Out-File $CXAList -Append -Width 240

}

# }

#XenappHotfixRU2Report 'XA600W2K8R2X64R02'

}

Workspace control on Citrix Xenapp published desktop

2014-01-28T15:57:00.005-08:00

What this means?

- If you login to published Desktop from computer 1 and then go to computer 2 and login from there, your session will automatically get disconnected from computer 1 and launch on computer 2. You will NOT get the error “you have reached the concurrent session limit”

- Auto- launch is disabled. This is a default from citrix that when workspace control is enabled, auto-launch will be disabled. This means, when you open the login page and enter your username and password, the session will NOT launch automatically. You will have to click on the Application icon to launch your session

Where this does not work?

- If a session is running on computer 1 and you attempt to launch another session to the same desktop, you WILL get the concurrent limit error.

How this is done?

In Citrix Appcenter, expand “applications” and right click on the application you want to enable this on and select application properties. Select the “Name” section and under “application description”, type in the following

KEYWORDS:TreatAsApp Auto

TreatAsApp forces the desktop to be treated as a published application and enables workspace control

Auto automatically subscribes to the application in the storefront and the user does not need to manually add the application.

Script to backup ESXi configurations for all hosts in the cluster

2014-01-07T14:32:00.001-08:00

This script is used to backup the ESXi configuration of all the hosts in the cluster. This backup will serve in the event if a VMware host has to be rebuild and if the configuration of a host goes corrupt or compromised.This is a powershell script that requires the vmware powercli module.

#This script will backup the hosts Configuration

#Created By: Ruby Nahal

#run as scheduled task using the a service account that has

#administrative rights on vcente. vcenter must have powercli installed

#this script creates the backup in D:\esxibackups and each backup is replaced # by the new one

# it also copies the backups to \\fileserver\share\backups

#Load the vmware powershell snapin

add-pssnapin vmware.vimautomation.core

#Connect to the vCenter Server

Connect-VIServer vcenter-01

#grab the list of hostnames

$esxihosts = get-VMhost | select name

#Print the list

Write-Output $esxihosts

#variable of type string to convert the object $esxihost to string value

[string]$esxihostname

Foreach ($esxihost in $esxihosts)

{

$esxihostname = $esxihost.Name

Write-Output $esxihostname

Get-VMHostFirmware -VMHost $esxihostname -BackupConfiguration -DestinationPath D:\esxibackups

}

Copy-Item -Path D:\esxibackups -Destination \\fileserver\share\backups -Recurse -Force

How to disable the save password option on the Citrix Reciever from Citrix Storefront 2.0

2014-01-01T16:15:00.000-08:00

When the users logged in using receiver for windows directly instead of using receiver for web (receiver for web is the webpage we access using the browser – the typical way we log on) it allowed you to save passwords and so next time you open it, you are already logged on to receiver. Bit too much of a security concern for us. The solution from citrix is to change a registry on the client computers which doesn’t quite work for us http://support.citrix.com/article/CTX134341. We wanted to do it at the infrastructure level and not at the client level. Although Citrix storefront did not provide this functionality out of the box, there is a workaround.

The following changes were made to the storefront servers to have the disable save password configured.

You can disable the ability to save passwords at the StoreFront level with the below instructions.

1. Locate the Authentication service directory on disk under the IIS root folder.

e.g. “C:\inetpub\wwwroot\Citrix\Authentication”

2. Open the file “\Views\ExplicitForms\Authenticate.aspx” in a text editor.

3. Comment the following line

<% Html.RenderPartial(“SaveCredentialsRequirement”, SaveCredentials); %>

By replacing it with

<%-- Html.RenderPartial(“SaveCredentialsRequirement”, SaveCredentials); --%>

4. Save the file.

How to deploy an ova template (vAPP) for vCenter Operations Manager (vCOPS) without a DRS license

2013-12-31T11:43:00.001-08:00

To deploy the vCenter operations manager which can be downloaded as a ova file, a DRS license is required. If you attempt to deploy the template without a DRS license, you will receive the following error

"Cannot deploy OVF descriptor on cluster-01 since it contains a vAPP and the cluster is not in DRS mode"

vCenter Operations manager foundation now is available free from the VMware website. So how to deploy it? Well, here is how

vAPP is essentially just a container for virtual machines that holds settings for the two virtual machines such as IP pools, boot order etc. vAPP can be treated as a single entity and has the options to turn on or off the vAPP. However, vAPP can be split and the individual virtual machines can be taken out from the vAPP by removing the vAPP layer. vCOPs contains two VMs - UI VM and Analytics VM which are part of the vAPP. Please do remember, a vCenter license is required. vCOPs cannot work without vCenter.

To split the vAPP we must first deploy the vAPP. For example I have a cluster of 3 hosts on the vCenter. Follow the steps below to deploy the vAPP

Remove one of the hosts from the cluster such that it is no longer part of the cluster but still a part of the datacenter. For achieving this, you can either drag and drop the host out of the cluster or you can remove the host to the cluster and add the host at the datacenter level. You will have to remove all VMs from the host and take it in maintenance mode.
Once the host is no longer a part of the cluster but just a part of the datacenter, deploy the vAPP on the host. You will have to configure the vAPP with IP settings, name etc.
Once the vAPP has been deployed, turn on the vAPP. Wait for both the VMs to be up and running. Once the VMs are up and running, make sure you can access the UI VM. Check the DNS settings and the make sure you have DNS entries for the VMs on your DNS servers. VMs must be able to resolve each other.
If you can access the registration page, we are good to go. You can either to register the vCOPs with the vCenter server now or later. You will need an account with admin privileges on vCenter to complete the registration.
Now turn off the vAPP and after the vAPP has completely turned off, move the server back into the cluster by dragging and dropping the server into the cluster. This will remove the vAPP layer and you will see two VMs in the cluster called UI VM and Analytics VM.
You might receive an error and you can choose to ignore it.
It is very IMPORTANT that you turn on the analytics VMs at least 10 minutes before the UI VM. You will break it if you mess up the boot order. The vAPP is no longer there to manage the boot order, so the boot order will have to be managed manually.

And this is it. vCenter operations manager has been deployed.

How to find if a balloon driver is enabled on Virtual Machines on a VMware host

2013-12-27T13:27:00.002-08:00

Open ssh connection to the host.

NOTE: you will have to enable the SSH service by connecting to the host in vsphere client. Go to configuration>security profile> services 'properties' > select SSH and click on options> start.

2. After logging in, type in the command esxtop, by default vmware displays CPU attributes.

3. By press 'm' to switch to memory view.

4. We are looking for the MCTL attribute which might not be in the list of attributes being displayed, so press 'f' to edit the list of attributes being displayed.

5. You can toggle the field to show or not by pressing the corresponding letter. Like in this case, press 'j' to display the MSTL attribute
6. The MCTL attribute will show 'y' if the balloon driver is enabled.

How to modify the replication interval between Domain controllers in different sites to less than 15 minutes?

2013-12-25T16:46:00.000-08:00

I recently came across a problem where DCs in our two data centers (Each datacenter is a different AD site) were not replicating fast enough. Because the DCs in question are in different sites, the minimum interval for replication between sites can only be 15 minutes, which is not good enough for us. So the workaround is to set the ADSI attribute “options” for the site link to 1 which means USE_NOTIFY attribute.

USE_NOTIFY forces the replication to occur when a change is made. But this does not work on manually created connections like we have. It only works on connections managed and created by KCC (these connections will have the value “automatically generated” shown under the first column. (details can be found at http://blogs.msdn.com/b/canberrapfe/archive/2012/03/26/active-directory-replication-change-notification-amp-you.aspx)

So what worked in our case to force the servers to override the default values by using integer value 12 for each server’s NTDS settings in ADSI. Details are in http://www.ryanjadams.com/2010/05/turbo-charge-active-directory/#axzz2PaGkqbUu. I have modified the attribute “options” for each server

a. Open ADSI edit

b. Connect to and expand the Configuration container

c. Navigate to Sites-> Sitename-> Servers-> servername-> NTDS Settings

d. Right click and select properties for each manual connection object in this folder.

e. Select the “Options” property. Note that if the value is 1 then it is an intrasite connection object and owned by the KCC. If the value is 5 then it is an intersite connection object and owned by the KCC. If it is one of these values and owned by the KCC then do NOT change it. It should be changed at the site level instead, otherwise if you change the value on a connection object that is owned by the KCC you force it out of KCC control and the KCC will no longer manage it.

f. In the edit attribute box, change the value to 12.

How to find out which device is the domain account getting locked out from?

2013-12-24T11:52:00.000-08:00

Great tool from Microsoft to use is LockOutStatus [Account Lockout and Management Tools]
http://www.microsoft.com/en-us/download/details.aspx?id=18465
These tools are mainly for Windows XP and Server 2003, but LockOutStatus.exe works on Server 2008 R2 Active Directories too.

Here you can see the list of you AD servers and last time that particular account was locked out.
View the servers Security Event Viewer on the Active Directory computer where the account was last locked out and filter for Event ID: 644 [2003] or 4740 [2008]

The Caller Machine Name is will indicate where the incorrect credentials are originating from

2) Find out what process/application is using the incorrect credentials.

If the Caller Machine Name is coming from a known server such as Exchange, to check is if the user has additional devices such as iphone, ipad which perhaps do not have the latest users credentials.

Microsoft Exchange 2010 - Monitoring Search indexer Crawl status

2013-08-09T15:56:00.005-07:00

Search indexer full crawl can take a long time depending on the size of the database. At the mailbox level, the powershell command can be used in verbose mode.

C:\Windows\system32>test-exchangesearch -identity testuser1 -verbose

VERBOSE: [18:34:15.191 GMT] Test-ExchangeSearch : Active Directory session settings for 'Test-ExchangeSearch' are: View

Entire Forest: 'False', Default Scope: 'domain.com', Configuration Domain Controller: 'dc.domain.com',

Preferred Global Catalog: 'dc.domain.com', Preferred Domain Controllers: '{ dc.domain.com }'

VERBOSE: [18:34:15.191 GMT] Test-ExchangeSearch : Runspace context: Executing user:

Domain.com/Admin Accounts/Individual/admin, Executing user organization: ,

Current organization: , RBAC-enabled: Enabled.

VERBOSE: [18:34:15.191 GMT] Test-ExchangeSearch : Beginning processing &

VERBOSE: [18:34:15.191 GMT] Test-ExchangeSearch : Instantiating handler with index 0 for cmdlet extension agent "Admin

Audit Log Agent".

VERBOSE: [18:34:15.238 GMT] Test-ExchangeSearch : Current ScopeSet is: { Recipient Read Scope: {{, }}, Recipient Write

Scopes: {{, }}, Configuration Read Scope: {{, }}, Configuration Write Scope(s): {{, }, }, Exclusive Recipient Scope(s):

{}, Exclusive Configuration Scope(s): {} }

VERBOSE: [18:34:15.238 GMT] Test-ExchangeSearch : Searching objects "testuser1" of type "ADUser" under the root "$null".

VERBOSE: [18:34:15.253 GMT] Test-ExchangeSearch : Previous operation run on domain controller 'dc.domain.com'.

VERBOSE: [18:34:15.253 GMT] Test-ExchangeSearch : Processing object "domain.com/Users/testuser1".

VERBOSE: [18:34:15.316 GMT] Test-ExchangeSearch : Searching objects "MBX06_DB01" of type "MailboxDatabase" under the

root "$null".

VERBOSE: [18:34:15.316 GMT] Test-ExchangeSearch : Previous operation run on domain controller 'dc.domain.com'.

VERBOSE: [18:34:15.316 GMT] Test-ExchangeSearch : Admin Audit Log: Entered Handler:Validate.

VERBOSE: Testing Exchange Search.

VERBOSE: [18:34:15.331 GMT] Test-ExchangeSearch : Resolved current organization: .

VERBOSE: [18:36:16.872 GMT] Test-ExchangeSearch : Getting catalog state for Mailbox Database "MBX_DB01"...

VERBOSE: [18:36:16.872 GMT] Test-ExchangeSearch : Catalog state: Crawling.

VERBOSE: [18:36:16.872 GMT] Test-ExchangeSearch : Getting the indexing state for mailbox "testuser1" ...

VERBOSE: [18:36:16.887 GMT] Test-ExchangeSearch : Mailbox state: NotStarted, Crawl start time: 0001-01-01T00:00:00.

VERBOSE: [18:36:16.887 GMT] Test-ExchangeSearch : Getting the message indexing state...

VERBOSE: [18:36:16.887 GMT] Test-ExchangeSearch : Document state: Queued, state = 0.

VERBOSE: [18:36:16.903 GMT] Test-ExchangeSearch : TimeSinceLastNotificationWasIndexed for MDB MBX06_DB01 is 0 seconds.

The mailbox state will tell if this mailbox has been crawled or not, in this case, the crawl has not yet started for this mailbox.

At the mailbox database level or server level, we can use the performance monitor to monitor the status of the crawl.

Most of the counters can be found under MSExchange Search Indices

Some of the common ones I used are

Full Crawl Mode Status - The Value is 0 when the crawl is completed

Number of backlogged items added to the retry table - if the value is more than 30000 after the crawl is complete, restart the Microsoft Exchange Search indexer service

Number of failed mailboxes - this will tell how many mailboxes failed to be crawled

Number of mailboxes left to crawl - if the crawl is still running, how many mailboxes are left to crawl

Microsoft Exchange 2010 - Search does not return all the results

2013-08-09T15:41:00.003-07:00

We recently had a problem with our Exchange 2010 search indexer where it would only return the results from today forward but none of the older results. When we checked the event log event ID 107 was reported:

Exchange Search Indexer has temporarily disabled indexing of the Mailbox Database Database1 (GUID = 32299692-d5af-4415-8ca0-36f9d485ddb6) due to an error (Microsoft.Mapi.MapiExceptionNotFound: MapiExceptionNotFound: Unable to update CI Retry table (hr=0x80004005, ec=-1601)

Diagnostic context:

Lid: 1494 ---- Remote Context Beg ----

Lid: 28242 StoreEc: 0xFFFFF9BF

Lid: 59457

Lid: 22610 StoreEc: 0xFFFFF9BF

Lid: 52115 dwParam: 0x1148628

Lid: 4000 StoreEc: 0xFFFFF9BF

Lid: 16793 StoreEc: 0xFFFFF9BF

Lid: 14126 StoreEc: 0xFFFFF9BF

Lid: 1750 ---- Remote Context End ----

Lid: 10098 StoreEc: 0xFFFFF9BF

Lid: 20226

Lid: 15346 StoreEc: 0xFFFFF9BF

at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)

at Microsoft.Mapi.ExRpcAdmin.CiUpdateRetryTable(Guid mdbGuid, Guid instanceGuid, UInt32[] documentIds, Guid[] mailboxGuids, Int32[] hresults, Int16[] initialStates)

at Microsoft.Exchange.Search.PropertyStore.UpdateRetryTable(PropertyStore ps, UInt32[] docIds, Guid[] mailboxGuids, Int32[] hresults, Int16[] initialStates, Object& unusedOut1, Object& unusedOut2)

at Microsoft.Exchange.Search.PropertyStore.DoRetriableOperation[InParameter1Type,InParameter2Type,InParameter3Type,InParameter4Type,OutParameter1Type,OutParameter2Type,ReturnType](Guid mdbGuid, InParameter1Type inParameter1, InParameter2Type inParameter2, InParameter3Type inParameter3, InParameter4Type inParameter4, OutParameter1Type& outParameter1, OutParameter2Type& outParameter2, OperationDelegate`7 operationDelegate)

at Microsoft.Exchange.Search.PropertyStore.UpdateDocumentStatesOnBatchCompletion(BatchInformation batchInformation)

After this error, it was almost that the server reached a checkpoint and would not return any older items in the search result. We restarted the Microsoft Exchange Search Indexer and the Microsoft Search (exchange) service but it did not resolve the problem. To test whether there are any corruption problems, we ran the script for testing exchange search

C:\Windows\system32>test-exchangesearch -identity Username -verbose

Everything seemed to be normal, except for the search for older items.

Exchange provides a range of scripts that are present in the \Scripts directory. By Default this is C:\Program Files\Microsoft\Exchange Server\V14\Scripts for Exchange 2010. Exchange 2010 SP1 introduced the Content Index Troubleshooter Script – Troubleshoot-CI.ps1. This script can monitor and perform troubleshooting on a server’s CI catalogs and can detect and resolve symptoms such as

Deadlock: Exchange Search deadlocks waiting on threads from MSSearch
Corruption: One or more search indices are corrupted
Stall: Similar to deadlock in that the indices are not getting updated
Backlog: The Search catalog is backlogged resulting in missing index searches

To fix this problem we had to rebuild the search indexer catalog. The easy way is to use the script under the scripts folder

ResetSearchIndex.ps1 -force -all

This will delete all the catalog folders and rebuild the catalog. The search will not be accessible until the catalog crawling is complete. We can monitor the search index to see how many mailboxes are left to crawl (I will write a blog about that soon). I do recommend running the rebuild after business hours if the database is anything bigger than 500GB as it does take a long time (>5 hours) to rebuild the catalog and may cause outlook to be lag if you are running in online mode.

Netapp Snapmanager for SQL - setting different snapshot retention settings for different backup jobs

2013-08-08T16:53:00.001-07:00

Netapp Snapmanager for SQL is an amazing product. I personally love it. Snapmanager allows you to SQL aware snapshots for the databases residing on the Netapp LUNs. It goes without saying that you have to follow the best practices for setting up the LUNs with databases and log files on separate LUNs for each instance to have this work correctly. The Jobs can be scheduled as SQL jobs or windows tasks. We can set retention on these backups according to the number of backups and number of days.

Now when the retention is simple, something like 7 days or 10 backups, it is fairly easy to specify that. However, the retention for our backups was a little more complicated than that. We required backups to occur every four hours at the business hours and retain those backups for 7 days and than have a once a day backup with a retention of 42 days. So in other words a four hourly backup for 7 days and a daily backup for 42 days needed to be the retention period. This is where the management groups come into place. Snapmanager for SQL has whats called backup management groups that allow you to set up different retention period for different backup jobs.

So for the 4-hourly backups we used the 'standard' management group. Under 'Backup' > 'backup and verify' > 'backup management group' - the option 'standard' is selected from the drop down.

For the daily backups we change this option to 'daily'. So when Snapmanager deletes the backups older than 7 days for the 4 hourly backups, it understands the daily management group and does not delete the backups under the 'daily' management group. That way we can have separate retention for different backup jobs.

GAL is blank after the mailboxes are moved to exchange 2010 from Exchange 2007

2012-12-10T19:52:00.001-08:00

We are in the process of migrating from Exchange 2007 to Exchange 2010. We have segregated AD because we host exchange for multiple companies. We use ACLs and msExchQueryBaseDN attribute for the segregation. Each company has a separate GAL, Address list and OAB. After the mailboxes were moved, the GAL appeared to be blank although the names could be resolved by outlook. After a lot of struggle to fix it, we found out why.

Exchange 2010 SP2 uses Address book policies to segregate the GAL access instead of the msExchQueryBaseDN attribute. So first of all we create an address book policy for each client. You can do so in the Exchange MMC> organization management > mailboxes > address book policies tab > new address book policy. Create it as shown in the image below. This is required if we do not want the users to have access to all the GALs.

Apply the policies to all the users under COMPANY1. You have to change it for individual user if you want to do in exchange MMC. You can do it by right clicking properties of the user mailbox> mailbox settings tab> select address book policies and click properties> select address book policy. Possibly can be scripted as well.

Step 2: check the permissions on the address lists for the client.

Open adsiedit > configuration > CN = configuration, DC=domain,DC=com > CN=services > CN=Microsoft Exchange > CN= i-worx > CN=Address lists container

1. expand CN= all address lists. Right click the address list for COMPANY1. All the users in COMPANY1 should have the following permissions

a. READ

b. Open address list

(see image below)

2. Under CN=Address lists container, now expand CN= All global address lists, Right click the GAL for the COMPANY1.The CLIENTCODE users group should have the following permissions

a. READ

b. Open address list

(see image below)

LAST STEP

Clear the msExchQueryBaseDN attribute for all users for company1.

What is Cloud Hosting?

2012-08-11T11:50:00.001-07:00

‘Cloud’ is the new buzz word in the world of business. Many businesses are moving towards cloud hosting and cloud hosting has become extremely popular in the past few years. The main concept behind cloud hosting is that the resources required for the infrastructure are spread across many servers probably across the world and are rendered on a per need basis. Cloud hosting comes with the benefits of reducing the downtime in case of hardware failures.

In cloud hosting, because the resources can be allocated on a per need basis, scalability becomes a huge advantage over dedicated systems. Also, this also means that load can be managed easily as load can be split on to other servers. Cloud hosting comes with the stress-free management without having to worry about the underlying hardware hosting your environment. Businesses have been moving towards cloud hosting to get rid of the overhead involved in management and maintenance of the dedicated environments. A typical example of cloud hosting is IaaS (Infrastructure as a service) offered by Amazon. Companies and businesses can have their environments hosted on the Amazon servers which are located all over the world. Amazon has been doing IaaS for a few years.

Each server in the cloud helps in carrying out a particular function. In case, one of the servers fail, it is temporarily taken out of the resources pool and another server kicks in to take over the functions of the failed server. Same is true if a site where these servers are located. If the site goes down because of a disaster, another site takes over the functions. The site recovery feature of course depends on the hosting provider and the money paid for the services.

This will also happen in case of an overload condition. The quality of hardware and the quality of service provided (meaning features like disaster recovery, zero downtime) depends on the hosting provider. In case of enterprise level cloud hosting, the hosting providers are generally utilizing high end hardware with Vmware which is considered to be the Rolls Royce version of virtualization and deliver extremely reliable cloud hosting services.

Cloud hosting has come a long way and has completely surpasses dedicated server environment and why not – you have multiple servers at your disposal whenever you need instead of a single server in a dedicated environment. In cloud hosting environment, because of the scalability and reliability, the businesses have the benefit of coping up with emergency situations as opposed to dedicated environments. The reduction in downtime and manageability are also some of the main reasons ahy businesses choose cloud hosting. With the cloud hosting services improving day by day, the cost of these services has come down in the last 4-5 years. Most businesses today want a hassle-free working environment where they can have peace of mind and focus on the business instead of dealing with the everyday dilemma of dedicated environments. The day is not far away when cloud hosting will be the choice of all businesses – big or small.

Solved - Cannot upgrade Xenapp 6.0 to Xenapp 6.5

2012-07-28T19:29:00.000-07:00

I was super excited about the new Xenpp 6.5 with so many improvements focused on user experience and more. We have about 40 Xenpp 6.0 servers with over 500 users. So testing this was crucial as once you uninstall Xenapp 6.0 and install Xenapp 6.5, the only way of going back is restoring from the backup. I tried the upgrade utility provided by Citrix to no avail (http://support.citrix.com/article/CTX130614/). Xenapp 6.0 appeared to uninstall just fine but installation of Xenapp 6.5 failed. As many of you might already know there is no direct upgrade path from Xenpp 6.0 to Xenapp 6.5. The utility is merely a script that uninstalls all the components of Xenapp 6.0 and installs 6.5 and joins to the new Xenapp 6.5 farm.

If i installed Xenapp 6.5 on a brand new server, it worked fine. But when i installed it on a server with Xenapp 6.0, using utility or manually uninstalling and installing it, it failed. After spending hours on research, i still couldnot make it work. There were many other people having the same problem, different solutions worked for different people. However, for me it was no luck. I finally managed to figured out that the problem was something to do with the Xenapp 6.0 not uninstalling properly, although all the components uninstalled successfully.
Out of those 15 times i tried, this is how it worked for me

1. unjoin the Xenapp 6 server from the farm and uninstall Xenapp 6.0
2. Then uninstall all the citrix components.

it might ask you to reboot after some of the uninstallations. I just rebooted it once after all the components were uninstalled.
3. after the reboot, delete all the folders under C:\Program files (x86)\citrix\ except C:\Program files (x86)\citrix\Xentools (only if the VM is a Xen VM - you will have the Xen tools installed). You might get an access denied error for the folder C:\Program files (x86)\citrix\farmGPO. Take the ownership of the folder and then delete.
4. Next, open regedit, and delete all the keys under HKLM\software\citrix

5. reboot
6. Now install Xenapp 6.5 and the installation will succeed

Hope this helps :)

Netapp - How to move a data from a non-qtree folder to a qtree for CIFS

2012-07-25T22:13:00.000-07:00

Following is the scenario

Lets say you have a CIFS share for a folder that is not a qtree, so you cannot impose a Quota on it. To impose a quota, you will have to move the share to a qtree. You can either create a qtree under another volume or the same volume. The following article explains both ways to achieve this.

How to resolve this?

The source folder /vol/volume1/folder1 on filer1 is shared as \\filer1\folder1$. First remove the existing share.
Rename the source folder to folder1.old (you may leave the folder with the existing name if you do not want to create a qtree with the same name or if the qtree will reside under another volume).

Now create a qtree called 'folder1' under volume 'volume1'. (or a different volume 'volume2')
NOTE: Please be careful with the case of the qtree name and the command as it is case sensitive.

To copy all the data from the source folder to the destination qtree we can use the 'ndmpcopy'. This command also copies the ACLs for the data.

The Syntax for the command is
ndmpcopy Source_location Destination_location

Use the following commands to copy the data with the ACLs to the newly created qtree

1. if the source folder has been renamed
ndmpcopy /vol/volume1/folder1.old /vol/volume1/folder1

where volume1 is the name of the volume under which the source folder and the new qtree resides.

2. If the source folder resides on a different volume than the new qtree

ndmpcopy /vol/volume1/folder1 /vol/volume2/folder1

where volume1 is the volume under which the source folder resides and volume2 is the volume under which the new qtree has been created

After the ndmpcopy is complete, share the newly created qtree. Ndmpcopy may take time to complete depending on the size of the data to be copied.

Citrix Xenapp 6.0 gives an error when users logon to the citrix web interface "An error occured when making the requested connection"

2012-07-25T20:52:00.001-07:00

This error can occur because of a number of reasons concerning the Xenapp 6.0 server. Besides the obvious reason - the server does not have internet connectivity or is shut down, the tricky cause can be the load value for the server. This can be checked using the following command on the culprit Xenapp 6.0 server

QFARM /LOAD

This command will give you the list of the servers in the farm and their load value. If the load value of the server is 20000, it will refuse any further connections and give the above error. One more symptom is that you might not see the server in the list at all. This can be corrected by doing the following

1. Open Command prompt as administrator

2. Run the command "net stop imaservice" to stop the imaservice

3. Run the command "net start imaservice" to start the imaservice

NOTE: this can also be done through services.msc console but i have experienced that the imaservice tends to hang when restarted through services console.

Once the service is restarted. The server will start accepting connections again.