Notes

Setting up Citrix Receiver on Ubuntu

2015-02-25T12:40:00.000-05:00

The goal of this project is to make use of the old laptops so that we can run Citrix Receiver on it, minus the headache of Windows patching, and antivirus.

So here is what I have to get it started.

an old laptop
Ubuntu 14.04 LTS 32-bit
Citrix Receiver version 13.1.0

The reason that the 32-bit Ubuntu was chosen was mainly the compatibility issue of 64 bit Citrix Receiver. I started with 64 bit, but the errors were horrible. It seems that the 64-bit Citrix Receiver was trying to use some 32-bit library in a 64-bit Ubuntu OS. So I reinstalled with Ubuntu 32-bit, and the installation process of 32-bit Citrix Receiver was so smooth that I was thrilled to see it.

And the .dev version of the Citrix Receiver failed with an error during installation as well. But the .tar.gz file went very smooth, so I am going to stick with the tar.gz file.

Here are the commands that I used to run to install Citrix Receiver 13.1.0.

cd /tmp/citrixclient

tar xzvf linuxx86-13.1.0.285639.tar.gz

sudo ./tmp/citrixclient/setupwfc

sudo ln -s /usr/share/ca-certificates/mozilla/* /opt/Citrix/ICAClient/keystore/cacerts/

sudo c_rehash /opt/Citrix/ICAClient/keystore/cacerts/

That is all for Citrix Receiver installation. In Firefox, before you can use the Citrix Receiver for the first time, you will have to activate it in Firefox.

Open the Firefox broswer, go to Tools->Add-ons
Change the option for Citrix Receiver for Linux from Ask to Activate to Always Activate. (Ask to Activate never works, because it never ask to activate). This needs to be done in each user account in Ubuntu, unless you set it in the skel profile, and all the new user created upon it will have the Always Activate set.

And that is it. You will be able to use this laptop to connect to Citrix sessions.

Next time I will show how to create a live Ubuntu image with Citrix Receiver installed, that can run from a USB.

How we separated our two office’s AD and Exchange

2014-01-08T12:09:00.000-05:00

Our company sold off a branch office last year, and we have been working on the separation of our business and our IT infrastructure. We are a Windows environment. Each office used to be a redundancy of each other, which means that each office has a full Active Directory forest, which contains domains controllers for root domain and all of its sub domains. Users in each office are in its dedicated site sub domain, with its own Exchange servers(in mix environment of Exchange 2007 and Exchange 2003). From the very beginning we came to an agreement with passive cleanup by having an identical AD forest and Exchange organization on each site, and when the cutoff time comes, we turn off the VPN network completely and start to clean up AD and Exchange by removing the objects in the other sites. It needs to be done at Exchange level, Domain level, and User level. We will use these names to in our case. I know, Microsoft doesn't support these type of removal, but to us, it is cheaper, less time, and sounded more straight forward. During the separation, we did learn a lot in a hard way.

root of the domain: core.com

Site North: NorthDC, NorthEX(Exchange server), external email domain is north-example.com

Site South: SouthDC, SouthEX(Exchange server), external email domain is south-example.com

We are in the south, so I will state the steps that I took from the point of SouthEX, SouthDC’s view. But in Site North, it needs to have the same things done to the SouthDC and SouthEX objects.

Before the disconnection, make sure you have redundant domain controllers in your forest. For example, if you have a single root domain controller in your site, you might want to create a second one before the disconnection. Never have a single domain controller in your organization.

After the disconnection, the first issue we found was that users in each sites won’t be able to email each other, and all the emails will get stuck in the unreachable queue. This should be expected, since north-example.com and south-example.com are still under the same Exchang organization.

Exchange Level: We will need to remove the orphaned Exchange objects that is meant for NorthDC and NorthEX

Remove Recipient Policies. Remove any policies that has anything to do with the other office’s domains. In SouthEX, we removed policies that applied to NorthEX. Otherwise it won’t let you remove north-example.com from Accepted domains in SouthEX Exchange server.
Removed north-example.com from Accepted Domains in SouthEX Exchange server,
Remove any connectors and routes to NorthEX
Remove NorthEX Exchange servers in SouthDC via ADSIEDIT.msc.

Exchange servers are under:

Configuration

CN=Configuration,DC=core,dc=com

Remove Exchange attributes for users in NorthDC

How to Use the Remove Exchange Attributes Option

There might be some emails already got stuck in an unreachable queue. You may issue the retry-queue PowerShell command to resubmit those emails:

retry-Queue -Identity "ServerName\Destination" -Resubmit $true

Domain Level: We need to remove the orphaned Domain Controller completely

Using Ntdsutil.exe to transfer or seize FSMO roles to a domain controller
You may need to configure a new authoritative timerver in the domain.
Remove old computer account by using "Active Directory Sites and Services" tool.
Remove old DNS and WINS records of the orphaned Domain Controller.
Use "ADSIEdit" to remove old computer records from the Active Directory:

OU=Domain Controllers,DC=domain,DC=local
CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=local
CN=Domain System Volume (SYSVOL share),CN=File Replication Service,CN=System,DC=domain,DC=local

Force Active Directory replication by using "Repadmin.exe" tool:

User level: Most of the users are ok at this time, however, certain users might see a NDR when sending emails to north-example.com with the following error:

IMCEAEX-_O=EXCHANGE+20Org_OU=First+20Administrative+20Group_cn=Recipients_cn=user@north-example.com

#550 5.1.1 RESOLVER.ADR.ExRecipNotFound; not found ##

This happens when Outlook use email address auto-complete cache for north-example.com, which was stored with X.500 address format when under the same Exchange organization. This old X.500 address was causing Exchange to reject the emails. Since Exchange 2000, the X.500 address has been stored in the leGacyExchangeDN attribute in AD. This value was set when a mailbox was created in Exchange 2003, includes the name of the Exchange administrative group where the mailbox belongs. The fix to it is either create a X.500 proxy address for North-example.com users, or clear the Outlook email address auto-complete cache.

It has been a month after the separations of two offices, we haven’t seen other issues yet. Drinking a cup of coffee, and crossing my fingers now ...

ESX fails to boot error: Error 15: Could not find file

2012-06-27T15:35:00.000-04:00

Recently when I used VMware Update Manager to patch a newly rebuilt vSphere 4.1, it couldn’t boot up when it was rebooting. Using DRAC to login into its console and found this error.

I did a search and found this in VMware forum communities. VMware KB1004574 kind of mentioned the reason behind this: /boot/partition ran out of space. My /boot/ at that time had 16 MB, and during patching it needs at least 24MB to boot properly, which is the size of the new initrd-2.6.18-xxx.ESX.img. When space is less than 24MB, the new initrd-2.6.18-xxx.ESX.img file (in my case, it was initrd-2.6.18-274.ESX.img) can’t be created, which causes initrd has a broken link to a non-existing file.

If you are able to login into Troubleshooting Mode, it would be an easy fix. In /boot, with command ls -l, you will able to see the red broken link. All you need to do is relink it to a valid existed initrd-2.6.18-xxx.ESX.img by using ln -sf initrd-2.6.18-238.ESX.img. Reboot the server now and it should come back up. However, in my case when I try to enter into Troubleshooting mode, it ran into an infinite loop of signature mismatch error.

Then I found this VMware KB1007908, and I chose the second option for me, which worked perfectly well. It uses a live CD to boot the server into Linux, and fixes the host’s grub.conf from a chroot environment to point to a valid img file. The main concept here is to have initrd file linked to a valid img file to make the host bootable. After the host is able to boot up, we will have to clean up the /boot/ partition, and run esxcfg-boot -b to fix the normal boot image and esxcfg-boot -t to fix the troubleshooting mode image. Here are the command I used when I boot into the live CD:
fdisk -l | more
mkdir /mnt/esx
mount /dev/sda2 /mnt/esx
mount /dev/sda1 /mnt/esx/boot
mount /dev/sda5 /mnt/esx/var/log
chroot /mnt/esx
bash
ln -sf initrd-2.6.18-238.ESX.img initrd.img

After the host booted up, I ran the following commands.
mv /boot/initrd-2.6.18-194.ESX.img /tmp/
esxcfg-boot -b #this will create initrd-2.6.18-274.ESX.img
ln -sf initrd-2.6.18-274.ESX.img initrd.img
mv /boot/initrd-2.6.18-238.ESX.img /tmp/

Now, everything is back to normal.

How to make DRACT run in a 64-bit system

2011-11-02T13:52:00.000-04:00

Dell™ Remote Access Configuration Tool (DRACT) is a tool that provides a central console to discover and configure Dell Remote Access Controller (DRAC) for all the servers in your network. It automates some of the common repetitive tasks, such as firmware upgrade, and AD authentication configuration.

As of Nov 2011, the latest version of DRACT is 1.0, which can be run in 32-bit Windows systems only. Nowadays, 64-bit systems are commonly used everywhere. Setting up a system just DRACT is kind of wasting resource. After some research, I found out that we CAN make it run in 64-bit environment.

You can have DRACT installed into a 64-bit system without problems. The symptom showing it is not working is that it can not discover any DRAC in your network. Here is what you need to do to make it work.
1) You need a file: corflags.exe in Microsoft SDK. You can install Microsoft SDK, or you can just copy corflags.exe from a system that has Microsoft SDK installed. Corflags.exe is a Conversion tool allows you to configure the CorFlags section of the header of a portable executable image.
2) Run the following command of the ract.exe executable file:

corflags.exe /32bit+ “D:\Program Files (x86)\Dell\RACT\RACT.exe”

The /32bit+ sets the 32-bit flag, so that the executable file always runs under WOW64 in a 64-bit system. (WOW64 is a compatibility environment that enables a 32-bit application to run on a 64-bit system. WOW64 is included in the system.)

That is it. Now DRACT will be able to discover the DRAC with the IPs that you provide in your network.

How to Cancel a Stuck VMware Tools Install

2010-11-17T13:48:00.000-05:00

Some co-worker ran into a problem with not able to VMotion a VM off a vSphere host. The error is “The virtual machine is installing VMware Tools and cannot initiate a migration operation.” Right click to “End VMware tools install” won’t help. After some digging, we found it can be resolved in two ways.

1) Click on the VM, go to the toolbar on the top in vSphere client, click on the CD/DVD icon -> CD/DVD Drive 1-> disconnect. This should resolve the issue and you should be able to VMotion this VM after this. This mostly happens to a Linux VM after it upgrades its VMware tools.

2) Found this in Bob Plankers’ The Lone Sysadmin blog. It works great too.
First, you need the ID of the VM (all on one line if it wraps):
/usr/bin/vmware-cmd /vmfs/volumes/datastore-name/vm-folder/vmx-file.vmx getid
Then you can do a:
/usr/bin/vmware-vim-cmd vmsvc/tools.cancelinstall idnumber

P2V fails at 94% after finished cloning disks

2010-09-08T16:45:00.000-04:00

Issue:
Physical server rebooted itself when VMware Converter reached 94% or 98%. By the time of its crash, it had already finished cloning its disks, and failed while it was trying to connect to the VM or reconfigure the VM. And the VM always goes to BSOD when it boots up.

Cause:
The cause the reboot of the physical server is unknown. It involves the physical hardware, the OS, and the application, and none of them can pinpoint where the failure comes from.
The cause of the BSOD in the VM is the virtual disk controller driver isn’t in place and the VM isn’t configured to be ready to run in a virtual environment.

Solution:
In the VMware Converter, run “Configure Machine” and use the wizard to configure the VM. After it is finished, we should be able to power on the VM into Windows.

VMware Update Manager 4.0

2009-11-19T12:33:00.001-05:00

Can't download patchs via VMware Update Manager 4.0

My environment is known for having complex firewalls and proxy. Lots of time it takes lots of effort and time to make application works. This time is no exception for VMware Update Manager 4.0.

The installation process for VUM 4 was very straight forward. Then it comes to the hard part. It can't download any patches for ESX hosts nor for VMs. After lots of testing, and some help from VMware's communities forum, finally I made it work. Here is what I have done.

1) Have proxy server opened up HTTP/HTTPS to the following 3 domains: *.vmware.com, *.shavlik.com, and *.microsoft.com. The last domain is not for downloading patches, but for Test Connection since that is one of the websites that it use to test connectivity.

2) Set up proxy setting. Do not put http in front of your proxy server name in the box. Make sure you have the right authentication information in it if it is required.

If you are seeing some errors like this, it could be caused by your proxy setting.

httpDownload, 730 Error 12007 from WinHttpSendRequest for url https://hostupdate.vmware.com/software/VUM/PRODUCTION/index.xml

'httpDownload' 4284 ERROR [httpDownload, 726] Error 12029 from WinHttpSendRequest for url https://hostupdate.vmware.com/software/VUM/PRODUCTION/index.xml

Please check out the following KB for more information.
http://kb.vmware.com/kb/1012926

How to replace SSL cert in VirtualCenter

2009-06-05T15:10:00.005-04:00

When replacing a SSL cert, there might be chance to cause encryption problems between ESX and VC. Doing it improperly will lose permissions setting, and it will take a long time for it to get stabilized.

To generate a new SSL cert, we can follow the instruction here at KB 1009092, or in Leo's blog. The way they listed to install a new SSL cert could be working for someone, but it didn't work on many other folks like this one, and this one. Here I list some basic steps that needed to perform in order to have a smooth change.

1) Disable HA

2) Disconnect ESX from VC

3) Stop VC service

4) Replace SSL cert

5) Start VC service

6) Reconnect ESX to VC with root/pw

If it still have issues, like having an error of vim.faultlogin, we can remove VC agent, and delete vpxuser from SC. Reconnecting ESX to VC will recreate the user and reinstall the VC agent.

Here are a couple KBs and articles might be helpful with changing SSL cert:

Enabling Server-Certificate Verification for Virtual Infrastructure Clients

Configuring Custom SSL Certificates in VirtualCenter 2.5

ESX WRITE10 error

2009-04-29T12:44:00.004-04:00

Recently a WRITE10 error in one of my ESX host caught my attention, and it occurs more than 10 times every second.

Apr 29 12:01:10 cla1011 vmkernel: 11:22:45:34.946 cpu4:1077)WARNING: VSCSI: 5291: WRITE10 past end of virtual device with 29365, length 128

After search on Google and VMware's communities, and still could not find detail information and solution about it, I turned to VMware technical support. The technical support engineer sent me their internal KB.

Symptoms

Repeatedly logging messages similar to either of the following in /var/log/vmkernel (or /var/log/messages on ESXi):

Feb 5 15:44:46 USPLVS02 vmkernel: 63:05:31:58.181 cpu3:1129)WARNING: VSCSI: 5292: WRITE10 past end of virtual device with 33554432 numBlocks, offset 33554351, length 128

Feb 12 17:03:04 pa-tse-h02 vmkernel: 156:05:50:47.889 cpu0:1174)WARNING: VSCSI: 3430: READ10 past end of virtual device with 20971520 numBlocks, offset 20980737, length 16

These messages indicate that I/O is being attempted that is outside the boundaries of the virtual device (virtual disk). In layman's terms, the VM has a list of ten items, and the guest OS is asking for the 12th item on the list.

Resolution

To find out which VM is responsible for these, the World ID (WID) must be determined from the log messages. The WID is after the cpu specifier, and before the WARNING in the above messages. In the case of the WRITE10 message above, the WID is 1129; for the READ10 message the WID is 1174.

If you look in /proc/vmware/sched/cpu, then the vcpu column (first one) will list the number identified in the logs.

To determine the VM responsible if it is not running:

cat `ls -rt vmkern*` | less

Find the first instance of the log message (using "/WRITE10" or "/READ10" will likely find it for you very well) Then search backwards in the logs for the WID value (in less this can be done with "?". ex: ?1129 Note: It searches beginning just before the top line on screen. Press 'n' to find the next match.) and keep searching earlier in logs until you find something similar to:

Feb 12 16:51:55 pa-tse-h02 vmkernel: 156:05:39:38.873 cpu2:1173)Sched: vm 1174: 4836: adding 'vmm0:ProblemVMName': group 'host/user/pool0': cpu: shares=2911 min=0 max=-1

The text will show you the name of the problematic VM after the vmm entry. In this case, "adding 'vmm0:ProblemVMName'" shows that the VM causing the issue is named ProblemVMName.

If you look at the contents of the descriptor file for the offending Virtual Machine's disks, you will find an entry listing the number of cylinders for the virtual disk. As an example:

ddb.geometry.cylinders = "2088"

In this case, the virtual disk has 2088 cylinders. Running "fdisk -l" against the flat file of the virtual disk will return information similar to:

You must set cylinders.

You can do this from the extra functions menu.

Disk ANSGOOD-flat.vmdk: 0 MB, 0 bytes

255 heads, 63 sectors/track, 0 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

ANSGOOD-flat.vmdk1 * 1 2089 16779861 7 HPFS/NTFS

Partition 1 has different physical/logical endings:

phys=(1023, 254, 63) logical=(2088, 254, 63)

Note that in this case, the end value for the partition/disk set in the partition table is 2089, exceeding the number of cylinders set in the descriptor file. If this were proper, it would show 2088 as the end value, instead of 2089. The operating system, as a result of this incorrect partition table

Extending the VMDK just enough to allow it to contain the size of the partition table might fix this, but because it is an invalid conglomeration of settings, it can not be safely assumed that this can be fixed. Other possible means of fixing the behavior is to cause the partition table to fit within the disk, by correcting it's ending value. Moving the data to a new, properly configured disk/partition table/file system is the best bet as the state of the file system by trying to modify the VMDK or partition table isn't known, and may be damaged by the changes, or already is damaged. Give the customer the options, and let them choose how to handle the changes to their system, as they can best judge how they want to protect their data.

The solution to it is either extend the VMDK file or shrink the partition table. Extending seems safer than shrink. The solution that I chose was using VMware Converter. By the way, VMware Converter 4 is offering some cool features over the previous version 3.0.3.

How to upgrade HBA firmware

2009-04-17T12:14:00.005-04:00

In my case, I was to upgrade firmware on Emulex HBA for my ESX 3.5 Update 1 host. Here is my hardware:

Server: Dell PowerEdge 6950

HBA: Emulex LPe1150-E

Download HBAnyware Library and Utilities Kit at Emulex Download section for Dell.

Download Emulex LPe1150-E latest firmware (as of this writing, it is 2.80A4)

Upload both the HBAnywhere and firmware to the ESX host

Verify the firmware version before the upgrade

cat /proc/scsi/lpfc/*

Install HBAnywhere by running

rpm -ivh elxvmwarecorekit-2.1a40-1.i386.rpm

The binary executable files will be store at /usr/sbin/hbanyware/

List the HBA that are manageable by HBAnywhere

/usr/sbin/hbanyware/hbacmd ListHBAs

Upgrade firmware in HBA

./hbacmd Download 10:00:00:00:c9:63:f4:19 /location_of_firmware(etc, /root/wf280a4.all)

Now we can verify the upgraded firmware version

cat /proc/scsi/lpfc/*

VMware VIMA

2009-04-08T14:03:00.002-04:00

While I was searching for a syslog appliance, ran across Duncan Epping's blog about VMotioning Service Console. Following his information and went over to VMware's VIMA webpage, downloaded it and gave it a try. I got to say it is like a bomb. Got to love it. It give ESX admins another central point of management of ESX, besides vCenter, via command lines. William Lam's VMware ESX and ESXi Scripts & Resources even gives some great examples on how to make use of this great appliance. With RCLI & VI Perl Toolkit integrated into the applicance, ESX admin now can make use of it from one central place. VIMA still got lots of places needed to improve, but it is really a greate starting point.

ESX 3.5 Update 4

2009-03-31T14:04:00.003-04:00

Just surprisingly found out VMware just released ESX 3.5 Update 4 yesterday on 30 Mar 2009. A couple months back, a consultant told me that VMware ESX 3.5 Update 3 was the last VI3 updates, and the next release is going to be ESX 4, or vSphere. Well, we will have to wait for a while before ESX 4 come out. It is not like I can't wait for ESX 4 coming out (well, maybe I do), I just wish VMware does a better job in QA. There is no need to have a buggy major release. I think I can wait.

After glancing at the release note, I found a couple things that I don't want to use ESX 3.5U4 in my production environment. Using these will japordize my currently stable environment. Normally when VMware has new update release for ESX, the first thing I would check out is What's New, which will contain the major change. The second thing is Known Issues, which will determine how soon I will be putting it into our production environment.

In this realse ESX 3.5U4, What's New mainly brings out more hardware support, and Network drivers, HBA drivers updates, along with a couple newer version of OS support. What really stop me from putting ESX 3.5U4 into production is the Known Issues.

And here are the major Known Issues that I don't like.

x64-based versions of Windows Vista and Windows Server 2008 guest operating systems require Microsoft hotfix x64-based versions of Windows Vista and Windows Server 2008 guest operating systems without Microsoft hotfix http://support.microsoft.com/kb/950772 might encounter a situation where the guest operating system stops responding and returns the following error:

MONITOR PANIC: vcpu-3:ASSERT vmcore/vmm/cpu/segment.c:430

Linux guest operating systems lose network connectivity after automatic tools upgrade

If the version of VMware tools in a Linux guest operating system becomes out of date and an automatic tools upgrade is performed, the guest operating system loses network connectivity. After an automatic tools upgrade, the guest operating system stops the network service and does not restart the service automatically after tools upgrade.

Workaround: Restart the network service in the guest operating system manually or reboot the guest operating system after an automatic tools upgrade.

The cursor disappears after VMware Tools has been configured

This issue has been observed on ESX Server 3.5 Update 4 with SUSE Linux Enterprise Server 8 (with or without SP4) guests. Immediately after you configure VMware Tools, the mouse cursor is not visible.

Workaround: Reboot the virtual machine.

Virtual Machines Might Not Power on After a Failover When the Host Is Isolated

Virtual machines might not start after a failover when a host is isolated and the isolation response is set to Guest Shutdown, which is the default configuration for a cluster. This might occur on clusters with fewer than five nodes and can happen on virtual machines
that take more time to complete the guest shutdown.

Workaround: Set the Isolation Response to either Leave powered on or Power off for clusters which have fewer than five nodes. To set the Isolation Response for a virtual machine, select the cluster, click the Edit Settings link, and select Virtual Machine Options under VMware HA. From the Isolation Response pop-up menu, select either Leave powered on or Power off options for the specific virtual machine.

There might be more dislikes in the Known Issues sections, since here is just what I have at the first glance. Another thing that make me think twice before an upgrade is the VMware tools upgrade. It is painful for Windows and Linux. Well, back to reality. All of these known issues wont' prevent me from installing it in my lab.

Could not power on VM II

2009-03-30T14:22:00.007-04:00

Here is a revisit the problem with not being able to power on a VM. In my previous post, Could not power on VM, it use a Linux command to kill the process so that the locked swap file can be release. Today I found an even easier way to fix the problem.

In my situation, I have an ESX server has some physical problem, which I haven't figured out what caused it yet. Basically two hostd and one vpxa processes are taking up 100% usage of CPU and 95% of 800MB memory and 50% of 1.6GB swap in service console. Contacted VMware technical support, and after he tried a couple things that I did to kill the hostd processes and vpxa without success, he gave me a solution, which is to reboot the ESX server. That is right, the solution is to reboot the ESX. In VC, the ESX has a "not responding" status. I have to export a list of the VM running in this server, record what datastore these VMs reside in. After that, I disconnected the ESX, and removed it from VC. Now I need to use RDP or SSH to connect to VM, and to shutdown the VM so that I can register the VMs in another ESX and power it on. Most of the VMs are working fine. But there always one or two VMs doesn't behave the way they are supposed to. One VM gave me an error of "Could not power on VM: No swap file. Failed to power on VM.". When I tried to remove the file from service console, it gave me an error of "Device or resource busy". The service console in this ESX isn't 100% fully functioning. "ps -ef | grep VM_name" doesn't returned anything. It means, killing the process won't work in this case. The solution I found is to rename the VM's vswp file.

Error: Could not power on VM: No swap file. Failed to power on VM.
Solution: Rename to vm_name-xxxxxx.vswp file name to something else, and VM can be powered on.

Here are two good articles about how to kill a VM: Gabe's virtual world and Technoblog. The reason I mentioned above is basically the last resort.

NetApp ESX Host Utilities Kit 6.0

2009-03-13T13:45:00.004-04:00

NetApp is going to have an ESX Host Utilities (6.0 replease) vCenter Plugin, with release date dependent upon VMware official release of vSphere. It was NetApp's showcase in VMworld Europe 2009.

This Host Utilities needs only a single installation to vCenter as a plugin instance, eliminates the hassle of the needs to install it into every ESX host.

Check out the video here

Improving ESX performance by unloading USB

2009-03-11T15:13:00.002-04:00

Many systems have shared interrupts between the USB controller and other components. In order to reduce any performance impact due to the shared interrupts, we can configure the ESX to avoid loading the USB drivers.

Configure the USB controller setting in BIOS to enable USB, so that the system supports USB devices both during and after ESX boot process even if USB drivers are not loaded by the ESX software.

Remove the following USB modules aliases from /etc/modules.conf

Save /etc/modules.conf and restart the system

This VMware KB 1290 has more detail information
on how you can tell if your system has shared interrupts.

Disabling ESX WebAccess

2009-01-27T12:51:00.004-05:00

Haven't thought about it, but it is interesting to know how to do that. Here is a discussion about it in VMware communities. Since vmware-hostd is using port 80 and 443, blocking firewall ports don't seem to be a right way to do it. Replacing the index page looks like a right way to go.

Another ESX bug

2009-01-21T13:14:00.004-05:00

I was planning to upgrade my VirtualCenter from version 2.5 Update 1 to version 2.5 Update 3, and then I received an email from VMware. This email states there is an issue with ESX 3.5 U3 could adversely effects my environment with I/O failure on SAN LUNs. I guess ESX 3.5 U3 can’t go in my environment, and I won’t be getting any benefits from VC 2.5 U3. I told my boss to hold off on this upgrade to VC 2.5 U3.

Here is the detail of this email:

ISSUE DETAILS:

VMware ESX and ESXi 3.5 U3 I/O failure on SAN LUN(s) and LUN queue is blocked indefinitely. This occurs when VMFS3 metadata updates are being done at the same time failover to an alternate path occurs for the LUN on which the VMFS3 volume resides. The effected releases are ESX 3.5 Update 3 and ESXi 3.5 U3 Embedded and Installable with both Active/Active or Active/Passive SAN arrays (Fibre Channel and iSCSI).

PROBLEM STATEMENT AND SYMPTONS:

ESX or ESXi Host may get disconnected from Virtual Center

All paths to the LUNs are in standby state

Esxcfg-rescan might take a long tome to complete or never complete (hung)

VMKernel logs show entries similar to the following:

Queue for device vml.02001600006006016086741d00c6a0bc934902dd115241 49442035 has been blocked for 6399 seconds.

Please refer to KB 1008130.

SOLUTION:

A reboot is required to clear this condition.

VMware is working on a patch to address this issue. The knowledge base article for this issue will be updated after the patch is available.

NEXT STEPS:

If you encounter this condition, please collect the following information and open an SR with VMware Support:

1. Collect a vsi dump before reboot using /usr/lib/vmware/bin/vsi_traverse.

2. Reboot the server and collect the vm-support dump.

3. Note the activities around the time where a first “blocked for xxxx seconds” message is shown in
the VMkernel.

How to check VMware tools version in a Linux VM not running X win

2008-12-11T13:11:00.002-05:00

Since in a Linux VM not running X windows, we can't use vmware-toolbox or .vmx to check the version of the VMware tools. We can check the version of VMware tools listed in vmware-config-tools.pl, although it could be the case that VMware tools got installed, but haven't been configured yet. Run the following command to grab the version of VMware tools have been installed.

#grep buildNr /usr/bin/vmware-config-tools.pl
my $buildNr;
$buildNr = '3.5.0 build-82663'
return remove_whitespaces($buildNr);

3.5.0 build-82663 is the VMware tools version number.

ESX intermittently disconnects from VirtualCenter

2008-11-18T09:09:00.004-05:00

Error:

Sometimes an ESX host could intermittently disconnects from VirtualCenter. There are many things could cause it, but here are two common, but haven't paying much attention places that we need to look for.

Solution:

1) Use "df -h" Check to see if / directory in ESX Service Console is filling up. After / directory is clean up and wait for a while before reconnecting ESX to VirtualCenter.

2) When seeing a message like below in /var/log/vmware/hostd.log, it is time to upgrade the memory in service console, and change its default soft and hard limit. If the memory checker reach its hard limit, bad things could happen to ESX.

/var/log/vmware/hostd.log

"[2008-11-14 22:05:13.319 'Memory checker' 20937648 warning] Current value 138020 exceeds soft limit 122880."

"Unable to fork: Cannot allocate memory"

SC memory: With VirtualCenter 2.5 and above, it is easy to upgrade the ESX service console memory to 800MB, but it requires a reboot. See KB1003313 and KB1003501 in VMware's website.

Soft/Hard limit: Edit the config.xml under /etc/vmware/hostd and add the following into the <config> section:

Do a restart on hostd process by:

service mgmt-vmware restart

Creating a floppy image

2008-11-12T14:52:00.002-05:00

Occasionally need to create floppy image, and an easy way to do it is to use Linux to create it. And you can mount it right there to add any files to it to.

The following command is to create a blank floppy, filled with zero.

/bin/dd if=/dev/zero of=/root/BlankFloppy.flp bs=512 count=2880

And this blank floppy can be formatted in Linux too.

/sbin/mkfs.vfat -v -c /root/BlankFloppy.flp

Mount the floppy image so that you can put files into it.

mount -o loop /root/BlankFloppy.flp /media/floppy

In Windows, you can use some Winimage, Floppyimage, undisker to create image of floppy.

Replacing O2 sensor in 2005 Honda CR-V LX

2008-07-15T23:26:00.000-04:00

After the Just Brakes scam, I decided to DIY on my car if it has problems. It didn’t take long before I had a chance.

My 2005 Honda CR-V LX 2WD had an engine light came on one day. I went over to the dealership close to my house the next morning. I was told that I would be charged $99 for diagnostic to find out why the light came on, and I would have to come back again since that day’s schedule was full. I was glad that he told me to come back; otherwise I would have to come up with my own reason to give him that I need to come back later. I don’t like they charge their customers like this. I remembered my co-worker told me that AutoZone offers free diagnostic on cars. After work I bought my car over to AutoZone, and the clerk gave me a printout of code P0135, which said it was O2 sensor’s error. I came back home and get online for some research. It took me a while. I spent quite a bit to time to see if my car is still under emission warranties. Finally, I downloaded my car’s Emission Warranty from Honda Owner Link website, and it confirmed with the information I got from two dealers’ service departments and America Honda – my car’s O2 sensor (A/F sensor) has 3 year/36K miles warranty only.

In the beginning I ordered the O2 sensor for $100 from The Parts Authority in Amazon. But it just wasted my time since the one they shipped me wasn’t a right one. Calling them back and found out that they don’t know which one is for my car. Finally they told me that they don’t have it. I ended up with paying both ways shipping for nothing. Finally I placed my order with Majestic Honda at www.hondaautomotiveparts.com. They have diagrams in their website for each part they are selling. It took about a week to get to me. I went to AutoZone to rent the O2 sensor socket, anti-seize and prepared to have a hard time to take off the original O2 sensor. I ended up taking less than one minute to take it off. Putting the new one on was very easy too. The engine light kept on in that day, but it went off in the next early morning. I felt pretty good about it. This is my first time doing maintenance on my car, and I am saving about $190 replacing it myself. Honda dealer was going to charge me $400 if I let them do it, plus $99 diagnostic.

Free tool secures VMware ESX

2008-06-23T00:18:00.001-04:00

ConfigCheck is a free utility that can compares VMware ESX configurations against best practices guidelines development by VMware Inc. The tool was jointly released Wednesday by VMware and configuration management software maker Tripwire Inc. ConsigCheck only works on ESX 3.5.

http://www.tripwire.com/configcheck/configcheckdownload.cfm

Just Brakes Scam

2008-06-23T00:17:00.000-04:00

Just Brakes is advertising $99 for 4 brake pads brake job to show their cares for their customers. What a scam it is. The one I went to completely lied to me and ripped me off.

After I drove into Just Brakes, at 11235 Alpharetta Rd, #100, Roswell, GA 30076, Dean was the technician did the quote to me. After he had someone take everything off, including tires, calipers, brake pads. He said my disk pads need to be replaced, which was $99.98. Since my 2005 Honda CR-V’s rear brake pads has clips on it, which are different from 2004’s Honda CR-V. They don’t have rear brake pads in stock, and I need to pay $75 so that they can have the rear brake pads. And he said disk hardware also needs to be replaced. When he brought me into the shop to show me what it was, he made it very vague. He pointed at the calipers and said the things on it bent, and needed to be replaced. Sorry for my naïve, but I didn’t know what caliper was, and what disk hardware was. I felt pretty confused. Disk hardware cost about $85. And bleeding is another $39.99. The total came to $304.83, which includes a $10 discount coupon. At that time, I was very confused, and they had taken everything apart. I wondered what would happened if I refused the deal. And I have taken half a day off already. What the heck. I agreed on the quote written on the back of a piece of paper. The whole thing was pretty thick though. They finished it in less than an hour. However, no one delivered any parts during that hour, and Dean was with me all the time. When a guy brought in my car key, Dean tried to explain that the delivery guy must come in from the back.

After I came back, and did my research online and found out this is a scam that Just Brakes has been doing to lots of its customers. The so-called Disk Hardware is just a clip on a caliper, costs about $10 online. I know part of this scam was my fault, for being too naïve on cars. I didn’t grow up in the US, and cars aren’t any hobbies to me. I guess it is about time for me to learn about cars. Secondly, I trusted Just Brakes too much. Car salesmen are infamous, now I guess cars shops are too. Certain business industries aren’t honors during their business practice.

2008-05-26T00:17:00.001-04:00

ESX3: ramcheck

I have been using Memtest86+ for all the ESX servers I deployed. It has been all good, but it requires booting from Memtest86+. During the memory check, ESX server isn’t running. Ran across a tool posted in xtravirt.com, which is actually an ESX built-in utility to do memory check. The command is ramcheck. The good thing about this is it is a built-in utility, and it runs in the background of ESX, which means no down time will be involved. Just run

service ramcheck start

It runs in a background and writes out a log file to /var/log/vmware/ramcheck.log and ramcheck-err.log. Since it not only checks the memory in Service Console, but also the memory in VMkernel, it has to run as a world in VMkernel space. It consumes only a few percent of CPU resource in the background, but the trade of is the time will take a couple weeks to complete. Run esxtop will show it as ramcheck.<id>. It will stop automatically once it finished all the memory check.

It might not be a complete replace of Memtest86+, but it does offer an alternative, non intrusive maintenance memory check during operation of an ESX server.

vmkping error: syscall version mismatch (compiled: 0x7616c4e3, kernel: 0x50f63116)

2008-05-07T14:45:00.000-04:00

Today after I configured an ESX 3.0.2 Update 1 server, I wanted to use vmkping to test vmkernel IP stack. However, it returned an error stating "syscall version mismatch (compiled: 0x7616c4e3, kernel: 0x50f63116)". Odd, since it doesn't look like an network issue, which was what I was always getting. After a search online, found a VMware KB 1002880 posted on Feb 2008 at http://kb.vmware.com/selfservice/dynamickc.do?externalId=1002880&sliceId=2&command=show&forward=nonthreadedKC&kcId=1002880. The KB states that only commands vmkping and ramcheck are affected, and all other system utilities, ESX Server stability and reliability are not jeopardized.Ok, I was planning to patch it anyway.