This ThinApp setup includes a legacy version of Java (7u25) which is super old but it should help with accessing the following (not exhaustive list, just things I have run into, please help expand list):

• UCS Manager 2.1.x
• vCloud Director 5.1.x Uploads
• HP iLO 2.x
• Legacy DRAC
• vCloud VPN
• EMC Unisphere

Onward, but first an important caveat:

*****************************************************

This process creates an insecure browser installation, DO NOT browse the internet with this configuration…ever.  

You have been warned.

*****************************************************

Note: I am not a ThinApp expert (and I usually avoid windows, hah!) so there may be a more elegant way to do this, if anyone has tips let me know!

First, collect the correct tools, I attached the versions that I used at the end of the post:

1. Firefox 29.0.1 (Link)
2. Java 7u25 (32 bit) (Link)
3. Flash Installer (for correct OS, I used Win8) (Link)
4. VMRC Installer for 5.1.x from your instance (or mine from 5.1.x below)
5. VMware ThinApp 5.x (Trial)
6. Clean Windows 7 or 8 Capture VM (Base install without security software is best)

After Windows is installed, the following are steps to take to create the package:

1. Take snapshot of base VM
2. Install ThinApp 5.x
3. Take snapshot (this is a theme, I have found a successful ThinApp capture is snapshot heavy)
4. Start “ThinApp Setup Capture” utility, Click Next
   
5. Click Prescan
   
6. Install Firefox
   • Choose Custom
   • De-select “Maintenance Service”
7. Install Flash (Latest version is fine)
8. Install Java
   • Choose “Change Destination Location”
   • Install to “c:\java”
9. Install VMRC Plugin
10. Launch Firefox, perform the following tasks
    1. Navigate to about:config in Firefox, Click “I’ll be careful, I promise!”
    2. Change the following settings
       • app.update.auto – false
       • app.update.enabled – false
       • app.update.silent – false
       • app.update.mode – 2
       • extensions.update.autoUpdateDefault – false
       • extensions.update.enabled – false
       • (optional, disables login remembering) signon.rememberSignons – false
       • (optional, enables pop-ups since vCD uses these) dom.disable_open_during_load – false
    3. Verify plugin updating is disabled:
       1. Navigate to Menu -> Addons
       2. Click Plugins
       3. Click the gear icon and make sure this option is not selected
          
    4. Perform the following tasks to verify that Java is installed
       1. Navigate to this link.
       2. Click “Activate”
          
       3. Click to “Allow and Remember”
          
       4. Check the “Don’t Ask Again…” Checkbox and Click Later
          
       5. Click the Run button at the prompt to verify the Java applet loads
11. Take Snapshot (just in case)
12. Back to the ThinApp Capture Utility, click Postscan
    
    
13. Click OK
    
14. This will take some time, if(Time.current > 17:00) { Fridge.get(‘beer’); } else { Counter.get(‘coffee’); }
15. De-select all entry points other than Firefox, click Next
    
16. Click Next
    
17. Click Next
    
    
18. Select “Restricted write access”, click Next.
    
19. Select “Same Directory…”, click Next.
    
20. Select No (unless you want to share), click next
    
21. Click next (I don’t use the ThinDirect feature because it only works with IE)
    
22. Update inventory name to “Firefox 29.0.1″ to differentiate it.
    
23. Select “Use one of the entry points” then pick “Mozilla Firefox.exe”, Check “Compress virtual package”, Click
    
    
24. Click OK
    
25. Ignore warnings (like a boss) and click Next
    
26. Click “Edit Package.ini”
    
27. In Package.ini, Locate the block “[Mozilla Firefox.exe]” and perform the following changes to the block
    1. Add “StatusbarDisplayName=BeyondVM Firefox 29.0.1 Build “
    2. Add  “CommandLine=%ProgramFilesDir%\Mozilla Firefox\firefox.exe -no-remote”
    3. Rename block from [Mozilla Firefox.exe] to [Mozilla Firefox 29.0.1.exe]
28. Save Package.ini
29. Click Build
30. Wait, this will also take some time.  Refer to previous formula: if(Time.current > 17:00) { Fridge.get(‘beer’); } else { Counter.get(‘coffee’); }
31. Click Finish
    

At this point you will have a functioning ThinApp package file all bundled up in the single exe file in the bin directory.  You might notice that this package is quite large, I performed the following additional steps to my package to get it down to about 350MB from over 800.

1. Edit Package.ini and perform the following modifications
   1. Add the following section below the compression type
      [FileList]
      ExcludePattern=*.bak,*.msi,*.mst,*.cab,*.msp
   2. Delete all entry points other than [Mozilla Firefox 29.0.1.exe]
2. Delete the following directories from the package (note: I have not exhaustively tested this, some of these may be required but it worked for me)
   1. %drive_C%\Users\All Users\Microsoft\Windows Defender
   2. %ProgramFilesDir(x64)%
   3. %SystemRoot%\WinSxS
   4. %SystemRoot%\assembly
   5. %SystemRoot%\Installer
   6. %SystemRoot%\Logs
3. Run build.bat as an Administrator

After that the package is pretty snappy and should be good to go.  It will create a profile directory next to the EXE to save settings in, this will include popup allowing and certificate exceptions.  The first time I logged into my vCloud instance I had to tell it to run the JRE again, I was unable to figure out how to get it to never ask during the build process but it wasn’t that bad to just do it the first time.

These are all of the files required to do this as well as the package I created following these directions. My pre-built package will expire in about 60d unless I can get VMware to give me a community license (hint hint).

BeyondVM Firefox 29 Bundle (215.8 MiB)

Firefox Setup 29.0 (27.7 MiB)

FirefoxPortableESR 24.7.0 English Paf (23.8 MiB)

VMware VMRC Plugin (34.3 MiB)

Please leave a comment if there is anything I can improve!

Lastly, the caveat we started with:

*****************************************************

This process creates an insecure browser installation, DO NOT browse the internet with this configuration…ever.  

You have been warned.

*****************************************************

Goodnight and Good Luck.

This week I had to connect an additional NetApp Storage System to my existing UCS environment through a different path than a similar shared storage platform that we utilize here.  This shouldn’t be a big deal but there were a few caveats:

• The Storage System was attached to a dedicated Nexus 5k for this customer
• The VLAN configured on the customer switch collided with one configured in UCS so VLAN translation was necessary through access ports.
• The traffic takes a different switching path from normal NFS traffic in this environment

I configured everything as one normally does when connecting to IP storage, jumbo frames and all.  There was only one problem:

# Standard Frames
~ # vmkping -s 1400 -d 10.0.0.253
PING 10.0.0.253 (10.0.0.253): 1400 data bytes
2508 bytes from 10.0.0.253: icmp_seq=0 ttl=255 time=0.232 ms
2508 bytes from 10.0.0.253: icmp_seq=1 ttl=255 time=0.198 ms
2508 bytes from 10.0.0.253: icmp_seq=2 ttl=255 time=0.265 ms

--- 10.0.0.253 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.198/0.232/0.265 ms
~ #

#Jumbo Frames
~ # vmkping -s 2500 -d 10.0.0.253
PING 10.0.0.253 (10.0.0.253): 2500 data bytes

--- 10.0.0.253 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
~ #

DOH!

Now, normally this is just a simple issue of the MTU being set incorrectly somewhere along the traffic path but as I dug deeper into this issue that turned out to not be the case, it was something much stranger and more interesting.  The IPs, VLANs and hostnames have been changed or obscured to protect the innocent.  Onward!

Note: To test pinging with Jumbo Frames in ESXi use the following command:

vmkping -s 2500 -d <ip of endpoint>

Background

I checked to make sure jumbo frames where configured correctly all along the traffic path (ESX, UCS vif and customer switch) but I was still not able to ping the Storage System from the ESXi host using jumbo frames.

For reference the configurations were as follows:

The Network

A little about the relevant network topology now, this is a basic diagram of how the network is laid out.  There is quite a bit more going on in this network but this diagram shows the relevant parts to this story.

I have marked the traffic path that was forced with the UCS Pin Group and verified via spanning tree.  Also included in this diagram is a similarly connected shared storage platform that becomes relevant in one of the troubleshooting steps.

First, this is the generic QoS policy that is configured on all network switches here in order to enable jumbo frames

system qos
  service-policy type network-qos jumbo
policy-map type network-qos jumbo
  class type network-qos class-default
    mtu 9216

Now some interface configurations, network diagram is back with the relevant interfaces marked.

Interface facing Customer NetApp (A):

interface Ethernet1/30
  description netapp01:E0B
  switchport mode trunk
  switchport trunk allowed vlan 60,177
  spanning-tree port type edge trunk

Customer side of uplink (B):

interface Ethernet1/29
  description cloud_access_a:eth1/17
  switchport access vlan 60

My side of uplink (C):

interface Ethernet1/17
  description customer_switch_a:1/29
  switchport access vlan 2244

Port Channel Facing UCS (D):

interface port-channel2
  description fi01:po1
  switchport mode trunk
  switchport trunk allowed vlan <truncated, long list>
  speed 10000

That is it as far as relevant networking goes, onto the VMware side.

ESXi

~ # esxcfg-vswitch -l
...
Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks   
vSwitch1        5632        6           128               9000    vmnic2,vmnic3

  PortGroup Name        VLAN ID  Used Ports  Uplinks   
  ipStorage-vmk             2244     1           vmnic2,vmnic3

~ #

Pretty simple configuration here, standard operating for the most part.  When this problem began this host had a standard switch but the configuration was just the same.  On to UCS.

UCS Configuration

The interesting thing that we did for troubleshooting was to use a pin group to force the traffic down one set of uplinks through the fabric so we could predict where the traffic was going to go.  These snippets don’t reflect the pin group as I removed it after troubleshooting but information on pin groups can be found here.

This is the running configuration of the vETH adapter:

fi01-A(nxos)# sh run interface vethernet 1841

interface Vethernet1841
  description server 1/3, VNIC stgA
  switchport mode trunk
  untagged cos 4
  no pinning server sticky
  pinning server pinning-failure link-down
  no cdp enable
  switchport trunk allowed vlan 
  bind interface port-channel1378 channel 1841
  service-policy type queuing input org-root/ep-qos-Storage
  no shutdown

This is the configuration for the CoS class of 4:

fi01-A /eth-server/qos # show eth-classified gold

Ethernet Classified Class:
    Priority: Gold
    CoS: 4
    Weight: 9
    BW Percent: 47
    Drop: Drop
    MTU: 9000
    Multicast Optimize: No
    Admin State: Enabled
ucpd01-A /eth-server/qos #

Now, onto the other end, the NetApp.

Storage System Configuration

Not too much to see here, but we can verify that the MTU is actually 9000.

filerA> ifconfig svif1-60
svif1-60: flags=0x2b4c863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 9000
        inet 10.128.160.253 netmask 0xffffff00 broadcast 10.0.0.255
        partner svif2-60 (not in use)
        ether 02:a0:98:1b:6c:18 (Enabled interface groups)
filberB> ifconfig svif2-60
svif2-60: flags=0x2b4c863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 9000
        inet 10.0.0.254 netmask 0xffffff00 broadcast 10.0.0.255
        partner svif1-60 (not in use)
        ether 02:a0:98:1b:67:98 (Enabled interface groups)

Despite all of that configuration I was still not able to get ping traffic to traverse the network when forcing jumbo frames.

Troubleshooting

The first thing that I did to try to see what was going on was to set up a simple packet capture on another host with jumbo frames enabled on the same switch as the Customer NetApp Storage System.

~ # vmkping 10.0.0.10 -s 2400 -d
PING 10.0.0.10 (10.0.0.10): 2400 data bytes

--- 10.0.0.10 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
~ #

#Source:
21:00:39.790337 IP truncated-ip - 2346 bytes missing! (tos 0x0, ttl 64, id 34311, offset 0, flags [DF], proto ICMP (1), length 2428)
    10.0.0.121 > 10.0.0.10: ICMP echo request, id 7342, seq 0, length 2408
21:00:40.792509 IP truncated-ip - 2346 bytes missing! (tos 0x0, ttl 64, id 34314, offset 0, flags [DF], proto ICMP (1), length 2428)
    10.0.0.121 > 10.0.0.10: ICMP echo request, id 7342, seq 1, length 2408
21:00:41.794613 IP truncated-ip - 2346 bytes missing! (tos 0x0, ttl 64, id 34320, offset 0, flags [DF], proto ICMP (1), length 2428)
    10.0.0.121 > 10.0.0.10: ICMP echo request, id 7342, seq 2, length 2408

#Destination:
21:01:34.825518 IP truncated-ip - 2346 bytes missing! 10.0.0.121 > 10.0.0.10: ICMP echo request, id 7342, seq 0, length 2408
21:01:34.825614 IP truncated-ip - 2346 bytes missing! 10.0.0.10 > 10.0.0.121: ICMP echo reply, id 7342, seq 0, length 2408
21:01:35.827732 IP truncated-ip - 2346 bytes missing! 10.0.0.121 > 10.0.0.10: ICMP echo request, id 7342, seq 1, length 2408
21:01:35.827825 IP truncated-ip - 2346 bytes missing! 10.0.0.10 > 10.0.0.121: ICMP echo reply, id 7342, seq 1, length 2408
21:01:36.829821 IP truncated-ip - 2346 bytes missing! 10.0.0.121 > 10.0.0.10: ICMP echo request, id 7342, seq 2, length 2408
21:01:36.829935 IP truncated-ip - 2346 bytes missing! 10.0.0.10 > 10.0.0.121: ICMP echo reply, id 7342, seq 2, length 2408

Note: The truncated-ip just means that tcpdump is only capturing headers, using ‘-s0′ switch turns this off.

What we see here is that on the source machine (Another ESXi Host) I can see the ICMP Echo Requests leave the machine and on the destination machine I see the ICMP Echo Requests AND ICMP Echo Replies.

WHAT.

Back to the network diagram:

I was doing this capture on a host located at point A, my next step was to check the other end of the path before UCS.  I set up a span port at point D to check it out.  I wired up a server with an available 10Gbit interface and connected it up.

On the CoreSwitch A the following was set up:

monitor session 1
  description vlan-224-tap
  source vlan 2244
  destination interface Ethernet1/15
  no shutdown

I fired up Wireshark with an ICMP capture filter (no need to crush the disk with 5GB/s of traffic!) and tried the ping again (this time to the filer again).  This was the result:

Again:

WHAT.

This was incredibly puzzling.  What this meant was that the traffic was correctly making it from the UCS server to the destination and back but was getting lost somewhere inside UCS.  I double and triple checked all of the configuration in UCS and ESXi and everything looked right.

One more test, add a known working VLAN to the vIF/vmnic.  This is where the shared array on the diagram comes in.  I set up a dummy vmkernel port on that VLAN and pinged the shared Storage System and guess what; IT WORKED.  So, what did this mean.  This means that by any logic we had ruled out both the transit network AND UCS.  That was impossible though since it clearly frames were still getting dropped somewhere.

My feeling at that moment can only be described by this meme:

There had to be something strange going on, something we had over-looked.  To save anyone else the 20 hours of rage I think it is best if I just skip to the end.

The Solution

This is the part of our story where its time to talk about Ethernet frames and VLAN tags briefly.  First, this is an great diagram of an ethernet frame (from this awesome NetApp article):

The important part about this frame is the VLAN tag, a 4 byte section added to the frame when using tagged VLANs, which we are in this environment.  Diving into that 4 bytes there are two parts, the first 2 bytes are the Tag Protocol Identifier (TPID) which is always set to 0x8100 for 802.1q frames.  The second two bytes are the Tag Control Information (TCI).  The TCI bytes are divided into two parts, the first three bits are to hold the 802.1p identification information and the remaining bits hold the VLAN ID.

This diagram (that I lifted from here) illustrates this a little bit better:

Now, at this point it probably isn’t apparent why I bring this up, but here it goes.  So back on UCS we have the definition of the storage vNIC as such, this time with the important parts marked:

fi01-A(nxos)# sh run interface vethernet 1841

interface Vethernet1841
  description server 1/3, VNIC stgA
  switchport mode trunk
  untagged cos 4
  no pinning server sticky
  pinning server pinning-failure link-down
  no cdp enable
  switchport trunk allowed vlan 
  bind interface port-channel1378 channel 1841
  service-policy type queuing input org-root/ep-qos-Storage
  no shutdown

That QoS policy maps to the system QoS policy of Gold, which is defined as follows, again with the important part bolded:

fi01-A /eth-server/qos # show eth-classified gold

Ethernet Classified Class:
    Priority: Gold
    CoS: 4
    Weight: 9
    BW Percent: 47
    Drop: Drop
    MTU: 9000
    Multicast Optimize: No
    Admin State: Enabled
ucpd01-A /eth-server/qos #

So, based on what we have learned about CoS and 802.1q frames, check out the network diagram with some more important information added:

Now, it took us quite some time to realize this and it ended up being an “ah hah!” moment from one of our network engineers but the issue was the fact that we were using an access port to do VLAN translation because CoS information is stored in the 802.1q part of the frame header.

So what was happening was the frames were traveling out and when they hit the first access port (C) the CoS information was lost when the VLAN tag was stripped.  That was all fine until the frame made it back to UcS which dropped the frames on ingress to the Fabric Interconnect because without a CoS tag they were too large (being jumbo frames).  This explains why regular frames worked but jumbo frames didn’t, the default CoS class allowed small frames through.

All things considered, the fix is simple, here are the updated access port configurations with the changes marked:

Customer side of uplink (B):

interface Ethernet1/29
  description cloud_access_a:eth1/17
  untagged cos 4
  switchport access vlan 60

My side of uplink (C):

interface Ethernet1/17
  description customer_switch_a:1/29
  untagged cos 4
  switchport access vlan 2244

With those changes in place, success!

~ # vmkping -s 2500 -d 10.0.0.253
PING 10.128.160.253 (10.0.0.253): 2500 data bytes
2508 bytes from 10.0.0.253: icmp_seq=0 ttl=255 time=0.232 ms
2508 bytes from 10.0.0.253: icmp_seq=1 ttl=255 time=0.198 ms
2508 bytes from 10.0.0.253: icmp_seq=2 ttl=255 time=0.265 ms

--- 10.0.0.253 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.198/0.232/0.265 ms
~ #

This was quite an interesting issue and it caused me to learn quite a bit about how ethernet works in general and as well as how UCS switches.

Another way to solve this is to set the default QoS class to accept jumbo frames but while this would have allowed the traffic to flow it may have caused other performance issues in the future in the event of fabric contention as the traffic would have not been prioritized correctly.

I hope this helps someone else that may run into this in the future!

—–

If you suddenly start seeing something the following error in vCloud Director 5.1.x:

This is because your Firefox upgraded to Firefox 30 automatically, as it does.  This error seems to be due to some sort of change that Mozilla added into Firefox 30, I haven’t tracked it down yet (if anyone has let me know!).  Even force enabling the plugin won’t help, Firefox 30 looks to have gone to an whitelist only model, doing so will make this error go away but the console sessions never connect.

That being said, VMware has stated that Firefox 30 will not be a supported browser for vCloud 5.1.x (the VMRC plugin doesn’t actually function so it is not a question of only supportability).  Unfortunately it is also not supported in IE 10, 11 or Chrome 35 so the only option is really Firefox 29 (or Firefox 3.6). The only solution is to upgrade to vCloud 5.5.x, which while true isn’t really a solution.

As a note, I have had success using ThinApp to isolate a Firefox 29.0.1 with the VMRC plugin, Flash and Java and Portable Firefox may be an option too. Either way, be sure to disable auto-update (Firefox Options -> Advanced -> Update) if you downgrade to Firefox 29.0.1.  There are some tips here on how to do that, the big one being the ‘-no-remote’ option to allow the ThinApp Firefox to launch its own process.

Update: It seems that installing the Firebug plugin and enabling that will allow the console to connect in Firefox 30 after forcing the plugins to activate.  While cumbersome, this is another workaround.

Update 2 (7/29/14): I have found another solution!  It is possible to use the Portable Firefox ESR Version (Extended Support Release), this is a Firefox 24 package that will run  in a sandbox.  It will use the installed plugins on your system so if your Java version is also incompatible that won’t work.  One thing to change, perform the following:

1. Locate FirefoxPortable.ini in your extracted ESR download (might be in Other/Source directory).
2. Copy to the root of Portable Firefox ESR install directory
3. Edit file and change the “AllowMultipleInstances” variable to true.
4. Save

This will allow the Portable Firefox and Installed Firefox to run at the same time.

Linux Setup

There are two modules to install for this setup, bonding and 8021q.  As follows:

#install packages
# NOTE: ensure ifenslave 2.6 is what gets installed, required for VLANs
root@filer:~# apt-get install -y ifenslave vlan

#load modules manually to be sure
root@filer:~# modprobe 8021q
root@filer:~# modprobe bonding

#add to modules for reboots
root@filer:~# echo 'bonding' >> /etc/modules
root@filer:~# echo '8021q' >> /etc/modules

Basic Bonding using LACP

Linux

This is the linux side of this link, the important part here is the bond-mode needs to be set to 802.3ad or mode 4.

#slave interfaces
auto eth4
iface eth4 inet manual
bond-master bond0

auto eth5
iface eth5 inet manual
bond-master bond0

#bond interface
auto bond0
iface bond0 inet static
        address 192.168.1.10
        netmask 255.255.255.0
        #there are several modes, this is also known as mode 4
        bond-mode 802.3ad 
        bond-miimon 100
        bond-slaves eth4 eth5

 Cisco 2960

This is the configuration of the two switch ports and the port channel on the Cisco switch.  The important part here is using ‘active’ mode on the channel group.

!vlan 10 is my storage VLAN
interface Port-channel1
 switchport access vlan 10 
end

interface GigabitEthernet0/43
 switchport access vlan 10
 channel-group 1 mode active
end

interface GigabitEthernet0/44
 switchport access vlan 10
 channel-group 1 mode active
end

 Bonding with VLAN Trunking

Most people need to use both VLANs and 802.3ad trunking in the real world, especially for storage, as it turns out that is pretty easy.  In this example I only have one VLAN but the setup can be repeated for each additional VLAN.  Also on the “base” interface I don’t have any configuration, this is where the “native” IP configuration would go.

Note: This configuration can also be applied to a non-bonded interface.

Linux

This is the linux side of this link, the important part here is the bond-mode needs to be set to 802.3ad or mode 4.

#slave interfaces
auto eth4
iface eth4 inet manual
bond-master bond0

auto eth5
iface eth5 inet manual
bond-master bond0

#bond interface
auto bond0
iface bond0 inet static
        #native vlan, need ip to configure
        address 1.1.1.1
        netmask 255.255.255.0
        bond-mode 802.3ad 
        bond-miimon 100 
        bond-slaves eth4 eth5 

auto vlan10 
iface vlan10 inet static 
        address 192.168.1.10 
        netmask 255.255.255.0 
        broadcast 192.168.1.255
        vlan-raw-device bond0

 Cisco 2960

This is also similar to the other configuration, the ‘switchport trunk allowed’ section is optional, I wanted to prune the VLANs for this link.

!vlan 10 is my storage VLAN
interface Port-channel1
 switchport trunk allowed vlan 10
 switchport mode trunk 
end

interface GigabitEthernet0/43
 switchport trunk allowed vlan 10
 switchport mode trunk
 channel-group 1 mode active
end

interface GigabitEthernet0/44
  switchport trunk allowed vlan 10
  switchport mode trunk
 channel-group 1 mode active
end

This is a pretty simple setup, post a comment with suggestions or requests for more information and I will keep this post up to date.

 References

• UbuntuBonding
• Cisco LACP Reference
• Ubuntu VLAN Wiki
• Ubuntu VLAN Interfaces manpage

I came up with this quick unix one-liner:

# NOTE: My datastore path is /vmfs/volumes/5317a80e-add165f6-ada9-001517599f73
# replace this with whatever datastore needs searching

#VMs
find /vmfs/volumes/5317a80e-add165f6-ada9-001517599f73 -name "*.vmx" -exec  vim-cmd solo/registervm {} \;

#Templates
find /vmfs/volumes/5317a80e-add165f6-ada9-001517599f73 -name "*.vtmx" -exec  vim-cmd solo/registervm {} \;

Which gave me this:

/vmfs/volumes/5317a80e-add165f6-ada9-001517599f73 # find /vmfs/volumes/5317a80e-add165f6-ada9-001517599f73 -name "*.vmx" -exec  vim-cmd solo/registervm {} \;
143
144
...
...
...
148
149
/vmfs/volumes/5317a80e-add165f6-ada9-001517599f73 #

There are more elegant ways to do this in PowerCLI that are probably more scaleable (not registering everything on one host, for example) but this is a quick, down and dirty way to get the job done!

In the ever present search for the “ultimate home lab box,” I ran across a new contender, the new Intel Avoton. This is a really interesting SoC (System on a Chip) that has some promise for home lab.  The Avoton C2000 series SoCs are based on the new 22nm manufacturing process that came with haswell which is a drastic improvement on the previous models.

One of the most interesting things about it is that it now allows for out of order pipeline execution. This is one of the things that made previous Atom CPUs/SoCs so much slower.  Another thing that is really intriguing about it is that it now supports both VT and is a 64bit CPU, so it can run both 64 bit and 32 bit guests as well as having 8 cores (there is a 4 core model available as well).  Also, this architecture has a maximum memory of 64GB which puts it right in the sweet spot for home lab nodes, though it is more likely that these boards will see 32GB (4x8gb) until the price of 16GB dimms/sodimms comes down.  And, the best part is that this SoC has a TDP of 20w – crazy!

The first motherboard I found to really support this CPU was the Asrock C2750D4I, I picked one up to do a little testing to see what it really could do.  Here is what the layout looks like, courtesy of the Asrock website:

This motherboard is pretty slick! It has a dedicated IPMI port with a simple web interface as well as dual on board NICs and 4 full size DIMM slots.  This particular board also features 12 SATA (mix of 3gb and 6gb) ports which would also make this a great home NAS base.

Note: To get ESXi 5.5 running on this board you will have to manually add the Intel igb driver to your ISO. I used esxi customizer for this because I am a lazy admin, but you can also do it with image builder. The driver was present in 5.1, and I am not sure why VMware removed it from 5.5. The driver can be found here: ESXi5X-INTEL-igb-4017

SuperMicro has also come out with a few boards but they were not widely available when I started testing.  The most interesting ones are A1SAi-2750F.  This board has a Marvell Alaska 88E1543 transceiver on it that allows it to have 4x 1GBe NICs. I am not sure how that will play with ESX but I plan on ordering one to test! It also has an onboard USB header which is always handy. However, it does have fewer SATA ports, but that isn’t a huge deal for an ESXi node. SuperMicro has a few uATX boards with an extra PCIe slot on it that look pretty cool as well, but those are a little big for me.

Now, by far the coolest use of this SOC is the HP Moonshot blade system.  I mean, just look at this thing (image from hp.com):

This thing is nuts! As you can see in the picture there are 45 small blades mounted vertically in a 4.5U chassis. This also includes a switched backplane that they all connect to. The blade that uses the Avoton is the m300 (image from hp.com):

I would love to get one of these chassis to play around with. It would be perfect for a lab of servers or mass scale small nodes. I wonder what the cost per VM would look like when using these blades for a farm of small VMs like a VDI environment, where each VM has about 2-4GB of memory, which would get you about 15-25 VMs on each node.  Interestingly, these blades don’t seem to support 64GB of ram. I wonder if that is a misprint or if HP does not have 16GB SODIMM modules available.

Testing

As a simple test of what kind of performance a CPU can give me, I like to set up two VMs and run a netperf test since this type of traffic involves mostly a CPU activity.  My test setup is pretty simple: I have two Ubuntu VMs, each with 2x vCPU and 4gb memory and a single VMXNET3 adapter.

For this test the VMs were connected to the same port group and assigned IP addresses, then iPerf was run with the following options:

Client: iperf -t 30 -c <server> -P 25
Server: iperf -s

This gives us a 30 second test with 25 connections, which should be enough to saturate any link.

This is a table of my results for this test. I had a few other servers with different CPU architectures sitting around, so I included those too. Also, where possible I measured the power draw with a killawatt device for curiosity or took the power reading out of the management software.

I know that this is not a very exhaustive test, but with only one board I can’t yet run something like vmmark. I plan to run something like that in the future when I have a more full blown lab based on these boards set up.  It is enough to convince me to pick a few more up for more investigating!  Also, I am going to add to this chart as I run across different CPU types to test.

Nested Virtualization

In the comments someone asked if this board supported nested virtualization, the details were sort of un-clear from Intel.  On the Intel ARK page it says that this CPU supports Intel VT but that isn’t the only requirement anymore.  Well, following William’s guide, I checked the mob for the property ‘nestedHVSupported’ which was true but in order to be thorough I also tested a virtual ESXi host (5.5 inside 5.5) and was able to boot both 32 bit and 64 bit nested guests and they performed pretty well so it looks like this is a definite yes, awesome!

Conclusion

The thing I found most interesting from these tests is that most of the Intel CPUs get about the same throughput (17GB/s) no matter the age or generation and the AMD CPUs were always substantially slower.  I am not sure if this is always true throughout the history of the two architectures, but I have noticed that Intel CPUs seem to feel faster in the past, even with similar clock speeds.

I think that this little motherboard (and others like it) is a real contender for the home lab where speed isn’t as large as a concern as is power savings.  I plan to load two or three of these up for my BeyondVM lab in the near future – a small and silent ESX farm is really appealing.  I will post updates as I get other performance statistics in the future. Leave a comment below for any test suggestions!

Update (2/3/14): Added a section on nested virtualization to answer some questions in the comments.

—-

This morning it came to my attention that my customers were no longer able to upload any media (OVFs or ISOs) to their vCloud catalogs. This seems to be due to the most recent Java JRE version released by Oracle.  The behavior I experienced was that the applet would appear to load but when I would click on the browse button nothing would happen.  This happens across all different browsers and browser versions.  The reason for this seems to be a change in the requirements for certificates and applet signing in JRE 7uU51.

Luckily, there is a quick workaround for this:

1. Open Configure Java control panel, navigate to Security then Edit Site List.

2. Type the URL for your vCloud instance in the blank and Click Add to add (repeat for multiple URLs), click OK

Thats it!  This really is a workaround, I am not sure what the final solution will be but this got my customers and I back into business for now.  If anyone has more information, let me know!

Update: It seems that this update actually breaks everything from HP iLO to vCenter Orchestrator, I am still searching for a better workaround if anyone knows of one.

Update 2: There are reports of this breaking Cisco UCS, Dell iDRAC, vCloud VPN and EMC Unisphere as well.  It might be safe to say that most Java based management tools are going to be effected.  VMware has released a KB article about their parts of it, check it out here.  One interesting thing in the KB is that the upload portion of this issue does not effect vCloud 5.5 as this uses the Client Integration Plugin.

Update 3: Setting the security level to “Medium” also seems to work for most things.

First, I needed to configure an interface to reach my storage and create an NFS datastore, easy enough:

~ # esxcfg-vswitch -a vSwitch1

~ # esxcfg-vswitch -L vmnic2 vSwitch1

~ # esxcfg-vswitch -L vmnic3 vSwitch1

~ # esxcfg-vswitch -A "IP Storage" vSwitch1
~ # esxcfg-vswitch -l 
Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks 
vSwitch0         3802        6           128               1500    vmnic0,vmnic1 
PortGroup Name        VLAN ID  Used Ports  Uplinks 
VM Network            0        0           vmnic0,vmnic1 
Management VMs        0        0           vmnic0,vmnic1 
Management Network    0        1           vmnic0,vmnic1 

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks 
vSwitch1         3802        6           128               1500    vmnic2,vmnic3 
PortGroup Name        VLAN ID  Used Ports  Uplinks IP 
Storage            0        1           vmnic2,vmnic3

~ # esxcfg-vmknic -a -i 10.1.1.20 -n 255.255.255.0 "IP Storage"

~ # esxcfg-vmknic -l

Interface  Port Group/DVPort   IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type

vmk0       Management Network  IPv4      10.1.0.20                               255.255.255.0   10.1.0.255      18:a9:05:58:63:7c 1500    65535     true    STATIC

vmk1       IP Storage          IPv4      10.1.1.20                               255.255.255.0   10.1.1.255      00:50:56:66:1e:35 1500    65535     true    STATIC

~ #

Next, I just need to mount the NFS Storage I had presented from OpenFiler (that is the topic of another article):

~ # esxcfg-nas -a -o 10.1.1.10 -s /mnt/aggr0/sharednfs/management0 "SharedNFS_Management0"

Now, double check it mounted:

~ # esxcfg-nas -l

SharedNFS_ManagementVMs is /mnt/aggr0/sharednfs/management0/ from 10.1.1.10 mounted available

~ #

Next, I installed OVFTool on my OpenFiler machine since it is just linux after all and I was working remotely so I didn’t want to have to push the vCenter appliance over a VPN incase it got interrupted, I could also install this directly in OSX.  I uploaded the ova file to a place on the local drive of my OpenFiler machine overnight.

Now to deploy:

[root@filer ~]# ovftool '--net:Network 1=VM Network' --datastore="SharedNFS_ManagementVMs" --diskMode=thin --name=vcenter.skunkworks.dohmein.com VMware-vCenter-Server-Appliance-5.5.0.5101-1398493_OVF10.ova vi://root:password@10.1.0.20 Opening OVA source: VMware-vCenter-Server-Appliance-5.5.0.5101-1398493_OVF10.ova The manifest validates Source is signed and the certificate validates Opening VI target: vi://root@10.1.0.20:443/ Deploying to VI: vi://root@10.1.0.20:443/ Transfer Completed Completed successfully [root@filer ~]#

Lets power this bad boy on, back on the ESXi Host:

~ # vim-cmd vmsvc/getallvms

Vmid                Name                                                     File                                           Guest OS      Version                                Annotation

6      vcenter.skunkworks.dohmein.com   [datastore1] vcenter.skunkworks.dohmein.com/vcenter.skunkworks.dohmein.com.vmx   sles11_64Guest   vmx-07    VMware vCenter Server Appliance

Version 5.5 of VC running on SLES 11

~ # vim-cmd vmsvc/power.on 6

Powering on VM:

~ #

Now here is the trick, how do I see where it is in the boot up process and to see what IP DHCP gave it.  Well, as it turns out, there is a path you can hit on an ESX host to get a screenshot of the VM, it is as follows:

https://<ip/hostname>/screen?id=<moid> (ID from the getallvms list above)

I am greeted with this:

Since I can’t login to the console without a windows computer, I decided to just try rebooting this VM to see if I can get DHCP to work (previously set up, had a flaky trunk between switches). Back to the ESXi command line:

~ # vim-cmd vmsvc/power.reset 6

Reset VM:

~ #

More refreshing of the screenshot page:

And here we go, now we have an IP address:

Sweet!  Now to just visit the address and finish the configuration!  No windows necessary!

Here is an example:

Now, this may seem great from a readability perspective but its terrible if you want to use excel (or anything for that matter) to sort or manipulate any data because it doesn’t know how to deal with the units.

What I really wanted was something like this:

Ruby to the rescue, I created a quick and dirty script to strip all of those ugly units out:

It will place the units from the first data line in the headers, there currently isn’t any checking on this so even if it doesnt make sense it will add it (like the Object column for example).

There are a few things that I will probably add in time, for example:

• Normalize units, i.e. GB/TB -> MB
• Convert KBps to IOPS (if possible)
• remove %
• Clean up headers

Other than that, its a great little hack I think.  Let me know any ideas.

The Code

The first major thing is that the first version of the dashboard was written using a framework called Dashing which is a sweet frame work for developing dashboards from any datasource really quickly but, as it turns out, is not great at efficiently handling hundreds of connections.

Because of this I re-wrote the dashboard from scratch using a simple Sinatra app, it ended up only being about 40 lines of actual logic (not including the HTML part) to get the job done, pretty awesome.

This simplicity also allowed me to throw some memcache caching up infront of it to handle just about any traffic I can throw at it.  I actually tested up to 250 hits/second using blitz.io and it functioned flawlessly.  Awesome.

I also used a totally sweet framework called isotope to do the layout of the tiles on that page that will allow me to do some even cooler things in the future.  One thing I did add though was the ability to filter by category, just click on the buttons in the toolbar to drill down a little, you can get views like this for LUNs:

The Stats

In this version we wanted to bring to the front a few new stats, most importantly being the storage vendor statistics.

1. Storage Stats – These stats may be interesting to some folks as NetApp is not among the top storage vendors on this list.  My theory on this is because NetApp admins typically (from my experience) provision large NFS volumes versus smaller VMFS volumes.  This stat is based on the number of LUNs not the total storage provided by each vendor (which would also be interesting to see, that will come later) so this list did not really surprise me personally.  As a little tease for what is to come,  I included a screenshot of the top 15 vendors sorted by amount of storage, interesting results as well.

• Environment Types – Another thing we decided to add was a breakdown of the environment types that we have gotten so far.  We thought this would be a great thing to display as it would give everyone an idea of what kind of data we have so far and who we need to bug as a community to upload.  I was happy to see that we had a huge portion of the environments being from production server based infrastructures, awesome work.
  
• Host Hardware – We increased the number of host vendors to 5 (after many asks from all of you!) and this shows basically what I expected as well.  I did expect Cisco to have a higher percentage but I suppose those are the larger environments that may not have reported in yet.  
• Country Standings – One note about these, I was actually not calculating these correctly (dividing submissions by number of users, doh!) so that is why these seem to have changed drastically.  The data is the same, I just am mathing correctly now.

Data Curation

People were asking if we combine DGC (EMC Clariion based storage) and EMC?

It seems like the census is to spend some time curating the data, combining Dell and Dell Inc for example, so I will be working on some code to do that as part of the next update.  This will probably be asynchronous (run as a job on the whole dataset as a whole on a schedule) so it won’t necessarily do it instantly on upload, this is to keep the time it takes to insert the data to a minimum.

The Future

Next up, the future.  I have been doing a lot of thinking about where to take this project in the future.  I was really astounded by the response that we have gotten from the community, you all have blown me away.  I really have some cool ideas for this project that I am not quite ready to talk about but I think everyone will really love.

My goal is to be able to give everyone the ability to mix this data as they want to see what the community can come up with with all of this data, I am sure some of you have some amazing ideas.  If you do have anything in particular send me an email and I would love to chat about it.

I also am looking to give some filtering and mashing up of the data on the private dashboards to narrow the scope of some of the averages, I know that is what people are really looking for.

One more thing (channeling The Steve), the Microsoft sized gorilla in the room.  I have thought about the ability to gather stats from other vendors environment in a similar manor to compare and I think this would be really amazing.  If any one is (or knows of) a rockstar that can work with the Hyper-V, Citrix, or OpenStack APIs get them in touch with me on twitter or by email.

Until then my head is going to be buried in refactoring and improving the main site codebase to get something really great out to the community, stand by.