random writes

A few VMware Virtual Volumes (VVol) tips

2016-10-02T00:00:00+00:00

VMware Virtual Volumes (VVols) are a vSphere feature which allow for storing VM virtual disks and other files natively on the storage system (instead of on VMFS datastores created on top of LUNs) and managing their placement on the storage system through vSphere storage policies. VVols are available since vSphere 6.0 and require a vSphere Standard or Enterprise Plus license.

VVols are a pretty fascinating piece of technology, which I’m guessing is set to completely replace VMFS datastores in a few years time. Since this is a fairly new feature without many implementation tips available online, here are a few things to bear in mind when deploying VVols:

in order to use VVols, besides a VVol certified storage system, HBAs in your ESXi hosts need to support Secondary LUN IDs - to check whether this is the case, refer to the the VMware Compatibility Guide for IO Devices and be sure to select Secondary LUNID (Enables VVols) in the Features section
- if your hosts are running ESXi 6, you can check for HBA VVol compatibility from the command line by running esxcli storage core adapter list and noting whether “Capabilites” column contains Second Level Lun ID for the vmhbas in question
although maximum data VVol (= virtual disk) size in vSphere 6.0 is 62 TB, you should check what are the limits on the storage system side, because they could be much less and force you to create multiple VVol datastores on the same system in order to consume its full available capacity; one such example is EMC Unity which allows you to consume up to 16 TB of a single storage pool for a VVol datastore
migration of VMs from VMFS to VVol datastores is performed using Storage vMotion and should be initiated solely from the web client; the vSphere C# client doesn’t offer the option to select a storage policy in the Storage vMotion wizard, which results in user-created storage policies being ignored and migrated VMs always being assigned the default “VVol No Requirements Policy”
if you are using Veeam Backup and Replication, VVols are supported since VBR 8.0U2b but with a few caveats:
- direct SAN transport mode is not supported
- Virtual Appliance (Hot Add) processing mode requires that all VBR proxy VM disks are located on the same VVol with the processed VM.

Trying out Docker client for Windows

2015-07-18T00:00:00+00:00

Docker client for Windows was released a few months ago, and recently I installed it inside of a Windows 10 CTP machine and tested it against a CentOS 7 serving as a Docker host.

For the installation I employed the Chocolatey package manager:

C:\> choco install docker -y

which will install the latest version of the client (1.7.0 at the time of writing).

By default, docker daemon listens only on unix:///var/run/docker.sock socket and therefore accepts only local connections, so in order to access it from the outside I needed to bind it to a TCP port. So I first stopped the Docker daemon on my container host:

$ sudo systemctl stop docker

and then ran it like this so it binds to TCP port 2375 in addition to the default socket:

$ sudo docker -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock -d &

Note: This was only for the purpose of a quick-and-dirty test, and you can read here why allowing the daemon to accept remote calls in this way isn’t such a great idea security-wise.

After this, everything was ready for connecting from my Windows client, or so I thought. I tried listing images on the docker host but was greeted with the following error:

C:\> docker -H 10.0.0.1:2375 info
Error response from daemon: client and server don't have same version (client : 1.19, server: 1.18)

After a quick docker -v, I realized that the daemon was running Docker version 1.6.2, while the client was running version 1.7.0.

I had no choice but to fire up Chocolatey again and install version 1.6.0 of the client:

C:\> choco uninstall docker -y
C:\> choco install docker -version 1.6.0 -y

and then everything worked as expected:

C:\> docker -H 10.0.0.1:2375 info
Containers: 0
Images: 33
Storage Driver: devicemapper
...
Kernel Version: 3.10.0-123.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 1
Total Memory: 458.4 MiB
Name: dockerhost
ID: DBST:ZQ5O:LOAJ:XG7D:FEBL:6FYO:2JZ3:CQN2:QPK6:6ARN:XBEZ:THVQ

If you go back to the Docker host, you’ll notice that your client commands result in calls to the Docker API:

...
INFO[0680] GET /v1.18/info                              
...

where v1.18 is the API version and when a client is running on a higher version than the daemon, a mismatch occurs in the API calls (trying to access /v1.19/info instead of /v1.18/info and similar). This explains the problem I originally had, although at the time it seemed kinda odd to me that you cannot use a higher version client to access a lower version daemon.

In order to avoid having to specify your Docker host parameters with every command, you can set a Windows environment variable DOCKER_HOST and assign it a value of tcp://<FQDN or IP of Docker host>:<TCP port>, e.g. tcp://10.0.0.1:2375 in my case. Afterwards, you can run the commands on your Windows client machine as if you’re located directly on the Docker host:

C:\> docker run -it ubuntu
root@825574d22c14:/# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.1 LTS"
root@825574d22c14:/# exit
C:\>

Pretty crazy, huh :) Kudos to the Microsoft Azure Linux team for porting the Docker client to Windows, and be sure to read more about their efforts on Ahmet Alp Balkan’s blog.

NetApp cDOT and ESXi SCSI UNMAP support

2015-06-22T00:00:00+00:00

I have recently noticed that LUNs created on NetApp cDOT storage systems by default don’t support space reclamation (aka SCSI UNMAP) which is a part of the vSphere VAAI Thin Provisioning primitive:

~ # esxcli storage core device vaai status get
naa.600a0980424733322f5d444f76513730
   VAAI Plugin Name: VMW_VAAIP_NETAPP
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: unsupported

and when you try to reclaim space, you are greeted with the following message:

~ # esxcli storage vmfs unmap -l cdot_01
Devices backing volume 549a787c-f5a4021c-e59d-68b599cbd47c do not support UNMAP

Based on a short review of a few systems, this seems to be the case only for the cDOT platform (tested on Ontap 8.2 and 8.3), while LUNs created on 7-mode systems show Delete Status: supported.

Explanation for this behavior can be found in the NetApp cDOT official documentation, namely the SAN Administration Guide. The document states that in order for space reclamation to be supported on a LUN:

LUN needs to be thinly provisioned on the storage system
space-allocation option needs to be enabled on the LUN

Information on how to enable space-allocation on a LUN is available in the same guide:

lun modify -vserver <vs> -volume <vol> -lun <lun> -space-allocation enabled

What’s a bugger is that the LUN needs to be offline in order to run this command, so best thing to do would be to enable this prior to presenting the LUN to the hosts.

One more thing to have in mind is that the VMware Compatibility Guide explicitly states that “VAAI Thin Provisioning Space Reclamation is not supported” for ESXi >= 5.5, and Data Ontap < 8.3 (both clustered and 7-mode). So, even if your NetApp LUNs are configured properly for space reclamation, be sure that you are running a supported configuration before initiating a SCSI UNMAP operation from an ESXi host.

Centralized ESXi syslog with ELK feat. SexiLog

2015-04-06T00:00:00+00:00

The ELK stack is a commonly used open source solution for log collection and analysis. It consists of three components:

Elasticsearch - distributed search engine, responsible for storing and searching through the received data;
Logstash - log collector, parser and forwarder;
Kibana - Elasticsearch web frontend, the end user interface for searching and visualizing log data.

There are a lot of tutorials on setting up the ELK stack, but if you’re looking into implementing ELK as a centralized (sys)log server for your vSphere environment, you should probably look no further than SexiLog.

To quote the official site - SexiLog is a specific ELK virtual appliance designed for vSphere environment. It’s pre-configured and heavily tuned for VMware ESXi logs. This means that the nice folks from Hypervisor.fr and VMdude.fr have gone through the trouble of installing ELK, optimizing it for vSphere log collection, adding a bunch of other useful tools and packaging everything as an easy to deploy VMware virtual appliance.

The appliance is a Debian-based VM with 2vCPU, 4GB RAM and 58GB hard drive space (8GB system disk + 50GB disk for storing indexed logs), and is sized for collecting up to 1500 events per second. Default credentials for the appliance are root / Sex!Log and after each log in you will be greeted with a menu that can be used for basic configuration and operations. Since this “SexiMenu” is a bash script which can be run manually from /root/seximenu/seximenu.sh, if you want to avoid it and log in directly to shell each time, you can simply comment out the last three lines from /root/.bashrc.

After deploying the appliance, you’ll need to configure your ESXi hosts to start sending syslog data, which can be done manually through vSphere client or ESXCLI or in an automated manner, e.g. with PowerCLI. Besides syslog, SexiLog also offers the possibility of collecting SNMP traps (and has Logstash SNMP grok patterns optimized for ESXi and Veeam Backup and Replication), as well as vCenter vpxd logs and Windows event logs with the help of the NXLog agent. For more info, RTFM :)

SexiLog goodies

SexiLog offers much more than just a vanilla ELK installation. Besides providing Logstash filters optimized for vCenter, ESXi and Veeam B&R log and SNMP trap collection, it also comes with a bunch of pre-configured Kibana dashboards.

Then, there is a notification service powered by Riemann, which receives warnings and alerts from Logstash and either sends them every 10 minutes (critical alerts) or aggregates them and e-mails them once per hour (all other alerts). You can check out which events are considered critical by looking at Logstash configuration files located at /etc/logstash/conf.d/ and searching for rules which add the achtung tag. In order for Riemann to do its job, it is necessary to configure SMTP parameters for the appliance, which can be done through the SexiMenu (option 7).

Another important part of SexiLog is Curator, which is used for purging Elasticsearch indices, in order for them not to fill up the /dev/sdb1 partition used for storing logs. Curator runs once per hour, as defined in /etc/crontab:

5 * * * * root curator delete --disk-space 40

and takes maximum allowed size of indices in GB as the input parameter. By default this is set to 40[GB] and should be changed if you decide to extend SexiLog’s hard disk no. 2.

Also, for more info about the health and performance of the Elasticsearch service, SexiLog provides two useful plugins - Head and Bigdesk, which can be accessed from http://<sexilog_IP_or_FQDN>/_plugin/head and http://<sexilog_IP_or_FQDN>/_plugin/bigdesk.

Feature requests

Two possible improvements have come to my mind while using SexiLog. First, it would be nice if there was a way to provide user authentication for accessing the Kibana interface (maybe kibana-authentication-proxy could help with this since Shield seems to be a commercial product?). Also, an option to export search results would be useful (e.g. when dealing with techical support), but this seems to be a current limitation of Elasticsearch/Kibana.

Further steps

For more information about SexiLog check out the site, and be sure to try out the demo. You can also contribute to this great project via Github.

Exploring VCSA embedded PostgreSQL database

2015-03-21T00:00:00+00:00

Since vSphere 5.0U1, VMware vCenter Server Appliance (VCSA) uses vPostgres - VMware flavored PostgreSQL as the embedded database. This post describes how to connect to the VCSA vPostgres server locally and remotely, and perform database backups using native PostgreSQL tools.

Note: Following procedures are probably unsupported by VMware and are given here just for the fun of hacking around VCSA. The instructions have been tested for VCSA 5.5 and 6.0 (VCSA 6.0 requires additional steps which can be found at the bottom of the post).

Connecting to PostgreSQL server locally

After logging to the VCSA over SSH or console, you can easily connect to the PostgreSQL server locally using psql:

/opt/vmware/vpostgres/current/bin/psql -U postgres

After connecting you can use psql or regular SQL commands, e.g.

# /opt/vmware/vpostgres/current/bin/psql -U postgres
psql.bin (9.3.5 (VMware Postgres 9.3.5.2-2444648 release))
Type "help" for help.

postgres=# \du
                             List of roles
 Role name |                   Attributes                   | Member of 
-----------+------------------------------------------------+-----------
 postgres  | Superuser, Create role, Create DB, Replication | {}
 vc        | Create DB                                      | {}

Here we see that there are two users defined in the PostgreSQL server, the postgres superuser, and the vc user, used by vCenter for connecting to its database.

Enabling remote PostgreSQL server access

By default, only local connections to the database server are allowed. In order to allow remote access (e.g. so that you can connect via GUI based administrative tools such as pgAdmin), first take a look at the following files:

/etc/vmware-vpx/embedded_db.cfg

/etc/vmware-vpx/vcdb.properties

embedded_db.cfg file stores general PostgreSQL server information (as well as the password for the postgres superuser), while vcdb.properties stores connection information for the vCenter server database VCDB, along with the password for the vc user. Take a note of these passwords, since you’ll be required to supply them for remote access.

Then, edit the /storage/db/vpostgres/pg_hba.conf configuration file in order to allow your IP to connect to the PostgreSQL server by adding the following line:

host    all             all             1.2.3.4/24          md5

replacing 1.2.3.4/24 with the actual IP address or range of addresses for which you want to allow access (e.g. 192.168.1.0/24).

Next, edit the /storage/db/vpostgres/postgresql.conf in order to configure PostgreSQL server to listen on all available IP addresses by adding the following line:

listen_addresses = '*'

Finally, restart the PostgreSQL server by running

/etc/init.d/vmware-vpostgres restart

Backing up the vCenter database

For information on using native PostgreSQL tools to perform VCDB backups and restores check out VMware KB 2034505. The requirement for the vCenter service to be stopped during the database backup seems kinda redundant, since pg_dump should perform consistent backups even if the database is in use.

Sample backup scripts and instructions on how to schedule them via cron can be found on Florian Bidabe’s and vNinja blogs. Since mount.nfs is available on the VCSA, it seems that you can even use an NFS share as a destination for your VCDB backups (haven’t tested it though).

VCSA 6.0 additional steps

VCSA 6.0 comes extra hardened compared to previous vSphere editions and additional steps are needed in order to allow remote access to the OS and then to the PostgreSQL server.

First, you need to enable SSH access to VCSA. This can be done during the deployment, or later, over VM console (similar interface to ESXi DCUI: F2 - Troubleshooting Mode Options - Enable SSH) or vSphere Web Client (Home - System Configuration - Nodes - Manage - Settings - Access).

After logging as root over SSH, you will be greeted with a limited shell called appliancesh. To switch to bash temporarily, run:

shell.set --enabled True
shell

For switching to bash permanently you can follow the instructions from this virtuallyGhetto post.

The final step is to allow external access to the PostgreSQL through the VCSA IPTables-based firewall. This can be done by editing the /etc/vmware/appliance/firewall/vmware-vpostgres file so that it looks like this:

{
  "firewall": {
    "enable": true,
    "rules": [
      {
        "direction": "inbound",
        "protocol": "tcp",
        "porttype": "dst",
        "port": "5432",
        "portoffset": 0
      }
    ]
  },
  "internal-ports": {
    "rules": [
      {
        "name": "server_port",
        "port": 5432
      }
    ]
  }
}

Afterwards, reload VCSA firewall by running

/usr/lib/applmgmt/networking/bin/firewall-reload

and PostgreSQL server should be accessible from the outside world after configuring pg_hba.conf and postgresql.conf as described above.

Trying out CoreOS on vSphere

2015-03-12T00:00:00+00:00

CoreOS is a minimal Linux distribution created to serve as a host for running Docker containers. Here’s some info on how to try it out on vSphere.

Since stable release 557, CoreOS comes packaged as an OVA and is supported as a guest OS on vSphere 5.5 and above. Deployment instructions can be found in VMware KB 2109161 and they are pretty much straightforward, with the exception of the need to modify boot parameters in order to change the password for the core user before logging in for the first time.

vCenter customization of CoreOS is currently not supported and customization can be done only through coreos-cloudinit. This is a CoreOS implementation of cloud-init, a mechanism for boot time customization of Linux instances, which can be employed on vSphere with some manual effort. More info on coreos-cloudinit and its configuration files called cloud-config can be found in the CoreOS Cloud-Config Documentation page, while for a tutorial on how to make Cloud-Config work in vSphere take a look at CoreOS with Cloud-Config on VMware ESXi.

CoreOS comes with integrated open-vm-tools, an open source implementation of VMware Tools, which is quite handy since CoreOS doesn’t offer a package manager or Perl, so no way to manually install VMware Tools after deployment. According to VMware Compatibility Guide, the open-vm-tools package has recently become the recommended way for running VMware Tools on newer Linux distros and vSphere editions (e.g. RHEL/CentOS 7.x and Ubuntu >=14.04 on vSphere 5.5 and above). For more info on open-vm-tools head to VMware KB 2073803.

As for CoreOS stable releases prior to 557, releases 522 and 494 are supported on vSphere as a Technical Preview. Since they don’t come prepackaged in a format that can be used directly on vSphere, their deployment involves one additional step - converting the downloaded .vmx file to OVF via OVF Tool. Check out VMware KB 2104303 for detailed installation instructions.

CoreOS quickstart

After logging in for the first time, you’ll probably want to start building and running containers. Obviously, Docker comes preinstalled and CoreOS currently uses BTRFS as the filesystem for storing images and containers:

$ docker -v
Docker version 1.4.1, build 5bc2ff8-dirty
$ docker info
Containers: 0
Images: 23
Storage Driver: btrfs
 Build Version: Btrfs v3.17.1
 Library Version: 101
Execution Driver: native-0.2
Kernel Version: 3.18.1
Operating System: CoreOS 557.2.0
CPUs: 2
Total Memory: 5.833 GiB
Name: core557
ID: 2BUV:642W:WTZQ:3L4O:FFIY:JOC5:XKO2:3QPC:ADEJ:LSCS:QS5K:XHKB

As mentioned before, CoreOS doesn’t offer a way to install additional packages directly to the OS, but provides Toolbox, which is by default a stock Fedora Docker container that can be used for installing sysadmin tools. Toolbox can be run using /usr/bin/toolbox, first execution of which will pull and run the container:

$ toolbox 
fedora:latest: The image you are pulling has been verified
00a0c78eeb6d: Pull complete 
834629358fe2: Pull complete 
834629358fe2: Pulling fs layer 
Status: Downloaded newer image for fedora:latest
core-fedora-latest
Spawning container core-fedora-latest on /var/lib/toolbox/core-fedora-latest.
Press ^] three times within 1s to kill container.
-bash-4.3#

After that, packages are yum install <package> away, while CoreOS filesystem is mounted inside the container to /media/root.

List host's modified advanced settings via ESXCLI

2015-02-06T00:00:00+00:00

Just wanted to share a great tip by Ather Beg on how to display all of ESXi host’s advanced settings which are different than the default values via ESXCLI:

esxcli system settings advanced list -d

This will return a list of advanced settings similar to:

...

   Path: /Net/TcpipHeapSize
   Type: integer
   Int Value: 32
   Default Int Value: 0
   Min Value: 0
   Max Value: 32
   String Value: 
   Default String Value: 
   Valid Characters: 
   Description: Initial size of the tcpip module heap in megabytes. (REQUIRES REBOOT!)

   Path: /NFS/MaxVolumes
   Type: integer
   Int Value: 256
   Default Int Value: 8
   Min Value: 8
   Max Value: 256
   String Value: 
   Default String Value: 
   Valid Characters: 
   Description: Maximum number of mounted NFS volumes. TCP/IP heap must be increased accordingly (Requires reboot)

...

where Int Value / String Value (depending whether the parameter stores its value as integer or string) shows the current value for the parameter, while Default Int Value / Default String Value show its default value.

Source: ProTip: How to remind yourself of Advanced Settings changes in ESXi

vSphere CBT bug (and PowerCLI to the rescue)

2015-01-24T00:00:00+00:00

A few months ago, VMware published information about a pretty catastrophic vSphere bug described in VMware KB 2090639. TLDR version: if you have expanded a CBT-enabled .vmdk past any 128GB boundary (e.g. 128GB, 256GB, 512GB etc.), all your backups since are possibly corrupted.

The above KB article provides workaround information for this issue, which involves resetting CBT for the affected VMs. This can be accomplished either via shutting down the VM and editing its Configuration Parameters or on-the-fly via PowerCLI / VDDK API. Veeam KB 1940 provides PowerCLI code which can be employed for resetting CBT for all CBT-enabled VMs in the vCenter inventory, without the need for shutting them down first.

I have adapted this code to a quick-and-dirty PowerCLI script, which can be found here (for those not familiar with Github, clicking on raw will take you directly to the script).

Just take notice that the script will create and instantly delete a snapshot for all CBT-enabled VMs, so it’s probably a good idea not to execute the script during work hours. Also, the first following backup will last longer, since backup software won’t be able to use CBT information and will have to scan the whole .vmdk for changed blocks.

Downloading files with wget on ESXi

2015-01-15T00:00:00+00:00

A limited set of familiar POSIX-like tools and utilities is available on ESXi, courtesy of the busybox executable:

~ # /usr/lib/vmware/busybox/bin/busybox --list
[
[[
addgroup
adduser
ash
awk
basename
cat
chgrp
chmod
chown
chvt
cksum
clear
cp
crond
cut
date
dd
delgroup
deluser
diff
dirname
dnsdomainname
du
echo
egrep
eject
env
expr
false
fdisk
fgrep
find
getty
grep
groups
gunzip
gzip
halt
head
hexdump
hostname
inetd
init
kill
ln
logger
login
ls
lzop
lzopcat
md5sum
mkdir
mkfifo
mknod
mktemp
more
mv
nohup
nslookup
od
passwd
poweroff
printf
readlink
reboot
reset
resize
rm
rmdir
sed
seq
setsid
sh
sha1sum
sha256sum
sha512sum
sleep
sort
stat
stty
sum
sync
tail
tar
tee
test
time
timeout
touch
true
uname
uniq
unlzop
unzip
usleep
vi
watch
wc
wget
which
who
xargs
zcat

As you can see, one of the tools present is wget which can be used for downloading files (e.g. installation ISOs, VIBs, offline bundles..) directly from the ESXi Shell, instead of first downloading locally to your desktop or jumphost and then uploading to hosts or datastores.

First, connect to ESXi Shell over SSH or DCUI and cd into the destination directory, which can be e.g. a shared datastore available to all hosts in your cluster:

cd /vmfs/volumes/datastore_name_here

or host’s /tmp directory (common choice for VIBs and offline bundles since it gets emptied on reboot). After that, just fire wget away as wget <file URL>:

~ # cd /vmfs/volumes/ISO_images/
.. # wget http://releases.ubuntu.com/14.04.1/ubuntu-14.04.1-server-amd64.iso
Connecting to releases.ubuntu.com (91.189.92.163:80)
ubuntu-14.04.1-serve 100% |*****************************|   572M  0:00:00 ETA

Here we downloaded Ubuntu Server installation ISO directly to our “ISO_images” datastore.

Direct installation of VIBs from an URL

Downloading installation ISOs is far from a best practice, since it’s probably not the best idea to use your host’s resources for downloading large files from the Internet, but the wget approach can save you same time if you’re often manually installing VIBs or offline bundles on your ESXi hosts.

For VIB files, an alternative to wget-ing and then installing the VIB would be to directly supply the URL of the VIB file to the esxcli software vib install command:

esxcli software vib install -v <URL of the VIB file>

e.g.

/tmp # esxcli software vib install -v http://vibsdepot.v-front.de/depot/DAST/iperf-2.0.5/iperf-2.0.5-1.x86_64.vib --no-sig-check
Installation Result
   Message: Operation finished successfully.
   Reboot Required: false
   VIBs Installed: DAST_bootbank_iperf_2.0.5-1
   VIBs Removed: 
   VIBs Skipped:

Here we installed hypervisor.fr iperf for ESXi VIB directly from the V-Front Software Depot. --no-sig-check is there to bypass signature verification, since the iperf VIB we are installing doesn’t have one.

As esxcli software vib install --help informs us, the supplied URL can be HTTP, HTTPS or FTP. URLs can be supplied only to the command’s -v switch, so this approach is limited to VIB files and not available for offline bundles.

Search available ESXCLI commands

2014-11-28T00:00:00+00:00

A way to search through available ESXCLI commands from ESXi Shell, vCLI or vMA:

esxcli esxcli command list | grep <keyword>

E.g.

~ # esxcli esxcli command list | grep snmp
system.snmp                                             get         
system.snmp                                             hash        
system.snmp                                             set         
system.snmp                                             test