I recently needed to migrate all VMs from one vCenter to another. I had planned to use the native Cross vCenter vMotion GUI, but I ran into a couple of issues. Although using it for one VM works fine-ish, using it for multiple VMs at a time becomes quite a burden.

The main problems stem from the GUI wants you to pick the portgroup and datastore on the destination vCenter. In theory, this makes sense as there might be differing storage at the destination vCenter. However, it doesn’t inform you during the process where the VMs currently reside. This is a bit tough to explain, so here is a screenshot:

In the picture, we see we are trying to migrate 2 VMs, and it wants destination storage to be selected. It gives no indication though of where the VMs are coming from. If identical storage is presented at both locations, and you want the VMs to land on the same datastore, there isn’t a way to do it through this GUI. You would need to first document where each VM is located before even beginning the migration process, which is silly.

If you do one VM at a time, you are given a VM Origin link during migration to check the source location:

But this doesn’t exist when migrating in batches.

With all this in mind, I created a script to assist with this process. In my case, identical storage and network port names were presented at each location so for the selection process, I only needed to have the script choose based on the port and datastore names the VM was already utilizing.

Two other caveats for this script. The first being I was paranoid about the VMs landing ok at the new vCenter. This means I have multiple networking checks both before and after the migration. Not all VMs have guest names, or even VMtools to check guest names/IPs, so I am checking networking by VMname, IP resolved by VMname, Guestname(where applicable), and Guest IP. So yeah, a little paranoid. When everything is working all right, the checks look something like this:

We can see it does check the IPv4 IP and VMname, before resolving some of the other stuff to IPv6. Regardless there were enough greens for me to proceed, where it would then do the same checks after the migration. Another thing to note is it would prompt before starting the migration, because paranoid.

The second caveat with the script is it will automatically ignore VMs with multiple portgroups or multiple datastores. In my use case, few VMs would fall into this category, and my paranoia demanded I do them manually anyway.

# Connect to Source and Target vCenter Servers
$sourceVC = "SourcevCenter"
$targetVC = "TargetvCenter"

# Define the list of VM names to migrate
#$vmNames = "SingleVM"  # Replace with the names of VMs you want to migrate
$targetClusterName = "Clustername"  # Replace with the target cluster name in the target vCenter
$vmhost = "VMHost" # Used for clearing out a specific ESXi host

# Connect to both vCenter servers
Connect-VIServer -Server $sourceVC 
Connect-VIServer -Server $targetVC 

# Get VMs hosted on the specified ESXi host from the source vCenter
$vmnames = Get-VMHost $vmhost -Server $sourceVC | Get-VM # Can be modified to get all VMs on vCenter by removing vmhost reference

# Loop through each VM and perform migration
foreach ($vmName in $vmNames) {
    
    # Retrieve the VM from the source vCenter
    $vm = Get-VM -Name $vmName -Server $sourceVC
    "VM"
    $vm

    # Get VM guest information
    $vmGuestInfo = Get-VMGuest -VM $vm
    $ipv4Addresses = $vmGuestInfo.IPAddress | Where-Object { $_ -match '^\d{1,3}(\.\d{1,3}){3}$' } | Select-Object -First 1

    # Initial network connectivity test
    Write-Host "Testing initial connectivity with IP $ipv4Addresses" -ForegroundColor Yellow
    $initialping = Test-NetConnection $ipv4Addresses 

    # Check initial ping response
    if ($ipv4Addresses -eq $null) {
        Write-Warning "No IPv4 Address found"
    } elseif ($initialping.PingSucceeded) {
        Write-Host "Initial ping succeeded" -ForegroundColor Green
    } else {
        Write-Warning "Initial ping failed"
    }

    # Test connectivity by VM Name
    Write-Host "Testing connectivity with VM Name $($vm.Name)" -ForegroundColor Yellow
    $pingtest = Test-NetConnection $vm.Name
    if (-not $pingtest.PingSucceeded) {
        Write-Warning "Ping failed for VM Name $($vm.Name)"
    } else {
        Write-Host "Ping succeeded for VM Name $($vm.Name)" -ForegroundColor Green
    }

    # Test connectivity by VM Guest Name
    Write-Host "Testing connectivity with VM Guest Name $($vmGuestInfo.HostName)" -ForegroundColor Yellow
    $pingtest = Test-NetConnection $vmGuestInfo.HostName
    if (-not $pingtest.PingSucceeded) {
        Write-Warning "Ping failed for VM Guest Name $($vmGuestInfo.HostName)"
    } else {
        Write-Host "Ping succeeded for VM Guest Name $($vmGuestInfo.HostName)" -ForegroundColor Green
    }

    # Retrieve source VM's datastore and portgroup
    $sourceDatastore = (Get-HardDisk -VM $vm | Get-Datastore)
    $sourcePortGroup = Get-VM $vm | Get-VirtualPortGroup -Distributed
    $sourcedatastorecenter = $sourceDatastore.Uid.Split("@")[1].Split("/")[0]

    Write-Host "Source Information:" -ForegroundColor Cyan
    $sourcePortGroup.Name
    $sourceDatastore.Name
    $sourcedatastorecenter

    # Validate VM network settings (Portgroup and Datastore)
    if ($sourcePortGroup.Count -gt 1) {
        Write-Warning "$vm has multiple portgroups, skipping."
        continue
    } elseif ($sourcePortGroup.Count -lt 1 -or $sourcePortGroup -eq $null) {
        Write-Warning "$vm has no portgroups, skipping."
        continue
    }

    if ($sourceDatastore.Count -gt 1) {
        Write-Warning "$vm has multiple datastores, skipping."
        continue
    } elseif ($sourceDatastore.Count -lt 1 -or $sourceDatastore -eq $null) {
        Write-Warning "$vm has no datastores, skipping."
        continue
    }

    # Retrieve corresponding datastore and portgroup in the target vCenter
    $targetDatastore = Get-Datastore -Name $sourceDatastore.Name -Server $targetVC
    $targetPortGroup = Get-VirtualPortGroup -Name $sourcePortGroup -Server $targetVC -Distributed
    $targetdatastorecenter = $targetDatastore.Uid.Split("@")[1].Split("/")[0]
  
    $targetCluster = Get-Cluster -Name $targetClusterName -Server $targetVC
    $targetHost = $targetCluster | Get-VMHost | Get-Random -Count 1

    Write-Host "Destination Information:" -ForegroundColor Cyan
    $targetPortGroup.Name
    $targetPortGroup.Uid.Split("@")[1].Split("/")[0]
    $targetDatastore.Name
    $targetdatastorecenter
    $targetHost.Name

    # Confirm migration
    $response = Read-Host "Are you sure you want to continue? (y/n)"
    if ($response -ne "y") {
        Write-Warning "Aborting migration for $vmName"
        continue
    }

    # Execute Cross vCenter vMotion
    Move-VM -VM $vm `
        -Destination $targetHost `
        -Datastore $targetDatastore `
        -NetworkAdapter (Get-NetworkAdapter -VM $vm) `
        -PortGroup $targetPortGroup `
        -Confirm:$true

    Write-Output "Migrated VM '$vmName' to target vCenter with portgroup '$sourcePortGroup' and datastore '$sourceDatastore'."

    # Post-migration connectivity check
    Start-Sleep -Seconds 2
    Write-Host "Testing post-migration connectivity with IP $ipv4Addresses" -ForegroundColor Yellow
    $pingtest = Test-NetConnection $ipv4Addresses
    if (-not $pingtest.PingSucceeded) {
        Write-Warning "Ping failed for VM, considering rollback."
    } else {
        Write-Host "Post-migration ping succeeded" -ForegroundColor Green
    }

    # Check post-migration connectivity by VM Name and Guest Name
    Write-Host "Testing with VM Name $($vm.Name)" -ForegroundColor Yellow
    $pingtest = Test-NetConnection $vm.Name
    if (-not $pingtest.PingSucceeded) {
        Write-Warning "Ping failed for VM Name $($vm.Name), considering rollback."
    } else {
        Write-Host "VM Name $($vm.Name) connectivity verified" -ForegroundColor Green
    }

    Write-Host "Testing with VM Guest Name $($vmGuestInfo.HostName)" -ForegroundColor Yellow
    $pingtest = Test-NetConnection $vmGuestInfo.HostName
    if (-not $pingtest.PingSucceeded) {
        Write-Warning "Ping failed for VM Guest Name $($vmGuestInfo.HostName), considering rollback."
    } else {
        Write-Host "VM Guest Name $($vmGuestInfo.HostName) connectivity verified" -ForegroundColor Green
    }
}

# Optional: Disconnect from vCenter Servers
# Disconnect-VIServer -Server $sourceVC -Confirm:$false
# Disconnect-VIServer -Server $targetVC -Confirm:$false

Some options for updates if I or someone else is ever interested, would be automatic rollback to source vCenter. In my case, with a good number of VMs “failing” tests because of inaccurate or inaccessible guest info this didn’t make sense, and with me monitoring each migrating I could manually move a VM back if necessary.

Second obvious one would be to update to account for multiple portgroups or datastores.

The script is also available on github, where I am going to start dumping more of the scripts I have written.

Continuing my journey with trying out varying Hypervisors, I thought Nutanix would be a great next stop after Proxmox.

Nutanix Community Edition is the free edition of the Nutanix Cloud Infrastructure platform and was first released in 2015. Although it is not meant for production use, it is a great way to get a feel for the Nutanix platform.

My overall install process was straightforward, with the only problems I experienced being of the normal homelab self-created variety. This post walking through the CE requirements and deployment process is very helpful. Unfortunately, my issues were caused by hopping right into trying to deploy without looking at the requirements as one might do in a homelab.

System Requirements

There are several other requirements listed in the post I mentioned, but the ones I wanted to focus on are these:

CPU

• Processor Type: Intel CPUs (Sandy Bridge or newer with VT-x and AVX) and AMD CPUs (Zen or newer).
• Core Requirement: Minimum of four cores, with two dedicated to the Controller VM (CVM).
• Intel E-core: Disable Intel Efficient cores (E-cores) in BIOS, as CE doesn’t support them.

System Memory

• Minimum: 32 GB of system memory.
• Recommended: 64 GB or more for AOS features like deduplication, compression, or multiple user VMs.

Storage

• Boot Device: At least 32 GB capacity, using a high-reliability device (internal or external, like SATA DOM, M.2 SSD, or SATA SSD/HDD).
• Cold Tier: Minimum of 500 GB, up to 18 TB (typically 3 x 6 TB HDDs), using HDD or SSD.
• Hot Tier (Flash): At least one non-NVMe SSD of 200 GB or larger.

I have a recent Intel NUC with 64GB of memory, so the CPU and Memory requirements weren’t an issue. Storage was my problem as I only had one NVMe disk installed. Once I booted up and realized I needed to select three different drives for different purposes it turned into a bit of a mess which took me a bit to get sorted.

I ended up with a 250 GB SSD I had lying around which I installed into the NUC, and I used a 64GB thumb drive for the boot device. With these, I was able to get the installation going.

Installation

To download and use Nutanix CE, I first needed to create an account. Here is the link. Shortly after signing up, I received an email with the direct link where I was able to download the iso for deployment.

After using Rufus to create a bootable thumb drive, I was able to start deploying to my NUC.

Deployment

Once Nutanix CE boots up I was presented with the installer.

I went through each disk and selected its role by using the “h”, “c”, or “d” keys. These are for the Hypervisor, CVM, and Data disks respectively.

Unlike VMware where a vCenter is needed to unlock most of the functionality, each node for Nutanix comes with its own “vCenter” in the form of the Cluster VM(CVM) which is always deployed. With this in mind, we see why this screen also requests separate IPs for the Host and the CVM.

Another note about the IPs is Nutanix will configure an internal IP for the CVM on 192.168.5.2. Although I am not sure of the impact if your home network is also on 192.168.5.0/24, I imagine that may cause some issues.

After finalizing the disks and IPs, the next page is the EULA.

The one thing to be aware of is you need to scroll down to the bottom of the EULA, either with the down arrow or Page Down. It will let you proceed to the next screen without doing so, however, that next screens tells you to go back and read the EULA.

I was honestly a bit surprised but this was all this was needed for the installer.

One other issue I had which is probably best mentioned here is I had the installer succeed here a couple times, but then started failing on some of the following configurations. The cause was I didn’t have the networking plugged in during the installation process. Even though I plugged it in shortly after, it seemed networking was required during the installation or something with the CVM broke. Sure, this seems obvious, but is more due to my homelab being a bit disjointed than not wanting to have it plugged in during the installation.

Configuration

After the reboot, we need to let the first boot processes complete. It is mentioned in the post this can take up to 20 minutes.

First, we log into the Hypervisor IP as root via ssh.

The default password is “nutanix/4u”.

Second, we log into the CVM from the Hypervisor as nutanix via SSH. The IP used here is not the one we previously configured, but the internal IP for the CVM. This is 192.168.5.2. The password is the same.

We run our command to create the cluster. This will be a single node so our command is:

cluster -s cvm_ip --redundancy_factor=1 create

There are other options if you are creating a multinode cluster. The documentation for this command and the following commands is here.

As the cluster starts creating, we see regular updates as various services are starting.

Finally, we see the cluster is up.

Next, we need to name the cluster.

ncli cluster edit-params new-name=cluster_name

Configure DNS.

ncli cluster add-to-name-servers servers=public_name_server_ip_address

And create the VIP. This would be more prominently used multinode cluster.

ncli cluster set-external-ip-address external-ip-address=cluster_ip_address

Once done, you can hit either the CVM IP or the VIP in a browser. Make sure it is HTTP though.

Password is again “nutanix/4u”. Then there is a prompt to update it.

Followed by a prompt to log in with the account you created on the Nutanix site.

Now we are all done and get into the dashboard.

For me, the first stop is to find Dark Mode.

Much better.

Hopefully, this has been helpful. Soon, I will have additional posts exploring the UI and creating VMs among other things.

Thanks!

Deploying a LinuX Container(LXC) seemed like the next logical step in this Proxmox journey. The process is fairly quick and straightforward.

Before we dig in it should first be noted that although Linux Containers and Docker Containers both have “container” in the name, they are very different. I like this Pure blog discussing the differences.

Create a Pool

The first step in deploying LXC is to create a Pool in Proxmox. Click Datacenter in the left-hand pane, then select Pools under permissions, then Create. This is not necessarily an intuitive location so hopefully this screenshot helps.

I decided to be very original and name it Pool 1.

Downloading Templates

Next, we need to grab a template for use with our LXC. Select the storage device, in my case local, and then CT Templates.

In this screenshot, I already have some from previous efforts, but this will be empty on a fresh Proxmox deployment. Fortunately, if you select the Templates button there are a ton of templates to easily add to Proxmox.

And this is only a small handful. Having used Ansible in the past, I was curious how the Ansible template would look.

After choosing to download the template a status screen appears to give updates.

Deploy Container

Finally, we can deploy our container. You can either right-click the host, then Create CT or select Create CT on the top-right of the window.

Choose a name for the container and password. Select the Resource Pool created earlier. The option to proceed won’t appear until all these fields are complete.

Choose the desired template, in this case the ansible one I just downloaded.

Determine disk size and location.

For this test, I stuck with defaults for CPU and Memory.

I did however want to use DHCP for the IP address of the container.

Lastly, enter DNS settings if you like, and finish

A new task window will appear to give status updates.

In this particular instance, the container failed to start after deployment, so I needed to start it manually which succeeded.

Once deployed, I try the IP in a browser

It has instructions to console into the container in Proxmox, and login as root, using the password I configured at deployment.

Next, I am instructed to create an admin password:

Then an optional API key, this optional though so I skip it.

It then asks a few more questions about configuring an email and installing security updates, but I skip those as well as they are optional and this is just a temporary container

Then finally, it is done and I can log in.

Turns out the Ansible template is for Semaphore, which is a UI for Ansible among other things. I may poke around with it later.

That is how to deploy a LXC on Proxmox. Have a great one.

After a couple of years of using my beefier NUC to run Ubuntu with VMware Workstation, I decided it was time for a switch. Naturally, this has nothing to do with the current state of VMware/Broadcom. Definitely not.

Looking for a switch and wanting to build my homelab out a bit more I thought Proxmox would be a great fit for me. Fortunately, it couldn’t have been easier.

After downloading and flashing the ISO to a USB, I fired it up. I also apologe in advance for the photo quality.

Booting to the USB, I see the install screen where I choose I want to GUI based installation.

As with most things, the next screen is a EULA:

Following the EULA, Proxmox asks where it should be installed.

This choice is at the bottom in tiny font. It can be easy to miss if you aren’t looking for it.

Basic geographic and time info is requested next:

Next, it wants an admin password and an email for notifications:

Networking is next, and it is where I had my only issue. In the screenshot, I chose to use the wifi adapter. Proxmox will install like this, but it won’t work. After much digging, it seemed the only way to get it to work on wifi was to make several additional networking changes after installation. I ended up going back and reinstalling at this point and chose the ethernet port for ease of use.

Lastly is the Summary page with all of the input information:

After the installation was completed and my NUC was rebooted, I hit IPAdress:8006 to login.

That’s it. I was honestly surprised with how simple and quick the installation was. I look forward to adding some VMs and containers to Proxmox and sharing my ongoing homelab updates here.

I recently completed my Linux Foundation Certified System Administration certification. Although I have used and managed Linux servers for several years as part of my job and homelab, it wasn’t my primary responsibility. The certification was a nice way for me to round off and solidify my Linux knowledge.

I thought it might be helpful to do some posts where I go back to basics with various Linux topics. Where better to start than what everyone needs to do at some point or another…..LOGS

This will not be an exhaustive guide for navigating and troubleshooting logs. It will be a starting point for those who are stumbling across this post and maybe have a tip or two for those with more experience.

Tail

Ahhhhh Tail! It is one of the first tools most learn when navigating text and log files. Being one of first means it is also very straightforward.

If we look at the man page the syntax for a tail command has this format:

tail [Option]  [File]

In practice it may look like this:

Without any parameters/flags tail will show the ten last or most recent lines in a file. The most common parameter used is probably -n, which changes how many of the most recent lines from a file are shown. In this instance, I am having tail show the most recent 20 lines from this log.

The second parameter for tail to know would be -f, where the f means follow. With this parameter tail will “follow” the file and as new lines are added to the file or log, tail will display them on the screen. This is really great for troubleshooting as you can see updates to the log in real time.

Grep

Grep allows us to search through log files for specific text. It is a tool with a LOT of depth to it as well, especially if you can pull out some decent regex skills.

I won’t be going too deep into it, but there are some basic examples worth sharing.

The syntax for grep is:

grep [OPTION...] PATTERNS [FILE...]

By default, grep is case sensitive, but if I use the -i flag, it will return entries regardless of case:

We see that both “error” and “Error” were returned.

If I want to search through all the syslog files so my command will look like this :

grep "error" /var/log/syslog*

This returns a lot of entries, but I can now see which file each entry corresponds to on the left.

And if I scroll down further, I see the other log file it searched:

If you use the -c flag it will tell you how many instances were found in each of the files.

This is very helpful if you are trying to determine which log file may have the info you need.

Journalctl

Journalctl is a tool which allows us to query logs from the systemd-journal service. In other words, it has all kinds of logs.

The most basic way is just to enter “journalctl” but that will return more than what is easy to parse and read. If we use -r it will return the newest entries first, with a pager, so we can consume the logs without our screen exploding.

And at the bottom, we see this:

This lets us know we can read more lines if we hit the spacebar.

Another option is –since which lets us define from how long ago we want logs.

Here I only did “2 minutes ago”, but hours, days…etc will work as well.

We can also use something similar to grep with the journalctl command by using -g. I will also use -n to limit the number of lines(like we did above).

Finally, use -u if you are looking for logs from a particular service.

This only scratches the surface, but hopefully it will help the next time you need to go diving into logs for troubleshooting.

A couple of months ago I was looking around for something new to try in the homelab. Since I had done Kubernetes last year when I was getting certified, I thought it might be a good idea to try out KubeVirt. Kubevirt sits on top of Kubenetes and allows one to deploy more traditional VMs. With my existing Kubernetes deployment and my history with VMware, this seemed like a good fit.

That is until my buddy Eric Shanks did a blog post about Harvester. After five minutes of researching Harvester, I knew this was what I would do next.

What is Harvester?

Harvester is a virtual machine and container platform built on Kubernetes. From the website, it is “a modern, open, interoperable, hyperconverged infrastructure (HCI) solution built on Kubernetes.”

Harvester uses several open-source products to build its platform.

• Linux OS. Elemental for SLE-Micro 5.3 is at the core of Harvester and is an immutable Linux distribution designed to remove as much OS maintenance as possible in a Kubernetes cluster.
• Built on top of Kubernetes. Kubernetes has become the predominant infrastructure language across all form factors, and Harvester is an HCI solution with Kubernetes under the hood.
• Virtualization management with KubeVirt. KubeVirt provides virtualization management using KVM on top of Kubernetes.
• Storage management with Longhorn. Longhorn provides distributed block storage and tiering.
• Observability with Grafana and Prometheus. Grafana and Prometheus provide robust monitoring and logging.

https://docs.harvesterhci.io/v1.3

This ticks a lot of boxes for me. First, I did a series of posts on Kubernetes a bit over a year ago. Second, I was already interested in Kubevirt. Lastly, although I didn’t make any posts about it, I did a Proof of Concept for a monitoring solution a few years ago which heavily used Grafana and Prometheus. All in all, Harvester seems like a great match for me.

I have some dedicated hardware where I want to deploy Harvester. However, I decided to first deploy it to VMware Workstation to get a feel for the process and poke at it until I get a more permanent home setup.

VM Deployment

We will start by creating a new VM in Workstation:

Select Custom as we will need to make some changes as we go which are different than just Typical

The compatibility will start with one for Workstation, but we need to ESXi. Whenever I did not choose ESXi I had network issues with the VM and could not finish the deployment.

Choose the Harvester ISO

For Operating System, select Linux and OpenSUSE 64bit

Name the VM and the location it will be stored

For processors, I found anything less than 4 and the required pods wouldn’t be able to spin up. The system requirements for Harvester are 8CPU minimum, but I was able to get it deployed with 4.

Memory is similar, the requirements list 32, but my deployment was able to stand up with 8.

Select Bridged Networking for the VM to utilize the host nic.

SCSI for disk

Create a new virtual disk

I tried to deploy this a few times with less than 250GB of space, but the installer wouldn’t let me proceed. 250 was the minimum limit for me.

After this finish up the wizard and boot up the VM to finish the deployment

Installing Harvester

This is a fresh Harvester install so we begin with choosing “Create a new Harvester cluster”.

I only have one disk attached to my VM so I choose the defaults here. Also, this screenshot is from a time I tried the install with a smaller disk size. Just pretend the 50G is 250G.

Decide on a hostname

Configure networking. If you deployed the VM with Workstation compatibility, you may have issues here. When I had select Workstation compatibility, my VM would not recognize the nic and so I could never proceed past this step.

I am choosing a static configuration.

Enter your DNS settings

Configure the Virtual IP

Choose a cluster token.

Create a password for the node.

NTP Settings

The next three screens you are able to leave blank which I did. They are Proxy settings, an option to import SSH keys, and optional Harvester config.

Assuming everything looks good your installation should kick off.

Once it is done your VM should look like this.

My installation took somewhere between 15 to 30 minutes before the status changed to “Ready”, so patience is important. If it takes too long though, it could be something is wrong and you may need to troubleshoot in the VM.

Hit the Management URL in a browser and choose your password for logging in.

Now at long last we are done with deployment and in Harvester! Enjoy!!!

I will be posting more soon as I mess around with my installation. Thanks!

For any familiar with PowerCLI, this is one of the first scripts someone may learn. It is just long enough to need more than one-liner, but short enough to feel like you don’t need to be an expert. Here is one I posted on this site over a decade ago:

param([string]$servers)
 
$spec= new-object vmware.vim.virtualmachineconfigspec
$spec.tools=new-object vmware.vim.toolsconfiginfo
$spec.tools.toolsupgradepolicy ="upgradeatpowercycle"
 
$vms= get-vm $servers
Foreach($vm in $vms){
$view=$vm|get-view
$view.reconfigvm($spec)
}

I share this so we have a baseline when looking at the Ansible version.

---
- name: Update VMTools Upgrade Policy
  hosts: vcenters
  gather_facts: no
  collections:
    - vmware.vmware_rest
  tasks:
    - name: Get all VMs
      vmware.vmware_rest.vcenter_vm_info:
        vcenter_hostname: "{{ inventory_hostname }}"
        vcenter_username: "{{ vcenter_username }}"
        vcenter_password: "{{ vcenter_passwords[inventory_hostname] }}"
        vcenter_validate_certs: no
      delegate_to: localhost
      register: vms

    - name: Change VMTools upgrade policy to UPGRADE_AT_POWER_CYCLE
      vmware.vmware_rest.vcenter_vm_tools:
        vcenter_hostname: "{{ inventory_hostname }}"
        vcenter_username: "{{ vcenter_username }}"
        vcenter_password: "{{ vcenter_passwords[inventory_hostname] }}"
        vcenter_validate_certs: no
        vm: "{{ item.vm }}"
        upgrade_policy: UPGRADE_AT_POWER_CYCLE
      register: _result
      delegate_to: localhost
      loop:
        "{{vms.value}}"

The two tasks here are straightforward even though they look and feel a more complicated than the PowerCLI version.

The first tasks gathers all the VMs in vCenter, and the second tasks changes the VMtools upgrade policy so VMtools upgrades during a reboot. That’s it.

Ansible’s benefit over PowerCLi is it can allow non-technical user to run pre-written playbooks very easily with little knowledge needed. Would I suggest everyone start transcribing all the Powershell/PowerCLI scripts to Ansible? Definitely not! But I do think exercises like this are important for learning Ansible or anything new as when we take a process or a script with which we are already familiar, it makes understanding the new tool or process that much easier.

Enjoy

Normally I frequently use Powershell when it comes to vSphere automation, but lately, I have been wanting to develop tools and scripts that are user-friendly and easily reusable. With those objectives in mind, it is hard to find a better tool than Ansible playbooks.

Ansible playbooks often use ssh for connecting to remote resources to run commands. With vSphere modules for Ansible. the experience is more similar to how powershell connects.

For these ansible playbooks I will be using a play which will grab some basic datacenter info from a vCenter. This module comes from the VMware Rest collection.

Local Playbook

The first example will be the simplest as all the info is contained within the playbook and we will not be using an inventory file. This is a “quick and dirty” approach as utilizing inventory files/var files is far more in line with the repeatable orchestration for which Ansible is known.

Here is my first playbook gatherinfolocal.yml:

---
- name: Gather Datacenter Information from VMware using REST API
  hosts: localhost
  gather_facts: no
  collections:
    - vmware.vmware_rest
  tasks:
    - name: Get Datacenter information
      vmware.vmware_rest.vcenter_datacenter_info:
        vcenter_hostname: "vnoobcenter.rosalita.lan"
        vcenter_username: "administrator@vsphere.local"
        vcenter_password: "MyPassword"
        vcenter_validate_certs: no
      register: datacenter_info

    - name: Display datacenter information
      debug:
        var: datacenter_info

All of the information for the vCenter, including the password, is in the playbook. I probably don’t need to mention how this is less than ideal and not secure.

Hosts is defined as localhost as we are not utilizing an inventory. The module is “vmware.vmware_rest.vcenter_datacenter_info” which utilizes vCenters REST API to pull some datacenter info back. That information is stored in the datacenter_info variable which is then return with debug.

Running this playbook is also straightforward and easy:

ansible-playbook  gatherinfolocal.yml

Which returns

Prompt for password

This second playbook we are going to abstract the info a bit by utilizing some variables in the playbook itself.

---
- name: Gather Datacenter Information from VMware using REST API
  hosts: localhost
  gather_facts: no
  collections:
    - vmware.vmware_rest

  vars:
    vcenter_hostname: 'vnoobcenter.rosalita.local'
    vcenter_username: 'administrator@vsphere.local'
  vars_prompt:
    - name: vcenter_password
      prompt: vCenter Password?

  tasks:
    - name: Get Datacenter information
      vmware.vmware_rest.vcenter_datacenter_info:
        vcenter_hostname: "{{ vcenter_hostname }}"
        vcenter_username: "{{ vcenter_username }}"
        vcenter_password: "{{ vcenter_password }}"
        vcenter_validate_certs: no
      register: datacenter_info

    - name: Display datacenter information
      debug:
        var: datacenter_info

This playbook is virtually the same as the one before except now there are variables in use. vCenter_Hostname and vCenter_username are effectively the same except the are just in a slightly different location in the playbook. These variables are then used during the task.

Next is vars_prompt. This will prompt the user to enter the variable when the playbook is run. In this instance it will be asking for the vCenter password. This is more secure than the first playbook as now password is stored in the playbook.

Running this playbook is exactly the same as the first

ansible-playbook  gatherinfo.yml

But now we are prompted for the password before the play executes:

Inventory and Ansible Vault

For the last example an inventory file and a variable file are created. The folder and file structure look something like

• vmware folder is arbitrary and where I am storing this work
• inv.yml and gatherinfo.yml are the inventory file and the playbook file respectively
• group_vars is the default location where variables for groups are stored
• vcenters.yml is the group we create in the inventory file

The inventory file:

vcenters:
  hosts:
    vnoobcenter.rosalita.lan:
      vcenter_username: administrator@vsphere.local

“vcenters” is the Ansible host group for our vCenter, this also correlates to the name of the group variable file vcenters.yml in the file structure. The host is vnoobcenter.rosalita.lan and a variable named vcenter_username is also created.

The playbook file:

---
- name: Gather Datacenter Information from VMware using REST API
  hosts: vcenters
  gather_facts: no
  collections:
    - vmware.vmware_rest
  tasks:
    - name: Get Datacenter information
      vmware.vmware_rest.vcenter_datacenter_info:
        vcenter_hostname: "{{ inventory_hostname }}"
        vcenter_username: "{{ vcenter_username }}"
        vcenter_password: "{{ vcenter_passwords[inventory_hostname] }}"
        vcenter_validate_certs: no
      delegate_to: localhost
      register: datacenter_info

    - name: Display datacenter information
      debug:
        var: datacenter_info

This looks much like the earlier versions, with one main change. For vcenter_password, we are referencing a nested variable. Think of vcenter_passwords as the group of variables and inventory_hostname as which specific variable. Since each host will have a different inventory_hostname, this will allow us to define different passwords for each vCenter.

The nested structure comes together when looking at the variable file vcenters.yml:

vcenter_passwords:
  vnoobcenter.rosalita.lan: MyPassword

This file is placed in the group_vars directory and when the playbook is run the variables are automatically loaded for the playbook to use.

Now we don’t want this variable to just be in plain text, so we need to use the ansible-vault command to encrypt it.

Once encrypted, the file looks like this:

When we run the playbook we specify the password we used to encrypt the variable file and Ansible will pull the passwords from the file for each of the hosts.

ansible-playbook  gatherinfoadv.yml -i inv.yml --ask-vault-pass

Ignore the dependency warnings

With this setup the Ansible playbooks and inventory can be expanded to account for as many vCenters as you like with any tasks.

I believe it was last year when I first heard about Project Keswick and how VMware wanted to streamline and change how ESXi hosts are deployed at the edge. It sounded cool and exciting, and I made a note to keep an eye out for it.

We fast forward to this year’s VMware Explore and Project Keswick is announced as a VMware Showcase. This allows users to try out the product and tinker all they like. With Project Keswick specifically geared towards edge usage, I knew it was a product I had to test in my homelab.

Let’s get started with a basic Keswick deployment.

Once in the Keswick portal you are greeted with this screen which pretty clearly lists out the steps one needs to take to setup Keswick.

However, there are a few prereqs we need to complete before we can walk through the steps in the portal.

GitHub Configuration

Following this process let’s start with the Git Repository. Once logged into github, I proceed to my profile page, the repositories tab, and then select New.

Choose a name for the repository, set it to private, and it doesn’t hurt to initialize it with a readme file.

Next, we need to create a yaml file for our host. This can be done with a text editor and using git commands. For this simple file howerer, I am just going to use the github UI.

Inside the repository, Add file-Create new file. Copy the contents from here

apiVersion: esx.vmware.com/v1alpha1
kind: HostConfiguration
metadata:
  name: keswick-host-config
  namespace: esx-system
spec:
  layertype: Incremental
  profile: |
    {
      "esx": {
        "system": {
          "system_time": {
            "enabled": true,
            "protocol": "NTP",
            "ntp_config": {
              "server": [{
                  "hostname": "0.pool.ntp.org",
                  "command": "SERVER"
                },
                {
                  "hostname": "1.pool.ntp.org",
                  "command": "SERVER"
                }
              ]
            }
          }
        }
      }
    }

This file was found on the Project Keswick documentation site.

Make sure the filename ends in yaml, and commit the changes.

Commit the changes to the main branch

Because this repository is private, we need to provide a way for Project Keswick to access this repository. Go to your picture in the top right of github, and select Settings.

At the very bottom, we want to select Developer Settings.

Creating and providing a Personal Access token will allow Keswick and the host to pull the newly created yaml file. Under Personal Access Token-Fine-grained token- Generate New Token.

The token name doesn’t matter, but we do want to make sure this token only has access to the private repository that was just created. By choosing “Only select repositories” and selecting the private repository, we can limit the token’s access.

Furthermore, under permissions we want this token to have Read-only access for the Contents permission.

Once created, github provides the token. You will never see this token again, so make sure to save it somewhere as you need it for the Project Keswick portion of this configuration.

Installing ESXi

I am installing my Project Keswick ESXi host on VMware Workstation which is located on an Ubuntu server I have here at home.

After downloading the “ova” file from the Keswick site, I first needed to extract the ova file as it came as a compressed file. After the extraction, open the file through Workstation.

Name the VM and select the location of the vmdk.

There are some firstboot and configuration files in the extracted files you can pre-configure for the ESXi host. I wasn’t doing that this time, so I left those fields blank.

On the other hand, I definitely wanted to set the root password and have SSH enabled from the start.

Funny thing, despite the multiple times I tried this deployment, my root password was never applied. The root password was always configured as empty or null. Basically, I would enter root as the user, then just hit enter again on the password field without typing anything.

You will need to note the Vendor/Model/Serial on your device in order to configure Keswick

Project Keswick

Start by getting logged into the Keswick site. From here it will present you with steps to get Keswick setup.

Let’s start at the beginning with Git Repository

Enter the basic details about the git repository. The access token we created will be used later.

That’s it for Git. Now we need to add a host. We use the the Vendor/Model/Serial earlier for the details.

Any Keswick host will attempt to check in with the Keswick service. Entering these details informs VMware/Keswick this host is yours.

After a minute or two you will see the host check in.

Choosing “Activate Host” will require the github access token we created earlier.

Once activated the host will be able to check the git repository and configure itself per the specification yaml.

If you are using the standard config I listed earlier, you probably won’t see much change to your host. To kick the tires a little more, we will add a basic nginx website.

MetalLB/Nginx

Fortunately for all of us, Alan Renouf has built and shared exactly what we need to quickly set up a simple Nginx page. Check out his repo here.

We need to copy his hello-keswick.yaml file to our own git hub repo. Then we also need to copy and edit the metallb.yaml file to our repo. The metallb.yaml is very long, but we only need to change some IP addresses at the end.

Adjust this IP range to conform to your own subnet, and you should be good to go.

After the change and placing the file in your own repo you should soon have your own test nginx page running on Keswick.

This has been fun getting to know Keswick a bit and its capabilities and limitations. I plan to put out a few more posts in the near future.

Enjoy!!

• Tanzu Homelab: Overview
• Tanzu Homelab: Storage Policy and Content Library
• Tanzu Homelab: Network Planning
• Tanzu Homelab: Deploy pfSense
• Tanzu Homelab: Deploy HAProxy
• Tanzu Homelab: Deploy Tanzu
• Tanzu Homelab: Final Configurations

In the last post, we got Tanzu configured, however, before we can get any services deployed there are a few more items we need set up.

The first is getting a Namespace setup, followed by permissions and storage. Fortunately, all of these items are pretty quick to configure.

From the Workload Management page, go to Namespaces and click Create Namespace.

Choose a name for the namespace and select the network.

Done! That was quick. Now on the namespace page, we need to choose Add Permissions.

I used the devops@vsphere.local account I setup earlier. Because this is a homelab, I went ahead and gave it owner permissions.

Now we need to Add Storage.

Select the policy you previously created.

The last thing we want to do is add a VM Class and Content Library. These are in the VM Service block/widget.

When we add a VM Class we are basically telling Tanzu what size of VMs it is able to deploy. For my lab, I want it to deploy the smallest possible, so best-effort-xsmall for me.

After selecting VM Class let’s choose a content library.

That’s it. Although we could have done this before now, let’s make sure we can access the supervisors.

From the Supervisor page you will see a Control Plane node address. This is actually one of the HAProxy VIPs fronting for the control nodes. Anyway, lets hit that IP in a browser.

Looks good. Download the CLI plug for your OS of choice.

Next, we need to extract the content of the zip into our PATH. To check your path on Windows, simply type path in a command prompt. You should see something like this.

PATH=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\dotnet\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\

You can extract the contents to any of those folders, or potentially add a new folder to your PATH. I just extracted it to an existing folder.

Once that is done you should be able to connect to you supervisor cluster.

Great! Now we are connected to Tanzu so we should be able to easily deploy some resources/workloads

Unfortunately, after getting vSphere with Tanzu deployed, it seems we still cannot deploy pods or any resources. The reason for this is the vSphere with Tanzu deployment is deploying a Tanzu/K8s management cluster. Being a management cluster, it won’t let us deploy workloads. Yes, a bit annoying.

We have two options, the first option is we now deploy and Tanzu Kubernetes Cluster to our vSphere with Tanzu. This would create a workload Tanzu cluster where we would be able to deploy pods/services.

The second option is to use William Lam’s workaround in order to allow to deploy to the existing Tanzu cluster we just deployed.

In the spirit of trying to do things “by the book”, I tried to get TKC deployed. For some reason I have not yet figured out, it would never deploy for me. In the midst of this over several weeks I had some storms and power outages. After each outage, I could not get Tanzu back into a healthy place and ended up needing to start over.

At the end of the day and because of my frustration, I landed on using William Lam’s method to get resources deployed.

After stepping through William Lam’s post, I was able to get a small nginx deployment stood up.

Woohoo! If you have stuck with me through all these posts, WOW! Thank You! Hopefully, you were also able to get everything working. Now you should be able to deploy services to Tanzu! Have Fun!