Fdo's Workspace

Terraform Troubleshooting: Server Failed to authenticate the request

2021-07-19T13:35:00.001-04:00

This is another one of those odd issues with my Ubuntu Linux Subsystem. Everything was working perfectly fine before I shutdown the laptop for the weekend and then it’s Monday. I would be lying if I say I did not touch laptop over the weekend. The truth is I did not use the Ubuntu Subsystem over the weekend and this morning I keep getting strange Terraform problems when running “terraform apply”.

Keeps complaining about:

Error: reading share properties for AzureRM Storage Account “diagstore###”: storage.FileServicesClient#GetServiceProperties: Failure responding to request: Status Code: 403/400 – Original Error: autorest/azure: Service returned an error. Status=400 Code=”AuthenticationFailed” Message=”Server failed to authenticate request. Make sure the value of Authentication header is formedd correctly including the signature.”

Bellow is a screenshot of the error:

Insufficient Priviledges?

Like usual did some browsing around these error messages and most of them pointed to insufficient access. To rule that out made myself added myself as contributor to the resource group this storage account was living and tried again.

Destroy and Rebuild?

Since this was my own dev environment it did not bother me to destroy and start over, but keep running into the same error when it reaches to the point where the storage account is been built

Time Drift? Seriously!!

I can’t find the post that mentioned this but apparently my Ubuntu subsystem had a huge time drift. I was too focused on fixing the problem and did not grab a screenshot the time drift but the fix was simply issue a #> wsl –shutdown from a PowerShell command prompt and start the subsystem again. After that everything was happy again.

Troubleshooting Terraform: Not the usual “Failed to query available provider packages” in Ubuntu Windows 10 WSL2

2021-07-15T12:07:00.000-04:00

I have a Windows 10 laptop with Ubuntu18 WSL2 installed and configured with bunch of tools including Terraform and Ansible. Because Ansible is does not run on Windows and can only manage Windows hosts, we are forced to use Windows System for Linux (Official Doc). Everything was working perfectly fine and all last week I’ve been working on Terraform in Windows because I did not have any Ansible work to do. Earlier this week I started the Ubuntu system and realized that when I executed “terraform init” I would get the following error:

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/azurerm: could not query provider registry
│ for registry.terraform.io/hashicorp/azurerm: the request failed after 2 attempts, please try again later: Get
│ "https://registry.terraform.io/v1/providers/hashicorp/azurerm/versions": net/http: request canceled while waiting for
│ connection (ssssssssssssssssssssssssssssssssssssssssssssssssss)

This clearly points to a network related problem but:

I was able to ping 8.8.8.8 perfectly fine
nslookup 8.8.8.8 > returns dns.google.
even tried curl https://registry.terraform.io/v1/providers/hashicorp/azurerm/versions > returns all the versions

At this point, I was almost convinced it was not a network problem but there was one more thing I did not try. I connected to the VPN and everything started working again.

So what was the problem????

Here is the sequence of events:

For a week I’ve been doing Terraform work from Windows because there was no need Ansible related work.
I have not been using VPN because been coding against my dedicated Sandbox.
This week I had to do some Ansible/Terraform work against my Sandbox and ran into the Terraform problem inside the subsystem.
After struggling for several days trying to overcome this issue, connected to the VPN and everything started working

This problem normally does not exist on a fresh WSL setup this was 100% caused by myself.

If you are familiar with the WSL2 network problem when connected to a VPN you will know what this is all about. To overcome the VPN problem inside the subsystem there was couple tweaks I had to make including modifying the DNS inside ubuntu. By default WSL automatically generates the /etc/resolv.conf files every time the subsystem is loaded. I needed a static /etc/resolv.conf because I needs it to resolve using the VPN assigned DNS servers when connected.

Default /etc/resolv.conf looks like:

# This file was automatically generated by WSL. To stop automatic generation of this file, add the following entry to /etc/wsl.conf:     
# [network]
# generateResolvConf = false

After adding those lines into the /etc/wsl.conf updated I was able to customize my /etc/resolv.conf to look like.

nameserver #.#.#.10

nameserver #.#.#.11

nameserver 8.8.8.8

The first and second row are the DNS assigned by the VPN, you can find this by running ipconfig /all from your Windows CMD prompt. Last line is just google dns which explains why resolution was working. The only hint was the delay during pings and nslookups which I did notice but did not pay close attention.

After few more test and re-reading the error message it totally makes sense now. “Client.Timeout exceeded while awaiting headers” Apparently, when “terraform init” is executed it tries to reach out to the provider urls and resolve against DNS1 > DNS2 before it hits DNS3 which is 8.8.8.8 it times out. I disabled DNS1 and DNS2 leaving 8.8.8.8 it was back to normal.

How was this fixed?

Solution to my problem was straight forward, I moved nameserver 8.8.8.8 to the top. If I have to work while connected to the VPN it will hit the second nameserver and timeout should cover the second try.

Hope this help someone. This was not the usual “Error: Failed to query available provider packages” error…

Learning Terraform From Zero

2021-07-08T12:11:00.000-04:00

This last year been really hectic and like everyone we have to do what we need to stay afloat. I recently started a new position with a new company and of course this is the beginning of a new adventure. If there is anyone following my blog you probably notice a shift in topics covered.

This article is only to provide some guidance and sharing my personal experience learning Terraform. This is by no mean a tutorial.

How hard is Terraform?

This depends completely on your background and your willingness to learn new technologies. I am by no means a software developer but I do write my fair share scripts, code and automation tools to perform various tasks. It could be mainly but not limited to PowerShell, Python, Ansible against Windows/Linux vms or VMware infra or Azure Cloud. In the past, I did tinker with Terraform and that was prior to choosing Ansible for a project I was working one. So I spent a good full weekend cranking on PluralSight training and YouTube videos to grab a solid foundation of the basics of Terraform. Learning Terraform was not terribly complicated, just had to be open minded.

My goal was to familiarize myself with all aspects of managing resources in Azure Cloud with Terraform.

Where to start?

Read, Learn and Practice.

You can find a lot of good reading material online. Best place to start is the main HashiCorp Site. Go thru the fundamentals and get an idea of what we dealing with:

https://learn.hashicorp.com/terraform

If you have access to online training course or willing to spend some money there are even more options. Fortunately I already had access to PluralSight and Here are couple good ones I used:

- Terraform – Getting Started by Ned Bellavance

- Terraform – Deep Dive by Ned Bellavance

- Implementing Terraform on Microsoft Azure – by Ned Bellavance

Those 3 courses help me setup a good development environment and a foundation.

There are of course plenty other good training sites like UDemy or even YouTube. Learning from YouTube help better if you are trying to expand on a specific topic. Because there is so much content in YouTube sometimes is hard to keep a good flow.

Where to practice?

Having a lab and place to get some hands on experience is key to learning. No amount of reading or watching video will give you the real experience. Terraform can be installed on OS X, Linux or Windows and you can manage resources in AWS, Azure, Google Cloud, Docker and etc. Since my goal is to learn how to manage Azure resources there is an option to create a free Azure account with $200 credit – link. I was fortunate enough to get a Visual Studio Subscription and monthly credit for my development projects.

Understanding How Terraform Works

Terraform – Infrastructure as Code

At first I kept trying to relate Terraform with Ansible which are completely different tools. In some cases you can perform similar task but there are pros and cons of doing it one way or the other.

In Terraform, you write one .tf file that contain your infrastructure as desired and you can build or destroy those resources by simply running a command. If you need to make some changes simple adjust the content of the .tf and apply again it will automatically update the resources. Depending on the type of changes made it might require deleting and rebuilding the resource.

In Ansible, you could also achieve the same by writing a playbook to build the resources and another one to destroy. It will require 2 different playbooks. Now the problem start when you need to make changes to the resources. You will have to write playbook to apply the changes, if changes require object deletion you will have to write that into the playbook.

Note: Terraform focuses on infrastructure automation, which is why it does a much better job a managing the infrastructure. You only have to make sure the tf file contains the resources with the desired configuration and Terraform will take care of building/updating those resources.

Where Terraform Don’t Shine

Terraform is a great tool and can certainly get the job done but there are times were you need to accept that there are easier ways.

Say the environment is now deployed, all the resources are deployed in Azure: resource groups, virtual network, load balancers, network security groups, and 100s of Linux and Windows servers. We need to configure all those Linux and Windows servers, they range from application servers, web servers, databases, so forth and so on.

With Terraform, you could configure servers by using writing long Bash script or PowerShell scripts and using the remote-exec provisioner to kick start the script remotely. Main challenge is Terraform will only be responsible of executing the task and will not be responsible of making sure the script is executed properly or not. It will be entirely reliant upon how well that Bash or PowerShell script is written. If your next Terraform job is depending on the successful completion of the configuration script, this will already be a big challenge.

With Ansible instead, you write your playbook or role you want to assign for each type of server and it will make sure the server is properly configured. You will have to create playbooks to configure web servers, application servers, db servers etc for Linux or Windows.

Why not use them together?

Using the right tool for the right job will make you life so much easier. Let Terraform handle all infrastructure deployment and Ansible do the configuration.

I’ve tested this and works harmonically. This is what the flow would look like.

Terraform deploys all the Azure resources
Terraform builds the Ansible inventory.yml file with the host and credentials or sshkeys
Terraform executes the Ansible Playbook against the inventory.yml file

This a quick workflow of one of the demo Terraform jobs I’ve been working on but hopefully I will start posting of those stuff very soon.

Visual Studio Code: Working against multiple User Accounts and Azure Subscriptions

2021-07-08T10:00:00.001-04:00

I started a new job and switched from a MacBook to a Dell Laptop. So I am forced to change my mindset and adjust, process been a bit saddening but I will survive. I use VSCode develop Azure PowerShell, Terraform and Ansible code to perform various task. I have multiple user accounts and even more Azure Subscriptions each for its own purpose. I often switch around depending on the tasks I am performing. The other day something really odd happen and thought I should document this and share.

Problem?

I was working on a Terraform job to deploy some resources in Azure but it was executing against the wrong subscription and user account. Obviously tried disconnecting all session using Disconnect-AzConnect and even Logout-AzAccount, then use Get-AzContext to make sure it was not returning anything. Used Connect-AzAccount again and Terraform was still deploying resources to the wrong place. I tried restarting computer, clearing browser cache, even searched how to clear Visual Studio Cache and no luck.

Cause?

What I did not realize was that there was an active Azure CLI (az login) session connected to the Azure Subscription the resources was been deployed to. Ran the “az account show” and it indeed show the subscription Terraform been deploying the to. Used “az logout” clear that. Terraform requires Azure CLI session not the Azure PowerShell session. Duh. This is what happens when you learn something new then shift your focus to work on a complete different project and come back from the holidays…

Lesson learned:

Terraform does not care whether you use Azure CLI or Azure PowerShell to establish to establish the Azure session. TF will pickup the active session and try to deploy resources.

Working against Multiple Azure Subscriptions in Visual Studio Code

Visual Studio Code is easily became “The Favorite IDE” because of its rich features/capabilities and language supports. But sometimes those rich features can work against you. My Environment have PowerShell Core 7.0 installed as well as the VSCode PowerShell Module that gives you the “PowerShell Integrated Console” terminal.

VSCode allows you to run multiple PowerShell Terminal Session but only one “PowerShell Integrated Console”. See Picture.

If you have an active Azure connection in any of the three “pwsh” terminals it will persist across the rest of the three “pwsh” shown in picture. So you can definitely start a different session from the PowerShell Integrated Console, if you really need to work with two different active session. I would not suggest doing this because it could do harm if executing wrong code against wrong subscription.

Note: Have WSL installed you could start another session.

Connecting to Azure via PowerShell

If using any of the AZ PowerShell modules you would need to connect using the Connect-AzAccount command. Easiest way is to type “Connect-AzAccount” and it will launch the browser asking to login with your account. At this point you can login with the desired account or select from any cached session. You can certainly use additional parameter to specify the tenant and subscription, for that please visit the official MS Doc

To disconnect simple run “Disconnect-AzAccount”.

Connecting using Login-AzAccount

Alternatively AZ module also have a Login-AzAccount which also allows you to connect to Azure from PowerShell It will also launch the the web browser but might you to copy and paste the Device Code. Once connection is establish you can run the same PS Commands.

To disconnect run “Logout-AzAccount”

Switching Azure Subscriptions with Set-AzContext

To select the Azure Subscription you want to work with:

Set-AzContext -Subscription "xxxx-xxxx-xxxx-xxxx"

The “Subscription” paramater can be either the Subscription ID or Subscription Name. Use Get-AzContext to verify session

Switching Azure Subscriptions with Select-AzContext

Another option is to use Select-AzContext which requires the the context name.

Example:

Select-AzContext ‘Test Subscripton (xxxxx-xxxx-xxx-xxx-xxx) – xxxxx-xxx-xxxx-xxxxx – test.user@outlook.com’

Note: If you have an active session, type Select-AzContext and hit TAB to toggle list the context available.

Switching Azure Subscription with Select-AzSubscription

Select-AzSubscription is probably becoming obsolete soon but it was still an option in the Az Module

Example:

Select-AzSubscription –Subscription “Name or ID”

Azure CLI

From Azure CLI simple use “az login” command to establish connection it will use web browser and access token:

az login

There are are plenty parameters you can use to sign in with different tenants or service principals. If interested refer to official MSDOC

To show active sessions:

az account show

To Switch Subscription:

az account set - -subscription “subscription name or ID”

To logout:

az logout

This is it for now. I am sure there are plenty other tips and tricks out there but this should cover the basics of connecting to Azure and switching from one Subscription to another.

ESXi Trying to access non-existent NFS .sunrpc

2020-03-31T15:27:00.000-04:00

Network team reached out to me stating that one of the ESXi host been flooding the switches with invalid error trying to hit a RPC or NFS server. From their end all they can see is:

esxi_hostip:922 -> destination_ip_address:111

So tracked down the host in question and it was a ESXi v6.7 host, accessed the ESXi host in via SSH and used tcpdump to see if that was true:

"tcpdump-uw | grep destination_ip_address"

19:01:17.575158 IP vm03cl1vc.840 > 10.159.18.20.sunrpc: Flags [S], seq 3024135077, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 3512365326 ecr 0], length 0

19:01:18.575482 IP 10.159.18.20.sunrpc > vm03cl1vc.839: Flags [R.], seq 0, ack 4202901431, win 0, length 0

As you can see from the screenshot it was trying to connect to "destination_i:sunrpc" and is not receiving a valid response. Further digging into .sunrpc port found it was an NFS mount. However the NFS no longer exist in our environment and there was no trace of that NFS server on the host.

Searching for the NFS Mount:

Here we are looking for anything that will trace back to the destination ip address in question.

Using ESXCLI: you can list all nfs storage available and in my case there was no trace of any NFS Datastore with that ip address.

Command: esxcli storage nfs list

Using ESXCFG-NAS: Similarly you can also use the esxcfg-nas command to list any network attached storage

Command: esxcfg-nas -l

What Other Options:

Now that we confirmed that there is really no traces of the NFS, we can dig a bit deeper into the ESXi system configuration. Warning: Tampering with esx.conf might severely impair your system. Make sure you are careful making changes.

In my case searched for any reference to the ip address in question inside the /etc/vmware/esx.conf file.

Ran command to display content of esx.conf file and filter out the ip address in question:

Now, I was able track down references to the non-existent NFS server and apparently it was due to a previous Rubrik setup in our environment. Now all I need to do is remove those entries. But before backup your esx.conf before making any changes. Once you done restart your esxi management agent.

Restart the ESXi host daemon and vCenter Agent services using these commands:

/etc/init.d/hostd restart

/etc/init.d/vpxa restart

And that was the solution for my problem.

VCSA v6.5 File Based Backup to FTP

2019-10-04T16:43:00.001-04:00

I wrote on my previous post how easy it was to backup VCSA v6.7 to a FTP server the entire process was straight forward and simple. Link to Post Building Windows 2016 FTP Server and Configure VCSA v6.7 File-Based Backup. For VCSA v6.5 all we had was some API to perform the task, which is still better than not having a backup.

So as promised I will share how we can backup VCSA v6.5.# using the same concept applied on vCenter 6.7. This is handy if you cannot upgrade your vCenter to v6.7 for whatever motive and need to keep vCenter protected. It will require some PowerShell but will get the job done.

Brief Overview

For vCenter v6.5 the file based backup concept was introduced, API released but not incorporated into the UI. But we can still backup the appliance using the Backup-VCSA module and some extra scripts.

How will this work??

The Backup-VCSA Module contains a function Backup-VCSAToFile that will take couple parameters:

- LocationType = 'FTPS', 'HTTP', 'SCP', 'HTTPS', 'FTP'

- Location = "10.1.1.200/vcsa_backup_folder/"

- LocationUser = ''ftp_service_account'

- LocationPassword = "ftp_service_account_pwd"

Sample Syntax:

Backup-VCSAToFile -BackupPassword $BackupPassword -LocationType $LocationType -Location $location -LocationUser $LocationUser -LocationPassword $locationPassword -Comment "This is a demo" -ShowProgress -FullBackup

For full detail refer to the link to source code: VMware Github

Next we will be using Windows Task Manager to schedule the job and optionally we can create another script to clean up files older than x amount of days.

Note: This document will only cover how backup to a ftp server.

Requirement:

- FTP Server / FTP server Account (if you do not have one visit my previous post to build one)

- vCenter Server Appliance v6.5.# ready for backup

- PowerShell Knowledge

- Understanding of Windows Task Scheduler works

- Backup-VCSA.psm1 module found at VMware Github wrote by Brian Graf

Building The Backup Script

For this exercise, I will be the Windows 2016 FTP server to store the scripts and run the backup jobs. The script was tested on Windows 2016 - PowerCLI 11.#

Familiarizing yourself with the backup script

Testing the backup script. This script is straight out of the documentation found in the Backup-VCSA module.

    
[VMware.VimAutomation.Cis.Core.Types.V1.Secret]$BackupPassword = “Passw0rd!”
 
$Comment = “First API Backup”
$LocationType = “FTP”
$location = “10.159.81.171/VC24_21/vcsabackup-$((Get-Date).ToString(‘yyyy-MM-dd-hh-mm’))”
$LocationUser = “ftp_svc1”
[VMware.VimAutomation.Cis.Core.Types.V1.Secret]$locationPassword = “Passw0rd!”


    
$secpasswd = ConvertTo-SecureString "VMware1!" -AsPlainText -Force
$creds = New-Object System.Management.Automation.PSCredential ("administrator@vsphere.local", $secpasswd)

$vcsa = "10.159.81.206"
$vi = Connect-CisServer $vcsa -Credential $creds


Backup-VCSAToFile -BackupPassword $BackupPassword  -LocationType $LocationType -Location $location -LocationUser $LocationUser -LocationPassword $locationPassword -Comment “This is a demo” -ShowProgress -FullBackup

If the script was executed properly you should see the job running:

And the files should show in the FTP directory specified.

PowerShell Script to Kick Start Backup Job

Here is a small script I wrote called Run-Backup.ps1 which takes as parameter a configuration file. The configuration file is nothing more than a json file with vCenter information, vCenter Creds, and the FTP parameters required to run the job.

This is what the Run-Backup.ps1 loos like.

<# 
    Run-Backup.ps1
    Script to Backup-VCSA 
#>
param (
        [String]$Config,
        [Switch]$Verbose
)

Get-Module -ListAvailable VMware.PowerCLI | Import-Module

$BackupModule = "$CurrDir\Backup-VCSA.psm1"

if (Test-Path $BackupModule){
    Write-Verbose "Backup Module Found"
    Import-Module $BackupModule
}
else{
    Write-Verbose "Backup Module not found"
    Write-Verbose "Exit"
    return 
}

if ($Config){
    Write-Verbose "Loading Config File"
    if (Test-Path $Config){
        $json = Get-Content -Raw $Config | ConvertFrom-Json
    }else{
        Write-Verbose "Failed to Load Config"
    }
}else{
    Write-Verbose "Missing Config File"
    Write-Verbose "Syntax:: RunBackup -Config [config.json]"
    return
}

Write-Verbose "Loading Variables"
$VCSA = $json.VC_IP
$VCName = $json.VC_Name
$VCUser = $json.VC_User
$VCPwd = $json.VC_Password | ConvertTo-SecureString 


$BackupPassword = $VCName
$LocationType = $json.LocationType
$location = "$($json.Location)/$($json.VC_Name)/vcsabackup-$((Get-Date).ToString(‘yyyy-MM-dd-hh-mm’))"
$LocationUser = $json.LocationUser
[VMware.VimAutomation.Cis.Core.Types.V1.Secret]$locationPassword = ([Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($($json.LocationPassword | ConvertTo-SecureString))))

$comment = "Backup $VCName to $location"

$creds = New-Object System.Management.Automation.PSCredential ($VCUser, $VCPwd)

$ci = Connect-CisServer -Server $VCSA -Credential $creds -Force

Backup-VCSAToFile -LocationType $LocationType -Location $location -LocationUser $LocationUser -LocationPassword $locationPassword -Comment $comment -ShowProgress -FullBackup

Disconnect-CisServer $ci -Force -Confirm:$false

As you might have noticed:
- The vCenter and FTP account password inside the JSON is a SecureString which is why needs to be decrypted before running.

- I am also adding a password to the backup job to protect the files, you will need this password in order to restore vCenter from these backups. To keep things simple I am using the vCenter name as the backup job password.

This is what the vcenter_config.json looks like:

{
    "VC_IP":  "10.10.10.207",
    "VC_Name":  "vc6_makisan",
    "VC_User":  "administrator@vsphere.local",
    "VC_Password":  "01000000d08c9ddf0115d1118c7a00c04fc297eb010000002760ab6290f6c749a1d36eb49f730646000000000200000000001066000000010000200000008ab43c2902ee38f81f8477bdf8513754389f1ebc002a3acb8c21e25af58e2d980da6986c70b34000000051bc2e08c3e380c7fd930a9a3917a38b3665f923d0cb763e311ab13df58d88fb17d9da553bd44c1d4391c6b5a1b6d8d87407469958d7acb725b6cdeff481435e",
    "LocationType":  "FTP",
    "Location":  "10.10.10.171",
    "LocationUser":  "ftp_svc1",
    "LocationPassword":  "01000000d08c9ddf0115d1118c7a00c04fc297eb010000002760ab6290f6c749a1d36eb49f73064600000000020000000000106600000001000020000000a34e546a76bcc678602303d20a9390cead042371a55acdf68216ad84faad953be2535c9955cda40000000a797810e57b51bad622f50c5bafab281dba3fc62a885c45df1616f889a1bce2b3aa31cdd241ca83c4937792620cc401f0a48eb20b88485d01bdb4653b14158ed"
}

To generate your password simply use ConvertTo-SecureString function:

"Your Password"| ConvertTo-SecureString -AsPlainText -Force | ConvertFrom-SecureString

to make things easier to copy you can try export to file

"Your Password" | ConvertTo-SecureString -AsPlainText -Force | ConvertFrom-SecureString > pwfile.txt

Testing new Run-Backup Job

If your JSON file was configured properly script should simply executing:

.\Run_Backup.ps1 -Config .\config_file.json

Now that the script works there is one more script we need before creating the Task Job

Creating calljob.bat for Task Scheduler

I found that Task Scheduler was not running the PS1 script properly for some reason so decided to just create a batch file to trigger the PowerShell script. If you manage to get around feel free to share how you were able to accomplish. In the mean time here is the batch script I used.

echo off
echo ****************************************************
echo Running batch file to call Run_Backup.ps1 script.
echo Syntax = calljob. [config.json]
echo ****************************************************



if exist %~dp0calljob.bat ( ^
goto :CAllJob) ^
else (goto :NotFound)

:NotFound
echo File not found
goto :EOF

:CallJob
echo Found Script
echo Call script 
REM powershell.exe -command "& '.\VMCLoneJob.ps1' %1"
powershell.exe -command "& '.\Run_Backup.ps1' %1 -Verbose"

Creating Windows Task to Initiate Backup

Launch a Task Scheduler : Administrative Tools | Task Scheduler

1 - Create a folder to keep all backup task together: This is optional

2 - Create a Task | Enter name of your task

- - Select - Run whether user is logged on or no - otherwise script willl not run

- - Check - Run with highest priviledge

3 - Task Trigger - This till trigger task to run on a schedule defined

- - I am running this daily

- - You can modify this to run more than once a day. For example you can run this job every 4 hours by checking "Repeat task every: 4hr for duration of 1day"

- - Check Enabled

4 - Action - Is the script that will be triggered

- - Action will be Start a program

- - Program/script: path to the bat file in my case E:\script\calljob.bat

- - Add Arguments: e:\scripts\vcenter_config.json

- - Start in: e:\scripts

5 - Conditions:
- - I think I left this default

6 - Settings:
- - Check - Allow task to be run on demand. This will allow you to test run the job when necessary.
- - Optionial: I set to stop task if runs longer than 2hrs. Might have to change this depending on the size of your vCenter and network latency between VC and FTP server.

7. Finally save the change and enter your administrator password. If you are having problem with the password go back to the General page and select the correct user or group that will own this job.

8. Finally kick start the job by selecting the job and RUN... If everything was configured properly task should run...

And that's it.. This is a fully automated backup solution for vCenter Appliance v6.5.#. This should keep your vCenter backed up until it is upgraded to v6.7..

Bonus Script - Deleting old Backups

Here is a script to delete jobs older than 30 days to reclaim some space. I am sure there are much better code but this is what I got:

Assuming E:\ftp\vcsa65 is the location directory all your backup jobs is pointing to.

Each vCenter_Backup job will have its own folder with the scheduled backup inside. Folder structure should look like:

E:\ftp\vcsa65\[vcenter_name]\[backup_job_date]

E:\ftp\vcsa65\vcenter_1\vcsabackup-date1
E:\ftp\vcsa65\vcenter_1\vcsabackup-date2
E:\ftp\vcsa65\vcenter_1\vcsabackup-date3
E:\ftp\vcsa65\vcenter_2\vcsabackup-date1
E:\ftp\vcsa65\vcenter_2\vcsabackup-date2
E:\ftp\vcsa65\vcenter_2\vcsabackup-date3
. . .

Script should delete [vcsabackup_data] jobs older than 30days by checking creation date. Make sure to test this properly and modify as see fit.

# Root Directory were all my vCenters are backed up
# structure E:\ftp\vcsa65\[vcenter_name]\backupjob_dates
$backup_path = "E:\ftp\vcsa65"

$vcenters = Get-ChildItem $backup_path

foreach ($vc in $vcenters){

    $backup_list = Get-ChildItem $vc.FullName
    #$backup_list | ? {$_.CreationTime -le (Get-Date).AddDays(-30) }
    foreach ($bk in  $backup_list){
        if ( $bk.CreationTime -le (Get-Date).AddDays(-30)){
            Write-Host "$($bk.FullName) - Over 30 Days Delete"
            Remote-Item $bk -Force -Confirm:$false
        }else{
            Write-Host "$($bk.FullName) - Not Over 30 Days, Skip"
        }

        
    }
}

Final Words:

Please make sure you test everything in a DEV environment before actually utilizing this in production. Everything found online must be fully tested.

Building Windows 2016 FTP Server and configure VCSA v6.7 File-Based Backup

2019-10-01T18:23:00.001-04:00

As the title says today we will be building a very simple Windows 2016 FTP server to backup vCenters Server Appliance. We will be using the File-Based backup feature that was built into vCenter 6.7.#. I was excited about this news but didn't have the chance to actually try it until now.

Back on vCenter v6.5 when file based backup was introduced, the only way to initiate the backups was via API. Now is part of the VAMI UI in v6.7.x

BTW, depending on the version the following backup target are supported

vCenter v6.5.x - FTP, FTPS, HTTP, HTTPS, or SCP
vCenter v6.7,x - FTP, FTPS, HTTP, HTTPS, NFS, SCP or SMB

Requirement:
- Basic Windows 2016 skills
- Windows 2016 Server Deployed and configured - VCSA should be able to reach this server.
- VCSA v6.7.x waiting for backups.

Installing FTP Server Role

From the Windows Server:
- Launch Server Manager
- Click - Add Role and Features
- Select Web Server (IIS)
- Role Features
- Select FTP Server
- Check FTP Services and FTP Extensibility

- Click Next to review confirmation and Finish to Install

Create the FTP Service Account

From Server Manager | Tools | Computer Management

- Local User and Groups

- Right Click on Users and create your user

- For example: ftp_svc

Creating the FTP Web Site

Go to Server Manager:
- Tools
- Internet Information Services (IIS) Manager
- Drill down to Sites and Right Click on Sites
- Select Add FTP Site

- Name your site and point to directory to be used as ftp_root directory (this is where files will be uploaded)

- Next to Binding and SSL Settings
- select no SSL and click next
- Next to Authentication and Authorization
- Authentication will be Basic
- Allow access to specified user and enter the service account created
- Check read and write permission

Test the FTP Site

Launch Internet Explorer and access ftp://localhost or ip address
- If everything was configured properly it should prompt for credentials
- Enter the service account user name and password

- You should be able to log into the root of the ftp site

Configure VCSA Backup

Log into your VCSA Appliance Management UI https://vcsa_ip:5480 and click Backup

- Click Configure

- Create Backup Schedule

- Backup location

- Backup server credentials

- Schedule: Daily, Monthly or Custom

- Encrypt backup password

- Number of backup to retain (this is cool to conserve and clean up old backups)

- Data check to retain Stats.

Once this is done you can click Backup Now to test and make sure everything is working

- Click Backup Now

- Enter the ftp_service account password and 'START'

- if everything is configured properly you should see files been copied to the FTP site

This is the easiest and cheapest solution to backup your vCenter Appliance.

Fix for Python not working in VirtualEnv - Reason: image not found.

2019-07-28T11:47:00.001-04:00

I've become a big fan of "virtualenvwrapper" because it makes managing multiple virtual environment super easy and it takes you to the root of the project folder you which to work on.

Few commands:
- mkproject "project_name": will create virtualenv and project folder
- workon : will list all the virtualenv
- workon project_name: will activate the virtuelenv and cd to the project folder
There are still lot of tips and tricks in the main website https://virtualenvwrapper.readthedocs.io/en/latest/

Lately been dealing with an odd issue where I can load the virtualenv but python is not working. When I try to launch python I get error:

$ python --version
dyld: Library not loaded: @executable_path/../.Python
Referenced from: /Users/maki/.virtualenvs/rest_foreman/bin/python
Reason: image not found
Abort trap: 6

Originally I would delete the virtualenv, create a new, but it will not change directory. Apparently deleting the virtualenv does not remove the project folder and there seems to be a bug where it doesn't register existing project folders. Only way to get around this and keeping the same project name as well as fixing change directory option was to rename the bad project folder. Do mkproject and move the file content to the new location. However it happened keep happening for some reason and I had to find a better solution.

What was the problem??

I browse to the .virtualenv/project/bin/ folder and notice the symlinks were messed up.

0 lrwxr-xr-x 1 maki staff 9 Jun 21 13:31 python -> python3.7 8 -rwxr-xr-x 1 maki staff 2362 Jun 21 13:31 python-config 0 lrwxr-xr-x 1 maki staff 9 Jun 21 13:31 python3 -> python3.732 -rwxr-xr-x 1 maki staff 12552 Jun 21 13:31 python3.7 8 -rwxr-xr-x 1 maki staff 253 Jun 21 13:31 wheel

Tried calling any of the 3 python# from the folder will give me same error message. Notice is a symlink to python3.7. Seems like a ENV variable missing. I am not sure why it stopped working but I know it was after rebooting my MAC and I don't reboot the MAC often.

Fixing the broken symlink:

From inside the virtualenv find your bin folder:

- $ workon project
- $ which python
output:
/Users/maki/.virtualenvs/rest_foreman/bin/python

Using another terminal session:

- $ cd to the bin folder
- $ which python3.7 (I only use python3 find location of your python3.7)

output:
/usr/local/bin/python3.7

# Run following command in the bin folder to replace broken links
- $ ln -s -f /usr/local/bin/python3.7 python
- $ ln -s -f /usr/local/bin/python3.7 python3.7
- $ ln -s -f /usr/local/bin/python3.7 python3

# Should look like this:
0 lrwxr-xr-x 1 maki staff 24 Jul 28 11:42 python -> /usr/local/bin/python3.7
8 -rwxr-xr-x 1 maki staff 2362 May 10 16:28 python-config
0 lrwxr-xr-x 1 maki staff 9 May 10 16:28 python3 -> python3.7
0 lrwxr-xr-x 1 maki staff 24 Jul 28 11:00 python3.7 -> /usr/local/bin/python3.7

Now it should work again. you can test by running python --version

Not sure if there is a better fix but if you do have one please share.. Thanks..

vCenter AD authentication taking over 60second???

2019-07-19T11:32:00.000-04:00

A very odd issue got escalated to me this week. Some of our in house tools will take forever to run and will time out. These tools uses AD service account to authenticate and issue is also encountered when login into vSphere Web Client. Browser will sit in login screen for a minute. BTW this particular environment was running v6.5.

I started by investigating the in-house application one of which I developed. Application will consistently timeout. Reported this issue and basic checks were performed on the vCenter. vCenter services, cpu/mem resources, disk space and all the usual stuff but everything seem fine.

In order to better understand the situation, created a very simply Python script to measure amount time time taken to connect to this vCenter. On every vCenter it will take ~0.15sec to establish connection. But this affected one was taking up to ~60sec consistently.

Once the account is authenticated everything worked fine, but waiting 60sec was unacceptable. Opened case with VMware, went thru whole process and the engineer finally pointed out /var/log/vmware/sso/vmware-sts-idmd.log error message complaining about cannot bind connection with ldap.

[ServerUtils] cannot bind connection: [ldap://ipaddres_of_primary_ad:389 , domain\userid]

[ServerUtils] Error received by LDAP client: com.vmware.identiry.interop.ldap.OpenLdapClientLibrary, error code: -1

Next step was to identify issue: Started with basic checking to make sure port were open between vCenter and AD server. "curl -v ad_server_ip:389" and of course it was failing. When it comes to production our role and access is very compartmentalized and reaching out to the appropriate team is required. Networking team verified ports were open and AD team verified no firewall rules were applied.

I tried several things to verify each side's statement. Since AD server and vCenter were in different network I wanted to make sure ports were not blocked. Wrote very simple Python script to listen to port 389, on the same network AD server was residing and tested port connection VC. It worked!! this rule out "network issue". Spent so much time looking at this problem that left something out.. We have a primary and a secondary AD server. Went back to the VC and tested port connecting against the secondary AD server and it worked!!! Well which perfectly explain the ~60sec delay before switching to the good AD.. Which also explain why the logs are flooded with error related to primary AD and no messages from the secondary.

Reported back to AD team and they had to schedule the reboot. Since we can't wait any longer easiest solution was to swap the identity sources used in vCenter. To do so:

-- Administration | Single Sign - On | Configuration | Identity Sources
-- Select your Domain | Edit | Swap your primary and secondary AD source
-- Save changes..

Tested and this fixed the problem for me...

Bonus Scripts used to troubleshoot this issue:

Test Python script to measure duration:

import atexit

from pyVim.connect import SmartConnectNoSSL, Disconnect

import requests

import json

from requests.auth import HTTPBasicAuth

import ssl

import time

start_time = time.time()

vcenter='vcenter_ip_address'

user='username@domain.com'

password='secret_password'

try:

print("Trying to connecto to ", vcenter)

service_instance = SmartConnectNoSSL(host=vcenter, user=user, pwd=password)

# doing this means you don't need to remember to disconnect your script/objects

atexit.register(Disconnect, service_instance)

except requests.exceptions.SSLError as e:

context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)

context.verify_mode = ssl.CERT_NONE

trying

service_instance = SmartConnectNoSSL(

host=vcenter, user=user, pwd=password, sslContext=context)

# doing this means you don't need to remember to disconnect your script/objects

atexit.register(Disconnect, service_instance)

except vim.fault.InvalidLogin:

print("Invalid Login - username or password.")

print('Script ended - trying to connect to VC -', vcenter)

print(service_instance)

print("--- %s seconds ---" % (time.time() - start_time))

Output

Trying to connecto to 10.10.10.10 Script ended - trying to connect to VC - 10.10.10.10 'vim.ServiceInstance:ServiceInstance' --- 0.15743589401245117 seconds ---

Port Listener Script - Credit to GeekFlare for creating this.

'''    Simple socket server using threads

'''

import socket

import sys

HOST = ''   # Symbolic name, meaning all available interfaces

PORT = 389 # Arbitrary non-privileged port

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

print( 'Socket created')

#Bind socket to local host and port

try:

s.bind((HOST, PORT))

except socket.error as msg:

print('Bind failed. Error Code : ' + str(msg[0]) + ' Message ' + msg[1] )

sys.exit()

print( 'Socket bind complete')

#Start listening on socket

s.listen(10)

print( 'Socket now listening')

#now keep talking with the client

while 1:

#wait to accept a connection - blocking call

conn, addr = s.accept()

print ('Connected with ' + addr[0] + ':' + str(addr[1]))

s.close()

Learning Flask and Jinja2 - Manipulating Data

2019-06-28T16:25:00.001-04:00

Lately I've been learning Flask to write tools that queries vCenter and this article very brief article that will show how to data generated in form of List or JSON in a Jinja2 template table.

Jinja2 - Dividing a List into columns using Batch

When you have a long list of data, instead of displaying a long list, you can plus the data into columns. Easiest way to do this is using the batch filter.

Following example will take a list of letter and divide into a table with 3 columns evenly distributed

import json
from jinja2 import Template
list_str = ['a','b','c','d','e','f','g','h','i','j','k','l','m']
list_str
tmpl = Template('''
<table>
{%- for row in data|batch(3, '&nbsp;') %}
  <tr>
  {%- for column in row %}
    <td>{{ column }}</td>
  {%- endfor %}
  </tr>
{%- endfor %}
</table>
''')
print(tmpl.render(data = list_str))
 

## Output

<table>
  <tr>
    <td>a</td>
    <td>b</td>
    <td>c</td>
  </tr>
  <tr>
    <td>d</td>
    <td>e</td>
    <td>f</td>
  </tr>
  <tr>
    <td>g</td>
    <td>h</td>
    <td>i</td>
  </tr>
  <tr>
    <td>j</td>
    <td>k</td>
    <td>l</td>
  </tr>
  <tr>
    <td>m</td>
    <td>&nbsp;</td>
    <td>&nbsp;</td>
  </tr>
</table>

Jinja2 - JSON to Table - Static

Data in JSON is the most common format but depending on how is displayed on your page it can be hard to read. Even when displaying into pretty JSON format often times is not good enough. Here is a quick example on how to turn a JSON output into 3 column table. This following example will export the data. Each row contains 3 fields: name, moid and guestfullname the Jinja loop will take loop thru each item into columns and rows.

import json
from jinja2 import Template

json_str = '''[{"name": "name", "moId": "moId", "guestFullName": "guestFullName"}, 
    {"name": "dummyvm023", "moId": "vm-2409", "guestFullName": "SUSE Linux Enterprise 11 (64-bit)"}, 
    {"name": "dummyvm034", "moId": "vm-9152", "guestFullName": "Red Hat Enterprise Linux 6 (64-bit)"}]'''


vm_list = json.loads(json_str)

tmpl1 = Template('''
<table>
   {% for vm in vm_list %}
   <tr>

       <td>{{ vm['name'] }}</td>

       <td>{{ vm['moId'] }}</td>

       <td>{{ vm['guestFullName'] }}</td>

   </tr>
   {% endfor %}
</table>
''')

# Sample Output
<table>
  <tr><td>name</td><td>moId</td><td>guestFullName</td></tr>
  <tr><td>dummyvm023</td><td>vm-2409</td><td>SUSE Linux Enterprise 11 (64-bit)</td></tr>
  <tr><td>dummyvm034</td><td>vm-9152</td><td>Red Hat Enterprise Linux 6 (64-bit)</td></tr>
</table>

Jinja2 - JSON to Table - Dynamic

Previous method is okay when the JSON data only have few fields/columns, but when dealing with large JSON files with lots of columns it becomes hard to code each field into the Jinja template. In the next example we will use keys and values to iterate thru the data and generate the table dynamically.

import json
from jinja2 import Template

json_str = '''[{"name": "name", "moId": "moId", "guestFullName": "guestFullName"}, 
    {"name": "dummyvm023", "moId": "vm-2409", "guestFullName": "SUSE Linux Enterprise 11 (64-bit)"}, 
    {"name": "dummyvm034", "moId": "vm-9152", "guestFullName": "Red Hat Enterprise Linux 6 (64-bit)"}]'''

tmpl2 = Template('''
<table>
   {% for vm in vmList %}
        <tr>
        {% for key, value in vm.items() %}
            <td>{{value}}</td>
        {% endfor %}
        </tr>
    {% endfor %}
</table>
''')
print(tmpl1.render(vmList = vmList))

## Sample Output

<table>
        <tr>
            <td>name</td>
            <td>moId</td>
            <td>guestFullName</td>
        </tr>
        <tr>
            <td>dummyvm023</td>
            <td>vm-2409</td>
            <td>SUSE Linux Enterprise 11 (64-bit)</td>
        </tr>
        <tr>
            <td>dummyvm034</td>
            <td>vm-9152</td>
            <td>Red Hat Enterprise Linux 6 (64-bit)</td>   
        </tr>
</table>

Here is a quick example of what it would look like on a real scenario. Here is a little tool I wrote where user can select the from a list of options they want in their report and sorted into 4 columns.

And here is the report generated:

Thats it...

Getting Started with Foreman 101 - Vagrant CentOS7

2019-06-14T11:42:00.001-04:00

One of the items I’ve been wanting to add to my toolbox was a bare metal solution. Goal here was to come up with solution to make the life of fellow colleagues easier. Previously, I built a PXE tool that will deploy ESXi VM and thought about extending this to deploy physical servers. Yeah I know we have AutoDeploy but it has its limitations. AutoDeploy doesn’t work for greenfields environments. It might work for capacity add’s but what if I manage many vCenters in different networks? We don't want to enable AutoDeploy everywhere you can imaging what would happen.

We also use Nested Datacenter for development work and I was able to accomplish this using vRO, PowerShell and my TFTP/PXE solution. Thought about extending my tool to do bare metal but didn’t take long to conclude this was a bad idea so started researching for tools online. So spent next few weeks researching and narrowed down to potential candidates. The time I spent evaluating each solution, I started to compile a list of nice to have:

easy to deploy and configure
rich in Rest API for integration purposes
plugins with third party tools is a big plus
UI would be nice
capable of provisioning Linux, Windows, ESXi
free, OpenSource is big plus
can deploy to vCenter plus
etc...

So far the big runner up was Foreman.

What is Foreman? “Foreman is a complete lifecycle management tool for physical and virtual servers. We give system administrators the power to easily automate repetitive tasks, quickly deploy applications, and proactively manage servers, on-premise or in the cloud.” https://www.theforeman.org/

Initial Thought - After evaluating different solutions one thing I found extremely annoying was lack of documentation how simple installer do not work as advertised. I was actually impressed with the completeness of Foreman documentation and initial install. The Quickstart process was no joke, I literally got started in less than 15 mins on my first try, compared to this other XXXX solution where I was stuck for 4 days troubleshooting their automated install. My colleague figured out after getting help from the community but I was already turned off by the product. When I found Foreman was capable of provisioning to vCenter that was awesome. Didn't take me long to get addicted.

My POC is MacBook with Vagrant and Virtual Box. Choice of OS platform is CentOS 7, but Foreman supports a wide variety of operating system. So far tested Centos and Ubuntu and they were both painless.

If you don’t are not using Vagrant that is not an issue can just deploy your basic CentOS 7 VM and run the install.

Let's Dive right into the setup process.

Deploying Vagrant/CentOS 7 with Foreman 1.22.0

1.22.0 - latest version at day of this post

Here is the config I added to my vagrantfile. Following code will deploy CentosVM and install Foreman with default configuration. Feel free to adjust the script, variables to meet your needs. Script_foreman is simply the yum update and the 5 commands to install Foreman. The rest is vagrant configuration. Note that hostname is a FQDN, which is required for Foreman install. Then update the IP address to match your network setting.

Vagrant Configuration File:

    $script_foreman = <<-SCRIPT
    sudo yum update -y
    sudo yum -y install https://yum.puppet.com/puppet6-release-el-7.noarch.rpm
    sudo yum -y install http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
    sudo yum -y install https://yum.theforeman.org/releases/1.22/el7/x86_64/foreman-release.rpm
    sudo yum -y install foreman-installer
    sudo foreman-installer
    SCRIPT
    config.vm.define "foreman" do |fore|
        fore.vm.box = "centos/7"
        fore.vm.hostname = "foreman.maki.com"
        fore.vm.network  "private_network", ip: "192.168.57.70"
        fore.vm.provider "virtualbox" do |vb|
            vb.memory= 4096
            vb.cpus = 4
        end
        fore.vm.provision "shell", inline: $script_foreman
    end

Once ready "vagrant up foreman" and within minutes your server should be up and running.

Installing Foreman on any CentOS 7 Server

This steps were straight from the Foreman website:

From the clean Centos7 Server runn the following commands:

sudo yum -y install https://yum.puppetlabs.com/puppet5/puppet5-release-el-7.noarch.rpm

Enable the EPEL (Extra Packages for Enterprise Linux) and the Foreman repos:

sudo yum -y install http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum -y install https://yum.theforeman.org/releases/1.20/el7/x86_64/foreman-release.rpm

Downloading the installer

sudo yum -y install foreman-installer

Running the installer

The installation run is non-interactive, but the configuration can be customized by supplying any of the options listed in foreman-installer --help, or by running foreman-installer -i for interactive mode. More examples are given in the Installation Options section. Adding -v will disable the progress bar and display all changes. To run the installer, execute:

sudo foreman-installer

Reference Links: https://www.theforeman.org/manuals/1.20/quickstart_guide.html#QuickstartGuide

Access Foreman from web browser. You can either add the FQDN to your hosts file or simply use the IP address.

And that is for Foreman Installation.

I am still learning the product and there is a long list of things I want to test so will be updating as I go. Most of my experiment will be against vCenter, once I get my hands on some physical gear will also test that.

Additional Info:

Documentation: https://www.theforeman.org/documentation.html Lot of good information. The community is extremely helpful and good place if you get stuck. Most of the answers can be found in Manual or Community old posts.

Deploying ESXi with Foreman: https://theforeman.org/2018/08/deploying-esxi-through-foreman.html. This was actually really cool project. Was one of the very first thing I did with Foreman.

vRealize Automation 7.5 - Dealing with Issues - Part 2

2019-03-29T11:26:00.000-04:00

This actually a continuation of my previous post: vRealize Automation 7.5 - Dealing with Issues Deploying vRA, IaaS and SQL I was finally able to complete a couple trainings on vRA 7.x. Now I have my instance of vRA 7.5 fully configured and deployed with some sample catalog items to Centos and Windows 2012R2 VMs. The Pluralsight training "Getting Started with vRealize Automation 7" is definitely a good start if you never had any exposure to vRA or want a refresher. Presenter does a good job covering most of the key points. However during the training I ran into some technical issues. Troubleshooting those issues definitely contributed in helping me understand the product better. Now I am looking for some more advanced tutorials or training materials to dive deeper into vRA.

Back to topic of the post which is about couple more issues I encountered through out my vRA learning process:

AD User Account not showing

After your domain is added as a "Directory" under Administration | Directory Management | Directories and synced, total number of domain users are shown under Synced Users. These domain user accounts should be consumable by vRA. It allow you to add any domain\user into Custom Groups or Business Groups, permissions, etc.

In my screenshot I have the default local domain and the second domain with 241 users synched.

During this exercise, I was trying to a new AD user to a Business Group and the account was not found because of the two following reasons:

New Account was created an AD and vRA is not aware of it

If you created a new user account is created in your AD, vRA is not aware of it unless is synced. Directory synch can be performed manually by click on "Sync Now" or wait until the next scheduled synch if configured.

Account is not new but still not showing in vRA

The other common scenario is when a vRA cannot find a user account, you are sure that the user exist in AD and is not a new account. This is usually because during vRA Directory configuration the user CNs configured does not include the user you are looking for.

Simple example is during initial configuration you decided to configure vRA to sync with CN=users,DC=example,DC=com but the user you looking for is part of CD=admins,DC=example,DC=com. Solution is just to specify/add another DNs.

Under: Administration | Directories | click on directory name you wish to reconfigure

- Click Sync Settings

- Click Users

- Here you add other CNs, once done simply Save and Sync.

Notice in my example: I originally had CN=users,DC=domain,DC=com. Then I added the DC=domain,DC=com, this will scan your entire AD for users. This is an extremely lazy way to do things and is not really recommended in production. I have a CN with a blank space in the name and couldn't figure out how add it to the list. Will update once I figure out.

VM templates not showing when designing blueprints

Another common mistake is when creating VM clone Blueprints the template you wish to use is not shown in the Template list. If you are stuck in this scenario make sure you check the following few items:

1 - Make sure the vm you are using as a template was converted to VM Template. In my environment, vm template is just a regular virtual machine to clone from and is not usually converted to a VM Template. Since vRA requires the images to be a template, right click on your target vm, from the menu select Template | Clone or Convert to Template

2 - Make sure the Template is in the right cluster / endpoint. If your vCenter has multiple clusters and during configuration you selected specific cluster for vRA consumption make sure your template is places there. To double check go to Fabric Group:

Infrastructure | Endpoints | Fabric Groups

In my example I had configured 2 different vCenters as endpoints, one of my vCenters have multiple clusters. I had to move the template to the selected cluster.

3 - Finally Sync the resources, vRA will sync vCenter object on its own schedule. If you need it immediately must manually recollect data. To do so go to: Infrastructure | Compute Resources | Compute Resources | right click on the cluster your template resides and click "Data Collection"

Once the Data Collection window launches click "Request Now" under Inventory to initiate manual sync. Once completed you can double check again to see if template shows in your blueprint.

This is all I got for now.

I am hoping dive deeper into vRA as time permits and start designing some more advanced Blueprints. If any one have have pointers to advance vRA tutorials feel to share in the comments section.

vRealize Automation 7.5 - Dealing with Issues Deploying vRA, IaaS and SQL

2019-03-22T17:18:00.002-04:00

vRealize Automation 7.5 is like a hot topic now. There is all sort of exciting news about it, all kinda of cool things it can do so couldn't help but check it out. Last year after VMworld 2018, I did attempt deploying the vRA 7.x but didn't get to far with the project. I have no experience with vRA but I am comfortable with vRealize Orchestrator (vRO).

Where to start? Well if you are vExpert, you should have a 1 year of free PluralSight Access. There is a good "Getting Started with vRealize Automation 7" which is bit old but concept is the same. I am using the training as my guide. Otherwise, there are plenty online tutorials on how to install vRA 7.x there are also YouTube videos.

I have been stuck in the first Chapter 1- Module: "A Simple Installation" which is 15min long for 3 days now. BTW, I have real work to do and vRA is only in my spare time. The goal of this post is not to show step by step how to do click thru the installation wizard but to deal with some the issues I came across.

vRA Appliance Root Account Lockout

I am a fast typer and not always accurate so 3 times before locking out is not good enough. Since this appliance is for learning purposes, disabling root account lockout is a must. Once you are locked out, only options are to wait for it to auto unlock, deploy a new one or log into single user mode and reset. To prevent this issue once and for all disable the lockout policy.

Boot into Single User Mode

Reboot your vRA appliance and console to it. Once you see the boot screen select the second option and enter 'e' to edit.

Scroll to the end of the line and append: rw init=/bin/bash

Once done ESC to go back to menu and 'b' to boot

Unlocking Root Account

To check if account is locked:

# pam_tally2 --user root

To reset

#pam_tally2 --user root --reset

Disabling Root Account Lockout

While you are here resetting the root account might as well disable the lockout policy. (This is only for your Dev/Testing purposes)

Use vi to edit /etc/pam.d/common-auth

# vi /etc/pam.d/common-auth

Enter 'i' to enter edit mode, once you comment out the line with "pam_tally.2.so deny=3..."
Hit ESC type ":wq!" to save

Now you reboot the appliance.

Reset the vRA Installation Wizard

If for whatever reason you made a mistake, browser closed or something else when wrong and when you go back to your https://vra_appliance_ip:5480 the installation wizard is not there. Fortunately deploying new OVA is not the only option. There is a way to bring that wizard back. I learned about this while dealing with the screen below.

Bring Back vRA Installation Wizard - Method 1

Once you are logged into your vRA appliance via SSH or console edit the "/etc/vcac/vami.ini"

Change the "start_wizard" from false to true

Restart vami-lighttp service

# service-lighttp restart command

Now you should see the wizard if you log back into https://vra_appliance_ip:5480

Bring Back vRA Installation Wizard - Method 2

Use the vcac-vami command to reset:

# /usr/sbin/vcac-vami installation-wizard activate
yes --> Confirm

Log back into https://vra_appliance_ip:5480 and try again.

Addressing Single Sign-On Page Issue

Unfortunately there is no easy way for this one. I was stuck here for sometime. The install process does have an option to not build the IAAS (Infrastructure as a Service) server but I think skipping that is also a factor to this problem. So I do recommend building the IAAS server. After that try again. Make sure you go back to the wizard and check the box to install IAAS server.

Error Message: PreparedStatementCallback; SQL [INSTER INTO installation_wizard_params (node, page, role, instance, param, value, sequence) values (?, ?, ?, ?, ?, ?, ?)]; ERROR: insert or update on table "installation_wizard_params" violates foreign key constraint "installation wizard params pk" Detail: Kye (node)=(cafe.node.128400245.15991) is not present in table "cluster_nodes".; nested exception is org.postgresql.util.PSQLException: ERROR: insert or update on table "installation_wizard_params" violates foreign key constraints "installation_wizard_params_pk" Detail: Key (node)=(cafe.node.128400245.15991) is not present in table "cluster_nodes".

Things to Try:

Deploy the IAAS server, I deployed a Windows 2012R2

Joined server to the Domain
Made sure DNS is fully working forward and reverse resolution from and to vRA
Time must be correct even thought the wizard will check for you
Install the Management Agent installer from the Wizard, if done properly your IAAS server will be discovered by the wizard.

Make sure the password is good, something like "VMware123!" which works for me.
If you did all that and the problem is still there you are out of luck, that happened to me.
Delete the vRA appliance and deploy fresh new one
Re-Install the Management Agent installer on the IAAS

Uninstall the agent - still pointing to old vRA
Revert Snapshot if you have one
Re-Install the Agent and point to new vRA

Almost positive this should fix the problem.

Microsoft SQL Server ISSUE

I wasn't sure if I was getting tired of vRA or the training video, this time I was stuck at the SQL Server screen. I failed the validation miserably, according to the error message it needs a SQL Server. I was under the impression it would at least load the SQL Express version for me. Sight... Video guy could have at least said something. Just kidding... Took a break from this, got fresh cup of coffee to refresh my mind and body. Several hours later...

Did browsing and found the vRA 7.5 Installation Guide - Requirement are in page 28. Following section is copied some info from the guide.

IaaS SQL Server Host A Windows server that hosts the IaaS SQL database must meet certain requirements.

Your SQL Server can reside on one of your IaaS Windows servers, or on a separate host.
When hosted together with IaaS components, these requirements are in addition to those for all IaaS Windows servers.
This release of vRealize Automation does not support the default SQL Server 2016 130 compatibility mode. If you separately create an empty SQL Server 2016 database for use with IaaS, use 100 or 120 compatibility mode. If you create the database through the vRealize Automation installer, compatibility is already configured.
When hosted together with IaaS components, configure Java.

Install 64-bit Java 1.8 update 181 or later. Do not use 32-bit. The JRE is enough.
You do not need the full JDK. n Set the JAVA_HOME environment variable to the Java installation folder. n Verify that %JAVA_HOME%\bin\java.exe is available.

Enable TCP/IP protocol for SQL Server.
SQL Server includes a model database that is the template for all databases created on the SQL instance.
For IaaS to install correctly, do not change the model database size.
Usually, the server needs more hardware than the minimums described in IaaS Windows Servers.

Installing/Configuring your SQL Server

I am installing SQL Server Express on my IAAS server, but you can certainly build a separate server.

Download SQL Server Express 2017 which is the latest version available for download on day of this article LINK . Whatever version you pick make sure is compatible with vRA.
Download SQL Server Management Studio Express LINK
Download install JAVA 64-bit JRE. I am a big fan of Chocolatey. Loaded my JRE and tools from via command line.

Install SQL Server Express 2017, SQL Management Studio and JAVA

Click on the SQL Express 2017 binary to start installation. Make sure you choose Custom, I did Basic and left out bunch of required components.

I left everything default up till the Feature Selection Page:

Database Engine Services

Sql Server Replication
Machine Learning Services

Shared Features

Client Tools Connectivity

Left everything else default.

Install SQL Server Management Studio and JAVA

Additional Configuration on your SQL Server

There are few other thing you must enable to fully pass validation

Launch SQL Server Configuration Manager

Set SQL Server Services | SQL Server Browser to start/automatic
Enable all Named Pipes and TCP/IP

Launch SQL Server Management Studio

Login using Windows Authentication
It will show your server as Localhost\SQLExpress
Browse Security | Logins
Create new login and find the domain user account you using to login and make this account sysadmin. It will be needed for DB creation process. Wizard will ask for this user account. I used the same domain account for vRA installation.

TRY AGAIN!

Go back to the vRA Installation Wizzard | Microsoft SQL Server screen and make sure you enter your hostname\[SQL INSTANCE] or it will fail again and again.

If everything went find it should pass the Microsoft SQL Server screen. During one my attempts I failed the final Validation process because the domain account I was using to did not have sufficient permission. So adding it as a "sysadmin" should fix the problem. Otherwise expand to check the actual error message.

On my last attempt passed validation successfully and started the installation.

While I was posting this Installation completed successfully.

Now I can finally move on to the chapter 2 of my vRealize Automation 7.x Training

PowerShell: Generating PScribo Report for DiskIOPs

2019-03-20T20:19:00.002-04:00

I've been VMUG member for sometime and never thought I would actually stand in front of a crowd and speak. My team had the opportunity to speak at DC VMUG in Reston,VA - "Mitigating Noisy Neighbor disk i/o situations" on March 14, 2019. It was my very first time ever speaking in front of a crowd and being nervous was an understatement.

So during the presentation I shared briefly a script to generate a Disk IOPs Report per VM per VMDK.

Following script will get all vms inside a "vmug-test" folder and generate a report with the storage IO allocation limit.

$vmlist = get-vm -location vmug-test
$rpt = @()

foreach ($vm in $vmlist){
    $disklist = $vm | Get-HardDisk
    foreach ($disk in $disklist){
        $tmp = $vm | select Name, HardDisk, SizeGB, IOPsLimit
        $tmp.Name = $vm.Name
        $tmp.HardDisk = $disk.Name
        $tmp.SizeGB = $disk.CapacityGB
        $tmp.IOPsLimit = $disk.ExtensionData.StorageIOAllocation.Limit
        $rpt += $tmp
    }
}
$rpt | Sort-Object Name
#To Generate CSV
$rpt | Sort-Object Name, HardDisk, SizeGB, IOPsLimit | Export-Csv -NoTypeInformation ".\IOPsRPT.csv"

######  Section Uses PS Scribo to Turn table in to HTML, TXT, DOC
$document = Document 'Report' -Verbose {
    #TOC -Name 'Table of Contents';
    #PageBreak;
    
    <# WARNING: 
        Microsoft Word will include paragraphs styled with 'Heading*' style names to the TOC. 
        To avoid this, define an identical style with a name not beginning with 'Heading'! 
    #>
    
    Paragraph -Style Heading1 'Disk IOPS Report'
    Section -Style Heading2 "Sample Report Sorted by VM name" {
            #Paragraph -Style Heading3 'Example of an autofit table width, autofit contents and no cell highlighting.'
            #Paragraph "Services ($($services.Count) Services found):"
            $rpt | Table -Columns Name,HardDisk,SizeGB,IOPsLimit -Headers Name,HardDisk,SizeGB,IOPsLimitcd -Width 0
    }
}
$document | Export-Document -Path .\ -Format Word,Html,Text -Verbose

Here is the sample output:

Sample script can be downloaded from GitHub

Installing Syntax Highlighter on Blogger

2019-03-20T15:23:00.001-04:00

Today I am scratching one more task off my list: ~~Adding Syntax Highlighter to my blog~~. Like everyone else I got nothing but excuses. Well today I spent few hours adding this feature into my blog. As usual ran into few issues/fixes which I shared as well.

First, what is Syntax Highlighter and why do we need this?

In simple word, is basically a tool that reformat blocks of code/scripts into a nicely colored pretty looking code block within your blog. I did some digging and decided to go with one written by Alex Gorbatchev, latest version as of today is 3.0.83. http://alexgorbatchev.com/SyntaxHighlighter

I opened this account Blogger.com so long ago and didn't bother switching to another service yet. Of course it has it’s pros and cons but that is a different topic.

Let’s get started.

Log into your Blogger.com account:

Click on Theme:
Before you do ANYTHING!! Please backup your theme. I found it a lot easier to download the team and make changes offline than directly on the browser. So many thing can go wrong.
Click on the Backup/Restore button

From the Theme > Backup / Restore window | Download the theme:

Side Note: I am sure, I am not the only one getting this annoying warning: “Custom themes that mix HTTP and HTTPS may affect the security and user experience of your blog when it is viewed over HTTPS. Learn more.”. Which is another reason why downloading the theme and modifying it offline then uploading it is the best way to get around this. Once file is saved Close this window.

Attempt to fix the mixd http issue (optional skip to next section)

Once to the back to the main Theme, click Edit HTML button. Notice you see that same warning “Editing your theme so it mixes HTTP and HTTPS may affect the security and user experience of your blog when it is viewed over HTTPS. Learn more. Hide warning” . If you are lucky you should be able to edit and paste the piece of code to enable Syntax Highlighter right from this window. In my case, I tried to fix all the http URLs by searching the entire code for http links and searching for a https version. The only ones I couldn’t able to replace were those on the top.

Here is the xmlns section causing those mixed http warning if anyone figure this one out please let me know.

<html b:dynamicViews='true' b:version='2' class='v2' expr:dir='data:blog.languageDirection' expr:lang='data:blog.locale' xmlns='http://www.w3.org/1999/xhtml' xmlns:b='http://www.google.com/2005/gml/b' xmlns:data='http://www.google.com/2005/gml/data' xmlns:expr='http://www.google.com/2005/gml/expr'>

Note: The only way to get around changes not saving is to download and modify offline

I clicked on the feedback button to report this problem. Let see what support comes back with.

Code to Install Syntax HighLighter

Here is the code you need to insert right above the </head> to enable Syntax Highlighter. This include all the different brushes provided. One thing I did do was find all the https links to the scripts.

<link href="http://alexgorbatchev.com/pub/sh/current/styles/shCore.css" rel="stylesheet" type="text/css">
<link href="http://alexgorbatchev.com/pub/sh/current/styles/shThemeDefault.css" rel="stylesheet" type="text/css">
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shCore.js" type="text/javascript">
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushCpp.js' type='text/javascript'/>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushCSharp.js' type='text/javascript'/>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushCss.js' type='text/javascript'/>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushJava.js' type='text/javascript'/>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushJScript.js' type='text/javascript'/>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushPhp.js' type='text/javascript'/>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushPython.js' type='text/javascript'/>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushRuby.js' type='text/javascript'/>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushSql.js' type='text/javascript'/>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushVb.js' type='text/javascript'/>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushXml.js' type='text/javascript'/>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushPerl.js' type='text/javascript'/>
<script language='javascript'>
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.all();
</script>

Installing from Browser

Find </head>, from a MAC use command+F, Windows CTRL+F.

After it is installed:

Now all Save theme and you all set with installation.

Editing Theme Offline

Use your favorite text editor, in my case Atom. Search for the XML file which is the theme downloaded on the previous step. Search for </head> and paste script provided.

Once ready upload the new theme to your blog using the Chose File.. and upload button.

Once done that’s it you are all set. Ready to test!

Using Syntax Highlighter

Now to use the SyntaxHighlighter place your code between the <pre /> tags and define your code type by setting the brush value.

<pre class="brush: Brush_Alias">
SCRIPT HERE
</pre>

For full list of supported Brush aliases check for manual for brushes http://alexgorbatchev.com/SyntaxHighlighter/manual/brushes/

Sample: To make PowerShell code block look like this:

    # Power Shell Script
    Write-Host "Simple script to get Windows Process"
    $process = Get-Process
    $process | Select-Object ProcessName, Id, CPU

Use this - brush type is ps or powershell

<pre class="brush: ps">
    # Power Shell Script
    Write-Host "Simple script to get Windows Process"
    $process = Get-Process
    $process | Select-Object ProcessName, Id, CPU
</pre>

If you are using Dynamic themes in your blog, you are not done. It will require adding additional lines of code to make it work. What are Dynamic View Themes? If you go under “Themes”, scrolld to the middle you can see Dynamic Views section:

Work around is to add the extra piece of code inside the right before closing </pre>

<script type="text/javascript">
SyntaxHighlighter.highlight();
</script>

So, for” Dynamic View” users, use the following format

<pre class="brush: ps">
    # Power Shell Script
    Write-Host "Simple script to get Windows Process"
    $process = Get-Process
    $process | Select-Object ProcessName, Id, CPU
    <script type="text/javascript">
    SyntaxHighlighter.highlight();
    </script>
</pre>

This should wrap up the Syntax Highlighter setup for you Blogger.com blog. To sum this up:

Intro to Syntax Highlighter 3.0.83 - latest version day blog was posted.
Backup/Restore Theme
Install System Highlighter into theme
How to use the Syntax Highlighter for those using Simple and Dynamic View themes.

Next I will try to figure out how to host the on my own share drive like: DropBox, OneDrive, Google Drive or anything down those lines.

VCSA 6.7 CLI Deployment issue with Target | Datacenter Parameter

2019-03-19T17:45:00.000-04:00

VCSA CLI deployment tools is one of the best way to automate the deployment of vCenter. Part of my role is to maintain a library of scripts to automate various task within vCenter as well as deploy and configure vCenter itself. Another really awesome use case is for vCenter Backup/Restore purposes. Say you backup your VCSA to FTP server, you could use this tool to deploy a VC then restore configuration from VCSA. When things go wrong is a lot easier to just destroy everything and start fresh.

vCenter Appliance v6.7 has been out for some time and I still haven’t found time to poke at it. Well today I finally had a reason to build a full v6.7 environment and ran into this annoying issue the parameters in the JSON file. Ran into this same issue during early days of VCSA v6.5 but somehow I did not document the fix and now I am scratching my head over this issue once again.

For those who are not sure what I am talking about. Inside the VCSA iso file there is folder with an executable vcsa-deploy that helps provision vCenters. In fact the VCSA UI deployment options facilitate json creation and then invokes that vcsa-deploy executable. The VCSA JSON Templates are located under following location: VMware-VCSA-all-6.7.0-8217866\vcsa-cli-installer\templates. I am looking under the install folder for new VCSA deployment. As you can see from the picture, there are two flavor of each VCSA or PSC type: on ESXi or on VC. You can deploy the appliance directly to a ESXi host or to a VC.

Purpose of this article is to address the "target" and "datacenter" parameter in the json file to deploy VCSA_on_VC

In this case, I wanted to use embedded_vCSA_on_VC.json template json. Populated all parameters according to the VMWare Deployment Configuration Parameters and I got the following error message.

================ [FAILED] Task: PrecheckTask: Running prechecks. execution
failed at 20:29:19 ================
Task 'PrecheckTask: Running prechecks.' execution failed because
[com.vmware.vcsa.installer.prechecks.host_configs: The path provided in the
target setting in section new_vcsa, subsection vc does not exist. Check the
value of the target setting in the JSON file provided to the installer. ],
possible resolution is []

VCSA-Deploy basically complaining about the "target" parameter: new_vcsa.vc.target. The same issue applies to the datacenter parameter.

Following section is extracted directly from the vCenter 6.7 | Deployment Configuration Parameters Documentation: VMWare Deployment Configuration Parameters

The target ESXi host or DRS cluster on which you want to deploy the appliance.

Important:

You must provide the name that is displayed in the vCenter Server inventory. For example, if the name of the target ESXi host is an IP address in the vCenter Server inventory, you cannot provide an FQDN.

If the target ESXi host or DRS cluster is located in a folder or a structure of folders, the value must be a comma-separated list of strings or a comma-separated list as a single string. For example,






["parent_folder", "child_folder", "esxi-host.domain.com"]






"parent_folder, child_folder, esxi-host.domain.com"

If the target ESXi host is part of a cluster, use a comma-separated list of strings or a comma-separated list as a single string to provide the path. For example,






["cluster_name", "esxi-host.domain.com"]






"cluster_name, esxi-host.domain.com"

Note:

The value is case-sensitive

For some reason i tried all those different combinations listed in the KB and did not work. Two important fact about this parameter is that it can be a string or array and is case sensitive. Using the cluster_name only worked. This is thanks to one of VirtuallyGhettos scripts.

Here is sample section that works:

"new_vcsa": {

"vc": {

"hostname": "10.10.10.10",

"username": "administrator@vsphere.local",

"password": "Passw0rd!",

"deployment_network": "Test-VL47",

"datacenter": "MakiDC",

"datastore": "Maki-Datastore02",

"target": "Maki-Cluster"

where:

- datacenter is the datacenter name

- target is the cluster name

Reference links:
JSON Templates for CLI Deplyment of the vCenter Server Appliance and Platform Services Controller Appliance

PowerCLI: Getting List of VMs Created in vCenter

2019-02-24T15:25:00.000-05:00

On my previous post, I was wrote about auditing reconfiguration task on specific VMs. But this time I simply wanted to get a full list of VMs created in vCenter. For this task I used a script from a website you probably already know "Virtu-Al.net":

Get-VIEvent -maxsamples 10000 |where {$_.Gettype().Name-eq"VmCreatedEvent" -or $_.Gettype().Name-eq "VmBeingClonedEvent" -or $_.Gettype().Name-eq "VmBeingDeployedEvent"} |SortCreatedTime -Descending |Select CreatedTime, UserName,FullformattedMessage -First 10

As you can see the script pulls max sample of 10K objects, what happen when I run this in my vCenter. Two things normally happens:

- You get couple or no VMs when you know that is not true. Then you try to increasing max sample to retrieve more events and you run into an issue.

- You get the deserializing error message when increasing the max sample number. Sometimes it will happen at 10k if not it will definitely occur if you increase that number to 100k:

Get-VIEvent: Error in deserializing body of reply message for operation 'ReadPreviousEvents'.

Well that is because you are using old version of PowerCLI. This occurred to me on a system running PowerCLI 6.5.x, and according to few articles and KB doc VMware was aware of this and was going to fix.

Well VMware did kept their promise. Tested the same script with PowerCLI 11.0# and issue was gone. Increased the max sample up to 300K and no error.

Keep in mind that the larger the number of samples you are trying to collect the longer it will take. In the next exercise, I will pull max sample of 400K and use the "measure-command" to measure how long this script took to complete.

If you are not familiar with the "Measure-Command" command-let you might want to check it out. Simple to use simply put in between the bracket code wish to execute. In this case, I want run Get-ViEvent and 400K max samples. As you see in the screenshot it took about 5mins to complete(not as bad as I thought) and I got 321800. This mean it only had 320K events.

If want to find out how far back you went simply subtract the "CreatedTime" from the last event from today's date. In my case it was about 30 days 17 hrs. So my vCenter currently retaining about 30days of data. To get more than that might require adjusting event retention on vCenter, this will cause your vCenter to eat up more disk space.

But let's work with what we got, I like doing break my scripts in sections. Notice I stored results of the "Get-ViEvent -MaxSample 400000" into $evt variable without using any filters. This way I can recycle the data, know how many events was collected in case, and how many of those events are actually relevant.

Now we can apply all filters, script takes lot less time to run:

$created = $evt | where {$_.Gettype().Name-eq "VmCreatedEvent" -or $_.Gettype().Name-eq "VmBeingClonedEvent" -or $_.Gettype().Name-eq "VmBeingDeployedEvent"}

After applying filters to get VMs created/cloned/deployed got 17 items.

To view these items in pretty formatted view apply another filter and sort by date.

Hopes this help understand when pulling "Get-VIEvents" from vCenter. If you need to pull other reports that might be stored in that $evt variable this might be the time to do them...

Simple PowerCLI Script to Audit a VM Task and Events - Reconfiguration Changes

2019-01-23T17:04:00.000-05:00

Purpose of this article is to show how generate a quick report to audit tasks and events on a target VM. The events in this report will be similar to what we see in vCenter VM Task/Events. If the events you are looking for are too old it might not show. However, this report will provide you with a more in depth detail of each task. This is useful when tracking specific tasks such as what reconfiguration was made, changes by who, relocated from to where to where, etc.

For this exercise, I deployed a test vm using one my scripts. Script will clone vm from template and perform a series of tasks. Next, we will be using the report to check what happened during the process. BTW, PowerShell v5.1 and PowerCli v10.1 was used to run the commands. Environment is running on a vCenter v6.0xx, but anything done in this exercise should work on any versions of vCenter.

Once the VM deployment was completed. Here list of Task shown in vSphere Web Client.

Here is the list of Events

Now lets run the script to get all events using Get-VIEvent:

$vmname = “testdp-1"
$vm = get-vm $vmname
$evt = $vm | Get-VIEvent
#or if you prefer a one liner
get-vm “tested-1” | Get-VIEvent

You should see the following output

Script returned a long list of events. To make this data bit easier to read we can grab CreatedDate and FullFormatedMessage field, format as table and sort it by date.

From this list we can see the following:

Date and Time where VM was created
Name of the VM it was cloned from
VM was Reconfigured several times
VM hardware compatibility was upgraded
VM hardware compatibility upgrade failed
VM was powered on and relocated
Started OS Customization
OS Customization succeed
VMWare Tools upgrade was initiated.

Now, lets try digging deeper into reconfiguration tasks. First, grab only task contains reconfigure. Here is a table format view to make sure the filter is working.
And here is list view of the same list of reconfigure events. Notice is the one that “Task: Reconfigure virtual machine” contains information regarding the reconfigure task, server name, event chain id of actual reconfigure task. Those starting with "Reconfigured.” are the one that contains the ConfigSpec details.

Lets take a closer look to the content of the task with ConfigSpec. You could do this couple different ways. By accessing the task and open the ConfigSpec parameter or export output into a JSON file.
Method 1 - $event[index].ConfigSpec
- This task was changing CPU and Memory

Method 2 - Export everything to a json file

$vmname = “testdp-1*"
$vm = get-vm $vmname
$evt = $vm | Get-VIEvent
$file = ".\" + $vmname.Replace("*","")+".json"
$evt | ConvertTo-Json -Depth 10 | Out-File $file

Sample output and checking out the ConfigSpec

Following Reconfigure.ConfigSpec was to change CPU, Memory settings.

Here another Reconfigure Task ConfigSpec that modified the VM NIC Mapping

As promised in the title this is a very simple script that can generate powerful report. This same report can be formated and ouput to a CSV file or even a HTML or Word document using PSCribo.

Cisco UCS-C220-M5 with ESXi v6.5x not Detecting 1GB X550-T2 NIC

2019-01-04T10:13:00.001-05:00

Today got reached out to deal with an issue 1GB NIC was not showing up on a bunch of Cisco UCSC-C220-M5##. Immediately suspected it was a missing driver issue on the original ESXi v6.5u2x. First thing first access the CIMC of the CISCO UCS host and checked Invenotry / Network Adapters and the Cisco LOM X550-T2 was showing fine.

The only to manage the host was using CIMC KVM/virtual console. Ran “esxcli software vib list” command and found ixgben v1.4.1.x and based on the VMware Compatibility Guide it should have worked.

If you track down the VMware ESXi 6.0 ixgben 1.4.1 driver notes it clearly states is compatible with ESXi6.5 and x550. For some reason the 10G device which uses the same driver is fine.

Anyway, on an quick attempt to fix issue this downloaded the latest ixgben version 1.7.10, created an iso, mount to host via CIMC virtual console, rebooted and still wouldn’t work. At the point the solution is to use the Cisco Custom Image for ESXi 6.5u2 GA ISO from the VMware Website.

This issue can be resolved by simply re-installing ESXi using the Cisco Custom Image for ESXi 6.5U2 GA Install CD ISO, but for learning purpose will be using the offline bundle zip file instead. Because we do not have network/ssh access, the only way present these files is via ISO and is already a build-in utility under Virtual Media to Create Image.

Creating ISO with Virtual Media / Create Image

Process is straight forward, simply click Create Image

Drop the zip file to create image

Download the ISO

Then, activate Virtual Devices and Map the ISO we just created.

Mounting the CD-ROM in ESXi using vsish

Once the ISO is mounted via CIMC / Virtual Device access your ESXi via shell and issue the following commands:

#start the cdrom module
vmkload_mod iso9660
# locate CD path
esxcfg-mpath -b | grep “CD-ROM"
#mount the device
vsish -e set /vmkModules/iso9660/mount mpx.vmhba##.C0:T0:L0
#access mounted device under /vmfs/volumes/
#in my case there was folder called mpx.vmhba33.C0:T0:L0

Now that CD-Drive is mounted and you can see the Custom Cisco Offline Bundle we can proceed with installation of vib file.

Note: Notice after issuing the esxcfg-mpath command it listed multiple paths (L0,L3,L5 and L6). I am not 100% why there are 4 different paths but mounting the one with L0 worked for me. I am assuming it can support mapping 4 iso files, or it might have detected the physical CD/DVD drive.

Installing ESXi Patch / VIB Drivers using ESXCLI

To install the patch use “esxcli software vib install -d “full path to zip file"

Once installed perform a final reboot and check to make sure the X550 device is now showing.

NSX v6.2-v6.3 - Adding CLI user to NSX Manager GUI via PowerShell

2018-11-05T16:34:00.007-05:00

This is actually an extention of my previous post "NSX 6.2.x-6.3.x - API against NSX Manager - 403 Forbidden”. On the post I used the curl command to add the cli user account into NSX Manager. Unfortunately the server where the command had to be run was a Windows machine so the only way around this was using PowerShell Invoke-WebRequest command.

Note: First part which is creating the CLI user account still needs to be done from the NSX Manager Console or SSH.

Here are the examples provided by KB 2150736:

Example 1:

POST https://<NSXMGR_IP>/api/2.0/services/usermgmt/role/api_username?isCli=true
<accessControlEntry>
<role>auditor</role>
<resource>
<resourceId>globalroot-0</resourceId>
</resource>
</accessControlEntry>

Example 2:

Continuing with the previous example from step 1, here is what the command would look like for curl:

curl -v -H "Content-Type:application/xml" -k -u admin -X POST -d "<accessControlEntry><role>auditor</role><resource><resourceId>globalroot-0</resourceId></resource></accessControlEntry>" https://<NSXMGR_IP>/api/2.0/services/usermgmt/role/api_username?isCli=true

Using both example as reference created a PowerShell script that will do the job:

PowerCLI Script:

#NSX Info
$nsx_admin = "admin"
$nsx_pwd = “adminPwd123!"
$nsx_mgr_ip= “nsxmgr_ip"
#NewCLIAccount
$nsx_user = "api_user"
####### No changes below this line - Will add user as an "Auditor"
$BasicAuth = $([Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("$($nsx_admin):$($nsx_pwd)")))
$Authorization = 'Basic {0}' -f $BasicAuth
$wrequest = "https://"+$nsx_mgr_ip+"/api/2.0/services/usermgmt/role/"+$nsx_user+"?isCli=true"
$body = @"
<accessControlEntry>
<role>auditor</role>
<resource>
<resourceId>globalroot-0</resourceId>
</resource>
</accessControlEntry>
"@
$headers = @{'Authorization' = $Authorization}
Invoke-WebRequest -Uri $wrequest -Method POST -ContentType "application/xml" -Body $body -Headers $headers

Updates:

If you are getting “The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel”. Add the following code before the Invoke-WebRequest and it should work.

# Disable SSL/TLS Certificate Verification (for non-publically accessible servers)
add-type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

Here is what final script should look like:

#NSX Info
$nsx_admin = "admin"
$nsx_pwd = "enter nsx admin password"
$nsx_mgr_ip= "#Nsx Manager Ip"
#NewCLIAccount
$nsx_user = "api_user"

####### No changes below this line - Will add user as an "Auditor"

$BasicAuth = $([Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("$($nsx_admin):$($nsx_pwd)")))
$Authorization = 'Basic {0}' -f $BasicAuth
$wrequest = "https://"+$nsx_mgr_ip+"/api/2.0/services/usermgmt/role/"+$nsx_user+"?isCli=true"

# Disable SSL/TLS Certificate Verification (for non-publically accessible servers)
add-type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

$body = @"
<accessControlEntry>
<role>auditor</role>
<resource>
<resourceId>globalroot-0</resourceId>
</resource>
</accessControlEntry>
"@
$headers = @{'Authorization' = $Authorization}
Invoke-WebRequest -Uri $wrequest -Method POST -ContentType "application/xml" -Body $body -Headers $headers

NSX 6.2.x-6.3.x - API against NSX Manager - 403 Forbidden

2018-11-02T11:50:00.000-04:00

I've started looking into all PowerShell Modules available to extract information from NSX. The environments I am working with runs NSX v6.2.x-v6.3.x. Started with Rest API commands and got stuck with the 403 Forbidden error message. I made sure it was not permission issue so elevated my test account to Enterprise Administrator. Also made sure it wasn't a port issue between the server and the NSX manager. Check here for Ports and Protocols Required by NSX.

Then decided to try PowerNSX from the PowerShell Gallery and was still getting the same error message:

After poking around noticed several thing:
- vSphere local admin account (administrator@vsphere.local has no issues connecting to the NSX Manager and the account is an “Enterprise Administrator”, and so is my test account.
- There is an admin account in v6.3.x is also able to connect using PowerNSX.
- Between NSX Manager 6.2.x and 6.3.x. In v6.3.x there is a Role called "NSX CLI User" which only the “admin” user has.
- Conclusion: Only the administrator@vsphere.local and NSX Cli User are allowed to run APIs.

And there is no way to assing the NSX CLI user role from the web client:

Only way around is to Create a CLI USER.

Step 1 - Creating the API/CLI user account in NSX Manager

SSH or Console into your NSX Manager with admin account

## From nsx manager commandline enter command:
enter
configure terminal
nsxmgr(config)#
nsxmgr(config)# user api_user password plaintext Password123!
nsxmgr(config)#
nsxmgr(config)# user api_user privilege web-interface
nsxmgr(config)# exit
nsxmgr#
nsxmgr# write memory

## To verify account was created
show running-config

Step 2 - Enable the new API/CLI account for web client use

Next is to create the same user via API to make it appear in the web client. This can be perform via PostMan, RestClient or curl. Since I am using a Mac will be using curl.

# Replace the NSXMGR_IP and the api_user
curl -v -H "Content-Type:application/xml" -k -u admin -X POST -d "<accessControlEntry><role>auditor</role><resource><resourceId>globalroot-0</resourceId></resource></accessControlEntry>" https://<NSXMGR_IP>/api/2.0/services/usermgmt/role/api_user?isCli=true

This command will add the api_user account to webclient and assign Auditor role. Log into webclient and make sure account was created. Change the role if desired.

Link to VMware KB Doc: Creating a User for NSX CLI and assigning API roles/permissions (2150736)

Now you should be able to connect using Connect-NSXServer command let. It will first ask for the NSX User credentials and next you will need to enter the vCenter credentials. If you API User account have enought right you can enter the credentials again. Mine is an “Auditor” so entering different vCenter credentials.

You can also pass the vCenter Credentials using the -VIUsername and VIPassword. To get help on this command simple run Get-Help | Connect-NSXServer

PowerCLI - Create vCenter Nested Folder Structure from File

2018-10-26T12:28:00.002-04:00

I was in the process of migrating everything off my old lab into a new one. Already had script to rebuild all the datastores, vdswitches, port groups, etc but not the VM folders. Found the Get-FolderPath script created by LucD the does the hard work. BTW, attended couple of LucD's session in VMworld and it was awesome. He was in the documenting virtual environment with PowerShell session if I am not mistaken.

Lets get right into it.

Script syntax:
Get-FolderPath -Folder (Get-Folder -Name “FolderName")

Extracting Folder Structure from Source vCenter

Now to here is a simple script to get a list all folders in your source datacenter. Must connect to your source vCenter first.

# Define dcname for VC with multiple datacenter

$dcname = "Maki-DC"
# Get list of VM folders only
$folder_list = Get-Datacenter $dcname | Get-Folder -Type VM
# Loop Through list and get the folder path
$folder_paths = $folders | % {(Get-FolderPath -folder $_).Path }
# Export into a txt file
$folder_paths | Out-File .\vm_folder_list.txt

Once script is completed it should generate the following text file:

Disconnect from source vCenter.

Create Folder Structure in Destination vCenter

Now here is the script to create the folder structure using text file generated.

$folder_list = get-content .\vm_folder_list.txt
$dcname = "Maki-DC"
$datacenter = Get-Datacenter -Name $dcname
# Define VM root folder
$rootFolder = $datacenter | Get-Folder -Name vm -Type VM
Foreach ($folderpath in $folder_list){
$fullpath = $folderpath
# Split fullpath into individual folder names
$directorylist = $fullpath.split('\')
# Skip datacenter folder name
$directorylist = $directorylist | Where {$_ -ne $dcname}
$parentfolder = $rootFolder
# Loop throught the list of directories and create them if not found.
# Folder are created row by row, from left to right
# Datacenter Root \ Subfolder1 \ Subfolder2 \ Subfolder3
# Once the folder is created it becomes the parent for subfolder.
Foreach ($directory in $directorylist){
$current = Get-Folder -name $directory -ErrorAction SilentlyContinue
if ($current -eq $null){
Write-Host "Creating new folder $directory" -ForegroundColor Green
$parentfolder = New-Folder -Name $directory -Location $parentfolder
}else{
Write-host "Skip - Found folder $directory" -ForegroundColor Yellow
$parentfolder = $current
}
}
}

Note: Before running the create folder structure script need to disconnect from your source vCenter and connect to the destination vCenter. It will cause issues if you are connected to both vCenter and both datacenter using same name.

This is what it looks like when script is running:

What the script is doing:

Once is done:

This was tested on vSphere v6.0 and v6.5...

PowerShell: Using PScribo to Generate Report

2018-10-25T20:42:00.000-04:00

I was doing on some data analysis against a CSV. Then I started using PowerShell to manipulate the data and do some calculation. Next to take it a step further thought about generating a pretty report in HTML format. Thought this might be a great opportinity to learn how to use PScribo. The very first time I heard about PScribo was in one of the VMworld sessions I think it was about documenting your virtual environment using PowerShell.

Downloaded the PScribo module from the PowerShell Gallery and downloaded the sample script provided by the creator (Github:iainbrighton\PScribo).

Now lets start by taking a look at the CSV data. CSV file contains nothing more than host information, cluster, cpu, memory and ESX version. From this information I want to process and get the amount of free memory and cpu.

Created a script that will process the CSV data, masage it, sort it and perform some simply calculation. Once the done here is the output on screen. Note: Didn’t cover how I processed the data. Might do that on a separate post.

There is no easy way to produce a nicely formated output into a document without without some tools. After few mins looking into the PSscribo sample I was able to produce a HTML and nicely formated text file.

Created Summary of all locations and highlighted location without enough resources to handle host a host failure.

Next section is break down of each location and individual host resource break down.

Here is the same output generated into a text file.

How this is done??

Once you have your data from the CSV file processed into tables or PSobject variables all you need to do process it using PScribo. Refer to the example provided by the creater is extremely easy to follow. I used sample to generate this output.

###### Section Uses PS Scribo to Turn table in to HTML, TXT, DOC
$document = Document 'ReportDoc' -Verbose {
# Section 1
Style -Name 'HighUsage' -Color White -BackgroundColor Firebrick -Bold
Paragraph -Style Heading1 'Summary Host Usage Report for All Locations'
Paragraph -Style Heading3 'Cell marked red means that there is not enough resource to handle a host failure'
Section -Style Heading2 "Host Usage for GEO" {
$reportclone = $report.Clone()
$reportclone | ? { ($_.MemFree) -lt ($_.HostMemSize) } | Set-Style -Property 'MemFree' -Style HighUsage
$reportclone | Table -Columns Datacenter,Cluster,NumHost,NumVCPU,CPUFree,MemoryMB,MemFree -Headers Datacenter,Cluster,NumHost,NumVCPU,CPUFree,MemoryMB,MemFreeMB -Width 0
}
PageBreak;
# Section 2
Paragraph -Style Heading1 'Detailed Host Usage Report Each Location'
Paragraph -Style Heading3 'Cell marked red means that memory usage is over 90%'
Foreach ($dc in $datacenters){
$dc_data = $csv_data | where {$_.Datacenter -eq $dc}
Section -Style Heading2 "Datastore Usage for $dc" {
$highusagetable = $dc_data.Clone()
$highusagetable | ? {$_.'Memory Usage %' -gt 90} | Set-Style -Property 'Memory Usage %' -Style HighUsage
$highusagetable | Table -Columns Host,DataCenter,Cluster, '# vCPUs', 'CPU usage %', '# Memory','Memory usage %', 'ESX Version' -Headers Host,DataCenter,Cluster, '# vCPUs','CPU usage %', '# Memory', 'Memory usage %', 'ESX Version' -Width 0
}
}
}
$document | Export-Document -Path .\ -Format Word,Html,Text -Verbose;

The script will generate a Word,Html and Text file. Simple notes:
- Paragraph is to define text there are different styles, heading format and colors you can use.
- Set-Style is to define styles. I used this to highlight values were needs attention.
- Table output: you can select the columns you want to output and even adjust the column name, width etc.

The second section will loop thru each location and list out host information. Each location with its respective host information.

This was the first time using PScribo and it was fairly user friendly. I attempted using vCheck to generate HTML files but requires a lot more work. This is a perfect solution for quick an easy documentation. Thumbs up to the creator!

vCenter 6.5 Health Rest API

2018-10-13T11:19:00.001-04:00

Exploring the vSphere v6.5 Rest API has been in my todo list since it was released. I’ve been pretty busy with support and engineering work as well as up keeping my existing code library with PowerCLI, Python (PyvMomi), VMware Orchestrator and even Ansible code. And of course I can only use personal time to blog. I have all these script languages why vCenter REST API now? If you take a closer look, there are plenty code on how to configure and create in almost anything in vCenter but nothing to monitor vCenter. So the piece that caught my attention was the vCenter Appliance Health API only available in REST.

So in this article we will be explore was is the Health API all about.

To get started we will be taking a quick look at the available APIs by access a vCenter v6.5.# landing page and click on the Browse vSphere REST API link.

Once inside the API Explorer main page, you can select the API from the drop down: appliance, cis, content, vapi and vcenter. For this exercise we will only be looking at the appliance.

To be even more specific we want to look at all the items listed under health:
- application management services
- database storage health
- load health
- memory health
- software packages - provides information on VUM repo packages.
- red: indicate security packages available
- orange: non security packages available
- green: no updates available
- gray: error retrieveing updates

- storage health
- swap health
- system health
- last health check time stamp

Part 1 - Using the vCenter API Explorer

Lets try health/applmgmt - expand to see the API, read through the content and click “TRY IT OUT!”.

It will show the full curl command, request URL, response body, reponse code and response header basically everything you would normally expect from this command. What do we get from this health/applmgnt API? From the response we get “green” meaning services all services are good and code 200 meaning request was successful.

It would be nice that in future releases they break down the status of each service. Perhaps green when everything is healthy, when orange or red pointing out which service is down.

Before I jump into PowerShell mode, I want to take a look at one more API. Check out the health/databasestorage API. Example value when “orange” when status 200 and sample response message when code 400. OK.

Now lets try out this health/databasestorage API. CLICK. Notice that we are getting a com.vmware.vapi.std.errors.unauthenticated in the response body and response code 401. This mean it failed to get a response and requires authentication in order to perform this request.

To get pass the authentication error click Login in the top right corner, enter your credentials and try again. This time you should see your vCenter’s database storage health and the response code 200.

Part 2 - Writing PS Scripts to get vCenter Health

This next section will translate everything we did in API Explorer to PowerShell.

Let’s use health/database-storage API for this exercise. From the API explorer we can obtain the Curl example, and the Request URL:

Curl:

curl -X GET —header 'Accept: application/json' --header 'vmware-api-session-id: 2b66ca49e7dc2e105d69bf52a4a75f77' 'https://vcenter_ip/rest/appliance/health/database-storage'

Request URL:

https://vcenter_ip/rest/appliance/health/database-storage

To satisfy the request we must obtain the “api-session-id” which will be used as an authentication token.
Following code will perfom the following task:
- Define target vCenter IP or FQDN
- Get Credentials to access vCenter
- Request session id from vCenter using Invoke-WebRequest
- Session ID will be used repeatedly throughout the rest of the script to perfom calls against the health API.

$vcenteraddr = "vCenter_IP_or_FQDN"
$credential = Get-Credential
$authentication = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credential.UserName+':'+$Credential.GetNetworkCredential().Password))
$header = @{
'Authorization' = "Basic $authentication"
}
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
# Request a token from cis, using vCenter Credentials
$response = Invoke-WebRequest -Uri https://$vcenteraddr/rest/com/vmware/cis/session -Method Post -Headers $header
$sessionid = (ConvertFrom-Json $response.Content).value
# this authentication header will be used for all future request
$header = @{'vmware-api-session-id' = $sessionid}

Once the session-id is stored in our new header we can proceed bu using the Invoke-RestMethod.

# Get database storage health status
$request_url = "https://$vcenteraddr/rest/appliance/health/database-storage"
$response = Invoke-RestMethod -Uri $request_url -Headers $header
Write-Host "Health status of database storage: $($response.value)"

Executing the code will give out following output:

That is how simple it is… Now we can finally add some health monitoring screen into our code library. With this you can certainly go as creative you want. Below I’ve included the rest of the code I am testing with.

$vcenteraddr = "vCenter_IP_or_FQDN"
$credential = Get-Credential
$authentication = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credential.UserName+':'+$Credential.GetNetworkCredential().Password))
$header = @{
'Authorization' = "Basic $authentication"
}
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$response = Invoke-WebRequest -Uri https://$vcenteraddr/rest/com/vmware/cis/session -Method Post -Headers $header
$sessionid = (ConvertFrom-Json $response.Content).value
$header = @{'vmware-api-session-id' = $sessionid}
# Get health status of applmgmt services
$request_url = "https://$vcenteraddr/rest/appliance/health/applmgmt"
$response = Invoke-RestMethod -Uri $request_url -Headers $header
Write-Host "Health status of applemgmt service: $($response.value)"

# Get database storage health status
$request_url = "https://$vcenteraddr/rest/appliance/health/database-storage"
$response = Invoke-RestMethod -Uri $request_url -Headers $header
Write-Host "Health status of database storage: $($response.value)"

# Get appliance health load
$request_url = "https://$vcenteraddr/rest/appliance/health/load"
$response = Invoke-RestMethod -Uri $request_url -Headers $header
Write-Host "VCSA load health: $($response.value)"
# Get appliance memory health
$request_url = "https://$vcenteraddr/rest/appliance/health/mem"
$response = Invoke-RestMethod -Uri $request_url -Headers $header
Write-Host "Health memory health: $($response.value)"
# Get appliance storage
$request_url = "https://$vcenteraddr/rest/appliance/health/storage"
$response = Invoke-RestMethod -Uri $request_url -Headers $header
Write-Host "Health storage health: $($response.value)"
# Get appliance swap
$request_url = "https://$vcenteraddr/rest/appliance/health/swap"
$response = Invoke-RestMethod -Uri $request_url -Headers $header
Write-Host "Health swap health: $($response.value)"
# Get Appliance system
$get_command = "https://$vcenteraddr/rest/appliance/health/system"
$response = Invoke-RestMethod -Uri $get_command -Headers $header
Write-Host "Health system health: $($response.value)"

Here is my vCenter’s Health:

ESXi Management Network Down After Host Upgrade

2018-09-10T18:05:00.001-04:00

Today was just one of those day. It all started with a simple ESXi host upgrade from v5.5u2 to v6.0u3x, in the process decided to do firmware upgrade. This was a Cisco UCS C-series host running CIMC v2.0 and couldn’t be happier updating it to the new v3.0x UI. While looking for the firmware saw 4.0(1a) was released, so decided to go with 4.0 instead. Few hours later, ESXi v6.0u3x was loaded and CIMC firmware 4.0 was loaded as well. From vCenter, noticed the host was still showing disconnected so launched the KVM Console and everything looked normal. Management network and everything was configured properly but couldn’t ping gateway or anything. I thought it could have been something related to the 4.0 firmware. Never had this issue during my previous firmware upgrades to v3.0. Spend the next hours or so doing all sort of things just to find out that the solution was as simple as two mouse clicks.

So what did I do?
After the Cisco UCS firmware was upgraded to 4.0(1a) I thought it might have been a driver issue. Went to the Cisco Hardware and Software Compatibility page, enter my Server Type, Model, OS Version, etc and a Driver ISO was available.

Downloaded the Driver ISO and decided to mount via Virtual Devices in the CIMC. Next thing I did was figure out how to mount the CDRom from inside the ESXi. Came accross vGhetto’s How to mount cdrom using vshish (https://www.virtuallyghetto.com/2011/04/how-to-mount-cdrom-using-vsish-on-esxi.html). Straight foward and simple instructions. Mounted the CDRROM, browsed all the way the the network driver folder just to find a html file. The hmtl files had a link to the Vmware website where driver could be downloaded.

Browsed to the link and downloaded the driver. Access the host via CIMC KVM Console, ran few commands and noticed the driver for the NIC was newer than the one listed in the VMware site.

The driver downloaded was in form of a ZIP file and the only way to make this available to the ESXi host was mounting ISO through the KVM. Searched online to found a utility to create and ISO and found (ISO Creator) free utility to convert a folder to ISO.
- Mounted the ISO to the active KVM Session
- Run few command to mount CDROM using vshish inside the host
- Browsed to the CD

Noticed that from the host, the content of the CD was weird. Tried to generate new ISO and same result. From windows, the CD content looked fine it had the vib and the zip file, but from the ESXi host it was just showing the partial file name with no extension.

Decided to run the installation commands against the IGB file anyway. Loaded the driver using the Cisco Driver Installation documentation.

To install the driver on ESXi Release 5.x/6.x:
esxcli software vib install –v /path/async-driver.vib

Ran the command agaist the IGBN file and it worked. It was installed successfully and needed a reboot to complete.

To finalize the installation I rebooted the server. Everything came up fine the driver was loaded but the management network still wasn’t working!!!

Decided to step back and check the network configurations and noticed something. From the ESXi host console vmnic0 was selected and that is usually our 1GB uplink for build/management purposes only.

When I ran the shell command to list all the vmnic devices. It was telling me that vmnic0 was now the 10GB uplink?!? Somehow the order of the NIC was reversed after the CIMC Firmware upgrade. Duh I did noticed vminc2 was the 1GB uplink but didn’t occur to me to select that device to begin with

Solution was simply to remove the deselecting vmnic0 and selecting vmnic2…

After all this unnecessary work what did I learn today?
- How to create ISO files from a folder
- How to mount a CDROM in ESXI using vsish
- How to check make sure the correct network adapter is selected before blaming it on the network driver