Seems like Dell fixed their repository and vLCM can now download the VIBs again as it should when adding a VIB to an image.

Here is the original article I wrote

We have been using the Dell depot in vLCM for a while but since a couple of days we see issues when we try to include some VIBs in an image. Including in the image works, but as soon as we try to remediate a host, vCenter try to download the VIB but that fails and thus remediating the host fails.

We added the Dell repository to our vCLM config:

Next thing we do is add a VIB from this repository to the image we will apply to our cluster:

But when we try to remediate the cluster, it tries to fetch the VIB from the Dell repo but fails …

It seems like the VIB that needs to be downloaded is not available at the URL vCLM gets from the Dell repository ….

I am able to access https://vmwaredepot.dell.com from my vCenter Appliance but trying to fetch the VIB gives me a 404 error: (In this screenshot a proxy is used, but even with a direct connection it fails)

Seems to me like Dell changed the directory structure on https://vmwaredepot.dell.com but forgot to update the xml index file ….. Getting in touch with Dell to get this fixed, will update this article when I get more info.

For this to work (due to the fact you can only configure the same subnet on the same front-end port for each node) this can only be done when in my scenario, port 2 of both Node A and Node B connect to the first switch, that only has storage network 1, and port 3 of both Node A and Node B connect to the second switch, that only has storage network 2.

So far so good, but as soon as I connected both ports 2 to switch 1 and ports 3 to switch 2, the PowerStore showed me an error: “Appliance port pairs are connected to the same data switch” Which is by design in my case and according to Dell’s best practices to use 2 storage networks for NVMe ….

But luckily there is a Dell knowledge base article (this one talks about iSCSI but it’s the same for NVMe) that says:

So in this scenario, “the warning can be safely ignored” which is what I did

First a part about the on-board mezzanine card:

This means the ports on the mezzanine card have about 63.04 Gbits/sec of bandwidth available. When using 4 ports at 25Gbit/sec, we can not use the full bandwidth these ports offer. So what about the I/O module slots?

This part is about the I/O module slots? Slot 0 is 16-lane PCIe Gen3 which gives you about 126 Gbit/sec, which should be sufficient for the 4 port 25Gbit/sec card …

But ….. Regarding the I/O module:

So the 4 x 25Gbit/sec I/O module itself is 8-lane PCIe Gen3 …. So even in slot 0 you can not use the full bandwidth of this card ….

Since PowerStore OS there is a new 2-Port Ethernet card that supports speeds of up to 100 Gb/s. This 100 GbE card is supported on PowerStore 1000-9200 models in I/O Module 0 slot, the 16-lane PCIe Gen3 slot, which is still limited to 126 Gbit/sec ….

If you want to combine ethernet connectivity and Fibre Channel, it gets even more interesting, since the preferred slot for the 4 port 32Gbit/sec FC I/O module is slot 0 …. In slot 0 it can run at full capacity. When adding more FC ports to slot 1, the combined ports bandwidth is again limited to 63.04 Gbits/sec

So when designing a high bandwidth PowerStore environment, keep this in the back of your mind ….

This was an HP host installed with the HP ISO image, but not sure if that is the cause of this issue.

When investigating the logs on the host I noticed that /var on the ramdisk was full

When issuing vdf -h available space for /var on the ramdisk was  0%

Looking in /var/log i noticed all logfiles where symlinks to /scratch except for the EMU directory, where some Emulex process seemed to fill up a log file …..

When removing the logfile /var/log/EMU/mili/mili2d.log and after restarting hostd, space was freed up on /var in the ramdisk, but the logfile /var/log/EMU/mili/mili2d.log returned and started filling up again.

Googeling I found a suggestion to remove the Emulex vibs when not using an Emulex HBA, but these hosts did have Emulex HBA’s

After some more research I found a fix that did not need a reboot or maintenance mode (which is great since vMotion stoped working on these hosts):

First, edit /etc/config/EMU/mili/libmili.conf and set MILILogLevel to 0

Next, remove the logfile /var/log/EMU/mili/mili2d.log

And then restart hostd with /etc/init.d/hostd restart

Check with vdf -h and /var should have diskspace available on the ramdisk

past these commands on a host that has the issues if you do not manually want to edit the conf file and want to fix the host at once.

First line sets log level to 0, the second line removes log file, the third line restarts hostd and the last line outputs disk utilisation so you can check if space is available again on var on ramdisk

sed -i ‘s/^MILILogLevel\ 4/MILILogLevel\ 0/g’ /etc/config/EMU/mili/libmili.conf
rm /var/log/EMU/mili/mili2d.log
/etc/init.d/hostd restart
vdf -h

First part of the solution was to uses FQDN on the ESXi hosts. Only short hostnames where configured on ESXi hosts, which probably caused Log Insight not being able to match the logs it received from the ESXi hosts to the registered hosts it learned from vCenter. Because of this all hosts where missing the vmw_vr_ops_id metadata and this metadata is used by vROps to pass to Log Insight to find the logs for the correct host.

After fixing this, one host still had no vmw_vr_ops_id metadata.

Seems like for whatever reason, matching of the ESXi hostname to the name used to register the host in vCenter is case sensitive. After changing case for th hostname on the ESXi server, the match was made, the metadata was added, and the in-context log search worked … Probably a bug ….

I did find some kb articles regarding some wrong vCenter database entries for LACP in previous upgrades, so I had a feeling this was related to LACP (which I do not use) …

So I decided to temporary configure LACP to maybe correct these wrong entries in the vCenter Database …

Since both vDS switches had two uplinks, and configuring LACP while your switches do not have LACP configured is generally a bad idea since you will break connectivity, I decided to first remove one of both physical adapters connected to each host from the uplink port group.

Next step was to enable LACP on the uplink port group.

Use Edit Uplink Portgroup settings on this vDS, Select LACP and set status to Enable, followed by save. Next, Use Edit Uplink Portgroup settings again, and set status to Disable.

Next thing I did is add the removed NIC to the uplink port group again.

After this quick workaround, I was able to upgrade my vDS to 6.5

After trying some stuff I found a log file which showed:

2016-11-15T20:47:38.547Z - error: vimService creation failed: Error: Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7fe24005a890] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)
2016-11-15T20:47:38.549Z - error: Login failed for source VC (Not found for PSC): Error: Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7fe24005a890] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

But I was 100% sure I did use the correct root password for the PSC

Googeling did not show the exact cause for this problem, but since I had seen issues before with expired passwords on the VCSA, I logged in to the PSC with the root account, which did work … So I decided to change the root password …

After changing the root password, I was able to upgrade the PSC

So most likely changing the root password “unexpired” the password. So the expiration for the linux password is not the same as the expiration for the password used for the API I presume …

By the way, if you would like to reuse the same password (by default you can not reuse a password that was already used before) you can issue the following command to clear the  password history before changing the password:

echo “” > /etc/security/opasswd

Keynote starts with Sanjay Poonen,General Manager of End-User Computing.

Sanjay starts talking about the transformation of education, healthcare, and low tech branches like tea estates, digitally transforming the end user.

Workspace one brings applications and eco systems together.

Sanjay is going to talk about the 3 layers, Apps & Identity, Desktop & Mobile, and Management & Security. Workspace One is the way to go. Sanjay promises live demo’s, let’s see … Ah, videos again …

Sanjay is doing a live commercial for Workspace One with recorded videos.

First shows some AirWatch features and Horizon with Nvidia support, nothing new yet, except for an application to approve T&E.

Sanjay invites Stephany Buscemi, Executive Vice President at Salesforce Marketing on stage. She talks about a Salesforce app with deep insight in the sales processes.

No talks about Fusion and Workstation, ant tels free license is available from the VMworld App, nice.

Back to AirWatch, customer success story video is shown.

Next is a demo for unified endpoint management, conditional access is shown, where work related info is not allowed to be posted on Twitter for instance. NSX is used to limit access to specific sets of the datacenter.

Next is a demo for VMware TrustPoint by Tanium, for threat detection and remediation.

Now ray O’Farrell takes the stage, VMware’s CTO. Ray invites Kit Colbert on stage talking about Cloud Native Applications. Who are using containers? … Hipsters …. Kits says he’s partially hipster, he loves hipsters …

Kit talks about extending existing virtual infrastructure to support containers, using vSphere integrated containers and/or Photon Platform.

Demo time, or actually, video time, showing provisioning containers from a repository.

Talking about a Virtual Container Host, which kind of looks like a vApp, where containers are provisioned. Shows standard vSphere tooling like NSX and vRops support containers, and resources for containers can be configured.

Now talks about using vRA for automation and governance. Using vRA to request an extension of the resources to be used for containers.

Next a video with talking about using vSphere integrated Containers.

Photon Platform is more targeted at greenfield installations, without vSphere.

Who versions at this moment, in the near future a version with Kubernetes and NSX support.

Ray is back on stage, talking about SDDC

New vSphere announcement will be done in Barcelona, not here.

Now Rajiv Ramaswami, EVP/GM Networking and Security talks about NSX.

Average cost of a data breach is $ 4.000.000 … NSX can prevent this using Micro segmentation.

NSX solves Security, Automation and Availability issues. Rajiv now mentions several customers and how they use NSX, followed by some customer videos.

Talking about vRealize Network Insight now (think this is the former Arkin product)

Accessible form the NSX gui and visualizing security policies, and able to convert seen traffic to security policies. Looks promising.

Next is Yanbing Lee, SVP Storage and availability at VMware, talking about VSAN.

High level story on why VSAN is great, not too many details unfortunately …. No face melting news so far.

Ah, demo time, using vRA to move workloads based on expected issues in the near future, not local, but to Softlayer in public cloud … Seems like integration finally starts to work.

Ray takes the stage again for a wrap up.

Pat Gelsinger, VMware’s CEO kicks of limping a little due to a broken foot.

21 Alumni Elite present, whom have attended all VMworlds (all US VMworlds to be more specific) They will all get lifetime access to VMworld for themselves and their spouses.

Let’s see if VMworld Europe Alumni Elite will get the same privileges …

Pat continues to teach us some history, and plays an audio fragment of Eric Schmidt from Google, taped in 2006 on a search engine conference talking about putting workloads in the cloud. Salesforce was founded that same year. This year 160 million workloads are running in private and public cloud. In 2021 it is expected 50% of all workloads will run in the cloud. And in 2030 50% of all workloads will run in public cloud.

Pat continues to talk about growth and IoT as driver of this growth, and stresses he thinks IT keeps growing and will not shrink as some others expect.

“Cloud will be a force that expands IT”

Vertical that embraces cloud most is the Technology Vendor vertical.

“Self-starting IT” aka shadow IT is growing faster than ever.

Traditional IT systems are doomed to fail …

It will be about Freedom v.s. Control, using the parent kid relation as an example.

Users want freedom, IT wants control and compliance for instance.

Pat plays videos from Raghu Raghuram from VMworld 2011 in Copenhagen talking about SDDC and customer stories by Zebra, iGov and SAIC.

We must address the next industry challenge.

Pat announces the “Cross-cloud Architecture” for cloud freedom and control and talks about the “VMware Cloud Foundation” product. See this as a hardware independent EVO product with integrated deployment and lifecycle management (in short)

Pat invites Robert Leblanc on stage, responsible for IBM cloud business to talk about their joint efforts to build public cloud. They are joined by Marriott being one of the customers running on the joined VMware and IBM cloud.

Pat continues to talk about Cross-cloud Services, and is joined by Guido Appenzeller, Chief Technology Strategy Officer at VMware. Guido talks about the changing role of IT in this time of Mega Clouds. Guido invites Motti Finkelstein from Citi on stages to talk about the changing role of the IT department. Key is using a hybrid model to ramp up resources when needed. Challenge is cross-cloud management, with a focus on security and compliance. So what has VMware in stock for us to do this …. drum roll …

Cross-cloud services sounds almost like “cloud virtualization”, have a single pane of glass to deploy workloads in different public cloud providers.

Guido gives a little demo with a tech preview version of the software, not very detailed.

Next John Spiegel from Columbia Sportswear is invited on stage. John starts showing his cloths that are all part of the Columbia brand family and recommends reading the book “The Phoenix Project” which still happens to be on my “to read” list. They continu to talk about the Arkin product, recently acquired by VMware to give insight in to network traffic in NSX.

The demo continues by deploying NSX in the public cloud and have the same features as NSX offers in your private cloud. This can be used to apply network policies to your network in the public cloud, and encrypt traffic between all VM’s for instance, pretty impressive!

Next customer on stage is Josh Warsop  from Johnson & Johnson talking about architectures for IoT, and show they move a database tier cross the globe and cross cloud providers, Amazon and Azure in this case.

Video from GE is shown about IoT OS Predix.

Pat is back on stage talking about the VMware Vision and plugs tomorrow’s keynote from Kit Colbert and Sanjay Poonan.

Next up is Michael Dell, CEO and Chairman for Dell. Funny to see lots of people starte running to get his picture, he’s almost like a rockstar, luckily I sat on from row yesterday during the TAM keynote where Michael showed up as well

This concludes todays keynote, more tomorrow.

Installing the PSC went fine, but when installing the vCenter appliance, I was not able to register it to the PSC. I kept getting the message “Invalid credentials” every time I entered the SSO administrator password. Redeployed the PSC several time, using different passwords, but no luck registering the VCSA.

Then I suddenly remembered I added and trusted the Certificate Authority certificate that I downloaded from the original PSC on the host I was trying to reinstall the PSC and VCSA from.

So I decided to delete the trusted CA root certificate from the certificate store on my windows host, and voila, I was able to register the VCSA with the PSC …

So seems like redeploying the PSC and registering the VCSA with the PSC when you added the CA root certificate on the PSC to your trusted Root certificates causes the registration to fail …. Probably since the CA has the same name, but is not the same certificate ….

And no, there was no KB article on this on the release date