[ERROR] Exception thrown when getting domainmanager workflows
[ERROR] AP tool is unable to determine the SDDC-M in progress day-n workflows.
AP tool is unable to determine if the SDDC-M has any in-progress day-n workflows. SDDC-M service updates are disruptive.

The Async Patch Tool in VCF is designed to streamline patching operations across your SDDC environment. However, the error message indicates that the tool is having difficulty identifying ongoing workflows/executions due to inconsistencies, often involving workflows that have incorrect names in the SDDC Manager database.

The issue generally arises because of ‘null‘ values assigned to workflow names within the SDDC Manager’s database – ‘executions’ table in ‘operationsmanager’ schema. These ‘null‘ values disrupt the tool’s ability to track and manage the progress of patching workflows, leading to the mentioned error.

To resolve this issue, you need to update any workflows with ‘null‘ names.

Here’s a step-by-step guide to applying the necessary updates:

1. Take a snapshot of SDDC Manager vm;
2. SSH to SDDC Manager as vcf user. Enter su to switch to the root user.
3. Connect to the SDDC Manager database (operationsmanager schema): psql -h localhost -U postgres -d operationsmanager

psql -h localhost -U postgres -d operationsmanager

4. Execute a query to find all workflows where the name is ‘null‘:

select * from execution where name is null;

5. Once you have identified the workflows with ‘null‘ names, you can update them:

update execution set name = '' where name is null;

6. Rerun the Async Patch Tool. With the database now clean of null names, rerun the Async Patch Tool. It should now be able to determine the in-progress workflows without encountering errors.

The post VCF 5.x – Async Patch Tool and SDDC-Manager in progress day-n workflows. appeared first on vconfig.pl.

1. Firstly, the certificate for the AVI (controller) needs to be added in the SSL/TLS Certificates. Additionally, the Aria Operations for Logs certificate should be added to a newly created PKI Profile, ensuring that CRL checking is disabled.

SSL/TLS Profile: go to Templates -> Security-> SSL/TLS Certificates

PKI Profile: go to Templates -> Security -> PKI Profile

2. Next, SSH into the controller using the admin account. After entering ‘shell’, you will be prompted to input your credentials again and run below commands:

> configure alertsyslogconfig AOL-syslog
alertsyslogconfig> syslog_servers
alertsyslogconfig:syslog_servers> syslog_server 192.168.79.1
alertsyslogconfig:syslog_servers> syslog_server_port 6514
alertsyslogconfig:syslog_servers> tls_enable
alertsyslogconfig:syslog_servers> no udp
alertsyslogconfig:syslog_servers> ssl_key_and_certificate_ref System-Default-Portal-Cert
alertsyslogconfig:syslog_servers> pkiprofile_ref vrli
alertsyslogconfig:syslog_servers> save
alertsyslogconfig> where
alertsyslogconfig> save

tls_enable  – set to true;
no udp – TCP port will be used;
pkiprofile_ref – name of PKI profile created for AOL certificate in step 1;
ssl_key_and_certificate_ref – certificate used for NSX ALB controller selected in step 1;

The syslog should be immediately visible in the NSX ALB UI:

3. Add notification to Alert Actions: go to Operations -> Alerts -> Alert Actions and edit each action, where you wish to include your recently configured syslog server.

4. To confirm that everything is functioning correctly, you can check the logs at: /var/lib/avi/log/alert_notifications_debug.log

5. View at NSX Advanced Load Balancer dashboards in Aria Operations for Logs:

The post NSX-ALB: Aria Operations for Logs as a syslog over TLS appeared first on vconfig.pl.

The VM “Edge VM” with moref id “vm-xxxxx” corresponding to the Edge Transport node “Edge Transport node id” vSphere placement parameters is found in NSX inventory but is not present in vCenter. Please check if the VM has been removed in vCenter or is present with a different VM moref id.

The cause of that error is configuration sync issue between NSX-T and vCenter server.

NSX-T alarm:

Edge Alarms are displayed in System – Nodes – Edge Transport Nodes page.

The ‘Recommended Action’ in the alarm details suggests that an configuration update (addOrUpdatePlacementReferences) of the Edge node should be performed.

However, before proceeding with such actions, attempt to resolve the issue by refreshing the node configuration (refresh_node_configuration) using the POST API command. Be sure to add the NSX admin user account for authorization and set the content type to ‘application/xml’:

POST https://<NSX_FQDN>/api/v1/transport-nodes/<edge-ID>?action=refresh_node_configuration&resource_type=EdgeNode&read_only=true

You can find <edge-ID> in System – Nodes – Edge Transport Nodes and click on the affected edge node:

Once successfully completed you should get https status 200. The alert should dissapeared immediately and no other action is required.

The post VCF on VxRail: Edge VM Present in NSX Inventory Not Present in vCenter appeared first on vconfig.pl.

Due to an invalid index file that was recently pushed to the VMware depot, the LCM service on SDDC Manager has downloaded several incorrect upgrade bundles. This is the reason why the LCM service crashes and is unable to recover.

It’s recommended to run this script (with steps described in the KB article) even if everything appears to be fine. This issue affects all SDDC Managers connected to the VMware depot. It only takes 5 minutes to complete.

Please follow the steps described in that KB. Below you can find a screenshot with the result of running the script.

python offline_bundle_cleanup.py delta_file

The post VCF 5.x – LCM service crashing on SDDC Manager appeared first on vconfig.pl.

‘VxManager HTTP Request failed with invalid status code’

If you open the Physical View of VxRail cluster in vCenter, page keeps loading or it shows ‘Unable to show physical view due to a failure of loading physical view data for the host’.

The reason was an expired vxm service account password – the SSO service account used for communication between VxRail Manager and vCenter Server. To validate it login to VxRail Manager as a mystic user and run:

curl --unix-socket /var/lib/vxrail/nginx/socket/nginx.sock -X GET "http://localhost/rest/vxm/internal/lockbox/v1/credentials?lockbox_name=SYSTEM&credential_names=management_account_vc"|jq

If ‘Connection_type’ is DISCONNECTED (screenshot below) that would indicate the password expired or the password was changed manually in vCenter and not updated on VxRail Manager.

Run below command to update/reset the password for service account used for communication between VxRail Manager and vCenter Server (use your username and password of course;) ). Once done please restart VxRail Manager vm.

curl -X PUT --unix-socket /var/lib/vxrail/nginx/socket/nginx.sock -H "accept: application/json" -H "Content-Type: application/json" -d '{"lockbox_name":"SYSTEM","credentials":[{"credential_name":"management_account_vc","username":"svc-xxx-xxx-xxx-vc@vsphere.local","password":"VnXXXXXXXXXIQ=="}]}' 'http://localhost/rest/vxm/internal/lockbox/v1/credentials';

The post VCF on VxRail 4.x – VXM-VC service account password reset appeared first on vconfig.pl.

^{2023-10-20T16:23:20.023+0000 [bringup,11c0e70c474a8970,5431] DEBUG [c.v.e.s.c.v.util.ResponseUtil,pool-2-thread-19] Build validation response: {“errorCode”:”VMNICS_MIN_SPEED.error”,”arguments”:[“vmnic0″,”reg02-esx02.xxxx.xx”,”1000″],”context”:{“severity”:”ERROR”,”bundleName”:”com.vmware.evo.sddc.common.validation.errors.messages”,”validation.taskId”:”7f000001-8b4d-18c5-818b-4de5a780010f”},”message”:”The speed of vmnic0 on host reg02-esx02.xxxx.xxx is 1000 MB/s but must be at least 10 GB/s according to minimum hardware requirements“}}

Convert your VCF Deployment Parameter Workbook from .xlsx to .json, transfer it to the Cloud Builder VM (in my case /tmp/vcf-5.0.0-lab.json), then run (replace admin user password with the correct one):

curl -k -u admin:'Pa$$w0Rd' -X POST https://localhost/v1/sddcs -H "Content-Type: application/json" -d "@/tmp/vcf-5.0.0-lab.json"

The post VCF 5.0 – how to disable the VMNICS_MIN_SPEED validation during bring-up process appeared first on vconfig.pl.

Backing to VCF bringup, my folder structure is as follow:

startVcfBringup.yml
/roles/
  /runVcfBringup/
    /defaults/
      main.yml
    /tasks/
      main.yml
      validateVcfBringup.yml
      startVcfBringup.yml

startVcfBringup.yml playbook

# Playbook performs validation and deployment of SDDC via CloudBuilder API
---
- name: Start VCF bringup
  hosts: localhost
  gather_facts: false
  vars_files:
    - group_vars/myVars
  tasks:
    - name: Import role runVcfBringup
      include_role:
        name: runVcfBringup

/roles/runVcfBringup/defaults/main.yml

• cloudBuilderRestApiSddcValidate – var for Cloud Builder API used for input specification validation.

• cloudBuilderRestApiSddcCreate: var for Cloud Builder API used triggering the task using the valid input specification.

• bringupJsonPath – var for location on my ansible host where Deployment Parameter Workbook file converted to json is located (input specification).

---
# defaults file for runVcfBringup

cloudBuilderRestApiSddcValidate: "https://{{ CloudBuilderFQDN }}/v1/sddcs/validations"
cloudBuilderRestApiSddcCreate: "https://{{ CloudBuilderFQDN }}/v1/sddcs"
bringupJsonPath: "/home/admin/vcfBringup.json"
cbAdminUser: "admin"
cbAdminUserPassword: "P@ssw0rd"

/roles/runVcfBringup/tasks/main.yml

---

- name: Validate JSON input for VCF bring up
  import_tasks: validateVcfBringup.yml

- name: Create VCF bringup basing on JSON input
  import_tasks: startVcfBringup.yml

/roles/runVcfBringup/tasks/validateVcfBringup.yml

---
#validateVcfBringup.yml

- name: Trigger VCF parameter Workbook JSON input validation
  uri:
    url: "{{ cloudBuilderRestApiSddcValidate }}"
    method: POST
    user: "{{ cbAdminUser }}"
    password: "{{ cbAdminUserPassword }}"
    validate_certs: no
    force_basic_auth: yes
    return_content: yes
    body_format: json
    body: "{{ lookup('file','{{ bringupJsonPath }}') }}"
    status_code: 200,202
  register: vcfBringupValidation

- name: Register validation task ID
  set_fact:
    validationId: "{{ vcfBringupValidation.json.id }}"

- name: Get validation status
  uri:
    url: "{{ cloudBuilderRestApiSddcValidate }}/{{ validationId }}"
    method: GET
    user: "{{ cbAdminUser }}"
    password: "{{ cbAdminUserPassword }}"
    validate_certs: no
    force_basic_auth: yes
    return_content: yes
    body_format: json
  register: vcfBringupValidationQuery
  until: (vcfBringupValidationQuery.json.resultStatus | lower == 'succeeded') or (vcfBringupValidationQuery.json.resultStatus | lower == 'failed')
  delay: 30
  retries: 200

- name: Show validation report
  debug: 
    var: vcfBringupValidationQuery
    verbosity: 1

- name: Error message if validation failed
  fail:
    msg: 
      - "Validation failed. Failed steps have to be fixed before continue."
  when: vcfBringupValidationQuery.json.resultStatus | lower == 'failed'

- name: No error message if validation is successful
  debug:
    msg: "Validation successful. Proceeding with VCF bring up."
  when: vcfBringupValidationQuery.json.resultStatus | lower == 'succeeded'

/roles/runVcfBringup/tasks/startVcfBringup.yml

---
#startVcfBringup.yml

- name: Start VCF bring-up process
  uri:
    url: "{{ cloudBuilderRestApiSddcCreate }}"
    method: POST
    user: "{{ cbAdminUser }}"
    password: "{{ cbAdminUserPassword }}"
    validate_certs: no
    force_basic_auth: yes
    return_content: yes
    body_format: json
    body: "{{ lookup('file','{{ bringupJsonPath }}') }}"
    status_code: 200,202
  register: vcfBringupCreation

- name: Register creation task ID
  set_fact:
    taskId: "{{ vcfBringupCreation.json.id }}"

- name: Get creation task status
  uri:
    url: "{{ cloudBuilderRestApiSddcCreate }}/{{ taskId }}"
    method: GET
    user: "{{ cbAdminUser }}"
    password: "{{ cbAdminUserPassword }}"
    validate_certs: no
    force_basic_auth: yes
    return_content: yes
    body_format: json
  register: vcfBringupCreationTask
  until: (vcfBringupCreationTask.json.status | lower == 'completed_with_success') or (vcfBringupCreationTask.json.status | lower == 'completed_with_failure')
  delay: 60
  retries: 200

- name: Show creation report
  debug: 
    var: vcfBringupCreationTask
    verbosity: 1

- name: Error message if creation failed
  fail:
    msg: 
      - "VCF bring up failed. Failed steps have to be fixed before continue. Task have to be retried. Task ID: {{ taskId }}"
  when: vcfBringupCreationTask.json.status | lower == 'completed_with_failure'

- name: No error message if creation is successful
  debug:
    msg: "VCF bring up performed successfully."
  when: vcfBringupCreationTask.json.status | lower == 'completed_with_success'

The post VCF 5.0 – VCF bring-up with Ansible appeared first on vconfig.pl.

1. Update the Deployment Parameter Workbook and convert it to json file i(f not yet done);
2. SSH to Cloud builder vm and search for last failure execution ID of the task in /var/log/vmware/vcf/bringup/vcf-bringup-debug.log. In my case it’s a ‘9d2a7232-a0a2-4a86-8093-8142183a7f92’ (use ID from the last entry on the list):

cat vcf-bringup-debug.log | grep 'End of'

3. Send updated Deployment Parameter Workbook (json file) back to the API and restart the task in CB UI:

curl -k -u admin -X PATCH https://localhost/v1/sddcs/9d2a7232-a0a2-4a86-8093-8142183a7f92 -H "Content-Type: application/json" -d "@/home/admin/<json_file_name.json>

The post VCF 5.0 – how to update the SDDC bring-up parameters appeared first on vconfig.pl.

NOTE: For security reasons It shouldn’t be unhidden and if you need it on your prod environment,please contact GSS

In VCF 4.x this functionality it’s enabled and in one of my previous article I described How to update vSAN license key in the failed ‘Add Cluster’ workflow.

Backing to VCF 5.x- if you run curl command:

curl -s http://localhost/domainmanager/internal/vault/<failed_task_ID>

you will get an error message: “Something went wrong. Please retry or contact the service provider and provide the reference token.“

To unhidden the endpoint, add ‘vcf.vault.http-access=true‘ to domainmanager config file:

1. ssh to SDDC Manager as a vcf user and su to root;
2. open etc/vmware/vcf/domainmanager/application-prod.properties;
3. add the following line: vcf.vault.http-access=true
4. save and exit the file;
5. restart domainmanager service:  systemctl restart domainmanager
6. wait 2-3 minutes and try to run curl command to get the input values from failed task’, i.e:

curl -s http://localhost/domainmanager/internal/vault/97b108a3-463d-4a10-a8de-5d2b778b3aab -XGET | json_pp > /tmp/testwf.json

The post VCF 5.x – how to modify the input values of failed task in SDDC Manager appeared first on vconfig.pl.

So what happens at the lower level when we create vmdk.

1. vCenter (vpxd) translates tasks to action vpxa agent (on the hosts)

2. vpxa communicates with the ESXi host agent (hostd) translating task to action internally.

QUICK REMINDER: vpxa acts as an intermediary between the vpxd and the hostd.

3. The host invokes required libraries/deamons such as DISKLIB/OSFSd to create vSAN objects (vmdk, snapshot,namespace,vswap…)

4. CLOM (Cluster Level Object Manager Daemon) takes care of the vSAN objects – it decides on the components and witnesses that need to be created and checks if there are enough available resources ( disk groups, free space etc.) to satisfy the policy.
This daemon also talks to the CLOM on other hosts to see what space is available.
After the objects are created CLOM is further responsible for monitoring the objects compliance status.

5. CLOM does all that thinks by communicating to the other nodes through the CMMDS (Cluster Monitoring, Membership and Directory Services).
CMMDS is a directory and content of it is used by other vSAN component technologies (DOM, LSOM) to determine the nodes storing the components of an object and the paths by which those nodes are reachable.
CMMDS also discovers and maintains the vSAN cluster, stores information such as node setup, policies, detecting failures in nodes and network paths.
Every host runs CMMDS and one of the vSAN cluster nodes is elected to be a Master. Another one is a Backup node – that holds a copy of the master data – will take over if the Master fails. All other nodes are Agents.
These roles are always system defined and cannot be changed by user/admin.

And here RDT (Reliable Datagram Transport) comes into play as well – the protocol used by vSAN for communication between nodes (cmmds sync, I/O flow). RDT is able to set up and tear down transport connections very quickly, depending on link healthstatus changes that are published by the CMMDS, in order to minimize any delay to the datagram transport as a result of link failures.

Below screenshots from vSAN 4-node cluster (1x master, 1x backup and 2x agents):

6. If everything looks good then CLOM instructs DOM (Distributed Object Manager) to talk with LSOM (step 7) to create and distribute the objects across the cluster .
DOM is responsible for initial I/O requests and is split up into three processes- Client, Owner, Component Manager.

– DOM Client runs on every node that contains components (runs on the same host as the VM) and performs the I/O to an object – all IO flows from the vSAN client to the owner.
– DOM Owner manages access to the object – determines which processes are allowed to send I/O to the object. Each object in a vSAN cluster has a DOM Owner and DOM Client. Once an object is created one of the vSAN cluster nodes is nominated as the DOM Owner for that object and this host is responsible to handle all IOPS to that object by locating the child components across cluster and redirecting the IO to respective components over vSAN network. The DOM owner may be a host that does not own any component of that object.
DOM Owner distributes data over different hosts through DOM Component Manager which is the local component of the DOM that connects with LSOM on each host.
– DOM Component Manager is responsible for managing the objects on hosts where components exists.

7 – 8. The LSOM ( Local Log-Structured Object Manager) is responsible only for handling the I/O (talking to PSA– Pluggable Storage Architecture which processes the system traffic). It receives I/O from the DOM and ACK when write operations have been completed. The LSOM is responsible as well for cache reading and writing (returns payloads for read operations and handles write buffering) and de-staging of data to capacity tier disks.

And that’s it. Any comments, questions – just ping me

The post vSAN – internal components and mechanisms appeared first on vconfig.pl.