The post VMware TAM Services – Data Services Manager 9.0.2 Update (Video) appeared first on CormacHogan.com.

Prepping DSM for Aria Automation integration

Let’s say, for example, I want my user dragomir to deploy certain versions of Postgres databases to the engineering-ns namespace, my user christos deploy certain versions of Postgres databases in the sales-ns namespace and my user paudie to deploy certain versions of Postgres databases to the marketing-ns. I want to achieve this via requests to provision a DBaaS catalog item in my Aria Automation Service Broker. The steps to implement this in DSM by a DSM administrator would be:

1. Assuming that these are Active Directory users, enable Directory Services in DSM & import the different AD groups (engineering, marketing, sales) containing the users into DSM as Directory Service Groups under Permissions. Alternatively, you could use local users in DSM if you wish.
2. Create different namespaces for the different groups of users (engineering, marketing, sales) and, if using Active Directory, add the appropriate Directory Service Group to the namespace.
3. Create Data Service Policies which associates the different namespaces with the permissions to provision data services. Choose the data service, which versions of service the users/groups are allowed to provision, select the appropriate infrastructure policies, select appropriate backup locations, etc.

For the purposes of this post, I have the following AD users configured in DSM:

Checking Active Directory – Users and Computers for dragomir, I can see here is in the correct AD group, DSMUsers-Engineering:

By checking the DSM Portal as a DSM Admin, I can also verify that the DSMUsers-Engineering AD Group has been added to the engineering-ns namespace in DSM:

Lastly, we can check the Data Service Policy that engineering-ns has been added to. This makes the relationship between the users/namespace and the data services and versions that they are allowed to provision. Here is a snippet of the DSP, showing that we are granting the members of the engineering-ns the ability to provision versions 16 & 17 of Postgres.

The relationship between Data Service Policy and Namespace can be shown as the following:

At this point the DSM side of things has been setup. Later on, we will see how to add these users to Aria Automation,  give them Service Broker user roles, and assign them as members of a Projects. These steps will allow them to provision databases to their own DSM namespaces through Aria Automation.

Aria Automation config.yaml changes

The config.yaml, which contains the configuration setting to integrate DSM with Aria Automation, now contains additional fields to map DSM Namespaces to Aria Automation Projects in version 9.0.2. For my particular setup, I have 3 users in different DSM namespaces that I want to be able to provision databases from Aria Automation. So I will have to create 3 different Projects in Aria Automation to achieve this:

Here is an example of the config.yaml which has the new project_to_namespace entries defined. Note that the DSM plugin for Aria Automation will only create the SVC_DSM_PROJECT referenced in the config.yaml manifest. It does not create the additional projects listed in the project_to_namespace in Aria Automation. These will need to be done manually by an Aria Automation admin after the plugin installer has been run.

---
dsm_hostname: 192.168.0.1
dsm_user_id: 'dsmadmin@ams.local'
dsm_password: 'VMware123!'
aria_base_url: https://vra.rainpole.com
orchestrator_base_url: https://vra.rainpole.com
aria_username: 'cormac'
aria_password: 'VMware123!'
org_id: 08969e76-5e6c-4581-9cf6-19c06384a9b2
blue_print_name: SVC DSM DBaaS
abx_action_name: SVC-DSM-DB-crud
cr_name: SVC_DSMDB
cr_type_name: Custom.DSMDB
env_name: SVC_DSM_ENV
project_name: SVC_DSM_PROJECT
default_namespace: marketing-ns
project_to_namespace:
  SVC_DSM_PROJECT: default
  SALES_DSM_PROJECT: sales-ns
  ENG_DSM_PROJECT: engineering-ns
  MK_DSM_PROJECT: marketing-ns
skip_certificate_check: 'False'
dsm_root_ca: |
-----BEGIN CERTIFICATE-----
MIIDcDCCAligAwIBAgIGAZrZgFtWMA0GCSqGSIb3DQEBCwUAMFkxCzAJBgNVBAYT
AlVTMQwwCgYDVQQKDANEU00xFTATBgNVBAsMDERTTSBQcm92aWRlcjElMCMGCSqG
SIb3DQEJARYWZHNtX3N1cHBvcnRAdm13YXJlLmNvbTAgFw0yNTExMjgxMDQwNTla
<--snip-->
GO10PoLXP5MZObWGarfkC/+IgY8=
-----END CERTIFICATE-----

After the deployment (running the command “python3 aria.py enable-blueprint-version-2“), the Aria Automation administrator will have to do the following:

1. Add users to Aria Automation, noting which namespace they have access to in DSM.
2. Assign these users the service role of Service Broker User
3. Create projects to match the project_to_namespace
4. Assign these users as member of their appropriate projects
5. Create new content sharing policies which grants the projects access to the DSM Content Source

1. Add users to Aria Automation with Service Broker User Role

In Aria Automation Identity and Access Management, assign the users the role of Service Broker User. This way, they will be able to trigger requests from the Service Broker DBaaS catalog item to provision databases from their own Project. If they have the role of Service Broker Admin, they would be able to see all projects, and would be able to request databases on namespaces which we do not want.  A Service Broker User role means that they only have access to their own project.

Note that there is no automatic correlation here between the DSM users and the Aria Automation users, other than the fact that, in this post, they are the same user in Active Directory (and both DSM and Aria automation support integration with AD). It is up to the Aria and DSM administrators to coordinate the correlation of the user in DSM and the user in Aria Automation to ensure that these users have access to the correct DSM resources.

2. Create Projects as per project_to_namespace in config.yaml

With the users configured, we can now login into Aria Automation as admin user, navigate to the Assembler and build the necessary projects that map to the config.yaml entries in project_to_namespace. With each project, assign one of the Service Broker users to the appropriate project:

It should look something like this. Of course, this is a simple example. If you have multiple users in a DSM namespace, these additional users would be added to the same project. To keep things simple, I am using a single user for each.

3. Create Content Sharing Policy Definitions

The final step for the administrator is to create new content sharing policies. These grant the projects access to the DSM Content Source so that they can ask DSM to provision databases. Navigate to the Service Broker, and under the Content & Policies tab, select Definitions (under Policies). In this example, I have created 3 policies, one per project. It should also be possible to create a single share and assign it to all projects.

And that completes the setup in Aria Automation. We can now go ahead and check if it possible to deploy a Postgres database as user dragomir from Aria Automation, and verify that it lands in the engineering-ns namespace in DSM.

Deploy a database via the Service Broker catalog

Begin by logging is to Aria Automation as the user ‘dragomir‘. As per the assigned role of Service Broker user, dragomir can only see the Service Broker service.

Clicking on the Service Broker, and then navigating to the Catalog, dragomir should only see a single item in the catalog, and that item should show that it is associated with only a single project, the ENG_DSM_PROJECT. So far, this looks like it is achieving our goal and ‘guard-railing’ the user dragomir into using the correct vSphere resources for data services, as defined by the Data Service Policy that is associated with the namespace where user dragomir resides on DSM.

Let’s validate this by making a request to provision a database. The only project available for selection in ENG_DSM_PROJECT, which is correct. Also, the list of available database versions matches those in the Data Service Policy to which the engineering-ns has been assigned in DSM. So far, so good. Continue with the deployment of the database.

Assuming that the deployment of the database is successful, dragomir can drill into the details of the database, and check which namespace was used for this deployment. And it does indeed look like that the deployment has been placed in the correct engineering-ns namespace as defined in the project_to_namespace directive in the config.yaml.

We can verify the same from the DSM Portal. Login as user dragomir, who we have seen has been added to the engineering-ns and check if the database is present. Since dragomir is a DSM user, a result of adding the DSMUsers-Engineering Directory Service Group in the Permissions section of DSM, he can access the UI.

Success. We have achieved our goal of integrating Aria Automation with the new RBAC features of DSM, and the relevant Data Service Policy controlling which users can provision which data services into which namespaces on DSM.

The post DSM 9.0.2 – Aria Automation plugin changes to support new RBAC features appeared first on CormacHogan.com.

The story so far …

By way of a recap, here is what we have delivered so far for the beta versions of MS SQL Server DSM data service:

• Click here for an overview of the original MS SQL Server tech preview release back in DSM 9.0.0.
• Click here and here for an overview of the MS SQL Server enhancements made in DSM 9.0.1, which include automated Active Directory integration and automatic DNS updates for MS SQL Server instances. This release also introduced an overview of the new RBAC mechanism based on Data Services Policies. These policies control user access to data services and resources.

Deploy MS SQL Server instance and user database

Let’s now take a look at the new SQL Server Agent support. I have already configured an Active Directory Domain for DSM. This allows me to create a privileged AD user account which can write SPNs (servicePrincipalNames). This in turn allows DSM to enable Windows Authentication for MS SQL Server database users. See the DSM 9.0.1 blog post mentioned previously for further details on this cool feature.

Next, we create an MS SQL Server instance or engine. I won’t repeat the steps here as they are very similar to how it is done in DSM version 9.0.1. However, there is a new section to enable SQL Server Agent. It also includes some configuration parameter around Job History when enabled. Later, we will create a job to see this in action.

Wait for the SQL Server instance to come online, verifying that Active Directory integration and automatic DNS updates are working correctly. This creates 4 System Databases on the MS SQL Server instance; master, model, msdb and tempdb. These are only accessible with administrator privileges. The details are available to the DSM admin when they view the summary page of the MS SQL Server instance, including a full connection string.

With the instance successfully deployed, we can create a user database on the instance. I am going to include a Windows Principal so that the user can use Windows Authentication to access the database. the format is ‘domain-name\user-name’.

After a few moment, the database will be online.

Now we can begin to look at the SQL Server Agent configuration and how we can use it to run some jobs.

Verify SQL Server Agent functionality

As part of this post, I am going to use some of the very popular scripts created by Ola Hallengren. The scripts are available here: https://ola.hallengren.com/downloads.html. In particular, I will look at the stored procedures that does database integrity checks. This is a stored procedure which we will install on the user database. First, we must installed another required stored procedure called CommandExecute.sql. We also need to run a script called CommandLog.sql to create a required table within the user database. Now we can run the DatabaseIntegrityCheck.sql stored procedure. Once this is completed, we can run the database integrity check.

After confirming that the stored procedure is working, we can then build a SQL Server Agent job to run the database integrity check, and modify the job to include a daily schedule to run it on a regular basis.

I am first going to login to the SQL Server as an administrator. The Admin login and password can be retrieved from the DSM UI, in the Summary view of the database instance.

Use this to login to the MS SQL Server Management Studio (SSMS) using SQL Server Authentication:

Let’s first check that the SQL Server Agent is running by opening a new query and running the following:

SELECT status_desc FROM sys.dm_server_services WHERE servicename LIKE 'SQL Server Agent%'; 

It appears to be running as expected. We can also check that the configuration values provided via the DSM UI for the SQL Server Agent have been implemented via the following command:

EXEC msdb.dbo.sp_get_sqlagent_properties; 

The max and min values of 1500 and 150 match the values that I provided during the deployment. So far, so good. Let’s now add the stored procedures on the user database for the integrity check.

Create Stored Procedure to do Database Integrity Checks

To add a stored procedure, select the user database (cormacdb01), and under Programmability, click on Stored Procedure > New > Stored Procedure, as shown below.

This will create a stored procedure template:

Replace the content of the stored procedure template with the Ola Hallengren scripts mentioned earlier, and execute them one at a time on the user database. Start with the CommandExecute script to build the CommandExecute procedure.

Next, open execute the CommandLog script to create the CommandLog table.

And finally execute the DatabaseIntegrityCheck script to create the Database Integrity Check procedure.

You can now run a command to verify that the database integrity check procedure is working. You can do this as the database owner, rather than the database admin. Logout from SSMS as the admin. Re-connect to the SSMS using the Windows Credentials of the owner, in this case ‘rainpole\cormac’. This is possible since we have given DSM the ability to automatically enable these logins through a privileged user account (see here) that we referenced when we created the database instance.

This user does not have access to any of the System database. The user only has access to its own user database. Since the SMSS connection always defaults to the ‘master’ database, we will need to change the database for this user to the user database (cormacdb01).

Once connected to the database, execute the following query on the user database, which is essentially running the procedure that we added earlier:

EXECUTE [dbo].[DatabaseIntegrityCheck] @Databases = 'USER_DATABASES', @LogToTable = 'Y'; 

You should observe the results of the integrity database check appear, reporting any issues with the database integrity:

Create a SQL Server Agent Job to do Database Integrity Checks

We will now create a SQL Server Agent job to run the database integrity checks. We can then add a schedule to it so that it checks the database integrity on a daily basis. Log out of the MS SQL Server Management Studio as the current Windows user and back in as the database admin using SQL Auth. Then run the following query to create the job. In the second EXEC, change the database_name to the name of the user database.

In a nutshell, we are creating a job called indexOptimize, and adding a step to the job to run the database integrity check on the user database. As this is run on the msdb, a system database, it can only be executed by the database admin.

USE msdb; 
    EXEC msdb.dbo.sp_add_job @job_name = indexOptimize; 
    EXEC msdb.dbo.sp_add_jobstep @job_name = IndexOptimize, @step_name = IndexOptimize, @database_name = <YOUR_DB_NAME>, @subsystem = N'TSQL', @command = 'EXECUTE [dbo].[DatabaseIntegrityCheck] @Databases = ''USER_DATABASES'', @LogToTable = ''Y'';'; 
    EXEC msdb.dbo.sp_add_jobserver @job_name = IndexOptimize, @server_name = N'(local)';

After running the above command, we should now see the job appear in the SQL Server Agent inventory view on the left hand view of the MS SQL Server Management Studio, as shown below.

Click on the job name (indexOptimize in this example) to see the job properties.

If you select the Steps page on the left hand side of the job properties, you can see a T-SQL type job step.

Click on the Edit underneath the Start Step in the job properties to verify that it is indeed a request to run the integrity check:

The job appears to have been successfully created.

Schedule a SQL Server Agent Job

Let’s now schedule this job to run regularly and report on the database integrity. Cancel the previous ‘Edit’ window to return back to the Job Properties. One of the pages listed in the left hand side is ‘Schedules’. Click on this to create a schedule for the job using the ‘New…’ button. In this case, I created a daily schedule for the integrity check.

With a schedule in place, we can check the job history and its runs using below command. You obviously won’t see any job runs reported until the job has run at least once.

EXEC msdb.dbo.sp_help_jobhistory @mode = 'FULL';

This is a sample of the output that the above command generates, showing multiple scheduled runs of the job:

Summary

And that completes the post demonstrating how SQL Server Agent support is now included in DSM 9.0.2 provisioned MS SQL Server instances. We continue to add features for MS SQL Server in each release, and we hope to make it generally available very soon indeed. If there are any questions around our Data Services Manager offering, or MS SQL Server integration, please reach out or leave a comment on the post.

The post DSM 9.0.2 – New Microsoft SQL Server Enhancements – SQL Server Agent Support appeared first on CormacHogan.com.

Deploy the Velero Client/CLI

Step 1 in the process is to deploy the Velero Client. This involves downloading a zipped up Velero binary to your desktop, and then placing it somewhere in your execution path (full instructions here). I downloaded it to an Ubuntu VM that I have in my environment. Once the Velero Client/CLI has been installed, you can check what the version is by using the command ‘velero version’. Ignore the server error. This is simply because we have not yet installed the server part of Velero onto the VKS cluster. However, there client version is 1.17.0. We should match this with the server version on the VKS cluster.

$ velero version 
Client:
         Version: v1.17.0_vmware.1
         Git commit: 3172d9f99c3d501aad9ddfac8176d783f7692dce-modified
 <error getting server version: unable to retrieve the complete list of server APIs: velero.io/v1: no matches for velero.io/v1, Resource=>

Deploy the Velero Server onto VKS

To being with, ensure that your KUBECONFIG is set up correctly, and that you are pointing to the correct VKS cluster, i.e., the one where you wish to install the Velero server. I usually check by running a kubectl get nodes to verify that I have the correct cluster context.

$ kubectl get nodes 
NAME                                                              STATUS   ROLES           AGE   VERSION 
kubernetes-cluster-dkpp-hkpfp-r6j66                               Ready    control-plane   41m   v1.33.3+vmware.1-fips 
kubernetes-cluster-dkpp-kubernetes-cluster-dkpp-np-pk3w-smkvs46   Ready    <none>          36m   v1.33.3+vmware.1-fips

Create a velero-data-values.yaml file

A yaml file is required to describe the details of the backup storage location (bsl). In this example, I am once again using MinIO to provide me with an S3 compatible bucket but any S3 compatible bucket will suffice. I have create a bucket called “velero-backups”. Since I have TLS enabled, I am using an https URL. This means I need to provider a Certificate Authority in the manifest, as shown below. This provides trust between Velero and the object store provider. I also need to provide credentials to access the object store.

$ cat velero-data-values.yaml
backupStorageLocation:
  bucket: velero-backups
  config:
    region: "minio"
    s3ForcePathStyle: "true"
    s3Url: "https://minio.rainpole.io:9000"
  caCert: |
    -----BEGIN CERTIFICATE-----
    MIIGFzCCA/+gAwIBAgICEBkwDQYJKoZIhvcNAQELBQAwgY0xEDAOBgNVBAMMB01J
    Tk9fQ0ExCzAJBgNVBAYTAklFMQ0wCwYDVQQIDARDb3JrMQ0wCwYDVQQHDARDb3Jr
    MRUwEwYDVQQKDAxWQ0YgRGl2aXNpb24xDTALBgNVBAsMBE9DVE8xKDAmBgkqhkiG
---> snip
    G+aB3AHlbfw45mkMlsk9jXl1sj21UYw/ZHJN
    -----END CERTIFICATE-----
  credential: |
    [default]
    aws_access_key_id=admin
    aws_secret_access_key=password

Use the VCF CLI to add a package repo to the cluster

Our next step is to use the VCF CLI (my binary is called vcf-cli) to create a package repo. The VCF CLI can be downloaded from any Supervisor Namespace Summary page. The repo, once created, will contain our Velero package which we can install once downloaded. The packages are retrieved from the VKS Standard Packages Repository. I mentioned the VCF CLI a few times recently, such as using it to troubleshoot DSM deployed on VKS clusters. It is a very useful tool for VCF administrators as it allow command line interaction with the Sueprvisor and VCF Automation. Before we create the repo, let’s first ensure that we are pulling down the latest packages. This URL for the VKS Release Notes will tell you which path to use for the packages. At the time of writing, VKS 3.5 was the latest release, so the RN reports that the VKS Standard Packages v3.5.0+20251218 repository is available here:

projects.packages.broadcom.com/vsphere/supervisor/vks-standard-packages/3.5.0-20251218/vks-standard-packages:3.5.0-20251218

Always check the Release Notes for the latest path information. We will use this path when we create the repo. Note that I am still in the KUBECONFIG context. Let’s add the repository. I am placing it in the namespace tkg-system.

$  vcf-cli package repository add standard-package-repo \
--url projects.packages.broadcom.com/vsphere/supervisor/packages/2025.10.22/vks-standard-packages:3.5.0-20251022 \
-n tkg-system
1:26:48PM: Updating package repository resource 'standard-package-repo' in namespace 'tkg-system'
1:26:48PM: Waiting for package repository reconciliation for 'standard-package-repo'
1:26:50PM: Fetching
            | apiVersion: vendir.k14s.io/v1alpha1
            | directories:
            | - contents:
            |   - imgpkgBundle:
            |       image: projects.packages.broadcom.com/vsphere/supervisor/packages/2025.10.22/vks-standard-packages@sha256:36b48ba005e884586512c2fda8c4598f426c6f78efa7f84f5b24087b49a6b52d
            |       tag: 3.5.0-20251022
            |     path: .
            |   path: "0"
            | kind: LockConfig
            |
1:26:50PM: Fetch succeeded
1:26:52PM: Template succeeded
1:26:52PM: Deploy started (3s ago)
1:26:56PM: Deploying
            | Target cluster 'https://10.96.0.1:443'
            | Changes
            | Namespace   Name                                                                     Kind             Age  Op      Op st.                      Wait to  Rs  Ri
            | tkg-system  ako.kubernetes.vmware.com                                                PackageMetadata  -    create  fallback on update or noop  -        -   -
            | ^           ako.kubernetes.vmware.com.1.13.4+vmware.1-vks.1                          Package          -    create  fallback on update or noop  -        -   -
            | ^           autoscaler.kubernetes.vmware.com                                         PackageMetadata  9m   delete  -                           -        ok  -
            | ^           cert-manager.kubernetes.vmware.com.1.17.2+vmware.1-vks.1                 Package          9m   delete  -                           -        ok  -
            | ^           cert-manager.kubernetes.vmware.com.1.17.2+vmware.2-vks.1                 Package          -    create  fallback on update or noop  -        -   -
            | ^           cert-manager.kubernetes.vmware.com.1.18.2+vmware.1-vks.1                 Package          9m   delete  -                           -        ok  -
            | ^           cert-manager.kubernetes.vmware.com.1.18.2+vmware.2-vks.2                 Package          -    create  fallback on update or noop  -        -   -
.
.
.

Let’s check if the Velero package is now included in the repository, and if so, what versions are available.

$ vcf-cli package available list -n tkg-system | grep velero
  velero.kubernetes.vmware.com                    velero


$ vcf-cli package available get velero.kubernetes.vmware.com
  NAME:                   velero.kubernetes.vmware.com
  DISPLAY-NAME:           velero
  CATEGORIES:             - data protection
  SHORT-DESCRIPTION:      Velero is an open source tool to safely backup and restore, perform disaster
recovery, and migrate Kubernetes cluster resources and persistent volumes.
  LONG-DESCRIPTION:       Velero is an open source tool to safely backup and restore, perform disaster
recovery, and migrate Kubernetes cluster resources and persistent volumes.
  PROVIDER:               VMware
  MAINTAINERS:            - name: Wenkai Yin
  SUPPORT-DESCRIPTION:    https://github.com/vmware-tanzu/velero
  VERSION                RELEASED-AT
  1.16.1+vmware.1-vks.1  2025-05-19 12:30:00 +0000 UTC
  1.16.2+vmware.1-vks.1  2025-08-05 12:30:00 +0000 UTC
  1.17.0+vmware.1-vks.1  2025-09-16 11:30:00 +0000 UTC

Looks good. Velero version 1.17.0 is an available package version, which will match the client version we are using.

Install the Velero Server Package

Let’s proceed with the install of the Velero server package. Specify the namespace of the repo where the package is stored, the name of the package itself, the version you wish to install and the path to the velero-data-values.yaml created earlier. The actual Velero server components will be installed from the package into a namespace called velero.

$ vcf-cli package install velero --namespace tkg-system \
--package velero.kubernetes.vmware.com \
--version 1.17.0+vmware.1-vks.1 \
--values-file velero-data-values.yaml

1:30:07PM: Pausing reconciliation for package installation 'velero' in namespace 'tkg-system'
1:30:08PM: Updating secret 'velero-tkg-system-values'
1:30:08PM: Resuming reconciliation for package installation 'velero' in namespace 'tkg-system'
1:30:08PM: Waiting for PackageInstall reconciliation for 'velero'
1:30:08PM: Waiting for generation 3 to be observed
1:30:09PM: Fetch started
1:30:09PM: Fetching
            | apiVersion: vendir.k14s.io/v1alpha1
            | directories:
            | - contents:
            |   - imgpkgBundle:
            |       image: projects.packages.broadcom.com/vsphere/supervisor/packages/2025.10.22/vks-standard-packages@sha256:5dd00ce6284efa836cae4abb351ab8987cf118f79d355c84ce2ba0a5ac5fbd29
            |     path: .
            |   path: "0"
            | kind: LockConfig
            |
1:30:09PM: Fetch succeeded
1:30:09PM: Template succeeded
1:30:10PM: Deploy started
1:30:10PM: Deploying
            | Target cluster 'https://10.96.0.1:443' (nodes: kubernetes-cluster-dkpp-hkpfp-r6j66, 1+)
            | Changes
            | Namespace  Name  Kind  Age  Op  Op st.  Wait to  Rs  Ri
            | Op:      0 create, 0 delete, 0 update, 0 noop, 0 exists
            | Wait to: 0 reconcile, 0 delete, 0 noop
            | Succeeded
1:30:10PM: Deploy succeeded

Check the Velero install status

We can now check if the package installed successfully. We can use kubectl to check on various objects in the velero namespace, such as deployments, replicaSets and Pods. We can check that the BackupStorageLocation (bsl) is available and by using the velero version command, we can verify that the server portion is now reporting correctly.

$ kubectl get deploy -n velero
NAME     READY   UP-TO-DATE   AVAILABLE   AGE
velero   1/1     1            1           4m38s


$ kubectl get rs -n velero
NAME                DESIRED   CURRENT   READY   AGE
velero-845bfc6654   1         1         1       4m59s


$ kubectl get pods -n velero
NAME                      READY   STATUS    RESTARTS   AGE
node-agent-f8qdp          1/1     Running   0          5m5s
velero-845bfc6654-69vqs   1/1     Running   0          5m5s


$ kubectl get bsl -n velero
NAME      PHASE       LAST VALIDATED   AGE     DEFAULT
default   Available   49s              5m22s   true


$ velero version
Client:
        Version: v1.17.0_vmware.1
        Git commit: 3172d9f99c3d501aad9ddfac8176d783f7692dce-modified
Server:
        Version: v1.17.0_vmware.1

Create a sample workload to backup and restore

To test out Velero backup and restore, I am going to use a simple Pod and PersistentVolumeClaim (pvc) combination. The PVC will have to use one of the existing StorageClasses (sc) in the VKS cluster. The Pod will also mount the volume onto the /demo folder in the busybox container.

As this is VKS on vSphere, the vSphere CSI driver is a necessary component for creating persistent volumes on vSphere storage. This will also mean that during the backup, Velero will request the creation of a CSI snapshot of the volume to backup the volume.  VKS clusters on vSphere have all of the necessary vSphere CSI components to achieve this.

Here are the manifests and steps to create the simple app. First we create a namespace called cormac-ns, which is where our app will live. Then we get the StorageClasses, and then build the appropriate PVC manifest using one of the storage classes. Once the PVC is created, we can reference it as a Volume in the Pod.

$ kubectl create ns cormac-ns
namespace/cormac-ns created 


$ kubectl get sc
NAME                                                              PROVISIONER              RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
sfo-w01-cl01-optimal-datastore-default-policy-raid1               csi.vsphere.vmware.com   Delete          Immediate              true                   25h
sfo-w01-cl01-optimal-datastore-default-policy-raid1-latebinding   csi.vsphere.vmware.com   Delete          WaitForFirstConsumer   true                   25h


$ cat example-pvc-cormac-ns.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: example-cormac-block-pvc
  namespace: cormac-ns
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: sfo-w01-cl01-optimal-datastore-default-policy-raid1


$ kubectl apply -f example-pvc-cormac-ns.yaml
persistentvolumeclaim/example-cormac-block-pvc created


$ kubectl get pvc -n cormac-ns -w
NAME                       STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                                          VOLUMEATTRIBUTESCLASS   AGE
example-cormac-block-pvc   Pending                                                                        sfo-w01-cl01-optimal-datastore-default-policy-raid1   <unset>                 4s
example-cormac-block-pvc   Pending   pvc-85cf01bc-52fb-4bc1-a74f-58990b2988bb   0                         sfo-w01-cl01-optimal-datastore-default-policy-raid1   <unset>                 4s
example-cormac-block-pvc   Bound     pvc-85cf01bc-52fb-4bc1-a74f-58990b2988bb   5Gi        RWO            sfo-w01-cl01-optimal-datastore-default-policy-raid1   <unset>                 5s


$ cat cormac-pod-with-volume.yaml
apiVersion: v1
kind: Pod
metadata:
  name: cormac-pod-1
  namespace: cormac-ns
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
    supplementalGroups: [4000]
  containers:
  - name: busybox
    image: "dockerhub.packages.vcfd.broadcom.net/busybox:latest"
    command: [ "sleep", "1000000" ]
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]
      runAsNonRoot: true
      runAsUser: 1000
      seccompProfile:
        type: "RuntimeDefault"
    volumeMounts:
      - mountPath: "/demo"
        name: demo-vol
  volumes:
    - name: demo-vol
      persistentVolumeClaim:
        claimName: example-cormac-block-pvc


$ kubectl apply -f cormac-pod-with-volume.yaml
pod/cormac-pod-1 created


$ kubectl get pods -n cormac-ns -w
NAME           READY   STATUS              RESTARTS   AGE
cormac-pod-1   0/1     ContainerCreating   0          8s
cormac-pod-1   1/1     Running             0          14s


$ kubectl get pod,pvc -n cormac-ns
NAME               READY   STATUS    RESTARTS   AGE
pod/cormac-pod-1   1/1     Running   0          33s

NAME                                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                                          VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/example-cormac-block-pvc   Bound    pvc-85cf01bc-52fb-4bc1-a74f-58990b2988bb   5Gi        RWO            sfo-w01-cl01-optimal-datastore-default-policy-raid1   <unset>                 2m15s

We now have an application. If you so wish, you can use kubectl exec to access the Pod, and create files and directories in the /demo directory. You can then check if the data is recovered after a backup and restore. Let’s now see if we can backup and restore this ‘critical’ application.

Backup using Velero CLI

I will now initiate a backup. I do not want to backup the full VKS cluster, only  the namespace cormac-ns as this is where my app is running. After the backup has completed, I can use the suggested ‘velero backup describe‘ command to look at it in more detail.

$ velero backup create cormac-backup-1 --include-namespaces cormac-ns --wait
Backup request "cormac-backup-1" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup 
will continue in the background.
......................................................................
Backup completed with status: Completed. You may check for more information using the 
commands `velero backup describe cormac-backup-1` and `velero backup logs cormac-backup-1`.


$ velero backup describe cormac-backup-1
Name:         cormac-backup-1
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.33.3+vmware.1-fips
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=33
Phase:  Completed
Namespaces:
  Included:  cormac-ns
  Excluded:  <none>
Resources:
  Included cluster-scoped:    <none>
  Excluded cluster-scoped:    volumesnapshotcontents.snapshot.storage.k8s.io
  Included namespace-scoped:  *
  Excluded namespace-scoped:  volumesnapshots.snapshot.storage.k8s.io
Label selector:  <none>
Or label selector:  <none>
Storage Location:  default
Velero-Native Snapshot PVs:    auto
File System Backup (Default):  false
Snapshot Move Data:            true
Data Mover:                    velero
TTL:  720h0m0s
CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s
Hooks:  <none>
Backup Format Version:  1.1.0
Started:    2026-01-14 13:42:45 +0000 UTC
Completed:  2026-01-14 13:43:55 +0000 UTC
Expiration:  2026-02-13 13:42:44 +0000 UTC
Total items to be backed up:  65
Items backed up:              65
Backup Item Operations:  1 of 1 completed successfully, 0 failed (specify --details for more information)
Backup Volumes:
  Velero-Native Snapshots: <none included>
  CSI Snapshots:
    cormac-ns/example-cormac-block-pvc:
      Data Movement: included, specify --details for more information
  Pod Volume Backups: <none included>
HooksAttempted:  0
HooksFailed:     0

Firstly, it looks like the backup has completed successfully. Secondly, it has indeed used CSI Snapshots, and moved the snapshot data to my backup storage location (S3 bucket). Note that this section of the output recommends using a –details options to see more information above the data movement. Let’s run that now (note that I have snipped some of the output for conciseness).

$ velero backup describe cormac-backup-1 --details
Name:         cormac-backup-1
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.33.3+vmware.1-fips
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=33
Phase:  Completed
Namespaces:
  Included:  cormac-ns
  Excluded:  <none>
Resources:
  Included cluster-scoped:    <none>
  Excluded cluster-scoped:    volumesnapshotcontents.snapshot.storage.k8s.io
  Included namespace-scoped:  *
  Excluded namespace-scoped:  volumesnapshots.snapshot.storage.k8s.io
Label selector:  <none>
Or label selector:  <none>
Storage Location:  default
Velero-Native Snapshot PVs:    auto
File System Backup (Default):  false
Snapshot Move Data:            true
Data Mover:                    velero
TTL:  720h0m0s
CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s
Hooks:  <none>
Backup Format Version:  1.1.0
Started:    2026-01-14 13:42:45 +0000 UTC
Completed:  2026-01-14 13:43:55 +0000 UTC
Expiration:  2026-02-13 13:42:44 +0000 UTC
Total items to be backed up:  65
Items backed up:              65
Backup Item Operations:
  Operation for persistentvolumeclaims cormac-ns/example-cormac-block-pvc:
    Backup Item Action Plugin:  velero.io/csi-pvc-backupper
    Operation ID:               du-f577e717-58bf-453b-8de8-c5f3b866855e.85cf01bc-52fb-4bcd93615
    Items to Update:
                           datauploads.velero.io velero/cormac-backup-1-mswhk
    Phase:                 Completed
    Progress description:  Completed
    Created:               2026-01-14 13:43:01 +0000 UTC
    Started:               2026-01-14 13:43:20 +0000 UTC
    Updated:               2026-01-14 13:43:49 +0000 UTC
Resource List:
  data.packaging.carvel.dev/v1alpha1/Package:
    - cormac-ns/ako.kubernetes.vmware.com.1.13.4+vmware.1-vks.1
    - cormac-ns/cert-manager.kubernetes.vmware.com.1.17.2+vmware.2-vks.1
    - cormac-ns/cert-manager.kubernetes.vmware.com.1.18.2+vmware.2-vks.2
<--snip-->
  data.packaging.carvel.dev/v1alpha1/PackageMetadata:
    - cormac-ns/ako.kubernetes.vmware.com
    - cormac-ns/cert-manager.kubernetes.vmware.com
    - cormac-ns/cluster-autoscaler.kubernetes.vmware.com
<--snip-->
  v1/ConfigMap:
    - cormac-ns/kube-root-ca.crt
  v1/Event:
    - cormac-ns/cormac-pod-1.188a9ca69877b51d
    - cormac-ns/cormac-pod-1.188a9ca70666357e
    - cormac-ns/cormac-pod-1.188a9ca9254c4de3
    - cormac-ns/cormac-pod-1.188a9ca963e97572
    - cormac-ns/cormac-pod-1.188a9ca96cda3faf
    - cormac-ns/cormac-pod-1.188a9ca974af3290
    - cormac-ns/example-cormac-block-pvc.188a9c7c780cab0a
    - cormac-ns/example-cormac-block-pvc.188a9c8eb1bdf906
    - cormac-ns/example-cormac-block-pvc.188a9c8eb1e499a8
    - cormac-ns/example-cormac-block-pvc.188a9c8fc1b87d38
  v1/Namespace:
    - cormac-ns
  v1/PersistentVolume:
    - pvc-85cf01bc-52fb-4bc1-a74f-58990b2988bb
  v1/PersistentVolumeClaim:
    - cormac-ns/example-cormac-block-pvc
  v1/Pod:
    - cormac-ns/cormac-pod-1
  v1/ServiceAccount:
    - cormac-ns/default
Backup Volumes:
  Velero-Native Snapshots: <none included>
  CSI Snapshots:
    cormac-ns/example-cormac-block-pvc:
      Data Movement:
        Operation ID: du-f577e717-58bf-453b-8de8-c5f3b866855e.85cf01bc-52fb-4bcd93615
        Data Mover: velero
        Uploader Type: kopia
        Moved data Size (bytes): 0
        Result: succeeded
  Pod Volume Backups: <none included>
HooksAttempted:  0
HooksFailed:     0

Now we can see even more details about the data movement. We can see that it is using the in-built velero data mover which is using Kopia. And we can in fact see more details about the snapshot and data movement by querying some other custom resources, as we will see next.

Backup: Under the covers

The following commands will examine some of the snapshot details, such as the snapshot location and class. You can add a “-o yaml” to the kubectl get commands, or use kubectl describe, to see more details. But it is the “dataupload” Custom Resource (CR) that is most useful. This is what ties the volume and resulting CSI snapshot tot he data mover and backup storage location, and if there are any issues with that operation, this is the CR to check.

$ kubectl get volumesnapshotlocations -n velero
NAME      AGE
default   22m


$ kubectl get volumesnapshotclasses
NAME                         DRIVER                   DELETIONPOLICY   AGE
volumesnapshotclass-delete   csi.vsphere.vmware.com   Delete           80m


$ kubectl describe datauploads cormac-backup-1-mswhk -n velero
Name:         cormac-backup-1-mswhk
Namespace:    velero
Labels:       velero.io/async-operation-id=du-f577e717-58bf-453b-8de8-c5f3b866855e.85cf01bc-52fb-4bcd93615
              velero.io/backup-name=cormac-backup-1
              velero.io/backup-uid=f577e717-58bf-453b-8de8-c5f3b866855e
              velero.io/pvc-uid=85cf01bc-52fb-4bc1-a74f-58990b2988bb
Annotations:  <none>
API Version:  velero.io/v2alpha1
Kind:         DataUpload
Metadata:
  Creation Timestamp:  2026-01-14T13:43:01Z
  Generate Name:       cormac-backup-1-
  Generation:          5
  Owner References:
    API Version:     velero.io/v1
    Controller:      true
    Kind:            Backup
    Name:            cormac-backup-1
    UID:             f577e717-58bf-453b-8de8-c5f3b866855e
  Resource Version:  12767
  UID:               f4da9b85-db30-4df7-9ebe-f61148c71c7c
Spec:
  Backup Storage Location:  default
  Csi Snapshot:
    Driver:           csi.vsphere.vmware.com
    Snapshot Class:   volumesnapshotclass-delete
    Storage Class:    sfo-w01-cl01-optimal-datastore-default-policy-raid1
    Volume Snapshot:  velero-example-cormac-block-pvc-f968m
  Operation Timeout:  10m0s
  Snapshot Type:      CSI
  Source Namespace:   cormac-ns
  Source PVC:         example-cormac-block-pvc
Status:
  Accepted By Node:      kubernetes-cluster-dkpp-kubernetes-cluster-dkpp-np-pk3w-smkvs46
  Accepted Timestamp:    2026-01-14T13:43:01Z
  Completion Timestamp:  2026-01-14T13:43:49Z
  Node:                  kubernetes-cluster-dkpp-kubernetes-cluster-dkpp-np-pk3w-smkvs46
  Node OS:               linux
  Path:                  /f4da9b85-db30-4df7-9ebe-f61148c71c7c
  Phase:                 Completed
  Progress:
  Snapshot ID:      4ff960c3f4d2f55253926f36a66d5e50
  Start Timestamp:  2026-01-14T13:43:20Z
Events:
  Type    Reason               Age                  From                   Message
  ----    ------               ----                 ----                   -------
  Normal  Data-Path-Started    4m2s                 cormac-backup-1-mswhk  Data path for cormac-backup-1-mswhk started
  Normal  Data-Path-Progress   4m2s (x2 over 4m2s)  cormac-backup-1-mswhk  {}
  Normal  Data-Path-Completed  4m1s                 cormac-backup-1-mswhk  {"snapshotID":"4ff960c3f4d2f55253926f36a66d5e50","emptySnapshot":false,"source":{"byPath":"/f4da9b85-db30-4df7-9ebe-f61148c71c7c","volumeMode":"Filesystem"}}
  Normal  Data-Path-Stopped    4m1s                 cormac-backup-1-mswhk  Data path for cormac-backup-1-mswhk stopped

This looks to have been successful. The final step of this post is to ensure we can do a successful restore of the application that we have just backed up. Let’s now do that.

Restore using Velero CLI

Start by deleting the “cormac-ns” namespace that was just backed up. It also deletes the Pod, PVC and resulting PV.

$ kubectl delete ns cormac-ns
namespace "cormac-ns" deleted


$ kubectl get pv
No resources found

Begin the velero restore, using the –backup command to identify which backup to restore from. Again, we can use the describe command to check the status of the restore.

$ velero restore create cormac-restore-260114 --from-backup cormac-backup-1
Restore request "cormac-restore-260114" submitted successfully.
Run `velero restore describe cormac-restore-260114` or `velero restore logs cormac-restore-260114` 
for more details.


$ velero restore describe cormac-restore-260114
Name:         cormac-restore-260114
Namespace:    velero
Labels:       <none>
Annotations:  <none>
Phase:                       Completed
Total items to be restored:  56
Items restored:              56
Started:    2026-01-14 13:57:37 +0000 UTC
Completed:  2026-01-14 13:58:06 +0000 UTC
Warnings:
  Velero:     <none>
  Cluster:    <none>
  Namespaces:
    cormac-ns:  could not restore, ConfigMap:kube-root-ca.crt already exists. Warning: the in-cluster version is different than the backed-up version
Backup:  cormac-backup-1
Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>
Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
  Cluster-scoped:  auto
Namespace mappings:  <none>
Label selector:  <none>
Or label selector:  <none>
Restore PVs:  auto
CSI Snapshot Restores:
  cormac-ns/example-cormac-block-pvc:
    Data Movement: specify --details for more information
Existing Resource Policy:   <none>
ItemOperationTimeout:       4h0m0s
Preserve Service NodePorts:  auto
Uploader config:
Restore Item Operations:  1 of 1 completed successfully, 0 failed (specify --details for more information)
HooksAttempted:   0
HooksFailed:      0

And just like we saw with the backup, the Data Movement details can be observed if you use the –details options with this command. This will include the following additional details in the output, as well as the list of resources that were restored:

Restore PVs:  auto
CSI Snapshot Restores:
  cormac-ns/example-cormac-block-pvc:
    Data Movement:
      Operation ID: dd-90ca852c-7601-4c70-bd8e-8df9e07c4a31.85cf01bc-52fb-4bcba3ea6
      Data Mover: velero
      Uploader Type: kopia
Existing Resource Policy:   <none>
ItemOperationTimeout:       4h0m0s
Preserve Service NodePorts:  auto
Uploader config:
Restore Item Operations:
  Operation for persistentvolumeclaims cormac-ns/example-cormac-block-pvc:
    Restore Item Action Plugin:  velero.io/csi-pvc-restorer
    Operation ID:                dd-90ca852c-7601-4c70-bd8e-8df9e07c4a31.85cf01bc-52fb-4bcba3ea6
    Phase:                       Completed
    Progress description:        Completed
    Created:                     2026-01-14 13:57:38 +0000 UTC
    Started:                     2026-01-14 13:57:46 +0000 UTC
    Updated:                     2026-01-14 13:58:01 +0000 UTC

The restore appears to have been successful. But lets verify using some kubectl commands.

$ kubectl get ns cormac-ns
NAME        STATUS   AGE
cormac-ns   Active   2m22s


$ kubectl get pod,pvc -n cormac-ns
NAME               READY   STATUS    RESTARTS   AGE
pod/cormac-pod-1   1/1     Running   0          2m32s

NAME                                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                                          VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/example-cormac-block-pvc   Bound    pvc-cf66b8fc-adc1-43e5-8a81-1fbac9e21e9a   5Gi        RWO            sfo-w01-cl01-optimal-datastore-default-policy-raid1   <unset>                 2m32s


$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                STORAGECLASS                                          VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-cf66b8fc-adc1-43e5-8a81-1fbac9e21e9a   5Gi        RWO            Delete           Bound    cormac-ns/example-cormac-block-pvc   sfo-w01-cl01-optimal-datastore-default-policy-raid1   <unset>                          2m37s

Success. It would appear as if everything has restored correctly. If there are issues with a restore, there are some useful CRs available once again.

Restore: Under the covers

Just like we saw with the backup, the following CRs, specifically downloadrequests and datadownlaods can be useful to query if there are some issue with a Velero restore operation on a VKS cluster using CSI snapshots.

$ kubectl get downloadrequests -n velero
NAME                                                         AGE
cormac-restore-260114-01fc60e9-d8c6-4efc-81f2-06f3e545272d   4m10s
cormac-restore-260114-05bd4b4e-5ad6-4e52-a261-f4b89fb0bae3   4m10s
cormac-restore-260114-2eba4517-ca71-4152-bb2a-fda141de8ed7   5m48s
cormac-restore-260114-7bbf1ae8-5e65-4a5b-86e7-54334b3ddd0a   4m45s
cormac-restore-260114-b926fbd2-8234-45c9-af1e-cd229d937504   4m10s
cormac-restore-260114-debbab74-3e50-449e-95b1-ccbe4415f8c9   5m48s
cormac-restore-260114-e133ec6f-6c8e-4951-8611-ffce0ee8ba8f   4m45s
cormac-restore-260114-fd6ed61e-d99e-43e1-a3db-4e2c8644242f   4m11s


$ kubectl describe downloadrequests cormac-restore-260114-01fc60e9-d8c6-4efc-81f2-06f3e545272d  -n velero
Name:         cormac-restore-260114-01fc60e9-d8c6-4efc-81f2-06f3e545272d
Namespace:    velero
Labels:       <none>
Annotations:  <none>
API Version:  velero.io/v1
Kind:         DownloadRequest
Metadata:
  Creation Timestamp:  2026-01-14T13:59:30Z
  Generation:          2
  Resource Version:    15170
  UID:                 db0c147b-7b67-4e0d-92f7-6ff12105b992
Spec:
  Target:
    Kind:  RestoreVolumeInfo
    Name:  cormac-restore-260114
Status:
  Download URL:  https://minio.rainpole.io:9000/velero-backups/restores/cormac-restore-260114/cormac-restore-260114-volumeinfo.json.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=admin%2F20260114%2Fminio%2Fs3%2Faws4_request&X-Amz-Date=20260114T135930Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=ddba7b633a6362e9ca648c301b7c5bb22fbf774230836c9ebe146f6c63a56ff5
  Expiration:    2026-01-14T14:09:30Z
  Phase:         Processed
Events:          <none>


$ kubectl get datadownloads  -n velero
NAME                          STATUS      STARTED   BYTES DONE   TOTAL BYTES   STORAGE LOCATION   AGE     NODE
cormac-restore-260114-v65f8   Completed   6m18s                                default            6m26s   kubernetes-cluster-dkpp-kubernetes-cluster-dkpp-np-pk3w-smkvs46


$ kubectl describe datadownloads cormac-restore-260114-v65f8  -n velero
Name:         cormac-restore-260114-v65f8
Namespace:    velero
Labels:       velero.io/async-operation-id=dd-90ca852c-7601-4c70-bd8e-8df9e07c4a31.85cf01bc-52fb-4bcba3ea6
              velero.io/restore-name=cormac-restore-260114
              velero.io/restore-uid=90ca852c-7601-4c70-bd8e-8df9e07c4a31
Annotations:  <none>
API Version:  velero.io/v2alpha1
Kind:         DataDownload
Metadata:
  Creation Timestamp:  2026-01-14T13:57:38Z
  Generate Name:       cormac-restore-260114-
  Generation:          5
  Owner References:
    API Version:     velero.io/v1
    Controller:      true
    Kind:            Restore
    Name:            cormac-restore-260114
    UID:             90ca852c-7601-4c70-bd8e-8df9e07c4a31
  Resource Version:  14953
  UID:               1c8f7949-b20e-4878-96b8-46a7859423cf
Spec:
  Backup Storage Location:  default
  Data Mover Config:
    Write Sparse Files:  false
  Node OS:               linux
  Operation Timeout:     10m0s
  Snapshot ID:           4ff960c3f4d2f55253926f36a66d5e50
  Source Namespace:      cormac-ns
  Target Volume:
    Namespace:  cormac-ns
    Pv:
    Pvc:        example-cormac-block-pvc
Status:
  Accepted By Node:      kubernetes-cluster-dkpp-kubernetes-cluster-dkpp-np-pk3w-smkvs46
  Accepted Timestamp:    2026-01-14T13:57:38Z
  Completion Timestamp:  2026-01-14T13:58:01Z
  Node:                  kubernetes-cluster-dkpp-kubernetes-cluster-dkpp-np-pk3w-smkvs46
  Phase:                 Completed
  Progress:
  Start Timestamp:  2026-01-14T13:57:46Z
Events:
  Type    Reason               Age              From                         Message
  ----    ------               ----             ----                         -------
  Normal  Data-Path-Started    7m               cormac-restore-260114-v65f8  Data path for cormac-restore-260114-v65f8 started
  Normal  Data-Path-Progress   7m (x2 over 7m)  cormac-restore-260114-v65f8  {}
  Normal  Data-Path-Completed  7m               cormac-restore-260114-v65f8  {"target":{"byPath":"/1c8f7949-b20e-4878-96b8-46a7859423cf","volumeMode":"Filesystem"}}
  Normal  Data-Path-Stopped    7m               cormac-restore-260114-v65f8  Data path for cormac-restore-260114-v65f8 stopped

And that completes the post. Hopefully this has demonstrated that Velero continues to be a very powerful command line tool for backing up and restoring Kubernetes workloads, not just vSphere Kubernetes Service (VKS) clusters. And if you have VCF Automation, remember that you do not need to do this manual approach, but instead you can use the VKS Management Data Protection tool via the UI.

The post Velero Revisited – Manually backing up VKS clusters using Velero appeared first on CormacHogan.com.

First of all, to get access to the new capabilities, you need to be logged in to the VCF Automation Organization Portal. From there, navigate to the Manage & Govern section. Select Kubernetes Management under VCF Services in the left hand navigation pane, then select clusters and this will reveal which VKS clusters are deployed. In this example, there is only one. The fact that it appears here suggests that VKSM has already discovered this cluster and has added the appropriate add-ons to allow it to be managed from VCF Automation. It is in a Healthy and Ready state, which is obviously good. This view alone, when you have many, many VKS clusters, can really help determine the status of your VKS estate at a glance.

Initial Setup

Initially you will find that Data Protection is not enabled on your VKS cluster. To enable Data Protection, you will need to have access to an S3 compatible object store bucket as this is where Data Protection sends the backup data. To enable Data Protection, click on the link highlighted below.

When you enable data protection, this popup appears. VKSM DP uses Velero to provide backup and restore functionality. Backups can use both File System Backup (FSB) and CSI snapshots. Later, we will see how we can choose between the different backup methods for volumes. Click on Enable to continue.

Once Data Protection has been enabled, you will see a new Data Protection tab for the cluster. You will also see a new data-protection agent/extension in the health view. Lastly, you will notice a warning about a missing backup target. This is the S3 bucket mentioned earlier. We will need to create a set of credentials for this target, as well as configure the target.

A similar, related message about missing credentials is seen if you try to configure the target location without first creating some credentials. Create the credentials by navigating to Configurations page, shown below. There are two configurations here – Credentials and Target Locations. Start with creating a new set of credentials.

The account credentials should contain an access key and secret key to allow Velero to access the S3 bucket. This is so that it can send the file backup data and potentially the CSI snapshot data, if this method is chosen for volume backups. More on this when we take a backup.

With the credential created, we can now begin to create the backup target. First step is to provide the credentials for the backup target which we just created in the previous step.

We then begin providing details about the storage provider and bucket. In this case, I am using a MinIO object store, but any compatible S3 API bucket should suffice.  Because I have TLS enabled on MinIO, I am providing a Certificate Authority from the MinIO server to that trust can be established to it by VKSM DP / Velero .

Next, from my list of VKS clusters, I select which ones are allowed to use this backup target. As I only have a single VKS, I only choose one in this example. Note that as you create further VKS clusters, you will also need to navigate to this point to assign your new cluster to the existing backup target location to allow it to use VKSM DP.

After giving the backup target a name, I save it. The backup target is now successfully configured.

Data Protection is now available on my VKS cluster.

This completes the setup. Let’s now try some backup and restore operations.

Create a stateful app on the VKS cluster

To provide something useful for the backup and restore, I made a very simple Pod and PVC using the below manifests, and deployed them into a new namespace in the VKS cluster called cormac-ns. Obviously this is a very simple example, as you can use VKSM DP to backup the whole of the VKS cluster. However, this simple app is just to show the backup and restore functionality in a way that is easy to follow. I won’t delve into describing the contents of each manifest. There is plenty of K8s documentation online that describe this already.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cormac-block-pvc
  namespace: cormac-ns
spec:
accessModes:
    – ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: vsan-default-storage-policy
—
apiVersion: v1
kind: Pod
metadata:
  name: cormac-pod
  namespace: cormac-ns
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
    supplementalGroups: [4000]
  containers:
  – name: busybox
    image: “busybox:latest”
    command: [ “sleep”, “1000000” ]
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: [“ALL”]
      runAsNonRoot: true
      runAsUser: 1000
      seccompProfile:
        type: “RuntimeDefault”
    volumeMounts:
      – mountPath: “/demo”
        name: demo-vol
  volumes:
    – name: demo-vol
      persistentVolumeClaim:
        claimName: cormac-block-pvc

Apply these manifests, and if you wish, use the kubectl exec onto the pod and create some files in the /demo folder which is where the volume has been mounted to.

VKSM DP Backup

Using KSM Data Protection, I can now initiate a backup. The first question that you are prompted with is to select the object or objects that you wish to backup. This could be the whole VKS cluster, a namespace, or a set of objects identified by a selector. In this case, I went with a single namespace backup, cormac-ns.

Now comes the interesting part which is determining “how” to do the backup. There are two options for the File Share Backup (FSB), opt-in and opt-out. If you choose FSB Opt-in, then you have to add some special annotation to the volumes (i.e., Opt-in) if you WANT them to be backed up by FSB. If you choose FSB Opt-out, again, you have add special annotations (i.e., Opt-out) if you DO NOT WANT the volumes to be backed up by FSB. Thus, if you want all volumes to be backed up using the CSI snapshot method, select both “Use FSB Opt-in approach” and “Use CSI Snapshot backup” as shown below. If no FSB annotations are added to any volumes, then CSI will be used for all volumes. See https://velero.io/docs/v1.10/file-system-backup/#using-the-opt-out-approach for more information on annotations. Note that you are annotating with the name of the volume as seen by the Pod, not the actual PVC name. The “do not meet prerequisites for CSI snapshot” is a warning and can be ignored in as VKS Clusters deployed via VCF Automation should automatically have all of the necessary components in place. However, there is a link provided to the documentation where you can verify that the pre-requisites are indeed being met.

Once you have decided on the volume backup method, click next. Now select the backup location that we created previously.

Set the backup schedule. You can create a schedule to meet your requirements, and take backup son a regular basic. Or, do what I am doing here and take a one-off backup right now.

Select a retention policy for the backup (how long to keep it). By default the retention is set to 30 days.

Give the backup a name, and hit the create button.

Backup will enter in progress state, and all going well, should soon report completion.

After completion, click on the backup to see the backup details. It should report that the backup method was CSI snapshot and not FSB in the Persistent Volumes section. If it reports FSB, check that you made the correct backup selection, and that the Pod does not have any FSB annotations if FSB Opt-in was chosen.

VKSM DP Restore

Let’s start a restore by first deleting the cormac-ns namespace. This will also delete the pod and pvc in the namespace.

$ kubectl delete ns cormac-ns

Now let’s use VKMS DP to restore the namespace and its contents from our previous backup. There is a restore backup button in the top right hand corner of the backup details screen which we click to begin the operation.

You are now prompted to select the scope of the restore. Is it the full backup you wish to restore, or perhaps just a single namespace or a set of objects identified by a label. I’ve selected a single namespace just to show you an example, but of course, we only backed up a single namespace anyway so we could have chosen to restore the entire backup. But hopefully you can see the flexibility available.

Next, select the volumes to restore (optional). As we have only backed up a single volume using CSI snapshots, then this is the only volume available to restore as well. However, if you did not want to restore the volume, then you could un-click this option and just restore the pod.

Finally, give the restore job a name, and click on the restore button.

Once the restore successfully completes, you should observe the namespace and its contents restored.

$ kubectl get pod,pvc -n cormac-ns
NAME            READY  STATUS   RESTARTS  AGE
pod/cormac-pod  1/1    Running  0         5m23s

NAME                                     STATUS  VOLUME                                    CAPACITY  ACCESS MODES  STORAGECLASS                 VOLUMEATTRIBUTESCLASS  AGE
persistentvolumeclaim/cormac-block-pvc   Bound   pvc-0f8fd0bc-c2c2-4a9e-89b9-b9e6f2b9d1f3  5Gi       RWO           vsan-default-storage-policy  <unset>                5m24s

Success. We have used VKSM Data Protection to backup and restore a namespace. That completes the post. I think this is a really nice feature which is now available as part of VKSM in VCFA 9.0.1 and later.

The post A first look at VKSM Data Protection (VKSM DP) appeared first on CormacHogan.com.

Getting the VCF CLI

The VCF CLI tool which is available via the Supervisor API URL, accessible from the Summary tab > Status window of any Namespace in the vSphere Client. In the Link to CLI Tools, click Open.

This will take you to the VCF Consumption CLI. Here you can download the VCF CLI tools to match your desktop Operating System. Note that you will also need access to a kubectl command on your desktop to do any meaningful troubleshooting.

Access the K8s objects (including database) running on VKS

Now that we have access to the VCF CLI and kubectl, we can begin to login to both the Supervisor as well as the VKS cluster itself. It might be useful to access the Supervisor directly since this is where the DSM Consumption Operator exists. The DSM Consumption Operator extends the Supervisor API to include the DSM API, so from here we can query the state of the databases. Let’s look at how to create the Supervisor context using the VCF CLI, and then use kubectl to do further queries. The following command, which points to the Supervisor Control Plane Node Address, and uses basic auth with the vSphere administrator login and password, creates the context for the Supervisor as well as all of the existing namespaces on the Supervisor:

> vcf.exe context create --endpoint=192.168.20.6  --username administrator@sfo-w01.local --auth-type basic
? Provide a name for the context: cormac-sv

? Provide a name for the context: cormac-sv
Provide Password: ***********

Logged in successfully.

You have access to the following contexts:
cormac-sv
cormac-sv:dsm-ns-lfyn4
cormac-sv:silver-tenant-ns-b8syh
cormac-sv:svc-auto-attach-domain-c10
cormac-sv:svc-consumption-operator-domain-c10
cormac-sv:svc-tkg-domain-c10
cormac-sv:svc-velero-domain-c10
cormac-sv:vks-ns-2vm7n
cormac-sv:vks-project-ns-74lj4

If the namespace context you wish to use is not in this list, you may need to
refresh the context again, or contact your cluster administrator.

To change context, use `vcf context use <context_name>`
[ok] successfully created context: cormac-sv
[ok] successfully created context: cormac-sv:svc-velero-domain-c10
[ok] successfully created context: cormac-sv:svc-tkg-domain-c10
[ok] successfully created context: cormac-sv:svc-auto-attach-domain-c10
[ok] successfully created context: cormac-sv:vks-project-ns-74lj4
[ok] successfully created context: cormac-sv:dsm-ns-lfyn4
[ok] successfully created context: cormac-sv:svc-consumption-operator-domain-c10
[ok] successfully created context: cormac-sv:silver-tenant-ns-b8syh
[ok] successfully created context: cormac-sv:vks-ns-2vm7n

In this example, the namespace dsm-ns-lfyn4 is the namespace used for landing my DSM databases infrastructure (i.e., VKS cluster). Requests for databases can be made from any tenant namespace as long as they have been given permission to do so via the Data Service Policy in VCF Automation. In this case, a tenant in the silver-tenant-ns-b8syh namespace has requested a database to be created, and the VKS cluster backing this database has bee created in the dsm-ns-lfyn4 namespace, in accordance with the infrastructure policy. Let’s use the Supervisor context and query its nodes using kubectl.

> vcf.exe context use cormac-sv
[ok] Token is still active. Skipped the token refresh for context "cormac-sv"
[i] Successfully activated context 'cormac-sv' (Type: kubernetes)
[i] Fetching recommended plugins for active context 'cormac-sv'...
[i] Installing the following plugins recommended by context 'cormac-sv':
  NAME                CURRENT  INSTALLING
  cluster             v3.3.1   v3.4.1
  kubernetes-release  v3.3.1   v3.4.1
  package             v3.3.1   v3.4.1
  registry-secret     v3.3.1   v3.4.1
[i] Installed plugin 'cluster:v3.4.1'
[i] Installed plugin 'kubernetes-release:v3.4.1'
[i] Installed plugin 'package:v3.4.1'
[i] Installed plugin 'registry-secret:v3.4.1'
[ok] Successfully installed all recommended plugins.

> kubectl.exe get nodes
NAME                                  STATUS   ROLES                  AGE    VERSION
423b4f3d337d2f1bd0ee1538ce627aa3      Ready    control-plane,master   4d2h   v1.31.6+vmware.3-fips
sfo01-w01-r01-esx01.sfo.rainpole.io   Ready    agent                  4d1h   v1.31.6-sph-vmware-clustered-infravisor-trunk-85-g71ed1bf
sfo01-w01-r01-esx02.sfo.rainpole.io   Ready    agent                  4d1h   v1.31.6-sph-vmware-clustered-infravisor-trunk-85-g71ed1bf
sfo01-w01-r01-esx03.sfo.rainpole.io   Ready    agent                  4d1h   v1.31.6-sph-vmware-clustered-infravisor-trunk-85-g71ed1bf

This matches my Supervisor environment which has a single control plane node, and is built on a cluster with 3 ESXi hosts reported as agents. Looks good. Now lets check the DSM Consumption Operator is installed correctly on the Supervisor buy running some DSM specific database commands in this context using kubectl.

> kubectl.exe get postgresclusters -A
NAMESPACE                NAME          STATUS   STORAGE   VERSION                AGE
silver-tenant-ns-b8syh   silver-pg01   Ready    20Gi      17.5+vmware.v9.0.1.0   2d3h

It would appear that there has been a single Postgres database created so far. The name of the database is silver-pg01 and the request to create the database originated from the silver-tenant-ns-b8syh namespace. However, the infrastructure for the database is in a different namespace as mentioned. It is dsm-ns-lfyn4 which we can confirm by doing a describe on the database from the tenant namespace:

> kubectl describe postgresclusters silver-pg01 -n silver-tenant-ns-b8syh
.
. <--snip
.
 Nodes:
Datacenter: sfo-w01-DC
Folder: Namespaces/cormac-sv/dsm-ns-lfyn4/silver-pg01-11706e
Host: sfo01-w01-r01-esx03.sfo.rainpole.io
Network:
Devices:
Network Name: silver-pg01-11706e-v69f5
Resource Pool: sfo-w01-cl01/Resources/Namespaces/dsm-ns-lfyn4/silver-pg01-11706e
Server: sfo-w01-vc01.sfo.rainpole.io
Storage Policy Name: vsan-default-storage-policy
Vm Moid: vm-139
Vm Name: silver-pg01-11706e-r88pq-2dx24
Vm Role: ControlPlane

If we run a query for all of the VKS clusters across all namespaces, we can see that the name of the VKS cluster that backs our Postgres database is the same name as the database (silver-pg01). There are some other VKS clusters deployed in other databases, but these are not used for DSM. They are used for other workloads, which is quite normal to see.

> kubectl get clusters -A
NAMESPACE              NAME                      CLUSTERCLASS                       PHASE         AGE     VERSION
dsm-ns-lfyn4           silver-pg01-11706e        dsmclusterclass-9-0-1-0-24917825   Provisioned   2d23h   v1.32.0+vmware.6-fips
vks-ns-2vm7n           kubernetes-cluster-i7qm   builtin-generic-v3.4.0             Provisioned   2d19h   v1.33.3+vmware.1-fips
vks-project-ns-74lj4   kubernetes-cluster-jkj8   builtin-generic-v3.4.0             Provisioned   6d21h   v1.33.3+vmware.1-fips

With the cluster name, you can look at the events associated with the VKS cluster by using the following kubectl command. This can be useful if there are some issues with the underlying cluster, which in turn prevents the database from coming online.

> kubectl events silver-pg01-11706e -n dsm-ns-lfyn4
LAST SEEN                 TYPE      REASON                         OBJECT                                                                                        MESSAGE
2d17h (x135 over 3d1h)    Normal    UpdateSuccess                  VirtualMachine/silver-pg01-11706e-r88pq-2dx24                                                 Update success
2d17h (x45 over 3d1h)     Normal    SuccessfulUpdate               NetworkInfo/dsm-ns-lfyn4                                                                      NetworkInfo CR has been successfully updated
12m                       Normal    KeyPairVerified                Issuer/silver-pg01-11706e-extensions-ca-issuer                                                Signing CA verified
11m                       Normal    UpdateSucceeded                CnsVolumeMetadata/a8bc3192-7805-46ee-ba2b-f0e5d144c42e-8c018544-40ac-4156-913d-18d95a9e38a2   ReconcileCnsVolumeMetadata: Successfully updated entry in CNS for instance with name "silver-pg01-monitor-0" and entity type "POD" in the guest cluster "a8bc3192-7805-46ee-ba2b-f0e5d144c42e".
11m                       Normal    UpdateSucceeded                CnsVolumeMetadata/a8bc3192-7805-46ee-ba2b-f0e5d144c42e-2fc122fe-9b9e-4ff8-ab21-3a840a4a12a0   ReconcileCnsVolumeMetadata: Successfully updated entry in CNS for instance with name "silver-pg01-monitor-silver-pg01-monitor-0" and entity type "PERSISTENT_VOLUME_CLAIM" in the guest cluster "a8bc3192-7805-46ee-ba2b-f0e5d144c42e".
11m                       Normal    UpdateSucceeded                CnsVolumeMetadata/a8bc3192-7805-46ee-ba2b-f0e5d144c42e-3633c6d4-7e96-409b-8ab6-84d1fe7092e3   ReconcileCnsVolumeMetadata: Successfully updated entry in CNS for instance with name "silver-pg01-pgdata-silver-pg01-0" and entity type "PERSISTENT_VOLUME_CLAIM" in the guest cluster "a8bc3192-7805-46ee-ba2b-f0e5d144c42e".
11m                       Normal    UpdateSucceeded                CnsVolumeMetadata/a8bc3192-7805-46ee-ba2b-f0e5d144c42e-b4909cde-8139-43ca-8a42-0c2838253b8f   ReconcileCnsVolumeMetadata: Successfully updated entry in CNS for instance with name "silver-pg01-0" and entity type "POD" in the guest cluster "a8bc3192-7805-46ee-ba2b-f0e5d144c42e".
11m                       Normal    UpdateSucceeded                CnsVolumeMetadata/a8bc3192-7805-46ee-ba2b-f0e5d144c42e-bf29d539-e02a-4ebc-8d66-3851d0e95428   ReconcileCnsVolumeMetadata: Successfully updated entry in CNS for instance with name "pvc-3633c6d4-7e96-409b-8ab6-84d1fe7092e3" and entity type "PERSISTENT_VOLUME" in the guest cluster "a8bc3192-7805-46ee-ba2b-f0e5d144c42e".
11m                       Normal    UpdateSucceeded                CnsVolumeMetadata/a8bc3192-7805-46ee-ba2b-f0e5d144c42e-c87be272-73aa-4632-a012-5a005ad9ee9c   ReconcileCnsVolumeMetadata: Successfully updated entry in CNS for instance with name "pvc-2fc122fe-9b9e-4ff8-ab21-3a840a4a12a0" and entity type "PERSISTENT_VOLUME" in the guest cluster "a8bc3192-7805-46ee-ba2b-f0e5d144c42e".
10m                       Normal    SuccessfulUpdate               SubnetSet/vm-default                                                                          SubnetSet CR has been successfully updated
9m59s                     Normal    SuccessfulUpdate               Service/silver-pg01-11706e                                                                    LoadBalancer service has been successfully updated
9m59s                     Normal    SuccessfulUpdate               Service/silver-pg01-11706e-88836bd1a14b724dad415                                              LoadBalancer service has been successfully updated
9m51s                     Normal    SuccessfulUpdate               SubnetPort/silver-pg01-11706e-r88pq-2dx24-silver-pg01-11706e-v69f5-eth0                       SubnetPort CR has been successfully updated
9m51s                     Normal    SuccessfulUpdate               SubnetSet/pod-default                                                                         SubnetSet CR has been successfully updated
9m51s                     Normal    SuccessfulUpdate               SubnetSet/silver-pg01-11706e-v69f5                                                            SubnetSet CR has been successfully updated
9m48s                     Normal    SuccessfulUpdate               Pod/jumpbox                                                                                   Pod CR has been successfully updated
9m30s                     Normal    SuccessfulUpdate               NetworkInfo/dsm-ns-lfyn4                                                                      NetworkInfo CR has been successfully updated
9m29s                     Normal    SuccessfulRealizeNSXResource   Service/silver-pg01-11706e-88836bd1a14b724dad415                                              Successful to update NSX resource for DLB Service
9m27s                     Normal    SuccessfulRealizeNSXResource   Service/silver-pg01-11706e                                                                    Successful to process DLB endpoint resource
4m52s (x20 over 12m)      Normal    UpdateSuccess                  VirtualMachine/silver-pg01-11706e-r88pq-2dx24                                                 Update success

With the cluster name, the VCF CLI can now be used to create a new context to access the VKS cluster backing the DSM database

Create a VKS context

The following commands create a new context for a VKS cluster (the one which backs the Postgres database called silver-pg01). To create this context, two additional parameters (workload-cluster-name and workload-cluster-namespace) must be included. Again, the auth-type is set to basic, but other authentication options are available (for details on how a tenant can use API tokens to access the VKS cluster, see this blog post from Tomas Fjota). Once the context is created, we can begin using this context and query the nodes and pods running in the cluster. Since this is a DSM provisioned single node database, it is normal to see a single control plane node provisioned. This node also allows DSM to create the Postgres pods on the control plane, something that is not allowed with vanilla VKS which always provisions a worker node for application workloads.

> vcf.exe context create --endpoint=192.168.20.6 --username administrator@sfo-w01.local 
--workload-cluster-name silver-pg01-11706e --workload-cluster-namespace dsm-ns-lfyn4 
--auth-type basic

? Provide a name for the context:  silver-pg01

? Provide a name for the context:  silver-pg01
Provide Password: *********

[i] Logging in to Kubernetes cluster (silver-pg01-11706e) (dsm-ns-lfyn4)
[i] Successfully logged in to Kubernetes cluster 192.168.22.4

You have access to the following contexts:
   silver-pg01
   silver-pg01:silver-pg01-11706e

If the namespace context you wish to use is not in this list, you may need to
refresh the context again, or contact your cluster administrator.
To change context, use `vcf context use <context_name>`
[ok] successfully created context: silver-pg01
[ok] successfully created context: silver-pg01:silver-pg01-11706e


> vcf.exe context list
  NAME                                           CURRENT  TYPE
  cormac-sv                                      true     kubernetes
  cormac-sv:dsm-ns-lfyn4                         false    kubernetes
  cormac-sv:silver-tenant-ns-b8syh               false    kubernetes
  cormac-sv:svc-auto-attach-domain-c10           false    kubernetes
  cormac-sv:svc-consumption-operator-domain-c10  false    kubernetes
  cormac-sv:svc-tkg-domain-c10                   false    kubernetes
  cormac-sv:svc-velero-domain-c10                false    kubernetes
  cormac-sv:vks-ns-2vm7n                         false    kubernetes
  cormac-sv:vks-project-ns-74lj4                 false    kubernetes
  silver-pg01                                    false    kubernetes
  silver-pg01:silver-pg01-11706e                 false    kubernetes
[i] Use '--wide' to view additional columns.


> vcf context use silver-pg01:silver-pg01-11706e

[ok] Token is still active. Skipped the token refresh for context "silver-pg01:silver-pg01-11706e"
[i] Successfully activated context 'silver-pg01:silver-pg01-11706e' (Type: kubernetes)
[i] Fetching recommended plugins for active context 'silver-pg01:silver-pg01-11706e'...
[ok] No recommended plugins found.

> vcf.exe context list

  NAME                                           CURRENT  TYPE
  cormac-sv                                      false    kubernetes
  cormac-sv:dsm-ns-lfyn4                         false    kubernetes
  cormac-sv:silver-tenant-ns-b8syh               false    kubernetes
  cormac-sv:svc-auto-attach-domain-c10           false    kubernetes
  cormac-sv:svc-consumption-operator-domain-c10  false    kubernetes
  cormac-sv:svc-tkg-domain-c10                   false    kubernetes
  cormac-sv:svc-velero-domain-c10                false    kubernetes
  cormac-sv:vks-ns-2vm7n                         false    kubernetes
  cormac-sv:vks-project-ns-74lj4                 false    kubernetes
  silver-pg01                                    false    kubernetes
  silver-pg01:silver-pg01-11706e                 true     kubernetes
[i] Use '--wide' to view additional columns.


> kubectl get nodes

NAME                             STATUS   ROLES           AGE     VERSION
silver-pg01-11706e-r88pq-2dx24   Ready    control-plane   2d23h   v1.32.0+vmware.6-fips


> kubectl get pods -A

NAMESPACE                       NAME                                                     READY   STATUS      RESTARTS       AGE
cert-manager                    cert-manager-5668b6499f-t29zf                            1/1     Running     2 (31m ago)    2d23h
cert-manager                    cert-manager-cainjector-859df965db-5zf4p                 1/1     Running     2 (31m ago)    2d23h
cert-manager                    cert-manager-webhook-7d66fb4668-bg9s8                    1/1     Running     1 (31m ago)    2d23h
d14a47-silver-tenant-ns-b8syh   default-incremental-backup-29422079-v8rmz                0/1     Completed   0              2d11h
d14a47-silver-tenant-ns-b8syh   default-incremental-backup-29423519-fhsmj                0/1     Completed   0              35h
d14a47-silver-tenant-ns-b8syh   default-incremental-backup-29424959-vl5wm                0/1     Completed   0              31m
d14a47-silver-tenant-ns-b8syh   silver-pg01-0                                            4/4     Running     4 (31m ago)    2d23h
d14a47-silver-tenant-ns-b8syh   silver-pg01-monitor-0                                    4/4     Running     4 (31m ago)    2d23h
kube-system                     antrea-agent-zmk8v                                       2/2     Running     5 (29m ago)    2d23h
kube-system                     antrea-controller-f5d6d787f-m2bbn                        1/1     Running     4 (29m ago)    2d23h
kube-system                     coredns-57db7b44f5-csfjm                                 1/1     Running     1 (31m ago)    2d23h
kube-system                     coredns-69b565fcb5-9bswf                                 0/1     Pending     0              2d23h
kube-system                     docker-registry-silver-pg01-11706e-r88pq-2dx24           1/1     Running     1 (31m ago)    2d23h
kube-system                     etcd-silver-pg01-11706e-r88pq-2dx24                      1/1     Running     1 (31m ago)    2d23h
kube-system                     image-puller-4jltj                                       1/1     Running     5 (31m ago)    2d23h
kube-system                     kube-apiserver-silver-pg01-11706e-r88pq-2dx24            1/1     Running     1 (31m ago)    2d23h
kube-system                     kube-controller-manager-silver-pg01-11706e-r88pq-2dx24   1/1     Running     3 (31m ago)    2d23h
kube-system                     kube-proxy-cts9x                                         1/1     Running     1 (31m ago)    2d23h
kube-system                     kube-scheduler-silver-pg01-11706e-r88pq-2dx24            1/1     Running     3 (31m ago)    2d23h
kube-system                     metrics-server-6ccf55cf87-4jbzt                          1/1     Running     1 (31m ago)    2d23h
kube-system                     snapshot-controller-7ccbcfddfd-4czzc                     1/1     Running     1 (31m ago)    2d23h
pinniped-concierge              pinniped-concierge-77ccbc897d-75l2z                      1/1     Running     1 (31m ago)    2d23h
pinniped-concierge              pinniped-concierge-77ccbc897d-ww9fl                      1/1     Running     1 (31m ago)    2d23h
pinniped-concierge              pinniped-concierge-kube-cert-agent-7449f8dbbb-vjt5b      1/1     Running     1 (31m ago)    2d23h
secretgen-controller            secretgen-controller-5cbf99f6c-t9bdn                     1/1     Running     1 (31m ago)    2d23h
telegraf                        telegraf-6d994786d8-cn82f                                1/1     Running     1 (31m ago)    2d23h
tkg-system                      kapp-controller-7ff74d9865-4989q                         2/2     Running     5 (29m ago)    2d23h
vmware-sql-postgres             postgres-operator-56ff7f7679-dd8vl                       1/1     Running     1 (31m ago)    2d23h
vmware-system-antrea            antrea-pre-upgrade-job-dsknp                             0/1     Completed   0              2d23h
vmware-system-auth              guest-cluster-auth-svc-lgv4x                             1/1     Running     1 (31m ago)    2d23h
vmware-system-cloud-provider    guest-cluster-cloud-provider-67f87c6699-qnhqd            1/1     Running     6 (29m ago)    2d23h
vmware-system-csi               vsphere-csi-controller-854ffbff6-w9lbf                   7/7     Running     7 (31m ago)    2d23h
vmware-system-csi               vsphere-csi-node-jp6jn                                   3/3     Running     13 (29m ago)   2d23h

In this VKS context, in the namespace d14a47-silver-tenant-ns-b8syh, I can see the pod for the primary database (silver-pg01-0) as well as the pod for the monitor (silver-pg01-monitor-0) which I’ve highlighted in blue above. Note that the primary pod has 4 containers called pg-container, instance-logging, reconfigure-instance and postgres-sidecar. Using kubectl, I can describe the pod, get events and look at the container logs, e.g., if  you want to look at the pg-container log, run the following command (-c for container):

> kubectl logs silver-pg01-0 -c pg-container -n d14a47-silver-tenant-ns-b8syh | more
2025-12-12T10:33:02.722Z INFO postgresinstance Removing post start tasks executed
2025-12-12T10:33:02.731Z INFO postgresinstance Removed post start tasks executed
2025-12-12T10:33:02.731Z INFO postgresinstance Running pre-start tasks
2025-12-12T10:33:02.731Z INFO postgresinstance executing {"task": "ConfigureDirectoryPermission"}
2025-12-12T10:33:02.731Z INFO postgresinstance executing {"task": "ConfigurePassFileTask"}
2025-12-12T10:33:02.732Z INFO postgresinstance executing {"task": "WaitForMonitorTask"}
2025-12-12T10:33:02.825Z INFO postgresinstance failed to connect to `user=autoctl_node database=pg_auto_failover`: hostname resolving error: lookup silver-pg01-monitor-0.silver-pg01-agent.d14a47-silver-tenant-ns-b8syh.svc.cluster.local on 10.96.0.10:53: no such host
2025-12-12T10:33:07.844Z INFO postgresinstance failed to connect to `user=autoctl_node database=pg_auto_failover`: 192.168.0.16:5432 (silver-pg01-monitor-0.silver-pg01-agent.d14a47-silver-tenant-ns-b8syh.svc.cluster.local): dial error: dial tcp 192.168.0.16:5432: connect: connection refused
2025-12-12T10:33:12.898Z INFO postgresinstance Connected to monitor
2025-12-12T10:33:12.898Z INFO postgresinstance executing {"task": "CleanUpDatabaseProcessTask"}
2025-12-12T10:33:12.898Z INFO postgresinstance Start cleanup process database...
pg_ctl: could not send stop signal (PID: 309): No such process
2025-12-12T10:33:12.901Z INFO postgresinstance executing {"task": "RemovePostmasterPidTask"}
2025-12-12T10:33:12.902Z INFO postgresinstance executing {"task": "ClearCustomTempDirTask"}
2025-12-12T10:33:12.903Z INFO postgresinstance executing {"task": "WriteCustomConfigFileTask"}
2025-12-12T10:33:12.923Z INFO postgresinstance Applying custom config {"config": {"Mode":"verify-ca","CAFilePath":"/etc/postgres_ssl/ca.crt","CertFilePath":"/etc/postgres_ssl/tls.crt","KeyFilePath":"/etc/postgres_ssl/tls.key","CustomConfigFilePath":"/pgsql/custom/postgresql-custom-override.conf","IsArchiveModeEnabled":true,"UnixSocketDirectories":["/pgsql/custom/tmp","/tmp"],"ArchiveCommand":"pgbackrest --stanza=d14a47-silver-tenant-ns-b8syh-silver-pg01-293d8da7-4489-4a93-a5c4-7a949b2960d4 archive-push %p","SharedPreloadLibraries":["pg_stat_statements","pgaudit","pg_cron"],"PostgresVersion":"17","PostgresLogDirectory":"/pgsql/logs/postgres","UserProvidedCustomPostgresConfigPath":"/etc/customconfig/postgresql.conf","BackupBasedContinuousRestoreMode":false,"SharedBuffers":"2654 MB","WorkMem":"26 MB","WalKeepSize":"96 MB","WalKeepSegments":6,"MaintenanceWorkMem":"530 MB","EffectiveCacheSize":"5309 MB","MaxSlotWalKeepSize":"1998 MB"}}
2025-12-12T10:33:12.924Z INFO postgresinstance executing {"task": "ConfigureMonitorConnectionStringTask"}
2025-12-12T10:33:12.939Z INFO postgresinstance executing {"task": "ConfigureSSLTask"}
2025-12-12T10:33:12.947Z INFO postgresinstance ssl.ca_file is already set to /etc/postgres_ssl/ca.crt
2025-12-12T10:33:12.954Z INFO postgresinstance ssl.cert_file is already set to /etc/postgres_ssl/tls.crt
2025-12-12T10:33:12.962Z INFO postgresinstance ssl.key_file is already set to /etc/postgres_ssl/tls.key
2025-12-12T10:33:12.962Z INFO postgresinstance executing {"task": "InitializeDatabaseTask"}
2025-12-12T10:33:12.962Z INFO postgresinstance Start initializing database...

So as you can clearly see, some low-level troubleshooting can be done using the new VCF CLI when a DSM database is provisioned via VCF Automation, using a Supervisor namespace infrastructure policy and is there fore using the vSphere Kubernetes Service to host the database.

Note: If you are running VCF v9.0.1 with DSM integrated, and you run the above commands and notice that there are no running containers (0/4) in the database pod, then you may have encountered an issue with the VKS Management service consuming most of the available resources in the control plane nodes of the VKS cluster. The VKS Management component automatically adds a bunch of agents to the control plane nodes of a VKS cluster, resulting in not enough resources to run the database. If you think this might be it, this is the KB which describes the workaround of adding the database namespace to the VKSM configMap – https://knowledge.broadcom.com/external/article?articleNumber=412306

SSH access to the VKS node

Now the final part of this section is to show you how to ssh onto the VKS node that hosts the database. Caution: With great power comes great responsibility. I would urge you to use extreme care if logging onto the VKS node, as you may end up doing something that impacts the database. However, there may be valid reasons where you need to do this, perhaps checking networking connectivity, etc. So, as per the official documentation, here is how to get ssh access to a VKS node using a jumpbox PodVM that has been deployed onto the same Namespace as the VKS cluster.

The first step is to switch contexts once more, and go back to the Supervisor context, cormac-sv. From here, you will need to list the Kubernetes “secrets” in the namespace where the database infra has been provisioned, in this case the dsm-ns-lfyn4. This ssh secret contains a private key. With this information,  you will be able to ssh onto the VKS node as a system user (vmware-system-user). The secret we are interested in is highlighted in blue below.

> vcf.exe context use cormac-sv
[ok] Token is still active. Skipped the token refresh for context "sv"
[i] Successfully activated context 'sv' (Type: kubernetes)
[i] Fetching recommended plugins for active context 'sv'...
[ok] All recommended plugins are already installed and up-to-date.

> kubectl.exe get secrets -n dsm-ns-lfyn4
NAME                                                        TYPE                                  DATA   AGE
cluster-autoscaler-secret                                   kubernetes.io/service-account-token   3      3d
silver-pg01-11706e-antrea-data-values                       Opaque                                1      3d
silver-pg01-11706e-auth-svc-cert                            kubernetes.io/tls                     3      3d
silver-pg01-11706e-ca                                       cluster.x-k8s.io/secret               2      3d
silver-pg01-11706e-control-plane-machine-agent-conf         Opaque                                1      3d
silver-pg01-11706e-encryption                               Opaque                                1      3d
silver-pg01-11706e-encryption-config                        Opaque                                1      3d
silver-pg01-11706e-etcd                                     cluster.x-k8s.io/secret               2      3d
silver-pg01-11706e-extensions-ca                            kubernetes.io/tls                     3      3d
silver-pg01-11706e-gateway-api-package                      clusterbootstrap-secret               0      3d
silver-pg01-11706e-guest-cluster-auth-service-data-values   Opaque                                1      3d
silver-pg01-11706e-kapp-controller-data-values              Opaque                                2      3d
silver-pg01-11706e-kubeconfig                               cluster.x-k8s.io/secret               1      3d
silver-pg01-11706e-ma-token                                 Opaque                                2      3d
silver-pg01-11706e-metrics-server-package                   clusterbootstrap-secret               0      3d
silver-pg01-11706e-pinniped-package                         clusterbootstrap-secret               1      3d
silver-pg01-11706e-proxy                                    cluster.x-k8s.io/secret               2      3d
silver-pg01-11706e-r88pq-2dx24                              cluster.x-k8s.io/secret               2      3d
silver-pg01-11706e-sa                                       cluster.x-k8s.io/secret               2      3d
silver-pg01-11706e-secretgen-controller-package             clusterbootstrap-secret               1      3d
silver-pg01-11706e-ssh                                      kubernetes.io/ssh-auth                1      3d
silver-pg01-11706e-ssh-password                             Opaque                                1      3d
silver-pg01-11706e-ssh-password-hashed                      Opaque                                1      3d
silver-pg01-11706e-user-trusted-ca-secret                   Opaque                                1      3d
silver-pg01-11706e-v69f5-ccm-secret                         kubernetes.io/service-account-token   3      3d
silver-pg01-11706e-v69f5-pvbackupdriver-secret              kubernetes.io/service-account-token   3      3d
silver-pg01-11706e-v69f5-pvcsi-secret                       kubernetes.io/service-account-token   3      3d
silver-pg01-11706e-vsphere-cpi-data-values                  Opaque                                1      3d
silver-pg01-11706e-vsphere-pv-csi-data-values               Opaque                                1      3d

The next step is to create a YAML manifest which describes the PodVM that we wish to deploy. Remember that this PodVM is deployed in the same namespace (dsm-ns-lfyn4) as the VKS cluster running the Postgres database pods. Here is an example of such as PodVM. It uses a Photon OS image and passes a command that copies the private key from the volume created from the secret which we referenced earlier. The secretName is highlighted in blue below once more. The volume holding the private key is mounted onto /root/ssh. We use yum to install openssh. The private key is then copied to a file called /root/.ssh/id_rsa. This now allows an ssh session as the ‘vmware-system-user’ onto the VKS node from the jumpbox PodVM. We pass the ssh command as an argument when we exec to the Pod (we will see how to do this shortly).

apiVersion: v1
kind: Pod
metadata:
  name: jumpbox
  namespace: dsm-ns-lfyn4
spec:
  containers:
  - image: "photon:5.0"
    name: jumpbox
    command: [ "/bin/bash", "-c", "--" ]
    args: [ "yum install -y openssh-server; mkdir /root/.ssh; cp /root/ssh/ssh-privatekey /root/.ssh/id_rsa; chmod 600 /root/.ssh/id_rsa; while true; do sleep 30; done;" ]
    volumeMounts:
      - mountPath: "/root/ssh"
        name: ssh-key
        readOnly: true
    resources:
      requests:
        memory: 2Gi
  volumes:
    - name: ssh-key
      secret:
        secretName: silver-pg01-11706e-ssh
  imagePullSecrets:
    - name: regcred

Next, apply the manifest to create the PodVM, and ensure it is running.

> kubectl.exe apply -f jumpbox-dsm.yml
pod/jumpbox created
 
> kubectl get pods -n dsm-ns-lfyn4 
NAMESPACE             NAME                READY   STATUS      RESTARTS        AGE
dsm-ns-lfyn4          jumpbox             1/1     Running     0               3m37s

Now, determine the IP address of the node that we wish to ssh onto. The following command, which queries virtual machines in the namespace, will provide this information.

> kubectl.exe get vm -o wide -n dsm-ns-lfyn4
NAME                             POWER-STATE   CLASS               IMAGE                   PRIMARY-IP4     AGE
silver-pg01-11706e-r88pq-2dx24   PoweredOn     best-effort-large   vmi-24554b66363a299c5   192.173.237.3   2d3h

The ssh session can now be initiated. Run a ‘kubectl exec’ command as show below. As soon as you ‘exec’ onto the jumpbox PodVM, the ssh command is run to the VKS node’s IP address is run. The PodVM will run the command and args as defined in the YAML manifest above to configure the private key for the ‘vmware-system-user’. This then allows the ssh command passed to the ‘kubectl exec’ onto the VKS node to succeed.

> kubectl exec -it jumpbox -n dsm-ns-lfyn4 -- /usr/bin/ssh vmware-system-user@192.173.237.3
The authenticity of host '192.173.237.3 (192.173.237.3)' can't be established.
ED25519 key fingerprint is SHA256:HEu9cZ9Uq4EFwQvR1KiWbJWlR0Jd3SXNK7lu8C6WBnY.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.173.237.3' (ED25519) to the list of known hosts.
cat: /var/run/motdgen/motd: Permission denied

vmware-system-user@silver-pg01-11706e-r88pq-2dx24 [ ~ ]$ ip a | more
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc prio state UP group default qlen 1000
    link/ether 04:50:56:00:78:00 brd ff:ff:ff:ff:ff:ff
    altname eno1
    altname enp11s0
    altname ens192
    inet 192.173.237.3/27 brd 192.173.237.31 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::650:56ff:fe00:7800/64 scope link
       valid_lft forever preferred_lft forever
.
.
.

And that completes the step to gain ssh access onto a VKS node that is backing a DSM provisioned database. Refer to the official documentation linked earlier for other methods (such as password) to gain access. If you need to lock down ssh access the the database, you can of course create an NSX Firewall Rule and block this port. There is a simple example of how to do this available here.

Summary

That concludes the post. Hopefully you have seen some of the ways in which it is possible to troubleshoot DSM database deployments on vSphere Kubernetes Services clusters. As mentioned, this is the default Kubernetes used when the infrastructure policy for the database is a Supervisor Namespace, typically provisioned via VCF Automation in VCF 9.x.

The post Using the VCF 9.x CLI to troubleshoot a DSM database running on VKS appeared first on CormacHogan.com.

In the example that I will show you here, I am going to stipulate that certain members of an organization need to set their retention policy for backups to 91 days rather than 30 days, which is the default. Let’s begin by navigating to the VCF Automation Organization Portal, and selecting the Manage & Govern view. Here you will find the Policies section in the left-hand navigation bar. We are going to create an IaaS Resource Policy. Click on that.

This will open the ‘New Policy’ wizard. Provide a name an optional description, and which Organization or Project(s) that the policy applies to. You can even get more granular in the criteria section and include Namespace names, Namespace classes or Regions to associate with the policy.

If you click on the “+ Load Policy Template” button, you will see some examples of pre-defined IaaS Resource Polices, which you can use as a template for further policies. There are a number of out-of-the-box templates such as a policy that enforces a maximum of one control plane/worker node on a Kubernetes cluster. This is just an FYI, so once you have had a chance to look at some of the examples, you can cancel the load template view and we can proceed with our database-specific IaaS policy.

Begin by filling in the name, scope, etc:

Next, add the code block for the validation, with a goal of requiring the retention policy for backups to be set to 91 days rather than the default value of 30 days. To do that, the code looks to match on the PostgresClusters API type object, and then match on the spec.backupConfig.backupRetentionDays field and express a constraint. Prior knowledge of the PostgresCluster specification is necessary to determine where to place to the constraints.

The IaaS Resource Policy type provides a policy-as-code approach using Kubernetes Validation Admission Policy. Validating admission policies use the Common Expression Language (CEL) to declare the validation rules of a policy. In the code above, it is using the `has()` function to safely check the the fields exist before accessing them, and then validates that `backupRetentionDays == 91`. If it does not match 91, then we will generate a message to the user indicating that this field is not set correctly.

Here is the actual code block. It is essentially the spec section from a Kubernetes Validating Admission Policy.

failurePolicy: Fail
matchConstraints:
  resourceRules:
  - apiGroups:   ["databases.dataservices.vmware.com"]
    apiVersions: ["v1alpha1"]
    operations:  ["CREATE", "UPDATE"]
    resources:   ["postgresclusters"]
validations:
- expression: "has(object.spec.backupConfig) && has(object.spec.backupConfig.backupRetentionDays) && object.spec.backupConfig.backupRetentionDays == 91"
  messageExpression: "'backupRetentionDays must be exactly 91, but got ' + string(object.spec.backupConfig.backupRetentionDays)"
  message: "backupRetentionDays must be exactly 91"
  reason: Invalid

Once created, the policy becomes visible in the “Definitions” view. If there are any issues with your policy definition, they will be highlighted here.

You can also see the scope of a policy, for example which Project and which Namespace it applies to.

And any time that this policy needs to do an enforcement on a database provisioning operation, this is visible in the Enforcements view:

With this in mind, let’s now try to provision a Postgres database with a retention policy set to the default value of 30 days from a tenant namespace in the project to which I have applied the policy. It should prevent me from doing so, and report the reason why … which it does.

The admission control policy prevented the deployment of a Postgres database due to the incorrect backup retention set by the user (30 days instead of 91). This is just one example of using the new IaaS Resource Policy engine for database resources.

If I try the exact same deployment, but this time specify a retention time of 91 days for backups, then the database deploys successfully as the condition in the IaaS Resource policy is met.

Here is another simple snippet of validation code, this time verifying that the name of the database is at least 8 characters long:

validations:
  - reason: Invalid
    message: databaseName must be greater than 8 chars
    expression: has(object.spec.databaseName) && size(object.spec.databaseName) > 8
    messageExpression: "'IaaS Policy Validation issue: database name must be greater
      than 8 characters, but got ' + string(size(object.spec.databaseName))"
failurePolicy: Fail
matchConstraints:
  resourceRules:
    - apiGroups:
        - databases.dataservices.vmware.com
      resources:
        - postgresclusters
      operations:
        - CREATE
        - UPDATE
      apiVersions:
        - v1alpha1

And now any attempt to create a database with a name that is less than 8 characters will be constrained by this policy:

Summary

Hopefully this post has highlighted the power of IaaS Resource Policies. Data Service Policies (DSP) used by DSM in VCF Automation do not cover all the fields of PostgresCluster API object. Also, DSPs are only available at the Provider level. IaaS Resource Policies are a great complement to DSPs. IaaS Resource Policies are available at the Organization/Tenant level and uses expressions that can put restrictions on any API object and on any field.

As seen in the above examples, we are able to restrict the backup retention and the database name length in a PostgresDatabase object. DSPs do not control this today. But using an IaaS Resource Policy, it is possible to say that you want to match on the PostgresCluster API type, and then you can match on the spec.databaseName or spec.backupConfig.backupRetentionDays field and express an appropriate constraint. Indeed, you can match of any of the fields and create any validation or constraint that you wish. This is a very powerful VCF and VCF Automation feature.

This GiHub repo contains the code shown in the above examples.

The post DSM 9.0.1 – Using IaaS Resource Policies to fine-tune database deployments in VCF Automation appeared first on CormacHogan.com.

Let’s start with the prerequisites. To begin with, a Data Service Policy will be required to allocate permissions and resources to tenants to create data services. If you wish to know more about Data Service Policies, there is a detailed description in this introductory blog post on DSM & VCFA integration. If you wish to provision a Postgres database, then the steps to create a Data Service Policy for Postgres are available in the VCFA UI. You can see how to do this, along with the other setup steps, in the blog I just referenced previously. However, if you wish to create a Data Service Policy for MySQL databases, then you will have to do this step via the API because, as mentioned, it is not yet plumbed up in the UI. The steps to do this are in my previous blog post, so you can follow the steps on how to create a MySQL DSP and database there.

We can now proceed with creating a database blueprint in VCFA. In the Tenant Organization, under Build and Deploy, select the Blueprint Design from the left-hand navigation menu. You can upload the blueprint code from a file, or create it directly. In the Organization select Blueprint Design > Blueprints > New From and then decide if you want to use a ‘Blank canvas’ to create the template or to ‘Upload’ it from a file.

Next, provide details about the blueprint template, such as a name, a description, and an optional custom icon. You can also select which Project within the Organization has access, and decide if, at some point, you want to grant other Projects in the same Organization access as well. That is what I have done here.

Let’s being with what a blueprint looks like from a blank canvas:

So we will populate both the inputs and the resources section. I will define my inputs which will be later used to form variables for my database, and also use a cciNamespace and dsmDb Supervisor resource. The latter is only achievable when the Consumption Operator is installed on the Supervisor, which provides the Supervisor with CRDs for DSM databases. Once those resource are in place, I can use a standard DSM YAML manifest for provisioning. Note that because I plan to use variables in my blueprint code, the formatVersion field will need to be bumped to 2.

I am making a few assumption about my VCFA and DSM environments. I am assuming that the namespaces in the code below have already been selected as valid DSM infrastructure policies. I am also assuming that there is a valid Data Service Policy created which grants access to the Postgres data service to the Organisation where my tenant namespace resides. With those assumption in mind, here is a blueprint for creating Postgres databases via VCFA using DSM. You can use this code sample to create your own blueprint. Lines which begin with a # are comments, and do not need to be included in the actual code.

# Blueprint to provision DSM Postgres Database
# Assumption is that Data Service Policy already exists and that the ORG where the tenant resides has been granted access to provision databases
# Another assumption is that the namespace where the database is provisioned to is part of an infrastructure policy
# Kudos to Eric Gray for initial template
#
formatVersion: 2
inputs:
  dbName:
    type: string
    title: Database Name
    description: Name of Postgres DB - leave blank to use deployment name; 4 character minimum
    #minLength: 4
    default: '' # indicates optional
  namespace:
    type: string
    title: Namespace
    description: Tenant namespace for this deployment
    default: same-ns-5hklb
  dbInfraPolicy:
    type: string
    title: Infrastructure Policy
    description: Target DSM infrastructure policy or namespace
    enum:
      - same-ns-5hklb
      - diff-ns-same-org-6mbgt
      - dsm-ns-01-5rvdd
    default: same-ns-5hklb
  storagePolicy:
    type: string
    title: Storage Policy
    description: vSphere Storage Policy
    default: vSAN Default Storage Policy
  diskSize:
    type: integer
    title: DB Disk Size
    maximum: 80
    minimum: 20
    default: 20
  backupLocation:
    type: string
    title: Backup Location
    description: S3 storage bucket configured for DSM backups
    enum:
      - none
      - dsm-backups
    default: dsm-backups
  dbAvailability:
    type: string
    title: Availability Model
    description: Single VM or Primary/Replica HA Cluster
    default: single
    oneOf:
      - title: Single VM
        const: single
      - title: HA Cluster
        const: cluster
  dbVersion:
    type: integer
    title: Postgres Major Version
    description: Major version of the database being provisioned
    maximum: 17
    minimum: 12
    default: 17
# Some of the inputs may need some pre-handling before getting back to the manifest below
variables:
  dbName: ${to_lower(input.dbName || env.deploymentName)}
  dbClass: ${input.dbAvailability == "single" ? "best-effort-large" :"best-effort-xlarge"}
  backup: ${input.backupLocation != "none" ? input.backupLocation :null}
  dbVersion: ${to_string(input.dbVersion)}
  backupConf:
    backupRetentionDays: 30
    schedules:
      - name: default-full-backup
        type: full
        schedule: 59 23 * * 6
      - name: default-incremental-backup
        type: incremental
        schedule: 59 23 1/1 * *
resources:
  cciNamespace:
    type: CCI.Supervisor.Namespace
    properties:
      name: ${input.namespace}
      existing: true
# Supervisor resource, in this case a DSM database, which is created in
# the Namespace context and will either succeed or fail
  dsmDb:
    type: CCI.Supervisor.Resource
    properties:
      context: ${resource.cciNamespace.id}
      wait:
        conditions:
          - status: 'True'
            type: Ready
          - status: 'True'
            type: TerminalFailure
            indicatesFailure: true
# This manifest is now identical to the YAML code used to create a database using the DSM API
# The only difference is that we are populating field based on the inputted variables above
# Essentially, the DSM APIs are made available to vSphere through the DSM Consumption Operator
# which is installed as a Supervisor Service, allowing VCF Automation to consume them
      manifest:
        apiVersion: databases.dataservices.vmware.com/v1alpha1
        kind: PostgresCluster
        spec:
          adminUsername: pgadmin
          databaseName: ${variable.dbName}
          version: ${variable.dbVersion}
          storageSpace: ${input.diskSize + 'Gi'}
          storagePolicyName: ${input.storagePolicy}
          infrastructurePolicy:
            name: ${input.dbInfraPolicy}
          replicas: ${input.dbAvailability == "single" ? 0 :1}
          vmClass:
            name: ${variable.dbClass}
          maintenanceWindow:
            duration: 6h
            startDay: SATURDAY
            startTime: '23:59'
          backupLocation: ${variable.backup != null ? {'name':input.backupLocation} :null}
          backupConfig: ${variable.backup != null ? variable.backupConf :null}
        metadata:
          name: ${variable.dbName}
          namespace: ${resource.cciNamespace.name}

The main sections of the code related to resources are input, variables and resources. My two resources are:

• Supervisor Namespace – CCI.Supervisor.Namespace – is used to create ( or select ) a Supervisor Namespace, which provides a Kubernetes-based workspace with resource limits, user access, and available Supervisor services so we can provision our VM and TKG resources based on our application needs.
• Supervisor Resource – CCI.Supervisor.Resource – is used to create any supported Supervisor K8s resource within a Supervisor Namespace, such as (using their K8s object kind), virtualmachines, virtualmachineservices, tanzukubernetesclusters, persistentvolumeclaims, secrets, and so on, depending on the K8s manifest we pass to the Supervisor resource we are configuring. Since we have extended the API with the Consumption Operator to add a DBaaS Supervisor Service, we can request a PostgresCluster as shown above. We would not be able to request this objectwithout the Consumption Operator in place.

Next, add the code to the blueprint. You can see here that the canvas has recognised the Supervisor Namespace element and the dsmDb element, and placed both of these on the canvas. These are CCI template elements included in the Cloud Consumption Interface. These can also be dragged, dropped and configured via the canvas, if you wish.

Once the code is added, we can do a check to verify that the syntax is correct. In the lower left hand side of the canvas there is a ‘TEST’ button. It will launch the inputs wizard for the database. Here you can test populating any necessary fields and check that any enum fields and any maximum/minimum fields are working correctly. If the syntax is working, the test result should show successful. Note that this does not validate the selections exist on the back-end. It is simply a syntax checker.

Next, click on the ‘VERSION’ button. This will allow us to push a version of the blueprint template to the catalog. You can add a description and a “Change log” entry, and select if you want this version pushed to the catalog.

Next, navigate to the catalog in the left-hand navigation menu. From here, the tile to provision a Postgres database should now be visible if it has been successfully published from the previous step. I already have a PhotonOS tile in my catalog which is why you are seeing two entries. If you have not added any other blueprints, then the Postgres database will be the only one visible.

Click on the Request link in the tile to start a database deployment. Populate the necessary details. Take a note of the version number and ensure it matches the version number you provided when publishing to the catalog (I’ve gone through a few versions, so your number may be different). In this example, I need to provide a deployment name. The database name will be set to the same name as the deployment if not provided separately. This is done in the code. We can also check if the infrastructure policy list that we defined in the code is working as a dropdown list, as shown here.

You can also check that the Postgres Major numbers are working, by selecting a number outside the range. It should error as shown below. Similar behaviour should be visible in the major number inputted is less than the supported minimum:

With the details populated, click on the ‘SUBMIT’ button. From the Instances view (still under Build & Deploy), you should now see the database begin to deploy.

All going well, the database should also appear in the DSM UI:

And of course, it should also appear in the Databases view of the Namespace that was chosen for the provisioning. This is only true for Postgres database. As mentioned a few times now, MySQL is still not plumbed up into VCFA, but we do plan to have it included in a future release. From this screen, I can click on the database to see the full YAML manifest as well as retrieve the connection string for the database.

And that completes the deployment of a DSM Postgres database in VCFA using a blueprint template. And of course, you can repeat the same thing for MySQL and create a blueprint template for those database types as well. Then from the catalog, you may also request a MySQL database to be provisioned.

When the database is provisioned, you will see it in the “Instances” view, but you will not see it in the Databases view for the reasons mentioned previously. However, this might be a sufficient way to track MySQL deployments presently, without having to context switch into the DSM UI.

If you’d like to use the blueprints as a starting point, feel free to download them from this public Github repo.

The post DSM 9.0.1 – Using VCFA Blueprints to provision databases appeared first on CormacHogan.com.

Create a MySQL Data Service Policy

To begin, login onto the DSM appliance as root. From here we can query some existing Data Service Policy, primarily one that is used in VCF Automation when provisioning Postgres databases. Use the command kg get DataServicePolicies -n dsm-system. Here is a Postgres Data Service Policy which allows all tenants in all VCFA organizations (matchCriteria is ‘*’) to provision all versions (again, matchCriteria is ‘*’) of Postgres databases. The infrastructure policy uses a Supervisor namespace as the destination for provisioned databases. This YAML output can now be used as the basis for creating a new MySQL Data Service Policy.

root@dsm [ ~ ]# kg get DataServicePolicies pg-for-everybody -n dsm-system -o yaml
apiVersion: infrastructure.dataservices.vmware.com/v1alpha1
kind: DataServicePolicy
metadata:
  creationTimestamp: "2025-09-26T11:01:07Z"
  generation: 6
  labels:
    dsm.vmware.com/created-in: vcfa
  name: pg-for-everybody
  namespace: dsm-system
  resourceVersion: "4349"
  uid: 0dbb1fa7-ecf2-4810-a7eb-1038500c71ab
spec:
  description: ""
  matchCriteria:
  - key: vcfa.vmware.com/org
    operator: in
    values:
    - '*'
  postgresPolicy:
    allowedReplicas:
    - -1
    allowedVersions:
    - '*'
    common:
      allowedBackupLocations:
      - dsm-backups
      allowedInfrastructurePolicies:
      - dsm-ns-synn6
  serviceType: vmware-sql-postgres
status: {}

Using the above output, we can now proceed with crafting a VCFA Data Services Policy for MySQL. Here is such an example which is essentially a copy of the Postgres one above, but now giving all tenants in all projects in all organizations the ability to provision any version of MySQL.This can be fine-tuned to meet your own specific requirements, such as granting only tenants in certain ORGs with the ability to provision a MySQL database or allowing multiple backup location or allowing multiple infrastructure policies to be selected.

apiVersion: infrastructure.dataservices.vmware.com/v1alpha1
kind: DataServicePolicy
metadata:
  labels:
    dsm.vmware.com/created-in: vcfa
  name: mysql-for-everybody
  namespace: dsm-system
spec:
  description: ""
  matchCriteria:
  - key: vcfa.vmware.com/org
    operator: in
    values:
    - '*'
  mysqlPolicy:
    allowedMembers:
    - -1
    allowedVersions:
    - '*'
    common:
      allowedBackupLocations:
      - dsm-backups
      allowedInfrastructurePolicies:
      - dsm-ns-synn6
  serviceType: vmware-sql-mysql

Use kubectl (or the kg shortcut) in the root shell on the DSM appliance to apply this manifest and create the MySQL Data Service Policy.

Create a MySQL Database

With the Data Service Policy now in place, we can turn our attention to creating a new MySQL database in VCF Automation. This step requires access to the VCF CLI tool which is available via the Supervisor API URL, accessible from the Summary tab > Status window of any Namespace in the vSphere Client. In the Link to CLI Tools, click Open.

This will take you to the VCF Consumption CLI. Here you can download the VCF CLI tools to match your desktop Operating System. Once downloaded, we can proceed to the next step of building a MySQL database in VCFA using our previously created MySQL Data Service Policy. Note that you will also need access to a kubectl command on your desktop to complete the process.

My desktop is running a Windows OS. For a Windows desktop, download the .exe file. I renamed the downloaded exe to vcf.exe for simplicity. You will now need your tenant admin to generate a token for access to VCFA. The token can be generated by logging into the ORG portal as the tenant admin, and from the tenant admin User / Account Settings view in the VCFA UI, select the tab for tokens and create a new one if necessary.

Finally, you will need to have the Certificate Authority (CA) from VCF Automation, typically called the VCF Operations Fleet Management Locker CA. This can be retrieved by opening a browser to VCFA and retrieving it that way, or from the Certificates section of your VCF Operations UI. Once the CA has been successfully retrieved and stored locally on your desktop, run the following command to create a context for your Org using the VCF CLI. Once the context is created for the Org, contexts for the different projects and namespaces in the Org also become available. The -e is the endpoint which points at the VCF Automation URL. Also include type ‘cci’ for Cloud Consumption Interface.

PS C:\Users\Administrator\Downloads> .\vcf.exe context create -e https://flt-auto01.rainpole.io 
--api-token <token-value> --tenant-name cjh-org-01 
--ca-certificate '.\VCF Operations Fleet Management Locker CA.crt' 
--type cci
? Provide a name for the context:  cjh-org-01
Successfully logged into flt-auto01.rainpole.io

You have access to the following contexts:
   cjh-org-01
   cjh-org-01:dsm-ns-synn6:default-project
   cjh-org-01:tenant-ns-4qvxq:default-project

If the namespace context you wish to use is not in this list, you may need to
refresh the context again, or contact your cluster administrator.

To change context, use `vcf context use <context_name>`
[ok] successfully created context: cjh-org-01
[ok] successfully created context: cjh-org-01:dsm-ns-synn6:default-project
[ok] successfully created context: cjh-org-01:tenant-ns-4qvxq:default-project
PS C:\Users\Administrator\Downloads>

Next, run the following commands to create the MySQL database in VCFA. In this example, the database creation request is emanating from a tenant namespace called tenant-ns-4qvxq in the Org cjh-org-01. However,  the objects that back the databases are provisioned to a different namespace called dsm-ns-synn6 namespace as this is how the Infrastructure Policy has been created (if you are new to VCFA and DSM integration, this earlier blog post might be worth reviewing to get an understanding on infra policies). Now with access to the VCF Consumption CLI, you can set the context to “organization:namespace:project”, check the MySQL Data Service Policy (DSP) which we created earlier is in place (binding) which means that this namespace is associated with the MySQL DSP, and finally request the creation of a MySQL database using kubectl.

PS C:\Users\Administrator\Downloads> .\vcf.exe context use cjh-org-01:dsm-ns-synn6:default-project
[ok] Token is still active. Skipped the token refresh for context "cjh-org-01:dsm-ns-synn6:default-project"
[i] Successfully activated context 'cjh-org-01:dsm-ns-synn6:default-project' (Type: cloud-consumption-interface)
[i] Fetching recommended plugins for active context 'cjh-org-01:dsm-ns-synn6:default-project'...
[i] No image repository override information was found
[ok] All recommended plugins are already installed and up-to-date.

PS C:\Users\Administrator\Downloads> .\kubectl.exe get dataservicepolicybindings
NAME      AGE
binding   5d1h

PS C:\Users\Administrator\Downloads> cat .\mysql.yml
apiVersion: databases.dataservices.vmware.com/v1alpha1
kind: MySQLCluster
metadata:
  name: mysql-backup-tester
  namespace: tenant-ns-4qvxq
spec:
  backupConfig:
    backupRetentionDays: 30
    schedules:
    - name: default-full-backup
      schedule: 59 23 * * 6
      type: full
  backupLocation:
    name: dsm-backups
  infrastructurePolicy:
    name: dsm-ns-synn6
  maintenanceWindow:
    duration: 6h0m0s
    startDay: SATURDAY
    startTime: "22:59"
  members: 1
  storagePolicyName: vSAN Default Storage Policy
  storageSpace: 20Gi
  supportAsynchReplicas: true
  version: 8.0.41+vmware.v9.0.1.0
  vmClass:
    name:  best-effort-large

PS C:\Users\Administrator\Downloads> .\kubectl.exe apply -f .\mysql.yml
mysqlcluster.databases.dataservices.vmware.com/mysql-backup-tester created

As stated, this database is created from the tenant namespace called called tenant-ns-4qxvq in the organization called cjh-org-01 in VCFA. This is possible since this namespace has been granted permission to provision MySQL databases via the MySQL DSP. However, the databases are provisioned to a different namespace called dsm-ns-synn6 as this is defined in the infrastructure policy. Since MySQL is not yet plumbed up in VCF Automation Data Services, it will not appear in the databases view in VCFA. However it will appear in the DSM UI:

And it will also be visible in the vSphere Client as a VKS cluster resource pool and VM in the dsm-ns namespace:

And that is how to create MySQL databases via the VCF Consumption CLI integrated with VCF Automation.

The post DSM 9.0.1 – MySQL Database deployments through VCF Automation appeared first on CormacHogan.com.

First, login to the DSM UI as a DSM Administrator. Navigate to the Settings menu on the left hand side. Here you will find the Metrics Forwarding tab. Select that tab to begin the setup.

Click on the “Configure Metrics Forwarding” button, and begin to populate details regarding the VCF Operations environment, in particular the Cloud Proxy component. A cloud proxy collects data from the end-point environment and uploads it to VCF Operations. Set the type to VCF Operations, then set the URL to be “https://<fqdn-of-cloud-proxy>:8443/opensource/default/metric“. Client Certificate Authentication must be used to connect to VCF Operations. To use this method, you will need to retrieve the certificate and private key from the cloud proxy. This is currently only possible via the API. See my previous 9.0 VCF Ops integration post for details on how to retrieve this information. It is not necessary to provide a “Basic Auth” username and password when using Client Certificate Authentication.

Unless you have already added the Cloud Proxy Certificate Authority to the DSM Trusted Root Certificates, you cannot enable Skip CA Certificate Verification. If you do not skip the CA Certificate Verification, you will be prompted to accept the Certificate Authority (CA) when it is presented to you during the configuration. The Custom Headers sections is for customers who want to add them to help routing, authentication, traceability, etc. Otherwise this field, which is optional, can be left blank. Similarly, Timeout in seconds should be fine at the default value of 5, but this is tuneable.

Review the settings. Click Configure. If everything is configured correctly, then the following Ready status should be observed in the Metrics Forwarding view:

We can now turn our attention to viewing those metrics in VCF Operations. After approximately 15 minutes, metrics from DSM should be flowing into VCF Operations (the existing DSM databases will need to restart to redirect their metrics to the VCF Operations endpoint). In VCF Operations UI, do a search on the keyword “dsm” to find the metrics. Look for the dsmpostgres_GENERIC for Postgres database or dsmmysql_Generic for MySQL databases. These are the objects with the actual actual metrics:

Drilling into these objects by clicking on them, you can locate the metrics that are being sent via the DSM Postgres database:

You can now proceed with building dashboards to highlight the metrics from your database that you are most interested in. This is a much improved user experience than what we had in version 9.0. Thanks for reading this far. Hope you find this information useful. You can find all my DSM related posts at this link.

The post DSM 9.0.1 – VCF Operations 9.0.1 Integration Improvements appeared first on CormacHogan.com.