Please note that Native Object Storage is still in tech preview, and is only available to some customers for testing. We hope to make it generally available to all our customers very soon.

Get an API Token

Login to your VCF Automation Organization as a tenant admin. In the top right hand corner, open My Account under the User Settings view. From, there select API Tokens, then click on New to create a new token. Store this token somewhere safe.

Login to the CCI context

With our token retrieve from VCF Automation, we can now login to the Cloud Consumption Interface, switch to the appropriate CCI context and query existing Object Stores using the VCF CLI. As you will see, I already have an existing Object Storage created via the UI.

$ vcf config init
[i] Some initialization of the CLI is required.
[i] Let's set things up for you.  This will just take a few seconds.
[i]
[i] Initialization done!
[i] ==
[ok] successfully initialized the config


$ vcf context create tenant-01-cci-context \
--endpoint https://flt-auto01.rainpole.io --type cci \
--api-token Ifi3Pka7E01oJzyeL3ltbfEqzxbgj5av --insecure-skip-tls-verify
? Provide Tenant Name:  tenant-01-org
Successfully logged into flt-auto01.rainpole.io

You have access to the following contexts:
   tenant-01-cci-context
   tenant-01-cci-context:tenant-01-ns-wmgrf:default-project

If the namespace context you wish to use is not in this list, you may need to
refresh the context again, or contact your cluster administrator.

To change context, use `vcf context use <context_name>`
[ok] successfully saved context: tenant-01-cci-context
[ok] successfully saved context: tenant-01-cci-context:tenant-01-ns-wmgrf:default-project


$ vcf context use tenant-01-cci-context
[ok] Token is still active. Skipped the token refresh for context "tenant-01-cci-context"
[i] Successfully activated context 'tenant-01-cci-context' (Type: cloud-consumption-interface)
[x] : failed to discover plugin sources from the system Harbor registry: unable to retrieve the complete list of server APIs: containerregistry.vmware.com/v1alpha1: no matches for containerregistry.vmware.com/v1alpha1, Resource=


$ kubectl get objectstores
NAME          REGION        VPC                   CAPACITY   PERFORMANCE CLASS   STATUS    AGE
tenant1-oss   cork-region   default-cork-region   1Ti        Small               Healthy   14d

Build a YAML Manifest for an Object Store

We can use the VCF Automation to create a sample Object Storage YAML manifest which can then be applied via the CCI. Login to the VCFA organisation as your organisation admin user, select Build & Deploy, then select Object Storage from the left-hand menu. Now select create a new object store, and populate the UI as normal. On the right-hand side, the Kubernetes Resource YAML for the Object Store is also populated as shown below.

Note  that in the UI, the certificate provided must be a wildcard (e.g., *.rainpole.io) to cover the IAM and storage endpoints, as well as the OSE. However, if HTTPS is disabled implying HTTP access only, you can provide a single certificate for the storage endpoint, avoiding the use of wildcard certs. The single certificate must contain 2 Subject Alternate Names (SANs), one for the <object-store-service-name>.<domain> and another for storage-<object-store-name>.<object-store-service-name>.<domain>. The naming convention is very important. Here is an example taken from the openssl.cnf file used in my environment, where the object store name is set to np-test and object store service name is set to nos. This results in the following SANs.

[ alt_names ] 
# Subject Alternate Names for server_cert 
DNS.1 = nos.rainpole.io 
DNS.2 = storage-np-test.nos.rainpole.io

The first entry (nos.rainpole.io) is used as domain name for the object store service. The second entry (storage-np-test.nos.rainpole.io) is used as the access endpoint FQDN for S3 access. Note that enableTLS is set to false in this manifest, so I do not need to provide a wildcard certificate. I only need to provide a valid certificate created with the two SANs mentioned previously for the storage endpoint, and of course, it must also include a private key. You can check the SAN entries are in the certificate by using the following openssl command:

% openssl x509 -in certs/ose-np-test.cert.pem -noout -text 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4157 (0x103d)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=CJH_CA, C=IE, ST=Cork, L=Cork, O=VCF Division, OU=OCTO, emailAddress=cormac.hogan@broadcom.com
        Validity
            Not Before: Oct 14 07:09:15 2026 GMT
            Not After : Oct 24 07:09:15 2027 GMT
        Subject: C=IE, ST=Cork, O=VCF Division, OU=OCTO, CN=storage-np-test.nos.rainpole.io, emailAddress=cormac.hogan@broadcom.com
        <<--snip-->>
                OpenSSL Generated Server Certificate
            X509v3 Subject Key Identifier: 
                F2:4E:A0:14:D0:A0:03:A6:05:D2:E2:F7:CF:99:DF:D6:3E:E3:BD:81
            X509v3 Subject Alternative Name: 
                DNS:nos.rainpole.io, DNS:storage-np-test.nos.rainpole.io
            X509v3 Authority Key Identifier: 
                keyid:85:5B:8C:EB:DE:31:79:69:29:80:A2:F2:38:78:AA:8E:7D:8A:33:3F

Here is an example of a YAML manifest, with the certificate and key entries truncated:

apiVersion: vsan.vmware.com/v1alpha1
kind: ObjectStore
metadata:
  name: np-test
spec:
  capacity: 100Gi
  cert: |
    -----BEGIN CERTIFICATE-----
    MIIGVzCCBD+gAwIBAgICEDwwDQYJKoZIhvcNAQELBQAwgYwxDzANBgNVBAMMBkNK
    ---<SNIP>---
    eNOCCp4u9pkp1Tp02lspRXCY8Qo8H/gX/I0PT/ic4j5kXOBcS81zy7C38w==
    -----END CERTIFICATE-----
  enableTLS: false
  key: |-
    -----BEGIN PRIVATE KEY-----
    MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC0CzYz0c2NCAQS
    ---<SNIP>---
    yvnT3LT9lYp1n/J8npzPMgnu
    -----END PRIVATE KEY-----
  performanceClass: Small
  regionID: urn:vcloud:region:c86a2585-26a5-41cd-9ef5-c90887a6d637
  regionName: cork-region
  vpcName: default-cork-region

If you need to retrieve the list of regions and their regionID, you can get it by running the following commands:

$ kubectl get regions
NAME
cork-region

$ kubectl get regions cork-region -o yaml
apiVersion: topology.cci.vmware.com/v1alpha2
kind: Region
metadata:
  annotations: {}
  finalizers: []
  labels: {}
  name: cork-region
  ownerReferences: []
  uid: c86a2585-26a5-41cd-9ef5-c90887a6d637
spec:
  description: ""
status:
  loadBalancerType: NSX_REGISTERED_AVI

Similarly, if you need to retrieve the name of the VPC, you can run the following:

$ kubectl get vpcs
NAME                  REGION        PRIVATE IPS     PROJECT   STATUS     AGE
default-cork-region   cork-region   172.30.0.0/16   <none>    Realized   36d

You now have everything you need to deploy a Native Object Storage from the API.

Apply the manifest, monitor the Object Storage creation

Now apply theYAML manifest created in the previous step. After applying the manifest, check the state of the object store. Note that in the example below, np-test is my new object store, whereas “tenant1-oss” is one that I previously created via the VCFA UI. Monitor the object stores from the CLI. It will change states from Init -> Deploying -> Degraded -> Healthy over the course of a few minutes.

$ kubectl apply -f tenant1-oss-np-test.yaml
objectstore.vsan.vmware.com/np-test created

$ kubectl get objectstores
NAME          REGION        VPC                   CAPACITY   PERFORMANCE CLASS   STATUS    AGE
np-test       cork-region   default-cork-region   100Gi      Small               Init      60s
tenant1-oss   cork-region   default-cork-region   1Ti        Small               Healthy   14d

$ kubectl get objectstores
NAME          REGION        VPC                   CAPACITY   PERFORMANCE CLASS   STATUS      AGE
np-test       cork-region   default-cork-region   100Gi      Small               Deploying   96s
tenant1-oss   cork-region   default-cork-region   1Ti        Small               Healthy     14d

$ kubectl get objectstores
NAME          REGION        VPC                   CAPACITY   PERFORMANCE CLASS   STATUS     AGE
np-test       cork-region   default-cork-region   100Gi      Small               Degraded   4m10s
tenant1-oss   cork-region   default-cork-region   1Ti        Small               Healthy    14d

$ kubectl get objectstores
NAME          REGION        VPC                   CAPACITY   PERFORMANCE CLASS   STATUS    AGE
np-test       cork-region   default-cork-region   100Gi      Small               Healthy   6m22s
tenant1-oss   cork-region   default-cork-region   1Ti        Small               Healthy   14d

You can monitor the creation of the PodsVMs which back the object store via the vSphere Client. Navigate to the namespace which is created on behalf of the object store (it will have the name of the object store), select the Compute view and monitor as the PodVMs start up. Here is an example from my lab.

Check the VCFA UI, tenant-01-org organization, and under Build & Deploy > Object Storage, your new object store should be visible. As per the CLI output previously, it should go through various states before eventually becoming Healthy.

The Object Store has been successfully deployed via the VCF CLI Cloud Consumption Interface. Note that the connections are now http rather than https, but that we were able to create this object store without the use of a wildcard certificate. Both the storage and the IAM are visible in the VCFA UI view of the object store. We can now drill down into the object store (as an org admin user) and create buckets, user credentials and bucket policies on our object store.

Summary

Object Storage is available via the VCF Automation Cloud Consumption Interface (CCI). Customers can manage Native Object Stores via the Kubernetes kubectl command line tool once a context is created. Even better, there is an option to avoid the use of wildcards in certificates for customers who are not in a position to use them in VCF 9.1.1 tech preview. The plan is to address this fully through both API and UI in time for GA.

The post Using the VCF CLI with CCI context to create Native Object Storage appeared first on CormacHogan.com.

Please note that Native Object Storage is still in tech preview, and is only available to some customers for testing. We hope to make it generally available to all our customers very soon.

Let’s begin by looking at the buckets that have been created for the different use-cases. In the DSM organization (dsm-org), I created a dedicated object store for DSM backups and logs. In the tenant-03-org, I also created another object store which the organization admin and users use for both VKS backups and Greenplum database external tables.  Here are the buckets created for the use-cases.

Buckets for VKS backups and Greenplum database external tables:

Buckets for DSM backups and logs:

VMware Data Services Manager Backups & Logs

Configuring the VMware Data Services Manager backup and log location is done via the DSM UI. Login as a DSM Admin. Navigate to Settings > Storage Settings. Then populate the entries for System Backup Repository UTL, System Log Repository URL and Database Backup Storage. The bucket names used below map to the bucket names created in the Native Object Store above. Note that DSM will require the CA used to create the Native Object Storage certificates to be added to its list of Trusted Root Certificates to ensure it can secure communication between itself and the Object Store.

This now enables users of DSM to generate log bundles and do point-in-time (PIT) restores of databases.

VKS Backups

When a Kubernetes (VKS) cluster is attached to VKS Management, Data Protection can be optionally enabled. In essence, this is adding the necessary Velero component add-ons to the VKS cluster. The VKSM UI can be used to add credentials and endpoints for backups/restores, mapping directly from the Native Object Store access control and buckets. Below is the VKSM UI in VCFA, showing a Target Location which maps back to the “vks-backups” bucket seen earlier.

Once correctly configured, Data Protection can be configures to take on-time backups or scheduled backups of your VKS and its objects. Below is a view of the Data Protection status of one of my VKS clusters by way of  an example:

Greenplum Database External Tables

Greenplum databases supports accessing external data with foreign tables. This includes data in S3 buckets. To achieve this, the Tanzu team have provided a complete set of instructions on how to use an external S3 bucket to write table data to, and read table data from. I won’t go through the complete setup – the instructions are very easy to follow in the official docs. I will show you a few simple psql commands to create some simple tables, one local one and ones which are used to read and write to S3. I will then write table data to the foreign table on S3, making a copy of the local table. I will drop the local table. I will recreate it and finally import the data from S3 back into the local table to show it working.

s3/s3.conf

First, there needs to be an s3/s3.conf file with permissions created for the Greenplum database user who is running the psql command. This configuration file holds the bucket credentials and other configuration information about how the bucket is accessed. This is a very simplified version shown here, with just accessid and secret. I am also asking it to ignore the lack of certificate information. This file must exist on all Greenplum database notes, both coordinators and segments.

[gpadmin@cjhgp01-2 ~]$ cat s3/s3.conf
[default]
accessid = "9O9GPGSIKJRNL6WRIDC1"
secret = "YBOYD9xnOrTC9bZExKPBismRYSpFlFgB1j27OPuJ"
verifycert = false

Create two foreign tables, one for S3 reads and one for S3 writes

These commands are all run after connecting to the Greenplum database. The connection string can be retrieved from the clusters view of Tanzu Data Intelligence UI, where the Greenplum database was provisioned from. Greenplum spins up a “sample” database which we can use to test with.

sample=# CREATE READABLE EXTERNAL TABLE S3FRUIT (id serial, name varchar(100), colour char(10), sku char(8))
LOCATION('s3://storage-tenant1-oss.rainpole.io:443/greenplum-ext-tables/dataset1/normal/
config=/home/gpadmin/s3/s3.conf')
FORMAT 'csv';
CREATE FOREIGN TABLE

sample=# CREATE WRITABLE EXTERNAL TABLE S3WRFRUIT (LIKE S3FRUIT)
LOCATION('s3://storage-tenant1-oss.rainpole.io:443/greenplum-ext-tables/dataset1/normal/
config=/home/gpadmin/s3/s3.conf')
FORMAT 'csv';
NOTICE:  table doesn't have 'DISTRIBUTED BY' clause, defaulting to distribution columns from LIKE table
CREATE FOREIGN TABLE

Create local table, populate with some sample data

sample=# create table fruit ( id serial, name varchar(100) not null, colour char(10), sku char(8) );
NOTICE:  Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'id' as the Greenplum Database data distribution key for this table.
HINT:  The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.
CREATE TABLE

sample=# insert into fruit (id, name, colour, sku) values (DEFAULT, 'Apple', 'Red', '4565'), (DEFAULT, 'Durian', 'Green', '4789');
INSERT 0 2

sample=# select * from fruit;
NOTICE:  One or more columns in the following table(s) do not have statistics: fruit
HINT:  For non-partitioned tables, run analyze <table_name>(<column_list>). For partitioned tables, run analyze rootpartition <table_name>(<column_list>). See log for columns missing statistics.
 id |  name  |   colour   |   sku
----+--------+------------+----------
  1 | Apple  | Red        | 4565
  2 | Durian | Green      | 4789
(2 rows)

Copy data from local table to foreign table on S3, then drop local table

sample=# insert into S3WRFRUIT
select * from fruit;
INSERT 0 2

sample=# select * from s3fruit;
 id |  name  |   colour   |   sku
----+--------+------------+----------
  2 | Durian | Green      | 4789
  1 | Apple  | Red        | 4565
(2 rows)

sample=# drop table fruit;
DROP TABLE

sample=# select * from s3fruit;
 id |  name  |   colour   |   sku
----+--------+------------+----------
  2 | Durian | Green      | 4789
  1 | Apple  | Red        | 4565
(2 rows)

sample=# select * from fruit;
ERROR:  relation "fruit" does not exist
LINE 1: select * from fruit;

Recreate local table, read data in from S3 foreign table

sample=# create table fruit ( id serial, name varchar(100) not null, colour char(10), sku char(8) );
NOTICE:  Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'id' as the Greenplum Database data distribution key for this table.
HINT:  The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.
CREATE TABLE

sample=# insert into fruit
sample-# select * from s3fruit;
INSERT 0 2

sample=# select * from fruit;
NOTICE:  One or more columns in the following table(s) do not have statistics: fruit
HINT:  For non-partitioned tables, run analyze <table_name>(<column_list>). For partitioned tables, run analyze rootpartition <table_name>(<column_list>). See log for columns missing statistics.
 id |  name  |   colour   |   sku
----+--------+------------+----------
  2 | Durian | Green      | 4789
  1 | Apple  | Red        | 4565
(2 rows)

So it would appear that we can successfully read and write data to foreign tables on S3 buckets from Greenplum. And to close, just to show you how this information is written to the S3 bucket in question, here is a screenshot of the contents:

Summary

Those are just a few of the potential use-cases that Native Object Storage can be used for. There are many others of course, some of which our customers have shared with us. Being able to have a landing ground for external tools and utilities is one such example. Being able to share data across different zones though bucket replication is another. And of course S3 object storage can play a significant role for AI workloads, in particular as a library for model files. Customers could download the model from  (for example) Hugging Face just once, scan it for viruses, and save the model onto an internal air-gapped S3 object store. Your AI containers then pull the model from there, safe in the knowledge that the model is validated and secure. We look forward to seeing many more use-cases once Native Object Storage is generally available.

The post Native Object Storage Use-Cases appeared first on CormacHogan.com.

Now you might be wondering why using the VCF CLI is necessary to do this step. The reason is this. As a DSM Admin, this customer has access to the DSM Gateway API from the DSM appliance. However, changes made to a database provisioned via VCF Automation through the DSM Gateway do not get synchronised to  the Supervisor. Thus it is possible that the change gets overridden by subsequent changes on the Supervisor. To make the change persistent, API changes should be made via the Supervisor. This synchronise changes back to DSM. However, to get access to the Supervisor to run kubectl API calls, you first need access to vCenter/vSphere. Not every DSM Admin has access to vCenter. Thus, we explored alternate methods, and found that changes to the database can also be made using using VCF CLI and a CCI context with a API token from VCF Automation. CCI, which stands for Cloud Consumption Interface, is a part of VCF Automation that enables dev-ops users to interact with the various services available in VCFA, such as VKS, VM Service and DSM.

First step is to get your token. Log into your organization in VCF Automation. In the top right hand corner, under User Settings, click My Account. From here you can generate your own API token.

Once the token is safely stored, we can commence with the VCF CLI. Note that the download instructions can be found in my previous post on this topic, and the steps have not changed since then. However, in this case, we won’t be accessing the Supervisor API using vSphere credentials, but the VCF Automation (VCFA) API using the token. Run the following VCF CLI command, choosing a name for the context and setting the type to CCI. I am not using a secure connection, but if you wanted to connect securely, there is an option to provide the Certificate Authority (CA) on the command line. The endpoint is my VCFA and not the Supervisor. You will be prompted to provide the name of the organization where the token was created by the user. In this case it is tenant-03-org.

$ vcf context create my-cci-context --endpoint https://flt-auto01.rainpole.io --type cci \
> --api-token xURVuNVWak0ItQa4JZnRLjhhDZH5Vx9V --insecure-skip-tls-verify

[i] Some initialization of the CLI is required.
[i] Let's set things up for you.  This will just take a few seconds.
[i]
[i] Initialization done!
[i] ==
? Provide Tenant Name:  tenant-03-org
Successfully logged into flt-auto01.rainpole.io
You have access to the following contexts:
   my-cci-context
   my-cci-context:tenant-03-ns-g8krt:default-project
If the namespace context you wish to use is not in this list, you may need to
refresh the context again, or contact your cluster administrator.
To change context, use `vcf context use <context_name>`
[ok] successfully saved context: my-cci-context
[ok] successfully saved context: my-cci-context:tenant-03-ns-g8krt:default-project
$

Now that we have successfully connected to the VCFA organization using the CCI, we need to set context:

$ vcf context use  my-cci-context:tenant-03-ns-g8krt:default-project
[ok] Token is still active. Skipped the token refresh for context "my-cci-context:tenant-03-ns-g8krt:default-project"
[i] Successfully activated context 'my-cci-context:tenant-03-ns-g8krt:default-project' (Type: cloud-consumption-interface)
[i] Using image repository hosts for context 192.168.41.71/vcf/vcf-cli-plugins/ga/plugin-inventory/v9-rel/plugin-inventory:latest
[i] Refreshing plugin inventory cache for "192.168.41.71/vcf/vcf-cli-plugins/ga/plugin-inventory/v9-rel/plugin-inventory:latest", this will take a few seconds.
[i] Fetching recommended plugins for active context 'my-cci-context:tenant-03-ns-g8krt:default-project'...
[i] No image repository override information was found
[i] Installing the following plugins recommended by context 'my-cci-context:tenant-03-ns-g8krt:default-project':
  NAME                INSTALLING
  addon               v3.7.0
  cluster             v3.7.0
  kubernetes-release  v3.7.0
  namespaces          v9.1.1
  package             v3.7.0
  registry-secret     v3.7.0
  vm                  v9.1.1
[i] Refreshing plugin inventory cache for "192.168.41.71/vcf/vcf-cli-plugins/ga/plugin-inventory/v9-rel/plugin-inventory:latest", this will take a few seconds.
[i] Refreshing plugin inventory cache for "192.168.41.71/vcf/vcf-cli-plugins/ga/plugin-inventory/v9-rel/plugin-inventory:latest", this will take a few seconds.
[i] Refreshing plugin inventory cache for "192.168.41.71/vcf/vcf-cli-plugins/ga/plugin-inventory/v9-rel/plugin-inventory:latest", this will take a few seconds.
[i] Refreshing plugin inventory cache for "192.168.41.71/vcf/vcf-cli-plugins/ga/plugin-inventory/v9-rel/plugin-inventory:latest", this will take a few seconds.
[i] Refreshing plugin inventory cache for "192.168.41.71/vcf/vcf-cli-plugins/ga/plugin-inventory/v9-rel/plugin-inventory:latest", this will take a few seconds.
[i] Refreshing plugin inventory cache for "192.168.41.71/vcf/vcf-cli-plugins/ga/plugin-inventory/v9-rel/plugin-inventory:latest", this will take a few seconds.
[i] Refreshing plugin inventory cache for "192.168.41.71/vcf/vcf-cli-plugins/ga/plugin-inventory/v9-rel/plugin-inventory:latest", this will take a few seconds.
[x] unable to automatically sync the plugins recommended by the active context. Please run 'vcf plugin sync' to sync plugins manually, error: [unable to list plugins from discovery source 'default': unable to fetch the inventory of discovery 'default' for plugins: plugins discovery image resolution failed. Please check that the repository image URL "192.168.41.71/vcf/vcf-cli-plugins/ga/plugin-inventory/v9-rel/plugin-inventory:latest" is correct: error getting the image digest: GET https://192.168.41.71/v2/vcf/vcf-cli-plugins/ga/plugin-inventory/v9-rel/plugin-inventory/manifests/latest: NOT_FOUND: resource not found: repo vcf/vcf-cli-plugins/ga/plugin-inventory/v9-rel/plugin-inventory, tag latest not found, unable to find plugin 'addon' matching version 'v3.7.0', unable to find plugin 'cluster' matching version 'v3.7.0', unable to find plugin 'kubernetes-release' matching version 'v3.7.0', unable to find plugin 'namespaces' matching version 'v9.1.1', unable to find plugin 'package' matching version 'v3.7.0', unable to find plugin 'registry-secret' matching version 'v3.7.0', unable to find plugin 'vm' matching version 'v9.1.1']
$

We can begin to query the databases, and even make our change. I can interact with the database and other objects using kubectl. In this example, I am going to modify the provisioned database and reduce the backup retention policy (spec.backupConfig.backupRetentionDays) from the default of 30 days down to 15 days. I list the PostgreSQL clusters in my organization, I check the current backup retention settings, I edit the retention value, wait for the cluster to reconcile before finally verifying that the change has taken effect. In the commands below, k is aliased to kubectl.

$ k get postgresclusters
NAME          STATUS   STORAGE   VERSION                AGE
pg-for-test   Ready    20Gi      18.4+vmware.v9.1.1.0   26h

$ k get postgresclusters pg-for-test -o jsonpath='{.spec.backupConfig.backupRetentionDays}{"\n"}'
30

$ k edit postgresclusters pg-for-test
postgrescluster.databases.dataservices.vmware.com/pg-for-test edited

$ k get postgresclusters
NAME          STATUS       STORAGE   VERSION                AGE
pg-for-test   InProgress   20Gi      18.4+vmware.v9.1.1.0   26h

$ k get postgresclusters pg-for-test
NAME          STATUS   STORAGE   VERSION                AGE
pg-for-test   Ready    20Gi      18.4+vmware.v9.1.1.0   26h

$ k get postgresclusters pg-for-test -o jsonpath='{.spec.backupConfig.backupRetentionDays}{"\n"}'
15

And if I check back on VCF Automation, selecting my database followed by Backup tab, I can see the reduced Retention value reflected in the UI.

So now we have a way of changing database configuration using the API, but without needing vCenter access or needing to log into the Supervisor.

The post Using the VCF CLI with CCI context to manage VCFA provisioned databases appeared first on CormacHogan.com.

First, let create a PostgreSQL cluster via VCF Automation version 9.1.1. I am going to build a PostgreSQL version 18 which is newly supported in this DSM 9.1.1 release.

During the setup, enable replication and add the configuration for the replication slot and replication user. There is a significant amount of text related to cross-region replication, when the primary and the replica are in different regions. Some firewall rules may need to be in place to enable communication between the primary and the read replica. If you are deploying the primary and the read replicas in different regions, pay particular attention to this note.

Once deployed, make a copy of the replication string. This will be needed to create the read replica.

We can now begin with the creation of a Read Replica. When it comes to the replication source section, populate the connection string saved from the primary, and add the replication slot. Note that the text related to SNAT IP address between the primary and replica is again mentioned.

Add the user connection string, the slot and any custom Certificate Authority that you might be using. The Extra Clients is only relevant if you are using the new DSM Network Security features, and creates rules in the NSX firewall to allow the IP addresses of these clients to have access to the database. See my post on DSM 9.1.1 Network Security for further information. Wait for the Read Replica to become ready:

And verify that the primary’s replication status is “Replicating”:

Now let’s test read only access to the replica. To do that, I made a simply psql script which created a table of ‘mock’ employee data and populated with some random values. Here is that script.

-- 1. Clean up existing table if you're re-running the script
DROP TABLE IF EXISTS mock_employees;

-- 2. Create the table structure
CREATE TABLE mock_employees (
    id SERIAL PRIMARY KEY,
    employee_code VARCHAR(10),
    age INT,
    salary NUMERIC(10, 2),
    department VARCHAR(50),
    is_active BOOLEAN,
    hire_date TIMESTAMP
);

-- 3. Generate and insert 10,000 rows of random data
INSERT INTO mock_employees (
    employee_code, 
    age, 
    salary, 
    department, 
    is_active, 
    hire_date
)
SELECT 
-- Generates a random UUID, converts to text, and grabs the first 8 characters
substr(gen_random_uuid()::text, 1, 8),

-- Generates a random age between 18 and 65
floor(random() * 48 + 18)::INT,

-- Generates a random salary between $30,000.00 and $130,000.00
(random() * 100000 + 30000)::NUMERIC(10, 2),

-- Picks a random department from an array of options
(ARRAY['Engineering', 'Sales', 'Marketing', 'HR', 'Finance', 'Legal'])[floor(random() * 6 + 1)],

-- Generates a random boolean (True/False)
random() > 0.5,

-- Generates a random timestamp somewhere within the last 5 years (1825 days)
NOW() - (random() * interval '1825 days')
FROM 
generate_series(1, 10000);

-- 4. Verify the row count and look at a sample of the data
SELECT count(*) FROM mock_employees;
SELECT * FROM mock_employees LIMIT 5;

I then ran the script as follows, suing the connection string to the primary:

$ psql postgresql://pgadmin:it0QI80D5Nm8x3LPUNYdiYQ6l88e0E@192.168.20.11:5432/tenant03-primary \
-f random.psql
DROP TABLE
CREATE TABLE
INSERT 0 10000
count
-------
10000
(1 row)

id | employee_code | age | salary | department | is_active | hire_date
----+---------------+-----+-----------+------------+-----------+----------------------------
1 | 42d9150f | 45 | 114332.05 | Sales | t | 2024-12-26 20:01:48.992993
2 | da648072 | 37 | 94975.66 | Legal | f | 2023-06-21 20:20:11.392203
3 | fa4fe54e | 31 | 108578.06 | Legal | f | 2026-06-14 10:41:03.661193
4 | 469205d7 | 55 | 92738.15 | Finance | t | 2024-01-16 03:31:14.870633
5 | 6ad4f01b | 65 | 121029.85 | HR | f | 2024-02-25 17:48:11.016703
(5 rows)

Afterwards, I ran another script every 30 seconds to keep updating the table. Here is that script:

-- 1. Generate and insert 100 rows of random data
INSERT INTO mock_employees (
    employee_code,
    age,
    salary,
    department,
    is_active,
    hire_date
)
SELECT
-- Generates a random UUID, converts to text, and grabs the first 8 characters
substr(gen_random_uuid()::text, 1, 8),

-- Generates a random age between 18 and 65
floor(random() * 48 + 18)::INT,

-- Generates a random salary between $30,000.00 and $130,000.00
(random() * 100000 + 30000)::NUMERIC(10, 2),

-- Picks a random department from an array of options
(ARRAY['Engineering', 'Sales', 'Marketing', 'HR', 'Finance', 'Legal'])[floor(random() * 6 + 1)],

-- Generates a random boolean (True/False)
random() > 0.5,

-- Generates a random timestamp somewhere within the last 5 years (1825 days)
NOW() - (random() * interval '1825 days')
FROM
generate_series(1, 100);

-- 4. Verify the row count and look at a sample of the data
SELECT count(*) FROM mock_employees;
SELECT * FROM mock_employees LIMIT 5;

And here is how I ran it every 30 seconds:

$ while true
do  
psql postgresql://pgadmin:it0QI80D5Nm8x3LPUNYdiYQ6l88e0E@192.168.20.11:5432/tenant03-primary \
-f random-100-new.psql 
sleep 30
done
INSERT 0 100
count
-------
10100
(1 row)
id | employee_code | age |  salary   | department | is_active |         hire_date
----+---------------+-----+-----------+------------+-----------+----------------------------
  1 | 42d9150f      |  45 | 114332.05 | Sales      | t         | 2024-12-26 20:01:48.992993
  2 | da648072      |  37 |  94975.66 | Legal      | f         | 2023-06-21 20:20:11.392203
  3 | fa4fe54e      |  31 | 108578.06 | Legal      | f         | 2026-06-14 10:41:03.661193
  4 | 469205d7      |  55 |  92738.15 | Finance    | t         | 2024-01-16 03:31:14.870633
  5 | 6ad4f01b      |  65 | 121029.85 | HR         | f         | 2024-02-25 17:48:11.016703
(5 rows)

INSERT 0 100
count
-------
10200
(1 row)

id | employee_code | age |  salary   | department | is_active |         hire_date
----+---------------+-----+-----------+------------+-----------+----------------------------
  1 | 42d9150f      |  45 | 114332.05 | Sales      | t         | 2024-12-26 20:01:48.992993
  2 | da648072      |  37 |  94975.66 | Legal      | f         | 2023-06-21 20:20:11.392203
  3 | fa4fe54e      |  31 | 108578.06 | Legal      | f         | 2026-06-14 10:41:03.661193
  4 | 469205d7      |  55 |  92738.15 | Finance    | t         | 2024-01-16 03:31:14.870633
  5 | 6ad4f01b      |  65 | 121029.85 | HR         | f         | 2024-02-25 17:48:11.016703
(5 rows)
...

Next, retrieve the connection string from the read replica. First, make sure we can read the replicated data.

$ psql postgresql://rep-user-1:91W2s6pWoT4jkEG72Gm33dRQHE5wDO@192.168.20.13:5432/tenant03-primary
psql (16.15 (Ubuntu 16.15-0ubuntu0.24.04.1), server 18.4 (VMware Postgres 18.4.0))
WARNING: psql major version 16, server major version 18.
Some psql features might not work.
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off)
Type "help" for help.

tenant03-primary=# SELECT count(*) FROM mock_employees;
count
-------
10900
(1 row)

tenant03-primary=# SELECT * FROM mock_employees LIMIT 5;
id  | employee_code | age | salary    | department | is_active | hire_date
----+---------------+-----+-----------+------------+-----------+----------------------------
1   | 42d9150f      | 45  | 114332.05 | Sales      | t         | 2024-12-26 20:01:48.992993
2   | da648072      | 37  | 94975.66  | Legal      | f         | 2023-06-21 20:20:11.392203
3   | fa4fe54e      | 31  | 108578.06 | Legal      | f         | 2026-06-14 10:41:03.661193
4   | 469205d7      | 55  | 92738.15  | Finance    | t         | 2024-01-16 03:31:14.870633
5   | 6ad4f01b      | 65  | 121029.85 | HR         | f         | 2024-02-25 17:48:11.016703
(5 rows)

That appears to be successful. The table exists, the number of entries seems to match the primary and the data looks reasonably good as well. What if we now try to write something on the read replica, such as trying to create a new table for example.

tenant03-primary=# CREATE TABLE new_mock_employees (id SERIAL PRIMARY KEY);
ERROR: cannot execute CREATE TABLE in a read-only transaction

As expected, this operation failed since this is a read replica and writes are not permitted. To summarise, those of you who are familiar with secondary databases from previous DSM versions should see similarities with this workflow. It is more or less identical to the creation of a secondary database for Disaster Recovery (DR) use-cases that we saw in prior releases. Once the read replica is created, it can be used by read-only workloads. This is a very useful feature for customers who wish to scale their databases for read intensive workloads.

As per the guidance in the official documentation, starting with version 9.1.1, Data Services Manager provisions each PostgreSQL cluster with support for up to 12 replication slots by default, controlled by the max_replication_slots parameter. Of these, two slots are always reserved for high availability (HA). The system holds the HA reservation regardless of whether the cluster is running as standalone or in HA mode, so that converting a standalone cluster to a primary cluster with read replicas can always succeed without changing the max_replication_slots parameter. This leaves 10 replication slots available for user workloads out of the box, implying you can have up to 10 read replicas.

The post Introducing PostgreSQL Read Replicas in DSM 9.1.1 appeared first on CormacHogan.com.

The first step is to gain SSH Access to the vCenter server. The steps are outlined in this Knowledge Base article on troubleshooting. Once you’ve managed to get a shell open on your vCenter server, the following commands can be used to retrieve the root password to gain access to the Supervisor.

Command> shell
Shell access is granted to root

root@sfo-w01-vc01 [ ~ ]# cd /usr/lib/vmware-wcp/
root@sfo-w01-vc01 [ /usr/lib/vmware-wcp ]# ./decryptK8Pwd.py
Read key from file
Connected to PSQL
Cluster: domain-c9:09ddf26c-f7e2-4333-b023-aa4a36024b79
IP: 192.168.1.200
PWD: AbCdEfGhIj
------------------------------------------------------------
root@sfo-w01-vc01 [ /usr/lib/vmware-wcp ]# sshpass -p 'AbCdEfGhIj' ssh -o \
stricthostkeychecking=no root@192.168.1.200
Last login: Fri Sep 18 10:45:31 2026 from 192.168.1.130
===============================================================================
EXERCISE EXTREME CAUTION when accessing a vSphere Supervisor control plane VM.
When running commands on a Supervisor control plane VM, you are acting with
elevated permissions that BYPASS PROTECTIONS. You may cause DATA LOSS, disrupt
running workloads, or PERMANENTLY DAMAGE the Supervisor, Supervisor Services
including the vSphere Kubernetes Service (VKS), or VKS Clusters.
Graceful recovery from changes made with these elevated permissions may not
be possible; it may be necessary to REDEPLOY THE ENTIRE SUPERVISOR, recreate
all VKS clusters, and redeploy all workloads.
===============================================================================
08:09:45 up 12 days, 22:03,  0 users,  load average: 7.72, 6.12, 5.87
tdnf update info not available yet!
root@423789b3f055e8e9961d56e456f323d6 [ ~ ]#

Once successfully logged onto the Supervisor, query the Data Service Manager deployment. In VCFA 9.1.1, the DSM is deploying using the VM Service feature. The follow commands can be used to check the name of the appliance and the opened ports on it. Note that on the Control Plane VM (Supervisor), the kubectl command is aliased to a simple ‘k’.

root@423789b3f055e8e9961d56e456f323d6 [ ~ ]# k get vm -A | grep dsm
svc-dsm-provider-operator-r77bq   dsm-appliance                     PoweredOn     10d


root@423789b3f055e8e9961d56e456f323d6 [ ~ ]# k get vmservice -n svc-dsm-provider-operator-r77bq
NAME                          TYPE           AGE
dsm-appliance-internal        ClusterIP      10d
dsm-appliance-lb              LoadBalancer   10d


root@423789b3f055e8e9961d56e456f323d6 [ ~ ]# k get vmservice -n svc-dsm-provider-operator-r77bq \
dsm-appliance-lb -o yaml
apiVersion: vmoperator.vmware.com/v1alpha5
kind: VirtualMachineService
metadata:
  creationTimestamp: "2026-09-10T13:25:35Z"
  finalizers:
  - vmoperator.vmware.com/virtualmachineservice
  generation: 1
  name: dsm-appliance-lb
  namespace: svc-dsm-provider-operator-r77bq
  ownerReferences:
  - apiVersion: provider.dataservices.vmware.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: DSMAppliance
    name: dsm-appliance
    uid: 14347c54-628d-4a35-ad8d-802d02ea23bc
  resourceVersion: "2065849"
  uid: a25607df-b720-4e58-bb86-cee6525ea006
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    dsm.vmware.com/appliance-name: dsm-appliance
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 192.168.41.72

By default, the DSM appliance is only servicing the https port, 443. This allows administrator to connect to the DSM UI via a browser using the loadBalancer IP address, but no other ports are accessible via that address. To allow SSH access (port 22), create a new YAML manifest similar to the above for an additional VM Service on the DSM Appliance. A sample manifest will look something similar to the following:

apiVersion: vmoperator.vmware.com/v1alpha5
kind: VirtualMachineService
metadata:
  name: dsm-appliance-lb-additional
  namespace: svc-dsm-provider-operator-r77bq
spec:
  ports:
  - name: ssh
    port: 22
    protocol: TCP
    targetPort: 22
  selector:
    dsm.vmware.com/appliance-name: dsm-appliance
  type: LoadBalancer

Apply the manifest. This creates a new VM Service. Note that the SSH port 22 is opened on a different IP address compared to the IP address associated with the https port above. However both redirect to the same DSM appliance:

root@423789b3f055e8e9961d56e456f323d6 [ ~ ]# k apply -f dsm-ssh.yaml
virtualmachineservice.vmoperator.vmware.com/dsm-appliance-lb-additional created


root@423789b3f055e8e9961d56e456f323d6 [ ~ ]# k get vmservice -n svc-dsm-provider-operator-r77bq
NAME                          TYPE            AGE
dsm-appliance-internal        ClusterIP       10d
dsm-appliance-lb              LoadBalancer    10d
dsm-appliance-lb-additional   LoadBalancer    14s


root@423789b3f055e8e9961d56e456f323d6 [ ~ ]# k get vmservice -n svc-dsm-provider-operator-r77bq dsm-appliance-lb-additional -o yaml
apiVersion: vmoperator.vmware.com/v1alpha5
kind: VirtualMachineService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"vmoperator.vmware.com/v1alpha5","kind":"VirtualMachineService","metadata":{"annotations":{},"name":"dsm-appliance-lb-additional","namespace":"svc-dsm-provider-operator-r77bq"},"spec":{"ports":[{"name":"ssh","port":22,"protocol":"TCP","targetPort":22}],"selector":{"dsm.vmware.com/appliance-name":"dsm-appliance"},"type":"LoadBalancer"}}
  creationTimestamp: "2026-09-21T08:27:34Z"
  finalizers:
  - vmoperator.vmware.com/virtualmachineservice
  generation: 1
  name: dsm-appliance-lb-additional
  namespace: svc-dsm-provider-operator-r77bq
  resourceVersion: "16164207"
  uid: ae4a417c-bedc-4d08-bd6b-b11264b47751
spec:
  ports:
  - name: ssh
    port: 22
    protocol: TCP
    targetPort: 22
  selector:
    dsm.vmware.com/appliance-name: dsm-appliance
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 192.168.41.73

At this point, SSH is opened to the DSM appliance via a new loadBalancer IP address. This means that we should be able to SSH to it. However, in order to be able to login as the root user, the root password must also be retrieved. This is help in a secret called dsm-appliance-root-cred in the same namespace. Here is just one example of how to retrieve it.

root@423789b3f055e8e9961d56e456f323d6 [ ~ ]# k get secrets -n svc-dsm-provider-operator-r77bq
NAME                                 TYPE                             DATA   AGE
dsm-appliance-root-cred              Opaque                           1      10d
dsm-po-values                        Opaque                           1      10d
dsm-provider-operator-pull-regcred   kubernetes.io/dockerconfigjson   1      10d


root@423789b3f055e8e9961d56e456f323d6 [ ~ ]# k get secrets -n svc-dsm-provider-operator-r77bq \
dsm-appliance-root-cred --template='{{.data.root_password}}' | base64 -d; echo
9L0b^9Lu%V@8mXl
root@423789b3f055e8e9961d56e456f323d6 [ ~ ]#

Now we have SSH opened, we have the IP address to reach the DSM appliance and we have the root password. Let’s see if we can successfully access it.

root@423789b3f055e8e9961d56e456f323d6 [ ~ ]# ssh root@192.168.41.73
Welcome to Photon 5.0 (\m) - Kernel \r (\l)
(root@192.168.41.73) Password: 9L0b^9Lu%V@8mXl

root@photon-6ca473c9efbb [ ~ ]# docker ps
CONTAINER ID   IMAGE                                                  COMMAND                  CREATED       STATUS       PORTS                      NAMES
dc797efaaa57   dsm-tsql-provisioner:9.1.1.0.25641961                  "/provisioner --conf…"   5 days ago    Up 5 days                               dsm-tsql-provisioner-service
77121de7aacd   vc-session-manager:9.1.1.0.25641961                    "/vc-session-manager…"   10 days ago   Up 10 days                              vc-session-manager-service
e155538f05d9   telegraf:v1.38.4_vmware.1                              "/bin/sh -c 'telegra…"   10 days ago   Up 10 days                              dsm-telegraf-service
c50cdede22b4   provider-update-service:9.1.1.0.25641960               "/bin/sh -c '/usr/ja…"   10 days ago   Up 10 days                              provider-update-service
8ecc8312b858   provider-monitoring-service:9.1.1.0.25641960           "/bin/sh -c '/usr/ja…"   10 days ago   Up 10 days                              monitoring-service
2ddc5884de06   registry:3                                             "/entrypoint.sh /etc…"   10 days ago   Up 10 days                              docker-registry
1297b75df54e   simplified-gateway:9.1.1.0.25641961                    "/gateway --v=5 --ds…"   10 days ago   Up 10 days   127.0.0.1:5443->6443/tcp   sgw-service
554efa07f087   dsm/cluster-api-controller:1.13.2_vmware.2             "/manager --leader-e…"   10 days ago   Up 10 days                              kubernetes-service-capi-1
aee34cb51ecf   dsm/kubeadm-bootstrap-controller:1.13.2_vmware.2       "/manager --leader-e…"   10 days ago   Up 10 days                              kubernetes-service-kadm-1
234b8a29a113   dsm/cluster-api-ipam-provider-in-cluster:1.1.0         "/manager --leader-e…"   10 days ago   Up 10 days                              kubernetes-service-caip-1
3d0d49c989fe   dsm/kubeadm-control-plane-controller:1.13.2_vmware.2   "/manager --leader-e…"   10 days ago   Up 10 days                              kubernetes-service-kcp-1
1718cea3e5e0   dsm/cluster-api-vsphere-controller:1.16.1_vmware.2     "/manager --leader-e…"   10 days ago   Up 10 days                              kubernetes-service-capv-1
cf767b7526af   kubernetes-service:9.1.1.0.25641963                    "/opt/moneta/kuberne…"   10 days ago   Up 10 days   0.0.0.0:6443->6443/tcp     kubernetes-service
a2b5b3972550   provider-service:9.1.1.0.25641960                      "/usr/bin/bash -c 'v…"   10 days ago   Up 10 days                              provider-service
3f4d4399ee9d   bitnami-influxdb:1.12.4-photon-5                       "/bin/sh -c 'influxd…"   10 days ago   Up 10 days                              influxdb-service
85df57a66487   provider-ui:9.1.1.0.25641967                           "/bin/sh -c '/usr/sb…"   10 days ago   Up 10 days   0.0.0.0:443->8443/tcp      provider-ui


root@photon-6ca473c9efbb [ ~ ]# kg get postgresclusters -A
NAMESPACE                   NAME                 STATUS   STORAGE   VERSION                AGE
7b6916-tenant-03-ns-g8krt   tenant-03-pg-db-01   Ready    20Gi      18.4+vmware.v9.1.1.0   10d

Everything looks to be working as expected. Remember that once you are finished with your troubleshooting on the DSM appliance, remove the VM Service that was created earlier. This will ensure that SSH access to the DSM appliance is once again removed. Retain the YAML manifest though, as you can re-apply it when you need to open SSH access once again.

The post How to enable SSH on DSM Appliance when deployed via VCFA 9.1.1 appeared first on CormacHogan.com.

Let’s use an LDAP user. In this case, my LDAP user is called tenant01. At the organisation level, this user is an org user with object store privileges. At the project level, this user has been given the role of Project Administrator in two projects in my organisation. The projects are called default-project and shared-project. This user also has a bucket in each project, built on the same object storage, proj-admin-bucket-01-def-proj (default-project) and proj-admin-bucket-02-shared-proj (shared-project). I am going to use the aws cli to demonstrate how it is not possible to copy from one bucket to another bucket, and then configure Native Object Storage bucket policies to show how it can be achieved. I will show you where to retrieve the account id shortly. I will also show you how to create the profiles for the user so that they can be used with the aws cli.

Initial setup

If using the aws cli, some configuration and credential files need to be created first. I am using an Ubuntu Linux jump-box, and have install the aws cli found here. I created a .aws/config and a .aws/credentials for my LDAP user tenant01. Note the relationship between the profile name used in the .aws/config and the entries in the credentials file. It is important to make these match. In the config file, the ca_bundle is optional. You can use –no-verify-ssl on the command line if you prefer. The CA file is the same CA that was used to sign the certificate for the object storage wildcard certificate when it was created in VCFA. The endpoint_url points to my object store. In the .aws/credentials, the key and secret can be obtain from the Native Object Storage UI in VCF Automation.

.aws/config

[profile tenant01-default-project]
region = cork-region
ca_bundle = /home/cormac/s3/ca-cert.pem
ignore_configure_endpoints_urls = true
endpoint_url = https://storage-tenant-01-obj-store-01.rainpole.io/

[profile tenant01-shared-project]
region = cork-region
ca_bundle = /home/cormac/s3/ca-cert.pem
ignore_configure_endpoints_urls = true
endpoint_url = https://storage-tenant-01-obj-store-01.rainpole.io/

.aws/credentials

[tenant01-default-project]
aws_access_key_id = TWM9F73VC7HDK9VUJ0ND
aws_secret_access_key = IXE9vVPHzzaqGXMvh2yOPLDBDsc33heHBPfwjYdy

[tenant01-shared-project]
aws_access_key_id = 5XYCB3GMJIPPBYEI3KDB
aws_secret_access_key = hWzoJV5GaQ3W1ClWJQctsjCnYcTSc8hB68iX2CVN

Test bucket access

We can now try to simple commands to list the contents of the buckets. We can start with the tenant01 LDAP user added to the default-project, and see if this user can use their default-project credentials to list the contents of their bucket in the default-project and then in the shared-project.

$ aws s3 ls s3://proj-admin-bucket-01-def-proj --summarize --human-readable --recursive \
--profile tenant01-default-project
2026-09-14 09:47:53 132 Bytes credentials_admin_20260910_151358.csv
2026-09-14 09:47:53 132 Bytes credentials_admin_20260910_152130.csv

Total Objects: 2
Total Size: 264 Bytes

$ aws s3 ls s3://proj-admin-bucket-02-shared-proj --summarize --human-readable --recursive \
--profile tenant01-default-project

aws: [ERROR]: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access denied

As expected, they can only list bucket contents in the. default-project as that is the project that the credentials are associated with. Let’s try this the other way around, and check if the LDAP user tenant01, who is also a project admin in the shared-project, can access the bucket contents in the default-project.

$ aws s3 ls s3://proj-admin-bucket-02-shared-proj --summarize --human-readable --recursive \
--profile tenant01-shared-project
2026-09-14 14:42:21 26.4 KiB supervisor-service-contour-legacy-v1.33.1_vmware.2-25276585.yml

Total Objects: 1
Total Size: 26.4 KiB

$ aws s3 ls s3://proj-admin-bucket-01-def-proj --summarize --human-readable --recursive \
--profile tenant01-shared-project

aws: [ERROR]: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access denied

So as expected, the same issue occurs. The tenant01 user who has credentials on the default-project cannot list objects in the shared-project since privileges on object storage are scoped at the project level. And vice-versa. The tenant01 user who has credentials on the shared-project cannot list objects in the default-project. Let’s see how we can build a policy that will allow the default-project user tenant01 access the proj-admin-bucket-02-shared-project bucket created in shared-project.

Build a bucket policy

To allow users from a remote project to access a bucket in a local bucket, a bucket policy must be created to allow it. The first thing you will need to retrieve before building a bucket policy is the account id of the project. We captured this in the table earlier. You can get this from the VCFA UI, as it is shown next to the Project name under the Native Object Storage view.

The default-project id is 38925014951642977:

The shared-project id is 67271507534621802:

You can use the aws cli to check that there is no existing bucket policy in place, making sure you use the correct credentials to do the query.

$ aws s3api get-bucket-policy –bucket proj-admin-bucket-02-shared-proj \
–profile tenant01-shared-project
aws: [ERROR]: An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist

We now want to set a bucket policy that allows users from the default-project ( account id beginning 389 ) to access a bucket (proj-admin-bucket-02-shared-project)in the shared-project. We can do this via the API /CLI or via Object Storage UI in VCF Automation. We want users from the remote project to be able to access the bucket read-only, so we are only allowing actions to list the bucket, get objects from the bucket and get the tags on these objects. We are not letting the remote users write anything to the bucket. The bucket policy, in JSON format, will look something like this:

{
    "Version": "2012-10-17",
    "Statement": [{
        "Sid": "AllowListBucket",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::38925014951642977:root"
        },
        "Action": "s3:ListBucket",
        "Resource": "arn:aws:s3:::proj-admin-bucket-02-shared-proj"
    }, {
        "Sid": "AllowGetObjects",
        "Effect": "Allow",
        "Principal": {
            "AWS": "38925014951642977"
        },
        "Action": ["s3:GetObject"],
        "Resource": "arn:aws:s3:::proj-admin-bucket-02-shared-proj/*"
    }, {
        "Sid": "AllowGetTags",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::38925014951642977:root"
        },
        "Action": ["s3:GetObjectTagging"],
        "Resource": "arn:aws:s3:::proj-admin-bucket-02-shared-proj"
    }]
}

Most of it is probably easy to understand, but the Principal might need a bit more explaining. Let’s break down the ARN in more detail:

• arn:aws: Amazon Resource Name (usually in the public AWS cloud).

• iam: The AWS service involved. In this case, Identity and Access Management.

• 38925014951642977: The Account ID. (Note: Standard AWS account IDs are 12 digits long but ours are 17 digits).

• root This does not literally mean the root user. Instead, in a policy context, it acts as a placeholder for the entire account. It basically delegates permission management to the administrator of that target account. It means: “I trust this account. The administrator of that account can now use their own IAM policies to grant their specific IAM users / roles to my S3 bucket.”

To create this policy via the API, run the following aws cli commands, where –policy file:// points to the JSON file above:

$ aws s3api put-bucket-policy --bucket proj-admin-bucket-02-shared-proj \
--policy file://enable-cross-bucket-iam-from-acc-38925014951642977-tenant01.json \
--profile tenant01-shared-project

If no error is returned, the policy has successfully taken effect. If you want to do this via the VCFA UI, login to the Organization where the Object Store has been created. Go to Build & Deploy > Object Storage. Select the project where you wish to apply the bucket policy, e.g., shared-project.  Click on the bucket to which the policy is to be applied. In this case, it is tenant-admin-bucket-02-shared-proj. Select Policy, and the Click Here to apply a bucket policy. Click JSON, and paste the bucket policy content created earlier.

Apply the bucket policy via the UI by clicking on Save. After saving, click Expand All to see the policy details. Note that this view will be available either way – if you applied the policy via the API or UI, it will appear here:

Now use the aws cli once more to see if it the bucket policy has taken effect.

$ aws s3api get-bucket-policy --bucket proj-admin-bucket-02-shared-proj \
--profile tenant01-shared-project 
{
"Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"AllowListBucket\",
\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"arn:aws:iam::38925014951642977:root\"},
\"Action\":\"s3:ListBucket\",\"Resource\":\"arn:aws:s3:::proj-admin-bucket-02-shared-proj\"},
{\"Sid\":\"AllowGetObjects\",\"Effect\":\"Allow\","Principal\":{\"AWS\":\"arn:aws:iam::38925014951642977:root\"},
\"Action\":[\"s3:GetObject\"],\"Resource\":\"arn:aws:s3:::proj-admin-bucket-02-shared-proj/*\"},
{\"Sid\":\"AllowGetObjectTags\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"arn:aws:iam::38925014951642977:root\"},
\"Action\":[\"s3:GetObjectTagging\"],\"Resource\":\"arn:aws:s3:::proj-admin-bucket-02-shared-proj/*\"}]}"
}

This looks good. Now see if the LDAP user tenant01, i.e., the project admin user from the default-project, can list the contents of the bucket that has been created on the same object storage but in a different project.

$ aws s3 ls s3://proj-admin-bucket-02-shared-proj --summarize \
--human-readable --recursive --profile tenant01-default-project
2026-09-14 14:42:21 26.4 KiB supervisor-service-contour-legacy-v1.33.1_vmware.2-25276585.yml

Total Objects: 1
Total Size: 26.4 KiB

Success! The user from the default-project can list the bucket contents of the shared-project bucket which resides in a different project, something this user could not do before the bucket policy was applied. Let’s now try to copy the contents of the bucket in the shared-project to a bucket in the default-project.

$ aws s3 cp s3://proj-admin-bucket-02-shared-proj s3://proj-admin-bucket-01-def-proj \
--recursive --profile tenant01-default-project
copy: s3://proj-admin-bucket-02-shared-proj/supervisor-service-contour-legacy-v1.33.1_vmware.2-25276585.yml to s3://proj-admin-bucket-01-def-proj/supervisor-service-contour-legacy-v1.33.1_vmware.2-25276585.yml

Let’s check if the file made it over to the default-project bucket:

$ aws s3 ls s3://proj-admin-bucket-01-def-proj --summarize --human-readable \
--recursive --profile tenant01-default-project
2026-09-14 09:47:53 132 Bytes credentials_admin_20260910_151358.csv
2026-09-14 09:47:53 132 Bytes credentials_admin_20260910_152130.csv
2026-09-16 09:06:38 26.4 KiB supervisor-service-contour-legacy-v1.33.1_vmware.2-25276585.yml

Total Objects: 3
Total Size: 26.7 KiB

Once again, success! If you have some issues, you can run some commands to check that access to the local bucket is working as expected. One useful command is checking if it is possible to upload a simple local file to the destination bucket from the jump-box using your locla project credentials. This will verify that the user has credentials to write to the local bucket.

$ aws s3 ls s3://proj-admin-bucket-01-def-proj --profile tenant01-default-project 
2026-09-14 09:47:53 132 Bytes credentials_admin_20260910_151358.csv
2026-09-14 09:47:53 132 Bytes credentials_admin_20260910_152130.csv
2026-09-16 09:06:38 26.4 KiB supervisor-service-contour-legacy-v1.33.1_vmware.2-25276585.yml

Total Objects: 3
Total Size: 26.7 KiB


$ echo "test" > test.txt

$ aws s3 cp test.txt s3://proj-admin-bucket-01-def-proj/ --profile tenant01-default-project
upload: ./test.txt to s3://proj-admin-bucket-01-def-proj/test3.txt

$ aws s3 ls s3://proj-admin-bucket-01-def-proj --profile tenant01-default-project
2026-09-14 09:47:53 132 Bytes credentials_admin_20260910_151358.csv
2026-09-14 09:47:53 132 Bytes credentials_admin_20260910_152130.csv
2026-09-16 09:06:38 26.4 KiB supervisor-service-contour-legacy-v1.33.1_vmware.2-25276585.yml
2026-09-16 09:11:41 5 Bytes test.txt

Total Objects: 4
Total Size: 26.7 KiB

Summary

Although still a tech preview in VMware Cloud Foundation version 9.1.1, Native Object Store has many production-ready features, especially around multi-tenancy. Whilst Object Storage is enabled at the scope of a Region, meaning that it is available to all organizations in a given region, the object stores themselves are organisation specific. Within an organisation, projects share the same underlying object store, but the access controls are set at the project level. As we have seen, an LDAP user who has been added to two projects may be allowed or denied access to the object store on a per project basis. Project Administrators decide which users are allowed to create, delete and access buckets, and the actions they can carry out on a bucket through the access control settings. Cross-Project bucket access can be also configured by a Project Admin through bucket policies, as seen above, to allow or deny remote access to buckets. Lots of nice granular controls in Native Object Storage.

The post Bucket Policies in VCFA 9.1.1 for Native Object Storage appeared first on CormacHogan.com.

1. As the Provider Admin, configure LDAP as an Identity Provider
2. As an Provider Admin, add the ‘Provider’ configured LDAP to an Organization
3. As an Org Admin, add LDAP user as an organization user
4. As an Org Admin, assign user a Project Admin role
5. As a Project Admin, observe Object Storage behaviour
6. As an Org Admin, create a new organization role to include additional object store privileges
7. As an Org Admin, change role of organization user (Project Admin)
8. As a Project Admin, login to organization and observe change in Object Storage behaviour
9. As an Org Admin, add a new LDAP user as organization user with additional object store privileges
10. As an Org Admin, assign Project Advanced User role to new LDAP user
11. As a Project Admin, assign Object Storage Access Controls to LDAP user
12. As an Project Advanced User, login to organization and observe Object Storage behaviour

At the end of this post, we should have a pretty good idea about how Native Object Storage access controls work for the different VCFA personas.

Step 1: Configure LDAP/Active Directory  on Provider

Begin by configuring LDAP as an Identity Provider for VCFA as the Provider Admin. You may get a ‘trust certificate’ popup similar to what I see here during testing. Once configured, it is worthwhile doing a quick test on some of the users to ensure you can query your LDAP Server / Active Directory successfully.

To do a user test, click on the Test icon, provide the password of the user used to configure LDAP in the first place, and then add the LDAP/AD user that you want to test. This will return a list of attribute and whether or not the are configured. If you are happy that LDAP is working, we can proceed to the Organization where Object Storage is configured. Below is an example of a Test output, displaying which user attributes are configured and which are not.

Step 2: Configure LDAP on Organization

Login to your Organization as the Provider Admin. Select the Administer tab, and then select Identity Providers. Next, select LDAP and then the Configure button, as shown below.

In the LDAP source selection list, choose VCF Automation system LDAP service, which is what was configured in step 1. Add a distinguished name for the Org. Save the settings.

Step 3: Import LDAP user as Organization User

Let’s import an LDAP user into our organization. You can do this as a Provider Admin, or you can log out and log in as the Org Admin. Return to the Administer section of the Organization but now select Access Control from the left hand menu list. At present there is only a single user, the tenant-03-admin. This is the first user that was added during the creation of the organization. Click on the Import Users button to add a new users from LDAP.

The Org Admin adds LDAP user ‘sadaf’. The Org Admin is also going to assign a role of “Organization User”. Click Import.

Step 4: Add LDAP user to Project as Project Admin

Now navigate to the Manage & Govern tab in the Organization. Select Projects, then click on the Project (e.g., default-project) that you wish to assign users to.

Click on Users, then Add Users. In the Users list, type in the LDAP User that was previously added to the organization, i.e., “sadaf”. Assign a Project Role. In this case, “sadaf” is assigned the role of Project Administrator. Click Add, and then make sure you click on the Save button located at the bottom of the window. I have missed this on a number of occasions, so be sure to do that step.

Check the users on the project and ensure that user ‘sadaf’ is listed as a Project Admin.

Looks good. Let’s proceed to trying to manage Native Object Storage as a Project Admin.

Step 5: Check Project Admin access to Native Object Storage

Log out of the organization as the provider admin / organization admin, and log back into this organization as ‘sadaf’, the Project Admin. From the Build & Deploy tab, select Object Storage. You should now observe something identical to this “Unable to Load Object Stores” message:

The reason for this is because, in tech preview in version 9.1.1, the “Organization User” role does not have sufficient privileges to either View or Manage object storage. We are going to have to create a new organization role with additional privileges to access the object storage. Let’s do that next.

Step 6: Create New Org Role with Object Storage Privileges

Log out of the tenant organization as the Project Admin and log back in as the Org Admin. Now, let’s take a look at the privileges associated with the Organization User. In the Organization, navigate to the Administer tab (which is only visible to Org Admins), and select Access Control once more from the left hand menu. Select Roles, then click on the Organization User to see the privileges/rights associated with that role. As you can see below there are no Object Store or vSAN Data Services privileges selected. Those are the ones needed for Object Storage.

We can now go ahead and clone the organization user role to a new role where we can edit the privileges. In the role listing, click on the 3 dots in front of the Organization User roles and select Clone.

Edit the new role, giving it a recognisable name, but more importantly, ensuring that the necessary Object Store and vSAN Data Service privileges are enabled. Note that in this case, I am selecting both management and view permissions. You may not want to do this, and instead only select View permissions for your Project Admins and Users. If you grant this role both sets of permissions, then anyone with this role will also be able to delete Object Stores in the Organisation, something you may not want to provide. Once the permsisions are selected, save this new Role.

Step 7: Assign new role to LDAP User

Staying in Access Control, select the Users view. Click the three dots before user ‘sadaf’ and select Edit. In the Assigned Roles section, remove the Organization User Role and add the new role with the object store privileges that you just cloned, in this case “Organization User with Object Store. Save the User settings.

Now log back into the Organization as the Project Admin ‘sadaf’ and see if the behaviour has changed with respect to Object Storage.

Step 8: Observe change in Object Storage behaviour

Now when Project Admin ‘sadaf’ logs into the organisation, navigates to Build & Deploy, and then select Object Storage from the left hand menu, she is immediately offered an access key and secret. This is much better than previous behaviour. Keep in mind that this is because ‘sadaf’ has already been added as a Project Admin to the default-project by the Org Admin, but now also has permissions to access to the object store. If multiple projects existed, and a different project other than default-project was selected, ‘sadaf’ would not have access to the object storage from that project unless the Org Admin adds ‘sadaf’ as a user to the project. ‘sadaf’ has only been given access to the default-project and no other projects. Access to the object store is scoped at the project level in VCF Automation.

After saving the key and secret somewhere safe, and closing the welcome popup, the Project Admin can proceed with other activities, such as bucket creation and assigning additional project users access to the object storage.

A Project Admin has access to Security Credentials, Access Control and Service Accounts in this Project. As a Project Admin, this user ‘sadaf’ can now control access to the Object Storage on behalf of other Project Users. Let’s look at that next.

Step 9: Import a new LDAP User as Project Advanced User

We have already seen how to do this. This will have to be done as the Org Admin (or Provider Admin). It cannot be done by a Project Admin, as the Project Admin does not have access to the Administer tab (as you can see in the previous screenshot in step 8). This time, when importing the user ‘paudie’ into the organization, ensure that the role is the org user + object storage privileges that we created earlier.

Step 10: Add new LDAP User as Project Advanced User

Next, add user “paudie” to the Project (default-project) as a Project Advanced User. This can be done by the Organization admin or as the Project admin.

Let’s login to the organisation and see if our newly added Project Advanced User “paudie” has access to the object store. The answer is no. They currently only have access to the Project, but not to Object Storage within the Project. Note that you are redirected to the Project Admin to resolve this access issue.

Step 11: Assign Object Storage Access Controls

Log out as Project Advanced User ‘paudie’, and login as Project Admin ‘sadaf’. Once logged in as a Project Admin, check the Access Controls associated with the Object Store. All policies and their users are displayed. As we can see, only the Org Admin (the first Org user created with the organisation) and user ”sadaf’ (Project Admin) have policies designed to give access to the object store in this project. Project Advanced User ‘paudie’ does not have any access to the object store.

Project Admin ‘sadaf’ can now go ahead and create some access rules for Project Advanced User ‘paudie’. We are not going to deal with granular rules here, but of course there are a range of actions that can be controlled, where we can grant or deny access to particular actions and to particular buckets. To keep things, simple, we are going to give the user ‘paudie’ access to all actions on all buckets on this object store. Give the policy a name, select the user to which it applies, and then setup your statement to allow or deny actions on buckets. Then click Create. Below is an example of an access control policy where ‘paudie’ is given access to all actions on all buckets on the object store in this project.

The policy will now be visible and can be viewed under Access Control. Note that this is only on the default-project only. This access control policy does not apply to other projects consuming the same object store.

Side note: In the Resources section, we placed a wildcard “*” in that field, giving user ‘paudie’ access to all buckets. If you simply wanted to give them access to a single bucket and its contents, replace the “*” with two ARN entries, e.g., arn:aws:s3:::your-bucket-name and arn:aws:s3:::your-bucket-name/*. This allows the user to have access to both the bucket and its contents. This is normally required for most use cases.

Step 12: Observe change in Object Storage behaviour

Let’s log in to the organization once again as user ‘paudie’ and see if access to the Object Storage has changed now that this user has been granted access control via a new policy.

Success. Project Advanced User ‘paudie’ can now access the object store, has access to buckets and can manage his own Security Credentials, i.e., create additional credentials. However, since this user is not a Project Admin, ‘paudie’ does not have the ability to manage or change the Access Controls, nor does he have access to Service Accounts. These controls are only available to the Project Admin. Thus, user ‘paudie’ is not able to onboard any more project users to enable them to access Object Storage. This has to be done by ‘sadaf’ who has Project Admin privileges (or by the Organization admin).

Summary

That completes the post. I hope it has gone some way to explain how VCF Automation roles and Project roles are used to implement Access Controls on Native Object Storage in the VCF 9.1.1 Technical Preview. This table should provide a useful recap. Note that many of the Org Admin tasks listed below could also be done by the Provider Admin:

The post Identity Providers and Access Controls in VCFA 9.1.1 for Native Object Storage appeared first on CormacHogan.com.

Other enhancements include setting an AD user as the  SQL Server admin rather than a SQL user, and support for gMSA – Group Managed Service Accounts – to perform group refreshes and group membership of users. The latter feature is interesting as gMSAs improves security. gMSAs automatically generating complex, 128-character passwords for the account and rotate it automatically every 30 days (by default). Using a gMSA in place of a normal AD account also get automatically granted permissions to manage its own Service Principal Names (SPNs). When the SQL Server engine starts, a gMSA can automatically register and unregister its SPN.

gMSA Support

Let’s look at how to set up a gMSA account during the creation of a SQL Server Instance through VCFA in more detail. This first screenshot shows where to create the SQL Server instances in VCF Automation. When you are logged into VCFA as a Provider Admin, navigate to the VCF Services > Data Services Manager in the navigation toolbar on the left hand side. As you can see, the user experience has changed quite a bit from the previous release, so take some time to familiarise yourself with it.

During the creation of a new SQL Server instance, you can now select to have a SQL user or an Active Directory user as the SQL Server administrator. If you choose a Windows Principal (AD user), provide the domain name as well as the user name, as shown below.

In the next section on Active Directory, a gMSA may be selected in place of an AD user account to handle database access. If gMSA is selected, then the AD Account Username must be populated with the gMSA ‘SamAccountName’. In many case, the SamAccountName has a trailing ‘$’ sign, and this must be included. gMSAs automatically have privileges to Write Service Principal Names so this field is automatically set to on as is not configurable. You can still use the account to Write DNS Names, but you must ensure that your privileged used added to the gMSA has this privilege in Active Directory.

If you want to test gMSA, here are the steps that I followed to create such as account in my active directory. These were run from a PowerShell window which I had opened with ‘Run as Administrator’ privileges:

1. Check that a Key Distribution Service (KDS) Root Key exists

Active Directory requires a Key Distribution Service (KDS) Root Key to generate encrypted gMSA passwords. Check to see if one exists. If it does not, you will have to create one.

PS C:\Users\Administrator> Get-KdsRootKey

AttributeOfWrongFormat :
KeyValue               : {74, 163, 251, 83...}
EffectiveTime          : 3/13/2026 11:30:05 PM
CreationTime           : 3/14/2026 9:30:05 AM
IsFormatValid          : True
DomainController       : CN=ARKHAM-DC01,OU=Domain Controllers,DC=arkham,DC=io
ServerConfiguration    : Microsoft.KeyDistributionService.Cmdlets.KdsServerConfiguration
KeyId                  : 6b591c9e-7676-59cf-6d44-bb745f0040a0
VersionNumber          : 1

2. Create a Security Group

PS C:\Users\Administrator> New-ADGroup -Name "CJH-gMSA-Hosts" `
-GroupScope Global -GroupCategory Security

3. Create the gMSA Identity

Create the actual gMSA account in Active Directory and assign the group permission to access its credentials. Note that the DNS Hostname can be anything in the following command:

PS C:\Users\Administrator> New-ADServiceAccount -Name "cjh_sql_gmsa" `
-DNSHostname "cjh_sql_gmsa.arkham.io" `
-PrincipalsAllowedToRetrieveManagedPassword "CJH-gMSA-Hosts"

4. Check that it the group managed service account was created successfully:

PS C:\Users\Administrator> Get-ADServiceAccount -Identity "cjh_sql_gmsa"
DistinguishedName : CN=cjh_sql_gmsa,CN=Managed Service Accounts,DC=arkham,DC=io
Enabled           : True
Name              : cjh_sql_gmsa
ObjectClass       : msDS-GroupManagedServiceAccount
ObjectGUID        : 890010fb-1b49-47d4-b2dc-552dc64e9bc0
SamAccountName    : cjh_sql_gmsa$
SID               : S-1-5-21-388335747-1197595944-2956950382-1608
UserPrincipalName :

PS C:\Users\Administrator> (Get-ADServiceAccount -Identity "cjh_sql_gmsa" `
-Properties "msDS-GroupMSAMembership").'msDS-GroupMSAMembership'.access
ActiveDirectoryRights : GenericAll
InheritanceType       : None
ObjectType            : 00000000-0000-0000-0000-000000000000
InheritedObjectType   : 00000000-0000-0000-0000-000000000000
ObjectFlags           : None
AccessControlType     : Allow
IdentityReference     : ARKHAM\CJH-gMSA-Hosts
IsInherited           : False
InheritanceFlags      : None
PropagationFlags      : None

5. Add your privileged AD user to the list of Group Members

PS C:\Users\Administrator> Add-ADGroupMember-Identity "CJH-gMSA-Hosts" -Members "cormac"

PS C:\Users\Administrator> Get-ADGroupMember-Identity "CJH-gMSA-Hosts"

distinguishedName : CN=cormac,CN=Users,DC=arkham,DC=io
name: cormac
objectClass : user
objectGUID: b5a2bd65-8832-469b-b278-8fa2fa6fcaea
SamAccountName: cormac
SID : S-1-5-21-388335747-1197595944-2956950382-1602

6. Test the gMSA.

This command should return True. If there is a misconfiguration, this should report back the reason why.

PS C:\Users\Administrator> Test-ADServiceAccount -Identity "cjh_sql_gmsa"
True

The account is now ready to be used when provisioning SQL Server instances via DSM Manager.

Encrypted Backup and Transparent Data Encryption

I already mentioned some of these new SQL Server features in my DSM version 9.1.1 announcement post, but some are worth repeating here as I know a number of customers have requested them. We have now validated Backup Encryption and TDE, Transparent Data Encryption, on DSM provisioned SQL Server databases. Both features are available through the API. Encrypted Backups can be activated by adding a PFX file to the SQL Server instance specification. This PFX (Personal Information Exchange) file is password-protected and contains the SSL/TLS public key certificate and its matching private key. It is added to the spec.serverConfig.certificates of the SQL Server instance, and then referenced via spec.serverConfig.backupEncryptionCertificate. All databases created on the SQL Server instance will now have encrypted backups if this is configured. Note that if you wish to restore an encrypted database to a new SQL Server instance, the same certificate used to encrypt the backup must be imported onto the target SQL Server instance.

Transparent Data Encryption also requires adding a certificate to the specification of the SQL Server instance, spec.serverConfig.certificates, as seen previously. Then, on the user database, the db_owner (database owner) can use TSQL commands to create an encryption key. Finally, the db_owner can turn on encryption by performing ALTER DATABASE .. SET ENCRYPTION ON.

Summary

I haven’t discussed some other important improvements such as support for SQL Server 2025 and on-demand backup for SQL Server databases. However, as mentioned at the outset, you will find the full set of enhancements in the release notes linked above. I am sure you will agree that this release contains plenty of SQL Server enhancements for those customers wishing to provide SQL Server Database as a Service (DBaaS) to their on-premises VMware Cloud Foundation customers.

The post SQL Server enhancements in Data Services Manager version 9.1.1 appeared first on CormacHogan.com.

By default, Net work Security is disabled. The VCFA Provider Admin will need to activate it. Navigate to VCF Services > Data Services. In the Data Services Manager menu, select Advanced. Here you will find the button to activate Network Security.

When the Activate button is clicked, a pop-up will appear explaining what Network Security is setting out to do. It is basically going to implement a set of rules so that only the tenant that requested the database is able to access “from that tenant”. It cannot be access from any other tenant, or from an external ip address outside of VCF. It will do this by implementing a set of gateway and distributed firewall rules on the dedicated DSM organization where the database instance is instantiated.

Now, do note that by default, Transit Gateway (TGW) Firewalls are disabled. Therefore either the DSM Org Admin or the Provider Admin will need to open the DSM Org and enable the Transit Gateway Firewall. A shortcut to this point is to simply click on the “Manage Rules” link found in the Advanced View of the Data Services Manager Network Security section show above. If you fail to enable the TGW firewall step, the gateway rules will not be implemented and will be left in an “In Progress” state, as shown here:

To turn on the Transit Gateway Firewall, select the Settings tab, select the region and turn the firewall on.

Once the Gateway Firewall is enabled, the state should change to Success and any existing DSM provisioned databases should have their ruleset added to the firewalls on the dedicated DSM Org. Below are the transit gateway rules which only allow the configured IP addresses (Supervisor Control Plane VMs and DSM Provider) to communicate to the K8s API server of the database, and only allow the SNAT’ed IP address of the tenant ORG to communicate to the database port.

So far, so good. Now let’s assume that someone outside of that tenant’s org wishes to access the database. Maybe it is an app running somewhere else on the infrastructure. In that case, all you have to do is to add the External IP Address to the database configuration, as shown here:

And once the database reconciles with the new external client ip address, a new Transit Gateway Rule will be in place to allow the connection to succeed. As shown below, this is added to the list of IP addresses that can access the Postgres database but not the list of IP addresses that are allowed to access the database’s K8s API server.

And to verify that we can now connect to the database from that IP address, run the following:

$ ip -f inet address show ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
altname enp2s1
inet 10.13.10.200/24 brd 10.13.10.255 scope global ens33
valid_lft forever preferred_lft forever

$ psql postgresql://pgadmin:43ux4ZwihNYooae00gUn5404ygLr0R@192.168.21.14:5432/ten2-pgdb
psql (16.15 (Ubuntu 16.15-0ubuntu0.24.04.1), server 16.14 (VMware Postgres 16.14.0))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off)
Type "help" for help.

ten2-pgdb=# \c
psql (16.15 (Ubuntu 16.15-0ubuntu0.24.04.1), server 16.14 (VMware Postgres 16.14.0))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off)
You are now connected to database "ten2-pgdb" as user "pgadmin".
ten2-pgdb=#

Everything looks good. The rules are allowing me to connect to the database from an external IP address.

A note about remote replication. If remote replication is configured on a database and the read replica and the primary instances are in different regions, you must manually authorise the network for connectivity. Edit the Primary database “Extra Clients” field like we did earlier and add the SNAT IP of the organization VPC where the read replica instance is created. Now perform the same action on the read replica database, adding the SNAT IP of the organization VPC where the primary instance was created as an “Extra Client”. This will enable data synchronisation and ready it for both read scale out and disaster recovery scenarios.

The post Data Services Manager version 9.1.1 Network Security appeared first on CormacHogan.com.

In this post, I wanted to share some of the setup and initial configuration steps so that you can see what’s involved. I’ll show you how a Provider Admin enables Object Storage on their VCF Private Cloud. Next, we will see how either the Provider Admin or the Org Admin creates the Object Stores. Finally, I’ll take you to the point where tenant users in a VCF Automation organization can begin to provision their own S3 compatible buckets. In a future post, we’ll look at how a Project Admin can configure access control to the object storage buckets for different project users and assign different levels of privileges to the different project users.

Be aware that by the time the service is generally available, the screenshots shown here may have changed. This post is simply to provide an overview, in the knowledge that the steps may be different in future releases.

Prep vCenter and ESXi

For the tech preview, you will need to enable some advanced options on both ESXi and vCenter to enable Object Storage. You won’t need this step when Native Object Storage is generally available, but for the tech preview, you will need to enable these settings manually.

1. ESXi – set /VSAN/VsanVBOSSEnabled to 1

[root@sfo01-w01-r01-esx01:~] esxcfg-advcfg -g /VSAN/VsanVBOSSEnabled
Value of VsanVBOSSEnabled is 0
[root@sfo01-w01-r01-esx01:~] esxcfg-advcfg -s 1 /VSAN/VsanVBOSSEnabled
Value of VsanVBOSSEnabled is 1

2. ESXi – restart vsanmgmtd

[root@sfo01-w01-r01-esx01:~] /etc/init.d/vsanmgmtd restart
Terminating vsanperfsvc with PID 2561654
vsanperfsvc stopped.
vsanperfsvc started.
[root@sfo01-w01-r01-esx01:~] /etc/init.d/vsanmgmtd status
vsanperfsvc is running
[root@sfo01-w01-r01-esx01:~]

3. vCenter – enable Advanced Setting config.vpxd.vsan.vboss.enabled

4. vCenter – restart vsan-health

root@sfo-w01-vc01 [ ~ ]# vmon-cli -r vsan-health
Completed Restart service request.

Download & Install Native Object Store Package

Tech preview customers will be provided with a run-book to step through how to download the necessary Native Object Storage package. You will need a Linux jump host. Once downloaded, the components need to be pushed up to VCF Automation VMSP Platform. Once that step is completed, tech preview customers can proceed with the installation of the Native Object Storage Service.

Note, however, that you will first need to create a Region object in VCF Automation. Native Object Storage is scoped at a region level, and one or more regions need to be selected when Native Object Storage is installed. The following section has some more specifics on Region settings.

To proceed with the installation, the VCFA Provider Admin used a new Native Object Store tile found under Services Management, as shown below. However, this tile is hidden in VCF 9.1.1 and only those customers participating in the tech preview will be provided with the steps to make it visible. Once the backend components for Object Storage are added to VCF, the VCF Automation Service Management tile for Native Object Storage can be used to do the registration and installation of the service at the Supervisor level.

With Native Object Storage installed, a new supervisor namespace called ‘svc-objectstore-xxx’ is created. In this namespace, the Native Object Storage operator is deployed, watching for requests from different VCFA organisations to build object stores.

The operator has two network services associated with it, compatibility-api and controller manager.

Once the install is complete, we can proceed with getting it ready for the tenant organizations.

VCFA Region Settings

I mentioned previously that Native Object Storage is scoped at the region level. Thus, at least one region must exist before starting the installation. The region is comprised of one or more Supervisors, backed by one or more vSphere clusters. There must be a vSAN cluster on at least one of the vSphere clusters to enable Native Object Storage. The main item to ensure is selected in the region settings is the vSAN Object Store Policy in the Storage Class settings. This is the storage class which is used to create object store volumes on vSAN, both for the metadata and for the data.

Note that there is no capability to choose which organization gets access to object storage in the 9.1.1 tech preview. If an organization is using a region quota from a region which is scoped to use the Native Object Storage, then that organization has the ability to create an object store.

Also note that if your region is a multi-zone, for example, the Supervisor is backed by multiple vSphere/vSAN clusters, the object store will only provision to the cluster placed in the first zone in the list. Tech Preview does not allow the provisioning of object storages across multiple zones.

VCFA Organization Settings

Once we have confirmed that the region does indeed include the vSAN Object Store Policy, we can proceed with the creation of the organizations. The organization region quota includes the vSAN Object Storage Policy.

There will probably be many questions about networking when it comes to Native Object Storage. Whilst different configuration are possible, I am going to keep the Object Store and the tenant on the same VPC. When creating this organization, I am selecting a network configuration whereby a number of default configuration items are automatically implemented, including VPC, Transit Gateway (TGW), SNAT and connectivity profile. I will continue to use this default VPC when creating the Object Store, as we will see later.

The final step in the Organization setup is to create a first user. In my case I am creating a first user who has the privileges of an Organisation Administrator. We will use this user later when we log onto the organisation we created.

When the Organisation is created, we can launch the Organisation portal directly from the Provider / System organization, automatically connecting to it as the Provider Admin. This approach does have some restrictions. In particular, whilst it should be possible for a Provider Admin to create Object Stores on behalf of organisation users, the Provider Admin should not be able to have access to buckets or bucket contents on the object store.

Create Object Store

As the Provider Admin in the tenant Organisation, I can select the Build & Deploy tab, and observe that there is now an Object Storage menu item on the left hand side. Click this should take me to the option of creating an object store, as shown below.

Let’s look at the information that must be provided when building an object store. Obviously, give the object store a name and select which region it is to be placed. The region quote from this region, which includes the vSAN Object Storage class will be used to place the volumes for the object store. You must select a VPC. In this case, it is the default VPC for the organisation but it could be a completely different VPC if you wish. The VPC must exist before you start the object store create process; it is not possible to create a new VPC in this workflow like we have with namespace creation. Set the size of the object store (100Gb minimum), and decide if you want to enable secure HTTPS access or not to the endpoints. If you do want secure access (recommended), you will need to provide a wildcard certificate (e.g., *.rainpole.io) and a private key to enable secure access on the object store endpoints (IAM and storage). Lastly, select a performance class (resources) for the object store.

This will initiate the creation of a new Supervisor namespace for the object store, based on the name of the object store. Initially you should observe the ‘telegraf’ and the ‘config manager’ vSphere pods starting up.

After a few moments, this should expand to 5 vSphere pods, and all should enter a running state if no issues are encountered.

A number of network services and endpoints for the object store should also be created in the namespace:

Checking back in VCFA, we should see the object store come online and healthy. Note that the IAM (Identity and Access Management) and storage endpoints should correlate to the LoadBalancer IP address of the Envoy service in the namespace network services view above. You should ensure that the object storage FQDN resolves to that IP address in your DNS.

As you are still connected to the organization as the Provider Admin, you can try to connect to the Object Store. You may need to accept the certificates depending on how they were created, and if your browser trusts the Certificate Authority used to build them.

But even after doing so, you should observe that your Provider Admin is not allowed to see any details regarding how the organization is using the object store. The Provider Admin cannot see any buckets or bucket contents, nor can they see Access Control info or any service accounts. This is by design. Let’s now try this step as an Org Admin.

Access Object Store as Org Admin

Log out as the Provider Admin, change organizations to the tenant organisation where the object store is created and login as the Org Admin using the first user credentials created when the Org was created.

Now when we connect, we can see the initial credentials pop up after accepting the certificates. This looks better than previous when we tried to use the Provider Admin to access the object store.

And finally we are in a position to create buckets and manage access control and additional security credentials to said buckets. In a future post, I will go into more detail regarding access controls, and what access different roles within a project have when it comes to object storage.

Summary

That completes the post. I hope it has given you an idea of how we have architected Native Object Storage, using vSAN, into VCF version 9.1.1. I will be following up with some additional post to go deeper into some of the aspects here, notably how to control who has access to the buckets from within an organisation. We will also look into some CLI (since Native Object Storage supports the standard S3 API, we can do lots of cool things via tools such as the aws cli, for example). Thanks for reading this far. I am sure you will agree that this is a very interesting feature in VCF 9.1.1.

The post A first look at Native Object Storage in VCF 9.1.1 appeared first on CormacHogan.com.