Next in our Extreme Performance Series mini-track, I’d like to highlight the following vCenter performance breakout.  Remember, you’ll want to attend the whole series to learn about performance across the stack.

VSVC5234 – Extreme Performance Series: vCenter of the Universe

Ravi Soundararajan, Principal Engineer, VMware
Justin King, Technical Marketing, VMware

Deploying vCenter server correctly can be a daunting task with many considerations and services. This advanced technical session will provide an understanding of the inner architecture and workings of vCenter to help the end user deploy and manage vCenter with the best performance in mind. Finally future directions will be discussed by product management and how they’ll positively impact scalability and performance.

While the abstract for this breakout might be light, I think we can all agree that when vCenter performs, administrators are happier people.  With so many virtualization functions dependent on vCenter, its performance needs to be well understood and managed.  So how do we ensure we get every last drop of performance out of it?  In this session, Ravi and Justin, two amazing vCenter gurus, will break down the architecture of vCenter, its current and future set of components, how far they can be scaled, and lastly, how you can tune its performance.  Additionally, learn about accessing the performance data it holds and how to exploit the programmatic interfaces with tools like StatsFeeder.  If you’ve ever worried about vCenter performance, don’t miss this session.

Excited to see you all there!

@vmMarkA

There will be a four part “Extreme Performance Series” set of breakouts that will allow the speakers to dive deep into their respective areas without overlapping of content.  This means you get to spend a whole breakout on an individual resource dimension: CPU/Memory Scheduling, Network, Storage and vCenter.  Dive deep into the architecture, new capabilities, recommend practices and troubleshooting of each resource dimension.  You’ll want to attend the whole series in order to learn about the complete stack.

I’ve created a blog post to highlight each session and why I’m excited about it.

VSVC4811 – Extreme Performance Series: Monster Virtual Machines

Seongbeom Kim, Performance Engineering, VMware
Peter Boone, GSS, VMware

Mission critical applications serving an entire organization represent the last hurdle to the organization looking to migrate its data center to a private cloud. Although, IT administrators realize the benefits of virtualization, they may be hesitant to virtualize the mission critical applications because of the resource demands of these applications and an inexplicable fear about the underlying hypervisor preventing the infrastructure from meeting the SLAs of these applications. vSphere, industry’s leading hypervisor platform, has been pushing the boundaries of infrastructure resources it can manage over various releases. Through its superior resource management capabilities, vSphere can easily scale horizontally to support many virtual machines (VMs) or vertically to support large VMs. Recently, vSphere extended the resource management envelope to such a level that even extreme resource-hungry applications such as databases can be run in VMs. In this talk, the speakers will give an overview of key vSphere features such as CPU and NUMA schedulers, and memory managers that propel monster VMs. The speakers will share their experimental results, experience gained, and lessons learned. Finally, the speakers will provide best practices to the audience for undertaking such an exercise. This talk should alleviate any hesitation to migrate the resource hungry, mission critical applications to a private cloud.

I can think of no better speakers than Seongbeom and Peter for this session.  Seongbeom’s daily efforts are about diagnosing and improving the performance of these resource dimensions.  Peter, an experienced performance guru in our support organization, and fellow Canadian, spends his days working on the front line with customer issues.  While the abstract above references Monster VMs a number of times, all the information in this breakout can be applied to environments of any size.  They explore processor technologies, how our schedulers work, NUMA and its performance implications, identifying and fixing scheduling contention, etc. so that next time you hear “my VM is running slow” you’ll have experience, tools and confidence to prove otherwise.

Excited to see you all there!

See the current VMworld 2013 content catalog here: https://vmworld2013.activeevents.com/connect/search.ww

Is there a good reason you don’t use VMware Auto Deploy?

Here at VMware we value our customers feedback and want to help make sure our product lines and features are in line with what is needed from your organization, as part of this we are trying to find out more details about how our customers use Auto Deploy, or if you don’t , how they don’t use Auto Deploy!

As part of this we have created a survey which will help prioritize efforts in the future and give us a clearer picture on how customers are or are not using Auto Deploy.

The survey takes you to different pages based upon your answers so please do not get scared by the number of pages at the top, this will quickly reduce and should take less than 5 minutes to complete.

As a thank you for filling this survey out, at the end you will have the chance to add your email address (optional) and be entered into a draw to receive 1 of 3 copies of VMware fusion, winners will be contacted after the end of the survey.

Thanks for taking the time to help make VMware products better.

Take the survey here

Well, like our friends from Down Under would say, “No Worries Mate”. What you are seeing here is a UI bug in the vSphere Windows Client. As you know, the vSphere Windows Client has been superseded by the new vSphere Web Client. But at the moment, it’s the main tool for configuration by those who connect to ESXi servers. With the vSphere Web Client being the current and future client user interface for vCenter Server managed objects and resources, the “old” vSphere Client may, at times, not be as current as we’d like.

In vSphere 5.1 we changed how we handled user permissions in ESXi. Prior to 5.1, it was done similar to how Unix does it. Usernames/passwords/groups all in different files in the /etc. directory. With 5.1, this was deprecated in favor of using vSphere API’s, Roles and Permissions. My colleague, Kyle Gleed, wrote a great blog post on this entitled “vSphere 5.1 – Full Admin Support of Named User Accounts”. As of 5.1, you can also use Named Accounts with full “root” privileges. (and like Kyle, I’d encourage you to add your hosts to an AD domain so you can use AD accounts and policies for those users)

Unfortunately, with these changes, the UI of the Windows Client was not updated to match and because the direction is the Web Client, the UI is not expected to be changed. Let me go into some detail about the changes and why the UI is not a concern.

Let’s take the example above. I have a user, TestUser. You can see that after I create the user, the checkbox is checked. (You will also note that the “Group” pull-down is empty as we are no longer using /etc/groups)

In the following videos we will go through some steps that show that the user does not have shell access unless they are in the Administrators group.

The first video shows that the user has the “Grant shell access” checkbox checked but because they are not in the Administrators group, they are unable to login via SSH to the ESXi server.

In the second video, I have added TestUser to the Administrators role. Upon doing that, TestUser can now SSH into the ESXi server. Easy enough, eh?

Now here’s something I found out when writing this blog. In the third video, I clone the Administrator role and assign TestUser to that role. When I try to SSH into the ESXi server, I’m denied. That’s because only users in the group Administrators can SSH into the ESXi server.

I hope that shows you that despite the UI bug, things are actually better by being able to leverage Role Based Access Controls (RBAC) at the ESXi level. Your ability to be more finely grained in what users can change is enhanced.

Further settings for SSH access are contained in the vSphere Hardening Guide. Some of those are:

• esxi-set-shell-interactive-timeout
• esxi-set-shell-timeout
• esxi-disable-ssh
• esxi-enable-lockdown-mode
• esxi-remove-authorized-keys
• esxi-verify-admin-group (if you are using Active Directory)

It’s good practice to use the shell timeout settings. Note that using lockdown mode does NOT disable SSH, nor does it prevent root users from accessing a host using SSH user authorization keys. More info can be found in the vSphere Hardening Guide.

If you have an Active Directory infrastructure, I’d highly recommend using the AD Directory Services. If only for dealing with password policies and user management via a centralized mechanism. Consistency is key!

To wrap up, let’s review what we’ve learned.

1. ESXi no longer uses /etc/group
2. User permissions are now governed by Roles and Permissions
3. The UI was not updated to reflect that change

Many thanks to my colleague William Lam who helped out in this blog.

Thanks for reading!

mike

Since VMware’s acquisition of Virsto earlier this year, many customers and folks in the community have expressed a great deal of interest in the product. Since so many folks have requested more information about the product, I’ve decide to write a series of in-depth blog articles that will discuss VMware Virsto’s capabilities, benefits, and targeted use cases for the product. VMware Virsto is a software-defined storage solution design to optimize the use of external block storage in vSphere virtual infrastructures. VMware Virsto enhances the use of external Storage Area Networks (SAN) by accelerating performance and increasing overall storage utilization. When considering the storage challenges that are faced today in virtual infrastructures, one of the primary concerns revolves around performance and space efficiency. Virtualized environments tend to be performance intensive and persistent with the presentation of random I/O.

It is no secret that block based storage platforms work best with sequential I/O and VMware Virsto addresses the challenge which currently exists in virtualized infrastructures by intercepting all of the randomized I/O at the vSphere Hypervizor level and writing them to dedicated write logs in a serialized format and later de-staged onto virtual disks on storage area networks (SAN). As a result of this design the product improves performance by accelerating random I/O and also eliminating the disk-sprawl caused by performance constraints. This is all achieved regardless of what the underlying block based storage platform type or vendor may be.

VMware Virsto introduces a level of flexibility and elasticity for storage resources in similar way to how vSphere hypervizor provide compute resources for servers by effectively allowing all aspects of the storage constructs and resource characteristics such as performance, capacity, and a number of different data services like snapshots, clones, thin-provisioning, etc. to be completely defined in software. The product complements and leverages the vSphere rich portfolio of management tools and workflows to provide and deliver rapid provisioning of high performance capabilities and space-efficient storage solution.

The services and capabilities delivered by VMware Virsto are focused on a virtual machine centric management model from a storage perspective with much a more intuitive approach. The management of block-based storage (FC, iSCSI) is based on logical unit number (LUN) level which is both complex and inefficient in a world where the VM are the key object. A VM-centric storage management model is not only more efficient, but much easier to understand for vSphere administrators who do not have deep storage backgrounds.

VMware Virsto is designed to manage logical volumes presented to it by heterogeneous block based storage platforms, and because of this there is no need for the product to be responsible for managing disk devices directly. It is assumed that the existence of a reliable and highly available storage subsystem with adequate RAID designs and configuration are implemented before volumes are presented to VMware Virsto. VMware Virsto sits on top of the RAID layer, and it is able to see all the presented and connected storage as a set of volumes. Any of the underlying volumes might represent individual disk devices, some RAID aggregation of disks, or entire intelligent storage arrays. The underlying storage could be heterogeneous, exhibiting various qualities of service, or it could be a set of homogeneous block based storage devices.

In short, VMware’s Virsto is a true software-define storage solution for existing vSphere virtualized infrastructures. It delivers virtual machine centric I/O performance optimization and efficient and agile data services. In Part Two of this blog series, I will cover the VMware Virsto architecture and construct definitions. In subsequent articles I will map out specific use cases for the product, including Virtualized Database, Tier 2, and Tier 3 Test/Dev workloads stored on block based storage.

For future notification on new articles around VMware storage virtualization solutions and products and more following me on Twitter: @PunchingClouds

Tomas Fojta, Michael Haines, Ray Budavari, Scott Harrison, Francois Misiak, Adrian Roberts, Raghvender Arni, Les Major, Jesse Schachter, Jagan Raghu

This month we have several trick and tips from everything from vCloud Director transfer storage to vCloud Connector Content Sync.  Without further delay…

vCloud Director and Alternate Authentication

Backstory:

vCloud Director 5.1 introduced support for the vCenter 5.1 Single-Sign-On service. It added a “Federation” option that allows you add SSO as a source for authentication – along side native support for Active Director and LDAP generally.

Q. Does vCloud Director v5.1.x work with Horizon?

A. vCloud Director 5.1 supports Horizon Workspace authentication for vCloud Organizations. See the whitepaper, Using VMware Horizon Workspace to Enable SSO in VMware vCloud Director 5.1.

vCloud Director and Certificate Keystores

Backstory:

The install of vCloud Director requires two certificates held in the certificates.ks store.  One is used to secure the connection to the vCloud Director administration portal, and the second one is used for any secured connection to a virtual machine – through what is referred to as the “Console Proxy”. In a publically accessible vCloud Director both FQDN would be registered in DNS, and you would configure the “Public Addresses” to match these FQDN. So that although the internal names of the vCloud Director servers (or cells) might be vcdcell01.corp.com and vcdcell02.corp.com – they would resolve to a load-balanced IP address to an FQDN like https://mycloud.corp.com.

When you setup a multiple vCloud Director servers the .bin installation can locate these certificates from the first instance you created, so there’s no need create certificates for every vCloud Director server you setup and install.

Q. After installing vCD, where can I find the certificate keystores?

A. If you are looking for the keystore file after the installation, then the installer saves this information to the /opt/vmware/vcloud-director/etc/responses.properties file (so it can be used as an input in a multi-cell deployment). Look for the user.keystore.path entry.

Alternatively, you can find it using the command:

find / -iname certificates.ks

Note: The keystore name of “certificates.ks” is merely a convention that you will see used in all our documentation. As such it’s just a convention and you may find that the name used for the keystore varies in your implementation.

It’s perhaps worth mention some best practices around the securing of your PKI infrastructure files. For example it wouldn’t be wise to store the files used to complete the request process in a user directory, but hold them in generic location available to those who need rights to manage them. Many of the ancillary files generate during the certificate request phase aren’t need on the vCloud Director cell once they have been imported into the keystore. So backing them and setting passwords on private keys generated is good way of making sure that wrong people don’t end up owning your certificates infrastructure.

vCloud Director Appliance Transfer Directory Size

Backstory:

The “transfer directory” is a staging array primarily used for uploading OVFs and media (.ISO/flp) files.  When you install vCloud Director you are asked to specify the shared directory location for this storage. It is recommended that you allocate a chunk of storage external to this. In my case I used my NetApp and its NFS storage to mount these storage to vCloud Director. The appliance is not designed (or supported) for production environments and its intended for PoC or homelab use.  As a result, it doesn’t support multi-cell.

Q. What is the default transfer directory size for the vCD 5.1 appliance?

A. The space for transfer area is around 21GB. It’s held on the / partition which is allocated 28GB of which about 5.8GB is used leaving 21GB in size.  The transfer location is located at /opt/vmware/vcloud-director/data/transfer

vCloud Director: MS SQL Server Cluster in vApp

Backstory:

VMware has support clustering technologies inside the VM all the way back to ESX 2.x. From ESX 2.5 onwards the recommendation back then was to use “Raw Device Mappings” (RDMs) to point to the shared and quorum volumes that make up a Microsoft Cluster.  Due to this history MSCS and RDM’s have had close relationship, although many of the original requirements – such as the support requirements to have the boot disks of the cluster on local storage were removed long ago. The important thing to remember with vCloud Director is that RDMs are not supported.

Q. I am trying to cluster MS SQL Server across 2 VMs in a vApp in vCD. Will this work?

A. Yes, but its recommended to use in-guest software iSCSI initiators supported by Microsoft.

vCloud Director: Copying vApps Across Catalogs

Backstory:

Each Organization with vCloud Director has a catalog for holding vApp Template and any other media the tenants require. These catalogs can be shared with people within the organization and they can also be “published” to every organization in the vCloud Director. This publishing is often done to make a standard collection of different builds available to every tenant. Often vCloud Director administrators create an “empty” organization with a published catalog purely for this purpose.

Q. Is it possible to copy a vApp from a private to a public catalog?

A. No, you can copy only from a public catalog to a private catalog, not the other way. As a workaround, you can give the org temporary catalog publishing privileges, make the private catalog public and then from the other org (which owns the genuine public catalog) initiate the copy. Of course, another option would be a more mundane download of the vApp to an OVF file in the private catalog, followed by uploading the OVF file to the public catalog – if this something that happens frequently

vCloud Director: Deploying Red Hat Enterprise Linux and Network Issues

Backstory:

Despite every attempt to standardize the deployment of VMs and guest operating systems within – differences and defaults with the GOS can and do result in unexpected outcomes – not least with Linux distributions.

Q. In vCD, when I deploy a RHEL template with a single vNIC, I receive a network interface name of eth1 or 2 or 3 and not eth0. Is this correct?

A. This is a known bug with RHEL. To fix this, go into the file /etc/udev/rules.d/70-persistant-net.rules which creates your ethX (Interfaces). RHEL always appends an entry to this file with a new MAC address. As a workaround, before you turn a VM into a template in vCD, delete all entries from that file, and delete all /etc/sysconfig/network-scripts/ifcfg-ethX files.

Q. When using vCD and RHEL customization, when RHEL is configured to use Network Manager it does not set the DNS entries in the /etc/resolv.conf file. How do I work around this?

A. Disable network manager and perform re-customization. Use the command:

ntsysv

from network manager service. Then Shutdown and power on the VM with force re-customization option through vCD.

SDRS and vCloud Director Media

Backstory:

vCloud Director 5.1 introduced support for Storage DRS. Storage DRS supports a maintenance mode which much like maintenance mode for compute DRS can trigger the evacuation of datastores.

Q. I have all my vCD media stored in a datastore with Storage DRS enabled. When I placed the datastore in maintenance mode, all VMs got migrated off but the media files remained. Is this to be expected?

A. Yes. Media files are objects unknown to vSphere infrastructure, so SDRS maintenance mode cannot migrate them.

Moving Out of the *(Any) Storage Profile

Backstory:

vCloud Director 5.1 introduced support for Storage Profiles – which allow you to classify your storage into different tiers or classes. The Storage Profiles are defined in the vSphere layer, and then are accessible from vCloud Director.

There is one Storage Profile that you will see within vCloud Director that you will not see or have to create within vSphere.  This is the ‘Any’ storage profile.  The ‘Any’ storage profile is a pseudo storage profile that refers to any available storage, including local attached storage.  This can be helpful in the event that you have ESX hosts that are not licensed to use Storage Profiles or if you have an older ESX host that does not support Storage Profiles.

Q. Can I move a vCD object out of the *(any) storage profile? For example, I noticed the Edge Gateway devices get placed there by default?

A. You can reconfigure vCD to stop using the *(any) storage profile using KB2045534.

vCNS 5.1 Edge Limits

Backstory:

When you define an Organization – you get the option to deploy an Edge Gateway at the same time. The Edge Gateway can be deployed into different sizes – by default vCloud Director offers “Compact” and “Full”.

Once deployed it can be upgraded from “compact” to “full” (but it cannot be downgraded). By right-clicking the Edge Gateway and selecting “Upgrade to Full Configuration”

If on the other hand you are using vCNS directly from vSphere, without vCloud Director – a right-click on vShield Manager allows you to change the size of the Edge Gateway from the “Actions” menu.

Note:  vCloud Director support the compact and large formats. However, it does not support the X-Large format. Remember though that the Edge Gateway can be deployed with vCloud Director, and in that case you will gain access to all options. Finally, remember downgrades are not supported of the Edge Gateway.

Q. What are the limits for vCNS 5.1 Edge devices? For example, maximum throughput based on Edge size (compact, large, X-large)?

A. The KB article KB2042799, vCloud Networking and Security 5.1 Edge configuration limits and throughput. There are two comparison tables in this KB that will give you an idea of the differences in terms of throughput:

vCNS Edge Ping

BackStory:

When an Organization or vApp Network is created there’s a good chance you will be using the vCNS Edge Gateway to keep the vApp(s) network configuration unique. The VMs within that network are likely to get their IP address from either DHCP range or from Static Pool of IP Addresses assigned to that pool. This IP assignment will include the “Default Gateway” address that is the “internal” interface of their respective Edge Gateway. Despite being on the same network, and internal – the Edge Gateway does not respond to pings.

Q. Does an Edge device respond when pinged?

A. Only if you create a rule for ICMP in the firewall settings. This is not on by default.

vCloud Connector: Scheduling Content Sync

Q. It appears that the synching of templates using the Content Sync feature in vCC takes place every 6 hours. Can this be scheduled?

A. No. The scheduler feature is a good one and we didn’t get around to implementing it for vCC 2.0. The default polling interval for synchronization is 6 hours. It can be changed, but the change is unsupported—to do this, edit the file on the vCC Server:

/usr/local/tcserver/vfabric-tc-server-standard/server/webapps/agent/WEB-INF/spring/appServlet/task.xml

Look for <property name=”jobExecutionIntervalInMinutes” value=”360″ />. After the change, restart the server.

vSphere Data Protection 5.1.11 Release Notes

vSphere Data Protection Advanced 5.1.21 Release Notes

Make sure you read the release notes in their entirety before upgrading. For more details on the upgrade process, please see “Upgrading the vSphere Data Protection Appliance” in the vSphere Data Protection Administration Guide.

More VDP resources:

vSphere Data Protection on vmware.com

vSphere Data Protection Advanced on vmware.com

vSphere Data Protection Evaluation Guide

@jhuntervmware

Webinar Registration

VMware vSphere Data Protection Advanced is a new edition of VMware’s backup and recovery lineup that extends the capabilities of the vSphere Data Protection software  included with most vSphere editions. With vSphere Data Protection Advanced, midsize customers can protect their environment with a virtual appliance that scales to 8TB of deduplicated data, using agent-less image-level backups or application-aware agents for MS SQL Server and Exchange. Attend this Webcast and learn how vSphere Data Protection Advanced enables you to:

• Dramatically reduce backup storage consumption and recovery times with a unique deduplication engine
• Save on storage and backup costs while improving availability and operational efficiency
• Simplify management for vSphere backup and recovery with a ”single pane of glass” solution designed specifically for seamless  integration with vSphere

@jhuntervmware

I am privileged to be a new addition to the Hands-on Labs team within Technical Marketing at VMware. I have been here just under 3 months, but I have been using our products for almost as long as we have had products, I am active in the community and have spent time in the field as both a customer and a partner. I hope that my background allows me to provide a unique perspective on what we do in my group. There were some things that I always wanted to know, and I’d like to share as much of that with you as I can.

As the new guy on this team, I have spent quite a bit of time understanding how the Hands-on Labs infrastructure is set up, where resources are located, and how we deploy labs to support various conferences, user groups, and the new 24×7 online activities.

Hands-on Labs Online

If you don’t know about the free HOL Online portal, you should stop reading and go sign up for an account right now. Seriously, point your browser to http://hol.vmware.com/ and get an account. Now. It’s in Public Beta and we’re signing people up continuously. Request an account and you should have one soon. Of course, I would appreciate it if you decided to come back here after you sign up.

Hands-on Labs @ VMworld

Most people know about the labs at VMworld. In fact, the labs have consistently been one of the highlights of the show for a large portion of attendees. I had been involved with the labs in the past as a content creator and presenter, so my new role is something that I am really excited about.

Several years back, we had two different types of labs: instructor-led labs and self-paced labs. Each type had its own benefits and drawbacks. For example, instructor-led labs were like classes and attendees had access to the people who actually developed the lab content because they tended to lead those sessions. Unfortunately, it was often very difficult to get into the lab sessions that you wanted because there were limited seats and sessions available. The capacity issue was addressed by the self-paced labs, and enhanced by a slick provisioning system that allowed any self-paced lab to be taken from any station in the pool. The drawback was that the people who created the labs were not always available when you wanted to take the lab.

As long as the labs were available, most attendees didn’t seem to mind, especially if we could direct them to a session where the lab’s topic was covered in more depth. However, our goal is to have as many subject-matter experts in the lab area as possible to answer questions as they arise. At conferences, we do our best to schedule lab resources so that one of the content contributors for each lab is on the floor during all posted lab hours.

Our current Hands-on Labs offering has evolved from this self-service model. At VMworld US 2012, we debuted our first BYOD capability. We expect to enhance that capability this year and provide several different types of lab experiences in addition to a whole batch of fresh, VMworld-exclusive content. It is no secret that we experienced some challenges with the labs at VMworld last year. We have listened to your feedback and made some changes. I firmly believe that your lab experience this year will be more satisfying.

Cloud!

When I think about it, even though our use case is somewhat unique, what we are doing here has many of the characteristics of “cloud”:

• On-demand Self-service: Anyone with an HOL online account can sign in and experience a lab. The environments are provisioned on-demand and presented for use. (Well, technically, we maintain some pre-deployed instances of each lab in order to save you time. This works similar to the way that VMware Horizon View pools work and is handled by our front-end application.)
• Measured Service: We don’t charge for this service, but the “cost” could be measured in minutes: when you enroll in a lab, you get to use the environment for set amount of time. When that time expires, your environment goes away. You can get a new one as many times as you would like, but nothing persists beyond the allocation.
• Leveraging Pooled Resources and Rapid Elasticity: Our labs are designed to be self-contained, deploy quickly, run for a finite period, then disappear. This is an incarnation of what I like to call the “Paper Towel” use case: need it, get it, use it, toss it. We deploy known, fixed blocks of capacity, which have been designed for a specific use case.

For our use case, availability is important, but not in the traditional sense:

• There are specific times during the year when we need 100% availability. At the VMworld and Partner Exchange conferences, attendees want the labs available during all of the hours that the labs are open — and more!
• The remainder of the year, people accessing our online portal would be inconvenienced if their lab disappeared due to a backend issue. However, it wouldn’t kill them to re-enroll and get a new copy. As long as we design for this kind of failover (i.e. not preserving any state), we are fine: we’re not running a reactor, mail server, or performing brain surgery here.

From a design and capacity perspective, we have to design for steady usage with some pretty massive spikes:

• Typical usage of labs via the HOL online portal’s public beta period averages 600-700 per week. We typically have 6-10 people taking labs concurrently unless there is an event of some kind going on. We have roughly 60 pre-staged copies of various labs deployed and waiting for you to use, with over 1000 VMs deployed within the tenant that services the HOL online portal.
• As for spikes, during a conference, we deploy and destroy an average of 8,000 VMs per hour!

What about Hardware?

All of that cloud stuff is well and good, but I’ll bet many of you want to know what’s behind the curtain: what gear do we use to run this environment, and what does this look like. I will have to save that for another post.

Installation

• Install vCloud Networking and Security Manager (aka vShield Manager) on a dedicated management cluster. Other components that get installed on this cluster are VMware vCenter Server, vCloud Director etc.
• vCloud Networking and Security Manager should be run on an ESXi host that is not affected by downtime, such as frequent reboots or maintenance mode operations. Use vSphere HA to increase the resilience of the Manager. Thus, a cluster with more than one ESXi host is recommended.
• Install vCloud Networking and Security App Firewall on all vSphere hosts within a cluster so that virtual machines remain protected as they migrate between vSphere hosts.
• The management interfaces of vCloud Networking and Security components should be placed in a common network, such as the vSphere management network. Manager requires IP connectivity to the vCenter Server, ESXi host, and App Firewall virtual machine. Refer the KB article for the network port requirements for vCloud Networking and Security. It is a best practice to separate management traffic from the production traffic.
• If the vCenter Server or vCenter Server database virtual machines are on the ESXi host on which you are installing App Firewall, migrate them to another host before installing App Firewall or exclude these virtual machines from vCloud Networking and Security App Firewall protection.
• Install VMware Tools on each Virtual Machine. The vCloud Networking and Security Manager collects the IP addresses of virtual machines from VMware Tools on each virtual machine. Use App Firewall SpoofGuard to authorize the IP addresses reported by VMware Tools to prevent spoofing.  With SpoofGuard use trust on first use to reduce the administrative overhead.

Firewall Policy Management

• Use vCenter containers (vApps, resource pools, port groups, etc.) and security groups (grouping of vApps, resource pools, port groups, vNICs etc.) instead of IP addresses for policy enforcement. This allows creating security policies that can follow virtual machines during the vMotion process, and are completely transparent to IP address changes and network renumbering. In addition, use of vCenter containers and security groups enable rules to be dynamic. When a new virtual machine joins the container or security group, the rules are applied automatically and not required to define new rules.
• Use service groups to combine multiple services to reduce the number of entries in the rule table.
• Ethernet rules control which higher-level protocols (like ARP, IPv6, PPP and so on) can communicate over L2. By assessing what communication is required between applications and each tier of the application, create Ethernet rules that block all unnecessary traffic with a default any to any allow at the end.  Ethernet rules are enforced before the General rules. When a packet is allowed by an Ethernet rule, it will be further inspected by General rules. When a packet is denied by Ethernet rule, General rules are not evaluated.
• General rules control the specific L3 traffic based on IP addresses, as well as L4 traffic based on TCP and UDP ports. Explicitly add rules to allow the communication required between applications and each tier of the application, with a default any to any deny at the end.
• Set a firewall rule to have L2 isolation between servers in a security group when applicable e.g. isolate one web server from another web server. This can prevent the spread of malware when one machine gets infected and provides PVLAN like capability, but more easily managed. App firewall provides better security than PVLANs particularly when combined with SpoofGuard.
• Set the Fail Safe mode to Block – in the remote event of App Firewall service virtual machine failure, this setting ensures to block the traffic to all virtual machines running on the host preventing any security vulnerability.
• App Firewall protects applications within the virtual datacenter, whereas Edge Firewall provides protection at the perimeter of the virtual datacenter. In multi-tenant deployments, each tenant would have separate Edge devices.  For firewalling between virtual machines within the same tenant use App Firewall. Whereas, for isolating traffic between tenants use Edge Firewall.
• In a multi-tenant deployment, App Firewall allows you to assign independent IP addresses to specific port groups. You can mark a port group as an independent namespace, and then the datacenter level firewall rules no longer apply to that port group. This is done automatically for VXLAN virtual wires i.e. a separate namespace created for each VXLAN network.

Day to Day Operations / Troubleshooting

• Regularly monitor the allowed/denied flows using Flow Monitoring to ensure that firewall rules are set up correctly. Use Flow Monitoring to audit network traffic, define and refine firewall policies, and identify threats to the network.
• Setup syslog servers on App Firewall for central logging. Enable logging on a per rule basis to send the Allow/Deny syslog messages to central syslog server. Use ‘Rule ID’ in the App Firewall rule table to correlate the syslog messages with the corresponding Firewall rules.
• Setup NTP to ensure accurate timestamps on log messages. All App Firewall instances use the NTP server configuration of the vCloud Networking and Security Manager.
• Use the comments field in each App Firewall rule to keep track of the changes.
• Use the Load History option to revert the vCloud Networking and Security App firewall configuration to a previous version. vCloud Networking and Security Manager saves the App firewall configuration each time new firewall rules are published and retains the previous ten configurations.
• Schedule periodic backup of vCloud Networking and Security Manager data, which can include configuration, events, and audit log tables.
• After creating a full backup of the vCloud Networking and Security Manager Database, shut down the virtual machine then take a snapshot or full clone of the virtual machine prior to any upgrades. Refer the KB article for additional information.

Get notification of these blogs and more vCloud Networking and Security information by following me on Twitter @vCloudNetSec.