As mentioned in earlier post here, vCloud Networking and Security (vCNS) Manager is responsible for mapping the logical Layer 2 networks to multicast group addresses. If you provide less number of multicast group addresses than the logical layer 2 networks, vCNS manager will assign the logical layer 2 networks to multicast addresses in a round robin fashion. For example, if there are 4 logical L2 networks (A1,A2,A3,A4) and 2 multicast group addresses (M1, M2), Logical networks A1 and A3 will be mapped to multicast group address M1 while A2 and A4 are mapped to M2.

Let’s take a look at the packet flows to understand any security or performance impact of mapping multiple logical networks on one multicast group. To simplify the drawing, as shown below, we will use a two logical Layer 2 network deployment with one multicast group address.

Example – Two logical networks mapped to one multicast group address

As shown in the diagram above, both logical networks have virtual machines connected. Virtual machine MAC1 and MAC2 are on VXLAN 5001 (green vwire) and virtual machines MAC3 and MAC4 are on VXLAN 5002 (red vwire). When the virtual machines are powered on, IGMP join requests to multicast group address 239.1.1.100 are sent out by the respective VTEPs on the host.

IGMP packet flows

The diagram above shows the packet flows

1. Virtual Machine MAC1 connected to VXLAN 5001 is powered on
2. IGMP join message to multicast group address 239.1.1.100 is sent out by VTEP on Host 1
3. Virtual Machine MAC 4 connected to VXLAN 5002 is powered on
4. IGMP join message to multicast group address 239.1.1.100 is sent out by VTEP on Host 3

To show the security implication in such deployment, we will take an example of broadcast traffic generated by virtual machine MAC 1. The virtual machine is connected to the logical network VXLAN 5001 (green vwire). The broadcast packet generated by this virtual machine should only be received by other virtual machines connected to VXLAN 5001 and not by virtual machines on VXLAN 5002.

Broadcast packet flows

As shown in the diagram above the packets flow as follows

1. Virtual Machine MAC1 sends a broadcast packet
2. VTEP on Host 1 encapsulates the packet in UDP header with destination IP address as the multicast group address 239.1.1.100.
3. The physical network delivers the packet to all the hosts that joined the multicast group 239.1.1.100.
4. The VTEP on the Host 4, after receiving the packet, checks if the VXLAN ID matches “5001” or “5002”. In this case the packet is sent from the virtual machine connected to the logical network with VXLAN 5001 (green vwire), and thus the packet will be only delivered to the virtual machine (MAC2) connected on that network.
5. The VTEPs on Host 2 and Host 3 will also receive the packet, because those hosts had also joined the multicast group 239.1.1.100. However, after VTEP checks that the packet is only destined to virtual machines connected to VXLAN 5001 (green vwire) the packets are dropped.

Even if the physical network delivers the broadcast packet from one logical network to all VTEPs, the individual VTEP on the Host do not forward packets unless they are destined to a logical network identified by the encapsulation header, and virtual machines are connected to the logical network on that Host. In this example, the VTEPs on Host 2 and Host 3 don’t have any virtual machines connected to logical network VXLAN 5001, and thus broadcast packet is not forwarded. The broadcast traffic on one logical network is not seen on the other logical network, even if the multicast group address is the same. So, there shouldn’t be any additional security concerns because of mapping multiple logical Layer2 networks to one multicast group address.

In terms of performance though, multicast traffic is suboptimal in such deployments. As we see in this example, Host 2 and Host 3 don’t have any virtual machines connected to logical network VXLAN 5001, but they still receive the broadcast traffic. This is because those hosts have virtual machines connected to logical network VXLAN 5002, and that logical network is associated with same multicast group address as VXLAN 5001. The physical network only knows that Host 2 and Host 3 are interested in listening to the same conversation as Host 1 and Host 4.

I hope this post clarifies the impact of mapping multiple logical networks to same multicast group address.

In the next post I will cover how VTEPs build forwarding table with virtual machine MAC address and associated VTEP IP address entries.

Here are the links to Part 1, Part 2, Part 3

Get notification of these blogs postings and more VMware Networking information by following me on Twitter:  @VMWNetworking

EMC World

What they demonstrated was Apache Hadoop running on Serengeti on vSphere. What attendees saw was some innovative thinking about how to get more mileage out of their data as well as their datacenter.

Why Virtualize Hadoop?

According to Gelsinger, over 500,000 Hadoop installations exist today on bare metal servers. Along with West’s demo, Gelsinger dove hard into explaining why each of these deployment teams should rethink that decision.

The first benefit users will see is setup is a fraction of what it is for setting it up the old fashioned way. Even the most seasoned administrators realizes a manual set up of a Hadoop cluster takes hours if not days to establish. With virtualization, a virtual machine image can be brought up in about six minutes.

Beyond that, as Serengeti Product Marketing Manager Joe Russell explained in an interview last week, by placing your Hadoop clusters on virtualized infrastructure, you have the opportunity to separate the data storage from the compute processing.  By treating them as independent entities, VMware shows that you optimize these separately.

Of course, for performance reasons, Serengeti knows to preserve data locality and keep the compute and data nodes collocated with each other, but the simple act of breaking them apart means that it is easier to scale up, scale down or even deploy completely different compute clusters on to the same data. Now, when you need more compute power to speed up a job, you can just add it.

It also means that you can reuse the same HDFS and apply completely different compute logic to the to the data, basically establishing multi-tenancy for the data through a shared file system. You don’t even have to use the same distribution of Hadoop. If you want to deploy Cloudera and MapR to the same data, you can. This means data can become multi-purpose, and eliminates redundant storage in the deployment footprint.

But West wasn’t finished there. He also demonstrated how to gracefully prioritize workloads. Using Serengeti baked into VMware’s vSphere and vCenter, big data administrators can assign priorities for their workloads and compute resources can be automatically allocated to the right set of compute processing.

So, if you are using MapR to run a recommendations engine for your website that helps target product recommendations to individual customers browsing your site, you can make sure that process is treated like a first-class citizen. Since this functionality is core to the business and is known to be a money-maker, we want to make sure this job always finishes first, or finishes within its promised service levels. Serengeti let’s you set that priority and automates resource allocation for you.

But it doesn’t prevent other jobs from accessing that data. For instance, a data scientist may have additional ideas for improvements to your recommendation engine, or may try to find some other product sales patterns that could provide useful insights to the business later on.  These data scientists could deploy a new compute node against the active data and get going right away. By setting the experimentation node up to be a secondary priority in the overall cluster, as demand for the higher priority job grows, compute resources will be gracefully starved from the second priority data experiment. Serengeti will know to do an orderly shutdown of TaskTracker nodes, and reassign tasks in the queue to a different queue.

At this point, the job will take longer to complete, but your data scientists will still make progress as long as they have any unused capacity left for compute. Similarly, as shopping demand dies overnight while your customers are sleeping, more resources can be opened up to complete the secondary job.

In other words, your compute processes for Hadoop just got elastic and multi-tenant.

The Result: Smarter Hadoop

In the era of big data, Gelsinger and West hit home with many of the attendees. By virtualizing Hadoop, companies open a window to get faster time-to-insight and at the same time optimize their data center to use hardware more effectively and prioritize performance. With benefits falling squarely in the “less work” and “better ROI”, I expect many in the crowd today to go home and rethink their big data strategy around virtualization.

Ideally, one logical Layer 2 network is associated with one multicast group address. Sixteen million logical Layer 2 networks can be identified in VXLAN, using 24 bit field in the encapsulation header, but the multicast group addresses are limited (224.0.0.0 to 239.255.255.255). In some scenarios it might not be possible to have one to one mapping of a logical Layer 2 network to multicast group address. In such scenarios the vCloud Networking and Security Manager maps multiple logical networks to a multicast group address. After the discussion on the association of multicast group to logical network, let’s take a look at some details on the logical network properties.

Logical Layer 2 networks can span across all the hosts managed by a vCenter Server. Virtual machines connected to a logical network behave as if they are connected to a single broadcast domain (equivalent to same VLAN). For example, in the diagram below, VXLAN 5001 is a logical Layer 2 network that spans across four hosts. Virtual machines running on Host 1 and Host 4 are connected to same logical network (VXLAN 5001).

The question is how broadcast traffic is handled on the logical network. Any broadcast packet from a device connected to the logical network should reach all the devices on that network. For example, in the diagram below, if virtual machine 1 on Host 1 sends a broadcast packet that packet has to reach virtual machine running on Host 4. As you can see the packet has to traverse through the VTEPs and the physical network to reach the virtual machine running on Host 4.

There are few communication options (as discussed in part 2) available to VTEP on the Host 1 when it comes to delivering broadcast packet from the logical network. It can use unicast, broadcast or multicast. Multicast is much more efficient in utilizing the resources of the physical network and it is used when sending broadcast packet from the logical network.

VXLAN deployment with one logical network

Let’s take a look in detail how the packet flows through the VTEP and the physical network.

We will take the same example of one logical network spanning across 4 Hosts. The physical topology provides a single VLAN 2000 to carry VXLAN transport traffic. In this case only IGMP snooping and IGMP querier is configured in the physical network. As we saw with multicast operation here , few things have to happen first before the physical network devices handle multicast packets.

IGMP Packet flows

In the diagram above the blue circles with number indicates the packet flow:

1. Virtual machine (MAC1) on Host1 is connected to the logical Layer 2 Network VXLAN 5001 and is powered on.
2. VTEP on Host 1 sends a IGMP join message to the network and joins the 239.1.1.100 multicast group address that is associated with VXLAN 5001 logical network
3. Similarly, virtual machine (MAC2) on Host 4 is connected to VXLAN 5001 and is powered on.
4. VTEP on Host 4 sends a IGMP join message to the network and joins the 239.1.1.100 multicast group address that is associated with VXLAN 5001 logical network

The Host 2 and Host 3 VTEPs don’t join the multicast group address because they don’t have any virtual machines running those are connected to VXLAN 5001 logical network. This is where the multicast optimization comes into play. Only VTEPs that are interested in listening to multicast group traffic, joins the group.

When a broadcast packet is generated by virtual machine on Host 1 this is how the packet flows through the physical topology and is delivered to the virtual machine running on host 4.

Multicast Packet flow

The following is the flow of packets:

1. Virtual machine (MAC1) on Host1 generates a broadcast frame.
2. VTEP on Host 1 encapsulates this broadcast frame into a UDP header with destination IP as multicast group address 239.1.1.100.
3. Physical network delivers the packet to the Host 4 VTEP, because it had joined the multicast group 239.1.1.100. The Host 2 and 3 VTEPs will not receive the broadcast packet.
4. VTEP on Host 4 first looks at the encapsulation header and if the 24-bit value of VXLAN identifier matches with the logical Layer 2 network ID, it removes the encapsulation header and delivers the packet to the virtual machine.

This is how multicast is used to deliver the broadcast traffic generated on any logical network. The other two traffic types on the logical network that will make use of multicast on the physical network are:

• Unknown Unicast frames
• Multicast frames from virtual machine

All other types of communication on the logical network is handled through normal unicast path in the physical network.

One of the question I always get is what happens if multiple logical networks are mapped to a single multicast group address and what are security and performance implication on that type of configuration. In the next post I will cover that.

Please let me know if you have any questions on these packet flows.

Here are the links to Part 1, Part 2, Part 4

Get notification of these blogs postings and more VMware Networking information by following me on Twitter:  @VMWNetworking

Use Case

Each application is deployed using a separate VXLAN network as shown below.  To keep the diagram simple, only one application is shown below.  The application has three tiers – web, app and db.

Enforce the following separation between tiers of the application using vCloud Networking and Security App Firewall.

1. Isolate Web Servers from one another
2. Allow HTTP/HTTPS traffic to Web Servers from any network other than Application VXLAN network.
3. Allow Web Server to App Server communication on port 8080
4. Allow App Server to Db Server communication on port 3036
5. Block all other traffic

vCenter Network view of the Application

vCenter Network view of the Application is shown below, where all virtual machines of the application are connected to the VXLAN port group “vxw-dvs-127-virtualwire-27-sid-5001-App1-vWire” as highlighted below.

App Firewall with VXLAN

In vCloud Networking and Security 5.1 release, each VXLAN network is created with an independent namespace for App Firewall. Datacenter level firewall rules no longer apply to the virtual machines attached to the VXLAN networks. We need to use Network Virtualization –> Networks section to define the App Firewall policy objects and rules.

Clicking on VXLAN network “App1-vWire”, you will see the following.Click on “Security” tab to create the App Firewall policy objects and rules for the VXLAN segment “App1-vWire”.

Firewall Rule Policy Objects

Security Groups

Security groups can include virtual network adapters, virtual wire, and other security groups. Let’s create three security groups Web-Srvr-SG, App-Srvr-SG, and Db-Srvr-SG.

Click on “+” icon in “Grouping” section to create a Security Group as highlighted below. Give a Name to the Security Group and select the Members.

Web-Srvr-SG is created with “App1-WebServer1” and “App1-WebServer2” network adapters as members. Similarly create two other security groups – App-Srvr-SG and Db-Srvr-SG. All the three security groups created are shown below.

Service and Service Groups

A service is a protocol-port combination and a service group is a combination of two or more services. Most commonly used services are pre-defined for convenience and ease of use. Create additional services and service groups from “Services” section. Services and service groups created for the application in this Use Case are highlighted below.

Firewall Rule Management

App Firewall Ethernet Rules

The first Ethernet rule below ensures micro-segmentation of web servers i.e. one web server cannot talk to another web server. If one of the web servers is compromised, it cannot be used to directly attack the other servers, even ARP and RARP will be denied. The second rule specifies a default Allow Ethernet rule. This is because Ethernet rules operate before General rules and a default deny Ethernet rule would not allow any traffic flow out of any virtual machine in this example. These rules satisfy the requirement 1 from the Use Case section.

App Firewall General Rules

The following General firewall rules are set up for the application to function properly satisfying the requirements 2 to 5 from the Use Case section.

• Rule 1 – Web-Access: Allows HTTP and HTTPS traffic to Web servers. Notice the negation used in the Source, wherein HTTP and HTTPS traffic to Web servers allowed from any network other than the “App1-vWire” VXLAN network. (Requirement 2)
• Rule 2 – Web-to-App-Access : Allow Web Server to App Server communication on App Port (Requirement 3).
• Rule 3 – App-to-Db-Access : Allow App Server to Db Server Communication on Db Port (Requirement 4).
• Rule 4 – Default Rule: Block all other traffic (Requirement 5).

Flow Monitoring

Flow Monitoring dashboard for the VXLAN network is shown below. The dashboard shows the percentage of allowed flows in green and blocked flows in red.

Clicking the Details link on the Flow Monitoring dashboard shows Allowed Flows and Blocked Flows for various services. Clicking on the rule id of the Flow Monitoring Details Allowed or Blocked Flow shows the details of the rule that allowed or blocked the traffic as shown below. Use Add Rule / Edit Rule link to create/edit the firewall rule.

In summary, we looked at how to use App Firewall rules and flow monitoring with vCloud Networking and Security 5.1 VXLAN networks.

Get notification of these blogs and more vCloud Networking and Security information by following me on Twitter @vCloudNetSec.

Before jumping into the solution, I would like to provide some background on what an OVF and OVA is. An OVF (Open Virtualization Format) is an open, secure, portable, efficient, and flexible format for the packaging and distribution of one or more virtual machines. An OVF usually contains several files that includes a descriptor file with .ovf extension, virtual disk file(s) with .vmdk extension and manifest file with .mf extension. The OVF format is a DMTF standard and it is also supported on other platforms besides VMware that provides support for the OVF format. Contrast that to an OVA (Open Virtual Appliance) which is just a single tar archive file that contains the contents of an OVF. An OVA can be thought of like a container which compresses the contents of the OVF for ease of distribution. Given this difference, I suspect this might be the reason why only the OVF format is only supported in vCloud Director. Though I can see why it maybe confusing when vSphere supports both the OVF and OVA format.

Note: vCloud Director supports both 1.0 and 1.1 of the OVF format.

Going back to our initial problem, you can use the ovftool to deploy an OVA directly to vCloud Director. The way this works is that the OVF is extracted out as part of the upload process from the OVA (not additional space required on the client side) prior to uploading to vCloud Director. This removes the manual two step process today which is to convert or extract the OVF from the OVA and then upload to vCloud Director.

The syntax for the ovftool to import to vCloud Director is as follows:

ovftool [OVA-FILE] [VCLOUD-LOCATOR]

Note: You can find  examples of the vCloud locator syntax in the ovftool documentation.

Here is an example of uploading the recent vCO 5.1u1 OVA to a vCloud Director 5.1 environment using the ovftool on Mac OS X:

"/Applications/VMware Fusion.app/Contents/Library/VMware OVF Tool/ovftool"  --acceptAllEulas /Users/lamw/Desktop/vCO_VA-5.1.1.0-1070383_OVF10.ova "vcloud://username:password@vcloud-director-ip-or-hostname:443?org=TechMarketing&vappTemplate=vCO-5.1u1&catalog=WorkInProgress&vdc=TM-Allocation1-ovDC"

Here is a screenshot of the upload:

The result of the above command is a new vAppTemplate called vCO-5.1u1 in the Catalog as well as a deployed vApp in my workspace. You can also opt out of deploying the vApp to the workspace by specifying –vCloudTemplate to be true.

If we login to our vCloud Director instance, we can see that our vAppTemplate has been deployed:

The nice thing about this solution is you can keep all of your OVA files for ease of distribution without having to convert them to an OVF. If you wish to convert your OVA files to OVF, you do not need to deploy to a vSphere environment first and then export it back out to an OVF, you can just use the ovftool to help with the conversion.

The syntax to convert an OVA to OVF is as follows:

ovftool [OVA-FILE] [OVF-FILE]

Here is an example of converting our vCO OVA to an OVF:

"/Applications/VMware Fusion.app/Contents/Library/VMware OVF Tool/ovftool" --acceptAllEulas /Users/lamw/Desktop/vCO_VA-5.1.1.0-1070383_OVF10.ova /Users/lamw/Desktop/vCO-5.1u1.ovf

Here is a screenshot of the conversion process:

The results are following three files:

• vCO-5.1u1-disk1.vmdk
• vCO-5.1u1.mf
• vCO-5.1u1.ovf

Get notification of new blog postings and more by following lamw on Twitter:  @lamw

In the diagram below you see three main types of communication modes that are common in a network – Unicast, Broadcast and Multicast.

Figure 1

Unicast mode (Fig1-A) is best for one to one communication while broadcast (Fig1-B) is best utilized when message has to be delivered to all nodes in a network. The devices in the network are capable of supporting Unicast and Broadcast traffic. However, when a message has to be delivered to a selected few nodes in the network as shown in Fig1-C, unicast and broadcast modes are not efficient. For example, if Unicast mode is used, the node on the left has to send a message to node 2 first, and then send the same message to node 2 again.

Multicast protocol support in the network allows optimal delivery for one to many communications. Instead of the end nodes sending multiple copies of message, the switches and routers perform that job.

How does Multicast work in IP network?

First of all, a unique IP address range is assigned as Multicast group IP address. It is a Class D address range from 224.0.0.0 to 239.255.255.255. Each address in this range designates a multicast group. Some of the addresses are reserved.

Any node (computer/user) in the network can join a multicast group using Internet Group Management Protocol (IGMP). For example, in Fig 1-C two nodes on the right have joined multicast group address 239.1.1.100.

After IGMP join requests, when an IP datagram with destination IP address of a multicast group is sent, it gets forwarded to every node that has joined that multicast group. For example, in Fig 1-C the node on the left sends a packet with destination IP address as 239.1.1.100. The network then delivers the packet to the two nodes on the right who had joined that multicast group earlier.

The devices in the network (Layer 2 switches and Layer 3 routers) run multicast protocols to support this optimal delivery of packets to the selected group of nodes. The following are some of the key protocols that are used in a multicast supported network

-       Internet Group Management Protocol (IGMP v1, v2, v3). IGMP manages multicast groups using Query and Report messages.

-       Multicast routing protocols – PIM – different modes (sparse, dense)

The network devices use these protocols to learn about which nodes have joined which multicast groups and where the nodes are in the network.

When it comes to VXLAN, the multicast support requirements in the physical network are dictated by the number of transport VLAN used in the design. As mentioned in the last post, the transport VLAN carries VXLAN encapsulated traffic. If you are using a single transport VLAN then there is no need for multicast routing protocol (PIM). However, you need the following functions enabled on the switches and routers

-       IGMP snooping on Layer 2 Switch

-       IGMP Querier on the Router

What is IGMP snooping? And how does it work?

We saw that multicast optimizes the delivery of the packets to the interested nodes. So the question is how does layer 2 network devices know which nodes are interested in which conversations or multicast groups?

The layer 2 switches monitor the IGMP query and report messages to find out which switch ports are subscribed to which multicast group. This functionality of a layer 2 switch is called IGMP snooping. The diagram below shows an example where there are two servers on the right streaming two different webcasts A and B. The users on the left choose to subscribe to a particular webcast by sending IGMP report messages.

IGMP Join request

The Layer 2 switch monitors IGMP packets sent by the users and makes entry in the forwarding table about the membership to particular multicast addresses. As you can see that multicast group address 239.1.1.100 is associated with Webcast A and 239.1.1.101 with Webcast B. In this example Port 1 and 2 are members of the multicast group 239.1.1.100 while Port 3 and 4 are members of 230.1.1.101.

The diagram below shows how the Webcast A packets with destination IP address 239.1.1.00 (Orange Arrow) sent to port 10 are only replicated to port 1 and 2 of the switch. Similarly the Webcast B traffic (Green Arrow) is only sent to port 3 and 4. User connected to port 5 is not subscribed to any Webcasts so it won’t receive any multicast traffic.

Multicast Packets

This shows how IGMP snooping capability on a physical switch optimizes the multicast packet delivery. Note that in this example each user has joined only one multicast group, but in reality they can join any number of multicast groups.

Why do you need IGMP querier ?

IGMP querier is the function of a router and it is important to enable that for a proper IGMP snooping operation on layer 2 switches. We looked at how users join a multicast group by sending IGMP query messages. These messages are sent to the multicast router or IGMP querier. Without an IGMP querier to respond to, users do not send periodic membership requests. As a result, the entries in the layer 2 switch times out and multicast traffic is not delivered.

I hope this clarifies some of the commonly used multicast terminology and how basic things work in multicast. In the next post, I will cover the following things:

-       Explain what is the relation between Layer 2 logical network in VXLAN and multicast group.

-       When and how multicast is used in VXLAN?

-       Explain that not all traffic is multicast in VXLAN deployment.

Here are the links to Part 1, Part 3, Part 4

Get notification of these blogs postings and more VMware Networking information by following me on Twitter:  @VMWNetworking

The next query I received was about this difference between Emulex & QLogic was how could it impact shared storage, e.g. one host has QLogic HBAs and another host has Emulex HBAs. First off, we do not support mixing HBAs from different vendors in the same host, so we don’t have to worry about that aspect. But mixing hosts that have HBAs from the same vendor is an interesting question, especially when the hosts are accessing the same datastore.

Overall, there won’t be an issue with hosts having HBAs from different vendors with different queue depths. They can successfully share access to a datastore, behave correctly, and they are fully supported.

There is one consideration however, and this is around Storage I/O Control (SIOC). I won’t go into all the details of SIOC in this post (more can be found in this white paper) but suffice to say that SIOC works by throttling the device queue depth across all hosts in order to prevent a noisy neighbor problem (where one VM on one host is impacting other VMs sharing the same datastore). Now, the reason for increasing the QLogic device queue depth from 32 to 64 was to give SIOC more slots to play with when it came to controller I/O. However, now we have a situation where some hosts may have a device queue depth of 64 and other hosts may have a device queue depth of 32. This could mean that those hosts which have Emulex HBAs are not getting the same fairness as those hosts that have QLogic HBAs when there is no I/O congestion on the shared datastore.

If you have hosts in your cluster, and some are using QLogic HBAs and others are using Emulex HBAs, and you are also using Storage I/O Control, and at times of I/O congestion, we don’t think customers need to change anything. There are two possible scenarios that can happen:

1. The hosts with QLogic HBAs installed has VIP VMs that need a larger share of queue depth. If the latency becomes higher than the configured SIOC congestion threshold, SIOC will bring down the device queue depth on hosts with QLogic HBAs since the limit is 64 on these hosts. In that case all is good because SIOC has more room to play with on these hosts and it will be able to keep higher queue depth even in periods of high congestion.

2. The hosts with Emulex HBAs installed has VIP VMs that need a larger share of queue depth.  Again, in a mixed environment, SIOC will throttle down the QLogic HBAs first before throttling down the Emulex HBAs (depending of share values. etc). The only concern with a mixed environment is when SIOC increases the queue depth on Emulex hosts during periods of low congestion, it will get stuck at 32 but on hosts using QLogic, SIOC will be able to go up to 64. This is completely normal and this is what SIOC does in order to utilize the array effectively.

You might think that we should increase the Emulex Queue depth value to match that of the QLogics and give fairness across the cluster. Unfortunately, at this time Emulex are recommending that a device queue depth of 32 is the sweet spot for their HBAs.

Hopefully Emulex will allow us to increase the queue depth of their HBAs up to 64 as well sometime in the future, but in the meantime we can leave the default 64 setting for QLogic and default 32 setting for Emulex as SIOC will work fine with these.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage

We hope you enjoy this month’s vCloud Suite Digest.  This is where we take some questions that we get and disseminate the answers in the hopes that it will help someone else who might have a similar question.  This month, we have some great tidbits on guest OS clustering, elastic VDCs, and networking among other things.  Enjoy!

vCloud Director

Guest OS Clustering

Backstory:

For some time vSphere has supported clustering technologies within the Guest Operating System, of which Microsoft Clustering Service (MSCS) is perhaps the most well known.  In the early days of ESX 2.x we used to get students to set up a NodeA/NodeB cluster-in-a-box configuration. The recommendation since the rise of VMware Distributed Resource Scheduler (DRS) is to use “anti-affinity” rules to ensure that NodeA and NodeB never reside on the same physical vSphere Host.

Q. Does vCloud Director (vCD) 5.1 support clustering within a guest OS?

A. It is OS-dependent. For example, you can create MS SQL Server failover cluster databases with vCD VMs using Windows 2008 R2. With that said, you should also note that there is a built-in method to ensure the VMs are running on different hosts other than deploying into different Provider vDCs. As each Provider vDC generally points to a different cluster – this should be enough to guarantee separation. Alternatively, you could use vCO or similar to apply anti-affinity rules once the VMs are deployed.

vApp with VMs Spanning Clusters

Backstory:

Since vCloud Director 5.1 it has been possible to add multiple VMware HA/DRS clusters into the same Provider vDC. Such a configuration is often referred to as an “Elastic vDC” as the compute resources of a single cluster do not limit it. It’s recommend (although not required) to use the VXLAN feature with an elastic vDC as this allows the administrator to configure networks that span domains.

Q. Can we define a vApp that has VMs that span clusters? With vCD, can a vApp’s network span multiple clusters?

A. Deploying a vApp in an elastic VDC may deploy VMs belonging to the same vApp in multiple clusters. This is not user-controlled, however. A vApp Network can span multiple clusters and even Layer 3 domains if a VXLAN-backed network pool is used.

An admin can define elastic VDCs and span clusters, and vApp networks can span clusters, but both of these are on the back end—transparent, not accessible, and not even knowable to an Organization user defining a vApp.  These rules also apply to every vApp in the Org vDC.

Whether what one creates spans or not is a happy accident based on settings and deploy-time distribution of resources, rather than a purposeful action on the vApp itself – even if the admin has defined everything such that it CAN span, there’s still nothing that is going to guarantee that it WILL, except for one approach that uses visibility to the storage to control where the VMs are placed – In this case you would create two storage tiers: cluster1 and cluster2 and assign to each cluster datastores (not shared between clusters). The user has control over which VM within the vApps uses which tier, and the placement engine takes care of the rest.

IP Masquerade in vCD 5.x

Q. In vCD 1.x there was an IP masquerade setting, but this seems to have disappeared in vCD 5.1. How do I achieve the equivalent functionality in vCD 5.1?

A. The behavior was changed in vCD 5.1. See KB article 2036040. Essentially, IP masquerade has been superseded by a new approach that improves the capabilities of vCloud Director. Now whenever a VM is created, its “internal” IP address is supplemented by an “external” IP address. This “external” IP addressed is allocated from a sub-allocation IP address range. You can see this mapping from the “Virtual Machines” tab of a vApp.

A combination of Source and Destination NAT rules (together with a firewall rule) allows you to grant VMs within the vApp access to the outside world, or to allow inbound access from the outside world.

Changes in Networking in an Upgrade from vCD 1.5 to 5.1

Q. When upgrading from vCD 1.5 to 5.1, what happens to an org network used in vCD 1.5 ?

A. Isolated and direct org networks get converted into an org VDC network. Routed org networks get converted into a gateway with two interfaces and an org VDC network.

Increasing vCD Cell Performance

Q. I am running vCloud Director 5.1 with 12GB systems RAM and increased JVM heap size per the best practices guide to 3GB and found the vCD cell response very good. Will increasing the memory size to 8GB help even more?

A. More memory does not necessarily mean better performance. You should profile your vCD cells to determine what, if any, bottlenecks exist. If you really want to optimize the memory and garbage collection options, see our white papers on Enterprise Java Applications on vSphere:

• http://www.vmware.com/files/pdf/techpaper/vmware-vfabric-sqlfire-best-practices-guide.pdf
• http://www.vmware.com/files/pdf/techpaper/Enterprise-Java-Applications-on-VMware-Best-Practices-Guide.pdf

vCloud Director Licensing: Partially Powered-on vApps

BackStory:

A partially powered-on vApp is where the vApp contains some VMs which are powered on, and others which are powered off. The vApp in vCloud Director is given a process ID, just like a VM. So it is possible to a vApp that is “powered on” when none of the VMs are actually powered on within it. If this happen then you would power off the vApp as normal.

Q. If I have some VMs in a vApp powered off, how does this count in licensing?

A. vCD is licensed at the VM level, counting the number of powered-on VMs. However, note that vCD itself does not enforce licensing based on the number of powered-on VMs – ensuring compliance is a manual process.

vCloud Network and Security (vCNS)

VPN Support

Q. Do we have a list of supported/non-supported third-party VPN products for vCNS?

A. VMware has tested our IPsec site-to-site VPN feature with Juniper and Cisco products, and this should work without any issues. Since IPsec is an open specification/protocol suite (IETF standard) we should be able to interoperate with any IPsec solution (of course there are limitations), but the typical deployments will work just fine.

The limitation of IPsec, which is also one of its core strengths, is its extensibility. Although nearly all products/solutions support the same base set of authentication and encryption algorithms, third-party vendors are free to add new algorithms as they come along.

vCNS Edge Gateway High Availability

BackStory:

vCNS introduced a new high-availability option for the Edge Gateway that can be enabled when it is being created – or enabled afterwards. This option can be enabled in vCloud Director on the properties of any “Edge Gateway” under the General tab; alternatively, if you want to use this feature without vCloud Director, consult the blogpost (see below) for a step-by-step guide to configuring with vCNS Manager.

Q. Where can I find an example illustrating how to set up the high availability features of vCNS Edge 5.1?

A. See: http://blogs.vmware.com/vsphere/2013/03/vcloud-networking-and-security-5-1-edge-gateway-high-availability.html

vCNS Edge Storage Placement

Backstory:

When you create a new Organization Network or vApp Network it is likely that an Edge Gateway will be deployed by vCloud Director. Using your default “Storage Profile” configured for the Organization, the new Edge Gateway appliance will deployed. This is done when the Organization Virtual Datacenter is defined and is referred to as the “Default Instantiation Profile”.

Q. In vCD, can we influence placement of the vShield Edge appliances?

A. The Edge will get placed in any valid datastore for the VDC, just like a regular VM, but there is no way to choose which datastore it will get placed in. If you are using storage profiles, then you can enable/disable storage profiles at the org VDC level to help control placement.

Q. How can I move an already-deployed Edge?

A. Reset the network to force the Edge to re-deploy.

Edge Gateway and Physical Servers

Backstory:

The Edge Gateway is NAT, VPN, Load-balancer, DHCP and Firewall all in one, and primarily acts as gateway device for VMs. As you would expect much of the automation is delivered to virtual machines, but it is possible to configure it for physical devices, too.

Q. Is there a licensing model for vCNS to protect physical servers with an Edge firewall?

A. VMware does not license for physical machines, only protected virtual machines. So any physical devices are protected for free.

Q. How would this be set up?

A. The portgroup being protected by the edge device would just be backed by VLAN in the physical world. The traffic patterns are essentially the same traffic patterns (in terms of tracing packets up and down through the switching fabric) that we see with a very typical firewall-on-a-stick deployment having the firewall attached to distribution layer multilayer switch.  The only difference is that the firewall-on-a-stick is now the Edge device, but the same number of traffic hops through the physical network. 

Hi,

In order to get a wide audience for this topic, I’ve cross posted this post from the VMware Security and Compliance Blog. Enjoy!

It has been a couple of weeks since the release of the vSphere 5.1 Hardening Guide. Right around that time there was a call for updated content for the VMware Mobile Knowledge Portal app Well, I really wanted to see the updated Hardening Guide available on that  platform. That presented a challenge. For most customers, the format of releasing it as an Excel spreadsheet meets their need but have you looked at a spreadsheet on an iPad? Not a pretty sight.

So, using Microsoft Word’s Mail Merge capability I whipped up a proof of concept and showed it to a couple of folks. When your boss says “That’s awesome!” you know you’re on the right track. After some updating, we came out with a decent template and I’m happy to say that it looks great on my old iPad.

Here’s some examples:

First, a picture of the Mobile Knowledge Portal app itself. You can see a list of all the Hardening Guide content.

Now the Introduction Page that explains what the Hardening Guide contains

Finally, here’s an example of Hardening Guide data reformatted for a tablet. This is useful for browsing through all the guidelines.

If you don’t have the VMKP, get it now for iPad and Android! Here’s some more info about the app and where to get it.

Get it for iPad on iTunes
Get it for Android tablets at Google Play

The VMKP is designed to provide a simple way for VMware customers to view technical collateral around the Datacenter & Cloud Infrastructure and Infrastructure & Operations Management products.
VMKP 2.0 adds the following enhancements over the original version
- Android and iPad support
- Ability to rate collateral
- Ability to provide feedback to VMware on pieces of collateral
- Integration with Facebook and Twitter to let others know what you have been reading on the VMKP
- Mechanism to request additional collateral items

I hope you find this useful. If you have feedback, please send it on!

Thanks,

mike

The papers that we have made available “Network virtualization Design Guide” and “VXLAN Deployment Guide”, provides some basic knowledge about the technology, Use cases, and step-by-step deployment instructions. However, some of the detailed packet flow scenarios are not explained in these papers. So I thought it would be a good idea to put together a series of post discussing the packet flows in a VXLAN environment. Also, there are many common questions that I would like to address as part of this series.

To start this series, I will first describe the different components of the VMware’s VXLAN implementation.

VXLAN Components

The diagram above shows a deployment of two compute clusters that is configured with VXLAN components running on each vSphere host.

VXLAN is an overlay network technology. Overlay network can be defined as any logical network that is created on top of the existing physical networks. VXLAN creates Layer 2 logical networks on top of the IP network. The following two are key traits of an overlay technology:

-       It encapsulates original packets into a new header. For example, IPSec VPN, an overlay technology, encapsulates original IP frame in another IP header.

-       Communication is typically established between two tunnel end points. For example, in an IPSec based VPN, which runs on the public internet, the tunnels are established between two sites.

When you apply those overlay technology traits to VXLAN, you will see that VXLAN encapsulates original MAC frames in to a UDP header (shown below), and all vSphere hosts participating in VXLAN acts as tunnel end points. They are called Virtual Tunnel Endpoints (VTEPs).

VXLAN – Encapsulation Header

VTEPs are the nodes that provide the encapsulation and de-encapsulation function. When we will go through the detail packet flows it will be clear how these VTEPs encapsulate and de-encapsulate traffic from any virtual machine connected to a VXLAN based Layer 2 logical network or virtual wire. The virtual tunnel endpoint (VTEP) configured on every vSphere host consists of the following three modules:

1) VMware Installation Bundle (VIB) or vmkernel module – VTEP functionality is part of the VDS and is installed as a VMware Installation Bundle (VIB). This module is responsible for VXLAN data path processing, which includes maintenance of forwarding tables and encapsulation and de-encapsulation of packets.

2) vmknic virtual adapter – This adapter is used to carry control traffic, which includes response to multicast join, DHCP, and ARP requests. As with any vmknic, a unique IP address is assigned per host. The IP address is used as the VTEP IP while establishing host-to-host tunnels to carry VXLAN traffic.

3) VXLAN port group – This is configured during the initial VXLAN configuration process. It includes physical NICs, VLAN information, teaming policy, and so on. These port group parameters dictate how VXLAN traffic is carried in and out of the host VTEP through the physical NICs. As shown in the diagram, VLAN 2000 is used as the transport VLAN for VXLAN traffic. The transport VLAN has no relation to the logical Layer 2 networks or virtual wires that you will create.

The configuration of the VTEP on each vSphere host is managed through a central place called vCloud Networking and Security Manager. One of the common questions I get is whether this manager acts as a controller similar to the Openflow controller. The answer is No. In VXLAN there is no special controller or control plane required. So then the question is how in VXLAN a forwarding table is created ? In physical switch infrastructure the forwarding table information helps deliver packets to the right destination.

In VXLAN all the learning about the virtual machine MAC address and its association with VTEP IP is performed through the support of physical network. One of the protocols utilized in the physical network is IP multicast. VXLAN makes use of this IP multicast protocol to populate the forwarding tables in the VTEP.

Before we dig into how IP multicast is utilized in VXLAN, in the next blog, we will take a look at some basics on IP Multicast.

Here are the links to Part 2, Part 3, Part 4

Get notification of these blogs postings and more VMware Networking information by following me on Twitter:  @VMWNetworking