SplitRXMode in VMware vSphere 5

My previous post was on how multicasting is handled in VMware vSwitch context. You can read about it here.

Now it is most apt to mention some advanced setting around it. The advanced setting we talk about is SplitRXMode.

While Multicasting worked fine for some multicast applications this still wasn’t sufficient enough for the more demanding multicast applications and hence stalled their virtualization. 

The reason being that in this case VMs would process the packet replication in a single shared context which ultimately led to constraints. This is because when there was a high VM to ESX ratio there was a consequent high packet rate that often caused large packet losses and bottlenecks. VMware vSphere 5 provides, the new splitRXMode to not only compensate for this problem but also enable the virtualization of demanding multicast applications.

SplitRx mode is an ESXi feature that uses multiple physical CPUs to process network packets received in a single network queue. This feature provides a scalable and efficient platform for multicast receivers. SplitRx mode typically improves throughput and CPU efficiency for multicast traffic workloads.

VMware recommends enabling splitRx Mode in situations where multiple virtual machines share a single physical NIC and receive a lot of multicast or broadcast packets.

NOTE: 

•         SplitRx mode is supported only on vmxnet3 network adapters. 
•         This feature is disabled by default.
•         SplitRx mode is individually configured for each virtual NIC.

To enable SplitRX do the following:

This feature, which is supported only for VMXNET3 virtual network adapters, is individually configured for each virtual NIC using the ethernetX.emuRxMode variable in each virtual machine’s .vmx file (where X is replaced with the network adapter’s ID).

The possible values for this variable are:

 ethernetX.emuRxMode = “0″

The above value disables splitRx mode for ethernetX.

ethernetX.emuRxMode = “1″

The above value enables splitRx mode for ethernetX.

To change this variable through the vSphere Client:

1. Select the virtual machine you wish to change, then click Edit virtual machine settings.
2. Under the Options tab, select General, then click Configuration Parameters.
3. Look for ethernetX.emuRxMode (where X is the number of the desired NIC). If the variable isn’t present, click Add Row and enter it as a new variable.
4. Click on the value to be changed and configure it as you wish.

The change will not take effect until the virtual machine has been restarted.

References and Credits: Chris Hendryx (it.toolbox.com).

Back to VMware basics - vMotion Deepdive

Ever thought on how vMotion works ? 

I plan to write a detailed post divided into various sub posts to help you understand how the process works and what happens in background.

Architecture

vSphere 5.5 vMotion transfers the entire execution state of a running virtual machine from the source VMware vSphere ESXi host to the destination ESXi host over a high speed network. The execution state primarily consists of the following components:

• The virtual machine’s virtual disks
• The virtual machine’s physical memory
• The virtual device state, including the state of the CPU, network and disk adapters, SVGA , and so on
•  External network connections

Lets see how vSphere 5.5 vMotion handles the challenges associated with the transfer of these different states of a virtual machine.

Migration of Virtual Machine’s Storage

vSphere 5.5 vMotion builds on Storage vMotion technology for transfer of the virtual machine’s virtual disks. We need to understand, Storage vMotion architecture briefly to provide the necessary context.

Storage vMotion uses a synchronous mirroring approach to migrate a virtual disk from one datastore to another datastore on the same physical host. This is implemented by using two concurrent processes. First, a bulk copy (also known as a clone) process proceeds linearly across the virtual disk in a single pass and performs a bulk copy of the disk contents from the source datastore to the destination datastore.

Concurrently, an I/O mirroring process transports any additional changes that occur to the virtual disk, because of the guest’s ongoing modifications. The I/O mirroring process accomplishes that by mirroring the ongoing modifications to the virtual disk on both the source and the destination datastores. Storage vMotion mirrors I/O only to the disk region that has already been copied by the bulk copy process. Guest writes to a disk region that the bulk copy process has not yet copied are not mirrored because changes to this disk region will be copied by the bulk copy process eventually.

Migration of Virtual Machine’s Memory

vSphere 5.5 vMotion builds on existing vMotion technology for transfer of the virtual machine’s memory. Both vSphere 5.5 vMotion and vMotion use essentially the same pre-copy iterative approach to transfer the memory contents. The approach is as follows:

•  [Phase 1] Guest trace phase The guest memory is staged for migration during this phase. Traces are placed on the guest memory pages to track any modifications by the guest during the migration.

• [Phase 2] Pre-copy phase Because the virtual machine continues to run and actively modify its memory state on the source host during this phase, the memory contents of the virtual machine are copied from the source ESXi host to the destination ESXi host in an iterative process. Each iteration copies only the memory pages that were modified during the previous iteration.

• [Phase 3] Switch-over phase During this final phase, the virtual machine is momentarily quiesced on the source ESXi host, the last set of memory changes are copied to the target ESXi host, and the virtual machine is resumed on the target ESXi host.

In contrast to vMotion prior to vSphere 5.1, vSphere 5.5 vMotion also must transport any additional changes that occur to the virtual machine’s virtual disks due to the guest’s ongoing operations during the memory migration. In addition, vSphere 5.5 vMotion must coordinate the several copying processes including the bulk copy process, I/O mirroring process, and memory copy process.

To allow a virtual machine to continue to run during the entire migration process, and to achieve the desired amount of transparency, vSphere 5.5 vMotion begins the memory copy process only after the bulk copy process completes the copy of the disk contents.The memory copy process runs concurrently with the I/O mirroring process, so the modifications to the memory and virtual disks, due to the guest’s ongoing operations, are reflected to the destination host.

Because both the memory copy process and I/O mirroring process contend for the same network bandwidth, the memory copy duration could be slightly higher in vSphere 5.5 vMotion compared to the memory copy duration during vMotion. Generally, this is not an issue because the memory dirtying rate is typically high compared to the rate at which disk blocks change.

vSphere 5.5 vMotion guarantees atomic switch-over between source and destination hosts by ensuring both memory and disk state of the virtual machine are in lock-step before switch-over, and fails back to source host and source disks in the event of any unexpected failure during disk or memory copy.

Further details on vMotion, Storage vMotion and some related concepts in my next post.