Virtual Machine (VM) live migration is an important feature to enhance the high availability of applications, guarantee quality of service, and simplify infrastructure maintenance operations. Specifically, it can help to:
As a result, it is almost a must-have feature for cloud infrastructure.
In VMware environment, the feature that supports VM live migration is vMotion, and it allows you to move an entire running VM from one physical server to another, without downtime, well, at least that is the goal. Because vMotion involves the transfer of the VM’s active memory and precise execution state across the network, the speed of vMotion network connecting these two hosts has a great impact on vMotion execution, and in some scenarios, even deciding whether vMotion can succeed or not.
In this blog, we compare the difference of performing vMotion over 40Gb/s network vs. 10Gb/s network. Our benchmark shows that:
We believe that the new generation of 25/50/100G interconnect networks will accelerate live migration in a similar fashion as 40G networks over 10G networks.
To understand the importance of a high-speed vMotion network, let’s first examine how vMotion works and the steps that vMotion takes to complete a live migration.
Out of the steps, Pre-copy, Iteration and Switchover all involves copying data from source server to target server and the network performance is critical in deciding whether iteration will ever converge, or switchover will fail or succeed.
In an ideal environment, the interation process converge in the following fashion (I borrowed the following table from an Opvisor blog):
|Pre-Copy Iteration||Main memory to be transferred||Time needed for the transfer||Change in memory during the transfer|
|1||2.048 MB||16 seconds||512 MB|
|2||512 MB||4 seconds||128 MB|
|3||128 MB||1 second||32 MB|
|4||32 MB||0,25 seconds||8 MB|
|5||8 MB||vMotion cutoff, since the residual transmission takes only ~0.06 seconds|
But if the application is dirtying the memory faster than the network ability to move the changes from source to destination server, this process may not converge.
In our benchmarking experiment, we ran ESXi6.0 over all-flash VSAN array. We looked at VMs with 128GB and 256GB memory configurations respectively. In these VMs, we run IOMeter to simulate applications with frequent read/write operations. We use the conventional network with 10Gb/s throughput as reference to examine the effect that Mellanox CloudX 40Gb/s high-speed interconnect can bring.
For these two VMs, when we performed vMotion over 40Gb/s network compared to 10Gb/s network, the completion time was reduced by 88% and 82% respectively.
Not surprisingly, when we monitor the bandwidth consumption during the vMotion process, we are seeing much higher bandwidth in the 40G vMotion network. Here we use the VM with 256GB memory to illustrate the network bandwidth consumption during vMotion precopy+interation phase, and during switchover phase.
As mentioned previously, in most cases, each pre-copy iteration should take less time to complete than the previous iteration, but if we get a very active VM that modifies memory contents faster than it can be transferred, the iterative copy might never converge. In that case, vMotion performs an action called Stun During Page-Send (SDPS) to ensure the memory modification rate is slower than the memory change-set transfer rate.
Upon activation, SDPS injects microsecond delays into the virtual machine execution and throttles its page dirty rate to a preferred rate, guaranteeing pre-copy convergence. The downside of SDPS is that it does run the risk of halting VM for an extended period of time and may cause read/write operations to fail. In our case, we run the IOMeter application in the VM, and we observed that during the SPDS phase of the vMotion, the I/O operations were halted, and resumed later. We ran OLTP workload with a VM with 256GB memory, and the duration of I/O halt is much longer on 10G vMotion networks compared to 40G vMotion networks.
As a conclusion, high-speed networks can really set vMotion into fast motion by transferring VM state data much more quickly from source server to destination server. In some special cases where we got active VMs with large memory, high-speed network is the only way to make vMotion converge and cause minimal downtime to applications.