Stressing Out with Hyper-V

I’ve been spending a lot of my time recently working with Hyper-V (pretty evident based upon some of the recent blog entries, I know). In Virtual Server 2005 R2 (or VPC) if I ran very intensive applications in the guest operating systems I saw the impact on the host processor’s operating system. This provided an easy method to determine if the host operating system was insufficient for the requirements of the guest operating system(s). While I have been working with Hyper-V this didn’t seem to be the same situation but I put that off to just timing between when I was looking at the processor utilization on the guest and host operating systems.

I was talking with one of the folks with Microsoft at TechEd last week about Hyper-V and Operations Manager and some different pieces started falling into place. In our discussion Clint Huffman (a really nice person and solid performance analysis guy among other things – blog at http://blogs.technet.com/clint_huffman/) explained that Hyper-V changes how things work from a performance analysis perspective. He referred me to work done by Tony Voellm related to how Hyper-V works and more of the internal details. Good stuff on this from Tony is available in a couple of sets of articles available at http://blogs.msdn.com/tvoellm/archive/2008/03/20/hyper-v-clocks-lie.aspx and http://blogs.msdn.com/tvoellm/archive/tags/Hyper-V+Performance+Counters/default.aspx.

From my perspective the long story made short on this: high processor utilization in the guest operating systems will not necessarily translate into high processor utilization on the host operating system running Hyper-V.  Task Manager or the standard performance monitor counters for processor usage on the host operating system in Hyper-V do not reflect the impact that is occurring on the guest operating system(s). There are Hyper-V specific counters which can supplement the standard processor performance counters which are discussed in the tvolellm blog entries above. Before accepting this on faith of the conversation I ran through my own tests.

My test environment was a Windows 2008 server with a dual-core and four gigs of memory running Hyper-V. Multiple guest operating systems were installed which are running in Hyper-V. These include ClusterServer1, ClusterServer2 and ClusterServer3 (see http://cameronfuller.spaces.live.com/blog/cns!A231E4EB0417CB76!1238.entry for background on this). I used cpustress (discussed in http://cameronfuller.spaces.live.com/blog/cns!A231E4EB0417CB76!154.entry) to create the heavy processor overhead I needed to really stress out the guest operating systems. The overall test plan was to track processor performance via Task Manager on the host operating system while I added load to the guest operating systems to track the impact.

Step 1:

I run a processor slaughtering application on Cluster Server 2 – maxing out the virtual processor.

Ready01

Assessed impacts to processor on the Server Manager – Virtual machine processor showing at 46% for Cluster Server 2.

Ready02

Checked Task Manager on the Host operating system – Processor utilization hovering around 7%.

Ready03 

Step 2:

I run the same processor slaughtering application on Cluster Server 3 – maxing out the virtual processor.

Ready04

Assessed impacts to processor on the Server Manager – Virtual machine processor showing at 38% for Cluster Server 2, and 46% for Cluster Server 3.

Ready05

Checked Task Manager on the Host operating system – Processor utilization hovering around 15%.

Ready06

Step 3:

I run the same processor slaughtering application on Cluster Server 1 – maxing out the virtual processor.

Ready07

Assessed impacts to processor on the Server Manager – Virtual machine processor showing at 29% for Cluster Server 2, and 25% for Cluster Server 3, and 33% for Cluster Server 1.

Ready08

Checked Task Manager on the Host operating system – Processor utilization hovering around 14%.

Ready09

Following this is the actual tasks using the processor on the host operating system… Summarized to – Not Much of anything.  

Ready10 

Lessons Learned:

·        Unlike Virtual Server 2005 R2 and VPC, processor utilization on the guests does not appear to have a direct correlation to processor utilization on the host. Additional Hyper-V specific counters may provide this information however.

·        This has two important impacts that I see. First off, the inability to use the Task Manager to determine the full processor usage on a system with Hyper-V installed that has guests running. While Task Manager can be used to determine tasks that are occurring with the Hyper-V host, the actual processor usage on the system may be significantly higher than is shown in the Task Manager.

·        This may have some interesting impacts from an Operations Manager perspective as well to either the Server 2008 operating system management pack or to the Hyper-V management pack. A server running Hyper-V may not alert on high processor usage even if the processor is maxed out. As an example, if we take a host system with a dozen virtuals, if each of these virtuals is using 10% of the processor the full processor on the host may be used but it would seem that the Server 2008 operating system management pack may be hard-pressed to determine this condition.