TurboTalk

January 28, 2010 By John Gannon

Three secrets of the Uber-Virtualized

In October 2009, Gartner estimated that only 16% of workloads worldwide are running in virtual machines, although tremendous growth is expected in the coming years.  Not surprisingly and roughly in line with Gartner’s estimates, most customers tend to be about 20% or 30% virtualized, with ambitious plans for growth in the coming year.

However, some organizations are outliers.  They have virtualized the majority of their IT environment and are seeing benefits above and beyond the typical server consolidation and disaster recovery use cases.

I like to call these folks the uber-virtualized, and in this post I’ll discuss some of the best practices we’ve learned from them!

1)  Pay attention to your storage environment, because there is a good chance it’s where the bottleneck lives!

The fear of storage bottlenecks keeps the uber-virtualized up at night.  When you’ve virtualized most of your IT environment, that is going to cause additional stress on your SAN because of all the virtual disks you’re storing there.  Rather than throwing storage capacity at the problem (at additional cost), much time and effort goes into poring over storage array and VirtualCenter data, trying to find optimization opportunities.

There are certainly some monitoring tools on the market which can aid in this process by gathering numerous bits of utilization and performance data from hosts and storage arrays.  However, these tools (just as VirtualCenter) leave the administrator to make the final decision about how to rebalance the environment and mitigate the risk of storage bottlenecks.  Fortunately, the uber-virtualized have been working with VMware technology for many years, and are often able to make the right decision based on their experience.

VMware has also recognized that storage challenges can really hurt virtualized IT deployments, and have responded by developing new technology like IO DRS.  This is a good first step, although the VMware administrator will need to have the experience to recognize the proper thresholds to configure to trigger a migration.  With 10 or even 100 VMs, this is fairly simple to do.  However, with hundreds or thousands of VMs in an environment that’s tightly managed (50%+ utilization), deeper analysis needs to be done looking at all resources together (CPU, memory, and I/O) before making workload balancing decisions.  Otherwise you’re risking performance problems and downtime.

2) Are you (CPU) Ready?

CPU Ready is one of the key parameters that the most experienced VMware administrators examine when they see a performance problem.  In fact, it is often the first thing they’ll check when debugging.  Learn to love this statistic and what it means, because it can help you identify virtual machines that may be oversized and that cause your environment to perform poorly.

(By the way, here is a nice Powershell script that will grab CPU Ready stats for all of your VMs!)

3) Automate lightly, young Padawan.

One thing that surprised me in talking to the uber-virtualized is that some are skeptical about the use of automation like DRS in their environments.   I thought everyone would be running DRS in fully automated mode, and have their feet up on the couch at home while drinking a beer since their environment was managing itself   But there were some organizations that felt that DRS didn’t give them everything they needed (particularly in the IO department).  It will be interesting to see how IO DRS, once released, will address some of these concerns.

Are you one of the uber-virtualized?  Care to share any of your secrets, tips, or tricks?  Please feel free to leave them in the comments, your fellow VMware administrators and architects will appreciate it!

Category: Performance

Posted on January 28, 2010 | Permalink | View Comments