Monday, September 8, 2008

One kink in the armor

If there is any issues with VI3, it has to be the reliability of Virtual Center after its host machine crashes.  In the past 2 years of running virtual center, the Windows box on which it runs has lost power due to facility issues maybe 5 times.  Of the five major power surges/outages, virtual center has failed to fully come online 4 times.  So far each time has required a reinstallation of the application, and a restoration of the last full db backup.  Luckily bringing the hosts back under management isn't a very arduous task.  Even without the database backup, you can still bring all of the hosts back under the VC umbrella rather quickly.  You do however lose all of your cluster configuration options.  This particular fault bit me just a few weeks back.

After a VC reinstallation due to a power outage, I had what I thought was a suspect database backup and current DB.  So I wiped the old db, and created a new one.  I brought all of the hosts back into management, but had forgotten about setting the DRS to partially automatic, rather than the default fully automatic.  This caused the VMs on my 75% utilized production cluster to start VMotioning at will about 3 every 2 minutes.  After about 40 or so vmotions in an hour, we finally had one puke.  Luckily it was our corporate exchange server.  After looking at the logs, and seeing all of the VMotions, it became clear to me that I had forgotten to go back in and setup all of the cluster settings as per my preference.  Once this was done, a VM reset, everything was fine.

Moral of the story, when recreating a VC database, be sure to go in and inspect all settings of a cluster, before re-adding the hosts!

No comments: