Strange VMware Tools issue while upgrading one of my Ubuntu 6.06.2 VMware Guests to Ubuntu 8.04.1.  The Ubuntu upgrade itself seemed to go just fine, but installing the updated VMware Tools gave me an error:
 

Your kernel was built with "gcc" version "4.2.3", while you are
 trying to use "/usr/bin/gcc" version "4.2.4". This configuration
is not recommended and VMware Player may crash if you’ll continue.
Please try to use exactly same compiler as one used for building
your kernel. Do you want to go with compiler "/usr/bin/gcc"
version "4.2.4" anyway? [no]

It didn’t take very long to find a thread about this - http://ge.ubuntuforums.com/showthread.php?t=963825.  I took the suggest of some of the users here and chose YES.  SNAPSHOT YOUR MACHINES BEFORE YOU TRY SOMETHING UNPROVEN!
 

Well now it complained that it couldnt locate the Linux C Header files:
 

The header files in /usr/include are generally for C libraries, not for the
running kernel. If you do not have kernel header files in your /usr/src
directory, you probably do not have the kernel-source package installed. Are yousure that /usr/include contains the header files associated with your running
kernel? [no]

Weird.  So now I had to manually install the Linux Headers.  I did this by issuing:

apt-get install linux-headers-`uname -r` build-essential

I now re-ran perl /usr/src/vmware-tools-distrib/vmware-install.pl.  I again of course had to tell the installer to continue compiling using gcc 4.2.4, but now it didnt complain about the C headers. The install completed successfully.

I had completely forgotten about this. WIth past ESX builds Ubuntu was only *sort of* supported.  In order to get VMWare Tools installed on Ubuntu you had to basically merge the native VMWare Tools with open-vmware-tools. 

The process of merging the two sets to tools together was simple enough, and successful enough - but doesnt it feel great when something is really supported and when it works the way it’s supposed to? For the shiny new supported VMWare Tools I simply ran ./vmware-install.pl from the console of my Guest, waited about 5 minutes and voila!

ESX 3.5 Update 3 Release Notes

I was doing some regular maintenance today in Virtual Center 2.5 today and realized Its been a couple months since I’ve installed any updates on my ESX 3.5 hosts.  I don’t think I’ve installed any updates since way back before August before the bugged Update 2 was release. Back then I had reinstalled VI 2.5 and forgot to reinstall Update Manager in VI.  Funny how you forget about things when they aren’t right in front of you.

When I first attempted to Scan for Updates I received the following error in VI:
 

metadata for patch missing

Searching through the forums suggested the ESX host FW wasn’t properly listening for connections and suggested I run:

esxcfg-firewall --openport 80,tcp,out,dynamicupdates 
(port 80 was specific to my environment and may vary for your setup)

Once I did this the Scan for Updates ran just fine

Onto the upgrade….

Before the upgrade

[root@esx-03 root]# vimsh -n -e ‘hostsvc/hostsummary’ | grep fullName
File not found.
         fullName = "VMware ESX Server 3.5.0 build-82663",

1.Reinstalled Update Manager and attached the Critical and Non-critical Baselines to the ESX hosts

2.Put my first ESX host into Maintenance Mode and agreed to all the informational messages that come with that

3.Selected all Guest VMs and migrated them to another ESX host (I’m still not absolutely sure if entering Maintenance Mode actually evacuates all the running VMs.  I just migrate them manually because I can see it working)

4.Right clicked on the ESX host and chose Remediate, chose both Baselines and Next’ed through the wizard.

5.30 minutes later the ESX host was back up and running with the newest patches and updates. Repeat steps on all other Hosts.

After the upgrade

[root@esx-03 root]# vimsh -n -e ‘hostsvc/hostsummary’ | grep fullName
File not found.
         fullName = "VMware ESX Server 3.5.0 build-123630",

One thing I noticed after the upgrade was that the SSH server didn’t automatically start even though it was set to

I clicked START and all was well.  I will do more troubleshooting on this the next time I need to reboot the Hosts.  I’d rather not evacuate all my Guest VMs for this reason alone. For now I’m happy being up to date.

I noticed when i did some maintenance and rebooted the ESX hosts and i could not get High Availability (HA) back up and running.  I kept receiving the error:

An error occurred during configuration of the HA agent on the Host
Enable firewall failed:  vim.fault.HostConfigFault

Strange.  Looks like neither ESX hosts are talking to each other.  VMWare communities suggest i run:

[root@syr-nav-esx-03 root]# esxcfg-firewall -q aamclient

And it gave me:

Can’t locate XML/DOM.pm in @INC (@INC contains: /usr/lib/vmware/esx-perl/perl5/site_perl/5.8.0 /opt/ActivePerl-5.8/site/lib /opt/ActivePerl-5.8/lib .) at /usr/sbin/esxcfg-firewall line 21.
BEGIN failed–compilation aborted at /usr/sbin/esxcfg-firewall line 21.

Turns out the problem is from the changes Asigra Televaulting DS-Client wanted me to make to each ESX host in order to get host based backups working.  The DS-Client requires Perl to be upgraded to at least 5.8.1. And in order to make sure ESX continues to function after the Perl upgrade they states on page 419 of dsuser.pdf:

After the Perl upgrade, the vmware-cmd command may no longer work since it is not compatible
with the new Perl. You will, however, need this command in order to register restored virtual
machines. To fix this problem, go to the /usr/bin/ directory and edit the vmware-cmd file as well as
the vmware*.pl files in this directory; change the first line in each file to refer to the old Perl
installation (replace #!/usr/bin/perl with #!/usr/bin/perl.old). Run vmware-cmd without
parameters to verify that it works.

So i guess they forget to mention /usr/bin/esxcfg-firewall also needs it’s perl path changed to /usr/sbin/perl.old.  This makes me wonder what else migth brake from the Perl upgrade.

My approach to backing up guest VM’s from ESX3.5 is two-headed:

• Backup the data like you would a normal machine by attaching via SMB, NFS, SQL, etc.  Just get the data
• Backup the Guest VM via the ESX host (pull a bare metal snapshot of the entire system)

Why backup everything twice you ask?

Well fortunately for you and unfortunately for me I have a real world experience that will help answer this question. I had a vm guest failure and had to rebuilt a vm then restore data and settings.  Suffice to say this took forever. So here’s why you can do both…

• If data is deleted you have backup sets that have just pure data - files that are used by applications and users. If a single file gets deleted, corrupted, or anything else _bad_ the restore for this file(s) is quick and easy.  Restoration of this file(s) doesn’t need to affect every other user connected to the system in question.
• If the OS becomes unusable, system files fail, vmdk (virtual disk) files on the host get corrupted then you now have the full machine backup to simply turn back on from date X/Y/Z. Asigra actually takes a native ESX snapshot at the time of backup.

Using Asigra Televaulting (or a small handful of other backup systems) allows us to perform backups that are compressed and bit level.  So, if I have a VM that has a 20GB vmdk disk, but really only has 8GB of data in the guest system, then the backups will actually be less than 8 because all that free space will be compressed down to nothing and then the 8GB of real data will further be compressed down.

My schedules are now setup to backup real data each night, but backup the OS (again using the native ESX snapshot) once a week.  For the most part, system settings and configurations aren’t happening each day. So, if the guest VM dies, I can simply restore the VM to the last weekly backup (as of at the most 6 days) then restore the real data to to that machine (as of at the most the night before).

A note for Asigra users - I’ve had a lot of problems backing up via VI.  So far attaching directly to the ESX hosts seems to be working great.  The one fallback for this is you need to setup rules to ensure that VM’s stay on the same ESX host.  Asigra only knows that a guest VM is on the host you originally configured it to be backed up from.

So I was having some pretty significant performance problems with a vm running Windows Server 2003.  I thought it might be due to the fact that I had so many snapshots.  Fellow ESX admins over at the VMWare communities confirmed this to be the case.

So my next step would be to combine all the snapshots and get rid of my delta’s by committing them all. I was going to run some guest updates first, so I again as always, made a snapshot. Something either guest or host related happened and the guest powered off.  When I went to turn it back on VI complained:

too many levels of redo logs

uh oh! the guest wouldn’t turn on!  it turns out that this snapshot was in fact the 35th snapshot taken for this machine. This 35th snapshot didn’t complete correctly and was corrupted. it also turns out there is a 32 snapshot limit for VM guests.  Good to know VI tells you about this! <sarcasm included>

After frantically searching the web and forums for a solutions someone pointed me in the direction of a post here: http://zealkabi.blogspot.com/2008/10/virtualcenter-shows-no-snapshot-but-it.html which clearly shows the process i need to use to commit my snapshots, specifically Solution B:

If solution A did not work then next step to follow is: use vmkfstools -i to consolidate snapshots.
1. You can export the disk with vmkfstools to recreate the virtual machine:
2. Execute the following command to create a directory for the new disk:\
# mkdir /vmfs/volumes/UUID/new_RHEL5
3. Execute the following command to point vmkfstools at the last snapshot file:
# vmkfstools -i RHEL5-000001.vmdk /vmfs/volumes/UUID/new_RHEL5/new_RHEL5.vmdk

Three hours later, snapshots 32 through 1 committed and a single vmdk.  While this commit process was running I realized i could have simply told VI to run vmdisk00032.vmdk instead of the final (and corrupt) vmdisk00035.vmdk.  this would have been the quick resolution to get me back up and running, and I could have don the snapshot committal at a better time.

Huge thanks to patrickds from the VMWare communities and SANJAT KABI (http://zealkabi.blogspot.com) for their knowledge!

Just noticed this syslog appliance was recently released. its free and runs rsyslog and phplogcon for the web interface. its pretty powerful, and it comes working out of the box…oh ya and it’s free.

One note - GD seems to be broken in the current release. its as simple as “sudo apt-get install php5-gd” then “sudo /etc/init.d/apache2 restart”

Syslog Appliance