The Microsoft parts were easy.  I was sure there would be no problem those. My one concern was with the Sage Timberline software package. This program never ran particularly well, and its obvious it is running using very old programming languages much like alot of the LOB apps companies use these days.

I needed to make sure this package would both a) run within a VMware environment and b) be supported by Sage from a technical perspective. When I called the Sage Customer Support desk and asked their response was:

Sage Timberline Office is not supported running as a virtual machine, however we will troubleshoot any issues you have to a reasonable extent, while not allowing any support calls to be escalated to the engineering department.”

That actually sounds pretty fair. The woman I spoke with sounded fairly knowledgable about the Sage product and had been one of the technicians to previously work with another Sage customer that had issues running their software in a virtual environment. I asked her what the specific issues encountered were and all she really could tell me was that the Pervasive Database performed very badly and lots of unexpected things would happen while using the software.

So here I was – ready to get a signature for a proposal that would cost my customer 10′s of thousands of dollars and I had no factual evidence that showed that the accounting package they used would even work. If there were ever going to be a problem with this software it would be related to performance alone, right? There only about 10 users that access this application, so that shouldn’t really be an issue, right?

Well it turns out Sage Timberline Accounting software runs just fine in a virtual environment. It runs better running in VMware ESX 4 than it ever did on its own dedicated Dell Poweredge 1650 with RAID5.

I guess the moral of the story is that if it should work and it isn’t, you’re probably doing something wrong. Fight through the problems with a sense of resolve, do your research, follow guidelines and procedures and most things will run just fine in a virtual environment….and never take no for an answer.

Today I realized it’s also beneficial to duplicate the HOSTS file on a machine that is being converted from physical to virtual using vCenter Converter (the machine I am converting). Before I appended those entries to the I was receiving “FAILED: A system error encountered during an operation.” each time I tried running the conversion.  The error cleared up right away.

I still don’t know why this continues to be an issue. The infrastructures are setup as vanilla and as “by the book” as possible.  DNS and WINS are always setup and working properly.  Short name and FQDN resolution always works from every host and client on the network.

I suppose it isn’t the end of the world. Seems like it should be an easy thing for VMware to “fix”…then again maybe they don’t consider it to be broken.

Luckily I run everything within VMware ESX Environment, so no data needed to be copied from one machine to another. I simply provisioned a new Windows Server 2003 guest and added the old server’s VMDK disk to the new machine.

• The first thing you need to do is backup your old IIS configuration. In IIS Manager, right click in IIS Manager > All Tasks > Backup/Restore Configuration.

• Probably the most imporant part of the whole process is to make sure you “Encrypt the backup using password”. This makes th e IIS config “portable”. If you don’t encrypt it, you won’t be able to import it on a different system as there are sessions keys stored withing the IIS metabase.

• By default the configuration backup goes to C:\WINDOWS\system32\inetsrv\MetaBack. So you will need to grab both files it has created and move them on over to the new server. Place them in the same location on the new server because IIS doesn’t give you a choice to browse for a backup – it looks in that directory each time.

• Now you’ll restore that configuration from within IIS Manager, right click in IIS Manager > All Tasks > Backup/Restore Configuration. Choose the one you want to restore and restore. It will ask for the encryptin password.
• Now we need to restart IIS. This will actually re-create the IUSR account in the Local User Manager, which kind of impressed me. The password will be wrong for this account so you need to set it to something manually by the normal Reset Password function.
• Now you need to need to tell the metabase about that new password. Microsoft has a nice article outlining how this is done.
  http://support.microsoft.com/kb/297989
• Restart IIS and for the most part you should be good to go.

I opted to simply reset the password for the IUSR account that the metabase created automatically for me. The name was wrong (hence IUSR_computer), however this meant I wouldnt need to change all of the special IUSR permissions set for special files such as MDB and other special case files. You can easily tell the metabase to use a new user account as the IUSR account using the metabase explorer, but I’m not going to go into that right now.

After starting this process on a test VM, after 3 hours of converting it failed.  The logs gave me the following:

UNKNOWN_METHOD_FAULT(vim.fault.NotAuthenticated)

I stumbled upon a post on the vmware communities that suggested I try the following

When setting up the conversion, don’t use “convert all disks and leave current size”, use “select volumes…” and leave all volumes selected and the default options checked.

I gave it another try and voila! Conversion worked without error.

I wonder if multiple disks has anything to do with the snapshot merging problems I’ve had so many times.  As I attempt to remember specifics of all past failed attempts, I believe they’ve all had multiple disks.  Maybe that was the problem all along?  As long as this Plan B works I’m happy!

The process of merging the two sets to tools together was simple enough, and successful enough – but doesnt it feel great when something is really supported and when it works the way it’s supposed to? For the shiny new supported VMWare Tools I simply ran ./vmware-install.pl from the console of my Guest, waited about 5 minutes and voila!

ESX 3.5 Update 3 Release Notes

When I first attempted to Scan for Updates I received the following error in VI:

metadata for patch missing

Searching through the forums suggested the ESX host FW wasn’t properly listening for connections and suggested I run:

esxcfg-firewall --openport 80,tcp,out,dynamicupdates
(port 80 was specific to my environment and may vary for your setup)

Once I did this the Scan for Updates ran just fine

Onto the upgrade….

Before the upgrade

[root@esx-03 root]# vimsh -n -e 'hostsvc/hostsummary' | grep fullName
File not found.
fullName = "VMware ESX Server 3.5.0 build-82663",

1.Reinstalled Update Manager and attached the Critical and Non-critical Baselines to the ESX hosts

2.Put my first ESX host into Maintenance Mode and agreed to all the informational messages that come with that

3.Selected all Guest VMs and migrated them to another ESX host (I’m still not absolutely sure if entering Maintenance Mode actually evacuates all the running VMs.  I just migrate them manually because I can see it working)

4.Right clicked on the ESX host and chose Remediate, chose both Baselines and Next’ed through the wizard.

5.30 minutes later the ESX host was back up and running with the newest patches and updates. Repeat steps on all other Hosts.

After the upgrade

[root@esx-03 root]# vimsh -n -e 'hostsvc/hostsummary' | grep fullName
File not found.
fullName = "VMware ESX Server 3.5.0 build-123630",

One thing I noticed after the upgrade was that the SSH server didn’t automatically start even though it was set to

I clicked START and all was well.  I will do more troubleshooting on this the next time I need to reboot the Hosts.  I’d rather not evacuate all my Guest VMs for this reason alone. For now I’m happy being up to date.

• Backup the data like you would a normal machine by attaching via SMB, NFS, SQL, etc.  Just get the data
• Backup the Guest VM via the ESX host (pull a bare metal snapshot of the entire system)

Why backup everything twice you ask?

Well fortunately for you and unfortunately for me I have a real world experience that will help answer this question. I had a vm guest failure and had to rebuilt a vm then restore data and settings.  Suffice to say this took forever. So here’s why you can do both…

• If data is deleted you have backup sets that have just pure data – files that are used by applications and users. If a single file gets deleted, corrupted, or anything else _bad_ the restore for this file(s) is quick and easy.  Restoration of this file(s) doesn’t need to affect every other user connected to the system in question.
• If the OS becomes unusable, system files fail, vmdk (virtual disk) files on the host get corrupted then you now have the full machine backup to simply turn back on from date X/Y/Z. Asigra actually takes a native ESX snapshot at the time of backup.

Using Asigra Televaulting (or a small handful of other backup systems) allows us to perform backups that are compressed and bit level.  So, if I have a VM that has a 20GB vmdk disk, but really only has 8GB of data in the guest system, then the backups will actually be less than 8 because all that free space will be compressed down to nothing and then the 8GB of real data will further be compressed down.

My schedules are now setup to backup real data each night, but backup the OS (again using the native ESX snapshot) once a week.  For the most part, system settings and configurations aren’t happening each day. So, if the guest VM dies, I can simply restore the VM to the last weekly backup (as of at the most 6 days) then restore the real data to to that machine (as of at the most the night before).

• A note for Asigra users – I’ve had a lot of problems backing up via VI.  So far attaching directly to the ESX hosts seems to be working great.  The one fallback for this is you need to setup rules to ensure that VM’s stay on the same ESX host.  Asigra only knows that a guest VM is on the host you originally configured it to be backed up from.

So my next step would be to combine all the snapshots and get rid of my delta’s by committing them all. I was going to run some guest updates first, so I again as always, made a snapshot. Something either guest or host related happened and the guest powered off.  When I went to turn it back on VI complained:

too many levels of redo logs

uh oh! the guest wouldn’t turn on!  it turns out that this snapshot was in fact the 35th snapshot taken for this machine. This 35th snapshot didn’t complete correctly and was corrupted. it also turns out there is a 32 snapshot limit for VM guests.  Good to know VI tells you about this! <sarcasm included>

After frantically searching the web and forums for a solutions someone pointed me in the direction of a post here: http://zealkabi.blogspot.com/2008/10/virtualcenter-shows-no-snapshot-but-it.html which clearly shows the process i need to use to commit my snapshots, specifically Solution B:

If solution A did not work then next step to follow is: use vmkfstools -i to consolidate snapshots.
1. You can export the disk with vmkfstools to recreate the virtual machine:
2. Execute the following command to create a directory for the new disk:\
# mkdir /vmfs/volumes/UUID/new_RHEL5
3. Execute the following command to point vmkfstools at the last snapshot file:
# vmkfstools -i RHEL5-000001.vmdk /vmfs/volumes/UUID/new_RHEL5/new_RHEL5.vmdk

Three hours later, snapshots 32 through 1 committed and a single vmdk.  While this commit process was running I realized i could have simply told VI to run vmdisk00032.vmdk instead of the final (and corrupt) vmdisk00035.vmdk.  this would have been the quick resolution to get me back up and running, and I could have don the snapshot committal at a better time.

Huge thanks to patrickds from the VMWare communities and SANJAT KABI (http://zealkabi.blogspot.com) for their knowledge!