I noticed when i did some maintenance and rebooted the ESX hosts and i could not get High Availability (HA) back up and running.  I kept receiving the error:

An error occurred during configuration of the HA agent on the Host
Enable firewall failed:  vim.fault.HostConfigFault

Strange.  Looks like neither ESX hosts are talking to each other.  VMWare communities suggest i run:

[root@syr-nav-esx-03 root]# esxcfg-firewall -q aamclient

And it gave me:

Can’t locate XML/DOM.pm in @INC (@INC contains: /usr/lib/vmware/esx-perl/perl5/site_perl/5.8.0 /opt/ActivePerl-5.8/site/lib /opt/ActivePerl-5.8/lib .) at /usr/sbin/esxcfg-firewall line 21.
BEGIN failed–compilation aborted at /usr/sbin/esxcfg-firewall line 21.

Turns out the problem is from the changes Asigra Televaulting DS-Client wanted me to make to each ESX host in order to get host based backups working.  The DS-Client requires Perl to be upgraded to at least 5.8.1. And in order to make sure ESX continues to function after the Perl upgrade they states on page 419 of dsuser.pdf:

After the Perl upgrade, the vmware-cmd command may no longer work since it is not compatible
with the new Perl. You will, however, need this command in order to register restored virtual
machines. To fix this problem, go to the /usr/bin/ directory and edit the vmware-cmd file as well as
the vmware*.pl files in this directory; change the first line in each file to refer to the old Perl
installation (replace #!/usr/bin/perl with #!/usr/bin/perl.old). Run vmware-cmd without
parameters to verify that it works.

So i guess they forget to mention /usr/bin/esxcfg-firewall also needs it’s perl path changed to /usr/sbin/perl.old.  This makes me wonder what else migth brake from the Perl upgrade.

My approach to backing up guest VM’s from ESX3.5 is two-headed:

• Backup the data like you would a normal machine by attaching via SMB, NFS, SQL, etc.  Just get the data
• Backup the Guest VM via the ESX host (pull a bare metal snapshot of the entire system)

Why backup everything twice you ask?

Well fortunately for you and unfortunately for me I have a real world experience that will help answer this question. I had a vm guest failure and had to rebuilt a vm then restore data and settings.  Suffice to say this took forever. So here’s why you can do both…

• If data is deleted you have backup sets that have just pure data - files that are used by applications and users. If a single file gets deleted, corrupted, or anything else _bad_ the restore for this file(s) is quick and easy.  Restoration of this file(s) doesn’t need to affect every other user connected to the system in question.
• If the OS becomes unusable, system files fail, vmdk (virtual disk) files on the host get corrupted then you now have the full machine backup to simply turn back on from date X/Y/Z. Asigra actually takes a native ESX snapshot at the time of backup.

Using Asigra Televaulting (or a small handful of other backup systems) allows us to perform backups that are compressed and bit level.  So, if I have a VM that has a 20GB vmdk disk, but really only has 8GB of data in the guest system, then the backups will actually be less than 8 because all that free space will be compressed down to nothing and then the 8GB of real data will further be compressed down.

My schedules are now setup to backup real data each night, but backup the OS (again using the native ESX snapshot) once a week.  For the most part, system settings and configurations aren’t happening each day. So, if the guest VM dies, I can simply restore the VM to the last weekly backup (as of at the most 6 days) then restore the real data to to that machine (as of at the most the night before).

A note for Asigra users - I’ve had a lot of problems backing up via VI.  So far attaching directly to the ESX hosts seems to be working great.  The one fallback for this is you need to setup rules to ensure that VM’s stay on the same ESX host.  Asigra only knows that a guest VM is on the host you originally configured it to be backed up from.

Anyone that knows me at all knows I’m a Crackberry Addict. I’ve been using a Blackberry for many years, I’ve converted many people to Blackberries, and I administer a big handful of Blackberry Servers.     

A while back RIM started giving away a version of BES (BES Express) that was the same as the normal BES except for a 15 user maximum limit.  User CALs for this version of the software still cost $100 each just like normal BES CALs, however you didn’t have to pay the ~$3000 for the full blown BES.

Well several months ago RIM stopped offering BES Express and changed it to Blackberry Professional Server (BPS).  BPS now had a 30 user limit.  Awesome! Mostly. RIM touted BPS as the new small business version of BES.  All the same great features and benefits. However the major change in this is that it is technically a completely different product.  Updates are released for BES separately from the updates for BPS.

Why does this matter you ask?  HTML and attachment editing support of course! A few months ago RIM announced a firmware upgrade for Blackberry devices that would support HTML and attachment editing.  This firmware update only does any good for you if the BES or BIS server you are using is also up to date with these new features. To this date, the service packs that have this new functionality built into them are still not available for BPS. In my eyes this is a travesty. RIM allows you to purchase an "Upgrade Key" for BPS to BES, but this of course will cost you the full price of BES - $3000.  This just isn’t possible for most companies running BPS - that’s why we’re running it in the first place of course!

I implore RIM to moving on this.  Of all companies RIM should be well aware that as more time goes on there will become many more alternate solutions to the Blackberry platform. I don’t imagine it is a good idea to upset the small business market.  One day we might not be a small business anymore and we might want to spend our $3000 on a different platform.

So I was having some pretty significant performance problems with a vm running Windows Server 2003.  I thought it might be due to the fact that I had so many snapshots.  Fellow ESX admins over at the VMWare communities confirmed this to be the case.

So my next step would be to combine all the snapshots and get rid of my delta’s by committing them all. I was going to run some guest updates first, so I again as always, made a snapshot. Something either guest or host related happened and the guest powered off.  When I went to turn it back on VI complained:

too many levels of redo logs

uh oh! the guest wouldn’t turn on!  it turns out that this snapshot was in fact the 35th snapshot taken for this machine. This 35th snapshot didn’t complete correctly and was corrupted. it also turns out there is a 32 snapshot limit for VM guests.  Good to know VI tells you about this! <sarcasm included>

After frantically searching the web and forums for a solutions someone pointed me in the direction of a post here: http://zealkabi.blogspot.com/2008/10/virtualcenter-shows-no-snapshot-but-it.html which clearly shows the process i need to use to commit my snapshots, specifically Solution B:

If solution A did not work then next step to follow is: use vmkfstools -i to consolidate snapshots.
1. You can export the disk with vmkfstools to recreate the virtual machine:
2. Execute the following command to create a directory for the new disk:\
# mkdir /vmfs/volumes/UUID/new_RHEL5
3. Execute the following command to point vmkfstools at the last snapshot file:
# vmkfstools -i RHEL5-000001.vmdk /vmfs/volumes/UUID/new_RHEL5/new_RHEL5.vmdk

Three hours later, snapshots 32 through 1 committed and a single vmdk.  While this commit process was running I realized i could have simply told VI to run vmdisk00032.vmdk instead of the final (and corrupt) vmdisk00035.vmdk.  this would have been the quick resolution to get me back up and running, and I could have don the snapshot committal at a better time.

Huge thanks to patrickds from the VMWare communities and SANJAT KABI (http://zealkabi.blogspot.com) for their knowledge!

Just noticed this syslog appliance was recently released. its free and runs rsyslog and phplogcon for the web interface. its pretty powerful, and it comes working out of the box…oh ya and it’s free.

One note - GD seems to be broken in the current release. its as simple as “sudo apt-get install php5-gd” then “sudo /etc/init.d/apache2 restart”

Syslog Appliance

Logs all incoming and outgoing calls from your Blacberry phone to the calendar. Has the ability to set the entries to private or to categorize them automatically. This app has been a huge help for me to keep track of my billable hours for customers. I bet they hate though.

CallLogger

Ubuntu 8.10 due Thursday. Profits? Not so fast | Latest Business Tech News - CNET News

I don’t think it will possible to make a lot of money, or maybe any money, selling the desktop,” Shuttleworth said. “We’re not going to try to make money selling the desktop. We force ourselves to look to services-oriented business models. I remain confident this is the right business model for the industry. Linux is the forcing function that (means) the broader software industry will shift in business models away from licensing the bits and to services.

Here we are with a new Ubuntu release, albeit a short term support desktop release….it’s exciting nonetheless.

I run Ubuntu on my laptop at home and I recently changed all my FreeBSD servers to Ubuntu 8.1 Hardy Heron and so far I have no regrets. Managing updates and software packages is a cinch. My servers run lean and mean now. No extra fluff. Packages are installed on an as-needed basis and the general structure of the Ubuntu system seems to make alot more logical sense.

The only question I have over the long term is “will Ubuntu to continue to grow and stay relevant”? Many Linux distro’s have come and gone. I’ve taken a gamble in the hopes that Ubuntu will be a long term solution and I hope many others will. I think the more people use it they will see that it is just as scalable as any other distro out there and spins like a top.

Good luck to Ubuntu and to me! Support Ubuntu by downloading it here!