An error occurred during configuration of the HA agent on the Host
Enable firewall failed: vim.fault.HostConfigFault

Strange.  Looks like neither ESX hosts are talking to each other.  VMWare communities suggest i run:

[root@syr-nav-esx-03 root]# esxcfg-firewall -q aamclient

And it gave me:

Can’t locate XML/DOM.pm in @INC (@INC contains: /usr/lib/vmware/esx-perl/perl5/site_perl/5.8.0 /opt/ActivePerl-5.8/site/lib /opt/ActivePerl-5.8/lib .) at /usr/sbin/esxcfg-firewall line 21.
BEGIN failed–compilation aborted at /usr/sbin/esxcfg-firewall line 21.


Turns out the problem is from the changes Asigra Televaulting DS-Client wanted me to make to each ESX host in order to get host based backups working.  The DS-Client requires Perl to be upgraded to at least 5.8.1. And in order to make sure ESX continues to function after the Perl upgrade they states on page 419 of dsuser.pdf:

After the Perl upgrade, the vmware-cmd command may no longer work since it is not compatible
with the new Perl. You will, however, need this command in order to register restored virtual
machines. To fix this problem, go to the /usr/bin/ directory and edit the vmware-cmd file as well as
the vmware*.pl files in this directory; change the first line in each file to refer to the old Perl
installation (replace #!/usr/bin/perl with #!/usr/bin/perl.old). Run vmware-cmd without
parameters to verify that it works.

So i guess they forget to mention /usr/bin/esxcfg-firewall also needs it’s perl path changed to /usr/sbin/perl.old.  This makes me wonder what else migth brake from the Perl upgrade.