2012/02/14

ESXTOP always run in batch mode

A colleague of mine pointed this one out. Seems something to do with the TERM parameter when you ssh to an ESX host. If you change it to plain xterm it works again in regular mode

/vmfs/volumes/4d2750de-8956fe05-a7a9-e61f133c816b # echo $TERM
xterm-color
/vmfs/volumes/4d2750de-8956fe05-a7a9-e61f133c816b # export TERM=xterm
/vmfs/volumes/4d2750de-8956fe05-a7a9-e61f133c816b # esxtop

Update ESX5: The host returns esxupdate error code:7.

Today I tried to upgrade some ESXi hosts via Update Manager. It's a nice tool since it can orchestrate your upgrade. However after the orchestration I was left with one host that didn't want to be update (Actually 2, one had a locked vm where I needed to do /etc/init.d/hostd restart so that the running tasks on ESX where cancelled).

The one that didn't want to upgrade gave me the following error in vcenter
Stage patches to entity
esx12
The host returns esxupdate error code:7.
Cannot download VIB. Check the Update
Manager log files and esxupdate log files
for more details.
....
14/02/2012 12:28:38
14/02/2012 12:28:38
14/02/2012 12:29:02

Strange, the errors seems to be a firewall issue but my other host did just fin. I went to look in the esxupdate.log

tail -n 100 -f /var/log/esxupdate.log
2012-02-14T11:28:55Z esxupdate: HostImage: INFO: Attempting to download VIB vmware-fdm
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: An esxupdate error exception was caught:
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: Traceback (most recent call last):
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/usr/sbin/esxupdate", line 216, in main
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: cmd.Run()
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esx5update/Cmdline.py", line 144, in Run
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/Transaction.py", line 243, in InstallVibsFromSources
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/Transaction.py", line 345, in _installVibs
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/Transaction.py", line 388, in _validateAndInstallProfile
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/HostImage.py", line 630, in Stage
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/HostImage.py", line 414, in _download_and_stage
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: VibDownloadError: (, 'Unable to download VIB from any of the URLs ')
2012-02-14T11:28:55Z esxupdate: esxupdate: DEBUG: <<<
2012-02-14T11:28:59Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetTimeout']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:28:59Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetRetries']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:28:59Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetRateLimit']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:28:59Z esxupdate: esxupdate: INFO: ---
Command: scan
Args: ['scan']
Options: {'nosigcheck': None, 'retry': 5, 'loglevel': None, 'cleancache': None, 'viburls': None, 'meta': ['http://d2vce01.d2:9084/vum/repository/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip'], 'proxyurl': None, 'timeout': 30.0, 'cachesize': None, 'hamode': True, 'maintenancemode': None}
2012-02-14T11:28:59Z esxupdate: BootBankInstaller.pyc: INFO: Unrecognized value "title=Loading VMware ESXi" in boot.cfg
2012-02-14T11:28:59Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/bootOption', '-rp']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:29:00Z esxupdate: downloader: DEBUG: Downloading http://d2vce01.d2:9084/vum/repository/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip to /tmp/tmpk1SPPZ...
2012-02-14T11:29:00Z esxupdate: Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file
2012-02-14T11:29:00Z esxupdate: BootBankInstaller.pyc: INFO: Unrecognized value "title=Loading VMware ESXi" in boot.cfg
2012-02-14T11:29:00Z esxupdate: HostImage: DEBUG: Live image has been updated but /altbootbank image has not. This means a reboot is not safe.
2012-02-14T11:29:00Z esxupdate: HostImage: DEBUG: Live image has been updated but /altbootbank image has not. This means a reboot is not safe.
2012-02-14T11:29:00Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/usr/sbin/vsish', '-e', '-p', 'cat', '/hardware/bios/dmiInfo']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:29:00Z esxupdate: esxupdate: INFO: All done!
2012-02-14T11:29:00Z esxupdate: esxupdate: DEBUG: <<<

And saw that there was a problem with the fdm (HA) agent package. I tried to uninstall via
/tmp # /opt/vmware/uninstallers/VMware-fdm-uninstall.sh
but this didn't really help


Finally I decided to do manual update so that repo/downloading would not be the issue. Again I was faced with a similar error
/vmfs/volumes/4d0f342d-b384b59f-b015-e61f133c792b # esxcli software vib install -d=/scratch/ESXi500-201109001.zip
[VibDownloadError]
Failed to download VIB.
url = vmware-fdm-5.0.0-455964
localfile = Unable to download VIB from any of the URLs
Please refer to the log file for more details.

No good so I decided to try to uninstall the fdm package, maybe that would help. After all if the host would exist the package would be pushed anyway
/vmfs/volumes/4d0f342d-b384b59f-b015-e61f133c792b # esxcli software vib remove -n=vmware-fdm
Removal Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed:
VIBs Removed: VMware_bootbank_vmware-fdm_5.0.0-455964
VIBs Skipped:

After that I could manually install the packages so once again I was a happy camper/vmfs/volumes/4d0f342d-b384b59f-b015-e61f133c792b # esxcli software vib install -d=/scratch/ESXi500-201109001.zip
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: VMware_bootbank_esx-base_5.0.0-0.3.474610, VMware_locker_tools-light_5.0.0-0.3.474610
VIBs Removed: VMware_bootbank_esx-base_5.0.0-0.0.469512, VMware_locker_tools-light_5.0.0-0.0.469512
VIBs Skipped: VMware_bootbank_ata-pata-amd_0.3.10-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-atiixp_0.4.6-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-cmd64x_0.2.5-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-pdc2027x_1.0-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-serverworks_0.4.3-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-sil680_0.4.8-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-via_0.3.3-2vmw.500.0.0.469512, VMware_bootbank_block-cciss_3.6.14-10vmw.500.0.0.469512, VMware_bootbank_ehci-ehci-hcd_1.0-3vmw.500.0.0.469512, VMware_bootbank_esx-tboot_5.0.0-0.0.469512, VMware_bootbank_ima-qla4xxx_2.01.07-1vmw.500.0.0.469512, VMware_bootbank_ipmi-ipmi-devintf_39.1-4vmw.500.0.0.469512, VMware_bootbank_ipmi-ipmi-msghandler_39.1-4vmw.500.0.0.469512, VMware_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.500.0.0.469512, VMware_bootbank_misc-cnic-register_1.1-1vmw.500.0.0.469512, VMware_bootbank_misc-drivers_5.0.0-0.0.469512, VMware_bootbank_net-be2net_4.0.88.0-1vmw.500.0.0.469512, VMware_bootbank_net-bnx2_2.0.15g.v50.11-5vmw.500.0.0.469512, VMware_bootbank_net-bnx2x_1.61.15.v50.1-1vmw.500.0.0.469512, VMware_bootbank_net-cnic_1.10.2j.v50.7-2vmw.500.0.0.469512, VMware_bootbank_net-e1000_8.0.3.1-2vmw.500.0.0.469512, VMware_bootbank_net-e1000e_1.1.2-3vmw.500.0.0.469512, VMware_bootbank_net-enic_1.4.2.15a-1vmw.500.0.0.469512, VMware_bootbank_net-forcedeth_0.61-2vmw.500.0.0.469512, VMware_bootbank_net-igb_2.1.11.1-3vmw.500.0.0.469512, VMware_bootbank_net-ixgbe_2.0.84.8.2-10vmw.500.0.0.469512, VMware_bootbank_net-nx-nic_4.0.557-3vmw.500.0.0.469512, VMware_bootbank_net-r8168_8.013.00-3vmw.500.0.0.469512, VMware_bootbank_net-r8169_6.011.00-2vmw.500.0.0.469512, VMware_bootbank_net-s2io_2.1.4.13427-3vmw.500.0.0.469512, VMware_bootbank_net-sky2_1.20-2vmw.500.0.0.469512, VMware_bootbank_net-tg3_3.110h.v50.4-4vmw.500.0.0.469512, VMware_bootbank_ohci-usb-ohci_1.0-3vmw.500.0.0.469512, VMware_bootbank_sata-ahci_3.0-6vmw.500.0.0.469512, VMware_bootbank_sata-ata-piix_2.12-4vmw.500.0.0.469512, VMware_bootbank_sata-sata-nv_3.5-3vmw.500.0.0.469512, VMware_bootbank_sata-sata-promise_2.12-3vmw.500.0.0.469512, VMware_bootbank_sata-sata-sil_2.3-3vmw.500.0.0.469512, VMware_bootbank_sata-sata-svw_2.3-3vmw.500.0.0.469512, VMware_bootbank_scsi-aacraid_1.1.5.1-9vmw.500.0.0.469512, VMware_bootbank_scsi-adp94xx_1.0.8.12-6vmw.500.0.0.469512, VMware_bootbank_scsi-aic79xx_3.1-5vmw.500.0.0.469512, VMware_bootbank_scsi-bnx2i_1.9.1d.v50.1-3vmw.500.0.0.469512, VMware_bootbank_scsi-fnic_1.5.0.3-1vmw.500.0.0.469512, VMware_bootbank_scsi-hpsa_5.0.0-17vmw.500.0.0.469512, VMware_bootbank_scsi-ips_7.12.05-4vmw.500.0.0.469512, VMware_bootbank_scsi-lpfc820_8.2.2.1-18vmw.500.0.0.469512, VMware_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.500.0.0.469512, VMware_bootbank_scsi-megaraid-sas_4.32-1vmw.500.0.0.469512, VMware_bootbank_scsi-megaraid2_2.00.4-9vmw.500.0.0.469512, VMware_bootbank_scsi-mpt2sas_06.00.00.00-5vmw.500.0.0.469512, VMware_bootbank_scsi-mptsas_4.23.01.00-5vmw.500.0.0.469512, VMware_bootbank_scsi-mptspi_4.23.01.00-5vmw.500.0.0.469512, VMware_bootbank_scsi-qla2xxx_901.k1.1-14vmw.500.0.0.469512, VMware_bootbank_scsi-qla4xxx_5.01.03.2-3vmw.500.0.0.469512, VMware_bootbank_uhci-usb-uhci_1.0-3vmw.500.0.0.469512