2012/02/14

Update ESX5: The host returns esxupdate error code:7.

Today I tried to upgrade some ESXi hosts via Update Manager. It's a nice tool since it can orchestrate your upgrade. However after the orchestration I was left with one host that didn't want to be update (Actually 2, one had a locked vm where I needed to do /etc/init.d/hostd restart so that the running tasks on ESX where cancelled).

The one that didn't want to upgrade gave me the following error in vcenter
Stage patches to entity
esx12
The host returns esxupdate error code:7.
Cannot download VIB. Check the Update
Manager log files and esxupdate log files
for more details.
....
14/02/2012 12:28:38
14/02/2012 12:28:38
14/02/2012 12:29:02

Strange, the errors seems to be a firewall issue but my other host did just fin. I went to look in the esxupdate.log

tail -n 100 -f /var/log/esxupdate.log
2012-02-14T11:28:55Z esxupdate: HostImage: INFO: Attempting to download VIB vmware-fdm
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: An esxupdate error exception was caught:
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: Traceback (most recent call last):
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/usr/sbin/esxupdate", line 216, in main
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: cmd.Run()
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esx5update/Cmdline.py", line 144, in Run
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/Transaction.py", line 243, in InstallVibsFromSources
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/Transaction.py", line 345, in _installVibs
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/Transaction.py", line 388, in _validateAndInstallProfile
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/HostImage.py", line 630, in Stage
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/HostImage.py", line 414, in _download_and_stage
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: VibDownloadError: (, 'Unable to download VIB from any of the URLs ')
2012-02-14T11:28:55Z esxupdate: esxupdate: DEBUG: <<<
2012-02-14T11:28:59Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetTimeout']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:28:59Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetRetries']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:28:59Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetRateLimit']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:28:59Z esxupdate: esxupdate: INFO: ---
Command: scan
Args: ['scan']
Options: {'nosigcheck': None, 'retry': 5, 'loglevel': None, 'cleancache': None, 'viburls': None, 'meta': ['http://d2vce01.d2:9084/vum/repository/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip'], 'proxyurl': None, 'timeout': 30.0, 'cachesize': None, 'hamode': True, 'maintenancemode': None}
2012-02-14T11:28:59Z esxupdate: BootBankInstaller.pyc: INFO: Unrecognized value "title=Loading VMware ESXi" in boot.cfg
2012-02-14T11:28:59Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/bootOption', '-rp']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:29:00Z esxupdate: downloader: DEBUG: Downloading http://d2vce01.d2:9084/vum/repository/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip to /tmp/tmpk1SPPZ...
2012-02-14T11:29:00Z esxupdate: Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file
2012-02-14T11:29:00Z esxupdate: BootBankInstaller.pyc: INFO: Unrecognized value "title=Loading VMware ESXi" in boot.cfg
2012-02-14T11:29:00Z esxupdate: HostImage: DEBUG: Live image has been updated but /altbootbank image has not. This means a reboot is not safe.
2012-02-14T11:29:00Z esxupdate: HostImage: DEBUG: Live image has been updated but /altbootbank image has not. This means a reboot is not safe.
2012-02-14T11:29:00Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/usr/sbin/vsish', '-e', '-p', 'cat', '/hardware/bios/dmiInfo']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:29:00Z esxupdate: esxupdate: INFO: All done!
2012-02-14T11:29:00Z esxupdate: esxupdate: DEBUG: <<<

And saw that there was a problem with the fdm (HA) agent package. I tried to uninstall via
/tmp # /opt/vmware/uninstallers/VMware-fdm-uninstall.sh
but this didn't really help


Finally I decided to do manual update so that repo/downloading would not be the issue. Again I was faced with a similar error
/vmfs/volumes/4d0f342d-b384b59f-b015-e61f133c792b # esxcli software vib install -d=/scratch/ESXi500-201109001.zip
[VibDownloadError]
Failed to download VIB.
url = vmware-fdm-5.0.0-455964
localfile = Unable to download VIB from any of the URLs
Please refer to the log file for more details.

No good so I decided to try to uninstall the fdm package, maybe that would help. After all if the host would exist the package would be pushed anyway
/vmfs/volumes/4d0f342d-b384b59f-b015-e61f133c792b # esxcli software vib remove -n=vmware-fdm
Removal Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed:
VIBs Removed: VMware_bootbank_vmware-fdm_5.0.0-455964
VIBs Skipped:

After that I could manually install the packages so once again I was a happy camper/vmfs/volumes/4d0f342d-b384b59f-b015-e61f133c792b # esxcli software vib install -d=/scratch/ESXi500-201109001.zip
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: VMware_bootbank_esx-base_5.0.0-0.3.474610, VMware_locker_tools-light_5.0.0-0.3.474610
VIBs Removed: VMware_bootbank_esx-base_5.0.0-0.0.469512, VMware_locker_tools-light_5.0.0-0.0.469512
VIBs Skipped: VMware_bootbank_ata-pata-amd_0.3.10-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-atiixp_0.4.6-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-cmd64x_0.2.5-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-pdc2027x_1.0-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-serverworks_0.4.3-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-sil680_0.4.8-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-via_0.3.3-2vmw.500.0.0.469512, VMware_bootbank_block-cciss_3.6.14-10vmw.500.0.0.469512, VMware_bootbank_ehci-ehci-hcd_1.0-3vmw.500.0.0.469512, VMware_bootbank_esx-tboot_5.0.0-0.0.469512, VMware_bootbank_ima-qla4xxx_2.01.07-1vmw.500.0.0.469512, VMware_bootbank_ipmi-ipmi-devintf_39.1-4vmw.500.0.0.469512, VMware_bootbank_ipmi-ipmi-msghandler_39.1-4vmw.500.0.0.469512, VMware_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.500.0.0.469512, VMware_bootbank_misc-cnic-register_1.1-1vmw.500.0.0.469512, VMware_bootbank_misc-drivers_5.0.0-0.0.469512, VMware_bootbank_net-be2net_4.0.88.0-1vmw.500.0.0.469512, VMware_bootbank_net-bnx2_2.0.15g.v50.11-5vmw.500.0.0.469512, VMware_bootbank_net-bnx2x_1.61.15.v50.1-1vmw.500.0.0.469512, VMware_bootbank_net-cnic_1.10.2j.v50.7-2vmw.500.0.0.469512, VMware_bootbank_net-e1000_8.0.3.1-2vmw.500.0.0.469512, VMware_bootbank_net-e1000e_1.1.2-3vmw.500.0.0.469512, VMware_bootbank_net-enic_1.4.2.15a-1vmw.500.0.0.469512, VMware_bootbank_net-forcedeth_0.61-2vmw.500.0.0.469512, VMware_bootbank_net-igb_2.1.11.1-3vmw.500.0.0.469512, VMware_bootbank_net-ixgbe_2.0.84.8.2-10vmw.500.0.0.469512, VMware_bootbank_net-nx-nic_4.0.557-3vmw.500.0.0.469512, VMware_bootbank_net-r8168_8.013.00-3vmw.500.0.0.469512, VMware_bootbank_net-r8169_6.011.00-2vmw.500.0.0.469512, VMware_bootbank_net-s2io_2.1.4.13427-3vmw.500.0.0.469512, VMware_bootbank_net-sky2_1.20-2vmw.500.0.0.469512, VMware_bootbank_net-tg3_3.110h.v50.4-4vmw.500.0.0.469512, VMware_bootbank_ohci-usb-ohci_1.0-3vmw.500.0.0.469512, VMware_bootbank_sata-ahci_3.0-6vmw.500.0.0.469512, VMware_bootbank_sata-ata-piix_2.12-4vmw.500.0.0.469512, VMware_bootbank_sata-sata-nv_3.5-3vmw.500.0.0.469512, VMware_bootbank_sata-sata-promise_2.12-3vmw.500.0.0.469512, VMware_bootbank_sata-sata-sil_2.3-3vmw.500.0.0.469512, VMware_bootbank_sata-sata-svw_2.3-3vmw.500.0.0.469512, VMware_bootbank_scsi-aacraid_1.1.5.1-9vmw.500.0.0.469512, VMware_bootbank_scsi-adp94xx_1.0.8.12-6vmw.500.0.0.469512, VMware_bootbank_scsi-aic79xx_3.1-5vmw.500.0.0.469512, VMware_bootbank_scsi-bnx2i_1.9.1d.v50.1-3vmw.500.0.0.469512, VMware_bootbank_scsi-fnic_1.5.0.3-1vmw.500.0.0.469512, VMware_bootbank_scsi-hpsa_5.0.0-17vmw.500.0.0.469512, VMware_bootbank_scsi-ips_7.12.05-4vmw.500.0.0.469512, VMware_bootbank_scsi-lpfc820_8.2.2.1-18vmw.500.0.0.469512, VMware_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.500.0.0.469512, VMware_bootbank_scsi-megaraid-sas_4.32-1vmw.500.0.0.469512, VMware_bootbank_scsi-megaraid2_2.00.4-9vmw.500.0.0.469512, VMware_bootbank_scsi-mpt2sas_06.00.00.00-5vmw.500.0.0.469512, VMware_bootbank_scsi-mptsas_4.23.01.00-5vmw.500.0.0.469512, VMware_bootbank_scsi-mptspi_4.23.01.00-5vmw.500.0.0.469512, VMware_bootbank_scsi-qla2xxx_901.k1.1-14vmw.500.0.0.469512, VMware_bootbank_scsi-qla4xxx_5.01.03.2-3vmw.500.0.0.469512, VMware_bootbank_uhci-usb-uhci_1.0-3vmw.500.0.0.469512

10 comments:

  1. Thanks, very helpful post. I had a similar issue. Tried to apply critical updates/extensions via baseline.

    It seems to bomb after the vmware-fdm line every time.

    ~ # cat /var/log/esxupdate.log | grep -i "vmware-fdm" -A3

    2012-05-08T17:32:30Z esxupdate: HostImage: INFO: Attempting to download VIB vmware-fdm
    2012-05-08T17:32:30Z esxupdate: esxupdate: ERROR: An esxupdate error exception was caught:
    2012-05-08T17:32:30Z esxupdate: esxupdate: ERROR: Traceback (most recent call last):

    Manually removed it...
    ~ # esxcli software vib remove -n=vmware-fdm
    Removal Result
    Message: Operation finished successfully.
    Reboot Required: false
    VIBs Installed:
    VIBs Removed: VMware_bootbank_vmware-fdm_5.0.0-623373
    VIBs Skipped:

    Baseline policies applied from Update Manager without issue.
    Rebooted and the host rejoined the cluster with HA reconfigured.

    ReplyDelete
  2. I also ran into this issue. Your fix worked for me. After removing the vmware-fdm VIB, I was able to Stage and then Remediate through the Update Manager tab.

    Thank you!

    ReplyDelete
  3. Thanks for the help! If you're looking for the command to execute the command on your server, it's:esxcli --server X.X.X.X software vib remove -n=vmware-fdm

    ReplyDelete
  4. You are the man Timo! Much respect

    ReplyDelete
  5. Awesome, this fixed my problem right away. Saved me many many possible headaches tonight.

    ReplyDelete
  6. This also worked for me.
    Got the error when trying to install the CIM Provider for a qLogic HBA.

    After exiting maintenance mode the HA agent (including fdm) gets reinstalled, so you don't even lose any functionality.

    ReplyDelete
  7. I also have the same issue. When I try to stage my ESXi 5.1 host I got error: The host returns esxupdate error code:7. Cannot download VIB.Check the Update Manager log files and esxupdate log files for more details.

    I just rebooted my host, and after that everything works perfectly.

    ReplyDelete
  8. I was also getting "The host returns esxupdate error code:7. Cannot download VIB.Check the UpdateManager log files and esxupdate log files for more details."
    I ran:
    /tmp # cp /opt/vmware/uninstallers/VMware-fdm-uninstall.sh /tmp/VMware-fdm-unins
    tall.sh
    /tmp # ./VMware-fdm-uninstall.sh

    The host was then able to begin updating again from VUM

    ReplyDelete
  9. This solution worked perfectly. Thanks for taking the time to post it.

    ReplyDelete