2012/02/14

Update ESX5: The host returns esxupdate error code:7.

Today I tried to upgrade some ESXi hosts via Update Manager. It's a nice tool since it can orchestrate your upgrade. However after the orchestration I was left with one host that didn't want to be update (Actually 2, one had a locked vm where I needed to do /etc/init.d/hostd restart so that the running tasks on ESX where cancelled).

The one that didn't want to upgrade gave me the following error in vcenter
Stage patches to entity
esx12
The host returns esxupdate error code:7.
Cannot download VIB. Check the Update
Manager log files and esxupdate log files
for more details.
....
14/02/2012 12:28:38
14/02/2012 12:28:38
14/02/2012 12:29:02

Strange, the errors seems to be a firewall issue but my other host did just fin. I went to look in the esxupdate.log

tail -n 100 -f /var/log/esxupdate.log
2012-02-14T11:28:55Z esxupdate: HostImage: INFO: Attempting to download VIB vmware-fdm
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: An esxupdate error exception was caught:
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: Traceback (most recent call last):
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/usr/sbin/esxupdate", line 216, in main
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: cmd.Run()
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esx5update/Cmdline.py", line 144, in Run
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/Transaction.py", line 243, in InstallVibsFromSources
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/Transaction.py", line 345, in _installVibs
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/Transaction.py", line 388, in _validateAndInstallProfile
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/HostImage.py", line 630, in Stage
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/HostImage.py", line 414, in _download_and_stage
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: VibDownloadError: (, 'Unable to download VIB from any of the URLs ')
2012-02-14T11:28:55Z esxupdate: esxupdate: DEBUG: <<<
2012-02-14T11:28:59Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetTimeout']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:28:59Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetRetries']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:28:59Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetRateLimit']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:28:59Z esxupdate: esxupdate: INFO: ---
Command: scan
Args: ['scan']
Options: {'nosigcheck': None, 'retry': 5, 'loglevel': None, 'cleancache': None, 'viburls': None, 'meta': ['http://d2vce01.d2:9084/vum/repository/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip'], 'proxyurl': None, 'timeout': 30.0, 'cachesize': None, 'hamode': True, 'maintenancemode': None}
2012-02-14T11:28:59Z esxupdate: BootBankInstaller.pyc: INFO: Unrecognized value "title=Loading VMware ESXi" in boot.cfg
2012-02-14T11:28:59Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/bootOption', '-rp']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:29:00Z esxupdate: downloader: DEBUG: Downloading http://d2vce01.d2:9084/vum/repository/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip to /tmp/tmpk1SPPZ...
2012-02-14T11:29:00Z esxupdate: Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file
2012-02-14T11:29:00Z esxupdate: BootBankInstaller.pyc: INFO: Unrecognized value "title=Loading VMware ESXi" in boot.cfg
2012-02-14T11:29:00Z esxupdate: HostImage: DEBUG: Live image has been updated but /altbootbank image has not. This means a reboot is not safe.
2012-02-14T11:29:00Z esxupdate: HostImage: DEBUG: Live image has been updated but /altbootbank image has not. This means a reboot is not safe.
2012-02-14T11:29:00Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/usr/sbin/vsish', '-e', '-p', 'cat', '/hardware/bios/dmiInfo']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:29:00Z esxupdate: esxupdate: INFO: All done!
2012-02-14T11:29:00Z esxupdate: esxupdate: DEBUG: <<<

And saw that there was a problem with the fdm (HA) agent package. I tried to uninstall via
/tmp # /opt/vmware/uninstallers/VMware-fdm-uninstall.sh
but this didn't really help


Finally I decided to do manual update so that repo/downloading would not be the issue. Again I was faced with a similar error
/vmfs/volumes/4d0f342d-b384b59f-b015-e61f133c792b # esxcli software vib install -d=/scratch/ESXi500-201109001.zip
[VibDownloadError]
Failed to download VIB.
url = vmware-fdm-5.0.0-455964
localfile = Unable to download VIB from any of the URLs
Please refer to the log file for more details.

No good so I decided to try to uninstall the fdm package, maybe that would help. After all if the host would exist the package would be pushed anyway
/vmfs/volumes/4d0f342d-b384b59f-b015-e61f133c792b # esxcli software vib remove -n=vmware-fdm
Removal Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed:
VIBs Removed: VMware_bootbank_vmware-fdm_5.0.0-455964
VIBs Skipped:

After that I could manually install the packages so once again I was a happy camper/vmfs/volumes/4d0f342d-b384b59f-b015-e61f133c792b # esxcli software vib install -d=/scratch/ESXi500-201109001.zip
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: VMware_bootbank_esx-base_5.0.0-0.3.474610, VMware_locker_tools-light_5.0.0-0.3.474610
VIBs Removed: VMware_bootbank_esx-base_5.0.0-0.0.469512, VMware_locker_tools-light_5.0.0-0.0.469512
VIBs Skipped: VMware_bootbank_ata-pata-amd_0.3.10-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-atiixp_0.4.6-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-cmd64x_0.2.5-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-pdc2027x_1.0-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-serverworks_0.4.3-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-sil680_0.4.8-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-via_0.3.3-2vmw.500.0.0.469512, VMware_bootbank_block-cciss_3.6.14-10vmw.500.0.0.469512, VMware_bootbank_ehci-ehci-hcd_1.0-3vmw.500.0.0.469512, VMware_bootbank_esx-tboot_5.0.0-0.0.469512, VMware_bootbank_ima-qla4xxx_2.01.07-1vmw.500.0.0.469512, VMware_bootbank_ipmi-ipmi-devintf_39.1-4vmw.500.0.0.469512, VMware_bootbank_ipmi-ipmi-msghandler_39.1-4vmw.500.0.0.469512, VMware_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.500.0.0.469512, VMware_bootbank_misc-cnic-register_1.1-1vmw.500.0.0.469512, VMware_bootbank_misc-drivers_5.0.0-0.0.469512, VMware_bootbank_net-be2net_4.0.88.0-1vmw.500.0.0.469512, VMware_bootbank_net-bnx2_2.0.15g.v50.11-5vmw.500.0.0.469512, VMware_bootbank_net-bnx2x_1.61.15.v50.1-1vmw.500.0.0.469512, VMware_bootbank_net-cnic_1.10.2j.v50.7-2vmw.500.0.0.469512, VMware_bootbank_net-e1000_8.0.3.1-2vmw.500.0.0.469512, VMware_bootbank_net-e1000e_1.1.2-3vmw.500.0.0.469512, VMware_bootbank_net-enic_1.4.2.15a-1vmw.500.0.0.469512, VMware_bootbank_net-forcedeth_0.61-2vmw.500.0.0.469512, VMware_bootbank_net-igb_2.1.11.1-3vmw.500.0.0.469512, VMware_bootbank_net-ixgbe_2.0.84.8.2-10vmw.500.0.0.469512, VMware_bootbank_net-nx-nic_4.0.557-3vmw.500.0.0.469512, VMware_bootbank_net-r8168_8.013.00-3vmw.500.0.0.469512, VMware_bootbank_net-r8169_6.011.00-2vmw.500.0.0.469512, VMware_bootbank_net-s2io_2.1.4.13427-3vmw.500.0.0.469512, VMware_bootbank_net-sky2_1.20-2vmw.500.0.0.469512, VMware_bootbank_net-tg3_3.110h.v50.4-4vmw.500.0.0.469512, VMware_bootbank_ohci-usb-ohci_1.0-3vmw.500.0.0.469512, VMware_bootbank_sata-ahci_3.0-6vmw.500.0.0.469512, VMware_bootbank_sata-ata-piix_2.12-4vmw.500.0.0.469512, VMware_bootbank_sata-sata-nv_3.5-3vmw.500.0.0.469512, VMware_bootbank_sata-sata-promise_2.12-3vmw.500.0.0.469512, VMware_bootbank_sata-sata-sil_2.3-3vmw.500.0.0.469512, VMware_bootbank_sata-sata-svw_2.3-3vmw.500.0.0.469512, VMware_bootbank_scsi-aacraid_1.1.5.1-9vmw.500.0.0.469512, VMware_bootbank_scsi-adp94xx_1.0.8.12-6vmw.500.0.0.469512, VMware_bootbank_scsi-aic79xx_3.1-5vmw.500.0.0.469512, VMware_bootbank_scsi-bnx2i_1.9.1d.v50.1-3vmw.500.0.0.469512, VMware_bootbank_scsi-fnic_1.5.0.3-1vmw.500.0.0.469512, VMware_bootbank_scsi-hpsa_5.0.0-17vmw.500.0.0.469512, VMware_bootbank_scsi-ips_7.12.05-4vmw.500.0.0.469512, VMware_bootbank_scsi-lpfc820_8.2.2.1-18vmw.500.0.0.469512, VMware_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.500.0.0.469512, VMware_bootbank_scsi-megaraid-sas_4.32-1vmw.500.0.0.469512, VMware_bootbank_scsi-megaraid2_2.00.4-9vmw.500.0.0.469512, VMware_bootbank_scsi-mpt2sas_06.00.00.00-5vmw.500.0.0.469512, VMware_bootbank_scsi-mptsas_4.23.01.00-5vmw.500.0.0.469512, VMware_bootbank_scsi-mptspi_4.23.01.00-5vmw.500.0.0.469512, VMware_bootbank_scsi-qla2xxx_901.k1.1-14vmw.500.0.0.469512, VMware_bootbank_scsi-qla4xxx_5.01.03.2-3vmw.500.0.0.469512, VMware_bootbank_uhci-usb-uhci_1.0-3vmw.500.0.0.469512
Posted by at

10 comments:

  1. Carl van EijkMay 8, 2012 at 8:54 PM

    Thanks, very helpful post. I had a similar issue. Tried to apply critical updates/extensions via baseline.

    It seems to bomb after the vmware-fdm line every time.

    ~ # cat /var/log/esxupdate.log | grep -i "vmware-fdm" -A3

    2012-05-08T17:32:30Z esxupdate: HostImage: INFO: Attempting to download VIB vmware-fdm
    2012-05-08T17:32:30Z esxupdate: esxupdate: ERROR: An esxupdate error exception was caught:
    2012-05-08T17:32:30Z esxupdate: esxupdate: ERROR: Traceback (most recent call last):

    Manually removed it...
    ~ # esxcli software vib remove -n=vmware-fdm
    Removal Result
    Message: Operation finished successfully.
    Reboot Required: false
    VIBs Installed:
    VIBs Removed: VMware_bootbank_vmware-fdm_5.0.0-623373
    VIBs Skipped:

    Baseline policies applied from Update Manager without issue.
    Rebooted and the host rejoined the cluster with HA reconfigured.

    ReplyDelete
  2. Andy ShinnJuly 2, 2012 at 1:13 AM

    I also ran into this issue. Your fix worked for me. After removing the vmware-fdm VIB, I was able to Stage and then Remediate through the Update Manager tab.

    Thank you!

    ReplyDelete
  3. neverknowDecember 15, 2012 at 8:35 AM

    Thanks for the help! If you're looking for the command to execute the command on your server, it's:esxcli --server X.X.X.X software vib remove -n=vmware-fdm

    ReplyDelete
  4. RobertFebruary 14, 2013 at 3:51 PM

    You are the man Timo! Much respect

    ReplyDelete
  5. Amos LabsMarch 5, 2013 at 6:14 AM

    Awesome, this fixed my problem right away. Saved me many many possible headaches tonight.

    ReplyDelete
  6. gesshokuMarch 22, 2013 at 5:17 PM

    This also worked for me.
    Got the error when trying to install the CIM Provider for a qLogic HBA.

    After exiting maintenance mode the HA agent (including fdm) gets reinstalled, so you don't even lose any functionality.

    ReplyDelete
  7. UnknownJune 4, 2013 at 2:24 PM

    I also have the same issue. When I try to stage my ESXi 5.1 host I got error: The host returns esxupdate error code:7. Cannot download VIB.Check the Update Manager log files and esxupdate log files for more details.

    I just rebooted my host, and after that everything works perfectly.

    ReplyDelete
  8. UnknownJune 26, 2013 at 9:41 PM

    I was also getting "The host returns esxupdate error code:7. Cannot download VIB.Check the UpdateManager log files and esxupdate log files for more details."
    I ran:
    /tmp # cp /opt/vmware/uninstallers/VMware-fdm-uninstall.sh /tmp/VMware-fdm-unins
    tall.sh
    /tmp # ./VMware-fdm-uninstall.sh

    The host was then able to begin updating again from VUM

    ReplyDelete
  9. UnknownFebruary 20, 2014 at 10:35 PM

    This solution worked perfectly. Thanks for taking the time to post it.

    ReplyDelete
  10. UnknownMarch 3, 2014 at 12:38 PM

    thank you so much...

    ReplyDelete

Note: Only a member of this blog may post a comment.

Subscribe to: Post Comments (Atom)