Storwize and ESX: change default path selection to RR

Round robin is now an acceptable psp for storwize. However if you don't want to change the psp for every lun you create, it might be handy to adjust the default satp for svc so that it uses RR for every (new) lun it detects.

First of all, lets see all the possible path selection protocols we can use (PSP)

~ # esxcli storage nmp psp list

Name Description

------------- ---------------------------------

VMW_PSP_MRU Most Recently Used Path Selection

VMW_PSP_RR Round Robin Path Selection

VMW_PSP_FIXED Fixed Path Selection

We can see the default MRU for storwize/svc, which we are going to change to RR

First of all we can see that ESX will use a certain SATP if it detects an SVC / Storwize

~ # esxcli storage nmp satp rule list

Name Device Vendor Model Driver Transport Options Rule Group Claim Options Default PSP PSP Options Description

------------------- ------ ------- ---------------- ------ --------- -------------------------- ---------- ----------------------------------- ----------- ----------- --------------------------------------------------------------------------

VMW_SATP_SVC IBM 2145 system

Conveniently this is VMW_SATP_SVC . We can now check which default PSP is associated with this SATP by executing the list rule

~ # esxcli storage nmp satp list

Name Default PSP Description

------------------- ---------------- ------------------------------------------

VMW_SATP_SVC VMW_PSP_MRU Supports IBM SVC

You can see that by default MRU is chosen. We can change with the following command

esxcli storage nmp satp set --default-psp yourpsp --satp yoursatp

So execute

~ # esxcli storage nmp satp set --default-psp VMW_PSP_RR --satp VMW_SATP_SVC

Default PSP for VMW_SATP_SVC is now VMW_PSP_RR

Now reboot your server, the paths should be reclaimed with RR

Based on : http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1017760

Centos 6.2 and Deluge 1.3.4

The following instructions might not be fined grained but they get the job done :). I executed everything as root :)

First of all you will need to get the source from deluge
http://dev.deluge-torrent.org/wiki/Download#Source

I got the tar.gz version 1.3.4

Then you will need the following key components. These are libraries/tools you need for compiling libtorrent
yum groupinstall "Development tools"
yum install boost-* python-setuptools m2crypto pyOpenSSL openssl openssl-devel python-devel

Then untar the package you've download
tar -xzvf deluge-1.3.4.tar.gz
cd deluge-1.3.4

Build and install the package (time to get some coffee)
python setup.py build
python setup.py install

Lastly you will need to install some extra python packages (eggs). Otherwise you will see import errors
easy_install chardet
easy_install twisted

You can now start deluge by running
deluge

ESXTOP always run in batch mode

A colleague of mine pointed this one out. Seems something to do with the TERM parameter when you ssh to an ESX host. If you change it to plain xterm it works again in regular mode

/vmfs/volumes/4d2750de-8956fe05-a7a9-e61f133c816b # echo $TERM
xterm-color
/vmfs/volumes/4d2750de-8956fe05-a7a9-e61f133c816b # export TERM=xterm
/vmfs/volumes/4d2750de-8956fe05-a7a9-e61f133c816b # esxtop

Update ESX5: The host returns esxupdate error code:7.

Today I tried to upgrade some ESXi hosts via Update Manager. It's a nice tool since it can orchestrate your upgrade. However after the orchestration I was left with one host that didn't want to be update (Actually 2, one had a locked vm where I needed to do /etc/init.d/hostd restart so that the running tasks on ESX where cancelled).

The one that didn't want to upgrade gave me the following error in vcenter
Stage patches to entity
esx12
The host returns esxupdate error code:7.
Cannot download VIB. Check the Update
Manager log files and esxupdate log files
for more details.
....
14/02/2012 12:28:38
14/02/2012 12:28:38
14/02/2012 12:29:02

Strange, the errors seems to be a firewall issue but my other host did just fin. I went to look in the esxupdate.log

tail -n 100 -f /var/log/esxupdate.log
2012-02-14T11:28:55Z esxupdate: HostImage: INFO: Attempting to download VIB vmware-fdm
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: An esxupdate error exception was caught:
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: Traceback (most recent call last):
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/usr/sbin/esxupdate", line 216, in main
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: cmd.Run()
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esx5update/Cmdline.py", line 144, in Run
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/Transaction.py", line 243, in InstallVibsFromSources
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/Transaction.py", line 345, in _installVibs
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/Transaction.py", line 388, in _validateAndInstallProfile
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/HostImage.py", line 630, in Stage
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-469512/bora/build/esx/release/python-2.6-lib-zip-stage/469512/visor/pylib/python2.6/site-packages/vmware/esximage/HostImage.py", line 414, in _download_and_stage
2012-02-14T11:28:55Z esxupdate: esxupdate: ERROR: VibDownloadError: (, 'Unable to download VIB from any of the URLs ')
2012-02-14T11:28:55Z esxupdate: esxupdate: DEBUG: <<<
2012-02-14T11:28:59Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetTimeout']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:28:59Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetRetries']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:28:59Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetRateLimit']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:28:59Z esxupdate: esxupdate: INFO: ---
Command: scan
Args: ['scan']
Options: {'nosigcheck': None, 'retry': 5, 'loglevel': None, 'cleancache': None, 'viburls': None, 'meta': ['http://d2vce01.d2:9084/vum/repository/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip'], 'proxyurl': None, 'timeout': 30.0, 'cachesize': None, 'hamode': True, 'maintenancemode': None}
2012-02-14T11:28:59Z esxupdate: BootBankInstaller.pyc: INFO: Unrecognized value "title=Loading VMware ESXi" in boot.cfg
2012-02-14T11:28:59Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/bootOption', '-rp']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:29:00Z esxupdate: downloader: DEBUG: Downloading http://d2vce01.d2:9084/vum/repository/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip to /tmp/tmpk1SPPZ...
2012-02-14T11:29:00Z esxupdate: Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file
2012-02-14T11:29:00Z esxupdate: BootBankInstaller.pyc: INFO: Unrecognized value "title=Loading VMware ESXi" in boot.cfg
2012-02-14T11:29:00Z esxupdate: HostImage: DEBUG: Live image has been updated but /altbootbank image has not. This means a reboot is not safe.
2012-02-14T11:29:00Z esxupdate: HostImage: DEBUG: Live image has been updated but /altbootbank image has not. This means a reboot is not safe.
2012-02-14T11:29:00Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/usr/sbin/vsish', '-e', '-p', 'cat', '/hardware/bios/dmiInfo']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2012-02-14T11:29:00Z esxupdate: esxupdate: INFO: All done!
2012-02-14T11:29:00Z esxupdate: esxupdate: DEBUG: <<<

And saw that there was a problem with the fdm (HA) agent package. I tried to uninstall via
/tmp # /opt/vmware/uninstallers/VMware-fdm-uninstall.sh
but this didn't really help


Finally I decided to do manual update so that repo/downloading would not be the issue. Again I was faced with a similar error
/vmfs/volumes/4d0f342d-b384b59f-b015-e61f133c792b # esxcli software vib install -d=/scratch/ESXi500-201109001.zip
[VibDownloadError]
Failed to download VIB.
url = vmware-fdm-5.0.0-455964
localfile = Unable to download VIB from any of the URLs
Please refer to the log file for more details.

No good so I decided to try to uninstall the fdm package, maybe that would help. After all if the host would exist the package would be pushed anyway
/vmfs/volumes/4d0f342d-b384b59f-b015-e61f133c792b # esxcli software vib remove -n=vmware-fdm
Removal Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed:
VIBs Removed: VMware_bootbank_vmware-fdm_5.0.0-455964
VIBs Skipped:

After that I could manually install the packages so once again I was a happy camper/vmfs/volumes/4d0f342d-b384b59f-b015-e61f133c792b # esxcli software vib install -d=/scratch/ESXi500-201109001.zip
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: VMware_bootbank_esx-base_5.0.0-0.3.474610, VMware_locker_tools-light_5.0.0-0.3.474610
VIBs Removed: VMware_bootbank_esx-base_5.0.0-0.0.469512, VMware_locker_tools-light_5.0.0-0.0.469512
VIBs Skipped: VMware_bootbank_ata-pata-amd_0.3.10-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-atiixp_0.4.6-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-cmd64x_0.2.5-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-pdc2027x_1.0-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-serverworks_0.4.3-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-sil680_0.4.8-3vmw.500.0.0.469512, VMware_bootbank_ata-pata-via_0.3.3-2vmw.500.0.0.469512, VMware_bootbank_block-cciss_3.6.14-10vmw.500.0.0.469512, VMware_bootbank_ehci-ehci-hcd_1.0-3vmw.500.0.0.469512, VMware_bootbank_esx-tboot_5.0.0-0.0.469512, VMware_bootbank_ima-qla4xxx_2.01.07-1vmw.500.0.0.469512, VMware_bootbank_ipmi-ipmi-devintf_39.1-4vmw.500.0.0.469512, VMware_bootbank_ipmi-ipmi-msghandler_39.1-4vmw.500.0.0.469512, VMware_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.500.0.0.469512, VMware_bootbank_misc-cnic-register_1.1-1vmw.500.0.0.469512, VMware_bootbank_misc-drivers_5.0.0-0.0.469512, VMware_bootbank_net-be2net_4.0.88.0-1vmw.500.0.0.469512, VMware_bootbank_net-bnx2_2.0.15g.v50.11-5vmw.500.0.0.469512, VMware_bootbank_net-bnx2x_1.61.15.v50.1-1vmw.500.0.0.469512, VMware_bootbank_net-cnic_1.10.2j.v50.7-2vmw.500.0.0.469512, VMware_bootbank_net-e1000_8.0.3.1-2vmw.500.0.0.469512, VMware_bootbank_net-e1000e_1.1.2-3vmw.500.0.0.469512, VMware_bootbank_net-enic_1.4.2.15a-1vmw.500.0.0.469512, VMware_bootbank_net-forcedeth_0.61-2vmw.500.0.0.469512, VMware_bootbank_net-igb_2.1.11.1-3vmw.500.0.0.469512, VMware_bootbank_net-ixgbe_2.0.84.8.2-10vmw.500.0.0.469512, VMware_bootbank_net-nx-nic_4.0.557-3vmw.500.0.0.469512, VMware_bootbank_net-r8168_8.013.00-3vmw.500.0.0.469512, VMware_bootbank_net-r8169_6.011.00-2vmw.500.0.0.469512, VMware_bootbank_net-s2io_2.1.4.13427-3vmw.500.0.0.469512, VMware_bootbank_net-sky2_1.20-2vmw.500.0.0.469512, VMware_bootbank_net-tg3_3.110h.v50.4-4vmw.500.0.0.469512, VMware_bootbank_ohci-usb-ohci_1.0-3vmw.500.0.0.469512, VMware_bootbank_sata-ahci_3.0-6vmw.500.0.0.469512, VMware_bootbank_sata-ata-piix_2.12-4vmw.500.0.0.469512, VMware_bootbank_sata-sata-nv_3.5-3vmw.500.0.0.469512, VMware_bootbank_sata-sata-promise_2.12-3vmw.500.0.0.469512, VMware_bootbank_sata-sata-sil_2.3-3vmw.500.0.0.469512, VMware_bootbank_sata-sata-svw_2.3-3vmw.500.0.0.469512, VMware_bootbank_scsi-aacraid_1.1.5.1-9vmw.500.0.0.469512, VMware_bootbank_scsi-adp94xx_1.0.8.12-6vmw.500.0.0.469512, VMware_bootbank_scsi-aic79xx_3.1-5vmw.500.0.0.469512, VMware_bootbank_scsi-bnx2i_1.9.1d.v50.1-3vmw.500.0.0.469512, VMware_bootbank_scsi-fnic_1.5.0.3-1vmw.500.0.0.469512, VMware_bootbank_scsi-hpsa_5.0.0-17vmw.500.0.0.469512, VMware_bootbank_scsi-ips_7.12.05-4vmw.500.0.0.469512, VMware_bootbank_scsi-lpfc820_8.2.2.1-18vmw.500.0.0.469512, VMware_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.500.0.0.469512, VMware_bootbank_scsi-megaraid-sas_4.32-1vmw.500.0.0.469512, VMware_bootbank_scsi-megaraid2_2.00.4-9vmw.500.0.0.469512, VMware_bootbank_scsi-mpt2sas_06.00.00.00-5vmw.500.0.0.469512, VMware_bootbank_scsi-mptsas_4.23.01.00-5vmw.500.0.0.469512, VMware_bootbank_scsi-mptspi_4.23.01.00-5vmw.500.0.0.469512, VMware_bootbank_scsi-qla2xxx_901.k1.1-14vmw.500.0.0.469512, VMware_bootbank_scsi-qla4xxx_5.01.03.2-3vmw.500.0.0.469512, VMware_bootbank_uhci-usb-uhci_1.0-3vmw.500.0.0.469512

Unable to access unspecified file since it is locked

Today we had some problems with snapshots that were hanging on the vms that were being backuped. Strange errors occured where we were unable to delete the snapshot (title error) or the snapshot was hidden.

This guy provided the solution. For us it was fixed after rebooting TSM, creating a new snapshot and then choosing the delete all option

However to quickly check all the vms i wrote a small powercli script so that we could quickly identify the machines that were having the problem. It is quite easy but might help you in moments of stress ;)

[regex]$disksnap = "[-][0-9]{6,6}[.]vmdk"
foreach($vm in (get-vm))
{
 $first = $true
 foreach($disk in ($vm.harddisks | select filename))
 {
  if($disk.filename -match $disksnap)
  {
   if ($first)
   {
    $first = $false
    write-host "-----------------------------"
    write-host "VM Name  : "$vm.Name
    write-host "Snapshot : "
    $snaps = $vm | get-snapshot
    foreach($snap in $snaps)
    {
     write-host $snap.name
    }
   }
   write-host ">   "$disk.filename
   
  }
 }
}

I one case, rebooting TSM was not enough. The fix was to restart the mgmt agents. This seems to clear the TSM agent after which we could consolidate the snapshot again.

(In ESXi you can do this via ssh using "services.sh restart")

BTW; some time ago I had some vms having some disks referencing delta disks. The only solution was cloning the disks via vmkfstools

vmkfstools -i "/vmfs/volumes/Datastore/examplevm/examplevm-000001.vmdk" "/vmfs/volumes/Datastore 2/newexamplevm/newexamplevm.vmdk"

Before you delete the disks from the vm (to configure the clone disks)
-Make a note of the scsi controller type
-Make a note of the order on vSCSI controller (disk 1 -> 0:0, disk 2 -> 0:1,...)

Physically ejecting a CDROM drive om esxi

eject /dev/cdrom/mpx.*

ESXi modules

Some commando's I always need to lookup so lets gather them a bit

First of all, ESXi modules work a lot like linux. You can find them in a specific location

~ # ls /usr/lib/vmware/vmkmod/bnx*
/usr/lib/vmware/vmkmod/bnx2.o /usr/lib/vmware/vmkmod/bnx2i.o /usr/lib/vmware/vmkmod/bnx2x.o

You can also list the active modules
~ # esxcfg-module -l | grep bnx
bnx2 1 180
bnx2i 0 88

More info about a module can be obtained with the -i switch
~ # esxcfg-module -i bnx2
esxcfg-module module information
input file: /usr/lib/vmware/vmkmod/bnx2.o
License: GPL
Version:
Name-space:
Required name-spaces:
com.vmware.vmkapi@v1_1_0_0
Parameters:
disable_msi: int
Disable Message Signaled Interrupt (MSI)
heap_initial: int
Initial heap size allocated for the driver.
heap_max: int
Maximum attainable heap size for the driver.
skb_heap_initial: int
Initial private socket buffer heap size allocated for the driver.
skb_heap_max: int
Maximum attainable private socket buffer heap size for the driver.
stop_on_tx_timeout: int
For debugging purposes, prevent a chip reset when a tx timeout occurs

You can then get the current options
~ # esxcfg-module -g bnx2
bnx2 enabled = 1 options = ''

Setting an options works with the -s parameter
~ # esxcfg-module -s "oneRequest=false" nfsclient
~ # esxcfg-module -g nfsclient
nfsclient enabled = 1 options = 'oneRequest=false'
~ # esxcfg-module -s "" nfsclient
~ # esxcfg-module -g nfsclient
nfsclient enabled = 1 options = ''

Enabling modules is done by using the -e parameter. To disable a module, you will need the -d switch. Notice, you will need to reboot to get the module enabled this way
~ # ls /usr/lib/vmware/vmkmod/ | grep tg3
tg3.o
~ # esxcfg-module -e tg3
~ # esxcfg-module -d tg3

Esxcfg that a lot of things that the vmkload_mod command does. You can load modules live by using this commando. You will not have to reboot the server
~ # vmkload_mod tg3
Module tg3 loaded successfully
~ # vmkload_mod -u tg3
Module tg3 successfully unloaded

You can also see that the vmkload_mod gives similar output as the esxcfg-module
~ # vmkload_mod -l | grep tg3
tg3 0 120
~ # vmkload_mod -s tg3
vmkload_mod module information
input file: /usr/lib/vmware/vmkmod/tg3.o
Version: Version 3.86.0.1-1vmw, Build: 320137, Interface: ddi_9_1 Built on: Nov 2 2010
License: GPL
Required name-spaces:
com.vmware.vmkapi@v1_1_0_0
Parameters:
skb_heap_max: int
Maximum attainable private socket buffer heap size for the driver.
skb_heap_initial: int
Initial private socket buffer heap size allocated for the driver.
heap_max: int
Maximum attainable heap size for the driver.
heap_initial: int
Initial heap size allocated for the driver.
tg3_debug: int
Tigon3 bitmapped debugging message enable value
~ #

Another fun thing to do with the vmkload_mod is loading the cdrom driver
~ # vmkload_mod iso9660
Module iso9660 loaded successfully

Then you can mount the cdrom with
vsish -e set /vmkModules/iso9660/mount $(esxcfg-mpath -b | grep "CD-ROM" | awk '{print $1}')

You should the see the cdrom mounted in /vmfs/volumes

Unmount it later on
vsish -e set /vmkModules/iso9660/umount $(esxcfg-mpath -b | grep "CD-ROM" | awk '{print $1}')

Afterwards unload the module
~ # vmkload_mod -u iso9660
Module iso9660 successfully unloaded

(CDROM commands are from (http://www.virtuallyghetto.com/2011/04/how-to-mount-cdrom-using-vsish-on-esxi.html)