Timo's Techie Corner

2011/10/17

Unable to access unspecified file since it is locked

Today we had some problems with snapshots that were hanging on the vms that were being backuped. Strange errors occured where we were unable to delete the snapshot (title error) or the snapshot was hidden.

This guy provided the solution. For us it was fixed after rebooting TSM, creating a new snapshot and then choosing the delete all option

However to quickly check all the vms i wrote a small powercli script so that we could quickly identify the machines that were having the problem. It is quite easy but might help you in moments of stress ;)


[regex]$disksnap = "[-][0-9]{6,6}[.]vmdk"
foreach($vm in (get-vm))
{
 $first = $true
 foreach($disk in ($vm.harddisks | select filename))
 {
  if($disk.filename -match $disksnap)
  {
   if ($first)
   {
    $first = $false
    write-host "-----------------------------"
    write-host "VM Name  : "$vm.Name
    write-host "Snapshot : "
    $snaps = $vm | get-snapshot
    foreach($snap in $snaps)
    {
     write-host $snap.name
    }
   }
   write-host ">   "$disk.filename
   
  }
 }
}

I one case, rebooting TSM was not enough. The fix was to restart the mgmt agents. This seems to clear the TSM agent after which we could consolidate the snapshot again.

(In ESXi you can do this via ssh using "services.sh restart")

BTW; some time ago I had some vms having some disks referencing delta disks. The only solution was cloning the disks via vmkfstools


vmkfstools -i "/vmfs/volumes/Datastore/examplevm/examplevm-000001.vmdk" "/vmfs/volumes/Datastore 2/newexamplevm/newexamplevm.vmdk"

Before you delete the disks from the vm (to configure the clone disks)
-Make a note of the scsi controller type
-Make a note of the order on vSCSI controller (disk 1 -> 0:0, disk 2 -> 0:1,...)

2011/07/20

Physically ejecting a CDROM drive om esxi

eject /dev/cdrom/mpx.*

2011/07/05

ESXi modules

Some commando's I always need to lookup so lets gather them a bit

First of all, ESXi modules work a lot like linux. You can find them in a specific location

~ # ls /usr/lib/vmware/vmkmod/bnx*
/usr/lib/vmware/vmkmod/bnx2.o /usr/lib/vmware/vmkmod/bnx2i.o /usr/lib/vmware/vmkmod/bnx2x.o

You can also list the active modules
~ # esxcfg-module -l | grep bnx
bnx2 1 180
bnx2i 0 88

More info about a module can be obtained with the -i switch
~ # esxcfg-module -i bnx2
esxcfg-module module information
input file: /usr/lib/vmware/vmkmod/bnx2.o
License: GPL
Version:
Name-space:
Required name-spaces:
com.vmware.vmkapi@v1_1_0_0
Parameters:
disable_msi: int
Disable Message Signaled Interrupt (MSI)
heap_initial: int
Initial heap size allocated for the driver.
heap_max: int
Maximum attainable heap size for the driver.
skb_heap_initial: int
Initial private socket buffer heap size allocated for the driver.
skb_heap_max: int
Maximum attainable private socket buffer heap size for the driver.
stop_on_tx_timeout: int
For debugging purposes, prevent a chip reset when a tx timeout occurs

You can then get the current options
~ # esxcfg-module -g bnx2
bnx2 enabled = 1 options = ''

Setting an options works with the -s parameter
~ # esxcfg-module -s "oneRequest=false" nfsclient
~ # esxcfg-module -g nfsclient
nfsclient enabled = 1 options = 'oneRequest=false'
~ # esxcfg-module -s "" nfsclient
~ # esxcfg-module -g nfsclient
nfsclient enabled = 1 options = ''

Enabling modules is done by using the -e parameter. To disable a module, you will need the -d switch. Notice, you will need to reboot to get the module enabled this way
~ # ls /usr/lib/vmware/vmkmod/ | grep tg3
tg3.o
~ # esxcfg-module -e tg3
~ # esxcfg-module -d tg3

Esxcfg that a lot of things that the vmkload_mod command does. You can load modules live by using this commando. You will not have to reboot the server
~ # vmkload_mod tg3
Module tg3 loaded successfully
~ # vmkload_mod -u tg3
Module tg3 successfully unloaded

You can also see that the vmkload_mod gives similar output as the esxcfg-module
~ # vmkload_mod -l | grep tg3
tg3 0 120
~ # vmkload_mod -s tg3
vmkload_mod module information
input file: /usr/lib/vmware/vmkmod/tg3.o
Version: Version 3.86.0.1-1vmw, Build: 320137, Interface: ddi_9_1 Built on: Nov 2 2010
License: GPL
Required name-spaces:
com.vmware.vmkapi@v1_1_0_0
Parameters:
skb_heap_max: int
Maximum attainable private socket buffer heap size for the driver.
skb_heap_initial: int
Initial private socket buffer heap size allocated for the driver.
heap_max: int
Maximum attainable heap size for the driver.
heap_initial: int
Initial heap size allocated for the driver.
tg3_debug: int
Tigon3 bitmapped debugging message enable value
~ #

Another fun thing to do with the vmkload_mod is loading the cdrom driver
~ # vmkload_mod iso9660
Module iso9660 loaded successfully

Then you can mount the cdrom with
vsish -e set /vmkModules/iso9660/mount $(esxcfg-mpath -b | grep "CD-ROM" | awk '{print $1}')

You should the see the cdrom mounted in /vmfs/volumes

Unmount it later on
vsish -e set /vmkModules/iso9660/umount $(esxcfg-mpath -b | grep "CD-ROM" | awk '{print $1}')

Afterwards unload the module
~ # vmkload_mod -u iso9660
Module iso9660 successfully unloaded

(CDROM commands are from (http://www.virtuallyghetto.com/2011/04/how-to-mount-cdrom-using-vsish-on-esxi.html)

2011/05/20

ESXi 4.0 U1 and DS3500 (SAS), doesn't show multipathing

Had this at a customer site. Basicaly creating a claimrule for the ds3500. Basic summary:
esxcli nmp satp addrule -s VMW_SATP_LSI -V IBM -M "^1746*" -e "IBM DS3500"

You need to reboot to use the claim rules or just force it on a disk via
esxcli corestorage claiming reclaim -d

Full article at
http://www-01.ibm.com/support/docview.wss?uid=isg3T1012434

2011/05/18

vCenter DB on full and no backups being taken

Results in logs that are to big. Changed the recovery model to simple and executed the following to truncated the log

USE vcms_nieuw
GO
DBCC SHRINKFILE("logicalname_log", 1)
BACKUP LOG vcms_nieuw WITH TRUNCATE_ONLY
DBCC SHRINKFILE("logicalname_log", 1)
GO

This cleaned up the logs. Waiting for the backup guys to take it over from here :)

2011/04/20

vCloudPHP 1.0.1 : Howto get it working on Centos 5.5

Start by downloading the tar.gz or the zip. You can also scp it to centos but i just used elinks :)
install the redhat repo so that you can install php-pear
You'll need the php-pear-HTTP-Request2 . I found it on the "remi' repo, so I added the repository. You can also install the rpm directly without adding the repository.
install php
Install the php-xml package
install php-pear-HTTP-Request2
You can now run helloVCloud.php

I noticed that you might have to restart the httpd service to reload the new PHP module


[root@request ~]# yum install elinks
[root@request ~]# elinks 'http://www.vmware.com/downloads/downloadBinary.do?downloadGroup=VCDSDKPHP101&vmware=downloadBinary&file=vcloudPHP-1.0.1-362508.zip'
[root@request ~]# tar -xzvf vcloudPHP-1.0.1-362508.tar.gz 
[root@request ~]# cd vcloudPHP-1.0.1/samples

[root@request samples]# rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
[root@request samples]# rpm -Uvh http://rpms.famillecollet.com/enterprise/remi-release-5.rpm
[root@request samples]# echo "enabled the repo"
[root@request samples]# cat /etc/yum.repos.d/remi.repo | grep -e "name" -e "enabled" -e "\["
[remi]
name=Les RPM de remi pour Enterprise Linux $releasever - $basearch
enabled=1
[remi-test]
name=Les RPM de remi en test pour Enterprise Linux $releasever - $basearch
enabled=0

[root@request samples]# yum install php
[root@request samples]# yum install php-pear-HTTP-Request2
[root@request samples]# yum install php-xml

[root@request samples]# php helloVCloud.php  -s '172.27.2.30' -u 'cloudadministrator@MyCloudOrg' -p 'testpass' -o='MyCloudOrg'

2011/04/19

Injecting drivers with ESX 4.1 and creating an ISO

Actually based on vGhetto Blog

I'm just referencing what I did for myself . If you need more details, visit the link above


[root@vad iso]# mkdir /mnt/iso
[root@vad iso]# mkdir /mnt/edit
[root@vad iso]# mkdir /mnt/driver
[root@vad edit]# mkdir /mnt/newiso
[root@vad vi-admin]# sudo mount -o loop VMware-VMvisor-Installer-4.1.0.update1-348481.x86_64.iso /mnt/iso/
[root@vad iso]# cd /mnt/edit/
[root@vad edit]# bunzip2 -c ../iso/imagedd.bz2 > imagedd
[root@vad edit]# vibddi -i imagedd -q
< oem-vmware-esx-drivers-net-vxge (400.2.0.28.21239-1OEM) >
< oem-vmware-esx-drivers-scsi-3w-9xxx (400.2.26.08.036vm40-1OEM) >
< vmware-esx-firmware (4.1.0-1.4.348481) >
< CMPI-1.0 >
< VMW_CMPI_CPP-1.0.5 >
< vmkapi_1_1_0_0 >
< vmknexus1kvapi-1-4 >
< vmkepsecapi_1.0.0.0 >
< DriverAPI-9.0 >
< DriverAPI-9.1 >
< vmware-esx-tools-light (4.1.0-1.4.348481) >
[root@vad edit]# mount /home/vi-admin/vmware-esx-drivers-scsi-megaraid-sas_400.5.29-1vmw.2.17.00000.379626.iso /mnt/driver/ -o loop
[root@vad edit]# vibddi -i imagedd -o ../driver/offline-bundle/LSI_5.29-offline_bundle-379626.zip -n
[root@vad edit]# vibddi -i imagedd -q
< oem-vmware-esx-drivers-net-vxge (400.2.0.28.21239-1OEM) >
< oem-vmware-esx-drivers-scsi-3w-9xxx (400.2.26.08.036vm40-1OEM) >
< vmware-esx-drivers-scsi-megaraid-sas (400.5.29-1vmw.2.17.00000) >
< vmware-esx-firmware (4.1.0-1.4.348481) >
< CMPI-1.0 >
< VMW_CMPI_CPP-1.0.5 >
< vmkapi_1_1_0_0 >
< vmknexus1kvapi-1-4 >
< vmkepsecapi_1.0.0.0 >
< DriverAPI-9.0 >
< DriverAPI-9.1 >
< vmware-esx-tools-light (4.1.0-1.4.348481) >
[root@vad edit]# vibddi -i imagedd -c
Checking bootbank filesystems
Bootbank filesystems are valid
[root@vad edit]# bzip2 -cv imagedd > imagedd.bz2  imagedd: 
 3.052:1,  2.621 bits/byte, 67.24% saved, 943718400 in, 309208798 out.
[root@vad edit]# cp -R /mnt/iso/ /mnt/newiso/
[root@vad iso]# cd /mnt/newiso/iso/
[root@vad iso]# cp /mnt/edit/imagedd.bz2 /mnt/newiso/iso/
cp: overwrite `/mnt/newiso/iso/imagedd.bz2'? y
[root@vad iso]# echo "$(echo $(md5sum imagedd.bz2)| cut -f1 -d' ')  VMware-VMvisor-big-4.1.0-348481-x86_64.dd.bz2" > imagedd.md5
[root@vad iso]# cat imagedd.md5 
0a7814146df74441063a50c6609f6d35  VMware-VMvisor-big-4.1.0-348481-x86_64.dd.bz2
[root@vad iso]# cd /mnt/newiso/iso/
[root@vad iso]# mkisofs -o ../VMware-VMvisor-ICustom-4.1.0.update1-348481.x86_64.iso -b isolinux.bin -c boot.catalog -no-emul-boot -boot-load-size 4 -boot-info-table ./
[root@vad iso]# ls /mnt/newiso/*.iso -alh
-rw-r--r-- 1 root root 394M Apr 19 07:19 /mnt/newiso/VMware-VMvisor-ICustom-4.1.0.update1-348481.x86_64.iso
[root@vad newiso]# ls -alh /home/vi-admin/VMware-VMvisor-Installer-4.1.0.update1-348481.x86_64.iso 
-rw-r--r-- 1 vi-admin root 394M Apr 19 05:25 /home/vi-admin/VMware-VMvisor-Installer-4.1.0.update1-348481.x86_64.iso