IBM Quick Links

• BOMC tool download
• Firmware bladecenter
• Quickpath box bladecenter in the right corner
• AMM firmware
• Electronic service calls

ESX and HS22 : some pointers

I recently had some problems with ESX(i) and HS22 blades. Thought I just blog them for future reference.

First of all 2 driver problems. You'll need to install the vsphere cli to solve them (http://www.vmware.com/support/developer/vcli/)

• If the management network for your HS22 blade is acting funny and you can't seem to get digital video (KVM) on your HS22 blade to work, you'll need to update the broadcom driver. It can be downloaded from the vmware site http://downloads.vmware.com/d/details/esx_esxi_40_broadcom_bnx2_dt/ZHcqYmRqZHRidGR3
• If you are using the Emulex CFFH oneconnect or CNA or whatever you want to call the card, you'll also need to install the driver before any of the nics show up. Same site, other link http://downloads.vmware.com/d/details/esx_esxi4x_serverengines_dt/ZHcqYmRAZXdiZHdlZQ==

The procedure for both drivers is similar:

• Extract the ISO
• Find the offline bundle f.e BCM-bnx2-2.0.7c-1.0.4.00000-offline_bundle-229199.zip in the offline-bundle directory
• Install the driver by executing : vihostupdate.pl --server --password --username root --install --bundle c:\path\to\zip\BCM-bnx2-2.0.7c-1.0.4.00000-offline_bundle-229199.zip

Another problem. If you have an ESXi recovery cd for you HS22 with Part number (P/N) 60Y0073 you might find that the cd is not booting. If you put the cd in a laptop you will see why. The manufacturer has burned the iso literally on the cd. This means you will need to copy the iso file to you hard disk and then properly burn it to a cd with for example cdburnerxp , brasero or disk utility in Mac. Alternatively, you can open a call at IBM and wait for a replacement part

Lastly while updating the HS22 I have been encountering a lot of problems with doing firmware upgrades. One neat trick I learned is that the uEFI will load it defaults settings after 3 unsuccessful boots! Another thing I learned is that it is better to update the IMM first and then the rest of the components. What I do is now

• Update the IMM via the AMM Blade Taks > Firmware Update. This allows me to push the IMM firmware update to multiple blade at the same time
• Wait until the blades get rediscovered and are stable. This means waiting around 10 minutes after the update
• Update the rest of the components via the BOMC (Bootable Media Creator). If you don't know this tool, here is the link http://www-947.ibm.com/systems/support/supportsite.wss/selectproduct?taskind=2&brandind=5000020&familyind=5375180&typeind=0&modelind=0&osind=0&psid=sp&continue.x=1&matrix=Y#Bootable%20Media%20Creator
• Update some components manually like the Emulex CNA (the first firmware update is not supported via the BOMC)

Electronic calls @ IBM

Problems with IBM hardware?! Open a call directly without calling to first line :)

http://bit.ly/bjvWA8

Firefox direct link if you are living in EU or Belgium

Don't you hate it when you want to install firefox on a windows 2003 server just to download some binairies? Here is a fast link for you guys so you don't have to accept 30 mirrors before the auto-redirect page finally hits the same page
Belnet Firefox

Somethings to think about when upgrading to vSphere 4

The main things to think about:

Upgrading from vSphere 4 is supposed to be a walk in the park. I prefer to scratch all the ESX hosts instead of hoping that the update process works 100% as it should do. vCenter upgrading is a next next finnish process (hopefully). One thing to remember. If you are scratching your vCenter, leave your old vCenter running until all your old hosts are migrated to ESX 4.0. Because the licensing system has changed, you will need to keep the license server running on you old vCenter and configure the vCenter vSphere version to delegate the work for the old ESX hosts to the old license server (Administration > VirtualCenter Management Server Configuration > License Server. ). You could also install an old license server on your new vCenter and transfer the licenses. However this is not worth the effort because once all the hosts are "upgraded" you don't need it anymore. Also a small remark here, if you have tools that intergrate with vCenter, you will need an own game plan. For example, VMware view is tightly integrated with vCenter

If you scratched your vCenter, remember that you will have to move all the settings from vCenter old to vCenter new. I'm talking about

• VirtualCenter Management Server Configuration (Like smtp and snmp)


• Resource pool settings and cluster settings (HA - DRS - DPM). For example I have a rule that put the DB for vCenter and vCenter on the same hosts so they can communicate localy


• Virtual Machines and templates folders


• Alarms and automated tasks


• Permissions


• Advanced settings, although mostly you don't change these


• Copy sysprep from a to b. Just a reminder, it is under "[X]:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\sysprep\"
  

Don't kill if me if this list is incomplete

Next you will need to disconnect and remove the ESX from the original cluster and add the hosts to the new cluster. This will not stop the running VM's on your ESX hosts. However disabling HA might be a good thing while moving hosts around. If you don't disable DRS you will keep your resource pools but they will get ugly at the other site. So make sure you have overcapacity.


Some things you should know about your ESX hosts before you scratch them.

• FQDN / IP / Subnet / Gateway / DNS / NTP server. You will need these settings while installing ESX. You also will need to know the mac address of the physical NIC that is being used for your service console + the vlan your service console is in


• Partitioning. If you have custom agents installed this is especially true. I like to oversize my ESX console a lot if the local disk are not being used. Just because I can. Here is an example "oversized" esx console
  /dev/sdh9 4.9G 1.6G 3.1G 34% /
  /dev/sdg1 1.1G 75M 952M 8% /boot
  /dev/sdh6 2.0G 36M 1.8G 2% /home
  /dev/sdh8 2.0G 88M 1.8G 5% /opt
  /dev/sdh7 3.9G 72M 3.6G 2% /tmp
  /dev/sdh5 3.9G 211M 3.5G 6% /var
  /dev/sdh2 3.9G 74M 3.6G 2% /var/log
  The boot partition was configured automatically. I also create a higher swap size (1600mb). Ow and remember if you had agents installed, you will have to reinstall them :)


• Time zone


• Have a valid license key


• Make scripts or a clear layout how your virtual networking is done. Important settings to think about are how you NICs are bound to your vswitches. Write down the mac address so you can link mac address to vswitches. I have seen NIC numbering (vmnicx vs mac) being changed while scratching. Also check your port groups. They might overwrite you vswitch settings. For example, I like to put vMotion on a active/standby NIC configuration. I mostly overwrite this in the port group if i don't have a lot of physical NICs. Also make sure you have consistent port groups names!


• Know the firewall setup, it is there under advanced settings. For example you might need to open up the ssh client so that you can scp/ssh between hosts

Writing down all the settings/configuration before you begin helps. You can upgrade the hosts one by one following your scheme

and watch out for theses ..

• If you are able to disconnect the SAN, do so! If not check carefully that you don't overwrite a lun while installing


• On HS ibm blades, you will need to enable the vt instructions and the no execution disable bit in the bios. You'll find it under advanced settings -> cpu


• Check if the local datastore is empty!

Before you add your scratched hosts to the cluster MAKE SURE THE NETWORK IS CORRECTLY SETUP. You don't want VMs to end in vlans that are not connected to the right uplinks. I also love to ping and vmkping from the console. Also check that you can lookup your hostname, fqdn and your reverse ip.



Some fun extra's here :

Ofcourse our job wouldn't be any fun if you didn't have troubles. Before you start check that all the VMs are using hardware 4/vmware tools. You cannot vmotion between 3.5 and 4 if the hardware is version 3. So before you do anything, check all the vms version and tools. (I learned the hard way that upgrading from 3 to 7 when everything is transfered can be a bummer).

While moving a vm, I also encoutered this nice error. "Troubleshooting Migration compatibility error: Virtual machine has CPU and/or memory affinities configured, preventing VMotion". Disabling the cpu affinity in the advanced settings of the VM didn't fix the issue. The VM was still starting on the wrong host. Removing the following out the vmx manually fixed the problem.
sched.cpu.affinity = "0"
sched.cpu.htsharing = "any"
sched.mem.affinity = "all"
Ofcourse this will require downtime of the VM :(

Installing an rpm via yum is possible. I always used rpm -Ivh but you can also use yum localinstall –disablerepo=* . This is nice when you are installing vmware tools on a redhat family VM and are using the vmware tools ISO (when you choose install guest tools in the menu).

PCoIP can frack up you VM's Resolution. We had a user that started a PCoIP session with a VM. We are not sure if the session was not cleanly shut down but when we connected with the console in vCenter our screen size was fracked. Rebooting the vm, didn't help. The only thing that worked was logging on with PCoIP and resize the screen with the PCoIP client.

Vyatta Demo VMware Router

You know the drill. You need to set up a mobile datacenter that you need to move around from one place to another. When you get on location they give you a single ethernet cable with dhcp on it to connect to the internet. Your mission:
-Have multiple VLANs and being able to create new vlans
-Connect those multiple VLANs
-Have internet on all your VM's
-Being able to plugin the demo rack anywhere without any modification.

Previously we had a "hardware" router. An IBM blade with centos and tg3 broadcom drivers doing simple forwarding. It's not to hard to set up but you'll get a ship load of extra services you don't need + you loose a blade just for routing.

The solution : vyatta. It is an open source router wanting to compete with the cisco cli. It's easy and it's free ... or at least the software is, not the documentation.

So how did we set up my VM:
-eth0 : An ethernet adapter E1000. Pushed it in vlan 4095. In vsphere this will just pass all the incoming packets on the vSwitch without removing the vlan tags, essentially creating a trunk between your vswitch and your vm
-eth1 : An ethernet adapter E1000. Pushed it in vlan 12. This vlan is presented as 1 port on one of our physicall switches. This is were you plugin you uplink
-2gb hard disk

Installing vyatta is simple. Download the live CD. Start with the cd. Login with root : vyatta. Once you are logged in you can execute
$ install-system
Follow the wizard and reboot. Congrats, you have a brand new virtual router

Login with vyatta and password vyatta after the reboot. I executed the following
$ configure
This will put you into vyatta configure modus... like configure t on cisco
$ set interfaces ethernet eth0 address 192.168.114.254/24
$ set interfaces ethernet eth0 vif 15 address 192.168.115.254/24
$ set interfaces ethernet eth0 vif 16 address 192.168.116.254/24
$ set interfaces ethernet eth1 address dhcp
Untagged traffic is handled by the first line. The other vlans (15 and 16) are configured with the second and third command. They essentially create a virtual adaptor on eth0. The vif attribute defines the vlan. The last command sets up dhcp on eth1

$ set system host-name vRouter
$ set system domain-name mo.ilnk
$ set system name-server 192.168.115.1
$ set service ssh
Set up host name and dns domain. Third command set the dns server. The last command enables ssh.

$ set service nat rule 1 source address 192.168.114.0/24
$ set service nat rule 1 outbound-interface eth1
$ set service nat rule 1 type masquerade
$ set service nat rule 2 source address 192.168.115.0/24
$ set service nat rule 2 outbound-interface eth1
$ set service nat rule 2 type masquerade
$ set service nat rule 3 source address 192.168.116.0/24
$ set service nat rule 3 outbound-interface eth1
$ set service nat rule 3 type masquerade
Set up 3 nat rules. The first line of each rule defines which source addresses should be nated. The second line defines the uplink. The third line sets up masquerading. This will transform each tcp/ip packet so that it all looks like the packets are comming from the input. Essentially masking you demo environment and allowing you to have internet anywhere in your vlans. Remember you'll need a rule per VLAN

$ run renew dhcp interface eth1
Run a dhcp renew on eth1, like doing dhclient eth1 on many linux boxes. This will also set default router if your dhcp anounces one

$ set service dhcp-server shared-network-name ETH0_16_POOL subnet 192.168.116.0/24 start 192.168.116.10 stop 192.168.116.200
$ set service dhcp-server shared-network-name ETH0_16_POOL subnet 192.168.116.0/24 default-router 192.168.116.254
$ set service dhcp-server shared-network-name ETH0_16_POOL subnet 192.168.116.0/24 dns-server 192.168.115.1
Defines a dhcp rule in vlan 16 so that you can plug and play. You can make more then one pool per vlan if you like. You define the vlan by defining the subnet

$ commit
$ save
$ exit
Commit is in essence put your work to run. Save is like copy run start on cisco. Save the running config you commited to the boot config so you can reboot

You don't need to set up a default gateway because you'll get it from your dhcp lease. However if you need to assign a static ip you can do it the same ways as you did on eth0. However you'll need the following command to setup a the default gateway in your router
$ set system gateway-address 192.0.2.99
Remember to commit and save ;)


Just for the fun of it, we love to sing bom vyatta at work

Ping loss with ESX 4.0

So a very short post here but for people out there and for myself as reference it might be handy.

Friday I was at a customer site having the problem that he lost 5 pings to virtual machines every now and then. Unrelated to other events, debugging it was a hell. What also was bizarre was the fact that we didn't have any loss on the network ( we checked statics with ethtool and on the switches). The packets were just delivered to late to the VM, which then ofcourse replied to late and caused a ping timeout (we saw this behavior with a network sniffer).

We start to analyse the variables in the game. What changed since the network problems occured. Well the SAN changed. Look further in the logs (/var/log/messages) revealed a multipathing problem (SATP,PSP and NMP warnings). What happened was that the client was migrating to a newer version of SVC and in effect replacing the nodes. Multipathing worked fine but the old nodes were still showing up as dead paths. Probably this path were fixed paths (because SVC likes fixed) and the ESX hypervisor was trying to jump back to this old paths. Rescanning the hba's or rebooting the ESX server solved the problem (for now?)