Netextender corrupts mDNSResponder

A great way to start the day. My boss had some problems with his mac today. He wasn't able to surf at all. I checked all the layers and stranded at the highest level :).

• nslookup www.google.com resolved the domain (same with dig)
• ping 8.8.8.8 replied
• ping www.google.com did not reply (can not resolve hostname)
• internet in a VM worked correctly

I was quite sure that it had something to do with the way Mac OS X resolved the domain names. I also quickly learned that mDNSresponder is responsible for DNS since 10.6.4. Focusing on that I stumbled onto an article where NetExtender corrupts the xml that the services uses to boot. Since we were using NetExtender I thought I should give it a shot. First of all I copied my own xml from my working mac to a usb stick

sudo cp /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist /Volumes/stickname/

I then went in console at my boss computer (spotlight > terminal) and executed the sudo su command to enter root mode , the sh# prompt should appear. I entered the correct directory

• cd /System/Library/LaunchDaemons/

and checked the files

• ls -al com.apple.mDNSResponder*

This showed up 3 files, the regular plist file an nxbk file and the helper file which is not important.

I then compared the nxbk file and the regular file with the one back upped from my computer by using diff

• diff com.apple.mDNSResponder.plist /Volumes/stickname/com.apple.mDNSResponder.plist
• diff com.apple.mDNSResponder.plist.nxbk /Volumes/stickname/com.apple.mDNSResponder.plist

The nxbk file made by netextender before altering the plist was completely the same as the one I had on my USB stick. I decided that netExtender had corrupted the plist file so shut down the service, copying back the backup and relaunching the service could be a solution. So I executed the following (actually I backed up the files by cping them to /Users/someuser/)

• launchctl unload /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist
• rm com.apple.mDNSResponder.plist
• mv com.apple.mDNSResponder.plist.nxbk com.apple.mDNSResponder.plist
• launchctl load /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist

After that everything was working again.

BTW here is an output of my plist

ESX disconnects from vCenter and Corrupt IBM ESXi embedded sticks

Sometime ago I was installing a vCenter and connecting the the ESX hosts. The host kept disconnecting after 1 minute or something. Reconnecting helped for 1 minute. The problem? I thought I disabled the firewall but this wasn't the case. The firewall was blocking the incoming connections. Disabling the firewall completely solved the problem. This leaves me to believe that the heart-beating is done in all directions ESX to ESX but also ESX to vCenter and vice versa ... interesting.

Another great thing I encountered is a corrupted ESXi embedded IBM USB stick. The ESX hosts was asking for a Diagnostic dump partition. This was strange because I worked with the sticks before and they never needed one (dumping on a local partition / vicfg-dumpart -l to show). Also installing the VMware tools was giving errors about missing the cd's. Watching under the / root partition (unsupported mode/ls -al) was not showing the /store partition. After examining a "working" and a "non-working" ESXi and comparing the fdisk -l output, I noticed that a partition was missing. I reinstalled the ESXi with the recovery cd that you get when buying the embedded USB keys, fixed the problem.

I attached some screenshots so you can see the differences. The last partition (store partition was missing 615 to 900)

XMLHttpRequest not refreshing

I have been playing for some days now with writing some interacting javascript (dynamic forms and so on). Although I already did that in the past I had never played around with the XMLHttpRequest object. To overcome that I tried the XMLHttpRequest so that I could split up data from the javascript code.

One thing I noticed was that the xml request was cached by IE. I tried refreshing the page but this didn't help. The only thing I could do was surf manually to the xml and refresh it so that the cache was updated. This ofcourse was not acceptable. I found two solutions that worked for me.

The first solution was creating a hidden iframe that showed the xml. When you refresh the page, it also refreshes the iframe and thus the cache is updated. Not elegant but it works for not so dynamic pages. Here is an example

<div style="visibility:hidden;display:none"><iframe src="http://tendertechie.tries.technology.com/myxml.xml" id="resframe"></iframe></div>

*** btw if you want to post html code like this but don't want it to be interpreted on blogger.com, you can change the lesser then sign < by the html equivalent &lt; and ofcourse the > by &gt; . To show this code again I used &ampgt; :D

The second one I came up with was while I was driving my car. I was thinking about an old webbased game I used to play (http://cq2.speedxs.nl/indexnl.php). I remembered that while traveling, the webmaster added a ?move=c first in the url and then a ?move=q for the following request. The pattern was constantly alternating. This allowed the page to be refreshed without having caching problems. When I came home I tried to add ?upd=randomdata to my xml url and see if it would display the data correctly (this was the case). This led me to believe that I could refresh the data by adding random new data every time I requested an xml. And was is more "random" then time (read is always different). I came up with the following code

function getXMLDocViaMethod(url,method,reload)

{

if (window.XMLHttpRequest)

{

xmlhttp=new XMLHttpRequest();

}

else

{

xmlhttp=new ActiveXObject("Microsoft.XMLHTTP");

}

if(reload)

{

var d = new Date();

url = url+"?updch="+d.getTime();

}

xmlhttp.open(method,url,false);

xmlhttp.send();

xmlDoc=xmlhttp.responseXML;

return xmlDoc;

}

If you want to read more about xmlhttprequest, I suggest using the w3schools site. They have good examples that you can interactively try out. Here is a quick link!

MOB

I have always been a big scripting fan. Scripts make the IT-admin-life so much easier. That is why I always am shocked when IT guys say they hate scripting.

When ESX 4.1 was released I was very happy to hear that VMware is now supporting kickstart scripting in ESXi. Trying it out, I found liam his article about kickstart scripting ESXi (http://www.virtuallyghetto.com/2010/09/automating-esxi-41-kickstart-tips.html). If you don't know liam, you have been living under a rock. This guy is one of the best vmware scripters out there and is quite famous for his perl linkedclone script.

One of the cool things he mentions is tip 7 on how to execute vi api commands via the mob web interface. Playing around with it I made some -hopefully useful- scripts for everybody to play with.

• date.py : set the date and time
• dns.py : to set your hostname,dns servers, domain and searchdomains
• domain.py : join your ESXi hosts to a domain
• maintmode.py : to enter/exit maintenaince mode
• gateway.py : to alter your gateway
• service.py : to start/stop/auto on or off a service like for example ntpd
• ntp.py : configure ntp servers and on or off of the service

The scripts use the simplymob.py to do the webhandling (so that i didn't have the copy it all over again). The script has to be in the same location of the script you are using.

You can find all the files here

vWorkspace locks VM

Recently a colleague of mine was playing with the vWorkspace broker from Quest Software. It was all fun and games until vWorkspace decided to lock one of the VMs. Result:

-Unable to delete snapshots (creating them was no problem)

-Unable to migrate the virtual machine

-Unable to delete the virtual machine

-Unable to remove the virtual machine from the inventory.

Notice, they did not throw an error. The options were just grayed out.

Of course I like a challenge so I started my quest ;)

First of all I noticed that I was able to create snapshots, remove the virtual machine from the inventory and so on directly from the ESX host. This was my first clue. Removing the machine from the ESX locally created an orphaned machine. Luckily since the VM was down I could power it on and it was reregistered and no longer orphaned. This lead me to believe that the problem was in the vCenter database.

I started searching the KB and I found the following article

So I started by shutting down vCenter. I then opened up the database and executed

• select ID from VPX_ENTITY where name ='';

This returned an ID. I then executed

• delete from VPX_VM where ID=;
• delete from VPX_ENTITY where ID=;

In my case this deleted 2 rows (one row for each VM)

Lastly I restarted the virtual center. Because the virtual machine was still available on the ESX it was "rediscovered" by vCenter after booting. Result : VM unlocked.

This let me believes that the vWorkspace changes something in the database that locks the virtual machine. Probably the result of some linked-clone locking...

Notice : I have no idea if this broke vWorkspace things though ;)

IBM Quick Links

• BOMC tool download
• Firmware bladecenter
• Quickpath box bladecenter in the right corner
• AMM firmware
• Electronic service calls

ESX and HS22 : some pointers

I recently had some problems with ESX(i) and HS22 blades. Thought I just blog them for future reference.

First of all 2 driver problems. You'll need to install the vsphere cli to solve them (http://www.vmware.com/support/developer/vcli/)

• If the management network for your HS22 blade is acting funny and you can't seem to get digital video (KVM) on your HS22 blade to work, you'll need to update the broadcom driver. It can be downloaded from the vmware site http://downloads.vmware.com/d/details/esx_esxi_40_broadcom_bnx2_dt/ZHcqYmRqZHRidGR3
• If you are using the Emulex CFFH oneconnect or CNA or whatever you want to call the card, you'll also need to install the driver before any of the nics show up. Same site, other link http://downloads.vmware.com/d/details/esx_esxi4x_serverengines_dt/ZHcqYmRAZXdiZHdlZQ==

The procedure for both drivers is similar:

• Extract the ISO
• Find the offline bundle f.e BCM-bnx2-2.0.7c-1.0.4.00000-offline_bundle-229199.zip in the offline-bundle directory
• Install the driver by executing : vihostupdate.pl --server --password --username root --install --bundle c:\path\to\zip\BCM-bnx2-2.0.7c-1.0.4.00000-offline_bundle-229199.zip

Another problem. If you have an ESXi recovery cd for you HS22 with Part number (P/N) 60Y0073 you might find that the cd is not booting. If you put the cd in a laptop you will see why. The manufacturer has burned the iso literally on the cd. This means you will need to copy the iso file to you hard disk and then properly burn it to a cd with for example cdburnerxp , brasero or disk utility in Mac. Alternatively, you can open a call at IBM and wait for a replacement part

Lastly while updating the HS22 I have been encountering a lot of problems with doing firmware upgrades. One neat trick I learned is that the uEFI will load it defaults settings after 3 unsuccessful boots! Another thing I learned is that it is better to update the IMM first and then the rest of the components. What I do is now

• Update the IMM via the AMM Blade Taks > Firmware Update. This allows me to push the IMM firmware update to multiple blade at the same time
• Wait until the blades get rediscovered and are stable. This means waiting around 10 minutes after the update
• Update the rest of the components via the BOMC (Bootable Media Creator). If you don't know this tool, here is the link http://www-947.ibm.com/systems/support/supportsite.wss/selectproduct?taskind=2&brandind=5000020&familyind=5375180&typeind=0&modelind=0&osind=0&psid=sp&continue.x=1&matrix=Y#Bootable%20Media%20Creator
• Update some components manually like the Emulex CNA (the first firmware update is not supported via the BOMC)