The Last Octet

Here are the symptoms and what we found resolved the matter.

We'd recently discovered that some our VM's were not successfully completing their VMware Update Manager baseline scans. Each of the problematic VM's had similar symptoms, we'd initiate the scan and then at about 55% of the way through the scan would stall for about ten minutes and then timeout.

After ensuring each of the troublesome VM's had a CD-ROM drive connected (required for VUM) we still continued to run into the stalling problem. Digging further into the issue found that the pattern was isolated to VM's that had been P2V'ed. One particular VM led us down the path to resolving the issue for the rest of the VM's. The VM in question had an exclamation mark over one of its IDE controllers in Windows device manager. Removing the conflicting device allowed VUM to properly mount the virtual CD it needed, and allowed the scan to complete properly on that machine.

The solution was similar on the remaining machines, but not as obvious. When we dug into the device manager of the other problematic machines they did not have the same Windows hardware conflicts as the first machine. Even after configuring device manager to show hidden devices, we found no offending hardware. Only after following the instructions in Microsoft KB article 315539 were we able to remove the offending legacy hardware. Those steps are basically:

1. Open the command prompt.
2. At the prompt enter: set devmgr_show_nonpresent_devices=1
3. Open device manager with: start devmgmt.msc
4. Finally, click "Show hidden devices" on the View menu

As it turns out all of the P2V'ed machines were never completely flushed of their previous Windows hardware profiles (although we'd done a regular clean up after they were migrated to remove legacy devices and OEM drivers and utilities). We also thought that we'd be able to obtain some of the problem indicators from the VUM logs, but the most we got out of those was that the scans were unable to mount the CD-ROM.

Another lesson learned.