|PC Virtual Machine Monitors|
VMware is basically the rich man's virtualizer. It can run any number of "Guest Operating Systems" and can boot from real media or image files. It comes with a GUI that makes it easier to configure than Bochs or QEMU.
Please note: VMware is not an emulator. There is lot of difference between an emulator and a virtual machine monitor. Emulators, like Bochs, emulate each instruction, whereas VMware (and Virtual PC) try to run most of the code on the host PC directly, and only emulate instructions that create invalid instruction faults or access system-level memory.
The VMware BIOS supports booting from a CD (use mkisofs to make a bootable CD, attach cdrom device to the .iso file) or a floppy drive (attach the floppy disk to a file), plus hard drives. These are the easiest options for loading your own kernel.
VMware's usefulness for hobbyists depends on the code's generation. Broadly:
- Workstation 5.0 (and previous). Not at all helpful for hobbyists. Error codes are designed for reporting to VMware, and there are no developer-centric features.
- Workstation 5.5 (Player 1.0, Server 1.0). Marginally more useful: a buggy guest operating system traces useful error messages. But in reality, go for something more modern. Supports SMP, if you are writing a multiprocessor kernel. Server and Player are free, which is a perk; go for Server.
- Workstation 6.0 (Fusion 1.0/1.1). Useful feature: gdb-based guest debug stub support. Useful feature: record-replay (though difficult to use), which might be helpful for reproducing race conditions.
- ??? (Server 2.0 beta in progress). Current generation; details unavailable.
- NOTE: VMware Player is totally free for use, and has smaller size than VMware Workstation. The new versions of VMware Player (from 3.0 up) are able to create new virtual machines. Thus this VMM is *perfect* for testing.
These options are valid in Workstation 6.0+, and should be set in the virtual machine's .vmx file.
- debugStub.listen.guest32 = "TRUE"
- debugStub.listen.guest64 = "TRUE"
If using these options, Workstation prints a message "VMware Workstation is listening for debug connection on port 8832." into the vmware.log file. Start a GDB session (using a copy of your kernel that includes debug information), then:
- target remote localhost:8832
Note that some versions of VMWare contain a bug: after you disconnect GDB from port 8832, VMWare will actually continue listening on port 8833 not 8832. The actual port can always be located by looking at the latest "Debug stub" entry in the vmware.log file:
vmx| W110: Debug stub: VMware Workstation is listening for debug connection on port 8833.
Standard gdb commands work in this mode (e.g. printing memory, backtrace). Note that this is a debug stub attached to the KERNEL, not a userspace program.
Other useful options:
- debugStub.listen.guest32.remote = "TRUE" # Allows debugging from a different computer / VM instead of localhost. # The IP for remote debugging will be that of the host.
- debugStub.listen.guest64.remote = "TRUE" # Same, but for 64-bit code
- monitor.debugOnStartGuest32 = "TRUE" # Breaks into debug stub on first instruction (warning: in BIOS!) # This will halt the VM at the very first instruction at 0xFFFF0, you could set the next breakpoint to break *0x7c00 to break when the bootloader is loaded by the BIOS
- debugStub.hideBreakpoints = "TRUE" # Enables the use of hardware breakpoints instead of software (INT3) breakpoints
- bios.bootDelay = "3000" # Delay booting the BIOS code.
VMware products emulate a triple fault by resetting the machine. On beta products, however, a guest triple fault results in Bug 19580. Sometimes this represents a bug in VMware's emulation; usually, this represents a bug in your kernel.
Compatibility with IDEs
- IDA Pro supports debugging Windows kernels via the VMWare stub 
- VisualKernel supports debugging Linux kernels from Visual Studio with VMWare gdb stub 
- http://www.vmware.com/ - VMware homepage.
- http://communities.vmware.com/index.jspa - Community forums. Engineers are present; if asked politely, some will provide very exact technical details
- http://www.pagetable.com/?p=25 - A description of a problem encountered using VMware on AMD Opteron RevC processors.
- http://stackframe.blogspot.com/2007/04/debugging-linux-kernels-with.html - VMware engineer describes how to debug Linux kernel with debug stub.
- http://sourceforge.net/projects/open-vm-tools/ - Open source client tools. Good source for VMware proprietary hardware and host/client hooks