GDB
GDB has become the standard debugger on Linux and other free Unices. It is a source-level debugger rather than a machine-level one such as the Bochs debugger; this could be good or bad, depending on your viewpoint.
While debugging system code is not its intended use, it does a fine job of it; it can be used directly with some emulators, without modifying your code at all, or it can be used over a serial line. The latter option involves implementing a Remote Serial Protocol stub in your operating system.
Remote Debugging
We are interested in the remote debugging facility of GDB. From the GDB manual, “If you are trying to debug a program running on a machine that cannot run GDB in the usual way, it is often useful to use remote debugging. For example, you might use remote debugging on an operating system kernel, or on a small system which does not have a general purpose operating system powerful enough to run a full-featured debugger.”
For remote debugging, a remote stub should be implemented in the remote program that needs to be debugged. That means the kernel should contain the remote stub to talk to the host GDB during the debug session. This requires a kernel source change and it is a must if you are running your kernel on a test(real) machine and debugging from another machine. If you are using an emulator (Bochs or QEMU), you can use the GDB stub compiled into the emulator.
Implementing GDB Stub
To debug (using GDB) a kernel running on a real machine, the kernel needs to contain a GDB stub. For the i386 platform, GDB source includes a reference implementation of gdb-stub.c. It requires the following three functions from the kernel to read/write from a serial port and to set up the exception handlers.
getDebugChar(); // to read a character from serial port
putDebugChar(int); // to write a character to serial port
exceptionHandler(int exception_number, void *exception_address); // to set exception handler
Implement the above functions in your kernel, include the gdb-stub.c in the compilation, and also call the following functions during kernel start to connect to the gdb host.
InitSerialPort(sys_gdb_port, 9600, UART_DATA_BIT_8, UART_PARITY_NONE, UART_STOP_BIT_1); /*set up the serial port*/
set_debug_traps(); /*setup exception handlers*/
kprintf("Waiting for GDB(0x%X) : ", sys_gdb_port );
__asm__("int3"); /*break point exception to sync with GDB host*/
Now start your test machine and your kernel will wait for a GDB host connection.
Using Emulator Stubs
QEMU - GDB can be connected to QEMU by simply starting qemu with the -s -S command line switches.
Bochs - For GDB to be able to interface with Bochs, Bochs will need to be configured with the --enable-gdb-stub argument. The bochssrc (or whatever configuration script is being used) will need to have the gdbstub line set to something like gdbstub: enabled=1, port=1234, text_base=0, data_base=0, bss_base=0.
Starting GDB
After an emulator/kernel is configured it will then wait for a connection from GDB. GDB on the host machine can be started like this:
$ gdb YOUR-KERNEL . . . (gdb) target remote :1234 Remote debugging using :1234 0x0000fff0 in ?? () (gdb)
If you are debugging a kernel running on a real machine then use target remote /dev/tty2 instead of the network port.
If it prompts you after the "target remote" line asking whether to kill a program that is already being debugged answer "n".
If you are using a 64-bit kernel, you may need to set up the address size using set command.
gdbinit files
Put commonly-used commands in a .gdbinit file in your working directory. For example:
file kernel.bin target remote localhost:1234
This will connect to GDB-stub whenever you start GDB and will load symbols from kernel.bin.
GDB in Emacs
Emacs is well integrated with GDB. Try using M-x gdb and when it asks you for command-line arguments to GDB, use --annotate=3. This will allow Emacs to follow along as you place breakpoints and single-step through your kernel.
Virtual Serial Ports
If you have gdbstub inside your kernel and run your kernel inside an emulator, you can use a COM Port redirector to create a virtual serial port. After creating a virtual serial port, one port in the pair should be used for the emulator; the other one should specified in gdb for remote debugging.
com0com can be used in Windows and a pseudo terminal might be used in Linux to create virtual serial port pairs.