(gdb) break *0x972

In this post series, I'll explain how to setup and play with system debugging of Linux kernel. We'll have to build the environment: Qemu virtual machine, Linux kernel and a Busybox filesystem. I'll show how to play with the debugger in the next post.

All the explanation here are for building a x86/x86_64 kernel. Everything will work the same with another instruction-set, but you'll have to setup yourself the cross-compiling environment!

Compile Linux Kernel in Debugging Mode

# Download and extract the sources of the kernel
KVERSION=3.17.4
wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-$KVERSION.tar.xz;
tar xvf linux-$KVERSION.tar.xz
cd linux-$KVERSION

# configure the kernel
cp /boot/config-$(uname -r) .config # copy Fedora kernel configuration
make oldconfig                      # set new options
make menuconfig

Make sure that these options are checked/unchecked:

Kernel hacking
--> Compile-time checks and compiler options
--> Compile-time checks and compiler options
--> [X] Compile the kernel with debug info
--> [ ] Strip assembler-generated symbols during link
[X] Kernel debugging

make run 

... and go grab a coffee, it will take a while!

Compile Qemu in Debugging Mode

If you want to study how Qemu communicates with the debugger, build it now with the debugging information (enabled by default), otherwise install it from your distribution packages.

QVERSION=2.1.2
wget http://wiki.qemu-project.org/download/qemu-$QVERSION.tar.bz2
tar xvf qemu-$QVERSION.tar.bz2
mkdir qemu-{build,install}
cd qemu-build
../qemu-$QVERSION/configure --prefix=$(readlink -f ../qemu-install) --disable-kvm --target-list="i386-softmmu x86_64-softmmu"
make && make install

Compile Busybox Filesystem

Busybox and initrd preparation come almost directly from mgalgs, thanks!

If you want to have a chance to study the link between user-level applications and the kernel, build Busybox with debugging information. Otherwise, just grab the precompiled binaries

BVERSION=1.19.4
wget http://busybox.net/downloads/busybox-$BVERSION.tar.bz2
tar xf busybox-$BVERSION.tar.bz2
cd busybox-$BVERSION/
make menuconfig

and make sure that the debugging options are checked:

Busybox Settings
--> Debugging Options

then compile and install (by default in _install sub-directory) make make install

Build Initrd Filesystem

Initrd provides an early filesystem, I assume it's preloaded in the shared memory by the BIOS (ie, Qemu):

mkdir initramfs
cd initramfs
# create standard filesystem directories
mkdir -pv bin lib dev etc mnt/root proc root sbin sys
# create standard file devices
sudo cp -va /dev/{null,console,tty} dev/
sudo mknod dev/sda b 8 0
# import busybox filesystem
cp ../busybox-$BVERSION/_install/* . -rv

We didn't recompile the glibc, so we need to import it from our local system (adapt it to what you see in ldd output)

# copy relevant shared libraries
ldd bin/busybox
# linux-vdso.so.1 =>  (0x00007fff9fdfe000) (virtual)
# libm.so.6 => /lib64/libm.so.6 (0x0000003d49e00000)
# libc.so.6 => /lib64/libc.so.6 (0x0000003d49200000)
# /lib64/ld-linux-x86-64.so.2 (0x0000003d48e00000)

mkdir lib
ln -s lib lib64 # make lib and lib64 identical
cp /lib64/libm.so.6 lib  # symlink to libm-2.18.so
cp /lib64/libm-2.18.so lib
cp /lib64/libc.so.6 lib # symlink to libc-2.18.so
cp /lib64/libc-2.18.so lib
cp /lib64/ld-linux-x86-64.so.2 lib

You can ensure that your shared library are correctly imported by running sudo chroot . /bin/sh in your initramfs directory.

Finally, prepare an init script that will setup the user-space environment:

cat > init << EOF
#!/bin/sh

/bin/mount -t proc none /proc
/bin/mount -t sysfs sysfs /sys
/bin/mount -t ext2 /dev/sda /mnt/root

exec /bin/sh
EOF
chmod 755 init

Prepare and Run the Virtual Machine

The initrd filesystem has to be packaged in a cpio archive. Each time you modify a file in initramfs you'll have to rebuild the archive:

cd initramfs && \
find . -print0 | cpio --null -ov --format=newc > ../my-initramfs.cpio \    
&& cd ..

Last step consists in creating a hard-disk for the system and format it in ext2:

SIZE=512M
qemu-img create disk.img $SIZE
mkfs.ext2 -F disk.img

Now you're ready to boot the virtual machine!

qemu-install/bin/qemu-system-x86_64 -nographic -hda disk.img -kernel linux-$KVERSION/arch/x86_64/boot/bzImage -initrd my-initramfs.cpio -append "console=ttyS0"

Options -nographic and -append "console=ttyS0" redirect the virtual machine output to the console, instead of creating a dedicated window. The others are straightforward, they pass the hard disk file, kernel file (compressed in bz format) and initrd filesystem.

Hit Ctrl-Alt-C to enter Qemu console, and quit to exit.

Debug Linux Kernel

Run Qemu with gdbserver listener (notice the -s, equivalent to -gdb tcp::1234):

qemu-install/bin/qemu-system-x86_64 -nographic -hda disk.img -kernel linux-$KVERSION/arch/x86_64/boot/bzImage -initrd my-initramfs.cpio -append "console=ttyS0" -s

and connect GDB to that port, with the uncompressed kernel as symbol file:

gdb linux-$KVERSION/vmlinux -ex "target remote localhost:1234"
...
Reading symbols from linux-3.17.4/vmlinux...done.
Remote debugging using localhost:1234
(gdb) where
#0  native_safe_halt () at .../irqflags.h:50
#1  arch_safe_halt () at .../paravirt.h:111
#2  default_idle () at .../process.c:311
#3  arch_cpu_idle () at .../process.c:302
#4  cpuidle_idle_call () at .../idle.c:120
#5  cpu_idle_loop () at .../idle.c:220
#6  cpu_startup_entry (state=<optimized out>) at .../idle.c:268
#7  rest_init () at init/main.c:418
#8  start_kernel () at init/main.c:680
#9  x86_64_start_reservations (real_mode_data=<optimized out>) at .../head64.c:193
#10 x86_64_start_kernel (real_mode_data=<optimized out>) at .../head64.c:182