Debugging with Qemu

Analyze Failures

Background

This question was a challenge because I have never done kernel debugging, or used QEMU before. But a few doses of Google later, and using simple deduction based on other experiences, I'm getting through.

> Get this KVM qcow2 image

KVM = Kernel Virtual Machine (hypervisor) compatible with wp:QEMU (https://QEMU.org) -- a machine emulator and virtualizer. File is an wp:Xz-compressed (like 7-zip; only GPL) "debug-crash.qcow"

So, I installed QEMU + dependencies on my Bionic workstation.

Command to Start

> Please show the commands you use to start via libvirt with this XML.

The xml had /usr/bin/kvm-spice in it as the emulator; so I searched dpkg -S /usr/bin/kvm-spice which turns out to be qemu-kvm (that I already have installed). So I assume that I'll need to run qemu-kvm. So then I researched how to start a qemu-kvm session/machine, using libvirt, and given an XML manifest / configuration file. I didn't find anything like that, so I went with basic commands for starting QEMU. (Failing fast and early reveals the path to perfection.) Listing the user and system emulators in /usr/bin/qemu-*, there is qemu-system-x86_64-spice. I started the system with:

/usr/bin/qemu-system-x86_64-spice -hda /home/greg/Downloads/debug-crash.qcow

But, from the Arch wiki I found that

To start QEMU in KVMmode, append -enable-kvm to the additional start options. To check if KVM is enabled for a running VM, enter the QEMU Monitor using Ctrl+Alt+Shift+2 , and type info kvm.

That informed me that I was not running in KVM mode (disabled). So I halted the machine. I then launched the GUI virt-manager sudo virt-manager, created a new machine using the qcow image, and started it. Looking at the ps axjfww output on the host, I could see that the command virt-manager used to start the VM was

qemu-system-x86_64 -enable-kvm -name guest=generic,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-generic/master-key.aes -machine pc-i440fx-bionic,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu Broadwell-noTSX-IBRS -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 4cbd5a21-bff7-4e66-8632-2e936507c658 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-generic/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/home/greg/Downloads/debug-crash.qcow,format=qcow2,if=none,id=drive-ide0-0-0 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=26,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:4e:3e:15,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardevspicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on

I also installed the qemu-virgil snap and was able to start the VM with KVM using

sudo qemu-virgil -enable-kvm -m 512 -device virtio-vga,virgl=on -display sdl,gl=on -redir tcp:10022::2 2 -soundhw ac97 /home/greg/Downloads/debug-crash.qcow

Turning to the libvirt project, I discovered that you can convert the XML to command-line args. So, after a quick sudo mkdir -p /var/lib/uvtool/libvirt/images/

, and placing the image there, I tried to run the VM with

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name guest=debug-crash,debug-threads=on -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain--1-debug-crash/master-key.aes -machine pc-i440fx-bionic,accel=kvm,usb=off,dump-guest-core=off -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 49cdd138-2fae-4f57-8fdc-d9bafabb92ac -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain--1-debug-crash/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/uvtool/libvirt/images/debug-crash.qcow,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:c9:bd:ff,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on

but that failed because the specified AES key -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain--1-debug-crash/master-key.aes file does not exist. Removing that option, it still fails on a permission problem. The socket file doesn't exist, but it also doesn't get created... even using sudo to execute it as libvirt-qemu (that's the owner of the socket directory). Noticing that the final directory is missing, I ran it again after creating the domain--1-debug-crash directory.

And looking again at my host system process list ps axjfww I could see that the command to start the VM is

qemu-system-x86_64 -enable-kvm -name guest=debug-crash,debug-threads=on -machine pc-i440fx-bionic,accel=kvm,usb=off,dump-guest-core=off -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 49cdd138-2fae-4f57-8fdc-d9bafabb92ac -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain--1-debug-crash/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/uvtool/libvirt/images/debug-crash.qcow,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:c9:bd:ff,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on

That still doesn't quite work, because I can't connect to the machine. It appears that I am supposed to create and sign a certificate for the machine?

Apache

> Apache is not starting, why?

There's a syntax error on line 153 of /etc/apache2/apache2.conf. More accurately, it can not open the file that it's trying to open at /etc/apache2/extra.conf. And actually, looking at that file, someone with a sense of humor symlinked it to /dev/urandom -- which is why Apache has a problem with the syntax :-D

Crash File

> What is going on with /var/crash/_usr_sbin_postconf.1000.crash?

TBD

Makefile Weirdness (LDFLAGS)

> It turns out this change actually made LDFLAGS empty. Why?

I thought this was a trick question, because it looked like the syntax for $CFLAGS and $CPPFLAGS are incorrect. But, that syntax actually does look correct. Since the sed expression also looks correct, I thought that somehow CFLAGS and/or LDFLAGS weren't configured correctly. But that was only a guess. Reading The Fine Manual on dpkg-buildflags, it tells me that LDFLAGS defaults to empty. So sed is a no-op, and LDFLAGS would still be empty.

Instructions

To login use
- User: ubuntu
- Password: ubuntu
- SSH server runs by default and guest does dhcp (or use the serial console)
In that image clarify the following things:
- Apache is not starting, why?
- What is going on with /var/crash/_usr_sbin_postconf.1000.crash (use debug
symbols and apport-retrace / apport-unpack to get started)
Makefile Weirdness
It was determined that a certain linker flag was causing problems in the build of a package, so
a change was added to debian/rules to remove that flag:
override_dh_auto_build:
CFLAGS="$(CFLAGS) $(CPPFLAGS)" \
LDFLAGS="$(dpkg-buildflags --get LDFLAGS \
| sed -e 's/-Wl,-Bsymbolic-functions//')" \
dh_auto_build
The flag was removed from LDFLAGS via a simple sed expression. It turns out this change
actually made LDFLAGS empty. Why?

General Public License = A sane copyright license

Extensible Markup Language = a system for markup such that you can create your own 'schema' of terms rather than a specific markup language.