Debugging with Qemu
Analyze Failures[edit]
Background[edit]
This question was a challenge because I have never done kernel debugging, or used QEMU before. But a few doses of Google later, and using simple deduction based on other experiences, I'm getting through.
> Get this KVM qcow2 image
KVM = Kernel Virtual Machine (hypervisor) compatible with wp:QEMU (https://QEMU.org) -- a machine emulator and virtualizer. File is an wp:Xz-compressed (like 7-zip; only GPL) "debug-crash.qcow"
So, I installed QEMU + dependencies on my Bionic workstation.
Command to Start[edit]
> Please show the commands you use to start via libvirt with this XML.
The xml had /usr/bin/kvm-spice in it as the emulator; so I searched dpkg -S /usr/bin/kvm-spice which turns out to be qemu-kvm (that I already have installed). So I assume that I'll need to run qemu-kvm. So then I researched how to start a qemu-kvm session/machine, using libvirt, and given an XML manifest / configuration file. I didn't find anything like that, so I went with basic commands for starting QEMU. (Failing fast and early reveals the path to perfection.) Listing the user and system emulators in /usr/bin/qemu-*, there is qemu-system-x86_64-spice. I started the system with:
/usr/bin/qemu-system-x86_64-spice -hda /home/greg/Downloads/debug-crash.qcow
But, from the Arch wiki[1] I found that
To start QEMU in KVMmode, append -enable-kvm to the additional start options. To check if KVM is enabled for a running VM, enter the QEMU Monitor using Ctrl+Alt+Shift+2 , and type
info kvm.
That informed me that I was not running in KVM mode (disabled). So I halted the machine. I then launched the GUI virt-manager sudo virt-manager, created a new machine using the qcow image, and started it. Looking at the ps axjfww output on the host, I could see that the command virt-manager used to start the VM was
qemu-system-x86_64 -enable-kvm -name guest=generic,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-generic/master-key.aes -machine pc-i440fx-bionic,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu Broadwell-noTSX-IBRS -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 4cbd5a21-bff7-4e66-8632-2e936507c658 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-generic/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/home/greg/Downloads/debug-crash.qcow,format=qcow2,if=none,id=drive-ide0-0-0 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=26,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:4e:3e:15,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardevspicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
I also installed the qemu-virgil snap and was able to start the VM with KVM using
sudo qemu-virgil -enable-kvm -m 512 -device virtio-vga,virgl=on -display sdl,gl=on -redir tcp:10022::2 2 -soundhw ac97 /home/greg/Downloads/debug-crash.qcow
Turning to the libvirt project, I discovered that you can convert the XML to command-line args. So, after a quick sudo mkdir -p /var/lib/uvtool/libvirt/images/
, and placing the image there, I tried to run the VM with
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name guest=debug-crash,debug-threads=on -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain--1-debug-crash/master-key.aes -machine pc-i440fx-bionic,accel=kvm,usb=off,dump-guest-core=off -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 49cdd138-2fae-4f57-8fdc-d9bafabb92ac -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain--1-debug-crash/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/uvtool/libvirt/images/debug-crash.qcow,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:c9:bd:ff,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on
but that failed because the specified AES key -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain--1-debug-crash/master-key.aes file does not exist. Removing that option, it still fails on a permission problem. The socket file doesn't exist, but it also doesn't get created... even using sudo to execute it as libvirt-qemu (that's the owner of the socket directory). Noticing that the final directory is missing, I ran it again after creating the domain--1-debug-crash directory.
And looking again at my host system process list ps axjfww I could see that the command to start the VM is
qemu-system-x86_64 -enable-kvm -name guest=debug-crash,debug-threads=on -machine pc-i440fx-bionic,accel=kvm,usb=off,dump-guest-core=off -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 49cdd138-2fae-4f57-8fdc-d9bafabb92ac -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain--1-debug-crash/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/uvtool/libvirt/images/debug-crash.qcow,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:c9:bd:ff,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on
That still doesn't quite work, because I can't connect to the machine. It appears that I am supposed to create and sign a certificate for the machine?
Apache[edit]
> Apache is not starting, why?
There's a syntax error on line 153 of /etc/apache2/apache2.conf. More accurately, it can not open the file that it's trying to open at /etc/apache2/extra.conf. And actually, looking at that file, someone with a sense of humor symlinked it to /dev/urandom -- which is why Apache has a problem with the syntax :-D
Crash File[edit]
> What is going on with /var/crash/_usr_sbin_postconf.1000.crash?
TBD
Makefile Weirdness (LDFLAGS)[edit]
> It turns out this change actually made LDFLAGS empty. Why?
I thought this was a trick question, because it looked like the syntax for $CFLAGS and $CPPFLAGS are incorrect. But, that syntax actually does look correct. Since the sed expression also looks correct, I thought that somehow CFLAGS and/or LDFLAGS weren't configured correctly. But that was only a guess. Reading The Fine Manual on dpkg-buildflags, it tells me that LDFLAGS defaults to empty. So sed is a no-op, and LDFLAGS would still be empty.
Instructions[edit]
To login use - User: ubuntu - Password: ubuntu - SSH server runs by default and guest does dhcp (or use the serial console) In that image clarify the following things: - Apache is not starting, why? - What is going on with /var/crash/_usr_sbin_postconf.1000.crash (use debug symbols and apport-retrace / apport-unpack to get started) Makefile Weirdness It was determined that a certain linker flag was causing problems in the build of a package, so a change was added to debian/rules to remove that flag: override_dh_auto_build: CFLAGS="$(CFLAGS) $(CPPFLAGS)" \ LDFLAGS="$(dpkg-buildflags --get LDFLAGS \ | sed -e 's/-Wl,-Bsymbolic-functions//')" \ dh_auto_build The flag was removed from LDFLAGS via a simple sed expression. It turns out this change actually made LDFLAGS empty. Why?
