Debugging with Qemu

This question was a challenge because I have never done kernel debugging, or used QEMU before. But a few doses of Google later, and using simple deduction based on other experiences, I'm getting through.

> Get this KVM qcow2 image

KVM = Kernel Virtual Machine (hypervisor) compatible with wp:QEMU (https://QEMU.org) -- a machine emulator and virtualizer. File is an wp:Xz-compressed (like 7-zip; only GPL) "debug-crash.qcow"

So, I installed QEMU + dependencies on my Bionic workstation.

> Please show the commands you use to start via libvirt with this XML.

The xml had /usr/bin/kvm-spice in it as the emulator; so I searched dpkg -S /usr/bin/kvm-spice which turns out to be qemu-kvm (that I already have installed). So I assume that I'll need to run qemu-kvm. So then I researched how to start a qemu-kvm session/machine, using libvirt, and given an XML manifest / configuration file. I didn't find anything like that, so I went with basic commands for starting QEMU. (Failing fast and early reveals the path to perfection.) Listing the user and system emulators in /usr/bin/qemu-*, there is qemu-system-x86_64-spice. I started the system with:

/usr/bin/qemu-system-x86_64-spice -hda /home/greg/Downloads/debug-crash.qcow

But, from the Arch wiki I found that

To start QEMU in KVMmode, append -enable-kvm to the additional start options. To check if KVM is enabled for a running VM, enter the QEMU Monitor using Ctrl+Alt+Shift+2 , and type info kvm.

That informed me that I was not running in KVM mode (disabled). So I halted the machine. I then launched the GUI virt-manager sudo virt-manager, created a new machine using the qcow image, and started it. Looking at the ps axjfww output on the host, I could see that the command "I" used to start the VM was

qemu-system-x86_64 -enable-kvm -name guest=generic,debug-threads=o n -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-generic/master-key.aes -machine pc-i440fx- bionic,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu Broadwell-noTSX-IBRS -m 1024 -realtime mlock=off -smp 1,socket s=1,cores=1,threads=1 -uuid 4cbd5a21-bff7-4e66-8632-2e936507c658 -no-user-config -nodefaults -chardev socket,id=charmonitor ,path=/var/lib/libvirt/qemu/domain-1-generic/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -r tc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -globa l PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterb us=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr= 0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-seria l0,bus=pci.0,addr=0x6 -drive file=/home/greg/Downloads/debug-crash.qcow,format=qcow2,if=none,id=drive-ide0-0-0 -device ide- hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=26,id=hostnet0 -device rtl8139,netdev=hostn et0,id=net0,mac=52:54:00:4e:3e:15,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id= serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel 0,id=channel0,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migr ation=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci .0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 -chardev spicevmc,id=c harredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 -device virtio-balloon-pci,id=ballo on0,bus=pci.0,addr=0x7 -msg timestamp=on

Apache

> Apache is not starting, why? There's a syntax error on line 153 of /etc/apache2/apache2.conf. More accurately, it can not open the file that it's trying to open at /etc/apache2/extra.conf. And actually, looking at that file, someone with a sense of humor symlinked it to /dev/urandom -- which is why Apache has a problem with the syntax :-D

LDFLAGS

> It turns out this change actually made LDFLAGS empty. Why? I thought this was a trick question, because it looked like the syntax for $CFLAGS and $CPPFLAGS are incorrect. But, that syntax actually does look correct. Since the sed expression also looks correct, I have to surmise that somehow CFLAGS and/or LDFLAGS weren't configured correctly. But that's only a guess. Reading The Fine Manual on dpkg-buildflags, it tells me that LDFLAGS defaults to empty. So sed is a no-op, and LDFLAGS would still be empty.

Instructions

To login use
- User: ubuntu
- Password: ubuntu
- SSH server runs by default and guest does dhcp (or use the serial console)
In that image clarify the following things:
- Apache is not starting, why?
- What is going on with /var/crash/_usr_sbin_postconf.1000.crash (use debug
symbols and apport-retrace / apport-unpack to get started)
Makefile Weirdness
It was determined that a certain linker flag was causing problems in the build of a package, so
a change was added to debian/rules to remove that flag:
override_dh_auto_build:
CFLAGS="$(CFLAGS) $(CPPFLAGS)" \
LDFLAGS="$(dpkg-buildflags --get LDFLAGS \
| sed -e 's/-Wl,-Bsymbolic-functions//')" \
dh_auto_build
The flag was removed from LDFLAGS via a simple sed expression. It turns out this change
actually made LDFLAGS empty. Why?