[pve-devel] How to compile pve-kernel with debug info?

[Nathan L]

Following up to answer my own questions and to ask a follow up question.

On Fri, Jun 4, 2021 at 8:18 PM Nathan L <nathanlangford.4 at gmail.com> wrote:

> What am I missing here?
1. The correct git commit to use if you want to match the published
package is specified in /usr/share/doc/insert-package-name/SOURCE. For
example:
# cat /usr/share/doc/pve-kernel-5.4.114-1-pve/SOURCE
git clone git://git.proxmox.com/git/pve-kernel.git
git checkout cd81f880a481ec8e3366cd493bb9d42821ebf967

2. As best I can tell, by using the git clone and git checkout
commands included in the SOURCE file with the published package, and
then running the make command, in theory you should get the same
output as that used for the published package. I say "in theory"
because there is some info like the hostname embedded in the vmlinuz
file that will cause checksums not to match between the published
version and your compiled version. I think there also may be some more
substantial differences that are causing the crash tool errors
"WARNING: kernel version inconsistency between vmlinux and dumpfile".
This leads to my follow up question at the end.

3. The above means that, in theory, all you need to do to get a
vmlinux file to debug your current kernel version is git clone ...;
git checkout ...; make; and then you should be able to use the vmlinux
file located at build/ubuntu-focal/vmlinux. This doesn't work in
practice though.

4. Point 3 seems to be true because if you compile the kernel as
mentioned previously, actually install your compiled
pve-kernel-5.4.114-1-pve_5.4.114-1_amd64.deb file, boot into that
kernel, and run 'echo c > /proc/sysrq-trigger' to create a crash dump,
that crash dump can be successfully opened in 'crash' using the
vmlinux file compiled and located at build/ubuntu-focal/vmlinux.
However, if you try to use the published pve-kernel .deb package and
your compiled vmlinux file, it won't work.

> Should I not use the compiled vmlinux file at ./build/ubuntu-focal/vmlinux?
No. That is the correct vmlinux but for some reason it doesn't work
with the published kernel package.

> Is there a flag I need to change or add somewhere to properly compile a version of the kernel I can use with crash?
No. The kernel is compiled into a vmlinux file with all debug info
included by default.

> Is there a difference between the vmlinux file in the build/ubuntu-focal directory vs what I could extract from the vmlinuz file in the build/debian directory? The latter had the debug info stripped when I extracted it.
Yes. A big difference. The kernel is compiled into a vmlinux file with
all debug info included by default. It's also 743MB. Which is why it's
further processed, added to, stripped of lots of data, etc to become
the final vmlinuz file that gets packaged. That vmlinuz file is 12MB.
There is no way to extract the full vmlinux from vmlinuz because so
much data, like the debug info, was stripped along the way.

Okay, follow up question:
Why doesn't my compiled vmlinux for pve-kernel-5.4.114-1-pve work for
debugging crash dumps from a machine running the same kernel version
but from the published package? (example below)

I should also make another thread or file a bug report on this but the
included makedumpfile version (1.6.5-1 I think) with Proxmox didn't
work to capture crash dumps at all. I had to upgrade it to the 1.6.8-4
version from the debian testing repo.



# cat /var/crash/202106081333/dmesg.202106081333 | grep "Linux version"
[    0.000000] Linux version 5.4.114-1-pve (build at proxmox) (gcc
version 8.3.0 (Debian 8.3.0-6)) #1 SMP PVE 5.4.114-1 (Sun, 09 May 2021
17:13:05 +0200) ()
###The published version of the kernel was installed and loaded when
this crash dump occurred.

# file /root/pve-kernel/build/ubuntu-focal/arch/x86/boot/bzImage
/root/pve-kernel/build/ubuntu-focal/arch/x86/boot/bzImage: Linux
kernel x86 boot executable bzImage, version 5.4.114-1-pve
(root at hostname) #1 SMP PVE 5.4.114-1 (Sun, 09 May 2021 17:13:05
+0200), RO-rootFS, swap_dev 0xB, Normal VGA
###The version I compiled is exactly the same. This bzImage is derived
from vmlinux used in the below crash command.

# crash ~/pve-kernel/build/ubuntu-focal/vmlinux
/var/crash/202106081333/dump.202106081333

crash 7.2.5
Copyright (C) 2002-2019  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

WARNING: kernel relocated [710MB]: patching 114251 gdb minimal_symbol values

WARNING: kernel version inconsistency between vmlinux and dumpfile

crash: seek error: kernel virtual address: ffff99711f606024  type:
"tss_struct ist array"

##Crash doesn't like this.

Best regards,

Nathan