QEMU调试Linux内核环境搭建

发表于 更新于 分类于 ，

摘要

本文主要通过 QEMU + Linux Kernel + Busybox 组合搭建 Linux Kernel 开发环境，

  • Qemu模拟ARM64运行环境
  • Busybox用于制作Linux运行的根文件系统

QEMU安装

1
2
3
4
5
6
7
8
9
wget https://download.qemu.org/qemu-9.0.1.tar.xz

tar Jxvf qemu-9.0.1.tar.xz

mkdir build
cd build
../configure --prefix=~/opt/qemu --target-list=aarch64-linux-user arm-linux-user i386-linux-user riscv32-linux-user riscv64-linux-user x86_64-linux-user aarch64-softmmu arm-softmmu i386-softmmu riscv32-softmmu riscv64-softmmu x86_64-softmmu --enable-debug
make

ARM64编译器安装

1
2
3
4
5
# download
wget https://developer.arm.com/-/media/Files/downloads/gnu/13.2.rel1/binrel/arm-gnu-toolchain-13.2.rel1-x86_64-aarch64-none-linux-gnu.tar.xz

# decompress
tar Jxvf arm-gnu-toolchain-13.2.rel1-x86_64-aarch64-none-linux-gnu.tar.xz

编译器路径加入环境变量

~/.bashrc中设置PATH 

1
export PATH=$PATH:~/arm-gnu-toolchain-13.2.rel1-x86_64-aarch64-none-linux-gnu/

Busybox编译

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# download
wget https://www.busybox.net/downloads/busybox-1.36.1.tar.bz2

# decompress
tar jxvf busybox-1.36.1.tar.bz2

# cd busybox
cd busybox-1.36.1

# make menuconfig
make ARCH=arm64 CROSS_COMPILE=aarch64-none-linux-gnu- menuconfig
# 选择如下选项
# Settings --->
#     [*] Build static binary (no shared libs)

# compiler
make ARCH=arm64 CROSS_COMPILE=aarch64-none-linux-gnu-

# make install
make ARCH=arm64 CROSS_COMPILE=aarch64-none-linux-gnu- install

# after make install generate _install dir

_install 目录下文件结构如下

1
2
3
4
5
6
7
8
.
├── bin
├── linuxrc -> bin/busybox
├── sbin
└── usr

4 directories, 1 file

基于 _install 制作 Linux kernel 启动根文件系统

1
2
3
mkdir initramfs
cd initramfs
cp -r ../busybox-1.36.1/_install/* .

然后执行如下脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
cd initramfs

mkdir dev etc lib sys proc tmp var home root mnt

cd etc

# create profile file and add following contents
echo "#!/bin/sh" > profile
echo "export HOSTNAME=qemu" >> profile
echo "export USER=root" >> profile
echo "export HOME=/home" >> profile
echo "export PS1=\"[$USER@$HOSTNAME \W]\# \"" >> profile
echo "PATH=/bin:/sbin:/usr/bin:/usr/sbin" >> profile
echo "LD_LIBRARY_PATH=/lib:/usr/lib:$LD_LIBRARY_PATH" >> profile
echo "export PATH LD_LIBRARY_PATH" >> profile

# create inittab file and add following contents
echo "::sysinit:/etc/init.d/rcS" > inittab
echo "::respawn:-/bin/sh" >> inittab
echo "::askfirst:-/bin/sh" >> inittab
echo "::ctrlaltdel:/bin/umount -a -r " >> inittab

# create fstab file and add following contents
echo "#device  mount-point    type     options   dump   fsck order" > fstab
echo "proc /proc proc defaults 0 0" >> fstab
echo "tmpfs /tmp tmpfs defaults 0 0" >> fstab
echo "sysfs /sys sysfs defaults 0 0" >> fstab
echo "tmpfs /dev tmpfs defaults 0 0" >> fstab
echo "debugfs /sys/kernel/debug debugfs defaults 0 0" >> fstab
echo "kmod_mount /mnt 9p trans=virtio 0 0" >> fstab

# create init.d directory and enter
mkdir init.d
cd init.d

## create rcS and add following contents
echo "mkdir -p /sys" > rcS
echo "mkdir -p /tmp" >> rcS
echo "mkdir -p /proc" >> rcS
echo "mkdir -p /mnt" >> rcS
echo "/bin/mount -a" >> rcS
echo "mkdir -p /dev/pts" >> rcS
echo "mount -t devpts devpts /dev/pts" >> rcS
echo "echo /sbin/mdev > /proc/sys/kernel/hotplug" >> rcS
echo "mdev -s" >> rcS
chmod 777 rcS

## cd to /etc directory
cd ..

# cd to root directory
cd ..

# cd to dev directory and mknod console and null node

cd dev
sudo mknod console c 5 1
sudo mknod null c 1 3

cd ..

# add .so file to lib

cp -a /home/w512/workarea/linux_kernel_debug/arm-gnu-toolchain-13.2.Rel1-x86_64-aarch64-none-linux-gnu/aarch64-none-linux-gnu/libc/lib64/lib*.so.* lib
cp -a /home/w512/workarea/linux_kernel_debug/arm-gnu-toolchain-13.2.Rel1-x86_64-aarch64-none-linux-gnu/aarch64-none-linux-gnu/libc/lib/*.so.* lib

# generate initramfs.cpio.gz
find . | cpio -o -H newc | gzip -c > ../initramfs.cpio.gz

cd ..

执行以上脚本后， 在脚本目录下生成对应的 initramfs.cpio.gz

Linux Kernel编译

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# download
wget https://mirrors.nju.edu.cn/kernel.org/linux/kernel/v6.x/linux-6.8.12.tar.xz

# decompress
tar jxvf linux-6.8.12.tar.xz

# configure
cd linux-6.8.12
cp arch/arm/configs/vexpress_defconfig  .config
make ARCH=arm64 CROSS_COMPILE=aarch64-none-linux-gnu- menuconfig
# No1. add hotplug support
# Device Drivers
#    -> Generic Driver
#        -> Support for uevent helper
#            (/sbin/hotplug) path to uevent helper
#
# No2. Modify Kernel Page Setting
# Kernel Features  --->
#    Page size(4KB)  --->
#    Virtual address space size(48-bit)--->
#
# No3. Add CONFIG_DEBUG_FS=y
# Kernel hacking  --->
#     Generic Kernel Debugging Instruments  --->
#         [*] Debug Filesystem
# compile
make ARCH=arm64 CROSS_COMPILE=aarch64-none-linux-gnu- all -j8

QEMU 运行 Linux Kernel

1
2
3
4
5
6
7
8
9
10
11
12
qemu-system-aarch64 \
	-machine virt \
	-cpu cortex-a57 \
	-machine type=virt \
	-m 1024 \
	-smp 4 \
	-kernel linux-6.8.12/arch/arm64/boot/Image \
	-initrd initramfs.cpio.gz \
	--append "rdinit=/linuxrc root=/dev/vda rw console=ttyAMA0 loglevel=8" \
	-nographic \
	--fsdev local,id=kmod_dev,path=kmodules,security_model=none \
	-device virtio-9p-device,fsdev=kmod_dev,mount_tag=kmod_mount

kmodules 目录用途

用于QEMU 和 Ubuntu Host之间共享的目录

QEMU + GDB 调试 Linux Kernel

qemu 端执行如下命令

1
2
3
4
5
6
7
8
9
10
11
12
13
qemu-system-aarch64 \
	-machine virt \
	-cpu cortex-a57 \
	-machine type=virt \
	-m 1024 \
	-smp 4 \
	-kernel linux-6.8.12/arch/arm64/boot/Image \
	-initrd initramfs.cpio.gz \
	--append "rdinit=/linuxrc root=/dev/vda rw console=ttyAMA0 loglevel=8" \
	-nographic \
	--fsdev local,id=kmod_dev,path=kmodules,security_model=none \
	-device virtio-9p-device,fsdev=kmod_dev,mount_tag=kmod_mount \
	-S -s

GDB 端执行如下命令

1
2
3
4
5
gdb-multiarch --tui vmlinux

(gdb)target remote localhost:1234  // 通过1234端口远程连接到qemu端
(gdb)b start_kernel                // start_kernel 处设置断点
(gdb)c

QEMU + GBD 调试 arm64 启动汇编代码

通过如下命令获取 vmlinux 中 section header 信息，如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
w512@w512-pc ~/w/linux_kernel_debug> aarch64-none-linux-gnu-readelf -S linux-6.8.12/vmlinux
There are 39 section headers, starting at offset 0xbfd5540:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .head.text        PROGBITS         ffff800080000000  00010000
       0000000000010000  0000000000000000  AX       0     0     65536
  [ 2] .text             PROGBITS         ffff800080010000  00020000
       00000000008ee000  0000000000000000  AX       0     0     65536
  [ 3] .rodata           PROGBITS         ffff800080900000  00910000
       000000000020f98e  0000000000000000  WA       0     0     4096
  ...

  [13] .rodata.text      PROGBITS         ffff800080b5d800  00b6d800
       0000000000005800  0000000000000000  AX       0     0     2048
  [14] .init.text        PROGBITS         ffff800080b70000  00b80000
       00000000000499f4  0000000000000000  AX       0     0     8
  [15] .exit.text        PROGBITS         ffff800080bb99f8  00bc99f8
       0000000000002324  0000000000000000  AX       0     0     8
  [16] .altinstructions  PROGBITS         ffff800080bbbd1c  00bcbd1c
       0000000000033378  0000000000000000   A       0     0     1
  [17] .init.data        PROGBITS         ffff800080bf5000  00c05000
       0000000000019888  0000000000000000  WA       0     0     8
    ...

  [25] .bss              NOBITS           ffff800080edb000  00eea200
       000000000004bed0  0000000000000000  WA       0     0     4096
    ...

  [38] .shstrtab         STRTAB           0000000000000000  0bfd538c
       00000000000001b2  0000000000000000           0     0     1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  D (mbind), p (processor specific)

其中关键的 section 对应的虚拟地址和物理地址如下：

section name virtual addr phy addr
.head.text ffff800080000000 0x40200000
.text ffff800080010000 0x40210000
.rodata ffff800080900000 0x40b00000
.rodata.text ffff800080b5d800 0x40d5d800
.init.text ffff800080b70000 0x40d70000
.init.data ffff800080bf5000 0x40df5000

GDB 启动时不要加载 vmlinux， 在 GDB 中通过 add-symbol-file 加载 vmlinux 中指定 section 要加载的物理地址

1
add-symbol-file linux-6.8.12/vmlinux -s .head.text 0x40200000 -s .text 0x40210000 -s .rodata 0x40b00000 -s .rodata.text 0x40d5d800 -s .init.text 0x40d70000 -s .init.data 0x40df5000

问题以及解决方案

问题1：mount: mounting debugfs on /sys/kernel/debug failed: No such file or directory

解决方案： 内核配置加上CONFIG_DEBUG_FS=y

参考