Gentoo on Odroid-M1 (aarch64) with upstream 6.1.x kernel, LVM and LUKS
Followup on earlier experience with Odroids M1 board here is guide to get Gentoo with upstream 6.1.x kernel running on Odroids M1 board while running on top of LUKS encrypted disk with LVM. (examples for both eMMC and NVMe installation are included)
Previous guide - Gentoo on Odroid-M1 (aarch64).
Requirements
- Odroid-M1 board - I have used 8GB RAM version, but I believe that 4GB should work too
- bootable Linux system that has some basic utilities on it (this manual assumes presence of
parted
,cryptsetup
andmkfs.xfs
which are fairly common on distributions these days) - Internet connection - for getting files to install Gentoo and packages
- approximately 4 hours of time :) - there are 2 steps where you have to wait for compilation to finish (first ~39 minutes for packages and later ~168 minutes for kernel compilation)
- 16GB (or bigger) eMMC or NVMe
Installation
- boot up some OS from SD card
- partition the eMMC card or NVME (this manual will assume same layout for eMMC and NVMe) - GPT is used here
- p1 - reservation for future use (custom u-boot)- not used in this manual
- p2 -
/boot
to hold kernel, initramfs, DTB and boot.scr files -
p3 - LUKS encrypted partition with LVM on it containing
/
LV for Gentoo - eMMC card partitioning:
# parted /dev/mmcblk0 mklabel gpt # parted /dev/mmcblk0 mkpart primary 2MiB 6MiB # parted /dev/mmcblk0 mkpart primary 6MiB 262MiB # parted /dev/mmcblk0 mkpart primary 262MiB 80%
# lsblk /dev/mmcblk0 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS mmcblk0 179:0 0 116.6G 0 disk ├─mmcblk0p1 179:1 0 4M 0 part ├─mmcblk0p2 179:2 0 256M 0 part └─mmcblk0p3 179:3 0 93.1G 0 part
- NVMe card partitioning:
# parted /dev/nvme0n1 mklabel gpt # parted /dev/nvme0n1 mkpart primary 2MiB 6MiB # parted /dev/nvme0n1 mkpart primary 6MiB 262MiB # parted /dev/nvme0n1 mkpart primary 262MiB 80%
# lsblk /dev/nvme0n1 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS nvme0n1 259:0 0 931.5G 0 disk ├─nvme0n1p1 259:1 0 4M 0 part ├─nvme0n1p2 259:2 0 256M 0 part └─nvme0n1p3 259:3 0 745.2G 0 part
- create
/boot
file system (petitboot on Odroid-M1 cannot read XFS, but can read EXT2/3/4)- on eMMC
# mkfs.ext2 /dev/mmcblk0p2
- on NVMe
# mkfs.ext2 /dev/nvme0n1p2
- on eMMC
- (optional) test the performance of LUKS encryption methods to determine which one to use
- Odroid-M1 cpu has acceleration for AES family ciphers
- NOTE: while eMMC card can sustains read/writes in 50-150/MBs the NVMe can easily go over 600MB/s therefore the encryption will become a bottleneck for performance on NVMe
# cryptsetup benchmark # Tests are approximate using memory only (no storage IO). PBKDF2-sha1 507539 iterations per second for 256-bit key PBKDF2-sha256 927943 iterations per second for 256-bit key PBKDF2-sha512 437636 iterations per second for 256-bit key PBKDF2-ripemd160 309497 iterations per second for 256-bit key PBKDF2-whirlpool 108863 iterations per second for 256-bit key argon2i 4 iterations, 381023 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time) argon2id 4 iterations, 384375 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time) # Algorithm | Key | Encryption | Decryption aes-cbc 128b 511.4 MiB/s 582.5 MiB/s serpent-cbc 128b 41.1 MiB/s 46.3 MiB/s twofish-cbc 128b 65.0 MiB/s 69.1 MiB/s aes-cbc 256b 439.6 MiB/s 533.3 MiB/s serpent-cbc 256b 41.8 MiB/s 46.3 MiB/s twofish-cbc 256b 65.4 MiB/s 69.0 MiB/s aes-xts 256b 536.5 MiB/s 536.9 MiB/s <---- we will use this one serpent-xts 256b 42.2 MiB/s 46.9 MiB/s twofish-xts 256b 66.5 MiB/s 69.7 MiB/s aes-xts 512b 511.3 MiB/s 510.8 MiB/s serpent-xts 512b 42.6 MiB/s 46.4 MiB/s twofish-xts 512b 67.8 MiB/s 69.6 MiB/s
- create and open LUKS encrypted partition for LVM with Gentoo system
- on eMMC
# cryptsetup --type luks2 --cipher aes-xts-plain64 --key-size 256 --hash sha256 --iter-time 2000 --use-random luksFormat /dev/mmcblk0p3 # cryptsetup luksOpen /dev/mmcblk0p3 data
- on NVMe
# cryptsetup --type luks2 --cipher aes-xts-plain64 --key-size 256 --hash sha256 --iter-time 2000 --use-random luksFormat /dev/nvme0n1p3 # cryptsetup luksOpen /dev/nvme0n1p3 data
- on eMMC
- create LVM VG and LV that will hold Gentoo
/
- XFS will be used for root file system but EXT4 is also not a bad option here
# vgcreate vg_system /dev/mapper/data # lvcreate -n root_lv -L10G vg_system # mkfs.xfs /dev/vg_system/root_lv
- XFS will be used for root file system but EXT4 is also not a bad option here
- mount (future) Gentoo
/
, download stage3 and unpack it there - I will use systemd variant of stage3 from nearby mirror# mount /dev/vg_system/root_lv /mnt # cd /mnt # wget http://ftp.kaist.ac.kr/gentoo/releases/arm64/autobuilds/20230219T224646Z/stage3-arm64-systemd-mergedusr-20230219T224646Z.tar.xz # tar xpvf stage3-*.tar.xz --xattrs-include='*.*' --numeric-owner
- mount
/boot
and other virtual file systems- on eMMC:
# mount /dev/mmcblk0p2 /mnt/boot # mount -t proc none /mnt/proc # mount --rbind /sys /mnt/sys # mount --rbind /dev /mnt/dev
# df -h /mnt /mnt/boot /mnt/proc /mnt/sys /mnt/dev Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_system-root_lv 10G 1.7G 8.3G 17% /mnt /dev/mmcblk0p2 238M 14K 226M 1% /mnt/boot none 0 0 0 - /mnt/proc sysfs 0 0 0 - /mnt/sys udev 10M 0 10M 0% /mnt/dev
- on NVMe:
# mount /dev/nvme0n1p2 /mnt/boot # mount -t proc none /mnt/proc # mount --rbind /sys /mnt/sys # mount --rbind /dev /mnt/dev
# df -h /mnt /mnt/boot /mnt/proc /mnt/sys /mnt/dev Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_system-root_lv 10G 1.7G 8.3G 17% /mnt /dev/nvme0n1p2 238M 14K 226M 1% /mnt/boot none 0 0 0 - /mnt/proc sysfs 0 0 0 - /mnt/sys udev 10M 0 10M 0% /mnt/dev
- on eMMC:
- chroot into Gentoo
/
# cp -L /etc/resolv.conf /mnt/etc # chroot /mnt /bin/bash # env-update # source /etc/profile
- configure
/etc/fstab
- use the UUIDs from your outputs (NOTE: UUID is not same thing as PARTUUID)- on eMMC:
# blkid |grep -E '(mmcblk0p2|root_lv)' /dev/mmcblk0p2: UUID="aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa" BLOCK_SIZE="1024" TYPE="ext2" PARTLABEL="primary" PARTUUID="aaaaaaaa-1111-1111-1111-111111111111" /dev/mapper/vg_system-root_lv: UUID="bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb" BLOCK_SIZE="512" TYPE="xfs" # echo 'UUID=bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb / xfs defaults 0 0' >> /etc/fstab # echo 'UUID=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa /boot ext2 noauto,noatime 0 0' >> /etc/fstab
- on NVMe:
# blkid |grep -E '(nvme0n1p2|root_lv)' /dev/nvme0n1p2: UUID="aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa" BLOCK_SIZE="1024" TYPE="ext2" PARTLABEL="primary" PARTUUID="aaaaaaaa-1111-1111-1111-111111111111" /dev/mapper/vg_system-root_lv: UUID="bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb" BLOCK_SIZE="512" TYPE="xfs" # echo 'UUID=bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb / xfs defaults 0 0' >> /etc/fstab # echo 'UUID=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa /boot ext2 noauto,noatime 0 0' >> /etc/fstab
- on eMMC:
- get portage tree
# emerge-webrsync
- configure timezone (adjust to your needs)
# echo "Asia/Seoul" > /etc/timezone # emerge --config sys-libs/timezone-data
- configure CPU flags for CPU (you can get these flags from
app-portage/cpuid2cpuflags
)# echo '*/* CPU_FLAGS_ARM: edsp neon thumb vfp vfpv3 vfpv4 vfp-d32 aes sha1 sha2 crc32 v4 v5 v6 v7 v8 thumb2' > /etc/portage/package.use/00cpuflags
- configure
/etc/portage/make.conf
- set architecture matching the CPU (you can also use
-march=native
) - set concurrency so we utilize all (4) CPU cores
# nano /etc/portage/make.conf ... COMMON_FLAGS="-O2 -pipe -march=armv8.2-a+crypto+fp16+rcpc+dotprod" ... MAKEOPTS="-j5"
- set architecture matching the CPU (you can also use
- emerge tools needed for building kernel (13pkgs at time of writing, compilation time ~39 minutes)
- to boot Gentoo
sys-kernel/linux-firmware
is not needed therefore lets disablefirmware
USE flag to save some space - to generate kernel and initramfs I will use
sys-kernel/genkernel
- to be able to use basic LVM utilities
sys-fs/lvm2
will be installed withlvm
USE flag - for management of encrypted LUKS partition package
sys-fs/cryptsetup
is needed - we will use
sys-kernel/gentoo-sources
to provide 6.1.x kernel sources from which kernel will be compiled# echo 'sys-kernel/genkernel -firmware' > /etc/portage/package.use/01_basic_os # echo 'sys-fs/lvm2 lvm' >> /etc/portage/package.use/01_basic_os # emerge -av sys-kernel/genkernel dev-embedded/u-boot-tools sys-fs/lvm2 sys-fs/cryptsetup sys-kernel/gentoo-sources
- to boot Gentoo
- lets select the installed kernel sources for 6.1.x kernel from
sys-kernel/gentoo-sources
package# eselect kernel set 1 # eselect kernel list Available kernel symlink targets: [1] linux-6.1.12-gentoo *
- additionally to sources download the Device Tree (DT) for Odroid M1 board and add it to kernel
- NOTE: this steps should be repeated each time the new kernel version is selected
# curl -o /usr/src/linux/arch/arm64/boot/dts/rockchip/rk3568-odroid-m1.dts https://raw.githubusercontent.com/tobetter/linux/odroid-6.1.y/arch/arm64/boot/dts/rockchip/rk3568-odroid-m1.dts # echo 'dtb-$(CONFIG_ARCH_ROCKCHIP) += rk3568-odroid-m1.dtb' >> /usr/src/linux/arch/arm64/boot/dts/rockchip/Makefile
- NOTE: this steps should be repeated each time the new kernel version is selected
- get some reasonable
.config
to start with for kernel compilation. For this guide I will use my minimized kernel config for 6.1 kernel# curl -o /root/2023-03-odroid-m1-config.cfg https://www.famera.cz/blog/assets/files/gentoo-odroid-m1/2023-03-odroid-m1-config.cfg
- configure genkernel
- disable microcode loading as there is no support for it on ARM CPUs
- add callback that will compile DTB file for the Odroid-M1 board that is needed to boot system and that will be copied into
/boot/
automatically same way as kernel and initramfs - enable LVM and LUKS support in initramfs
# nano /etc/genkernel.conf ... CMD_CALLBACK="make dtbs; copy_image_with_preserve gentoo.dtb arch/arm64/boot/dts/rockchip/rk3568-odroid-m1.dtb gentoo.dtb" MICROCODE="no" LVM="yes" LUKS="yes"
- compile the kernel (~168 minutes)
- you can customize kernel config by adding
--menuconfig
option - subsequent compilations are faster (around 45-50 minutes) as genkernel caches binaries for initramfs
# genkernel --kernel-config=/root/2023-03-odroid-m1-config.cfg all
- you can customize kernel config by adding
- get text version of
boot.scr
(boot.txt
) - this guide will use the minimized version ofboot.scr
based on one that came from Odroid’s Ubuntu# cd /boot # curl -o /boot/boot.txt https://www.famera.cz/blog/assets/files/gentoo-odroid-m1/2022-08-boot.txt
- customize
/boot/boot.txt
file to match your system- edit UUID of Gentoo
/
in the downloaded file to match your system - edit UUID for partition with LUKS
- enable TRIM support on root file system with
root_trim=yes
-
fk_kvers
should match the version of kernel that we are compiling - on eMMC:
# blkid |grep -E '(mmcblk0p3|root_lv)' /dev/mmcblk0p3: UUID="cccccccc-cccc-cccc-cccc-cccccccccccc" TYPE="crypto_LUKS" PARTLABEL="primary" PARTUUID="cccccccc-2222-2222-2222-222222222222" /dev/mapper/vg_system-root_lv: UUID="bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb" BLOCK_SIZE="512" TYPE="xfs" # nano boot.txt ... setenv bootlabel "Gentoo 6.1.12 LUKS LVM" ... setenv fk_kvers "6.1.12-gentoo-arm64" ... setenv bootargs " ${bootargs} dolvm crypt_root=UUID=cccccccc-cccc-cccc-cccc-cccccccccccc root=UUID=bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb root_trim=yes" ... # mkimage -A arm64 -T script -O linux -n 'boot script' -C none -d boot.txt boot.scr
- on NVMe:
# blkid |grep -E '(nvme0n1p3|root_lv)' /dev/nvme0n1p3: UUID="cccccccc-cccc-cccc-cccc-cccccccccccc" TYPE="crypto_LUKS" PARTLABEL="primary" PARTUUID="cccccccc-2222-2222-2222-222222222222" /dev/mapper/vg_system-root_lv: UUID="bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb" BLOCK_SIZE="512" TYPE="xfs" # nano boot.txt ... setenv bootlabel "Gentoo 6.1.12 LUKS LVM" ... setenv fk_kvers "6.1.12-gentoo-arm64" ... setenv bootargs " ${bootargs} dolvm crypt_root=UUID=cccccccc-cccc-cccc-cccc-cccccccccccc root=UUID=bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb root_trim=yes" ...
- edit UUID of Gentoo
- generate the binary version (
boot.scr
) that is recognized by petitboot (mkimage
is part ofdev-embedded/u-boot-tools
package)# mkimage -A arm64 -T script -O linux -n 'boot script' -C none -d boot.txt boot.scr
- (optional) check the content of
/boot
# ls -l /boot/ total 32093 -rw-r--r-- 1 root root 4259292 Feb 24 13:46 System.map-6.1.12-gentoo-arm64 -rw-r--r-- 1 root root 2046 Feb 25 10:26 boot.scr -rw-r--r-- 1 root root 1974 Feb 25 10:26 boot.txt -rw-r--r-- 1 root root 57456 Feb 24 14:09 gentoo.dtb -rw-r--r-- 1 root root 5261180 Feb 24 15:39 initramfs-6.1.12-gentoo-arm64.img drwx------ 2 root root 12288 Feb 24 11:55 lost+found -rw-r--r-- 1 root root 25513992 Feb 24 13:46 vmlinuz-6.1.12-gentoo-arm64
- configure root password, exit chroot and REBOOT
# passwd root # exit # reboot
- once the system starts booting you should be able to see the Gentoo 6.1.12 LUKS LVM entry from which you can boot the Gentoo
Petitboot (dev.20220424) Hardkernel ODROID-M1 --------------------------------------------------------------- [Disk: mmcblk0p2 / bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb] <=== Gentoo 6.1.12 LUKS LVM <=== ...
Petitboot (dev.20220424) Hardkernel ODROID-M1 --------------------------------------------------------------- [Disk: nvme0n1p2 / bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb] <=== Gentoo 6.1.12 LUKS LVM <=== ...
- during system boot you will be asked to unlock the LUKS partition which holds LVM with Gentoo
/
file system and after that system should boot into login console and can be used in usual way.
Lessons learned / Additional notes
- Odroid M1 hardware seems to be well supported by upstream Linux kernel with exception of boards DT file
- The repository from which DT for Odroid M1 comes has additional patches compared to vanilla upstream kernel worth exploring if something doesn’t work well
- notably Wake-on-lan support for integrated NIC
- HDMI changes which on monitor I have enables use of monitors native resolution on boot (instead of lower 1080p)
- to enable M2 slot with NVMe kernel config
CONFIG_PCIE_ROCKCHIP_DW_HOST=y
is used to compile in PCIe for this board- if support was compiled in properly then ‘PCI bridge’ should be visible in
lspci
output# lspci 0002:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01) <=== PCIe bridge 0002:01:00.0 Non-Volatile memory controller: SK hynix Gold P31/PC711 NVMe Solid State Drive <=== NVMe device (shown if it is present)
- if support was compiled in properly then ‘PCI bridge’ should be visible in
- for remote unlock of LUKS partition via SSH you can use genkernel SSH function - Gentoo genkernel wiki - Adding SSH support to initramfs
Additional resources and future plans
- 2022-08: previous guide - Gentoo on Odroid-M1 (aarch64)
- Odroid-M1 wiki: https://wiki.odroid.com/odroid-m1/odroid-m1
- Github tobetter/linux 6.1.y
Future plans/ideas:
- figuring out how to nicely integrate DT for M1 board into
gentoo-sources
workflow (possibly via/etc/portage/pathes
) - script for generating
boot.txt
/boot.scr
with correct kernel version automatically - versioning of
gentoo.dtb
to match the kernel for which it was compiled
Final thoughts
Revisiting this board after half year looks very promising in terms of long term support and I’m quite happy to see the progress that creators made to get the needed into Linux kernel.
In case you got stuck in the above procedure at some point feel free to drop me an email.