Ondrej Famera - top logo

Gentoo on Odroid-M1 (aarch64) with upstream 6.1.x kernel, LVM and LUKS

Followup on earlier experience with Odroids M1 board here is guide to get Gentoo with upstream 6.1.x kernel running on Odroids M1 board while running on top of LUKS encrypted disk with LVM. (examples for both eMMC and NVMe installation are included)

Previous guide - Gentoo on Odroid-M1 (aarch64).

Requirements

  • Odroid-M1 board - I have used 8GB RAM version, but I believe that 4GB should work too
  • bootable Linux system that has some basic utilities on it (this manual assumes presence of parted, cryptsetup and mkfs.xfs which are fairly common on distributions these days)
  • Internet connection - for getting files to install Gentoo and packages
  • approximately 4 hours of time :) - there are 2 steps where you have to wait for compilation to finish (first ~39 minutes for packages and later ~168 minutes for kernel compilation)
  • 16GB (or bigger) eMMC or NVMe

Installation

  • boot up some OS from SD card
  • partition the eMMC card or NVME (this manual will assume same layout for eMMC and NVMe) - GPT is used here
    • p1 - reservation for future use (custom u-boot)- not used in this manual
    • p2 - /boot to hold kernel, initramfs, DTB and boot.scr files

    • p3 - LUKS encrypted partition with LVM on it containing / LV for Gentoo

    • eMMC card partitioning: 
      # parted /dev/mmcblk0 mklabel gpt
# parted /dev/mmcblk0 mkpart primary 2MiB 6MiB
# parted /dev/mmcblk0 mkpart primary 6MiB 262MiB
# parted /dev/mmcblk0 mkpart primary 262MiB 80%

      # lsblk /dev/mmcblk0
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0 116.6G  0 disk
├─mmcblk0p1 179:1    0     4M  0 part
├─mmcblk0p2 179:2    0   256M  0 part
└─mmcblk0p3 179:3    0  93.1G  0 part
    • NVMe card partitioning: 
      # parted /dev/nvme0n1 mklabel gpt
# parted /dev/nvme0n1 mkpart primary 2MiB 6MiB
# parted /dev/nvme0n1 mkpart primary 6MiB 262MiB
# parted /dev/nvme0n1 mkpart primary 262MiB 80%

      # lsblk /dev/nvme0n1
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme0n1     259:0    0 931.5G  0 disk 
├─nvme0n1p1 259:1    0     4M  0 part
├─nvme0n1p2 259:2    0   256M  0 part 
└─nvme0n1p3 259:3    0 745.2G  0 part
  • create /boot file system (petitboot on Odroid-M1 cannot read XFS, but can read EXT2/3/4)
    • on eMMC 
      # mkfs.ext2 /dev/mmcblk0p2
    • on NVMe 
      # mkfs.ext2 /dev/nvme0n1p2
  • (optional) test the performance of LUKS encryption methods to determine which one to use
    • Odroid-M1 cpu has acceleration for AES family ciphers
    • NOTE: while eMMC card can sustains read/writes in 50-150/MBs the NVMe can easily go over 600MB/s therefore the encryption will become a bottleneck for performance on NVMe 
      # cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1       507539 iterations per second for 256-bit key
PBKDF2-sha256     927943 iterations per second for 256-bit key
PBKDF2-sha512     437636 iterations per second for 256-bit key
PBKDF2-ripemd160  309497 iterations per second for 256-bit key
PBKDF2-whirlpool  108863 iterations per second for 256-bit key
argon2i       4 iterations, 381023 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
argon2id      4 iterations, 384375 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
#     Algorithm |       Key |      Encryption |      Decryption
      aes-cbc        128b       511.4 MiB/s       582.5 MiB/s
  serpent-cbc        128b        41.1 MiB/s        46.3 MiB/s
  twofish-cbc        128b        65.0 MiB/s        69.1 MiB/s
      aes-cbc        256b       439.6 MiB/s       533.3 MiB/s
  serpent-cbc        256b        41.8 MiB/s        46.3 MiB/s
  twofish-cbc        256b        65.4 MiB/s        69.0 MiB/s
      aes-xts        256b       536.5 MiB/s       536.9 MiB/s   <---- we will use this one
  serpent-xts        256b        42.2 MiB/s        46.9 MiB/s
  twofish-xts        256b        66.5 MiB/s        69.7 MiB/s
      aes-xts        512b       511.3 MiB/s       510.8 MiB/s
  serpent-xts        512b        42.6 MiB/s        46.4 MiB/s
  twofish-xts        512b        67.8 MiB/s        69.6 MiB/s
  • create and open LUKS encrypted partition for LVM with Gentoo system
    • on eMMC 
      # cryptsetup --type luks2 --cipher aes-xts-plain64 --key-size 256 --hash sha256 --iter-time 2000 --use-random luksFormat /dev/mmcblk0p3
# cryptsetup luksOpen /dev/mmcblk0p3 data
    • on NVMe 
      # cryptsetup --type luks2 --cipher aes-xts-plain64 --key-size 256 --hash sha256 --iter-time 2000 --use-random luksFormat /dev/nvme0n1p3
# cryptsetup luksOpen /dev/nvme0n1p3 data
  • create LVM VG and LV that will hold Gentoo /
    • XFS will be used for root file system but EXT4 is also not a bad option here 
      # vgcreate vg_system /dev/mapper/data
# lvcreate -n root_lv -L10G vg_system
# mkfs.xfs /dev/vg_system/root_lv
  • mount (future) Gentoo /, download stage3 and unpack it there - I will use systemd variant of stage3 from nearby mirror 
    # mount /dev/vg_system/root_lv /mnt
# cd /mnt
# wget http://ftp.kaist.ac.kr/gentoo/releases/arm64/autobuilds/20230219T224646Z/stage3-arm64-systemd-mergedusr-20230219T224646Z.tar.xz
# tar xpvf stage3-*.tar.xz --xattrs-include='*.*' --numeric-owner
  • mount /boot and other virtual file systems
    • on eMMC: 
      # mount /dev/mmcblk0p2 /mnt/boot
# mount -t proc none /mnt/proc
# mount --rbind /sys /mnt/sys
# mount --rbind /dev /mnt/dev

      # df -h /mnt /mnt/boot /mnt/proc /mnt/sys /mnt/dev
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/vg_system-root_lv   10G  1.7G  8.3G  17% /mnt
/dev/mmcblk0p2                 238M   14K  226M   1% /mnt/boot
none                              0     0     0    - /mnt/proc
sysfs                             0     0     0    - /mnt/sys
udev                            10M     0   10M   0% /mnt/dev
    • on NVMe: 
      # mount /dev/nvme0n1p2 /mnt/boot
# mount -t proc none /mnt/proc
# mount --rbind /sys /mnt/sys
# mount --rbind /dev /mnt/dev

      # df -h /mnt /mnt/boot /mnt/proc /mnt/sys /mnt/dev
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/vg_system-root_lv   10G  1.7G  8.3G  17% /mnt
/dev/nvme0n1p2                 238M   14K  226M   1% /mnt/boot
none                              0     0     0    - /mnt/proc
sysfs                             0     0     0    - /mnt/sys
udev                            10M     0   10M   0% /mnt/dev
  • chroot into Gentoo / 
    # cp -L /etc/resolv.conf /mnt/etc
# chroot /mnt /bin/bash
# env-update
# source /etc/profile
  • configure /etc/fstab - use the UUIDs from your outputs (NOTE: UUID is not same thing as PARTUUID)
    • on eMMC: 
      # blkid |grep -E '(mmcblk0p2|root_lv)'
/dev/mmcblk0p2: UUID="aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa" BLOCK_SIZE="1024" TYPE="ext2" PARTLABEL="primary" PARTUUID="aaaaaaaa-1111-1111-1111-111111111111"
/dev/mapper/vg_system-root_lv: UUID="bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb" BLOCK_SIZE="512" TYPE="xfs"
# echo 'UUID=bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb / xfs defaults 0 0' >> /etc/fstab
# echo 'UUID=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa /boot ext2 noauto,noatime 0 0' >> /etc/fstab
    • on NVMe: 
      # blkid |grep -E '(nvme0n1p2|root_lv)'
/dev/nvme0n1p2: UUID="aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa" BLOCK_SIZE="1024" TYPE="ext2" PARTLABEL="primary" PARTUUID="aaaaaaaa-1111-1111-1111-111111111111"
/dev/mapper/vg_system-root_lv: UUID="bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb" BLOCK_SIZE="512" TYPE="xfs"
# echo 'UUID=bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb / xfs defaults 0 0' >> /etc/fstab
# echo 'UUID=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa /boot ext2 noauto,noatime 0 0' >> /etc/fstab
  • get portage tree 
    # emerge-webrsync
  • configure timezone (adjust to your needs) 
    # echo "Asia/Seoul" > /etc/timezone
# emerge --config sys-libs/timezone-data
  • configure CPU flags for CPU (you can get these flags from app-portage/cpuid2cpuflags) 
    # echo '*/* CPU_FLAGS_ARM: edsp neon thumb vfp vfpv3 vfpv4 vfp-d32 aes sha1 sha2 crc32 v4 v5 v6 v7 v8 thumb2' > /etc/portage/package.use/00cpuflags
  • configure /etc/portage/make.conf
    • set architecture matching the CPU (you can also use -march=native)
    • set concurrency so we utilize all (4) CPU cores
    # nano /etc/portage/make.conf
...
COMMON_FLAGS="-O2 -pipe -march=armv8.2-a+crypto+fp16+rcpc+dotprod"
...
MAKEOPTS="-j5"
  • emerge tools needed for building kernel (13pkgs at time of writing, compilation time ~39 minutes)
    • to boot Gentoo sys-kernel/linux-firmware is not needed therefore lets disable firmware USE flag to save some space
    • to generate kernel and initramfs I will use sys-kernel/genkernel
    • to be able to use basic LVM utilities sys-fs/lvm2 will be installed with lvm USE flag
    • for management of encrypted LUKS partition package sys-fs/cryptsetup is needed
    • we will use sys-kernel/gentoo-sources to provide 6.1.x kernel sources from which kernel will be compiled 
      # echo 'sys-kernel/genkernel -firmware' > /etc/portage/package.use/01_basic_os
# echo 'sys-fs/lvm2 lvm' >> /etc/portage/package.use/01_basic_os
# emerge -av sys-kernel/genkernel dev-embedded/u-boot-tools sys-fs/lvm2 sys-fs/cryptsetup sys-kernel/gentoo-sources
  • lets select the installed kernel sources for 6.1.x kernel from sys-kernel/gentoo-sources package 
    # eselect kernel set 1
# eselect kernel list
Available kernel symlink targets:
[1]   linux-6.1.12-gentoo *
  • additionally to sources download the Device Tree (DT) for Odroid M1 board and add it to kernel
    • NOTE: this steps should be repeated each time the new kernel version is selected 
      # curl -o /usr/src/linux/arch/arm64/boot/dts/rockchip/rk3568-odroid-m1.dts https://raw.githubusercontent.com/tobetter/linux/odroid-6.1.y/arch/arm64/boot/dts/rockchip/rk3568-odroid-m1.dts
# echo 'dtb-$(CONFIG_ARCH_ROCKCHIP) += rk3568-odroid-m1.dtb' >> /usr/src/linux/arch/arm64/boot/dts/rockchip/Makefile
  • get some reasonable .config to start with for kernel compilation. For this guide I will use my minimized kernel config for 6.1 kernel 
    # curl -o /root/2023-03-odroid-m1-config.cfg https://www.famera.cz/blog/assets/files/gentoo-odroid-m1/2023-03-odroid-m1-config.cfg
  • configure genkernel
    • disable microcode loading as there is no support for it on ARM CPUs
    • add callback that will compile DTB file for the Odroid-M1 board that is needed to boot system and that will be copied into /boot/ automatically same way as kernel and initramfs
    • enable LVM and LUKS support in initramfs
    # nano /etc/genkernel.conf
...
CMD_CALLBACK="make dtbs; copy_image_with_preserve gentoo.dtb arch/arm64/boot/dts/rockchip/rk3568-odroid-m1.dtb gentoo.dtb"
MICROCODE="no"
LVM="yes"
LUKS="yes"
  • compile the kernel (~168 minutes)
    • you can customize kernel config by adding --menuconfig option
    • subsequent compilations are faster (around 45-50 minutes) as genkernel caches binaries for initramfs
    # genkernel --kernel-config=/root/2023-03-odroid-m1-config.cfg all
  • get text version of boot.scr (boot.txt) - this guide will use the minimized version of boot.scr based on one that came from Odroid’s Ubuntu 
    # cd /boot
# curl -o /boot/boot.txt https://www.famera.cz/blog/assets/files/gentoo-odroid-m1/2022-08-boot.txt
  • customize /boot/boot.txt file to match your system
    • edit UUID of Gentoo / in the downloaded file to match your system
    • edit UUID for partition with LUKS
    • enable TRIM support on root file system with root_trim=yes

    • fk_kvers should match the version of kernel that we are compiling

    • on eMMC: 
      # blkid |grep -E '(mmcblk0p3|root_lv)'
/dev/mmcblk0p3: UUID="cccccccc-cccc-cccc-cccc-cccccccccccc" TYPE="crypto_LUKS" PARTLABEL="primary" PARTUUID="cccccccc-2222-2222-2222-222222222222"
/dev/mapper/vg_system-root_lv: UUID="bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb" BLOCK_SIZE="512" TYPE="xfs"
# nano boot.txt
...
setenv bootlabel "Gentoo 6.1.12 LUKS LVM"
...
setenv fk_kvers "6.1.12-gentoo-arm64"
...
setenv bootargs " ${bootargs} dolvm crypt_root=UUID=cccccccc-cccc-cccc-cccc-cccccccccccc root=UUID=bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb root_trim=yes"
...
# mkimage -A arm64 -T script -O linux -n 'boot script' -C none -d boot.txt boot.scr
    • on NVMe: 
      # blkid |grep -E '(nvme0n1p3|root_lv)'
/dev/nvme0n1p3: UUID="cccccccc-cccc-cccc-cccc-cccccccccccc" TYPE="crypto_LUKS" PARTLABEL="primary" PARTUUID="cccccccc-2222-2222-2222-222222222222"
/dev/mapper/vg_system-root_lv: UUID="bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb" BLOCK_SIZE="512" TYPE="xfs"
# nano boot.txt
...
setenv bootlabel "Gentoo 6.1.12 LUKS LVM"
...
setenv fk_kvers "6.1.12-gentoo-arm64"
...
setenv bootargs " ${bootargs} dolvm crypt_root=UUID=cccccccc-cccc-cccc-cccc-cccccccccccc root=UUID=bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb root_trim=yes"
...
  • generate the binary version (boot.scr) that is recognized by petitboot (mkimage is part of dev-embedded/u-boot-tools package) 
    # mkimage -A arm64 -T script -O linux -n 'boot script' -C none -d boot.txt boot.scr
  • (optional) check the content of /boot 
    # ls -l /boot/
total 32093
-rw-r--r-- 1 root root  4259292 Feb 24 13:46 System.map-6.1.12-gentoo-arm64
-rw-r--r-- 1 root root     2046 Feb 25 10:26 boot.scr
-rw-r--r-- 1 root root     1974 Feb 25 10:26 boot.txt
-rw-r--r-- 1 root root    57456 Feb 24 14:09 gentoo.dtb
-rw-r--r-- 1 root root  5261180 Feb 24 15:39 initramfs-6.1.12-gentoo-arm64.img
drwx------ 2 root root    12288 Feb 24 11:55 lost+found
-rw-r--r-- 1 root root 25513992 Feb 24 13:46 vmlinuz-6.1.12-gentoo-arm64
  • configure root password, exit chroot and REBOOT 
    # passwd root
# exit
# reboot
  • once the system starts booting you should be able to see the Gentoo 6.1.12 LUKS LVM entry from which you can boot the Gentoo 
    Petitboot (dev.20220424)                   Hardkernel ODROID-M1
---------------------------------------------------------------
[Disk: mmcblk0p2 / bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb]         <===
Gentoo 6.1.12 LUKS LVM                                           <===
...

    Petitboot (dev.20220424)                   Hardkernel ODROID-M1
---------------------------------------------------------------
[Disk: nvme0n1p2 / bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb]         <===
Gentoo 6.1.12 LUKS LVM                                           <===
...
  • during system boot you will be asked to unlock the LUKS partition which holds LVM with Gentoo / file system and after that system should boot into login console and can be used in usual way.

Lessons learned / Additional notes

  • Odroid M1 hardware seems to be well supported by upstream Linux kernel with exception of boards DT file
  • The repository from which DT for Odroid M1 comes has additional patches compared to vanilla upstream kernel worth exploring if something doesn’t work well
    • notably Wake-on-lan support for integrated NIC
    • HDMI changes which on monitor I have enables use of monitors native resolution on boot (instead of lower 1080p)
  • to enable M2 slot with NVMe kernel config CONFIG_PCIE_ROCKCHIP_DW_HOST=y is used to compile in PCIe for this board
    • if support was compiled in properly then ‘PCI bridge’ should be visible in lspci output 
      # lspci
0002:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01)  <=== PCIe bridge
0002:01:00.0 Non-Volatile memory controller: SK hynix Gold P31/PC711 NVMe Solid State Drive     <=== NVMe device (shown if it is present)
  • for remote unlock of LUKS partition via SSH you can use genkernel SSH function - Gentoo genkernel wiki - Adding SSH support to initramfs

Additional resources and future plans

Future plans/ideas:

  • figuring out how to nicely integrate DT for M1 board into gentoo-sources workflow (possibly via /etc/portage/pathes)
  • script for generating boot.txt/boot.scr with correct kernel version automatically
  • versioning of gentoo.dtb to match the kernel for which it was compiled

Final thoughts

Revisiting this board after half year looks very promising in terms of long term support and I’m quite happy to see the progress that creators made to get the needed into Linux kernel.

In case you got stuck in the above procedure at some point feel free to drop me an email.

Last change .
© 2011-2022, backend by Jekyll