User:Jessesimpson36/HiFive Unmatched Rev-B RISC-V Installation guide
HiFive Unmatched Rev-B RISC-V Installation guide
The following is some notes I came across when setting up my RISC-V board. It's also my first time trying to write gentoo documentation, so hopefully I follow through with adding details.
My installation journey
When I first tried to install Gentoo on this board, I mostly struggled with U-boot and getting any sort of serial output. So how I went about installing it was by taking the default SD Card provided by the board, and flashing a new sd card with the same partition scheme and files, with the exception of the linux filesystem, which instead of Ubuntu, I used my own Gentoo filesystem. I am able to boot with that, but I'm eventually going to want to swap out u-boot with a newer version of u-boot that I compile myself (and is more up-to-date), and I will want to compile my own kernel.
I also foolishly bought a USB wifi adapter that I will need to compile my own out-of-tree kernel module in order to get a network connection.
Some of the steps in this guide may be redundant, since I'm first going to write this to mirror my installation journey, and then I will optimize it to be a more generic install guide.
What comes in the default SD Card provided with the board
With the Unmatched board, there is an SD card with the following partition table.
Disk /dev/mmcblk0: 29.72 GiB, 31914983424 bytes, 62333952 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 1FB1F4BB-00FC-4169-920A-567D008A5404 Device Start End Sectors Size Type /dev/mmcblk0p1 34 2081 2048 1M HiFive FSBL /dev/mmcblk0p2 2082 10273 8192 4M HiFive BBL /dev/mmcblk0p3 16384 282623 266240 130M Microsoft basic data /dev/mmcblk0p4 286720 13918207 13631488 6.5G Linux filesystem
mmcblk0p1: FSBL Partition
explain what FSBL is
mmcblk0p2: BBL Partition
explain what BBL is
mmcblk0p3: NTFS filesystem
This filesystem is a typical /boot mount with a configuration of extlinux, as well as the device tree and a kernel.
mmcblk0p4: Root linux filesystem
This is where typical userspace linux files go. By default, this filesystem seems to be Ubuntu.
Setting up RISC-V cross-compiling chroot environment
Compiling_with_QEMU_user_chroot
The above link helped me set up a chroot environment.
Setting up Serial port
The first thing you'll want to be able to do is be able to connect to the board and send commands in. There are 3 places that you need to supply options for the serial port to work.
- U-boot
- Kernel boot logs
- Login prompt
Serial port: U-boot
I don't recall if I did anything special to configure U-boot for this. I'll think on it.
Serial port: Kernel
During the boot process, the kernel needs to know where to send log messages. The following kernel command line parameter will tell the kernel to send log messages to the serial port.
console=ttySIF0,115200
Serial port: Login prompt
After being able to access U-boot and the kernel, you'll want to be able to log in with a username and password. Otherwise you'll only see kernel logs. To do this, you'll need to have the following line in your /etc/inittab
s0:12345:respawn:/sbin/agetty -L 115200 ttySIF0 vt100
Client serial port setup
On the client, when you plug in your serial port, you will see 2 devices
- /dev/ttyUSB0
- /dev/ttyUSB1
/dev/ttyUSB0 is unused.
Use this command to connect to the serial port:
root #minicom -b 115200 -D /dev/ttyUSB1or screen:
root #screen -L /dev/ttyUSB1 115200If you get no serial output at all when you start your RISC-V machine, try using this baud rate to get a very early boot error code and refer to the HiFive Software Reference Manual for a list of error codes:
root #screen -L /dev/ttyUSB1 89856U-Boot commands for a proper boot flow
Below is a successful boot prompt. To drop into a shell, with a serial connection open, hit space within the first few seconds of booting. This will halt the automatic bootflow and open a shell.
U-Boot 2021.01 (Apr 07 2021 - 17:59:15 +0000) CPU: rv64imafdc Model: SiFive HiFive Unmatched A00 DRAM: 16 GiB MMC: spi@10050000:mmc@0: 0 EEPROM: SiFive PCB EEPROM format v1 Serial number: SF105SZ233800206 PCB revision: 4 Ethernet MAC address: 70:b3:d5:92:fc:42 CRC: 946d66d2 EEPROM dump: (0x25 bytes) 00: F1 5E 50 45 01 02 00 04 42 00 53 46 31 30 35 53 10: 5A 32 33 33 38 30 30 32 30 36 01 70 B3 D5 92 FC 20: 42 D2 66 6D 94 found SiFive v1 In: serial@10010000 Out: serial@10010000 Err: serial@10010000 Model: SiFive HiFive Unmatched A00 Net: eth0: ethernet@10090000 Hit any key to stop autoboot: 0 => env set bootargs root=/dev/mmcblk0p4 rootfstype=ext4 rootwait console=ttySIF0,115200 => mmc rescan => load mmc 0:4 0x82000000 boot/Image.gz 7247925 bytes read in 4725 ms (1.5 MiB/s) => load mmc 0:3 0x88000000 hifive-unmatched-a00.dtb 10473 bytes read in 12 ms (851.6 KiB/s) => booti 0x82000000 - 0x88000000 Uncompressing Kernel Image Moving Image from 0x82000000 to 0x80200000, end=810b2000 ## Flattened Device Tree blob at 88000000 Booting using the fdt blob at 0x88000000 Using Device Tree in place at 0000000088000000, end 00000000880058e8 Starting kernel ... <kernel messages follow>
Explanation of commands
Set kernel command line options
In U-Boot, you load the kernel command line arguments as an environment variable.
env set bootargs root=/dev/mmcblk0p4 rootfstype=ext4 rootwait console=ttySIF0,115200
Why I set each kernel command line option
- root=/dev/mmcblk0p4 - This declares the block device of the root filesystem
- rootfstype=ext4 - I don't think this is necessary, but tells the kernel what filesystem it is.
- rootwait - SD cards tend to load in the kernel slower than other storage devices, this prevents mounting root before the SD card is available.
- console=ttySIF0,115200 - Output logs to the serial port ttySIF0 with a baud rate of 115200.
Rescan block devices
mmc rescan will just reload the disk. I'm unsure if this is necessary.
mmc rescan
Load kernel into memory
Load the kernel into memory address 0x82000000. There is nothing special about this memory address other than it isn't reserved for anything useful. If you need to find your kernel file from the u-boot command line, check out the help page for ext4ls in the u-boot command line.
load mmc 0:4 0x82000000 boot/Image.gz
Explanation
- mmc - read from an SD card interface
- 0:4 - device number 0 with partition number 4
- 0x82000000 - nonspecific memory address
- boot/Image.gz - path to a kernel
Load device tree into memory
Load from the device tree binary
load mmc 0:3 0x88000000 hifive-unmatched-a00.dtb
Explanation
- mmc - read from an SD card interface
- 0:3 - read the file from the /boot partition (partition 3)
- 0x88000000 - nonspecific memory address that doesn't overlap with the previous address (0x82000000)
- hifive-unmatched-a00.dtb - device tree binary file
Boot
Boot from the memory addresses that were loaded.
booti 0x82000000 - 0x88000000
- 0x82000000 - memory address we used to load the kernel
- dash - the dash indicates that we will not be using an initramfs. If you are using an initramfs, place the memory address of it here.
- 0x88000000 - memory address we used to load the device tree binary.
Beyond initial boot
I'd like to be able to load my boot instructions from a file so that I don't have to keep copy/pasting the u-boot commands. Newer versions of U-boot have a "bootflow" file which can set a default and boot once the device is turned on. For that I will need to compile my own, newer version of u-boot.
Moving from a Micro SD card to an NVMe Drive
Lately I've been dissatisfied with the performance and reliability with my micro-sd cards that I'm using to boot off of. I found that my SD cards would get corrupted fairly easily, and that they were quite slow when doing things like compiling kernels. And what I thought would be a simple operation (duplicating the disk to a new NVMe drive and booting) is actually more complicated. This section will go over the problem I faced and how I was able to work around it.
Problem
The kernel version 5.11.10, which is the kernel shipped on the SD card with the Unmatched RISC-V board, does not support using an NVMe drive on the motherboard. The patchset that allows the kernel to do this is in this commit: https://lwn.net/ml/linux-kernel/cover.1614681831.git.greentime.hu@sifive.com/
Which the first commit in that patch set is dated for Tue, 02 Mar 2021 18:59:11 +0800
And according to https://en.wikipedia.org/wiki/Linux_kernel_version_history the commit would not be introduced until a 5.12 kernel version.
I first started by trying to cross-compile the kernel using my chroot environment. Unfortunately, this produced a kernel with a faulty magic-number and was unable to boot on the risc-v machine.
My next thought was to use the risc-v machine to compile a new kernel onto the sd card. This lasted all night and I woke up finding that the micro-sd card was corrupted from too many write operations. (Thankfully I take backups).
My current approach will be to compile a kernel by booting into the sd card, but this time, attaching a different disk and all the writes will go on the new disk. I found that USB external hard drives did not work on the board, so I will now attempt to mount an NFS and compile the kernel on there (using qemu-nbd and a block-device file on the NFS because NFS doesn't preserve file permissions).
Creating a dtb file
dtb files are device tree binaries and contain necessary information about how the hardware interacts with the kernel. When compiling the linux kernel with genkernel, you are likely generating a bunch of dts files for different boards, and each of these dts files can be converted to a dtb file which can be used at boot in the u-boot prompt. The following is how to compile the dtb from a c++ source file.
cpp -nostdinc -I include -undef -x assembler-with-cpp ./arch/riscv/boot/dts/sifive/hifive-unmatched-a00.dts > hifive-unmatched-a00.tmp.dts dtc -O dtb -b 0 -o hifive-unmatched-a00.dtb hifive-unmatched-a00.tmp.dts
Once booted on a kernel with nvme support
At this point, you should be booted into the mmc device, and running fdisk -l should report an NVMe drive. the goal is to now copy the data off the mmc drive onto the nvme drive. thankfully, the Hi-Five FSBL partition and the Hi-Five BBL partitions are only read while used in the u-boot phase of booting, so those can safely be copied.
Recommended packages
Install sys-apps/gptfdisk for sgdisk and sys-fs/dosfstools for mkfs.vfat.
creating the new partitions
Follow the instructions from https://docs.u-boot.org/en/latest/board/sifive/unmatched.html#flashing
root #sgdisk -g --clear -a 1 \
--new=1:34:2081 --change-name=1:spl --typecode=1:5B193300-FC78-40CD-8002-E86C45580B47 \
--new=2:2082:10273 --change-name=2:uboot --typecode=2:2E54B353-1271-4842-806F-E436D6AF6985 \
--new=3:16384:282623 --change-name=3:boot --typecode=3:0x0700 \
--new=4:286720:13918207 --change-name=4:root --typecode=4:0x8300 \
/dev/nvme0n1we will also want to expand the fourth partition to be larger. I used cfdisk for this. theres probably a way to edit the above command to use the rest of the disk space for the fourth partition.
root #dd if=/dev/mmcblk0p1 of=/dev/nvme0n1p1 bs=4M status=progressroot #dd if=/dev/mmcblk0p2 of=/dev/nvme0n1p2 bs=4M status=progress
root #mkfs.vfat /dev/nvme0n1p3root #mkfs.ext4 /dev/nvme0n1p4/boot is also not often used much after being loaded during initial boot, so mount the nvme partition 3 and cp the contents of /boot into there.
The more complicated part will be copying the root filesystem. If you took a backup of the root filesystem like I did, then you can mount the nfs drive and copy the backup onto the new root filesystem.
fixing /etc/fstab to the nvme drive
final boot commands in u-boot
env set bootargs root=/dev/nvme0n1p4 rootfstype=ext4 rootwait console=ttySIF0,115200 nvme scan fatls nvme 0:3 / load nvme 0:3 0x82000000 vmlinuz-6.9.3-gentoo-riscv64 load nvme 0:3 0x88000000 hifive-unmatched-a00-6.9.3.dtb booti 0x82000000 - 0x88000000
automating u-boot login
I spent a while debugging why my extlinux/extlinux.conf file wasn't able to be booted from. As it turns out, for GPT partition schemes, u-boot requires the partition with your extlinux/extlinux.conf file to be marked as bootable. cfdisk does not support setting the bootable flag on GPT so you will need to use fdisk to do it.
fdisk x - enable expert commands A - toggle the legacy BIOS bootable flag 3 - partition number 3 (the one with /extlinux/extlinux.conf or /boot/extlinux/extlinux.conf
Once I had that, I was able to get a boot to happen on NVMe by explicitly setting
setenv boot_targets nvme bootmeth order extlinux bootflow scan
Otherwise, it seems that network booting takes higher priority than NVMe. I'll add more to this section to figure out how to make it boot from NVMe as higher precedence. also I will give my extlinux.conf file as an example, but not tonight.
fixing the time
Unfortunately, the hifive unmatched revb board does not support the RTC module in the kernel yet. From their official software manual
> There is no support for the RTC function on the HiFive Unmatched yet. Time information can be acquired from an Ethernet or WiFi network. If a user isn’t connected to a network where time information is available then some functions that depend on correct time may not work such as ssh.
https://www.sifive.com/boards/hifive-unmatched-revb
However, you can still fix the time by using ntp. I recommend not installing openntpd, because they have a max drift that is hard to force update the clock. so I would instead install net-misc/ntp
Useful links
u-boot-sifive-board-specific-docs
hifive-unmatched-software-reference-manual
Compiling_with_QEMU_user_chroot
https://youtu.be/INWghYZH3hI?si=ECEsmj0Xu3kFGOGR