User:Jessesimpson36/HiFive Unmatched Rev-B RISC-V Installation guide

HiFive Unmatched Rev-B RISC-V Installation guide

The following is some notes I came across when setting up my RISC-V board. It's also my first time trying to write gentoo documentation, so hopefully I follow through with adding details.

My installation journey

When I first tried to install Gentoo on this board, I mostly struggled with U-boot and getting any sort of serial output. So how I went about installing it was by taking the default SD Card provided by the board, and flashing a new sd card with the same partition scheme and files, with the exception of the linux filesystem, which instead of Ubuntu, I used my own Gentoo filesystem. I am able to boot with that, but I'm eventually going to want to swap out u-boot with a newer version of u-boot that I compile myself (and is more up-to-date), and I will want to compile my own kernel.

I also foolishly bought a USB wifi adapter that I will need to compile my own out-of-tree kernel module in order to get a network connection.

Some of the steps in this guide may be redundant, since I'm first going to write this to mirror my installation journey, and then I will optimize it to be a more generic install guide.

What comes in the default SD Card provided with the board

With the Unmatched board, there is an SD card with the following partition table.

Disk /dev/mmcblk0: 29.72 GiB, 31914983424 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 1FB1F4BB-00FC-4169-920A-567D008A5404

Device          Start      End  Sectors  Size Type
/dev/mmcblk0p1     34     2081     2048    1M HiFive FSBL
/dev/mmcblk0p2   2082    10273     8192    4M HiFive BBL
/dev/mmcblk0p3  16384   282623   266240  130M Microsoft basic data
/dev/mmcblk0p4 286720 13918207 13631488  6.5G Linux filesystem

mmcblk0p1: FSBL Partition

explain what FSBL is

mmcblk0p2: BBL Partition

explain what BBL is

mmcblk0p3: NTFS filesystem

This filesystem is a typical /boot mount with a configuration of extlinux, as well as the device tree and a kernel.

mmcblk0p4: Root linux filesystem

This is where typical userspace linux files go. By default, this filesystem seems to be Ubuntu.

Setting up RISC-V cross-compiling chroot environment

Compiling_with_QEMU_user_chroot

The above link helped me set up a chroot environment.

Setting up Serial port

The first thing you'll want to be able to do is be able to connect to the board and send commands in. There are 3 places that you need to supply options for the serial port to work.

U-boot
Kernel boot logs
Login prompt

Serial port: U-boot

I don't recall if I did anything special to configure U-boot for this. I'll think on it.

Serial port: Kernel

During the boot process, the kernel needs to know where to send log messages. The following kernel command line parameter will tell the kernel to send log messages to the serial port.

console=ttySIF0,115200

Serial port: Login prompt

After being able to access U-boot and the kernel, you'll want to be able to log in with a username and password. Otherwise you'll only see kernel logs. To do this, you'll need to have the following line in your /etc/inittab

s0:12345:respawn:/sbin/agetty -L 115200 ttySIF0 vt100

Client serial port setup

On the client, when you plug in your serial port, you will see 2 devices

/dev/ttyUSB0
/dev/ttyUSB1

/dev/ttyUSB0 is unused.

Use this command to connect to the serial port:

root #minicom -b 115200 -D /dev/ttyUSB1

or screen:

root #screen -L /dev/ttyUSB1 115200

If you get no serial output at all when you start your RISC-V machine, try using this baud rate to get a very early boot error code and refer to the HiFive Software Reference Manual for a list of error codes:

root #screen -L /dev/ttyUSB1 89856

U-Boot commands for a proper boot flow

Below is a successful boot prompt. To drop into a shell, with a serial connection open, hit space within the first few seconds of booting. This will halt the automatic bootflow and open a shell.

U-Boot 2021.01 (Apr 07 2021 - 17:59:15 +0000)

CPU:   rv64imafdc
Model: SiFive HiFive Unmatched A00
DRAM:  16 GiB
MMC:   spi@10050000:mmc@0: 0
EEPROM: SiFive PCB EEPROM format v1
Serial number: SF105SZ233800206
PCB revision: 4
Ethernet MAC address: 70:b3:d5:92:fc:42
CRC: 946d66d2
EEPROM dump: (0x25 bytes)
00: F1 5E 50 45 01 02 00 04 42 00 53 46 31 30 35 53 
10: 5A 32 33 33 38 30 30 32 30 36 01 70 B3 D5 92 FC 
20: 42 D2 66 6D 94 
found SiFive v1
In:    serial@10010000
Out:   serial@10010000
Err:   serial@10010000
Model: SiFive HiFive Unmatched A00
Net:   eth0: ethernet@10090000
Hit any key to stop autoboot:  0 
=> env set bootargs root=/dev/mmcblk0p4 rootfstype=ext4 rootwait console=ttySIF0,115200
=> mmc rescan
=> load mmc 0:4 0x82000000 boot/Image.gz
7247925 bytes read in 4725 ms (1.5 MiB/s)
=> load mmc 0:3 0x88000000 hifive-unmatched-a00.dtb
10473 bytes read in 12 ms (851.6 KiB/s)

=> booti 0x82000000 - 0x88000000
   Uncompressing Kernel Image
Moving Image from 0x82000000 to 0x80200000, end=810b2000
## Flattened Device Tree blob at 88000000
   Booting using the fdt blob at 0x88000000
   Using Device Tree in place at 0000000088000000, end 00000000880058e8

Starting kernel ...
<kernel messages follow>

Explanation of commands

Set kernel command line options

In U-Boot, you load the kernel command line arguments as an environment variable.

env set bootargs root=/dev/mmcblk0p4 rootfstype=ext4 rootwait console=ttySIF0,115200

Why I set each kernel command line option

root=/dev/mmcblk0p4 - This declares the block device of the root filesystem
rootfstype=ext4 - I don't think this is necessary, but tells the kernel what filesystem it is.
rootwait - SD cards tend to load in the kernel slower than other storage devices, this prevents mounting root before the SD card is available.
console=ttySIF0,115200 - Output logs to the serial port ttySIF0 with a baud rate of 115200.

Rescan block devices

mmc rescan will just reload the disk. I'm unsure if this is necessary.

mmc rescan

Load kernel into memory

Load the kernel into memory address 0x82000000. There is nothing special about this memory address other than it isn't reserved for anything useful. If you need to find your kernel file from the u-boot command line, check out the help page for ext4ls in the u-boot command line.

load mmc 0:4 0x82000000 boot/Image.gz

Explanation

mmc - read from an SD card interface
0:4 - device number 0 with partition number 4
0x82000000 - nonspecific memory address
boot/Image.gz - path to a kernel

Load device tree into memory

Load from the device tree binary

load mmc 0:3 0x88000000 hifive-unmatched-a00.dtb

Explanation

mmc - read from an SD card interface
0:3 - read the file from the /boot partition (partition 3)
0x88000000 - nonspecific memory address that doesn't overlap with the previous address (0x82000000)
hifive-unmatched-a00.dtb - device tree binary file

Boot

Boot from the memory addresses that were loaded.

booti 0x82000000 - 0x88000000

0x82000000 - memory address we used to load the kernel
dash - the dash indicates that we will not be using an initramfs. If you are using an initramfs, place the memory address of it here.
0x88000000 - memory address we used to load the device tree binary.

Beyond initial boot

I'd like to be able to load my boot instructions from a file so that I don't have to keep copy/pasting the u-boot commands. Newer versions of U-boot have a "bootflow" file which can set a default and boot once the device is turned on. For that I will need to compile my own, newer version of u-boot.

Moving from a Micro SD card to an NVMe Drive

Lately I've been dissatisfied with the performance and reliability with my micro-sd cards that I'm using to boot off of. I found that my SD cards would get corrupted fairly easily, and that they were quite slow when doing things like compiling kernels. And what I thought would be a simple operation (duplicating the disk to a new NVMe drive and booting) is actually more complicated. This section will go over the problem I faced and how I was able to work around it.

Problem

The kernel version 5.11.10, which is the kernel shipped on the SD card with the Unmatched RISC-V board, does not support using an NVMe drive on the motherboard. The patchset that allows the kernel to do this is in this commit: https://lwn.net/ml/linux-kernel/cover.1614681831.git.greentime.hu@sifive.com/

Which the first commit in that patch set is dated for Tue, 02 Mar 2021 18:59:11 +0800

And according to https://en.wikipedia.org/wiki/Linux_kernel_version_history the commit would not be introduced until a 5.12 kernel version.

I first started by trying to cross-compile the kernel using my chroot environment. Unfortunately, this produced a kernel with a faulty magic-number and was unable to boot on the risc-v machine.

My next thought was to use the risc-v machine to compile a new kernel onto the sd card. This lasted all night and I woke up finding that the micro-sd card was corrupted from too many write operations. (Thankfully I take backups).

My current approach will be to compile a kernel by booting into the sd card, but this time, attaching a different disk and all the writes will go on the new disk. I found that USB external hard drives did not work on the board, so I will now attempt to mount an NFS and compile the kernel on there (using qemu-nbd and a block-device file on the NFS because NFS doesn't preserve file permissions).

Creating a dtb file

dtb files are device tree binaries and contain necessary information about how the hardware interacts with the kernel. When compiling the linux kernel with genkernel, you are likely generating a bunch of dts files for different boards, and each of these dts files can be converted to a dtb file which can be used at boot in the u-boot prompt. The following is how to compile the dtb from a c++ source file.

cpp -nostdinc -I include -undef -x assembler-with-cpp ./arch/riscv/boot/dts/sifive/hifive-unmatched-a00.dts > hifive-unmatched-a00.tmp.dts

dtc -O dtb -b 0 -o hifive-unmatched-a00.dtb hifive-unmatched-a00.tmp.dts

Once booted on a kernel with nvme support

At this point, you should be booted into the mmc device, and running fdisk -l should report an NVMe drive. the goal is to now copy the data off the mmc drive onto the nvme drive. thankfully, the Hi-Five FSBL partition and the Hi-Five BBL partitions are only read while used in the u-boot phase of booting, so those can safely be copied.

Recommended packages

Install sys-apps/gptfdisk for sgdisk and sys-fs/dosfstools for mkfs.vfat.

creating the new partitions

Follow the instructions from https://docs.u-boot.org/en/latest/board/sifive/unmatched.html#flashing

root #

sgdisk -g --clear -a 1 \
 --new=1:34:2081         --change-name=1:spl --typecode=1:5B193300-FC78-40CD-8002-E86C45580B47 \
 --new=2:2082:10273      --change-name=2:uboot  --typecode=2:2E54B353-1271-4842-806F-E436D6AF6985 \
 --new=3:16384:282623    --change-name=3:boot --typecode=3:0x0700 \
 --new=4:286720:13918207 --change-name=4:root --typecode=4:0x8300 \

  /dev/nvme0n1

we will also want to expand the fourth partition to be larger. I used cfdisk for this. theres probably a way to edit the above command to use the rest of the disk space for the fourth partition.

root #dd if=/dev/mmcblk0p1 of=/dev/nvme0n1p1 bs=4M status=progress

root #dd if=/dev/mmcblk0p2 of=/dev/nvme0n1p2 bs=4M status=progress

root #mkfs.vfat /dev/nvme0n1p3

root #mkfs.ext4 /dev/nvme0n1p4

/boot is also not often used much after being loaded during initial boot, so mount the nvme partition 3 and cp the contents of /boot into there.

The more complicated part will be copying the root filesystem. If you took a backup of the root filesystem like I did, then you can mount the nfs drive and copy the backup onto the new root filesystem.

fixing /etc/fstab to the nvme drive

final boot commands in u-boot

env set bootargs root=/dev/nvme0n1p4 rootfstype=ext4 rootwait console=ttySIF0,115200
nvme scan
fatls nvme 0:3 /
load nvme 0:3 0x82000000 vmlinuz-6.9.3-gentoo-riscv64
load nvme 0:3 0x88000000 hifive-unmatched-a00-6.9.3.dtb
booti 0x82000000 - 0x88000000

automating u-boot login

I spent a while debugging why my extlinux/extlinux.conf file wasn't able to be booted from. As it turns out, for GPT partition schemes, u-boot requires the partition with your extlinux/extlinux.conf file to be marked as bootable. cfdisk does not support setting the bootable flag on GPT so you will need to use fdisk to do it.

fdisk
x - enable expert commands
A - toggle the legacy BIOS bootable flag
3 - partition number 3 (the one with /extlinux/extlinux.conf or /boot/extlinux/extlinux.conf

Once I had that, I was able to get a boot to happen on NVMe by explicitly setting

setenv boot_targets nvme
bootmeth order extlinux
bootflow scan

Otherwise, it seems that network booting takes higher priority than NVMe. I'll add more to this section to figure out how to make it boot from NVMe as higher precedence. also I will give my extlinux.conf file as an example, but not tonight.

fixing the time

Unfortunately, the hifive unmatched revb board does not support the RTC module in the kernel yet. From their official software manual

> There is no support for the RTC function on the HiFive Unmatched yet. Time information can be acquired from an Ethernet or WiFi network. If a user isn’t connected to a network where time information is available then some functions that depend on correct time may not work such as ssh.

https://www.sifive.com/boards/hifive-unmatched-revb

However, you can still fix the time by using ntp. I recommend not installing openntpd, because they have a max drift that is hard to force update the clock. so I would instead install net-misc/ntp

Useful links

compiling-device-trees

ac1300 kernel driver

u-boot-sifive-board-specific-docs

hifive-unmatched-board

hifive-unmatched-software-reference-manual

Compiling_with_QEMU_user_chroot

https://youtu.be/INWghYZH3hI?si=ECEsmj0Xu3kFGOGR