User:Fearedbliss/Installing Gentoo Linux On ZFS
Install Gentoo Linux on OpenZFS
Author: Jonathan Vasquez (fearedbliss)
Last Updated: 2022-10-13-2145
Preface
This guide will show you how to install Gentoo Linux on x86_64 with:
* UEFI-GPT (EFI System Partition - Unencrypted FAT32 partition as per UEFI Spec) * /boot on ZFS (Featureless & Unencrypted) * /, /home on ZFS (Encrypted ZFS if desired) * swap on a regular partition * OpenZFS 2.1.4 * GRUB 2.04+ * OpenRC (or systemd) * Gentoo Stable (x86_64)
Required Tools
You will need an ISO that contains OpenZFS. Luckily, the Gentoo Admin CD provides the needed packages. You can download the "Admin CD" from the Gentoo Downloads page. After that, we'll use it to make a bootable USB.
Linux
For the following commands, we will assume that your USB is /dev/sdg.
Format the USB
root #dd if=admincd-amd64-20220612T170541Z_OpenZFS_2.1.4.iso of=/dev/sdg bs=1M status=progress
root #syncWindows
Rufus is the USB Utility I recommend when on Windows. You can Download Rufus here.
- Start Rufus
- Select your USB Device from the Device drop down.
- Select your ISO by clicking SELECT.
- Partition Scheme: MBR
- Target system: BIOS or UEFI
- Volume label: ADMINCD
- File system: FAT32
- Cluster size: 4096 bytes (Default)
- Click START.
This should be all that's necessary to have a bootable USB.
Assumptions
- Only installing Gentoo on a single drive (Multiple drives in the same pool should automatically work).
- The /boot pool is featureless and unencrypted.
- The /boot/efi is an unencrypted FAT32 partition as per UEFI spec.
- Your swap partition is outside of ZFS and on a dedicated partition.
- GRUB 2.04+ is being used
- Gentoo Admin CD (Contains OpenZFS)
- Kernel: gentoo-kernel-bin
- Initramfs: bliss-initramfs
- Init: OpenRC
You are free to substitute any of the above for whatever you want. However, support will
only be provided from me when the above configuration is used. Also, this guide is the way
I install Gentoo, not exactly the way the handbook has it.
Boot your system into the ISO
Since this is highly computer dependent, you will need to figure out how to boot your USB on your system and get to the live environment. You may need to disable Secure Boot if that causes your USB to be rejected. Make sure your system BIOS/UEFI is set up to boot UEFI devices, rather than BIOS devices (Legacy).
Confirm that you booted in UEFI Mode
After you boot into the Live CD, make sure that you booted into UEFI mode by typing the following:
root #ls /sys/firmware/efiIf the above directory is empty or doesn't exist, you are not in UEFI mode. Reboot and boot into UEFI mode.
Continuing the installation without being in UEFI mode will most likely yield an unbootable system. If you want to install in BIOS mode, you will need a different setup.
Partition
We will now partition the drive and aim to create the following layout (for the rest of the guide we will assume /dev/nvme0n1 is your main drive):
/dev/nvme0n1p1 | 512 MiB | EFI System Partition | /boot/efi /dev/nvme0n1p2 | 1024 MiB | Boot Partition (ZFS, No Feature Flags) | /boot /dev/nvme0n1p3 | 2048 MiB | swap | swap /dev/nvme0n1p4 | Rest of Disk | ZFS (or Encrypted ZFS) | /, /home, ...
There are many UEFI motherboard firmwares that are extremely buggy. We will attempt to use a 512 MiB FAT32 partition configuration to increase success.
Open up your drive in GNU parted and tell it to use optimal alignment:
root #parted -a optimal /dev/nvme0n1Keep in mind that all of the following operations will affect the disk immediately. GNU parted does not stage changes like fdisk or gdisk.
Set the units to Mebibytes
(parted)unit mibCreate GPT partition layout
This will delete all partitions and create a new GPT table.
(parted)mklabel gptCreate and label your partitions
(parted)mkpart esp 1 513
(parted)mkpart boot 513 1537
(parted)mkpart swap 1537 3585
(parted)mkpart rootfs 3585 -1
Set the bootable flag on the ESP partition
(parted)set 1 boot onFinal View
(parted)printModel: ATA VBOX HARDDISK (scsi) Disk /dev/nvme0n1: 40960MiB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1.00MiB 513MiB 512MiB primary 2 513MiB 1537MiB 1024MiB boot 3 1537MiB 3585MiB 2048MiB swap 4 3585MiB 40959MiB 37374MiB rootfs
Exit the application
(parted)quitFormat your drives
Create your boot partition
root #mkfs.fat -F32 /dev/nvme0n1p1This partition needs to be FAT32 due to it being an UEFI requirement. If it isn't, your system will not boot!
Create your zpool
Create your zpool which will contain your drives and datasets:
root #zpool create -f -o ashift=12 -o cachefile= -O compression=lz4 -O atime=off -m none -R /mnt/gentoo tank /dev/nvme0n1p4If you want to use native zfs encryption (with a passphrase), simply specify the -O encryption=on -O keyformat=passphrase options as well.
Create your zfs datasets
We will keep it simple and just create a few datasets for /, and /home. ZFS is extremely flexible and you can easily add or remove datasets in the future.
root #zfs create tank/os
root #zfs create -o mountpoint=/ tank/os/main
root #zfs create -o mountpoint=/home tank/home
Create your boot pool
It is safer for us to create a separate zpool that has all feature flags disabled. This is because even though grub currently supports the latest feature flags for the 0.6.5 release of ZFS, new feature flags added to ZFS without proper bootloader support can make your system unbootable. Since GRUB 2 has solid support for a zpool with no feature flags, we will create a separate featureless boot pool. However, our main system will still have all features enabled.
root #zpool create -f -d -o ashift=12 -o cachefile= -m /boot -R /mnt/gentoo boot /dev/nvme0n1p2Never do a zpool upgrade or enable encryption on your boot pool! Doing this will leave your system unbootable.
Create your swap
root #mkswap -f /dev/nvme0n1p3
root #swapon /dev/nvme0n1p3
* Do not put your swap inside ZFS in any capacity (dataset nor zvol). I've experienced lockups when my RAM is 100% and the system starts swapping while the swap is on ZFS. It doesn't crash when the swap is on a normal partition.
Verify everything looks good
You can verify that all of these things worked by running the following:
root #zpool status pool: boot
state: ONLINE
status: The pool is formatted using a legacy on-disk format. The pool can
still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done, the
pool will no longer be accessible on software that does not support
feature flags.
scan: none requested
config:
NAME STATE READ WRITE CKSUM
boot ONLINE 0 0 0
nvme0n1p2 ONLINE 0 0 0
errors: No known data errors
pool: tank
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
nvme0n1p4 ONLINE 0 0 0
errors: No known data errors
root #zfs listNAME USED AVAIL REFER MOUNTPOINT boot 36.2M 742M 35.1M /boot tank 315G 402G 192K none tank/home 320K 402G 320K /home tank/os 310G 402G 310G / tank/os/main 310G 402G 310G /
Now we are ready to install Gentoo!
Installing Gentoo
Set your date and time
Let's say it's June 18, 2022 @ 11:48 PM (will be 23:48 in 24 hour time), we would do the following:
root #date 06182348202206 18 2348 2022 (Month, Date, 24 Hour Time, Year). It will be UTC at the moment but you can fix it once you reboot.
Preparing to chroot
First let's mount our EFI boot partition in our chroot directory:
root #cd /mnt/gentoo
root #mkdir boot/efi
root #mount /dev/nvme0n1p1 boot/efi
and download the OpenRC (or systemd) amd64 image here and extract it:
root #wget <file>
root #tar xpf <file>
You can also find a mirrored copy of the admin cd on my server: https://xyinn.org/gentoo/livecd/.
Copy zpool cache
root #mkdir etc/zfs
root #cp /etc/zfs/zpool.cache etc/zfs
Copy network settings
root #cp /etc/resolv.conf etc/Mount required devices
root #mount --rbind /dev dev
root #mount --rbind /proc proc
root #mount --rbind /sys sys
root #mount --make-rslave dev
root #mount --make-rslave proc
root #mount --make-rslave sys
Chroot into your environment
root #env -i HOME=/root TERM=$TERM chroot . bash -lEdit fstab
Everything is on zfs so we don't need anything in here except for the efi directory and swap entries. My fstab looks as follows:
root #nano /etc/fstab/dev/nvme0n1p1 /boot/efi vfat noauto 1 2 /dev/nvme0n1p3 none swap sw 0 0
Do not auto-mount the /boot/efi partition. If you do, the init system will create the /boot/efi folder if it doesn't exist, and that will prevent the boot zpool from mounting.
Modify make.conf
Let's modify our /etc/portage/make.conf so we can start installing stuff with a good base (Change it to what you need):
root #nano /etc/portage/make.confUSE="-branding" # This should be your number of processors + 1 MAKEOPTS="-j5" EMERGE_DEFAULT_OPTS="--with-bdeps=y --keep-going=y --quiet-build=y" FEATURES="buildpkg" LINGUAS="en en_US" # This is required so that when we compile GRUB later, EFI support is built. GRUB_PLATFORMS="efi-64"
Get the portage tree
Copy the default example portage config
root #cp /usr/share/portage/config/repos.conf /etc/portage/repos.conf
root #emerge --sync
Kernel Installation
For simplicity, we will just use the prebuilt gentoo kernel.
Disable 'initramfs' USE flag
Since we are using bliss-initramfs, we will disable the 'initramfs' USE flag on gentoo-kernel-bin.
root #echo "sys-kernel/gentoo-kernel-bin -initramfs" >> /etc/portage/package.use/gentoo-kernel-binInstall the kernel
root #emerge gentoo-kernel-binConsider pinning the kernel version that was installed so that when we do a future kernel upgrade, an emerge --depclean won't delete our kernel files from the /boot directory. If the gentoo-kernel ebuild that emerge pulled to install was: gentoo-kernel-bin-5.15.48, we would do:
root #emerge --noreplace =gentoo-kernel-bin-5.15.48Install required applications
Enable ZFS support in GRUB
root #echo "sys-boot/grub libzfs" >> /etc/portage/package.useNow install the apps:
root #emerge bliss-initramfs grub zfsInstalling the bootloader onto your drive
We will need to install the bootloader onto the drive. Before we do that however, let's see if GRUB can detect our /boot and /boot/efi filesystem types:
root #grub-probe /bootThis should say 'zfs'. If it doesn't, then something is wrong and your system will not boot!
root #grub-probe /boot/efiThis should say 'fat'. If it doesn't, then something is wrong and your system will not boot!
Before we install the bootloader, we will need to have read/write access to the efi nvram variables. Let's remount our efivars now:
root #mount -o remount,rw /sys/firmware/efi/efivars/Now run the following to install the bootloader to the drive:
root #grub-install --efi-directory=/boot/efiThe above command will install the grub bootloader files into /boot and the efi files into /boot/efi. It should return a "Installation finished. No error reported." message. If it doesn't, then something is wrong and your system will not boot!
If after a reboot your UEFI motherboard doesn't detect your bootloader, it might be because of a buggy motherboard firmware. Try to pass the "--removable" flag to grub install and try again. This will install the efi file to the EFI specification's fallback directory.
Make the GRUB 2 configuration file
You can use the following configuration file as a basis for your system:
root #nano /boot/grub/grub.cfg
set default=0
set timeout=1
insmod part_gpt
insmod fat
insmod efi_gop
insmod efi_uga
menuentry "Gentoo - 5.15.48-gentoo-dist" {
linux /@/vmlinuz-5.15.48-gentoo-dist root=tank/os/main by=id elevator=noop quiet logo.nologo refresh
initrd /@/initrd-5.15.48-gentoo-dist
}
If you are using native ZFS encryption, add the encrypted flag to your kernel line as well.
Generating new zpool.cache file before/after reboot
ZFS is very sensitive about the data that is contained in the zpool.cache file and at this point, when we reboot, the information in it might not be completely accurate. To ensure we have a good cache file, we have instructed bliss-initramfs in the bootloader config above, to ignore the current cachefile on the system, and make a new one that is up-to-date. We only need to do this once.
Generate the initramfs and move the file to its correct location
Before generating the initramfs, make sure to edit the /etc/bliss-initramfs/settings.json and include any kernel modules needed to boot your system. In this guide, we will need to load the nvme driver since gentoo-kernel-bin has this driver compiled as a module. If your system fails too boot, it may be related to a missing kernel module not being loaded. Follow the instructions in the recovery section for getting back into your environment.
root #nano /etc/bliss-initramfs/settings.json
"modules": {
"files": ["nvme"]
},Now generate your initramfs and move it to the correct location:
root #bliss-initramfs -k 5.15.48-gentoo-dist
root #mv initrd-5.15.48-gentoo-dist /boot
Final steps before reboot
OpenRC
root #rc-config add zfs-import boot
root #rc-config add zfs-mount boot
systemd
root #systemctl enable zfs.target
root #systemctl enable zfs-import-cache
root #systemctl enable zfs-mount
root #systemctl enable zfs-import.target
And now for the magic commands (cross fingers lol):
root #passwd
root #exit
root #reboot
After you reboot
Remove zpool.cache refresh flag from bootloader configuration
Open up your grub.cfg and remove the refresh flag from the kernel line.
root #nano /boot/grub/grub.cfgImport your boot pool
You can mount your boot pool now so that your /boot directory will be available on subsequent reboots. The boot pool will be automatically remembered through the cachefile:
root #zpool import -o cachefile= boot
Take a snapshot of your new system
Since we now have a working system, we will snapshot it in case we ever want to go back or recover files:
root #zfs snapshot boot@2022-06-18-2241-55-INSTALL
root #zfs snapshot -r tank@2022-06-18-2241-55-INSTALL
You can view the contents of these snapshots by checking their respective and hidden .zfs directories:
root #ls /boot/.zfs/snapshots
root #ls /.zfs/snapshots
root #ls /home/.zfs/snapshots
We are naming the snapshots in this format because it's the format that my 'bliss-zfs-scripts' use, and it is also very readable.
Limiting the ARC size
If you want to cap the ZFS ARC from growing past a certain point, you can put the number of bytes inside the /etc/modprobe.d/zfs.conf file, and then remake your initramfs. When the system starts up, and the module is loaded, these options will be passed to the zfs kernel module. (Temporary) Change the ARC max for the running system to 4 GB
root #echo 4294967296 >> /sys/module/zfs/parameters/zfs_arc_max(Permanent) Save the 4 GB ARC cap as a loadable kernel parameter
root #echo "options zfs zfs_arc_max=4294967296" >> /etc/modprobe.d/zfs.confOnce we have the above file created, let's regenerate the initramfs. bliss-initramfs will automatically detect that this file exists and copy it into the initramfs. When you reboot your machine, the initramfs will load up the zfs kernel module with the parameters found in the file.
root #bliss-initramfs -k 5.15.48-gentoo-dist
root #mv initrd-5.15.48-gentoo-dist /boot
Recovery
If you need to get back into your zpool from a livecd environment, you'll simply need to import your pool again using the following commands:
root #zpool import -R /mnt/gentoo tankroot #zpool import -R /mnt/gentoo bootIf your pool is encrypted, you will also want to either pass the -l (lower case L) option your the zpool import command above (Which will ask you for the pool's passphrase and thus load the keys), or you can use the following command after you imported the pool without keys:
root #zpool load-key tankThis will also ask you for the passphrase and load the key for the pool. Once you do that, you should be able to see the available value for your encrypted pool by doing:
root #zfs get keystatus tankYou'll also probably want to import your boot pool and your efi partition as well:
root #zpool import -R /mnt/gentoo boot
root #mount /dev/nvme0n1p1 /mnt/gentoo/boot/efi
{{Note|If you have trouble booting your pool upon reboot, try adding the refresh flag during boot.}
Other stuff you might want to install
Bliss ZFS Scripts
The following scripts allow you to automatically: * Take snapshots of your pool * Replicate the pool to another pool (Full and Incremental Backups) * Clean the old snapshots on your pools. You can download, customize, and install the scripts into your /usr/local/sbin directory. Github. And that's it. Enjoy!