Project:Infrastructure/Servers/Susuwatari

From Gentoo Wiki
Jump to:navigation Jump to:search
Hostname susuwatari.arm.dev.gentoo.org
DNS alias(es)
Service(s) ARM64 Build
CPU TODO
RAM 128GB
Storage 2x MZQL2960HCJR-00A07 Samsung 960GB NVME
Owner Hetzner
Location Finland
DSA SSH key fingerprint (MD5 hash)
RSA SSH key fingerprint (MD5 hash)
ECDSA SSH key fingerprint (MD5 hash)
ED25519 SSH key fingerprint (MD5 hash)
Has IPv6 connectivity Yes
Is virtual machine No
Is in service Yes


Hetzner bootstrap

Hetzner installer bootstrap - first time, to see Hetzner's bootloader config and then pivot-install to Gentoo 

# cat >/tmp/install.conf <<EOF
DRIVE1 /dev/nvme0n1
SWRAID 0
SWRAIDLEVEL 1
HOSTNAME susuwatari.arm.dev.gentoo.org
USE_KERNEL_MODE_SETTING yes
LV susuwatari   root   /        ext4         50G
LV susuwatari   home   /home    ext4         50G
PART /boot/efi esp 256M
PART swap swap 4G
PART /boot ext3 1024M
PART lvm susuwatari all
IMAGE /root/.oldroot/nfs/install/../images/Ubuntu-2404-noble-arm64-base.tar.gz
EOF
# installimage -a -c /tmp/install.conf 
# sfdisk -d /dev/nvme0n1 |sed 's/, uuid=.*//g;/label-id/d'  |sfdisk /dev/nvme1n1
# vgextend susuwatari /dev/nvme1n1p4
# sysctl -w dev.raid.speed_limit_max=20000000
# lvconvert -y --type raid1 susuwatari/root
# lvconvert -y --type raid1 susuwatari/home

Bugs in the hetzner bootstrap

  • Extra swap is created: LVM $VGNAME/swap
    • Wait until after boot to fix it


Notes:

  • Need to verify what RAID1 EFI & /boot works on the ARM64 UEFI.
  • Bootloader config is slightly different than Hetzner AMD64
    • AMD64 Hetzner strictly boots the /boot/efi/EFI/boot/bootx64.efi file
  • Capture of the boot setup, with redaction of MAC & UUID
# efibootmgr 
BootCurrent: 0004
Timeout: 1 seconds
BootOrder: 0002,0003,0004,0001
Boot0001  UEFI: Built-in EFI Shell	VenMedia(5023b95c-db26-429b-a648-bd4766xxxxxx)0000424f
Boot0002* UEFI: PXE IPv4 Intel(R) Network D8:5E:D3:xx:xx:xx	PcieRoot(0x50000)/Pci(0x5,0x0)/Pci(0x0,0x0)/MAC(d85ed3xxxxxx,1)/IPv4(0.0.0.00.0.0.0,0,0)0000424f
Boot0003* UEFI: PXE IPv4 Intel(R) Network D8:5E:D3:xx:xx:xx	PcieRoot(0x50000)/Pci(0x5,0x0)/Pci(0x0,0x1)/MAC(d85ed3xxxxxx,1)/IPv4(0.0.0.00.0.0.0,0,0)0000424f
Boot0004* ubuntu	HD(1,GPT,35151a19-f2ec-4386-8112-85d62cxxxxxx,0x1000,0x80000)/File(\EFI\ubuntu\grubaa64.efi)0000424f

# find /boot/efi/EFI/ -type f -ls
       12    148 -rwx------   1 root     root       151552 Jul 31 08:01 /boot/efi/EFI/BOOT/BOOTAA64.EFI
       14    148 -rwx------   1 root     root       151552 Jul 31 08:01 /boot/efi/EFI/ubuntu/grubaa64.efi

# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-6.8.0-38-generic root=/dev/mapper/susuwatari-root ro consoleblank=0 systemd.show_status=true consoleblank=0 console=ttyAMA0 console=tty0

# sed '/\/vmlinu/{ s/^[[:space:]]*//g; p}' -n /boot/grub/grub.cfg | uniq
linux	/vmlinuz-6.8.0-38-generic root=/dev/mapper/susuwatari-root ro consoleblank=0 systemd.show_status=true consoleblank=0 console=ttyAMA0 console=tty0
linux	/vmlinuz-6.8.0-38-generic root=/dev/mapper/susuwatari-root ro recovery nomodeset consoleblank=0 systemd.show_status=true

/etc/default/grub 

GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`( . /etc/os-release; echo ${NAME:-Ubuntu} ) 2>/dev/null || echo Ubuntu`
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX="consoleblank=0 systemd.show_status=true"
GRUB_TERMINAL=console

Hetzner files:
<pre>
# /etc/modprobe.d/blacklist-hetzner.conf
### Hetzner Online GmbH - installimage
### unwanted kernel modules
blacklist pcspkr
blacklist snd_pcsp
### buggy kernel modules
blacklist mei
blacklist mei-me
blacklist sm750fb

# /etc/modprobe.d/hetzner.conf
### Hetzner Online GmbH - installimage
options drm edid_firmware=edid/1280x1024.bin

# /etc/sysctl.d/99-hetzner.conf
### Hetzner Online GmbH installimage
# sysctl config
#net.ipv4.ip_forward=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.icmp_echo_ignore_broadcasts=1
# ipv6 settings (no autoconfiguration)
net.ipv6.conf.default.autoconf=0
net.ipv6.conf.default.accept_dad=0
net.ipv6.conf.default.accept_ra=0
net.ipv6.conf.default.accept_ra_defrtr=0
net.ipv6.conf.default.accept_ra_rtr_pref=0
net.ipv6.conf.default.accept_ra_pinfo=0
net.ipv6.conf.default.accept_source_route=0
net.ipv6.conf.default.accept_redirects=0
net.ipv6.conf.all.autoconf=0
net.ipv6.conf.all.accept_dad=0
net.ipv6.conf.all.accept_ra=0
net.ipv6.conf.all.accept_ra_defrtr=0
net.ipv6.conf.all.accept_ra_rtr_pref=0
net.ipv6.conf.all.accept_ra_pinfo=0
net.ipv6.conf.all.accept_source_route=0
net.ipv6.conf.all.accept_redirects=0

# /etc/apt/sources.list.d/hetzner-mirror.sources
X-Repolib-Name: hetzner-mirror
Types: deb
URIs: http://mirror.hetzner.com/ubuntu-ports/packages
Suites: noble noble-updates noble-backports noble-security
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

# /etc/apt/apt.conf.d/99hetzner
Acquire::PDiffs "false";
Acquire::Languages "en";

# /etc/systemd/timesyncd.conf.d/hetzner.conf
[Time]
NTP=ntp1.hetzner.de ntp2.hetzner.com ntp3.hetzner.net 

# /etc/systemd/system/mdcheck_start.timer.d/hetzner.conf
[Timer]
OnCalendar=
OnCalendar=*-*-26 02:14:00
# /etc/default/grub.d/hetzner.cfg
GRUB_HIDDEN_TIMEOUT_QUIET=false
GRUB_CMDLINE_LINUX_DEFAULT="consoleblank=0 console=ttyAMA0 console=tty0"

# only use text mode - other modes may scramble screen
GRUB_GFXPAYLOAD_LINUX="text"
  • Converting /boot to RAID1
# rsync -a --one-file-system /boot/ /boot.ubuntu/
# umount /boot/efi
# umount /boot
(TODO: show how to convert p3 to RAID partition type; be sure to change uuid & label-id)
# mdadm --create /dev/md1 -e 1.0 /dev/nvme0n1p3 /dev/nvme1n1p3 -l 1 -n 2
# dd if=/dev/zero bs=1M of=/dev/md1
# uuid=$(grep '/boot ' /etc/fstab  |cut -f1 -d' ' |cut -d= -f2)
# mkfs.ext3 /dev/md1 -U $uuid -L /boot
# mount /boot
# rsync -a --one-file-system /boot.ubuntu/ /boot/
# mkdir -p /boot/efi.nvme0 /boot/efi.nvme1
# mkfs.vfat /dev/nvme1n1p1
# adjust fstab:
UUID=359A-6CDB /boot/efi.nvme0 vfat umask=0077 0 1
UUID=C849-23A5 /boot/efi.nvme1 vfat umask=0077 0 1
# mount /dev/efi.nvme0
# mount /dev/efi.nvme1
# dpkg-reconfigure grub-efi-arm64
 - select both devices
 - no nvram

# efibootmgr  -o 0002,0003,0000,0004,0001
BootCurrent: 0004
Timeout: 1 seconds
BootOrder: 0002,0003,0000,0004,0001
Boot0000* Ubuntu	HD(1,GPT,247450c3-4d52-497d-abb6-373b52160734,0x1000,0x80000)/File(\EFI\ubuntu\grubaa64.efi)
Boot0001  UEFI: Built-in EFI Shell	VenMedia(5023b95c-db26-429b-a648-bd47664c8012)0000424f
Boot0002* UEFI: PXE IPv4 Intel(R) Network D8:5E:D3:E5:B4:4A	PcieRoot(0x50000)/Pci(0x5,0x0)/Pci(0x0,0x0)/MAC(d85ed3e5b44a,1)/IPv4(0.0.0.00.0.0.0,0,0)0000424f
Boot0003* UEFI: PXE IPv4 Intel(R) Network D8:5E:D3:E5:B4:4B	PcieRoot(0x50000)/Pci(0x5,0x0)/Pci(0x0,0x1)/MAC(d85ed3e5b44b,1)/IPv4(0.0.0.00.0.0.0,0,0)0000424f
Boot0004* Ubuntu	HD(1,GPT,35151a19-f2ec-4386-8112-85d62c3db7b6,0x1000,0x80000)/File(\EFI\ubuntu\grubaa64.efi)


(TODO)
# Show how to rebuild the initramfs w/ RAID
Retrieved from "https://wiki.gentoo.org/index.php?title=Project:Infrastructure/Servers/Susuwatari&oldid=1306391"