User:Fog Watch/A server on RAID dm-crypt and LXC

From Gentoo Wiki
Jump to:navigation Jump to:search

Introduction

This is a "how I did it", not a "how to". Accordingly it is written in first person.

The document describes a server built for home use.

Gentoo was chosen because maintenance is tremendous. Ext4 was chosen as its recent and current.

This latest incarnation of the server is a migration from vserver to LXC. The change was undertaken because one is not in the kernel, whereas the other is. Also networking under LXC is more flexible.

There doesn't seem much point in containerising the operating system without putting them in containers, so everything, including root, is put into a logical volume.

I don't want my data to go wandering through physical theft or disk RMA's. The physical volume is therefore a dm-crypt device. I don't use LUKS.

The dm-crypt is put onto a /dev/md1 RAID 1 device, just to add a little protection from disk failure. Both disks are partitioned about the same. Except, /dev/sda2 is used to store bootable ISOs and /dev/sdb2 is swap.

Layers

Gentoo will be installed on the following stack

  • Gentoo
  • LXC
  • ext4
  • LVM2 logical volumes
  • dm-crypt
  • RAID 1
  • GPT disk partitions.

GPT disk partitions

root #gdisk -l /dev/sda | tail -n 6
Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048          206847   100.0 MiB   EF00  EFI system partition
   2          206848         8595455   4.0 GiB     8300  ISOs
   3         8595456        16984063   4.0 GiB     8200  Swap
   4        16984064        17188863   100.0 MiB   FD00  Boot
   5        17188864       117231374   47.7 GiB    FD00  LVM Physical volume


RAID 1

root #mdadm --create --verbose /dev/md0 --level=mirror --raid-devices=2 /dev/sda4 missing
root #mdadm --create --verbose /dev/md1 --level=mirror --raid-devices=2 /dev/sda5 missing

dm-crypt

Create keyfile:

user $base64 /dev/random | head -c 1024 > /tmp/keyfile

If entropy is a problem and /dev/random is a little empty this will be too slow. I found success on a wireless access point running sys-apps/haveged.

root #cryptsetup create --cipher aes-xts-plain64 --key-file /mnt/floppy/keyfile cryptmd1 /dev/md1

LVM2 logical volumes

root #pvcreate /dev/mapper/cryptmd1

And then create the necessary volume group, logical volumes and filesystems.

Initramfs

An Initramfs is needed to establish the underpinnings of the root logical volume. The following is a fairly comprehensive list of commands that were used to construct an initial RAM file system.

user $mkdir initramfs-4.1.12-gentoo
user $cd initramfs-4.1.12-gentoo/
user $mkdir -p lib/modules lib/firmware/bnx2 root mnt/root bin dev sbin etc sys proc tmp usr/lib
user $cp -a /bin/busybox bin/
user $cd bin
user $ln -sf busybox sh
user $cd ..
user $cp -a /etc/mdadm.conf /containers/lxc-root/rootfs/root/initramfs-4.1.12-gentoo/etc/
FILE /root/initramfs-4.1.12-gentoo/init
#!/bin/busybox sh

#load_modules() {
#    MODULES="libphy.ko bnx2.ko md-mod.ko raid1.ko dm-crypt.ko"
#    MOD_PATH="lib/modules/"
#    for MODULE in ${MODULES} ; do
#        insmod -f ${MOD_PATH}/${MODULE}
#    done
#}

rescue_shell() {
    echo "Something went wrong. Dropping you to a shell."
    busybox --install -s
    exec /bin/sh
}

# Mount the /proc and /sys filesystems.
mount -t proc none /proc
mount -t sysfs none /sys

mount -t devtmpfs none /dev

#load_modules || rescue_shell #The kernel was made with the essential modules so that the initramfs does not need to be rebuilt every time with a new kernel.

mdadm -As

sleep 2

mount LABEL=Keyfile /mnt/usb  || rescue_shell #The label is an ext filesystem label, not the label of the partition.

cryptsetup create --cipher aes-xts-plain64 --key-file /mnt/usb/keyfile cryptmd1 /dev/md1

#sleep 2

lvm vgchange -ay  || rescue_shell
lvm vgscan --mknodes > /dev/null 2>&1

mount -o ro /dev/mapper/vg-root /mnt/root || rescue_shell

# Clean up.
umount /proc
umount /sys
umount /mnt/usb

# Boot the real thing.
exec switch_root /mnt/root /sbin/init || rescue_shell
FILE /root/initramfs-4.1.12-gentoo/initramfs.sh
find . -print0 | cpio --null -ov --format=newc | gzip -9 > /boot/initramfs-4.1.12-gentoo.img

The libraries to be copied into the initramfs vary over time. So the following is only an approximation, and only for 1 Jan 2016. 

ln -sf lib lib64
cd usr
ln -sf lib lib64

cp -a /sbin/cryptsetup sbin/

ldd sbin/cryptsetup #and copy in the necessary files.
cp -a /usr/lib64/libcryptsetup.so.4 usr/lib64/libcryptsetup.so.4
cp -a /usr/lib64/libpopt.so.0 usr/lib64/libpopt.so.0
cp -a /lib64/libc.so.6 lib64/libc.so.6
cp -a /lib64/libuuid.so.1 lib64/libuuid.so.1
cp -a /lib64/libdevmapper.so.1.02 lib64/libdevmapper.so.1.02
cp -a /usr/lib64/libgcrypt.so.20 usr/lib64/libgcrypt.so.20
cp -a /usr/lib64/libgpg-error.so.0 usr/lib64/libgpg-error.so.0
cp -a /lib64/ld-linux-x86-64.so.2 lib64/ld-linux-x86-64.so.2
cp -a /lib64/libudev.so.1 lib64/libudev.so.1
cp -a /lib64/libpthread.so.0 lib64/libpthread.so.0
cp -a /lib64/librt.so.1 lib64/librt.so.1
cp -a /lib64/libm.so.6 lib64/libm.so.6
cp -a /lib64/libresolv.so.2 lib64/libresolv.so.2
cp -a /lib64/libcap.so.2 lib64/libcap.so.2
cp -a /lib64/libattr.so.1 lib64/libattr.so.1
cp -a /lib/ld-2.21.so .
cp -a /lib/libattr.so.1.1.0 .
cp -a /lib/libcap.so.2.24 .
cp -a /lib/libc-2.21.so .
cp -a /lib/libm-2.21.so .
cp -a /lib/libpthread-2.21.so .
cp -a /lib/libresolv-2.21.so .
cp -a /lib/librt-2.21.so .
cp -a /lib/libudev.so.1.6.4 .
cp -a /lib/libuuid.so.1.3.0 .
cd ../usr/lib
cp -a /usr/lib/libcryptsetup.so.4.6.0 .
cp -a /usr/lib/libgcrypt.so.20.0.3 .
cp -a /usr/lib/libgpg-error.so.0.15.0 .
cp -a /usr/lib/libpopt.so.0.0.0 .

cp -a /sbin/lvm sbin/ # again ldd is used to find out which libraries are needed.
# previously I have still had to copy libraries in when built "static", so I don't bother with that any more.

cp -a /lib64/libudev.so.1 .
cp -a /lib64/libdl.so.2 .
cp -a /lib64/libblkid.so.1 .
cp -a /lib64/libdevmapper-event.so.1.02 .
cp -a /lib64/libdevmapper.so.1.02 .
cp -a /lib64/libreadline.so.6 .
cp -a /lib64/libc.so.6 .
cp -a /lib64/librt.so.1 .
cp -a /lib64/libm.so.6 .
cp -a /lib64/libresolv.so.2 .
cp -a /lib64/libcap.so.2 .
cp -a /lib64/libpthread.so.0 .
cp -a /lib64/ld-linux-x86-64.so.2 .
cp -a /lib64/libuuid.so.1 .
cp -a /lib64/libncurses.so.5 .
cp -a /lib64/libattr.so.1 .

cp -a /lib/libblkid.so.1.1.0 .
cp -a /lib/libdl-2.21.so .
cp -a /lib/libncurses.so.5.9 .
cp -a /lib/libreadline.so.6.3 .

cp -a /sbin/mdadm sbin/

cp -a /lib/libnss_files.so.2 lib
cp -a /lib/libnss_files-2.21.so lib

#cd sbin
#ln -sf ../bin/busybox mdev

#cd ../lib/modules/
#cp -a /lib/modules/4.1.12-gentoo/kernel/drivers/net/ethernet/broadcom/bnx2.ko .
#cp -a /lib/modules/4.1.12-gentoo/kernel/drivers/md/dm-crypt.ko .
#cp -a /lib/modules/4.1.12-gentoo/kernel/drivers/net/phy/libphy.ko .
#cp -a /lib/modules/4.1.12-gentoo/kernel/drivers/md/md-mod.ko .
#cp -a /lib/modules/4.1.12-gentoo/kernel/drivers/md/raid1.ko .

#cp -a /lib/firmware/bnx2/bnx2-rv2p-09-6.0.17.fw .
#cp -a /lib/firmware/bnx2/bnx2-mips-06-6.2.1.fw .
#cp -a /lib/firmware/bnx2/bnx2-mips-06-6.2.3.fw .
#cp -a /lib/firmware/bnx2/bnx2-rv2p-06-6.0.15.fw .
#cp -a /lib/firmware/bnx2/bnx2-rv2p-09ax-6.0.17.fw .
#cp -a /lib/firmware/bnx2/bnx2-mips-09-6.2.1a.fw .

LXC

Migrating from Vserver to LXC

When the kernel is patched with vserver, networking inside a container is unavailable. This makes migrating from one to the other difficult.

To migrate, first make an initial container from a tarball or such like. Then chroot into this and follow the handbook. The container is then only useful when the kernel is changed over to one that is unpatched.

System

Default system config:

FILE /etc/lxc/lxc.conf
lxc.lxcpath = /containers
lxc.default_config = /etc/lxc
lxc.bdev.lvm.vg = vg

Use the other options from the wiki.

root #mkdir /containers

Container

Privileged

All of the containers will be internally facing. Because of this and the additional problems with unprivileged containers, privileged will be used here.

Establish reasonable default container config:

FILE /etc/lxc/default.conf
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0.1
lxc.network.name = net0
lxc.start.auto = 1
root #lxc-create -B lvm -t gentoo -n test --lvname=test --fstype=ext4 --fssize=5G -- -a amd64

-f /etc/lxc/default.conf causes problems with bringing up the interface. Search /usr/share/lxc/templates/lxc-gentoo for usage to explore template parameters. Of particular interest might be --mirror and --tarball.

FILE tail -n 3 /containers/test2/config
lxc.include = /etc/lxc/default.conf
lxc.include = /containers/test2/net.conf
lxc.mount = /containers/test2/fstab
FILE /containers/test2/net.conf
lxc.network.ipv4 = 192.168.2.10/24
lxc.network.ipv4.gateway = 192.168.2.1
lxc.network.hwaddr = 12:b0:49:3b:c5:af

The full path for fstab is needed, otherwise the file is not found.

No, you don't have to mount /dev/vg/test2 or add a line to /etc/fstab

root #lxc-start -n test2

Unprivileged

Unprivileged are good for providing services to the Internet. Use the -t download of lxc-create. The suggested config is about right:

You don't need a users, but you may as well have one.

root #useradd -m -G users -s /bin/bash lxc

Comply with prerequisites.

root #echo "=sys-apps/shadow-4.2.1* ~amd64" >> /etc/portage/package.accept_keywords
root #touch /etc/{subuid,subgid}
root #usermod --add-subuids 100000-100100 lxc
root #usermod --add-subgids 100000-100100 lxc
root #echo "app-admin/cgmanager ~amd64" >> /etc/portage/package.accept_keywords
root #emerge -1av cgmanager
root #/etc/init.d/cgmanager start
root #rc-update add cgmanager default

Make

root #su - lxc
FILE ~/.config/lxc/lxc.conf
'"`UNIQ--pre-00000011-QINU`"'
FILE ~/.config/lxc/default.conf
'"`UNIQ--pre-00000014-QINU`"'
user $mkdir -p .config/lxc
user $mkdir -p .local/lxc
root #lvcreate -n template -L 5G vg
root #mkfs.ext4 /dev/vg/template
user $mkdir ~/.local/lxc/template
root #echo "/dev/vg/template /home/lxc/.local/lxc/template ext4 user,defaults,noatime,exec 0 1" >> /etc/fstab
user $mount .local/lxc/template/
root #chown 100000:lxc .local/lxc/template && chmod 770 .local/lxc/template/
user $lxc-create -l debug -P ~/.local/lxc/ -t download -n template -- -d gentoo -r current -a amd64

Stop all

Obtain lxc-stop-all.sh.

Autostart

Need to be able to initiate a user script at boot up from a shell. First, need a self-signed certificate:

user $ssh-keygen -t rsa
user $ssh-copy-id localhost

And then set up fcron to run a script.

user $fcrontab -l | grep lxc-autostart
@reboot ssh lxc@localhost /home/lxc/lxc-autostart.sh
FILE ~/lxc-autostart.sh
#!/bin/sh
sudo /usr/bin/cgm create all $USER
#sudo /usr/bin/cgm chown all $USER $(id -u) $(id -g)
sudo /usr/bin/cgm chown all $USER 1000 1000
cgm movepid all $USER $$
lxc-autostart
#cat /proc/self/cgroup #to check

Sudo can be tied down for this.

FILE /etc/sudoers.d/lxc
'"`UNIQ--pre-0000001B-QINU`"'
user $grep lxc.start.auto ~/.local/share/lxc/test3/config
lxc.start.auto = 1

Copy

It was not obvious how to clone an unprivileged container, so I used the following and manually tidied things like /etc/conf.d/hostname /home/lxc/.local/lxc/hf/config /home/lxc/.local/lxc/hf/local.conf

root #rsync -avx template/ hf/
Retrieved from "https://wiki.gentoo.org/index.php?title=User:Fog_Watch/A_server_on_RAID_dm-crypt_and_LXC&oldid=515404"