User:NeddySeagoon/Pi3 Build Root
Building For Memory Constrained Systems
There are several ways to build for a memory constrained system like the Raspberry Pi.
* Build locally with a large swap space * Build locally with the help of cross distcc. * Build in an QEMU chroot on a completely different architecture. * Build in an QEMU chroot on a completely different architecture with the aid of cross distcc.
The first has the advantage of requiring no special setup.
The second speeds thing up considerably as building and perhaps preprocessing, are delegated to a more powerful system. The Memory Constrained system still has to do the linking and there are limits to how much swap can be used.
The third is slower as the target CPU is emulated in software by QEMU but it can use all the hosts CPU(s) and RAM. Its also possible to add cross distcc to the chroot to call the coss compiler on the chroot host or other systems.
A QEMU chroot is not a silver bullet. Not all syscalls are implemented, some missing will be tolerated others must be avoided.
Motivation
The driver to do this was Firefox-54. Firefox-54 has a hard dependency on rust. Rust comes with a bundled LLVM which adds to the burden. Rust has its own package manager, cargo, which needs rust to build.
A Raspberry Pi 3 with 1GB RAM will not build rust. No rust, no cargo, no Firefox-54 or later on a Raspberry Pi 3 or other similar aarch64 systems.
Aarch64 Metrics - Using glibc
glibc was choosen as a benchmark as it supports parallel builds well.
It appears that the chroot is about half the speed of the real hardware when its hosted on a AMD Phenom II X6 1090T.
AMD Phenom(tm) II X6 1090T Processor Six Core Native Build 3.2GHz
As a reference point. This is as good as it gets. Its both the qemu chroot host and the cross distcc helper in these benchmarks.
emerge -av1 =sys-libs/glibc-2.24-r2
Sat Jun 17 17:57:13 2017 >>> sys-libs/glibc-2.24-r2
merge time: 4 minutes and 35 seconds.
qemu chroot
emerge -av1 =sys-libs/glibc-2.24-r2
Sat Jun 17 18:49:04 2017 >>> sys-libs/glibc-2.24-r2
merge time: 47 minutes and 49 seconds.
The major difference from the above is the arm64 emulation.
qemu chroot with cross distcc
The compiles are pushed over the network to 127.0.0.2, so they stand out inDISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-text 5
Preprocessing is still carried out in the qemu chroot. In exchange for native compiling, the chroot has to drive the network.
FEATURES=distcc emerge -av1 =sys-libs/glibc-2.24-r2
Sat Jun 17 19:33:51 2017 >>> sys-libs/glibc-2.24-r2
merge time: 38 minutes and 28 seconds.
qemu chroot with cross distcc and pump mode
FEATURES="distcc distcc-pump" emerge -av1 =sys-libs/glibc-2.24-r2
Sat Jun 17 20:51:05 2017 >>> sys-libs/glibc-2.24-r2
merge time: 37 minutes and 54 seconds.
Still with a lot of localhost preprocessing
Raspberry Pi 4 Cores at 1.2GHz
FEATURES="-distcc -distcc-pump" MAKEOPTS="-j5" emerge glibc -1av
Mon Jun 19 14:00:54 2017 >>> sys-libs/glibc-2.24-r2
merge time: 32 minutes and 18 seconds.
Raspberry Pi 4 Cores at 1.2GHz with cross distcc
FEATURES="distcc -distcc-pump" MAKEOPTS="-j5" emerge glibc -1
Mon Jun 19 14:43:47 2017 >>> sys-libs/glibc-2.24-r2
merge time: 24 minutes and 55 seconds.
and with MAKEOPTS="-j8" as that's what the helper normally runs at.
Mon Jun 19 15:49:45 2017 >>> sys-libs/glibc-2.24-r2
merge time: 23 minutes and 49 seconds.
Raspberry Pi 4 Cores at 1.2GHz with pump mode cross distcc
Now the helpers can help with preprocessing at the expense of the Pi running compression for the network link.
FEATURES="distcc distcc-pump" MAKEOPTS="-j8" emerge glibc -1
Mon Jun 19 16:16:06 2017 >>> sys-libs/glibc-2.24-r2
merge time: 23 minutes and 45 seconds.
No preprocessing was observed to be sent to the helper. That probably indicates a problem on the Pi.
Setting Up An Aarch64 Chroot For a Pi 3
The chroot target directory is /usr/aarch64-unknown-linux-gnu/ which was provided by crossdev when the cross toolchain was built.
crossdev -t aarch64-unknown-linux-gnu
See the Raspberry_Pi_3_64_bit_Install guide. Unless some pure cross compiling has been performed, its almost empty. Fix the /usr/aarch64-unknown-linux-gnu/etc/portage/make.profile symlink. The default setting of embedded is not useful.
The chroot needs to be populated with arm64 packages, there are two possible sources.
- Use existing aarch64 packages from outside the chroot, e.g. from your arm64 install.
- Using the arm64 stage3 tarball
Using Existing Aarch64 Packages From Outside The Chroot
Binary packages have been saved to /packages on your existing arm64 install. If not, you can't use this method.
Packaging the entire install with quickpkg is possible but is left as an exercise for the reader.
Copy the existing world file /var/lib/portage/world to /usr/aarch64-unknown-linux-gnu/var/lib/portage/world
Copy the existing arm64 /etc/portage/ to /usr/aarch64-unknown-linux-gnu/etc/portage/ EXCEPT make.conf
The existing /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf has some cross compile settings that will be needed.
Merge the existing arm64 USE, INPUT_DEVICES, VIDEO_CARDS settings and so on, into /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf
Mount the existing Aarch64 packages directory to /usr/aarch64-unknown-linux-gnu/packages
emerge-wrapper -t aarch64-unknown-linux-gnu --init aarch64-unknown-linux-gnu-emerge -KuDNav @world
Will offer to bring @world up to date using the binaries. Review the output for any changed USE flags and so on. Adjust /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf as required.
Answer y when it looks good.
The command will fail if the dependency tree cannot be satisfied with the provided binaries. Either provide the binaries or change the -K to -k, which will cause portage to attempt to cross compile any required packages.
Reusing the world file
Users missing the binary packages can still start with the stage3 tarball and their world file.
Using the arm64 stage3 tarball
Users that are sill waiting for their aarch64 system to arrive can populate /usr/aarch64-unknown-linux-gnu/ with the current arm64 stage3 tarball and do a handbook install. The Handbook:AMD64 will serve as a guide as there is no arm64 handbook yet.
Do follow the steps for your host kernel and binfmt later in this guide, or
chroot /usr/aarch64-unknown-linux-gnu/ /bin/bash --login will fail.
Host Kernel
See Embedded Handbook/General/Compiling with qemu user chroot
<*> Kernel support for MISC binaries
Is required. This setting has a <M> option too. However, switching the option on changes other parts of the kernel, so a full kernel build, install and reboot is required.
Static QEMU
QEMU will emulate the target processor in software. Its going to be installed in /usr/aarch64-unknown-linux-gnu/. It is the only program in /usr/aarch64-unknown-linux-gnu/ that will actually run on the host. As everything else in /usr/aarch64-unknown-linux-gnu/ is built for the target CPU, QEMU cannot call any librares. It must be self contained.
This means that its a bad idea to use the chroot to emerge qemu. Add qemu to package.mask inside the chroot.
Building and Installing the Static QEMU
See Embedded Handbook/General/Compiling with qemu user chroot
binfmt service
The binfmt service allows the host to direct forigen binaries to QEMU. Its provided by sys-apps/openrc. Systemd users need to do their homework.
Either add it to the default runlevel or /etc/init.d/binfmt start as required.
Preparaing to Chroot
This is best done with a script, since it needs to be performed every time the build root is used. It would be possible to add the mounts to the hosts /etc/fstab too.
#!/bin/bash # run this script to prepare the arm64 chroot # for QEMU building # start the binfmt service on the host # harmless warning if its already running /etc/init.d/binfmt start cd /usr/aarch64-unknown-linux-gnu/ mount --bind /usr/portage usr/portage mount --bind /usr/portage/distfiles usr/portage/distfiles # not /usr/portage/packages as they are for the host! mount --bind /proc proc mount --bind /sys sys mount --bind /dev dev mount --bind /dev/pts dev/pts # must be after the bind mount of dev mount tmpfs -t tmpfs -o rw,nosuid,nodev,noexec dev/shm # don't actually do the chroot
Do not mount the Hosts packages directory. It should be safe ...
USE Flags
app-misc/pax-utils must be built with USE=-seccomp for use inside the chroot.
There will be an unimplemented sysctl 277 error at package install time when using the chroot for builds.
This will prevent packages being installed
app-misc/pax-utils will cross compile, so its no hardship to fix.
Ready Made Raspberry Pi 3 Aarch64 Stage4 chroot
Download from NeddySeagoons Pi3 64 Bit stuff.
Be sure to read the README_chroot since getting it wrong will ruin your whole day.
Its just over 4G on my filesystem.
No kernel is required as this is a chroot.
The chroot host still needs kernel support and the binfmt service started.
So far, its untested since the upload. If it works for you, feedback would be appreciated if it fails, I would like to fix it.
This approach has the advantage of including rust and cargo already built.
Note: qemu should be masked in this chroot but isn't. Updating qemu will overwrite the static qemu built for the host.
Chroot
chroot /usr/aarch64-unknown-linux-gnu/ /bin/bash --login source /etc/profile export PS1="aarch64 $PS1"
If all is well, your host is running an arm64 /bin/bash with QEMU emulating an arm64. If not, you got an error message from the chroot command.
In some cases the aarch64 emulation needs to be registered with the kernel, its one single long command.
echo ':aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-aarch64:' > /proc/sys/fs/binfmt_misc/register
Loose Ends
Cross Distcc
Install distcc into the aarch64 chroot in the normal way.
In the /etc/distcc/hosts file list a helper at 127.0.0.2/8,cpp,lzo
The /8 is send up to 8 jobs. cpp allows pump mode and lzo is the compression to be used with pump mode.
On the host, distcc needs to --allow 127.0.0.0/8
The 127.0.0.2 address is used so that its clear in distcc-mon, what is being compiled where.
Add any more helpers you may have too.
Add distcc to FEATURES in make.conf.
DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-text 5 11249 Compile gconv_simple.c 127.0.0.2[5] 11337 Compile gconv.c 127.0.0.2[6] 11330 Compile gconv_open.c 127.0.0.2[7] 11374 Preprocess localhost[0] 11386 Preprocess localhost[2] 11391 Preprocess localhost[3]
Objects processed by 127.0.0.2 use the cross compiler which runs natively on the host but emits code for arm64. This avoids the QEMU emulation overhead for gcc.
Cross Distcc with Pump Mode
Set up cross distcc as above,
Add distcc-pump to FEATURES in make.conf.
Xorg
The intersection of the set of GPUs that appear on aarch64 systems and the set which appear on AMD/Intel systems is the empty set. Most graphics cards in AMD/Intel systems provide a BIOS ROM and require that code from that ROM is executed to initialise the card. Aarch64 systems cannot execute that code.
It looks line an interesting challenge. To run Xorg from an emulated aarch64 system executing on an AMD/Intel platform.
Firefox >=54.0
First we need rust.
Rust
Unfortunately rust needs rust to build. That's just like icedtea and gcc. To make that work the ebuild must be patched. The aarch64 support is there upstream, its just not included in the in tree ebuild.
Copy the ebuild to your overlay and apply the following patch.
--- rust-1.16.0.ebuild 2017-03-23 20:18:30.074368422 +0000
+++ rust-1.16.0-r100.ebuild 2017-06-15 19:47:05.857067998 +0100
@@ -26,6 +26,7 @@
STAGE0_VERSION="1.$(($(get_version_component_range 2) - 1)).1"
RUST_STAGE0_amd64="rustc-${STAGE0_VERSION}-x86_64-unknown-linux-gnu"
RUST_STAGE0_x86="rustc-${STAGE0_VERSION}-i686-unknown-linux-gnu"
+RUST_STAGE0_arm64="rustc-${STAGE0_VERSION}-aarch64-unknown-linux-gnu"
DESCRIPTION="Systems programming language from Mozilla"
HOMEPAGE="http://www.rust-lang.org/"
@@ -33,6 +34,7 @@
SRC_URI="https://static.rust-lang.org/dist/${SRC} -> rustc-${PV}-src.tar.gz
amd64? ( https://static.rust-lang.org/dist/${RUST_STAGE0_amd64}.tar.gz )
x86? ( https://static.rust-lang.org/dist/${RUST_STAGE0_x86}.tar.gz )
+ arm64? ( https://static.rust-lang.org/dist/${RUST_STAGE0_arm64}.tar.gz )
"
LICENSE="|| ( MIT Apache-2.0 ) BSD-1 BSD-2 BSD-4 UoI-NCSA"
Don't forget to make the digest.
It needs to be ~arm64 keyworded too, or added to package.accept_keywords.
Cargo
Cargo is much the same. Aarch64 support is available. However, the 2016-09-01 nghtly snapshot, which is what the ebuild would fetch, segfaults in the QEMU chroot. The 2016-11-28 version seems to build.
Copy the ebuild to your overlay and apply the following patch.
--- cargo-0.17.0.ebuild 2017-03-18 10:41:17.000000000 +0000
+++ /root/cargo-0.17.0-r100.ebuild 2017-06-18 21:04:41.164444192 +0100
@@ -3,7 +3,7 @@
EAPI=6
-CARGO_SNAPSHOT_DATE="2016-09-01"
+CARGO_SNAPSHOT_DATE="2016-11-28"
CRATES="
advapi32-sys-0.2.0
aho-corasick-0.5.3
@@ -93,12 +93,16 @@
amd64? (
https://static.rust-lang.org/cargo-dist/${CARGO_SNAPSHOT_DATE}/cargo-nightly-x86_64-unknown-linux-gnu.tar.gz ->
cargo-snapshot-amd64-${CARGO_SNAPSHOT_DATE}.tar.gz
+ )
+ arm64? (
+ https://static.rust-lang.org/cargo-dist/${CARGO_SNAPSHOT_DATE}/cargo-nightly-aarch64-unknown-linux-gnu.tar.gz ->
+ cargo-snapshot-aarch64-${CARGO_SNAPSHOT_DATE}.tar.gz
)"
RESTRICT="mirror"
LICENSE="|| ( MIT Apache-2.0 )"
SLOT="0"
-KEYWORDS="~amd64 ~x86"
+KEYWORDS="~amd64 ~arm64 ~x86"
IUSE="doc libressl"
@@ -126,6 +130,7 @@
# where <arch> could be 'x86_64' (amd64) or 'i686' (x86)
use amd64 && CTARGET="x86_64-unknown-linux-gnu"
use x86 && CTARGET="i686-unknown-linux-gnu"
+ use arm64 && CTARGET="aarch64-unknown-linux-gnu"
# NOTE: 'disable-nightly' is used by crates (such as 'matches') to entirely
# skip their internal libraries that make use of unstable rustc features.
Firefox-54
With rust and cargo in packages, firefox itself might build on the Pi with help from cross distcc. (it won't, not even at -j1).
Rust is profile masked on arm64 but its a hard dependency for Firefox-54.
Continue to build firefox in the QEMU chroot
If the build fails with
File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 75, in __init__
sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 38] Function not implemented
this indicates that /dev/shm is not available in the chroot.
mount tmpfs -t tmpfs -o rw,nosuid,nodev,noexec /dev/shm will fix it.
Here, the build ends with
>>> Install firefox-54.0 into /var/tmp/portage/www-client/firefox-54.0/image/ category www-client * PT_PAX marking -m /var/tmp/portage/www-client/firefox-54.0/work/firefox-54.0/ff/dist/bin/xpcshell with scanelf * XATTR_PAX marking -me /var/tmp/portage/www-client/firefox-54.0/work/firefox-54.0/ff/dist/bin/xpcshell with setfattr * Adding prefs from mozconfig to /var/tmp/portage/www-client/firefox-54.0/work/firefox-54.0/ff/dist/bin/browser/defaults/preferences/all-gentoo.js make -j8 DESTDIR=/var/tmp/portage/www-client/firefox-54.0/image/ install make[1]: Entering directory '/var/tmp/portage/www-client/firefox-54.0/work/firefox-54.0/ff/browser/installer'
which is the stalled optimise seen in versions of firefox>=52 on arm64.
There is a patch for firefox, create /etc/portage/patches/www-client/firefox/no-optimise.patch with the following content
--- ./browser/config/mozconfigs/linux32/common-opt.orig 2017-03-27 16:47:57.039428674 -0700
+++ ./browser/config/mozconfigs/linux32/common-opt 2017-03-27 16:43:26.004591584 -0700
@@ -3,6 +3,7 @@
ac_add_options --enable-update-channel=${MOZ_UPDATE_CHANNEL}
ac_add_options --with-google-api-keyfile=/builds/gapi.data
ac_add_options --with-mozilla-api-keyfile=/builds/mozilla-desktop-geoloc-api.key
+ac_add_options --disable-startupcache
. $topsrcdir/build/unix/mozconfig.linux32
--- ./browser/config/mozconfigs/linux64/common-opt.orig 2017-03-27 16:46:54.794982020 -0700
+++ ./browser/config/mozconfigs/linux64/common-opt 2017-03-27 16:47:05.345170624 -0700
@@ -3,6 +3,7 @@
ac_add_options --enable-update-channel=${MOZ_UPDATE_CHANNEL}
ac_add_options --with-google-api-keyfile=/builds/gapi.data
ac_add_options --with-mozilla-api-keyfile=/builds/mozilla-desktop-geoloc-api.key
+ac_add_options --disable-startupcache
. $topsrcdir/build/unix/mozconfig.linux
note that the patch ends with a blank line
This patch alone does not work for me, it also takes
FEATURES="-sandbox -usersandbox" MAKEOPTS="-j8" emerge firefox
Next step, install on the Pi
Chroot Build Times
aarch64 NeddySeagoon_Static / # genlop -t rust cargo firefox
* dev-lang/rust
Sun Jun 18 04:30:42 2017 >>> dev-lang/rust-1.16.0-r100
merge time: 7 hours, 8 minutes and 6 seconds.
* dev-util/cargo
Sun Jun 18 10:07:42 2017 >>> dev-util/cargo-0.17.0
merge time: 36 minutes and 7 seconds.
* www-client/firefox
Tue Jun 20 21:14:21 2017 >>> www-client/firefox-54.0
merge time: 2 hours, 51 minutes and 8 seconds.
Install the Packages on the Target
Move the packages and the ebuilds that they were built with to the target and emerge the binaries.
emerge -kav firefox
These are the packages that would be merged, in order: Calculating dependencies... done! [binary N ] dev-util/cargo-0.17.0-r100::gentoo USE="-debug (-doc) (-libressl)" 0 KiB [binary N ] dev-lang/rust-1.16.0-r100:stable/1.16::gentoo USE="-clang -debug (-doc) -libcxx" 0 KiB [binary U ] www-client/firefox-54.0::gentoo [51.0.1::gentoo] USE=" ..."
Careful readers will note that the writer did not use an overlay for the patched ebuilds.
Complete Aarch64 KVM on an amd64 Host
It should work but this is left as an exercise to the reader.
The boot loader will probably need to be u-boot, since it will need to run on the arm64 emulator and neither grub, nor the Raspberry Pi bootcode.bin will do that.
Hint: u-boot can be used to boot the Pi. bootcode.bin is then used to load u-boot and u-boot does the rest.