Embedded Handbook/General/Compiling with QEMU user chroot

General topics
Introduction
Compiling with QEMU user chroot
Creating a cross-compiler
Cross-compiling with Portage
Cross-compiling the kernel
Frequently asked questions
Emulators
Qemu
Armulator
Hercules
Bootloaders
Das U-Boot
NeTTrom
RedBoot
SH-LILO

Compiling with QEMU user chroot.

This wiki page explains the required steps needed to be able to chroot into a root filesystem whose platform (e.g. aarch64) is different from the running system (e.g. amd64). Software can then be compiled within the chroot transparently using QEMU emulation.

Installation

Kernel

The build system's kernel must support miscellaneous binary formats. This can be enabled with CONFIG_BINFMT_MISC=m or CONFIG_BINFMT_MISC=y in the the kernel's .config file.

Tip
A system restart is required after building this module before it can be used.

KERNEL Enable CONFIG_BINFMT_MISC

Executable file formats  --->
  <*> Kernel support for MISC binaries

USE Flags

USE flags for app-emulation/qemu QEMU + Kernel-based Virtual Machine userland tools

+aio	Enables support for Linux's Async IO
+curl	Support ISOs / -cdrom directives via HTTP or HTTPS.
+doc	Add extra documentation (API, Javadoc, etc). It is recommended to enable per package instead of globally
+fdt	Enables firmware device tree support
+filecaps	Use Linux file capabilities to control privilege rather than set*id (this is orthogonal to USE=caps which uses capabilities at runtime e.g. libcap)
+gnutls	Enable TLS support for the VNC console server. For 1.4 and newer this also enables WebSocket support. For 2.0 through 2.3 also enables disk quorum support.
+jpeg	Enable jpeg image support for the VNC console server
+oss	Add support for OSS (Open Sound System)
+pin-upstream-blobs	Pin the versions of BIOS firmware to the version included in the upstream release. This is needed to sanely support migration/suspend/resume/snapshotting/etc... of instances. When the blobs are different, random corruption/bugs/crashes/etc... may be observed.
+png	Enable png image support for the VNC console server
+seccomp	Enable seccomp (secure computing mode) to perform system call filtering at runtime to increase security of programs
+slirp	Enable TCP/IP in hypervisor via net-libs/libslirp
+vhost-net	Enable accelerated networking using vhost-net, see https://www.linux-kvm.org/page/VhostNet
+vnc	Enable VNC (remote desktop viewer) support
accessibility	Adds support for braille displays using brltty
alsa	Enable alsa output for sound emulation
bpf	Enable eBPF support for RSS implementation.
bzip2	Enable bzip2 compression support
capstone	Enable disassembly support with dev-libs/capstone
debug	Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces
fuse	Enables FUSE block device export
glusterfs	Enables GlusterFS cluster fileystem via sys-cluster/glusterfs
gtk	Add support for x11-libs/gtk+ (The GIMP Toolkit)
infiniband	Enable Infiniband RDMA transport support
io-uring	Enable efficient I/O via sys-libs/liburing.
iscsi	Enable direct iSCSI support via net-libs/libiscsi instead of indirectly via the Linux block layer that sys-block/open-iscsi does.
jack	Add support for the JACK Audio Connection Kit
jemalloc	Enable jemalloc allocator support
keyutils	Support Linux keyrings via sys-apps/keyutils
lzo	Enable support for lzo compression
multipath	Enable multipath persistent reservation passthrough via sys-fs/multipath-tools.
ncurses	Enable the ncurses-based console
nfs	Enable NFS support
nls	Add Native Language Support (using gettext - GNU locale utilities)
numa	Enable NUMA support
opengl	Add support for OpenGL (3D graphics)
pam	Add support for PAM (Pluggable Authentication Modules) - DANGEROUS to arbitrarily flip
pipewire	Enable pipewire output for sound emulation
plugins	Enable qemu plugin API via shared library loading.
pulseaudio	Enable pulseaudio output for sound emulation
python	Add optional support/bindings for the Python language
rbd	Enable rados block device backend support, see https://docs.ceph.com/en/mimic/rbd/qemu-rbd/
sasl	Add support for the Simple Authentication and Security Layer
sdl	Enable the SDL-based console
sdl-image	SDL Image support for icons
selinux	!!internal use only!! Security Enhanced Linux support, this must be set by the selinux profile or breakage will occur
smartcard	Enable smartcard support
snappy	Enable support for Snappy compression (as implemented in app-arch/snappy)
spice	Enable Spice protocol support via app-emulation/spice
ssh	Enable SSH based block device support via net-libs/libssh2
static	Build the User and Software MMU (system) targets as well as tools as static binaries
static-user	Build the User targets as static binaries
systemtap	Enable SystemTap/DTrace tracing
test	Enable dependencies and/or preparations necessary to run tests (usually controlled by FEATURES=test but can be toggled independently)
udev	Enable virtual/udev integration (device discovery, power and storage device support, etc)
usb	Enable USB passthrough via dev-libs/libusb
usbredir	Use sys-apps/usbredir to redirect USB devices to another machine over TCP
vde	Enable VDE-based networking
virgl	Enable experimental Virgil 3d (virtual software GPU)
virtfs	Enable VirtFS via virtio-9p-pci / fsdev. See https://wiki.qemu.org/Documentation/9psetup
vte	Enable terminal support (x11-libs/vte) in the GTK+ interface
xattr	Add support for getting and setting POSIX extended attributes, through sys-apps/attr. Requisite for the virtfs backend.
xdp	Enable support for XDP through net-libs/xdp-tools
xen	Enables support for Xen backends
zstd	Enable support for ZSTD compression

Data provided by the Gentoo Package Database · Last update: 2024-10-31 01:03 More information about USE flags

Important
app-emulation/qemu must be emerged with USE=static-user to use user targets.

Warning
QEMU_SOFTMMU_TARGETS and QEMU_USER_TARGETS are empty by default and must be defined to utilize user targets.

Note
Activating the static-user will require supporting libraries to be build with static-libs support.

FILE /etc/portage/package.use/qemuEnable aarch64 user target and static-libs in supporting libraries.

# Enable static-user and add the aarch64 target
app-emulation/qemu static-user QEMU_SOFTMMU_TARGETS: aarch64 QEMU_USER_TARGETS: aarch64
# required by app-emulation/qemu::gentoo[static,static-user]
# required by qemu (argument)
dev-libs/glib static-libs
# required by app-emulation/qemu::gentoo[-static,static-user]
# required by qemu (argument)
sys-libs/zlib static-libs
# required by app-emulation/qemu::gentoo[-static,static-user,xattr]
# required by qemu (argument)
sys-apps/attr static-libs
# required by dev-libs/glib::gentoo
# required by app-emulation/qemu::gentoo[-static,static-user]
# required by qemu (argument)
dev-libs/libpcre2 static-libs

Warning
If using LTO and the static-user, GCC will use huge amount of RAM when compiling. Because of this, it is generally recommended to disable LTO while compiling in this configuration or use Clang if LTO is required.

See also
bug #883419

Note
When xattr is not enabled emerge may fail to install packages in the chroot'ed environment.

QEMU target configuration

By default, app-emulation/qemu does not define any QEMU_SOFTMMU_TARGETS or QEMU_USER_TARGETS. The example configuration above only includes aarch64 targets.

Tip
The output of emerge -pv app-emulation/qemu contains the full target list.

Tip
To build for a Raspberry Pi running a 32-bit stage3 build, be sure to select QEMU_USER_TARGETS="arm" here. For 64-bit stage3 builds, select aarch64. When in doubt both can be set.

To build all targets:

root #echo 'QEMU_SOFTMMU_TARGETS="*"' >> /etc/portage/make.conf

root #echo 'QEMU_USER_TARGETS="*"' >> /etc/portage/make.conf

Alternatively, the USE_EXPAND syntax can be used to keep make.conf free of USE_EXPAND values:

FILE /etc/portage/package.use/qemuConfigure QEMU to build all targets.

app-emulation/qemu QEMU_SOFTMMU_TARGETS: * QEMU_USER_TARGETS: *

Emerge

root #emerge --ask --update --newuse --deep app-emulation/qemu

Configuration

Group configuration

For non-root users to use QEMU, they must be added to the kvm group.

To add larry to the kvm group:

root #gpasswd -a larry kvm

OpenRC

To start the qemu-binfmt service:

root #rc-service qemu-binfmt start

It may be wise for the services to be started by default on boot:

root #rc-update add qemu-binfmt default

systemd

The systemd-binfmt service must be configured by adding files containing the desired handler registration strings to /etc/binfmt.d/.

Modern versions of qemu ship a binfmt configuration file that supports all binary formats. Simply link it to /etc for binary format support, then skip the following manual file creation steps.

root #ln -s /usr/share/qemu/binfmt.d/qemu.conf /etc/binfmt.d/qemu.conf

Manual binfmt configuration

Alternatively, to only make support for aarch64 and arm architectures, separate files can manually be created:

FILE /etc/binfmt.d/qemu-aarch64-static.confSpecifying an aarch64 binfmt-misc handler for systemd-binfmt

:aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:

FILE /etc/binfmt.d/qemu-arm-static.confSpecifying an arm binfmt-misc handler for systemd-binfmt

:arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:

See #Binary format handlers for a list of possible handlers.

Note
In the FileBox above, the eighth byte of the mask has been set to fc (rather than ff), to allow programs like gcc to be executed. See for example Debian bug #799120.

With the file(s) created in /etc/binfmt.d/, systemd will now find them automatically at system boot time. See man 5 binfmt.d for more information.

Since the files were likely created just a few moments ago, it will be necessary to restart the service once to register the binary formats:

root #systemctl restart systemd-binfmt

To confirm the service is running properly after restarting:

root #systemctl status systemd-binfmt

Usage

Preparing the chroot

To be able to chroot into a system of a different platform (e.g. aarch64 while using an amd64 system), mounted in e.g. /mnt/gentoo, the QEMU static-user binary must be copied into the environment.

This file is named qemu-<architecture> under /usr/bin/ and should be copied to usr/bin/ within the build environment.

To use an aarch64 chroot environment at /mnt/gentoo-aarch64:

user $cp /usr/bin/qemu-aarch64 /mnt/gentoo-aarch64/usr/bin

Chrooting

Once the environment has been prepared, sys-apps/arch-chroot can be used:

root #arch-chroot /mnt/gentoo-aarch64

Portage configuration

Currently QEMU does not support the pid-sandbox (bug #703278) and network-sandbox (bug #703276) Portage features.

To disable these features::

FILE /etc/portage/make.conf

FEATURES="-pid-sandbox -network-sandbox"

Additional Usage

Binary format handlers

Binary format handlers look rather confusing at first, but they can be simply broken down as: :name:type:offset:magic:mask:interpreter:flags

With each field representing, in order of appearance:

• name being the name of the binary format, a new /proc file will be created with this name at /proc/sys/fs/binfmt_misc
• type being either E or M
  • E means the executable file format is identified by its file extension
  • M means the format is identified by the magic number mask
• offset represents the offset of the magic/mask in the file in bytes, the default is 0
• magic is a byte sequence that binfmt matches for to know what files to pass through to the interpreter
• interpreter is a program that is to be run with the matching file as an argument
• flags is a field that controls different aspects of interpreter when it is invoked
  • P - preserve-argv[0]: adds an argument to the argument vector to preserve the original argv[0]
  • O - open-binary: opens the file for reading and pass its descriptor as an argument rather than the full path
  • C - credentials: current behavior of binfmt_misc is to calculate the credentials and security of the new process according to the interpreter, but including this flag will be calculated according to the binary. (Implies the O flag)
  • F - fix binary: currently, the binary is spawned lazily when the misc format file is invoked, however, this does not work very well in the face of mount namespaces and changroots so F allows the binary to be opened always once it is installed, regardless of environment changes.

Some restrictions do apply with binfmt:

• The whole register string may not exceed 1920 characters
• The magic must reside in the first 128 bytes of the file
• The interpreter string may not exceed 127 characters

Architecture	Binfmt Handler String
AARCH64	:aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:
ARM	:arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:
ARMeb	:armeb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-armeb:
Alpha	:alpha:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x26\x90:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-alpha:
DOS	:DOSWin:M::MZ::/etc/eselect/wine/bin/wine:
LOONGARCH64	:loongarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x02\x01:\xff\xff\xff\xff\xff\xff\xff\xfc\x00\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-loongarch64:
M68K	:m68k:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x04:\xff\xff\xff\xff\xff\xff\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-m68k:
MIPS	:mips:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-mips:
MIPS64	:mips64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mips64:
MIPSel	:mipsel:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mipsel:
MIPS N32	:mipsn32:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-mipsn32:
MIPSel N32	:mipseln32:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mipseln32:
PPC	:ppc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x14:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-ppc:
PPC64	:ppc64:M::\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x15:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-ppc64:
RISCV64	:riscv64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-riscv64:
SH4	:sh4:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff\xff\xff:/usr/bin/qemu-sh4:
SH4eb	:sh4eb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sh4eb:
SPARC	:sparc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x02:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sparc:

Manual setup

First, mount the binfmt_misc handler if it is not already mounted, then register the format with the kernel via the procfs:

root #[ -d /proc/sys/fs/binfmt_misc ] || modprobe binfmt_misc

root #[ -f /proc/sys/fs/binfmt_misc/register ] || mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc

Important
Do not register a handler that matches the host machine (in this example, registers for x86 and AMD64 has been excluded since the host machine).

To register a handler, pipe its format to /proc/sys/fs/binfmt_misc/register, for example

root #echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:' > /proc/sys/fs/binfmt_misc/register

See also
#Binary format handlers for a list of possible handlers.

Advanced Tips

Sometimes we'll need to pass additional args to QEMU (CPU model), so we'll create a wrapper script (in C) that'll call QEMU with it:

FILE qemu-wrapper.c

/*
 * Pass arguments to qemu binary
 */

#include <string.h>
#include <unistd.h>

int main(int argc, char **argv, char **envp) {
	char *newargv[argc + 3];

	newargv[0] = argv[0];
	newargv[1] = "-cpu";
	newargv[2] = "cortex-a8"; /* here you can set the cpu you are building for */

	memcpy(&newargv[3], &argv[1], sizeof(*argv) * (argc -1));
	newargv[argc + 2] = NULL;
	return execve("/usr/bin/qemu-arm", newargv, envp);
}

Compile the wrapper with:

root #gcc -static qemu-wrapper.c -O3 -s -o qemu-wrapper

Then copy into the chroot. Notice the first example ARM entry in the binfmt_misc section uses this method.

References

← Introduction Home Creating a cross-compiler →

---

This page is based on a document formerly found on our main website gentoo.org.
The following people contributed to the original document: Mike Frysinger, Ned Ludd, Robin H. Johnson, Alex Tarkovsky, Alexey Shvetsov, Raúl Porcel, Joshua Saddler on April 28, 2013.
They are listed here because wiki history does not allow for any external attribution. If you edit the wiki article, please do not add yourself here; your contributions are recorded on each article's associated history page.