QEMU

From Gentoo Wiki
(Redirected from KVM)
Jump to:navigation Jump to:search
This page contains changes which are not marked for translation.


QEMU (Quick EMUlator) is a generic, open source hardware emulator and virtualization suite.

Introduction

QEMU is a Type-2 hypervisor that runs within user space and performs virtual hardware emulation.

  • Firstly, QEMU is a type 2 hypervisor.
  • QEMU can be paired with KVM to run VMs at near native speed. This is accomplished by using hardware extensions such as: Intel VT-x or AMD-V.
  • It can then emulate for user-level processes that allow applications compiled for one architecture to run on a different one.
  • Multiple operating modes: User-mode emu, System emu, KVM hosting, and Xen Hosting,
  • QEMU can save and restore the state of VMs of all its running programs.
  • QEMU VMs can interface with many types of physical host hardware such as CD-ROM Drives, USB Devices, Audio Interfaces, Hard Disks, Network Cards
  • Format of Virtual Disk image defaults to the `qcow2` format. The `qcow2` only uses as much host disk space as the Guest OS grows to use. Using snapshot method, guest OS can revert back to its desire state in time.
  • It does not depend on graphical output methods on the host system, instead making use of an integrated VNC server to access the screen of the guest OS.
  • QEMU on a host CPU can execute multiple virtual CPUs in parallel.


QEMU has support for acceleration plug-ins.

Available QEMU plugins are:

Virtualizer Accelerator Virtualization type Description Gentoo package name
qemu tcg full[1]/software-emulation QEMU's own Tiny Code Generator. This is the default. More frequently denoted as qemu and not qemu/tcg so often. app-emulation/qemu
qemu hvf[2] paravirtualization[3] Apple's Hypervisor.framework based on Intel VT.
qemu whpx[4] hybrid Microsoft's Windows Hypervisor Platform based on Intel VT or AMD-V.
qemu kvm paravirtualization[5] Linux Type-2 Hypervisor. This is the common choice for host using amd64, arm64, or mips[6]. Supports Microsoft Windows. app-emulation/qemu
qemu haxm[7] paravirtualization[8] Intel VT, by Intel Corporation.

QEMU when used in conjunction with an accelerator becomes a Type-1 hypervisor that runs in kernel space, that allows a user space program access to the hardware virtualization features of various processors. Such accelerator can be KVM (Kernel-based Virtual Machine) or Xen.

If no accelerator is used, QEMU will run entirely in user-space using its built in binary translator TCG (Tiny Code Generator). Using QEMU without an accelerator is relatively inefficient and slow.

Note
This article typically uses KVM as the accelerator of choice due to its GPL licensing and availability. Without KVM nearly all commands described here will still work (unless KVM specific).


The following sub-articles provide detailed instructions on QEMU configurations and options:


Installation

BIOS and UEFI firmware

In order to utilize KVM either Vt-x (vmx) or AMD-V (svm) must be supported by the processor. Vt-x or AMD-V are Intel and AMD's respective technologies for permitting multiple operating systems to concurrently execute operations on the processors.

To inspect hardware for virtualization support issue the following command:

user $grep --color -E "vmx|svm" /proc/cpuinfo

For a period manufacturers were shipping with virtualization turned off by default in the system's firmware. Note that toggling this feature in the firmware may actually require full removal of power from the system to take effect. If restarting the system does not work try shutting down, unplugging the system, and pressing the power button in an unplugged state to discharge any residual energy from the power supply unit (PSU). Reapply power to the system to verify success.

If KVM support is available there should be a "kvm" device listed at /dev/kvm. This will take effect after the system has booted to a KVM enabled kernel.

Kernel

Described below are the basic requirements for KVM kernel configuration for the host OS. A more complete and up-to-date list can be found at the KVM Tuning Kernel page.

Note
Different guest (virtualized) OS may require additional kernel options. These are covered in the corresponding QEMU section pages.
KERNEL Enable high resolution timer support (CONFIG_HIGH_RES_TIMERS)
General setup  --->
    Timers subsystem  --->
        <*>   High Resolution Timer Support
Note
This includes support for ARM64 processors.
Physical CPU processor support - Host

If KVM support is not available, insert CONFIG_KVM=y into the /usr/src/linux/.config and rebuild/reinstall the kernel (and its initramfs image). Come back here after the host gets rebooted.


KERNEL Enable KVM Support (CONFIG_KVM)
[*] Virtualization  --->
    <*>   Kernel-based Virtual Machine (KVM) support
Note
This includes support for ARM64 processors.

Processor Support

KERNEL Enable KVM support for Intel processors (CONFIG_KVM_INTEL)
[*] Virtualization  --->
    <M>   KVM for Intel processors support
KERNEL Enable KVM support for AMD processors (CONFIG_KVM_AMD)
[*] Virtualization  --->
    <M>   KVM for AMD processors support
Warning
If both "KVM for Intel processors support" and "KVM for AMD processors support" are set as built into the kernel (*) an error message will appear from kprint from early boot. Since the system has only one type processor (Intel or AMD) enabling one or both options as modules (M) will make the error message disappear.

Handling kernel config at CLI

To set the various kernel configuration settings from the command lines, the linux/scripts/kconfig/merge_config.sh shall be used here:

Mandatory kernel configuration options to set:

FILE /usr/src/kernel-kconfig-qemu-host.config
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=y
CONFIG_KVM_INTEL=y
CONFIG_KVM_AMD=y
root #cd /usr/src/linux
root #scripts/kconfig/merge_config.sh .config /usr/src/kernel-kconfig-qemu-host.config

Useful kernel configuration options to use:

FILE /usr/src/kernel-kconfig-qemu-host-optional.config
CONFIG_VHOST_NET=y
CONFIG_HIGH_RES_TIMER=y
CONFIG_HPET=y
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_KSM=y
CONFIG_SYSFS=y
CONFIG_PROC_FS=y
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_CGROUPS=y
root #scripts/kconfig/merge_config.sh .config /usr/src/kernel-kconfig-qemu-host-optional.config

Networking

Accelerated networking, required for vhost-net USE flag (recommend):

KERNEL vhost-net kernel 5.7 and later (CONFIG_VHOST_NET)
Device Drivers  --->
    [*] VHOST drivers  --->
        <*> Host kernel accelerator for virtio net
KERNEL vhost-net (before kernel 5.7)
[*] Virtualization --->
    <*> Host kernel accelerator for virtio net
KERNEL Optional advanced networking support (CONFIG_NET_CORE, CONFIG_TUN)
Device Drivers  --->
    [*] Network device support  --->
        [*] Network core driver support
            <*> Universal TUN/TAP device driver support

Needed for 802.1d Ethernet bridging:

KERNEL Enabling 802.1d Ethernet Bridging support (CONFIG_IPV6, CONFIG_BRIDGE)
[*] Networking support  --->
        Networking options  --->
            <*> The IPv6 protocol
            <*> 802.1d Ethernet Bridging

Intel VT-g (integrated graphics adapter virtualization)

Mediated device passthrough for Intel GPUs (Broadwell to Comet Lake) [1].

KERNEL Intel VT-g (CONFIG_VFIO_MDEV, CONFIG_DRM_I915_GVT, CONFIG_DRM_I915_GVT_KVMGT)
Device Drivers  --->
        <*> VFIO Non-Privileged userspace driver framework
            <*> Mediated device driver framework
        Graphics Support  --->
            <*> Intel 8xx/9xx/G3x/G4x/HD Graphics
                [*] Enable Intel GVT-g graphics virtualization host support
            <*> Enable KVM host support Intel GVT-g graphics virtualization


USE flags

Some packages are aware of the qemu USE flag.

Review the possible USE flags for QEMU:

USE flags for app-emulation/qemu QEMU + Kernel-based Virtual Machine userland tools

+aio Enables support for Linux's Async IO
+curl Support ISOs / -cdrom directives via HTTP or HTTPS.
+doc Add extra documentation (API, Javadoc, etc). It is recommended to enable per package instead of globally
+fdt Enables firmware device tree support
+filecaps Use Linux file capabilities to control privilege rather than set*id (this is orthogonal to USE=caps which uses capabilities at runtime e.g. libcap)
+gnutls Enable TLS support for the VNC console server. For 1.4 and newer this also enables WebSocket support. For 2.0 through 2.3 also enables disk quorum support.
+jpeg Enable jpeg image support for the VNC console server
+oss Add support for OSS (Open Sound System)
+pin-upstream-blobs Pin the versions of BIOS firmware to the version included in the upstream release. This is needed to sanely support migration/suspend/resume/snapshotting/etc... of instances. When the blobs are different, random corruption/bugs/crashes/etc... may be observed.
+png Enable png image support for the VNC console server
+seccomp Enable seccomp (secure computing mode) to perform system call filtering at runtime to increase security of programs
+slirp Enable TCP/IP in hypervisor via net-libs/libslirp
+vhost-net Enable accelerated networking using vhost-net, see https://www.linux-kvm.org/page/VhostNet
+vnc Enable VNC (remote desktop viewer) support
accessibility Adds support for braille displays using brltty
alsa Enable alsa output for sound emulation
bpf Enable eBPF support for RSS implementation.
bzip2 Enable bzip2 compression support
capstone Enable disassembly support with dev-libs/capstone
debug Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces
fuse Enables FUSE block device export
glusterfs Enables GlusterFS cluster fileystem via sys-cluster/glusterfs
gtk Add support for x11-libs/gtk+ (The GIMP Toolkit)
infiniband Enable Infiniband RDMA transport support
io-uring Enable efficient I/O via sys-libs/liburing.
iscsi Enable direct iSCSI support via net-libs/libiscsi instead of indirectly via the Linux block layer that sys-block/open-iscsi does.
jack Add support for the JACK Audio Connection Kit
jemalloc Enable jemalloc allocator support
keyutils Support Linux keyrings via sys-apps/keyutils
lzo Enable support for lzo compression
multipath Enable multipath persistent reservation passthrough via sys-fs/multipath-tools.
ncurses Enable the ncurses-based console
nfs Enable NFS support
nls Add Native Language Support (using gettext - GNU locale utilities)
numa Enable NUMA support
opengl Add support for OpenGL (3D graphics)
pam Add support for PAM (Pluggable Authentication Modules) - DANGEROUS to arbitrarily flip
pipewire Enable pipewire output for sound emulation
plugins Enable qemu plugin API via shared library loading.
pulseaudio Enable pulseaudio output for sound emulation
python Add optional support/bindings for the Python language
rbd Enable rados block device backend support, see https://docs.ceph.com/en/mimic/rbd/qemu-rbd/
sasl Add support for the Simple Authentication and Security Layer
sdl Enable the SDL-based console
sdl-image SDL Image support for icons
selinux !!internal use only!! Security Enhanced Linux support, this must be set by the selinux profile or breakage will occur
smartcard Enable smartcard support
snappy Enable support for Snappy compression (as implemented in app-arch/snappy)
spice Enable Spice protocol support via app-emulation/spice
ssh Enable SSH based block device support via net-libs/libssh2
static Build the User and Software MMU (system) targets as well as tools as static binaries
static-user Build the User targets as static binaries
systemtap Enable SystemTap/DTrace tracing
test Enable dependencies and/or preparations necessary to run tests (usually controlled by FEATURES=test but can be toggled independently)
udev Enable virtual/udev integration (device discovery, power and storage device support, etc)
usb Enable USB passthrough via dev-libs/libusb
usbredir Use sys-apps/usbredir to redirect USB devices to another machine over TCP
vde Enable VDE-based networking
virgl Enable experimental Virgil 3d (virtual software GPU)
virtfs Enable VirtFS via virtio-9p-pci / fsdev. See https://wiki.qemu.org/Documentation/9psetup
vte Enable terminal support (x11-libs/vte) in the GTK+ interface
xattr Add support for getting and setting POSIX extended attributes, through sys-apps/attr. Requisite for the virtfs backend.
xdp Enable support for XDP through net-libs/xdp-tools
xen Enables support for Xen backends
zstd Enable support for ZSTD compression
Data provided by the Gentoo Package Database · Last update: 2024-10-31 01:03 More information about USE flags

Note
More than one USE flag (gtk, ncurses, sdl, or spice) can be enabled for graphical output. If graphics are desired it is generally recommended to enable more than one graphical USE flag.
Note
If virt-manager is going to be used, be sure to enable the usbredir and spice USE flags on the qemu package for correct operation.

USE_EXPAND

Additional ebuild configuration frobs are provided as the USE_EXPAND variables QEMU_USER_TARGETS and QEMU_SOFTMMU_TARGETS. See app-emulation/qemu for a list of all the available targets (there are many; most are very obscure and may be ignored; leaving these variables at their default values will disable almost everything which is probably just fine for most users).

For each target specified, a qemu executable will be built. A softmmu target is the standard qemu use-case of emulating an entire system (like VirtualBox or VMWare, but with optional support for emulating CPU hardware along with peripherals). user targets execute user-mode code only; the (somewhat shockingly ambitious) purpose of these targets is to "magically" allow importing user-space linux ELF binaries from a different architecture into the native system (like multilib, without the awkward need for a software stack or CPU capable of running it).

In order to enable QEMU_USER_TARGETS and QEMU_SOFTMMU_TARGETS we can edit the variables globally in /etc/portage/make.conf, i.e.:

FILE /etc/portage/make.conf
QEMU_SOFTMMU_TARGETS="arm x86_64 sparc"
QEMU_USER_TARGETS="x86_64"


Or, the /etc/portage/package.use file(s) can be modified. Two equivalent syntaxes are available: traditional USE flag syntax, i.e.:

FILE /etc/portage/package.use
app-emulation/qemu qemu_softmmu_targets_arm qemu_softmmu_targets_x86_64 qemu_softmmu_targets_sparc
app-emulation/qemu qemu_user_targets_x86_64


Another alternative is to use the newer USE_EXPAND-specific syntax:

FILE /etc/portage/package.use
app-emulation/qemu QEMU_SOFTMMU_TARGETS: arm x86_64 sparc QEMU_USER_TARGETS: x86_64


Emerge

After reviewing and adding any desired USE flags, emerge app-emulation/qemu:

root #emerge --ask app-emulation/qemu


Configuration

The following sub-articles provide detailed instructions on QEMU configurations and options:

  • Usage options - Contains common configuration options used with QEMU (graphics/display, networking, RAM, storage, processor, etc).
  • QEMU/Linux guest — describes the setup of a Gentoo Linux guest in QEMU using Gentoo bootable media.
  • QEMU/Windows guest — setup of a Windows guest using QEMU
  • OS2WarpV3 guest - Describes the configuration steps needed to setup a virtualized OS2WarpVs=3 guest with QEMU.

Environment variables

  • G_MESSAGES_DEBUG
  • LISTEN_FDS
  • LISTEN_PID
  • QEMU_AUDIO_DRV
  • QEMU_MODULE_DIR
  • XDG_RUNTIME_DIR

Files

Files that QEMU uses.

Single File

  • /etc/libvirt/qemu.conf - QEMU configuration file.
  • /etc/libvirt/qemu-lockd.conf - QEMU lock files
  • /etc/libvirt/qemu-sanlock.conf - QEMU SAN lock
  • /etc/libvirt/qemu/<domain-name>.xml - Domain XML setting for a virtual machine or container.
  • /etc/libvirt/qemu/autostart/<domain-name>.xml - Autostart this domain (virtual machine or container).
  • /etc/libvirt/qemu/networks/<network-name>.xml - Network XML setting file for a network connection
  • /etc/libvirt/qemu/networks/autostart/<network-name>.xml - Autostart this network connection.
  • /var/lib/libvirt/qemu/channel/target/<domain-name>/<socket-file> - UNIX socket file for Libvertd daemon API
  • /var/cache/libvirt/qemu/capabilities/<hash-value>.xml - Host OS capabilities in XML-format
  • /var/lib/libvirt/qemu/checkpoint/
  • /var/lib/libvirt/qemu/<domain-9-XXXX>/ - holds UNIX sockets and AES keys for this domain.
  • /var/lib/libvirt/qemu/dump/
  • /var/lib/libvirt/qemu/nvram/
  • /var/lib/libvirt/qemu/ram/
  • /var/lib/libvirt/qemu/save/ - holding directory of hibernation images
  • /var/lib/libvirt/qemu/snapshot/ - holding directory of snapshots
  • /var/run/libvirt/qemu - various UNIX socket and PID files for libvirtd daemon.

Image File

QEMU supports the following disk image formats:

  • QEMU copy-on-write (.qcow2, .qed, .qcow, .cow)
  • VirtualBox Virtual Disk Image (.vdi)
  • CD/DVD (ISO-9660) images (.iso)
  • Raw images (.img), that guest OS can control
  • VFAT-16
  • VMware Virtual Machine Disk (.vmdk)
  • Virtual PC Virtual Hard Disk (.vhd)
  • Parallels disk image (.hdd, .hds) – Read-only
  • Apple macos Universal Disk Image Format (.dmg) – Read-only
  • Bochs – Read-only
  • Linux cloop – Read-only


Additional software

User name qemu is required; Defined by acct-user/qemu; evoked by sys-emulator/qemu package.

Group name qemu is required. Defined by acct-group/qemu; evoked by sys-emulator/qemu package.


Usage

Qemu can be used in two ways, with GUI front ends and through the command line. The configuration of QEMU depends on which method is employed:

  • GUI (Front-End) - To make life easier, there are multiple user-friendly front ends to QEMU: See QEMU front-ends
  • CLI

Invocation

QEMU supports around 34 different CPU architectures. To find the desired architecture, list what is installed.

user $ls /usr/bin/qemu-system-*
/usr/bin/qemu-system-aarch64       /usr/bin/qemu-system-mips      /usr/bin/qemu-system-rx
/usr/bin/qemu-system-alpha         /usr/bin/qemu-system-mips64    /usr/bin/qemu-system-s390x
/usr/bin/qemu-system-arm           /usr/bin/qemu-system-mips64el  /usr/bin/qemu-system-sh4
/usr/bin/qemu-system-avr           /usr/bin/qemu-system-mipsel    /usr/bin/qemu-system-sh4eb
/usr/bin/qemu-system-cris          /usr/bin/qemu-system-nios2     /usr/bin/qemu-system-sparc
/usr/bin/qemu-system-hppa          /usr/bin/qemu-system-or1k      /usr/bin/qemu-system-sparc64
/usr/bin/qemu-system-i386          /usr/bin/qemu-system-ppc       /usr/bin/qemu-system-tricore
/usr/bin/qemu-system-loongarch64   /usr/bin/qemu-system-ppc64     /usr/bin/qemu-system-x86_64
/usr/bin/qemu-system-m68k          /usr/bin/qemu-system-ppc64le   /usr/bin/qemu-system-x86_64-microvm
/usr/bin/qemu-system-microblaze    /usr/bin/qemu-system-riscv32   /usr/bin/qemu-system-xtensa
/usr/bin/qemu-system-microblazeel  /usr/bin/qemu-system-riscv64   /usr/bin/qemu-system-xtensaeb
user $qemu-system-x86_64 -help
QEMU emulator version 7.2.9 (Debian 1:7.2+dfsg-7+deb12u5)
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
usage: qemu-system-x86_64 [options] [disk_image]

'disk_image' is a raw hard disk image for IDE hard disk 0

Standard options:
-h or -help     display this help and exit
-version        display version information and exit
-machine [type=]name[,prop[=value][,...]]
                selects emulated machine ('-machine help' for list)
                property accel=accel1[:accel2[:...]] selects accelerator
                supported accelerators are kvm, xen, hax, hvf, nvmm, whpx or tcg (default: tcg)
                vmport=on|off|auto controls emulation of vmport (default: auto)
                dump-guest-core=on|off include guest memory in a core dump (default=on)
                mem-merge=on|off controls memory merge support (default: on)
                aes-key-wrap=on|off controls support for AES key wrapping (default=on)
                dea-key-wrap=on|off controls support for DEA key wrapping (default=on)
                suppress-vmdesc=on|off disables self-describing migration (default=off)
                nvdimm=on|off controls NVDIMM support (default=off)
                memory-encryption=@var{} memory encryption object to use (default=none)
                hmat=on|off controls ACPI HMAT support (default=off)
                memory-backend='backend-id' specifies explicitly provided backend for main RAM (default=none)
                cxl-fmw.0.targets.0=firsttarget,cxl-fmw.0.targets.1=secondtarget,cxl-fmw.0.size=size[,cxl-fmw.0.interleave-granularity=granularity]
                sgx-epc.0.memdev=memid,sgx-epc.0.node=numaid
-cpu cpu        select CPU ('-cpu help' for list)
-accel [accel=]accelerator[,prop[=value][,...]]
                select accelerator (kvm, xen, hax, hvf, nvmm, whpx or tcg; use 'help' for a list)
                igd-passthru=on|off (enable Xen integrated Intel graphics passthrough, default=off)
                kernel-irqchip=on|off|split controls accelerated irqchip support (default=on)
                kvm-shadow-mem=size of KVM shadow MMU in bytes
                split-wx=on|off (enable TCG split w^x mapping)
                tb-size=n (TCG translation block cache size)
                dirty-ring-size=n (KVM dirty ring GFN count, default 0)
                notify-vmexit=run|internal-error|disable,notify-window=n (enable notify VM exit and set notify window, x86 only)
                thread=single|multi (enable multi-threaded TCG)
-smp [[cpus=]n][,maxcpus=maxcpus][,sockets=sockets][,dies=dies][,clusters=clusters][,cores=cores][,threads=threads]
                set the number of initial CPUs to 'n' [default=1]
                maxcpus= maximum number of total CPUs, including
                offline CPUs for hotplug, etc
                sockets= number of sockets on the machine board
                dies= number of dies in one socket
                clusters= number of clusters in one die
                cores= number of cores in one cluster
                threads= number of threads in one core
Note: Different machines may have different subsets of the CPU topology
      parameters supported, so the actual meaning of the supported parameters
      will vary accordingly. For example, for a machine type that supports a
      three-level CPU hierarchy of sockets/cores/threads, the parameters will
      sequentially mean as below:
                sockets means the number of sockets on the machine board
                cores means the number of cores in one socket
                threads means the number of threads in one core
      For a particular machine type board, an expected CPU topology hierarchy
      can be defined through the supported sub-option. Unsupported parameters
      can also be provided in addition to the sub-option, but their values
      must be set as 1 in the purpose of correct parsing.
-numa node[,mem=size][,cpus=firstcpu[-lastcpu]][,nodeid=node][,initiator=node]
-numa node[,memdev=id][,cpus=firstcpu[-lastcpu]][,nodeid=node][,initiator=node]
-numa dist,src=source,dst=destination,val=distance
-numa cpu,node-id=node[,socket-id=x][,core-id=y][,thread-id=z]
-numa hmat-lb,initiator=node,target=node,hierarchy=memory|first-level|second-level|third-level,data-type=access-latency|read-latency|write-latency[,latency=lat][,bandwidth=bw]
-numa hmat-cache,node-id=node,size=size,level=level[,associativity=none|direct|complex][,policy=none|write-back|write-through][,line=size]
-add-fd fd=fd,set=set[,opaque=opaque]
                Add 'fd' to fd 'set'
-set group.id.arg=value
                set <arg> parameter for item <id> of type <group>
                i.e. -set drive.$id.file=/path/to/image
-global driver.property=value
-global driver=driver,property=property,value=value
                set a global default for a driver property
-boot [order=drives][,once=drives][,menu=on|off]
      [,splash=sp_name][,splash-time=sp_time][,reboot-timeout=rb_time][,strict=on|off]
                'drives': floppy (a), hard disk (c), CD-ROM (d), network (n)
                'sp_name': the file's name that would be passed to bios as logo picture, if menu=on
                'sp_time': the period that splash picture last if menu=on, unit is ms
                'rb_timeout': the timeout before guest reboot when boot failed, unit is ms
-m [size=]megs[,slots=n,maxmem=size]
                configure guest RAM
                size: initial amount of guest memory
                slots: number of hotplug slots (default: none)
                maxmem: maximum amount of guest memory (default: none)
NOTE: Some architectures might enforce a specific granularity
-mem-path FILE  provide backing storage for guest RAM
-mem-prealloc   preallocate guest memory (use with -mem-path)
-k language     use keyboard layout (for example 'fr' for French)
-audio-help     show -audiodev equivalent of the currently specified audio settings
-audio [driver=]driver,model=value[,prop[=value][,...]]
                specifies the audio backend and device to use;
                apart from 'model', options are the same as for -audiodev.
                use '-audio model=help' to show possible devices.
-audiodev [driver=]driver,id=id[,prop[=value][,...]]
                specifies the audio backend to use
                Use ``-audiodev help`` to list the available drivers
                id= identifier of the backend
                timer-period= timer period in microseconds
                in|out.mixing-engine= use mixing engine to mix streams inside QEMU
                in|out.fixed-settings= use fixed settings for host audio
                in|out.frequency= frequency to use with fixed settings
                in|out.channels= number of channels to use with fixed settings
                in|out.format= sample format to use with fixed settings
                valid values: s8, s16, s32, u8, u16, u32, f32
                in|out.voices= number of voices to use
                in|out.buffer-length= length of buffer in microseconds
-audiodev none,id=id,[,prop[=value][,...]]
                dummy driver that discards all output
-audiodev alsa,id=id[,prop[=value][,...]]
                in|out.dev= name of the audio device to use
                in|out.period-length= length of period in microseconds
                in|out.try-poll= attempt to use poll mode
                threshold= threshold (in microseconds) when playback starts
-audiodev oss,id=id[,prop[=value][,...]]
                in|out.dev= path of the audio device to use
                in|out.buffer-count= number of buffers
                in|out.try-poll= attempt to use poll mode
                try-mmap= try using memory mapped access
                exclusive= open device in exclusive mode
                dsp-policy= set timing policy (0..10), -1 to use fragment mode
-audiodev pa,id=id[,prop[=value][,...]]
                server= PulseAudio server address
                in|out.name= source/sink device name
                in|out.latency= desired latency in microseconds
-audiodev sdl,id=id[,prop[=value][,...]]
                in|out.buffer-count= number of buffers
-audiodev sndio,id=id[,prop[=value][,...]]
-audiodev spice,id=id[,prop[=value][,...]]
-audiodev dbus,id=id[,prop[=value][,...]]
-audiodev wav,id=id[,prop[=value][,...]]
                path= path of wav file to record
-device driver[,prop[=value][,...]]
                add device (based on driver)
                prop=value,... sets driver properties
                use '-device help' to print all possible drivers
                use '-device driver,help' to print all possible properties
-name string1[,process=string2][,debug-threads=on|off]
                set the name of the guest
                string1 sets the window title and string2 the process name
                When debug-threads is enabled, individual threads are given a separate name
                NOTE: The thread names are for debugging and not a stable API.
-uuid %08x-%04x-%04x-%04x-%012x
                specify machine UUID

Block device options:
-fda/-fdb file  use 'file' as floppy disk 0/1 image
-hda/-hdb file  use 'file' as hard disk 0/1 image
-hdc/-hdd file  use 'file' as hard disk 2/3 image
-cdrom file     use 'file' as CD-ROM image
-blockdev [driver=]driver[,node-name=N][,discard=ignore|unmap]
          [,cache.direct=on|off][,cache.no-flush=on|off]
          [,read-only=on|off][,auto-read-only=on|off]
          [,force-share=on|off][,detect-zeroes=on|off|unmap]
          [,driver specific parameters...]
                configure a block backend
-drive [file=file][,if=type][,bus=n][,unit=m][,media=d][,index=i]
       [,cache=writethrough|writeback|none|directsync|unsafe][,format=f]
       [,snapshot=on|off][,rerror=ignore|stop|report]
       [,werror=ignore|stop|report|enospc][,id=name]
       [,aio=threads|native|io_uring]
       [,readonly=on|off][,copy-on-read=on|off]
       [,discard=ignore|unmap][,detect-zeroes=on|off|unmap]
       [[,bps=b]|[[,bps_rd=r][,bps_wr=w]]]
       [[,iops=i]|[[,iops_rd=r][,iops_wr=w]]]
       [[,bps_max=bm]|[[,bps_rd_max=rm][,bps_wr_max=wm]]]
       [[,iops_max=im]|[[,iops_rd_max=irm][,iops_wr_max=iwm]]]
       [[,iops_size=is]]
       [[,group=g]]
                use 'file' as a drive image
-mtdblock file  use 'file' as on-board Flash memory image
-sd file        use 'file' as SecureDigital card image
-snapshot       write to temporary files instead of disk image files
-fsdev local,id=id,path=path,security_model=mapped-xattr|mapped-file|passthrough|none
 [,writeout=immediate][,readonly=on][,fmode=fmode][,dmode=dmode]
 [[,throttling.bps-total=b]|[[,throttling.bps-read=r][,throttling.bps-write=w]]]
 [[,throttling.iops-total=i]|[[,throttling.iops-read=r][,throttling.iops-write=w]]]
 [[,throttling.bps-total-max=bm]|[[,throttling.bps-read-max=rm][,throttling.bps-write-max=wm]]]
 [[,throttling.iops-total-max=im]|[[,throttling.iops-read-max=irm][,throttling.iops-write-max=iwm]]]
 [[,throttling.iops-size=is]]
-fsdev proxy,id=id,socket=socket[,writeout=immediate][,readonly=on]
-fsdev proxy,id=id,sock_fd=sock_fd[,writeout=immediate][,readonly=on]
-fsdev synth,id=id
-virtfs local,path=path,mount_tag=tag,security_model=mapped-xattr|mapped-file|passthrough|none
        [,id=id][,writeout=immediate][,readonly=on][,fmode=fmode][,dmode=dmode][,multidevs=remap|forbid|warn]
-virtfs proxy,mount_tag=tag,socket=socket[,id=id][,writeout=immediate][,readonly=on]
-virtfs proxy,mount_tag=tag,sock_fd=sock_fd[,id=id][,writeout=immediate][,readonly=on]
-virtfs synth,mount_tag=tag[,id=id][,readonly=on]
-iscsi [user=user][,password=password]
       [,header-digest=CRC32C|CR32C-NONE|NONE-CRC32C|NONE
       [,initiator-name=initiator-iqn][,id=target-iqn]
       [,timeout=timeout]
                iSCSI session parameters

USB convenience options:
-usb            enable on-board USB host controller (if not enabled by default)
-usbdevice name add the host or guest USB device 'name'

Display options:
-display spice-app[,gl=on|off]
-display sdl[,gl=on|core|es|off][,grab-mod=<mod>][,show-cursor=on|off]
            [,window-close=on|off]
-display gtk[,full-screen=on|off][,gl=on|off][,grab-on-hover=on|off]
            [,show-tabs=on|off][,show-cursor=on|off][,window-close=on|off]
            [,show-menubar=on|off]
-display vnc=<display>[,<optargs>]
-display curses[,charset=<encoding>]
-display egl-headless[,rendernode=<file>]
-display dbus[,addr=<dbusaddr>]
             [,gl=on|core|es|off][,rendernode=<file>]
-display none
                select display backend type
                The default display is equivalent to
                "-display gtk"
-nographic      disable graphical output and redirect serial I/Os to console
-spice [port=port][,tls-port=secured-port][,x509-dir=<dir>]
       [,x509-key-file=<file>][,x509-key-password=<file>]
       [,x509-cert-file=<file>][,x509-cacert-file=<file>]
       [,x509-dh-key-file=<file>][,addr=addr]
       [,ipv4=on|off][,ipv6=on|off][,unix=on|off]
       [,tls-ciphers=<list>]
       [,tls-channel=[main|display|cursor|inputs|record|playback]]
       [,plaintext-channel=[main|display|cursor|inputs|record|playback]]
       [,sasl=on|off][,disable-ticketing=on|off]
       [,password=<string>][,password-secret=<secret-id>]
       [,image-compression=[auto_glz|auto_lz|quic|glz|lz|off]]
       [,jpeg-wan-compression=[auto|never|always]]
       [,zlib-glz-wan-compression=[auto|never|always]]
       [,streaming-video=[off|all|filter]][,disable-copy-paste=on|off]
       [,disable-agent-file-xfer=on|off][,agent-mouse=[on|off]]
       [,playback-compression=[on|off]][,seamless-migration=[on|off]]
       [,gl=[on|off]][,rendernode=<file>]
   enable spice
   at least one of {port, tls-port} is mandatory
-portrait       rotate graphical output 90 deg left (only PXA LCD)
-rotate <deg>   rotate graphical output some deg left (only PXA LCD)
-vga [std|cirrus|vmware|qxl|xenfb|tcx|cg3|virtio|none]
                select video card type
-full-screen    start in full screen
-vnc <display>  shorthand for -display vnc=<display>

i386 target only:
-win2k-hack     use it when installing Windows 2000 to avoid a disk full bug
-no-fd-bootchk  disable boot signature checking for floppy disks
-no-acpi        disable ACPI
-no-hpet        disable HPET
-acpitable [sig=str][,rev=n][,oem_id=str][,oem_table_id=str][,oem_rev=n][,asl_compiler_id=str][,asl_compiler_rev=n][,{data|file}=file1[:file2]...]
                ACPI table description
-smbios file=binary
                load SMBIOS entry from binary file
-smbios type=0[,vendor=str][,version=str][,date=str][,release=%d.%d]
              [,uefi=on|off]
                specify SMBIOS type 0 fields
-smbios type=1[,manufacturer=str][,product=str][,version=str][,serial=str]
              [,uuid=uuid][,sku=str][,family=str]
                specify SMBIOS type 1 fields
-smbios type=2[,manufacturer=str][,product=str][,version=str][,serial=str]
              [,asset=str][,location=str]
                specify SMBIOS type 2 fields
-smbios type=3[,manufacturer=str][,version=str][,serial=str][,asset=str]
              [,sku=str]
                specify SMBIOS type 3 fields
-smbios type=4[,sock_pfx=str][,manufacturer=str][,version=str][,serial=str]
              [,asset=str][,part=str][,max-speed=%d][,current-speed=%d]
              [,processor-id=%d]
                specify SMBIOS type 4 fields
-smbios type=8[,external_reference=str][,internal_reference=str][,connector_type=%d][,port_type=%d]
                specify SMBIOS type 8 fields
-smbios type=11[,value=str][,path=filename]
                specify SMBIOS type 11 fields
-smbios type=17[,loc_pfx=str][,bank=str][,manufacturer=str][,serial=str]
               [,asset=str][,part=str][,speed=%d]
                specify SMBIOS type 17 fields
-smbios type=41[,designation=str][,kind=str][,instance=%d][,pcidev=str]
                specify SMBIOS type 41 fields

Network options:
-netdev user,id=str[,ipv4=on|off][,net=addr[/mask]][,host=addr]
         [,ipv6=on|off][,ipv6-net=addr[/int]][,ipv6-host=addr]
         [,restrict=on|off][,hostname=host][,dhcpstart=addr]
         [,dns=addr][,ipv6-dns=addr][,dnssearch=domain][,domainname=domain]
         [,tftp=dir][,tftp-server-name=name][,bootfile=f][,hostfwd=rule][,guestfwd=rule][,smb=dir[,smbserver=addr]]
                configure a user mode network backend with ID 'str',
                its DHCP server and optional services
-netdev tap,id=str[,fd=h][,fds=x:y:...:z][,ifname=name][,script=file][,downscript=dfile]
         [,br=bridge][,helper=helper][,sndbuf=nbytes][,vnet_hdr=on|off][,vhost=on|off]
         [,vhostfd=h][,vhostfds=x:y:...:z][,vhostforce=on|off][,queues=n]
         [,poll-us=n]
                configure a host TAP network backend with ID 'str'
                connected to a bridge (default=br0)
                use network scripts 'file' (default=/etc/qemu-ifup)
                to configure it and 'dfile' (default=/etc/qemu-ifdown)
                to deconfigure it
                use '[down]script=no' to disable script execution
                use network helper 'helper' (default=/usr/lib/qemu/qemu-bridge-helper) to
                configure it
                use 'fd=h' to connect to an already opened TAP interface
                use 'fds=x:y:...:z' to connect to already opened multiqueue capable TAP interfaces
                use 'sndbuf=nbytes' to limit the size of the send buffer (the
                default is disabled 'sndbuf=0' to enable flow control set 'sndbuf=1048576')
                use vnet_hdr=off to avoid enabling the IFF_VNET_HDR tap flag
                use vnet_hdr=on to make the lack of IFF_VNET_HDR support an error condition
                use vhost=on to enable experimental in kernel accelerator
                    (only has effect for virtio guests which use MSIX)
                use vhostforce=on to force vhost on for non-MSIX virtio guests
                use 'vhostfd=h' to connect to an already opened vhost net device
                use 'vhostfds=x:y:...:z to connect to multiple already opened vhost net devices
                use 'queues=n' to specify the number of queues to be created for multiqueue TAP
                use 'poll-us=n' to specify the maximum number of microseconds that could be
                spent on busy polling for vhost net
-netdev bridge,id=str[,br=bridge][,helper=helper]
                configure a host TAP network backend with ID 'str' that is
                connected to a bridge (default=br0)
                using the program 'helper (default=/usr/lib/qemu/qemu-bridge-helper)
-netdev l2tpv3,id=str,src=srcaddr,dst=dstaddr[,srcport=srcport][,dstport=dstport]
         [,rxsession=rxsession],txsession=txsession[,ipv6=on|off][,udp=on|off]
         [,cookie64=on|off][,counter][,pincounter][,txcookie=txcookie]
         [,rxcookie=rxcookie][,offset=offset]
                configure a network backend with ID 'str' connected to
                an Ethernet over L2TPv3 pseudowire.
                Linux kernel 3.3+ as well as most routers can talk
                L2TPv3. This transport allows connecting a VM to a VM,
                VM to a router and even VM to Host. It is a nearly-universal
                standard (RFC3931). Note - this implementation uses static
                pre-configured tunnels (same as the Linux kernel).
                use 'src=' to specify source address
                use 'dst=' to specify destination address
                use 'udp=on' to specify udp encapsulation
                use 'srcport=' to specify source udp port
                use 'dstport=' to specify destination udp port
                use 'ipv6=on' to force v6
                L2TPv3 uses cookies to prevent misconfiguration as
                well as a weak security measure
                use 'rxcookie=0x012345678' to specify a rxcookie
                use 'txcookie=0x012345678' to specify a txcookie
                use 'cookie64=on' to set cookie size to 64 bit, otherwise 32
                use 'counter=off' to force a 'cut-down' L2TPv3 with no counter
                use 'pincounter=on' to work around broken counter handling in peer
                use 'offset=X' to add an extra offset between header and data
-netdev socket,id=str[,fd=h][,listen=[host]:port][,connect=host:port]
                configure a network backend to connect to another network
                using a socket connection
-netdev socket,id=str[,fd=h][,mcast=maddr:port[,localaddr=addr]]
                configure a network backend to connect to a multicast maddr and port
                use 'localaddr=addr' to specify the host address to send packets from
-netdev socket,id=str[,fd=h][,udp=host:port][,localaddr=host:port]
                configure a network backend to connect to another network
                using an UDP tunnel
-netdev stream,id=str[,server=on|off],addr.type=inet,addr.host=host,addr.port=port[,to=maxport][,numeric=on|off][,keep-alive=on|off][,mptcp=on|off][,addr.ipv4=on|off][,addr.ipv6=on|off]
-netdev stream,id=str[,server=on|off],addr.type=unix,addr.path=path[,abstract=on|off][,tight=on|off]
-netdev stream,id=str[,server=on|off],addr.type=fd,addr.str=file-descriptor
                configure a network backend to connect to another network
                using a socket connection in stream mode.
-netdev dgram,id=str,remote.type=inet,remote.host=maddr,remote.port=port[,local.type=inet,local.host=addr]
-netdev dgram,id=str,remote.type=inet,remote.host=maddr,remote.port=port[,local.type=fd,local.str=file-descriptor]
                configure a network backend to connect to a multicast maddr and port
                use ``local.host=addr`` to specify the host address to send packets from
-netdev dgram,id=str,local.type=inet,local.host=addr,local.port=port[,remote.type=inet,remote.host=addr,remote.port=port]
-netdev dgram,id=str,local.type=unix,local.path=path[,remote.type=unix,remote.path=path]
-netdev dgram,id=str,local.type=fd,local.str=file-descriptor
                configure a network backend to connect to another network
                using an UDP tunnel
-netdev vde,id=str[,sock=socketpath][,port=n][,group=groupname][,mode=octalmode]
                configure a network backend to connect to port 'n' of a vde switch
                running on host and listening for incoming connections on 'socketpath'.
                Use group 'groupname' and mode 'octalmode' to change default
                ownership and permissions for communication port.
-netdev vhost-user,id=str,chardev=dev[,vhostforce=on|off]
                configure a vhost-user network, backed by a chardev 'dev'
-netdev vhost-vdpa,id=str[,vhostdev=/path/to/dev][,vhostfd=h]
                configure a vhost-vdpa network,Establish a vhost-vdpa netdev
                use 'vhostdev=/path/to/dev' to open a vhost vdpa device
                use 'vhostfd=h' to connect to an already opened vhost vdpa device
-netdev hubport,id=str,hubid=n[,netdev=nd]
                configure a hub port on the hub with ID 'n'
-nic [tap|bridge|user|l2tpv3|vde|vhost-user|socket][,option][,...][mac=macaddr]
                initialize an on-board / default host NIC (using MAC address
                macaddr) and connect it to the given host network backend
-nic none       use it alone to have zero network devices (the default is to
                provided a 'user' network connection)
-net nic[,macaddr=mac][,model=type][,name=str][,addr=str][,vectors=v]
                configure or create an on-board (or machine default) NIC and
                connect it to hub 0 (please use -nic unless you need a hub)
-net [user|tap|bridge|vde|socket][,option][,option][,...]
                old way to initialize a host network interface
                (use the -netdev option if possible instead)

Character device options:
-chardev help
-chardev null,id=id[,mux=on|off][,logfile=PATH][,logappend=on|off]
-chardev socket,id=id[,host=host],port=port[,to=to][,ipv4=on|off][,ipv6=on|off][,nodelay=on|off]
         [,server=on|off][,wait=on|off][,telnet=on|off][,websocket=on|off][,reconnect=seconds][,mux=on|off]
         [,logfile=PATH][,logappend=on|off][,tls-creds=ID][,tls-authz=ID] (tcp)
-chardev socket,id=id,path=path[,server=on|off][,wait=on|off][,telnet=on|off][,websocket=on|off][,reconnect=seconds]
         [,mux=on|off][,logfile=PATH][,logappend=on|off][,abstract=on|off][,tight=on|off] (unix)
-chardev udp,id=id[,host=host],port=port[,localaddr=localaddr]
         [,localport=localport][,ipv4=on|off][,ipv6=on|off][,mux=on|off]
         [,logfile=PATH][,logappend=on|off]
-chardev msmouse,id=id[,mux=on|off][,logfile=PATH][,logappend=on|off]
-chardev vc,id=id[[,width=width][,height=height]][[,cols=cols][,rows=rows]]
         [,mux=on|off][,logfile=PATH][,logappend=on|off]
-chardev ringbuf,id=id[,size=size][,logfile=PATH][,logappend=on|off]
-chardev file,id=id,path=path[,mux=on|off][,logfile=PATH][,logappend=on|off]
-chardev pipe,id=id,path=path[,mux=on|off][,logfile=PATH][,logappend=on|off]
-chardev pty,id=id[,mux=on|off][,logfile=PATH][,logappend=on|off]
-chardev stdio,id=id[,mux=on|off][,signal=on|off][,logfile=PATH][,logappend=on|off]
-chardev braille,id=id[,mux=on|off][,logfile=PATH][,logappend=on|off]
-chardev serial,id=id,path=path[,mux=on|off][,logfile=PATH][,logappend=on|off]
-chardev tty,id=id,path=path[,mux=on|off][,logfile=PATH][,logappend=on|off]
-chardev parallel,id=id,path=path[,mux=on|off][,logfile=PATH][,logappend=on|off]
-chardev parport,id=id,path=path[,mux=on|off][,logfile=PATH][,logappend=on|off]
-chardev spicevmc,id=id,name=name[,debug=debug][,logfile=PATH][,logappend=on|off]
-chardev spiceport,id=id,name=name[,debug=debug][,logfile=PATH][,logappend=on|off]

TPM device options:
-tpmdev passthrough,id=id[,path=path][,cancel-path=path]
                use path to provide path to a character device; default is /dev/tpm0
                use cancel-path to provide path to TPM's cancel sysfs entry; if
                not provided it will be searched for in /sys/class/misc/tpm?/device
-tpmdev emulator,id=id,chardev=dev
                configure the TPM device using chardev backend

Boot Image or Kernel specific:
-bios file      set the filename for the BIOS
-pflash file    use 'file' as a parallel flash image
-kernel bzImage use 'bzImage' as kernel image
-append cmdline use 'cmdline' as kernel command line
-initrd file    use 'file' as initial ram disk
-dtb    file    use 'file' as device tree image

Debug/Expert options:
-compat [deprecated-input=accept|reject|crash][,deprecated-output=accept|hide]
                Policy for handling deprecated management interfaces
-compat [unstable-input=accept|reject|crash][,unstable-output=accept|hide]
                Policy for handling unstable management interfaces
-fw_cfg [name=]<name>,file=<file>
                add named fw_cfg entry with contents from file
-fw_cfg [name=]<name>,string=<str>
                add named fw_cfg entry with contents from string
-serial dev     redirect the serial port to char device 'dev'
-parallel dev   redirect the parallel port to char device 'dev'
-monitor dev    redirect the monitor to char device 'dev'
-qmp dev        like -monitor but opens in 'control' mode
-qmp-pretty dev like -qmp but uses pretty JSON formatting
-mon [chardev=]name[,mode=readline|control][,pretty[=on|off]]
-debugcon dev   redirect the debug console to char device 'dev'
-pidfile file   write PID to 'file'
-singlestep     always run in singlestep mode
--preconfig     pause QEMU before machine is initialized (experimental)
-S              freeze CPU at startup (use 'c' to start execution)
-overcommit [mem-lock=on|off][cpu-pm=on|off]
                run qemu with overcommit hints
                mem-lock=on|off controls memory lock support (default: off)
                cpu-pm=on|off controls cpu power management (default: off)
-gdb dev        accept gdb connection on 'dev'. (QEMU defaults to starting
                the guest without waiting for gdb to connect; use -S too
                if you want it to not start execution.)
-s              shorthand for -gdb tcp::1234
-d item1,...    enable logging of specified items (use '-d help' for a list of log items)
-D logfile      output log to logfile (default stderr)
-dfilter range,..  filter debug output to range of addresses (useful for -d cpu,exec,etc..)
-seed number       seed the pseudo-random number generator
-L path         set the directory for the BIOS, VGA BIOS and keymaps
-enable-kvm     enable KVM full virtualization support
-xen-domid id   specify xen guest domain id
-xen-attach     attach to existing xen domain
                libxl will use this when starting QEMU
-xen-domid-restrict     restrict set of available xen operations
                        to specified domain id. (Does not affect
                        xenpv machine type).
-no-reboot      exit instead of rebooting
-no-shutdown    stop before shutdown
-action reboot=reset|shutdown
                   action when guest reboots [default=reset]
-action shutdown=poweroff|pause
                   action when guest shuts down [default=poweroff]
-action panic=pause|shutdown|exit-failure|none
                   action when guest panics [default=shutdown]
-action watchdog=reset|shutdown|poweroff|inject-nmi|pause|debug|none
                   action when watchdog fires [default=reset]
-loadvm [tag|id]
                start right away with a saved state (loadvm in monitor)
-daemonize      daemonize QEMU after initializing
-option-rom rom load a file, rom, into the option ROM space
-rtc [base=utc|localtime|<datetime>][,clock=host|rt|vm][,driftfix=none|slew]
                set the RTC base and clock, enable drift fix for clock ticks (x86 only)
-icount [shift=N|auto][,align=on|off][,sleep=on|off][,rr=record|replay,rrfile=<filename>[,rrsnapshot=<snapshot>]]
                enable virtual instruction counter with 2^N clock ticks per
                instruction, enable aligning the host and virtual clocks
                or disable real time cpu sleeping, and optionally enable
                record-and-replay mode
-watchdog-action reset|shutdown|poweroff|inject-nmi|pause|debug|none
                action when watchdog fires [default=reset]
-echr chr       set terminal escape character instead of ctrl-a
-incoming tcp:[host]:port[,to=maxport][,ipv4=on|off][,ipv6=on|off]
-incoming rdma:host:port[,ipv4=on|off][,ipv6=on|off]
-incoming unix:socketpath
                prepare for incoming migration, listen on
                specified protocol and socket address
-incoming fd:fd
-incoming exec:cmdline
                accept incoming migration on given file descriptor
                or from given external command
-incoming defer
                wait for the URI to be specified via migrate_incoming
-only-migratable     allow only migratable devices
-nodefaults     don't create default devices
-chroot dir     chroot to dir just before starting the VM
-runas user     change to user id user just before starting the VM
                user can be numeric uid:gid instead
-sandbox on[,obsolete=allow|deny][,elevateprivileges=allow|deny|children]
          [,spawn=allow|deny][,resourcecontrol=allow|deny]
                Enable seccomp mode 2 system call filter (default 'off').
                use 'obsolete' to allow obsolete system calls that are provided
                    by the kernel, but typically no longer used by modern
                    C library implementations.
                use 'elevateprivileges' to allow or deny the QEMU process ability
                    to elevate privileges using set*uid|gid system calls.
                    The value 'children' will deny set*uid|gid system calls for
                    main QEMU process but will allow forks and execves to run unprivileged
                use 'spawn' to avoid QEMU to spawn new threads or processes by
                     blocking *fork and execve
                use 'resourcecontrol' to disable process affinity and schedular priority
-readconfig <file>
                read config file
-no-user-config
                do not load default user-provided config files at startup
-trace [[enable=]<pattern>][,events=<file>][,file=<file>]
                specify tracing options
-plugin [file=]<file>[,<argname>=<argvalue>]
                load a plugin
-async-teardown enable asynchronous teardown
-msg [timestamp[=on|off]][,guest-name=[on|off]]
                control error message format
                timestamp=on enables timestamps (default: off)
                guest-name=on enables guest name prefix but only if
                              -name guest option is set (default: off)
-dump-vmstate <file>
                Output vmstate information in JSON format to file.
                Use the scripts/vmstate-static-checker.py file to
                check for possible regressions in migration code
                by comparing two such vmstate dumps.
-enable-sync-profile
                enable synchronization profiling

Generic object creation:
-object TYPENAME[,PROP1=VALUE1,...]
                create a new object of type TYPENAME setting properties
                in the order they are specified.  Note that the 'id'
                property must be set.  These objects are placed in the
                '/objects' path.

During emulation, the following keys are useful:
ctrl-alt-f      toggle full screen
ctrl-alt-n      switch to virtual console 'n'
ctrl-alt        toggle mouse and keyboard grab

When using -nographic, press 'ctrl-a h' to get some help.

See <https://qemu.org/contribute/report-a-bug> for how to report bugs.
More information on the QEMU project at <https://qemu.org>.

Permissions

In order to run a KVM accelerated virtual machine without logging as root, add normal users to the kvm group. Replace <username> in the example command below with the appropriate user(s):

root #gpasswd -a <username> kvm

Creation of a disk image

To create a raw disk image with with 40G size:

user $qemu-img create -f raw my-systems-disk-image.img 40G

TO create a raw image with copy-on-write disabled (nocow): "nocow" is a file attribute. (check with lsattr)

user $qemu-img create -f raw my-systems-disk-image.img -o nocow=on 40G

This would create a qcow2 image (useful if your filesystem doesn't support sparse files):

user $qemu-img create -f qcow2 my-systems-disk-image.qcow2 40G

Preparation of a bootable disk image from scratch

A system can be copied on to a disk image when not using a cdrom installation medium. By default qemu uses a "bios-firmware" to boot the system. The disk can be prepared with a msdos disklabel and a gap between the end of the 512 byte MBR (Master Boot Record) and the start of the first partition. The gap is needed for boot loaders like grub that place boot-code in the gap.

A raw disk image can be prepared by attaching it as a loop device:

user $losetup -fP /path/to/my-systems-disk-image.img
  • -f find the first unused loop device
  • -P scans for the partitions

List the loop devices with this command:

user $losetup -l

Then the loop device can be formatted like a normal disk.
Print the partition table:

user $parted /dev/loop000 print

Create a msdos disklabel with:

user $parted /dev/loop000-number-of-the-device-whose-data-will-be-lost mklabel msdos

Create an ext4 partition:

user $parted /dev/loop000 mkpart primary ext4 1Mib 40GiB

Set the boot flag:

user $parted /dev/loop000 set 1 boot on

Create a filesystem:

user $mkfs.ext4 /dev/loop000

Mount it somewhere

user $mount /dev/loop000 /mnt/my-new-fs

Create a boot/grub folder for grub.

user $mkdir -p /mnt/my-new-fs/boot/grub

Install grub on the loop device and advice grub to install its files in boot/grub

user $grub-install --boot-directory=/mnt/my-new-fs/boot/grub /dev/loop000

Unmount the filesystem and detach the loop device

user $umount /mnt/my-new-fs
user $losetup -d /dev/loop000

If the loop device is busy it will not return an error. This is verified with

user $losetup -l

This is enough to boot into a grub2 boot prompt. This is can be used as the basis for a bootable system.

CPU selection

QEMU has "accelerators" like kvm(Kernel Virtual Machine) or tcg (Tiny Code Generator) or Xen (wikip[2]).

The accelerator can usually only "accelerate" the features that are available on the host cpu. So the selection of the cpu affects the performance.

To get a list of cpus:

user $qemu-system-x86_64 -cpu help
Available CPUs:
x86 486                   (alias configured by machine type)
x86 486-v1                
x86 Broadwell             (alias configured by machine type)
x86 Broadwell-IBRS        (alias of Broadwell-v3)
x86 Broadwell-noTSX       (alias of Broadwell-v2)
x86 Broadwell-noTSX-IBRS  (alias of Broadwell-v4)
x86 Broadwell-v1          Intel Core Processor (Broadwell)
x86 Broadwell-v2          Intel Core Processor (Broadwell, no TSX)
x86 Broadwell-v3          Intel Core Processor (Broadwell, IBRS)
x86 Broadwell-v4          Intel Core Processor (Broadwell, no TSX, IBRS)
x86 Cascadelake-Server    (alias configured by machine type)
x86 Cascadelake-Server-noTSX  (alias of Cascadelake-Server-v3)
x86 Cascadelake-Server-v1  Intel Xeon Processor (Cascadelake)
x86 Cascadelake-Server-v2  Intel Xeon Processor (Cascadelake) [ARCH_CAPABILITIES]
x86 Cascadelake-Server-v3  Intel Xeon Processor (Cascadelake) [ARCH_CAPABILITIES, no TSX]
x86 Cascadelake-Server-v4  Intel Xeon Processor (Cascadelake) [ARCH_CAPABILITIES, no TSX]
x86 Cascadelake-Server-v5  Intel Xeon Processor (Cascadelake) [ARCH_CAPABILITIES, EPT switching, XSAVES, no TSX]
x86 Conroe                (alias configured by machine type)
x86 Conroe-v1             Intel Celeron_4x0 (Conroe/Merom Class Core 2)
x86 Cooperlake            (alias configured by machine type)
x86 Cooperlake-v1         Intel Xeon Processor (Cooperlake)
x86 Cooperlake-v2         Intel Xeon Processor (Cooperlake) [XSAVES]
x86 Denverton             (alias configured by machine type)
x86 Denverton-v1          Intel Atom Processor (Denverton)
x86 Denverton-v2          Intel Atom Processor (Denverton) [no MPX, no MONITOR]
x86 Denverton-v3          Intel Atom Processor (Denverton) [XSAVES, no MPX, no MONITOR]
x86 Dhyana                (alias configured by machine type)
x86 Dhyana-v1             Hygon Dhyana Processor
x86 Dhyana-v2             Hygon Dhyana Processor [XSAVES]
x86 EPYC                  (alias configured by machine type)
x86 EPYC-IBPB             (alias of EPYC-v2)
x86 EPYC-Milan            (alias configured by machine type)
x86 EPYC-Milan-v1         AMD EPYC-Milan Processor
x86 EPYC-Rome             (alias configured by machine type)
x86 EPYC-Rome-v1          AMD EPYC-Rome Processor
x86 EPYC-Rome-v2          AMD EPYC-Rome Processor
x86 EPYC-v1               AMD EPYC Processor
x86 EPYC-v2               AMD EPYC Processor (with IBPB)
x86 EPYC-v3               AMD EPYC Processor
x86 Haswell               (alias configured by machine type)
x86 Haswell-IBRS          (alias of Haswell-v3)
x86 Haswell-noTSX         (alias of Haswell-v2)
x86 Haswell-noTSX-IBRS    (alias of Haswell-v4)
x86 Haswell-v1            Intel Core Processor (Haswell)
x86 Haswell-v2            Intel Core Processor (Haswell, no TSX)
x86 Haswell-v3            Intel Core Processor (Haswell, IBRS)
x86 Haswell-v4            Intel Core Processor (Haswell, no TSX, IBRS)
x86 Icelake-Server        (alias configured by machine type)
x86 Icelake-Server-noTSX  (alias of Icelake-Server-v2)
x86 Icelake-Server-v1     Intel Xeon Processor (Icelake)
x86 Icelake-Server-v2     Intel Xeon Processor (Icelake) [no TSX]
x86 Icelake-Server-v3     Intel Xeon Processor (Icelake)
x86 Icelake-Server-v4     Intel Xeon Processor (Icelake)
x86 Icelake-Server-v5     Intel Xeon Processor (Icelake) [XSAVES]
x86 Icelake-Server-v6     Intel Xeon Processor (Icelake) [5-level EPT]
x86 IvyBridge             (alias configured by machine type)
x86 IvyBridge-IBRS        (alias of IvyBridge-v2)
x86 IvyBridge-v1          Intel Xeon E3-12xx v2 (Ivy Bridge)
x86 IvyBridge-v2          Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS)
x86 KnightsMill           (alias configured by machine type)
x86 KnightsMill-v1        Intel Xeon Phi Processor (Knights Mill)
x86 Nehalem               (alias configured by machine type)
x86 Nehalem-IBRS          (alias of Nehalem-v2)
x86 Nehalem-v1            Intel Core i7 9xx (Nehalem Class Core i7)
x86 Nehalem-v2            Intel Core i7 9xx (Nehalem Core i7, IBRS update)
x86 Opteron_G1            (alias configured by machine type)
x86 Opteron_G1-v1         AMD Opteron 240 (Gen 1 Class Opteron)
x86 Opteron_G2            (alias configured by machine type)
x86 Opteron_G2-v1         AMD Opteron 22xx (Gen 2 Class Opteron)
x86 Opteron_G3            (alias configured by machine type)
x86 Opteron_G3-v1         AMD Opteron 23xx (Gen 3 Class Opteron)
x86 Opteron_G4            (alias configured by machine type)
x86 Opteron_G4-v1         AMD Opteron 62xx class CPU
x86 Opteron_G5            (alias configured by machine type)
x86 Opteron_G5-v1         AMD Opteron 63xx class CPU
x86 Penryn                (alias configured by machine type)
x86 Penryn-v1             Intel Core 2 Duo P9xxx (Penryn Class Core 2)
x86 SandyBridge           (alias configured by machine type)
x86 SandyBridge-IBRS      (alias of SandyBridge-v2)
x86 SandyBridge-v1        Intel Xeon E312xx (Sandy Bridge)
x86 SandyBridge-v2        Intel Xeon E312xx (Sandy Bridge, IBRS update)
x86 Skylake-Client        (alias configured by machine type)
x86 Skylake-Client-IBRS   (alias of Skylake-Client-v2)
x86 Skylake-Client-noTSX-IBRS  (alias of Skylake-Client-v3)
x86 Skylake-Client-v1     Intel Core Processor (Skylake)
x86 Skylake-Client-v2     Intel Core Processor (Skylake, IBRS)
x86 Skylake-Client-v3     Intel Core Processor (Skylake, IBRS, no TSX)
x86 Skylake-Client-v4     Intel Core Processor (Skylake, IBRS, no TSX) [IBRS, XSAVES, no TSX]
x86 Skylake-Server        (alias configured by machine type)
x86 Skylake-Server-IBRS   (alias of Skylake-Server-v2)
x86 Skylake-Server-noTSX-IBRS  (alias of Skylake-Server-v3)
x86 Skylake-Server-v1     Intel Xeon Processor (Skylake)
x86 Skylake-Server-v2     Intel Xeon Processor (Skylake, IBRS)
x86 Skylake-Server-v3     Intel Xeon Processor (Skylake, IBRS, no TSX)
x86 Skylake-Server-v4     Intel Xeon Processor (Skylake, IBRS, no TSX)
x86 Skylake-Server-v5     Intel Xeon Processor (Skylake, IBRS, no TSX) [IBRS, XSAVES, EPT switching, no TSX]
x86 Snowridge             (alias configured by machine type)
x86 Snowridge-v1          Intel Atom Processor (SnowRidge)
x86 Snowridge-v2          Intel Atom Processor (Snowridge, no MPX)
x86 Snowridge-v3          Intel Atom Processor (Snowridge, no MPX) [XSAVES, no MPX]
x86 Snowridge-v4          Intel Atom Processor (Snowridge, no MPX) [no split lock detect, no core-capability]
x86 Westmere              (alias configured by machine type)
x86 Westmere-IBRS         (alias of Westmere-v2)
x86 Westmere-v1           Westmere E56xx/L56xx/X56xx (Nehalem-C)
x86 Westmere-v2           Westmere E56xx/L56xx/X56xx (IBRS update)
x86 athlon                (alias configured by machine type)
x86 athlon-v1             QEMU Virtual CPU version 2.5+
x86 core2duo              (alias configured by machine type)
x86 core2duo-v1           Intel(R) Core(TM)2 Duo CPU     T7700  @ 2.40GHz
x86 coreduo               (alias configured by machine type)
x86 coreduo-v1            Genuine Intel(R) CPU           T2600  @ 2.16GHz
x86 kvm32                 (alias configured by machine type)
x86 kvm32-v1              Common 32-bit KVM processor
x86 kvm64                 (alias configured by machine type)
x86 kvm64-v1              Common KVM processor
x86 n270                  (alias configured by machine type)
x86 n270-v1               Intel(R) Atom(TM) CPU N270   @ 1.60GHz
x86 pentium               (alias configured by machine type)
x86 pentium-v1            
x86 pentium2              (alias configured by machine type)
x86 pentium2-v1           
x86 pentium3              (alias configured by machine type)
x86 pentium3-v1           
x86 phenom                (alias configured by machine type)
x86 phenom-v1             AMD Phenom(tm) 9550 Quad-Core Processor
x86 qemu32                (alias configured by machine type)
x86 qemu32-v1             QEMU Virtual CPU version 2.5+
x86 qemu64                (alias configured by machine type)
x86 qemu64-v1             QEMU Virtual CPU version 2.5+
x86 base                  base CPU model type with no features enabled
x86 host                  processor with all supported host features 
x86 max                   Enables all features supported by the accelerator in the current host

Recognized CPUID flags:
  3dnow 3dnowext 3dnowprefetch abm ace2 ace2-en acpi adx aes amd-no-ssb
  amd-ssbd amd-stibp amx-bf16 amx-int8 amx-tile apic arat arch-capabilities
  arch-lbr avic avx avx-vnni avx2 avx512-4fmaps avx512-4vnniw avx512-bf16
  avx512-fp16 avx512-vp2intersect avx512-vpopcntdq avx512bitalg avx512bw
  avx512cd avx512dq avx512er avx512f avx512ifma avx512pf avx512vbmi
  avx512vbmi2 avx512vl avx512vnni bmi1 bmi2 bus-lock-detect cid cldemote
  clflush clflushopt clwb clzero cmov cmp-legacy core-capability cr8legacy
  cx16 cx8 dca de decodeassists ds ds-cpl dtes64 erms est extapic f16c
  flushbyasid fma fma4 fpu fsgsbase fsrm full-width-write fxsr fxsr-opt
  gfni hle ht hypervisor ia64 ibpb ibrs ibrs-all ibs intel-pt intel-pt-lip
  invpcid invtsc kvm-asyncpf kvm-asyncpf-int kvm-hint-dedicated kvm-mmu
  kvm-msi-ext-dest-id kvm-nopiodelay kvm-poll-control kvm-pv-eoi kvm-pv-ipi
  kvm-pv-sched-yield kvm-pv-tlb-flush kvm-pv-unhalt kvm-steal-time kvmclock
  kvmclock kvmclock-stable-bit la57 lahf-lm lbrv lm lwp mca mce md-clear
  mds-no misalignsse mmx mmxext monitor movbe movdir64b movdiri mpx msr
  mtrr nodeid-msr npt nrip-save nx osvw pae pat pause-filter pbe pcid
  pclmulqdq pcommit pdcm pdpe1gb perfctr-core perfctr-nb pfthreshold pge
  phe phe-en pks pku pmm pmm-en pn pni popcnt pschange-mc-no pse pse36
  rdctl-no rdpid rdrand rdseed rdtscp rsba rtm sep serialize sgx sgx-debug
  sgx-exinfo sgx-kss sgx-mode64 sgx-provisionkey sgx-tokenkey sgx1 sgx2
  sgxlc sha-ni skinit skip-l1dfl-vmentry smap smep smx spec-ctrl
  split-lock-detect ss ssb-no ssbd sse sse2 sse4.1 sse4.2 sse4a ssse3 stibp
  svm svm-lock svme-addr-chk syscall taa-no tbm tce tm tm2 topoext tsc
  tsc-adjust tsc-deadline tsc-scale tsx-ctrl tsx-ldtrk umip v-vmsave-vmload
  vaes vgif virt-ssbd vmcb-clean vme vmx vmx-activity-hlt
  vmx-activity-shutdown vmx-activity-wait-sipi vmx-apicv-register
  vmx-apicv-vid vmx-apicv-x2apic vmx-apicv-xapic vmx-cr3-load-noexit
  vmx-cr3-store-noexit vmx-cr8-load-exit vmx-cr8-store-exit vmx-desc-exit
  vmx-encls-exit vmx-entry-ia32e-mode vmx-entry-load-bndcfgs
  vmx-entry-load-efer vmx-entry-load-pat vmx-entry-load-perf-global-ctrl
  vmx-entry-load-pkrs vmx-entry-load-rtit-ctl vmx-entry-noload-debugctl
  vmx-ept vmx-ept-1gb vmx-ept-2mb vmx-ept-advanced-exitinfo
  vmx-ept-execonly vmx-eptad vmx-eptp-switching vmx-exit-ack-intr
  vmx-exit-clear-bndcfgs vmx-exit-clear-rtit-ctl vmx-exit-load-efer
  vmx-exit-load-pat vmx-exit-load-perf-global-ctrl vmx-exit-load-pkrs
  vmx-exit-nosave-debugctl vmx-exit-save-efer vmx-exit-save-pat
  vmx-exit-save-preemption-timer vmx-flexpriority vmx-hlt-exit vmx-ins-outs
  vmx-intr-exit vmx-invept vmx-invept-all-context vmx-invept-single-context
  vmx-invept-single-context vmx-invept-single-context-noglobals
  vmx-invlpg-exit vmx-invpcid-exit vmx-invvpid vmx-invvpid-all-context
  vmx-invvpid-single-addr vmx-io-bitmap vmx-io-exit vmx-monitor-exit
  vmx-movdr-exit vmx-msr-bitmap vmx-mtf vmx-mwait-exit vmx-nmi-exit
  vmx-page-walk-4 vmx-page-walk-5 vmx-pause-exit vmx-ple vmx-pml
  vmx-posted-intr vmx-preemption-timer vmx-rdpmc-exit vmx-rdrand-exit
  vmx-rdseed-exit vmx-rdtsc-exit vmx-rdtscp-exit vmx-secondary-ctls
  vmx-shadow-vmcs vmx-store-lma vmx-true-ctls vmx-tsc-offset
  vmx-tsc-scaling vmx-unrestricted-guest vmx-vintr-pending vmx-vmfunc
  vmx-vmwrite-vmexit-fields vmx-vnmi vmx-vnmi-pending vmx-vpid
  vmx-wbinvd-exit vmx-xsaves vmx-zero-len-inject vpclmulqdq waitpkg
  wbnoinvd wdt x2apic xcrypt xcrypt-en xfd xgetbv1 xop xsave xsavec
  xsaveerptr xsaveopt xsaves xstore xstore-en xtpr

Show the available accelerators:

user $qemu-system-x86_64 -accel help
Accelerators supported in QEMU binary:
tcg
kvm
hvf
whpx
haxm


Starting QEMU

This is how to start a virtual machine with the same feature set as the host cpu, a raw disk image and 2G of ram. By default a vnc server is started that runs with no password protection and listens on the loop interface. QEMU is advised to listen on a local UNIX socket with the following command. Set the file permissions appropriately to protect the VNC server from unauthorized access. A cdrom installation and boot medium is added with "-cdrom filename.img"

user $qemu-system-x86_64 -vnc unix:/home/user/.qemu-vnc-socket -cpu host -drive file=/var/virt/rootfs-build-tc,format=raw -m 2G
Warning
Starting the server with -vnc :0 it listens on port 5900 (first display) on all interfaces with no password protection.


Troubleshooting

"kvm: already loaded the other module"

Sometimes during the early boot splash the error message "kvm: already loaded the other module" can be seen. This message indicates both the Intel and the AMD kernel virtual machine settings have been enabled in the kernel. To fix this, enable as a module or disable either the Intel or AMD KVM option specific to the system's processor in the kernel configuration. For example, if the system has an Intel processor enable the Intel KVM, then make sure the AMD KVM is set as a module (M) or is disabled (N). The relevant options to enable or disable can be found in the kernel's .config file via the CONFIG_KVM_INTEL and CONFIG_KVM_AMD variables or in the configuration section above.

Creating TUN/TAP device - No such file or directory

Sometimes this error can occur if TUN/TAP support cannot be found in the kernel. To solve this, try loading the driver:

root #modprobe tun

If that works, add this to a file in /etc/modules-load.d/ to load on startup:

FILE /etc/modules-load.d/qemu-modules.conf
tun

Configuration does not support video model 'qxl'

This is usually the case if QEMU is not built with the spice USE flag. To resolve this issue, try to build QEMU with the correct USE flag.

First add spice to via a package.use file:

FILE /etc/portage/package.use/qemu
app-emulation/qemu spice

Then rebuild the package:

root #emerge --ask app-emulation/qemu

My qemu has kvm support on some guest architectures

KVM works only for the same architecture. An ARM64 host cannot handle x86_64 instructions.

Invalid context errors on SELinux systems

By default, Libvirt generates a random SELinux MCS label for the QEMU process when it is started. If the loaded SELinux policy does not support MCS categories, the resulting security context will be invalid:

CODE SELinux error from virt-manager
Error starting domain: unable to set socket security context 'system_u:system_r:svirt_t:s0:c123,c456': Invalid argument
CODE SELinux error from the kernel
kernel: SELinux:  Context system_u:object_r:svirt_image_t:s0:c123,c456 is not valid (left unmapped).

The solution is either to switch to one of the policy types which supports MCS categories or manually set the virtual machine's security labels, without MCS categories:

CODE Libvirt domain XML with manually specified seclabel fields
<domain type="kvm">
  <name>fedora</name>
  ...
  <devices>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2"/>
      <source file="/var/lib/libvirt/images/fedora.qcow2">
        <seclabel model='selinux' relabel='yes'>
          <label>system_u:object_r:svirt_image_t</label>
        </seclabel>
      </source>
      <target dev="vda" bus="virtio"/>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </disk>
  ...
  <seclabel type='static' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t</label>
  </seclabel>
</domain>

Static-user and LTO

GCC will use huge amount of RAM when LTO is enabled on the system if any of the ppc64 options are enabled while using the static-user flag, because of this is recommended to disable LTO while compiling in this configuration or use clang if LTO is required. See bug #883419


lto1: internal compiler error: original not compressed with zstd

This is caused by a mismatch of GCC used to compile zlib and glib to the one being used to compile qemu, this can be fixed by rebuilding both before compiling qemu again.

root #emerge --ask sys-libs/zlib dev-libs/glib

BSOD when booting Windows 10

Create this file:

FILE /etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1

and restart the system.

Removal

There may be image files left behind after removal of QEMU package.

Unmerge

root #emerge --ask --depclean --verbose app-emulation/qemu


See also


External resources

References

  1. ↑ https://en.wikipedia.org/wiki/Full_virtualization
  2. ↑ https://developer.apple.com/documentation/hypervisor
  3. ↑ https://en.wikipedia.org/wiki/Paravirtualization
  4. ↑ https://github.com/RceNinja/notes/blob/master/notes/build_qemu_with_enabled_hyper-v_acceleration_(whpx)_on_windows.md
  5. ↑ https://en.wikipedia.org/wiki/Paravirtualization
  6. ↑ QEMU / KVM CPU model configuration
  7. ↑ https://github.com/intel/haxm
  8. ↑ https://en.wikipedia.org/wiki/Paravirtualization
Retrieved from "https://wiki.gentoo.org/index.php?title=QEMU&oldid=1314095"