Portage TMPDIR on tmpfs

Important
It is unlikely that tmpfs will provide any performance gain for modern systems that have -pipe set in the CFLAGS environment variable, even for systems with a spinning drive.

When emerging packages it is possible to build them in tmpfs (RAM) space instead of having build files pushed and pulled to Hard Disk Drive (HDD) or Solid State Drive (SSD) space. Building packages in tmpfs might speed up emerge times. It will, however, reduce SSD wear and fragmentation within the file system.

Still, the mentioned (might-be)-benefits should never be the primary basis for implementing such a system, because tmpfs (or zram) as Portage TMPDIR adds complexity and has additional caveats: firstly, additional steps are needed to set it up. Secondly, constant mind-fullness is absolutely necessary during operation, e.g. to prevent running out of disk space (which is, in the case of tmpfs: RAM) and out of memory (RAM). Most importantly it is impossible to be certain if there even is a benefit at all, because all of this is very hard to quantify. For example, modern filesystems are designed to prevent fragmentation, and modern operating systems use RAM as disk cache which prevents unnecessary writes and thereby wear (meaning: the more RAM, the more caching, the less wear). Contrary, if the tmpfs is set up to use too much RAM this will in turn also cause more swapping, and this will mean additional wear on an SSD if it is used for swap space.

Generally speaking, unless the system has a large amount of RAM, it may be more of a hassle to setup Portage in tmpfs. Larger packages such as sys-devel/gcc (see below) will fail on less then 4 GBs of tmpfs. Keep this in mind when proceeding!

Note
On slow systems tmpfs is the fastest, but RAM usage is high. On fast systems it might be a good alternative to setup "Portage TMPDIR on zram" instead, where the choice of compression-algorithm is a tradeoff between preserving RAM space and performance. Especially source files are well compressible; at the same configured size, running out of space is less likely with zram as it would be with uncompressed tmpfs. E.g. for profiles with systemd, see Using systemd zram-generator for an example configuration...

Configuration

Considering tmpfs size

The system's tmpfs space should be large enough to handle the largest packages to be compiled on the system. If the tmpfs space were to ever become completely full then the emerge will fail. Most packages would not need more than 1 GB for compilation, but there are a few that are very large and would need more. Those still wanting to compile these packages on tmpfs should verify enough free tmpfs space exists. The following list contains estimates of how much space is allocated on each package. Some of these are based on the minimum space requirements specified in the ebuilds. Note that the actual allocated size may vary depending on the features included when building the package, and it may also vary also on every version update.

An example of a size check failure:

* There is NOT at least 13 GiB disk space at "/var/tmp/portage/www-client/firefox-81.0.1/temp"
*
* Space constraints set in the ebuild were not met!
* The build will most probably fail, you should enhance the space
* as per failed tests.

Package	Memory usage (uncompressed)
app-office/libreoffice	~6 GB with 512 MB of extra system memory
dev-db/mysql	8 GB (versions >= 8.0.19-r1)
dev-lang/ghc	> 4 GB (will fail with less)
dev-lang/mono	> 4 GB (will fail with less)
dev-lang/rust	~7 GB; ~10 GB if {C,CXX,F,FC}FLAGS has -ggdb set
dev-lang/spidermonkey	> 6.4 GB (will fail with less)
dev-qt/qtwebengine	Builds a fork of Chromium in the background: ~10 GB with 3 GB of extra system memory
mail-client/thunderbird	Approximately the same as Firefox
sci-libs/tensorflow	~5 GB per Python target, e.g. if targeting both Python 3.6 and Python 3.7, it will require ~10 GB
sys-devel/clang	8.8 GB
sys-devel/gcc	> 4 GB (will fail with less), even more if Java and Objective C are also included
sys-devel/llvm	8 GB
www-client/chromium	~10 GB with 3 GB of extra system memory
www-client/firefox	~4.5 GB; ~13 GB if the pgo, debug, or test USE flag has been enabled

When using ccache to assist in resuming compiles, it should be noted an equal size of the /var/tmp and /var/tmp/portage directories is necessary. Alternatively the per-package choices as shown below for large packages requiring large amounts of space can be implemented.

ccache creates a directory in /var/cache/ccache to store compiled elements for resuming.

fstab

Mount Portage's TMPDIR (/var/tmp/portage is where Portage stores files while building packages. ) to tmpfs by adding the following to the system's /etc/fstab config file:

FILE /etc/fstabtmpfs fstab example

tmpfs		/var/tmp/portage		tmpfs	size=4G,uid=portage,gid=portage,mode=775	0 0

Note
If using SELinux, the following option can be added: rootcontext=system_u:object_r:portage_tmp_t.

The size parameter can be adjusted in /etc/fstab to set the max size of the tmpfs, limiting RAM usage. Systems with large amounts of RAM can increase the value quite significantly.

Note
If mounting Portage's tmp dir as a tmpfs, there is no need to change Portage's PORTAGE_TMPDIR in /etc/portage/make.conf.

Important
If setting PORTAGE_TMPDIR and using a different path, Portage will automatically add /portage to the specified path. Setting this to /tmp actually uses /tmp/portage so the tmpfs should be mounted there.

After /etc/fstab has been modified, mount Portage's TMPDIR to RAM by running the mount command followed by the directory location outline in fstab (this does not need to be done manually after every boot. After a reboot /var/tmp/portage should be mounted by /etc/fstab automatically):

root #mount /var/tmp/portage

In the unlikely event that the entire /var/tmp/ directory is already mounted as tmpfs, it can be worked around by the special x-mount.mkdir mount option:

FILE /etc/fstabtmpfs fstab example

tmpfs /var/tmp         tmpfs rw,nosuid,nodev,size=4G,mode=1777 0 0
tmpfs /var/tmp/portage tmpfs rw,nosuid,nodev,size=4G,mode=775,uid=portage,gid=portage,x-mount.mkdir=775 0 0

Per-package choices at compile time

Portage can be configured to build large packages outside of the tmpfs space on a per-package basis.

Create a file to tell Portage where to place the temporary files directory:

FILE /etc/portage/env/notmpfs.conf

PORTAGE_TMPDIR="/var/tmp/notmpfs"

Create a separate temporary file directory outside of the tmpfs mount location:

root #mkdir /var/tmp/notmpfs

root #chown portage:portage /var/tmp/notmpfs

root #chmod 775 /var/tmp/notmpfs

Create a special Portage file called package.env in /etc/portage/ and list all the packages that are too large to be compiled using tmpfs:

FILE /etc/portage/package.env

app-office/libreoffice		notmpfs.conf
dev-lang/ghc			notmpfs.conf
dev-lang/mono			notmpfs.conf
dev-lang/rust			notmpfs.conf
dev-lang/spidermonkey		notmpfs.conf
mail-client/thunderbird		notmpfs.conf
sys-devel/clang                 notmpfs.conf
sys-devel/gcc			notmpfs.conf
sys-devel/llvm                  notmpfs.conf
www-client/chromium		notmpfs.conf
www-client/firefox		notmpfs.conf
dev-qt/qtwebengine              notmpfs.conf
games-strategy/0ad              notmpfs.conf

Tips

Resizing tmpfs

To resize the current tmpfs instance in /var/tmp/portage, run:

root #mount -o remount,size=N /var/tmp/portage

Where N is in the form of bytes. It can also be suffixed with k, m, or g to respectively have the form of (k)ilobytes, (m)egabytes or (g)igabytes. It can also be suffixed with a % to limit the tmpfs instance to the percentage of current physical RAM, the default being 50% when the parameter is not specified.

The resized tmpfs will not persist to the next boot unless the size parameter is modified in /etc/fstab. This is not necessary since a larger tmpfs is only needed during large package compilations.

It is recommended to leave-out at least 1 GB of space for the system to prevent out-of-memory problems. Using swap-disks for some heavy compile-time and link-time instances which are unexpected may also be helpful. Now even if swap-disks are used, reads and writes to it would only be minimal compared to having a physical filesystem behind /var/tmp/portage.

Here is a note about the size parameter in Linux kernel's documentation which can be found in /usr/src/linux/Documentation/filesystems/tmpfs.txt as long as a kernel has been emerged:

FILE tmpfs.txtTMPFS information

size: The limit of allocated bytes for this tmpfs instance. The default is half of your physical RAM without swap. If you oversize your tmpfs instances the machine will deadlock since the OOM handler will not be able to free that memory.

Besides the obvious danger of choking the system by allocating too much memory for tmpfs space, it should be generally safe to enlarge the tmpfs during an emerge as this would only increase the size limit of the tmpfs without destroying any data from the emerge process.

For example, if a system has 12 GB of RAM and 3 disks with 2 GB of swap space working in parallel on each disk, then it would be pretty safe to choose size limit equal to 16G. 16 GB size is usually enough to compile Libreoffice and Chromium in parallel (usual emerge -1uDN @world) while reading Internet in a web browser.

It's not often that you'll ever have to do it and emerge would tell you that tmpfs is too small however there are instances that the package's ebuild would be not accurate at estimating the amount of disk space necessary for building the package. Newer packages may end up allocating more space, whereas using lesser USE flags would make it allocate less.

The solution for this is to either enlarge tmpfs, or add the exception to /etc/portage/package.env, and then run emerge again.

Save an emerge and resume later

Note
This is experimental. ebuild experts should be queried about how reliable this command is; when to use it and when not to. bug #549580 tracks possibly doing this automatically in Portage.

Example: emerging webkit-gtk can take a long time. I want to reboot into another OS and resume this ebuild later.

Optional: I use app-portage/genlop to inspect the current emerge session. I like using it to remind me of the ebuild version number or hopefully to get an estimated time remaining.

user $genlop -c

Currently merging 1 out of 2

 * net-libs/webkit-gtk-2.4.8 

       current merge time: 4 hours, 27 minutes and 35 seconds.
       ETA: unknown.

Press Ctrl+c to quit the current emerge session.

Since I am rebooting, I'll have to use cp -a or tar -cpf to save /var/tmp/portage/* while preserving permissions. Otherwise the tmpfs contents will be lost; You may want to inspect the memory size of /var/tmp/portage by using du:

root #du -sh /var/tmp/portage/

251M	/var/tmp/portage/

Reboot, do other stuff, come back later.

Restore /var/tmp/portage/*.

Resume the ebuild with ebuild <repository_directory>/<category>/<package_name>-<version>.ebuild merge:

root #ebuild /var/db/repos/gentoo/net-libs/webkit-gtk/webkit-gtk-2.6.5.ebuild merge

>>> Existing ${T}/environment for 'webkit-gtk-2.6.5' will be sourced. Run
>>> 'clean' to start with a fresh environment.
>>> Checking webkitgtk-2.6.5.tar.xz's mtime...
>>> WORKDIR is up-to-date, keeping...
 * checking ebuild checksums ;-) ...                                                                                                                   [ ok ]
 * checking auxfile checksums ;-) ...                                                                                                                  [ ok ]
 * checking miscfile checksums ;-) ...                                                                                                                 [ ok ]
>>> It appears that 'setup' has already executed for 'webkit-gtk-2.6.5'; skipping.
>>> Remove '/var/tmp/portage/net-libs/webkit-gtk-2.6.5/.setuped' to force setup.
>>> It appears that 'unpack' has already executed for 'webkit-gtk-2.6.5'; skipping.
>>> Remove '/var/tmp/portage/net-libs/webkit-gtk-2.6.5/.unpacked' to force unpack.
>>> It appears that 'prepare' has already executed for 'webkit-gtk-2.6.5'; skipping.
>>> Remove '/var/tmp/portage/net-libs/webkit-gtk-2.6.5/.prepared' to force prepare.
>>> Configuring source in /var/tmp/portage/net-libs/webkit-gtk-2.6.5/work/webkitgtk-2.6.5 ...
>>> Working in BUILD_DIR: "/var/tmp/portage/net-libs/webkit-gtk-2.6.5/work/webkit-gtk-2.6.5_build"
..
..
.

If you're using other repository sources besides gentoo like layman overlays, make sure that you're using the correct repository directory of the ebuild as one package can also belong to other repositories and be chosen to be installed over the one in gentoo. You can get the repository name of the current package by reading the last action entry in /var/log/emerge.log or reading the build.log file in the package's build directory with a command like:

user $fgrep Repository: /var/tmp/portage/net-libs/webkit-gtk-2.6.5/temp/build.log

Do not use the .ebuild file found in /var/tmp/portage/<category>/<package_name>-<version>/build-info/<package_name>-<version>.ebuild as it seems to be only a reference. Perhaps there's a way to use it, but one would have to thoroughly understand how ebuild and ebuild.sh work.

Happy hacking!

Troubleshooting

No space left on device

If you encounter a not-enough space error or anything similar, there are basically two things to do:

1. Check the /var/tmp/portage directory for old package directories from previously failed compiles. Any packages found therein should be deleted; with exceptions made for any failed packages the user would like to resume compiling later.
2. Resize the tmpfs.
3. In case you still get messages related to exhausted disk space during emerge, even though the allocated tmpfs size is not nearly exceeded (check with du -h during emerge), you may have stumbled upon an inodes shortage. So far it definitely may be a problem for the www-client/chromium package, for it's grand storage requirements, but can be expected for other large packages as well. To workaround - append nr_inodes=0 to the list of your options for the tmpfs mount in the /etc/fstab file. For additional information refer to 'tmpfs' section in man mount.

As last instance, add a (temporary) swap file somewhere on your system with enough capacity:

root #touch /swap.img

root #chmod 600 /swap.img

root #dd if=/dev/zero bs=1024M of=/swap.img count=8

This will create the file /swap.img with 8GB filled with zeroes.

Set up the swap area using:

root #mkswap /swap.img

Enable the file for swapping:

root #swapon /swap.img

Check, if the swap file is active:

root #swapon --summary

Filename                     Type            Size    Used    Priority
/swap.img                    file            8388604 3876352 -2

Compile the packages which would deadlock the computer because of high RAM usage (e.g. chromium):

root #emerge --ask www-client/chromium

Alternatively, disable and remove the swap file when finished:

root #swapoff /swap.img

root #rm /swap.img

See also

• Zram — a Linux kernel feature and userspace tools for creating compressible RAM-based block devices.
• Zswap — a lightweight compressed cache for swap pages.