Catalyst-Lab

Important
This documentation and the resources mentioned herein are not officially part of the Gentoo project. They are provided as part of a personal project and may not be endorsed or supported by Gentoo.

Catalyst-Lab is an automation tool designed to streamline the process of building various Gentoo release stages and binhost packages. It manages the dependencies between stages, prepares the necessary files, and performs the builds. Catalyst-Lab operates by processing stage templates that include key details such as the spec file, portage configuration directory, overlays, root overlays, filesystem scripts (fsscript), and more.

The tool ensures ease of transfer between different host systems without the need for template modifications, even if the new host system uses a different architecture. Catalyst-Lab is particularly useful for maintaining scheduled releases and binhost packages by automating the build process for all or selected templates without altering pre-existing templates.

To get Catalyst-Lab script visit https://github.com/damiandudycz/catalyst-lab

Templates

Templates are stored in the /etc/catalyst-lab/templates directory and contain all the necessary data to generate the final spec file. Each template is organized in a structured directory with the following format: PLATFORM/RELEASE/STAGE. For example: ps3/23.0-default/stage3-cell-openrc.

Template Structure Elements

PLATFORM: Defines the device or collection of devices sharing the same CPU architecture (e.g., ppc64, ps3, rpi5, amd64).
RELEASE: Combines stages for a specific system configuration and base profile (e.g., 23.0-default, 23.0-llvm).
STAGE: Contains information related to the build stage, including the spec file template, portage configuration directory, overlays, root overlays, and fsscript.

Users should prepare templates in the /etc/catalyst-lab/templates directory to define the builds they intend to create. By default, Catalyst-Lab builds every template, but specific templates can be selected for building. The tool will automatically detect and build required source stages if they are unavailable.

Releng

Catalyst-Lab can utilize releng Portage configuration directories as the base for templates. Releng configuration is setup automatically, based on stage target build by every stage. The template’s portage folder can include additional files to be merged during the build.

By default, the releng repository is downloaded to /opt/releng.

Portage Snapshots

Catalyst-Lab automatically handles Portage snapshots. It checks for available snapshots and downloads them if none are found. Users can also specify the --update-snapshot flag to ensure the latest snapshot is fetched.

QEMU

Catalyst-Lab detects if a QEMU interpreter is required for each stage and adds the relevant information to the final spec file. Users do not need to manually specify the interpreter in template spec files. Ensure the arch variable in platform.conf is set correctly for this functionality. This enables template sharing across different architectures without modifications.

Catalyst Configuration

Templates can provide custom catalyst.conf files for specific builds. This allows users to have different Catalyst configurations for platforms, releases, or individual stages. The catalyst.conf file can be placed in the following directories:

PLATFORM – applies to all builds within the platform.
PLATFORM/RELEASE – applies to all builds within the release.
PLATFORM/RELEASE/STAGE – applies to a single stage.

If no custom catalyst.conf is provided, the default system catalyst.conf will be used.

Platform Configuration

Each PLATFORM directory in the templates should include a platform.conf file containing basic platform information:

arch - Defines base arch and subarch (subarch is optional if is the same as base arch) (e.g., amd64/core2, ppc64/cell, amd64).
chost (optional) - CHost of build stages.
common_flags (optional) - customized common_flags property of build stages.
repos (optional) – A comma-separated list of additional overlay repositories, either local or Git-based, used by all templates within the platform.

Inheritance

Catalyst-Lab determines the build order and stage inheritance by examining the template configuration files. Fields like target, version_stamp, and source_subpath must be set correctly to ensure proper inheritance. For example, to ensure that stage4 uses stage3 as its seed, the following values should be set:

stage4:

source_subpath: @REL_TYPE@/stage3-@SUB_ARCH@-desktop-openrc-@TIMESTAMP@

stage3 location:

ps3/23.0-default/stage3-desktop-openrc

The name of stage folder defines version_stamp (in this example it would be set to desktop-openrc-@TIMESTAMP@). User can overwrite this value in stage.spec if chooses to.

Placeholders such as @PLATFORM@, @REL_TYPE@, and @TIMESTAMP@ are automatically replaced by the script with the correct values.

Binary Packages

Catalyst-Lab uses and creates binary package repositories (binrepo) during the stage build process. By default, binrepos are stored in /var/cache/catalyst/binpkgs/local/PLATFORM/RELEASE. Users can specify a different directory for specific stages by setting the pkgcache value in the stage.spec file.

Stage.spec Template Structure

The stage.spec file serves as a template for the specification file that will be generated automatically by the script. This template does not need to include all the fields required in the final specification file, and users may omit certain values. These are detailed later in this section. The stage spec template also allows for the specification of keywords that will be replaced with automatically generated values, such as @TIMESTAMP@, @PLATFORM@, etc.

Below is a list of common values that should or should not be included in the spec template, along with a brief description:

Include	Parameter	Description	Example
no	target	Can be specified, but script can also detect it automatically from template name	stage1
no	version_stamp	Can be specified, but script can also detect it automatically from template name. Is set manually, it is advisable to append -@TIMESTAMP@ to this value to generate and utilize timestamp-marked files.	openrc-@TIMESTAMP@
no	profile	Only set this in stages that changes profile from their parents. Users can utilize the @BASE_ARCH@ placeholder to retrieve the value from platform.config.	default/linux/@BASE_ARCH@/23.0
yes	source_subpath	needs to be defined correctly, as it is used to determine inheritance. It can contain templates to automatically fill in the values. It is critical to ensure that the source can be correctly identified from this value.	@PLATFORM@/@REL_TYPE@/stage3-ppc64-openrc-@TIMESTAMP@
no	compression_mode	If not specified, pixz is used by default	pixz
no	update_seed	It's added automatically by the script	yes
no	update_seed_command	It's added automatically by the script	--update --deep --changed-use --usepkg --buildpkg --with-bdeps=y @system @world
yes	use, packages, rm, rcadd, unmerge, empty	should be included for liveCD stages and stage4 stages if appropriate
no	subarch	do not set this value; it will be automatically populated from platform.config.
no	rel_type	do not set this value; it will be automatically determined based on the template directory.
no	portage_confdir	do not include this value; it will be automatically created.
no	snapshot_treeish	do not include this value unless you wish to specify a very specific treeish.
no	repos	only add this if you wish to specify custom repositories for this build; otherwise, it will be automatically populated from platform.conf.
no	portage_prefix	do not include this value if you are using a releng_base file for the portage confdir template. If the portage confdir template is created without using releng, it may be included.
no	pkgcache_path	do not include this value unless you want to specify a custom directory for pkgcache. By default, this is set by the script to /var/cache/catalyst-binpkgs/${PLATFORM}/${RELEASE}.
no	interpreter	do not include this value; it will be added automatically if emulation is needed.
no	/overlay, /root_overlay, */fsscript	do not include these values; they will be added automatically if the corresponding files and directories exist in the template.

Examples

Example of stage1 spec template - ps3/23.0-default/stage1:

source_subpath: @FAMILY_ARCH@/gentoo/stage3-@BASE_ARCH@-openrc-@TIMESTAMP@

Example of stage3 spec template - ps3/23.0-default/stage3-openrc:

source_subpath: @REL_TYPE@/stage1-@SUB_ARCH@-@TIMESTAMP@

Example of livecd-stage1 template - ps3/23.0-default/livecd-stage1-cell:

source_subpath: @REL_TYPE@/stage3-@SUB_ARCH@-openrc-@TIMESTAMP@

use:
 	ps3
	compile-locales
	fbcon
	socks5
	unicode
	xml

packages:
	sys-apps/ps3-gentoo-installer
	sys-apps/ps3vram-swap
	sys-block/zram-init
 	app-portage/gentoolkit
 	net-misc/ntp
	sys-block/zram-init
	net-misc/networkmanager
	app-accessibility/brltty
	app-admin/pwgen
	app-arch/lbzip2
	app-arch/pigz
	app-arch/zstd
	app-crypt/gnupg
	app-misc/livecd-tools
	app-portage/mirrorselect
	app-shells/bash-completion
	app-shells/gentoo-bashcomp
	net-analyzer/tcptraceroute
	net-analyzer/traceroute
	net-misc/dhcpcd
	net-misc/iputils
	net-misc/openssh
	net-misc/rdate
	net-wireless/iw
	net-wireless/iwd
	net-wireless/wireless-tools
	net-wireless/wpa_supplicant
	sys-apps/busybox
	sys-apps/ethtool
	sys-apps/fxload
	sys-apps/gptfdisk
	sys-apps/hdparm
	sys-apps/ibm-powerpc-utils
	sys-apps/ipmitool
	sys-apps/iproute2
	sys-apps/lsvpd
	sys-apps/memtester
	sys-apps/merge-usr
	sys-apps/ppc64-diag
	sys-apps/sdparm
	sys-apps/usbutils
	sys-auth/ssh-import-id
	sys-block/parted
	sys-fs/bcache-tools
	sys-fs/btrfs-progs
	sys-fs/cryptsetup
	sys-fs/dosfstools
	sys-fs/e2fsprogs
	sys-fs/f2fs-tools
	sys-fs/iprutils
	sys-fs/lsscsi
	sys-fs/lvm2
	sys-fs/mdadm
	sys-fs/mtd-utils
	sys-fs/sysfsutils
	sys-fs/xfsprogs
	sys-libs/gpm
	sys-process/lsof
	www-client/links

Setting Up a Catalyst-Lab Template

To configure a Catalyst-Lab template, follow these steps:

Create a platform directory in /etc/catalyst-lab/templates that corresponds to your platform. For example:

mkdir /etc/catalyst-lab/templates/ps3

Inside the platform template directory, create a file named platform.conf and populate it with the following information:

Parameter	Description
arch	Architecture of system to build. Contains base arch and subarch (subarch is optional if it's the same as base arch),
chost (optional)	CHost of stages to build. This can usually be obtained from catalyst toml files using correct arch_subarch and can usually be skipped here.
common_flags (optional)	Customized common flags for the build.
repos (optional)	Comma-separated list of additional overlay repositories, applicable for the entire platform template. This can be a local directory or a link to a Git repository.

Ensure you replace the sample values with those that match your specific requirements. Example of platform.conf file:

arch=ppc64/cell

# Optional:
chost="powerpc64-unknown-linux-gnu"
common_flags="-O2 -pipe -mcpu=cell -mtune=cell -mabi=altivec -mno-string -mno-update -mno-multiple"
repos="https://github.com/damiandudycz/ps3-gentoo-overlay"

If you wish to use a custom catalyst.conf file for all builds in this platform, create the catalyst.conf file in the platform template directory:

/etc/catalyst-lab/templates/ps3/catalyst.conf

Create a directory for the release version:

mkdir /etc/catalyst-lab/templates/ps3/23.0-default

If you wish to use a custom catalyst.conf file for all builds in this release, add it to the release template directory:

/etc/catalyst-lab/templates/ps3/23.0-default/catalyst.conf

Create a directory for the specific stage:

mkdir /etc/catalyst-lab/templates/ps3/23.0-default/stage1

If you wish to use a custom catalyst.conf file for this specific stage only, place it in the stage template directory:

/etc/catalyst-lab/templates/ps3/23.0-default/stage1/catalyst.conf

Inside the stage directory, create a file named stage.spec:

/etc/catalyst-lab/templates/ps3/23.0-default/stage1/stage.spec

This file serves as a stage specification template, not the final specification file. The script will automatically populate this template with various generated values. You can use keywords such as @PLATFORM@, @RELEASE@, @TIMESTAMP@, and others, which are detailed later in this document.

(optional) Within the stage directory, create a portage directory:

mkdir /etc/catalyst-lab/templates/ps3/23.0-default/stage1/portage

This directory serves as your portage_confdir. Here, you can make any desired changes to your system, such as adding package.use, env, etc. Content of this directory will be merged with releng portage base directory. Here you can add only additional files that you need on top of releng base.

Optionally add the overlay, root_overlay directories, and fsscript.sh if you intend to use them. There is no need to specify these in the stage.spec template; the script will automatically include these entries if the corresponding directories and files exist.

Your template is now ready to be built using the Catalyst-Lab script.

Usage

After preparing the necessary templates, you can initiate the build process by executing the following command:

./catalyst-lab.sh --build

This command will prepare the final stage files, download the required seeds, set up the releng and portage snapshots, and build all specified stages.

If you wish to build only specific stages, you can specify them directly in the script:

./catalyst-lab.sh --build ps3/23.0-default/stage3-cell-base-openrc

You can include one or more templates in this command. If a stage requires the construction of a seed prior to its build, the necessary seed will be added automatically. If a previous version of the corresponding seed exists, it will be utilized instead.

It is also possible to use wildcard when selecting stages to build, for example:

./catalyst-lab.sh --build "*/*/livecd*"

If you want to see the stages to be build without actually building, just omit --build flag

The following additional flags can be used with the script:

--update-snapshot - Retrieve the latest portage snapshot before initiating the build process.
--update-releng - Pull the latest changes from the releng repository prior to building.
--update-repos - Pull the latest changes from overlay repositories used by current stage.
--clean - Force the build of all required seeds, even if previous builds are available.
--purge - Remove old builds, leaving only the newest one created.

User:Damiandudycz/catalyst-lab

Contents