Project:Mozilla/Version Bump Guide

From Gentoo Wiki
Jump to:navigation Jump to:search

This page provides scripts and general guidance on how to maintain several products offered by Mozilla in Gentoo. It is recommended to have a separate environment for development, e.g. a chroot, container, virtual machine and so. See Package_testing for specifics.

Some scripts to help with maintainer's tasks are available at https://github.com/juippis/my-gentoo-mozilla-scripts - they are referenced below. See the Readme file for a short introduction to these scripts.

Important
Always study the shell scripts you get from the Internet before executing them!


General notes

I recommend using the provided scripts, and a dedicated test environment for these bumps. git clone the script repo and add the bin/ directory to your user's $PATH. git pull before working on Mozilla products in Gentoo (or just occassionally).

Other tools that are recommended: pkg-testing-tools in your test environment, gentoolkit in your host, pkgcheck in your host.

Join the other project members in Libera.chat's #gentoo-mozilla channel!

General resources: https://firefox-source-docs.mozilla.org/index.html

Firefox

https://wiki.mozilla.org/Release_Management/Calendar

https://wiki.mozilla.org/NSS:Release_Versions

Only Firefox-ESR gets stabilized.


Firefox ESR major version bump

When bumping major ESR version, e.g., from 78 to 91, it's better to base the ebuild on the new parallel rapid version. Upon release, rapid and esr will have their development branches diverged immediately, so in theory rapid-91.0.2 and esr-91.0.2 can be different, but in practice probably not.

It's suggested to still keep offering old ESRs before the new one is "officially" released and biggest bugs weeded out, Mozilla will usually take a while before the new ESR is "production-ready" even if it's offered, and old ESR is kept alive for few months.


NSS

https://wiki.mozilla.org/NSS:Release_Versions

Each version gets a detailed "release notes" shipped by upstream. After unpacking the sources, see ./nss-"newversion"/nss/doc/rst/releases/nss_"newversion".rst. Or head to https://hg.mozilla.org/projects/nss/file/tip/doc/rst/releases

Upstream includes their CI scripts and configs in the release tarball, so the ./nss/automation/ directory can be studied for packaging help.

NSS's build documentation recommends to use gyp to build nss, but upstream says otherwise. There's some work currently on-hold regarding gyp support.

Only NSS-ESR gets stabilized.

Spidermonkey

Spidermonkey's sources get shipped with Firefox-ESR's source tarball. The rapid version of Spidermonkey is often broken and should not be used. Therefore Spidermonkey's ebuild follows closely Firefox-ESR's. Spidermonkey should match Firefox-ESR's patch set, but also its own spidermonkey patch set in Gentoo. 

FIREFOX_PATCHSET="firefox-91esr-patches-05j.tar.xz"
SPIDERMONKEY_PATCHSET="spidermonkey-91-patches-03j.tar.xz"

I.e., for spidermonkey-91.7.1 match the patch set used in if firefox-91.7.1.

https://github.com/mozilla/gecko-dev/tree/master/js

https://github.com/mozilla-spidermonkey/spidermonkey-embedding-examples/tree/next

https://github.com/mozilla-spidermonkey/spidermonkey-embedding-examples/tree/next/docs

https://github.com/mozilla-spidermonkey/spidermonkey-embedding-examples/tree/next/.github/workflows CI configuration.


Thunderbird

Thunderbird is built on top of Firefox-ESR and a new version is often released just a couple of days after Firefox-ESR. Not every ESR change goes to Thunderbird, and therefore not every Firefox-ESR is followed by a new Thunderbird release. E.g, purely cosmetic changes don't usually warrant a new Thunderbird release.

As a general note, whatever was done with the matching Firefox-ESR's ebuild has to be replicated on to Thunderbird as well. Firefox-ESR and Thunderbird share their Gentoo patch sets, e.g. 

FIREFOX_PATCHSET="firefox-91esr-patches-06j.tar.xz"

is replicated in Thunderbird as well.

https://www.thunderbird.net/en-US/thunderbird/releases/

Periodic Maintainer checks

Use packages.gentoo.org or gentoo_mozilla_maintainer_check.sh to check occassionally for maintainer's tasks. Add yourself to Mozilla's mail alias to receive notifications from bugzilla, and mails directly to the alias. bugs assigned to mozilla@g.o.

user $DEVREPO=~/git/gentoo gentoo_mozilla_maintainer_check.sh


Version Bump Guidance

cbindgen

Launch your container or other dedicated test environment. Install cargo-ebuild.

root #emerge --ask dev-util/cargo-ebuild
user $cd /tmp
user $cd cbindgen
user $git checkout v0.21.0
user $cargo ebuild
user $cd ~/where/your/gentoo/dev/repo/is/dev-util/cbindgen
user $mv /tmp/cbindgen/cbindgen-0.21.0.ebuild .
user $pkgdev manifest

modify ebuild here. Add unstable keywords from the previous ebuild e.g. ekeyword ~amd64 ~arm ~arm64 ~ppc64 ~riscv ~x86 cbindgen-0.21.0.ebuild. Add 

https://github.com/eqrion/cbindgen/archive/refs/tags/v${PV}.tar.gz -> ${P}.tar.gz

to SRC_URI if needed. Remember to run pkgdev manifest again after. Add QA_FLAGS_IGNORED="usr/bin/cbindgen" to prevent rust-related QA warnings from popping up. Proceed testing it with

user $pkg-testing-tool --append-emerge '--autounmask=y --oneshot' --extra-env-file 'test.conf' --test-feature-scope once --max-use-combinations 6 -p '=dev-util/cbindgen-0.21.0'

Do reverse-bdep testing, https://packages.gentoo.org/packages/dev-util/cbindgen/reverse-dependencies

Firefox

For major version bumps (e.g. 95.0.3 -> 96.0), use the gentoo_mozilla_products_vbump_differ.sh script to sniff out any relevant changes done in the build system. More relevant files are listed at the top. Firefox-bin packages can be bumped pretty much after browser/locales/all-locales shows differences, or no differences.

user $gentoo_mozilla_products_vbump_differ.sh firefox-95.0.3 firefox-96.0

For minor version bumps (e.g. 98.0 -> 98.0.1), there's usually less than 10 changes so they can most likely be checked from Mozilla's mercurial web interface. minor bump with tags example.

Although it doesn't hurt to run the gentoo_mozilla_products_vbump_differ.sh for those either.

https://hg.mozilla.org/releases/mozilla-release/log rapid

https://hg.mozilla.org/releases/mozilla-esr91/log ESR

Sort out patching if you've been following upstream bugs and are aware some patches are redundant, or if some additional ones are needed. See Patches and Patching. Open up a container, chroot or whatever is your preferred way of testing new ebuilds.

user $cd /var/db/repos/gentoo/www-client/firefox
user $cp firefox-98.0.ebuild firefox-98.0.1.ebuild
user $pkgdev manifest

Do your ebuild modifications. (It's good to test the new patch set applies with a simple ebuild firefox-98.0.1.ebuild clean prepare. For major version bumps it's good to do a single run with default USE flags:

user $emerge -1av www-client/firefox:rapid --buildpkg-exclude www-client/firefox --usepkg-exclude www-client/firefox
user $emerge -1av www-client/firefox:esr --buildpkg-exclude www-client/firefox --usepkg-exclude www-client/firefox

If it passes, use the proper script to test it.

user $test_mozilla_products-amd64.sh www-client/firefox-98.0.1

Even for minor bumps it's good to do a thorough testing, since sometimes updated dependencies might be the cause of new failures. E.g. media-libs/dav1d and libvpx are infamous for that. Do runtime testing between builds if possible.

With good package-testing-settings your terminal output will show the final size for the build directory. These can be found from saved elogs too. Example: 

no pgo, no lto
 * Final size of build directory: 6879180 KiB (  6.5 GiB)
 * Final size of installed tree:   278032 KiB (271.5 MiB)

gcc, +lto +pgo 
 * Final size of build directory: 13954940 KiB ( 13.3 GiB)
 * Final size of installed tree:    228356 KiB (223.0 MiB)

Increase the size requirements in the ebuild if needed.

mozilla-vpn-client

nss

user $gentoo_mozilla_products_vbump_differ.sh nss-3.75 nss-3.76

Open up chroot, container etc. Copy and edit the ebuild as needed. Fist do a basic installation to see that patches apply, and that it compiles with default USE flags.

root #emerge --ask --oneshot --verbose --buildpkg-exclude dev-libs/nss --usepkg-exclude dev-libs/nss =dev-libs/nss-3.76

Install libabigail and iwdevtools if you don't have them already.

root #emerge --ask app-portage/iwdevtools dev-util/libabigail
user $ebuild nss-3.75.ebuild clean install
user $ebuild nss-3.76.ebuild clean install
user $qa-cmp -I dev-libs/nss

Edit the ebuild if needed.

Then do few test runs with pkg-testing-tools,

user $pkg-testing-tool --append-emerge '--autounmask=y --oneshot' --extra-env-file 'test.conf' --test-feature-scope once --max-use-combinations 8 -p '=dev-libs/nss-3.76'

Testing reverse dependencies wouldn't hurt at this point either, but if qa-cmp didn't report any significant changes, it may not be that important.

https://qa-reports.gentoo.org/output/genrdeps/rindex/dev-libs/nss

A very simple rdeptester.sh.

nspr

Run the script to find important build-related differences between versions:

user $gentoo_mozilla_products_vbump_differ.sh nspr-4.33 nspr-4.34

Open up chroot, container etc. Copy and edit the ebuild as needed. Fist do a basic installation to see that patches apply, and that it compiles.

root #emerge --ask -1av =dev-libs/nspr-4.34 --buildpkg-exclude dev-libs/nspr --usepkg-exclude dev-libs/nspr

Remember to ekeyword ~all nspr-4.34.ebuild your new ebuild.

Install libabigail and iwdevtools if you don't have them already, and check there are no breaking library changes.

root #emerge --ask app-portage/iwdevtools dev-util/libabigail
user $ebuild nspr-4.33.ebuild clean install
user $ebuild nspr-4.34.ebuild clean install
user $qa-cmp -I dev-libs/nspr

Do better test runs with stricter test.env. This will also run FEATURES="test" once.

root #pkg-testing-tool --append-emerge '--autounmask=y --oneshot' --extra-env-file 'test.conf' --test-feature-scope once --max-use-combinations 6 -p '=dev-libs/nspr-4.34'

Do reverse dependency testing, just in case.

Spidermonkey

Again, remember to replicate whatever was done in Firefox-ESR's ebuild between the changed versions. Base a major version bump (e.g., 78 -> 91) in Firefox-ESR's ebuild.

user $gentoo_mozilla_products_vbump_differ.sh spidermonkey-91.7.1 spidermonkey-91.7.2

Update patch set if needed. Start a container, chroot etc. Edit the ebuild as needed. Always good to do a light run to test that patches apply and that the package compiles with default USE flags etc.

user $emerge -1av dev-lang/spidermonkey --buildpkg-exclude dev-lang/spidermonkey --usepkg-exclude dev-lang/spidermonkey

Install libabigail and iwdevtools if you don't have them already.

root #emerge --ask app-portage/iwdevtools dev-util/libabigail
user $ebuild spidermonkey-91.7.1.ebuild clean install
user $ebuild spidermonkey-91.7.2.ebuild clean install
user $qa-cmp -I dev-lang/spidermonkey

Do a thorough test run.

user $pkg-testing-tool --append-emerge '--autounmask=y' --extra-env-file 'test.conf' --test-feature-scope once --max-use-combinations 12 -p '=dev-lang/spidermonkey-91.7.2'
Important
Spidermonkey runs different tests with +jit and -jit so remember to run the test phase at least twice!


Thunderbird

Firefox-ESR should be bumped at this point. Base the initial work on that, e.g. update patch set to match Firefox-ESR-91.5.0.

user $gentoo_mozilla_products_vbump_differ.sh thunderbird-91.4.2 thunderbird-91.5.0

Do your ebuild modifications. (It's good to test the new patch set applies with a simple ebuild thunderbird-91.5.0.ebuild clean prepare. For major version bumps it's good to do a single run with default USE flags:

user $emerge -1av mail-client/thunderbird --buildpkg-exclude mail-client/thunderbird --usepkg-exclude mail-client/thunderbird

If it passes, use the proper script to test it.

user $test_mozilla_products-amd64.sh mail-client/thunderbird-91.5.0

Even for minor bumps it's good to do a thorough testing, since sometimes updated dependencies might be the cause of new failures. E.g. media-libs/dav1d and libvpx are infamous for that. Do runtime testing between builds if possible.


Patches and Patching

All firefox patches are stored in a firefox-patches/ directory that's compressed and downloaded via SRC_URI. See the "FIREFOX_PATCHSET" variable in the ebuild. Originally the patches were based on git repository submissions, but I haven't been able to find the origin of this repository, so now the patches are just named in git-format for consistency's sake.

Try to use a consistent patch naming convention. "PN"-"majorversion"-patches-"revision".tar.xz.

For single patch files inside the tarball, we attempt to link it to a relevant Gentoo/Mozilla bug via the filename. E.g., 0011-spidermonkey-94.0-bmo-1687417-silence-static_assert-for-nojit.patch.

user $cd /tmp
user $tar xf /var/cache/distfiles/firefox-"oldversion"-patches-"revision".tar.xz
user $cd firefox-patches/

Now wget / copy your new patch here, note the naming convention.

user $cd ..
user $tar -cJf firefox-"newversion"-patches-"newrevision".tar.xz firefox-patches/

Upload the file to your devspace.

user $scp firefox-"newversion"-patches-"newrevision".tar.xz dev.gentoo.org:~/public_html/mozilla/patchsets/

Update FIREFOX_PATCHSET variable in the new ebuild.

If you need to remove an older patch due to upstream fixing it etc, do

user $cd /tmp
user $tar xf /var/cache/distfiles/firefox-"newversion"-patches-"currentrevision".tar.xz
user $cd firefox-patches/
user $rm 0010-firefox-custom-patch.patch
user $gentoo_firefox_patch_order.sh .
user $cd ..
user $tar -cJf firefox-"newversion"-patches-"newrevision".tar.xz firefox-patches/
user $scp firefox-"newversion"-patches-"newrevision".tar.xz dev.gentoo.org:~/public_html/mozilla/patchsets/
Retrieved from "https://wiki.gentoo.org/index.php?title=Project:Mozilla/Version_Bump_Guide&oldid=1293146"