User:NeddySeagoon/HOWTO Update Old Gentoo
Major editing in progress to replace gentoo-x86 CVS with the gentoo-historical git repo
HOWTO Update Old Gentoo
This is mostly for the educational experience and to write this document to show what can be done if you really really have to do it.
Reinstalling a system several years old will be faster than this process
Background
Some time ago, it looks like mid 2011, my four or five discrete systems were retired in favour of a Gen7 HP Microserver running KVMs to replace the physical servers. The power savings paid for the Microserver in 18 months and its still going.
The poor wee thing struggled in only 8G of RAM and I wanted to run another KVM. I upgraded to 16G RAM, that's double the maximum its supposed to take but that's another HOWTO.
In 2011, one of the services moved to a KVM was my router. It was originally Smoothwall 3 on a 233MHz Pentium. To be able to test the KVM implementation I created a number of throw away KVMs for testing, before I exposed real hardware to the new router.
Of course, being throw away KVMs, they were never thrown away, they were just shut down and left.
Being in need of another KVM, I thought it might be fun to attempt to upgrade a 2011 Gentoo install, rather that install a new KVM.
Overview
This process is going through some of the missed updates. It does imply that nothing should break as a result of following this process, if you can get all the source files required for each step. That's a big if for systems more than three years out of date.
The Starting Point
The datestamp on the gentoo repo
user $cat /usr/portage/metadata/timestampThu Aug 11 17:36:42 UTC 2011
Any time in 2019, a 2011 install qualifies as old and out of date
Profile
user $eselect profile listdefault/linux/amd64/10.0 default/linux/amd64/10.0/desktop default/linux/amd64/10.0/desktop/gnome default/linux/amd64/10.0/desktop/kde default/linux/amd64/10.0/developer default/linux/amd64/10.0/no-multilib default/linux/amd64/10.0/server hardened/linux/amd64 hardened/linux/amd64/selinux hardened/linux/amd64/no-multilib * hardened/linux/amd64/no-multilib/selinux selinux/2007.0/amd64 selinux/2007.0/amd64/hardened selinux/v2refpolicy/amd64 selinux/v2refpolicy/amd64/desktop selinux/v2refpolicy/amd64/developer selinux/v2refpolicy/amd64/hardened selinux/v2refpolicy/amd64/server
binutils
Blue_Test ~ #binutils-config -l[1] x86_64-pc-linux-gnu-2.21.1 *
gcc
Blue_Test ~ #gcc-config -l[1] x86_64-pc-linux-gnu-4.5.3 * [2] x86_64-pc-linux-gnu-4.5.3-hardenednopie [3] x86_64-pc-linux-gnu-4.5.3-hardenednopiessp [4] x86_64-pc-linux-gnu-4.5.3-hardenednossp [5] x86_64-pc-linux-gnu-4.5.3-vanilla
Kernel
Blue_Test ~ #uname -aLinux Blue_Test 2.6.39-hardened-r8 #1 SMP Thu Jul 28 19:39:39 BST 2011 x86_64 AMD Phenom(tm) 9550 Quad-Core Processor AuthenticAMD GNU/Linux
World
It was a fairly minimal install for firewall testing
/var/lib/portage/worldapp-admin/metalog
app-admin/sudo
app-portage/gentoolkit
app-portage/ufed
app-text/wgetpaste
net-analyzer/tcpdump
net-misc/dhcpcd
net-misc/ntp
sys-apps/ethtool
sys-apps/portage
sys-apps/usermode-utilities
sys-boot/grub-static
sys-devel/gettext
sys-kernel/hardened-sources
sys-libs/gpm
sys-process/dcron
The process described here should work for any type of install. The ebuilds are available. The hard bit is finding old source code. Its a process document, not a step by step guide. This example does not have a desktop or even a window manager. Updating KDE or GNOME should be more of the same.
Plan A - Just Go For It
This is harmless but unlikely to work on systems over a year out of date
- Install git
- Move the old repo out of the way, so we can still refer to it for bits and pieces.
- Get an up to date ::gentoo repo
Blue_Test ~ #
emerge -uDNav worldBlue_Test ~ #
etc-updateAn eight year old portage with a current repo ...
Plan A - Implementation
Install git
Blue_Test ~ #emerge git -av* IMPORTANT: 2 news items need reading for repository 'gentoo'. * Use eselect news to read news items. * Last emerge --sync was 8y 21d 4h 12m 6s ago. These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild R ] dev-vcs/git-1.7.6 USE="blksha1 curl iconv perl python threads webdav -bash-completion -cgi -cvs -doc -emacs -gtk (-ppcsha1) -subversion -tk -xinetd" 0 kB Total: 1 package (1 reinstall), Size of downloads: 0 kB Would you like to merge these packages? [Yes/No]
Git is already installed. That was an unexpected bonus.
Shuffle repos
Move the old repo out of the way.
Blue_Test ~ #mv /usr/portage /usr/portage_oldClone the git ::gentoo repo
Blue_Test ~ #mkdir /usr/portage
Blue_Test ~ #cd /usr/portage
Blue_Test ~ #git clone https://anongit.gentoo.org/git/repo/gentoo.git ./Fix the Profile Definition
The current profile has been removed, so to put off dealing with profile updates change the /etc/make.profile symbolic link to point to the copy in the repo we saved earlier.
This install predates the migration to /etc/portage/..
Blue_Test ~ #cd /etc
Blue_Test ~ #rm make.profile
Blue_Test ~ #ln -s ../usr/portage_old/profiles/hardened/linux/amd64/no-multilib make.profilePlan A Summary
The old system is intact, with its original profile and a current ::gentoo repo.
In theory, in a perfect world, emerge -uDNav world just works.
Blue_Test ~ #emerge -uDNav worldThese are the packages that would be merged, in order: Calculating dependencies \ * Missing digest for '/usr/portage/sys-fs/sysfsutils/sysfsutils-2.1.0.ebuild' / * Manifest not found for '/usr/portage/virtual/pager/pager-0.ebuild' | * Manifest not found for '/usr/portage/virtual/ssh/ssh-0.ebuild' ...
The old portage sys-apps/portage-2.2.0_alpha50 knows nothing of the new manifest format, and lots of other things besides, so this approach ends here.
Plan B Use a git/CVS Repo
Gentoo git history is split into two repos. From switch from CVS to git and the later move of CVS to git. Depending on how of of date the install is, you may need repos. The first commit to the new git repo was at Saturday Aug 8 2015.
- Move the old repo out of the way, so we can still refer to it for bits and pieces.
- Use a git repo, to do easy checkouts by dates.
*Fall back to a CVS repo if required
- Incrementally update stage 1 by running /usr/portage/scripts/bootstrap.sh
/usr/portage/scripts/bootstrap.sh is the script used to do the stage 1 part of a stage 1 install. Its still in the git tree. Its not going to be quite that easy as there will be at least two profile changes along the way.
Plan B - Implementation
The first two steps were completed as part of Plan A. We need to see how far git goes back ... git checkout `git rev-list -n 1 --first-parent --before="2015-09-01" master` looks about the limit. Its not far enough.
Follow the instructions in the next section for using git but clone the historical.git repo and work with that.
user $git clone https://anongit.gentoo.org/git/repo/gentoo/historical.gitNow we have three copies of the ::gentoo
- /usr/portage as it was in 2011
- from git with all the git history
- from
CVSuntil it was migrated to git
That's all the ebuilds that were ever in the ::gentoo repository.
After rummaging around with help from Google, I found all the bits I needed to make the update run. It was quite a nostalgia trip, using a portage from 2011.
My system has moved forward a year. There have been no profile updates yet.
Plan C
This is a refinement of plan B and probably the right way.
Gentoo tries to keep an upgrade path for about a year, so lets try one year steps. Before you cross an EAPI boundary, you need a new portage that understands the new EAPI.
Profile updates will be covered by News items. That's /10.0/ to /13.0/ to /17.0/ to /17.1/ to /23.0/. They must be performed as they become due.
Overview
- Get a suitable ebuild repo
(CVS or git) - GENTOO_MIRRORS=""
FEATURES="digest assume-digests"emerge -uDNavf world --keep-going to fetch sources. - Find missing sources or delete old ebuilds that won't be used ... or both.
- emerge binutils gcc, then switch to the new versions.
- GENTOO_MIRRORS="" FEATURES="digest assume-digests" emerge -uDNav world --keep-going to actually update.
- perl-cleaner --all
- python-updater
- revdep-rebuild or emerge @preserved-rebuild
- etc-update
- read NEWS and act on it
- update kernel
- emerge --depclean (carefully)
- reboot into new kernel
- tidy up as required
- Rinse and repeat at annual intervals.
Timesavers
Set GENTOO_MIRRORS="" That's not the same as leaving GENTOO_MIRRORS unset. The former instructs portage not to use the gentoo mirror system at all, the latter permits the use of a default set of mirrors.
Source code is removed from the gentoo mirror system at the same time as ebuilds are removed from the tree. This setting tells portage to use the SOURCE_URI list in the ebuild to locate source files without checking the mirrors first.
This is normally regarded as poor netiquette, that is why Gentoo provides a mirror system in the first place. Revert this setting later
The NEWS System
Gentoo News items are normally distributed with emerge --sync. As this process uses cvs and git repos directly, other arrangements need to be made for reading news items. https://www.gentoo.org/support/news-items/
What action, if any, needs to be taken in response to news is install specific.
At each step, act on any News items that apply to your install
Take care that you only act on news items up to the date of your portage tree. Do not act on any future news items. They will have their turn later.
==== Problems With the CVS Repro ====
The Attic
Many things will have been moved to the Attic. This changes the headers in the ebuilds but not the digests.
Blue_Test ~ #emerge -avuDN @worldCalculating dependencies - * Digest verification failed: * /usr/gentoo-x86/sys-libs/glibc/glibc-2.15-r2.ebuild * Reason: Filesize does not match recorded size * Got: 8000 * Expected: 7994
Notice the header # $Header: /var/cvsroot/gentoo-x86/sys-libs/glibc/Attic/glibc-2.15-r2.ebuild,v 1.29 2013/02/12 15:53:08 jer Exp $
Its grown an "/Attic" to make the ebuild 6 characters longer.
The choices are fix the ebuild headers or fix the digests. Fixing headers to match the existing digests is more secure. Fixing the digests is faster since portage offers FEATURES="digest"
Understand the security implications before you make your choice. I used the FEATURES="digest" method. My sed foo is far too weak to fix the ebuild headers.
Blue_Test ~ #GENTOO_MIRRORS="" FEATURES="digest assume-digests" emerge -1av portageFEATURES variable contains unknown value(s): fixpackages * IMPORTANT: 2 news items need reading for repository 'gentoo'. * Use eselect news to read news items. * * The FEATURES=digest setting can prevent corruption from being noticed. * The `repoman manifest` command is the preferred way to generate * manifests and it is capable of doing an entire repository or category at * once. *
Problems With Finding Source Files
All the distfiles I have ever downloaded since mid 2006 are at distfiles along with a smattering of older ones too. Consider adding http://bloodnoc.org/~roy/olde-distfiles/ to the top of your GENTOO_MIRRORS= list in make.conf. https://bloodnoc.org/~roy/olde-distfiles/ is supported by the server but requires the use of common cryto and certificates, which may not be possible on old systems.
https://dev.gentoo.org/~vapier/dist/ and https://dev.gentoo.org/~sam/distfiles/sys-devel/gcc/ are reported be useful sources of old distfiles too.
The Repo Itself
Recreating all the digests for packages that portage wants to install requires all the sources for all the ebuilds in the ebuild leaf directory. That's fine if you have or can find all the sources.
In practice, any ebuild that will not be used can be removed. That limits source code finding to the absolute minimum.
https:// connections
Don't even think about it on an old system.
Source files that are on a SSL protected server are out of reach of on old system. Certificates have been changed and SSL protocols have changed. Download such source files via an intermediate system.
Trying is harmless.
http://bloodnoc.org/~roy/olde-distfiles/ is http://
Warts on My System
My system is 64 bit only. It boots using grub-static, which requires 32 bit support in the kernel to install. My kernels do not have 32 bit support so installing grub-static fails. Use portages --exclude="grub-static" for cases like this. More permanently, use package.mask
Other Pitfalls
In no particular order.
Its /etc/make.conf, not /etc/portage/make.conf
Old emerge --depclean is a bit aggressive. Its a very bad thing to let it rip out eselect-python as that breaks emerge. Fixing it is left as an exercise for the reader.
Don't forget about
perl-cleaner --all python-updater revdep-rebuild etc-update DEVTMPFS=y in the kernel CPU_FLAGS_X86= keeping the kernel up to date switch from udev to eudev sshd changed to prohibitpassword for root
Not all of these may apply to your case. You might even find a few more.
Progress
==== Using cvs ====
For testing and working out how to use a cvs repo as the ::gentoo tree.
cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot co -D "2011-09-01 GMT" gentoo-x86
cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot co -D "2012-09-01 GMT" gentoo-x86
The first step up. That was difficult. Finding sources and removing ebuilds that were not required were the problem areas.
cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot co -D "2013-09-01 GMT" gentoo-x86
This was actually two steps as EAPI 5 was introduced. A portage that understands EAPI 5 was required to get to 2013-09-01 GMT.
The /10.0/ to /13.0/ profile change was here too.
cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot co -D "2014-09-01 GMT" gentoo-x86
Switched to eudev.
Regressing Too Far
If you don't know how old your install is and you regress too far, your emerge output will contain lots of downgrades These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild UD ] sys-libs/glibc-2.23-r3:2.2::gentoo [2.23-r4:2.2::gentoo] USE="rpc -audit -caps -debug -gd (-hardened) (-multilib) -nscd -profile (-selinux) -suid -systemtap -vanilla" 13.448 KiB
[ebuild UD ] app-arch/bzip2-1.0.6-r7::gentoo [1.0.6-r8::gentoo] USE="-static -static-libs" 764 KiB
[ebuild UD ] sys-apps/opentmpfiles-0.1.2::gentoo [0.1.3::gentoo] USE="(-selinux)" 6 KiB
[ebuild UD ] app-arch/unrar-5.4.5:0/0::gentoo [5.5.5-r1:0/5::gentoo] 222 KiB
[ebuild UD ] net-mail/tnef-1.4.12::gentoo [1.4.15::gentoo] 3.916 KiB
[ebuild UD ] dev-libs/crypto++-5.6.5:0/5.6::gentoo [5.6.5-r1:0/5.6::gentoo] USE="-static-libs" 4.122 KiB
[ebuild UD ] dev-libs/gobject-introspection-common-1.48.0::gentoo [1.50.0::gentoo] 1.340 KiB
...
Advance the date in your portage tree (CVS or git) and try again.
Some packages, like sys-libs/glibc, must not be downgraded or you may break your system.
Tidying Up
Depending on how much free space is available, this step may need to be performed more or less frequently. Its documented here as this is a good time, before moving to a git ::gentoo repo.
Remove old kernels from /usr/src
Remove /var/tmp/portage, or its contents.
eclean -d {packages distfiles}
Remove /var/log/portage
Moving to git
To free up space, the
gentoo-x86 cvs repo can be removed.
Get the entire ::gentoo git repo.
root #cd /usr root #git clone https://github.com/gentoo/gentoo.git ./gentoo-gitFix the /usr/portage symlink to point to gentoo-git, rather than gentoo-x86. That's the entire ::gentoo git repo as it was at the time of the clone. For me, the newest commit is Date: Mon Sep 9 12:51:11 2019 +0200 As we are still 5 years out of date, we can't use that as is.
Using git
The first commit (all of the cvs gentoo-x86) was Saturday Aug 8 2015. The next step is from git. That fits nicely with the annual Sep 1st steps.
root #git checkout `git rev-list -n 1 --first-parent --before="2015-09-01" master`This puts the HEAD of the git repo at Date: Tue Sep 1 13:30:01 2015 +0200
The FEATURES="digest" is no longer required. Git does not have an attic.
As the repo path was changed from /usr/gentoo-x86 to /usr/gentoo-git
Fix /etc/portage/repos.conf
Fix /etc/portage/make.profile
If ssh password login for root is required, check sshd_config before the reboot.
2016
root #git checkout `git rev-list -n 1 --first-parent --before="2016-09-01" master`Only sources for eight packages no longer available from the SOURCE_URI= in the ebuilds. Once you get to three years old, the GPL requires that sources can still be obtained, so it gets easier then. Obtained does not mean on line.
There in no need to detail the steps any more.
2017
root #git checkout `git rev-list -n 1 --first-parent --before="2017-09-01" master`2018
root #git checkout `git rev-list -n 1 --first-parent --before="2018-09-01" master`2019
root #git checkout `git rev-list -n 1 --first-parent --before="2019-09-01" master`Up To Date
root #git checkout master root #git pull I still have the migration to the 17.1 profile to do.
Further Help
A Case Study
There is a blow by blow real update on the Gentoo Forums
This follows using CVS and git. The topic runs to about 300 posts before ending with success.
Posting on the Forum
The Unsupported Software forum is the right place to post for more detailed help with your particular case.