Talk:Binary package guide
Before creating a discussion or leaving a comment, please read about using talk pages. To create a new discussion, click here. Comments on an existing discussion should be signed using
~~~~:
A comment [[User:Larry|Larry]] 13:52, 13 May 2024 (UTC) : A reply [[User:Sally|Sally]] 08:41, 5 November 2024 (UTC) :: Your reply ~~~~
My Server and Client(s) Setup
When I get more time, I'll try studying the article. In the meantime, for the past few years, I've used two of my own scripts called "$HOME/bin/sync-server.sh" & "$HOME/bin/sync-clients.sh" with HOSTNAME checking to make sure they're host or client as well as incorporating some of the commands from "Binary Package Guide". I used well structured Bash Scripting techniques with Functions. Again, I'll try posting them sometime (or I can EMail them upon request), but they're pretty basic. The only trivial part is checking for updating the kernel if it's needed, if so then cp the .config file over and build/install the kernel. Then re-run emerge world as some packages break if there's no .config file immediately following a kernel upgrade. The client scripts finally EMail the server to notify emerge completion. The only rsync'ing I do these days, are /etc/portage files and /usr/local/portage containing my custom ebuilds. No NFS utilization as it complicates things greatly. Everything depends on emerge & layman for syncing. Signed: Roger 18:33, 15 November 2011 (UTC)
- I'm so glad you like your scripts, Roger. Since you have left the wiki editors with no easy way to reach you, I am closing this discussion. --Davidbryant (talk) 14:41, 6 August 2020 (UTC)
32 bit and 64 bit Package Binary Server
I've just upgrade to a 64 bit platform (i7 8x3.5Ghz CPU, w/ 32G RAM) and seem quite capable of building both 32 and 64 bit entire system packages within minutes or an hour or so, and distributing to my older 32 bit platforms versus building them specifically for each arch, on each arch. Would be nice to see some Wiki info on making Gentoo build 32 and 64 bit binary packages. From what I'm guessing, the 32 bit builds performed on a 64 bit platform need to be performed within a chrooted build environment?
After figuring or documenting this, making sure the USE flags are compatible across the 32 & 64 bit binaries need to be addressed. ... shrugs ... I'd love to push the older 32 bit platforms into a binary distro, but nothing seems reasonably stable on any distro except Gentoo, including having Gentoo's flexibility with having newer packages/ebuilds easily available. Roger 10:18, 13 December 2012 (UTC)
- I guess nobody else really cares. It's been eight years since Roger wrote this, and he's not readily accessible. --Davidbryant (talk) 14:38, 6 August 2020 (UTC)
- I care. And currently I'm unable to setup crossdev environment with i686-pc-linux-gnu-emerge and all the necessary layout under /usr/i686-pc-linux-gnu
Is the support for multiple binary package servers still incomplete ?
The support for multiple binary package servers is somewhat incomplete. If several servers serve a binary package for the same package version, then only the first one will be considered. This can be problematic when these binary packages differ in their USE variable configuration and the USE variable configuration of a later binary package would match the systems configuration.
In my own testing I've not been able to reproduce any of the caveats described. Shouldn't we remove this note ? Sdrik (talk) 15:25, 27 March 2018 (UTC)
- I'm pretty sure the caveat is still important. You won't run into this unless you're maintaining your own private repositories, and compile two different versions of the same program. --Davidbryant (talk) 14:24, 6 August 2020 (UTC)
What about compiler flags when -march=native is used?
Am I missing something? building a package and control USE flags is one thing. However, given I have on my hots, i.e. builder machine, -march=native in CFLAGS, it will invoke gcc for its own processor, including processor flags at the stage of compilation. And such a code may be useless on another machine. This should be somehow reflected in the document especially given that the handbook suggests -march=native as the most standard approach. --Alex-kas (talk) 19:41, 2 February 2020 (UTC)
- Good point. But it's already addressed in "Using Binary Packages":
For binary packages to be usable on other systems they must fulfill some requirements:
* The client and server architecture and CHOST must match.
- Doesn't that answer your question? --Davidbryant (talk) 14:33, 6 August 2020 (UTC)
We should list available bin package compression formats
Portage supports BINPKG_COMPRESS variable which allows for different compression formats to be used, such as lz4. We should list them all here.
Juippis (talk) 13:49, 15 December 2021 (UTC)
- Thanks for the heads up! Sounds like a very good idea, but I don't use this, so I wouldn't know what to add off the top of my head. I have a project that builds packages though, so I may get a chance to come to this with a bit of knowledge some time.
- Comments like this are great, because they focus attention on particular changes that need to be made, so someone will surely get around to this eventually. Feel free to make any changes to fix this yourself however, if you know what needs doing, and don't mind: this seems like the sort of change that users will benefit from :).
- Boilerplate: Remember that no matter if changes aren't perfect, or if the formatting is a bit off, as long as edits make things a bit better, it's a step in the right direction ;). The contributor's guide gives a few tips on how to easily make contributions ;).
- I will close this discussion soon, if nothing more is added, as this seems like a pretty complete informational comment in and of itself (see closing discussions).
Clarifying the information in the "Understanding the binary package format" section
i came to the "Understanding the binary package format" section of this page due to trying to understand the format of the xpak that gets downloaded by the gentoo-kernel-bin ebuild. In that particular case, the compression format is xz, not bz2; but qtbz2 can nevertheless be used to split that file into its .tar.xz and the xpak archive components, despite the command name, and despite this not being documented in the man page for qtbz2. Should this section be changed to reflect all this? Flexibeast (talk) 12:52, 9 March 2022 (UTC)
- Excellent observation. Perhaps the developer of the sys-kernel/gentoo-kernel-bin package has set custom compression formats on the build machine that is assembling the package... which was discovered by you as you manually disassembled the package step-by-step. binpkg compression can be changed to support various compression algorithms such as bzip2, gzip, lz4, lzip, lzop, xz, and zstd (search for BINPKG_COMPRESS on the man make.conf) for more information... --Maffblaster (talk) 00:28, 10 March 2022 (UTC)
Mixing the formats XPAK and GPKG
Are there any implications when switching an existing build server to GPKG? Is it enough to configure the format in make.conf? Do clients using the binary packages get along with some as XPAK and others as GPKG? According to this page it does not seem necessary to rebuild all packages to GPKG nor clean XPAK from the packages dir? Is that correct?--Onkobu (talk) 15:48, 2 January 2023 (UTC)
- I don't think there are any implications, as long as all clients support all used formats (GPKG is a newer addition). You should be able to just set the format in make.conf on the server, doing so on the client is not necessary. Clients will use whatever the build server sets in $PKGDIR/Packages. Portage saves the binary package filename separately for each package in $PKGDIR/Packages so it will know which file to use. Rebuilding all packages to the same format is not necessary. After switching file format, the old binary package file will remain until cleaned with eclean packages. Jjakob (talk) 15:31, 22 May 2023 (UTC)
Binary package OpenPGP signing
I tried to enable this on my machine. I symlinked root's gpg-agent socket to my user's agent extra socket so that root can use my user's gpg smartcard. Imported my gpg public key into root's truststore and set it to ultimate trust. I tried signing some git commits as root and it worked so I know this part works. Enabled signing by setting
"/etc/portage/make.conf"BINPKG_FORMAT="gpkg"
FEATURES="buildpkg binpkg-signing"
BINPKG_GPG_SIGNING_GPG_HOME="/root/.gnupg"
BINPKG_GPG_SIGNING_KEY="my key fingerprint"
When I run emerge the package is successfully built and signed (I am requested to press the UIF touch button on my key 3 times after the package is built) but then it fails to verify its signature:
root #"emerge -1 dev-libs/tree-sitter"... cut ...
>>> Completed installing dev-libs/tree-sitter-0.20.8 into /var/tmp/portage/dev-libs/tree-sitter-0.20.8/image
... cut ...
>>> Done.
!!!
gpg: keyblock resource '/etc/portage/gnupg/pubring.kbx': No such file or directory
[GNUPG:] ERROR add_keyblock_resource 33587281
[GNUPG:] PLAINTEXT 74 0
[GNUPG:] NEWSIG
gpg: Signature made Mon 22 May 2023 04:51:16 PM CEST
gpg: using EDDSA key FE4A3C99DCC974A683D406031E936484FFE8BA69
[GNUPG:] ERROR keydb_search 33554445
[GNUPG:] ERROR keydb_search 33554445
[GNUPG:] ERRSIG 1E936484FFE8BA69 22 10 01 1684767076 9 FE4A3C99DCC974A683D406031E936484FFE8BA69
[GNUPG:] NO_PUBKEY 1E936484FFE8BA69
gpg: Can't check signature: No public key
gpg: can't create `/etc/portage/gnupg/random_seed': No such file or directory
!!! Invalid binary package: '/var/cache/binpkgs/dev-libs/tree-sitter-0.20.8.gpkg.tar.3271', GPG verify failed
Exception in callback AsynchronousTask._exit_listener_cb(<bound method...7f26fed69c40>>)
handle: <Handle AsynchronousTask._exit_listener_cb(<bound method...7f26fed69c40>>)>
Traceback (most recent call last):
File "/usr/lib/python3.11/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/usr/lib/python3.11/site-packages/_emerge/AsynchronousTask.py", line 210, in _exit_listener_cb
listener(self)
File "/usr/lib/python3.11/site-packages/_emerge/TaskSequence.py", line 49, in _task_exit_handler
self._start_next_task()
File "/usr/lib/python3.11/site-packages/_emerge/TaskSequence.py", line 43, in _start_next_task
self._start_task(task, self._task_exit_handler)
File "/usr/lib/python3.11/site-packages/_emerge/CompositeTask.py", line 112, in _start_task
task.start()
File "/usr/lib/python3.11/site-packages/_emerge/AsynchronousTask.py", line 34, in start
self._start()
File "/usr/lib/python3.11/site-packages/_emerge/EbuildBuild.py", line 518, in _start
self.ebuild_build._record_binpkg_info(self.ebuild_binpkg)
File "/usr/lib/python3.11/site-packages/_emerge/EbuildBuild.py", line 560, in _record_binpkg_info
"BINPKGMD5": f"{pkg._metadata['MD5']}\n",
^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute '_metadata'
Terminated
Is the section missing an instruction to also follow the Binary_package_guide#Verify_binary_package.27s_OpenPGP_signature section? Or is my global "USE=verify-sig" causing this?
Jjakob (talk) 15:16, 22 May 2023 (UTC)
I am having the same issue, emerge --info | grep -i home yields : PORTAGE_GPG_SIGNING_COMMAND="gpg --sign --digest-algo SHA256 --clearsign --yes --default-key "${PORTAGE_GPG_KEY}" --homedir "${PORTAGE_GPG_DIR}" "${FILE}""
I didn't see either PORTAGE_GPG_DIR or PORTAGE_GPG_KEY in the environmental variables. so I created them in the make.conf (this could be a version issue/oversight)
Also, I moved the homedir to portage (/var/lib/portage/home), as I use the portage user. After setting the read permissions on 'other' I was able for the process to read trusted.db, and I was able to successfully build a gpkg package. Also, depending on the situation, the .gnupg directory appears to need 'other' accesss, even though emerge uses root. which is weird. anyways.
I am happy to allow portage to have it's own gpg home directory/key. And I am avoiding using the root directory for now Also, if I opt not to have all four (BINPKG or PORTAGE prefixed) variables set, i get errors. during binpkg creation.
- ALSO I ADDED :
BINPKG_GPG_VERIFY_GPG_HOME=".../.gnupg" ... I have to say, its quite convoluted, perhaps simplifying the gpg variables if its possible, or perhaps just a rewrite in the docs.
--ciaran
pcxmac (talk) 08:32, 13 Oct 2023 (PST)
gpg-keepalive feature
I have tried to gpg-keepalive feature, and it does not seem to work. If the compilation last too long, the GPG passphrase is again asked at the end. If I type it, it appears in clear text. Then if I type enter, it hangs, the binary package is not created and the package is not installed. Any idea on this ?-- FrancoisVal (talk) 13:34, 5 January 2024 (CET)
- Have you tried contacting support, FrancoisVal ? If you find a way to fix this, please do add a tip or troubleshoot entry to the article so that future readers can avoid the issue :) -- Ris (talk) 08:46, 2 March 2024 (UTC)
"chroot" and "cross-compile" not enough for my (common ?) use case
Feel free to reopen
My use case is that I want to build on amd64 for amd64 but with completely different cpu features.
Sometimes the target has features that are not on the host.
I know about these 3 ways :
- crosscompiling with emerge-x86_64-mycustomtargetname-linux-gnu instead of emerge-x86_64-pc-linux-gnu (made by crossdev)
- emerge --root=/usr/mycustomtargetname/ --sysroot=/usr/mycustomtargetname
- chroot
The problems I had:
- with 1. : crossdev refused to build a cross compiler with the same tuple (emerge-x86_64-pc-linux-gnu). Then I used another tuple like : cross-x86_64-mycustomtargetname-linux-gnu-gcc
- with 2. : some packages built with the wrong flags and illegal instructions were embedded. I think sqlite was the culprit.
- with 3. : what is described under point 8. "What Goes Wrong" here in this article: User:NeddySeagoon/Build_In_A_Chroot
Way 1 seems to work.
My hope is that the packages respect the flags more strictly if they think
that they are cross-compiling because the CHOST and CBUILD tuples are different
with way 1.
For the binpkgs to be accepted by the target host I also had to change the target hosts
CHOST. But this worked like so: Changing_the_CHOST_variable
Way 1 is the only way I found to do this.
A nice side effect of this approach is that it automatically refuses to install packages with the wrong cflags/opcodes, because of the different CHOST.
Isn't this a somewhat common use case ?
Alexander-n8hgeg5e (talk) 16:03, 29 February 2024 (UTC)
Now I figured out that some packages like dev-libs/gobject-introspection
refuse to cross-compile.
I guess "Way 3" but inside a virtual machine is the prevailing method.
I think it should be mentioned in the article.
Alexander-n8hgeg5e (talk) 16:03, 29 February 2024 (UTC)
- I don't think it's possible to use a chroot with CPU functions that are not available on the host. Me, personally, I've just started to use a chroot on an amd64 Ryzen Arch Linux as host (would be march=znver3, but since it's Arch it's actually x86-64 baseline), the target is an amd64 Intel Core i-3000 Gentoo (march=ivybridge). All the functions that the chrooted environment use are available on the host, even though one is Intel and one is AMD, because the Intel arch is so much older. At least I hope that that's the case, but so far everything worked.
- If both systems are AMD64, the CHOST should be identical anyways: x86_64-pc-linux-gnu. I would recommend to step down to the lowest CFLAGS on the target system that is also supported by the host. psABI architecture levels also come to mind, as in e.g. x86-64-v2. Ubuntu has a blog page that sums this up nicely, here. GCC uses them with option -march=, so in your target system's /etc/portage/make.conf you'd use CFLAGS="-march=x86-64-v2". Or -v3 or none, just "x86-64", whatever both systems support.
- I tried crossdev (cross-compiling) once on an amd64 host for a ppc64 target, but I couldn't make it work.
- Luttztfz (talk) 20:46, 29 February 2024 (UTC)
- Hm, I think you are right. Now after reading your comment I think way 3(chroot) and way 2(emerge --root= --syroot=) is the usual approach if you are on the same architecture like amd64.
- The virtual machine seems not to be the solution because the vm could have a big performance problem if it had to emulate a feature inside the chroot.
- I got very far with this cross compile approach. In one instance it compiled the whole system (ceph node) and only one package (scipy) had to be compiled on the target. But I couldn't automatically fetch the package back to the subtree (/usr/x86_64-mycustomtargetname-linux-gnu) because portage refused somehow to use a binhost declared inside the subtree.
- I used the same profile and everything as on the target and somehow I had to recompile cross-x86_64-something-linux-gnu/gcc a few times.
- Alexander-n8hgeg5e (talk) 23:51, 1 March 2024 (UTC)
- Looks like you might get more help from support, Alexander-n8hgeg5e . If you manage to find out the best way to cross compile for your use, please do make the article more clear for everyone else :) -- Ris (talk) 08:42, 2 March 2024 (UTC)
Verify binary package OpenPGP signatures
I think this section should provide a recommended package and therefore this paragraph:
1. Generate (or use an existing) key with signing abilities, and export the public key to a file.
might be altered to state (perhaps including a colorized console consistent with this page's style):
1. If you do not have an existing key with signing abilities, emerge app-crypt/gnupg and then generate a key. See Gentoo's wiki GnuPG.
Jlpoole (talk) 02:18, 30 March 2024 (UTC)
Selective usage of binhost for only some (jumbo) packages?
I want to use this, just for qtwebengine and libreoffice, and am surprised to see that -bin packages are still a thing.
In my eyes, ideally, setting package.use to bindist would automatically ignore any non-matching make option, USE, etc, and just pull in a binary package when you emerge that package. Making -bin packages unnecessary.
But there is no mention of anything like that in this article, and currently, it seems to be an all or nothing approach with emerge. (Using any of -k -K -g -G with Gentoo’s binary package repo enabled, results in all packages being binary, modulo some details.)
The only workaround I know, is to pull only those packages from Gentoo’s binary repo manually, and place them in a local binary repo, disable Gentoo’s one, and then enable -k. But then I have to absolutely make sure (in a sync hook) that those packages are available in the right version locally, and still mess with package-specific options, no?
I’m new again to Gentoo. (Had to not use it since ~2017.) So excuse me, if I’m still not up to date on some modern knowledge. :)
— Evi1m4chine (talk) 14:22, 21 October 2024 (UTC)