User:Hummer12007/pomu

pomu is a project conceived to simplify importing packages from several sources into a portage repository (managed by git). It shall allow importing and maintaining (with updates) packages from multiple conflicts, searches of gpo.zugaina.org, management of updates.

Rationale

The portage tree can not fully satisfy needs of every user. For this reason, overlays exist to provide packages which can not make it (due to various reasons) in the main tree. Additionally, some packages in the tree (or overlays) have not been subjected to rigorous QA procedures and may exhibit some issues including, but not limited to, build failure on some configurations. Occasionally, package versions lag behind upstream for months due to lack of manpower. So one sometimes has to temporarily edit ebuilds in the tree until the bugs are fixed,or add ebuilds for newer versions, or even make some changes which would not be accepted in the main tree.

Occasionally, fixes are already written (but stuck in stale bugs on bugzilla or github PRs), so fetching them from there may be useful.

Using many overlays may turn out problematic as well (especially ones with high divergence from main tree),causing package conflicts and unexpected behaviour. To minimize risk, one may want to reduce the number of overlays used to a minimum, only pulling required packages in a local overlay (which sometimes requires importing other packages and/or eclasses).

As such, one can see a need for a means to semi-automatically import ebuilds from varied sources: main/overlay trees, bugzilla, PRs/VCSs, and text files; maintaining changesets, with operations done atomically (with atomic git commits).

Therefore, proposed is a utility, which would integrate ebuilds from various sources into a local ebuild repository,which, preferably, may be managed with git.

Prerequisites

pomu requires the following packages to be installed:

sys-apps/portage	: Portage repository querying and management
app-portage/repoman	: Manifest generation
dev-python/click	: CLI implementation
dev-python/git-python	: Git repository management and initialization

In addition to the necessary dependencies, pomu requires a git ebuild repository (optionally set up with portage), though it can create a new one as well (with or without setting it up in portage).

Installation

An ebuild is available at https://github.com/Hummer12007/h-overlay/blob/master/app-portage/pomu/pomu-9999.ebuild (one could add the https://github.com/Hummer12007/h-overlay overlay).

To install pomu, one should

root #emerge --ask pomu

Design

pomu is convieved as a command-line utility which would allow the user to import packages (ebuilds) from different sources into a local ebuild repository (and set the repository up), with an option to sync the ebuilds subsequently (with remote changes). Removal of ebuilds should, of course, be possible. Another planned feature is gpo.zugaina.org searching.

Planned functionality

1) create/use a git repository/directory, in a configurable location (default may be /usr/local/portage), with an option to set it up as a repo in /etc/portage/repos.conf;

2) import ebuilds and related files (FILESDIR, eclasses, dependent ebuilds (if required) etc.) in atomic commits with meaningful descriptions;

3) import packages from:

• the portage tree
• overlays (either in fs or in remote repositories, including, but not limited to, overlays in the layman

catalog, arbitrary git repos (with proper layout) etc.)

• local/remote text files
• bugzilla tickets (from ebuild/patch attachments, and (optionally) from links in comments (at least (may

be extended) serving text/plain content))

• github pull requests (may be extended to support repos other than gentoo/gentoo)
• (possibly) some sane pastebin services (like github gists, paste.pound-python.org etc.);

4) when importing packages:

• (from overlays) detect dependencies/eclasses unique to the overlay (unavailable in parent overlays/main

tree) and (prompt the user to) import them

• (from overlays/portage) detect user changes and merge remote changes hunk by hunk (in a manner

similar to dispatch-conf)

• allow the user to specify package category/name (when importing from files/pastebins/patches, and new

package version

• generate required files (Manifests etc.)

4) search for packages, interfacing with gpo.zugaina.org;

5) update ebuilds pulled in from portage/overlays, merging upstream and user changes;

6) remove packages from the repository (if possible, by simply reverting the relevant commits, unless some non-leaf packages/eclasses pulled in by the package are required for another one, ergo we should track dependencies).

The utility shall operate on a (git-managed) portage overlay, storing the required data under the metadata/pomu directory.

Metadata

The required data may include a list of the root imported packages (directly imported by the user), and a directory hierarchy (similar to portage's /var/db/pkg) storing data required for the packages. For any given package, the following may be stored: a list of its parents/children (packages and eclasses), as well as user-made changes stored as patches and the original file hashsum (if applicable), a version etc.

Metadata would include a metadata/pomu/pkgs file (a list of packages pulled in by the user (non-significant untill introduction of dependencies) in the format app-category/name). For each added package, it would maintain a metadata/pomu/app-category/name directory, containing: details about the package source (portage repo, uri to the ebuild, path to the local file), (in future) metadata required for tracking changes, for portage source: full package atom of the current version in the repo, list of all files added by the package. Included files, for the time being:

FILES (package contents), BACKEND (used package source), VERSION (package version), PORTAGE_DATA (portage info for portage packages), FS_ORIG_PATH (original file path for local ebuilds).

Dependencies

When importing a package from a source, it may be required to additionally import other packages and/org eclasses from the repository to complement those missing on the user's system. Therefore dependencies should be tracked, so that the deps and/or eclasses are updated on sync as well, and unnecessary packages can be removed.

Package source modules and specifications

The potential list of package sources: portage repos, layman overlays, other remote repos, local and remote (uri-specifed) ebuild files (including some pastebin services), bgo bugs, github prs (potentially not only gentoo/gentoo ones).

A package can be specified the following (fully qualified) way: SOURCE:NAME, where source is the name of the source (e.g. bug:, pr:, uri:, portage:, and (more complex) portage/REPONAME (for a given portage repo), layman/REPONAME, repo/URI, pr/USER/REPO). The range of allowed values for NAME is determined by the package source, for instance, portage: may accept a package atom (which would match the best (may differ from pms definition) version), or a package name (with or without a category), which would select a package from a repo with highest priority (the best version), and some stuff similar to this. portage/repo: would select a repo. Similar names should be given to layman and repo sources. file: shall accept a filesystem path, uri: shall accept a uri, pr: and bug: shall accept a ticket number.

Of course, some details can be omitted (which would be inferred from other details), for instance a filesystem path can be specified without a file: prefix which is inferred, similarly with uris (where b.g.o uris can resolve to bug: values), a package atom shall refer to local portage repos et cetera.

Implemented package source modules: portage (portage packages), fs (ebuilds from the local filesystem).

Importing user changes

User changes can be imported (accounted for and commited) with the help of the pomu commit command. When run (without any options) it selects changes for each tracked package (included in the metadata/pomu directory), incorporates them in the data, and performs the commit (separately for each tracked package).

The following changes are possible:

• user modifies the directory in the repository (edits, the ebuild or some files in the filesdir, moves something (version bump), adds files to the FILESDIR etc.)
• user modifies the patches in the metadata directory
• user adds a patch to the metadata directory
• user edits the PATCHES_ORDER file (reordering or removing lines)
• user adds a patch AND edits the PATCHES_ORDER file

The proper way to accomodate some of these changes is yet to be determined.

The command should pregenerate the commit message (editable though), offer a command to commit everything in one commit, or set the commit message.

If the directory in the repository is edited, the command shall generate a patch (from commited state to new state) and add it to the patches of the package (append it to the PATCHES_ORDER file, patches/ directory etc.)

Command line syntax

Usage: pomu [OPTIONS] COMMAND [OPTIONS] [ARGS]...

Options: --repo (specify the repository) --help --version

Commands:

 init		Initialize pomu for a repository (a new one, or a created one)
 fetch         Fetch a package, listing all its files (a directory can be specified)
 status	Displays pomu status (whether initialized in any repo)
 install	Imports an ebuild specified by a string
 uninstall	Removes the ebuild specified by a string
 show		Displays metadata for an installed package
 sync		Synchronizes the ebuilds with remotes
 search	Searches gpo.zugaina.org

The init command should initialize pomu for a provided (portage) repository or a newly created one (in a specified directory, with a default), which is then added to portage.

The status command is self-descriptive.

All the following commands shall operate on the highest/lowest priority portage repository with pomu, unless another one is provided (with pomu as well).

The import command shall import a package from a source. The package is to be specified in the form mentioned above. It may (optionally) import other packages/eclasses to complement those missing on the system.

The sync command shall fetch the newest version of the package from it source, if the source supports it and then import it, merging (if the original was modified by the user) the changes, either automatically, or by discarding changes from either side, or doing it hunk-by-hunk (interactively), or in another way (an open question). It may (optionally) remove the older version, or, if the changes of the remote have led it be the same as the state in the repo, offer to remove the package.

The remove command shall remove an ebuild and (if applicable) its descendants. A version may be provided.

The search command shall search gpo.zugaina.org by a keyword (scraping the search result page).