Project:Portage/Sync

Starting with Portage version 2.2.16, Portage now has a modular, plug-in sync system. This new sync system will ease code maintenance as well as make it possible for third party modules to be installed. Many of the users will be learning about the repos.conf style changes that have been in Portage for several years now for the first time.

Portage has been migrating away from make.conf's PORTDIR and PORTDIR_OVERLAY variable settings for quite some time. Those settings are very limited in their capabilities as compared to the repos.conf style of repository configuration. The new sync system is the next step in that migration.

Using PORTDIR and PORTDIR_OVERLAY in make.conf limited Portage by only defining where the repository was located. Portage had no controls to specify other important attributes about that repository. The repos.conf configuration style allows for settings to be defined on a per repository, per sync-type basis. Along with this new expandability and the plug-in sync system, it will be much easier to change to new repository syncing methods.

General help

Note
While Portage can handle repos.conf as either a file or a directory of files. The preferred method is to be used as a directory. Other tools like layman and mirrorselect require and expect it to be a directory. Layman creates and manages its own layman.conf file to register installed overlays with Portage. mirrorselect looks for a repos.conf/gentoo.conf file in order to modify the sync-uri parameter for the gentoo repository.

Changes

/etc/portage/repos.conf/*
- New auto-sync setting for all repository types (needed):
  auto-sync = yes # default, Possible values are: yes, no, true, false
- New for git sync-type:
  sync-depth = n where n = {0,1,2,3,...}
  - 0 is equivalent to full git history
  - 1 shallow clone, only current state (default if option is absent)
  - 2, 3, ... history depth to download.
- New sync-type modules:
  - sync-type = svn
    Sync a subversion repository
  - for <=sys-apps/portage-2.2.18 sync-type = websync
  - for >=sys-apps/portage-2.2.19 sync-type = webrsync
    Perform an emerge-webrsync operation on the repo. By using this sync-type, a user can do a normal emerge --sync or emaint sync -a instead of running the emerge-webrsync command.
  - sync-type = laymansync
    (if installed) runs layman's code to sync the repository (overlay)
- New native portage postsync hooks
  - /etc/portage/postsync.d/*
    Runs all hook scripts found once, after all repos have been synced. Compatible with all existing postsync.d scripts. Replaces the portage-utils installed post_sync hook script.
  - /etc/portage/repo.postsync.d/*
    Runs each hook script found with three arguments: repo name, sync-uri, location
    Each script is run at the completion of each repository that was synced. Using these arguments, the hook script can decide if it needs to run or just exit. This is useful for when only a certain repo has been updated/synced and needs to trigger some other process or update system.

Migration

The primary reason for upgrading from make.conf style PORTDIR (deprecated, only there for backward compatibility with old tools, the repos.conf setting is used for its value) and PORTDIR_OVERLAY (still functional, but deprecated) listings and the new repos.conf style definitions is functionality. The repos.conf method is a lot more flexible and powerful for controlling the repositories. The PORTDIR_OVERLAY listings are simple listing which are not capable of listing the options to control the new sync features and abilities, let alone other portage capabilities such as priority.

Portage configuration

If the /etc/portage/repos.conf directory does not exist:

root #mkdir /etc/portage/repos.conf

Note
The next step is only needed in order to change the default settings for the gentoo ebuild repository

root #cp /usr/share/portage/config/repos.conf /etc/portage/repos.conf/gentoo.conf

Then edit the file to the desired settings, continue as needed with the remaining migration instructions. Edit all repos.conf/*.conf files, add the auto-sync option to each defined repository.

For sync-type, set it to one of the installed supported types.

Current supported sync types include:

rsync
git
svn
webrsync or websync (equivalent to running emerge-webrsync separately).^[1]
cvs
laymansync (if installed by Layman)

FILE /etc/portage/repos.conf/gentoo.conf

[gentoo]
location = /var/db/repos/gentoo
sync-type = rsync
sync-uri = rsync://rsync.gentoo.org/gentoo-portage
auto-sync = yes

Example local overlay sync-able from a git remote:

FILE /etc/portage/repos.conf/the-pit.conf

[the-pit]
location = /var/db/repos/the-pit
sync-type = git
sync-uri = git://me.org/me/the-pit.git
auto-sync = no

The above overlay can be synced with

user $emaint sync -r the-pit

Layman configuration

Manually editing /etc/portage/repos.conf/layman.conf

For an existing repos.conf/layman.conf file:

1) Change/add the sync-type:

FILE /etc/portage/repos.conf/layman.conf

[wtk]
priority = 50
location = /var/lib/layman/wtk
layman-type = git
sync-type = laymansync
sync-uri = git://tremily.us/wtk-overlay.git
auto-sync = yes

2) Ensure Layman 2.3.0 or higher has been installed with the laymansync module.

layman-updater method

Perform the next series of steps when migrating from the old make.conf layman conf_type to repos.conf type.

1) Ensure the new plug-in sync Portage version is installed, upgrading portage if necessary:

root #emerge \>=sys-apps/portage-2.2.16

2) Install the new laymansync enabled layman version. To install layman with the portage plugin, the sync-plugin-portage USE flag must be enabled:

FILE /etc/portage/package.use

app-portage/layman sync-plugin-portage

root #emerge \>=app-portage/layman-2.3.0

Note
If the repos.conf conversion has been completed prior to installing the new plug-in sync Portage and Layman versions (with its laymansync module), then the existing file will be missing the sync-type, sync-uri, and auto-sync settings. Skip to step 4 to re-generate the file with these definitions. Layman will use its own installed database to generate the Portage configuration file.

3) Edit layman.cfg change it to the repos.conf config type:

FILE /etc/layman/layman.cfg

conf_type : repos.conf

Save and exit the file.

4) Generate the new repos.conf/layman.conf file:

root #layman-updater -R

5) Delete layman's make.conf file:

root #rm /var/lib/layman/make.conf

6) Edit /etc/portage/make.conf: remove the source /var/lib/layman/make.conf line.

Done

Operation

Primary control of all sync operations has been moved from emerge to emaint. emerge --sync now runs the emaint sync module with the --auto option. This --auto option performs a sync on only those repositories with the auto-sync setting set to yes or true. If the auto-sync option is not set to yes or is absent, then emerge --sync may not sync any repositories.

Note
As a result of the default auto-sync = True/Yes setting, commands like eix-sync, esync -l, emerge --sync && layman -S will cause many repositories to be synced multiple times in a row. Please adjust configuration files or scripts as necessary for the new operation.

Warning
Due to the above default. For any repositories that should not be synced when emerge --sync or emaint sync --auto is run, be sure to set to set the auto-sync = no key/value on each repository.

The emaint sync module operates similar to layman. It can sync single or multiple repos. It can also sync repos with auto-sync = no set in the respective repos.conf file. This gives users and administrators more flexibility in how and when repositories are synced:

root #emaint sync

  emaint sync options:
    -a, --auto      Sync auto-sync enabled repos only (includes defaulted repos)
    -A, --allrepos  Sync all repos that have a sync-url defined
    -r REPO, --repo REPO
                    Sync the specified repo
    --sync-submodule {glsa,news,profiles}
                    Restrict sync to the specified
                    submodule(s)  (only rsync supported currently)

Examples

root #emaint sync -a

Equivalent to emerge --sync. Sync all repositories where auto-sync = true is set.

Note
Due to the default auto-sync = true setting, this command will sync all repositories that do not have auto-sync = no explicitly set.

root #emaint sync -r foo

Sync the foo repo (ignores auto-sync setting):

root #emaint sync --allrepos

Sync all repositories with a valid sync-type and sync-url defined. (ignores auto-sync setting)

Plug-in sync API

Module specifications

Directory structure

/usr/lib/portage/

sync/

modules/

rsync/

__init__.py

rsync.py

git/

__init__.py

git.py

__init__.py

config_checks.py

controller.py

getaddrinfo_validate.py

old_tre_timestamp.py

syncbase.py

Module's init.py

Example module __init__.py definition:

FILE portage/sync/modules/git/__init__.py

# Copyright 2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

"""Git plug-in module for portage.
Performs a git pull on repositories
"""

from portage.sync.config_checks import CheckSyncConfig

module_spec = {.
	'name': 'git',
	'description': __doc__,
	'provides':{
		'git-module': {
			'name': "git",
                        'sourcefile': "git",
			'class': "GitSync",
			'description': __doc__,
			'functions': ['sync', 'new', 'exists'],
			'func_desc': {
				'sync': 'Performs a git pull on the repository',
				'new': 'Creates the new repository at the specified location',
				'exists': 'Returns a boolean of whether the specified dir ' +
					'exists and is a valid Git repository',
			},
			'func_parameters': {
				'kwargs': {
					'type': dict,
					'description': 'Standard python **kwargs parameter format' +
						'Please refer to the sync modules specs at ' +
						'"https://wiki.gentoo.org:Project:Portage" for details',
				},
			},
			'validate_config': CheckSyncConfig,
		}
	}
}

Definition: "module_spec"

Field Name	Type	Comments
name	string	The name of the module The name of the file providing the exported classes
description	string	Typically is a pointer to the files docstring
provides : [module1] [module2]	dictionary dictionary dictionary	Dictionary of of the different sub modules it provides. Note the field identifiers must be unique, but are not used outside of the plugin system. Unique module identifier. Unique module identifier.

Definition: module_spec["provides"]["module1"]

Field Name	Type	Comments
name	string	Unique identifier of the exported class.
sourcefile	string	The filename that contains the class without the ".py" extension
class	string	the id/name of the class exported
description	string	Description of the service it performs
functions	list	The list of commands it exports
func_desc	dictionary, its keys are the same as the 'functions' list members above	strings describing each of the exported functions
validate_config	pointer	Assign it the CheckSyncConfig derived class used to validate repos.conf settings for this repo type. For simple modules, the base CheckSyncConfig is suitable. For others please refer to Validating repos.conf variables section of this page.

Validating repos.conf variables

Depending on the needs of the sync module, the "validate_config" variable can be assigned in the module's spec the portage.sync.config_checks.CheckSyncConfig class or subclass it, adding any checks for additional variables needed to be defined in the repos.conf definition for that sync type.

Example module adding an additional validation test

CODE portage/sync/modules/cvs/__init__.py

# Copyright 2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

"""CVS plug-in module for portage.
Performs a cvs up on repositories
"""


from portage.localization import _
from portage.sync.config_checks import CheckSyncConfig
from portage.util import writemsg_level


class CheckCVSConfig(CheckSyncConfig):

	def __init__(self, repo, logger):
		CheckSyncConfig.__init__(self, repo, logger)
		self.checks.append('check_cvs_repo')


	def check_cvs_repo(self):
		if self.repo.sync_cvs_repo is None:
			writemsg_level("!!! %s\n" %
				_("Repository '%s' has sync-type=cvs, but is missing sync-cvs-repo attribute")
				% self.repo.name, level=self.logger.ERROR, noiselevel=-1)


module_spec = {
	'name': 'cvs',
	'description': __doc__,
	'provides':{
		'cvs-module': {
			'name': "cvs",
			'class': "CVSSync",
			'description': __doc__,
			'functions': ['sync', 'new', 'exists'],
			'func_desc': {
				'sync': 'Performs a cvs up on the repository',
				'new': 'Creates the new repository at the specified location',
				'exists': 'Returns a boolean of whether the specified dir ' +
					'exists and is a valid CVS repository',
			},
			'func_parameters': {
				'kwargs': {
					'type': dict,
					'description': 'Standard python **kwargs parameter format' +
						'Please refer to the sync modules specs at ' +
						'"https://wiki.gentoo.org:Project:Portage" for details',
					'required-keys': ['options', 'settings', 'logger', 'repo',
						'xterm_titles', 'spawn_kwargs'],
				},
			},
			'validate_config': CheckCVSConfig,
		}
	}
}

Common classes used by sync modules

class SyncBase(object):........ Base Sync class meant to be subclassed. It defines some common functions which can be used or overriden as needed by the target module. It provides stubs for all functions needed by the main sync controller.
- def name():
- def can_progressbar(self, func):
- def __init__(self, bin_command, bin_pkg):
- def _kwargs(self, kwargs):.... Sets internal variables from kwargs
- def exists(self, **kwargs):.... Tests whether the repo actually exists
- def sync(self, **kwargs):...... Stub only: Sync the repository
- def post_sync(self, portdb, location, emerge_config):.... Stub only: repo.sync_type == "Blank" NOTE: Do this after reloading the config, in case it did not exist prior to sync, so that the config and portdb properly account for its existence.

class NewBase(object):........ Base Sync class which subclasses SyncBase and also meant to be subclassed. It adds a predefined sync(), and stubs for new() and update(). It forms the base for a module which requires separate new and update operations.
- def sync(self, **kwargs):...... Conditionally syncs the repository using either the new or update functions
- def new(self, **kwargs):....... Stub only: Do the initial download and install of the repository
- def update(self):............... Stub only: Update existing repository

class CheckSyncConfig(object):..... Base repos.conf settings checks class
- def __init__(self, repo=None, logger=None):.... Class init function
  - self.checks = ['check_uri', 'check_auto_sync']...... List of predefined checks to perform. When subclassing append to it or redfine as needed
- def repo_checks(self):......................... Perform all checks available
- def check_uri(self):.............................. Check the sync_uri setting
- def check_auto_sync(self):................. Check the auto_sync setting

↑ Renamed from websync to webrsync in Portage 2.2.19 and higher for consistency and popular demand.

[1] Renamed from websync to webrsync in Portage 2.2.19 and higher for consistency and popular demand.

[1]

Project:Portage/Sync

General help

Changes

Migration

Portage configuration

Layman configuration

Manually editing /etc/portage/repos.conf/layman.conf

layman-updater method

Operation

Examples

Plug-in sync API

Module specifications

Directory structure

Module's __init__.py

Definition: "module_spec"

Definition: module_spec["provides"]["module1"]

Validating repos.conf variables

Common classes used by sync modules

Module's init.py