Project:Portage/Proposals

This is where we will flesh out ideas for the direction to take areas of portage's code.

Portage sync

Note
This proposal is now fully implemented. Please refer to the Project:Portage/Sync wiki page for it's current specification.

This is the proposed layout of a totally revamped sync system. It uses the plug-in manager already in portage.

portage.sync.pdf

Advantages

1. Flexible
2. Easily extended
3. Less maintenance, core sync manager does not need changes for new modules
4. Add on modules can be user installed for special types

Note
The basic modular sync code is working, but please remember only minimal changes were done to the existing code in order to make it work. There is still much re-factoring to do to optimize the code for the new system. It will be open for code review at a later time. Please restrict code review to the basic design algorithm.

Proposed To Do

1. Determine the information the module_spec needs to provide to the sync manager and clients <== InProgress
2. Move the plug-in manager code out of emaint's namespace, into portage's
3. Put a freeze on the current sync code to reduce the rebasing needed for the merge on completion <== Done
4. Create a new feature branch to use for this change <== Done, plugin-sync
5. Code the sync manager code <== Preliminary coding done.
6. Determine the method to be able to do concurrent repo syncs
7. Possibly move the TaskHandler class into the SyncManager class or subclass it
8. Make the sync manager provide any common services functions such as dropping from root down to user privileges <== Done
9. Move the sync code to a new rsync module <== Done
10. Move the git code to it's own module <== Done
11. re-write the code in action_sync(). <== minimally migrated to the new SyncManager class
12. Begin testing, debugging <== Rsync, git tested working. CVS untested
13. Add to module_spec for repos.conf variables it accepts/requires
14. Add repos.conf validation checks functions
15. Determine the location/directory for 3rd party sync modules to be installed to.
16. Re-factor the code to optimize for the new structure.
17. Move the web-rsync code from emerge into a module
18. Modify emerge-webrsync cli to use the new webrsync module
19. MORE testing, debugging.
20. merge the branch back into master

Examples

CODE __init__.py from the new rsync plug-in module. (Very basic)

# Copyright 2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

"""Rsync plug-in module for portage.
   Performs rsync transfers on repositories
"""


module_spec = {
	'name': 'rsync',
	'description': __doc__,
	'provides':{
		'module1': {
			'name': "rsync",
			'class': "RsyncSync",
			'description': __doc__,
			'functions': ['sync',],
			'func_desc': {'sync': 'Performs rsync transfers on the repository'}
			}
		}
	}

Example Proposed extensions

CODE __init__.py for a new extended git plug-in module

# Copyright 2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

"""Rsync plug-in module for portage.
   Performs a git operations on repositories
"""


module_spec = {
	'name': 'git',
	'description': __doc__,
	'provides':{
		'module1': {
			'name': "git",
			'class': "GitSync",
			'description': __doc__,
			'functions': ['sync', 'install', 'validate',],
			'func_desc': {
                            'sync': 'Performs a git pull on the repository',
                            'install': 'Installs a new git clone of the repository',
                            'validate': 'performs validation checks on the repos.conf ' \
                                        'entries for the repository',
                            },
                         'conf': {
                             'location': 'Repo location eg: /var/lib/gentoo/repos/gentoo',
                             'sync-type': 'The sync module type associated with the repo.  eg: "git"',
                             'sync-uri': 'the uri to use for initial installation',
                             {,
			}
		}
	}

Discussion

portage-dev mail list threads

• topic: New proposed modular sync system

1. gmane thread

To be discussed at the next portage team meeting, Feb. 9, 2014 @ 19:00 UTC in #gentoo-meetings

Build environment plugins

Introduction

Portage is getting overcomplex with growing number of build-env quirks and FEATURES. Would be good to figure out a nice plugin API to handle them.

Potential build-env plugins

The following features could be split into separate plugins:

• ccache,
• distcc,
• fakeroot,
• fuse-sandbox (in TODO),
• ipc-sandbox,
• network-sandbox,
• sandbox,
• userpriv.

Required facilities

Based on the potential plugins, the plugin system needs to provide at least:

• entry/exit hooks for phase execution,
• environment manipulation,
• convenient wrapper creation and PATH manipulation,
• ordering facilities — i.e. plugins requiring root privileges would have to be ordered before userpriv.