Project:Infrastructure/Mirrors/Rsync

Rsync (Portage tree) mirrors distribute the Gentoo ebuild repository to our users.

The Gentoo project operates the master mirror as well as the so-called main rotation mirrors in a few locations worldwide. Additionally, community members provide resources in form of community mirrors that help distribute the load and service locations that are not covered by any main rotation mirrors.

This document provides direction for parties interested in hosting a community rsync mirror.

Requirements

Bandwidth: You need to have a minimum of 5 MBit/s upstream capacity on a permanent connection. DSL, cable, or other dial-up connections are not suited for hosting a mirror.
Storage: Hosting a mirror requires a minimum of 2 GiB available disk space.
Transmission protocols: As the name indicates, your mirror needs to serve its contents via the rsync protocol. IPv6 connectivity is strongly encouraged.
Simultaneous connections: We ask that you support a minimum of 15 and preferably 30 or more concurrent user connections.
Update frequency: Your mirror needs to update every 30 minutes: Once between :00 and :10, and once between :30 and :40. Updates need to be exactly 30 minutes apart (for instance at :08 and at :38).
Contact and information policy: Your administrative contact must react to enquiries as soon as possible and resolve issues within two weeks. We expect to be notified in the event of any changes or prolonged outages.

Becoming a community mirror

If your hardware and connection fulfill the requirements stated above, we would love to have you as a community mirror.

Note
Please note that we currently have enough rsync mirrors in many regions such as Europe or North America.
Your request to become a community mirror may not be processed for quite a bit of time unless your mirror is in a region lacking coverage.
You can contact us via email or IRC to find out if your location requires additional bandwidth.

That being said, we are looking for additional main rotation mirrors worldwide, i.e. ones that are managed by Gentoo, with you just providing hardware and connectivity. Should you be interested, please get in touch!

Initial setup

Fetching data

First, you need to fetch the mirror data. The following example script illustrates the process:

FILE /opt/gentoo-rsync/rsync-gentoo-portage.sh

#!/bin/bash
  
RSYNC="/usr/bin/rsync"
OPTS="--quiet --recursive --links --perms --times -D --delete --timeout=300 --checksum"
#Uncomment the following line only if you have been granted access to masterportage.gentoo.org
#SRC="rsync://masterportage.gentoo.org/gentoo-portage"
#If you are waiting for access to our master mirror, select one of our mirrors to mirror from:
#SRC="rsync://rsync.de.gentoo.org/gentoo-portage" # for Europe
#SRC="rsync://rsync.us.gentoo.org/gentoo-portage" # for the rest of the world
DST="/space/gentoo/rsync/"
  
echo "Started update at" `date` >> $0.log 2>&1
logger -t rsync "re-rsyncing the gentoo-portage tree"
${RSYNC} ${OPTS} ${SRC} ${DST} >> $0.log 2>&1
  
echo "End: "`date` >> $0.log 2>&1

Uncomment the appropriate SRC line depending on where your mirror is located. You will get access to masterportage.gentoo.org when your mirror is accepted as a community mirror.

Serving data

rsyncd needs to be configured to serve the repository via the gentoo-portage rsync module. Other things like the storage path are of course up to you.

Important
For security reasons, the use of a chrooted environment is required!

This is a sample configuration:

FILE /etc/rsyncd.conf

uid = nobody
gid = nobody
use chroot = yes
max connections = 15
pid file = /var/run/rsyncd.pid
motd file = /etc/rsync/rsyncd.motd
log file = /var/log/rsync.log
transfer logging = yes
log format = %t %a %m %f %b
syslog facility = local3
timeout = 300
  
[gentoo-portage]
path = /gentoo/rsync
comment = Gentoo Linux Portage tree mirror
exclude = distfiles

Your rsyncd.motd should contain your IP address and other relevant information about your mirror, such as information about the host providing the Portage mirror and an administrative contact.

Requesting inclusion

When your mirror has completed fetching the data and your update cronjob is in place, please file a ticket to inform us about your new mirror: File ticket

You need to have an account on bugs.gentoo.org and be logged in to file the ticket. Please fill in all fields with the relevant information, and include whatever else you think may be important to let us know.

Inclusion process

When we are ready to process your request, your mirror is added to our mirror monitoring system for a period of at least 2 weeks. During this time, we will update you with any issues we find on the ticket you have filed.

Should the service quality of your mirror meet our standards, we will then add it to the community mirror listing and provide you with access to the master rsync mirror. Your host will also be aliased with a name of the form: rsync[num].[country code].gentoo.org.

If any issues remain after a certain period of time that would impact the service negatively, we may decline your request. In this case, you can re-request the inclusion of your mirror after a period of 6 months.

After the inclusion

First of all, thank you for contributing to the Gentoo project.

Please ensure that your administrative contact is subscribed to the low traffic gentoo-mirrors mailing list which is used to publish announcements for all mirror admins. You can do so by sending an empty message to gentoo-mirrors+subscribe@lists.gentoo.org.

When anything changes (be it that you're physically moving, or need to change the IP addresses, or have a new admin contact), please be sure to let us know immediately. You can reach us via email at mirror-admin@gentoo.org, or you can file a ticket on bugs.gentoo.org in the Mirrors product.