Project:Infrastructure/Mirrors/Wrangling

From Gentoo Wiki
Jump to:navigation Jump to:search

This guide lists the steps needed to properly wrangle (manage) new Gentoo distfile mirrors.

Mirror management

Preparing and testing mirrors

1. New mirrors should file a bug under the "Mirrors" product, "New Server" component. If a request comes in via email, ask them to file the request on Bugzilla.

2. Ask for the following information on the bug, if not provided already:

  • Official mirror name (preferably not the mirror's hostname, but the name of the organization hosting the mirror).
  • Mirror type: release/distfiles and/or gentoo-portage. Note that gentoo-portage (i.e. the Gentoo ebuild repository / ::gentoo) is always rsync only. Some mirrors may have rsync for release/distfiles as well. Do not be confused by the difference!
  • URLs: full protocol and path for release/distfile mirrors, hostname or IP for gentoo-portage mirrors
  • Hardware specifications
  • Bandwidth specifications (transfer speed, eg 100Mbit, and any limits that should be mentioned)
  • Location
  • Intended concurrent users (for each protocol, mirror type)
  • Admin contact name and email address (This address should be registered in Bugzilla so we can CC you to any bug reports)

3. Modify the summary to have the type of mirror and the city as well as country if possible.

4. Inform the mirror of testing policies, tell them the due date (when testing will be completed) and give them the following links to make sure their mirror(s) have been setup correctly:

Our policy right now for testing duration is as follows:

  • New distfiles/gentoo-portage mirrors: 2 weeks from date of infra accepting the bug
  • Returning distfiles/gentoo-portage mirrors: Same hardware/IP - 1 week. Different hardware/IP - 2 weeks
  • Problematic mirrors: 1 week to fix the issue, else they're removed from rotation unless we're provided with valid reasons by the admins.
  • Rsync mirrors in any continental rotation on DNS with issues should be removed immediately from those until the issue is resolved. For example, rsync.namerica.gentoo.org, rsync.europe.gentoo.org. You can leave the mirror in its country's rotation.

Now the mirror is ready to be added officially.

Adding mirrors

For distfiles mirrors

  1. Add the mirror distfiles.xml file in the api.g.o repository: files/mirrors/distfiles.xml (follow the same markup format!).
  2. Send email to mirror admin(s) (and CC mirror-admin@gentoo.org!)
CODE Example email
Hi,

We're glad to have you as our newest source mirror, welcome aboard!

Use this information to sync with our private mirror every 4 hours from
Midnight local time. This is for mirror admins only, please do not share
this information.

----------------------------------------

rsync module: masterdistfiles-{eu,us}.gentoo.org::gentoo

rsync -rlptDq --delete -H masterdistfiles-{eu,us}..gentoo.org::gentoo /your/local/path/goes/here

Please note that we have received several reports of problems when using
the -z (compress) option of rsync.  Thus, please DO NOT use this option
when syncing against this mirror.

---------------------------------------

Please also make sure you are subscribed to gentoo-mirrors@gentoo.org as
we often send out administrative notices and policy changes via this
list.

Thanks for helping Gentoo!

Cheers,
$name
  1. Add the mirror/admin-email to the infra wiki's mirrorcontact page.
  2. Put in the following comment and close the bug:
CODE Reply comment template
Thanks for helping out Gentoo!

I have added your mirror to our list at https://www.gentoo.org/downloads/mirrors/.

Please start syncing from masterdistfiles.gentoo.org as the email I have sent to you instructs.

We would like you to subscribe to the gentoo-mirrors Mailing List by sending an email to gentoo-mirrors+subscribe@lists.gentoo.org, as we send out notifications of updates, releases, and changes in policy to the mirrors.

Please also monitor https://mirrorstats.gentoo.org/rsync/ as this application is used to keep track of the up-to-date nature of our mirrors. Thanks for the hardware and bandwidth!

For gentoo-portage mirrors

Note
TTL for rotations are obviously lower, so that they can be modified quickly if needed.
  1. Add new host to the ACLs. Follow the format, re-use existing numbers for a country before adding new ones. To add an IP to the ACL:
user $${EDITOR} rsync.yaml
user $bin/gen_masterportage_acl < rsync.yaml > hosts.allow.master
user $git commit rsync.yaml hosts.allow.master
  1. To add a new country, just make sure it isn't already there, then create one.
    Note
    If you've added a new country to this file, chances are countryTLD.gentoo.org doesn't exist, check FIRST and if nothing, create one using the "Add A Domain Page". Email infra@gentoo.org (CC mirror-admin@gentoo.org) to notify them of the new sub-domain.
  2. Add the IP into the country (and if needed, continent) DNS rotation(s) in git-based DNS. For example, a new mirror in the US would be rsyncNN.us.gentoo.org, so you will add the new IP:
CODE Managing DNS rotation
rsyncNN.us.gentoo.org - TTL - 86400

(US/Country rotation)
rsync.us.gentoo.org - TTL - 1800

(Continental rotation)
rsync.namerica.gentoo.org - TTL - 1800
  1. Add the mirror to git git.gentoo.org/data/api/files/mirrors/rsync.xml (check the format of the doc, follow the same).
  2. Add the mirror/admin-email to the infra wiki.
  3. Put in the following comment (modify as needed) and close the bug:
CODE Reply comment template
Thanks for helping out Gentoo.

I have added the IP you specified in the bug to our access database, please start syncing to masterportage.gentoo.org per our rsync document at wiki.gentoo.org/wiki/Project:Infrastructure/Mirrors/Rsync.

We would like you to subscribe to the gentoo-mirrors mailing list by sending an email to gentoo-mirrors+subscribe@lists.gentoo.org, as we send our notifications of updates, releases, and changes in policy to the mirrors.

Once again, thanks for the hardware and bandwidth! Please also monitor https://mirrorstats.gentoo.org/rsync/ as this application is used to keep track of the up-to-date nature of our mirrors.

You are mirror rsyncNN.$country.gentoo.org in the rsync.$country.gentoo.org rotation and have been added to the rsync.$continent.gentoo.org rotation as well.

Removing mirrors

For distfiles mirrors

  1. Remove the mirror from git: git.gentoo.org/data/api/files/mirrors/distfiles.xml (check the format of the doc, follow the same).
  2. Remove the mirror/admin-email from the infra wiki.
  3. Put in the following comment and close the bug:
Removed $server_name from mirrors listings. Thanks for the hardware and bandwidth donation, your services helped many Gentoo Users. (You have no more obligation to monitor the gentoo-mirrors mailing list.)

For gentoo-portage mirrors

Note
If you are removing the last mirror in a rsync.$CC rotation, you should talk to infra and make sure there is a solution so users do not try to sync to an dead address.
  1. Remove the IP from osprey.gentoo.org at /home/gmirror/rsync-hosts-allow/hosts.allow.master. Leave the existing rsyncX.$CC comment so that it is known that it should be reused.
CODE Removing an IP from the ACL
$ git checkout git+ssh://git@git.gentoo.org/infra/masterportage-acl
$ ${EDITOR} hosts.allow.master
$ git commit hosts.allow.master
  1. Remove the IP from the country DNS rotation(s) in git-based DNS
  2. Remove the mirror from CVS gentoo/xml/htdocs/main/en/mirrors-rsync-data.xml (check the format of the doc, follow the same).
  3. Put in the following comment (modify as needed) and close the bug:
Removed $server_name from ACLs, removed from DNS. Thanks for the hardware and bandwidth donation, your services helped many Gentoo users. (You have no more obligation to monitor the gentoo-mirrors mailing list.)

This page is based on a document formerly found on our main website gentoo.org.
The following people contributed to the original document: robbat2, fox2mike, nightmorph
They are listed here because wiki history does not allow for any external attribution. If you edit the wiki article, please do not add yourself here; your contributions are recorded on each article's associated history page.

Retrieved from "https://wiki.gentoo.org/index.php?title=Project:Infrastructure/Mirrors/Wrangling&oldid=1259269"