Project:Infrastructure/Gitlab

Gitlab

Gitlab is currently deployed in a testing capacity on gitlab.gentoo.org and is not yet publicly available.

Gitlab for users

Gitlab Authentication

Currently only "Gentoo SSO" is supported. This means only developers can login at this time. We expect to add other omniauth login sources later (google, github, gitlab, etc.)

Backups

Gitlab backups are taken nightly.

Updates

The current pace by gitlab upstream is 1 minor release per month. We try stay within 3 minor releases of :latest.

SSH keys

Currently we do not synchronize SSH keys with any identity platform, but we likely need to add syncing of ssh keys from LDAP.

Groups

We currently do not synchronize any group data from anywhere. Again this is an open item we need to address before going public.

SSH

The physical machine hosting gitlab has 2 IPs (both on v4 and v6.) sshing to gitlab.gentoo.org will try to connect to the specific IP for gitlab and you will be connected to gitlab's ssh.

Gitlab's ssh uses its own set of host keys and wrappers like a normal gitlab.

Gitlab for Infrastructure

Gitlab is configured a bit by puppet (see dist/gitlab) and a bit by hand.

We use the omnibus containers to deploy gitlab. The gitlab config is at /etc/gitlab/docker-compose.yml.

Starting gitlab

cd /etc/gitlab/ docker-compose up -d

Stopping gitlab

cd /etc/gitlab docker-compose down

Upgrading gitlab

We need to upgrade about every 2-4 weeks to stay up to date with gitlab development. Upgrades cause downtime, but its typically brief (15-20 minutes.) Announce it in #gentoo-dev beforehand, then:

Head to the admin area to see what version we are on: https://gitlab.gentoo.org/admin
Then head to the gitlab docker repo to see what version are available: https://hub.docker.com/r/gitlab/gitlab-ce/tags. For security updates, infra will have received an advisory including the updated versions.
Don't jump more than 1 minor version at a time (minor versions are the second version component.)
Be sure to check changelog for any breaking changes, especially if changing major or minor versions

The docker image is gitlab/gitlab-ce:<VERSION> (so for example: gitlab/gitlab-ce:14.10.0-ce.0) You can see an example tag in dist/gitlab/manifests/server.pp. Construct the new docker tag based on the next version of gitlab we need.

So, if we want to upgrade to 14.3.2; the image would be gitlab/gitlab-ce:14.3.2-ce.0. Verify this on the dockerhub!

Change the $image variable in puppet's dist/gitlab/manifests/server.pp to the new version ($image = 'gitlab/gitlab-ce:14.3.2-ce.0'), commit and push.
Optionally run puppet agent --no-daemonize --verbose on towhee; and it should take the update. If not running manually, be sure to watch puppet mails from towhee to check the output for errors.
This start may take 15-20 minutes (to run the upgrade.)
After getting successful puppet output, ensure the running version is what you expected to update to by checking https://gitlab.gentoo.org/help.

SSHing into the gitlab host for infra

Currently gitlab runs on towhee, you need to 'ssh towhee.gentoo.org' to get to the host; sshing into 'gitlab.gentoo.org' will ssh into the gitlab container, which you do not want.

Upgrading gitlab-runner

Currently gitlab-runner is being ran on inca, so you would need to 'ssh inca.gentoo.org' to get to the host. In puppet repository, bump the version for gitlab-runner under dist/gitlab/manifests/runner.pp for a matching version in dev-util/gitlab-runner. Then run puppet agent --no-daemonize --verbose to apply the changes (you can watch in htop to see it's progress).

What about Gitolite?

Currently we plan to keep gentoo repos mastered in gitolite. We can set up automatic pushes to gitlab in gitolite configs. We will consider migrating repos to gitlab in the future.

TODOs for gitlab setup

Add Prometheus monitoring/alerting
Add infra-status.gentoo.org lines for gitlab
gitlab.com omniauth
Add ssh key sync
add gpg key sync
add group sync
add more admins to gitlab
add terraform for administration?