I run a GitLab server that I use to collaborate with coworkers.¹ It’s a custom-built machine with a RAID, lots of memory, redundant power supplies, and is running Ubuntu. Every day, a simple cron script to makes a backup copy of the data on the server to another machine. The script looks something like this:

#!/bin/sh

cd /home/gitlab/gitlab
sudo -u gitlab -H bundle exec rake gitlab:backup:create RAILS_ENV=production
rsync -avrcpt /home/gitlab/gitlab/tmp/backups/ user@remotemachine:/path/to/gitlabtars
rsync -avrcpt /home/git/repositories/ user@remotemachine:/path/to/barerepos

This script copies a GitLab-friendly backup file to a remote machine and syncs bare Git repositories to a different place on the same remote machine (just in case GitLab goes out of style).

That keeps the data (relatively) safe, but if the primary GitLab server goes down, I’d have to build a machine from scratch, install GitLab and restore the backup before anybody could make any pushes or pulls.

The machine I’m using to store backups is also an Ubuntu machine, so I decided to make that machine a functioning “clone.” Here’s how I did it.

From the Top

First of all, disclaimer: I take no responsibility for what you do to your machine, whether you’re taking advice from this page or not. Whatever happens to your machine is your responsibility; I make no guarantees, warranties or promises. Understand what each command does before you execute it.

Second of all, this writeup explains the process I had to use to clone from a specific machine to a different specific machine, both running GitLab 4.0. It may not work perfectly for you! When I upgrade to a newer version of GitLab, I’ll write another post detailing the clone process.

On a machine with the same operating system as the primary server (ideally), use the GitLab installation instructions to get a copy of GitLab running on the clone machine. When checking out the GitLab source, checkout the same commit number that’s on the primary server onto the clone. You can find this number by issuing the git log command on the primary server from the directory where GitLab was cloned (in my case, /home/gitlab/gitlab/). It’ll be the top commit number. So, in step 6 of the GitLab install directions, change the checkout command to:

sudo -u gitlab -H git checkout [commit number]

As per the instructions of people who solved restoration problems, it’s necessary to make sure all the permissions on the repository directories are correct.²

sudo chmod -R ug+rwX,o-rwx /home/git/repositories/
sudo chmod -R u-s /home/git/repositories/
sudo find /home/git/repositories/ -type d -print0 | sudo xargs -0 chmod g+s

In GitLab 4.0, the restore backup task contains a couple sudo commands that end up halting the task and asking for the gitlab user’s password. If you followed the GitLab installation directions properly, when you created the gitlab user you specified the --disabled-login option, which means there’s nothing you can do to make those commands complete. Might as well remove them. So, edit the file /home/gitlab/gitlab/lib/tasks/gitlab/backup.rake to remove the problematic commands. Lines 139-142 should be changed to look like this:³

permission_commands = [
         "",
         ""
]

Now run the backup restore task:

sudo -u gitlab -H bundle exec rake gitlab:backup:restore RAILS_ENV=production

Hopefully the backup restore task completed successfully, and if so, you should ensure the base repository directory has the correct permissions and run two more tasks:

sudo chmod -R ug+rwXs,o-rwx /home/git/repositories/
sudo -u gitlab -H bundle exec rake gitlab:gitolite:update_repos RAILS_ENV=production
sudo -u gitlab -H bundle exec rake gitlab:gitolite:update_keys RAILS_ENV=production

With any luck, those tasks completed without errors. Now check to see if GitLab says everything is okay by running this task:

sudo -u gitlab -H bundle exec rake gitlab:check RAILS_ENV=production

That final task might result in errors but its output is pretty helpful in those cases.

Bam! You should be able to clone any repositories from the clone machine that you have access to on the primary machine, and the web interface should show the same information as the primary machine when the backup was made.

Automating the Cloning Process

With a clone of my GitLab server functioning properly, I can rest a little easier. However, there’s no reason for me to have to go through all that nonsense when I want to sync the clone, so I made a cron script for the clone machine. I’ve listed the script below—it needs to run as super user.

If you read through the script, you’ll notice it assumes a vanilla GitLab install and does other weird things like deleting and reinstating the gitolite-admin repository and setting permissions more than you’d expect. These are hacks that made the script run successfully and are required by what I assume are bugs that the GitLab team has fixed in subsequent releases. Again, use the script at your own risk. Enjoy!

#!/bin/bash

cd /home/gitlab/gitlab

# ensure permissions are correct
echo ensuring correct permissions
sudo chmod -R ug+rwX,o-rwx /home/git/repositories/
sudo chmod -R u-s /home/git/repositories/
sudo find /home/git/repositories/ -type d -print0 | sudo xargs -0 chmod g+s

# remove gitolite-admin repo to prevent errors that happen sometimes, and build new one
echo rebuilding gitolite-admin repository
sudo rm -r /home/git/repositories/gitolite-admin.git
sudo -u git -H sh -c "PATH=/home/git/bin:$PATH; gitolite setup -pk /home/git/gitlab.pub"

# remove old backup first lol!
echo removing old backup files
sudo -u gitlab -H rm tmp/backups/*

# copy new backup from external drive
echo copying new backup from backup media
bu_file=`ls /path/to/gitlabtars | tail -n 1`
sudo -u gitlab -H cp /path/to/gitlabtars/$bu_file tmp/backups/$bu_file

# restore the backup.
echo initiating gitlab backup restore rake task
sudo -u gitlab -H bundle exec rake gitlab:backup:restore RAILS_ENV=production
# ensure permissions are correct, as a matter of paranoia
sudo chmod -R ug+rwXs,o-rwx /home/git/repositories/

# update the repos and keys
echo updating gitolite repos and keys
sudo -u gitlab -H bundle exec rake gitlab:gitolite:update_repos RAILS_ENV=production
sudo -u gitlab -H bundle exec rake gitlab:gitolite:update_keys RAILS_ENV=production

# run a check.
echo running the final gitlab check to look at problems
sudo -u gitlab -H bundle exec rake gitlab:check RAILS_ENV=production