I run a GitLab server that I use to collaborate with coworkers.1 It’s a custom-built machine with a RAID, lots of memory, redundant power supplies, and is running Ubuntu. Every day, a simple
cron script to makes a backup copy of the data on the server to another machine. The script looks something like this:
1 2 3 4 5 6
This script copies a GitLab-friendly backup file to a remote machine and syncs bare Git repositories to a different place on the same remote machine (just in case GitLab goes out of style).
That keeps the data (relatively) safe, but if the primary GitLab server goes down, I’d have to build a machine from scratch, install GitLab and restore the backup before anybody could make any pushes or pulls.
The machine I’m using to store backups is also an Ubuntu machine, so I decided to make that machine a functioning “clone.” Here’s how I did it.
From the Top
First of all, disclaimer: I take no responsibility for what you do to your machine, whether you’re taking advice from this page or not. Whatever happens to your machine is your responsibility; I make no guarantees, warranties or promises. Understand what each command does before you execute it.
Second of all, this writeup explains the process I had to use to clone from a specific machine to a different specific machine, both running GitLab 4.0. It may not work perfectly for you! When I upgrade to a newer version of GitLab, I’ll write another post detailing the clone process.
On a machine with the same operating system as the primary server (ideally), use the GitLab installation instructions to get a copy of GitLab running on the clone machine. When checking out the GitLab source, checkout the same commit number that’s on the primary server onto the clone. You can find this number by issuing the
git log command on the primary server from the directory where GitLab was cloned (in my case,
/home/gitlab/gitlab/). It’ll be the top commit number. So, in step 6 of the GitLab install directions, change the checkout command to:
As per the instructions of people who solved restoration problems, it’s necessary to make sure all the permissions on the repository directories are correct.2
1 2 3
In GitLab 4.0, the restore backup task contains a couple
sudo commands that end up halting the task and asking for the
gitlab user’s password. If you followed the GitLab installation directions properly, when you created the
gitlab user you specified the
--disabled-login option, which means there’s nothing you can do to make those commands complete. Might as well remove them. So, edit the file
/home/gitlab/gitlab/lib/tasks/gitlab/backup.rake to remove the problematic commands. Lines 139-142 should be changed to look like this:3
1 2 3 4
Now run the backup restore task:
Hopefully the backup restore task completed successfully, and if so, you should ensure the base repository directory has the correct permissions and run two more tasks:
1 2 3
With any luck, those tasks completed without errors. Now check to see if GitLab says everything is okay by running this task:
That final task might result in errors but its output is pretty helpful in those cases.
Bam! You should be able to clone any repositories from the clone machine that you have access to on the primary machine, and the web interface should show the same information as the primary machine when the backup was made.
Automating the Cloning Process
With a clone of my GitLab server functioning properly, I can rest a little easier. However, there’s no reason for me to have to go through all that nonsense when I want to sync the clone, so I made a
cron script for the clone machine. I’ve listed the script below—it needs to run as super user.
If you read through the script, you’ll notice it assumes a vanilla GitLab install and does other weird things like deleting and reinstating the
gitolite-admin repository and setting permissions more than you’d expect. These are hacks that made the script run successfully and are required by what I assume are bugs that the GitLab team has fixed in subsequent releases. Again, use the script at your own risk. Enjoy!
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
This post should theoretically be published in a blog on my employer’s official website, but pushing the blog onto the production server isn’t a priority I suppose, so I’m putting this here. ↩
When restoring the backup for the first time, in version 4.0, it may be necessary to create the top level directories for all the repositories in
The backup task was changed after the GitLab 4.1 release so these changes wouldn’t be necessary. ↩