I run a GitLab server that I use to collaborate with coworkers.1 It’s a custom-built machine with a RAID, lots of memory, redundant power supplies, and is running Ubuntu. Every day, a simple
cron script to makes a backup copy of the data on the server to another machine. The script looks something like this:
#!/bin/sh cd /home/gitlab/gitlab sudo -u gitlab -H bundle exec rake gitlab:backup:create RAILS_ENV=production rsync -avrcpt /home/gitlab/gitlab/tmp/backups/ user@remotemachine:/path/to/gitlabtars rsync -avrcpt /home/git/repositories/ user@remotemachine:/path/to/barerepos
This script copies a GitLab-friendly backup file to a remote machine and syncs bare Git repositories to a different place on the same remote machine (just in case GitLab goes out of style).
That keeps the data (relatively) safe, but if the primary GitLab server goes down, I’d have to build a machine from scratch, install GitLab and restore the backup before anybody could make any pushes or pulls.
The machine I’m using to store backups is also an Ubuntu machine, so I decided to make that machine a functioning “clone.” Here’s how I did it.
From the Top
First of all, disclaimer: I take no responsibility for what you do to your machine, whether you’re taking advice from this page or not. Whatever happens to your machine is your responsibility; I make no guarantees, warranties or promises. Understand what each command does before you execute it.
Second of all, this writeup explains the process I had to use to clone from a specific machine to a different specific machine, both running GitLab 4.0. It may not work perfectly for you! When I upgrade to a newer version of GitLab, I’ll write another post detailing the clone process.
On a machine with the same operating system as the primary server (ideally), use the GitLab installation instructions to get a copy of GitLab running on the clone machine. When checking out the GitLab source, checkout the same commit number that’s on the primary server onto the clone. You can find this number by issuing the
git log command on the primary server from the directory where GitLab was cloned (in my case,
/home/gitlab/gitlab/). It’ll be the top commit number. So, in step 6 of the GitLab install directions, change the checkout command to:
sudo -u gitlab -H git checkout [commit number]
As per the instructions of people who solved restoration problems, it’s necessary to make sure all the permissions on the repository directories are correct.2
sudo chmod -R ug+rwX,o-rwx /home/git/repositories/ sudo chmod -R u-s /home/git/repositories/ sudo find /home/git/repositories/ -type d -print0 | sudo xargs -0 chmod g+s
In GitLab 4.0, the restore backup task contains a couple
sudo commands that end up halting the task and asking for the
gitlab user’s password. If you followed the GitLab installation directions properly, when you created the
gitlab user you specified the
--disabled-login option, which means there’s nothing you can do to make those commands complete. Might as well remove them. So, edit the file
/home/gitlab/gitlab/lib/tasks/gitlab/backup.rake to remove the problematic commands. Lines 139-142 should be changed to look like this:3
permission_commands = [ "", "" ]
Now run the backup restore task:
sudo -u gitlab -H bundle exec rake gitlab:backup:restore RAILS_ENV=production
Hopefully the backup restore task completed successfully, and if so, you should ensure the base repository directory has the correct permissions and run two more tasks:
sudo chmod -R ug+rwXs,o-rwx /home/git/repositories/ sudo -u gitlab -H bundle exec rake gitlab:gitolite:update_repos RAILS_ENV=production sudo -u gitlab -H bundle exec rake gitlab:gitolite:update_keys RAILS_ENV=production
With any luck, those tasks completed without errors. Now check to see if GitLab says everything is okay by running this task:
sudo -u gitlab -H bundle exec rake gitlab:check RAILS_ENV=production
That final task might result in errors but its output is pretty helpful in those cases.
Bam! You should be able to clone any repositories from the clone machine that you have access to on the primary machine, and the web interface should show the same information as the primary machine when the backup was made.
Automating the Cloning Process
With a clone of my GitLab server functioning properly, I can rest a little easier. However, there’s no reason for me to have to go through all that nonsense when I want to sync the clone, so I made a
cron script for the clone machine. I’ve listed the script below—it needs to run as super user.
If you read through the script, you’ll notice it assumes a vanilla GitLab install and does other weird things like deleting and reinstating the
gitolite-admin repository and setting permissions more than you’d expect. These are hacks that made the script run successfully and are required by what I assume are bugs that the GitLab team has fixed in subsequent releases. Again, use the script at your own risk. Enjoy!
#!/bin/bash cd /home/gitlab/gitlab # ensure permissions are correct echo ensuring correct permissions sudo chmod -R ug+rwX,o-rwx /home/git/repositories/ sudo chmod -R u-s /home/git/repositories/ sudo find /home/git/repositories/ -type d -print0 | sudo xargs -0 chmod g+s # remove gitolite-admin repo to prevent errors that happen sometimes, and build new one echo rebuilding gitolite-admin repository sudo rm -r /home/git/repositories/gitolite-admin.git sudo -u git -H sh -c "PATH=/home/git/bin:$PATH; gitolite setup -pk /home/git/gitlab.pub" # remove old backup first lol! echo removing old backup files sudo -u gitlab -H rm tmp/backups/* # copy new backup from external drive echo copying new backup from backup media bu_file=`ls /path/to/gitlabtars | tail -n 1` sudo -u gitlab -H cp /path/to/gitlabtars/$bu_file tmp/backups/$bu_file # restore the backup. echo initiating gitlab backup restore rake task sudo -u gitlab -H bundle exec rake gitlab:backup:restore RAILS_ENV=production # ensure permissions are correct, as a matter of paranoia sudo chmod -R ug+rwXs,o-rwx /home/git/repositories/ # update the repos and keys echo updating gitolite repos and keys sudo -u gitlab -H bundle exec rake gitlab:gitolite:update_repos RAILS_ENV=production sudo -u gitlab -H bundle exec rake gitlab:gitolite:update_keys RAILS_ENV=production # run a check. echo running the final gitlab check to look at problems sudo -u gitlab -H bundle exec rake gitlab:check RAILS_ENV=production
- This post should theoretically be published in a blog on my employer’s official website, but pushing the blog onto the production server isn’t a priority I suppose, so I’m putting this here. [return]
- When restoring the backup for the first time, in version 4.0, it may be necessary to create the top level directories for all the repositories in
- The backup task was changed after the GitLab 4.1 release so these changes wouldn’t be necessary. [return]