How To Backup Azure DevOps Git Repositories

7 min read

In this article, I will show you how to automate the backup of Azure DevOps Git repositories and then verify the restore operation.

Introduction

DevOps has been an emerging trend in the software development world for the past several years. While the term may be relatively new, it is really a convergence of a number of practices that have been evolving for decades. DevOps is a revolutionary way to release software quickly and efficiently while maintaining a high level of security.

Azure DevOps— formerly known as Visual Studio Team Services (VSTS)— is a Software as a service (SaaS) platform from Microsoft that provides an end-to-end DevOps toolchain for developing and deploying software. It also integrates with most leading tools on the market and is a great option for orchestrating a DevOps toolchain. It also integrated perfectly with both Visual Studio and Visual Studio Code. While the code is perfectly safe on Azure infrastructure as committed by Microsoft here (Azure DevOps data protection overview), including periodic backups and Geo-replication, but we do not have any control over it. And it does not prevent from unintentional or malicious actions leading to data loss.

There are also cases where a centralized local backup of all projects and repositories is needed. These might include corporate policies, business continuity, and disaster recovery plans (BCDR).

Unfortunately, there is no out of the box backup solution as confirmed by Microsoft here. However, what we could do is to download the repository manually from Azure DevOps as a Zip file as shown in the figure below.

How To Backup Azure DevOps Git Repositories 1

But this may not be practical if we have a considerable amount of projects and repositories and need them backed up on a regular basis.

Let’s explore other methods and see how to automate the backup of the repositories.

Prerequisites

To follow this article, you need to have the following:

  1. Azure subscription – If you don’t have an Azure subscription, you can create a free one here.
  2. Azure storage account – To create a general-purpose storage account, follow the instructions described here.
  3. Azure DevOps – If you don’t have one, you can create a free one here.
    • You have an existing Azure Repos.
    • You are familiar with Azure Pipelines and YAML syntax.

Automate Azure DevOps backup repository

To automate the backup repository, you have a couple of different options as follows:

  1. Programmatically back up source code from Azure DevOps using Azure DevOps REST API. Then you can use Azure Functions with a time trigger to run this on schedule.
  2. Create a git bash script to mirror and backup the Git repository.
  3. Clone the latest repository on your machine and then automate the backup using your favorite on-premises backup application such as SCDPM, Veeam, etc. OR,
  4. You can leverage Azure DevOps service to automate the backup and copy the repository to Azure Blob storage or to an Azure VM.

For the remainder of this article, I will use option 4 to automate the backup of the repository using the YAML Pipeline and command-line tasks. The process looks like this:

  1. Create an Azure DevOps Personal Access Token (PAT).
  2. Create a YAML Pipeline including the following tasks:
    • Clone the Git repository.
    • Archive the Git repository.
    • Copy the archive file to Azure Blob storage.

Create an Azure DevOps Personal Access Token

Sign in to the Azure DevOps portal and take the following steps:

On the top right corner of the portal, you have user settings next to your account picture. Clicking on it reveals the account menu where you can find the ‘Personal access tokens‘ option as shown in the figure below.

How To Backup Azure DevOps Git Repositories 2

The ‘Personal access tokens‘ page will open. Click on ‘New Token‘ to create one. For this example, we only need to check the ‘Read‘ for Code for shown in the figure below. Next, give a ‘Name‘ to the token, then set the ‘Expiration‘ date and click ‘Create‘.

How To Backup Azure DevOps Git Repositories 3

Once the token is created you will get the value, but since it won’t be shown again, you must copy and save it elsewhere. We will need this token in the next section to clone the repository.

Create a YAML Pipeline

Azure Pipelines supports continuous integration (CI) and continuous delivery (CD) to constantly and consistently test and build your code and ship it to any target. You accomplish this by defining a pipeline. And if there’s any update with your repository, the pipeline will be triggered to run and copy the entire repository.

To create a YAML pipeline, take the following steps:

  1. Navigate to your team project on Azure DevOps in a new browser tab.
  2. Navigate to the Pipelines hub on the left-hand side.
  3. Click ‘New pipeline‘. We will use the wizard to automatically create the YAML definition based on our project. How To Backup Azure DevOps Git Repositories 4
  4. Next, select the ‘Azure Repos Git‘ as the source hosting platform. How To Backup Azure DevOps Git Repositories 5
  5. Then select your repository. In my example, the repo is called ‘infra-code-cn‘. How To Backup Azure DevOps Git Repositories 6
  6. Next, select the ‘Starter pipeline‘ template as the starting point for the backup pipeline. How To Backup Azure DevOps Git Repositories 7
  7. The starter pipeline will open including the following default YAML definition. How To Backup Azure DevOps Git Repositories 8
  8. Next, we need to update the YAML definition. The full YAML definition syntax to monitor and backup all branches looks like this (the PAT, azure Subscription, and storage account have been intentionally changed in this example). Please refer to the logic details below the figure. How To Backup Azure DevOps Git Repositories 9
    • Trigger (Continuous Integration): The wildcard (*) can monitor all your branches. So if there’s any update with your repository, the pipeline will be triggered to run.
    • Pool (windows-latest): This task is written in PowerShell and thus works only when running on Windows agents.
    • The CmdLine Task: This command calls ‘git clone –mirror‘ to make a copy of your git repository. Here you need to use the personal access token that we created in the previous steps. The full syntax of the command line task looks like this: 'git clone --mirror https://{yourPAT}@dev.azure.com/{yourOrganizationName}/{yourProjectName}/_git/{yourRepoName}'
    • The Archive Files Task: This will take the git repository which was cloned from the previous step and then zipped to Backup.zip and overwrite the existing archive file on the VM image used in the Pool. The archive files task includes a range of standard archive formats such as .zip, .jar, .war, .ear, .tar, .7z, and more.
    • The File Copy Task: The copy task will take the archive Backup.zip file and send it over to the Azure Blob storage. In this task, you need to specify the target Azure subscription, storage account, and container name.
  9. Finally, click Save and run and then commit. How To Backup Azure DevOps Git Repositories 10
  10. Once you commit, you need to give the pipeline the needed permission to access the resource before it can run for the first time. Click ‘View‘. How To Backup Azure DevOps Git Repositories 11
  11. On the Waiting for review page, click ‘Permit‘, and then click ‘Permit‘ on the confirmation access window. How To Backup Azure DevOps Git Repositories 12
  12. Once you permit, the pipeline will kick in and run the job. The job should be completed in about 2 minutes. How To Backup Azure DevOps Git Repositories 13
  13. You will also receive an email if the Build failed or succeeded similar to the below. How To Backup Azure DevOps Git Repositories 14

Verify the Backup

Now to verify the backup, sign in to the Azure portal and browse to the storage account. Go to Containers under Blob service.

You will see a new container named ‘azure-devops-container‘ as we set in the YAML definition.

How To Backup Azure DevOps Git Repositories 15

Click on the container name and then open the folder which is the Blob Prefix (az-devops-backup), you will see Backup.zip archive file as shown in the figure below.

How To Backup Azure DevOps Git Repositories 16

Restore from Backup

To restore from backup, you need first to download the archive Backup.zip file from the container and extract it locally on your machine.

Once you extract the file, you will see the repository name with .git extension. Open command prompt window assuming you already have Git installed on your machine, browse to the .git folder and then run the following command to push and restore your repository. Please note that you can restore to the same project/repository or you can restore to a new project/new repository.

How To Backup Azure DevOps Git Repositories 17

The --mirror option is used with both the clone for backup and push command. This option ensures that all branches and other attributes are replicated in the new repo.

The good news is, once you restore the repository, it also brings along all the revision information (full-fidelity history) as shown in the figure below.

How To Backup Azure DevOps Git Repositories 18

That’s it there you have it!

Summary

The logic of the YAML Pipeline is, if there’s any change to your source repository, the pipeline will be triggered and it takes the latest copy in System.DefaultWorkingDirectory (CmdLine task) and archive this copy into a Backup.zip file, then the Azure File copy task will copy the .zip file to Azure blob storage. You can also copy it to Azure VM which allows you to create a daily/weekly backup for your VM, please check the Azure File Copy task for more information.

With the help of simple tasks, we can automate and produce a full copy of the repository that could be easily restored or imported into the new project.

__
Thank you for reading my blog.

If you have any questions or feedback, please leave a comment.

-Charbel Nemnom-

About Charbel Nemnom 578 Articles
Charbel Nemnom is a Cloud Architect, Swiss Certified ICT Security Expert, Microsoft Most Valuable Professional (MVP), and Microsoft Certified Trainer (MCT), totally fan of the latest's IT platform solutions, accomplished hands-on technical professional with over 17 years of broad IT Infrastructure experience serving on and guiding technical teams to optimize the performance of mission-critical enterprise systems. Excellent communicator is adept at identifying business needs and bridging the gap between functional groups and technology to foster targeted and innovative IT project development. Well respected by peers through demonstrating passion for technology and performance improvement. Extensive practical knowledge of complex systems builds, network design, business continuity, and cloud security.

Be the first to comment

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.