Effective Way To Backup Azure DevOps Repositories

Updated – 07/09/2023 – This article has been updated to include and back up all Azure DevOps Git repositories within a project. Please refer to the following section for more details.

Updated – 24/08/2023 – This article has been updated to include the date and time of the Azure DevOps repo archive file, so all previous backup versions are kept and not overwritten.

Updated – 24/07/2023 – This article has been updated to use the system-generated token instead of the Personal Access Token (PAT) to improve the security and reliability of the Pipeline.

In this article, we will show you how to effectively automate Azure DevOps Backup Git repositories and then verify the restore operation.

Introduction

DevOps has been an emerging trend in the software development world for the past several years. While the term may be relatively new, it is really a convergence of a number of practices that have been evolving for decades. DevOps is a revolutionary way to release software quickly and efficiently while maintaining a high level of security.

Azure DevOps — formerly known as Visual Studio Team Services (VSTS)— is a Software as a service (SaaS) platform from Microsoft that provides an end-to-end DevOps toolchain for developing and deploying software. It also integrates with most leading tools on the market and is a great option for orchestrating a DevOps toolchain. It also integrated perfectly with both Visual Studio and Visual Studio Code. While the code is perfectly safe on Azure infrastructure as committed by Microsoft here (Azure DevOps data protection overview), including periodic backups and Geo-replication, we do not have any control over it. And it does not prevent unintentional or malicious actions leading to data loss.

There are also cases where a centralized local backup of all projects and repositories is needed. These might include corporate policies, business continuity, and disaster recovery plans (BCDR).

Unfortunately, there is no out-of-the-box backup solution as confirmed by Microsoft here. However, what we could do is download the repository manually from Azure DevOps as a Zip file as shown in the figure below.

But this may not be practical if we have a considerable amount of projects and repositories and need them backed up on a regular basis.

Let’s explore other methods and see how to automate the backup of the repositories.

Prerequisites

To follow this article, you need to have the following:

1) Azure subscription – If you don’t have an Azure subscription, you can create a free one here.

2) Azure storage account and container – To create a general-purpose v2 storage account, follow the instructions described here.

3) Azure DevOps – If you don’t have one, you can create a free one here.

• You have an existing Azure Repos.
• You are familiar with Azure Pipelines and YAML syntax.

4) Azure DevOps Service Connection – To create a new service connection, follow the steps described in this article. You can assign the scope at the Management Group or Azure Subscription Level.

Automate Azure DevOps backup repository

To automate the backup repository, you have a couple of different options as follows:

1) Programmatically back up source code from Azure DevOps using Azure DevOps REST API. Then you can use Azure Functions with a time trigger to run this on schedule.

2) Create a git bash script to mirror and back up the Git repository.

3) Clone the latest repository on your machine and then automate the backup using your favorite on-premises backup application such as SCDPM, Veeam, etc. OR,

4) You can leverage the Azure DevOps service to automate the backup and copy the repository to Azure Blob storage or to an Azure VM.

For the remainder of this article, we will use option 4 to automate the backup of the repository using the YAML Pipeline and command-line tasks. At a high level, the process looks like this:

1) Create an Azure DevOps Personal Access Token (PAT).

2) Create a YAML Pipeline including the following tasks:

• Clone the Git repository.
• Archive the Git repository.
• Copy the archive file to Azure Blob storage.

Create an Azure DevOps Personal Access Token

Updated – 24/07/2023 – You can skip this section and move to the next section (Create a YAML Pipeline), you don’t need to create an Azure DevOps Personal Access Token (PAT). We’ll use the system-generated token instead of using (PAT) to improve the security and prevent the token from getting expired.

Sign in to the Azure DevOps portal and take the following steps:

On the top right corner of the portal, you have user settings next to your picture account. Clicking on it reveals the account menu where you can find the ‘Personal access tokens‘ option as shown in the figure below.

The ‘Personal access tokens‘ page will open. Click on ‘New Token‘ to create one. For this example, we only need to check the ‘Read‘ for Code as shown in the figure below. Next, give a ‘Name‘ to the token, then set the ‘Expiration‘ date and click ‘Create‘.

Once the token is created you will get the value, but since it won’t be shown again, you must copy and save it elsewhere. We will need this token in the next section to clone the repository.

Create a YAML Pipeline

Azure Pipelines supports continuous integration (CI) and continuous delivery (CD) to constantly and consistently test and build your code and ship it to any target. You accomplish this by defining a pipeline. And if there’s any update with your repository, the pipeline will be triggered to run and copy the entire repository.

To create a YAML pipeline, take the following steps:

1) Navigate to your team project on Azure DevOps in a new browser tab.

2) Navigate to the Pipelines hub on the left-hand side.

3) Click ‘New pipeline‘. We will use the wizard to automatically create the YAML definition based on our project.

4) Next, select ‘Azure Repos Git‘ as the source hosting platform.

5) Select your repository. In this example, the repo is called ‘infra-code-cn‘.

6) Next, select the ‘Starter pipeline‘ template as the starting point for the backup pipeline.

7) The starter pipeline will open including the following default YAML definition.

8) Next, we need to update the YAML definition. The full YAML definition syntax to monitor and back up all branches looks like the following. Please refer to this section to get the latest version of the YAML definition file that includes a backup date and time for the archive file, so you will have all the previous backup versions of the files.

The YAML file includes the following logic:

• Trigger (Continuous Integration): The wildcard (*) can monitor all your branches. So if there’s any update with your repository, the pipeline will be triggered to run.
• Pool (Windows-latest): This task is written in PowerShell and thus works only when running on Windows agents.
• Variables: We define the date and time the pipeline kicked off, so we can append it to the archive backup file name.
• CmdLine Task: This command calls ‘git clone –mirror‘ to make a copy of your git repository. Here you need to use the personal access token that we created in the previous step. The System.AccessToken is a special variable that carries the security token used by the running build. The full syntax of the command line task looks like this:

steps:
  - task: CmdLine@2
  - bash: 'git clone --mirror https://${SYSTEM_ACCESSTOKEN}@dev.azure.com/{yourOrganizationName}/{yourProjectName}/_git/{yourRepoName}'
    env:
      SYSTEM_ACCESSTOKEN: $(System.AccessToken)

• The Archive Files Task: This will take the git repository which was cloned from the previous step and then zipped to Date-Time_AzDevOpsBackup.zip and overwrite the existing archive file (if exists) on the VM image used in the Pool. The archive files task includes a range of standard archive formats such as .zip, .jar, .war, .ear, .tar, .7z, and more.
• The File Copy Task: The copy task will take the archive Date-Time_AzDevOpsBackup.zip file and send it over to the Azure Blob storage. In this task, you need to specify the target service connection in Azure DevOps for the Azure subscription, storage account, and container name.

9) Finally, click Save and Run and then commit.

10) Once you commit, you need to give the pipeline the needed permission to access the resource before it can run for the first time. Click ‘View‘.

11) On the Waiting for Review page, click ‘Permit‘, and then click ‘Permit‘ on the confirmation access window.

12) Once you permit, the pipeline will kick in and run the job. The job should be completed in about 2 minutes.

13) You will also receive an email if the Build failed or succeeded similar to the one below.

Verify the Backup

Now to verify the backup, sign in to the Azure Portal and browse to the storage account, then go to Containers under Blob service.

You will see a new container named ‘azure-devops-container‘ as we set in the YAML definition.

Click on the container name and then open the folder which is the Blob Prefix (az-devops-backup), you will see the Backup.zip archive file as shown in the figure below.

Restore from Backup

To restore from the backup, you need first to download the archive Date-Time_AzDevOpsBackup.zip blob from the storage container and extract it locally on your machine as shown in the figure below.

Once you extract the file, you will see the repository name with the .git extension. Open the command prompt window assuming you already have Git installed on your machine, browse to the .git folder, and then run the following command to push and restore your repository. Please note that you can restore to the same project/repository or you can restore to a new project and new repository.

The “--mirror” parameter is used with both the clone for backup and push commands. This option ensures that all branches and other attributes are replicated in the new repo.

The good news is, that once you restore the repository, it also brings along all the revision information (full-fidelity history) as shown in the figure below.

That’s it there you have it!

Backup Azure DevOps YAML

You can find the latest version of the YAML pipeline below which will back up your Azure DevOps Git repository and keep your previous versions intact, in this case, the old backup versions won’t be overwritten, so you can restore from earlier backups if needed.

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
# Date: 24-July-2023
# Update: 24-August-2023

trigger:
  branches:
    include:
    - '*'

pool:
  vmImage: 'windows-latest'

variables: 
 backup_date: $[format('{0:yyyy}-{0:MM}-{0:dd}T{0:HH}-{0:mm}-{0:ss}', pipeline.startTime)]

steps:
  - task: CmdLine@2
  - bash: 'git clone --mirror https://${SYSTEM_ACCESSTOKEN}@dev.azure.com/{yourOrganizationName}/{yourProjectName}/_git/{yourRepoName}'      
    env:
      SYSTEM_ACCESSTOKEN: $(System.AccessToken)

  - task: ArchiveFiles@2
    displayName: 'Building archive file locally'
    inputs:
      rootFolderOrFile: '$(System.DefaultWorkingDirectory)'
      includeRootFolder: true
      archiveType: 'zip'
      archiveFile: '$(Build.ArtifactStagingDirectory)/$(backup_date)_AzDevOpsBackup.zip'
      replaceExistingArchive: true

  - task: AzureFileCopy@3
    displayName: 'Azure Blob File Copy'
    inputs:
      SourcePath: '$(Build.ArtifactStagingDirectory)/$(backup_date)_AzDevOpsBackup.zip'
      azureSubscription: 'AzDevOps-Bkp-Svc'
      Destination: 'AzureBlob'
      storage: 'storageaccountnamehere'
      ContainerName: 'azure-devops-container'
      BlobPrefix: 'az-devops-backup'

Backup Multiple Repos in Azure DevOps

A common question that keeps coming up is the following:

How can we apply the steps described above to include and back up multiple Azure DevOps repositories?

The good news is that I finally tackled this functionality, so now we can back up all or specific repos in a specific Azure DevOps project. For this to work, please follow the steps described in this section.

First, you need to take the following YAML definition and update all the parameters marked with # CHANGE < very important.

This YAML file will be referenced later when we create the new pipeline. Save this file in the desired repository and folder with the .yml extension:

# This is totally optional, you can remove all parameters and using hardcoded values in stages below
# Parameters are useful if you are running this pipeline manually and you need sometimes to pick only required components
parameters:
  - name: backupSourceCode
    displayName: 'Backup Azure DevOps Source Code'
    type: boolean
    default: true # Backup Azure DevOps Source Code (true/false)
  - name: backupVariableGroups
    displayName: 'Backup Azure DevOps Variable Groups'
    type: boolean
    default: false # Backup Azure DevOps Variable Groups (true/false)

trigger:
  none

stages:
  - stage: MAIN # CHANGE - the name of the stage, you can omit if you have only one stage and use jobs directly    
    jobs:
    - template: templates/multi-repo-backup-template.yml # CHANGE - the path and name of the backup template .yml file below
      parameters:        
        backupSourceCode: ${{ parameters.backupSourceCode}} # can be changed to hardcoded value true or false
        backupVariableGroups: ${{ parameters.backupVariableGroups}} # can be changed to hardcoded value true or false
        repositories: # List of all repositories that you want to backup.
          - self # Self means current repo where this file will be located. The current repo will be backed up by default.
          - "git://SameProjectName/RepoName-1" # CHANGE - Please use syntax "git://ProjectName/RepoName" for all other repos. If Project name and/or Repo name contains spaces, then use it as it is without escaping.
          - "git://SameProjectName/RepoName-2" # CHANGE
          - "git://SameProjectName/RepoName-3" # CHANGE
        serviceConnectionName: 'AzDevOps-Bkp-Svc' # CHANGE - name of the service connection in Azure DevOps for the Azure subscription
        storageAccountName: 'devopsstorageaccount'  # CHANGE - name of the target Azure storage account. MUST EXISTS before execution!
        backupContainerName: 'azure-devops-container' # CHANGE - name of the container in the storage account. MUST EXISTS before execution!
        blobDirectoryName: 'az-devops-backup' # CHANGE - name of the blob directory where you wish to store the backup
        # Other optional parameters:
        #backupArtifacts: true/false - enable backup of artifact, default is false
        #feedPath: path to artifact's feed
        #packageName: package name
        #packageVersion: pacakge version, usually from naming template

The second YAML definition that you need, is the backup template itself which contains all the logic to back up the source code, variable groups (optionally), and/or artifacts (optionally). You don’t need to update any parameters in this file, all the parameters are defined in the above YAML file.

parameters:
  - name: serviceConnectionName
    displayName: 'Service Connection Name'  
  - name: storageAccountName
    displayName: 'Storage account Name'    
  - name: backupContainerName
    displayName: 'Container name in storage account'    
  - name: blobDirectoryName
    displayName: 'Blob Directory Name' 
  - name: envName
    displayName: 'Environment'
    type: string
    default: 'DevOps Backup'

  - name: backupSourceCode
    displayName: 'Backup Azure DevOps Source Code'
    type: boolean
    default: true
  - name: repositories
    displayName: Repositories to backup (use "git://ProjectName/RepoName" for other repos)
    type: object
    default:
       - self
       #- "git://ProjectName/RepoName"
  
  - name: backupVariableGroups
    displayName: 'Backup Azure DevOps Variable Groups'
    type: boolean
    default: true
  - name: devOpsOrgUrl
    displayName: 'DevOps Organization URL'
    type: string
    default: ' '

  - name: backupArtifacts
    displayName: 'Backup Azure DevOps Artifacts'
    type: boolean
    default: false
  - name: feedPath
    displayName: 'Feed path'
    type: string
    default: ' '
  - name: packageName
    displayName: 'Package name in feed'
    type: string
    default: ' '
  - name: packageVersion
    displayName: 'Package version (usually from naming)'
    type: string
    default: ' '

jobs:
  - ${{ if eq(parameters.backupVariableGroups, true) }}:
    - deployment: backupVariableGroup
      variables:
      - name: backupDateTime
        value: $[format('{0:yyyy}-{0:MM}-{0:dd}T{0:HH}-{0:mm}-{0:ss}', pipeline.startTime)]
      - ${{ if eq(parameters.devOpsOrgUrl, ' ') }}:
        - name: actualDevOpsUrl
          value: ${{ variables['System.TeamFoundationCollectionUri'] }}
      displayName: Backup Variable Group
      environment: ${{ parameters.envName }}
      strategy:
        runOnce:
          deploy:
            steps:
              # Getting access token for the Repo
              - checkout: self
                persistCredentials: true
                clean: true
              # You don't need to use az devops login in case of using AZURE_DEVOPS_EXT_PAT env variable
              # Proof: https://learn.microsoft.com/en-us/azure/devops/cli/log-in-via-pat?view=azure-devops&tabs=windows
              - script: az devops configure --defaults organization=$(actualDevOpsUrl) project="$(System.TeamProject)" 
                displayName: 'Set default Azure DevOps organization and project'
                env:
                  AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

              - script: az pipelines variable-group list
                displayName: 'Test Az pipelines command'
                env:
                  AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)
              
              - pwsh: |
                  $backupFolder = New-item (Join-Path "backup" "$(backupDateTime)")  -ItemType Directory -Force
                  $bgFolder = Join-Path $backupFolder.FullName "variable-groups"
                  # Get all variable groups
                  $groups = ConvertFrom-Json "$(az pipelines variable-group list)"
                  # echo $groups
                  $groups | foreach {
                    $groupName = $_.name

                    # Prepend VariableGroups folder name
                    $filePath = Join-Path $bgFolder "$groupName.json"

                    # Save the variable group to a file
                    ConvertTo-Json $_ | New-Item $filePath -Force
                  }
                displayName: 'Save variable groups'
                workingDirectory: $(System.DefaultWorkingDirectory)
                env:
                  AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)
              - task: CmdLine@2
                inputs:
                  script: |
                    az storage blob upload-batch -d ${{ parameters.backupContainerName }} --source "$(System.DefaultWorkingDirectory)/backup" --connection-string "${{ parameters.backupStorageAccountConectionString }}" --overwrite
                  workingDirectory: '$(System.DefaultWorkingDirectory)'

  - ${{ if eq(parameters.backupArtifacts, true) }}:
    - deployment: backupArtifacts
      displayName: Backup DevOps Artifacts
      variables:
        - name: backupDateTime
          value: $[format('{0:yyyy}-{0:MM}-{0:dd}T{0:HH}-{0:mm}-{0:ss}', pipeline.startTime)]
      environment: ${{ parameters.envName }}
      strategy:
        runOnce:
          deploy:
            steps:
              #frontendbuild
              - task: DownloadPackage@1
                displayName: Download artifacts
                inputs:
                  packageType: 'upack'
                  feed: ${{parameters.feedPath}} #'$(az._global.projectName)/$(az._global.artifacts.frontendbuild)@local' # stage=local/Release/Prerelease
                  definition: ${{ parameters.packageName }}
                  version: '*'
                  downloadPath: '$(System.DefaultWorkingDirectory)/backup/$(backupDateTime)/Artifacts/${{ parameters.packageName }}/${{parameters.packageVersion}}'

              - script: |
                    az storage blob upload-batch -d ${{ parameters.backupContainerName }} --source "$(System.DefaultWorkingDirectory)/backup" --connection-string "${{ parameters.backupStorageAccountConectionString }}" --overwrite
                workingDirectory: '$(System.DefaultWorkingDirectory)'
                displayName: 'Publish to storage account'

  - ${{ if eq(parameters.backupSourceCode, true) }}:
    - deployment: backupSourceCode
      displayName: Backup Source Code
      variables:
        - name: backupDateTime
          value: $[format('{0:yyyy}-{0:MM}-{0:dd}T{0:HH}-{0:mm}-{0:ss}', pipeline.startTime)]
      environment: ${{ parameters.envName }}
      strategy:
        runOnce:
          deploy:
            steps:
            - ${{ each repoToBackup in parameters.repositories }}:
              - checkout: ${{ repoToBackup }}
                persistCredentials: true
                clean: true
                path: 
            - script: |
                  /bin/mkdir -p backup/$(backupDateTime)/SourceCode
                  ls
                  find . -maxdepth 1 ! \( -name backup -o -name '.' \) -exec cp -r {} backup/$(backupDateTime)/SourceCode/ \;
              workingDirectory: '$(System.DefaultWorkingDirectory)'
              displayName: Prepare backup folder
            - ${{ if le(length(parameters.repositories), 1) }}:
              - script: |
                    zip -r -m "devOpsCodeBackup.zip" .
                    ls -Al                   
                workingDirectory: '$(System.DefaultWorkingDirectory)/backup/$(backupDateTime)/SourceCode'
                displayName: Backup single repo
            - ${{ if gt(length(parameters.repositories), 1) }}:
              - script: |
                    for i in */; do zip -r -m "${i%/}.zip" "$i"; done
                    ls -Al                                      
                workingDirectory: '$(System.DefaultWorkingDirectory)/backup/$(backupDateTime)/SourceCode'
                displayName: Backup all repos
            - task: AzureCLI@2
              displayName: Azure Storage Blob Upload Batch
              inputs:
                scriptType: 'pscore'
                scriptLocation: 'inlineScript'         
                azureSubscription: ${{ parameters.serviceConnectionName }}              
                inlineScript: | 
                    az storage blob upload-batch -d ${{ parameters.backupContainerName }} --account-name ${{ parameters.storageAccountName }} --destination-path ${{ parameters.blobDirectoryName }} --source "$(System.DefaultWorkingDirectory)/backup" --overwrite

Now we have both files saved in our main (self) repository as shown in the figure below:

In the next step, you need to create a new pipeline and select the Git repository where you saved the first YAML file. In this example, the file is called “backup-all-repos.yml“. Then click Continue.

Next, click Run and test the pipeline. When you run the pipeline for the first time, you need to give it permission to access before the first run can continue. Please note that this is only needed once. Select the waiting “Backup Source Code” job, then click View and Permit twice to confirm. Please note that granting permission will permit the use of Environment ‘DevOps Backup’ for all waiting and future runs of this pipeline.

Last, you can configure and create a schedule on when you want to run the pipeline and trigger the backup, then click Save.

If you run the pipeline more than once per day or manually, then the backup will be created based on the date and time when you run the pipeline. The backup will be uploaded to a separate blob directory appended by the date and time as shown in the figure below.

There you have it. Happy Azure DevOps Backup!

Summary

In this article, we illustrated all the steps that you need to back up a single Azure DevOps repository, as well as how to back up multiple repositories in a single Azure DevOps project, and optionally variable groups and artifacts backup.

The logic of the YAML Pipeline is, that if there’s any change to your source repository, the pipeline will be triggered and it takes the latest copy in the System.DefaultWorkingDirectory (CmdLine task) and archive this copy into a Backup.zip file, then the Azure File copy task will copy the .zip file to Azure blob storage.

You can also copy it to Azure VM which allows you to create a daily/weekly backup for your VM, please check the Azure File Copy task for more information.

With the help of simple tasks, we can automate and produce a full copy of the repository that could be easily restored or imported into the new Azure DevOps project.

__
Thank you for reading my blog.

If you have any questions or feedback, please leave a comment.

-Charbel Nemnom-