Sync Between Two Azure File Shares for Disaster Recovery

15 Min. Read

Azure Files enables you to set up highly available network file shares that can be accessed by using the standard Server Message Block (SMB) protocol or the Network File System (NFS) protocol. That means that multiple VMs can share the same files with both read and write access. You can also read the files using the REST interface or the storage client libraries.

One thing that distinguishes Azure Files from files on a corporate file share is that you can access the files from anywhere in the world using a URL that points to the file and includes a shared access signature (SAS) token. You can generate SAS tokens; they allow specific access to a private asset for a specific amount of time.

In this article, I will share with you how to sync between two Azure file shares for Disaster Recovery in two different storage accounts.

Introduction

If you have been working with file shares for quite some time, you have probably noticed that Azure Files provides two options to connect to an Azure file share:

  1. You can access the Azure file share directly using SMB, NFS, or REST API protocols.
  2. You can create a cache of the Azure file share on a Windows Server with Azure File Sync and access the data from the Windows Server whether running on-premises or in Azure (IaaS VM).

Azure file shares can be mounted concurrently by cloud or on-premises deployments. Azure Files SMB file shares are accessible from Windows, Linux, and macOS clients. Azure Files NFS file shares are accessible only from Linux or macOS clients.

Microsoft supports locally redundant storage (LRS), zone redundant storage (ZRS), and geo-redundant storage (GRS) to ensure that Azure services are always available. GRS asynchronously replicates your data to a secondary paired region. However, you can only access your data in the secondary region if Microsoft initiates a failover from the primary to secondary region from an account configured with GRS, so Microsoft recommends that you treat the new region as a backup of data only.

One of the most requested features by customers is to backup Azure file share(s) to a secondary region of their choice for disaster recovery. Additionally, Azure File Sync does NOT automatically begin syncing with the new (primary) region after failover. Lastly, Microsoft does not support GRS for premium file shares.

It’s very important to design your application for high availability from the start. Microsoft recommends keeping the following best practices in mind for maintaining high availability for your Azure Storage data:

  • Disks: Use Azure Backup to back up the VM disks used by your Azure virtual machines. Also, consider using Azure Site Recovery to protect your VMs in the event of a regional disaster.
  • Block blobs: Turn on soft delete to protect against object-level deletions and overwrites, or copy block blobs to another storage account in a different Azure region using AzCopy, Azure PowerShell, or the Azure Data Movement library.
  • Files: Use Azure Backup to back up your file shares. Also, enable soft delete to protect against accidental file share deletions. For geo-redundancy in cases where GRS is not available, use AzCopy or Azure PowerShell to copy your files to another storage account in a different region.
  • Tables: Use AzCopy to export table data to another storage account in a different region.

What if you want to have a secondary copy of your data that resides in an Azure file share in the region of your choice without using a geo-redundant storage (GRS) storage account?

In this article, I will share with you how to sync between two different Azure file shares in two different storage accounts and two different regions, so you can have a copy of your data in a secondary region of your choice while maintaining security and business continuity.

For this article, I will make use of the AzCopy tool, which is a command-line utility that you can use to sync or copy files to or from a storage account. I will also use Azure Container Instances to simplify and automate the AzCopy script in Runbook, which will run as part of the container. In this way, we can run the container on a simple schedule to sync the data and only get billed for the time the container was used.

Prerequisites

To follow this article, you need to have the following:

  1. Azure subscription – If you don’t have an Azure subscription, you can create a free one here.
  2. You need to have two different storage accounts either in the same region and same subscription or in different regions and subscriptions.
  3. You also need to create two Azure file shares across two different storage accounts either in the same region and same subscription or in different regions and subscriptions.
  4. Lastly, you need to have some files in the first Azure file share (which is considered the primary share). You can add these files directly in the portal or by mounting the share, or you can sync data from Windows Servers to the file share using Azure File Sync.

At a high level, the deployment design is represented by the following diagram.

Sync Between Two Azure File Shares for Disaster Recovery
Deployment Design – Sync between two Azure File Shares for Disaster Recovery

Get started

First, we need to create an Azure automation account that will help you to automate the synchronization and backup process without user interaction. This will also make sure to respect the security access of your storage account without exposing access keys to users.

Create Automation Account

In this step, I will create an Azure automation resource with a Run As account. Run As accounts in Azure Automation are used to provide authentication for managing resources in Azure with the Azure cmdlets. When you select ‘Create an Azure Run As account‘, it creates a new service principal user in Azure Active Directory (Azure AD) and assigns the Contributor role to the service principal at the subscription level where the Automation Account is created.

Open the Azure portal, click All services found in the upper left-hand corner. In the list of resources, type Automation. As you begin typing, the list filters based on your input. Select Automation Accounts.

Click +Add. Enter the automation account name, choose the right subscription, resource group, location, and then click Create.

Create Azure Automation Account

Grant Run As account permissions in other subscriptions

Please note that you can use the same workflow described in this article to sync your data between two Azure file shares in two different Azure subscriptions as well. In this scenario, you need to manually assign the service principal account that get’s created as part of the automation account to the target Resource Group in the second subscription.

Azure Automation supports using a single Automation account from one subscription and executing Runbooks against Azure Resource Manager resources across multiple subscriptions. You assign the Run As account service principal the Contributor role in the other resource group in the second subscription. I highly recommend assigning the Contributor role on the resource group level where you have created the source and target storage accounts, do not assign the Contributor role on the subscription level for additional security. For more information, please check the Role-based access control in Azure Automation.

To assign the Run As account to the role in the other subscription, the user account performing this task needs to be a member of the Owner role in that subscription.

You can find the service principal name in Azure AD in the automation account under Account Settings | Run as accounts | Azure Run Az Account as shown in the figure below (In this example, my automation account name is aac-sync-afs-blob_xxxx).

Azure Automation Azure AD App Registrations

As mentioned in the previous section, when you create an automation account with Run as Account, it creates a new service principal in Azure Active Directory (Azure AD) and assigns the Contributor role to the service principal at the subscription level by default. To maintain a high level of security, please remove the Contributor role for the service principal from the subscription level and assign it on the resource group level instead.

Import Az modules from Gallery

In the next step, you need to import the required modules from the Modules gallery. In your list of Automation Accounts, select the account that you created in the previous step. Then from your automation account, select Modules under Shared Resources. Click the Browse Gallery button to open the Browse Gallery page. You need to import the following modules from the Modules gallery in the order given below:

  1. Az.Accounts
  2. Az.ContainerInstance
  3. Az.Storage

Sync Between Two Azure File Shares for Disaster Recovery 1

At the time of this writing, AzCopy is still not part of the Azure Automation Runbook. For this reason, I will be creating an Azure Container instance with AzCopy as part of the container, so we can automate the entire synchronization and snapshots backup process.

Create PowerShell Runbook

In this step, you can create multiple Runbooks based on which set of Azure file shares you want to sync/copy. PowerShell Runbooks are based on Windows PowerShell. You directly edit the code of the Runbook using the text editor in the Azure portal. You can also use any offline text editor such as Visual Studio Code and import the Runbook into Azure Automation.

From your automation account, select Runbooks under Process Automation. Click the ‘+ Create a runbook‘ button to open the Create a runbook blade. Select PowerShell as Runbook type.

Create PowerShell Runbook

In this example, I will create a Runbook to synchronize and copy all my files and directories changes from a primary Azure file share (snapshot) to a secondary file share in a different storage account in a different Azure region. You can also be creative as much as you want and cover different Azure file shares, etc.

Edit the Runbook

Once you have the Runbook created, you need to edit the Runbook, then write or add the script to choose which Azure file shares you want to sync. Of course, you can create scripts that suit your environment.

As mentioned earlier, in this example, I will create a Runbook to synchronize and copy all my data from one Azure file share to another Azure file share which can be in a different storage account and region. And to maintain a high level of security, I will NOT use the storage account keys. Instead, I will create a time limit SAS token URI for each file share, and the SAS token will expire automatically after 30 minutes. So, if you regenerate your storage account keys in the future, the automation process won’t break.

The script as follows. You can copy the script from the ‘Code Block’ below or from GitHub here.

<#
.DESCRIPTION
A Runbook example that automatically creates incremental backups of an Azure Files system on a customer-defined schedule and stores the backups in a separate storage account.
It does so by leveraging AzCopy with the sync parameter, which is similar to Robocopy /MIR. Only changes will be copied with every backup, and any deletions on the source will be mirrored on the target.
In this example, AzCopy is running in a Container inside an Azure Container Instance using Service Principal in Azure AD.

.NOTES
Filename : SyncBetweenTwoFileShares
Author1  : Sibonay Koo (Microsoft PM)
Author2  : Charbel Nemnom (Microsoft MVP/MCT)
Version  : 1.0
Date     : 13-February-2021
Updated  : 13-April-2021

.LINK
To provide feedback or for further assistance please email:
azurefiles@microsoft.com
#>

Param (
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $sourceAzureSubscriptionId,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $sourceStorageAccountRG,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $targetStorageAccountRG,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $sourceStorageAccountName,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $targetStorageAccountName,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $sourceStorageFileShareName,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $targetStorageFileShareName
)

# Azure File Share maximum snapshot support limit by the Azure platform is 200
[Int]$maxSnapshots = 200

$connectionName = "AzureRunAsConnection"

Try {
    #! Get the connection "AzureRunAsConnection "
    $servicePrincipalConnection = Get-AutomationConnection -Name $connectionName
    Write-Output "Logging in to Azure..."
    Connect-AzAccount -ServicePrincipal `
        -TenantId $servicePrincipalConnection.TenantId `
        -ApplicationId $servicePrincipalConnection.ApplicationId `
        -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint
}
Catch {
    If (!$servicePrincipalConnection) {
        $ErrorMessage = "Connection $connectionName not found..."
        throw $ErrorMessage
    }
    Else {
        Write-Error -Message $_.Exception
        throw $_.Exception
    }
}

# SOURCE Azure Subscription
Select-AzSubscription -SubscriptionId $sourceAzureSubscriptionId

#! Source Storage Account in the primary region
# Get Source Storage Account Key
$sourceStorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $sourceStorageAccountRG -Name $sourceStorageAccountName).Value[0]

# Set Azure Storage Context
$sourceContext = New-AzStorageContext -StorageAccountKey $sourceStorageAccountKey -StorageAccountName $sourceStorageAccountName

# List the current snapshots on the source share
$snapshots = Get-AzStorageShare `
    -Context $sourceContext.Context | `
Where-Object { $_.Name -eq $sourceStorageFileShareName -and $_.IsSnapshot -eq $true}

# Delete the oldest (1) manual snapshot in the source share if have 180 or more snapshots (Azure Files snapshot limit is 200)
# This leaves a buffer such that there can always be 6 months of daily snapshots and 10 yearly snapshots taken via Azure Backup
# You may need to adjust this buffer based on your snapshot retention policy
If ((($snapshots.count)+20) -ge $maxSnapshots) {
    $manualSnapshots = $snapshots | where-object {$_.ShareProperties.Metadata.Keys -eq "AzureBackupProtected"}
    Remove-AzStorageShare -Share $manualSnapshots[0].CloudFileShare -Force
}

# Take manual snapshot on the source share
# When taking a snapshot using PowerShell, the snapshot's metadata is set to a key-value pair with the key being "AzureBackupProtected"
# The value of "AzureBackupProtected" is set to "True" or "False" depending on whether Azure Backup is enabled
$sourceShare = Get-AzStorageShare -Context $sourceContext.Context -Name $sourceStorageFileShareName
$sourceSnapshot = $sourceShare.CloudFileShare.Snapshot()

# Generate source file share SAS URI
$sourceShareSASURI = New-AzStorageShareSASToken -Context $sourceContext `
  -ExpiryTime(get-date).AddDays(1) -FullUri -ShareName $sourceStorageFileShareName -Permission rl
# Set source file share snapshot SAS URI
$sourceSnapSASURI = $sourceSnapshot.SnapshotQualifiedUri.AbsoluteUri + "&" + $sourceShareSASURI.Split('?')[-1]

#! TARGET Storage Account in a different region
# Get Target Storage Account Key
$targetStorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $targetStorageAccountRG -Name $targetStorageAccountName).Value[0]

# Set Target Azure Storage Context
$destinationContext = New-AzStorageContext -StorageAccountKey $targetStorageAccountKey -StorageAccountName $targetStorageAccountName

# Generate target SAS URI
$targetShareSASURI = New-AzStorageShareSASToken -Context $destinationContext `
    -ExpiryTime(get-date).AddDays(1) -FullUri -ShareName $targetStorageFileShareName -Permission rwl

# Create AzCopy syntax command
$sourceSnapshotSASURI = "'" + $sourceSnapSASURI + "'"
$targetFileShareSASURI = "'" + $targetShareSASURI + "'"

# Check if target file share contains data
$targetFileShare = Get-AzStorageFile -Sharename $targetStorageFileShareName -Context $destinationContext.Context

# If target share already contains data, use AzCopy sync to sync data from source to target
# Else if target share is empty, use AzCopy copy as it will be more efficient
if ($targetFileShare) {
     $command = "azcopy " + "sync " + $sourceSnapshotSASURI + " " + $targetFileShareSASURI + " --preserve-smb-info" + " --preserve-smb-permissions" + " --recursive"
}
Else {
     $command = "azcopy " + "copy " + $sourceSnapshotSASURI + " " + $targetFileShareSASURI + " --preserve-smb-info" + " --preserve-smb-permissions" + " --recursive"
}
# Create Azure Container Instance and run the AzCopy job
# The container image (peterdavehello/azcopy:latest) is publicly available on Docker Hub and has the latest AzCopy version installed
# You could also create your own private container image and use it instead
# When you create a new container instance, the default compute resources are set to 1vCPU and 1.5GB RAM
# We recommend starting with 2vCPU and 4GB memory for larger file shares (E.g. 3TB)
# You may need to adjust the CPU and memory based on the size and churn of your file share
New-AzContainerGroup -ResourceGroupName $sourceStorageAccountRG `
         -Name azcopyjob -image peterdavehello/azcopy:latest -OsType Linux `
         -Cpu 2 -MemoryInGB 4 -Command $command `
         -RestartPolicy never

# List the current snapshots on the target share
$snapshots = Get-AzStorageShare `
    -Context $destinationContext.Context | `
Where-Object { $_.Name -eq $targetStorageFileShareName -and $_.IsSnapshot -eq $true}

# Delete the oldest (1) manual snapshot in the target share if have 190 or more snapshots (Azure Files snapshot limit is 200)
If ((($snapshots.count)+10) -ge $maxSnapshots) {
    $manualSnapshots = $snapshots | where-object {$_.ShareProperties.Metadata.Keys -eq "AzureBackupProtected"}
    Remove-AzStorageShare -Share $manualSnapshots[0].CloudFileShare -Force
}

# Take manual snapshot on the target share
# When taking a snapshot using PowerShell, the snapshot's metadata is set to a key-value pair with the key being "AzureBackupProtected"
# The value of "AzureBackupProtected" is set to "True" or "False" depending on whether Azure Backup is enabled
$targetShare = Get-AzStorageShare -Context $destinationContext.Context -Name $targetStorageFileShareName
$targetShareSnapshot = $targetShare.CloudFileShare.Snapshot()

Write-Output ("")

Save the script in the CMDLETS pane as shown in the figure below.

Edit Azure Automation Runbook Script

Then test the script using the “Test pane” button to verify it’s working as intended before you publish it. On the Test page, you need to supply the following parameters manually and then click the Start button to test the automation script.

  • SOURCEAZURESUBSCRIPTIONID
  • SOURCESTORAGEACCOUNTRG
  • TARGETSTORAGEACCOUNTRG
  • SOURCESTORAGEACCOUNTNAME
  • TARGETSTORAGEACCOUNTNAME
  • SOURCESTORAGEFILESHARENAME
  • TARGETSTORAGEFILESHARENAME

Test Azure Automation Runbook Script

Once the test is completed successfully, publish the Runbook by clicking on Publish. This is a very important step!

Schedule the Runbook

In the final step, you need to schedule the Runbook to run based on your desired time to sync and backup the changes from the primary Azure file share to the secondary file share in a different region.

Within the same Runbook that you created in the previous step, select Schedules and then click + Add schedule.

So, if you need to schedule the Runbook to run twice a day, then you need to create the following schedule with Recur every 12 Hours with Set expiration to No. You can also run it on-demand if you wish to do so.

Schedule Azure Automation Runbook

While scheduling the Runbook, you need to pass on the required parameters for the PowerShell script to run successfully. In this scenario, you need to specify the following:

  • Azure Subscription ID where the primary file share is created.
  • Source Storage Resource Group name where the primary storage account is created.
  • Target Storage Resource Group name where the secondary storage account is created.
  • Source Storage Account name.
  • Target Storage Account name.
  • Source Azure File Share name that you want to backup from.
  • Target Azure File Share name that you want to backup to.

The automation script takes those parameters as input as shown in the figure below.

Schedule Azure Automation Runbook with Parameters

Once done, click OK.

Test the Runbook

In this quick demo, I will test the Runbook and request on-demand file share sync to copy the data from the primary Azure file share to a secondary Azure file share in a different region.

This scenario simulates when an application or user adds or modifies files directly in Azure File Share and/or Azure File Sync, and then the automation script takes a snapshot (backup) on the primary file share and then sync/copy the snapshot to the secondary Azure file share, and finally takes a snapshot on the target file share.

Monitor the Runbook

You can monitor the success or failure of the automation backup using the ‘Jobs‘ tab of Runbooks under Resources as shown in the figure below.

Monitor Azure Automation PowerShell Runbook

You can also see the completion of each automation job by clicking on any completed job and then select the Output tab as shown in the figure below. In the output log, you will see if the script runs successfully by looking at the ‘State: Succeeded’. Note that the success state for the automation job does not mean the actual copy/sync job has been completed successfully.

Azure Automation Output Log

You can see the actual copy job completion details by going to the Azure Container Instance: Settings | Containers | Logs as shown in the figure below.

AzCopy Job Container Log

That’s it there you have it!

What you should know

You may need to make certain adjustments to the script based on your particular configuration.

The time for the sync between source and target to complete will depend on the churn on the source, the data size and the number of files being synced, and the available IOPS on both the source and target shares.

You should test the script before deploying it in production to ensure that AzCopy can fully sync your data within your scheduled time frame. If using premium file shares, ensure you provision enough IOPs such that running the script does not disrupt your regular processes. You may also want to modify your AzCopy configuration (for example by increasing the thread count). Likewise, you may need to adjust the settings of the container you provision based on your file share’s size and churn. Additionally, you may need to increase the SAS URI Token expiry time based on the amount of data you have to copy. The SAS must be valid throughout the whole job duration since we need it to interact with the service. I would suggest padding the expiration a bit just to be safe.

An additional point that you want to be aware of if you have a restricted storage account(s) for public access. The AzCopy job running inside the container is using the public IP address of the Azure Container Instance (ACI) to connect to the source and destination storage accounts to perform the Copy/Sync jobs. For this scenario, you have two options as follows:

  1. You could deploy the Azure Container Instance (ACI) into a vNet, and then use a Service Endpoint on your Storage Account(s) down to the ACI subnet. You’d have a single vNet, create a subnet for ACI and then tie the two storage accounts down to that subnet. This means that the storage accounts can only be accessed from resources within that subnet, which is good from a security point of view, but it doesn’t help with data exfiltration concerns (check option 2 below). And if you are going across regions (not region pairs), then you should be able to use the public IP of the ACI in the storage account firewall. That said, service endpoints for storage do work across paired regions, so if you were copying between North Europe/West Europe then the service endpoints would work. When you go to the Firewall on the Storage Account and choose Selected Networks, you can choose subnets in both the primary and paired regions.
  2. Or, you could assign a Private Endpoint for the Storage Account in the same vNet as the ACI and connect via that. Private Endpoints offer higher security as each individual storage account has a private IP in a separate subnet in the vNet, so has no public endpoint access at all. That said, you then have to manage DNS more carefully, and you pay for traffic through the Private Endpoints, so that’s another consideration. Option 2 is more complex than option 1.

I would recommend option 1 to sync between two Azure File Shares for backup and disaster recovery.

How it works…

The runbook runs a command which automatically creates incremental backups (snapshots) of an Azure Files system and stores the backups in a separate storage account. It does so by leveraging the AzCopy tool with the sync parameter, which is similar to Robocopy /MIR. Only changes will be copied with every backup, and any deletions on the source will be also mirrored on the target file share.

How the script works…

The script starts by taking the following 7 required parameters:

  1. sourceAzureSubscriptionId
  2. sourceStorageAccountRG
  3. targetStorageAccountRG
  4. sourceStorageAccountName
  5. targetStorageAccountName
  6. sourceStorageFileShareName
  7. targetStorageFileShareName

We set the maximum snapshots variable with 200 as a value. Why 200, because at the time of this writing, every single Azure file share supports a maximum of 200 snapshots by design. We need to track the number of manual snapshots taken by the script since you might already be using Azure Backup to protect the share in the same region. Azure Backup controls the retention of snapshots it takes, but it does NOT automatically delete manual snapshots if the snapshot limit is hit, and instead fails backup with an error. Thus, we will be deleting only the oldest (1) manual snapshot which was taken by the script once we reach above 180 snapshots on the primary file share, however, on the target file share we set it to delete once we reach more than 190 snapshots. If we are using Azure Backup, we can calculate the maximum number of snapshots Azure Backup will store based on our retention policies. Using this, we should adjust the threshold for deleting the oldest manual snapshot on the source to ensure there is always enough room for our scheduled backup jobs to take snapshots. For example, a limit of 180 allows us to take 6 months of daily backups and leaves room for 20 ad hoc backups.

Once we determine the number of snapshots is below the limit, we take a backup (snapshot) on the source file share. For added security, we generate a file share SAS URI which is only valid for 1 day with Read (R) and List (L) permissions and assigns it to the source snapshot. We then generate a file share SAS URI for the target file share which is also valid for 1 day with Read (R), Write (W), and List (L) permissions.

Next, we check if the target file share contains data because if the destination share is empty, then performing a copy makes sense since it uses fewer resources compare to sync. Then we spin a lightweight container and run the AzCopy Copy command if the target file share is empty, and the AzCopy Sync command if the target file share is NOT empty while preserving the SMB information and SMB Permissions (i.e. ACLs and metadata) in recursive mode. So with Sync, only the changes will be copied with every backup, and any deletions on the source will be also removed on the target file share. In this scenario, AzCopy is running in a Container inside an Azure Container Instance using Service Principal in Azure AD.

Once the synchronization is completed between the source and the target file share, we check the current snapshot numbers on the target share to see if we are still below the limit, then we delete any snapshot if needed.

Finally, we take a manual snapshot of the target file share to make sure we have a consistent backup.

Summary

In this article, I showed you how to sync and copy from a primary Azure file share to a secondary Azure file share in a different Azure region using the AzCopy tool running in a container. In this way, we can run the container with sync jobs on a simple schedule and only get billed for the time the container is used.

Learn more:

This is version 1.0 of the sync between two Azure File Shares for disaster recovery (DR) that was developed in coordination with Microsoft, if you have any feedback or changes that everyone should receive, please feel free to leave a comment below.

__
Thank you for reading my blog.

If you have any questions or feedback, please leave a comment.

-Charbel Nemnom-

Related Posts

Previous

Automate Failback for SQL AlwaysOn Availability Group

Protecting Azure Blobs using Azure Backup

Next

Let me know what you think, or ask a question...

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe to Stay in Touch

Never miss out on your favorite posts and our latest announcements!

The content of this website is copyrighted from being plagiarized! You can copy from the 'Code Blocks' in Black.

Please send your feedback to the author using this form for any 'Code' you like.

Thank you for visiting!

ads