You dont have javascript enabled! Please enable it!

Sync Between Two Azure File Shares for Disaster Recovery

17 Min. Read

Azure Files enables you to set up highly available network file shares that can be accessed by using the standard Server Message Block (SMB) protocol or the Network File System (NFS) protocol. That means that multiple VMs can share the same files with both read and write access. You can also read the files using the REST interface or the storage client libraries.

One thing that distinguishes Azure Files from files on a corporate file share is that you can access the files from anywhere in the world using a URL that points to the file and includes a shared access signature (SAS) token. You can generate SAS tokens; they allow specific access to a private asset for a specific amount of time.

In this article, we will share with you how to sync between two Azure file shares for Disaster Recovery in two different storage accounts.

Introduction

If you have been working with file shares for quite some time, you have probably noticed that Azure Files provides two options to connect to an Azure file share:

1) You can directly access the Azure file share using SMB, NFS, or REST API protocols.

2) You can create a cache of the Azure file share on a Windows Server with Azure File Sync and access the data from the Windows Server whether running on-premises or in Azure (IaaS VM).

Azure file shares can be mounted concurrently by cloud or on-premises deployments. Azure Files SMB file shares are accessible from Windows, Linux, and macOS clients. Azure Files NFS file shares are accessible only from Linux or macOS clients.

Microsoft supports locally redundant storage (LRS), zone redundant storage (ZRS), and geo-redundant storage (GRS) to ensure that Azure services are always available. GRS asynchronously replicates your data to a secondary paired region. However, you can only access your data in the secondary region if Microsoft initiates a failover from the primary to secondary region from an account configured with GRS, so Microsoft recommends that you treat the new region as a backup of data only.

One of the most requested features by customers is to backup Azure file share(s) to a secondary region of their choice for disaster recovery. Additionally, Azure File Sync does NOT automatically begin syncing with the new (primary) region after failover. Lastly, Microsoft does not support GRS for premium file shares.

It’s very important to design your application for high availability from the start. Microsoft recommends keeping the following best practices in mind for maintaining high availability for your Azure Storage data:

  • Disks: Use Azure Backup to back up the VM disks used by your Azure virtual machines. Also, consider using Azure Site Recovery to protect your VMs in the event of a regional disaster.
  • Block blobs: Turn on soft delete to protect against object-level deletions and overwrites, or copy block blobs to another storage account in a different Azure region using AzCopy, Azure PowerShell, or the Azure Data Movement library.
  • Files: Use Azure Backup to back up your file shares. Also, enable soft delete to protect against accidental file share deletions. For geo-redundancy in cases where GRS is not available, use AzCopy or Azure PowerShell to copy your files to another storage account in a different region.
  • Tables: Use AzCopy to export table data to another storage account in a different region.

What if you want to have a secondary copy of your data that resides in an Azure file share in the region of your choice without using a geo-redundant storage (GRS) storage account?

In this article, we will share with you how to sync between two different Azure file shares in two different storage accounts and two different regions, so you can have a copy of your data in a secondary region of your choice while maintaining security and business continuity.

For this article, we will make use of the AzCopy tool, which is a command-line utility that you can use to sync or copy files to or from a storage account. We will also use Azure Container Instances to simplify and automate the AzCopy script in Runbook, which will run as part of the container. In this way, we can run the container on a simple schedule to sync the data and only get billed for the time the container was used.

Prerequisites

To follow this article, you need to have the following:

1) Azure subscription – If you don’t have an Azure subscription, you can create a free one here.

2) You need to have two different storage accounts either in the same region and same subscription or in different regions and subscriptions.

3) You also need to create two Azure file shares across two different storage accounts either in the same region and same subscription or in different regions and subscriptions.

4) Lastly, you need to have some files in the first Azure file share (which is considered the primary share). You can add these files directly in the portal or by mounting the share, or you can sync data from Windows Servers to the file share using Azure File Sync.

At a high level, the design deployment is represented by the following diagram:

Sync Between Two Azure File Shares for Disaster Recovery
Deployment Design – Sync between two Azure File Shares for Disaster Recovery

Get started

First, we need to create an Azure automation account that will help you to automate the synchronization and backup process without user interaction. This will also make sure to respect the security access of your storage account without exposing access keys to users.

Create Automation Account

In this step, we will create an Azure automation resource with a Run As account. Run As accounts in Azure Automation are used to provide authentication for managing resources in Azure with the Azure cmdlets. When you select ‘Create an Azure Run As account‘, it creates a new service principal user in Azure Active Directory (Azure AD) and assigns the Contributor role to the service principal at the subscription level where the Automation Account is created.

Open the Azure portal, and click All services found in the upper left-hand corner. In the list of resources, type Automation. As you begin typing, the list filters based on your input. Select Automation Accounts.

Click +Add. Enter the automation account name, choose the right subscription, resource group, and location, and then click Create.

Create Azure Automation Account

Grant Run As account permissions in other subscriptions

Please note that you can use the same workflow described in this article to sync your data between two Azure file shares in two different Azure subscriptions as well. In this scenario, you need to manually assign the service principal account that gets created as part of the automation account to the target Resource Group in the second subscription.

Azure Automation supports using a single Automation account from one subscription and executing Runbooks against Azure Resource Manager resources across multiple subscriptions. In the second subscription, you assign the Run As account service principal the Contributor role in the other resource group.

We highly recommend assigning the Contributor role on the resource group level where you have created the source and target storage accounts, do not assign the Contributor role on the subscription level for added security. For more information, please check the Role-based access control in Azure Automation.

To assign the Run As account to the role in the other subscription, the user account performing this task needs to be a member of the Owner role in that subscription.

You can find the service principal name in Azure AD in the automation account under Account Settings | Run as accounts | Azure Run Az Account as shown in the figure below (In this example, my automation account name is aac-sync-afs-blob_xxxx).

Azure Automation Azure AD App Registrations

As mentioned in the previous section, when you create an automation account with Run as Account, it creates a new service principal in Azure Active Directory (Azure AD) and assigns the Contributor role to the service principal at the subscription level by default. To maintain a high level of security, please remove the Contributor role for the service principal from the subscription level and assign it to the resource group level instead.

Updated – 11/11/2021 – You can now create an Azure automation account with a Managed Identity. Microsoft recommends using Managed Identities for the Automation accounts instead of using Run As accounts. Managed identity would be more secure and offers ease of use since it doesn’t require any credentials to be stored. Azure Automation support for Managed Identities is now generally available. When you create a new Automation Account, System assigned identity is enabled.

We highly recommend moving away from the Run As account to Managed Identity, we kept both documented in this article as a reference.

When you create an Automation Account with Managed Identity, it creates a new service principal user in Azure Active Directory (Azure AD) by default. Next, you must assign the appropriate (Azure RBAC) Contributor role to allow access to Azure Storage for the service principal at the resource group level. If you have two different subscriptions and two different resource groups, then you must assign the RBAC Contributor role for the service principal on the source and target resource group.

Always keep in mind and follow the principle of least privilege and carefully assign permissions only required to execute your runbook.

Import Az modules from Gallery

In the next step, you need to import the required modules from the Modules gallery. In your list of Automation Accounts, select the account that you created in the previous step. Then from your automation account, select Modules under Shared Resources. Click the Browse Gallery button to open the Browse Gallery page. You need to import the following modules from the Modules gallery in the order given below:

1) Az.Accounts

2) Az.ContainerInstance

3) Az.Storage

Sync Between Two Azure File Shares for Disaster Recovery 1

Updated – 11/11/2021 – Starting in September 2021, automation accounts will now have Az modules by default installed. You don’t need to import the modules from the gallery as shown in the figure above. Please note that you can also update the modules to the latest Az version from the modules blade as shown in the figure below.

Update Az Modules
Update Az Modules

At the time of this writing, AzCopy is still not part of the Azure Automation Runbook. For this reason, we’ll be creating an Azure Container instance with AzCopy as part of the container to automate the entire synchronization and snapshots backup process.

Create PowerShell Runbook

In this step, you can create multiple Runbooks based on which set of Azure file shares you want to sync/copy. PowerShell Runbooks are based on Windows PowerShell. You directly edit the code of the Runbook using the text editor in the Azure portal. You can also use any offline text editor such as Visual Studio Code and import the Runbook into Azure Automation.

From your automation account, select Runbooks under Process Automation. Click the ‘+ Create a runbook‘ button to open the Create a runbook blade. Select PowerShell as Runbook type, and then select the runtime version to be used for the PowerShell runbook type. In this example, we are using the 7.1 (preview) Runtime version. Please note that runtime version 5.1 will also work.

> 5.1 Runtime version executes the runbook in PowerShell 5.1 runtime.
> 7.1 (preview) Runtime version executes the runbook in PowerShell 7.1 (preview) runtime.

Create PowerShell Runbook
Create PowerShell Runbook

In this example, we will create a Runbook to synchronize and copy all files and directories changes from a primary Azure file share (snapshot) to a secondary file share in a different storage account in a different Azure region. You can also be creative as much as you want and cover different Azure file shares, etc.

Edit the Runbook

Once you have the Runbook created, you need to edit the Runbook, then write or add the script to choose which Azure file shares you want to sync. Of course, you can create scripts that suit your environment.

As mentioned earlier, in this example, we will create a Runbook to synchronize and copy all data from one Azure file share to another Azure file share which can be in a different storage account and region. And to maintain a high level of security, we WON’T use the storage account keys. Instead, we will create a time limit SAS token URI for each file share, and the SAS token will expire automatically after 1 day. So, if you regenerate your storage account keys in the future, the automation process won’t break.

The script is as follows. You can copy the script from the ‘Code Block’ below or GitHub here.

Updated – 27/12/2021 – The script below has been updated and tested with the latest Az.ContainerInstance module version 2.1 and above.

<#
.DESCRIPTION
A Runbook example that automatically creates incremental backups of an Azure Files system on a customer-defined schedule and stores the backups in a separate storage account.
It does so by leveraging AzCopy with the sync parameter, which is similar to Robocopy /MIR. Only changes will be copied with every backup, and any deletions on the source will be mirrored on the target.
In this example, AzCopy is running in a Container inside an Azure Container Instance using Service Principal in Azure AD.

.NOTES
Filename : SyncBetweenTwoFileShares
Author1  : Sibonay Koo (Microsoft PM)
Author2  : Charbel Nemnom (Microsoft MVP/MCT)
Version  : 2.1
Date     : 13-February-2021
Updated  : 18-April-2022
Tested   : Az.ContainerInstance PowerShell module version 2.1 and above

.LINK
To provide feedback or for further assistance please email: azurefiles[@]microsoft.com
#>

Param (
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $sourceAzureSubscriptionId,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $sourceStorageAccountRG,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $targetStorageAccountRG,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $sourceStorageAccountName,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $targetStorageAccountName,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $sourceStorageFileShareName,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $targetStorageFileShareName
)

# Azure File Share maximum snapshot support limit by the Azure platform is 200
[Int]$maxSnapshots = 200

# Ensures you do not inherit an AzContext in your runbook
Disable-AzContextAutosave -Scope Process

# Connect to Azure with system-assigned managed identity (automation account)
Connect-AzAccount -Identity

# SOURCE Azure Subscription
Select-AzSubscription -SubscriptionId $sourceAzureSubscriptionId

#! Source Storage Account in the primary region
# Get Source Storage Account Key
$sourceStorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $sourceStorageAccountRG -Name $sourceStorageAccountName).Value[0]

# Set Azure Storage Context
$sourceContext = New-AzStorageContext -StorageAccountKey $sourceStorageAccountKey -StorageAccountName $sourceStorageAccountName

# List the current snapshots on the source share
$snapshots = Get-AzStorageShare `
    -Context $sourceContext.Context | `
Where-Object { $_.Name -eq $sourceStorageFileShareName -and $_.IsSnapshot -eq $true}

# Delete the oldest (1) manual snapshot in the source share if have 180 or more snapshots (Azure Files snapshot limit is 200)
# This leaves a buffer such that there can always be 6 months of daily snapshots and 10 yearly snapshots taken via Azure Backup
# You may need to adjust this buffer based on your snapshot retention policy
If ((($snapshots.count)+20) -ge $maxSnapshots) {
    $manualSnapshots = $snapshots | where-object {$_.ShareProperties.Metadata.Initiator -eq "Manual"}
    Remove-AzStorageShare -Share $manualSnapshots[0].CloudFileShare -Force
}

# Take manual snapshot on the source share
# When taking a snapshot using PowerShell, CLI or from the Portal, the snapshot's metadata is set to a key-value pair with the key being "Manual"
# The value of "AzureBackupProtected" is set to "True" or "False" depending on whether Azure Backup is enabled
$sourceShare = Get-AzStorageShare -Context $sourceContext.Context -Name $sourceStorageFileShareName
$sourceSnapshot = $sourceShare.CloudFileShare.Snapshot()

# Generate source file share SAS URI
$sourceShareSASURI = New-AzStorageShareSASToken -Context $sourceContext `
  -ExpiryTime(get-date).AddDays(1) -FullUri -ShareName $sourceStorageFileShareName -Permission rl
# Set source file share snapshot SAS URI
$sourceSnapSASURI = $sourceSnapshot.SnapshotQualifiedUri.AbsoluteUri + "&" + $sourceShareSASURI.Split('?')[-1]

#! TARGET Storage Account in a different region
# Get Target Storage Account Key
$targetStorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $targetStorageAccountRG -Name $targetStorageAccountName).Value[0]

# Set Target Azure Storage Context
$destinationContext = New-AzStorageContext -StorageAccountKey $targetStorageAccountKey -StorageAccountName $targetStorageAccountName

# Generate target SAS URI
$targetShareSASURI = New-AzStorageShareSASToken -Context $destinationContext `
    -ExpiryTime(get-date).AddDays(1) -FullUri -ShareName $targetStorageFileShareName -Permission rwl

# Check if target file share contains data
$targetFileShare = Get-AzStorageFile -Sharename $targetStorageFileShareName -Context $destinationContext.Context

# If target share already contains data, use AzCopy sync to sync data from source to target
# Else if target share is empty, use AzCopy copy as it will be more efficient
# By default, the replicated files that you deleted from the source does not get deleted on the destination.
# The deletion on the destination is optional because some customers want to keep them as backup in the DR location.
# If you want to delete the files, then you want to add the " --delete-destination true" flag to the $command below.
# The "--delete-destination" defines whether to delete extra files from the destination that are not present at the source.
if ($targetFileShare) {
     $command = "azcopy","sync",$sourceSnapSASURI,$targetShareSASURI,"--preserve-smb-info","--preserve-smb-permissions","--recursive"
}
Else {
     $command = "azcopy","copy",$sourceSnapSASURI,$targetShareSASURI,"--preserve-smb-info","--preserve-smb-permissions","--recursive"
}

# The container image (peterdavehello/azcopy:latest) is publicly available on Docker Hub and has the latest AzCopy version installed
# You could also create your own private container image and use it instead
# When you create a new container instance, the default compute resources are set to 1vCPU and 1.5GB RAM
# We recommend starting with 2vCPU and 4GB memory for larger file shares (E.g. 3TB)
# You may need to adjust the CPU and memory based on the size and churn of your file share
# The container will be created in the $location variable based on the source storage account location. Adjust if needed.
$location = (Get-AzResourceGroup -Name $sourceStorageAccountRG).location
$containerGroupName = "syncafsdrjob"

# Set the AZCOPY_BUFFER_GB value at 2 GB which would prevent the container from crashing.
$envVars = New-AzContainerInstanceEnvironmentVariableObject -Name "AZCOPY_BUFFER_GB" -Value "2"

# Create Azure Container Instance Object
$container = New-AzContainerInstanceObject `
-Name $containerGroupName `
-Image "peterdavehello/azcopy:latest" `
-RequestCpu 2 -RequestMemoryInGb 4 `
-Command $command -EnvironmentVariable $envVars

# Create Azure Container Group and run the AzCopy job
$containerGroup = New-AzContainerGroup -ResourceGroupName $sourceStorageAccountRG -Name $containerGroupName `
-Container $container -OsType Linux -Location $location -RestartPolicy never

# List the current snapshots on the target share
$snapshots = Get-AzStorageShare `
    -Context $destinationContext.Context | `
Where-Object { $_.Name -eq $targetStorageFileShareName -and $_.IsSnapshot -eq $true}

# Delete the oldest (1) manual snapshot in the target share if have 190 or more snapshots (Azure Files backup snapshot limit is 200)
If ((($snapshots.count)+10) -ge $maxSnapshots) {
    $manualSnapshots = $snapshots | where-object {$_.ShareProperties.Metadata.Initiator -eq "Manual"}
    Remove-AzStorageShare -Share $manualSnapshots[0].CloudFileShare -Force
}

# Take manual snapshot on the target share
# When taking a snapshot using PowerShell, CLI or from the Portal, the snapshot's metadata is set to a key-value pair with the key being "Manual"
# The value of "AzureBackupProtected" is set to "True" or "False" depending on whether Azure Backup is enabled
$targetShare = Get-AzStorageShare -Context $destinationContext.Context -Name $targetStorageFileShareName
$targetShareSnapshot = $targetShare.CloudFileShare.Snapshot()

Write-Output ("")

Save the script in the CMDLETS pane as shown in the figure below.

Edit Azure Automation Runbook Script

Then test the script using the “Test pane” button to verify it’s working as intended before you publish it. On the Test page, you need to supply the following parameters manually and then click the Start button to test the automation script.

  • SOURCEAZURESUBSCRIPTIONID
  • SOURCESTORAGEACCOUNTRG
  • TARGETSTORAGEACCOUNTRG
  • SOURCESTORAGEACCOUNTNAME
  • TARGETSTORAGEACCOUNTNAME
  • SOURCESTORAGEFILESHARENAME
  • TARGETSTORAGEFILESHARENAME

Test Azure Automation Runbook Script

Once the test is completed successfully, publish the Runbook by clicking on Publish. This is a very important step!

Schedule the Runbook

In the final step, you need to schedule the Runbook to run based on your desired time to sync and backup the changes from the primary Azure file share to the secondary file share in a different region.

Within the same Runbook that you created in the previous step, select Schedules and then click + Add schedule.

So, if you need to schedule the Runbook to run twice a day, then you need to create the following schedule with Recur every 12 Hours with Set expiration to No. You can also run it on-demand if you wish to do so.

Schedule Azure Automation Runbook

While scheduling the Runbook, you need to pass on the required parameters for the PowerShell script to run successfully. In this scenario, you need to specify the following:

  • Azure Subscription ID where the primary file share is created.
  • Source Storage Resource Group name where the primary storage account is created.
  • Target Storage Resource Group name where the secondary storage account is created.
  • Source Storage Account name.
  • Target Storage Account name.
  • Source Azure File Share name that you want to backup from.
  • Target Azure File Share name that you want to backup to.

The automation script takes those parameters as input as shown in the figure below.

Schedule Azure Automation Runbook with Parameters

Once done, click OK.

Test the Runbook

In this quick demo, I will test the Runbook and request on-demand file share sync to copy the data from the primary Azure file share to a secondary Azure file share in a different region.

This scenario simulates when an application or user adds or modifies files directly in Azure File Share and/or Azure File Sync, and then the automation script takes a snapshot (backup) on the primary file share and then sync it (copy) the snapshot to the secondary Azure file share, and finally takes a snapshot on the target file share.

Monitor the Runbook

You can monitor the success or failure of the automation backup using the ‘Jobs‘ tab of Runbooks under Resources as shown in the figure below.

Monitor Azure Automation PowerShell Runbook

You can also see the completion of each automation job by clicking on any completed job and then selecting the Output tab as shown in the figure below. In the output log, you will see if the script runs successfully by looking at the ‘State: Succeeded’. Note that the success state for the automation job does not mean the actual copy/sync job has been completed successfully.

Azure Automation Output Log

You can see the actual copy job completion details by going to the Azure Container Instance: Settings | Containers | Logs as shown in the figure below.

AzCopy Job Container Log

That’s it there you have it!

What you should know

You may need to make certain adjustments to the script based on your particular configuration.

The time for the sync between source and target to complete will depend on the churn on the source, the data size and the number of files being synced, and the available IOPS on both the source and target shares.

You should test the script before deploying it in production to ensure that AzCopy can fully sync your data within your scheduled time frame. If using premium file shares, ensure you provision enough IOPs such that running the script does not disrupt your regular processes.

You may also want to modify your AzCopy configuration (for example by increasing the concurrency value). Likewise, you may need to adjust the settings of the container you provision based on your file share’s size and churn (data change). Additionally, you may need to increase the SAS URI Token expiry time based on the amount of data you have to copy. The SAS must be valid throughout the whole job duration since we need it to interact with the service. We would suggest padding the expiration a bit just to be safe.

An additional point that you want to be aware of is if you have a restricted storage account(s) for public access. The AzCopy job runs inside the container is using the public IP address of the Azure Container Instance (ACI) to connect to the source and destination storage accounts to perform the Copy/Sync jobs. For this scenario, you have two options as follows:

1) You could deploy the Azure Container Instance (ACI) into a vNet, and then use a Service Endpoint on your Storage Account(s) down to the ACI subnet. You’d have a single vNet, create a subnet for ACI and then tie the two storage accounts down to that subnet. This means that the storage accounts can only be accessed from resources within that subnet, which is good from a security point of view, but it doesn’t help with data exfiltration concerns (check option 2 below).

And if you are going across regions (not region pairs), then you should be able to use the public IP of the ACI in the storage account firewall. That said, service endpoints for storage do work across paired regions, so if you were copying between North Europe/West Europe then the service endpoints would work. When you go to the Firewall on the Storage Account and choose Selected Networks, you can choose subnets in both the primary and paired regions:

2) Or, you could assign a Private Endpoint for the Storage Account in the same vNet as the ACI and connect via that. Private Endpoints offer higher security as each storage account has a private IP in a separate subnet in the vNet, so has no public endpoint access at all. That said, you then have to manage DNS more carefully, and you pay for traffic through the Private Endpoints, so that’s another consideration. Option 2 is more complex than option 1:

We would recommend option 1 to sync between two Azure File Shares for backup and disaster recovery.

How it works…

The runbook runs a command which automatically creates incremental backups (snapshots) of an Azure Files system and stores the backups in a separate storage account. It does so by leveraging the AzCopy tool with the sync parameter, which is similar to Robocopy /MIR. Only changes will be copied with every backup, and any deletions on the source will be also mirrored on the target file share.

How the script works…

The script starts by taking the following 7 required parameters:

  1. sourceAzureSubscriptionId
  2. sourceStorageAccountRG
  3. targetStorageAccountRG
  4. sourceStorageAccountName
  5. targetStorageAccountName
  6. sourceStorageFileShareName
  7. targetStorageFileShareName

We set the maximum snapshots variable with 200 as a value. Why 200, because at the time of this writing, every single Azure file share supports a maximum of 200 snapshots by design. We need to track the number of manual snapshots taken by the script since you might already be using Azure Backup to protect the share in the same region.

Azure Backup controls the retention of snapshots it takes, but it does NOT automatically delete manual snapshots if the snapshot limit is hit, and instead fails backup with an error. Thus, we will be deleting only the oldest (1) manual snapshot which was taken by the script once we reach above 180 snapshots on the primary file share, however, on the target file share we set it to delete once we reach more than 190 snapshots.

If we are using Azure Backup, we can calculate the maximum number of snapshots Azure Backup will store based on our retention policies. Using this, we should adjust the threshold for deleting the oldest manual snapshot on the source to ensure there is always enough room for our scheduled backup jobs to take snapshots. For example, a limit of 180 allows us to take 6 months of daily backups and leaves room for 20 ad hoc backups.

Once we determine the number of snapshots is below the limit, we take a backup (snapshot) on the source file share. For added security, we generate a file share SAS URI which is only valid for 1 day with Read (R) and List (L) permissions and assigns it to the source snapshot. Then we generate a file share SAS URI for the target file share which is also valid for 1 day with Read (R), Write (W), and List (L) permissions.

Next, we check if the target file share contains data because if the destination share is empty, then performing a copy makes sense since it uses fewer resources compared to sync. Then we spin a lightweight container and run the AzCopy Copy command if the target file share is empty, and the AzCopy Sync command if the target file share is NOT empty while preserving the SMB information and SMB Permissions (i.e. ACLs and metadata) in recursive mode.

With Sync, only the changes will be copied with every backup, and any deletions on the source will be also removed on the target file share. In this scenario, AzCopy is running in a Container inside an Azure Container Instance using Service Principal in Azure AD.

Once the synchronization is completed between the source and the target file share, we check the current snapshot numbers on the target share to see if we are still below the limit, then we delete any snapshot if needed.

Finally, we take a manual snapshot of the target file share to make sure we have a consistent backup.

Storage Accounts with disabled network access

Azure Container Instances with virtual network (VNET) integration using PowerShell.

If you need to create a container instance with VNET integration rather than using a public IP, then you need to add the -SubnetId parameter followed by the resource ID of your virtual network and subnet to this section of the script:

# Create Azure Container Group and run the AzCopy job
# Create the subnet resource ID hashtable for the container group
$subnetId = @{
    Id = '/subscriptions/subscription-id/resourceGroups/resource-group-name/providers/Microsoft.Network/virtualNetworks/virtual-network-name/subnets/subnet-name'
    Name = 'Friendly name of your virtual subnet'
}
$containerGroup = New-AzContainerGroup -ResourceGroupName $sourceStorageAccountRG -Name $containerGroupName `
-Container $container -OsType Linux -Location $location `
-SubnetId $subnetId -RestartPolicy never

This assumes that you already have an existing virtual network and dedicated subnet created for the container instance (ACI).

Next, you create Private endpoints for both storage accounts with Private DNS Zone entries for those.

Last but not least, you create a VNET peering between your VNETs of Azure Region Pairs.

Summary

In this article, we showed you how to sync and copy from a primary Azure file share to a secondary Azure file share in a different Azure region using the AzCopy tool running in a container. In this way, we can run the container with sync jobs on a simple schedule and only get billed for the time the container is used.

Learn more:

This is version 2.0 of the sync between two Azure File Shares for disaster recovery (DR) that was developed in coordination with Microsoft, if you have any feedback or changes that everyone should receive, please feel free to leave a comment below.

__
Thank you for reading my blog.

If you have any questions or feedback, please leave a comment.

-Charbel Nemnom-

Related Posts

Previous

Automate Failback for SQL AlwaysOn Availability Group

Protecting Azure Blobs using Azure Backup

Next

48 thoughts on “Sync Between Two Azure File Shares for Disaster Recovery”

Leave a comment...

  1. Hello Charbel,
    First of all, thanks for this! It really helps me move data between storage accounts.

    What I do see is that your script uses snapshots to copy/sync the data. When I use your script now, it does not sync the entire data from one storage account to another. Looks like not all the data is in the snapshot?

    Is there a way to amend the script that it uses the entire storage account for synchronization? And not a snapshot?

  2. Hello Jaap, thanks for the comment!
    I am glad to hear that you were able to move data between storage accounts.
    The snapshot should contain all the data, if the target file share contains data, it will sync, if not, it will copy all the data (one time), and then sync afterward.
    Yes, of course, you could amend the script that it uses the entire storage account (file share) for synchronization and not a snapshot.
    Please refer to the following example as described in this article and adjust it to sync between two Azure File Shares.
    Hope it helps!

  3. Hi Charbel,

    Thanks for publishing this article. I am able to get the file sync working with the storage accounts open to all networks. However, I am struggling to lock down the storage account for private access only. I have created the necessary storage endpoints, an endpoint for the automation account, and assigned the container instance a private IP. I keep being thrown this error which has left me really puzzled:

    Environments
    ————
    {[AzureChinaCloud, AzureChinaCloud], [AzureCloud, AzureCloud], [AzureGermanCloud, AzureGermanCloud], [AzureUSGovernme…

    This request is not authorized to perform this operation. HTTP Status Code: 403 – HTTP Error Message: This request is not authorized to perform this operation.
    ErrorCode: AuthorizationFailure
    ErrorMessage: This request is not authorized to perform this operation.
    RequestId:8d086b1c-301a-0008-7a2c-2ea640000000
    Time:2022-03-02T11:53:36.3616691Z
    This request is not authorized to perform this operation. HTTP Status Code: 403 – HTTP Error Message: This request is not authorized to perform this operation.
    ErrorCode: AuthorizationFailure
    ErrorMessage: This request is not authorized to perform this operation.
    RequestId: cc7c957a-501a-0031-012c-2ee6e4000000
    Time: Wed, 02 Mar 2022 11:53:37 GMT
    System.Management.Automation.RuntimeException: You cannot call a method on a null-valued expression.

    However when I open up the storage accounts again the file sync works.

    Thanks,

  4. Hi Charbel,

    When removing public access to a storage account, the step that gets snapshots (e.g. $snapshots = Get-AzStorageShare) starts to fail – this is with managed identities and service endpoints in place.

    I’m presuming this is to do with a lack of service endpoint support for Automation Accounts? The commands are therefore running outside of the allowed container instance and getting blocked.

    Have I missed something?

  5. Hello Chris, thanks for your comment and for sharing your experience!
    Yes exactly, if you look at the Networking section for the storage account under Resource instance (Resource type), we don’t see Automation Accounts so we can allow it and have access to the storage account based on the system-assigned managed identity.
    As a workaround what you could do is, you can allow the Public IPs for Azure Automation service from a specific Azure Region. You can download the list of public IPs by service and region from here.
    Hopefully, Microsoft will add support for Automation Accounts natively to Azure storage.
    Hope it helps!

  6. Hello Aaron, thanks for your comment!
    Azure Automation Accounts lacks service endpoint support to Azure Storage when you want to lock down the storage account for private access only.
    If you look at the Networking section for the storage account under Resource instance (Resource type), we don’t see Automation Accounts so we can allow it and have access to the storage account based on the system-assigned managed identity.
    As a workaround what you could do is, you can allow the Public IPs for Azure Automation service only from a specific Azure Region.
    You can download the list of public IPs by service and region from here.
    Hopefully, Microsoft will add support for Automation Accounts natively to Azure storage.
    Hope it helps!

  7. Great write-up, Charbel! This will be a helpful starting point for a project I am working on.

    Reviewing your script, I think the metadata for snapshots has been changed.

    Instead of `$manualSnapshots = $snapshots | where-object {$_.ShareProperties.Metadata.Keys -eq “AzureBackupProtected”}`, I am needing to use below to get manual snaps. Snaps always appear to have an ‘Initiator’ metadata key, with values either of ‘AzureBackup’ or ‘Manual’:
    `$manualSnapshots = $snapshots | where-object {$_.shareproperties.metadata.initiator -eq ‘Manual’}`

  8. Thank you Matthew for the comment and for sharing your experience!
    Yes, Microsoft has recently updated the metadata for manual snapshots.
    I have updated the script accordingly.

  9. Hello Joe, thanks for the comment and feedback!
    In this case, where NFS does not support snapshots, you can take out (remove) the snapshot section from the script and then sync only or copy the data between the two storage accounts (file shares) in different Azure regions.
    Hope it helps!

  10. First of all – thanks for writing/sharing this it’s extremely helpful and saves folks like myself a lot of time.

    Question – What exactly needs to be modified in the script if I’m going to create a VNET integrated container instance – I’m fine with using the image you provide but need it VNET integrated? It looks like part of the script provisions one however it isn’t VNET integrated.

    Thanks!

  11. Hello Charbel,

    Thanks for the wonderful article, I have set up the automation account via Managed Identity and while running the script I am getting below error at Container instances

    “failed to parse user input due to error: preserve-permissions is set but persistence for up/downloads is a Windows-only feature”

  12. Hello Jaymin, thanks for the comment!
    Please check the permissions that you gave for the Managed Identity.
    You must assign the appropriate RBAC role (i.e. Contributor or Storage Account Contributor) to allow access to Azure Storage for the service principal at the resource group level.
    If you have two different subscriptions and two different resource groups, then you must assign the RBAC Contributor role for the service principal on the source and target resource group.
    Please also make sure that the managed identity has Read permissions only at the subscription level.
    Hope it helps!

  13. Hello Daniel, thanks for the comment!
    I am glad to hear this was helpful to you.
    What you need to modify in the script if you want to create a container instance with VNET integration is the following:
    You need to add the -SubnetId parameter followed by the resource ID of the virtual network and subnet to this section:

    # Create Azure Container Group and run the AzCopy job
    # Create the subnet resource ID hashtable for the container group
    $subnetId = @{
        Id = '/subscriptions/subscription-id/resourceGroups/resource-group-name/providers/Microsoft.Network/virtualNetworks/virtual-network-name/subnets/subnet-name'
        Name = 'Friendly name of your virtual subnet'
    }
    $containerGroup = New-AzContainerGroup -ResourceGroupName $sourceStorageAccountRG -Name $containerGroupName `
    -Container $container -OsType Linux -Location $location `
    -SubnetId $subnetId -RestartPolicy never

    This assumes that you already have an existing virtual network and dedicated subnet created for the container instance.
    Hope it helps!

  14. Thanks for the info on the Subnet integration! Just FYI it does look like you actually have to store the -SubnetId variable in a hash table.

  15. Thank you Daniel for the feedback, I am glad it’s working for you!
    I have updated the script with an example to define the $subnetId variable as a hashtable.

  16. I’m still having one additional problem which is getting a 403 every time I run the playbook. I assume this is a communication issue from the Automation Account to the Storage Accounts. When I open up the storage account it works without issue. I have my ACI VNET Integrated with service endpoints to my storage accounts, Automation Account system managed identity setup with contributor rights on both Storage accounts but to no avail. I even tried service endpoint on the private endpoint on automation account to storage accounts and that did not help either. Looking at Public IPs for automation accounts it’s not sorted by region so that’s not really feasible. Curious how others have this working with firewall-enabled Storage Accounts.

  17. Hello Daniel, thanks for the comment!
    I’ve seen that you posted the same question four times in the comment section :)
    Sorry, I don’t provide support or consulting services in the comment section.
    If you would like to work on this, you can contact me on this page.
    As you probably noticed, Azure Automation Accounts lacks service endpoint support for Azure Storage when you want to lock down the storage account with network firewall/private access.
    If you look at the Networking section for the storage account under Resource instance (Resource type), we don’t see Automation Accounts, so we can allow it to access the storage account based on the system-assigned managed identity.
    As a workaround what you could do is, you can allow the range of Public IPs for the Azure Automation service only from a specific Azure Region.
    You can download the list of public IPs by service and region here.
    Hope it helps!

  18. Thanks for your reply.

    I am using a user-assigned managed identity. I have already given appropriate permission like a contributor role at the resource group level. Also given the reader permission at the subscription level.

    Today also I tried with system-assigned managed identity, but still facing the same issue.

  19. Hello Jaymin, thanks for the update!
    Question: Did you change anything from the script or added any new parameters?
    It looks like the issue is around the AzCopy command not able to perform the initial copy.
    What I would suggest is, to run the script manually piece by piece without Automation Account (runbook), so you can find out which section is failing.
    Make sure to check the variable content before you move and run the next command.
    Sorry, I can’t help further. We need to debug further.
    If you would like to work with me on this, you can contact me on this page.
    Hope it helps!

  20. Just wanted to let you know I did find a solution for my Automation Account access problem – it’s to use a hybrid runbook on the Automation Account + register the AZ modules on the VM you use for it. Works great.

  21. Hello Daniel, thanks for the update!
    I am happy to hear that it’s working for you now.
    In this case, you moved out from using the ACI solution and you used a full VM instead of a Container.
    From a price and operational point of view, it’s a bit expensive compared to a full VM vs ACI.
    And if you deployed the VM on-premises with a hybrid runbook worker, then it’s NOT a full cloud solution!!!
    ACI does support now managed identity, you could leverage it in combination with Automation Account managed identity to access the storage account without SAS Token ;)
    Cheers,

Let me know what you think, or ask a question...

error: Alert: The content of this website is copyrighted from being plagiarized! You can copy from the 'Code Blocks' in 'Black' by selecting the Code. Thank You!