Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync

Updated – 20/07/2023 – This article has been updated to use Azure automation with Managed Identity instead of Azure Automation Run As Account which will retire on September 30, 2023.

Updated – 30/11/2021 – Microsoft has improved the Storage Sync Change Detection and the 10,000 item limit no longer applies when scanning the entire share. Please check the following section for more details.

In this article, we will show you how to enable immediate Sync on Azure File Share with Azure File Sync.

Introduction

Azure File Sync can help you centralize your files in Azure and then install a sync agent on Windows Server whether it’s on-premises or in Azure (IaaS VM) to provide fast local access to your files. Your Windows Server and Azure Files are constantly in sync, so you have one centralized location for your files with multi-site access powered by a fast local cache and cloud tiering.

The cloud tiering feature enables frequently accessed files to be cached locally such that the full file content is present on the server, whereas less frequently accessed files are tiered to the cloud. The tiered files (AFS reparse points) will be recalled on-demand when a user or application accesses it on the local server.

For more information about Azure File Sync, please make sure to check my previous articles.

Since the first release of Azure File Sync, one of the most requested features by many customers, was the ability to sync back when the changes occur directly on the Azure File Share. The user’s voice is very active in improving this capability, you can add your vote here.

The good news is, that Microsoft just added a new way to trigger sync that happens on files that are placed directly in the Azure File Share. Normally it can take up to 24 hours before sync will detect changes that happen directly in the Azure File Share. With this new Azure PowerShell cmdlet introduced by the Azure Storage Team, you can point sync to particular files, directories, or sub-directories and have it look for changes, then sync back all the changes.

This new capability is intended for scenarios where some type of automated process in Azure is doing the file edits or migrations done by an administrator (like moving a new directory of files into the file share). You can also use the same approach when a user adds or changes files directly in Azure File Share.

Microsoft recommends that for end-user direct changes on the Azure File Share, to install the Azure File Sync agent in an IaaS VM, and then give the end-user access through that. This way all changes will quickly sync to other agents without the need to call the Azure PowerShell cmdlet, but nothing stops you from using this method to tackle that scenario as well.

In this article, we will show you how to automate the immediate sync process and make sure that all the changes made on the Azure File Share will sync back to your Windows Server whether it’s on-premises or in Azure.

Invoke-AzStorageSyncChangeDetection Improvements

Prior to the Azure File Sync agent version 14 release, if you made changes directly in the Azure file share, you could use the Invoke-AzStorageSyncChangeDetection cmdlet to detect the changes and sync them to the file servers in your sync group. However, the cmdlet would fail to run if the path specified contained more than 10,000 items (objects).

The good news is, that Microsoft has improved the Invoke-AzStorageSyncChangeDetection cmdlet and the 10,000 item limit no longer applies when scanning the entire share.

Please make sure to download Azure File Sync Agent version 14 or later by following the step-by-step guide.

Create an Azure Automation Account

When you create an Automation Account, it creates a new service principal in Azure Active Directory (Azure AD) by default. Using a Managed Identity instead of the Automation Run As account makes management simpler. You don’t have to renew the certificate used by the Automation Run As account. Additionally, you don’t have to specify the Run As connection object in your runbook code. You can access resources using your Automation account’s managed identity from a runbook without creating certificates, connections, Run As accounts, etc.

Please note that Azure Automation Run As Account will retire on September 30, 2023, and will be replaced with Managed Identities.

Next, you must assign the appropriate (Azure RBAC) role to allow access to the storage account and the storage sync service for the Managed Identity at the resource group, subscription, or management group level.

You need to clone the Storage Account Contributor role first, and then add the following custom Microsoft.StorageSync permissions to it for the script to work properly. Always keep in mind to use the principle of least privilege (PoLP) when assigning permissions.

You can do that by going to Access Control (IAM) and selecting Add Custom Role. Next, clone the Storage Account Contributor role.

Click Next, then select + Add permissions and add the following permissions to the custom role:

Microsoft.StorageSync/storageSyncServices/syncGroups/cloudEndpoints/read
Microsoft.StorageSync/storageSyncServices/syncGroups/cloudEndpoints/triggerChangeDetection/action
Microsoft.StorageSync/storageSyncServices/syncGroups/cloudEndpoints/operationresults/read

Next, in the Azure portal, click All Services in the upper left-hand corner. In the list of resources, type Automation. As you begin typing, the list filters based on your input. Select Automation Accounts.

Click +Add. Enter the automation account name, choose the right subscription, resource group, and location, and then click Create.

Import Modules from Gallery

In the next step, you need to import the required modules from the Modules gallery. In your list of Automation Accounts, select the account that you created in the previous step.

From your Automation Account, select Modules under Shared Resources. Click Browse Gallery to open the Browse Gallery page. You need to import the following modules from the Modules gallery in the order given below:

1. Az.Accounts
2. Az.StorageSync

The good news is that starting in September 2021, Automation Accounts will now have the Az modules installed by default. You don’t need to import the modules from the gallery as we used to do in the past. Please note that you can also update the modules to the latest Az version from the modules blade as shown in the figure below.

The most common PowerShell modules are provided by default in each Automation account. See the default modules imported on this page. As the Azure team updates the Azure modules regularly, changes can occur with the included cmdlets.

Create PowerShell Runbook

In this step, you can create multiple Runbooks based on which set of Azure File Shares you want to sync back the changes. PowerShell Runbooks are based on Windows PowerShell. You directly edit the code of the Runbook using the text editor in the Azure portal. You can also use any offline text editor such as Visual Studio Code and import the Runbook into Azure Automation.

In this example, we will create a Runbook to detect and check the files and directories changes in a specific Sync Group Name, and in a specific Cloud Endpoint Name. You can also be creative as much as you want and cover multiple Azure File Shares / Sync Groups / Cloud Endpoints / Directories.

Edit The Runbook

Once you have the Runbook created, you need to edit the Runbook, then write or add the script to choose which Azure File Share you want to detect and sync back the changes. Of course, you can create scripts that suit your environment.

As mentioned earlier, in this example, we will create a Runbook to detect and check the files and directories changes in a specific Sync Group Name / Cloud Endpoint Name and within a particular (file share) directory.

The script is as follows:

<#
.DESCRIPTION
A Runbook example which continuously check for files and directories changes in recursive mode
For a specific Azure File Share in a specific Sync Group / Cloud Endpoint
Using the Managed Identity (Service Principal in Azure AD)

.NOTES
Filename : Enable-ImmediateFileSync
Author   : Charbel Nemnom (Microsoft MVP/MCT)
Version  : 1.4 
Date     : 24-August-2019 
Updated  : 25-July-2023 

.LINK To provide feedback or for further assistance please visit: 
https://charbelnemnom.com
#>
 
Param ( 
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()] 
    [String] $AzureSubscriptionId, 
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $ResourceGroupName, 
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $StorageSyncServiceName, 
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $SyncGroupName, 
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $Path
) 

# Ensures you do not inherit an AzContext in your runbook 
Disable-AzContextAutosave -Scope Process 

# Connect to Azure with system-assigned managed identity (automation account) 
Connect-AzAccount -Identity 

# Set Azure Subscription context
Set-AzContext -SubscriptionId "$AzureSubscriptionId"

#! Get Cloud Endpoint Name
$azsync = Get-AzStorageSyncCloudEndpoint -ResourceGroupName "$ResourceGroupName" `
   -StorageSyncServiceName "$StorageSyncServiceName" -SyncGroupName "$SyncGroupName" 
Write-Output "Get Azure Storage Sync Cloud Endpoint Name: $($azsync.CloudEndpointName)"

#! Invoke-AzStorageSyncChangeDetection
Write-Output "Check for files and directories changes for $StorageSyncServiceName in $SyncGroupName" 
Invoke-AzStorageSyncChangeDetection -ResourceGroupName "$ResourceGroupName" `
  -StorageSyncServiceName "$StorageSyncServiceName" -SyncGroupName "$SyncGroupName" `
  -CloudEndpointName $azsync.CloudEndpointName -DirectoryPath "$Path" -Recursive 

Write-Output ("")

Save the script in the CMDLETS pane as shown in the following screenshot.

Then test the script using “Test Pane” to verify it’s working as intended before you publish it. Once the test is completed, publish the Runbook by clicking Publish.

Schedule Azure File Sync Change Detection

In the final step, you need to schedule the Runbook to run based on your desired time to detect the Azure File Share changes.

Within the same Runbook that you create in the previous step, select Schedules and then click + Add Schedule.

So, if you need to schedule the Runbook to run every hour, then you need to create the following schedule with Recur every 1 Hour with Set expiration to No. You can also run it on-demand if you wish to do so.

While scheduling the Runbook, you can pass on the parameters required for the PowerShell Script. In my example, I need to specify the Azure Subscription ID, Resource Group Name, Storage Sync Service Name, Sync Group Name, and the Directory Name that I want to detect the changes. The sample script takes those parameters as input.

Once done, click OK twice.

Test the Runbook

In this demo, we will test the Runbook by uploading a file directly to Azure File Share, and then we will request on-demand storage sync to detect the changes so the file will sync back to on-premises as well.

This scenario will simulate when the user adds or change files directly in Azure File Share.

Monitor the Runbook

You can monitor the success or failure of these schedules using the “Jobs” tab of Runbooks under Resources. You can also see the next run schedule, in my example, the Runbook will run every hour, and so forth…

That’s it there you have it!

As a side note, running Invoke-AzStorageSyncChangeDetection will work for small data sets. Keep in mind that this will fail after 10K objects. Please note that this limit no longer applies when scanning the entire share. Please check the following section for more details.

This is still version 1.3, if you have any feedback or changes that everyone should receive, please feel free to leave a comment below.

Summary

Azure File Sync extends on-premises file servers into Azure providing cloud benefits while maintaining performance and compatibility. Azure File Sync provides:

• Multi-site access – provide write access to the same data across Windows servers and Azure Files.
• Cloud tiering – stores only recently accessed data on local servers.
• Integrates with Azure backup – no need to back up your data on-premises.
• Fast disaster recovery – restore file metadata immediately and recall data as needed.

I hope you find this guide useful. To learn more about Azure File Sync, please check the following articles.

__
Thank you for reading my blog.

If you have any questions or feedback, please leave a comment.

-Charbel Nemnom-