You dont have javascript enabled! Please enable it! Step-by-Step – Enable Immediate Sync On Azure File Share With Azure File Sync - CHARBEL NEMNOM - MVP | MCT | CCSP | CISM - Cloud & CyberSecurity

Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync

7 Min. Read

Updated – 20/07/2023 This article has been updated to use Azure automation with Managed Identity instead of Azure Automation Run As Account which will retire on September 30, 2023.

Updated – 30/11/2021 Microsoft has improved the Storage Sync Change Detection and the 10,000 item limit no longer applies when scanning the entire share. Please check the following section for more details.

In this article, we will show you how to enable immediate Sync on Azure File Share with Azure File Sync.

Introduction

Azure File Sync can help you centralize your files in Azure and then install a sync agent on Windows Server whether it’s on-premises or in Azure (IaaS VM) to provide fast local access to your files. Your Windows Server and Azure Files are constantly in sync, so you have one centralized location for your files with multi-site access powered by a fast local cache and cloud tiering.

The cloud tiering feature enables frequently accessed files to be cached locally such that the full file content is present on the server, whereas less frequently accessed files are tiered to the cloud. The tiered files (AFS reparse points) will be recalled on-demand when a user or application accesses it on the local server.

For more information about Azure File Sync, please make sure to check my previous articles.

Since the first release of Azure File Sync, one of the most requested features by many customers, was the ability to sync back when the changes occur directly on the Azure File Share. The user’s voice is very active in improving this capability, you can add your vote here.

The good news is, that Microsoft just added a new way to trigger sync that happens on files that are placed directly in the Azure File Share. Normally it can take up to 24 hours before sync will detect changes that happen directly in the Azure File Share. With this new Azure PowerShell cmdlet introduced by the Azure Storage Team, you can point sync to particular files, directories, or sub-directories and have it look for changes, then sync back all the changes.

This new capability is intended for scenarios where some type of automated process in Azure is doing the file edits or migrations done by an administrator (like moving a new directory of files into the file share). You can also use the same approach when a user adds or changes files directly in Azure File Share.

Microsoft recommends that for end-user direct changes on the Azure File Share, to install the Azure File Sync agent in an IaaS VM, and then give the end-user access through that. This way all changes will quickly sync to other agents without the need to call the Azure PowerShell cmdlet, but nothing stops you from using this method to tackle that scenario as well.

In this article, we will show you how to automate the immediate sync process and make sure that all the changes made on the Azure File Share will sync back to your Windows Server whether it’s on-premises or in Azure.

Invoke-AzStorageSyncChangeDetection Improvements

Prior to the Azure File Sync agent version 14 release, if you made changes directly in the Azure file share, you could use the Invoke-AzStorageSyncChangeDetection cmdlet to detect the changes and sync them to the file servers in your sync group. However, the cmdlet would fail to run if the path specified contained more than 10,000 items (objects).

The good news is, that Microsoft has improved the Invoke-AzStorageSyncChangeDetection cmdlet and the 10,000 item limit no longer applies when scanning the entire share.

Please make sure to download Azure File Sync Agent version 14 or later by following the step-by-step guide.

Create an Azure Automation Account

When you create an Automation Account, it creates a new service principal in Azure Active Directory (Azure AD) by default. Using a Managed Identity instead of the Automation Run As account makes management simpler. You don’t have to renew the certificate used by the Automation Run As account. Additionally, you don’t have to specify the Run As connection object in your runbook code. You can access resources using your Automation account’s managed identity from a runbook without creating certificates, connections, Run As accounts, etc.

Please note that Azure Automation Run As Account will retire on September 30, 2023, and will be replaced with Managed Identities.

Next, you must assign the appropriate (Azure RBAC) role to allow access to the storage account and the storage sync service for the Managed Identity at the resource group, subscription, or management group level.

You need to clone the Storage Account Contributor role first, and then add the following custom Microsoft.StorageSync permissions to it for the script to work properly. Always keep in mind to use the principle of least privilege (PoLP) when assigning permissions.

You can do that by going to Access Control (IAM) and selecting Add Custom Role. Next, clone the Storage Account Contributor role.

Create a custom role
Create a custom role

Click Next, then select + Add permissions and add the following permissions to the custom role:

Microsoft.StorageSync/storageSyncServices/syncGroups/cloudEndpoints/read
Microsoft.StorageSync/storageSyncServices/syncGroups/cloudEndpoints/triggerChangeDetection/action
Microsoft.StorageSync/storageSyncServices/syncGroups/cloudEndpoints/operationresults/read
Create Storage Sync Services custom role
Create Storage Sync Services custom role

Next, in the Azure portal, click All Services in the upper left-hand corner. In the list of resources, type Automation. As you begin typing, the list filters based on your input. Select Automation Accounts.

Click +Add. Enter the automation account name, choose the right subscription, resource group, and location, and then click Create.

Create Automation Account with managed identity
Create Automation Account with managed identity

Import Modules from Gallery

In the next step, you need to import the required modules from the Modules gallery. In your list of Automation Accounts, select the account that you created in the previous step.

From your Automation Account, select Modules under Shared Resources. Click Browse Gallery to open the Browse Gallery page. You need to import the following modules from the Modules gallery in the order given below:

  1. Az.Accounts
  2. Az.StorageSync

Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync 1

The good news is that starting in September 2021, Automation Accounts will now have the Az modules installed by default. You don’t need to import the modules from the gallery as we used to do in the past. Please note that you can also update the modules to the latest Az version from the modules blade as shown in the figure below.

Automation Account Update Az Module
Automation Account Update Az Module

The most common PowerShell modules are provided by default in each Automation account. See the default modules imported on this page. As the Azure team updates the Azure modules regularly, changes can occur with the included cmdlets.

Create PowerShell Runbook

In this step, you can create multiple Runbooks based on which set of Azure File Shares you want to sync back the changes. PowerShell Runbooks are based on Windows PowerShell. You directly edit the code of the Runbook using the text editor in the Azure portal. You can also use any offline text editor such as Visual Studio Code and import the Runbook into Azure Automation.

In this example, we will create a Runbook to detect and check the files and directories changes in a specific Sync Group Name, and in a specific Cloud Endpoint Name. You can also be creative as much as you want and cover multiple Azure File Shares / Sync Groups / Cloud Endpoints / Directories.

Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync 2

Edit The Runbook

Once you have the Runbook created, you need to edit the Runbook, then write or add the script to choose which Azure File Share you want to detect and sync back the changes. Of course, you can create scripts that suit your environment.

As mentioned earlier, in this example, we will create a Runbook to detect and check the files and directories changes in a specific Sync Group Name / Cloud Endpoint Name and within a particular (file share) directory.

The script is as follows:

<#
.DESCRIPTION
A Runbook example which continuously check for files and directories changes in recursive mode
For a specific Azure File Share in a specific Sync Group / Cloud Endpoint
Using the Managed Identity (Service Principal in Azure AD)

.NOTES
Filename : Enable-ImmediateFileSync
Author   : Charbel Nemnom (Microsoft MVP/MCT)
Version  : 1.5 
Date     : 24-August-2019 
Updated  : 09-October-2023 

.LINK To provide feedback or for further assistance please visit: 
https://charbelnemnom.com
#>
 
Param ( 
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()] 
    [String] $AzureSubscriptionId, 
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $ResourceGroupName, 
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $StorageSyncServiceName, 
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $SyncGroupName, 
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $Path
) 

# Ensures you do not inherit an AzContext in your runbook 
Disable-AzContextAutosave -Scope Process 

# Connect to Azure with system-assigned managed identity (automation account) 
Connect-AzAccount -Identity 

# Set Azure Subscription context
Set-AzContext -SubscriptionId "$AzureSubscriptionId"

#! Get Cloud Endpoint Name
$azsync = Get-AzStorageSyncCloudEndpoint -ResourceGroupName "$ResourceGroupName" `
  -StorageSyncServiceName "$StorageSyncServiceName" -SyncGroupName "$SyncGroupName"

Write-Output "Get Azure Storage Sync Cloud Endpoint Name: $($azsync.CloudEndpointName)"

#! Invoke-AzStorageSyncChangeDetection
Write-Output "Check for files and directories changes for $StorageSyncServiceName in $SyncGroupName" 
Invoke-AzStorageSyncChangeDetection -ResourceGroupName "$ResourceGroupName" `
  -StorageSyncServiceName "$StorageSyncServiceName" -SyncGroupName "$SyncGroupName" `
  -CloudEndpointName "$($azsync.CloudEndpointName)" -DirectoryPath "$Path" -Recursive 

Write-Output ("")

Save the script in the CMDLETS pane as shown in the following screenshot.

Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync 3

Then test the script using “Test Pane” to verify it’s working as intended before you publish it. Once the test is completed, publish the Runbook by clicking Publish.

Schedule Azure File Sync Change Detection

In the final step, you need to schedule the Runbook to run based on your desired time to detect the Azure File Share changes.

Within the same Runbook that you create in the previous step, select Schedules and then click “+ Add a Schedule“.

So, if you need to schedule the Runbook to run every hour, then you need to create the following schedule with Recur every 1 Hour with Set expiration to No. You can also run it on-demand if you wish to do so.

Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync 4

While scheduling the Runbook, you can pass on the parameters required for the PowerShell Script. In my example, I need to specify the Azure Subscription ID, Resource Group Name, Storage Sync Service Name, Sync Group Name, and the Directory Name that I want to detect the changes. The sample script takes those parameters as input.

Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync 5

Once done, click OK twice.

Test the Runbook

In this demo, we will test the Runbook by uploading a file directly to Azure File Share, and then we will request on-demand storage sync to detect the changes so the file will sync back to on-premises as well.

This scenario will simulate when the user adds or changes files directly in Azure File Share.

Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync 6

Monitor the Runbook

You can monitor the success or failure of these schedules using the “Jobs” tab of Runbooks under Resources. You can also see the next run schedule, in my example, the Runbook will run every hour, and so forth…

Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync 7

That’s it there you have it!

As a side note, running Invoke-AzStorageSyncChangeDetection will work for small data sets. Keep in mind that this will fail after 10K objects. Please note that this limit no longer applies when scanning the entire share. Please check the following section for more details.

This is still version 1.3, if you have any feedback or changes that everyone should receive, please feel free to leave a comment below.

Summary

Azure File Sync extends on-premises file servers into Azure providing cloud benefits while maintaining performance and compatibility. Azure File Sync provides:

  • Multi-site access – provide write access to the same data across Windows servers and Azure Files.
  • Cloud tiering – stores only recently accessed data on local servers.
  • Integrates with Azure backup – no need to back up your data on-premises.
  • Fast disaster recovery – restore file metadata immediately and recall data as needed.

I hope you find this guide useful. To learn more about Azure File Sync, please check the following articles.

__
Thank you for reading my blog.

If you have any questions or feedback, please leave a comment.

-Charbel Nemnom-

Photo of author
About the Author:
Charbel Nemnom
Charbel Nemnom is a Senior Cloud Architect with 21+ years of IT experience. As a Swiss Certified Information Security Manager (ISM), CCSP, CISM, Microsoft MVP, and MCT, he excels in optimizing mission-critical enterprise systems. His extensive practical knowledge spans complex system design, network architecture, business continuity, and cloud security, establishing him as an authoritative and trustworthy expert in the field. Charbel frequently writes about Cloud, Cybersecurity, and IT Certifications.
Previous

Download FREE e-book, and take a chance to Win $20 Amazon Vouchers!

Passed Exam: EXIN Privacy and Data Protection Based on The EU GDPR

Next

43 thoughts on “Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync”

Leave a comment...

  1. In the Path I can specify just one folder of entire share that I synchronise ? It is only working like that. What if I want Invoke synchronisation of entire share with all subfolders ? (whole path in whole endpoint directory that is synchronising not just specific subfolder.

  2. Hello Marcin,
    Thanks for the comment.
    Please note that running Invoke-AzStorageSyncChangeDetection will work for small data sets. Keep in mind that this will fail after 10,000 objects.
    You can target and specify the entire share, subfolder or set of files. However, a maximum of 10,000 items can be detected. If the scope of changes is known to you, I recommend limit the execution of this command to parts of the namespace, so change detection can finish quickly and within the 10,000 item limit.
    Hope that helps!

  3. which path do we use for this? Script completed but nothing happens so i am thinking i mess up with the path

  4. Hello Rocky, thanks for your comment. Please note that the path is the volume+folder (E.g. D:\Data or D:\Data\Marketing) that you want to sync back on-premises. Hope this helps!

  5. Getting the error :
    “Failed The term ‘Connect-AzAccount’ is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.” is not recognized from your script. Any idea on how to fix this?

  6. I did get this figured out by getting the correct modules installed. but now I’m getting the following error when getting the script to complete:
    “Invoke-AzStorageSyncChangeDetection : Long running operation failed with status ‘Failed’. Additional Info:’Change
    detection operation failed.

    Additional information:
    Error code: 0x80C87005 MgmtInternalError
    Error message: Failed to trigger change detection on the replica group ‘(ID)”
    Code: MgmtChangeDetectionOperationError
    Message: Change detection operation failed.

    Please let me know what I can do to solve this? I haven’t been able to find anything on it.

    Thanks!

  7. Hello Wade,
    Please note that running Invoke-AzStorageSyncChangeDetection will work for small data sets. Keep in mind that this will fail after 10,000 objects.
    You can target and specify the entire share, subfolder or set of files. However, a maximum of 10,000 items can be detected that why you got the error noted above. If the scope of changes is known to you, I would recommend limit the execution of this command to a folder(s) instead of the entire share, so change detection can finish quickly and within the 10,000 item limit.
    Hope that helps!

  8. I ended up figuring out the issue. I was getting the error above due to the Path being wrong. You just want to specify the folder on the share, not the whole path of where the folder is located on the server. For example, you want to put just “Files”, not E:\Data\Files.

  9. Hello – I got this deployed, but I can’t see any results. I’ve made sure the path I entered has 10k files. if I enter ssmedia\SoundSamles\Docs, then no errors, but nothing happens. The file I added to the storage account folder isn’t syncd to the on-prem server.

  10. Hello Mark. Please make sure that the path is the volume+folder (E.g. D:\SoundSamples or D:\Data\SoundSamples\Docs) that you want to sync back on-premises. Hope this helps!

  11. Hi Charbel , Thank you for the post . I get the below error , if I pass the the path as volume+folder and if I pass only the folder name then nothing happens , i mean no error however nothing is getting syncing back to local drive.
    arget: At line:62 char:1 + Invoke-AzStorageSyncChangeDetection -ResourceGroupName $ResourceGroup … + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : CloseError: (:) [Invoke-AzStorageSyncChangeDetection], StorageSyncCloudException + FullyQualifiedErrorId : Microsoft.Azure.Commands.StorageSync.CloudEndpoint.InvokeChangeDetectionCommand

  12. Hello Anup, please note that the Invoke-AzStorageSyncChangeDetection cmdlet is only for Azure File Sync deployment. Could you please confirm you are using Azure File Sync?
    The Invoke-AzStorageSyncChangeDetection cmdlet will not detect the following changes in the Azure file share:
    * Files that are deleted.
    * Files that are moved out of the share.
    * Files that are deleted and created with the same name.
    Please note that the above-mentioned 3 changes will be detected when the change detection job runs which is every 24 hours.
    Hope this helps!

  13. Hi Charbel, I just figured it out , the path i was giving was the path of the local folder , when I corrected it with file path of file share , the file got synced to local folder. Yes I am using Azure file sync .
    There is another query is it possible to have the automation less than 1 hour (e.g 15 mins) , I see the minimum reoccurrence is 1 hour.

  14. Hello Anup, yes, the path should be the Azure file share and NOT the local path of the server. Unfortunately, today the minimum query possible is 1 hour, we cannot do it less than that. Stay tuned, there is a new improvement coming!

  15. As Azure run as accounts are being retired in September, do you have an updated script that uses managed identities? Thanks!

  16. Hello PMD, thanks for the comment and feedback!
    Yes, I have updated the script to support Managed Identities instead of Azure Automation Run As Account.
    Give it a try and let me know if it works for you.
    Thanks!

Let us know what you think, or ask a question...