You dont have javascript enabled! Please enable it! Step-by-Step – Enable Immediate Sync On Azure File Share With Azure File Sync - CHARBEL NEMNOM - MVP | MCT | CCSP | CISM - Cloud & CyberSecurity

Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync

7 Min. Read

Updated – 20/07/2023 This article has been updated to use Azure automation with Managed Identity instead of Azure Automation Run As Account which will retire on September 30, 2023.

Updated – 30/11/2021 Microsoft has improved the Storage Sync Change Detection and the 10,000 item limit no longer applies when scanning the entire share. Please check the following section for more details.

In this article, we will show you how to enable immediate Sync on Azure File Share with Azure File Sync.

Introduction

Azure File Sync can help you centralize your files in Azure and then install a sync agent on Windows Server whether it’s on-premises or in Azure (IaaS VM) to provide fast local access to your files. Your Windows Server and Azure Files are constantly in sync, so you have one centralized location for your files with multi-site access powered by a fast local cache and cloud tiering.

The cloud tiering feature enables frequently accessed files to be cached locally such that the full file content is present on the server, whereas less frequently accessed files are tiered to the cloud. The tiered files (AFS reparse points) will be recalled on-demand when a user or application accesses it on the local server.

For more information about Azure File Sync, please make sure to check my previous articles.

Since the first release of Azure File Sync, one of the most requested features by many customers, was the ability to sync back when the changes occur directly on the Azure File Share. The user’s voice is very active in improving this capability, you can add your vote here.

The good news is, that Microsoft just added a new way to trigger sync that happens on files that are placed directly in the Azure File Share. Normally it can take up to 24 hours before sync will detect changes that happen directly in the Azure File Share. With this new Azure PowerShell cmdlet introduced by the Azure Storage Team, you can point sync to particular files, directories, or sub-directories and have it look for changes, then sync back all the changes.

This new capability is intended for scenarios where some type of automated process in Azure is doing the file edits or migrations done by an administrator (like moving a new directory of files into the file share). You can also use the same approach when a user adds or changes files directly in Azure File Share.

Microsoft recommends that for end-user direct changes on the Azure File Share, to install the Azure File Sync agent in an IaaS VM, and then give the end-user access through that. This way all changes will quickly sync to other agents without the need to call the Azure PowerShell cmdlet, but nothing stops you from using this method to tackle that scenario as well.

In this article, we will show you how to automate the immediate sync process and make sure that all the changes made on the Azure File Share will sync back to your Windows Server whether it’s on-premises or in Azure.

Invoke-AzStorageSyncChangeDetection Improvements

Prior to the Azure File Sync agent version 14 release, if you made changes directly in the Azure file share, you could use the Invoke-AzStorageSyncChangeDetection cmdlet to detect the changes and sync them to the file servers in your sync group. However, the cmdlet would fail to run if the path specified contained more than 10,000 items (objects).

The good news is, that Microsoft has improved the Invoke-AzStorageSyncChangeDetection cmdlet and the 10,000 item limit no longer applies when scanning the entire share.

Please make sure to download Azure File Sync Agent version 14 or later by following the step-by-step guide.

Create an Azure Automation Account

When you create an Automation Account, it creates a new service principal in Azure Active Directory (Azure AD) by default. Using a Managed Identity instead of the Automation Run As account makes management simpler. You don’t have to renew the certificate used by the Automation Run As account. Additionally, you don’t have to specify the Run As connection object in your runbook code. You can access resources using your Automation account’s managed identity from a runbook without creating certificates, connections, Run As accounts, etc.

Please note that Azure Automation Run As Account will retire on September 30, 2023, and will be replaced with Managed Identities.

Next, you must assign the appropriate (Azure RBAC) role to allow access to the storage account and the storage sync service for the Managed Identity at the resource group, subscription, or management group level.

You need to clone the Storage Account Contributor role first, and then add the following custom Microsoft.StorageSync permissions to it for the script to work properly. Always keep in mind to use the principle of least privilege (PoLP) when assigning permissions.

You can do that by going to Access Control (IAM) and selecting Add Custom Role. Next, clone the Storage Account Contributor role.

Create a custom role
Create a custom role

Click Next, then select + Add permissions and add the following permissions to the custom role:

Microsoft.StorageSync/storageSyncServices/syncGroups/cloudEndpoints/read
Microsoft.StorageSync/storageSyncServices/syncGroups/cloudEndpoints/triggerChangeDetection/action
Microsoft.StorageSync/storageSyncServices/syncGroups/cloudEndpoints/operationresults/read
Create Storage Sync Services custom role
Create Storage Sync Services custom role

Next, in the Azure portal, click All Services in the upper left-hand corner. In the list of resources, type Automation. As you begin typing, the list filters based on your input. Select Automation Accounts.

Click +Add. Enter the automation account name, choose the right subscription, resource group, and location, and then click Create.

Create Automation Account with managed identity
Create Automation Account with managed identity

Import Modules from Gallery

In the next step, you need to import the required modules from the Modules gallery. In your list of Automation Accounts, select the account that you created in the previous step.

From your Automation Account, select Modules under Shared Resources. Click Browse Gallery to open the Browse Gallery page. You need to import the following modules from the Modules gallery in the order given below:

  1. Az.Accounts
  2. Az.StorageSync

Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync 1

The good news is that starting in September 2021, Automation Accounts will now have the Az modules installed by default. You don’t need to import the modules from the gallery as we used to do in the past. Please note that you can also update the modules to the latest Az version from the modules blade as shown in the figure below.

Automation Account Update Az Module
Automation Account Update Az Module

The most common PowerShell modules are provided by default in each Automation account. See the default modules imported on this page. As the Azure team updates the Azure modules regularly, changes can occur with the included cmdlets.

Create PowerShell Runbook

In this step, you can create multiple Runbooks based on which set of Azure File Shares you want to sync back the changes. PowerShell Runbooks are based on Windows PowerShell. You directly edit the code of the Runbook using the text editor in the Azure portal. You can also use any offline text editor such as Visual Studio Code and import the Runbook into Azure Automation.

In this example, we will create a Runbook to detect and check the files and directories changes in a specific Sync Group Name, and in a specific Cloud Endpoint Name. You can also be creative as much as you want and cover multiple Azure File Shares / Sync Groups / Cloud Endpoints / Directories.

Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync 2

Edit The Runbook

Once you have the Runbook created, you need to edit the Runbook, then write or add the script to choose which Azure File Share you want to detect and sync back the changes. Of course, you can create scripts that suit your environment.

As mentioned earlier, in this example, we will create a Runbook to detect and check the files and directories changes in a specific Sync Group Name / Cloud Endpoint Name and within a particular (file share) directory.

The script is as follows:

<#
.DESCRIPTION
A Runbook example which continuously check for files and directories changes in recursive mode
For a specific Azure File Share in a specific Sync Group / Cloud Endpoint
Using the Managed Identity (Service Principal in Azure AD)

.NOTES
Filename : Enable-ImmediateFileSync
Author   : Charbel Nemnom (Microsoft MVP/MCT)
Version  : 1.5 
Date     : 24-August-2019 
Updated  : 09-October-2023 

.LINK To provide feedback or for further assistance please visit: 
https://charbelnemnom.com
#>
 
Param ( 
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()] 
    [String] $AzureSubscriptionId, 
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $ResourceGroupName, 
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $StorageSyncServiceName, 
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $SyncGroupName, 
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $Path
) 

# Ensures you do not inherit an AzContext in your runbook 
Disable-AzContextAutosave -Scope Process 

# Connect to Azure with system-assigned managed identity (automation account) 
Connect-AzAccount -Identity 

# Set Azure Subscription context
Set-AzContext -SubscriptionId "$AzureSubscriptionId"

#! Get Cloud Endpoint Name
$azsync = Get-AzStorageSyncCloudEndpoint -ResourceGroupName "$ResourceGroupName" `
  -StorageSyncServiceName "$StorageSyncServiceName" -SyncGroupName "$SyncGroupName"

Write-Output "Get Azure Storage Sync Cloud Endpoint Name: $($azsync.CloudEndpointName)"

#! Invoke-AzStorageSyncChangeDetection
Write-Output "Check for files and directories changes for $StorageSyncServiceName in $SyncGroupName" 
Invoke-AzStorageSyncChangeDetection -ResourceGroupName "$ResourceGroupName" `
  -StorageSyncServiceName "$StorageSyncServiceName" -SyncGroupName "$SyncGroupName" `
  -CloudEndpointName "$($azsync.CloudEndpointName)" -DirectoryPath "$Path" -Recursive 

Write-Output ("")

Save the script in the CMDLETS pane as shown in the following screenshot.

Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync 3

Then test the script using “Test Pane” to verify it’s working as intended before you publish it. Once the test is completed, publish the Runbook by clicking Publish.

Schedule Azure File Sync Change Detection

In the final step, you need to schedule the Runbook to run based on your desired time to detect the Azure File Share changes.

Within the same Runbook that you create in the previous step, select Schedules and then click “+ Add a Schedule“.

So, if you need to schedule the Runbook to run every hour, then you need to create the following schedule with Recur every 1 Hour with Set expiration to No. You can also run it on-demand if you wish to do so.

Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync 4

While scheduling the Runbook, you can pass on the parameters required for the PowerShell Script. In my example, I need to specify the Azure Subscription ID, Resource Group Name, Storage Sync Service Name, Sync Group Name, and the Directory Name that I want to detect the changes. The sample script takes those parameters as input.

Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync 5

Once done, click OK twice.

Test the Runbook

In this demo, we will test the Runbook by uploading a file directly to Azure File Share, and then we will request on-demand storage sync to detect the changes so the file will sync back to on-premises as well.

This scenario will simulate when the user adds or changes files directly in Azure File Share.

Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync 6

Monitor the Runbook

You can monitor the success or failure of these schedules using the “Jobs” tab of Runbooks under Resources. You can also see the next run schedule, in my example, the Runbook will run every hour, and so forth…

Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync 7

That’s it there you have it!

As a side note, running Invoke-AzStorageSyncChangeDetection will work for small data sets. Keep in mind that this will fail after 10K objects. Please note that this limit no longer applies when scanning the entire share. Please check the following section for more details.

This is still version 1.3, if you have any feedback or changes that everyone should receive, please feel free to leave a comment below.

Summary

Azure File Sync extends on-premises file servers into Azure providing cloud benefits while maintaining performance and compatibility. Azure File Sync provides:

  • Multi-site access – provide write access to the same data across Windows servers and Azure Files.
  • Cloud tiering – stores only recently accessed data on local servers.
  • Integrates with Azure backup – no need to back up your data on-premises.
  • Fast disaster recovery – restore file metadata immediately and recall data as needed.

I hope you find this guide useful. To learn more about Azure File Sync, please check the following articles.

__
Thank you for reading my blog.

If you have any questions or feedback, please leave a comment.

-Charbel Nemnom-

Photo of author
About the Author
Charbel Nemnom
Charbel Nemnom is a Senior Cloud Architect with 21+ years of IT experience. As a Swiss Certified Information Security Manager (ISM), CCSP, CISM, Microsoft MVP, and MCT, he excels in optimizing mission-critical enterprise systems. His extensive practical knowledge spans complex system design, network architecture, business continuity, and cloud security, establishing him as an authoritative and trustworthy expert in the field. Charbel frequently writes about Cloud, Cybersecurity, and IT Certifications.
Previous

Download FREE e-book, and take a chance to Win $20 Amazon Vouchers!

Passed Exam: EXIN Privacy and Data Protection Based on The EU GDPR

Next

43 thoughts on “Step-by-Step – Enable Immediate Sync on Azure File Share with Azure File Sync”

Leave a comment...

  1. Thanks again. Script is now fixed. Although I cannot get it to work. I’m sure everything else is configured correctly:

    “The client ‘xxx’ with object id ‘xxx’ does not have authorization to perform action ‘Microsoft.StorageSync/storageSyncServices/syncGroups/cloudEndpoints/read’ over scope ‘/subscriptions/xxx/resourceGroups/Wob/providers/Microsoft.StorageSync/storageSyncServices/wobsfilesync/syncGroups/wobsfilesync_group01’ or the scope is invalid. If access was recently granted, please refresh your credentials.”

    (xxx = IDs etc edited out!)

  2. Hello PMD, thanks for confirming!
    The error that you are facing now is related to the Managed Identity permission.
    Have you assigned the Managed Identity (Service Principal) of the Automation Account name at the subscription or resource group level where the Storage Account and the Storage Sync Services are created?
    Did you create a new Automation Account or you used an existing one?
    For testing purposes, you could assign the RBAC Storage Account Contributor role first, and once the script is working, you can lower the permissions.
    Thanks!

  3. Ah, missed this bit, this is where it’s failing, I think.

    “Get Azure StorageSync Cloud Endpoint Name:
    Operation returned an invalid status code ‘NotFound'”

  4. Hello PMD, I’ve updated the script to get the correct Azure Storage Sync Cloud Endpoint Name.
    Give it a try.

  5. Hi. I still get an error. Still looks like it’s failing when trying to get the Cloud Endpoint Name

    “Get Azure Storage Sync Cloud Endpoint Name:

    Check for files and directories changes for XXX in XXX
    System.Management.Automation.ParameterBindingValidationException: Cannot validate argument on parameter ‘Name’. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again. —> System.Management.Automation.ValidationMetadataException: The argument is null or empty. Provide an argument that is not null or empty, and then try the command again.”

  6. Hello PMD, do you have spaces in your “Storage Sync Service” name?
    I have updated the script to handle spaces in the Storage Sync Service and Sync Group names.
    Give it a try again. Thanks!

  7. I do not have spaces, the updated script still gives the same error. The only non alphanumeric I have is an underscore in my Sync Group Name. No spaces or special characters other than that.

  8. Hello PMD, thanks for confirming!
    Please note that I have just tried the entire workflow in my environment and it’s working without any issue.
    I am sorry, but I cannot troubleshoot further this issue in the comment section.
    There could be many reasons for this to happen which is specific to your environment.
    I would say revise all the steps again and make sure the PowerShell code is correct.
    If you are interested in working on this together, please feel free to reach out using this form.
    Thank you for understanding!

  9. Hello, I have found the issue PDM above has (had?) is due to permissions.
    After some troubleshooting I noticed, regardless of scope (subscription, resource group, etc.) the Storage Account Contributor role lacks these Microsoft.StorageSync permissions needed to properly execute this script:
    Microsoft.StorageSync/storageSyncServices/syncGroups/cloudEndpoints/read Microsoft.StorageSync/storageSyncServices/syncGroups/cloudEndpoints/triggerChangeDetection/action Microsoft.StorageSync/storageSyncServices/syncGroups/cloudEndpoints/operationresults/read
    You can create a custom role by cloning the Storage Account Contributor and add the above permissions to keep things PoLP.

  10. Hello MM, thanks for the comment and for sharing your experience here.
    Yes, I strongly recommend creating custom role dedicated for the Storage Sync Service to follow the principle of least privilege (PoLP) when assigning permissions.
    Thanks!

  11. Hi Charbel,

    Get the error message when I test the script:
    Invoke-AzStorageSyncChangeDetection: C:\Temp\x1oln1om.2yl\c8c44e30-b4e8-492d-ba39-709815f70c12.ps1:49

    Line |
    49 | -CloudEndpointName $azsync.CloudEndpointName -DirectoryPath “$Path” – …
    | ~~~~~~~~~~~~~~~~~~~~~~~~~
    | Cannot validate argument on parameter ‘Name’. The argument is null or
    | empty. Provide an argument that is not null or empty, and then try the
    | command again.

    can you assist please?

  12. Hello Wyne, thanks for the comment!
    I have made a slight change to the script.
    Could you please get a new copy and try again?
    Please make sure that you provide the “-DirectoryPath” parameter correctly.
    Hope it helps!

  13. Charbel,

    The slight change you made to the script, worked. no error message.
    The only problem is when I test the runbook by uploading files to an existing folder and create a folder with files or make changes to a file, and I request on-demand storage sync to detect changes so the files or folders sync back town to the server it does not detect no changes. Have you seen this issue?

    Thank you.

  14. Hello Wyne, it’s good to hear that the error message is gone now!
    As for the behavior that you are seeing, when you request on-demand storage sync to detect changes, you need to make sure that you pass the correct “folder Name” of the on-premises (connected) file server to the “-DirectoryPath” parameter that you want to detect the changes for and sync back.
    Please note that you can also target the entire share, subfolder or set of files. If the scope of changes is known to you, limit the execution of this command to parts of the namespace, so individual item change detection can finish quickly and within the 10,000 items limit. Otherwise, run the “Invoke-AzStorageSyncChangeDetection” command without the “-DirectoryPath” or “-Path” parameters to invoke full share-level change detection.
    Hope it helps!

  15. Hi Charbel,
    Full share-level change detection worked great.
    Still issue with on-demand sync with specific folder from the share.
    In the Directory Path I entered the specific folder name. example I entered “550 Madison Ave” script run with detecting any changes. I tried the full path “H:\D\LZ DATA\DATA\Drive\550 Madison Ave” script run fine but detect no changes. Any suggestions

  16. Hello Wyne, please note that if you run “Invoke-AzStorageSyncChangeDetection” with “-DirectoryPath” or “-Path” parameters, the command will not detect the following changes in the Azure file share:
    1) Files that are deleted.
    2) Files that are moved out of the share.
    3) Files that are deleted and created with the same name.
    Please confirm if you fall in any of the above restrictions?
    However, if share-level change detection is invoked, all of these changes will be detected as you confirmed.
    Here is an example on how to detect changes in the “Data” and “Reporting\Templates” directories of a syncing Azure file share. All paths are relative to the root of the Azure file share namespace.

    Invoke-AzStorageSyncChangeDetection -ResourceGroupName "ResourceGroupName" `
     -StorageSyncServiceName "StorageSyncServiceName" -SyncGroupName "SyncGroupName" `
     -CloudEndpointName "CloudEndpointName" -Path "Data","Reporting\Templates"

    Hope it helps!

  17. Hi Charbel,
    You have done us a great service with this, thank you!

    I already have AFS running and want to automate it away from the 24-hour default.
    Is it possible to have your tweaked runbook script, please? Plus it is currently protected.

    Kind Regards,
    Mark

Let us know what you think, or ask a question...