How to Enable Immediate Sync on Azure File Share with Azure File Sync

6 Min. Read

Updated – 30/11/2021 Microsoft has improved the Storage Sync Change Detection and the 10,000 item limit no longer applies when scanning the entire share. Please check the following section for more details.

In this article, I will show you how to enable immediate Sync on Azure File Share with Azure File Sync.

Introduction

Azure File Sync can help you centralize your files in Azure and then install a sync agent on Windows Server whether it’s on-premises or in Azure (IaaS VM) to provide fast local access to your files. Your Windows Server and Azure Files are constantly in sync, so you have one centralized location for your files with multi-site access powered by a fast local cache and cloud tiering.

The cloud tiering feature enables frequently accessed files to be cached locally such that the full file content is present on the server, whereas less frequently accessed files are tiered to the cloud. The tiered files (AFS reparse points) will be recalled on-demand when a user or application accesses it on the local server.

For more information about Azure File Sync, please make sure to check my previous articles.

Since the first release of Azure File Sync, one of the most requested features by many customers, was the ability to sync back when the changes occur directly on the Azure File Share. The user’s voice is very active to improve this capability, you can add your vote here.

The good news is, Microsoft just added a new way to trigger sync that happens on files that are placed directly in the Azure File Share. Normally it can take up to 24 hours before sync will detect changes that happen directly in the Azure File Share. With this new Azure PowerShell cmdlet introduced by the Azure Storage Team, you can point sync to particular files, directories, or sub-directories and have it look for changes, then sync back all the changes.

This new capability is intended for scenarios where some type of automated process in Azure is doing the file edits or migrations are done by an administrator (like moving a new directory of files into the file share). You can also use the same approach when a user adds or change files directly in Azure File Share.

Microsoft recommends that for end-user direct changes on the Azure File Share, is to install Azure File Sync agent in an IaaS VM, and then give the end-user access through that. This way all changes will quickly sync to other agents without the need to call the Azure PowerShell cmdlet, but nothing stops you from using this method to tackle that scenario as well.

In this article, I will show you how to automate the immediate sync process and make sure that all the changes made on the Azure File Share will sync back to your Windows Server whether it’s on-premises or in Azure.

Invoke-AzStorageSyncChangeDetection Improvements

Prior to the Azure File Sync agent version 14 release, if you made changes directly in the Azure file share, you could use the Invoke-AzStorageSyncChangeDetection cmdlet to detect the changes and sync them to the file servers in your sync group. However, the cmdlet would fail to run if the path specified contained more than 10,000 items (objects).

The good news is, Microsoft has improved the Invoke-AzStorageSyncChangeDetection cmdlet and the 10,000 item limit no longer applies when scanning the entire share.

Please make sure to download Azure File Sync agent version 14 or later by following the step-by-step guide.

Create an Azure Automation Account

First, I need to create an Azure automation resource with a Run As account. Run As accounts in Azure Automation are used to provide authentication for managing resources in Azure with the Azure cmdlets. When you create a Run As account, it creates a new service principal user in Azure Active Directory and assigns the Contributor role to this user at the subscription level.

Open the Azure portal, click All services found in the upper left-hand corner. In the list of resources, type Automation. As you begin typing, the list filters based on your input. Select Automation Accounts.

Click +Add. Enter the automation account name, choose the right subscription, resource group, location, and then click Create.

How to Enable Immediate Sync on Azure File Share with Azure File Sync 1

Import Modules from Gallery

In the next step, you need to import the required modules from the Modules gallery. In your list of Automation Accounts, select the account that you created in the previous step.

From your Automation Account, select Modules under Shared Resources. Click Browse Gallery to open the Browse Gallery page. You need to import the following modules from the Modules gallery in the order given below:

  1. Az.Accounts
  2. Az.StorageSync

How to Enable Immediate Sync on Azure File Share with Azure File Sync 2

Create PowerShell Runbook

In this step, you can create multiple Runbooks based on which set of Azure File Shares you want to sync back the changes. PowerShell Runbooks are based on Windows PowerShell. You directly edit the code of the Runbook using the text editor in the Azure portal. You can also use any offline text editor such as Visual Studio Code and import the Runbook into Azure Automation.

In this example, I will create a Runbook to detect and check the files and directories changes in a specific Sync Group Name, and in a specific Cloud Endpoint Name. You can also be creative as much as you want and cover multiple Azure File Shares / Sync Groups / Cloud Endpoints / Directories.

How to Enable Immediate Sync on Azure File Share with Azure File Sync 3

Edit The Runbook

Once you have the Runbook created, you need to edit the Runbook, then write or add the script to choose which Azure File Share you want to detect and sync back the changes. Of course, you can create scripts that suit your environment.

As mentioned earlier, in this example, I will create a Runbook to detect and check the files and directories changes in a specific Sync Group Name / Cloud Endpoint Name and in a specific directory.

The script is as follows:

<#
.DESCRIPTION
A Runbook example which continuously check for files and directories changes in recursive mode
For a specific Azure File Share in a specific Sync Group / Cloud Endpoint
Using the Run As Account (Service Principal in Azure AD)

.NOTES
Filename : Enable-ImmediateFileSync
Author   : Charbel Nemnom
Version  : 1.2
Date     : 24-August-2019
Updated  : 30-October-2021

.LINK
To provide feedback or for further assistance please visit:
https://charbelnemnom.com 
#>

Param (
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $AzureSubscriptionId,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $ResourceGroupName,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $StorageSyncServiceName,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $SyncGroupName,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $Path
)

$connectionName = "AzureRunAsConnection"

Try {
    #! Get the connection "AzureRunAsConnection "
    $servicePrincipalConnection = Get-AutomationConnection -Name $connectionName
    Write-Output "Logging in to Azure..."
    Connect-AzAccount -ServicePrincipal `
        -TenantId $servicePrincipalConnection.TenantId `
        -ApplicationId $servicePrincipalConnection.ApplicationId `
        -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint
}
Catch {
    If (!$servicePrincipalConnection) {
        $ErrorMessage = "Connection $connectionName not found..."
        throw $ErrorMessage
    }
    Else {
        Write-Error -Message $_.Exception
        throw $_.Exception
    }
}

Select-AzSubscription -SubscriptionId $AzureSubscriptionId

#! Get Cloud Endpoint Name
$azsync = Get-AzStorageSyncCloudEndpoint -ResourceGroupName "$ResourceGroupName" -StorageSyncServiceName $StorageSyncServiceName `
    -SyncGroupName "$SyncGroupName"
Write-Output "Get Azure StorageSync Cloud Endpoint Name: $($azsync.CloudEndpointName)"

#! Invoke-AzStorageSyncChangeDetection
Write-Output "Check for files and directories changes for $StorageSyncServiceName in $SyncGroupName"
Invoke-AzStorageSyncChangeDetection -ResourceGroupName $ResourceGroupName -StorageSyncServiceName $StorageSyncServiceName `
    -SyncGroupName $SyncGroupName -CloudEndpointName $azsync.CloudEndpointName -DirectoryPath $Path -Recursive

Write-Output ("")

Save the script in the CMDLETS pane as shown in the following screenshot.

How to Enable Immediate Sync on Azure File Share with Azure File Sync 4

Then test the script using “Test Pane” to verify it’s working as intended before you publish it. Once the test is completed, publish the Runbook by clicking Publish.

Schedule Azure File Sync Change Detection

In the final step, you need to schedule the Runbook to run based on your desired time to detect the Azure File Share changes.

Within the same Runbook that you create in the previous step, select Schedules and then click + Add schedule.

So, if you need to schedule the Runbook to run every hour, then you need to create the following schedule with Recur every 1 Hour with Set expiration to No. You can also run it on-demand if you wish to do so.

How to Enable Immediate Sync on Azure File Share with Azure File Sync 5

While scheduling the Runbook, you can pass on the parameters required for the PowerShell Script. In my example, I need to specify the Azure Subscription ID, Resource Group Name, Storage Sync Service Name, Sync Group Name, and the Directory Name that I want to detect the changes. The sample script takes those parameters as input.

How to Enable Immediate Sync on Azure File Share with Azure File Sync 6

Once done, click OK.

Test the Runbook

In this demo, I will test the Runbook by uploading a file directly to Azure File Share, and then I will request on-demand storage sync to detect the changes so the file will sync back to on-premises as well. This scenario will simulate when the user adds or change files directly in Azure File Share.

How to Enable Immediate Sync on Azure File Share with Azure File Sync 7

Monitor the Runbook

You can monitor the success or failure of these schedules using the “Jobs” tab of Runbooks under Resources. You can also see the next run schedule, in my example, the Runbook will run every hour, and so forth…

How to Enable Immediate Sync on Azure File Share with Azure File Sync 8

That’s it there you have it!

As a side note, running Invoke-AzStorageSyncChangeDetection will work for small data sets. Keep in mind that this will fail after 10K objects. Please note that this limit no longer applies when scanning the entire share. Please check the following section for more details.

This is still version 1.2, if you have any feedback or changes that everyone should receive, please feel free to leave a comment below.

Summary

Azure File Sync extends on-premises file servers into Azure providing cloud benefits while maintaining performance and compatibility. Azure File Sync provides:

  • Multi-site access – provide write access to the same data across Windows servers and Azure Files.
  • Cloud tiering – store only recently accessed data on local servers.
  • Integrates with Azure backup – no need to back up your data on-premises.
  • Fast disaster recovery – restore file metadata immediately and recall data as needed.

I hope you find this guide useful. To learn more about Azure File Sync, please check the following articles.

__
Thank you for reading my blog.

If you have any questions or feedback, please leave a comment.

-Charbel Nemnom-

Related Posts

Previous

Download FREE e-book, and take a chance to Win $20 Amazon Vouchers!

Passed Exam: EXIN Privacy and Data Protection Based on The EU GDPR

Next

17 thoughts on “How to Enable Immediate Sync on Azure File Share with Azure File Sync”

Leave a comment...

  1. In the Path I can specify just one folder of entire share that I synchronise ? It is only working like that. What if I want Invoke synchronisation of entire share with all subfolders ? (whole path in whole endpoint directory that is synchronising not just specific subfolder.

  2. Hello Marcin,
    Thanks for the comment.
    Please note that running Invoke-AzStorageSyncChangeDetection will work for small data sets. Keep in mind that this will fail after 10,000 objects.
    You can target and specify the entire share, subfolder or set of files. However, a maximum of 10,000 items can be detected. If the scope of changes is known to you, I recommend limit the execution of this command to parts of the namespace, so change detection can finish quickly and within the 10,000 item limit.
    Hope that helps!

  3. which path do we use for this? Script completed but nothing happens so i am thinking i mess up with the path

  4. Hello Rocky, thanks for your comment. Please note that the path is the volume+folder (E.g. D:\Data or D:\Data\Marketing) that you want to sync back on-premises. Hope this helps!

  5. Getting the error :
    “Failed The term ‘Connect-AzAccount’ is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.” is not recognized from your script. Any idea on how to fix this?

  6. I did get this figured out by getting the correct modules installed. but now I’m getting the following error when getting the script to complete:
    “Invoke-AzStorageSyncChangeDetection : Long running operation failed with status ‘Failed’. Additional Info:’Change
    detection operation failed.

    Additional information:
    Error code: 0x80C87005 MgmtInternalError
    Error message: Failed to trigger change detection on the replica group ‘(ID)”
    Code: MgmtChangeDetectionOperationError
    Message: Change detection operation failed.

    Please let me know what I can do to solve this? I haven’t been able to find anything on it.

    Thanks!

  7. Hello Wade,
    Please note that running Invoke-AzStorageSyncChangeDetection will work for small data sets. Keep in mind that this will fail after 10,000 objects.
    You can target and specify the entire share, subfolder or set of files. However, a maximum of 10,000 items can be detected that why you got the error noted above. If the scope of changes is known to you, I would recommend limit the execution of this command to a folder(s) instead of the entire share, so change detection can finish quickly and within the 10,000 item limit.
    Hope that helps!

  8. I ended up figuring out the issue. I was getting the error above due to the Path being wrong. You just want to specify the folder on the share, not the whole path of where the folder is located on the server. For example, you want to put just “Files”, not E:\Data\Files.

  9. Hello – I got this deployed, but I can’t see any results. I’ve made sure the path I entered has 10k files. if I enter ssmedia\SoundSamles\Docs, then no errors, but nothing happens. The file I added to the storage account folder isn’t syncd to the on-prem server.

  10. Hello Mark. Please make sure that the path is the volume+folder (E.g. D:\SoundSamples or D:\Data\SoundSamples\Docs) that you want to sync back on-premises. Hope this helps!

  11. Hi Charbel , Thank you for the post . I get the below error , if I pass the the path as volume+folder and if I pass only the folder name then nothing happens , i mean no error however nothing is getting syncing back to local drive.
    arget: At line:62 char:1 + Invoke-AzStorageSyncChangeDetection -ResourceGroupName $ResourceGroup … + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : CloseError: (:) [Invoke-AzStorageSyncChangeDetection], StorageSyncCloudException + FullyQualifiedErrorId : Microsoft.Azure.Commands.StorageSync.CloudEndpoint.InvokeChangeDetectionCommand

  12. Hello Anup, please note that the Invoke-AzStorageSyncChangeDetection cmdlet is only for Azure File Sync deployment. Could you please confirm you are using Azure File Sync?
    The Invoke-AzStorageSyncChangeDetection cmdlet will not detect the following changes in the Azure file share:
    * Files that are deleted.
    * Files that are moved out of the share.
    * Files that are deleted and created with the same name.
    Please note that the above-mentioned 3 changes will be detected when the change detection job runs which is every 24 hours.
    Hope this helps!

  13. Hi Charbel, I just figured it out , the path i was giving was the path of the local folder , when I corrected it with file path of file share , the file got synced to local folder. Yes I am using Azure file sync .
    There is another query is it possible to have the automation less than 1 hour (e.g 15 mins) , I see the minimum reoccurrence is 1 hour.

  14. Hello Anup, yes, the path should be the Azure file share and NOT the local path of the server. Unfortunately, today the minimum query possible is 1 hour, we cannot do it less than that. Stay tuned, there is a new improvement coming!

Let me know what you think, or ask a question...

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe to Stay in Touch

Never miss out on your favorite posts and our latest announcements!

The content of this website is copyrighted from being plagiarized!

You can copy from the 'Code Blocks' in 'Black' by selecting the Code.

Please send your feedback to the author using this form for any 'Code' you like.

Thank you for visiting!