In This Article
In this article, we will show you how to sync between two Azure Blob Storage accounts, how to sync between Azure File Shares, as well as how to sync between Azure Blob and Azure Files. This scenario is very useful if you have an application that is designed to read/write to a single file share and you have another application that is deployed in another Azure subscription that requires access to the same files. In this way, you can sync between Azure File Shares or Blob Storage.
For this scenario, we will make use of the AzCopy tool which is a command-line utility that you can use to copy/sync blobs or files to or from a storage account. If you are new to this tool, then make sure to check the get started document from Microsoft here. The good news is, Microsoft recently added Sync support for AzCopy starting with version 10.3.0 and later.
The prerequisites are very simple as follows:
2) You need to have two different storage accounts either in the same region, the same subscription or in different regions and subscriptions.
3) You also need to create at least one container in each blob storage OR one Azure File Share across two different storage accounts.
4) Download Microsoft Azure Storage Explorer from here if you don’t have it yet, I will use it to create the Shared Access Signature (SAS) tokens. You can also generate SAS tokens using the Azure Portal, as well as using PowerShell. Personally, I prefer to use Azure Storage Explorer to generate SAS tokens.
As a side note, SAS is more secure than the storage account key. One of the main reasons is with SAS, you can ACL the IPs that can access the account, you can control the permissions on the account in a more granular fashion, and when the token will expire, as well as which service you want to have access to (Blobs, Files, Queues, Tables). Please note that SAS tokens are signed with your key, so invalidating your storage account key should invalidate the SAS token as well, keep that in mind!
5) Last, you need to upload some files to the Blob container(s) or Azure file share(s).
Sync Between Azure File Shares
Assuming you have some files in each blob container or file share. Once you are ready, you can take the following steps:
In this example, I have the following files uploaded to the first Azure File Share named (az-fileshare-01), I have 10 files.
In the second Azure File Share named (az-fileshare-02), I have similar files but without the Copy version (5 files).
To sync an entire Azure File Share from one storage account with SAS to another storage account with SAS, you can use the following syntax. This command will go through all the files in the source file share in recursive mode and sync the contents to the destination Azure file share in the second storage account.
azcopy sync "https://[sourceaccount].file.core.windows.net/[Share]?[SAS]" "https://[targetaccount].file.core.windows.net/[Share]?[SAS]" --recursive=true
The output will show the total number of files at the Source, Destination, and the Total Number of Copy Transfers which are 5 files, as well as the time in Minutes.
Open the second Azure file share in the second storage account and verify the files are synced over (total 10 files).
If I run the sync command again, you can see that the source and destination file shares are already in sync (10 files).
Sync Between Azure Blob Storage
You can use the same syntax to sync a blob container as well. To sync an entire blob storage container from one storage account with SAS to another storage account with SAS, you can use the following syntax. This command will go through all the files in the source blob storage container in recursive mode and sync the contents to the destination blob container in the second storage account.
azcopy sync "https://[sourceaccount].blob.core.windows.net/[Container]?[SAS]" "https://[targetaccount].blob.core.windows.net/[Container]?[SAS]" --recursive=true
Last but certainly not least, you can automate and schedule the entire process using Azure Automation Runbook to run/sync for example every 1 hour.
Sync Between Azure Blob and Azure Files
Updated – 01/02/2022 – Starting with the AZCopy version 10.13.0 and later, Microsoft added sync support between Azure Blob <-> Azure Files instead of only copy.
You can use the same syntax to sync a blob container to Azure files and vice versa. To sync a specific container to Azure file share (Azure Blob -> Azure Files) from one storage account with SAS to another storage account, you can use the following syntax.
azcopy sync "https://[sourceaccount].blob.core.windows.net/[container-name]?[SAS]" "https://[targetaccount].file.core.windows.net/[fileshare-name]?[SAS]" --recursive=true
And the reverse is true (Azure Files -> Azure Blob).
azcopy sync "https://[sourceaccount].file.core.windows.net/[fileshare-name]?[SAS]" "https://[targetaccount].blob.core.windows.net/[container-name]?[SAS]" --recursive=true
Please note that if you specify the –delete-destination=true flag is set to true or prompt, then the sync will delete files and blobs at the destination that are not present at the source.
How it works…
Please note that synchronization is one-way. In other words, you choose which of the two endpoints is the source and which one is the destination.
The sync command compares file names and last modified timestamps. You can set the –delete-destination optional flag to a value of “true” or “prompt” to delete files in the destination directory if those files no longer exist in the source directory. If you set the –delete-destination flag to true, AzCopy deletes files without providing a prompt. If you want a prompt to appear before AzCopy deletes a file, then set the –delete-destination flag to “prompt” only.
To prevent accidental deletions, please make sure to enable the soft delete feature on your Azure Storage blobs before you use the –delete-destination=prompt|true flag option.
For more information about AzCopy Sync syntax with additional parameters, please check the following document from Microsoft.
Do you want to learn more about Azure Storage including Azure Blobs and Azure File Shares? Make sure to check my recently published online course here: Azure Storage Essential Training.
That’s it there you have it.
Thank you for reading my blog.
If you have any questions or feedback, please leave a comment.