Migrating data can result in a lot of downtime, and losing file and folder permissions can be a hassle as it requires remapping everything. It’s important to grant access to others before panicking and hiding in your IT room.
If you have an SMB share on a NAS, Linux, or Windows server, you can now migrate it to Azure file shares using Azure Storage Mover, a convenient managed migration service. Our comprehensive guide will walk you through a new way to move your on-premises SMB and NFS file shares to Azure file shares using Azure Storage Mover.
Table of Contents
Introduction
In April 2023, Microsoft announced Azure Storage Mover is generally available (GA). This means you can use this service to migrate your files and folders to Azure Storage with confidence and ease.
Azure Storage Mover is a fully managed migration service that handles all the complexities of moving your data to the cloud. You can choose from different migration options, such as online, offline, or hybrid, depending on your needs and preferences. You can also monitor and control the migration process through a simple user interface or a REST API. Azure Storage Mover helps you reduce the downtime and risk of migrating your data to Azure Storage.
In August 2023, Microsoft announced Azure Storage Mover support for SMB and Azure Files (public preview) that lets you migrate SMB shares to Azure file shares with full fidelity and opens various migration scenarios:
1) You can migrate your NAS share to an Azure file share, then use the Azure file share or hook up Azure File Sync.
2) You can even leverage Azure Storage Mover for Data Box catch-up copies. If you’re struggling with limited bandwidth, Data Box can help with the initial bulk migration. Then, you can use Storage Mover to keep your share up-to-date with the latest changes before transitioning your workload or adding Azure File Sync.
Prerequisites
To follow this article, you need to have the following:
1) Azure subscription – If you don’t have an Azure subscription, you can create a free one here.
2) On-premises file share or Network Attached Storage (NAS) so we can use it as the source to migrate.
3) You need to have at least one storage account and one Azure file share to be used as a target. To create a General Purpose v2 (GPv2) storage account, you can follow the instructions described here.
Related: Check Azure Storage Essential Training.
4) Azure Storage Mover resource (more details in the next section).
5) Azure Storage Mover agent VM (more details in the next section). Storage Mover is going to use agents to get read access to your data and then copy all of that to Azure file shares using AzCopy (like Robocopy but for the cloud). So how many agents are you going to need? Well, that depends on how many sources you have and how many of them you want to move at the same time.
For best performance, you need to have two points as close together as possible. So, if you have five different file servers in five different locations and you want to collapse all of those into one file share in Azure as fast as possible, then you’ll need at least one agent for every file server.
But what if one of those file servers has hundreds of millions of files and you need to get that done faster? Now there’s no explicit limit on the number of files that a single agent can move, so it may take a while to process that 100 million files, but one agent will get the job done.
To increase the speed further, consider allocating additional CPU and RAM to the agent. Alternatively, you may set up multiple agents in the same location, targeting specific sub-folders for a divide-and-conquer approach.
If you want it to go even faster, you could give that agent more CPU and RAM, or you could set up multiple agents in that one location, focusing on sub-folders, and then you kind of divide and conquer.
Now, if you have a huge data set, but a really slow Internet connection. Well, then you can use the Azure data box service to copy that data and ship it over to Microsoft, who will then upload that data into your storage account.
Then you can use the Storage Mover to sync up the final delta changes and complete your migration project.
6) Outbound network connectivity. The Azure Storage Mover appliance agent VM should have access to the following URIs:
https://microsoft.comhttps://xdmpingaccount1.blob.core.windows.nethttps://management.azure.comhttps://login.microsoftonline.comhttps://login.windows.nethttps://gbl.his.arc.azure.comhttps://pas.windows.nethttps://guestnotificationservice.azure.comhttps://agentserviceapi.guestconfiguration.azure.comhttps://portal.azure.comhttps://dc.services.visualstudio.com
7) For SMB shares migration projects, you need an Azure Key Vault where you can store the on-premises SMB credentials (username and password) as secrets. You can use an existing Key Vault or create a new one. You can follow the quick start guide to create a Key Vault using the Azure portal.
Creating secrets is going to be a manual operation, you need two of them, one for the username and one for the password as shown in the figure below. The secret value(s) is the name of the actual user and the password who has ONLY read access to your on-premises file share.
You also need to set the Azure role-based access control (RBAC) permissions on the Key Vault under Settings > Access configuration, and then give the “Key Vault Secrets User” role access to the Storage Mover agent VM name (more details in the next section), so the appliance can retrieve and read secrets from the Key Vault.
Assuming you have all the prerequisites in place, let’s see how to create and deploy a new Azure Storage Mover service.
Create Storage Mover Resource
Jump to the Azure portal and take the following steps:
1) From the left menu blade in the Azure portal, click Create a resource to build your new resource and search for Azure Storage Mover.
2) Next, select your subscription and resource group as always, and then give it a name and select your region. The region you select here only determines where control messages are sent and metadata about your migration is stored. The data that is migrated, is sent directly from the agent to the target in Azure Storage.
A Storage Mover can be deployed to any region without affecting the migration speed. It does not need to be co-located with any other Azure resources to perform migrations. Click Next > to continue.
3) Then select your Log Analytics workspace for monitoring. This is going to keep track of all of your data copy logs, which is very important. Click Next > to continue.
4) Add your Tags like you usually do for all of your cloud resources to track things like the cost center, the owner, and how to get in touch with your team.
5) Click Next > to continue, and then click Review + Create to create a Storage Mover resource.
Deploy Storage Mover Agent VM
Now every migration process has five basic steps (Discover, Assess, Review, Plan, and Migrate). In this step, we will start with the discovery phase and that’s with deploying Storage Mover agents.
As mentioned earlier, Azure Storage Mover is going to use agents to get read access to your data and then copy all of that to your cloud.
Once the Storage Mover resource is provisioned, you need to go to Registered agents under Resource management. The first step is to download the agent VM from Microsoft Download Center. Agents are on-premises VMs that establish a secure connection between your source files and Azure. They also provide the computational processing to run your migration jobs.
The agent will come in as a virtual hard drive (VHDX) file that you can set up in Hyper-V today. The VMware images will be released very soon. Stay Tuned!
The agent is a Linux-based VM appliance and the requirements are 4 vCPUs and eight GB of RAM with 20 disks that will handle all of the copy processing.
Now once you have the file downloaded, extract your virtual hard drive (VHDX), then you need to open Hyper-V and create a new virtual machine. To speed up the process, you can run the following PowerShell commands to create the VM in Hyper-V with the right settings. At the time of this writing, we are using Azure Storage Mover agent VM version 2.0.287.
# Define Variables
$VMName = "Storage Mover Agent"
# Create Storage Mover Agent VM with minimum 8GB of RAM
# You can give the VM more RAM if you need it to work faster
New-VM `
-Name $VMName `
-MemoryStartupBytes 8GB `
-SwitchName "InternalNAT" `
-Path "V:\Hyper-V\" `
-VHDPath "V:\Hyper-V\Storage Mover Agent\AzureStorageMover_2.0.287_amd64.vhdx" `
-Generation 1
# Disable Dynamic Memory
Set-VMMemory -VMName $VMName -DynamicMemoryEnabled $false
# Set the VM processor count for the VM to minimum of 4 vCPUs
Set-VM -VMname $VMName -ProcessorCount 4
# Open a VM Connect window, and start the VM
vmconnect.exe localhost $VMName
Start-Sleep -Seconds 7
Start-VM -Name $VMNameAs of today, you need to specify Generation 1 Hyper-V VM for the appliance, if you specify Generation 2, the appliance won’t boot.
Once the appliance is booted, you will see a login prompt, the user name is “admin” and the default password is also “admin“. Then you should go through the password change process and make it something way more secure.
Once you reset the password, you can also login to the agent VM using the following SSH command: ssh admin@ip-address
Next, you need to go to your network configuration and test your connectivity by entering number 2 in the xdmsh> prompt, you have the option to show or update network configuration (IP Address/mask) if needed and test your network connectivity (number 3). This is exactly what you should see. Everything is in green <Ok> and we’re good to go.
Then you can just press number 4 to Quit back to the main menu and next, we need to register our agent to the Storage Mover service. Enter number 4 on the first prompt, and then provide the following information. You can copy these details from the Microsoft Entra admin center (Identity > Overview blade), and from the Storage Mover > Overview blade in the Azure portal.
- Tenant Id?
- Subscription Id?
- Resource Group Name?
- Storage Mover Name?
- Agent Name? This name is shown for the agent in the Azure portal. Select a name that identifies this agent VM for you (Do not use dashes, spaces, or special characters for the Agent Name). Otherwise, you will encounter the following error: Agent VM registration was not completed within the wait time. You may try the registration again. Failed to register with Hybrid Compute!
- Private Link Scope (Optional)? Azure Private Link is used to set up a dedicated private IP address within your Azure virtual network for you to privately communicate with. You can use this feature to connect securely and migrate to Azure file shares. If you have a Private Link already set up, then you need to supply the entire resource ID for the Private Link Scope as shown in the example below. In this particular environment, we don’t have a Private Link. Press <
ENTER> to continue.
/subscriptions/123-456-789-0000-12345678/resourcegroups/resource-group-name/providers/microsoft.network/privateendpoints/private-name01
Next, you need to open a web browser, go to https://microsoft.com/devicelogin, and then use the provided code to sign in and enter your credentials. For authentication, the agent utilizes the device authentication flow with Microsoft Entra ID. In a few seconds, the registration should be completed successfully, and you can go back to the Azure portal.
Of course, you need to repeat the same steps for all of your other agents. The appliance VM will be automatically onboarded to Azure Arc Machines as well.
Create a Project Plan
Once you have all of your VM agent(s) registered with the Storage Mover service, you need to create a project plan.
Back to the Azure Portal > Storage Mover > Project Explorer and select + Create project. Give it a name, and a description, and then hit Create. Please note that you won’t be able to change the project name later, so choose a descriptive name here.
Now all of this processing is done through migration jobs, where we tell the agent what source and target we want it to use, and then it just gets to work.
Select the newly created project name and then click + Create job definition. We’re going to need a name for that and a description, then click and select your agent from the drop-down and click Next.
Then you need to provide the Source endpoint (Hostname or IP address) and the file share name. Then you select if this is an NFS or SMB share, and since SMB requires authentication, we’ll need credentials to access the file share.
As mentioned in the prerequisites section, you need to select the Azure Key Vault that you’ve already set up. Then select the secret for the Username, and select the secret for the Password as the input method as shown in the figure below. You can even add a Description if you like.
Then scroll to the bottom where you can optionally specify a Sub-path from the full source UNC path that the agent is going to use. In the Sub-path field, you enter: Sub-folder-1\Sub-folder-2. The full path will look something like this: \\192.168.0.211\Media\Sub-Folder-1\Sub-Folder-2.
A sub-path is optional and specifies a sub-folder. If you don’t define a sub-path, a job will run starting from the root of your share. If the sub-path is not found in the source, nothing will be copied.
The agent will work to copy that entire file share, all of its subfolders, and all of its contents. But remember, as mentioned in the prerequisites section, if you have a massive file share and you want everything to go through faster, you could have multiple agents. Well, to do that the right way, you’d want to assign each agent to a subfolder so that the two agents don’t overlap each other because that will lead to problems.
Then click Next to specify the target endpoint. For the Target Endpoint, you need to specify where in Azure Files we want to copy this data. You need to select your Subscription, and the Storage account you want to use, select the target File share, and add a Description. You can also add a Target sub-path if you need to. Then click Next.
Next, you have to select the Copy mode, and this is either going to be Merge or Mirror. Now there’s a lot of information to read on this page so you know exactly what’s happening and what kind of migration scenario you want, but at a high level, mirror is where we’ll make the target match the source, and merge is where we’re adding data from the source into the target, just like Robocopy. So, if you’re bringing in data from multiple sources into a single target, you’d want to do a Merge because Mirror will delete the data from source one and replace it with data from source two. Click Next.
In the last step, you should review all the settings to make sure all the details are correct, and then click Create.
You could also add, or delete storage endpoints by navigating to the left menu under Resource Management > Storage Endpoints. You will see two tabs for the Source and Target endpoints that we just created as part of our project.
But you can also click at the top to create a new endpoint(s) and then you can add those in as jobs to your project later on, that way you have full control in planning out your migration from end to end.
Start and Monitor Migration Job
Now we’re ready to start the migration. Go to Project Explorer, select your migration project, and then click on the migration job.
As shown in the figure below, we have two main tabs: the “Properties” tab, where all the data is visible, and the “Run History” tab, where all the run jobs are displayed. The only thing that you need to do, is click Start job at the top.
Now the job will go into Queue and wait for the agent VM to authenticate with the Key vault credentials over to the on-premises file share and it only needs read access, then it will start copying the files over to your target Azure file share.
The cool thing is your users can just keep on working if the agent runs into something like a file lock situation, there is no issue because the job can be run as many times as you want.
The job has been assigned to the agent. You can view the service and job status by selecting number 3 in the appliance VM’s main menu, and then look at the Job Summary by selecting number 2, you will see the “JOB ID” number and the status “SUCCESSFUL” as shown in the figure below.
Back into the Azure portal, on the migration job page, click Refresh at the top, and you will see a new Monitoring tab where you can see a lot more data of the processed files and folders in detail. In this example, the job is completed successfully.
Next, click on the Run history tab, and you can see all of the times you tried to run the migration job. In this case, the job run Succeeded.
Now anything that goes wrong, for example, files aren’t copied or any kind of issue you run into, is a Failed job. If you encounter such issues, you can click on the failed job run, and then at the top, you’ll have a deep link that you can go to, which will pull in all that data from your Log Analytics workspace about your copy jobs. The important thing, it gives you the job error code where you can click the Mitigation link at the top that will redirect you to the official Microsoft troubleshooting guide, and then you can just search for the error code.
To verify that the SMB file share was copied to Azure Files, you can switch to the Storage account and browse the Azure file share. In this example, we can see all the media files were copied to our media SMB file share in Azure.
Please note that the Storage Mover service automatically assigns the “Storage File Data Privileged Contributor” RBAC role to the Azure file share that you select as your Target endpoint. This is done for the Storage Mover agent VM, which you are using (in this case, it is called MoverAgentVMCH01). This RBAC role is assigned directly to the file share and not to the Storage account.
This privileged role is required to complete the migration job. Once the migration project is complete, you can remove the role for added security.
There you have it. Happy file shares migration with Azure Storage Mover!
Wrapping Up
In this guide, we showed a new effective way that you can use to move and migrate on-premises SMB and NFS file shares to Azure file shares using Azure Storage Mover.
Besides the existing general available capability to migrate from an on-premises NFS share to an Azure Blob container, Storage Mover supports many additional source and target combinations. Whether your SMB share is on a NAS or a Linux/Windows server, you can now use Mover’s managed migration service to migrate your SMB share to Azure file shares with full-fidelity migration and differential replications.
Storage Mover is a new service that will migrate your NFS and SMB storage into the cloud with no downtime, saving your permissions and never giving access to your data to anyone else.
The Storage Mover service works through a migration appliance VM. You will run it on your organization’s infrastructure, close to the source storage endpoint. The agent VM needs to be registered and can then be fully managed from the Storage Mover resource in the Azure portal.
__
Thank you for reading my blog.
If you have any questions or feedback, please leave a comment.
-Charbel Nemnom-
