Enable SFTP Support for Azure Blob Storage

SSH File Transfer Protocol (SFTP) is a network protocol used for secure file transfer over a secure shell. FTP/SFTP is still a very common protocol used by many customers in several industries. Up until today, Microsoft did not have a fully managed SFTP service in Azure yet, and if you are purely an Azure customer, then this is a less desirable solution.

A while ago, we wrote a blog post on how to deploy a Secure FTP (SFTP) service on Microsoft Azure by leveraging the Azure Container Instance (ACI) powered by Azure File Shares. You can read more about this solution in this article.

After more than two years of waiting, finally Microsoft released a fully managed SFTP (PaaS) service based on Microsoft Azure Blob Storage. In this article, we will show you how to deploy and enable SFTP support for Azure Blob Storage.

Introduction

SFTP support for Azure Blob Storage is now available in public preview. You can use an SFTP client like OpenSSH or PuTTY to connect to an SFTP-enabled storage account, manage your objects using file system semantics and add users to access the storage account, and specify permissions for each user.

The capabilities of this new SFTP solution require an ADLS Gen2 Storage Account (also referred to as Hierarchical Namespace account, HNS account, or Azure Data Lake Storage Gen2 account).

Why Microsoft has built the SFTP solution to use a hierarchical namespace (HNS) over blob data?

The key mechanism that allows Azure Data Lake Storage Gen2 to provide file system performance at object storage scale and prices is the addition of a hierarchical namespace. This allows the collection of objects/files within a storage account to be organized into a hierarchy of directories and nested subdirectories in the same way that the file system on your computer is organized.

To learn more about the benefits of using a hierarchical namespace, please check the official documentation from Microsoft.

Prerequisites

To follow this article, you need to have the following:

1) An Azure subscription. If you don’t have an Azure subscription, you can create a free one here.

2) Before you can enable SFTP support, you must register the SFTP feature with your subscription. The following steps will enable the feature on your subscription, you can choose between PowerShell or the Azure CLI.

> Open Windows Terminal if you have the Azure CLI installed on your machine, or use the Azure Cloud Shell at https://shell.azure.com and run the following commands:

Azure CLI

# Set the Azure context for the desired subscription
az account set --subscription "xxxx-xxxx-xxxx-xxxx" 

# Check if the live tier feature is registered first 
az feature show --namespace Microsoft.Storage --name AllowSFTP

# Register the live tier feature on your subscription
az feature register --namespace Microsoft.Storage --name AllowSFTP

PowerShell

Here is the PowerShell version for the same:

# Set the Azure context for the desired subscription 
Set-AzContext -Subscription "xxxx-xxxx-xxxx-xxxx"

# Check if the live tier feature is registered first
Get-AzProviderFeature -FeatureName "AllowSFTP" -ProviderNamespace "Microsoft.Storage"

# Register the live tier feature on your subscription
Register-AzProviderFeature -FeatureName "AllowSFTP" -ProviderNamespace "Microsoft.Storage"

3) An Azure storage account Gen2 with hierarchical namespace enabled. Please note that you can’t enable a hierarchical namespace on an existing storage account. You can enable a hierarchical namespace only when you create a storage account. See the instructions on how to create a storage account to use with Azure Data Lake Storage Gen2. To automate the creation process, please check the next section.

4) You need a dedicated storage account to use for SFTP. At the time of this writing, an SFTP-enabled account will not support other protocols like REST or NFS. This means that blobs won’t be accessible by tools that use the REST API including Azure CLI, and PowerShell.

5) The storage account SKU type should be set to (Standard_LRS/Standard_ZRS), or premium Block blobs (Premium_LRS/Premium_ZRS).

6) If you’re connecting from an on-premises network, make sure that your client allows outgoing communication through port 22. The SFTP support for Azure Blob Storage uses only port 22. You cannot use a different port as we used to do with Azure Container Instance (ACI) powered by Azure File Shares.

7) The SFTP Public Preview is available in the following 13 Azure regions, the list will expand with more regions soon:

• North Central US
• East US 2
• East US 2 EUAP
• Central US EUAP
• Canada East
• Canada Central
• West Europe
• North Europe
• Australia East
• Switzerland North
• Germany West Central
• East Asia
• France Central

8) You have a couple of options to deploy SFTP, you can either use the Azure Cloud Shell, Visual Studio (VS) Code, the new Windows Terminal, or the Azure Portal. The deployment method described in this article works with PowerShell 5.1 or PowerShell 7.2.x (core) with the Az module installed. You can use the following PowerShell command to install and update the “Az module” locally on your machine.

# Make sure you have the latest version of PowerShellGet installed
Install-Module -Name PowerShellGet -Force 

# Install and update to the latest Az PowerShell module
Install-Module -Name Az -AllowClobber -Force

SFTP authentication and access

A Local User is a new form of identity management provided with SFTP for Blob Storage. You need to create and configure users within your SFTP-enabled storage account and grant access permissions to one or more root containers. At the time of this writing, you can have up to 1,000 users per account, and each user can access up to 100 containers. You can specify Read/Write/Create/List/Delete permissions for each container. Microsoft is looking at how to expand the users’ limit per storage account because 1,000 users are not enough for large organizations.

For SFTP authentication, you can use Public Key or Password to authenticate to your SFTP service.

A Public Key is the most common form used for SSH authentication. As you probably know, with Public Key authentication, the user must possess an SSH Keypair and the server (Azure Storage in this case) must possess the Public Key. When the user begins to connect, it sends a message with the public key and signature. The storage account will validate the message and check that the user and key are recognized in the account.

For Password authentication, this method uses an Azure-generated password (more on this in the next section).

What about the integration with Azure AD for user authentication instead of using Local Users?

The Azure AD identity team is actively working on Azure AD integration, Stay Tuned!

For the remainder of this article, we’ll use a public key to authenticate to our SFTP enabled storage account.

Create SFTP Enabled Storage account

In this section, we will generate SSH keys, then create a storage account for SFTP, and finally, authenticate and connect to the blob storage using OpenSSH.

For the purpose of the article, we will be using the Azure Resource Manager (ARM) template SFTP deployment, as well as showing you the Azure Portal experience.

Please note that at the time of this writing, PowerShell, or Azure CLI access to SFTP and Local User management are not yet supported. We can use Azure Resource Manager (ARM), the Azure Portal, and the ARMClient command-line tool to deploy and manage SFTP. Microsoft will progressively enable other methods soon.

Once you have all the prerequisites in place, take now the following steps:

Generate SSH Keys

Before we deploy the SFTP service and create a storage account, we’ll need to generate an SSH key pair to be used for public authentication.

First, you need to make sure you have OpenSSH installed on your machine. If you are using Windows OS (Client or Server), you can run the following PowerShell command to quickly verify:

Get-WindowsCapability -Online | ? Name -like 'OpenSSH*'

If OpenSSH Client is not installed on your machine (State: Not Present), then you can run the following command to install it:

# Install the OpenSSH Client
Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0

Once you have the OpenSSH client installed, you can run the following command to generate an SSH key pair.

ssh-keygen

Please note that by default, the ssh-keygen command will place keys and public key files into the users .ssh folder (C:\Users\username\.ssh) if you did not choose a different location than the default.

Next, you need to read (get) the contents of the public key by running the following command. Take note of the entire output because you’ll need it when deploying the SFTP service with public key authentication. We have black-boxed the secret for obvious reasons.

cat ~/.ssh/id_rsa.pub

Deploy SFTP service

To deploy the SFTP service, open PowerShell, and connect to your Azure account by running the following command:

Connect-AzAccount 

Select your desired Azure subscription if you have multiple subscriptions by running the following command:

Select-AzSubscription -SubscriptionId "<sub_id>"

Next, you need to create a resource group in the supported region by running the following command:

New-AzResourceGroup -Name "<rg_name>" -Location "francecentral"

Next, you need to download the SFTP account creation public key ARM template and note the path to the downloaded file, we’ll use it in the next step.

If you want to use password authentication instead of public keys, then you can download the JSON ARM template from here.

To deploy the ARM template to the resource group you created earlier, run the following command:

New-AzResourceGroupDeployment -ResourceGroupName "<rg_name>" -TemplateUri "<path_to_template>"

During the deployment, you will be prompted to enter the following values:

1) Provide a string value for ‘storageAccountName’ – (Type ? for help). This will be used to create a new storage account (you can also use an existing storage account if it meets the prerequisites).

2) Provide a string value of the primary ‘user Name’ – (Type ? for help). This will be used to create a new local user for the storage account

3) Provide string value for ‘home Directory’ – (Type ? for help). This represents the local user’s default remote directory. This can be a container or subdirectory within a container.

4) Finally, paste the string value for the ‘public Key’ – (Type ? for help). The value for the public key will be the entire string contents of your locally-generated public key. Please check the following section on how to generate an SSH key pair.

The deployment looks like the following:

Connect with SFTP Clients

Now that you have an SFTP enabled account that is accessible via a public key authentication! The storage account is created as “Standard_LRS” and located in “West Europe” by default (specified in template). You can also deploy the storage account in another supported Azure region, and with a different SKU type “Standard_ZRS” if you want.

"storageAccountType": {
      "type": "string",
      "defaultValue": "Standard_LRS",
      "allowedValues": ["Standard_LRS", "Standard_ZRS"],
      "metadata": { "description": "Storage Account type" }
    },
    "location": {
      "type": "string",
      "defaultValue": "westeurope",
      "allowedValues": ["westeurope", "northcentralus", "eastus2", "eastus2euap", "centralus", "canadaeast", "canadacentral", "northeurope", "australiaeast", "switzerlandnorth", "germanywestcentral", "eastasia", "francecentral"],
      "metadata": { "description": "Region" }
}

The storage account is also enabled for the hierarchical namespace (required) with a single container created for the home directory, and with a single local user created with full permissions to their home directory (check the next section if you want to change the user permissions).

Once you deployed the SFTP service, you could connect with any supported SFTP client such as OpenSSH, Putty psftp.exe, Cyberduck, or WinSCP. Microsoft recommends using the OpenSSH client. Please refer to the following section to install Windows OpenSSH.

Here is an example of how to connect with OpenSSH to SFTP on Blob Storage:

sftp <storage_account_name>.<localuser_name>@<endpoint>
# E.g. sftp testaccount.user1@testaccount.blob.core.windows.net

If you received a warning message about POSSIBLE DNS SPOOFING DETECTED! then make sure to add the correct host key in C:\\Users\\username/.ssh/known_hosts to get rid of this message. Or you could delete the known_hosts file and then try to connect again.

Here is what the user experience looks like after you connect successfully and upload some files.

Here is what it looks like from the storage account (storage browser) experience in the Azure Portal.

A quick cheat sheet for various OpenSSH operation commands that you could use:

OpenSSH Operation	Description
cd path	Change remote directory to 'path'
get remote [local]	Download file from SFTP
ls [-1afhlnrSt] [path]	Display remote directory listing
mkdir path	Create remote directory
put localfile	Upload file to SFTP
pwd	Display remote working directory
rename oldpath newpath	Rename remote file
rm path	Delete remote file
rmdir path	Remove remote directory

If you are using the Command Prompt console instead of Windows Terminal, then make sure to turn on Ctrl+Shift+C/V for copy and paste for Windows OpenSSH client as shown in the figure below. The copy and paste in Windows Terminal are enabled by default.

If you are new to Windows Terminal, we highly recommend checking it out here. You can download it for free from the Microsoft Store.

Connect with Private Endpoint

In the previous section, we connected to our SFTP blob container over a public endpoint. You could also configure the storage account to be accessible over a private endpoint only, so you can transfer files to SFTP over a private IP instead of pubic IP addresses. This scenario is useful if you have a site-to-site VPN between your on-premises network and Azure, or you need an SFTP service within your Azure environment. In this case, you keep SFTP access private for added security.

Here are the quick steps to configure your storage account with Private Endpoint and make it ready for SFTP access:

1) Go into the “Networking” tab of your storage account that you created in the previous step and add a new private endpoint as shown in the figure below.

2) In Create private endpoint wizard enter or select the following information:

• Basics
  • Select your Azure Subscription
  • Select your Resource Group
  • The region is selected by default
  • Enter your desired PrivateEndpoint name
• Resource
  • Target sub-resource: blob
• Networking
  • Select your existing virtual network
  • Select your existing virtual subnet
• Private DNS integration
  • Integrate with private DNS zone Leave the default Yes
  • Private DNS Zone Leave the default (New) privatelink.blob.core.windows.net
• Select Review + create and then click Create.

3) Go into the network security group that was created and then add an Inbound Security Rule allowing SSH over TCP port 22 as shown in the figure below.

4) Finally, test the connection over the private link by using the following command:

sftp <accountname>.<user>@<accountname>.privatelink.blob.core.windows.net

Change SFTP user permissions

If you want to limit the user’s access to SFTP, then you can change the value for “permissions” under “permission scopes” in the JSON template as shown in the figure below and then redeploy it.

You can refer to the permissions table below.

Permission	Permission Code	Description
Read	r	Read file contents
Write	w	Upload file Create directory Upload directories
List	l	List contents within container List contents within directories
Delete	d	Delete files/directories
Create	c	Upload file if file doesn't exist Create directory if it doesn't exist Create directories

From the Azure Portal, you can also add and edit (modify) the SFTP user permissions as shown in the figure below.

Add multiple users for SFTP

For large projects, you require to have multiple users accessing your SFTP service. In this section, we’ll explore how to add multiple users for SFTP enabled storage account in the Azure Portal.

In this section, you’ll learn how to add an additional local user, choose an authentication method, and then assign permissions for that local user.

Launch the Azure Portal and navigate to your storage account.

Under Settings, select SFTP, and then select Add local user as shown in the figure below:

In the Add local user configuration pane, add the name of a user, and then select which methods of authentication you’d like to associate with this local user. You can associate a password and/or an SSH key as shown in the figure below.

> If you select Secure with a password, then your password will appear when you’ve completed all of the steps in the Add local user configuration pane.

> If you select Secure with SSH public key, then select Add key source to specify a key source (Generate a new key pair, Use existing key stored in Azure, or Use existing public key).

Microsoft noted that while you can enable both forms of authentication, however, SFTP clients can only connect by using ONE form of authentication.

Click Next to move to the Container permissions tab of the configuration pane.

In the Container permissions tab, select the containers that you want to make available to this local user. Then, select which types of operations you want to enable this local user to perform such as (Read, Write, List, Delete, or Create).

In the Home directory field, type and enter the name of the container or the directory path (including the container name) that will be the default location associated with this local user. The Home directory is a container/directory path relative to the storage account to be considered a default directory for the user (E.g. container-name/directory-sales or container-name). If you don’t set the Home directory, the user won’t have the container and/or the directory associated as the default location when connecting.

Select the Add button to add the local user as shown in the figure below.

If you enabled password authentication, then the Azure-generated password appears in a dialog box after the local user has been added as shown in the figure below. Similar to Azure AD applications secret, you can’t retrieve this value later, so make sure to copy the password, and then store it in a place where you can find it. You can also regenerate the password later on if needed.

And if you chose to generate a new key pair, then you’ll be prompted to download the private key of that keypair after the local user has been added.

SFTP blob storage pricing

During the public preview, Microsoft announced that the use of SFTP does not incur any additional charges. However, the standard transaction, storage, and networking prices for the underlying Azure Data Lake Store Gen2 account still apply. SFTP might incur additional charges when the feature becomes generally available.

The SFTP transactions (downloads/uploads) will be mapped to the equivalent blob storage transactions (hierarchical namespace enabled). To learn more, please refer to the pricing page under the Transaction section. The deletion from SFTP is free.

Let’s take a look at the following real-world example based on the (West US 2) region. We use SFTP to backup our legacy system on-premises, hence, the upload is more than download based on our use case.

> Monthly upload 5 TB.

> Monthly download 28 GB.

> SFTP message size 100 KB.

Please note that the 4MB (write/read) transaction is irrelevant for SFTP currently because SFTP clients never can send a message that big.

Description	Price in US Dollar
Write operation cost (every 4MB, per 10,000)	$0.065
Read operation cost (every 4MB, per 10,000)	$0.005
SFTP Message size (KB)	100
Number of SFTP messages in 1 GB	(1,000,000/100) = 10,000
Write (Upload) GB for 1 month	5,000
Read (Download) GB for 1 month	28
Write cost for 1 month	(10,000*5,000/10,000)*0.065 = $325
Read cost for 1 month	(10,000*28/10,000)*0.005 = $0.14
Blob storage cost 5TB ($0.0184 per GB Hot)	$94.208
Total cost per month	$419.34

Please note that the price example above is tentative and might change when the SFTP service becomes generally available (GA).

Keep in mind that if the SFTP message size is bigger (e.g. 256 KB), this means it will be cheaper because we can send more data compared to 100 KB and generate fewer transactions. Check if you have control to set the message (buffer) size in the SFTP client(s) that you use.

The best way to optimize your costs is to increase the buffer (message) size and use the largest SFTP message size if possible. The SFTP message size limit is NOT controlled by Azure Blob Storage. It is configured by the SFTP client so depending on the client and OS used, you could increase the SFTP message beyond 100KB, we have tried 256KB with OpenSSH on Linux. Increasing the buffer size and using multiple concurrent connections can significantly improve speed and reduce cost (more on this in the next section).

SFTP Performance

Upload performance with default settings for some clients can be slow. Some of this is expected because SFTP is a chatty protocol and sends small message requests. Increasing the buffer size and using multiple concurrent connections can significantly improve speed.

For WinSCP, you can use a maximum of 9 concurrent connections to upload multiple files.

For OpenSSH on Windows, you can increase buffer size to 100000 Bytes (100 KB):

sftp -B 100000 testaccount.user1@testaccount.blob.core.windows.net

For OpenSSH on Linux, you can increase buffer size to 262000 Bytes (256 KB):

sftp -B 262000 -R 32 testaccount.user1@testaccount.blob.core.windows.net

There’s a 4-minute timeout for idle or inactive connections. OpenSSH will appear to stop responding and then disconnect. Some SFTP clients reconnect automatically.

That’s it there you have it!

Summary

In this article, we showed you how to deploy, manage, and enable SFTP support for Azure Blob Storage, so you take the advantage of a fully managed PaaS service without worrying about maintaining any IaaS VM or Azure Container Instances.

At the time of this writing, you could perform any data plane operations with Azure CLI, Azure PowerShell, and the REST API. However, CLI and PowerShell support for the management plane such as (enabling/disabling SFTP and managing local users) will come later.

Learn more

> Secure File Transfer Protocol (SFTP) support for Azure Blob Storage (preview).

> Connect to Azure Blob Storage by using the Secure File Transfer (SFTP) protocol (preview).

> Host keys for Secure File Transfer Protocol (SFTP) support in Azure Blob Storage (preview).

> Known issues with Secure File Transfer (SFTP) protocol support in Azure Blob Storage (preview).

> Deploy an SFTP Service on Files.com and integrate with Microsoft Azure, check the following step-by-step guide.

> Deploy an SFTP Service based on Azure Container Instances and Azure Files, check the following step-by-step guide.

Until then, stay tuned for the SFTP Azure blob storage GA release in 2022!

__
Thank you for reading my blog.

If you have any questions or feedback, please leave a comment.

-Charbel Nemnom-

Related Posts