How to Deploy a Secure FTP (SFTP) Service on Microsoft Azure

| ,

Published on | Updated on February 24, 2021

8 Min. Read

In this article, I will share with you how to deploy a secure (SFTP) service based on Microsoft Azure and Azure File Shares.

Updates

Based on the user’s feedback and comments, the SFTP deployment in Azure was updated with 4 new options as follows:

  • Updated – 01/02/2021 A new scenario was created to address and limit the access of who can upload files via SFTP. In this scenario, you will use the container to upload files via SFTP from specific public IPs only for added security, then you can access those files via a web service (HTTP/HTTPS) from anywhere. Please check the following section for more details on how to deploy that scenario.
  • Updated – 11/11/2020 A new scenario was created to address the authentication scenario using SSH keys instead of using usernames and passwords. Please check the following section for more details on how to deploy it.
  • Updated – 06/10/2020 – The ARM Template has been updated to include the new file share access tier to reduce cost, you can choose now between (Hot, Cool, or Transaction Optimized). How to Deploy a Secure FTP (SFTP) Service on Microsoft Azure 1
  • Updated – 05/10/2020 – The ARM Template has been updated to mount the volume inside the container instance with the same Azure file share name. This is useful if you want to create another folder with a certain name that you need for a specific project.How to Deploy a Secure FTP (SFTP) Service on Microsoft Azure 2

What would you like to see next? You are welcome to share your thoughts in the comment section below.

Introduction

SSH File Transfer Protocol (SFTP) is a network protocol used for secure file transfer over a secure shell. FTP/SFTP is still a very common protocol used by many customers in several industries. Microsoft does not have a fully managed SFTP service in Azure yet, however, Amazon AWS has an SFTP as a service, and if you are purely an Azure customer, then this is obviously a less desirable solution.

Azure should have SFTP as-a-service, so it will be so much easy to implement rather than creating some IaaS VMs and enable FTP because creating a VM-based SFTP is costly and require high-maintenance. The user-voice is very active and many customers are requesting an SFTP as a service on Azure. Please add your vote here.

So if you are still want to deploy a secure FTP on Azure today, you still have a couple of options as follows:

  1. Get a FileZilla Pro license and send the files to Azure blob storage, however, this is still an IaaS solution and not a full PaaS solution.
  2. Use a third-party solution such as SFTP on Files.com and integrate with Microsoft Azure blob storage. Files.com is a great solution that integrates seamlessly with Azure. Check the following step-by-step guide and see how to mount Azure blob storage on top of FTP/SFTP service.
  3. Use an Azure Container Instance (ACI) powered by an Azure File Share as a storage back-end for a less VM approach. This solution will be a good workaround for a cost-effective SFTP solution in Azure which is backed by durable persistent storage. ACI service is inexpensive and requires very little maintenance, while data is stored in Azure Files which is a fully managed SMB service in the cloud.

In this article, I will share with you how to deploy an SFTP service based on Azure Container Instance (ACI) and Azure File Shares.

Deploy SFTP Service on Azure

Microsoft has released two Azure ARM Templates to create an on-demand SFTP Service on Azure for two different scenarios:

  1. Scenario 1: Create an SFTP Service with a new Azure Files share.
  2. Scenario 2: Create SFTP Service with an existing Azure Files share.

This template creates an on-demand SFTP server using an Azure Container Instance (ACI). It creates a Storage Account and a File Share via the Azure CLI using another ACI. This File Share is then mounted into the main ACI to provide persistent storage after the container is terminated. The container is Linux-based. The beauty of this solution is, once you transfer/upload are completed, you can stop the ACI and the files will remain accessible. You can also delete/recreate the ACI and mount the same file share to copy more files.

Please note that the templates published by Microsoft above will create a (general purpose v1) storage account. To this end, I have updated the ARM template to support (general purpose v2) storage account type besides the improvement mentioned in the updates section. Additionally, the price per GB for general-purpose v1 and general-purpose v2 storage account is the same. So why not use the latest Azure storage features.

Click on the “Deploy to Azure” button and follow the steps as shown in the video below to deploy an SFTP service with a new Azure storage account and file share (Scenario 1).

Deploy To Azure

Please take note of the username and password during the deployment since you will need to use them to access the SFTP service in the next step.

Last but not least, copy the public IP address from the container group (sftp-group), and then connect securely to the SFTP service with your desired FTP client such as (FileZilla). Enjoy :)

Restrict Public IP Access

Now the SFTP service is publicly accessible from anywhere over a secure shell. You have a new requirement to whitelists specific IPs to connect to the SFTP service. In other words, you want to restrict access to the SFTP service in Azure and allow only a certain set of IP ranges.

What you could do is the following, you can implement a Network Security Group (NSG) on the subnet in Azure and then only allow Inbound communications from specific public IP ranges. Now in order to use an NSG, you’ll need to deploy the Azure Container Instances (ACI) into a virtual network (VNET) as documented by Microsoft here. However, as noted in the “limitations” section there, Public IP isn’t supported in this scenario (Container groups deployed to a virtual network don’t currently support exposing containers directly to the internet with a public IP address or a fully qualified domain name), so you would need to proxy the connection through something else that would support the IP restrictions such as using an Azure Firewall for example. Hopefully, the Azure Container Team will address that limitation in the near future.

To restrict access to the SFTP service, please check the following section where you can enable an IP allowlist and co-locate Nginx as front-end with SFTP in the backend.

Restricted SFTP access with NGINX Reverse Proxy

NGINX is one of the most popular web servers in the world. Not only is NGINX a fast and reliable static web server, but it is also used by a ton of developers as a reverse-proxy that sits in front of their APIs.

A reverse proxy server is a server that typically sits in front of other web servers in order to provide additional functionality that the web servers may not provide themselves.

For example, a reverse proxy can provide SSL termination, load balancing, request routing, caching, compression, or even A/B testing.

When running web and SFTP services in docker containers, it can be useful to run a reverse proxy in front of the containers so you can have more control over who can access your files and serve static content using Nginx for example.

In this section, I will share with you a useful and interesting scenario where you can use the container with SFTP to upload files from a restricted set of public IP addresses, and then use Nginx reverse proxy as a front-end to access those files securely via HTTPS.

The entire process is the following:

  1. Update the following Nginx.conf file with your public IP addresses that you want to allow who can upload files via SFTP. In this configuration, I have two ports so we can connect via SFTP to the public IP using port 2222 (which is passed to port 22 of the SFTP container) and also connect natively to port 80 with HTTP.
    user  nginx;
    worker_processes  1;
    
    error_log  /var/log/nginx/error.log warn;
    pid        /var/run/nginx.pid;
    
    events {
        worker_connections  1024;
    }
    
    http {
        default_type  application/octet-stream;
    
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';
    
        access_log  /var/log/nginx/access.log  main;
    
        sendfile        on;
        #tcp_nopush     on;
    
        keepalive_timeout  65;
    
        #gzip  on;
    
        include /etc/nginx/conf.d/*.conf;
    
        server {
            root /mnt/sftp;
            listen 80;
            location / {
                autoindex on;
            }
        }
    }
    
    stream {
        server {
            listen 2222;
            proxy_pass localhost:22;
            allow add_your_public_ip_1;
            allow add_your_public_ip_2;
            allow add_your_public_ip_3;
            # Need this IP range for the ACI monitoring infrastructure
            allow 10.240.0.0/16;
            deny all;
        }
    }
  2. Upload the updated Nginx.conf file to an existing Azure file share, assuming you already have a storage account deployed in Microsoft Azure. In this way, Nginx will be configured at runtime automatically.
  3. Deploy the SFTP service using the ARM template (more on this in a bit).
  4. Connect using the public IP address or the container DNS (FQDN) via TCP Port 2222, then authenticate to the SFTP service using your favorite FTP/SFTP client (FileZilla), and upload some files.
  5. Access the SFTP Service using the public IP address or the container DNS (FQDN) from your favorite browser and enjoy!

Click on the “Deploy to Azure” button and follow the steps as shown in the video below to deploy the SFTP service with a username and password authentication (for SSH keys authentication, check the next deployment option). The template will create a new Azure storage account and one file share, assuming that you have already uploaded the Nginx.conf file to an existing Azure storage account and file share.

Deploy To Azure

For SSH keys authentication, click on the “Deploy to Azure” button below to deploy the SFTP service with Nginx reverse proxy. The template will create a new Azure storage account and one file share, assuming that you have already uploaded the Nginx.conf file to an existing Azure storage account and file share.

Deploy To Azure

With this scenario, you will use the container for SFTP to upload files from restricted public IP addresses, then you can access those files as static via HTTP/HTTPS using Nginx reverse proxy as front-end.

Logging in with SSH keys

If you interact regularly with SSH commands and remote hosts, you may find that using a key pair instead of passwords can be convenient. Instead of the remote system prompting for a password with each connection, authentication can be automatically negotiated using public and private key pairs.

In this section, I will share with you another interesting scenario that you can use to logging in and authenticating with SSH keys to your SFTP service based on Azure Container Instance (ACI) instead of using passwords.

The entire process is the following:

  1. Generate your SSH (public/private) keys with OpenSSH: ssh-keygen -t rsa -b 4096 -f ssh_sftp_rsa_key
  2. Deploy the SFTP service using the new ARM template (more on this in a bit).
  3. Upload your public key (xxxxx.pub) to the Azure File Share where the SSH key will be stored (e.g. adminsftp).
  4. Restart the Azure Container Instance (sftp-group).
  5. Load the private key on your machine using (Pageant from PuTTY) for example.
  6. Connect using the public IP address and authenticate to the SFTP service using your favorite FTP/SFTP client (FileZilla) for example.

Click on the “Deploy to Azure” button and follow the steps as shown in the video below to deploy the SFTP service with a new storage account and two Azure file shares, one for the SFTP data upload, and the second one will be used to store the SSH public keys.

Deploy To Azure

That’s it there you have it!

Summary

In this article, I showed you how to deploy an SFTP service based on Azure Container Instance (ACI) and Azure File Shares in two different scenarios. However, if you want to deploy a secure FTP with Azure blob storage without using containers, then you can create an SFTP service based on Files.com today, and then sync your data to Microsoft Azure blob storage. This is a great solution because your data can be mounted directly or synced to/from Microsoft Azure, thus reducing additional storage costs.

The good news is, Microsoft is actively investigating to create a fully managed SFTP (PaaS) service on Azure, I will update this article as soon as I can share what’s coming. Please stay tuned!

The following survey has been recently closed by Microsoft which was open to collect customers feedback and use cases: To this end, they’ve created a survey, please take a minute and fill it out here: http://aka.ms/ftprequirements

So until Microsoft releases a fully managed SFTP as service, you can create an SFTP service based on Azure Container Instance (ACI), and then once the PaaS solution is available, you can switch and use the same Azure File Share. Your data will be intact.

__
Thank you for reading my blog.

If you have any questions or feedback, please leave a comment.

-Charbel Nemnom-

Related Posts

Previous

How To Set Multiple IP Addresses on One NSG Rule in Azure Stack

Passed Exam: Business Continuity Management Based on ISO/IEC 22301

Next

30 thoughts on “How to Deploy a Secure FTP (SFTP) Service on Microsoft Azure”

Leave a comment...

  1. Great solution, thanks for that! 2 Quick questions from a linux noob – Can I create additional users who have access to that same upload folder? or better how could I make use of AAD to control access – or combination of both

    Reply
    • Thanks David for the comment!

      This is unfortunately quite a complex task, as the default behavior of the container is to force a separate folder for each user.
      To get around this, you would need to create a new container image based on the existing one, and modify a few things in there.
      For the use of AAD to control access… this isn’t something that’s natively possible at the moment. There’s a VM extension that has been in Preview for years, but I’m not sure what the future of that one is.

      Thanks!

      Reply
  2. Thanks for the detail outline which I was looking for my presentation, planning to move in Azure soon but the missing part of SFTP solution was not really clear. Even I have asked Microsoft so many time how this can achieve without 3rd party and IaaS solution, they told me that it will be done via Data Factory which not make sense much really, didn’t see it DF can do what SFTP server capable.
    Question for you what’s the missing part using of ACI than having SFTP as a service, can you add on this? Thanks again

    Reply
  3. Thanks Charbel for the fast replay. I’m more focusing with Azure stand alone solutions ACI will work for now until Azure come up with SFTP as a service, I’m dealing inbound and outbound connection initiation to sftp limiting with IP whitelist to the container will work for access. Since I’m planning to apply for gov cloud using 3rd party solution very titend to get approval but as you mention files.come, sftp gateway lot out there most of them also require to run VM (cons cost will be high). Thanks again great article.

    Reply
  4. Hello!

    Thank you for the awesome demo.

    I have a question about the SSH setup. I followed your steps and deployed a ACI. I also generated a SSH key pair on my laptop.

    However, my confusion lies somewhere in the later steps (steps 5-6). I uploaded the public key to the file-share (as it is, not in any hidden folder) and restarted the Instance. However, before performing step 5, I wanted to test my connection. So, on FileZilla, I entered the host name, user name & password, but did NOT upload the private key file. However, it still connected. I uploaded the private key file (through FileZilla) and the same result was produced. My concern is that the host should not even allow a connection from the client that does not have the private key. What should I do or what I have done wrong?

    Note: unlike the demo video/GIF which creates a new resource, I used the option to use existing resources. I also understand I may not have given all details necessary, so if you require any other detail please ask me!

    Thank you in advance! :)

    Reply
    • Thanks Varun for the comment!
      Please note that if you deployed a new ACI following my SSH template scenario, there is no Password to enter during the deployment.
      Which template did you use? if you followed the existing Azure File share (Scenario 2) here, then this won’t work because this template only accept username and password as authentication. Which existing resources do you have already (storage account and file share)?
      If you want to test the SSH scenario, please click Deploy to Azure in this section Logging in with SSH keys.
      I need to address your scenario in an upcoming update for SSH Setup with existing Azure File Shares.
      Hope that helps!
      Best,
      -Charbel

      Reply
  5. Hey Charbel,

    Thank you so much for the clarification. I think I should be able to manage that.

    However, my next question is if it’s possible to make the .ssh folder a hidden folder. When I connect through FileZilla the .ssh folder shows up, but is it possible to completely hide it. (the client should not be able to see it at all).

    Another question I have is what would you recommend the best method for having multiple users (different volumes)?

    This website has been a saviour and I absolutely love your template! Thank you so so much again!!

    Reply
    • Hello Varun,
      I think that the .ssh folder requires very specific permissions, so it will always need to be readable by the user.
      By default any folder starting with a (.) will be hidden in Linux, but is probably shown by default in FileZilla.
      Could you please try a different FTP/SFTP client and check?
      For the second question, what I can think of that you need to have a different Azure File share(s) per user: one for each SSH public key, with the appropriate mappings in the ARM template.
      Hope this helps!
      -Charbel

      Reply
  6. Hi Charbel.

    Thank you so much, I believe I have figured it out.

    I do have a few more questions I would want to ask you. Do you know how add multiple users for the SFTP in different File Shares?

    Thanks in advance,
    – Varun Sampat

    Reply
    • Do you mean for (Read, Write, Modify/Delete)?
      This could be achieved using the Linux directory/file permissions.
      Yes, you can create more users and do the appropriate mapping (user => File share).

      Reply
  7. Great article and followup questions. Do you have a recommendation on how best to assign a static IP to the SFTP ACI? Or, would it be better to have this behind an Application Gateway?

    Reply
  8. For the above service of SFTP server I want to create multiple users with different folder level permissions in the storage account. From the portal we can create only 1 user account. Also in which region is this service supported. Please let us know.

    Reply
    • Hello Syed,
      You can deploy the SFTP service following this article in any Azure region.
      You can create multiple Azure File Shares and assign each one for a specific user.
      You need to update the template to match your requirements.
      Hope this helps!

      Reply
  9. Hello dear, do you know any way to implement a web service to access these existing files via HTTPS? I will use the container with SFTP only to upload files, but I need to implement an HTTPS service to be able to access the files. I would appreciate your guidance. Thank you very much for the article, it is very helpful.
    Regards.

    Reply
  10. Am I correct that if I wanted to have the Nginx / SFTP solution work entirely with port 22, one would just change the Azure template and nginx.conf file’s 2222 entries to 22?

    Reply
    • Hello Bill,
      Yes correct, you would need to update the template and nginx.conf file’s from port 2222 entries to 22.
      I would recommend using a different port for security purposes. For this reason, I selected a different port.
      Thanks!

      Reply
  11. I did some more testing and found that i can’t use the Nginx container to map Internet port 22 to the SFTP container’s port 22. I’ve temporarily switched things back to port 2222 externally, but it would be nice to find a simple way to have the Nginx/SFTP solution advertise port 22 to the Internet

    Reply
    • Hello Bill, here is a quick update:
      Please note that since the SFTP container is listening on port 22, you can’t have nginx listening on the same port. However, you could modify the SFTP container to listen on 2222 instead, and then you could expose port 22 via nginx to the Internet.
      Hope this helps!

      Reply
  12. Hello again–I’ve got most things working, but my testing indicates that the strategy of including Allow public IP addresses does not work. IP addresses that are NOT one of the Allow entries can still access the SFTP server on the back end. The log files on the Ngnix Container show that Azure is NAT’ing all public IP addresses to one of the private IP addresses in the CIDR block that’s included in the Allow list.

    Essentially, it seems that an Azure Nginx Container does not get access to the public IP addresses for any of its Azure traffic. Therefore, it can’t white list them.

    Thoughts?

    Reply
    • Hello Bill, thanks for your comment. Yes, you are right. I saw the same thing here.
      After investigating, I found out that the Azure Container Instances is not preserving the client IP address. You can read about it here.
      I am checking with the team to see when they will support preserving the client IP address so we can filter incoming public IP addresses.
      Hope this helps!

      Reply

Leave a comment below...

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe to Charbel Nemnom’s Blog

Get the latest posts delivered right to your inbox

The content of this website is copyrighted from being plagiarized! However, you can copy from the 'Code Blocks'.

Please send your feedback to the author using this form for any 'Code' you like.

Thank you for visiting!