The App Service resource provider in Azure Stack is a PaaS-based service that is deployed and maintained by the Azure Stack Cloud Operator.
Users and tenants can use the App Service resource provider to create web applications, mobile application back ends, RESTful APIs, and more. It is designed to help solve many of the problems faced by web developers by focusing on delivering web hosting at the cloud scale.
The App Service provides features and frameworks that are required for building, deploying, and maintaining enterprise-level applications. You can use many popular development platforms and languages including Java, PHP, Node.js, Python, and the Microsoft .NET Framework languages.
By default, when you deploy the App Service resource provider in Azure Stack, it creates the following infrastructure for you:
- Website Controller: This role server is responsible for provisioning and managing the other roles in the App Service.
- Management: This role is the App Service REST API that Azure Stack uses.
- Front-End: This role accepts the inbound HTTP/HTTPS requests from consumers of the web app, it routes the request to the worker server where the web app is being executed and returns responses from the worker to the client.
- Publisher: This role is responsible for publishing the DevOps-created content for web apps. It is the File Transfer Protocol (FTP) endpoint and is responsible for integration with code repositories such as GitHub.
- Worker (in Shared mode): This role is responsible for executing the web app that the client is connecting to.
- File Server: This stores the content for each web app that is uploaded to App Service. The worker roles obtain the required files for the web app from this role.
It is possible to scale these instances to meet the requirements of the App Service resource provider. For more information about the App Service resource provider in Azure Stack, please check the following article.
The other day, I was preparing for the App Service resource provider deployment on Azure Stack Integrated Systems, and I was going through all the prerequisites steps before I begin with the actual deployment.
Since it’s a production deployment, it’s recommended and important to deploy the App Service in a highly available configuration. Otherwise, the service might become unavailable if any of the components fail. For more information about the HA deployment for the App Service, please check the following guide.
Once the HA deployment is completed successfully, I was able to launch the App Service dashboard from the admin portal.
However, after a couple of the days, the dashboard didn’t open anymore, and I received the following error message:
Failed to load App Service extension. Please check whether App Service resource provider and sql database is up and running.
Finding the cause
As part of the highly available App Service infrastructure VMs deployment, the following resources are created:
- A virtual network and required subnets.
- Network security groups for the file server, SQL Server, and Active Directory Domain Services (AD DS) subnets.
- Storage accounts for VM disks and cluster cloud witness.
- One internal load balancer for SQL VMs with private IP bound to the SQL AlwaysOn listener.
- Three availability sets for the AD DS, file server cluster, and the SQL cluster.
- Two node SQL cluster.
- Two node file server cluster (Storage Spaces Direct).
- Two domain controllers.
Since it’s a highly available deployment, I went to the virtual machines blade and I restarted one of the SQL server nodes.
After a couple of minutes, I went back to the App Service dashboard and I noticed that the dashboard is loading now!
So it looks like the issue is around the App Service SQL database.
I went back to the HA deployment document here and I read the instructions again to see if I missed any step. What I found is, at the end of the document, there is a comment by a user that he hits the same issue stating the following:
The most common reason for that would be missing the post install steps to add the Databases to the Always-On availability group. Unfortunately, this step is documented in the release notes here and not in the deployment guide.
In the Post-deployment Steps, Microsoft is stating the following:
As you can see, the document is misleading and the steps are not documented properly.
Fixing the issue
At the time of this writing, if you encountered this issue, then after the App Service installation is completed, you need to go to the SQL Server, back up the two databases and then add them to the Availability group – If you don’t do this, the databases will not be available if the Availability group listener moves to the 2nd SQL node. Thus, the App Service extension won’t load.
To resolve this issue, please follow the steps below:
- Open a remote desktop session to the CN0-VM (which has a public IP address). But before doing so, you need to Add inbound RDP port rule in the network security group (NSG) to allow access for that VM, because RDP access is denied from the Internet by default. Please note that you need to use the “Other Roles Virtual Machine Admin username and password” to login.
- From the CN0-VM, you need to open another RDP session to one of the SQL Servers clusters. Please note that you need to use the domain admin username and password for the domain controller that gets deployed as part of the App Service infrastructure.
- Launch SQL Server Management Studio (SSMS), and open a connection to both SQL servers at the same time. At this point, you should see one SQL Server has the two app service databases and one server does not.
- Right-click each database (appservice_hosting and appservice_metering) and then choose Tasks -> Back Up… and continue next through the wizard. This step is optional but better safe than sorry!
- Once the databases are backed up, open the Always On High Availability groups on both servers and note which server is the (Primary).
- Right Click on the alwayson-ag (Primary) and select Add Database…
- Select the two App Service Databases and click on Next > to continue.
- In the Connect to Existing Secondary Replicas step, click on Connect… to connect to the secondary replica SQL Server. Click on Next > to continue.
- In the Select Initial Data Synchronization step, keep the default option Automatic seeding and click on Next > to continue.
- In the Validation step, make sure that all results passed successfully, and click on Next > to continue.
- In the Summary step, verify the choices you made in the wizard, and click on Finish.
- In the final step, verify the wizard completed successfully, and click on Close.
- That’s it there you have it. The databases will now be on both SQL servers. In this example, SQL-1 is the Primary.
- If one server is rebooted or down, the App Service will remain available to the users (the Secondary server will be inaccessible – as expected), the Primary server will be only active. In this example, SQL-0 is the Primary now.
- Last but not least, don’t forget to remove and delete the RDP access rule that you set on CN0-VM (NSG) in Step 1.
Thanks to the Azure Stack team for their help in getting to the bottom of this.
Hope this helps someone out there!
Thank you for reading my blog.
If you have any questions or feedback, please leave a comment.