(Solution) Failed To Create Backup in Azure Stack Development Kit 1807 Update #AzureStack #AzureStackDevKit #ASDK

| ,

Published on | Updated on December 27, 2020

2 Min. Read

I recently had a problem where I could not get Azure Stack Infrastructure Backup to work correctly for me whatsoever.

Just to be clear here, I am referring to Azure Stack Development Kit (ASDK) and NOT Azure Stack Integrated Systems (multi-nodes).

The two issues I had were the following error:

  • Failed to create the backup. Error message: backup failed with status Failed.
  • Infrastructure backup failed because of an unknown error.

(Solution) Failed To Create Backup in Azure Stack Development Kit 1807 Update #AzureStack #AzureStackDevKit #ASDK 1

As you can see in the screenshot above, the error is unclear!!! So I followed the remediation steps:

  1. I started a new Backup through the portal and with PowerShell. I ended with the same error after approximately 10 minutes.
  2. I collected the Azure Stack logs.

I reached out to the Azure Stack product group and they investigated over one night (time zone difference). Eventually many hours of troubleshooting, the logs identified the problem.

  • ERCS VM is running high memory usage, and from the log we saw bunch of the OutOfMemory Exceptions.
  • This issue is only on ASDK 1807 update as the ERCS VM default memory on ASDK is 2GB.

(Solution) Failed To Create Backup in Azure Stack Development Kit 1807 Update #AzureStack #AzureStackDevKit #ASDK 2

  • The workaround is to increase the ERCS VM memory to 4GB manually.
  • The fix will be addressed in the next update.

So I bumped the ERCS VM to 4GB and even to 8 GB and retried the backup, but I encountered another issue.

  • Since I ran many backups, the Backup service got to a point when the service got choked on memory and died, which caused all backups attempts later to failed completely.
  • To resolve this issue we had to reboot the ERCS VM and canceled all previous backup actions. Then we checked that the cluster is in a healthy state.

(Solution) Failed To Create Backup in Azure Stack Development Kit 1807 Update #AzureStack #AzureStackDevKit #ASDK 3

We triggered a new Infrastructure Backup, and things finally worked.

(Solution) Failed To Create Backup in Azure Stack Development Kit 1807 Update #AzureStack #AzureStackDevKit #ASDK 4

The ERCS VM memory issue won’t appear again in the next update.

Many Thanks to Tony, Peter, Vijay, Michela and Charles in the Azure Stack team for their help in getting to the bottom of this.

Hope this helps someone out there!

__
Thank you for reading my blog.

If you have any questions or feedback, please leave a comment.

-Charbel Nemnom-

Previous

Configure Azure Stack Automatic Infrastructure Backup With PowerShell #AzureStack #AzureStackDevKit #ASDK

Getting Started With CloudBerry Explorer PRO For Microsoft Azure Blob Storage @CloudBerryLab

Next

Leave a comment below...

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe to Charbel Nemnom’s Blog

Get the latest posts delivered right to your inbox

The content of this website is copyrighted from being plagiarized! However, you can copy from the 'Code Blocks'.

Please send your feedback to the author using this form for any 'Code' you like.

Thank you for visiting!