Enable Diagnostic Settings for Storage Accounts to Event Hub

9 Min. Read

This article will demonstrate how to enable and configure diagnostic settings for storage accounts to Event Hub with Azure Policy.

A common theme in cloud environments is enforcing organizational standards and adopting cloud governance since day one. And this is very important since it will give you the ability to define policies, processes, and procedures. These policies then dictate what can be done and verify that what does exist is correct. A service from Microsoft called Azure Policy is a great way to make that happen and take proactive action.

Introduction

Azure Policy is a service in Azure that you use to create, assign, and manage policies. These policies enforce different rules and effects over your resources, so those resources stay compliant with your corporate standards and service level agreements. Azure Policy meets this need by continuously evaluating your resources for non-compliance with assigned policies.

You have a policy in your organization that dictates to automatically send all Azure storage accounts logs to third-party Security Information and Event Management (SIEM) solutions such as Splunk or IBM QRadar. And when you have critical applications and business processes that rely on Azure storage, you want to monitor those resources for their availability, performance, and operation.

Instead of remembering that you need to enable diagnostic settings after a new storage account is deployed, you can leverage Azure Event Hubs to stream and ingest storage account logs to your SIEM system and automate this process using Azure Policy.

Azure Storage | Diagnostic settings
Azure Storage | Diagnostic settings

Azure Event Hubs is a cloud-based, event-processing service that can receive and process millions of events per second. Event Hubs act as a front door for an event pipeline, to receive incoming data and store this data until processing resources are available.

At the time of this writing, Azure Storage logs are in public preview and are available in all public and US Government cloud regions. This enables logs for blobs (which includes Azure Data Lake Storage Gen2), files, queues, and tables. This feature is available for all storage accounts that are created with the Azure Resource Manager deployment model.

Please note that you can follow the same concept described in this article to send/archive storage account logs to another storage account and/or to Log Analytics workspace, or send to partner solution as well.

The integration of third-party (SIEM) solutions with Azure is out of the scope in this article.

Prerequisites

To follow this article, you need to have the following:

1) An Azure subscription. If you don’t have an Azure subscription, you can create a free one here.

2) An Event Hub namespace and an event hub in your Azure subscription. Learn how to create an event hub.

  • Take a note of the event hub namespace and the shared access policy name called (authorization rule), we will use them when we assign the custom policy definition.
  • The event hub authorization rule Id is in the following format: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.EventHub/namespaces/{eventHubNamespaceName}/AuthorizationRules/RootManageSharedAccessKey

3) One or more Azure storage accounts that are deployed in your subscription – To create a general-purpose v2 storage account, you can follow the instructions described here.

4) The Event Hub namespace and the storage accounts should be in the same region to enable diagnostic settings. You can create multiple Event Hub namespaces to cover all your storage accounts in different Azure regions.

Please note that you can enable and configure diagnostic settings by using the Azure Portal, PowerShell, the Azure CLI, an Azure Resource Manager template, or Azure Policy (the scope of this article).

Create Custom Policy definition

A quick overview of Azure Policy effects. Each policy definition that you create in the Azure Policy has a single effect. That effect determines what happens when the policy rule is evaluated to match. The effects behave differently if they are for a new resource, an updated resource, or an existing resource.

In this example, we need to set the diagnostics settings to stream to an event hub for newly deployed Azure storage accounts, as well as for an existing resource(s) in our environment. For this to work, we need to use the “DeployIfNotExists” policy. To understand how the Azure Policy effect works with the “DeployIfNotExists” policy definition, please check the official documentation from Microsoft.

The example below will evaluate all storage accounts to determine if Diagnostic Settings is enabled to stream to an event hub. If not, then a deployment to enable is executed for all the logs and metrics. At the time of this writing, the collection of the following Logs and Metrics are supported for the storage account, as well as for each individual storage service that is available on the account:

> Storage account:

  • Metrics: Transaction

> Blob service:

  • Logs: StorageRead | StorageWrite | StorageDelete
  • Metrics: Transaction

> File service:

  • Logs: StorageRead | StorageWrite | StorageDelete
  • Metrics: Transaction

> Table service:

  • Logs: StorageRead | StorageWrite | StorageDelete
  • Metrics: Transaction

> Queue service:

  • Logs: StorageRead | StorageWrite | StorageDelete
  • Metrics: Transaction

Open the Azure Portal, click “All services” and then search for “Policy” and then click on “Definitions” → “+ Policy definition”.

For the “Definition location“, select the location by clicking the ellipsis [] and select either a management group or a subscription.

In the “Name“ field, give a descriptive name for the policy definition such as – “Enable diagnostic settings for Azure Storage Accounts to Event Hub” and add a Description.

In the “Category” section, select the appropriate category for this policy. In this example, we will choose “Storage” as shown in the figure below.

New Policy Definition
New Policy Definition

For the “POLICY RULE“, paste the following policy definition in JSON format and then click “Save“.

For the “Role definitions“, the Contributor role is selected by default as part of the policy.

The roleDefinitionIds: b24988ac-6180-42a0-ab88-20f7382dd24c is “Contributor”

Get-AzRoleDefinition "Contributor"
Get-AzRoleDefinition "Contributor"
Get-AzRoleDefinition “Contributor”

The contributor role is required to set and enable managed identity on the resources.

Here is the complete Policy definition in JSON format:

{
    "mode": "All",
    "policyRule": {
      "if": {
        "allOf": [
            {
              "field": "type",
              "equals": "Microsoft.Storage/storageAccounts"
            },
            {
              "field": "name",
              "notLike": "diag*"
            },
            {
              "field": "location",
              "equals": "[parameters('Location')]"
            }
          ]
        },
      "then": {
        "effect": "[parameters('effect')]",
        "details": {
          "type": "Microsoft.Insights/diagnosticSettings",
          "roleDefinitionIds": [
            "/providers/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c"
          ],
          "existenceCondition": {
            "allOf": [
              {
                "field": "Microsoft.Insights/diagnosticSettings/metrics.enabled",
                "equals": "True"
              },
              {
                "field": "Microsoft.Insights/diagnosticSettings/logs.enabled",
                "equals": "True"
              },
              {
                "field": "Microsoft.Insights/diagnosticSettings/eventHubAuthorizationRuleId",
                "matchInsensitively": "[parameters('eventHubAuthorizationRuleId')]"
              }
            ]
          },
          "deployment": {
            "properties": {
              "mode": "incremental",
              "template": {
                "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
                "contentVersion": "1.0.0.0",
                "parameters": {
                  "servicesToDeploy": {
                    "type": "array"
                  },
                  "diagnosticsSettingNameToUse": {
                  "type": "string"
                 },
                  "resourceName": {
                    "type": "string"
                  },
                  "eventHubAuthorizationRuleId": {
                    "type": "string"
                  },
                  "location": {
                    "type": "string"
                  },
                  "Transaction": {
                    "type": "string"
                  },
                  "StorageRead": {
                    "type": "string"
                  },
                  "StorageWrite": {
                    "type": "string"
                  },
                  "StorageDelete": {
                    "type": "string"
                  }
                },
                "variables": {},
                "resources": [
                  {
                    "condition": "[contains(parameters('servicesToDeploy'), 'blobServices')]",
                    "type": "Microsoft.Storage/storageAccounts/blobServices/providers/diagnosticSettings",
                    "apiVersion": "2017-05-01-preview",
                    "name": "[concat(parameters('resourceName'), '/default/', 'Microsoft.Insights/', parameters('diagnosticsSettingNameToUse'))]",
                    "location": "[parameters('location')]",
                    "dependsOn": [],
                    "properties": {
                      "eventHubAuthorizationRuleId": "[parameters('eventHubAuthorizationRuleId')]",
                      "metrics": [
                        {
                          "category": "Transaction",
                          "enabled": "[parameters('Transaction')]"
                        }
                      ],
                      "logs": [
                        {
                          "category": "StorageRead",
                          "enabled": "[parameters('StorageRead')]"
                        },
                        {
                          "category": "StorageWrite",
                          "enabled": "[parameters('StorageWrite')]"
                        },
                        {
                          "category": "StorageDelete",
                          "enabled": "[parameters('StorageDelete')]"
                        }
                      ]
                    }
                  },
                  {
                    "condition": "[contains(parameters('servicesToDeploy'), 'fileServices')]",
                    "type": "Microsoft.Storage/storageAccounts/fileServices/providers/diagnosticSettings",
                    "apiVersion": "2017-05-01-preview",
                    "name": "[concat(parameters('resourceName'), '/default/', 'Microsoft.Insights/', parameters('diagnosticsSettingNameToUse'))]",
                    "location": "[parameters('location')]",
                    "dependsOn": [],
                    "properties": {
                      "eventHubAuthorizationRuleId": "[parameters('eventHubAuthorizationRuleId')]",
                      "metrics": [
                        {
                          "category": "Transaction",
                          "enabled": "[parameters('Transaction')]"
                        }
                      ],
                      "logs": [
                        {
                          "category": "StorageRead",
                          "enabled": "[parameters('StorageRead')]"
                        },
                        {
                          "category": "StorageWrite",
                          "enabled": "[parameters('StorageWrite')]"
                        },
                        {
                          "category": "StorageDelete",
                          "enabled": "[parameters('StorageDelete')]"
                        }
                      ]
                    }
                  },
                  {
                    "condition": "[contains(parameters('servicesToDeploy'), 'tableServices')]",
                    "type": "Microsoft.Storage/storageAccounts/tableServices/providers/diagnosticSettings",
                    "apiVersion": "2017-05-01-preview",
                    "name": "[concat(parameters('resourceName'), '/default/', 'Microsoft.Insights/', parameters('diagnosticsSettingNameToUse'))]",
                    "location": "[parameters('location')]",
                    "dependsOn": [],
                    "properties": {
                      "eventHubAuthorizationRuleId": "[parameters('eventHubAuthorizationRuleId')]",
                      "metrics": [
                        {
                          "category": "Transaction",
                          "enabled": "[parameters('Transaction')]"
                        }
                      ],
                      "logs": [
                        {
                          "category": "StorageRead",
                          "enabled": "[parameters('StorageRead')]"
                        },
                        {
                          "category": "StorageWrite",
                          "enabled": "[parameters('StorageWrite')]"
                        },
                        {
                          "category": "StorageDelete",
                          "enabled": "[parameters('StorageDelete')]"
                        }
                      ]
                    }
                  },
                  {
                    "condition": "[contains(parameters('servicesToDeploy'), 'queueServices')]",
                    "type": "Microsoft.Storage/storageAccounts/queueServices/providers/diagnosticSettings",
                    "apiVersion": "2017-05-01-preview",
                    "name": "[concat(parameters('resourceName'), '/default/', 'Microsoft.Insights/', parameters('diagnosticsSettingNameToUse'))]",
                    "location": "[parameters('location')]",
                    "dependsOn": [],
                    "properties": {
                      "eventHubAuthorizationRuleId": "[parameters('eventHubAuthorizationRuleId')]",
                      "metrics": [
                        {
                          "category": "Transaction",
                          "enabled": "[parameters('Transaction')]"
                        }
                      ],
                      "logs": [
                        {
                          "category": "StorageRead",
                          "enabled": "[parameters('StorageRead')]"
                        },
                        {
                          "category": "StorageWrite",
                          "enabled": "[parameters('StorageWrite')]"
                        },
                        {
                          "category": "StorageDelete",
                          "enabled": "[parameters('StorageDelete')]"
                        }
                      ]
                    }
                  },
                  {
                    "condition": "[contains(parameters('servicesToDeploy'), 'storageAccounts')]",
                    "type": "Microsoft.Storage/storageAccounts/providers/diagnosticSettings",
                    "apiVersion": "2017-05-01-preview",
                    "name": "[concat(parameters('resourceName'), '/', 'Microsoft.Insights/', parameters('diagnosticsSettingNameToUse'))]",
                    "location": "[parameters('location')]",
                    "dependsOn": [],
                    "properties": {
                      "eventHubAuthorizationRuleId": "[parameters('eventHubAuthorizationRuleId')]",
                      "metrics": [
                        {
                          "category": "Transaction",
                          "enabled": "[parameters('Transaction')]"
                        }
                      ]
                    }
                  }
                ],
                "outputs": {}
              },
              "parameters": {
                "diagnosticsSettingNameToUse": {
                  "value": "[parameters('diagnosticsSettingNameToUse')]"
               },
                "eventHubAuthorizationRuleId": {
                  "value": "[parameters('eventHubAuthorizationRuleId')]"
                },
                "Location": {
                  "value": "[field('location')]"
                },
                "resourceName": {
                  "value": "[field('name')]"
                },
                "Transaction": {
                  "value": "[parameters('Transaction')]"
                },
                "StorageDelete": {
                  "value": "[parameters('StorageDelete')]"
                },
                "StorageWrite": {
                  "value": "[parameters('StorageWrite')]"
                },
                "StorageRead": {
                  "value": "[parameters('StorageRead')]"
                },
                "servicesToDeploy": {
                  "value": "[parameters('servicesToDeploy')]"
                }
              }
            }
          }
        }
      }
    },
    "parameters": {
      "eventHubAuthorizationRuleId": {
        "type": "String",
        "metadata": {
            "displayName": "Event Hub Shared Access Policy Authorization Rule Id",
            "description": "Specify Event Hub Shared Access Policy Authorization Rule Id"
            }
        },
      "Location": {
       "type": "String",
        "metadata": {
        "displayName": "Resource Location",
        "description": "Resource Location must be the same as the Event Hub Location",
        "strongType": "location"
        }
      },
        "servicesToDeploy": {
        "type": "Array",
        "metadata": {
          "displayName": "Storage services to deploy",
          "description": "List of Storage services to deploy"
        },
        "allowedValues": [
          "storageAccounts",
          "blobServices",
          "fileServices",
          "tableServices",
          "queueServices"
        ],
        "defaultValue": [
          "storageAccounts",
          "blobServices",
          "fileServices",
          "tableServices",
          "queueServices"
        ]
      },
      "diagnosticsSettingNameToUse": {
        "type": "String",
        "metadata": {
          "displayName": "Setting name",
          "description": "Name of the diagnostic settings."
        },
        "defaultValue": "diagSetByAzPolicyEventHub"
      },
      "effect": {
        "type": "String",
        "metadata": {
          "displayName": "Effect",
          "description": "Enable or disable the execution of the policy"
        },
        "allowedValues": [
          "DeployIfNotExists",
          "Disabled"
        ],
        "defaultValue": "DeployIfNotExists"
      },
      "StorageDelete": {
        "type": "String",
        "metadata": {
          "displayName": "StorageDelete - Enabled",
          "description": "Whether to stream StorageDelete logs to the Log Analytics workspace - True or False"
        },
        "allowedValues": [
          "True",
          "False"
        ],
        "defaultValue": "True"
      },
      "StorageWrite": {
        "type": "String",
        "metadata": {
          "displayName": "StorageWrite - Enabled",
          "description": "Whether to stream StorageWrite logs to the Log Analytics workspace - True or False"
        },
        "allowedValues": [
          "True",
          "False"
        ],
        "defaultValue": "True"
      },
      "StorageRead": {
        "type": "String",
        "metadata": {
          "displayName": "StorageRead - Enabled",
          "description": "Whether to stream StorageRead logs to the Log Analytics workspace - True or False"
        },
        "allowedValues": [
          "True",
          "False"
        ],
        "defaultValue": "True"
      },
      "Transaction": {
        "type": "String",
        "metadata": {
          "displayName": "Transaction - Enabled",
          "description": "Whether to stream Transaction logs to the Log Analytics workspace - True or False"
        },
        "allowedValues": [
          "True",
          "False"
        ],
        "defaultValue": "True"
      }
    }
}

In this custom policy, we are looking for all storage accounts where their names do NOT start with (diag*), we need to exclude those storage accounts from logs and metrics, then we are checking if the “metrics” and “logs” are enabled, and finally we are checking if the destination is set to the event hub authorization rule Id that we specified.

The ExistenceCondition option in the template above is super useful for the Azure storage accounts that have already metrics and logs enabled. If this condition is true, the policy definition will skip the deployment defined in the “DeployIfNotExists” section.

To learn more about the ExistenceCondition option for the “DeployIfNotExists” properties, please check the official documentation here.

Assign Custom Policy definition

To assign the custom policy definition, take the following steps:

If you want to automate the deployment and assignment of Azure Policies via Azure DevOps instead of using the Azure Portal, then please check the following step-by-step guide.

Open the Azure Portal, click “All services” and then search for “Policy” and then click on “Assignments”. An assignment is a policy that has been assigned to take place within a specific scope.

Select “Assign policy” from the top of the “Policy | Assignments” page.

On the “Assign Policy” page, select the Scope by clicking the ellipsis [] and select either a management group or subscription. You can optionally select a resource group if you want. A scope determines what resources or grouping of resources the policy assignment gets enforced on. Then click Select at the bottom of the Scope page.

Select the Policy definition ellipsis [] to open the list of available definitions. Choose the custom policy that we created in the previous step. The Policy enforcement is Enabled by default. Click Next to continue.

On the Parameters page, specify the following parameters for this policy assignment:

> Event Hub Shared Access Policy Authorization Rule Id – Please refer to the prerequisites section to see how to get the Event Hub Authorization Rule Id.
> Resource Location – The Azure region where your storage accounts are deployed.
> Storage services to deploy – By default all 5 services are selected.
> Setting name – The diagnostic settings default name is set in the template.
> Effect – The default value is set to “DeployIfNotExists“, you can disable the effect of this policy later on by setting the effect to “Disabled“.
> StorageDelete – Enabled – The default value is set to “True“.
> StorageWrite – Enabled – The default value is set to “True“.
> StorageRead – Enabled – The default value is set to “True“.
> Transaction – Enabled – The default value is set to “True“.

Assign Policy - Parameters
Assign Policy – Parameters

Click Next to continue. In the Remediation page, by default “Create a Managed Identity” with system assigned is selected for you because existing resources can be updated via a remediation task after the policy is assigned only.

Choose the default location for the “System assigned identity location“, this is required because policies with the “deployIfNotExists” and “Modify” effect types need the ability to modify resources and edit the configuration on existing resources respectively. To do this, a managed identity will be created automatically for each policy assignment. This identity will also be given the “Contributor” permissions on the scope that you have selected earlier. Click Next to continue.

On the Non-compliance message page, set the desired message (e.g. Diagnostic Settings for Storage accounts should be enabled to Event Hub to monitor those resources). Then click the “Review + create” button.

Non-compliance message
Non-compliance message

Finally, click “Create” to create the assignment.

Verify Diagnostic settings for Storage accounts

To verify that the custom policy is deployed successfully, you need to wait for at least 30 minutes after a resource has been created or updated. The policy won’t be triggered immediately, this is by design. To trigger the policy compliance scan manually (immediately), open the cloud shell and run the following command:

$job = Start-AzPolicyComplianceScan -AsJob
$job | ft -AutoSize

The scan job will run in the background as shown in the figure below.

Trigger Azure Policy Compliance Evaluation Manually
Trigger Azure Policy Compliance Evaluation Manually

If you want to check Azure Policy compliance status and remediate non-compliant resources via Azure DevOps Pipelines, then please check the following step-by-step guide.

Behind the scene, Azure Policy will create a Remediation task as shown in the figure below. To remediate existing resources, select the desired policy and then click on the “Remediate” button.

Policy Remediation - Enable Diagnostic settings for Azure Storage Accounts
Policy Remediation – Enable Diagnostic settings for Azure Storage Accounts

Please note that during an evaluation cycle, the policy definition with a “DeployIfNotExists” effect that matches resources is marked as non-compliant only, but no action is taken on that resource. For this reason, the remediation task was created when you assign the policy definition as described in the previous section.

Open the Azure Portal, click “All services” and then search for “Policy” and then click on “Remediation”. In the Policy | Remediation page, select the Remediation tasks tab, and check the status of the auto-remediation task. It’s completed successfully without any error as shown in the figure below.

Azure Policy - Remediation tasks
Azure Policy – Remediation tasks

If we look at the Storage account resource under Monitoring | Diagnostic settings, we can see that the “Diagnostics status” status is Enabled for all storage services.

Diagnostics status
Diagnostics status

If we select any of the resources/storage services to view its diagnostic settings, we can see that Diagnostic settings are set to Event hub as shown in the figure below.

Diagnostic settings | Event hub
Diagnostic settings | Event hub

Switching to Event Hubs Namespace | Event Hubs, we can verify the insights for logs and metrics are created for (StorageRead,  StorageWrite, StorageDelete, and Transaction).

Event Hubs Namespace | Event Hubs
Event Hubs Namespace | Event Hubs

That’s it there you have it!

Summary

In this article, we showed you how to enable and configure diagnostic settings for storage accounts to Event Hub with Azure Policy, so you can monitor your critical applications and business processes that rely on Azure storage for their availability, performance, and operation.

Azure Policies are becoming increasingly popular, as they provide a wide range of management capabilities over Azure resources. Not only do they allow us to enforce policy rules to control Azure resources, but they can also enable us to automatically evaluate compliance and remediate non-compliant resources.

Do you want to learn more about Azure Storage including Azure Blobs and Azure File Shares? Make sure to check my recently published online course about Azure Storage Essential Training.

We hope you find this guide useful.

__
Thank you for reading my blog.

If you have any questions or feedback, please leave a comment.

-Charbel Nemnom-

Related Posts

Previous

Recover and Repair Microsoft SQL Corrupted Database

Web Application Security Testing – Types, Best Practices, and Checklist

Next

Let me know what you think, or ask a question...

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe to Stay in Touch

Never miss out on your favorite posts and our latest announcements!

The content of this website is copyrighted from being plagiarized!

You can copy from the 'Code Blocks' in 'Black' by selecting the Code.

Please send your feedback to the author using this form for any 'Code' you like.

Thank you for visiting!