Enable Azure Resource Diagnostic Settings To Event Hub Using Azure Policy

8 min read

Introduction

Azure Policy is a service in Azure that you use to create, assign, and manage policies. These policies enforce different rules and effects over your resources, so those resources stay compliant with your corporate standards and service level agreements. Azure Policy meets this need by continuously evaluating your resources for non-compliance with assigned policies.

For example, you have a policy in your organization that dictates to automatically forward all azure resources logs to third-party Security Information and Event Management (SIEM) solutions such as Splunk and IBM QRadar. Instead of remembering that you need to enable diagnostic settings after a new resource is deployed, you can leverage Azure Event Hubs to stream and integrate Azure Platform logs to your SIEM system and automate this process using Azure Policy.

Azure Event Hubs is a cloud-based, event-processing service which can receive and process millions of events per second. Event Hubs acts as a front door for an event pipeline, to receive incoming data and stores this data until processing resources are available.

Platform logs in Azure, including Activity logs (Azure Subscription), Resource logs (Azure Resources), and Azure Active Directory logs provide detailed diagnostic and auditing information for Azure resources and the Azure platform they depend on. For the complete list of all the applicable Azure services that support resource (diagnostic) logs can be found in this document.

Please note that you can follow the same concept described in this article to forward Azure resource logs to a storage account and/or to Log Analytics workspace as well.

In this article, I will show you how to enable diagnostic settings for an Azure resource to an event hub using Azure Policy so you can send the data to an external third-party SIEM systems.

Prerequisites

To follow this article, you need to have the following:

  1. An Azure subscription. If you don’t have an Azure subscription, you can create a free one here.
  2. An Event Hub namespace and an event hub in your Azure subscription. Learn how to create an event hub.
    • Take a note of the event hub namespace and the shared access policy name called (authorization rule), we will use them when we assign the custom policy definition.
    • The event hub authorization rule Id is in the following format: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.EventHub/namespaces/{eventHubNamespaceName}/AuthorizationRules/{authorizationRuleName}
  3. One or more Azure resource that is deployed in your subscription. You can find the list of supported resource logs here. In this example, I have an Azure SQL database deployed.
  4. The Event Hub namespace and the resources should be in the same region to enable diagnostic settings. You can create multiple Event Hub namespaces to cover all your workloads in different Azure regions.

Please note that integrating third-party (SIEM) solutions with Azure is out of the scope in this article.

Create Custom Policy definition

A quick overview of Azure Policy effects. Each policy definition that you create in the Azure Policy has a single effect. That effect determines what happens when the policy rule is evaluated to match. The effects behave differently if they are for a new resource, an updated resource, or an existing resource.

For this example, we need to set the diagnostics settings to stream to an event hub for newly deployed resources, as well as for an existing resource(s) in our environment. For this to work, we need to use “DeployIfNotExists” effect policy. To understand how Azure Policy effect works with “DeployIfNotExists” policy definition, please check the official documentation from Microsoft here.

The example below will evaluates SQL Server databases to determine if Diagnostic Settings is enabled to stream to an event hub. If not, then a deployment to enable is executed for all the logs and metrics. At the time of this writing, the collection of the following SQL data logs and metric are supported:

  • SQLInsights
  • AutomaticTuning
  • QueryStoreRuntimeStatistics
  • QueryStoreWaitStatistics
  • Errors
  • DatabaseWaitStatistics
  • Timeouts
  • Blocks
  • Deadlocks
  • Basic
  • InstanceAndAppAdvanced
  • WorkloadManagement

Take now the following steps:

  1. Open the Azure Portal, click “All services” and then search for “Policy” and then click on “Definitions” → “+ Policy definition”.
  2. For the “Definition location“, select the location by clicking the ellipsis [] and select either a management group or subscription.
  3. In the “Name“ field, give a descriptive name for the policy definition.
  4. In the “Category” section, select the appropriate category for this policy. In this example, I will choose “SQL“.
  5. For the “POLICY RULE“, paste the following policy definition and then click “Save“.
  6. For the “Role definitions“, make sure that the Contributor is selected. The contributor role is required to set and enable diagnostic settings on the resources.

Here is the Policy definition in JSON format:

{
    "mode": "All",
    "parameters": {
            "resourceLocation": {
                "type": "string",
                "metadata": {
                    "displayName": "Resource Location",
                    "description": "Resource Location must be the same as the Event Hub",
                    "strongType": "location"
                }
            },
            "eventHubName": {
                "type": "string",
                "metadata": {
                    "displayName": "Event Hub Namespace",
                    "description": "Specify the namespace name of the Event Hub"
                }
            },
            "eventHubAuthorizationRuleId": {
                "type": "string",
                "metadata": {
                    "displayName": "Event Hub Authorization Rule Id",
                    "description": "Specify Event Hub Authorization Rule Id"
                }
            }
        },
        "policyRule": {
            "if": {
                "allOf": [
                    {
                        "field": "type",
                        "equals": "Microsoft.Sql/servers/databases"
                    },
                    {
                        "field": "location",
                        "equals": "[parameters('resourceLocation')]"
                    }
                ]
            },
            "then": {
                "effect": "deployIfNotExists",
                "details": {
                    "type": "Microsoft.Insights/diagnosticSettings",
                    "roleDefinitionIds": [
                        "/providers/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c"
                    ],
                    "existenceCondition": {
                        "allOf": [
                            {
                                "field": "Microsoft.Insights/diagnosticSettings/logs.enabled",
                                "equals": "True"
                            },
                            {
                                "field": "Microsoft.Insights/diagnosticSettings/metrics.enabled",
                                "equals": "True"
                            },
                            {
                                "field": "Microsoft.Insights/diagnosticSettings/eventHubAuthorizationRuleId",
                                "matchInsensitively": "[parameters('eventHubAuthorizationRuleId')]"
                            },
                            {
                                "field": "Microsoft.Insights/diagnosticSettings/eventHubName",
                                "matchInsensitively": "[parameters('eventHubName')]"
                            }
                        ]
                    },
                    "deployment": {
                        "properties": {
                            "mode": "incremental",
                            "template": {
                                "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
                                "contentVersion": "1.0.0.0",
                                "parameters": {
                                    "resourceName": {
                                        "type": "string"
                                    },
                                    "eventHubName": {
                                        "type": "string"
                                    },
                                    "eventHubAuthorizationRuleId": {
                                        "type": "string"
                                    },
                                    "location": {
                                        "type": "string"
                                    }
                                },
                                "variables": {},
                                "resources": [
                                    {
                                        "type": "Microsoft.Sql/servers/databases/providers/diagnosticSettings",
                                        "apiVersion": "2017-05-01-preview",
                                        "name": "[concat(parameters('resourceName'), '/', 'Microsoft.Insights/diagSetByAzPolicyEventHub')]",
                                        "location": "[parameters('location')]",
                                        "dependsOn": [],
                                        "properties": {
                                            "eventHubName": "[parameters('eventHubName')]",
                                            "eventHubAuthorizationRuleId": "[parameters('eventHubAuthorizationRuleId')]",
                                            "metrics": [
                                                {
                                                    "category": "AllMetrics",
                                                    "enabled": true,
                                                    "retentionPolicy": {
                                                        "days": 0,
                                                        "enabled": false
                                                    },
                                                    "timeGrain": null
                                                }
                                            ],
                                            "logs": [
                                                {
                                                    "category": "SQLInsights",
                                                    "enabled": true
                                                },
                                                {
                                                    "category": "AutomaticTuning",
                                                    "enabled": true
                                                },
                                                {
                                                    "category": "QueryStoreRuntimeStatistics",
                                                    "enabled": true
                                                },
                                                {
                                                    "category": "QueryStoreWaitStatistics",
                                                    "enabled": true
                                                },
                                                {
                                                    "category": "Errors",
                                                    "enabled": true
                                                },
                                                {
                                                    "category": "DatabaseWaitStatistics",
                                                    "enabled": true
                                                },
                                                {
                                                    "category": "Timeouts",
                                                    "enabled": true
                                                },
                                                {
                                                    "category": "Blocks",
                                                    "enabled": true
                                                },
                                                {
                                                    "category": "Deadlocks",
                                                    "enabled": true
                                                }
                                            ]
                                        }
                                    }
                                ],
                                "outputs": {}
                            },
                            "parameters": {
                                "eventHubName": {
                                    "value": "[parameters('eventHubName')]"
                                },
                                "eventHubAuthorizationRuleId": {
                                    "value": "[parameters('eventHubAuthorizationRuleId')]"
                                },
                                "location": {
                                    "value": "[field('location')]"
                                },
                                "resourceName": {
                                    "value": "[field('fullName')]"
                            }
                         }
                    }
                }
             }
        }
    }
}

The ExistenceCondition option in the template above is super useful for the Azure resource(s) that has already diagnostics settings connected to Event Hub. If this condition is true, the policy definition will skip the deployment defined in the “DeployIfNotExists” section in the template. To learn more about ExistenceCondition option for the “DeployIfNotExists” properties, please check the documentation here.

Assign Custom Policy definition

To assign the custom policy definition, take the following steps:

  1. Open the Azure Portal, click “All services” and then search for “Policy” and then click on “Assignments”. An assignment is a policy that has been assigned to take place within a specific scope.
  2. Select “Assign policy” from the top of the “Policy | Assignments” page.
  3. On the “Assign Policy” page, select the Scope by clicking the ellipsis [] and select either a management group or subscription. You can optionally select a resource group. A scope determines what resources or grouping of resources the policy assignment gets enforced on. Then click Select at the bottom of the Scope page.
  4. Select the Policy definition ellipsis [] to open the list of available definitions. Choose the custom policy that we created in the previous step. The Policy enforcement is Enabled by default. Click Next to continue.
  5. In the Parameters page, specify the parameters for this policy assignment (Resource Location, Event Hub Namespace, and Event Hub Authorization Rule Id). Please refer to the prerequisites section to see how to get the Event Hub Authorization Rule Id. Click Next to continue.
  6. In the Remediation page, select “Create a remediation task“, because existing resources can be updated via a remediation task after the policy is assigned only. This is a very important step, otherwise the existing resources will be marked as non-compliant without any auto-remediation. Choose the default location for the “Managed Identity“, this is required because policies with the “deployIfNotExists” and “Modify” effect types need the ability to deploy resources and edit tags on existing resources respectively. To do this, a managed identity will be created automatically for each policy assignment. Click Next to continue.
  7. In the Review + Create page, review the settings and then click “Create“.

Verify Custom Policy definition

To verify that the custom policy is deployed successfully, you need to wait for at least 15 minutes after a resource has been created or updated. The policy won’t be triggered immediately, this is by design. Behind the scene, Azure Policy will create a Remediation task as shown in the following screenshot.

Please note that during an evaluation cycle, the policy definition with a “DeployIfNotExists” effect that match resources are marked as non-compliant only, but no action is taken on that resource. For this reason, you need to create a remediation task when you assign the policy definition as described in the previous section.

Open the Azure Portal, click “All services” and then search for “Policy” and then click on “Remediation”. In the Policy | Remediation page, select Remediation tasks tab, and check the status of the auto-remediation task. It’s completed successfully without any error.

If you look at the SQL database resource under Monitoring | Diagnostic settings, you can see that diagnostic settings is enabled to Event Hub.

If you click on “Edit setting“, you will see all the logs and metrics are enabled and the destination is set to stream to an event hub.

If we look at the compliance policy state, we will see that all SQL databases in my environment are compliant and enabled for diagnostic settings.

That’s it there you have it!

Summary

In this article, I showed you how to create and deploy an Azure custom Policy definition to automate and enable Azure resource diagnostic settings to stream to an Event Hub.

Please note that you need to have a separate custom policy for each Azure resource that you want to enable diagnostic settings and stream to Event Hub. What you could do is to create all the needed policy definitions for your environment and then add them all to an initiative definition and then assign the initiative scope based on Management Group or Subscription, I highly recommend to use Management Groups if you are not doing so already.

To learn more about Azure Policy, please check the official documentation from Microsoft here.

To learn more about Azure Event Hub, please check the official documentation from Microsoft here.

__
Thank you for reading my blog.

If you have any questions or feedback, please leave a comment.

-Charbel Nemnom-

About Charbel Nemnom 545 Articles
Charbel Nemnom is a Cloud Architect, ICT Security Expert, Microsoft Most Valuable Professional (MVP), and Microsoft Certified Trainer (MCT), totally fan of the latest's IT platform solutions, accomplished hands-on technical professional with over 17 years of broad IT Infrastructure experience serving on and guiding technical teams to optimize the performance of mission-critical enterprise systems. Excellent communicator is adept at identifying business needs and bridging the gap between functional groups and technology to foster targeted and innovative IT project development. Well respected by peers through demonstrating passion for technology and performance improvement. Extensive practical knowledge of complex systems builds, network design, business continuity, and cloud security.

Be the first to comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.