Azure Policy is a service in Azure that you use to create, assign, and manage policies. These policies enforce different rules and effects over your resources, so those resources stay compliant with your corporate standards and service level agreements. Azure Policy meets this need by continuously evaluating your resources for non-compliance with assigned policies.
In this article, we will show you how to enable diagnostic settings for an Azure resource to an event hub using Azure Policy so you can send the data to external third-party SIEM systems.
Table of Contents
Introduction
You have a policy in your organization that dictates to automatically forward all Azure resources logs to third-party Security Information and Event Management (SIEM) solutions such as Splunk and IBM QRadar. Instead of remembering that you need to enable diagnostic settings after a new resource is deployed, you can leverage Azure Event Hubs to stream and integrate Azure Platform logs to your SIEM system and automate this process using Azure Policy.
Azure Event Hubs is a cloud-based, event-processing service that can receive and process millions of events per second. Event Hubs act as a front door for an event pipeline, to receive incoming data and store this data until processing resources are available.
Platform logs in Azure, including Activity logs (Azure Subscription), Resource logs (Azure Resources), and Azure Active Directory logs provide detailed diagnostic and auditing information for Azure resources and the Azure platform they depend on. The complete list of all the applicable Azure services that support resource (diagnostic) logs can be found in this document.
Please note that you can follow the same concept described in this article to forward Azure resource logs to a storage account and/or to Log Analytics workspace as well.
Prerequisites
To follow this article, you need to have the following:
1) An Azure subscription. If you don’t have an Azure subscription, you can create a free one here.
2) An Event Hub namespace and an event hub in your Azure subscription. Learn how to create an event hub.
- Take note of the event hub namespace and the shared access policy name called (authorization rule), we will use them when we assign the custom policy definition.
- The event hub authorization rule Id is in the following format: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.EventHub/namespaces/{eventHubNamespaceName}/AuthorizationRules/{authorizationRuleName}
3) One or more Azure resources that are deployed in your subscription. You can find the list of supported resource logs here. In this example, we have an Azure SQL database deployed.
4) The Event Hub namespace and the resources should be in the same region to enable diagnostic settings. You can create multiple Event Hub namespaces to cover all your workloads in different Azure regions.
Please note that integrating third-party (SIEM) solutions with Azure is out of the scope of this article.
Create a Custom Policy definition
A quick overview of Azure Policy effects. Each policy definition that you create in the Azure Policy has a single effect. That effect determines what happens when the policy rule is evaluated to match. The effects behave differently if they are for a new resource, an updated resource, or an existing resource.
In this example, we need to set the diagnostics settings to stream to an event hub for newly deployed resources, as well as for an existing resource(s) in our environment. For this to work, we need to use the “DeployIfNotExists” policy. To understand how the Azure Policy effect works with the “DeployIfNotExists” policy definition, please check the official documentation from Microsoft here.
The example below will evaluate SQL Server databases to determine if Diagnostic Settings is enabled to stream to an event hub. If not, then a deployment to enable is executed for all the logs and metrics. At the time of this writing, the collection of the following SQL data logs and metrics are supported:
- SQLInsights
- AutomaticTuning
- QueryStoreRuntimeStatistics
- QueryStoreWaitStatistics
- Errors
- DatabaseWaitStatistics
- Timeouts
- Blocks
- Deadlocks
- Basic
- InstanceAndAppAdvanced
- WorkloadManagement
Take now the following steps:
1) Open the Azure Portal, click “All services” and then search for “Policy” and then click on “Definitions” → “+ Policy definition”.
2) For the “Definition location“, select the location by clicking the ellipsis […] and select either a management group or subscription.
3) In the “Name“ field, give a descriptive name for the policy definition.
4) In the “Category” section, select the appropriate category for this policy. In this example, we will choose “SQL“.
5) For the “POLICY RULE“, paste the following policy definition and then click “Save“.
6) For the “Role definitions“, make sure that the Contributor is selected. The contributor role is required to set and enable diagnostic settings on the resources.
Here is the Policy definition in JSON format:
{
"mode": "All",
"parameters": {
"resourceLocation": {
"type": "string",
"metadata": {
"displayName": "Resource Location",
"description": "Resource Location must be the same as the Event Hub",
"strongType": "location"
}
},
"eventHubName": {
"type": "string",
"metadata": {
"displayName": "Event Hub Namespace",
"description": "Specify the namespace name of the Event Hub"
}
},
"eventHubAuthorizationRuleId": {
"type": "string",
"metadata": {
"displayName": "Event Hub Authorization Rule Id",
"description": "Specify Event Hub Authorization Rule Id"
}
}
},
"policyRule": {
"if": {
"allOf": [
{
"field": "type",
"equals": "Microsoft.Sql/servers/databases"
},
{
"field": "location",
"equals": "[parameters('resourceLocation')]"
}
]
},
"then": {
"effect": "deployIfNotExists",
"details": {
"type": "Microsoft.Insights/diagnosticSettings",
"roleDefinitionIds": [
"/providers/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c"
],
"existenceCondition": {
"allOf": [
{
"field": "Microsoft.Insights/diagnosticSettings/logs.enabled",
"equals": "True"
},
{
"field": "Microsoft.Insights/diagnosticSettings/metrics.enabled",
"equals": "True"
},
{
"field": "Microsoft.Insights/diagnosticSettings/eventHubAuthorizationRuleId",
"matchInsensitively": "[parameters('eventHubAuthorizationRuleId')]"
},
{
"field": "Microsoft.Insights/diagnosticSettings/eventHubName",
"matchInsensitively": "[parameters('eventHubName')]"
}
]
},
"deployment": {
"properties": {
"mode": "incremental",
"template": {
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"resourceName": {
"type": "string"
},
"eventHubName": {
"type": "string"
},
"eventHubAuthorizationRuleId": {
"type": "string"
},
"location": {
"type": "string"
}
},
"variables": {},
"resources": [
{
"type": "Microsoft.Sql/servers/databases/providers/diagnosticSettings",
"apiVersion": "2017-05-01-preview",
"name": "[concat(parameters('resourceName'), '/', 'Microsoft.Insights/diagSetByAzPolicyEventHub')]",
"location": "[parameters('location')]",
"dependsOn": [],
"properties": {
"eventHubName": "[parameters('eventHubName')]",
"eventHubAuthorizationRuleId": "[parameters('eventHubAuthorizationRuleId')]",
"metrics": [
{
"category": "AllMetrics",
"enabled": true,
"retentionPolicy": {
"days": 0,
"enabled": false
},
"timeGrain": null
}
],
"logs": [
{
"category": "SQLInsights",
"enabled": true
},
{
"category": "AutomaticTuning",
"enabled": true
},
{
"category": "QueryStoreRuntimeStatistics",
"enabled": true
},
{
"category": "QueryStoreWaitStatistics",
"enabled": true
},
{
"category": "Errors",
"enabled": true
},
{
"category": "DatabaseWaitStatistics",
"enabled": true
},
{
"category": "Timeouts",
"enabled": true
},
{
"category": "Blocks",
"enabled": true
},
{
"category": "Deadlocks",
"enabled": true
}
]
}
}
],
"outputs": {}
},
"parameters": {
"eventHubName": {
"value": "[parameters('eventHubName')]"
},
"eventHubAuthorizationRuleId": {
"value": "[parameters('eventHubAuthorizationRuleId')]"
},
"location": {
"value": "[field('location')]"
},
"resourceName": {
"value": "[field('fullName')]"
}
}
}
}
}
}
}
}
The ExistenceCondition option in the template above is super useful for the Azure resource(s) that has already diagnostics settings connected to Event Hub. If this condition is true, the policy definition will skip the deployment defined in the “DeployIfNotExists” section in the template. To learn more about the ExistenceCondition option for the “DeployIfNotExists” properties, please check the documentation here.
Assign Custom Policy definition
To assign the custom policy definition, take the following steps:
1) Open the Azure Portal, click “All services” and then search for “Policy” and then click on “Assignments”. An assignment is a policy that has been assigned to take place within a specific scope.
2) Select “Assign policy” from the top of the “Policy | Assignments” page.
3) On the “Assign Policy” page, select the Scope by clicking the ellipsis […] and select either a management group or subscription. You can optionally select a resource group. A scope determines what resources or grouping of resources the policy assignment gets enforced on. Then click Select at the bottom of the Scope page.
4) Select the Policy definition ellipsis […] to open the list of available definitions. Choose the custom policy that we created in the previous step. The Policy enforcement is Enabled by default. Click Next to continue.
5) In the Parameters page, specify the parameters for this policy assignment (Resource Location, Event Hub Namespace, and Event Hub Authorization Rule Id). Please refer to the prerequisites section to see how to get the Event Hub Authorization Rule Id. Click Next to continue.
6) In the Remediation page, by default “Create a Managed Identity” is selected for you because existing resources can be updated via a remediation task after the policy is assigned only. Choose the default location for the “Managed Identity“, this is required because policies with the “deployIfNotExists” and “Modify” effect types need the ability to modify resources and edit the configuration on existing resources respectively. To do this, a managed identity will be created automatically for each policy assignment. Click Next to continue.
7) In the Review + Create page, review the settings and then click “Create“.
Verify Custom Policy definition
To verify that the custom policy is deployed successfully, you need to wait for at least 15 minutes after a resource has been created or updated. The policy won’t be triggered immediately, this is by design. Behind the scene, Azure Policy will create a Remediation task as shown in the following screenshot.
Please note that during an evaluation cycle, the policy definition with a “DeployIfNotExists” effect that matches resources is marked as non-compliant only, but no action is taken on that resource. For this reason, the remediation task was created when you assign the policy definition as described in the previous section.
Open the Azure Portal, click “All services” and then search for “Policy” and then click on “Remediation”. In the Policy | Remediation page, select the Remediation tasks tab, and check the status of the auto-remediation task. It’s completed successfully without any errors.
If you look at the SQL database resource under Monitoring | Diagnostic settings, you can see that diagnostic settings are enabled to Event Hub.
If you click on “Edit setting“, you will see all the logs and metrics are enabled and the destination is set to stream to an event hub.
If we look at the compliance policy state, we will see that all SQL databases in my environment are compliant and enabled for diagnostic settings.
That’s it there you have it!
Transform DeployIfNotExists Policy to AuditIfNotExists
A recent question that has been asked by one of my readers is the following:
We need to transform the effect for a Built-in Policy which goes like “Deploy Diagnostic Settings for Key Vault to Event Hub” and a few others, the current Effect is “DeployIfNotExists” and we need the policy to only have “AuditIfNotExists”.
Now, there is no direct or quick conversion tool that transforms policies between these two effects.
What you need to do is, evaluate each policy individually that you want to transform, and then remove the “deployment” section part where the policy actually deploys and changes the configuration.
At a high level, you need to remove and update the following sections in the “DeployIfNotExists” policy to convert it to “AuditIfNotExists”:
- Change the policy mode from “Indexed” to “All“. The mode determines which resource types are evaluated for a policy definition. The “mode”: “Indexed” should only be applied to resources that support tags and locations, and “mode”: “All” evaluate resource groups, subscriptions, and all resource types.
- Remove the “roleDefinitionIds”: [ ] section because we don’t need to assign permissions to remediate or take corrective actions by using remediation tasks.
- Remove the “deployment”: { } section including all the sub-sections “properties”: { }, “resources”: [ ], and “parameters”: { }.
- You also need to update the outer “parameters”: { } section of the policy to remove or specify what you want to evaluate under the “existenceCondition”: { } when you assign the policy.
- You need to update the effect from “DeployIfNotExists” to “auditIfNotExists” as we need to audit and not deploy. Pay attention to the “a” of the “auditIfNotExists” effect, it should be in lowercase.
- Finally, you need to create a new custom policy definition and assign it to the appropriate scope.
Understanding the effect of “DeployIfNotExists” and “AuditIfNotExists” is crucial to transform the policy correctly.
As documented by Microsoft (understand Azure policy effects), AuditIfNotExists enables auditing of resources related to the resource that matches the “if” condition but doesn’t have the properties specified in the “details” of the “then” condition. Similar to AuditIfNotExists, a DeployIfNotExists policy definition executes a template “deployment” when the condition is met.
Here is an example of transforming the Built-in “Deploy Diagnostic Settings for Key Vault to Event Hub” policy to “Audit if Diagnostic Settings is enabled for Key Vault to Event Hub“:
{
"mode": "All",
"policyRule": {
"if": {
"allOf": [
{
"field": "type",
"equals": "Microsoft.KeyVault/vaults"
},
{
"anyOf": [
{
"value": "[parameters('eventHubLocation')]",
"equals": ""
},
{
"field": "location",
"equals": "[parameters('eventHubLocation')]"
}
]
}
]
},
"then": {
"effect": "[parameters('effect')]",
"details": {
"type": "Microsoft.Insights/diagnosticSettings",
"existenceCondition": {
"allOf": [
{
"count": {
"field": "Microsoft.Insights/diagnosticSettings/logs[*]",
"where": {
"field": "Microsoft.Insights/diagnosticSettings/logs[*].enabled",
"equals": "[parameters('logsEnabled')]"
}
},
"greaterOrEquals": 1
},
{
"count": {
"field": "Microsoft.Insights/diagnosticSettings/metrics[*]",
"where": {
"field": "Microsoft.Insights/diagnosticSettings/metrics[*].enabled",
"equals": "[parameters('metricsEnabled')]"
}
},
"greaterOrEquals": 1
}
]
}
}
}
},
"parameters": {
"effect": {
"type": "String",
"metadata": {
"displayName": "Effect",
"description": "Enable or disable the execution of the policy"
},
"allowedValues": [
"auditIfNotExists",
"Disabled"
],
"defaultValue": "auditIfNotExists"
},
"eventHubLocation": {
"type": "String",
"metadata": {
"displayName": "Event Hub Location",
"description": "Optional - The location the Event Hub resides in. Only Key Vaults in this location will be checked.",
"strongType": "location"
},
"defaultValue": ""
},
"metricsEnabled": {
"type": "String",
"metadata": {
"displayName": "Enable metrics",
"description": "Whether to check if metrics stream to the Event Hub is enabled - True or False"
},
"allowedValues": [
"True",
"False"
],
"defaultValue": "True"
},
"logsEnabled": {
"type": "String",
"metadata": {
"displayName": "Enable logs",
"description": "Whether to check if logs stream to the Event Hub is enabled - True or False"
},
"allowedValues": [
"True",
"False"
],
"defaultValue": "True"
}
}
}When you assign the policy, you can set the following parameters (Policy Effect, Event Hub Location, Enable metrics, and Enable logs) to evaluate and audit against the specified parameters as shown in the figure below.
Summary
In this article, we showed you how to create and deploy an Azure custom Policy definition to automate and enable Azure resource diagnostic settings to stream to an Event Hub.
Please note that you need to have a separate custom policy for each Azure resource that you want to enable diagnostic settings and stream to Event Hub. You could create all the needed policy definitions for your environment and then add them all to an initiative definition and then assign the initiative scope based on Management Group, Subscription, or Resource Group. I highly recommend using Management Groups if you are not doing so already.
To learn more about Azure Policy, please check the official documentation from Microsoft here.
To learn more about Azure Event Hub, please check the official documentation from Microsoft here.
__
Thank you for reading my blog.
If you have any questions or feedback, please leave a comment.
-Charbel Nemnom-
Hi Charbel, I wanted to test the policy, could you please share the json document on git / googledrive or any by any means possible, unable to copy the contents. That would be really helpful.
Hello Vinod,
Thanks for your message.
You can get a copy of the JSON policy from here.
Let me know if it works for you.
Thanks!
Many Thanks, it worked !
Hi Charbel,
How to transform the “deployifnotexist” policy to “auditifnotexist”.
What things do we need to clutter from policy and what we should keep?
Please address.
Thanks
Hello Vipul, thank you for the comment!
This is a great question.
I have added a new section to address how to transform the DeployIfNotExists Policy to AuditIfNotExists.
Please check it and let me know if it works for you.
Thanks!
Good article. This details how you specify the Event Hub Namespace, but how would you go about changing the policy so that the diagnostic settings go into a specific Event Hub behind that namespace? (Where you don’t want Azure to create a hub name of its own choosing) ?
Hello Daz, thanks for the comment and the great question!
Yes, you can create a custom Event Hub Instance within the Event Hub Namespace, then go to the Event Hub that you created and add a new SAS authorization policy.
When you assign the policy as described in this section, in the Parameters Tab, set the following Event Hub Authorization Rule Id, you need to update the variables between { } to match your environment.
Hope it helps!
Hi Charbel,
Let’s say if I have multiple resources in various regions and I have event hub deployed for all those various regions. Then is it possible to send diagnostic logs of a resource to the event hub which is deployed in the same region and ultimately send it to SIEM. Let’s say I want to enable all the logs and metrics for all the resources. If this is possible, could you please provide a sample.
Hello Ashish, thanks for the comment!
Yes, this is completely possible, however, you need to have a single policy for each resource/service that you have deployed in your environment, then send the logs and metrics to Azure Event Hub.
In this article, I shared an example policy for Azure SQL database, you need to create a policy for other resources as well (E.g. Azure Firewall, API Management, App Gateway, etc.).
One way that you can use to accelerate this process, is to use the Azure Diagnostics Policy Generator script to create the policies for the resource types you have within your Azure Subscription.
The script will create Azure Custom Policies for Azure resource types that support Azure Diagnostics logs and metrics. The Policies can be created for both Event Hub and Log Analytics as target.
Hope it helps!
Hi Charbel, thank you for your response!
It is quite helpful and knowledgeable but I have one more query, is it possible to deploy a policy at either the Management Group or Subscription Level for the above ask by using tags or something else? For example, we use a location tag in every resource and azure policy automatically creates and routes diagnostic logs to that specific region’s event hub. Any sample would be much appreciated!
Hello Ashish, yes sure you can assign the policy at the Management Group or Subscription Level.
I recommend at the Management Group level if possible.
Then you can update the policy definition under the “policyRule” section, to add the following field to match the Tag and Value that you want:
Then under the “parameters” section at the beginning of the policy definition, you need to add the following inclusionTagName and inclusionTagValue:
Hope it helps!
While applying policies we get option to select discussion only… How to apply policies at Management Group level?
Hello Vipul, yes you can assign the policy at the Management Group level.
But to do that, you must first create the Policy definition OR the Initiative definition and Set the (Policy/Initiative location) at the Management Group Level.
For example, if you create a policy definition at the management group level, a policy assignment of that definition can be scoped down to a subscription level within that management group.
Once you do that, then you can assign and apply the policies at Management Group level.
From a permissions perspective, you need to make sure that you have the Resource Policy Contributor role assigned at the Management Group level.
Hope it helps!