Azure Policy Compliance and Remediation via Azure DevOps

8 Min. Read

Today, a common theme in cloud environments is enforcing organizational standards and adopting cloud governance since day one. And this is very important since it will give you the ability to define policies, processes, and procedures. These policies then dictate what can be done and verify that what does exist is correct. A service from Microsoft called Azure Policy is a great way to make that happen and take corrective action.

This article will demonstrate how to check Azure Policy compliance status and remediate non-compliant resources via Azure DevOps Pipelines.

Introduction

Azure Policy is a service in Azure that you use to create, assign, and manage policies. These policies enforce different rules and effects over your resources, so those resources stay compliant with your corporate standards and service level agreements. Azure Policy meets this need by continuously evaluating your resources for non-compliance with assigned policies.

If you develop many Azure Policy definitions with ARM templates as I do, you find that using the Azure Portal is not efficient to deploy, remediate and check the compliance status of those policies. The good news is, we could leverage Azure DevOps to perform such policy operations.

Azure DevOps — formerly known as Visual Studio Team Services (VSTS)— is a Software as a service (SaaS) platform from Microsoft that provides an end-to-end DevOps toolchain for developing and deploying software. It also integrates with most leading tools on the market and is a great option for orchestrating a DevOps toolchain.

In this guide, I will walk you step-by-step on how to check Azure Policy compliance status and remediate non-compliant resources via Azure DevOps Pipelines.

Before you continue with the remainder of this article, please make sure to check the first part on how to deploy and assign Azure policy via Azure DevOps Pipelines.

Prerequisites

To follow this article, you need to have the following:

1) Azure subscription – If you don’t have an Azure subscription, you can create a free one here.

2) Azure DevOps – If you don’t have one, you can create a free one here.

3) Azure DevOps Organization/Project – If you don’t have one, follow these steps to create an organization.

4) Azure DevOps Git repository – If you don’t have one, follow these steps to create a new Git repository in your project.

5) Azure DevOps service connection for Azure Pipelines – The service connection (principal) can be created at the Subscription or Management Group level. I highly recommend using Management Groups where possible. To create a service connection, please follow the steps described here.

6) Last but not least, you need to give the appropriate permissions for the Azure DevOps service principal to manage Azure Policy (please check the following article on how to grant and assign the ‘Resource Policy Contributor’ to the Azure DevOps service principal).

Assuming you have all the prerequisites in place, take now the following steps:

Create a new variable group

In this section, we will create three variables (ClientID, ClientSecret, and TenantID) that we need to use in our YAML pipeline in the next section.

Variable groups store values and secrets that you might want to be passed into a YAML pipeline or make available across multiple pipelines. You can share and use variables groups in multiple pipelines in the same project.

Sign in to the Azure DevOps portal, select your organization and take the following steps:

On the left-hand side of the blade, select Library under Pipelines as shown in the figure below, and then click on +Variable group.

Create a new variable group
Create a new variable group

Next, you need to enter a name and description for the variable group.

Enter the name and value for each variable to include in the group, choosing + Add for each one. I highly recommend encrypting and securely store the values, choose the “lock” icon at the end of the row. When you’re finished adding variables, select Save.

Enter secret values for each variable
Enter secret values for each variable

The Client ID and Client Secret is the service principal that we created as part of the prerequisites. To get the service principal (Client ID and Client Secret), you can navigate to the service connection page in Azure DevOps (Select Project settings > Service connections) and then click on ‘Manage Service Principal‘ as shown in the figure below.

Manage Service Principal
Manage Service Principal

The enterprise application page will open where you can copy the Application (client ID and create a secret) values.

You can get the Tenant ID from your Azure Active Directory (AAD) overview page.

Create a new pipeline

In this section, we will create a new pipeline for policy compliance and remediation.

Sign in to the Azure DevOps portal and take the following steps:

On the left-hand side of the blade, select Pipelines under Pipelines as shown in the figure below and then click on All. Select New folder and enter a descriptive name for the folder (e.g. Azure Policy Compliance & Remediate), and then click Create.

Pipelines - Create a new folder
Pipelines – Create a new folder

A new folder will be created. Expand the folder name and then select ‘Create a new pipeline here‘ as shown in the figure below.

Pipelines - Create a new pipeline
Pipelines – Create a new pipeline

The new pipeline wizard will open where you can choose the location of your code. In this example, we are using Aure Repos Git.

Select Aure Repos Git, and then select your Git repository. Select ‘Starter pipeline‘ as shown in the figure below (we need to update it, more on this in a bit).

Configure your pipeline
Configure your pipeline

You can also do it the other way around. You copy and save the YAML file below in your Git repository, and then configure your pipeline by selecting ‘Existing Azure Pipelines YAML file‘ and select the .yml file instead.

Delete the starter pipeline code and replace it with the following. Please note that you need to update the following variables based on your environment.

1) variables: -group: <variable group name>

2) stage: PolicyCompliance -filePath: ‘<your git repository folder>\get-policycompliance.ps1′

3) stage: RemediatePolicy -filePath: ‘<your git repository folder>\remediate-policy.ps1′

Please check the next section for the ‘get-policycompliance.ps1‘ and ‘remediate-policy.ps1‘ task files.

trigger: none

parameters:
- name: type
  displayName: Scope
  type: string
  values: 
  - Subscription
  - ManagementGroup
  - ResourceGroup

- name: id
  displayName: Id
  type: string

variables:
- group: CharbelNemnom

stages:
- stage: PolicyCompliance
  jobs:  
  - job: GetPolicyCompliance
    pool: 
      vmImage: 'windows-2019'
    steps:
    - task: PowerShell@2
      inputs:
          filePath: 'AZ-PolicyCompliance\get-policycompliance.ps1'
          pwsh: true
          arguments: '-type "${{ parameters.type }}" -id "${{ parameters.id }}" -TenantID $(TenantID) -AppID $(ClientID) -AppSecret $(ClientSecret)'
      displayName: 'Run PowerShell Script'

- stage: Validation
  jobs:  
  - job: waitForValidation
    displayName: Wait for external validation  
    pool: server
    timeoutInMinutes: 4320
    steps:
    - task: ManualValidation@0
      timeoutInMinutes: 1440
      inputs:
        notifyUsers: |
          
        instructions: 'Please validate the build configuration and resume'
        onTimeout: 'reject'

- stage: RemediatePolicy
  jobs:  
  - job: RemediatePolicy
    pool: 
      vmImage: 'windows-2019'
    steps:
    - task: PowerShell@2
      inputs:
          filePath: 'AZ-PolicyCompliance\remediate-policy.ps1'
          pwsh: true
          arguments: '-type "${{ parameters.type }}" -id "${{ parameters.id }}" -TenantID $(TenantID) -AppID $(ClientID) -AppSecret $(ClientSecret)'
      displayName: 'Run PowerShell Script'

When you’re finished adding and updating the code, select Save as shown in the figure below.

Review and save your pipeline
Review and save your pipeline

Finally, enter a commit message, extended description as optional, and then click Save.

Get Azure Policy Compliance stage

In this step, we will define and create our first stage to get Azure Policy states summary for resources.

Select and copy the code below and save it in a file named: get-policycompliance.ps1

param(
    [Parameter(Mandatory = $false)] [string] $type,
    [Parameter(Mandatory = $false)] [string] $id,
    [Parameter(Mandatory = $true)] [string] $AppID,
    [Parameter(Mandatory = $true)] [string] $AppSecret,
    [Parameter(Mandatory = $true)] [string] $tenantid
)

# .Import PowerShell Modules
Get-ChildItem -Path C:\Modules\az_*\Az.Accounts -Filter "*.psd1" -Recurse | Import-Module
Get-ChildItem -Path C:\Modules\az_*\Az.Resources -Filter "*.psd1" -Recurse | Import-Module
Get-ChildItem -Path C:\Modules\az_*\Az.PolicyInsights -Filter "*.psd1" -Recurse | Import-Module

# .Authentification
$CredApp = New-Object pscredential -ArgumentList ($AppID, ($AppSecret | ConvertTo-SecureString -Force -AsPlainText))
$Login = Login-AzAccount -TenantId $tenantid -Credential $CredApp -ServicePrincipal

switch ($type) {
    subscription { 
        Write-Output "This is the latest policy compliance states generated in the last day for all resources within the specified Subscription Id: $id"
        Set-AzContext -SubscriptionId $id
        Get-AzPolicyStateSummary -SubscriptionId $id
        }
    managementgroup {
        Write-Output "This is the latest policy compliance states generated in the last day for all resources within the specified Management Group Id: $id"
        Get-AzPolicyStateSummary -ManagementGroupName $id 
        }
    resourcegroup { 
        Write-Output "This is the latest policy compliance states generated in the last day for all resources within the specified Resource Group Name: $id"
        Get-AzPolicyStateSummary -ResourceGroupName $id
        }
}

The PowerShell script will get the parameter(s) as input from the pipeline (more on this in a bit), and then give you a summary view of the latest policy compliance states generated in the last day for all resources within the specified scope that could be either management group id, subscription id, or resource group name. It includes only non-compliant policy states.

Next, upload and push the script file to the git repository under your desired folder as shown in the figure below.

Get Azure Policy states summary for resources stage
Get Azure Policy states summary for resources stage

Validation stage

The pipeline also includes a validation stage. This is a very important step. In this stage, you can review and validate the non-compliance policy assignments and decide if you want to resume or reject the remediation stage.

By default, the wait for external user validation timeout is set to 24 hours (4320 minutes), and if non-response (onTimeout), the pipeline will be canceled (rejected).

You can update the validation stage parameters in the YAML file as needed:

- stage: Validation
  jobs:  
  - job: waitForValidation
    displayName: Wait for external validation  
    pool: server
    timeoutInMinutes: 4320
    steps:
    - task: ManualValidation@0
      timeoutInMinutes: 1440
      inputs:
        notifyUsers: |
          
        instructions: 'Please validate the build configuration and resume'
        onTimeout: 'reject'

Remediate Azure Policy stage

In this step, we will define and create our final stage to create and start policy remediation for non-compliant policy assignments.

Select and copy the code below and save it in a file named: remediate-policy.ps1 

param(
    [Parameter(Mandatory = $false)] [string] $type,
    [Parameter(Mandatory = $false)] [string] $id,
    [Parameter(Mandatory = $true)] [string] $AppID,
    [Parameter(Mandatory = $true)] [string] $AppSecret,
    [Parameter(Mandatory = $true)] [string] $tenantid
)

# .Import PowerShell Modules
Get-ChildItem -Path C:\Modules\az_*\Az.Accounts -Filter "*.psd1" -Recurse | Import-Module
Get-ChildItem -Path C:\Modules\az_*\Az.Resources -Filter "*.psd1" -Recurse | Import-Module
Get-ChildItem -Path C:\Modules\az_*\Az.PolicyInsights -Filter "*.psd1" -Recurse | Import-Module

# .Authentification
$CredApp = New-Object pscredential -ArgumentList ($AppID, ($AppSecret | ConvertTo-SecureString -Force -AsPlainText))
$Login = Login-AzAccount -TenantId $tenantid -Credential $CredApp -ServicePrincipal

switch ($type) {
    subscription {
        $nonCompliantPolicies = Get-AzPolicyState -SubscriptionId $id | Where-Object { $_.ComplianceState -eq "NonCompliant" -and $_.PolicyDefinitionAction -eq "deployIfNotExists" -and $_.PolicyAssignmentScope -like "*subscriptions*" }         
        Set-AzContext -SubscriptionId $id
        foreach($policy in $nonCompliantPolicies)
        {
            Write-Output "Start remediation: $($policy.PolicyDefinitionName)"
            $startremediation = Start-AzPolicyRemediation -Name "rem-$($policy.PolicyDefinitionName)" -PolicyAssignmentId $policy.PolicyAssignmentId -PolicyDefinitionReferenceId $policy.PolicyDefinitionId
        }
    }
    managementgroup {
        $nonCompliantPolicies = Get-AzPolicyState -ManagementGroupName $id | Where-Object { $_.ComplianceState -eq "NonCompliant" -and $_.PolicyDefinitionAction -eq "deployIfNotExists" -and $_.PolicyAssignmentScope -like "*managementGroups*" }          
        foreach($policy in $nonCompliantPolicies)
        {
            Write-Output "Start remediation: $($policy.PolicyDefinitionName)"
            $startremediation = Start-AzPolicyRemediation -Name "rem-$($policy.PolicyDefinitionName)" -PolicyAssignmentId $policy.PolicyAssignmentId -PolicyDefinitionReferenceId $policy.PolicyDefinitionId
        }
    }
    resourcegroup {
        $nonCompliantPolicies = Get-AzPolicyState -ResourceGroupName $id | Where-Object { $_.ComplianceState -eq "NonCompliant" -and $_.PolicyDefinitionAction -eq "deployIfNotExists" -and $_.PolicyAssignmentScope -like "*$id*" }
        foreach($policy in $nonCompliantPolicies)
        {
            Write-Output "Start remediation: $($policy.PolicyDefinitionName)"
            $startremediation = Start-AzPolicyRemediation -Name "rem-$($policy.PolicyDefinitionName)" -PolicyAssignmentId $policy.PolicyAssignmentId -PolicyDefinitionReferenceId $policy.PolicyDefinitionId
        }
    }
}

The PowerShell script will get the parameter(s) as input from the pipeline (more on this in a bit), and then create a policy remediation task for all non-compliant policy assignments. Please note that all non-compliant resources at or below the remediation’s scope will be remediated. Remediation is only supported for policies with the ‘deployIfNotExists’ effect.

The script will get all non-compliant policies that can be remediated.

As a side note, if you don’t want to remediate all non-compliant policy assignments, you can update the script to accept a list of an array of specific policy assignment(s) that you want to remediate instead.

Similar to the previous stage, upload and push the script file to the git repository under your desired folder as shown in the figure below.

Start policy remediation for non-compliant policy assignments stage
Start policy remediation for non-compliant policy assignments stage

Run Azure Policy Pipeline

In this section, we will run the new pipeline for policy compliance and remediation.

Sign in to the Azure DevOps portal and take the following steps:

On the left-hand side of the blade, select Pipelines under Pipelines as shown in the figure below and then click on All.

Expand the pipeline folder that you created in the previous step (e.g. Azure Policy Compliance & Remediate), then select your Git repository and click ‘Run pipeline‘ as shown in the figure below.

Run Azure DevOps Pipeline
Run Azure DevOps Pipeline

Next, select your desired scope ‘Type‘ whether you want to get the non-compliant policy assignments by scope (subscription id, management group id, or resource group name), and then enter the corresponding ‘Id‘. In this example, I will use a management group scope.

When you’re finished entering the details, select ‘Run‘ to kick the pipeline as shown in the figure below.

Enter Pipeline parameters
Enter Pipeline parameters

Next, you can see that the policy compliance job is completed successfully and the validation stage is waiting for approval.

Policy compliance stage
Policy compliance stage

You can click on the policy compliance job and see the list of Non-Compliant Policies output as shown in the figure below.

Check Azure Policy Assignments
Check Azure Policy Assignments

Next, we’ll validate and Resume to continue remediating the non-compliance policies as shown in the figure below.

Validate the build configuration and resume
Validate the build configuration and resume

The remediation policy stage will kick in and be complete successfully as shown in the figure below.

Start Azure policy remediation stage
Start Azure policy remediation stage

That’s it there you have it.

Azure DevOps Summary Pipeline job
Azure DevOps Summary Pipeline job

Happy Azure Policy Remediation with Azure DevOps!

Summary

In this guide, I showed you how to check Azure Policy compliance status and remediate non-compliant resources via Azure DevOps Pipelines.

With the help of simple stages in our build pipeline, we can automate and simplify the remediation process of Azure Policies via Azure DevOps Pipelines at scale. You also get the power of version control using Git repositories where you can store and author all your PowerShell scripts and YAML files in Visual Studio code.

To learn more on how to deploy and assign Azure Policy via Azure DevOps Pipelines, please check the following step-by-step guide.

__
Thank you for reading my blog.

If you have any questions or feedback, please leave a comment.

-Charbel Nemnom-

Related Posts

Previous

AZ-801 Study Guide: Configuring Windows Server Hybrid Advanced Services

Monitor Azure AD Guest Users With Azure Sentinel

Next

Let me know what you think, or ask a question...

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe to Stay in Touch

Never miss out on your favorite posts and our latest announcements!

The content of this website is copyrighted from being plagiarized!

You can copy from the 'Code Blocks' in 'Black' by selecting the Code.

Please send your feedback to the author using this form for any 'Code' you like.

Thank you for visiting!