Microsoft Sentinel watchlist enables you to collect data from external data sources for correlation with the events in your Microsoft Sentinel environment. Once created, you can use watchlists in your search, detection rules, threat hunting, and response playbooks.
This article will show you how to backup and restore Microsoft Sentinel Watchlists, so you can import them to another environment, or if you need to upgrade an existing solution from Content Hub that includes Watchlists, and you want to take a backup just in case something goes wrong.
Better safe than sorry!
Table of Contents
Introduction
Microsoft Sentinel is a cloud-native Security Information Event Management (SIEM) and Security Orchestration Automated Response (SOAR) solution. Microsoft Sentinel delivers intelligent security analytics and threat intelligence across the enterprise, providing a single solution for alert detection, threat visibility, proactive hunting, and threat response.
Microsoft Sentinel Watchlist is generally available, you can use it to import a list of details and transform them into a log format stored in Log Analytics Workspace for use within Sentinel. The list can be made by uploading a CSV file of data or made via the Microsoft Sentinel API. The information uploaded can be detailed within the logs ingested into Microsoft Sentinel or external data used to enrich information within Sentinel.
Once created, watchlists can be used within Analytic Rules, Threat Hunting, Playbooks, and anything else that involves running KQL queries.
I often get asked by customers, what are the common use cases for using watchlists? The common scenarios according to Microsoft documentation, watchlists can be used for:
1) Investigating threats and responding to incidents quickly with the rapid import of IP (list of TOR IP addresses for example), file hashes, and other data from CSV files. Once imported, you can use watchlist name-value pairs for joins and filters in alert rules, threat hunting, workbooks, notebooks, and general KQL queries.
2) Importing business data as a watchlist. For example, you can import user lists with privileged system access, VIP users, or recently terminated/resigned employees, and then use the watchlist to create allow and deny lists used to detect or prevent those users from logging in to the network.
3) Reducing alert fatigue. You can create allow lists to suppress alerts from a group of users or domains, such as users from authorized IP addresses or domains that perform tasks that would normally trigger the alert, and prevent benign events from becoming alerts.
4) Enriching event data. You can use watchlists to enrich your event data with name-value combinations derived from external data sources.
Prerequisites
To follow this article, you need to have the following:
1) Azure subscription – If you don’t have an Azure subscription, you can create a free one here.
2) Log Analytics workspace – To create a new workspace, follow the instructions here Create a Log Analytics workspace.
3) Enable Microsoft Sentinel at no additional cost on an Azure Monitor Log Analytics workspace for the first 31-days, follow the instructions here. Once Microsoft Sentinel is enabled on your Azure Monitor Log Analytics workspace, every GB of data ingested into the workspace can be retained at no charge for the first 90-days.
4) Create a new watchlist as described in this article.
Once you have all the prerequisites in place, take the following steps.
Backup Sentinel Watchlists
To back up your watchlist, you have two options, you can either backup all Watchlist records in one step, or you can backup individual Watchlists.
Individual Watchlists backup are useful in case you are doing an update of any of your Content Hub solutions that has Watchlist(s) in their content type.
Backup all Watchlists
To back up all Watchlist records in one step, open the Log Analytics workspace and run the following KQL query:
union isfuzzy=true Watchlist, ConfidentialWatchlist
| where TimeGenerated < now()
| where _DTItemType == 'watchlist-item'
| summarize hint.shufflekey=_DTItemId arg_max(_DTTimestamp, _DTItemStatus, LastUpdatedTimeUTC, SearchKey, WatchlistItem, WatchlistAlias) by _DTItemId
| order by WatchlistAlias
| where _DTItemStatus != 'Delete'
| project-away _DTItemId, _DTTimestamp, _DTItemStatus, LastUpdatedTimeUTC, SearchKey, WatchlistAlias
| evaluate bag_unpack(WatchlistItem)This query will retrieve all the Watchlists that you have in your environment, and remove (project-away) all the columns that you don’t need as part of your export.
Next, you need to export to a CSV file using the Export feature, then choose to Export to CSV – displayed columns as shown in the figure below.
Now, the CSV file will contain all the columns for all watchlists, and some columns will be empty because they don’t exist in one or more watchlists, but since I am filtering by WatchlistAlias, they will be exported by order so they can be easily identified and filtered in the CSV file.
Backup Individual Watchlists
To back up individual Watchlists, you first need to get the name of the watchlist(s) that you need to back up in your environment.
Once you have the watchlist names ready, switch to the Log Analytics blade and run the following KQL query. You need to update the Watchlist Name:
_GetWassssatchlist('Watchlist-Name-Here')
| project-away LastUpdatedTimeUTC, _DTItemId, SearchKey
Then you export to CSV file using the Export feature, then choose to Export to CSV – displayed columns as shown in the figure below.
Save the CSV file in your desired ‘safe’ location, and then you need to repeat the same steps for others Watchlists that you have.
Restore Sentinel Watchlists
Now to restore a Microsoft Sentinel Watchlist, if you are doing an update to an existing solution from Content Hub, then you can do the update now before you proceed to restore your watchlists.
To restore your watchlist, take the following steps:
1) Open your Sentinel workspace and navigate to watchlists. Select your watchlist > Update Watchlist > Edit Watchlist items.
2) In the watchlist item editor, select all items and click Delete to purge the contents of the watchlist as shown in the figure below.
3) Then navigate back to the watchlist’s blade. Select your watchlist > Bulk Update > Local file > and Load your CSV file to restore your watchlist.
You can also backup your Watchlist items to an Azure storage account, and then set the Source type as Azure Storage (Preview) to restore them as shown in the figure below.
As a side note, you can’t restore all watchlists with KQL into the Watchlist Table in Log Analytics. You need to import them from CSV files using the Bulk Update option.
Automate Backup Sentinel Watchlists
The following Logic App Playbook is designed to perform regular backups of your Microsoft Sentinel watchlists and store them in your Azure storage account (blob storage).
This playbook utilizes an Azure AD app registration to authenticate to the Azure API.
As a prerequisite, you want to create a new app registration in Azure AD before deploying this template (keep the Client ID and Secret at hand).
The ARM template will create the following resources:
- The logic app with a managed identity.
- A key vault that stores the client’s secret for your AAD app registration.
- A storage account and blob storage container to hold the Watchlist backups.
The template will also assign the following roles:
- The logic App managed identity will be assigned a Sentinel Reader role at the resource group level.
- The logic App managed identity will be assigned Storage Blob Data Contributor at the resource level.
- The logic App managed identity will be assigned to the Key Vault Secrets User at the resource level.
Click on the “Deploy to Azure” button and follow the wizard.
Here is the sequence of steps that are performed as part of this playbook.
That’s it there you have it!
Summary
This article showed you how to backup and restore Microsoft Sentinel Watchlists so that you can import them to another or the same environment, this is useful especially when you have a large number of watchlist items, and you have made any modifications to the watchlists, either in terms of data or columns. It becomes critical to maintain a valid backup before making any changes.
To be safe, we recommend you also take a backup of your watchlist and then re-import them after you do an upgrade of any of your content hub solutions that has a Watchlist in their content type.
You can use watchlists in Microsoft Sentinel to investigate threats and respond to incidents quickly with the rapid import of IP addresses, file hashes, and other business data from CSV files. The incident will automatically trigger a security playbook to inform the organization’s Security Operation Center (SOC) team of this malicious attempt, so they can carry out further investigations.
This is one of the many features in Microsoft Sentinel that can be utilized to provide immense value to your security operations team.
The power of Microsoft Sentinel comes from the ability to detect, investigate, respond to, and remediate threats.
__
Thank you for reading my blog.
If you have any questions or feedback, please leave a comment.
-Charbel Nemnom-
