How to Use Watchlist in Azure Sentinel

10 Min. Read

Updated – 06/08/2021 – Watchlists templates are now in public preview. More information about using the watchlists templates can be found in this section.

Azure Sentinel watchlist enables you to collect data from external data sources for correlation with the events in your Azure Sentinel environment. Once created, you can use watchlists in your search, detection rules, threat hunting, and response playbooks.

This article will show you how to use watchlists in Azure Sentinel to investigate threats and respond to incidents quickly with the rapid import of IP addresses, file hashes, and other business data.

Introduction

Azure Sentinel is a cloud-native Security Information Event Management (SIEM) and Security Orchestration Automated Response (SOAR) solution. Azure Sentinel delivers intelligent security analytics and threat intelligence across the enterprise, providing a single solution for alert detection, threat visibility, proactive hunting, and threat response.

At the time of this writing, Azure Sentinel Watchlist is in (Preview) that you can use to import a list of details and transform them into a log format stored in Log Analytics Workspace for use within Azure Sentinel. The list can be made by uploading a CSV file of data or made via the Azure Sentinel API. The information uploaded can be detailed within the logs ingested into Azure Sentinel or external data used to enrich information within Sentinel. Once created, watchlists can be used within Analytic Rules, Threat Hunting, Playbooks, and anything else that involves running KQL queries.

I often get asked by customers, what are the common use cases for using watchlists? The common scenarios according to Microsoft documentation, watchlists can be used for:

  1. Investigating threats and responding to incidents quickly with the rapid import of IP (list of TOR IP addresses for example), file hashes, and other data from CSV files. Once imported, you can use watchlist name-value pairs for joins and filters in alert rules, threat hunting, workbooks, notebooks, and general KQL queries.
  2. Importing business data as a watchlist. For example, you can import user lists with privileged system access, or recently terminated/resigned employees, and then use the watchlist to create allow and deny lists used to detect or prevent those users from logging in to the network.
  3. Reducing alert fatigue. You can create allow lists to suppress alerts from a group of users, such as users from authorized IP addresses that perform tasks that would normally trigger the alert, and prevent benign events from becoming alerts.
  4. Enriching event data. You can use watchlists to enrich your event data with name-value combinations derived from external data sources.

For the remainder of this article, I will cover the first scenario (investigate threats and respond to incidents) by importing the list of TOR Exit Nodes that I want to watch closely in my environment. As you know, TOR is an anonymous IP, so we wouldn’t want anybody who’s trying to hide themselves accessing the environment, right?

Prerequisites

To follow this article, you need to have the following:

  1. Azure subscription – If you don’t have an Azure subscription, you can create a free one here.
  2. Log Analytics workspace – To create a new workspace, follow the instructions here Create a Log Analytics workspace.
  3. Azure Sentinel – To enable Azure Sentinel at no additional cost on an Azure Monitor Log Analytics workspace for the first 31-days, follow the instructions here. Once Azure Sentinel is enabled on your Azure Monitor Log Analytics workspace, every GB of data ingested into the workspace can be retained at no charge for the first 90-days.
  4. At least one data connector is connected in Azure Sentinel. To configure the Activity Log data connector in Azure Sentinel to collect activity logs, please check the following guide. You can also collect other log data such as Microsoft 365 Defender, Office 365, Security Events, Azure AD, etc.
  5. Create a new watchlist (more on this in the next section).

Once you have all the prerequisites in place, you could:

Create a new watchlist

As noted previously, I will import the list of all TOR Exit Nodes. You can download the list of all IP addresses for TOR from the below URL:

  • TOR Exit Nodes – TOR exit node is the last Tor node that traffic passes through in the Tor network before exiting onto the internet.

Please note that you might need to download it again because this list is regularly checked and updated. At the time of this writing, you cannot update the watchlist from the Azure Portal unless you delete it and then upload a new one, however, you can update it via API. Please check the following guide on how to add or update a watchlist item with REST-API.

Next, copy and save the list in a CSV file as shown in the figure below with a header. In this example, the header is set to ‘TorIPAddress‘ which is used in the SearchKey field. The SearchKey field is used to optimize query performance when using watchlists for joins with other tables (more on this in a bit).

Prepare Watchlists List as CSV
Prepare Watchlists List as CSV

Once you have the list ready, take the following steps to create a new watchlist:

Open Azure Portal and sign in with a user who has Azure Sentinel Contributor permissions.

Click All services found in the upper left-hand corner. In the list of resources, type Azure Sentinel. As you begin typing, the list filters based on your input.

Click on Azure Sentinel and then select the desired Log Analytics Workspace.

From Azure Sentinel’s sidebar, select Watchlist under the Configuration section, then click + Add new as shown in the figure below.

Add a new Azure Sentinel Watchlist
Add a new Azure Sentinel Watchlist

On the General page (Watchlist wizard), provide the name, description, and alias for the watchlist as shown in the figure below, and then select Next: Source >. The Watchlist alias name will be used as a reference later when we start building KQL queries.

Azure Sentinel Watchlist Wizard
Azure Sentinel Watchlist Wizard

On the Source page, a CSV file with a header is set as default (currently only CSV is supported), then enter the number of lines before the header row in your CSV data file (0 in my example), and then choose the ‘TorBulkExitIPList.csv‘ as a file to upload in one of two ways:

  • You can click the Browse for files link in the Upload file box and select your CSV file to upload.
  • Or, drag and drop your CSV file onto the Upload file box.

Please note that file uploads are currently limited to files of up to 3.8 MB in size.

Next, you will see a preview of the first 50 rows of results in the Watchlist wizard (right-hand side). Then in the SearchKey field below, select the name of the column in your watchlist that you expect to use as a join with other data or a frequent object of searches. In this example, the name of the column is ‘TorIPAddress‘ as shown in the figure below. Click Next: Review and Create > to continue.

Upload Azure Sentinel Watchlist
Upload Azure Sentinel Watchlist

Finally, review the information, verify that it is correct, wait for the Validation passed message, and then click Create as shown in the figure below.

Review and validate Azure Sentinel Watchlist
Review and validate Azure Sentinel Watchlist

Once the watchlist is created, you will receive a notification message in the Azure Portal as shown in the figure below.

Azure Sentinel Watchlist created
Azure Sentinel Watchlist created

Create a hunting query

Now that we know we have all the capabilities of using watchlists for investigation and threat hunting. Then watchlists can be used before, during, or after an investigation, so we can monitor, track and detect suspicious activities and many other Azure Sentinel actions.

In this step, we’ll be leveraging the Azure Activity logs by creating a new hunting query to monitor in real-time when someone is performing any Azure activity through TOR Exit Nodes. You can perform similar queries with other collected/ingested logs.

Now you may ask, why do we need to create a Hunting query instead of an Analytic query rule? In fact, you can do both, with an analytic rule, the minimum query schedule is 5 minutes or above, however, with a hunting query, it’s nearly real-time (live stream). You can think of it as proactive versus reactive.

For the remainder of this article, I will use both approaches with Hunting to create a live stream session and then create an analytic rule. The good news is, when the custom query is created through hunting, you can create an analytic rule from the Hunting blade directly.

Open Azure Portal and sign in with a user who has Azure Sentinel Contributor permissions.

Click All services found in the upper left-hand corner. In the list of resources, type Azure Sentinel. As you begin typing, the list filters based on your input.

Click on Azure Sentinel and then select the desired Workspace.

From Azure Sentinel’s sidebar, select Hunting under the Threat management section, then click + New Query as shown in the figure below.

Create Azure Sentinel Hunting Query
Create Azure Sentinel Hunting Query

Enter a descriptive Name and Description. And In the Custom query section enter the following KQL query to be alerted when any Azure activity is performed through the list TOR IP Addresses.

In this example, we used the let statement which allows for the creating variables to be used in later statements. In this example, ‘TorIPAddress‘ is created and used as part of the AzureActivity “join” clause. We only included events from TOR IP addresses in the watchlist. As noted previously, for optimal query performance, we use SearchKey which representing the field I defined in creating the watchlist as the key for joins in this query.

let TorIPAddress = _GetWatchlist('TorExitNodes');
AzureActivity
| where CallerIpAddress != ''
| extend WhoDidIt = Caller, ResourceName = tostring(parse_json(Properties).resource)
| join TorIPAddress on $left.CallerIpAddress == $right.TorIPAddress
| project TimeGenerated, SearchKey, OperationNameValue, Type, SubscriptionId, WhoDidIt, ResourceName, ResourceGroup
Azure Sentinel Hunting Custom Query
Azure Sentinel Hunting Custom Query

Next, you need to map the entities to the appropriate columns available in your query results. This enables Azure Sentinel to recognize the entities that are part of the alerts for further analysis. The entity type must be a string. Next, give the right Tactics for this query such as (Initial Access), and then click on the Create button.

Azure Sentinel Hunting Entity Mapping
Azure Sentinel Hunting Entity Mapping

Once the custom query is created, navigate to Threat management > Hunting > Queries tab and then +Add a new filter as the (Provider), and value set to (Custom Queries).

Now to monitor Azure activities in real-time and receive notifications when a new event occurs, locate the hunting query that we created in the previous step (Watch TOR IP Exit Nodes), right-click the custom query, and then select Add to Livestream as shown in the figure below.

Azure Sentinel Hunting Add to Livestream
Azure Sentinel Hunting Add to Livestream

Now to view your Livestream session in action, navigate to Threat management > Hunting > Livestream tab. You will see that the query is in ‘Running‘ status as shown in the figure below.

Azure Sentinel Hunting Running query
Azure Sentinel Hunting Running query

Simulate an alert

To trigger an alert, you need to log into the Azure Portal to simulate suspicious activity by selecting any Azure resource you have.

I have logged into another virtual machine and I have installed the TOR browser that I will use to log in to the Azure Portal to simulate an Azure activity from a TOR IP.

Launch the TOR browser and navigate to portal.azure.com and use your account to log in. Type the account email and click Next. Then enter the password and click Sign in. If this the first time you are accessing this account, you will be asked to update your password, and on the next screen, click Yes if you would like to stay signed in for a while.

So now, you have successfully logged into the Azure portal using your account via the TOR browser. Next, you need to navigate to any Azure resource you already have (select a storage account for example), it’s not necessary to list the Keys, nor performing any activity to receive an alert.

Next, we need to wait for the notification to pop up. Because Livestream notifications for new events use Azure Portal notifications, you will see these notifications whenever you use the Azure portal. In my example, it took around 10 minutes for the notification to pop up after I navigated to my storage account through a TOR browser as shown in the figure below.

Azure Sentinel Livestream portal notification
Azure Sentinel Livestream portal notification

Select the notification to open the Livestream pane. Next, you can click Open Livestream session for this query where you can view the query results > including all the details for this notification.

Create an analytic rule

Now you can promote a Livestream session to a new alert by creating an analytic rule.

From within the same Livestream session, click on the Create analytics rule as shown in the figure below.

Azure Sentinel Livestream - Create an analytic rule
Azure Sentinel Livestream – Create an analytic rule

Give the analytic rule a meaningful ‘Name‘ and ‘Description‘, then select the corresponding ‘Tactics‘ (Initial Access). Those tactics are based on the MITRE ATT&CK Matrix for Enterprise. Then select ‘High‘ for the Severity and then click Next to Set rule logic.

In the Set rule logic tab, you will see the same rule query that we used in the previous hunting query. You can update it or leave it as it is.

Next, you need to enrich the alert, so I will map the following entities to the rule under the Alert enrichment section, as well as the new ‘Custom‘ entity details to enrich the alert for further investigation:

  • Account | Name = WhoDidIt
  • IP | Address = SearchKey
  • Azure Resource | ResourceId = ResourceName

In the Query scheduling section, I will schedule this query to run every 5 minutes and lookup data from the last 5 minutes. I will not change any other setting in the Set rule logic tab. Click Next to configure the Incident settings.

I will keep the default options for the Incident settings as well. However, I will enable group-related alerts, triggered by this analytics rule, into incidents. And keep the default settings: Grouping alerts into a single incident if all the entities match (recommended). Click Next to configure the Automated response.

In the Automated response tab, I will select the automated playbook that I created earlier to post a message in the Microsoft Teams Channel to inform the SOC team members about this threat. Click Next to review and create.

In the Review and create the page, validate the settings and click Create to start the rule creation process.

Azure Sentinel - Create a new analytic rule
Azure Sentinel – Create a new analytic rule

Now repeat the alert simulation described above and verify that a new incident is created in the Incidents blade.

Azure Sentinel - Incidents blade
Azure Sentinel – Incidents blade

Let’s see if I have received any message on the Microsoft Team channel. After waiting for a couple of minutes, a message popped up in my team channel as shown in the figure below.

Azure Sentinel - Team channel notification
Azure Sentinel – Team channel notification

That’s it there you have it. Happy Azure Sentinel watchlisting!

Watchlists templates

Azure Sentinel now provides built-in watchlist templates, which you can customize for your environment and use during investigations.

After those watchlists are populated with data, you can correlate that data with analytics rules, view it in the entity pages and investigation graphs as insights, create custom uses such as tracking VIP or sensitive users, and more.

At the time of this writing, Watchlist templates (preview) include:

  • High-Value Assets. A list of devices, resources, or other assets that have critical value in the organization.
  • Identity Correlation. A list of related user accounts that belong to the same person.
  • Network Addresses. A list of IP subnets and their respective organizational contexts.
  • Service Accounts. A list of service accounts and their owners.
  • Terminated Employees. A list of user accounts of employees that have been, or are about to be, terminated.
  • VIP Users. A list of user accounts of employees that have high impact value in the organization.

From the Azure portal, navigate to Azure Sentinel > Configuration > Watchlist > Templates (Preview).

Select a template from the list to view details on the right as shown in the figure below, and then select Create from template to create your watchlist.

Create a new watchlist using a template
Create a new watchlist using a template

Then continue in the Watchlist wizard as you would normally do. Please note that when using a watchlist template, the watchlist’s Name, Description, and Watchlist Alias values are all read-only.

Next, select Download Schema to download a CSV file that contains the relevant schema expected for the selected watchlist template. Each built-in watchlist template has its own set of data listed in the CSV file attached to the template.

Populate your local version of the CSV file, and then upload it back into the wizard.

Finally, continue as you would when creating a new watchlist from scratch, and then use your watchlist with queries and analytics rules.

Summary

This article showed you how to use watchlists in Azure Sentinel to investigate threats and respond to incidents quickly with the rapid import of IP addresses, file hashes, and other business data from CSV files. The incident will automatically trigger a security playbook to inform the organization’s Security Operation Center (SOC) team of this malicious attempt, so they can carry out further investigations.

This is one of the many features in Azure Sentinel that can be utilized to provide immense value to your security operations team.

Additional resources I highly encourage you to check:

The power of Azure Sentinel comes from the ability to detect, investigate, respond to, and remediate threats.

__
Thank you for reading my blog.

If you have any questions or feedback, please leave a comment.

-Charbel Nemnom-

Related Posts

Previous

Detect a Brute Force Attack with Azure Sentinel

Automatic VM Activation for Windows Server in Windows Admin Center

Next

Let me know what you think, or ask a question...

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe to Stay in Touch

Never miss out on your favorite posts and our latest announcements!

The content of this website is copyrighted from being plagiarized!

You can copy from the 'Code Blocks' in 'Black' by selecting the Code.

Please send your feedback to the author using this form for any 'Code' you like.

Thank you for visiting!