You dont have javascript enabled! Please enable it! Monitor Log Flow For Devices In Microsoft Sentinel - CHARBEL NEMNOM - MVP | MCT | CCSP | CISM - Cloud & CyberSecurity

Monitor Log Flow For Devices in Microsoft Sentinel

7 Min. Read

You are ingesting multiple devices and appliances to Microsoft Sentinel through the Common Event Format (CEF) via AMA, and you want to ensure that the logs flow regularly to the Log Analytics workspace.

In this article, we will show you how to monitor log flow for devices in Microsoft Sentinel and be alerted when a device stops sending logs.

Monitor Log Flow For Devices

Microsoft Sentinel is a cloud-native Security Information Event Management (SIEM) and Security Orchestration Automated Response (SOAR) solution. Microsoft Sentinel delivers intelligent security analytics and threat intelligence across the enterprise, providing a single solution for alert detection, threat visibility, proactive hunting, and threat response.

When it comes to monitoring log flow ingestion for network devices to Microsoft Sentinel, we can use multiple methods and different options to achieve this. This could be at the source device, in between (Log forwarder) machines, or at the target in Sentinel. We must also easily maintain a list of those devices without operating overhead. As new devices are added to the network and old devices are being retired in large organizations, we need to monitor the flow logs continuously being ingested into Microsoft Sentinel.

One option is to use the Microsoft Sentinel watchlist to achieve this purpose. Watchlists allow us to create dynamic tables inside Microsoft Sentinel, where we can add/remove data. According to Microsoft documentation, the common use cases for watchlists are used for:

1) Investigating threats and responding to incidents quickly with the rapid import of IP (list of TOR IP addresses, for example), file hashes, and other data from CSV files. Once imported, you can use watchlist name-value pairs for joins and filters in alert rules, threat hunting, workbooks, notebooks, and general KQL queries.

2) Importing business data as a watchlist. For example, you can import user lists with privileged system access, VIP users, or recently terminated/resigned employees and then use the watchlist to create allow and deny lists to detect or prevent those users from logging in to the network.

3) Reducing alert fatigue. You can create allow lists to suppress alerts from a group of users or domains, such as users from authorized IP addresses or domains performing tasks that normally trigger the alert and prevent benign events from becoming alerts.

4) Enriching event data. Watchlists can enrich your event data with name-value combinations derived from external data sources.

We can also leverage watchlists to monitor assets like networks and security appliances.

Monitor Log Flow for Network and Security Devices
Monitor Log Flow for Network and Security Devices

This article describes using Syslog via AMA and Common Event Format (CEF) via AMA connectors, including those in Common Event Format (CEF), from Linux machines, networks, and security devices and appliances that can be used to monitor the Log Flow for those devices and send an alert if we detect that we are not receiving logs from a particular device.

Prerequisites

To follow this article, you need to have the following:

1) Azure subscription – If you don’t have an Azure subscription, you can create one here for free.

2) Log Analytics workspace – To create a new workspace, follow the instructions to create a Log Analytics workspace.

3) Enable Microsoft Sentinel at no additional cost on an Azure Monitor Log Analytics workspace for the first 31 days; follow the quick onboarding process. Once Microsoft Sentinel is enabled on your Azure Monitor Log Analytics workspace, every GB of data ingested into the workspace can be retained at no charge for the first 90 days.

4) Designated Linux VM (Log forwarder) to collect logs from your network security devices and appliances using the Common Event Format (CEF) via AMA to the “CommonSecurityLog” table. The machine could be deployed in Azure or on-premises. If your log forwarder isn’t an Azure virtual machine, it must have Azure Arc (connected machine agent) installed and enabled.

// Related: Simulate and Validate CEF Logs to Microsoft Sentinel.

5) Create Microsoft Sentinel Watchlist (more on this in the next section).

6) Create Microsoft Sentinel Alert (more on this in the next section).

Once you have all the prerequisites in place, take the following steps:

Create Microsoft Sentinel Watchlist

In this section, we will create a new Watchlist to add all network and security devices and appliances we want to monitor.

Microsoft Sentinel Watchlist is generally available; you can use it to import a list of details and transform them into a log format stored in Log Analytics Workspace for use within Sentinel. The list can be made by uploading a CSV data file or via the Microsoft Sentinel API. The information uploaded can be detailed within the logs ingested into Microsoft Sentinel or external data used to enrich information within Sentinel.

1) First, let’s open a blank Microsoft Excel workbook and, in cell A1, insert Hostname; in cell B1, insert Identifier; and in cell C1, insert IpAddress. Then, under each column, enter the desired values for each network device that you want to monitor, as shown in the following screenshot:

Add data to the Excel table
Add data to the Excel table

2) Click on File, then Save as, choose a location, enter the name, and choose to save it as a CSV file.

3) Next, go to Microsoft Sentinel, select the Watchlist tab, and click + New, as shown in the following screenshot:

Add a new watchlist from the Watchlist tab
Add a new watchlist from the Watchlist tab

4) In the General tab, enter the name and alias of the watchlist (WC_NetworkDevices) give it a description (Lists of all known network devices across the company network, with their Identifier as the search field.), and click on Next: Source, as shown in the figure below:

General tab in the watchlist wizard
General tab in the watchlist wizard

5) Click on Browse for files in the next window and upload the saved CSV file. For the SearchKey value, select Identifier and click on Next: Review and create, as shown in the figure below:

Upload the file and select a SearchKey value
Upload the file and select a SearchKey value

6) Check your configuration and click Create. It will take around 3 to 5 minutes for us to query this data.

7) After approximately 5 minutes, we can click Refresh to see whether the number of Rows changed from 0 to 2 (based on the network of devices you entered). If it has, click on View in logs. If not, wait a few more minutes, as shown in the figure below. This list can be easily maintained afterward by adding or removing network devices.

View watchlist data in the logs
View watchlist data in the logs

8) This will open this watchlist in Logs, and we can now create a Microsoft Sentinel Schedule rule from it by selecting + New alert rule and Create Microsoft Sentinel alert, as shown in the figure below.

Create a new detection rule from Logs
Create a new detection rule from Logs

// Related: Step-by-Step Guide – Backup and Restore Microsoft Sentinel Watchlists.

Create Microsoft Sentinel Alert

Next, we need to create the Microsoft Sentinel Schedule rule logic to get alerted when a device stops sending logs.

1) On the Analytics rule creation wizard. (We name it – “WC FortiGates FlowMon,” and we give it a description – “Log flow monitoring rule for FortiGates logs in CommonSecurityLog“), then, for MITRE ATT&CK, we select Impact > T0826 – Loss of Availability and then click on Next: Set rule logic, as shown in the figure below:

Analytics rule wizard – General tab
Analytics rule wizard – General tab

2) In the Set rule logic, we need to adjust the query under the Rule query to match our “DeviceVendor.” In this example, we are monitoring the log flow for FortiGates devices. So, the KQL query is as follows. This query filters security logs to include only events related to Fortinet devices and then joins these filtered logs with our watchlist of the network devices we created above. It selects and summarizes the identifiers of network devices that do not have corresponding security events in the filtered logs, potentially indicating that the device(s) stopped sending security logs.

CommonSecurityLog
| where DeviceVendor == "Fortinet"
| join kind=rightanti _GetWatchlist("WC_NetworkDevices") on $left.DeviceExternalID == $right.SearchKey
| project SearchKey
| summarize by SearchKey

3) Next, we will set Custom details under Alert enhancement by entering “DeviceId” for the key, and select “SearchKey” for the value, as shown in the figure below:

Entity mapping in the Set rule logic tab
Entity mapping in the Set rule logic tab

4) The last step in the Set rule logic will be to change the “Run query every” from 5 Hours to 15 Minutes and the “Lookup data from the last” from 5 Hours to 15 Minutes; then we can click on Next: Incident settings, as shown in the figure below:

Run query scheduling configuration in the Set rule logic tab
Run query scheduling configuration in the Set rule logic tab

As a side note, Microsoft Sentinel scheduled alert rules are configured to have a 5-minute look-back period by default. However, each data source (i.e., CommonSecurityLog) may have an individual ingestion delay so that the 5-minute look-back period could be short. When joining multiple data types, you must understand the different delays for each data type to configure the look-back period correctly. Based on our example, the 15-minute look-back period is a sweet spot. The Workspace Usage Report Workbook, provided in Microsoft Sentinel out-of-the-box, includes a dashboard that shows latency and delays for the different data types flowing into your workspace.

Microsoft Sentinel | Workspace Usage Report
Microsoft Sentinel | Workspace Usage Report

5) We will leave the Incident settings unchanged and continue to Automated response. In the Automated response, we will create an automation rule that will run when this analytics rule is created. We will click on Add new, enter the name, and select When incident is created as the trigger, as shown in the figure below. As an action, we select run a playbook and choose one of our existing Playbooks (Logic App) to notify our SOC Team by posting a message in Microsoft Teams and sending an email. Then click on Apply.

Attach a playbook
Attach a playbook

6) Select Next: Review to continue to the last step in analytic rule creation. In Review + Create, we can check our configuration and then click on Save, as shown in the figure below:

Review the configuration and create a new scheduled rule
Review the configuration and create a new scheduled rule

This will create our analytics rule and run almost immediately since it is enabled. To test the rule, you can turn off the logs ingestion for one of the network devices and check that the alert and incident are created. The figure below shows an example of an incident posted in Microsoft Teams, including the network “Device ID” to identify the faulty device immediately.

Microsoft Sentinel Incident Alert in Microsoft Teams
Microsoft Sentinel Incident Alert in Microsoft Teams

That’s it, there you have it. Happy Flow Log Monitoring for Network and Security Devices in Microsoft Sentinel!

In Conclusion

In this article, we provided a comprehensive guide on monitoring log flow for devices in Microsoft Sentinel and setting up alerts when log flow stops. Microsoft Sentinel is a powerful cloud-native Security Information Event Management (SIEM) and Security Orchestration Automated Response (SOAR) solution that offers intelligent security analytics, threat intelligence, and proactive threat response capabilities.

The article outlines the common use cases for watchlists in Microsoft Sentinel, including investigating threats, importing business data, reducing alert fatigue, and enriching event data. It emphasizes the versatility of watchlists in monitoring assets and network security appliances.

Ultimately, by following the instructions in this article, we can effectively monitor log flow for network and security devices in Microsoft Sentinel, ensuring proactive detection and response to ensure that we continuously receive logs from our network devices. By implementing automated response actions, such as notifying the SOC team via Microsoft Teams or emails, you can effectively enhance the security posture and mitigate risks.

__
Thank you for reading our blog.

Please let us know in the comments section below if you have any questions or feedback.

-Charbel Nemnom-

Photo of author
About the Author
Charbel Nemnom
Charbel Nemnom is a Senior Cloud Architect with 21+ years of IT experience. As a Swiss Certified Information Security Manager (ISM), CCSP, CISM, Microsoft MVP, and MCT, he excels in optimizing mission-critical enterprise systems. His extensive practical knowledge spans complex system design, network architecture, business continuity, and cloud security, establishing him as an authoritative and trustworthy expert in the field. Charbel frequently writes about Cloud, Cybersecurity, and IT Certifications.
Previous

Azure Key Vault vs HashiCorp Vault – Which is the Best Solution?

Let us know what you think, or ask a question...