Maintaining robust security operations is more critical than ever in today’s rapidly evolving cybersecurity landscape. Microsoft Sentinel, a cloud-native SIEM solution, empowers organizations with real-time insights and proactive threat detection. One key feature of Sentinel is Summary Rules, which help aggregate large volumes of data for faster querying and more efficient analysis.
In this comprehensive guide, we’ll explore how to monitor Summary Rules in Microsoft Sentinel by configuring diagnostic settings, understanding key KQL queries, and ensuring that your environment remains secure, cost-effective, and high-performing.
Table of Contents
Introduction
Microsoft Sentinel is a modern, cloud-native SIEM (Security Information and Event Management) and SOAR (Security Orchestration, Automation, and Response) solution that integrates with Microsoft Defender XDR to provide advanced threat detection, investigation, and response. Its ability to ingest data from a wide range of sources—from endpoints to cloud services—enables security teams to gain a holistic view of their security posture. Sentinel’s scalable architecture allows organizations to handle massive amounts of log data without compromising performance, while powerful query capabilities via Kusto Query Language (KQL) make it easier to extract actionable insights.
Key features include:
- Real-Time Analytics: Immediate detection of threats with automated alerts.
- Scalability: Easily accommodates growing data volumes.
- Integration: Works seamlessly with various Microsoft and third-party security tools.
- Automation: Streamlines incident response through playbooks and machine learning.
- Unified Security Operations: Microsoft Sentinel’s integration with the unified security operations (SecOps) platform marks a significant milestone in unifying SecOps workflows.
Microsoft Sentinel helps organizations stay one step ahead in today’s dynamic threat environment by centralizing log management and threat analysis.
Summary Rules in Microsoft Sentinel
Summary Rules in Microsoft Sentinel represent a strategic advancement in log analytics. They are designed to aggregate raw log data into concise, precompiled tables, significantly reducing the time required to run complex queries over large datasets. With Summary Rules, you can enjoy the benefits of both performance and cost savings:
- Accelerated Queries: Precompiled summary data enables rapid searches across historical records.
- Cost Efficiency: Retain only necessary data in the analytics tier, reducing storage costs. For example, why keep 1,000 logs that say the same thing when we can summarize that data and keep only the important values we are interested in?
- Enhanced Data Privacy: Sensitive details can be obfuscated during summarization, ensuring compliance with privacy standards.
- Historical Analysis: Facilitates long-term trend analysis and incident investigations by retaining summary data beyond the standard retention period.
Currently, in preview, Summary Rules are limited to 30 rules per Log Analytics workspace, making careful planning and monitoring of their performance essential.
Health and Diagnostic Settings for Summary Rules
Ensuring that your Summary Rules are running smoothly is a top priority. Microsoft Sentinel offers diagnostic settings that help you track the performance and health of your Summary Rules. These settings capture vital information such as run history, execution times, and any errors or failures encountered during rule execution.
Why Monitor Rule Status?
- Operational Insights: Diagnostic logs provide detailed information on the execution and performance of Summary Rules.
- Proactive Issue Resolution: Early detection of issues prevents minor glitches from evolving into major disruptions.
- Cost Management: Monitoring diagnostic logs helps in understanding and controlling the associated costs.
By monitoring these metrics, you can maintain a high-performing environment and ensure that your security operations are both efficient and reliable.
Enabling “SummaryLogs” Diagnostic Settings
Enabling the “SummaryLogs” diagnostic settings under the Log Analytics workspace where your Sentinel instance is deployed is highly recommended for optimal monitoring. These logs are crucial for:
- Tracking historical run statuses and identifying patterns in rule failures.
- Providing an audit trail that can be used for compliance and troubleshooting.
- Helping you optimize Summary Rules by analyzing performance trends over time.
Steps to Enable SummaryLogs
There are two primary approaches to enabling diagnostic settings for Summary Rules in Microsoft Sentinel:
1) Global Enablement During Rule Creation: When creating a new Summary Rule, you have the option to enable “SummaryLogs” automatically by simply selecting Enable, as shown in the figure below. This ensures that the rule is monitored right from the start.
2) Manual Configuration via Log Analytics Workspace: Alternatively, for more detailed configuration, you can directly configure the diagnostic settings in the Log Analytics workspace. This method will allow for advanced customization and monitoring of system performance. You can choose ‘Configure advanced diagnostic settings‘, as shown in the figure above, or you can access the Log Analytics workspace, then from the workspace’s settings menu, select “Diagnostic settings“, then check the box to enable “Summary Logs“, as shown in the figure below, and decide where to forward these logs—be it another Log Analytics workspace, a storage account, or Event Hubs.
A significant advantage of enabling “SummaryLogs” for Summary Rules, whether by configuring advanced diagnostic settings at the workspace level or during rule creation, is that it applies universally to all Summary Rules. This means that after the initial setup, you don’t need to configure diagnostics for each rule individually. With Summary Rules still in preview and a maximum of 30 rules per workspace, this one-time configuration approach saves both time and administrative overhead.
Once configured and enabled, these settings remain in effect for all Summary Rules you create in the future, simplifying management and ensuring consistency across your security environment.
Benefits of Summary Rules Monitoring
The SummaryLogs comprehensively record each Rule’s execution, including performance metrics and error logs. This detailed logging is essential for several reasons:
- Historical Data Analysis: By reviewing past logs, you can detect recurring issues or performance bottlenecks.
- Performance Optimization: Identify long-running queries and fine-tune your summary logic accordingly.
- Incident Auditing: Maintain a detailed audit trail to support compliance and forensic investigations.
Monitoring SummaryLogs helps troubleshoot and provides a basis for continuous improvement of your security operations.
Monitoring Summary Rules
Embracing a proactive monitoring strategy for your Summary Rules in Microsoft Sentinel is crucial to safeguard your organization effectively. Continuous optimization and monitoring are the keys to successful security operations.
Evaluating Costs for Summary Rules Logs
While the benefits of Summary Rules are clear, it’s important to be aware of the additional costs associated with their logs. The more often you run the Summary rule (i.e., every 20, 30 minutes, or every 1, 2, 3, 6, 12, 24 hours), the more data and logs are written, thus raising the cost. On the other hand, the longer time you run the rule, the fewer logs you get.
The use case you build usually determines how frequently you want to run the summary rule. For example, if the use case is to build a baseline of multiple days and see how it fluctuates over the weeks of weekends, then running the summary rule once per day makes more sense.
To help manage and predict these costs, you can run a KQL query that calculates the total billable data for SummaryLogs. Consider the following query:
Usage
| where TimeGenerated > ago(32d)
| where StartTime >= startofday(ago(31d)) and EndTime < startofday(now())
| where DataType == "LASummaryLogs"
| summarize BillableDataGB = sum(Quantity) / 1000 by Solution, DataTypeDetailed Query Breakdown:
Time Frame Filtering: The initial where TimeGenerated > ago(32d) filter selects records from the past 32 days. This ensures the analysis covers a recent period.
Full-Day Aggregation: The clause where StartTime >= startofday(ago(31d)) and EndTime < startofday(now()) restricts the data to complete days—starting from 31 days ago up to the start of today. This avoids partial data that could skew cost calculations.
Data Type Specification: where DataType == "LASummaryLogs" ensures that the query only considers logs related to Summary Rules, ignoring other types of data in your workspace.
Summarization and Conversion: Finally, the summarize operator aggregates the total data quantity (dividing by 1000 to convert the logged units into gigabytes) and groups the result by Solution and DataType. This provides a clear breakdown of the costs associated with SummaryLogs usage.
In this particular environment, we have very low “LASummaryLogs” billable data in the table.
Using this query, you can proactively monitor and manage the costs incurred by diagnostic settings of your Summary Rules, ensuring that your security operations remain within budget.
Monitoring Query Duration for Summary Rules
Another important metric to monitor is the query duration of your Summary Rules. Long-running queries can indicate performance issues that need addressing. The following KQL query helps you measure and analyze query durations:
LASummaryLogs
| summarize QueryDurationInSeconds = sum(QueryDurationMs)/1000 by RuleName, BinStartTime
| sort by QueryDurationInSeconds descDetailed Query Breakdown:
Data Source: The query begins by referencing the LASummaryLogs table, which holds data on the execution of Summary Rules.
Summarization of Duration: Using the summarize operator, the query groups data by RuleName and BinStartTime (a time bucket indicating when the rule ran). It sums up the QueryDurationMs values (recorded in milliseconds) and converts them to seconds by dividing by 1000.
Sorting Results: The sort by QueryDurationInSeconds desc line orders the results in descending order, so the longest-running queries appear at the top. This makes it easy to identify which rules may require further optimization.
Monitoring the duration of your Summary Rules queries can provide valuable insights into performance issues, helping you fine-tune your Summary Rules and improve overall system efficiency.
Creating Analytic Rules to Monitor Summary Rules Status
To automate the monitoring process and receive proactive alerts, you can create an analytical (detection) rule that leverages KQL to detect failed Summary Rules. The following query provides a snapshot of rule failures:
LASummaryLogs
| where Status == "Failed"
| summarize arg_max(TimeGenerated, *), count() by RuleName, Message
Related: How to create Analytic Rules in Microsoft Sentinel.
Detailed Query Breakdown:
Filtering Failures: The query starts by filtering the LASummaryLogs table to only include records where the Status is “Failed.” This isolates instances where a rule did not execute or start successfully.
Aggregating Latest Failure Data: The summarize arg_max(TimeGenerated, *), count() by RuleName, Message function groups the data by RuleName and the corresponding failure Message. The arg_max function selects the most recent record for each group, ensuring that you have the latest failure details along with a count of how many times the failure occurred.
This summarized view of failed Summary Rules enables your security team to pinpoint issues quickly, analyze failure trends, and take corrective action before these issues impact overall operations.
Final Thoughts
Monitoring Summary Rules in Microsoft Sentinel is an essential component of maintaining a robust and efficient security operations center (SOC). With the ability to precompile and aggregate vast datasets, Summary Rules enable faster query execution and cost-effective data management. However, without proper monitoring and enabling Summary Logs, even the most advanced systems can encounter performance issues.
By integrating the analytical rules outlined in this guide into your Microsoft Sentinel environment, you can automate alerting processes and enhance your overall security posture.
With Summary Rules still in preview and a limit of 30 rules per workspace, it is important to plan your deployment carefully. Enable “SummaryLogs” diagnostic settings early on, monitor their usage to manage costs, and use the provided KQL queries as a basis for ongoing performance optimization. This proactive approach will help you maintain a secure, scalable, and efficient security infrastructure.
Remember, you can always support us in developing tools and creating content via Why Donate? – Charbelnemnom.com Cloud & Cybersecurity
__
Thank you for reading our blog.
Please let us know in the comments section below if you have any questions or feedback.
-Charbel Nemnom-
