Demystifying Microsoft Sentinel Multi-Tier Logging

Updated — 01/04/2025 — Starting 1 May 2025, Microsoft will begin billing for queries and search jobs on logs ingested into the Auxiliary Logs plan. Querying Auxiliary Logs will be charged $0.005 per GB of data scanned (US East), and Search jobs on Auxiliary Logs will incur a cost of $0.005 per GB of data scanned (US East) plus search results ingestion costs.

Updated — 28/02/2025 — Starting 1 April 2025, Auxiliary Logs will be generally available (GA). Ingestion billing will begin at $0.15 per GB (US East), and long-term retention billing will be at $0.02 per GB (US East).

Multi-tier logging in Azure Monitor Log Analytics and Microsoft Sentinel offers a structured approach to managing diverse logging needs. Categorizing logs into Analytics, Basic, and Auxiliary tiers addresses various requirements for monitoring, compliance, and security. This method streamlines log management, making it both cost-effective and operationally efficient.

In this article, we will demystify Azure Monitor (Log Analytics workspace) and Microsoft Sentinel Multi-Tier Logging and dive into Microsoft’s recent announcements of Auxiliary Logs and Summary Rules. Both of these are in public preview.

Introducing Multi-Tier Logging

Microsoft recently announced the new Auxiliary Logs tier and Summary Rules. We now have three tiers, and we can select which one to use. Each table in the Log Analytics workspace is designed so that you can designate which log goes to which tier.

So, the first type is Analytics Logs. These are the same logs that we have known from Azure Monitor logs for a long time. We can do really high-end things with that, like running super complex queries. It includes all the queries; there is no additional cost per query. We start with 31 days or 90 days if you have Microsoft Sentinel, and you can have interactive retention of queries for up to two years.

A while ago, Microsoft introduced the ability to do long-term retention, or what we used to call archiving. Microsoft changed the name to long-term retention, so you can keep these logs for 12 years. It’s just quite a lot of time, and it’s probably covering all your log and compliance needs.

Analytics Logs are on the high end and relatively expensive. For Azure Monitor without Sentinel, it starts at $2.3 per GB, and all the prices noted below are East US prices (list prices). We go all the way down to $1.48 per GB when you’re doing something called the Commitment Tiers. For Microsoft Sentinel, it starts at $4.3 per GB, and we go all the way down to $2.06 per GB with the Commitment Tiers.

So, if you commit to a large ingest volume you can make the price go down and down and down. This price is before discounts, and this price is in the East US region, just to set the expectation. You can go to the Azure Monitor pricing page or to the Azure Monitor pricing calculator, which takes this into account. So, this is the tier that we have had for 9 years now in this way or the other.

Two years ago, Microsoft added Basic Logs. Basic Logs is built for medium-touch telemetry data and is much cheaper. For Azure Monitor without Sentinel, it’s $0.50 per GB, and for Sentinel, it’s $1 per GB of data ingested, but it has additional charges for KQL queries. So, it’s not built for the data that you query all the time; it’s built for data that you ingest and again for incidents or troubleshooting once in a while. Microsoft also allows you to extend the retention up to 12 years in the long-term retention fees.

At the end of July 2024, Microsoft introduced Auxiliary Logs, a third tier, which is much cheaper. It’s like $0.10 per GB for Azure Monitor without Sentinel and $0.15 per GB with Sentinel, and they allow us to run KQL queries. It has additional charges for queries, but the queries are slower because there is a trade-off about the cost.

One thing to note is that all of this is a managed service, meaning you don’t need to do anything other than configure the stuff. It’s a full-blown SaaS service, meaning that Microsoft takes care of everything behind the scenes. There are lots of technologies behind the scenes, and the scale is a challenge of its own. Microsoft invested a lot in terms of scale, security, and all that stuff you get this out of the box. You just configure and send your logs into these tiers. You don’t need to deal with the scalability of the service or anything else; it’s just there for you.

	Analytics Logs	Basic Logs	Auxiliary Logs
Use cases	Powerful all inclusive logs ideal for real-time monitoring, alerting and dashboards	Medium-touch telemetry data needed for troubleshooting and incident response	Low-touch telemetry data intended for high verbose logs, auditing and compliance
Azure Monitor price	$2.3-$1.48 per GB*	$0.5 per GB*	$0.1 per GB*
Microsoft Sentinel price	$4.30-$2.06 per GB*	$1 per GB*	$0.15 per GB*
Query cost	Including unlimited queries	Additional charges for queries	Additional charges for queries
KQL	Fast query performance with full KQL support	Fast query performance with full KQL on a single table and lookup to Analytics tables	Slow query performance with full KQL on a single table and lookup to Analytics tables
Interactive retention	31/90 days (free) and can extend to 2 years ($0.10 per GB per month)	30 days only (free)	30 days only (free)
Long-term retention	Up to 12 years $0.02 per GB per month	Up to 12 years $0.02 per GB per month	Up to 12 years $0.02 per GB per month

* Prices are East US list prices before discounts for ingestion and include interactive retention. For full details, please check the Azure Monitor pricing page, the Azure Monitor pricing calculator, the Microsoft Sentinel pricing page, and the Microsoft Sentinel pricing calculator.

Benefits of Multi-Tier Logging

So, what are the benefits of putting all of this inside a single Azure Monitor (Log Analytics) product? Because we can do this with all kinds of other products, like Azure Data Explorer (ADX). It means that we have the same skills and integrations that apply to all the logs.

We have the same query language, API, and integration with other tools; for example, if we configure the platform with Terraform, we have the same Terraform integration. We ingest data exactly the same way. We also have the pipeline that controls all of the tiers (more on this below).

It’s also easy to switch between different log tiers. We can start with one type and move to the other tier. You can set the table plan to Auxiliary only when you create a custom table by using the REST API (more on this below). Please note that not all tables apply to all log types, but if a table is applicable, you can go back and forth with some limitations (you can switch a table’s plan once a week). You have the same compliance, so you don’t need to manage different systems. It’s a single system that goes all the way and deals with that.

Another point is that Microsoft has a lot of out-of-the-box capabilities. So, for example, we need to keep the logs for a very long time, but we don’t want to pay a lot for them, and how long-term retention, which used to be called Archive, is a super cheap option, it’s like $0.02 per GB per month for both Azure Monitor and Sentinel. If you look at it, it’s roughly and might be even cheaper than what you pay for storage, and you don’t need to do anything; you just need to configure it, and Microsoft takes care of the long-term retention.

Prerequisites

To follow this article, you need to have the following:

1) Azure subscription – If you don’t have an Azure subscription, you can create one here for free.

2) Log Analytics workspace – To create a new workspace, follow the instructions to create a Log Analytics workspace.

3) Enable Microsoft Sentinel at no additional cost on an Azure Monitor Log Analytics workspace for the first 31 days; follow the quick onboarding process. Once Microsoft Sentinel is enabled on your Azure Monitor Log Analytics workspace, every GB of data ingested into the workspace can be retained at no charge for 90 days.

4) Ensure you have the following roles assigned on the following resources:

• Microsoft Sentinel Contributor on the Log Analytics workspace where Microsoft Sentinel is deployed.
• Or Log Analytics Contributor on the Log Analytics workspace.

5) A data collection endpoint (DCE) and data collection rule (DCR) to manage data flow between log sources and your custom table with the Auxiliary plan.

6) Understanding of log table architecture and schema in Log Analytics, especially the “TimeGenerated” column.

Assuming you have all the prerequisites in place, take the following steps:

Create a custom table with the Auxiliary plan

As of today, we can only use the REST API to create a custom table with the Auxiliary plan. Hopefully, Microsoft will enable this capability using the Azure portal, CLI, and PowerShell as well.

To make things easier and faster, we will use PowerShell from Azure Cloud Shell to make REST API calls using the Azure Monitor Tables API to create the custom table with the Auxiliary plan.

Open the Cloud Shell at (https://shell.azure.com), then sign in with your account and ensure the environment is set to PowerShell and NOT to Bash.

Then copy the following PowerShell code and paste it into the Cloud Shell prompt to run it. You could use the Log Analytics REST API to get and export the entire schema for an existing table, the “CommonSecurityLog” native table, for example, and use it to create the custom table “CommonSecurityLog_CL“ by running the following command.

Please note that you need to replace the variables in the Path parameter with the appropriate values of your subscription, resource group, and workspace name:

$customTableParams = (Invoke-AzRestMethod -Method GET -Path "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.OperationalInsights/workspaces/{workspaceName}/tables/{tableName}?api-version=2023-01-01-preview").Content

Next, we need to update the value for “totalRetentionInDays” to 365 days (1 year) and set the “plan” to Auxiliary, as well as changing the custom table schema name for the JSON payload using Regex by running the following PowerShell commands:

# Set TotalRetentionInDays to 365 days
$customTableParams = $customTableParams -replace '("totalRetentionInDays"\s*:\s*)\d+', '${1}365'

# Set Plan type to Auxiliary
$customTableParams = $customTableParams -replace '("plan"\s*:\s*")[^"]*', '${1}Auxiliary'

# Remove the retentionInDays properties from the original table schema
$customTableParams = $customTableParams -replace '"retentionInDays":\s*90\s*', ''
$customTableParams = $customTableParams -replace '"retentionInDaysAsDefault":\s*true,\s*', ''
$customTableParams = $customTableParams -replace '"totalRetentionInDaysAsDefault":\s*true,\s*', ''

# Split the JSON string into two parts: before and after the first "name" match
$parts = $customTableParams -split '("name"\s*:\s*")[^"]*', 2 

# Reconstruct the JSON with the desired replacement for the first match
$customTableParams = $parts[0] + '"name": "CommonSecurityLog_CL' + $parts[2]

Last, replace the variables in the Path parameter with the appropriate values of your subscription, resource group, and workspace name in the “Invoke-AzRestMethod” command below to create the Auxiliary custom table:

Invoke-AzRestMethod -Method PUT -payload $customTableParams -Path "/subscriptions/{subscriptionId}/resourcegroups/{resourcegroup}/providers/microsoft.operationalinsights/workspaces/{workspacename}/tables/CommonSecurityLog_CL?api-version=2023-01-01-preview"

You should receive a similar output to the one below with the “StatusCode: 202“.

Next, switch to the Azure portal, then go to your “Log Analytics workspace > Tables” and verify that the new custom table has been created with the Auxiliary plan.

To automate the entire process further, you can deploy the following ARM template to your Microsoft Sentinel workspace to create a custom table with the Auxiliary plan enabled, including data collection rules (DCR) and a data collection endpoint (DCE):

Send data to a table with the Auxiliary plan

Next, we need to send data to a table with the Auxiliary plan. As of today, there are currently two ways to ingest data to a custom table with the Auxiliary plan:

With Azure Monitor Agent, you can collect logs from a text file or a JSON file. The methods are documented below:

• Collect logs from a text file with Azure Monitor Agent
• Collect logs from a JSON file with Azure Monitor Agent

The next option is to send data to Azure Monitor using the Logs ingestion API. To learn more, follow this tutorial: Send data to Azure Monitor using Logs ingestion API (Resource Manager templates).

You need a Microsoft Entra application and a Data Collection Rule (DCR). In the DCR endpoint JSON file, you need to update the following:

• streamDeclarations: Column definitions of the incoming data. This should include the same columns you set when you create a custom table with the Auxiliary plan. You can copy the columns from the $customTableParams variable. Before “columns:[ { } ]“, you should update the custom table name (i.e. Custom-CommonSecurityLog_CL), note “Custom-” before the table name.
• destinations: Is the name of your Log Analytics workspace.
• dataFlows: Matches the stream with the destination workspace and specifies the transformation query and the destination custom table (i.e. Custom-CommonSecurityLog_CL), note “Custom-” before the table name. The output of the destination query is sent to the destination custom table.

Learn more on how to create data transformations using DCR.

Summary Rules in Microsoft Sentinel

The next announcement we will discuss is Summary Rules. We think Summary Rules will be highly usable and that many customers will use them. They allow you to run recurrent rules that aggregate data.

We have the data coming up, and you can use it to do a couple of things. First of all, you can handle a lot of high-verbose data, so you have a lot of data coming, but you don’t need the actual logs themselves; you just need the aggregation, an hourly aggregation, for example. The Summary rules are basically KQL that runs every 20 minutes as minimum or hour or day as maximum and creates these aggregations. Based on our experience, customers usually can use the aggregations for the day-to-day and not the actual raw data.

Another thing you can do with Summary rules is improve query performance. So when it comes to the very large-scale logs, you cannot query them every time that you want to do something at the end of the day; there is physics, and it’s limited to what you can do. So, if you build your Summary rules like hourly aggregation, they are very compact, and you can query them, and then the query runs very fast and immediately gets exactly what you want.

There’s no extra cost for Summary rules. You only pay for the query and the ingestion of results to the destination table based on the table plan of the source table on which you run the query.

Create and Use Summary Rules

To create and use summary rules:

1) Ensure you have the necessary permissions and Microsoft Sentinel is enabled. In the Azure portal, from the Microsoft Sentinel navigation menu, under Configuration, select Summary rules (Preview). In the Microsoft Defender portal, select Microsoft Sentinel > Configuration > Summary rules.

2) Test your summary rule query on the Logs page before creating your rule. Verify that the query doesn’t reach or near the query limit, and check that the query produces the intended schema and expected results.

3) Create a new rule in Configuration > Summary Rules (Preview). In the Summary rule wizard, give it a name and description, and select your existing custom table with an Analytics plan. You can also create a new custom log table as Analytics directly from here. Please note that the plan for the summarized table can only be Analytics and not Basic.

To access the full summary rules experience, it’s recommended to enable SummaryLogs diagnostic settings or select Configure diagnostic settings for advanced configuration, as shown in the figure below. If the diagnostic data is forwarded to a Log Analytics workspace, it can be queried with KQL. The query below can be used to list all failures related to Summary Logs.

AzureActivity
| where OperationNameValue startswith @"MICROSOFT.OPERATIONALINSIGHTS/WORKSPACES/SUMMARYLOGS"
| where ActivityStatusValue != "Success"

4) Next, define parameters, enter your query logic, and set scheduling options. In this example, we configure the summary to run daily, so the query adds new summary records every day until it expires.

The configuration of Summary Rules is important; this decides how often your rule puts data in the new table. The more often you run the rule, the more data is written, thus raising the cost. In this case, we only run it once every 24 hours. The delay is also important because most of the ingested data to Sentinel has a delay of several (in some cases, a lot more) minutes.

Lastly, the rules can be scheduled to start at a specific time in UTC. This is especially important if you summarize in combination with bin() operators.

5) Last, preview the results and create the rule.

Sample Scenarios for Summary Rules

Summary rules in Microsoft Sentinel offer flexible solutions for various log management scenarios. Here is a practical example of how these rules can be applied to different use cases:

Detecting Malicious IP Activity:

As a threat hunter, one of our team’s objectives is to identify all occurrences within the last 90 days where a malicious IP address interacted in the network traffic logs during an active incident. Microsoft Sentinel currently ingests multiple terabytes of network logs per day, making it necessary to swiftly sift through them to locate matches for the malicious IP address.

You can use the following KQL logic to create a summary data set for each IP address related to the incident, including the SourceIP, DestinationIP, MaliciousIP, RemoteIP, each listing important attributes, such as IPType, FirstTimeSeen, and LastTimeSeen.

let csl_columnmatch=(column_name: string) {
CommonSecurityLog
| where TimeGenerated > startofday(ago(1d))
| where isnotempty(column_name)
| extend
    Date = format_datetime(TimeGenerated, "yyyy-MM-dd"),
    IPaddress = column_ifexists(column_name, ""),
    FieldName = column_name
| extend IPType = iff(ipv4_is_private(IPaddress) == true, "Private", "Public")
| where isnotempty(IPaddress)
| project Date, TimeGenerated, IPaddress, FieldName, IPType, DeviceVendor
| summarize count(), FirstTimeSeen = min(TimeGenerated), LastTimeSeen = min(TimeGenerated) by Date, IPaddress, FieldName, IPType, DeviceVendor
};
union csl_columnmatch("SourceIP")
    , csl_columnmatch("DestinationIP") 
    , csl_columnmatch("MaliciousIP")
    , csl_columnmatch("RemoteIP")
// Further summarization can be done per IPaddress to remove duplicates per day on larger timeframe for the first run
| summarize make_set(FieldName), make_set(DeviceVendor) by IPType, IPaddress

By using summary rules, you can achieve more efficient data management and analysis in Microsoft Sentinel. This approach can significantly reduce the volume of data stored while maintaining valuable insights for security operations.

Summary rules in Microsoft Sentinel are a game-changer for organizations dealing with large volumes of security data. They offer a perfect balance between data reduction and maintaining critical security insights.

You can find more use cases for Summary rules in the official Microsoft Sentinel documentation.

Combining Multi-Tier Logging

So, how does this Multi-Tier Logging fit all together?

We have a common ingestion pipe in Azure Monitor. All the logs come through the pipe, whether through the API, the agent, or Azure resources. The Azure Monitor pipeline has a lot of capabilities. First of all, you can filter logs. It’s something that Microsoft added over the last year. We can configure the pipeline to filter logs so we can write our KQL query that basically removes stuff or removes entire lines of records.

So, we can filter it in the pipe before we get to the actual storage, and we don’t pay for them; however, if you filter too much, there is some payment that is going on, but up to 50% filtering, it’s free. Read about the cost of transformations.

// Related: Optimize Log Ingestion and Access in Microsoft Sentinel.

The next point is about transformations. In many cases, the logs are not really coming from all kinds of different sources that look different and have different things, and you want to make them also easy to consume. So, instead of doing query time, and we saw many customers who do magic in query time, you can also take care of this in the ingestion pipeline.

The magic in all of this in the Common Ingestion Pipe is that it happens in an instant instantly and immediately; there is no latency. You don’t need to deploy another product, and everything is happening immediately. The ingestion there is optimized by Microsoft for scale because when this pipe handles petabytes of data, it cannot do petabytes of data if, for example, they run a program per each type, so it’s very precise and very optimized to do that. And we configure all of this using data collection rules (DCRs) that also could define the routing of the data.

The magic is happening where we are going very large, and we don’t want to store the data; we don’t want the processing of logs to be very long because then the alerts would be long, and then the dashboard would not load. Microsoft is investing a lot of effort to make sure that this is very fast, and most of it is included inside the log ingestion and doesn’t require any additional cost.

Then, we have the three options we mentioned previously: Analytics logs, Basic logs, and Auxiliary Logs. You can select whether the logs coming in each table are of this type or the other. As we mentioned previously, the Summary rules are a way to parse the data, and we also have interactive queries on all the data that is interactive retention.

The long-term retention data is queried using something called Search Jobs. Search Jobs is a kind of asynchronous query that you run, and it is optimized; for example, if you want to get all the records that have IP address blocks or find all activities done by a specific user. This is what a Search job is built for; it’s running asynchronous, and it can scan petabytes of data. It’s available for you as much as you want as a managed service, so you don’t need to do all kinds of piping, fetching, and hydrating data.

Long-term retention (formerly known as Archive) can also be for two months only; you don’t need to do it for 12 years; it’s up to 12 years, and you can do it for two months. So you have the interactive, and on top of that, you add the long-term retention.

For most needs, Microsoft has designed and listed several use cases, such as forensic analysis, auditing, and so on, and this is exactly why they built Search Jobs. Search Jobs is built exactly for these kinds of scenarios; the full power of a KQL query is available only for a small period of time, like 30 days, but you still have the full ability to run Search Jobs that get you data for how much time you want.

Comparing Multi-Tier Logging Plans

Let’s look at the details and compare all the logging tier (Analytics, Basic, and Auxiliary Logs) plans.

As we see in the table below, since the early days of Sentinel, we have had interactive retention days for Analytics logs. Microsoft has extended the interactive retention for Basic logs to 30 days; it used to be 8 days, which was too low, and customers want more days to be interactive retention for Basic logs. Microsoft has just extended that way more than we used to have 8 days, and we have 30 days for Auxiliary logs.

If you have Microsoft Sentinel or Application Insight, we have 90 days included retention for Analytics logs. Again, you can extend this interactive retention with an Analytics log for up to 2 years.

Now, when it comes to queries, Analytics logs have full KQL capabilities, and they provide rather fast query capabilities. Then, when we go to Basic logs, there are some limitations, so the queries are still fast because, behind the scenes, they are fully indexed. Microsoft is using Azure Data Explorer (ADX) behind the scenes, but we have some limitations on the query, so Microsoft had very strict limitations in Basic logs before, and they just extended this dramatically recently. So, we can basically run all KQL on a single table, and we can lookup other tables that are Analytics logs, which is the most common way we use them.

// Related: Optimize Microsoft Sentinel Log Retention With Azure Data Explorer.

Lookups, if you don’t know them, are kind of like leftouter join, but they are much easier to use than join operator, and they are much more effective and require fewer resources. Leftouter means that it is distributed, so it’s kind of like running in parallel on every one of the nodes of the cluster. As a best practice, you should start using lookup (leftouter join) instead of join, so you would get much better results, it would work faster, and the system would also like to invest much fewer resources. This is something Microsoft has changed and recently added more capabilities. So, go ahead and try them if you have Basic logs.

Now, with Auxiliary logs, Microsoft gives you the same KQL query capabilities, but by design, it would be slower. There is slightly different technology behind the scenes. Microsoft is using storage accounts, which are slower and not fully indexed; that’s one of the reasons that it’s slow. Microsoft is trying to make it very efficient. If you do like very pinpoint in-time queries, if you do something else, it might run slower than what you need, but again, you have a Search job, which is the recommended way of scanning all this data.

When it comes to query cost, there is no query cost in Analytics logs. However, there is a cost per GB scan for Basic and Auxiliary logs. Based on our experience, most of the customers don’t spend money too much here. In most cases, these kinds of logs are logs that you less frequently query, and with something like the new Summary rule, you don’t really need to query that because it is the raw data that you use later in case just for investigations, so you don’t really query day in and day out, and this is still the Analytics log where you do that.

The last thing is on Insights and Alerts. Not all the tables are supported on Basic and Auxiliary logs. Every insight, like the application insight, the container insight, and others, has a different rule about what’s supported and whatnot. Alerts also work only on Analytics logs because we need this power when we run these log alerts.

Capabilities	Analytics Logs	Basic Logs	Auxiliary Logs
Included interactive retention	31 days, 90 days for Sentinel and App Insights	30 days	30 days
Can add interactive retention	Up to 2 years	-	-
KQL Queries	Full analytics	Full KQL on a single table with lookup to analytics	Full KQL on a single table with lookup to analytics
Query performance	Fast	Fast	Slower
Query cost	No	Cost per scanned GB	Cost per scanned GB
Insights and Alerts	Supported	-	-

Multi-Tier Logging Example

Let’s see an example of how to use Multi-Tier Logging in Microsoft Sentinel using firewall logs.

So, we have a system where you have firewall logs and firewall logs are something pretty big and highly verbose, right? Many organizations need to keep all the records, though some of the records are completely useless if you have internal IP addresses talking to each other. However, we need to keep those logs for compliance, for example, especially if it’s the network that is coming in or out of your network, so we need to keep it, but we don’t want to pay much for it. This is why we have Auxiliary logs.

If there is data that is entirely unnecessary, for example, internal traffic within the organization, there are many situations where the internal traffic in the organization goes into the firewall log so you can write your filter and just get the data out. You don’t need to pay anything for that, not even the cost of Auxiliary logs, which are much cheaper.

Then we have these huge chunks of logs, but no one actually needs to query daily. We just need the summaries of the results. This is why we have a Summary rule. As we see in the diagram below, we go to our raw log and define the Summary rule on top of it. A Summary rule, for example, could be something like taking the source and destination IPs and creating hourly summaries so you have a record per every tuple (five-tuple, two-tuple, or three-tuple) that your firewall will sync over an hour.

Now, for most of our investigations, that data would actually be fine since we don’t need all of the logs. When we talk to security investigators, they usually come and say like, hey, show me which IP addresses, do I have the specific IP address in my network? And if we have a specific IP address that we want to investigate, we want to see which other IPs talk to this IP address. We don’t really need all the details; we need all the details if something really bad happens. The pattern that we recommend our customers to do is to have very compact aggregations.

This is just one example of firewall logs; you can find more log sources to use for Auxiliary log ingestion.

Combining Auxiliary Logs and Summary Rules

Combining Auxiliary Logs and Summary Rules in Microsoft Sentinel provides a cost-effective approach to handling large volumes of log data. This strategy optimizes log ingestion and retention, delivering benefits such as cost savings and improved security monitoring.

Auxiliary Logs are designed for verbose logs typically required for compliance and security scenarios. These logs are ingested at a lower cost compared to Analytics Logs and are ideal for data that doesn’t need frequent querying but still requires long-term retention.

Summary Rules enable data aggregation at the point of ingestion. By summarizing data, you pre-process it into a more compact form, making subsequent analysis and querying faster and more efficient. These summaries can be stored in tables under the Analytics plan, leveraging the performance benefits of Analytics Logs without incurring high storage costs.

Benefits of Using Auxiliary Logs with Summary Rules

Cost Savings: Storing verbose logs in Auxiliary Logs reduces storage costs, while summaries derived from these logs are stored in Analytics Logs for quick retrieval and analysis.

Efficient Log Management: Using Summary Rules to aggregate data reduces the volume of data stored in high-cost tiers. You can summarize logs hourly or daily, ensuring critical data is always available for immediate query in the Analytics Logs.

Improved Security Monitoring: Summarized data provides a high-level view of log activity, making it easier to detect anomalies and potential security threats.

Implementing Auxiliary Logs and Summary Rules

Set Up Auxiliary Logs: Configure your Log Analytics workspace and set up tables for Auxiliary Logs.

Configure Data Ingestion: Use the Azure Monitor Agent or Logs Ingestion API to ingest data into your Auxiliary Logs tables.

Define Summary Rules: Develop KQL queries to summarize your data based on specific security and monitoring needs.

Store Summary Data in Analytics Logs: Configure summary rules to store aggregated data in custom tables under the Analytics plan.

Monitor and Optimize: Use the summarized data for continuous monitoring, set up alerts on critical metrics, and optimize the configuration based on query performance and storage costs.

This integration allows for balancing extensive log retention with cost management, ensuring necessary data is accessible for daily operations and compliance requirements without overwhelming budgets. By leveraging the strengths of both Auxiliary Logs and Summary Rules, organizations can achieve a more robust and cost-effective security monitoring solution.

The combination of Auxiliary Logs and Summary Rules in Microsoft Sentinel represents a paradigm shift in log management, offering a perfect blend of comprehensive data retention and efficient analysis capabilities.

Limitations of Auxiliary Logs

Auxiliary Logs, currently in public preview, were announced at the end of July 2024 and offer the potential for cost-effective and efficient log management. However, they come with certain limitations that customers should be aware of:

* Billing: Although the Auxiliary plan is in public preview and the billing model is well defined, the actual billing is NOT yet enabled. So, if you ingest data into an Auxiliary table, the ingestion, long-term retention if you keep the data for longer than 30 days, queries, and search jobs are unbilled at this point in time.

* Regional Availability: Support is limited to specific and top regions, including Central and East US and select areas in Europe, Asia Pacific, and the Middle East. This may affect organizations requiring uniform log management across multiple regions.

* Retention Period: Microsoft declares that they support up to 12 years of long-term retention in all three plans: Analytics, Basic, and Auxiliary logs. Today, Auxiliary Logs have a fixed retention period of 365 days (1 year), which may not meet some organizations’ long-term storage requirements.

On the other hand, Microsoft currently doesn’t support shorter retention periods if you want to keep your data for less than a year. However, as of today, Microsoft won’t charge you extra for keeping your data for a longer period. If you are required by regulations to keep the data for only 30 days, then Auxiliary logs are not yet ready for that, as the data is retained for one year.

* Data Structure: Auxiliary Logs do not support dynamic data columns, limiting flexibility in the log data schema. They support only standard types. Today, you will have to parse the data into standard types first. This means that you can technically push a dynamic object into a column as text, but since the datatype isn’t supported, you’ll need to convert (parse_json(), todynamic(), or whatever) upon the query. Check the dynamic data type in Kusto.

* Ingestion Process: Data ingestion requires a data collection rule (DCR-based) custom table created through the Tables – Create Or Update API. This approach ensures structured logs but may be seen as restrictive by some users. Another limitation is that ingestion-time transformation, where you can filter some of the data, is not yet supported for the Auxiliary table. You can use third-party tools like Logstash or Cribl to parse the data.

* Search Capabilities: While supported, comprehensive queries on historical data are performed through search jobs, as direct queries are slower by design.

* Data Routing: Each data collection rule can direct logs to only a single custom table, potentially requiring multiple DCR rules for multi-destination routing.

All limitations are documented on the official documentation page. Some main limitations benefit you, and some are less beneficial. As Microsoft refines Auxiliary Logs based on customer feedback, potential future improvements may include:

1. Enable billing
2. Expanded geographic availability
3. Flexible retention periods beyond 365 days up to 12 years
4. Support for dynamic data columns
5. Enhanced ingestion methods
6. Data transformation capabilities during ingestion

By understanding these current limitations and potential enhancements, you can better align the log management strategies with the evolving capabilities of Azure Monitor (Log Analytics workspace) and Microsoft Sentinel.

In Summary

By integrating Auxiliary Logs and Summary Rules, you can achieve a balanced, cost-effective approach to log management and security monitoring. This combination allows for efficient data retention and quick access to critical information, enhancing your overall operational efficiency and security posture. The integration of Auxiliary Logs and Summary Rules represents a significant step forward in optimizing log management and security monitoring for organizations of all sizes.

Summary rules in Azure Monitor and Microsoft Sentinel aggregate large data sets, offering benefits in analysis, cost efficiency, and security that complement Auxiliary Logs. They pre-aggregate data during ingestion and store summaries in custom log tables under the Auxiliary, Basic, or Analytics data plan.

__
Thank you for reading our blog.

Please let us know in the comments section below if you have any questions or feedback.

-Charbel Nemnom-