In this article, we will share with you how to optimize and reduce your costs in Microsoft Sentinel and save on your bill.
Table of Contents
Optimize Costs in Microsoft Sentinel
Microsoft Sentinel is a cloud-native Security Information Event Management (SIEM) and Security Orchestration Automated Response (SOAR) solution. Microsoft Sentinel delivers intelligent security analytics and threat intelligence across the enterprise, providing a single solution for alert detection, threat visibility, proactive hunting, and threat response.
A quick example, SIEM has the ability to correlate events from one or more devices to provide additional details about an incident. For instance, an attacker performing a brute-force password attack on a web server may be generating alerts on the web server itself and also on the firewall and intrusion detection system. A SIEM would portray the incident using events from these and possibly other devices to give you a richer depiction of the incident.
Microsoft recently announced a new set of features (in public preview) that will help to reduce and optimize your costs in Microsoft Sentinel. Those features are called: Basic Logs, Archive tier, Search jobs, and Data restoration.
In this article, we will dive into those new features, discuss their pros and cons, and see how they function to bring your costs down.
Basic Logs
Basic logs are a new way of ingesting data at a low cost into Microsoft Sentinel. Every environment has telemetry that has security value but it’s normally not needed for detections, so ideally we would like to be able to keep all the data in the SIEM so that security analysts can search over it, but at the same time, we don’t need to incur the high ingestion costs for this data.
With basic logs, you have the best of both worlds. You can have this data in your SIEM at very low ingestion costs. Microsoft claimed is around -75% cheaper than the normal data ingestion cost.
If we jump over to the Microsoft Sentinel Pricing page, we can see that the normal data ingestion cost is $2.46 per GB-ingested under the Pay-As-You-Go tier, and for Basic Logs, it’s $0.50 per GB of data ingested. However, searching in Basic Logs is not free compared to Analytics Logs, the search queries are at $0.007 per GB of data scanned.
This is great, so let’s abandon the normal data ingestion logs and switch over to Basic logs. No, not so fast… There are some restrictions. Keep reading!
First of all, basic logs cannot be used in the analytic rules or log alerts. Also, the KQL language is limited, it has a great subset of features like filtering (where), parsing, projecting, and extending, including the following operators:
You can use all functions and binary operators within these operators. But you can’t do advanced things like join, union, and sort. Basic logs are really scoped to whatever you can do with the where operator.
The second restriction is, that not all tables are supported for Basic Logs yet.
Another important thing is, that these logs are retained for eight days only before they’re moved to the archive tier, and how long you keep them in the archive tier is up to you (more on this in the next section).
Configure Basic Logs
Now how to configure Basic Logs?
As we just discussed, configuring a table’s log data plan to Basic Logs lets you save on the cost of storing high-volume verbose logs you use for debugging, troubleshooting, and auditing, but not for analytics and alerts.
By default, all tables in your Log Analytics are Analytics tables. You need to configure particular tables to use Basic Logs. At the time of this writing (public preview), you can configure the following tables for Basic Logs only and more sources will be supported over time:
> All tables that are created with the Data Collection Rule (DCR)-based custom logs API.
> ContainerLog and ContainerLogV2, which Container Insights uses and which include verbose text-based log records.
> AppTraces, which contains freeform log records for application traces in Application Insights.
If you want to convert and enable basic logs on an existing table, you would need first to send custom logs to Azure Monitor Logs with REST API as described in Microsoft Docs. Then you would need to ingest the Microsoft Defender 365 (M365D data) for example as _CL rather than via the built-in data connector for them to be eligible to use for basic ingestion.
At the time of this writing, configuring Basic Logs cannot be set in the Azure portal yet. You need to use the REST API call or Azure CLI. In this example, we will show you the CLI option.
To configure a table for Basic Logs or Analytics Logs, you need to run the az monitor log-analytics workspace table update command and then set the –plan parameter to either Basic or Analytics.
\Example:
# Set Table to Basic Logs
az monitor log-analytics workspace table update --subscription SubID --resource-group RG-WEU-LAW --workspace-name LAWorkspace --name ContainerLog --plan Basic
# Set Table to Analytics Logs
az monitor log-analytics workspace table update --subscription SubID --resource-group RG-WEU-LAW --workspace-name LAWorkspace --name ContainerLog --plan Analytics
To check the configuration settings of a table, you could use the Azure portal, REST API, or Azure CLI.
You need to run the az monitor log-analytics workspace table show command below.
# Check Table configuration
az monitor log-analytics workspace table show --subscription SubID --resource-group RG-WEU-LAW --workspace-name LAWorkspace --name AppTraces --output table
This will give you the Plan details for the table (Basic or Analytics).
Name Plan ProvisioningState ResourceGroup
——— —— ——————- ——————–
AppTraces Basic Succeeded rg-az-security
Please note that you can switch plans (Analytics Logs <=> Basic Logs) once per week ONLY.
Archive Tier
This new archive tier provides cheaper retention costs for long-term data storage compared to the log analytics retention costs. Microsoft claimed to be -80% savings compared to the normal log analytics data retention costs.
So while keeping this data available for queries for security analysts, this new archive tier can be used/enabled for both basic logs and analytic logs. Microsoft now called what we used to have as normal logs or regular logs, analytic logs. So now we have two ingestion plans (Analytics Logs and Basic Logs).
The great news is, that the new archive tier can keep data for up to seven years. As you are probably aware up until now, we could only keep the data for up to two years (730 days). This is a great improvement to meet your security compliance log retention requirements.
Configure Archive Tier
Now how to configure long-term retention for the archive tier?
At the time of this writing, the long-term data retention cannot be configured in the Azure portal yet. You need to use the REST API call, PowerShell, or Microsoft Sentinel workbook which uses the API calls. Microsoft released a nifty PowerShell tool that will automate the entire process for you.
You first need to download the most up-to-date PowerShell script from this URL. Next, you need to extract the folder and open “Configure-Long-Term-Retention.ps1” either in Visual Studio Code or PowerShell.
Please note that the script runs from the user’s machine. You must allow PowerShell script execution. To do so, run the following command:
Set-ExecutionPolicy -Scope Process -ExecutionPolicy BypassThe script will prompt you to enter your Azure Tenant Id, then you are prompted to authenticate with your Azure credentials, once you’re authenticated, you will be prompted to select: The desired subscription, the Log Analytics Workspace for Sentinel, Table Plan (choose between basic logs or analytic logs), and the Archive Days.
Please note that the analytics plan is only per single table configuration and not for the entire log analytics workspace. So you need to choose one or multiple tables that you want to apply for long-term retention as shown in the figure below.
Then you need to enter the value for the archive days that you want to apply. The value beyond two years is restricted to full years. Allowed values are: [4-730], 1095, 1460, 1826, 2191, 2556 days.
The archive retention in days is calculated by using the RetentionInArchive = TotalLogRetention – RetentionInWorskspace.
Last, you select OK to apply and update the desired table with the new archive value for long-term retention.
Tables in your Log Analytics workspace always had the ability to deviate from the global workspace retention level to shorten or even extend the lifetime of your data. But because you could only configure this via an API call, this was considered a little “secret”.
Now, with Basic and Archive logs Microsoft also updated the UI in Log Analytics Workspace to incorporate table-level retention in (preview) with the added ability to configure the totalRetentionInDays as shown in the figure below.
Search
Once the data is in the Archive Tier, how do we access it?
The archive tier is accessed through the new Search menu under the General blade in Microsoft Sentinel.
The search feature allows you to use the same set of KQL commands that you would do with Basic Logs, it’ll execute your search jobs in the background and it will return those results even if your search needs to sweep through a lot of data to a search table in the Workspace that has all your search results in it. And once that data is in your search results table, you can do joins and any kind of KQL queries you want to perform on that data.
In the Search box, enter the search term. Then select the appropriate time range (this could be up to 7 years back with the archive tier).
Next, you need to select the Table that you want to search. And when you’re ready to start the search job, select Search.
When the search job starts, you’ll receive a notification and the job status show on the search page.
The search is essentially like a MapReduce job that breaks your search up into multiple jobs in parallel runs and executes them out all at the same time so you can perform a search across petabytes of data, the timeout is 24 hours, so you can imagine all these jobs running at the same time.
You can view your saved searches. The search results get populated into a table within the workspace, so once the custom table is created in your workspace, you can just do any kind of KQL query on it. There are no restrictions on the queries.
So when you click on any of the saved search results, it gives you the full status of the current search. You can see the search KQL query that ran in the background, you can delete it if you want, or you can also restore it (more on this in the next section).
Please note that search jobs are not free and will be charged by the amount of data scanned to complete the search. The price is $0.0062 per GB of data scanned.
Restore
Imagine you need to perform a full investigation or deep investigation on logs that are in the archive tier. In that case, the search feature might be suboptimal because it only will return back the search results that match your search.
The new restore capability in Microsoft Sentinel allows you to bring back everything in the Archive tier into Analytic logs so you can perform full KQL operations on that data.
So how data restoration works?
You take data from the archive tier, then it will automatically load it into the hot cache, meaning that it gets loaded into memory. The KQL queries are very fast, it’s the equivalent of having data just recently ingested in Analytics Logs. Then you can do all kinds of KQL queries and support more advanced query types.
The idea for the restore is to use it when you have to manage a big amount of data, for small amounts of data, you can use the search feature, but when you need to bring a large amount of data into the hot cache, it’s better to use the restore option. I strongly recommend looking for the search feature first before you restore your data and see if that is sufficient for you.
One scenario that is useful with restore is, if you have a really intense investigation case and you want to look at every single log entry or every single record event in the logs during a certain time frame, that’s what you would use restore for.
Please note that data restoration is not free and will be charged by the amount of data restored per day. The price is $0.13 per GB per day.
Summary
In this article, we showed you how to optimize your costs in Microsoft Sentinel and save on your bill.
To recap, we can now ingest data at cheaper rates with the basic logs and we can also retain that data for a longer time up to 7 years with a great discount with the archive tier.
The Microsoft Sentinel Search was developed for searching extremely large data sets across long time periods in minutes. You can search for IPs, hashes, users, etc. using a simple text search. Once a search is initiated, you can track its progress, and see instant results as search results are discovered. Next, you can leverage the Restoration capability to rehydrate archived data and make it available for high-performance KQL queries with just a few simple clicks.
With search, you can search across all sources, it doesn’t have to be basic logs. In some cases, it might make sense to use search if you’re trying to search across long time periods or across a lot of data, or an interactive type of search may not be appropriate for you because it timeouts, in this case, you would use search. Search works across all data types and restoration as well.
Microsoft is working on improving the Search and Restoration capabilities to make it easier for you, stay tuned!
> Learn more about searching large datasets in Microsoft Sentinel.
> Learn more about restoring archived logs from search in Microsoft Sentinel.
> Learn more about configuring data retention and archive policies in Azure Monitor Logs.
> Learn how to configure Basic Logs in Microsoft Sentinel.
__
Thank you for reading my blog.
If you have any questions or feedback, please leave a comment.
-Charbel Nemnom-
