Windows Event Forwarding (WEF) reads any operational or administrative event log on a device in your organization and forwards the events you choose to a Windows Event Collector (WEC) server. By using Windows Event Forwarding (WEF) connector, we could stream our logs from any Windows Servers connected to the Microsoft Sentinel workspace using Azure Monitor Agent (AMA).
In this article, we will share how to resolve the Windows Forwarded Events connector not populating the computer names in the WindowsEvent table in Log Analytics.
Table of Contents
Windows Forwarded Events and Microsoft Sentinel
Windows Forwarded Events is a feature of the Windows operating system that allows event logs to be collected and forwarded to a central location for analysis. This feature helps you to monitor and troubleshoot events across multiple computers from a central location.
When a computer running Windows generates an event, it writes the event to its local event log. With Windows Forwarded Events, you can configure computers to forward events to a central collector computer known as Windows Event Collector (WEC), which can then analyze the events from multiple sources.
The forwarded events can be filtered based on criteria such as event ID, source, and keywords to help you focus on specific types of events. Windows Event Forwarding (WEF) isn’t something new, I believe it has been around for more than 20 years now. This feature can be particularly useful for large-scale deployments, but the ability to query the logs has never been its strong point, plus the storage can be an issue.
Fortunately, with Microsoft Sentinel, we could leverage the built-in Windows Forwarded Events connector, which is based on the new Azure Monitor Agent (AMA), to stream our logs from the WEF/WEC server to the Log Analytics workspace, and overcome the challenging issue of querying and storing the logs, and without having the AMA agent installed on each computer or server.
The Issue
We deployed Windows Forwarded Events (Preview) connector to non-Azure VMs using Azure Arc and the AMA extension. We also enabled Asim normalization support on the connector page.
We have followed exactly the steps as documented by Microsoft: Forward On-Premises Windows Security Event Logs to Microsoft Sentinel. We have noticed that Windows Forwarded Events (WEF) connector is not populating the individual computer names in the WindowsEvent Table in Log Analytics, we can see only the computer name of the WEF/WEC server as shown in the figure below.
WindowsEvent
| summarize count() by Computer
And the event details for the collected machines are located under the Computer Name of the collector machine > EventData > TargetUserName_ as shown in the figure below.
WindowsEvent
| where EventID == '4624'
| extend TargetUserName_ = tostring(EventData.TargetUserName)
| project Computer, TargetUserName_, EventID, EventLevelName
| take 1
However, we can see all the “Forwarded Events” logs in Event Viewer populating from subscribers to the WEC subscription on the forwarder server, and we can see the reporting in the event itself belongs to the subscribed servers, but in Log Analytics, we can see only the WEF/WEC computer name in the WindowsEvents table.
Now, based on what Microsoft has documented, we are supposed to see all the subscribed servers in the WindowsEvents table as shown in the figure below:
So, what is the issue then?
Troubleshooting
As you probably know, the new Windows Event Forwarding (WEF) connector based on AMA uses a combination of Data Connection Rules (DCR) and Data Connector Rules Association (DCRA). DCRs define what data to collect and where it should be sent.
What we have done, we’ve filtered the events at the connector level by creating a DCR rule and using custom XPath queries under the Collect tab as shown in the figure below.
In this example, we are collecting only Security events with Event ID = 4624
Security!*[System[(EventID=4624)]]And we are collecting multiple Security events with EventID=4634, EventID=4740, EventID=4672, and so on.
Security!*[System[(EventID=4634 or EventID=4740 or EventID=4672 or EventID=4625 or EventID=4782 or EventID=4767 or EventID=4728 or EventID=4732 or EventID=4756)]]> Learn More: Windows security event IDs that can be sent to Microsoft Sentinel.
And by the way, if you would like to test and verify your XPath queries before you use it, you could run the following PowerShell command as Administrator:
Get-WinEvent -LogName Security -FilterXPath '*[System[(EventID=4624 or EventID=4634 or EventID=4740 or EventID=4672 or EventID=4625 or EventID=4767 or EventID=4728)]]' | FT -AutoSize
For more detailed information about setting up the Windows Security Events connector with the Azure Monitor Agents (AMA), please take a look at the Microsoft documentation.
Finding the Cause
After we started eliminating the potential root causes following the detailed guide by Microsoft, we noticed that they are collecting All Events and they did not set a DCR filter to collect a specific event in the Windows Event Forwarding (WEF) connector page.
They are also collecting “All Event IDs” for the subscribed computers on the WEF collector side. This is because of the example demonstrated in the guide but not as a best practice.
As you probably know, you’re charged for any data you collect in a Log Analytics workspace. Therefore, as a best practice, we should only collect the event data that we need.
For more information about filtering events using XPath queries, please check the official documentation.
Fixing the Issue
Now to fix this issue, we’ve decided to use the “Query Filter” page on the WEF collector side to only forward events that we are interested in capturing, and keep collecting “All Events” on the WEF connector page (Sentinel side).
As a side note, the “Query Filter” page in the Subscription Properties, allows you to filter and only forward events that you are interested in capturing as shown in the figure below. This filter will be used by all computer subscribers that are forwarding events. These events will all be sent to the WEC server, and then ingested into Sentinel.
As a second option, if you would like the WEC server to capture all events but filter this list before sending it to the Log Analytics Workspace, there is a second filter definition on the Microsoft Sentinel Windows Event Forwarding (WEF) connector page that you could set. However, this option did not work for us as we mentioned earlier.
After we’ve set the XPath query in the “Query Filter” page through the Event Viewer UI, as shown in the example below:
<QueryList>
<Query Id="0" Path="Security">
<Select Path="Security">
*[System[(EventID=4624 or EventID=4634 or
EventID=4740 or EventID=4672 or
EventID=4625 or EventID=4782 or
EventID=4767 or EventID=4728 or
EventID=4688 or EventID=1102 or
EventID=4732 or EventID=4756)]]
</Select>
<Suppress Path="Security">
*[System[(EventID='5156')]] and
(*[EventData[Data[@Name="DestAddress"] = "192.168.2.5"]])
</Suppress>
</Query>
</QueryList>We started to see the correct computer names (subscribed servers) in the WindowsEvent table in Log Analytics as shown in the figure below.
Please note that the <Suppress Path=”Security”>*</Suppress> is optional here. This is an XPath query that identifies what to exclude in the result set. In this example, we are suppressing Event ID 5156 on the WEC collector server. The XPath query is part of a QueryList node which allows you to define and run multiple queries at once.
As a side note, you can use Event Viewer UI to create XPath queries for Microsoft Sentinel DCRs. You open up Event Viewer on any Windows system and then select the log file which you want to pull Event IDs from.
[1] Choose the Filter Current Log… option, then…
[2] Enter the Event IDs you want to collect, and then…
[3] Go to the XML tab in the filter to find the XPath query.
[4] Lastly, take the XPath query part as shown in the figure below, and copy and paste it into your new DCR in the Windows Security Events or Windows Forwarded Events Data Connector page.
That’s it! You now know how to fix the Windows Event Forwarding (WEF) connector not populating the subscribed computer’s names in the WindowsEvent table.
The WEF Collector Server is Down
Now you may be asking, what happens in the event that the WEF/WEC collector server is down for either update reasons or just plain crashes?
Do computers that are sending logs, keep the current state, and then forward the logs once the collector becomes available, or all the logs that were sent during the collector server that was down are lost?
We’ve tested that exact scenario by taking down the WEF collector for a few minutes and then bringing it back online, and we have timed the power-off and power-on.
The good news is that the logs are kept locally on clients until the WEF collector becomes available and then forwarded. So you don’t lose logs if WEF is down, it will just forward when it becomes available without losing the track.
Another important point is the timestamp for the logs stays correct. So, if the WEF server goes down at 17:00 and comes back online at 17:10, when clients send logs that were generated in between those times, in Microsoft Sentinel, you will see logs time as 17:01, 17:03, 17:05, etc. So, keep this in mind when you are building Analytic Rules.
Summary
In this article, we showed you how to resolve the Windows Forwarded Events connector not populating the computer names in the WindowsEvent table in Log Analytics Workspace.
WEF can be particularly useful for large-scale deployments, where you need to monitor events from multiple computers and identify potential issues or security threats, and send them to Microsoft Sentinel instead of installing the Azure Monitor Agent (AMA) on each computer.
__
Thank you for reading my blog.
If you have any questions or feedback, please leave a comment.
-Charbel Nemnom-
