Securiment Blogs

Developing and Using Advanced Security Information Model (ASim) Parsers in Analytics Rule

Microsoft Sentinel ingests data from many sources. Working with various data types and tables together requires you to understand each of them, and write and use unique sets of data for analytics rules, workbooks, and hunting queries for each type or schema.

Sometimes, you'll need separate rules, workbooks, and queries, even when data types share common elements, such as firewall devices. Correlating different kinds of data during an investigation and hunting can also be challenging.

The Advanced Security Information Model (ASIM) is a layer between these diverse sources and the user. ASIM follows the robustness principle: "Be strict in what you send, be flexible in what you accept". Using the robustness principle as a design pattern, ASIM transforms Microsoft Sentinel's inconsistent and hard-to-use source telemetry into user-friendly data.

ASIM components

The following image shows how non-normalized data can be translated into normalized content and used in Microsoft Sentinel. For example, you can start with a custom, product-specific, non-normalized table, and use a parser and a normalization schema to convert that table to normalized data. Use your normalized data in both Microsoft and custom analytics, rules, workbooks, queries, and more.

ASIM includes the following components:

Microsoft has offered an Authentication schema for authentication event standardization. The out-of-the-box parsers provided by Microsoft are for

• SignInLogs

• AWSCloudTrail

• M365Defender

• MD4IoT

• MicrosoftWindowsEvent

• OktaSSO

These parsers do not include Syslog authentication events.

In our example we’ll Parse Syslog messages in ASim format keeping in mind ASim recommended fields for ASimAuthentication schema.

Normalizing Syslog Authentication Message for ASim.

Syslog messages usually don’t follow event schemes, some messages are clearer in context than others. In our exercise, we need authentication messages specifically “authentication failure and authentication success”

In the Syslog authentication failure event, we have essentially two different kinds of Events.

• One event is for the account that is present on the Linux machine.

• Another event is where the user is not present on Linux. It gives us an “Invalid user” within the event.

Authentication success via SSH usually falls into the authpriv facility. The parsing of the event is shown below.

Before diving into the parser itself we have to look into the naming convention set by Microsoft to build a source-specific parser on the lowest level.

Naming Convention

We are going to name our parsers starting with vim (vendor information model). The reason being our parsers are vendor-specific parsers with filtering.

Authentication failure via Invalid user

//Description

// This parser will first filter out failed password events in the info level.

Syslog

| where SyslogMessage contains "failed password" and SeverityLevel == "info"

| limit 100

| parse SyslogMessage with Activity:string

" for " TargetUserName

" from " IpAddress

" port " IpPort

" " Protocol

| where TargetUserName contains "invalid user"// Filtering where the invalid user is mentioned in the event

| extend tmp_Username = split(TargetUserName," ")

| extend TargetUserName = tostring(tmp_Username[2]) //breaking the username string to find the actual username

| extend

EventVendor = 'Linux'

, EventProduct = 'Syslog'

, EventCount=int(1)

, EventSchemaVersion='0.1.0'

, EventResult = iff (Facility ==0, 'Success', 'Failure')

, EventStartTime = TimeGenerated

, EventEndTime= TimeGenerated

, EventType= 'Logon'

, SrcDvcId=tostring(Computer)

, SrcDvcHostname =tostring(HostName)

, SrcDvcOs=tostring(Computer)

| project-rename

EventOriginalUid =ProcessID

, LogonMethod = ProcessName

| project-reorder

TimeGenerated

,EventProduct

, EventOriginalUid

, EventResult

, EventStartTime

, EventEndTime

, LogonMethod

, SrcDvcId

, SrcDvcHostname

, SrcDvcOs

// Created 22-09-2022

When it's set correctly we can save it as the function:

Authentication failure via Valid-user

//Description

// This parser will first filter out authpriv facility and filtered out login failed events.

Syslog

| where Facility has "authpriv" and SeverityLevel has "info" and SyslogMessage contains "Failed password for root"

| parse SyslogMessage with Activity:string

" for " TargetUserName

" from " IPAddress

" port " IpPort

" " Protocol

| extend

EventVendor = 'Linux'

, EventProduct = 'Syslog'

, EventCount=int(1)

, EventSchemaVersion='0.1.0'

, EventResult = iff (Facility ==0, 'Success', 'Failure')

, EventOriginalResultDetails = coalesce(Facility, SeverityLevel)

, EventStartTime = TimeGenerated

, EventEndTime= TimeGenerated

, EventType= 'Logon'

, SrcDvcId=tostring(Computer)

, SrcDvcHostname =tostring(HostName)

, SrcDvcOs=tostring(SourceSystem)

, EventOriginalUid=tostring(ProcessID)

| project-rename

LogonMethod = ProcessName

| project-reorder

TimeGenerated

,EventProduct

, EventOriginalUid

, EventResult

, EventOriginalResultDetails

, EventStartTime

, EventEndTime

, LogonMethod

, SrcDvcId

, SrcDvcHostname

, SrcDvcOs

| where TargetUserName !contains "invalid user" // Filtering TargetUserName in the end because field was created during the KQL

// Created 22-09-2022

Saving as a function:

Authentication success

//Description

// This parser will first filter out authpriv facility and then check for Accepted password in the SyslogMessage

Syslog

| where Facility has "authpriv" and SyslogMessage contains "Accepted password"

| parse SyslogMessage with Activity:string

" for " TargetUserName

" from " IpAddress

" port " IpPort

" " Protocol

| extend

EventVendor = 'Linux'

, EventProduct = 'Syslog'

, EventCount=int(1)

, EventSchemaVersion='0.1.0'

, EventResult = 'Success' //We are hardcoding success in the result because it's already determined that the message is a login success

, EventStartTime = TimeGenerated

, EventEndTime= TimeGenerated

, EventType= 'Logon'

, SrcDvcId=tostring(Computer)

, SrcDvcHostname =tostring(HostName)

, SrcDvcOs=tostring(Computer)

| project-rename

EventOriginalUid =ProcessID

, LogonMethod = ProcessName

| project-reorder

TimeGenerated

,EventProduct

, EventOriginalUid

, EventResult

, EventStartTime

, EventEndTime

, LogonMethod

, SrcDvcId

, SrcDvcHostname

, SrcDvcOs

// Created 22-09-2022

Saving as the function:

All three parsers are saved separately as functions “vimSyslogAuthenticationSuccessCustom, vimSyslogInvalidUserAuthenticationCustom, vimSyslogValidAuthenticationCustom“ which then be called in the ASimAuthentication union parser.

Call Custom Parsers in ASimAuthentication

Finally, load the function code of ASimAuthentication Parser and add our custom functions at the end of the function code.

let DisabledParsers=materialize(_GetWatchlist('ASimDisabledParsers') | where SearchKey in ('Any', 'ExcludeASimAuthentication') | extend SourceSpecificParser=column_ifexists('SourceSpecificParser','') | distinct SourceSpecificParser);

let ASimAuthenticationDisabled=toscalar('ExcludeASimAuthentication' in (DisabledParsers) or 'Any' in (DisabledParsers));

union isfuzzy=true

vimAuthenticationEmpty

, ASimAuthenticationAADManagedIdentitySignInLogs (ASimAuthenticationDisabled or ('ExcludeASimAuthenticationAADManagedIdentitySignInLogs' in (DisabledParsers) ))

, ASimAuthenticationAADNonInteractiveUserSignInLogs (ASimAuthenticationDisabled or ('ExcludeASimAuthenticationAADNonInteractiveUserSignInLogs' in (DisabledParsers) ))

, ASimAuthenticationAADServicePrincipalSignInLogs (ASimAuthenticationDisabled or ('ExcludeASimAuthenticationAADServicePrincipalSignInLogs' in (DisabledParsers) ))

, ASimAuthenticationSigninLogs (ASimAuthenticationDisabled or ('ExcludeASimAuthenticationSigninLogs' in (DisabledParsers) ))

, ASimAuthenticationAWSCloudTrail (ASimAuthenticationDisabled or ('ExcludeASimAuthenticationAWSCloudTrail' in (DisabledParsers) ))

, ASimAuthenticationOktaSSO (ASimAuthenticationDisabled or ('ExcludeASimAuthenticationOktaSSO' in (DisabledParsers) ))

, ASimAuthenticationM365Defender (ASimAuthenticationDisabled or ('ExcludeASimAuthenticationM365Defender' in (DisabledParsers) ))

, ASimAuthenticationMicrosoftWindowsEvent (ASimAuthenticationDisabled or ('ExcludeASimAuthenticationMicrosoftWindowsEvent' in (DisabledParsers) ))

, ASimAuthenticationMD4IoT (ASimAuthenticationDisabled or ('ExcludeASimAuthenticationMD4IoT' in (DisabledParsers) ))

, vimSyslogAuthenticationSuccessCustom

, vimSyslogInvalidUserAuthenticationCustom

, vimSyslogValidAuthenticationCustom

Below is the result of events in ASimAuthentication Parser which includes our Syslog events along with the rest of the devices sending authentication logs.

Analytics Rule

Once all important authentication messages are parsed and called in the union ASimAuthentication Parser, we’ll now be able to use that in our Brute force Analytics Rule.

The rule is defined with the condition of authentication failure of 10 times and more.

Because of our parsing technique, building analytics rule becomes easy, As it’s shown below.

ASimAuthentication

| where EventResult has "Failure"

| summarize count() by EventProduct, EventVendor, EventResult, TargetUsername, SrcDvcHostname, SrcDvcIpAddr

| where count_ >= 10

summarization allows us to call important artefacts in the Alert enrichment i.e. TargetUsername, SrcDvcipAddr, and SrcDvcHostname.

The rest of the alert settings can be set according to the liking.

Example Alert

Lastly, we confirmed that our rule is firing as intended.

Deploying in Microsoft Sentinel Environment

The below method is to deploy these parsers quickly in your environment. Provided link is of the GitHub repository.

https://github.com/LaraibKhan555/Parsers

Summary of present Parsers

vimSyslogAuthenticationSuccess -- This parser will filter and parse Authentication Success messages from the Syslog table in ASIM Schema format.

vimSyslogInvalidUserAuthenticationFailure -- This parser will filter and parser Authentication Failure messages from invalid users (random user hits Bruteforce style) from the Syslog data table.

vimSyslogValidAuthenticationFailure -- This parser will filter and parse Authentication failure messages on users that are configured and present on the Linux machine coming in the Syslog Data Table.

This deployment link will deploy all parsers available in the parser repository.

Conclusion

I hope Syslog parsers also become useful for others. And I also hope that this article gave you a glimpse of how you can parse data with KQL.