Techniques to make Azure Monitor alerts from Log Analytics more useful: Creating custom alerts with Azure Automation

In the first blog post of this series we showed how you can make your alerts more useful by cleaning up the underlying query (which provides a quick way to include only what is relevant in the alert). In this blog post we will show how to use the power of Azure Automation to customize your Azure Monitor alerts in any way you want to.

This blog post will discuss the approach to put together this solution through these major steps:

The steps involved in this process to reformat the alert
How to create the Azure Automation runbook which reformats the alerts
Creating a call to the Webhook in an Action Group
Adding the Action Group to an alert
Creating the Azure Automation runbook which emails the alerts (including credentials)
Documentation for additional reference
Results of this automation

The end of the blog also has copies of the two sample scripts in the “SAMPLE SCRIPTS” section and are also available for download here.

Steps to reformat the alert:

The process to reformat the alert goes through the following steps:

The Log Analytics condition is detected via an alert which uses an Action Group
The Action Group contains a webhook which calls the Azure Automation runbook
The first runbook reformats the content sent to the runbook via the Webhook
The first runbook calls a second runbook which sends the email to the appropriate email addresses using the credentials passed to the runbook

NOTE: The two runbooks in this example could have been combined but I chose to keep them separate to facilitate re-use of the second runbook which can be used to send an email for basically any condition. The runbooks are available both in this blog post and are also available here for download.

Creating the Azure Automation runbook which reformats the alerts

The first runbook is a PowerShell script which reformats the alerts based on the content sent to the script. The content of this runbook is at the end of this blog post in the “PowerShell script CustomAlert” section. We start by creating the Azure Automation runbook and then save and publish it (if you don’t already have an Azure Automation account, you will need to create that first).

When creating the runbook specify the appropriate name and for this example we are using “PowerShell”.

Once we have created the runbook, we add a Webhook for the runbook as shown below. This is done in the Webhooks section of the runbook as shown below.

And we give the Webhook name and expiration date.

NOTE: This runbook calls the second runbook

Creating an Action Group which uses the Webhook

The next step is to create an action group which will contain the Webhook (or add the call to the Webhook to an existing Action Group). To add an action to call the Webhook either create a new Action Group (in Azure Monitor under Settings / Action Groups) or open the existing Action Group you have chosen.

And then create a new action for that Action Group by specifying the Action Name and the Action Type of Webhook. When you are adding the details you will need to provide the Webhook URL which occurred when you defined the Webhook for the runbook.

Adding the Action Group to an alert

Now that we have an Action Group which includes the call to the Webhook, I recommend creating an alert for testing and tying the alert to the Action Group. This is done in Azure Monitor under Alerts, Managing Rules, and then opening the alert. For testing I recommend sending an email with the original alert format as well as calling the Webhook to reformat the alert (this helps greatly with debugging steps). An example of this showing two Action Groups (one to send the email, another to reformat and send the email) is shown below. This could also be done by creating a single Action Group which contains both an action to send the email and one to send the Webhook. After this solution is in place, the original email notification can be removed.

Creating the Azure Automation runbook which emails the alerts

The next step is to create the PowerShell workflow runbook which sends out the email. The content of this runbook is at the end of this blog post in the “PowerShell workflow script CustomAlert” section. We start by creating the Azure Automation runbook and then save and publish it.

When creating the runbook specify the appropriate name and for this example we are using “PowerShell Workflow”.

The second runbook uses credentials which actually send the email. These credentials will need to be defined in the Azure Automation account under Credentials.

We define a name and a user account and password which will be able to send email within this Azure subscription.

Documentation for additional reference

This blog post would not have been possible without a lot of other folks who had already put together pieces of this. These include:

How to start an Azure Automation runbook via a Webhook: https://docs.microsoft.com/en-us/azure/automation/automation-webhooks
Using Azure Automation to take on alerts: https://azure.microsoft.com/en-us/blog/using-azure-automation-to-take-actions-on-azure-alerts/
Handling a JSON payload from upgraded Log Analytics workspaces: http://soren.cloud/handling-json-payload-from-upgraded-log-analytics-workspaces-new-issues/
Find the script used to email from Azure Automation: https://gallery.technet.microsoft.com/office/Sending-email-from-Azure-24ee27cd
The original script to send email from Azure Automation using Office 365 (Secure SMTP): https://s1events.azure-automation.net/webhooks?token=vZ%2bb1YUcpvxcAYzF9sZNCxK1YM5YTFt7CrWoAYhg39A%3d

Results of this automation

Original email:

Email after the blog post on cleaning up the query:

And here is our final email:

Summary: Using Azure Automation you can reformat your Log Analytics alerts into any structure that you would like. Once the scripts are in place this is pretty easy to maintain and provides a much more targeted alert. The runbooks are shown below in the “SAMPLE SCRIPTS” section and are also available for download here. As a third part in this blog series, Stefan Roth wrote a blog on how to perform this same type of formatting with Logic Apps which is available here!

SAMPLE SCRIPTS

PowerShell script CustomAlert

NOTE: In this script you will need to define <EMAILADDRESS>, <AZUREAUTOMATIONNAME>, <RESOURCEGROUP>, and the subject of the email.

# <#

#.SYNOPSIS

#Take an alert passed to this script via a webhook and convert it from JSON to a formatted email structure

#>

param (

[Parameter(Mandatory=$false)]

[object] $WebhookData

)

# If runbook was called from Webhook, WebhookData will not be null.

if ($WebhookData) {

write-output "Webhook data $WebhookData"

# Logic to allow for testing in Test Pane

if(-Not $WebhookData.RequestBody)

{

$WebhookData = (ConvertFrom-Json -InputObject $WebhookData)

write-output "test Webhook data $webhookData"

}

$Spacer = ": "

$RequestBody = ConvertFrom-JSON -InputObject $WebhookData.RequestBody

# Get all metadata properties

$AlertRuleName = $RequestBody.AlertRuleName

$AlertThresholdOperator = $RequestBody.AlertThresholdOperator

$AlertThresholdValue = $RequestBody.AlertThresholdValue

$AlertDescription = $RequestBody.Description

$LinktoSearchResults =$RequestBody.LinkToSearchResults

$ResultCount =$RequestBody.ResultCount

$Severity = $RequestBody.Severity

$SearchQuery = $RequestBody.SearchQuery

$WorkspaceID = $RequestBody.WorkspaceId

$SearchWindowStartTime = $RequestBody.SearchIntervalStartTimeUtc

$SearchWindowEndTime = $RequestBody.SearchIntervalEndtimeUtc

$SearchWindowInterval = $RequestBody.SearchIntervalInSeconds

$message = ""

$linespacer = "n"

# Get detailed search results

if($RequestBody.SearchResult -ne $null)

{

$SearchResultRows = $RequestBody.SearchResult.tables[0].rows

$SearchResultColumns = $RequestBody.SearchResult.tables[0].columns;

foreach ($SearchResultRow in $SearchResultRows)

{

$Column = 0

$Record = New-Object –TypeName PSObject

$message = ""

foreach ($SearchResultColumn in $SearchResultColumns)

{

$Name = $SearchResultColumn.name

$ColumnValue = $SearchResultRow[$Column]

$Record | Add-Member –MemberType NoteProperty –Name $name –Value $ColumnValue -Force

if ($Name -eq "Computer")

{

$Computer = $ColumnValue

}

$message = $message + "$Name $Spacer $ColumnValue"

$message = $message + $linespacer

write-output $message

$Column++

}

# Connect to Azure with RunAs account

$ServicePrincipalConnection = Get-AutomationConnection -Name 'AzureRunAsConnection'

Add-AzureRmAccount -ServicePrincipal -TenantId $ServicePrincipalConnection.TenantId -ApplicationId $ServicePrincipalConnection.ApplicationId -CertificateThumbprint $ServicePrincipalConnection.CertificateThumbprint

$AzureContext = Select-AzureRmSubscription -SubscriptionId $ServicePrincipalConnection.SubscriptionID

write-output "Message is $message"

$params = @{"To"="<EMAILADDRESS>";"Subject"="High processor usage on $Computer";"Body"=$message}

Start-AzureRmAutomationRunbook –AutomationAccountName '<AZUREAUTOMATIONNAME>' –Name 'SendO365Email-PSWorkflow' -ResourceGroupName '<RESOURCEGROUP>' –Parameters $params

}

PowerShell workflow script SendO365Email-PSWorkflow

NOTE: In this script you should define <DEFAULTCRED> and <DEFAULTFROMADDRESS>.

<#

.SYNOPSIS

Sending email from Azure Automation using Office 365 (Secure SMTP)

.DESCRIPTION

This Runbook sample demonstrates how Azure Automation can be used to send emails using a secure SMTP

service like Office 365.

This Runbook use global assets defined in the Azure Automation Account.

For more information about how to create assets please check our this article:

http://azure.microsoft.com/blog/2014/07/29/getting-started-with-azure-automation-automation-assets-2/

WARNING: For this runbook to work, you must have created the

following Assets: Powershell Credentials for the email account.

.PARAMETER To (required)

String name to send the email to.

.PARAMETER Body (optional, defaulted)

String that contains the body of the email that are send to the users

.PARAMETER Subject (optional, defaulted)

String that contains the body of the email that are send to the users

.PARAMETER AzureOrgIdCredential (optional, defaulted)

String name of the PSCredential stored in global assets for authentication against the

Secure SMTP Service

.PARAMETER From (optional, defaulted)

String name to send the email from.

.PARAMETER CC (optional, defaults to the same as To)

String name to send the email from via CC.

.PARAMETER BCC (optional, defaults to the same as To)

String name to send the email from via BCC.

.NOTES

Author: Peter Selch Dahl from ProActive A/S

Last Updated: 12/21/2014

Version 1.1

Author: Cameron Fuller from Catapult Systems

Last Updated: 8/15/2018

Version 1.2

#>

workflow SendO365Email-PSWorkflow

{

##############################################################################

# This section contains the input variables that are needed for the runbook

# to work as expected. Input parameters can be replaced with hardcoded variables

# within the script if needed. See inspiration below.

##############################################################################

Param

(

[parameter(Mandatory=$true)]

[String]

$To,

[parameter(Mandatory=$true)]

[String]

$Subject = "Email subject",