How can I auto-close a variety of OpsMgr alerts on a scheduled basis? (#SCOM, #powershell)

In Operations Manager alerts can be generated by either a monitor or a rule (for background information on this see “The Rule of the Monitor”). Alerts which are generated by rules do not impact health state, and they will not auto-close themselves. This results in a situations where there may be alerts in your environment which may need to be manually closed on a daily or hourly basis. I worked with an organization which was using a large number of custom rules which they wanted the email notification to occur, but once that had occurred the alert could be closed. These are examples of situations where it is useful to have specific rule-generated alerts auto-close on a scheduled basis. This is all about decreasing the clutter in the OpsMgr console. By auto-closing these alerts we can focus on more actionable alerts which are occurring in the environment.

A while back I wrote a simple management pack called the Scheduled Result Management pack (or SRU). This management pack is good for auto-closing a small number of rule-generated alerts using overrides generated in the console. It does not work well however with large number of alerts to auto-close and it does not work well when the list will be constantly changing. In this type of a situation we can easily configure a scheduled task which will auto-close alerts which we specify based upon the content of a file on the system.

This blog article will discuss the steps required to create the folder and file structure, steps to create this task, how to debug the execution of the task, how to auto-close additional alerts, and will provide the sample file and script to perform the auto-close.

Creating the folder and file structure:

On the Root Management Server (RMS) we are creating a folder structure of c:\SCOMAutomation\CloseAlertByTextFile. The two files are CloseAlertByTextFile.ps1 (script later in this article) and TargetRules.txt (sample content later in this article).

Task Creation Steps:

Once the folder structure and files are available we’re going to create this in the task scheduler (this could also be created with a rule which is scheduled to run in OpsMgr but for simplicity of administration we are keeping this to the task scheduler). In Windows 2008 and higher this is available in:

Start, programs, administrative Tools, Task Scheduler

In the task scheduler use the Create Task option highlighted below:

For the task we need to configure the general tab with the name (SCOM – Close Alert by Text file) and to run whether user is logged on or not. Also – avoid using the Hidden option as it’s harder to find the task to manually run the task after it is created.

On the Actions pane we need to create a new action:

Action = Start a program
Program/Script = powershell.exe
Add Arguments = -command “& ‘c:\ScomAutomation\CloseAlertByTextFile\CloseAlertByTextFile.ps1’ “
Start in = c:\ScomAutomation\CloseAlertByTextFile

On the schedule task we are creating this to run daily every hour, depending upon the requirements for your environment this might only occur once a day or once a week.

Once this is created we can manually run the task to test it’s functionality.

Debugging the Task execution:

The PowerShell script includes logging for events which are created in the Application Log. These include:

EventID 10002 – created with each rule and includes count of how many closed

Event 10010 – summarization of what was done and home many types of alerts were closed.

Event 10009 – Created if nothing was done

Event 10000 – Created as an error if the RMS server can’t be contacted.

Adding to the alerts to auto-close:

The default alert listed in this article will not exist in your environment. For your environment I recommend that you find alerts generated by rules which you want to auto-close, and you can cut and paste from the OpsMgr console into the TargetRules.txt file as shown below (highlight the name of the alert shown in the Alert Details pane, control-C to copy).

TargetRules.txt

(There should not be any blank lines at the end of this file when it is created)

A SQL job failed to complete successfully.

CloseAlertByTextFile.ps1

(You will need to change the PS1 script to refer to “localhost” – assuming it’s going to run on the RMS or to the name of the RMS in your environment)

<#
.SYNOPSIS
    Reads a text file and closes all OPS Manager Alerts that meet the criteria
.DESCRIPTION
    Looks for a file called TargetRules.txt in the same folder as this script was ran from.
    Logs a name and count for each rule that is closed in the EventLog under “Source” ScomAutomation.
    The following events are logged if needed
    10000 Exited with an error. The parameter wasn’t passed to the function correctly
    10001 Lists how many instances of each Scom Event are closed
    10002 Shows when the script exits each time it has ran.
    10009 Shows if nothing is done.

.NOTES
    File Name : CloseAlertByTextFile.ps1
    Author     : Larry Brown – [email protected]
    Requires   : PowerShell V2; Ops Manager PowerShell Dll Registered

#> 

Function Out-ToEventLog
{
Param ($messageContents, $typeOfAlert, $NumberForOutput)
    $EventLog.WriteEntry($PrependInfo + $messageContents, $typeOfAlert, $NumberForOutput)
   
}

Function Clear-Alert
{
Param([string]$Alertname)
    $intInstanceCounter = 0
    If ($Alertname.Length -le 0){
            Out-ToEventLog "Alertname empty.. Nothing to close. Check the Alert text file for extra lines." "Warning" "10001"
            }
    Else{  
        $criteria = [string]::Format("Name = '{0}' AND IsMonitorAlert = 0 AND ResolutionState < 255",$Alertname);
        Get-Alert -Criteria $criteria | ForEach-Object {
            [String]::Format("Closing Alert '{0}'",$_.Name);
            $_ | Resolve-Alert -Comment "Closed by Automation Script" | out-null;   
            $intInstanceCounter = $intInstanceCounter + 1}
            }
    If($intInstanceCounter -gt 0){
        $global:intTypeCounter = $global:intTypeCounter + 1
        Out-ToEventLog "Closed $intInstanceCounter instance(s) of $Alertname" "Information" "10002"
       }
}

$INST = Split-Path -Parent $MyInvocation.MyCommand.Path

$global:intTypeCounter = 0

$rms = "RmsServerName"

$PrependInfo = "CloseAlertByText: "

Add-PSSnapin "Microsoft.EnterpriseManagement.OperationsManager.Client";

$EventLog = Get-EventLog -list | Where-Object {$_.Log -eq “Application”}
$EventLog.Source="ScomAutomation"

$mgConn = New-ManagementGroupConnection -connectionString:$rms;
if ( $mgConn -eq $null ){
    Out-ToEventLog "Not able to connect to the RMS: $rms. Exiting with error." "Error" "10000"
    return;
}
Set-Location "OperationsManagerMonitoring::";
Set-Location $rms;

foreach ($a in Get-Content "$INST\TargetRules.txt") {Clear-Alert $a}

if($global:intTypeCounter -gt 0){
    Out-ToEventLog "closed $global:intTypeCounter type(s) of Alert(s)" "Information" "10010"
    }
else{
    Out-ToEventLog "Exiting, No Alerts Closed." "Information" "10009"
}

Notes & Kudos!

This script and the process was created by Larry Brown – well done man! You can follow Larry on twitter at: @lbrownfromtx
This is designed to ONLY work for alerts generated by rules, not monitors. We do not want to auto-close monitor generated alerts (see the Rule of the Monitor at the start of this article for details).

In Operations Manager alerts can be generated by either a monitor or a rule (for background information on this see ” Scheduled Result Management pack (or SRU). This management pack is good for auto-closing a small number of rule-generated alerts using overrides generated in the console. It does not work well however with large number of alerts to auto-close and it does not work well when the list will be constantly changing. In this type of a situation we can easily configure a scheduled task which will auto-close alerts which we specify based upon the content of a file on the system. This blog article will discuss the steps required to create the folder and file structure, steps to create this task, how to debug the execution of the task, how to auto-close additional alerts, and will provide the sample file and script to perform the auto-close. Creating the folder and file structure: On the Root Management Server (RMS) we are creating a folder structure of c:SCOMAutomationCloseAlertByTextFile. The two files are CloseAlertByTextFile.ps1 (script later in this article) and TargetRules.txt (sample content later in this article). Task Creation Steps: Once the folder structure and files are available we’re going to create this in the task scheduler (this could also be created with a rule which is scheduled to run in OpsMgr but for simplicity of administration we are keeping this to the task scheduler). In Windows 2008 and higher this is available in: Start, programs, administrative Tools, Task Scheduler In the task scheduler use the Create Task option highlighted below: For the task we need to configure the general tab with the name (SCOM – Close Alert by Text file) and to run whether user is logged on or not. Also – avoid using the Hidden option as it’s harder to find the task to manually run the task after it is created. On the Actions pane we need to create a new action: Action = Start a program Program/Script = powershell.exe Add Arguments = -command “& ‘c:ScomAutomationCloseAlertByTextFileCloseAlertByTextFile.ps1’ ” Start in = c:ScomAutomationCloseAlertByTextFile On the schedule task we are creating this to run daily every hour, depending upon the requirements for your environment this might only occur once a day or once a week. Once this is created we can manually run the task to test it’s functionality. Debugging the Task execution: The PowerShell script includes logging for events which are created in the Application Log. These include: EventID 10002 – created with each rule and includes count of how many closed Event 10010 – summarization of what was done and home many types of alerts were closed. Event 10009 – Created if nothing was done Event 10000 – Created as an error if the RMS server can’t be contacted. Adding to the alerts to auto-close: The default alert listed in this article will not exist in your environment. For your environment I recommend that you find alerts generated by rules which you want to auto-close, and you can cut and paste from the OpsMgr console into the TargetRules.txt file as shown below (highlight the name of the alert shown in the Alert Details pane, control-C to copy). TargetRules.txt (There should not be any blank lines at the end of this file when it is created) A SQL job failed to complete successfully. CloseAlertByTextFile.ps1 (You will need to change the PS1 script to refer to “localhost” – assuming it’s going to run on the RMS or to the name of the RMS in your environment) <# .SYNOPSIS Reads a text file and closes all OPS Manager Alerts that meet the criteria .DESCRIPTION Looks for a file called TargetRules.txt in the same folder as this script was ran from. Logs a name and count for each rule that is closed in the EventLog under “Source” ScomAutomation. The following events are logged if needed 10000 Exited with an error. The parameter wasn’t passed to the function correctly 10001 Lists how many instances of each Scom Event are closed 10002 Shows when the script exits each time it has ran. 10009 Shows if nothing is done. .NOTES File Name : CloseAlertByTextFile.ps1 Author : Larry Brown – [email protected] Requires : PowerShell V2; Ops Manager PowerShell Dll Registered #> Function Out-ToEventLog { Param ($messageContents, $typeOfAlert, $NumberForOutput) $EventLog.WriteEntry($PrependInfo + $messageContents, $typeOfAlert, $NumberForOutput) } Function Clear-Alert { Param([string]$Alertname) $intInstanceCounter = 0 If ($Alertname.Length -le 0){ Out-ToEventLog “Alertname empty.. Nothing to close. Check the Alert text file for extra lines.” “Warning” “10001” } Else{ $criteria = [string]::Format(“Name = ‘{0}’ AND IsMonitorAlert = 0 AND ResolutionState < 255”,$Alertname); Get-Alert -Criteria $criteria | ForEach-Object { [String]::Format(“Closing Alert ‘{0}'”,$_.Name); $_ | Resolve-Alert -Comment “Closed by Automation Script” | out-null; $intInstanceCounter = $intInstanceCounter + 1} } If($intInstanceCounter -gt 0){ $global:intTypeCounter = $global:intTypeCounter + 1 Out-ToEventLog “Closed $intInstanceCounter instance(s) of $Alertname” “Information” “10002” } } $INST = Split-Path -Parent $MyInvocation.MyCommand.Path $global:intTypeCounter = 0 $rms = “RmsServerName” $PrependInfo = “CloseAlertByText: ” Add-PSSnapin “Microsoft.EnterpriseManagement.OperationsManager.Client”; $EventLog = Get-EventLog -list | Where-Object {$_.Log -eq “Application”} $EventLog.Source=”ScomAutomation” $mgConn = New-ManagementGroupConnection -connectionString:$rms; if ( $mgConn -eq $null ){ Out-ToEventLog “Not able to connect to the RMS: $rms. Exiting with error.” “Error” “10000” return; } Set-Location “OperationsManagerMonitoring::”; Set-Location $rms; foreach ($a in Get-Content “$INSTTargetRules.txt”) {Clear-Alert $a} if($global:intTypeCounter -gt 0){ Out-ToEventLog “closed $global:intTypeCounter type(s) of Alert(s)” “Information” “10010” } else{ Out-ToEventLog “Exiting, No Alerts Closed.” “Information” “10009” } Notes & Kudos! This script and the process was created by Larry Brown – well done man! You can follow Larry on twitter at: @ lbrownfromtx This is designed to ONLY work for alerts generated by rules, not monitors. We do not want to auto-close monitor generated alerts (see the Rule of the Monitor at the start of this article for details).