DefinIT Because if IT were easy, everyone would do it…

25Jan/1220

SCOM 2007 R2: Daily Health Check Script

An updated version of this script has been released: http://www.definit.co.uk/2012/05/scom-2007-r2-daily-health-check-script-v2/

MSFT-System-Center-logoI've been working with a Microsft SCOM PFE (Premier Field Engineer) for the last few months and part of the engagement is an environment health check for the SCOM setup. Based on this Microsoft recommend a series of health checks to for the environment that should be carried out every day. This is summarised as the following:

  1. Check the health of all Management Servers and Gateways
  2. Check the RMS is not in maintenance mode
  3. Review Outstanding Alerts
  4. Review Agent's Health Status
  5. Review Backup Status
  6. Review any Management Group Alerts
  7. Review the Pending Management status
  8. Review Database Sizes (Operations, Data warehouse, ACS)
  9. Review Volume of Alerts
  10. Review Alert Latency
  11. Document any changes 

From this, there are certain aspects that can't be automated so easily, or shouldn't be - e.g:

3. Review Outstanding Alerts: There is no point in scripting a capture of this as they need to be dealt with in the Console.

5. Review Backup Status: Backups are generally monitored elsewhere and by other software, you could query SQL to find the last backup of the DBs if required.

8. Review Database Sizes: This one I don't agree with being a daily check - if you have sized and tuned your environment correctly then these can be checked on a weekly or monthly schedule.

11. Document any changes: Clearly this isn't going to work as a scripted healthcheck!

Check the health of all Management Servers and Gateways

$ReportOutput = "<H2>Management Servers not in Healthy States</H2>"
$Count = Get-ManagementServer | where {$_.HealthState -ne "Success"} | Measure-Object
if($Count.Count -gt 0) {
 $ReportOutput += Get-ManagementServer | where {$_.HealthState -ne "Success"} | select Name,HealthState,IsRootManagementServer,IsGateway | ConvertTo-HTML -fragment
} else {
 $ReportOutput += "<p>All management servers are in healthy state.</p>"
}

Check the RMS is not in maintenance mode

$RMS = Get-ManagementServer | where {$_.IsRootManagementServer -eq $True}
$criteria = new-object Microsoft.EnterpriseManagement.Monitoring.MonitoringObjectGenericCriteria("InMaintenanceMode=1")
$objectsInMM = (Get-ManagementGroupConnection).ManagementGroup.GetPartialMonitoringObjects($criteria)
$is = "is not"
foreach ($MM in $objectsInMM){
 if($MM.Displayname -eq $RMS.Name){
   $is = "is"
 }
}
$ReportOutput += "<h2>RMS in Maintenance Mode</h2><p>"+ $RMS.Name +" "+$is+" in maintenance mode</p>"

Review Agent's Health Status

$ReportOutput += "<h2>Agents where Health State is not Green</h2>"
$ReportOutput += Get-Agent | where {$_.HealthState -ne "Success"} | select Name,HealthState | ConvertTo-HTML -fragment$ReportOutput += "<h2>Agents where the Monitoring Class is not available</h2>"
$AgentMonitoringClass = get-monitoringclass -name "Microsoft.SystemCenter.Agent"
$ReportOutput += Get-MonitoringObject -monitoringclass:$AgentMonitoringClass | where {$_.IsAvailable -eq $false} | select DisplayName | ConvertTo-HTML -fragment

Review any Management Group Alerts

$ManagementServers = Get-ManagementServer
foreach ($ManagementServer in $ManagementServers){
 $ReportOutput += "<h3>Alerts on " + $ManagementServer.ComputerName + "</h3>"
 $ReportOutput += get-alert -Criteria ("NetbiosComputerName = '" + $ManagementServer.ComputerName + "'") | where {$_.ResolutionState -ne '255' -and $_.MonitoringObjectFullName -Match 'Microsoft.SystemCenter'} | select TimeRaised,Name,Description,Severity | ConvertTo-HTML -fragment
}

Review the Pending Management status

$ReportOutput += "<h2>Agents in Pending State</h2>"
$ReportOutput += Get-AgentPendingAction | sort AgentPendingActionType | select AgentName,ManagementServerName,AgentPendingActionType | ConvertTo-HTML -fragment

Review Volume of Alerts

$ReportOutput += "<h2>Top 10 Repeating Alerts</h2>"
$ReportOutput += get-alert -Criteria 'ResolutionState < "255"' | Sort -desc RepeatCount | select-object –first 10 Name, RepeatCount, MonitoringObjectPath, Description | ConvertTo-HTML -fragment$ReportOutput += "<h2>Agents in Pending State</h2>"
$ReportOutput += Get-AgentPendingAction | sort AgentPendingActionType | select AgentName,ManagementServerName,AgentPendingActionType | ConvertTo-HTML -fragment

The Final Product

My full Daily Health Check Script, with some formatting, output for console and sending an email:

$Head = "<style>"
$Head +="BODY{background-color:#CCCCCC;font-family:Verdana,sans-serif; font-size: x-small;}"
$Head +="TABLE{border-width: 1px;border-style: solid;border-color: black;border-collapse: collapse; width: 100%;}"
$Head +="TH{border-width: 1px;padding: 0px;border-style: solid;border-color: black;background-color:green;color:white;padding: 5px; font-weight: bold;text-align:left;}"
$Head +="TD{border-width: 1px;padding: 0px;border-style: solid;border-color: black;background-color:#F0F0F0; padding: 2px;}"
$Head +="</style>"write-host "Getting Management Health Server States" -ForegroundColor Yellow
$ReportOutput = "<H2>Management Servers not in Healthy States</H2>"
$Count = Get-ManagementServer | where {$_.HealthState -ne "Success"} | Measure-Object
if($Count.Count -gt 0) {
 $ReportOutput += Get-ManagementServer | where {$_.HealthState -ne "Success"} | select Name,HealthState,IsRootManagementServer,IsGateway | ConvertTo-HTML -fragment
} else {
 $ReportOutput += "<p>All management servers are in healthy state.</p>"
}
write-host "Getting RMS Maintenance Mode" -ForegroundColor Yellow
$RMS = Get-ManagementServer | where {$_.IsRootManagementServer -eq $True}
$criteria = new-object Microsoft.EnterpriseManagement.Monitoring.MonitoringObjectGenericCriteria("InMaintenanceMode=1")
$objectsInMM = (Get-ManagementGroupConnection).ManagementGroup.GetPartialMonitoringObjects($criteria)
$is = "is not"
foreach ($MM in $objectsInMM){
 if($MM.Displayname -eq $RMS.Name){
   $is = "is"
 }
}
$ReportOutput += "<h2>RMS in Maintenance Mode</h2><p>"+ $RMS.Name +" "+$is+" in maintenance mode</p>"write-host "Getting Agent Health Status" -ForegroundColor Yellow
$ReportOutput += "<h2>Agents where Health State is not Green</h2>"
$ReportOutput += Get-Agent | where {$_.HealthState -ne "Success"} | select Name,HealthState | ConvertTo-HTML -fragment$ReportOutput += "<h2>Agents where the Monitoring Class is not available</h2>"
$AgentMonitoringClass = get-monitoringclass -name "Microsoft.SystemCenter.Agent"
$ReportOutput += Get-MonitoringObject -monitoringclass:$AgentMonitoringClass | where {$_.IsAvailable -eq $false} | select DisplayName | ConvertTo-HTML -fragmentwrite-host "Getting Management Server Alerts" -ForegroundColor Yellow
$ReportOutput += "<h2>Management Server Alerts</h2>"
$ManagementServers = Get-ManagementServer
foreach ($ManagementServer in $ManagementServers){
 $ReportOutput += "<h3>Alerts on " + $ManagementServer.ComputerName + "</h3>"
 $ReportOutput += get-alert -Criteria ("NetbiosComputerName = '" + $ManagementServer.ComputerName + "'") | where {$_.ResolutionState -ne '255' -and $_.MonitoringObjectFullName -Match 'Microsoft.SystemCenter'} | select TimeRaised,Name,Description,Severity | ConvertTo-HTML -fragment
}write-host "Getting Top 10 Unresolved Alerts" -ForegroundColor Yellow
$ReportOutput += "<h2>Top 10 Unresolved Alerts</h2>"
$ReportOutput += get-alert -Criteria 'ResolutionState < "255"'  | Group-Object Name | Sort-object Count -desc | select-Object -first 10 Count, Name | ConvertTo-HTML -fragmentwrite-host "Getting Top 10 Repeating Alerts" -ForegroundColor Yellow
$ReportOutput += "<h2>Top 10 Repeating Alerts</h2>"
$ReportOutput += get-alert -Criteria 'ResolutionState < "255"' | Sort -desc RepeatCount | select-object –first 10 Name, RepeatCount, MonitoringObjectPath, Description | ConvertTo-HTML -fragmentwrite-host "Getting Agents in Pending State" -ForegroundColor Yellow
$ReportOutput += "<h2>Agents in Pending State</h2>"
$ReportOutput += Get-AgentPendingAction | sort AgentPendingActionType | select AgentName,ManagementServerName,AgentPendingActionType | ConvertTo-HTML -fragmentwrite-host "Getting Overrides in Default Management Pack" -ForegroundColor Yellow
$ReportOutput += "<h2>Overrides in Default Management Pack</h2>"$OverrideCount = Get-ManagementPack | where {$_.DisplayName -match "Default Management Pack"} | get-override | measure-objectif($OverrideCount.Count -gt 2){
 foreach ($monitor in Get-ManagementPack | where {$_.DisplayName -match "Default Management Pack"} | get-override | where {$_.monitor}) {
  $ReportOutput += get-monitor | where {$_.Id -eq $monitor.monitor.id} | select-object DisplayName,Description | ConvertTo-HTML -fragment
  $ReportOutput += "<br />"
 }
 foreach ($rule in Get-ManagementPack | where {$_.DisplayName -match "Default Management Pack"} | get-override | where {$_.rule}) {
  $ReportOutput += get-rule | where {$_.Id -eq $rule.rule.id} | select-object DisplayName,Description | ConvertTo-HTML -fragment
  $ReportOutput += "<br />"
 }
} else {
 $ReportOutput += "<p>There are no unexpected overrides in the Default Management Pack</p>"
}$Body = ConvertTo-HTML -head $Head -body "$ReportOutput"$SmtpClient = New-Object system.net.mail.smtpClient
$MailMessage = New-Object system.net.mail.mailmessage
$SmtpClient.Host = "smtp.definit.co.uk"
$mailmessage.from = "[email protected]"
$mailmessage.To.add("<a href="mailto:[email protected]">[email protected]</a>")
#$mailmessage.To.add("<a href="mailto:[email protected]">[email protected]</a>")
$mailmessage.Subject = "SCOM Daily Healthcheck Report"
$MailMessage.IsBodyHtml = 1
$mailmessage.Body = $Body$smtpclient.Send($mailmessage)
 

You can download the file here (zip): Get-HealthCheck

Creative Commons License
SCOM 2007 R2: Daily Health Check Script by DefinIT, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.

Posted by Sam McGeown

Comments (20) Trackbacks (0)
  1. Hi

    Please can you tell me how to run this script

    getting errors if I only run the .ps1 file

  2. Getting loads of errors when trying to execute powershell script first part is:-
    Getting Management Health Server States
    Get-ManagementServer : The ‘Path’ parameter is empty or the required provider locat
    ion is not set.”
    At C:\gethealthcheck.ps1:10 char:30
    + $Count = Get-ManagementServer <<<< | where {$_.HealthState -ne "Success"} | Meas
    ure-Object
    + CategoryInfo : InvalidArgument: (:String) [Get-ManagementServer], A
    rgumentOutOfRangeException
    + FullyQualifiedErrorId : InvalidParameter,Microsoft.EnterpriseManagement.Oper
    ationsManager.ClientShell.GetManagementServerCmdlet

    • Hi – it sounds like you have not loaded the SCOM management shell correctly before running the script – if you are not running directly from the “Operations Manager Shell” you need to add something similar to the following at the beginning of the script:

      Set-Location "C:\Program Files\System Center Operations Manager 2007\"
      Load-PSSnapin -Snapin "Microsoft.EnterpriseManagement.OperationsManager.Client"
      .\Microsoft.EnterpriseManagement.OperationsManager.ClientShell.Startup.ps1

      The gets you into the correct context to run the healthcheck script.

      Thanks,

      Sam

      • I was running it from the Operations Manager Shell on my RMS, but i’ll try a normal windows powershell window and add the SCOM snap-in manually to see what happens

  3. Still getting the same error, I am running the script on the RMS could this be the issue?

  4. Might be worth mentioning I have 2 management groups in my SCOM 2007 R2 setup so I will have 2 RMS servers, 1 for each management group, perhaps this may be contributing to the problem? Unfortunately I have no scripting experience.

    • Ok, the multiple management groups might be the problem – there should be no problem running it from the RMS. When you load the Operations Manager Shell on the RMS normally, what is the result of running “get-managementserver” by itself? I don’t have an environment with multiple Management Groups to test.

  5. Ok the result of running the get-managementserver cmdlet is that it returns details of the RMS and MS in the management group.

    • If get-managementserver works like that, there should be no issue with it in the script. Are you sure there are no extra characters or anything in the script? Feel free to drop me an email (first name at last name dot co dot uk) with the script saved as a .txt and I’ll take a look.

    • Hi,

      I can simulate the same error doing the following:
      Error

      Try adding “Set-Location Monitoring:” at the beginning of the script, after the OpsMgr shell stuff I posted before.

      Sam

  6. Sam,

    Brilliant! that one line of code “Set-Location Monitoring:” put at the beginning of the powershell script has resolved the problem I reported earlier. Just need to get my smtp details sorted out so the output can be viewed in an e-mail.

    Appreciate your help with this.

    Incidently is it easy to amend the script so that the output is exported to csv or txt file(s) as opposed to an e-mail?

    • Phew – got there in the end! Glad it’s working for you! The script pulls lots of different data sources, so it doesn’t lend itself to a CSV export, but it could be modified fairly easily to dump out a text file.

  7. Now working perfectly including html mail output, thanks again for your help!

  8. Hi

    I tried all, but always gettings errors like “Unexpected token ‘write-host’ in expression or statement.” for teh variables, can you please help me to correct this

    LMS

  9. Hi

    I am trying to run your script and it seems to be getting so far then I get an error. Please see below any ideas much appreciated.

    Many thanks

    Reuben

    >c:\support\get-healthcheck.ps1
    Getting Management Health Server States
    Getting RMS Maintenance Mode
    Getting Agent Health Status
    Getting Management Server Alerts
    Getting Top 10 Unresolved Alerts
    Getting Top 10 Repeating Alerts
    Getting Agents in Pending State
    Getting Overrides in Default Management Pack
    ConvertTo-Html : A positional parameter cannot be found that accepts argument ‘New-
    Object’.
    At C:\support\get-healthcheck.ps1:74 char:23
    + $Body = ConvertTo-HTML <<<< -head $Head -body "$ReportOutput"$SmtpClient = New-O
    bject system.net.mail.smtpClient
    + CategoryInfo : InvalidArgument: (:) [ConvertTo-Html], ParameterBind
    ingException
    + FullyQualifiedErrorId : PositionalParameterNotFound,Microsoft.PowerShell.Com
    mands.ConvertToHtmlCommand

    Property 'Host' cannot be found on this object; make sure it exists and is settable
    .
    At C:\support\get-healthcheck.ps1:76 char:13
    + $SmtpClient. <<<

  10. Great thanks very much the SCOM part of the script seems to run fine now, but I dont get the email of the report. I have tested my email connectivity using the following and it works fine.

    thanks
    Reuben

    #$smtpServer = “mail.tlcmarketing.com”
    #$msg = new-object Net.Mail.MailMessage
    #$att = new-object Net.Mail.Attachment($file)
    #$smtp = new-object Net.Mail.SmtpClient($smtpServer)
    #$msg.From = “[email protected]
    #$msg.To.Add(“[email protected]”)
    #$msg.Subject = “SCOM Report test”
    #$msg.Body = “SCOM Report test$dt!”
    #$msg.Attachments.Add($att)
    #$smtp.Send($msg)
    #$att.Dispose()


Leave a comment

No trackbacks yet.