Tuesday, December 31, 2013

Simple way to integrate Nagios with Slack messaging

At work we recently switched messaging applications from Skype to a new platform called Slack. Slack just launched in August 2013. I have read it is similar to Campfire but I've never used that platform so I can't really comment on that but it is much more useful than a basic chat client like Skype. With Slack you can share files, easily search message history for text or files and integrate with 3rd party applications. Plus it is private for just your team or company. Slack has quite a few preconfigured integrations plus the ability to create your own custom integrations. First we setup the Github integration which allows all of our commit messages to dump into a channel. Next we setup the Trello integration to dump card changes from our main board into another channel. Then I went to setup the Nagios integration and ran into problems. They have a prebuilt integration for Nagios but I could not get it to work. It would post alert messages into the channel but the messages contained no information:


I mucked with their provided perl script quite a bit but I simply could not get it to work. It just kept posting empty messages. Being impatient and a do-it-yourselfer I set about trying to find another way to accomplish this. I looked through the list of integrations and noticed that they had a custom one called Incoming WebHooks which is an easy way to get messages from external sources posted into Slack. The simplest way to utilize Incoming WebHooks is to use curl to post the message to Slack's API. I wrote a little bash script that provides a detailed Nagios alert, a link back to the Nagios web page and conditional emoji's! Each warning level (OK, WARNING, CRITICAL and UNKNOWN) has it's own emoji icon. Here are some example messages in my Slack client:


Here is my bash script that posts to Slack. I placed it in /usr/local/bin

#!/bin/bash
# This script is used by Nagios to post alerts into a Slack channel
# using the Incoming WebHooks integration. Create the channel, botname
# and integration first and then add this notification script in your
# Nagios configuration.
#
# All variables that start with NAGIOS_ are provided by Nagios as
# environment variables when an notification is generated.
# A list of the env variables is available here:
# http://nagios.sourceforge.net/docs/3_0/macrolist.html
#
# More info on Slack
# Website: https://slack.com/
# Twitter: @slackhq, @slackapi
#
# My info
# Website: http://matthewcmcmillan.blogspot.com/
# Twitter: @matthewmcmillan
#Modify these variables for your environment
MY_NAGIOS_HOSTNAME="nagios.yourdomain.com"
SLACK_HOSTNAME="yourslack.slack.com"
SLACK_TOKEN="xyxyxyourslackkey"
SLACK_CHANNEL="#alerts"
SLACK_BOTNAME="nagios"
#Set the message icon based on Nagios service state
if [ "$NAGIOS_SERVICESTATE" = "CRITICAL" ]
then
ICON=":exclamation:"
elif [ "$NAGIOS_SERVICESTATE" = "WARNING" ]
then
ICON=":warning:"
elif [ "$NAGIOS_SERVICESTATE" = "OK" ]
then
ICON=":white_check_mark:"
elif [ "$NAGIOS_SERVICESTATE" = "UNKNOWN" ]
then
ICON=":question:"
else
ICON=":white_medium_square:"
fi
#Send message to Slack
curl -X POST --data "payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_USERNAME}\", \"text\": \"${ICON} HOST: ${NAGIOS_HOSTNAME} SERVICE: ${NAGIOS_SERVICEDISPLAYNAME} MESSAGE: ${NAGIOS_SERVICEOUTPUT} <https://${MY_NAGIOS_HOSTNAME}/cgi-bin/nagios3/status.cgi?host=${NAGIOS_HOSTNAME}|See Nagios>\"}" https://${SLACK_HOSTNAME}/services/hooks/incoming-webhook?token=${SLACK_TOKEN}
view raw slack_nagios.sh hosted with ❤ by GitHub
Here are the Nagios config lines that are added to commands.cfg

###############################
# Slack notifications
###############################
# 'notify-service-by-slack' command definition
define command {
command_name notify-service-by-slack
command_line /usr/local/bin/slack_nagios.sh > /tmp/slack.log 2>&1
}
# 'notify-host-by-slack' command definition
define command {
command_name notify-host-by-slack
command_line /usr/local/bin/slack_nagios.sh > /tmp/slack.log 2>&1
}
view raw commands.cfg hosted with ❤ by GitHub
And finally lines I added to contacts.cfg

define contact {
contact_name slack
alias Slack
service_notification_period 24x7
host_notification_period 24x7
service_notification_options w,u,c,r
host_notification_options d,r
service_notification_commands notify-service-by-slack
host_notification_commands notify-host-by-slack
}
define contactgroup{
contactgroup_name admins
alias Nagios Administrators
members root,slack
}
define contactgroup{
contactgroup_name admins-page
alias Nagios Administrators
members root,slack
}
view raw contacts.cfg hosted with ❤ by GitHub
I'm not sure why Slack's prebuilt Nagios integration didn't work for me but I really like what I came up with. No Perl modules to install and the only outside dependency is curl. It's also pretty easy to modify the info in the alert message by adding or removing NAGIOS_ env variables in the curl statement.