I mucked with their provided perl script quite a bit but I simply could not get it to work. It just kept posting empty messages. Being impatient and a do-it-yourselfer I set about trying to find another way to accomplish this. I looked through the list of integrations and noticed that they had a custom one called Incoming WebHooks which is an easy way to get messages from external sources posted into Slack. The simplest way to utilize Incoming WebHooks is to use curl to post the message to Slack's API. I wrote a little bash script that provides a detailed Nagios alert, a link back to the Nagios web page and conditional emoji's! Each warning level (OK, WARNING, CRITICAL and UNKNOWN) has it's own emoji icon. Here are some example messages in my Slack client:
Here is my bash script that posts to Slack. I placed it in /usr/local/bin
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
# This script is used by Nagios to post alerts into a Slack channel | |
# using the Incoming WebHooks integration. Create the channel, botname | |
# and integration first and then add this notification script in your | |
# Nagios configuration. | |
# | |
# All variables that start with NAGIOS_ are provided by Nagios as | |
# environment variables when an notification is generated. | |
# A list of the env variables is available here: | |
# http://nagios.sourceforge.net/docs/3_0/macrolist.html | |
# | |
# More info on Slack | |
# Website: https://slack.com/ | |
# Twitter: @slackhq, @slackapi | |
# | |
# My info | |
# Website: http://matthewcmcmillan.blogspot.com/ | |
# Twitter: @matthewmcmillan | |
#Modify these variables for your environment | |
MY_NAGIOS_HOSTNAME="nagios.yourdomain.com" | |
SLACK_HOSTNAME="yourslack.slack.com" | |
SLACK_TOKEN="xyxyxyourslackkey" | |
SLACK_CHANNEL="#alerts" | |
SLACK_BOTNAME="nagios" | |
#Set the message icon based on Nagios service state | |
if [ "$NAGIOS_SERVICESTATE" = "CRITICAL" ] | |
then | |
ICON=":exclamation:" | |
elif [ "$NAGIOS_SERVICESTATE" = "WARNING" ] | |
then | |
ICON=":warning:" | |
elif [ "$NAGIOS_SERVICESTATE" = "OK" ] | |
then | |
ICON=":white_check_mark:" | |
elif [ "$NAGIOS_SERVICESTATE" = "UNKNOWN" ] | |
then | |
ICON=":question:" | |
else | |
ICON=":white_medium_square:" | |
fi | |
#Send message to Slack | |
curl -X POST --data "payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_USERNAME}\", \"text\": \"${ICON} HOST: ${NAGIOS_HOSTNAME} SERVICE: ${NAGIOS_SERVICEDISPLAYNAME} MESSAGE: ${NAGIOS_SERVICEOUTPUT} <https://${MY_NAGIOS_HOSTNAME}/cgi-bin/nagios3/status.cgi?host=${NAGIOS_HOSTNAME}|See Nagios>\"}" https://${SLACK_HOSTNAME}/services/hooks/incoming-webhook?token=${SLACK_TOKEN} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
############################### | |
# Slack notifications | |
############################### | |
# 'notify-service-by-slack' command definition | |
define command { | |
command_name notify-service-by-slack | |
command_line /usr/local/bin/slack_nagios.sh > /tmp/slack.log 2>&1 | |
} | |
# 'notify-host-by-slack' command definition | |
define command { | |
command_name notify-host-by-slack | |
command_line /usr/local/bin/slack_nagios.sh > /tmp/slack.log 2>&1 | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
define contact { | |
contact_name slack | |
alias Slack | |
service_notification_period 24x7 | |
host_notification_period 24x7 | |
service_notification_options w,u,c,r | |
host_notification_options d,r | |
service_notification_commands notify-service-by-slack | |
host_notification_commands notify-host-by-slack | |
} | |
define contactgroup{ | |
contactgroup_name admins | |
alias Nagios Administrators | |
members root,slack | |
} | |
define contactgroup{ | |
contactgroup_name admins-page | |
alias Nagios Administrators | |
members root,slack | |
} |
Hi, great post. I found that the the variable SLACK_BOTNAME is not sent in the post to Slack but the undefined variable SLACK_USERNAME is.
ReplyDelete[root@splunk bin]# /usr/local/bin/slack_nagios.sh
ReplyDelete/usr/local/bin/slack_nagios.sh: line 47: unexpected EOF while looking for matching `}'
/usr/local/bin/slack_nagios.sh: line 48: syntax error: unexpected end of file
[root@splunk bin]#
I double checked my curly braces and they are correct. Maybe you lost one when you copied and pasted the code. At the bottom of the Gist above there is link that says 'view raw'. Or you can go here to download it: https://gist.github.com/matt448/8200821
DeleteIt can be avoided by doing
Deletedos2unix slack_nagios.sh
I've met the similar issue and contacted the Slack team for the support.
ReplyDeleteThey advised me to use configuration recommended for "older versions of Nagios" (like `command_line /usr/local/bin/slack_nagios.pl -field slack_channel=#alerts -field HOSTALIAS="$HOSTNAME$" -field SERVICEDESC="$SERVICEDESC$" -field SERVICESTATE="$SERVICESTATE$" -field SERVICEOUTPUT="$SERVICEOUTPUT$" -field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"` ) and it worked just fine.
Thanks for the update. Last year there wasn't much documentation for the Nagios integration. I'm sure they have improved the perl script and documentation since then.
DeleteI have tried all of the above and still no luck. The closest I've come is "Host '' is"
ReplyDeleteMatt, thanks for sharing!
ReplyDeleteAbove is a mix up of Perl (slack_nagios.pl) and Shell (slack_nagios.sh). I am using the shell script. Mine looks like (one line):
curl -X POST --data-urlencode "payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_BOTNAME}\", \"text\": \"${ICON} +++ HOST: ${1} +++ SERVICE: ${2} +++ MESSAGE: ${4} +++ \", \"icon_emoji\": \":ghost:\"}" https://${SLACK_HOSTNAME}/services/hooks/incoming-webhook?token=${SLACK_TOKEN}
My Nagios commancs.cfg looks like:
define command {
command_name notify-service-by-slack
command_line /usr/local/bin/slack_nagios.sh \
"$HOSTNAME$" \
"$SERVICEDESC$" \
"$SERVICESTATE$" \
"$SERVICEOUTPUT$" \
"$NOTIFICATIONTYPE$"
}
Thank you for this post! BASH didnt have many examples. I added this script to track myself: https://gist.github.com/tony-caffe/9d84265b36d92d89aa5ec04b6a0c81a8
DeleteI used the original slack.pl with this command config which works for me:
ReplyDeletedefine command {
command_name notify-service-by-slack
command_line /usr/lib/nagios/plugins/slack.pl -field slack_channel=#nagios -field HOSTALIAS="$HOSTNAME$" -field SERVICEDESC="$SERVICEDESC$" -field SERVICESTATE="$SERVICESTATE$
" -field SERVICEOUTPUT="$SERVICEOUTPUT$" -field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"
}
define command {
command_name notify-host-by-slack
command_line /usr/lib/nagios/plugins/slack.pl -field slack_channel=#nagios -field HOSTALIAS="$HOSTNAME$" -field HOSTSTATE="$HOSTSTATE$"
}
Hi Matt,
ReplyDeletei want to get all my critical nagios alert at my webhook address.can u help me in that.
please
I have this setup on 4 Nagios instances and it's working great on 3 of the 4. On the 4th one, I am not receiving any notifications for any of my Hosts that begin with eu. Any insight here?
ReplyDeleteThank you Mathew for this wonderful writeup. This still take us to the right direction. For some reason I was not able to get a full format message posted on the slack channel. Then I used the Slack given slack_nagios.pl and along with some params in the command file: /usr/local/nagios/etc/objects/commands.cfg.
ReplyDelete# 'notify-service-by-slack' command definition
define command {
command_name notify-service-by-slack
command_line /usr/local/bin/slack_nagios.pl -field slack_channel=nagios-notification -field HOSTALIAS="$HOSTNAME$" -field SERVICEDESC="$SERVICEDESC$" -field SERVICESTATE="$SERVICESTATE$" -field SERVICEOUTPUT="$SERVICEOUTPUT$" -field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"
}
# 'notify-host-by-slack' command definition
define command {
command_name notify-host-by-slack
command_line /usr/local/bin/slack_nagios.pl -field slack_channel=nagios-notification -field HOSTALIAS="$HOSTNAME$" -field SERVICEDESC="$SERVICEDESC$" -field SERVICESTATE="$SERVICESTATE$" -field SERVICEOUTPUT="$SERVICEOUTPUT$" -field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"
}
Check for syntax error:
/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
Restart nagios:
service nagios restart
Message started coming in with the right format.
In my slack.log i get
ReplyDelete` % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 289 0 957 --:--:-- --:--:-- --:--:-- 957
0 8 0 8 0 289 12 455 --:--:-- --:--:-- --:--:-- 0
No hooks`
May be your webhooks are not correct .. generating a new slack token worked for me.
Deletehttps://${SLACK_HOSTNAME}/services/hooks/incoming-webhook?token=${SLACK_TOKEN}
I have the same issue. Can the API token be created from any user or from the Bot ?
Delete