I mucked with their provided perl script quite a bit but I simply could not get it to work. It just kept posting empty messages. Being impatient and a do-it-yourselfer I set about trying to find another way to accomplish this. I looked through the list of integrations and noticed that they had a custom one called Incoming WebHooks which is an easy way to get messages from external sources posted into Slack. The simplest way to utilize Incoming WebHooks is to use curl to post the message to Slack's API. I wrote a little bash script that provides a detailed Nagios alert, a link back to the Nagios web page and conditional emoji's! Each warning level (OK, WARNING, CRITICAL and UNKNOWN) has it's own emoji icon. Here are some example messages in my Slack client:
Here is my bash script that posts to Slack. I placed it in /usr/local/bin
Here are the Nagios config lines that are added to commands.cfg
And finally lines I added to contacts.cfg
I'm not sure why Slack's prebuilt Nagios integration didn't work for me but I really like what I came up with. No Perl modules to install and the only outside dependency is curl. It's also pretty easy to modify the info in the alert message by adding or removing NAGIOS_ env variables in the curl statement.
Hi, great post. I found that the the variable SLACK_BOTNAME is not sent in the post to Slack but the undefined variable SLACK_USERNAME is.
ReplyDelete[root@splunk bin]# /usr/local/bin/slack_nagios.sh
ReplyDelete/usr/local/bin/slack_nagios.sh: line 47: unexpected EOF while looking for matching `}'
/usr/local/bin/slack_nagios.sh: line 48: syntax error: unexpected end of file
[root@splunk bin]#
I double checked my curly braces and they are correct. Maybe you lost one when you copied and pasted the code. At the bottom of the Gist above there is link that says 'view raw'. Or you can go here to download it: https://gist.github.com/matt448/8200821
DeleteIt can be avoided by doing
Deletedos2unix slack_nagios.sh
I've met the similar issue and contacted the Slack team for the support.
ReplyDeleteThey advised me to use configuration recommended for "older versions of Nagios" (like `command_line /usr/local/bin/slack_nagios.pl -field slack_channel=#alerts -field HOSTALIAS="$HOSTNAME$" -field SERVICEDESC="$SERVICEDESC$" -field SERVICESTATE="$SERVICESTATE$" -field SERVICEOUTPUT="$SERVICEOUTPUT$" -field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"` ) and it worked just fine.
Thanks for the update. Last year there wasn't much documentation for the Nagios integration. I'm sure they have improved the perl script and documentation since then.
DeleteI have tried all of the above and still no luck. The closest I've come is "Host '' is"
ReplyDeleteMatt, thanks for sharing!
ReplyDeleteAbove is a mix up of Perl (slack_nagios.pl) and Shell (slack_nagios.sh). I am using the shell script. Mine looks like (one line):
curl -X POST --data-urlencode "payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_BOTNAME}\", \"text\": \"${ICON} +++ HOST: ${1} +++ SERVICE: ${2} +++ MESSAGE: ${4} +++ \", \"icon_emoji\": \":ghost:\"}" https://${SLACK_HOSTNAME}/services/hooks/incoming-webhook?token=${SLACK_TOKEN}
My Nagios commancs.cfg looks like:
define command {
command_name notify-service-by-slack
command_line /usr/local/bin/slack_nagios.sh \
"$HOSTNAME$" \
"$SERVICEDESC$" \
"$SERVICESTATE$" \
"$SERVICEOUTPUT$" \
"$NOTIFICATIONTYPE$"
}
Thank you for this post! BASH didnt have many examples. I added this script to track myself: https://gist.github.com/tony-caffe/9d84265b36d92d89aa5ec04b6a0c81a8
DeleteI used the original slack.pl with this command config which works for me:
ReplyDeletedefine command {
command_name notify-service-by-slack
command_line /usr/lib/nagios/plugins/slack.pl -field slack_channel=#nagios -field HOSTALIAS="$HOSTNAME$" -field SERVICEDESC="$SERVICEDESC$" -field SERVICESTATE="$SERVICESTATE$
" -field SERVICEOUTPUT="$SERVICEOUTPUT$" -field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"
}
define command {
command_name notify-host-by-slack
command_line /usr/lib/nagios/plugins/slack.pl -field slack_channel=#nagios -field HOSTALIAS="$HOSTNAME$" -field HOSTSTATE="$HOSTSTATE$"
}
Hi Matt,
ReplyDeletei want to get all my critical nagios alert at my webhook address.can u help me in that.
please
I have this setup on 4 Nagios instances and it's working great on 3 of the 4. On the 4th one, I am not receiving any notifications for any of my Hosts that begin with eu. Any insight here?
ReplyDeleteThank you Mathew for this wonderful writeup. This still take us to the right direction. For some reason I was not able to get a full format message posted on the slack channel. Then I used the Slack given slack_nagios.pl and along with some params in the command file: /usr/local/nagios/etc/objects/commands.cfg.
ReplyDelete# 'notify-service-by-slack' command definition
define command {
command_name notify-service-by-slack
command_line /usr/local/bin/slack_nagios.pl -field slack_channel=nagios-notification -field HOSTALIAS="$HOSTNAME$" -field SERVICEDESC="$SERVICEDESC$" -field SERVICESTATE="$SERVICESTATE$" -field SERVICEOUTPUT="$SERVICEOUTPUT$" -field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"
}
# 'notify-host-by-slack' command definition
define command {
command_name notify-host-by-slack
command_line /usr/local/bin/slack_nagios.pl -field slack_channel=nagios-notification -field HOSTALIAS="$HOSTNAME$" -field SERVICEDESC="$SERVICEDESC$" -field SERVICESTATE="$SERVICESTATE$" -field SERVICEOUTPUT="$SERVICEOUTPUT$" -field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"
}
Check for syntax error:
/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
Restart nagios:
service nagios restart
Message started coming in with the right format.
In my slack.log i get
ReplyDelete` % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 289 0 957 --:--:-- --:--:-- --:--:-- 957
0 8 0 8 0 289 12 455 --:--:-- --:--:-- --:--:-- 0
No hooks`
May be your webhooks are not correct .. generating a new slack token worked for me.
Deletehttps://${SLACK_HOSTNAME}/services/hooks/incoming-webhook?token=${SLACK_TOKEN}
I have the same issue. Can the API token be created from any user or from the Bot ?
Delete