Tuesday, December 31, 2013

Simple way to integrate Nagios with Slack messaging

At work we recently switched messaging applications from Skype to a new platform called Slack. Slack just launched in August 2013. I have read it is similar to Campfire but I've never used that platform so I can't really comment on that but it is much more useful than a basic chat client like Skype. With Slack you can share files, easily search message history for text or files and integrate with 3rd party applications. Plus it is private for just your team or company. Slack has quite a few preconfigured integrations plus the ability to create your own custom integrations. First we setup the Github integration which allows all of our commit messages to dump into a channel. Next we setup the Trello integration to dump card changes from our main board into another channel. Then I went to setup the Nagios integration and ran into problems. They have a prebuilt integration for Nagios but I could not get it to work. It would post alert messages into the channel but the messages contained no information:


I mucked with their provided perl script quite a bit but I simply could not get it to work. It just kept posting empty messages. Being impatient and a do-it-yourselfer I set about trying to find another way to accomplish this. I looked through the list of integrations and noticed that they had a custom one called Incoming WebHooks which is an easy way to get messages from external sources posted into Slack. The simplest way to utilize Incoming WebHooks is to use curl to post the message to Slack's API. I wrote a little bash script that provides a detailed Nagios alert, a link back to the Nagios web page and conditional emoji's! Each warning level (OK, WARNING, CRITICAL and UNKNOWN) has it's own emoji icon. Here are some example messages in my Slack client:


Here is my bash script that posts to Slack. I placed it in /usr/local/bin

Here are the Nagios config lines that are added to commands.cfg

And finally lines I added to contacts.cfg

I'm not sure why Slack's prebuilt Nagios integration didn't work for me but I really like what I came up with. No Perl modules to install and the only outside dependency is curl. It's also pretty easy to modify the info in the alert message by adding or removing NAGIOS_ env variables in the curl statement.

16 comments:

  1. Hi, great post. I found that the the variable SLACK_BOTNAME is not sent in the post to Slack but the undefined variable SLACK_USERNAME is.

    ReplyDelete
  2. [root@splunk bin]# /usr/local/bin/slack_nagios.sh
    /usr/local/bin/slack_nagios.sh: line 47: unexpected EOF while looking for matching `}'
    /usr/local/bin/slack_nagios.sh: line 48: syntax error: unexpected end of file
    [root@splunk bin]#

    ReplyDelete
    Replies
    1. I double checked my curly braces and they are correct. Maybe you lost one when you copied and pasted the code. At the bottom of the Gist above there is link that says 'view raw'. Or you can go here to download it: https://gist.github.com/matt448/8200821

      Delete
    2. It can be avoided by doing

      dos2unix slack_nagios.sh

      Delete
  3. I've met the similar issue and contacted the Slack team for the support.
    They advised me to use configuration recommended for "older versions of Nagios" (like `command_line /usr/local/bin/slack_nagios.pl -field slack_channel=#alerts -field HOSTALIAS="$HOSTNAME$" -field SERVICEDESC="$SERVICEDESC$" -field SERVICESTATE="$SERVICESTATE$" -field SERVICEOUTPUT="$SERVICEOUTPUT$" -field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"` ) and it worked just fine.

    ReplyDelete
    Replies
    1. Thanks for the update. Last year there wasn't much documentation for the Nagios integration. I'm sure they have improved the perl script and documentation since then.

      Delete
  4. I have tried all of the above and still no luck. The closest I've come is "Host '' is"

    ReplyDelete
  5. Matt, thanks for sharing!

    Above is a mix up of Perl (slack_nagios.pl) and Shell (slack_nagios.sh). I am using the shell script. Mine looks like (one line):
    curl -X POST --data-urlencode "payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_BOTNAME}\", \"text\": \"${ICON} +++ HOST: ${1} +++ SERVICE: ${2} +++ MESSAGE: ${4} +++ \", \"icon_emoji\": \":ghost:\"}" https://${SLACK_HOSTNAME}/services/hooks/incoming-webhook?token=${SLACK_TOKEN}

    My Nagios commancs.cfg looks like:
    define command {
    command_name notify-service-by-slack
    command_line /usr/local/bin/slack_nagios.sh \
    "$HOSTNAME$" \
    "$SERVICEDESC$" \
    "$SERVICESTATE$" \
    "$SERVICEOUTPUT$" \
    "$NOTIFICATIONTYPE$"
    }

    ReplyDelete
    Replies
    1. Thank you for this post! BASH didnt have many examples. I added this script to track myself: https://gist.github.com/tony-caffe/9d84265b36d92d89aa5ec04b6a0c81a8

      Delete
  6. I used the original slack.pl with this command config which works for me:
    define command {
    command_name notify-service-by-slack
    command_line /usr/lib/nagios/plugins/slack.pl -field slack_channel=#nagios -field HOSTALIAS="$HOSTNAME$" -field SERVICEDESC="$SERVICEDESC$" -field SERVICESTATE="$SERVICESTATE$
    " -field SERVICEOUTPUT="$SERVICEOUTPUT$" -field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"
    }

    define command {
    command_name notify-host-by-slack
    command_line /usr/lib/nagios/plugins/slack.pl -field slack_channel=#nagios -field HOSTALIAS="$HOSTNAME$" -field HOSTSTATE="$HOSTSTATE$"
    }

    ReplyDelete
  7. Hi Matt,
    i want to get all my critical nagios alert at my webhook address.can u help me in that.
    please

    ReplyDelete
  8. I have this setup on 4 Nagios instances and it's working great on 3 of the 4. On the 4th one, I am not receiving any notifications for any of my Hosts that begin with eu. Any insight here?

    ReplyDelete
  9. Thank you Mathew for this wonderful writeup. This still take us to the right direction. For some reason I was not able to get a full format message posted on the slack channel. Then I used the Slack given slack_nagios.pl and along with some params in the command file: /usr/local/nagios/etc/objects/commands.cfg.

    # 'notify-service-by-slack' command definition
    define command {
    command_name notify-service-by-slack
    command_line /usr/local/bin/slack_nagios.pl -field slack_channel=nagios-notification -field HOSTALIAS="$HOSTNAME$" -field SERVICEDESC="$SERVICEDESC$" -field SERVICESTATE="$SERVICESTATE$" -field SERVICEOUTPUT="$SERVICEOUTPUT$" -field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"
    }

    # 'notify-host-by-slack' command definition
    define command {
    command_name notify-host-by-slack
    command_line /usr/local/bin/slack_nagios.pl -field slack_channel=nagios-notification -field HOSTALIAS="$HOSTNAME$" -field SERVICEDESC="$SERVICEDESC$" -field SERVICESTATE="$SERVICESTATE$" -field SERVICEOUTPUT="$SERVICEOUTPUT$" -field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"
    }

    Check for syntax error:
    /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

    Restart nagios:
    service nagios restart

    Message started coming in with the right format.

    ReplyDelete
  10. In my slack.log i get
    ` % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed

    0 0 0 0 0 289 0 957 --:--:-- --:--:-- --:--:-- 957
    0 8 0 8 0 289 12 455 --:--:-- --:--:-- --:--:-- 0
    No hooks`

    ReplyDelete
    Replies
    1. May be your webhooks are not correct .. generating a new slack token worked for me.
      https://${SLACK_HOSTNAME}/services/hooks/incoming-webhook?token=${SLACK_TOKEN}

      Delete
    2. I have the same issue. Can the API token be created from any user or from the Bot ?

      Delete

Please note all comments are moderated by me before they appear on the site. It may take a day or so for me to get to them. Thanks for your feedback.