Monitor S3 file ages with Nagios

I have started using Amazon S3 storage for a for a couple different things like static image hosting and storing backups. My backup scripts tar and gzip files and then upload the tarball to S3. Since I don't have a central backup system to alert me of failed backups or to delete old backups I needed to handle those tasks manually. S3 has built in lifecycle settings which I do utilize but as with everything AWS it doesn't always work perfectly. As for alerting on failed backups I decided to handle that by watching the age of the files stored in S3 bucket. I ended up writing a Nagios plugin that can monitor both the minimum and maximum age of files stored in S3. In addition to monitoring the age of backup files I think this could also be useful in monitoring the age of files if you use an S3 bucket as a temporary storage area for batch processing. In this case old files would indicate a missed file or possibly a damaged file that couldn't be processed.

I wrote this my favorite new language Python and used the boto library to access S3. The check looks through every file stored in a bucket and checks the file's last_modified property against the supplied min and/or max. The check can be used for either min age, max age or both. You will need to create a .boto file in the home directory of the user executing the Nagios check with credentials that have at least read access to the S3 bucket.

The check_s3_file_age.py file is available on my github nagios-checks repository here: https://github.com/matt448/nagios-checks.

To use this with NRPE add an entry something like this:

command[check_s3_file_age]=/usr/lib/nagios/plugins/check_s3_file_age.py --bucketname myimportantdata --minfileage 24 --maxfileage 720

Here is output from --help:

./check_s3_file_age.py --help

usage: check_s3_file_age.py [-h] --bucketname BUCKETNAME
                            [--minfileage MINFILEAGE]
                            [--maxfileage MAXFILEAGE] [--listfiles] [--debug]

This script is a Nagios check that monitors the age of files that have been
backed up to an S3 bucket.

optional arguments:
  -h, --help            show this help message and exit
  --bucketname BUCKETNAME
                        Name of S3 bucket
  --minfileage MINFILEAGE
                        Minimum age for files in an S3 bucket in hours.
                        Default is 0 hours (disabled).
  --maxfileage MAXFILEAGE
                        Maximum age for files in an S3 bucket in hours.
                        Default is 0 hours (disabled).
  --listfiles           Enables listing of all files in bucket to stdout. Use
                        with caution!
  --debug               Enables debug output.

I am a better sys admin than I am a programmer so please let me know if you find bugs or see ways to improve the code. The best way to do this is to submit an issue on github.

Here is sample output in Nagios

Compiling libhid for Raspbian Linux on a Raspberry Pi

My son and I are working on a project using a Raspberry Pi and I needed to be able to talk to a USB HID device. This requires a software library called libhid but unfortunately it is not available as a package on Raspbian linux. I downloaded the source and attempted to compile it but ran into an error:

lshid.c:32:87: error: parameter ‘len’ set but not used [-Werror=unused-but-set-parameter]
cc1: all warnings being treated as errors
make[2]: *** [lshid.o] Error 1
make[2]: Leaving directory `/root/libhid-0.2.16/test'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/libhid-0.2.16'
make: *** [all] Error 2

After some googling I found a couple others on the Raspberry Pi forums that ran into the same problem. One of the commenters came up with a simple fix that requires a quick edit of the source code. In ~/libhid-0.2.16/test you need to edit the file lshid.c

Here is the code before making the edit:

39 /* only here to prevent the unused warning */
40 /* TODO remove */
41 len = *((unsigned long*)custom);
42
43 /* Obtain the device's full path */


Here is the code after the edit.
You need to comment out line 41 and then add len = len; and custom = custom;

39 /* only here to prevent the unused warning */
40 /* TODO remove */
41 //len = *((unsigned long*)custom);
42 len = len;
43 custom = custom;
44
45 /* Obtain the device's full path */

After editing the file simply run configure, make and make install like normal. The library will be put into /usr/local. Make sure you run sudo ldconfig before trying to compile any software that uses libhid. Thanks Raspberry Pi forums!

Bandwidth limits for guest wifi on an ASA 5505

At work we have free wifi for our customers as a nicety and so they can download our smartphone app if needed. Initially I set it up with no bandwidth limits with the idea of keeping an eye on it and locking it down if there was abuse. Over the past few weeks my MRTG graphs showed several spikes where the free wifi hit 10mbps. That is a big chunk of our internet connection so I decided it was time to limit the bandwidth. Since I'm not a Cisco expert it took some Googling to find the best way to do this. I found a couple resources that helped me put together what I needed. The free wifi network is on a separate VLAN with it's own IP subnet.

Here is interface definition for the VLAN

interface Vlan92
  nameif freewifi
  security-level 50
  ip address 192.168.92.1 255.255.255.0


Here is the syntax I used to limit the freewifi VLAN to 2mbps. The limit is applied to the subnet used by the freewifi VLAN.

access-list ip-qos extended permit ip 192.168.92.0 255.255.255.0 any

access-list ip-qos extended permit ip any 192.168.92.0 255.255.255.0

class-map qos

  description qos policy

  match access-list ip-qos

policy-map qos

  class qos

    police output 2000000 2000000

    police input 2000000 2000000

service-policy qos interface freewifi

Testing

My initial thought for testing the bandwidth limits was to connect to the freewifi VLAN and simply use one of the internet speed testing web sites. The speed test web sites worked fine for download speeds but the upload tests kept reporting that they were getting the full bandwidth of the connection. It seemed like the upload limit wasn't being enforced. I tried all of the popular speed testing sites and got the same result. Downloads were limited to 2mbps and uploads were running at the full speed of the connection. Hmmm...

I reviewed my settings on the ASA and everything seemed like it was correct. I decided to do a different type of test to see if I would get a different result. I created a 10MB file and then tested uploading and downloading it to and from a server out on the internet using scp. This test gave me the results I was expecting. Both upload and download of this test file took about 35 seconds which is inline for a 2mbps connection. I then tested transferring the same file on the inside VLAN which has no bandwidth limits and the scp transfer time was 4 seconds. I'm not sure what was going on with the speed test sites but the upload speeds were not reporting accurately for me.


Monitoring status

You can watch the bandwidth limits in action using the 'show service-policy police' command. If the limit is exceeded the output will show the number of packets and bytes that have exceeded the bandwidth limit.

This is the command output before sending any traffic:

asa5505# show service-policy police

Interface freewifi:
  Service-policy: qos
    Class-map: qos
      Output police Interface freewifi:
        cir 2000000 bps, bc 2000000 bytes
        conformed 1306 packets, 907993 bytes; actions:  transmit
        exceeded 0 packets, 0 bytes; actions:  drop
        conformed 0 bps, exceed 0 bps
      Input police Interface freewifi:
        cir 2000000 bps, bc 2000000 bytes
        conformed 1072 packets, 192021 bytes; actions:  transmit
        exceeded 0 packets, 0 bytes; actions:  drop
        conformed 0 bps, exceed 0 bps


This is the output after transmitting several test files:

asa5505# show service-policy police

Interface freewifi:
  Service-policy: qos
    Class-map: qos
      Output police Interface freewifi:
        cir 2000000 bps, bc 2000000 bytes
        conformed 149813 packets, 127878453 bytes; actions:  transmit
        exceeded 10273 packets, 14716462 bytes; actions:  drop
        conformed 3384 bps, exceed 360 bps
      Input police Interface freewifi:
        cir 2000000 bps, bc 2000000 bytes
        conformed 157493 packets, 123699017 bytes; actions:  transmit
        exceeded 15083 packets, 21214456 bytes; actions:  drop
        conformed 4928 bps, exceed 760 bps





Resources

Source: http://community.spiceworks.com/topic/137645-bandwidth-limit-vlan-on-asa-5505

Source: http://www.youtube.com/watch?v=Xb2A3Xekp7Q

Template Nagios check for a JSON web service

I wrote two different custom Nagios checks for work last week and realized I could make a useful template out of them. After writing the first check I was able to reuse most of the code for the second check. The only changes I had to make had to do with the data returned. So I decided to make this into a generic template that I can reuse in the future. The check first verifies that the web service is responding correctly and then checks various data returned in JSON format.  While writing this template I found this really cool service (www.jsontest.com) that let me code against a service available to anyone who wants to try out this Nagios check before customizing it. This is the first time I have used Python's argparse function and I have to say it is fantastic. It makes adding command line arguments very easy and the result is professional looking.

My github repo can be found here: https://github.com/matt448/nagios-checks

Here is the code in a gist:

Nagios file paths on Ubuntu and simple backup script

This is more of a note to myself than anything else but might be helpful to others. Here are the config and data directories for Nagios when installed using packages on Ubuntu 12.04.


Config files
----------------------
/etc/nagios3/
/etc/nagios3/conf.d
/etc/nagios-plugins/config
/etc/nagios

Plugin executables
---------------------
/usr/lib/nagios/plugins

Graphing (pnp4naigos)
----------------------
/usr/share/pnp4nagios/html
/var/lib/pnp4nagios/perfdata

Other
-----------------------
/var/lib/nagios
/var/lib/nagios3



Here is a very simple backup script for Nagios on Ubuntu 12.04

Relaying Postfix through AuthSMTP on an alternate port

AuthSMTP is an authenticated SMTP relay service that you can use with web applications or any situation you need to send outbound e-mail. Because it is an authenticated service it is a little trickier to configure Postfix to relay through their service. I found this post really helpful in configuring the sasl options but one thing I couldn't find a clear answer on was how to use a port other than 25 for the relay host. AuthSMTP offers alternative ports (23, 26, 2525) for SMTP because some ISP's block port 25. To use an alternative port just put a colon after the host name and add the port number. Like this (in main.cf):

relayhost = mail.authsmtp.com:2525


The entry in your sasl-passwords file must match the relayhost name like this:

mail.authsmtp.com:2525 username:secretpassword


Just a quick tip about something that wasn't obvious to me and hopefully this helps out someone else.

File transfer through Windows Remote Desktop client

I can't believe I have never known about this feature of the Windows Remote Desktop client. I have been using Windows since version 3.1 and I use the Remote Desktop client (RDP) almost everyday at work. I have done an informal poll of several people I work with and none of them knew about this either. The feature I am talking about is mapping local drives to a remote machine using only the Remote Desktop client. I only ran across this myself while doing some reading about managing Amazon EC2 instances. I'm not sure why this feature is buried so deep in the options. Anyway here is how to use it:


Launch Remote Desktop and click on the 'Local Resources' tab.

I have used the Local Resources tab many times to change printer and audio settings. That 'More...' button hides some neat features. Click on 'More...' 

After clicking 'More...' you will have a list of all the drives available on your local PC. Check the box next to the drives you want to have mapped to the remote system.

After you have made the Remote Desktop connection to the remote machine this is how the drive appears on the remote system. There is an 'Other' section in the list of drives. Now you can to copy files back and forth to the remote system directly through the Remote Desktop client connection!