Arduino - Improving refresh rate of a TFT LCD display

I'm working on a digital speedometer project for my car and I ordered a couple different displays from Adafruit to experiment with. One of the displays I ordered is a 2.2" color TFT LCD display. I was really hoping to use this display to show the current speed of my car. One concern I had about using it for that purpose was the speed that an Arduino board could refresh the information on the screen. To test the display I wrote a simple sketch that just displays the number of seconds since start. Here is a video of that first attempt:



The redraw time causes the numbers to look like they are flashing which is very annoying. The mistake I made was blanking out the entire screen with tft.fillScreen(ILI9340_BLACK). It takes quite a while to redraw the entire screen with black.

Next I tried drawing a black rectangle just on the area of the screen where I was drawing numbers. Here is the result:



Huge improvement. Only blanking out that small area greatly sped up the redraw time.
In this next video I have highlighted the redraw area in red to make it easier to see what I am talking about:



What I am realizing at this point is you should only redraw information that has changed. If you watch the last video again, once the number goes past single digits the left digit looks like it is flashing. This is because it is being blanked out and redrawn every time the number changes. When you redraw something that hasn't changed it looks like flashing. Well that's it for now. I am going to continue to work on this more and figure out how to selectively redraw only digits that change. Once I figure that out I'll write up another blog post.
[UPDATE] - Here is part two:
 http://matthewcmcmillan.blogspot.com/2014/08/arduino-tft-lcd-display-refresh-rate-part2.html


Here is the code I was using to do my testing.



[Updated 2014-08-08 - Added link to part 2 of this topic]

Daemonized rvm ruby tasks using start-stop-daemon

I am very new to Ruby so the solution described in this post might be very obvious to some but I could not find all the parts of this solution in one place. I cobbled together bits and pieces of other peoples startup and capistrano scripts to get this working. My research has also shown there are ruby gems that might handle some of this better but I wasn't trying to reinvent the wheel. Anyhow, on with the show. At work, during our capistrano deployment we have a ruby process that has to be launched in the background. We are using start-stop-daemon to daemonize the process but our use of rvm complicates running rake because the rake binary is stored in the .rvm/gems directory. Normally the path to the rake binary is set from .bashrc and .bash_profile when you log in through a shell but when you execute things from cron, start-up scripts or other non-shell environments the paths don't exist. After some googling and tinkering I finally got something that worked:

task :start_mytask, :roles => :mytask, :except => { :no_release => true } do
  run "RAILS_ENV=#{rails_env} start-stop-daemon --start -b -m -o -d ~/current -p ~/pids/mytask.pid -a /home/ubuntu/.rvm/gems/ruby-1.9.3-p194@global/bin/rake mytask"
end

The downside with this script is I was calling rake for a specific version of ruby. At the time this was good enough to get us by. Fast forward several months and now we are preparing to upgrade to ruby 2.0. Our existing start script needed to be updated to use ruby 2.0 but I wanted to find a better way that wouldn't need to be tweaked each time we upgrade ruby. My first shot at updating the script I switched to executing rvm and calling 'bundle exec rake':

task :start_mytask, :roles => :mytask, :except => { :no_release => true } do
   run "RAILS_ENV=#{rails_env} start-stop-daemon --start -b -m -o --chdir ~/current --pidfile ~/pids/mytask.pid --exec ~/.rvm/bin/rvm -- current do bundle exec rake mytask"
end

This eliminates directly calling rake for a specific version of ruby but introduces a different issue. Executing it this way causes rvm to launch in a bash shell which then executes the rake task. start-stop-daemon creates a pid file based off of the first process started which is the bash process not the rake task. So when you try to stop it start-stop-daemon tries to kill the bash process which won't end because it has a child process running the rake task. I was getting closer but it wasn't perfect. More googling ensued and I discovered I could use rvm-exec instead of rvm. The rvm-exec command fixes the bash shell problem. It was created specifically for calling rvm in scripts. Here is what I finally came up with:

task :start_mytask, :roles => :mytask, :except => { :no_release => true } do
  run "RAILS_ENV=#{rails_env} start-stop-daemon --start -b -m -o --chdir ~/current --pidfile ~/pids/mytask.pid --exec ~/.rvm/bin/rvm-exec -- current bundle exec rake mytask"
end

It will now run independent of a ruby version number and since only one process is launched the pid file is created correctly. One other benefit I realized is now it is much easier to monitor this process. Previously the process was listed as just "rake". If you have more than one running that would get tricky to monitor. Launching it with this new script the process is listed as "rake mytask".

Finally to stop the daemonized rake task run this command:

task :stop_mytask, :roles => :mytask, :except => { :no_release => true } do
   run "start-stop-daemon -o -p ~/pids/mytask.pid --stop"
end


[UPDATE 01/13/2014] I realized today after doing more testing that I should be calling 'current bundle exec rake' instead of 'default bundle exec rake'. When you are upgrading ruby versions it doesn't necessarily mean you want to change the default ruby version in rvm. Using 'current' will cause the script to use whatever ruby is specified in the .rvmrc. I have edited the above scripts to reflect my new findings.

Simple way to integrate Nagios with Slack messaging

At work we recently switched messaging applications from Skype to a new platform called Slack. Slack just launched in August 2013. I have read it is similar to Campfire but I've never used that platform so I can't really comment on that but it is much more useful than a basic chat client like Skype. With Slack you can share files, easily search message history for text or files and integrate with 3rd party applications. Plus it is private for just your team or company. Slack has quite a few preconfigured integrations plus the ability to create your own custom integrations. First we setup the Github integration which allows all of our commit messages to dump into a channel. Next we setup the Trello integration to dump card changes from our main board into another channel. Then I went to setup the Nagios integration and ran into problems. They have a prebuilt integration for Nagios but I could not get it to work. It would post alert messages into the channel but the messages contained no information:

I mucked with their provided perl script quite a bit but I simply could not get it to work. It just kept posting empty messages. Being impatient and a do-it-yourselfer I set about trying to find another way to accomplish this. I looked through the list of integrations and noticed that they had a custom one called Incoming WebHooks which is an easy way to get messages from external sources posted into Slack. The simplest way to utilize Incoming WebHooks is to use curl to post the message to Slack's API. I wrote a little bash script that provides a detailed Nagios alert, a link back to the Nagios web page and conditional emoji's! Each warning level (OK, WARNING, CRITICAL and UNKNOWN) has it's own emoji icon. Here are some example messages in my Slack client:

Here is my bash script that posts to Slack. I placed it in /usr/local/bin

Here are the Nagios config lines that are added to commands.cfg

And finally lines I added to contacts.cfg

I'm not sure why Slack's prebuilt Nagios integration didn't work for me but I really like what I came up with. No Perl modules to install and the only outside dependency is curl. It's also pretty easy to modify the info in the alert message by adding or removing NAGIOS_ env variables in the curl statement.

Upgrading existing Solr installation to new version of Jetty

At work we have been running into a problem with Apache Solr crashing. Depending on how much it was used we would get several weeks of usage out of it before it crashed. Now it is only running for five days at a time. So this fire has started burning hot enough to be at the top of my to-do list.
When it crashes it throws errors saying "Too many open files". Running lsof showed it wasn't actually open files but thousands of orphaned sockets left open. The sockets looked like this in the lsof output:

java 2428 root 2173u sock 0,7 0t0 123291433 can't identify protocol

There won't be anything listed in netstat. These sockets don't have open connections to anything. The Solr log file will start showing errors similar to this:

SEVERE: java.io.FileNotFoundException: /usr/local/apache-solr-3.5.0/example/solr/data/index/_dgf.frq (Too many open files)

SEVERE: SolrIndexWriter was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!

SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@./solr/data/index/write.lock

Initially we dealt with this problem by monitoring the number of open files for the java process and running a reindex when it got close to the limit. Not a great solution but at the time there weren't enough hours in the day to put a bunch of effort into figuring this out. In my case the limit when Solr blew up was 4000 open sockets. Once Solr had that many sockets open it would just throw 500 errors.

Usually the answer to a situation like this is upgrade Solr to a newer version. Unfortunately I couldn't do that in this case because we have a ruby gem that is dependent on Solr version 3.5. My research pointed to Jetty as the source problem and not Solr. Once I found this post I knew for sure Jetty was causing the orphaned sockets. Solr 3.5.0 is packaged with Jetty 6.1.26 which has a bug that causes the orphaned sockets under certain conditions. Because Jetty 6 is fairly old the developers are not going to fix it. At this point I set about upgrading Jetty to version 7.

The first thing I had to figure out was what stuff was Solr and what stuff was Jetty. Turns out most of the package is Jetty. Solr is contained in apache-solr-3.5.0/example/solr and apache-solr-3.5.0/example/webapps/solr.war. So I decided to try and stuff Solr 3.5.0 into Jetty 7.6.13. Later I may try moving to the latest version of Jetty 9 but I'm just trying to solve this orphaned socket problem right now and was worried the older version of Solr might have problems with a newer Jetty.

Upgrading Jetty

Here are the steps I took to upgrade Solr 3.5.0 to Jetty 7

Download latest Jetty 7 (jetty-distribution-7.6.13.v20130916.tar.gz at the time this was written) from here http://download.eclipse.org/jetty/7.6.13.v20130916/dist/

Untar jetty-distribution-7.6.13.v20130916.tar.gz

tar xfvz jetty-distribution-7.6.13.v20130916.tar.gz

Create destination directory for all the new files

mkdir /usr/local/apache-solr-3.5.0-jetty-7.6.13

mkdir /usr/local/apache-solr-3.5.0-jetty-7.6.13/example

copy the contents of jetty-distribution-7.6.13.v20130916 to new directory

cp -a jetty-distribution-7.6.13.v20130916/* /usr/local/apache-solr-3.5.0-jetty-7.6.13/example

Copy solr files from old solr installation to new Jetty directory

cp -a /usr/local/apache-solr-3.5.0/example/solr  /usr/local/apache-solr-3.5.0-jetty-7.6.13/example

cp -a /usr/local/apache-solr-3.5.0/example/webapps/solr.war /usr/local/apache-solr-3.5.0-jetty-7.6.13/example/webapps/

Edit the jetty.xml config file to change the listening port

vi usr/local/apache-solr-3.5.0-jetty-7.6.13/example/etc/jetty.xml

Change this line

 <Set name="port"><Property name="jetty.port" default="8080"/></Set>

To this

 <Set name="port"><Property name="jetty.port" default="8983"/></Set>

At this point solr will run but there are some example war files and config files that aren't needed for Solr and should be cleaned up. 

- Edit /usr/local/apache-solr-3.5.0-jetty-7.6.13/example/start.ini

   vi /usr/local/apache-solr-3.5.0-jetty-7.6.13/example/start.ini

   Comment out the line

   etc/jetty-testrealm.xml

   so it reads 

   #etc/jetty-testrealm.xml

- Clean up example war files

  cd /usr/local/apache-solr-3.5.0-jetty-7.6.13/example/webapps

  mkdir BAK

  mv test.war spdy.war BAK

- Clean up example config files
  cd /usr/local/apache-solr-3.5.0-jetty-7.6.13/example/etc
  mkdir BAK
  mv jetty-spdy.xml jetty-spdy-proxy.xml jetty-testrealm.xml BAK
  cd /usr/local/apache-solr-3.5.0-jetty-7.6.13/example/contexts
  mkdir BAK
  mv test.xml BAK

I use a symbolic link for the installation directory so the start script doesn't have to be modified. Before restarting I have to switch that sym link.
  service solr stop
  cd /usr/local
  rm solr
  ln -s apache-solr-3.5.0-jetty-7.6.13 solr
  service solr start

Then you can test hitting the service locally.
  curl localhost:8983/solr/

it should return html that says something like this:
  <title>Welcome to Solr</title>
  </head>

  <body>
  <h1>Welcome to Solr!</h1>

You will probably need to run a reindex if transactions have been taking place while solr was down for the upgrade.

Resources used to compile this post
http://comments.gmane.org/gmane.comp.ide.eclipse.jetty.user/919
https://github.com/umars/jetty-solr
http://stackoverflow.com/questions/6425759/how-to-upgrade-update-the-solr-jetty-ubuntu-package
https://jira.codehaus.org/browse/JETTY-1458
http://grokbase.com/t/lucene/solr-user/123e6et8e0/too-many-open-files-lots-of-sockets

Arduino - Using digital potentiometers (AD8403)

I have seen several blog posts covering the use of digital potentiometers with an Arduino but I haven't seen any that demonstrate digital pots that contain a shutdown circuit. I'm working on a project that needs this particular feature.

The shutdown circuit is of interest to me because I plan on interfacing with something that controls movement. In that application any value on the digital pot will cause a motor to move. Lower resistive values move forward and higher resistive values move back. I need a way to completely disable the potentiometer. My first prototype used two relays, two transistors and two resistors with different values. The Arduino would turn off and on the relays to cause movement. It worked because I could simply turn off both relays but it was quite bulky and the mechanical relays were noisy. A friend recommend a digital potentiometer as a replacement. A single digital pot chip would replace six components. I began researching them and found that not all of them include a shutdown circuit. I settled on the Analog Devices AD8403 chip. It is a quad channel, 256 position digitally controlled variable resistor. Communication is done through an SPI interface.

So lets start with a demonstration of what happens when there is no shutdown circuit on a digital potentiometer and the Arduino is reset. The circuit and code cycles through each digital pot varying the brightness of an LED by running up and down all 255 positions of the pot. While this is running I press the reset button on the Arduino.



In this situation the digital pot chip still has power and the value of one the pots is stuck while the Arduino resets (demonstrated by the LED staying lit). In my real application this would cause movement to continue until the board finished restarting which is highly undesirable. This effect could be minimized somewhat by including code to reset values of the four pots on startup but there would still be a second or two of uncontrolled movement while the board restarts.

Next is a demonstration of the same circuit but with the shutdown circuit enabled. The shutdown pin is connected to a pull down resistor. This causes the chip to go into shutdown mode if the Arduino resets or loses power. Again I press the reset button on the Arduino as it is running the code.



This is getting close to what I want but there is still the problem of the pot being set to some unknown value when the AD8B403 is taken out of shutdown mode. To handle this I added code to the setup function that sets the pots to a known value (zero in this case) before taking the chip out of shutdown mode.

Circuit

Here is a photo and Fritzing diagram of the circuit.

The connections are:

  * All A pins of AD8403 connected to +5V

  * All B pins of AD8403 connected to ground

  * An LED and a 220-ohm resisor in series connected from each W pin to ground

  * RS - to +5v

  * SHDN - to digital pin 7 and a 10k ohm pull down resistor

  * CS - to digital pin 10  (SS pin)

  * SDI - to digital pin 11 (MOSI pin)

  * CLK - to digital pin 13 (SCK pin)

The AD8403 is the 10k ohm version.

The Fritzing diagram is available here: https://github.com/matt448/arduino/raw/master/SPI_Digital_Pot_AD8403/SPI_Digital_Pot_AD8403.fzz

Code

The most update to date version of the code is available here: https://github.com/matt448/arduino/blob/master/SPI_Digital_Pot_AD8403/SPI_Digital_Pot_AD8403.ino

The code to control this isn't very complex. This is my first time using an SPI device and I found it to be very straight forward. The Arduino IDE has an example sketch for controlling SPI digital pots under File > Examples > SPI > DigitalPotControl. I started with the example and modified it to include shutdown control. That example code also ran a bit slow because of the serial console messages so I removed all the serial communication code.

Comment on shutdown pin

So after getting through all this testing I noticed a limitation of the shutdown pin, it shuts down all four pots at the same time. In my real application I plan on controlling movement on four separate motors and moving them to a certain positions. Once the motor reaches it's position I want it to stop. But the only way to stop the motor is activate the shutdown which deactivates all four pots. This is a problem because the other three motors may not have reached their position when I need to shutdown. I did some more looking and found that Microchip makes a digital pot that has individually controllable software shutdown for each pot. I'm going to order a couple MCP4251 chips and I'll write up another post when I test them out.

UPDATE 3/17/2014: Post about the MCP4251 is available here
UPDATE 3/27/2014: Added list of connections and resistor values. Also added links to github repo.

Arduino - Sending data over a CAN bus

I have been tinkering with CAN buses due to my interest in cars. It's fascinating to me that packets are flying around a modern vehicle controlling nearly everything. Gauges, lights, locks, engine sensors, etc. To have a better understanding of the basics of a CAN bus I wanted to build the simplest possible setup to send and receive CAN messages. I chose two Arduino Uno's with a Seeed Studio CAN-BUS shield attached to each Uno. The Seeed shield is very straight forward and inexpensive. The Sparkfun CAN-BUS shield has an SD card slot, LCD connector and GPS connector. All of which are cool but drive up the price and complexity. The Seeed shield only does CAN bus and includes screw terminals which are handy for testing.

Arduino Uno R3	Seeed CAN-BUS Shield

What I wanted to do with this experiment was transmit the value of an analog pin hooked up to a linear potentiometer. The data would be sent from one Arduino to another over a CAN bus and then display that value on an LCD connected to the second Arduino. Here is a picture of my setup. (Ignore the Mega2560 above the LCD. It's not used here.)

And here is a Fritzing diagram minus the CAN-BUS shields.

CAN bus termination

A CAN bus requires 120 Ohm termination resistors at each end of the bus. The Seeed Studio shields have built in termination resistors. When you connect two Seeed CAN bus shields togther like I did in this example you will have a properly terminated CAN bus. If you plan on connecting into an existing CAN bus that already has termination you can disable the built in termination resistors. To disable termination you can cut trace P1 or you can desolder resistor R1.

Close up view of the Seeed CAN bus shield
 termination resistors.

**Note: I have recently discovered the Seeed Studio CAN-BUS shield v1.0 uses a 60 ohm termination resistor for R3. While that worked for this small demo I later ran into issues when trying to use this shield with other nodes on a CAN bus. This 60 ohm resistor caused me many hours of frustration. If you are going to use this shield with on a bus with multiple nodes I would recommend desoldering R3 and using the correct 120 ohm resistance at the ends of your bus. 

Connecting into an existing CAN bus

If you are planning on connecting into an existing CAN bus (like in a car) you need to remove/disable the termination resistor on the shield as explained above. The CAN bus in a vehicle already has termination resistors. Adding a new node with a termination resistor will cause errors and disrupt communication on the bus.

Another important step is to connect a common ground between your Arduino board and the vehicle. If you are connecting at the OBD2 port pin 5 provides a signal ground. If you can't find a signal ground wire a chassis ground will suffice.

CAN bus messages

So I should probably explain a bit about CAN bus messages. Each message is made up of an id and some data. The id's in hex start at 0x000 and go to 0x7FF or 0 to 2047 in decimal. In most systems lower id values are considered more important. The bus handles collisions by letting the lower id win the collision. The data can be between 1 and 8 bytes for each message. Each byte can have a value from 0 to 255 or in hex 0x00 to 0xFF. When you send a CAN bus message you transmit the id, how many bytes you are sending (this is called DLC) and the actual data. The receiver will only read the number of bytes you said should be in the message. So if you send a DLC of 4 but the message contains 8 bytes the receiver will only read the first 4 bytes. Eight bytes per message is a bit limiting but the tradeoff is the high reliability of the bus. So sometimes you have to be creative with stuffing data into those bytes. If the value you are sending is less than 255 you can just use a single byte. Larger numbers will require using multiple bytes. Ascii codes can be sent but only eight characters per message. Whatever method you use to stuff the data in will also have to be used to un-stuff the data on the receiver. In my simple example here I did some math to limit the range of values to 0-255. An analog pin produces values between 0-1024. I simply divided the result by four to give me data I could send in a single byte.

CAN buses can operate at several different speeds up to 1 Mbit/s. Typical rates are 100 kbit/s, 125 kbit/s and 500 kbit/s. Slower rates allow for longer length buses. All devices on a bus must transmit at the same speed. The CAN bus wikipedia page is a good place to start if you want to learn more about the CAN protocol.

Code

I started with the example code provided by Seeed and modified it to add in the LCD output on the 'receiver' device and added reading of the potentiometer on A0 for the value that is transmitted. They have basic examples for send and receive. You can find some good info on their wiki page. Their libraries are available here. On my Mac I created the directory ~/Documents/Arduino/libraries/CAN_BUS_Shield for the library files. I unzipped the file and copied over the .h and .cpp files into that new directory. The zip file also contains the send and receive examples.

Note that normally devices on a CAN bus are both receivers and transmitters of data. This is a simplified example where each device is only doing one task.

Sender code

Receiver code

Video

[Updated 2014-05-25: Noted value of A0 potentiometer in the Fritzing diagram]
[Updated 2014-07-21: Added section about termination resistors]
[Updated 2014-09-25: Added note about incorrect value of resistor R3 on Seeed's shield]
[Updated 2015-03-10: Added additional notes about termination resistors]
[Updated 2017-03-27: Added new section 'Connecting into an existing CAN bus']
[Updated 2018-06-15: Fixed broken links for Seeed-Studio wiki and libraries]

Nagios monitoring for Amazon SQS queue depth

I have found that a bunch of messages stacking up in my SQS queue's can be the first sign of something breaking. Several things can cause messages to stack up in the queue. I have seen malformed messages, slow servers and dead processes all cause this at different times. So to monitor the queue depth I wrote this Nagios check / plug-in. The check simply queries the SQS api and finds out the count of messages in each queue. Then it compares the count to the warning and critical levels.

This check is written in python and uses the boto library. It includes perfdata output so you can graph the number of messages in the queue. The the AWS API for SQS does wildcard matching of queue names so you can monitor a bunch of queues with one check if they have some sort of common prefix to the name. The way I use this is I have several individual checks using the complete explicit name of the queue and then a catchall using a wildcard set to a higher number that will catch any queues that have been added. Make sure you have a .boto file for the user that will be running this nagios check. It only requires read permissions.

Some queues may be more time sensitive than others. That is the case for my setup. For queues that are time sensitive I set the warning and critical counts to low values. Less time sensitive queues are set to higher count values. This screenshot is an example of that:

Config

Here is the command definition I use for Naigos:

# 'check_sqs_depth' command definition
define command{
        command_name    check_sqs_depth
        command_line    /usr/lib/nagios/plugins/check_sqs_depth.py --name '$ARG1$' --region '$ARG2$' --warn '$ARG3$' --crit '$ARG4$'
        }

and here is the service definition I'm using

define service{
        use                                 generic-service   
        host_name                      sqs.us-east-1
        service_description          example_name SQS Queue
        contact_groups                admins,admins-page,sqs-alerts
        check_command             check_sqs_depth!example_name!us-east-1!150!300!
        }

Code

The code is available on my github nagios-checks repository here: https://github.com/matt448/nagios-checks and I have posted it as a gist below. My git repository will have the most up-to-date version