Tech Notes From The Trenches

Steps to monitor mongrel_rails in any environment, using monit (http://mmonit.com/monit/)

1. Monit must be installed in the server

Preferred version:
This is monit version 5.0_beta6
Copyright© 2008 by Tildeslash Ltd. All Rights Reserved.

Note: If not installed, download and install it.

2. Monit must have a configuration file in /etc/monitrc (or in other pre-specified places)

Sample monitrc:

set daemon 120 with start delay 60
set mailserver localhost

set mail-format {

from: sysadmin@aycron.com

subject: [aycron-server] $SERVICE $EVENT at $DATE

}
set alert etagwerker@aycron.com
set init
set logfile /var/log/monit

set httpd port 2812 and

use address localhost # only accept connection from localhost

allow localhost # allow localhost to connect to the server

##### mongrel 4010 #####

check process mongrel-4010 with pidfile /mnt/myapp.com/current/tmp/pids/mongrel.4010.pid

start program = ”/usr/bin/mongrel_rails cluster::start -C /mnt/myapp.com/shared/config/myapp.com.yml—clean—only 4010”

stop program = ”/usr/bin/ruby /usr/bin/mongrel_rails stop -P /mnt/myapp.com/current/tmp/pids/mongrel.4010.pid”

if totalmem is greater than 200.0 MB for 5 cycles then restart # eating up memory?

if cpu is greater than 50% for 2 cycles then alert # send an email to admin

if cpu is greater than 80% for 3 cycles then restart # hung process?

if loadavg(5min) greater than 10 for 8 cycles then restart # bad, bad, bad

#if 3 restarts within 5 cycles then timeout # something is wrong, call the sys-admin

if failed port 4010 protocol http # check for response

with timeout 120 seconds

then restart #it could also be then alert

group myapp_mongrels

##### mongrel 4011 #####

##### etc

##### …

Note: If it doesn’t exist, create it and define it for each mongrel_rails you want to follow.

3. Monit daemon must be running

To start it:

sudo monit -c /etc/monitrc &

Note: This will not start the processes defined in your /etc/monitrc – You could add it to inittab (in case system needs to reboot)

4. Fire up your processes (using the group name you defined in your /etc/monitrc)

sudo monit -g myapp_mongrels start all

You should see something like this:

Cleartext credentials needed for basic authorization!
start service ‘mongrel-4010’ on user request
monit daemon at 21065 awakened
Awakened by User defined signal 1
‘ip-10-251-214-192’ Monit has not changed
‘mongrel-4010’ start: /usr/bin/mongrel_rails
Cleartext credentials needed for basic authorization!
start service ‘mongrel-4011’ on user request
monit daemon at 21065 awakened
Cleartext credentials needed for basic authorization!
start service ‘mongrel-4012’ on user request
monit daemon at 21065 awakened
Cleartext credentials needed for basic authorization!
start service ‘mongrel-4013’ on user request
monit daemon at 21065 awakened
Cleartext credentials needed for basic authorization!
start service ‘mongrel-5010’ on user request
monit daemon at 21065 awakened
missing process: removing tmp/pids/mongrel.4010.pid
starting port 4010
‘mongrel-4010’ start action done
‘mongrel-4010’ start action done
‘mongrel-4011’ start: /usr/bin/mongrel_rails
starting port 4011
‘mongrel-4011’ start action done
‘mongrel-4011’ start action done
‘mongrel-4012’ start: /usr/bin/mongrel_rails
starting port 4012
‘mongrel-4012’ start action done
‘mongrel-4012’ start action done
‘mongrel-4013’ start: /usr/bin/mongrel_rails
starting port 4013
‘mongrel-4013’ start action done
‘mongrel-4013’ start action done
‘mongrel-5010’ start: /usr/bin/mongrel_rails
starting port 5010
‘mongrel-5010’ start action done
‘mongrel-5010’ start action done
Awakened by User defined signal 1

Note #1: You could add this command (or a restart command) to your cap file.

Note #2: To get rid of ‘Cleartext credentials needed for basic authorization!’, you will need to add something like this ‘allow username:password’ right below ‘allow localhost’ (or some other authentication method, more sophisticated approaches can be found in the monit manual)

5. You’re done!

If everything worked out correctly, you should receive a bunch of notifications in the mail box you defined in /etc/monitrc

Note: To test it, you could “kill -9” one of the mongrel_rails you’re ‘moniting’

There are better ways to define monitoring, but this will definitely be useful if you have nothing monitoring your production processes/mongrels. You can find more information about monit here: http://mmonit.com/monit/documentation/monit.html – Have fun!