Dave Elkins

While working on a project with SOLR we wanted to setup SOLR to run as a daemon and monitor it with God.  While researching how to do this I found that while there some information, there was not a complete answer that completely worked for me.

So I thought I would share what I did.

Steps to install SOLR as a daemon

1. create /etc/init.d/solr.production and copy the below contents into the file. Please change all necessary paths.
2. change directory to /etc/init.d
3. run sudo /sbin/chkconfig --add solr.production
4. run sudo /sbin/service solr.production start
5. run ps -Af | grep start.jar to check that it is running
6. run cat /var/run/solr.production.pid (change path if needed) to see if PID value is correct.

/etc/init.d/solr.production

#!/bin/bash
# chkconfig: 2345 98 02
# description: Starts and stops Solr production

# source function library
. /etc/rc.d/init.d/functions

set -e

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

SOLR_HOME=/var/apps/solr
PIDFILE=/var/run/solr.production.pid

START_COMMAND="/usr/lib/jdk1.6.0_03/bin/java -Xms512M -Xmx1024M -jar start.jar"
LOG_FILE="$SOLR_HOME/logs/console.log"
NAME="Solr (production)"

start() {
  echo -n "Starting $NAME"
  if [ -f $PIDFILE ]; then
    echo -n "$PIDFILE exists. $NAME may be running."
  else
    cd $SOLR_HOME
    $START_COMMAND 2 > $LOG_FILE &
    sleep 2
    echo `ps -ef | grep -v grep | grep "$START_COMMAND" | awk '{print $2}'` > $PIDFILE
    echo "Done"
  fi
  return 0
}

stop() {
  echo -n "Stopping $NAME"
  kill `cat $PIDFILE`
  rm $PIDFILE
  echo "Done"
  return 0
}

case "$1" in
  start)
    start
  ;;
  stop)
    stop
  ;;
  restart)
    stop
    sleep 5
    start
  ;;

*)
echo "Usage: $0 (start | stop | restart)"
exit 1
esac

exit $?

UPDATE [03/10/2009]: The line `START_COMMAND=”/usr/lib/jdk1.6.0_03/bin/java -DSTOP.PORT=8079 -DSTOP.KEY=ftasolrstop -Xms512M -Xmx1024M -jar start.jar”` has now been changed to `START_COMMAND=”/usr/lib/jdk1.6.0_03/bin/java -Xms512M -Xmx1024M -jar start.jar”`

Thank you Cody for your comment and input. The reason for the change is that since I am just using the PID to kill the process I do not need those command line arguments because those are need for a different method of killing the process.

Setting Up Monitoring with God

1. install and setup God
2. create /etc/godrc/solr-production.god and copy the below contents into the file. Change paths and names as necessary.
3. restart God and SOLR will now be monitored by God

/etc/godrc/solr-production.god

God.watch do |w|
  w.name = "solr-production"
  w.group = "solr-production"
  w.interval = 30.seconds
  w.start = "/sbin/service solr.production start"
  w.stop = "/sbin/service solr.production stop"
  w.restart = "/sbin/service solr.production restart"
  w.start_grace = 10.seconds
  w.restart_grace = 10.seconds
  w.pid_file = "/var/run/solr.production.pid"
  w.behavior(:clean_pid_file)

  w.start_if do |start|
    start.condition(:process_running) do |c|
      c.interval = 5.seconds
      c.running = false
    end
  end

  w.restart_if do |restart|
    restart.condition(:memory_usage) do |c|
      c.above = 1024.megabytes
      c.times = [3,5] #3 out of 5 intervals
    end

    restart.condition(:cpu_usage) do |c|
      c.above = 25.percent
      c.times = 5
    end
  end

  w.lifecycle do |on|
    on.condition(:flapping) do |c|
      c.to_state = [:start, :restart]
      c.times = 5
      c.within = 5.minute
      c.transition = :unmonitor
      c.retry_in = 10.minutes
      c.retry_times = 5
      c.retry_within = 2.hours
    end
  end
end

Those are the config files to setup to run SOLR as daemon and monitor it with God.  If you have suggestions on how to did it better, please let me know.  I hope you find this helpful.

References

1. http://tldp.org/LDP/abs/html/
2. http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html
3. http://www.tech-recipes.com/rx/205/determine-if-file-exists-in-a-bournebash-shell-script/
4. http://pauldoerwald.ca/2008/7/29/hints-for-deploying-solr-for-rails-projects
5. http://breaksalot.org/solrinit

Updates

1. Fixed some html escaped characters in the code. Thanks, Mike for catching those.