In another thread, I had posted that we rotate our mongrel production
logs once a week as a means to keep the performance degradations that
happen with large log files at bay. A few of you asked me to post how
we do it, so here it is. We run our sites on RedHat Enterprise 4
machines. Our logrotate.conf file looks like this:
see “man logrotate” for details
rotate log files weekly
weekly
keep 4 weeks worth of backlogs
rotate 4
create new (empty) log files after rotating old ones
create
uncomment this if you want your log files compressed
#compress
RPM packages drop log rotation information into this directory
system-specific logs may be also be configured here.
mongrel logs
/path/to/rails/root/shared/log/*.log {
copytruncate
weekly
missingok
dateext
compress
sharedscripts
olddir /path/to/rails/root/old_mongrel_logs
rotate 28
postrotate
for i in ls /path/to/rails/root/shared/log/mongrel*.pid; do
kill -USR2 cat $i
done
endscript
}
Incidentally, we also use monit to monitor our various processes
(mongrel, Apache, ferretDRb MySQL, etc), so if for some odd reason
this log rotation would not bring one of our mongrels back up cleanly,
we’d be alerted and monit would try to restart it. So far, this has
not happened at all across any of our 6 production setups just like
this.
None of the sites are extremely high traffic, so it’s possible that
you’d want to rotate mongrel log files more frequently than weekly.
postrotate
for i in `ls /path/to/rails/root/shared/log/mongrel*.pid`; do
kill -USR2 `cat $i`
done
endscript
Incidentally, we also use monit to monitor our various processes
(mongrel, Apache, ferretDRb MySQL, etc), so if for some odd reason
this log rotation would not bring one of our mongrels back up cleanly,
we’d be alerted and monit would try to restart it.
Why not ask monit to restart your mongrels? It seems kind of nice to
have monit be the only guy in town touching the processes it
monitors… but maybe there’s a technical reason why not?
this log rotation would not bring one of our mongrels back up cleanly,
we’d be alerted and monit would try to restart it.
Why not ask monit to restart your mongrels? It seems kind of nice to
have monit be the only guy in town touching the processes it
monitors… but maybe there’s a technical reason why not?
-Nate
No technical reason. Simply timing. Monit wakes itself every X
minutes to check if processes are running, so there could be a delay
of up X minutes until the mongrel process was restarted. The approach
in the logrotate restarts the mongrels immediately. Interestingly,
monit does realize that the process ID for each mongrel has changed
and alerts of that fact, but since we know that will happen every
Sunday at 3:00 AM, we can safely ignore it.
Why not ask monit to restart your mongrels? It seems kind of nice to
have monit be the only guy in town touching the processes it
monitors… but maybe there’s a technical reason why not?
No technical reason. Simply timing. Monit wakes itself every X
minutes to check if processes are running, so there could be a delay
of up X minutes until the mongrel process was restarted. The approach
in the logrotate restarts the mongrels immediately. Interestingly,
monit does realize that the process ID for each mongrel has changed
and alerts of that fact, but since we know that will happen every
Sunday at 3:00 AM, we can safely ignore it.
I’m pretty sure that doing:
monit restart all -g your_mongrel_group
will wake monit up and have it restart your app before hitting its
timeout. This also avoids the worried emails from monit; ignoring ‘PID
changed’ messages from monit (even if you’re expecting them) is a bad
habit to get into.