Selectively failing when mongrel restarted with monit [was:

OK. Made a bit of progress on this. Have setup a duplicate server
instance with mongrel_cluster on it (really glad I went with Xen now)
and also done a bit of following back the code with ruby-debug.
I can now reproduce the error, but only if I (re)start the mongrels with
monit. Then, the particular page fails every time. Here’s the particular
part of the montirc file that restarts the mongrel instances:

check process mongrel_8001 with pidfile
/var/run/mongrel_cluster/mongrel_carstuff.8001.pid
  start program = "/usr/bin/mongrel_rails cluster::start -C
/home/frankg/sites/carstuff/current/config/mongrel_cluster.yml
--clean --only 8001"
  stop program = "/usr/bin/mongrel_rails cluster::stop -C
/home/frankg/sites/carstuff/current/config/mongrel_cluster.yml
--clean --only 8001"

However, if I manually start a mongrel instance with:
/usr/bin/mongrel_rails cluster::start -C
/home/frankg/sites/carstuff/current/config/mongrel_cluster.yml --clean
–only 8001
the page is rendered with no problems.

Couple of other things, What seemed to be happening on that page was
that the Photo model was being reloaded, as if it was in the development
environment. It fails the first time (ImageScience fails on the line env
= ENV[‘INLINEDIR’] || ENV[‘HOME’] – both are nil). Why it occurs on
that page and not on others, I still haven’t figured out.

Will keep plugging away at this, but if anyone’s got any bright ideas
would welcome them.

Thanks
Chris

-------- Original Message --------
From: - Wed Apr 04 18:25:04 2007
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00800000
Message-ID: [email protected]
Date: Wed, 04 Apr 2007 18:25:02 +0100
From: Chris T [email protected]
Reply-To: [email protected]
User-Agent: Thunderbird 1.5.0.10 (Macintosh/20070221)
MIME-Version: 1.0
To: [email protected]
Subject: Suggestions needed for strange problem on only one mongrel
instance out of cluster
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

I’m about to release my app (was going to today), but have found a last
minute bug that I’m have difficulty tying down, never mind fix. It’s
occurring sporadically, and is fixed (temporarily) by restarting the
instance of the mongrel, but is not always caused by the same page.
Honestly, I’m not sure I know where to start trying to track it down,
and any help would be appreciated.

What’s happening is this. If I hit one particular page (have observed
this on only two pages so far but of course may affect others) with
photos on it a number of times it renders fine (the photos are linked to
the model via attachment_fu, which is using Image Science, not RMagick).

Then suddenly, it blows up, with one of two errors. The first error is
the following (can post full stack trace if anyone thinks it’s useful):

A ActionView::TemplateError occurred in modtypes#show:

exit
On line #1 of app/views/shared/_main_photo.rhtml

1: <% if main_photo.medium_image_url %>
2:


3: <%= link_to image_tag(main_photo.medium_image_url, {:alt =>
main_photo.title, :title => main_photo.caption}), photo_path(main_photo)
%>
4:
<%= link_to(
truncate(main_photo.caption,80, “… see more”), photo_path(main_photo))
if main_photo.caption %>

/usr/lib/ruby/gems/1.8/gems/RubyInline-3.6.2/inline.rb:70:in exit' /usr/lib/ruby/gems/1.8/gems/RubyInline-3.6.2/inline.rb:70:in rootdir’
/usr/lib/ruby/gems/1.8/gems/RubyInline-3.6.2/inline.rb:84:in
directory' /usr/lib/ruby/gems/1.8/gems/RubyInline-3.6.2/inline.rb:257:in so_name’
/usr/lib/ruby/gems/1.8/gems/RubyInline-3.6.2/inline.rb:291:in
load_cache' /usr/lib/ruby/gems/1.8/gems/RubyInline-3.6.2/inline.rb:627:in inline’
/usr/lib/ruby/gems/1.8/gems/image_science-1.1.1/lib/image_science.rb:84
/usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:32:in
gem_original_require' /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:32:in require’
#{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:496:in
require' #{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:343:in new_constants_in’
#{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:496:in
require' #{RAILS_ROOT}/vendor/plugins/attachment_fu/lib/technoweenie/attachment_fu/processors/image_science_processor.rb:1 /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in gem_original_require’
/usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in
require' #{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:496:in require’
#{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:343:in
new_constants_in' #{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:496:in require’
#{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:105:in
require_or_load' #{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:249:in load_missing_constant’
#{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:453:in
const_missing' #{RAILS_ROOT}/vendor/plugins/attachment_fu/lib/technoweenie/attachment_fu.rb:82:in const_get’
#{RAILS_ROOT}/vendor/plugins/attachment_fu/lib/technoweenie/attachment_fu.rb:82:in
has_attachment' #{RAILS_ROOT}/app/models/photo_image.rb:8 /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in gem_original_require’
/usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in
require' #{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:496:in require’
#{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:343:in
new_constants_in' #{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:496:in require’
#{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:105:in
require_or_load' #{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:249:in load_missing_constant’
#{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:453:in
const_missing' #{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:465:in const_missing’
#{RAILS_ROOT}/vendor/rails/activerecord/lib/active_record/base.rb:1355:in
compute_type' #{RAILS_ROOT}/vendor/rails/activerecord/lib/active_record/reflection.rb:125:in send’
#{RAILS_ROOT}/vendor/rails/activerecord/lib/active_record/reflection.rb:125:in
klass' #{RAILS_ROOT}/vendor/rails/activerecord/lib/active_record/reflection.rb:129:in table_name’
#{RAILS_ROOT}/vendor/rails/activerecord/lib/active_record/associations/has_one_association.rb:74:in
construct_sql' #{RAILS_ROOT}/vendor/rails/activerecord/lib/active_record/associations/has_one_association.rb:6:in initialize’
#{RAILS_ROOT}/vendor/rails/activerecord/lib/active_record/associations.rb:882:in
new' #{RAILS_ROOT}/vendor/rails/activerecord/lib/active_record/associations.rb:882:in photo_image’
#{RAILS_ROOT}/app/models/photo.rb:45:in medium_image_url' #{RAILS_ROOT}/app/views/shared/_main_photo.rhtml:1:in _run_rhtml_47app47views47shared47_main_photo46rhtml’

Then if the same instance is hit again it returns a slightly different
error:

A ActionView::TemplateError occurred in modtypes#show:

Expected
/home/frankg/sites/carstuff/current/config/…/app/models/photo_image.rb
to define PhotoImage
On line #1 of app/views/shared/_main_photo.rhtml

1: <% if main_photo.medium_image_url %>
2:


3: <%= link_to image_tag(main_photo.medium_image_url, {:alt =>
main_photo.title, :title => main_photo.caption}), photo_path(main_photo)
%>
4:
<%= link_to(
truncate(main_photo.caption,80, “… see more”), photo_path(main_photo))
if main_photo.caption %>

#{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:250:in
load_missing_constant' #{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:453:in const_missing’
#{RAILS_ROOT}/vendor/rails/activesupport/lib/active_support/dependencies.rb:465:in
const_missing' #{RAILS_ROOT}/vendor/rails/activerecord/lib/active_record/base.rb:1355:in compute_type’
#{RAILS_ROOT}/vendor/rails/activerecord/lib/active_record/reflection.rb:125:in
send' #{RAILS_ROOT}/vendor/rails/activerecord/lib/active_record/reflection.rb:125:in klass’
#{RAILS_ROOT}/vendor/rails/activerecord/lib/active_record/reflection.rb:129:in
table_name' #{RAILS_ROOT}/vendor/rails/activerecord/lib/active_record/associations/has_one_association.rb:74:in construct_sql’
#{RAILS_ROOT}/vendor/rails/activerecord/lib/active_record/associations/has_one_association.rb:6:in
initialize' #{RAILS_ROOT}/vendor/rails/activerecord/lib/active_record/associations.rb:882:in new’
#{RAILS_ROOT}/vendor/rails/activerecord/lib/active_record/associations.rb:882:in
photo_image' #{RAILS_ROOT}/app/models/photo.rb:45:in medium_image_url’
#{RAILS_ROOT}/app/views/shared/_main_photo.rhtml:1:in
`_run_rhtml_47app47views47shared47_main_photo46rhtml’

While all this is going on, the other mongrel instances are serving up
the page with no probs. I’m running Ruby 1.8.5 with Rails 1.2.1
(frozen), on a number of Xen instances running Debian etch.

The page above is simply a ‘show’ page. There’s no dynamic resizing of
photos, or anything like that (so not sure why Image Science is being
invoked in the first error). It’s a really very straightforward page
from a controller/view perspective.

If anyone has any ideas, or point me towards a strategy for tracking
down the problem, I’d be very grateful.
Many thanks
Chris

Chris-

On Apr 5, 2007, at 10:37 AM, Chris T wrote:

check process mongrel_8001 with pidfile

/home/frankg/sites/carstuff/current/config/mongrel_cluster.yml --clean

Will keep plugging away at this, but if anyone’s got any bright ideas
would welcome them.

Thanks
Chris

Monit starts your mongrels with a fully clean environemnt. Meaning
none of your normal ENV vars are set. WHat you need to do is put this
in the top of your environment.rb

ENV[‘HOME’] = “/home/foobar”

Then restart monit and all the mongrels. This will fix the problem.

Cheers-

– Ezra Z.
– Lead Rails Evangelist
[email protected]
– Engine Y., Serious Rails Hosting
– (866) 518-YARD (9273)

Ezra Z. wrote:

monit. Then, the particular page fails every time. Here’s the
–clean --only 8001"
environment. It fails the first time (ImageScience fails on the

Cheers-

Ezra
Thanks so much for this. Much appreciated.
Chris