Erratic behavior with ferret 0.95 and acts_as_ferret

Okay previous post related to me trying to fix this problem with an
upgrade to ferret 0.10.0 but acts as ferret is obviously not compatible
with this new version of ferret.
My app is about to go into production and Ferret acting wierdly is a
major problem. It seems to crash randomly. Below is a list of the
Lighttpd error logs and a selection of code that sometimes produces
errors and sometimes doesn’t even with the same input.

Would dearly love for ferret to play nice and consistently.

On the development log of rails i also get stuff like

*** glibc detected *** corrupted double-linked list: 0x08aa20e0 ***

scary eh?

Is this normal? Do people manage to get ferret working nicely without
any scary errors and crashes? What setups do they use? What am i doing
wrong? I can supply and other logs and code etc. I’m just not sure what
else is of use? Is ferret 0.10. more stable? When will acts_as_ferret be
compatible? Should i be running a different web server or compilling
ferret differently?
Thank you in advance for any and all advice.
Regards
Caspar

I’m using
ruby 1.8.4
rails 1.1.6
ferret 0.95.0
acts_as_ferret
ubuntu dapper

def VoObject.browse_results(section,sale_category,page)

bq = Ferret::Search::BooleanQuery.new
	search_input  = '*'
	sort_fields = []
	sort_fields << Ferret::Search::SortField.new("ferret_date_registered", 

:reverse => :true)

	if section
		if !section.empty?
			qp = Ferret::QueryParser.new("section")
		query = qp.parse("\"#{section}\"")
			bq.add_query(query, Ferret::Search::BooleanClause::Occur::MUST)
			filter_on =true
		end
	end

	if sale_category
		if !sale_category.empty?
			qp = Ferret::QueryParser.new("sale_category")
		query = qp.parse("\"#{sale_category}\"")
			bq.add_query(query, Ferret::Search::BooleanClause::Occur::MUST)
			filter_on =true
		end
	end

if page == 1
		page = 0
	else
		page = (page-1)*20
end

	if filter_on
 	filter = Ferret::Search::QueryFilter.new(bq)
	 	results2 = VoObject.find_by_contents(search_input,:filter => filter, 

:sort => sort_fields,:num_docs => 20000)
results = VoObject.find_by_contents(search_input,:filter => filter,
:sort =>sort_fields,:first_doc=>page,:num_docs => 20)
else
results2 = VoObject.find_by_contents(search_input,:sort
=>sort_fields,:num_docs => 20000)
results = VoObject.find_by_contents(search_input,:sort =>
sort_fields,:first_doc=>page,:num_docs => 20)
end
[results,results2]
end

06-08-23 01:39:22: (mod_fastcgi.c.2536) FastCGI-stderr:
/home/ghost/user/V_O_2/public/…/config/…/app/views/search/_browse_results.rhtml:70:
warning: parenthesize argument(s) for future version
/home/ghost/user/V_O_2/public/…/config/…/app/views/search/_browse_results.rhtml:118:
warning: parenthesize argument(s) for future version

2006-08-23 01:39:22: (mod_fastcgi.c.2536) FastCGI-stderr:
2006-08-23 01:40:26: (mod_fastcgi.c.2430) unexpected end-of-file
(perhaps the fastcgi process died): pid: 25456 socket:
unix:/home/ghost/user/V_O_2/tmp/sockets/fcgi.socket-0
2006-08-23 01:40:26: (mod_fastcgi.c.3230) response already sent out, but
backend returned error on socket:
unix:/home/ghost/user/V_O_2/tmp/sockets/fcgi.socket-0 for /dispatch.fcgi
, terminating connection
2006-08-23 01:40:38: (mod_fastcgi.c.1739) connect failed: Connection
refused on unix:/home/ghost/user/V_O_2/tmp/sockets/fcgi.socket-0
2006-08-23 01:40:38: (mod_fastcgi.c.2851) backend died, we disable it
for a 5 seconds and send the request to another backend instead:
reconnects: 0 load: 1
2006-08-23 01:40:38: (mod_fastcgi.c.2624) child signaled: 6
2006-08-23 01:40:40: (mod_fastcgi.c.2536) FastCGI-stderr:
/home/ghost/user/V_O_2/public/…/config/…/app/views/search/_browse_results.rhtml:70:
warning: parenthesize argument(s) for future version
/home/ghost/user/V_O_2/public/…/config/…/app/views/search/_browse_results.rhtml:118:
warning: parenthesize argument(s) for future version

2006-08-23 01:40:40: (mod_fastcgi.c.2536) FastCGI-stderr:

On Sat, Aug 26, 2006 at 05:28:25PM +0200, Clare wrote:

I am getting this issue also… Does anyone know what this is? When will
Acts as Ferret be available for v10?

the current svn trunk of the plugin is compatible to acts_as_ferret,
besides more_like_this not working and queries not being ANDed by
default. You should give this a try if you experience these problems
with 0.9.5

Jens

errors and sometimes doesn’t even with the same input.
Posted via http://www.ruby-forum.com/.


Ferret-talk mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/ferret-talk


webit! Gesellschaft für neue Medien mbH www.webit.de
Dipl.-Wirtschaftsingenieur Jens Krämer [email protected]
Schnorrstraße 76 Tel +49 351 46766 0
D-01069 Dresden Fax +49 351 46766 66

I am getting this issue also… Does anyone know what this is? When will
Acts as Ferret be available for v10?

Thanks for your help in desperation!

Caspar wrote:

Okay previous post related to me trying to fix this problem with an
upgrade to ferret 0.10.0 but acts as ferret is obviously not compatible
with this new version of ferret.
My app is about to go into production and Ferret acting wierdly is a
major problem. It seems to crash randomly. Below is a list of the
Lighttpd error logs and a selection of code that sometimes produces
errors and sometimes doesn’t even with the same input.

Would dearly love for ferret to play nice and consistently.

On the development log of rails i also get stuff like

*** glibc detected *** corrupted double-linked list: 0x08aa20e0 ***

Hello,

I’m using acts_as_ferret with a single model class in my rails app. I
have a cron job that batch-loads several hundred objects of this
class at a time. While the batch job is running, attempting to run a
search from the web app results in a lock error of one type or
another. So far I’ve seen the two types of errors below. I see in the
archives several messages suggesting that using :auto_flush => true
should resolve locking problems. Looking at the acts_as_ferret code
suggests to me that :auto_flush => true is being used, but I’m not
100% sure. Can someone shed some light on this for me?

I’m using ferret 0.10.1 and acts_as_ferret from svn trunk rev 84.

Thanks,

-Dave

/usr/local/lib/ruby/gems/1.8/gems/ferret-0.10.1/lib/ferret/index.rb:
409:in `close’: Lock Error occured at <except.c>:79 in xraise
(Ferret::Store::lock::LockError)
Error occured in index.c:5237 - iw_flush_ram_segment
Couldn’t obtain commit lock to write segments file

     from /usr/local/lib/ruby/gems/1.8/gems/ferret-0.10.1/lib/

ferret/index.rb:409:in flush' from /usr/local/lib/ruby/1.8/monitor.rb:229:insynchronize’
from /usr/local/lib/ruby/gems/1.8/gems/ferret-0.10.1/lib/
ferret/index.rb:406:in flush' from /usr/local/lib/ruby/gems/1.8/gems/ferret-0.10.1/lib/ ferret/index.rb:214:in<<’
from /usr/local/lib/ruby/1.8/monitor.rb:229:in synchronize' from /usr/local/lib/ruby/gems/1.8/gems/ferret-0.10.1/lib/ ferret/index.rb:186:in<<’
from /home/digest/digest.sialia.com/helper_scripts/…/
config/…/vendor/plugins/acts_as_ferret/lib/acts_as_ferret.rb:567:in
ferret_create' from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.14.4/ lib/active_record/callbacks.rb:344:incallback’
… 11 levels…
from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.14.4/
lib/active_record/transactions.rb:126:in `save’

/usr/local/lib/ruby/gems/1.8/gems/ferret-0.10.1/lib/ferret/index.rb:
98:in `initialize’: Lock Error occured at <except.c>:103 in
xpop_context (Ferret::Store::lock::LockError)
Error occured in index.c:5371 - iw_open
Couldn’t obtain write lock when opening IndexWriter

     from /usr/local/lib/ruby/gems/1.8/gems/ferret-0.10.1/lib/

ferret/index.rb:98:in initialize' from /usr/local/lib/ruby/1.8/monitor.rb:229:insynchronize’
from /usr/local/lib/ruby/gems/1.8/gems/ferret-0.10.1/lib/
ferret/index.rb:96:in initialize' from /home/digest/digest.sialia.com/helper_scripts/../ config/../vendor/plugins/acts_as_ferret/lib/acts_as_ferret.rb:320:increate_index_instance’
from /home/digest/digest.sialia.com/helper_scripts/…/
config/…/vendor/plugins/acts_as_ferret/lib/acts_as_ferret.rb:312:in
ferret_index' from /home/digest/digest.sialia.com/helper_scripts/../ config/../vendor/plugins/acts_as_ferret/lib/acts_as_ferret.rb:567:inferret_create’
from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.14.4/
lib/active_record/callbacks.rb:344:in callback' from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.14.4/ lib/active_record/callbacks.rb:341:incallback’
… 10 levels…
from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.14.4/
lib/active_record/transactions.rb:126:in `save’

On Sat, Aug 26, 2006 at 09:08:46AM -0700, David R. wrote:

100% sure. Can someone shed some light on this for me?
Yes, aaf uses auto_flush. In theory, searching the index should not
result in any write access to the index (unless the index doesn’t exist
yet, in that case aaf tries to rebuild it before running the search)

Where do you see these errors ? The importer script or the web app ?

The second stack trace is strange, as the simple initialization of the
Index instance results in a locking error.

Jens

Error occured in index.c:5237 - iw_flush_ram_segment
from /usr/local/lib/ruby/gems/1.8/gems/ferret-0.10.1/lib/

ferret/index.rb:96:in initialize' lib/active_record/callbacks.rb:344:in callback’
from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.14.4/
lib/active_record/callbacks.rb:341:in callback' ... 10 levels... from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.14.4/ lib/active_record/transactions.rb:126:in save’


Ferret-talk mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/ferret-talk


webit! Gesellschaft für neue Medien mbH www.webit.de
Dipl.-Wirtschaftsingenieur Jens Krämer [email protected]
Schnorrstraße 76 Tel +49 351 46766 0
D-01069 Dresden Fax +49 351 46766 66

Jens,

Thanks for your response. I see errors in either the importer (batch)
script or in the web app. The traces I happened to post previously
were both from the batch script. Here are the relevant lines from a
typical error found in the rails app’s error log.

-Dave

ActionView::TemplateError (Lock Error occured at <except.c>:79 in xraise
Error occured in index.c:3089 - ir_is_latest
Error trying to commit the index. Commit lock already obtained

) on line #2 of app/views/messages/_message_tables_simple.rhtml:
1: <% if search_term %>
2: <% messages = Message.find_by_search_term(params[:search_term],
session[:timezone_offset]) %>
3: <% elsif current_filter(“filter_name”) %>
4: <% messages = Message.find_by_current_filter(session
[“current_filter”], nil, nil, session[:timezone_offset], params
[:days_ago], params[:alldays]) %>
5: <% elsif params[“listid”] %>

 /usr/local/lib/ruby/gems/1.8/gems/ferret-0.10.1/lib/ferret/

index.rb:537:in latest?' /usr/local/lib/ruby/gems/1.8/gems/ferret-0.10.1/lib/ferret/ index.rb:537:inensure_reader_open’
/usr/local/lib/ruby/gems/1.8/gems/ferret-0.10.1/lib/ferret/
index.rb:264:in []' /usr/local/lib/ruby/1.8/monitor.rb:229:insynchronize’
/usr/local/lib/ruby/gems/1.8/gems/ferret-0.10.1/lib/ferret/
index.rb:263:in []' #{RAILS_ROOT}/vendor/plugins/acts_as_ferret/lib/ acts_as_ferret.rb:486:infind_id_by_contents’
/usr/local/lib/ruby/gems/1.8/gems/ferret-0.10.1/lib/ferret/
index.rb:252:in search_each' /usr/local/lib/ruby/gems/1.8/gems/ferret-0.10.1/lib/ferret/ index.rb:251:insearch_each’
/usr/local/lib/ruby/1.8/monitor.rb:229:in synchronize' /usr/local/lib/ruby/gems/1.8/gems/ferret-0.10.1/lib/ferret/ index.rb:247:insearch_each’
#{RAILS_ROOT}/vendor/plugins/acts_as_ferret/lib/
acts_as_ferret.rb:484:in find_id_by_contents' #{RAILS_ROOT}/vendor/plugins/acts_as_ferret/lib/ acts_as_ferret.rb:344:infind_by_contents’
#{RAILS_ROOT}/app/models/message.rb:87:in `find_by_search_term’

On Mon, Aug 28, 2006 at 01:20:29AM -0700, David R. wrote:

Error occured in index.c:3089 - ir_is_latest
Error trying to commit the index. Commit lock already obtained

ok, it seems that the check to see if the current IndexReader is based
on
the latest version of the Ferret index wants to obtain a write lock and
fails, since the batch indexing job already has locked it.

This check is done by the Ferret::Index::Index class before every
search. Maybe Dave could shed some light on how to handle these errors ?
Imho Ferret should retry some more times to obtain a lock before failing
that way.

As a short term fix, one could catch the lock error in
Ferret::Index::Index#ensure_reader_open and continue as if
index_reader.latest?
had returned true. That would prevent the search from failure due to
a locked index.

In the opposite direction, i.e. if latest? grabbed the lock and the
indexing failed, a simple retry (calling record.save another time) could
solve the problem, too.

Jens


webit! Gesellschaft für neue Medien mbH www.webit.de
Dipl.-Wirtschaftsingenieur Jens Krämer [email protected]
Schnorrstraße 76 Tel +49 351 46766 0
D-01069 Dresden Fax +49 351 46766 66

On 8/29/06, David R. [email protected] wrote:

Jens,

Thanks again, that makes sense. I’ll implement some workarounds in my
code until we hear from Dave on the matter.

Hey Dave,

I’ve added a small sleep to the lock-obtain loop which should fix this
problem. There is also a single 2 second sleep and retry in the Index
class (you can change this with the :lock_retry_time parameter).
Please let me know whether or not this works for you. It’s already in
the subversion repository and it’ll be out in version 0.10.2.

Cheers,
Dave

Jens,

Thanks again, that makes sense. I’ll implement some workarounds in my
code until we hear from Dave on the matter.

-Dave