Ruby corrupts after a period of time

I have recently upgraded from Ruby 1.8.7 to 1.9.2p180 on Fedora. After a
period of time, a day perhaps, the installation goes corrupt. It works
perfectly fine initially then everything fails. I am rolling this from
source with typical ./configure/make/make install type process. A few
examples:

#irb
internal:lib/rubygems/custom_require:29:in require': closed stream (IOError) from <internal:lib/rubygems/custom_require>:29:in require’
from /usr/lib/ruby/1.9.1/irb.rb:13:in <top (required)>' from <internal:lib/rubygems/custom_require>:29:in require’
from internal:lib/rubygems/custom_require:29:in require' from /usr/bin/irb:9:in

gem list

internal:lib/rubygems/custom_require:29: [BUG] Segmentation fault
ruby 1.9.2p180 (2011-02-18 revision 30909) [x86_64-linux]

– control frame ----------
c:0005 p:---- s:0018 b:0018 l:000017 d:000017 CFUNC :require
c:0004 p:0013 s:0014 b:0014 l:000013 d:000013 METHOD
internal:lib/rubygems/custom_require:29
c:0003 p:0011 s:0009 b:0009 l:0012e8 d:0001f0 EVAL /usr/bin/gem:8
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:0000 s:0002 b:0002 l:0012e8 d:0012e8 TOP

– Ruby level backtrace information

/usr/bin/gem:8:in <main>' <internal:lib/rubygems/custom_require>:29:in require’
internal:lib/rubygems/custom_require:29:in `require’

– C level backtrace information

/usr/bin/ruby(rb_vm_bugreport+0xa1) [0x529241]
/usr/bin/ruby [0x56ce16]
/usr/bin/ruby(rb_bug+0xb1) [0x56cf91]
/usr/bin/ruby [0x4b6f8f]
/lib64/libpthread.so.0 [0x34b680ed30]
/lib64/libc.so.6(memcpy+0x286) [0x34b5883a56]
/usr/bin/ruby(rb_iseq_compile_node+0xc1) [0x5089d1]
/usr/bin/ruby(rb_iseq_new_with_opt+0xa6) [0x510336]
/usr/bin/ruby [0x4ff1a1]
/usr/bin/ruby [0x501d0a]
/usr/bin/ruby [0x500cba]
/usr/bin/ruby(rb_iseq_compile_node+0x991) [0x5092a1]
/usr/bin/ruby(rb_iseq_new_with_opt+0xa6) [0x510336]
/usr/bin/ruby [0x4ff1a1]
/usr/bin/ruby [0x5021f7]
/usr/bin/ruby [0x500cba]
/usr/bin/ruby(rb_iseq_compile_node+0x780) [0x509090]
/usr/bin/ruby [0x50fd5a]
/usr/bin/ruby(rb_iseq_new_top+0x2c) [0x50fffc]
/usr/bin/ruby [0x570f54]
/usr/bin/ruby(rb_require_safe+0x44f) [0x57228f]
/usr/bin/ruby [0x5270ff]
/usr/bin/ruby [0x519439]
/usr/bin/ruby [0x51e698]
/usr/bin/ruby(rb_iseq_eval_main+0x290) [0x51ea20]
/usr/bin/ruby [0x417cf3]
/usr/bin/ruby(ruby_run_node+0x36) [0x4199e6]
/usr/bin/ruby(main+0x4d) [0x416f2d]
/lib64/libc.so.6(__libc_start_main+0xfa) [0x34b581e32a]
/usr/bin/ruby [0x416e19]

[NOTE]
You may have encountered a bug in the Ruby interpreter or extension
libraries.
Bug reports are welcome.
For details: http://www.ruby-lang.org/bugreport.html

Aborted

What does one do to even troubleshoot this let alone fix it?

On Tue, Mar 22, 2011 at 2:26 PM, Chip B. [email protected]
wrote:

I have recently upgraded from Ruby 1.8.7 to 1.9.2p180 on Fedora. After a
period of time, a day perhaps, the installation goes corrupt. It works
perfectly fine initially then everything fails.

What does one do to even troubleshoot this let alone fix it?

On Mar 22, 2011, at 11:26 , Chip B. wrote:

I have recently upgraded from Ruby 1.8.7 to 1.9.2p180 on Fedora. After a
period of time, a day perhaps, the installation goes corrupt. It works
perfectly fine initially then everything fails. I am rolling this from
source with typical ./configure/make/make install type process. A few
examples:

What C extensions might you have loaded?

run it in gdb to get the backtrace?

Ryan D. wrote in post #988782:

What C extensions might you have loaded?

Forgive my ignorance as I am a syadmin and don’t delve into those parts.
How can I find this information? I am certain this is not a hardware
issue as this same problem has arisen on every box I have ever tried
this on… all systems being either Fedora or CentOS.

unknown wrote in post #989053:

On Thu, Mar 24, 2011 at 1:58 PM, Chip B. [email protected]
wrote:

Ryan D. wrote in post #988782:

What C extensions might you have loaded?

Forgive my ignorance as I am a syadmin and don’t delve into those parts.
How can I find this information out? I am certain this is not a hardware
issue as this same problem has arisen on every box I have ever tried
this on.

Can you provide more details of your process?

  • Install Fedora x on y (e.g. 13 on x86_64)
  • Install Ruby 1.8.7-? via yum or source
    • if source, what configure flags/prefix/etc
  • Download and extract 1.9.2-?
    • ./configure
      • any flags here? are you overwriting 1.8.7 in-place or
        installing into a prefix

Install Fedora/CentOS. It has been a mixed bag thus far. Mostly x86_64
systems. CentOS 5.3-5.5 or Fedora 8-13.

Ruby 1.8.7 is installed via source.
./configure --prefix=/usr
make
make install

Gems then installed via Ruby setup.rb. I have always just grabbed the
latest/greatest gem version. 1.3.9, 1.6.2, etc

Use gems to install passenger, rails, etc.

6 months later developer says “I gotta have 1.9.2!”

Ok, now for 1.9.2 I follow the same procedures.
./configure --prefix=/usr
make
make install

And being that gems is included in 1.9x, I skip the gem install. Then
use gem to install rails etc.

Everything works just peachy… gems install, apps run… then
mysteriously a day or so later everything curls up and dies like noted
above.

On Fri, 25 Mar 2011, Chip B. wrote:

above.
You should check to see if you have the auto updates using yum
configured.

– Matt
It’s not what I know that counts.
It’s what I can remember in time to use.

On Thu, Mar 24, 2011 at 1:58 PM, Chip B. [email protected]
wrote:

Ryan D. wrote in post #988782:

What C extensions might you have loaded?

Forgive my ignorance as I am a syadmin and don’t delve into those parts.
How can I find this information out? I am certain this is not a hardware
issue as this same problem has arisen on every box I have ever tried
this on.

Can you provide more details of your process?

  • Install Fedora x on y (e.g. 13 on x86_64)
  • Install Ruby 1.8.7-? via yum or source
    • if source, what configure flags/prefix/etc
  • Download and extract 1.9.2-?
    • ./configure
      • any flags here? are you overwriting 1.8.7 in-place or
        installing into a prefix

On Thu, Mar 24, 2011 at 3:08 PM, Chip B. [email protected]
wrote:

6 months later developer says “I gotta have 1.9.2!”

Have you looked into something like RVM to help manage multiple
versions of Ruby installed in independent prefixes?

above.
Thanks. Do you have any hosts running 1.9 that was not installed over
a 1.8 install? (I assume your hosts that only ever had 1.8 installed
never behaved like this.)

Matt L. wrote in post #989062:

On Fri, 25 Mar 2011, Chip B. wrote:

above.
You should check to see if you have the auto updates using yum
configured.

– Matt
It’s not what I know that counts.
It’s what I can remember in time to use.

Yum is absolutely off. I can’t imagine if it were on. Everything would
be blowing out everywhere as PHP, Python, Perl, etc are all compiled
from source and yum would butcher all of them. The OS installs are
always bare bones. About the only thing installed via yum is gcc, MySQL,
Apache and any required devel packages to compile things.
(openssl-devel, curl-devel, bzip2-devel, etc) Even Apche modules are
compiled (PHP, mod_python, mod_ruby, etc)

unknown wrote in post #989064:

On Thu, Mar 24, 2011 at 3:08 PM, Chip B. [email protected]
wrote:

6 months later developer says “I gotta have 1.9.2!”

Have you looked into something like RVM to help manage multiple
versions of Ruby installed in independent prefixes?

above.
Thanks. Do you have any hosts running 1.9 that was not installed over
a 1.8 install? (I assume your hosts that only ever had 1.8 installed
never behaved like this.)

Correct. 1.8.x installs have been stable for a long while starting with
1.8.5 several years back. And these are all indeed upgrades from 1.8.x.

RVM has been discussed but the thought was get 1.9.x working first as a
proof of concept, but I suppose RVM could work the opposite way and
insulate versions. However with Passenger in the mix, they are going to
be using a consistent version of Ruby in any event (or so I understand)
Regardless, one would think you could get a global
install of 1.9.2 working as Fedora and CentOS aren’t exactly back water
odd ball distros.

On Mar 24, 2011, at 12:08 , Chip B. wrote:

unknown wrote in post #989053:

On Thu, Mar 24, 2011 at 1:58 PM, Chip B. [email protected]
wrote:

Ryan D. wrote in post #988782:

What C extensions might you have loaded?

Forgive my ignorance as I am a syadmin and don’t delve into those parts.
How can I find this information out? I am certain this is not a hardware
issue as this same problem has arisen on every box I have ever tried
this on.

kk. no problem. Basically, are there any .so files in any of the lib
dirs of the gems you’ve installed?

What’s this say?

% gem env path

then:

% ls $(gem env path)/gems//lib/**/.so

In my case:

% ls $(gem env path)/gems//lib/**/.bundle
/Library/Ruby/Gems/1.8/gems/nokogiri-1.4.4/lib/nokogiri/nokogiri.bundle
/Library/Ruby/Gems/1.8/gems/sqlite3-1.3.3/lib/sqlite3/sqlite3_native.bundle

use gem to install rails etc.

Everything works just peachy… gems install, apps run… then
mysteriously a day or so later everything curls up and dies like noted
above.

% env | egrep “GEM|RUBY”

In my case:

10006 % env | egrep “GEM|RUBY”
GEM_PATH=/Library/Ruby/Gems/1.8

which can cause hell on wheels when I run ruby 1.9

On Thu, Mar 24, 2011 at 3:44 PM, Chip B. [email protected]
wrote:

install of 1.9.2 working as Fedora and CentOS aren’t exactly back water
odd ball distros.

I have run 1.9.x on CentOS (PREFIX=/usr/local) with no issues, but I
don’t know why you later have trouble installing it over 1.8.7 in
/usr.

Some more news. I built a clean Fedora 13 x86_64 box last night to test
1.9.2 in a non-upgrade environment. Exact same results. By morning ruby
was spaghetti.

Ok, to answer Ryan’s questions (Thank you very much for taking the time
on this by the way). Of course those commands fail when in the corrupt
state.

‘gem env path’ says
/root/.gem/ruby/1.9.1:/usr/lib/ruby/gems/1.9.1

Instead of ls I used find.

#find /usr/lib/ruby/gems/1.9.1 -name *.so

Returned nothing and the other path in root doesn’t exist.

#env | egrep “GEM|RUBY”

Also nothing

On Fri, Mar 25, 2011 at 10:43 AM, Chip B.
[email protected] wrote:

Some more news. I built a clean Fedora 13 x86_64 box last night to test
1.9.2 in a non-upgrade environment. Exact same results. By morning ruby
was spaghetti.

Thanks for testing that. Did you also test installing to a different
PREFIX?

unknown wrote in post #989307:

On Fri, Mar 25, 2011 at 10:43 AM, Chip B.
[email protected] wrote:

Some more news. I built a clean Fedora 13 x86_64 box last night to test
1.9.2 in a non-upgrade environment. Exact same results. By morning ruby
was spaghetti.

Thanks for testing that. Did you also test installing to a different
PREFIX?

Did you ever resolve this issue? I am running into the same problem on
RHEL4. I had everything running great yesterday, and today I can’t run
anything. Was thinking it was something I had added to the
$LD_LIBRARY_PATH yesterday that isn’t automatically put there upon login
but I guess that’s not the case. It is segment faulting upon any type of
require…

I am 99% sure you are running into problems with prelink. On any centos
or
Fedora OS where we run custom compiled rubies (1.9.2-p180 currently),
prelink runs once a day and corrupts the executable by doing something
to
.so’s it uses or the binary executables themselves. The only way we
have
been able to keep prelink from doing anything is to completely uninstall
the
prelink rpm. I am sure if we spent the time, we could blacklist all
ruby
files from the prelink configuration file, but it wasn’t worth our time
and
effort to do so. prelink is pretty much obsolete (or so I have heard)
on
modern OS’s and we have not had any problems with it uninstalled.

Uninstall prelink, then recompile and reinstall from scratch your ruby
interpreter. I am very sure this will solve your issue.

Any prelink gurus out there want to A) tell me that it is necessary to
have
installed and B) tell me how I tell prelink not to touch my custom
compiled
ruby?

-Doug Seifert

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/22/2011 02:26 PM, Chip B. wrote:

from /usr/lib/ruby/1.9.1/irb.rb:13:in <top (required)>' -- control frame ---------- <internal:lib/rubygems/custom_require>:29:in require’
/usr/bin/ruby(rb_iseq_compile_node+0xc1) [0x5089d1]
/usr/bin/ruby [0x50fd5a]
/lib64/libc.so.6(__libc_start_main+0xfa) [0x34b581e32a]

What does one do to even troubleshoot this let alone fix it?

All right,
That’s weird. I agree. I don’t have a lot of experience using
distribution-provided rubies. So I’ll try to take my point the clearest
I can. First than anything that kind of output doesn’t really help a
lot, all I can see here is that it came to crash on some kind of memory
issue which you can’t determine by the backtrace at all. Attaching gdb
to the process and watching that thing crash or getting a core dump will
bring more light to that.
On the other hand, do you appear to have left-overs from an older
install? Are you sure you’re not somehow getting to load a shared
library (.so file) belonging to an old installation?
If you haven’t solved or went through a workaround by the time you read
this, then let me know I’d be trying to help you through it.


Best Regards,

Adrian P.

Twitter: @blackxored | Skype: blackxored | Google Voice: (559) 744 3201
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iQIcBAEBAgAGBQJNjLBKAAoJEF9FgjLdFu0kJZ8P/1c5KOe3Vlm0E6vl86ZjEyl+
plfKuP4WHGw/DEIdQjMfaVaqosMqDudPRph5oi1h4rVgN8CaVDimq7q0oAn9h23L
xoT+h3zlBFV/Hr9UKE41/W3zWnf3jWhy5YospIRCDnkopvlibiHo0RLWl0SWgc0S
N4gwSzVAX5NEmnbAff9dhXReeKv8XOGtQzUYK8yv2PQpZz/vgUJAWoUUSkmTKgmR
H8M/cEJhHsS3Hj+1zesT0XWqkjU4U2Y+dSJoN/GCulERX1sBHR7to2lfu/Cz5G28
WSK/Sj2Xp9nyg/iZ8AvcoV9XJHxxRu8J61Y89CgJEFHmpTZzrttrYQCs0sU+eQYV
6TkpoQ2bm3wwTGQGJIpwY3PgBFW0cBWCOq0dbEKwYYWB4qca9Arb1zFlYpYRlAVT
n4h9IoRZmHC24SuU1Ej4LB0UkGCuwhJx/Re289QGWA1ILicE7fzRHv9A9/7rKpEK
LVzo/axzhDr047u1RmfhjbG4f6M7YedWOwtMO6TcdBONLhxvd9orvDtuLtwH8Rcp
DkPXDQQl/yPGg7K8Vewj5p+JULc4Jr3hQZGaTsfknrHrrrq+Ly0MM0ZGeisoGzN2
JD2bSTNi5zNPYJraedUEkExCUSfKvPerwIcv1NNKZcOGRPrNASpph9voElYnJAz8
dghNcfl8PGe9Nhf8qMnJ
=OLxU
-----END PGP SIGNATURE-----

On Wed, Jul 27, 2011 at 11:06 PM, Douglas S. [email protected]
wrote:

A) tell me that [prelink] is necessary to have installed

From the latest NSA Linux guide[1]:

  • Prelink can be safely disabled by setting the following setting
    ‘PRELINKING=no’ in /etc/sysconfig/prelink:
  • and run /usr/sbin/prelink -ua to restore binaries to a non-prelinked
    state

And recommends doing so:

  • [prelink] is undesirable because it provides a stable address for an
    attacker to use during an exploitation attempt.

[1]
http://www.nsa.gov/ia/guidance/security_configuration_guides/operating_systems.shtml#linux2