Ruby 1.9.0/1.8.7/1.8.6/1.8.5 new releases (Security Fix)

Ruby
61

Cheri Anaclerio wrote:

Could somebody please explain how to apply the Smartleaf Stanislav and
Hongli’s patches? I too am experiencing glibc segmentation problems and
would like to fix my local copy of rails 1.8.6.

Thanks!

Cheri

  1. Download the 1.8.6p230 source here:
    Arbitrary code execution vulnerabilities

  2. Download the patch here (and and stick it in the ruby-1.8.6-p230
    source directory): http://dev.smartleaf.com/misc/p230_fixit_patch.txt

  3. In your terminal:
    $ cd ruby-1.8.6-p230
    $ patch < ./p230_fixit_patch.txt

  4. Then compile per usual. Ex:
    $ ./configure
    $ make
    $ sudo make install

62

Jason C. wrote:

  1. Then compile per usual. Ex:
    $ ./configure
    $ make
    $ sudo make install

Jason, thanks for the explanation. However, I’d argue against that last
step. Doing step 4 like that will put the files directly into the user’s
/usr/local hierarchy, which will make removing or upgrading that versino
difficult.

I’d strongly recommend that you use GNU Stow
[http://www.gnu.org/software/stow/manual.html] and change the steps to:

$ ./configure --prefix=/usr/local/stow/ruby-1.8.6p230smartleaf
$ make
$ sudo mkdir -p /usr/local/stow
$ sudo make install
$ sudo stow -d /usr/local/stow ruby-1.8.6p230smartleaf

With those commands, you’ll end up with a symlink at /usr/local/bin/ruby
that points to the installed version deeper within the /usr/local/stow
hierarchy.

Using stow will let you easily install, uninstall and switch between
versions of a compiled applications.

-igal

PS: Another good alternative is to use Ruby Enterprise Edition
[http://rubyenterpriseedition.com/] which already includes the patches
and installs itself by default into directory within /opt.

63

Jason, thanks for the explanation. However, I’d argue against that last
step. Doing step 4 like that will put the files directly into the user’s
/usr/local hierarchy, which will make removing or upgrading that versino
difficult.

I’d strongly recommend that you use GNU Stow

Igal,

Good call with that. I just wanted to demonstrate to Cheri that at that
point in the steps, you can do anything you’d normally do with a
functional Ruby source.

Thanks for the additional info!

-Jason

64

Cheri Anaclerio wrote:

The problem is that the copy of RubyGems installed via Yum
(/usr/bin/gem) is using your old interpreter (/usr/bin/ruby) and knows
nothing about your new manually-compiled interpreter
(/usr/local/bin/ruby).

The solution is to install RubyGems for the new interpreter, e.g.:

wget http://rubyforge.org/frs/download.php/38646/rubygems-1.2.0.tgz
tar xvfz rubygems-1.2.0.tgz
cd rubygems-1.2.0
sudo /usr/local/bin/ruby setup.rb --no-ri --no-rdoc
sudo /usr/local/bin/gem install rake --no-ri --no-rdoc

Because the above changes the stowed directory structure, you’ll want to
restow it so that the new files are linked into /usr/local, e.g.:

pushd /usr/local/stow; sudo stow --restow ruby-1.8.6p230smartleaf;

popd

There’s a way to make multiple Ruby interpreters share a copy of the
installed gems by setting the GEM_HOME environmental variable. However,
this seems like a bad idea and you’re best off just installing new
copies of the gems you need.

-igal

65

Thanks, Igal! So when I want to go back to using /usr/bin/ruby and it’s
associated gems, would I just delete the stow directory

$ sudo stow -d /usr/local/stow ruby-1.8.6p230smartleaf

I am contemplating the solution of just upgrading to Ruby 1.8.7 and
Rails 2.1 since my project is still in development. However, from
reading the forums it looks like there are many plugins that won’t work
with Rails 2.1 and other issues that still need to be resolved. Any
thoughts on the upgrade route?

  • Cheri

Igal K. wrote:

Cheri Anaclerio wrote:

The problem is that the copy of RubyGems installed via Yum
(/usr/bin/gem) is using your old interpreter (/usr/bin/ruby) and knows
nothing about your new manually-compiled interpreter
(/usr/local/bin/ruby).

The solution is to install RubyGems for the new interpreter, e.g.:

wget http://rubyforge.org/frs/download.php/38646/rubygems-1.2.0.tgz
tar xvfz rubygems-1.2.0.tgz
cd rubygems-1.2.0
sudo /usr/local/bin/ruby setup.rb --no-ri --no-rdoc
sudo /usr/local/bin/gem install rake --no-ri --no-rdoc

Because the above changes the stowed directory structure, you’ll want to
restow it so that the new files are linked into /usr/local, e.g.:

pushd /usr/local/stow; sudo stow --restow ruby-1.8.6p230smartleaf;

popd

There’s a way to make multiple Ruby interpreters share a copy of the
installed gems by setting the GEM_HOME environmental variable. However,
this seems like a bad idea and you’re best off just installing new
copies of the gems you need.

-igal

66

Cheri Anaclerio wrote:

Thanks, Igal! So when I want to go back to using /usr/bin/ruby and it’s
associated gems, would I just delete the stow directory

Not quite. When you want to get rid of a version, first tell stow to
remove the symlinks:
sudo stow -d /usr/local/stow --delete ruby-1.8.6p230smartleaf

And then you can “rm -rf” the “/usr/local/stow/ruby-1.8.6p230smartleaf”
directory. You should keep the “/usr/local/stow directory” though, in
case you need to stow away something else.

However, please note that although the p230smartleaf fixes the
segmentation faults, the p230 version it relies on seems to change the
Ruby API in a number of ways. I’m working with folks in the ruby-core
mailing list to try to make sense of the situation and determine how
significant the breakage is.

For the moment, it may be safest to use one of the backport patches
developed against p111 or p114, or the Ruby Enterprise Edition which
includes these patches. Also note that many OS vendors have shipped new
packages with some of these fixes, including FreeBSD, Ubuntu, and
Fedora. However, it’s unclear how good these unofficial patches are, and
it’d be best if the Ruby developers can ship an officially blessed fix
for this in the near future.

I am contemplating the solution of just upgrading to Ruby 1.8.7 and
Rails 2.1 since my project is still in development. However, from
reading the forums it looks like there are many plugins that won’t work
with Rails 2.1 and other issues that still need to be resolved. Any
thoughts on the upgrade route?
If you don’t have to promise availability for your app, it may be worth
trying out 1.8.7 and reporting your experiences. However, please
understand that very few people are using that version and that if
earlier versions of Rails don’t run on it, this may indicate deeper
issues, so this will be somewhat adventurous.

I ported a moderate sized Rails app from 2.0.2 to 2.1.0 recently, it
works well and I really liked that it provides built-in solutions for a
number of features we’d implemented on our own and it was nice to
replace those with the much nicer bundled solutions.

-igal

67

Hi,

I run Arch and just upgraded to ruby-1.8.7_p22 but it seems the string
issue is partially fixed.

irb(main):001:0> str = “A”*(2**16); nil
=> nil
irb(main):002:0> while true
irb(main):003:1> str << str
irb(main):004:1> end
(irb):3: [BUG] Segmentation fault
ruby 1.8.7 (2008-06-20 patchlevel 22) [i686-linux]

Aborted

While the following works fine:
irb(main):001:0> str = “A”*(2**16); nil
=> nil
irb(main):002:0> while true
irb(main):003:1> str = str+str
irb(main):004:1> end
ArgumentError: negative string size (or size too big)
from (irb):3:in `+’
from (irb):3
from :0

I have another computer with Ubuntu and ruby 1.8.6 in which both cases
worked fine.

68

Ok, I took the stow route and followed the steps below. However, now
when I start the webrick server, Rubygems 0.9.4.1 can’t be found even
though it is installed. Looks like the directory I’m pointing to
doesn’t know anything about Rubygems. What can I do to fix this?
Thanks! Cheri

(See full trace by running task with --trace)
Rails requires RubyGems >= 0.9.4. Please install RubyGems and try again:
http://rubygems.rubyforge.org
[root@localhost cz]# yum install rubygems
fedora 100% |=========================| 2.1 kB
00:00
updates 100% |=========================| 2.3 kB
00:00
adobe-linux-i386 100% |=========================| 951 B
00:00
Setting up Install Process
Parsing package install arguments
Package rubygems - 0.9.4-1.fc8.noarch is already installed.
Nothing to do
[root@localhost cz]#

I’d strongly recommend that you use GNU Stow
[http://www.gnu.org/software/stow/manual.html] and change the steps to:

$ ./configure --prefix=/usr/local/stow/ruby-1.8.6p230smartleaf
$ make
$ sudo mkdir -p /usr/local/stow
$ sudo make install
$ sudo stow -d /usr/local/stow ruby-1.8.6p230smartleaf

With those commands, you’ll end up with a symlink at /usr/local/bin/ruby
that points to the installed version deeper within the /usr/local/stow
hierarchy.

Using stow will let you easily install, uninstall and switch between
versions of a compiled applications.

-igal

PS: Another good alternative is to use Ruby Enterprise Edition
[http://rubyenterpriseedition.com/] which already includes the patches
and installs itself by default into directory within /opt.

69

Hi Igal,

Are you sure these are the right steps for installing gems for the
Smartleaf patched interpreter? I’ve followed all your steps outlined
below, but when I get to

sudo /usr/local/bin/gem install rake --no-ri --no-rdoc,

the command is not found.

i.e.,
[Cheri@localhost rubygems-1.2.0]$ sudo /usr/local/bin/gem install rake
–no-ri --no-rdoc
sudo: /usr/local/bin/gem: command not found

I don’t think I’m installing RubyGems in the right directory path.

Thanks in advance for your help.

  • Cheri

Igal K. wrote:

Cheri Anaclerio wrote:

The problem is that the copy of RubyGems installed via Yum
(/usr/bin/gem) is using your old interpreter (/usr/bin/ruby) and knows
nothing about your new manually-compiled interpreter
(/usr/local/bin/ruby).

The solution is to install RubyGems for the new interpreter, e.g.:

wget http://rubyforge.org/frs/download.php/38646/rubygems-1.2.0.tgz
tar xvfz rubygems-1.2.0.tgz
cd rubygems-1.2.0
sudo /usr/local/bin/ruby setup.rb --no-ri --no-rdoc
sudo /usr/local/bin/gem install rake --no-ri --no-rdoc

Because the above changes the stowed directory structure, you’ll want to
restow it so that the new files are linked into /usr/local, e.g.:

pushd /usr/local/stow; sudo stow --restow ruby-1.8.6p230smartleaf;

popd

There’s a way to make multiple Ruby interpreters share a copy of the
installed gems by setting the GEM_HOME environmental variable. However,
this seems like a bad idea and you’re best off just installing new
copies of the gems you need.

-igal

70

Igal K. wrote:

As far as I can tell, both of these patches provide a working version of
MRI Ruby. Stanislav and Hongli’s p111 backport relies on them having
correctly cherry-picked the right fixes. Smartleaf’s p230 fix relies on
the rest of that Ruby version to be correct, and I have some concerns
about other lurking bugs if they shipped a version with such an obvious
flaw.

How do we choose between these?

-igal

Our production Rails environment has been having some memory problems
for a while. Once a week or our mongrel daemons swell in memory usage
and it pushes beyond what our box can comfortably handle. We’re hunting
for the root cause, but a simple restart of the mongrel processes clears
it up for another week or so.

We installed the smartleaf patched ruby on our production environment on
Monday. Since that time we had our processes balloon at least a dozen
times in two days. After reverting the patch to ruby it has returned to
normal. I don’t know if our situation is a corner case or not. I’m
just wanting to let folks know who are trying to decide what to do that
we’ve had problems with the smartleaf patch.

71

Doug Alcorn wrote:

just wanting to let folks know who are trying to decide what to do that
we’ve had problems with the smartleaf patch.

Thanks for reporting that, Doug.

Can you or one of your engineers try to figure out how to trigger this
memory ballooning problem? That’d be really valuable to the Ruby
community.

The Smartleaf patch doesn’t introduce any new code, merely reverts some
problematic refactoring, so whatever is making your problem worse may
very likely still be in the latest Ruby code that the maintainers plan
to publish soon, so it’d be best to identify the cause ASAP.

-igal

72

Igal K. wrote:

Can you or one of your engineers try to figure out how to trigger this
memory ballooning problem? That’d be really valuable to the Ruby
community.

Not only to the community, but also my client. I’ve been working on it
on and off for several weeks. I’m not sure I have the right skills atm
to do memory profiling. I’d be interested in some good pointers on how
to memory profile ruby/rails apps

73

Cheri,

I think I see what went wrong. You’ll probably need to restow before the
newly installed gem command becomes visible. Try these:

wget http://rubyforge.org/frs/download.php/38646/rubygems-1.2.0.tgz
tar xvfz rubygems-1.2.0.tgz
cd rubygems-1.2.0
sudo /usr/local/bin/ruby setup.rb --no-ri --no-rdoc
sudo stow -d /usr/local/bin --restow ruby-1.8.6p230smartleaf
sudo /usr/local/bin/gem install rake --no-ri --no-rdoc
sudo stow -d /usr/local/bin --restow ruby-1.8.6p230smartleaf

Basically, whenever new executables, like “gem”, are added within the
stowed directory, you need to restow. However, at least on my system,
once gem is installed, it seems to be doing something clever to symlink
things like rake and such into /usr/local/bin automatically without the
need for a restow.

If these commands don’t work, can you paste the final results of the
“setup.rb” command? When it finishes installing it on my system, it
prints something like:

RubyGems installed the following executables:
   /usr/local/stow/ruby-1.8.6-p230smartleaf/bin/gem

When (re)stowed, the contents of that “bin” directory above are
symlinked into /usr/local/bin, and that’s where /usr/local/bin/gem is
supposed to come from.

-igal

74

Doug Alcorn wrote:

Not only to the community, but also my client. I’ve been working on it
on and off for several weeks. I’m not sure I have the right skills atm
to do memory profiling. I’d be interested in some good pointers on how
to memory profile ruby/rails app

I’ve got potentially bad news according to Robin Ward at
http://weblog.rubyonrails.com/2008/6/21/multiple-ruby-security-vulnerabilities:

I installed the Smartleaf p230 patch above and I can confirm that my
Ruby app is running fine in production without any segfaults. […]
Disregard what I said above about the smartleaf patch: After running
for a while the memory usage of all my ruby processes climbed through
the room. There are definitely memory leaks present.
Because the bug isn’t in the Smartleaf patch, but rather in the Ruby
interpreter, it’s very likely that the problem is still in there.

Can anyone please provide code to reproduce this?

-igal

75

Doug Alcorn wrote:

to memory profile ruby/rails apps

Your findings may be very important. Most people are running versions of
Ruby with patches backported to p111/p114, whereas you’re running a
patch against p230. Relatively few people are using that newer release,
but its code is much closer to what the Ruby maintainers plan to ship in
the near future. Because you’re closer to the edge, you’ve likely hit
something the rest of us haven’t.

As for identifying the problem, unless someone has a better suggestion,
I think you just have to use each part of your application until you
identify an action that triggers this bug. You should clone your
production environment as closely as possible, including the database
and run the app in production mode, and use that for your testing. You
should also keep a written list of all controllers, actions and logical
paths within actions, and then check these off as you’ve validated them
so you can be systematic about checking everything. After each action,
glance at the application’s memory consumption to see if you caught the
problem. Once you know what sequence of user actions will cause it to
balloon, you can start narrowing down the code causing it by commenting
stuff out and such.

A good way to keep an eye on memory consumption is to start up a
terminal and run a command like this in it:
watch -n1 ‘ps aux | egrep “ruby|PID” | egrep -v egrep’

The above scans the output of “ps” each second and displays all the
processes matching the “ruby”. You may want to substitute
“mongrel_rails” or whatever the name is for your server process, and
then just watch the RSS and VSZ columns.

Another possible way to identify the problem may be NewRelic’s RPM
(http://newrelic.com/). Be sure to get the “Production Mode” with the 2
month free trial and enable the feature which uploads reporting data
back to them. Among other useful features, this Rails plugin will help
you quickly identify the actions that are taking a long time to
complete. If lucky, this will help you identify the actions causing
leakage; and if less lucky, you’ll know where to prioritize your tuning
efforts later, which is still useful.

Anyone else have ideas on how to identify the problem?

-igal

76

Igal K. wrote:

atm to do memory profiling. I’d be interested in some good pointers
I think you just have to use each part of your application until you

(http://newrelic.com/). Be sure to get the “Production Mode” with the 2

“watch” is for wimps … just about every Linux system has “top”. :slight_smile:

What’s really nice about “top” is that you can do “top -b > logfile”
and it will do a “top” at the default interval to the log file. Then you
can parse that log file with Ruby and watch your process leak memory.

I’m just about to set up some shell scripts (sorry, all you Rake fans –
my Rakefiles are too ugly so I’m dropping back to bash :slight_smile: ) to build a
profiling Ruby. I might just take the opportunity to learn how to use
“valgrind” while I’m at it.

77

Doug Alcorn wrote:

to memory profile ruby/rails apps
I’ve never used it, but I believe “valgrind” is the tool one uses to
check for memory leaks in C code – like MRI, for example. :slight_smile:

78

WARNING: Do not use Ruby 1.8.6 releases p230 through p238 in production!
There are bugs in the Ruby interpreter that cause it to leak memory.
Please use a Ruby release that’s based on a patched version of the p111
or p114 code until this is resolved.

I’ve put together code which demonstrates this at:
http://pastie.org/226715

-igal

RedMine ticket: http://redmine.ruby-lang.org/issues/show/216
ruby-core thread: http://www.ruby-forum.com/topic/158334#new

79

M. Edward (Ed) Borasky wrote:

I’ve never used it, but I believe “valgrind” is the tool one uses to
check for memory leaks in C code – like MRI, for example. :slight_smile:

In many cases, yes — but I tried it for the 1.8.6-p230 segfault
problem, and regrettably, MRI’s GC confused the hell out of it.

FWIW, I’m the guy who produced the “Smartleaf patch” which eliminates
the immediate segfault problems (though not, apparently, this reported
leak). What this amounted to was identifying the particular patch
(of many that went into p230 as released) that was causing the
trouble,
and reverting that one patch (so, the “smartleaf patch” comes straight
from
a reversed “svn diff”). The strategy I used was a little
time-consuming,
but it has the advantage of not requiring expertise in much of anything;
it was bisection search (i.e., binary search among the 1.8.6 patch
stream).

The idea of a bisection search is basically this: There’s an ordered
series of svn revisions that turned 1.8.6-p111 into 1.8.6-p230 (or
whatever), and among those, there’s going to be a first revision that
exhibits the problem. If we assume that nothing else broke in the
course of the stream of revisions, then we can find that first revision
by binary search: Check out the release midway between the two we’re
trying to evaluate, and see if that release leaks storage the way p238
does. If the memory leak problem is already there, then it must have
been introduced somewhere in the series of revisions between 1.8.6-p111
and [midpoint]. If not, then it was introduced between [midpoint] and
1.8.6-p230. Either way, we’ve now bracketed it to within an interval
half the original size. If we skip to the midpoint of that interval,
we can cut it in half again, and proceed to the one patch that
introduced
the problem. (The git suite of tools can actually automate a lot of
this,
if you happen to have the relevant history in a local git repo — see
“git bisect”.)

In this case, of course, there’s one complication: something else did
break in the middle — in particular, the segfaults that start
happening
at revision 17222. So, if you get to (or past) that point in the
history,
you’d have to revert that particular change to look for the storage
leak.
But if anyone out there can keep track of that complication, and has
time
to evaluate a dozen or so ruby-1.8.6 revisions to see if they have this
problem, that’s enough to identify the particular change which first
introduced the problem. It doesn’t need to be that many, I believe —
but I’m allowing for screwups.

(BTW, It’s conceivable that there’s a subsequent patch which also
introduced
a leak, or depends in some other way on the one that did — in which
case,
you’ll have to root around in the subsequent history to get a more
complete
picture. But the most likely thing is that if you take the patch you
find,
and you revert it against 1.8.6-p238, you’ll find the problem goes
away.)

One last note: this quite likely gives you something that you can apply
to p238 to eliminate the problem. It does not give the Japanese
maintenance
team a simple, self-contained piece of code which replicates the problem

and if they’re reluctant to deal with bug reports that can only be
replicated
by installing tens of thousands of lines of other peoples’ code, some
with
native code extensions, I do have a certain degree of sympathy. For the
segfault problem, I was eventually able to produce a short,
self-contained
script which also blew up — see separate thread here:

http://www.ruby-forum.com/topic/157617

No rocket science there either. I’d identified the revision 17222
patch as a likely culprit, and also figured out what operations were
affected by that patch. So, I wrote a simple script with a long loop,
kept stuffing it with those operations until I saw segfaults, and
then eliminated irrelevant lines from that script until I had the
smallest thing I could which still blew up. (Again, there are no
absolute guarantees here: the suspect patch could conceivably
introduce more than one leak, and a minimal script would replicate
only one. Which is why I said "may be the simplest demonstration
in the post linked above. But while this isn’t guaranteed to find
the problem, it’s generally likely to.)

So, ahem… why am I not doing this? Well, for one thing, I don’t
use mongrel, so I can’t follow Igal’s replication recipe exactly —
and for another, I’m really pressed for time this week. I may
have time soon, but I’d be happier if someone else gets there
first ;-).

HTH,
rst@{ai,alum}.mit.edu

80

Robert Thau wrote:

[ stuff that got mangled by the line-wrapper at ruby-forum.com. ]

Gotta get a real NNTP client. Sigh…

rst