Hello folks,
I’m wondering if anyone knows an option I can put in my gemspec to
prevent rubygems from requiring all my dependencies as soon as I
require my library.
It’s okay that gems will inforce their dependencies, (throw an error
if the gems arent there), but I don’t want them all to load at once.
In my library, I’ve got a bunch of requires in my methods, and I want
them to require those libs only when they are called.
so for instance :
require “rubygems”
require “ruport” #depends on fastercsv, but should not load it at this
point
Ruport::DataSet.load(“foo.csv”) #now fastercsv should be loaded
This works fine when running via ruby -Ilib over my sources, but when
I use it as a gem, it autoloads all it’s deps, which takes between 3
and 5 seconds and is really annoying.
Suggestions?
My Rakefile with gemspec is here:
http://rubyurl.com/nb2
On Jun 13, 2006, at 3:21 PM, Gregory B. wrote:
so for instance :
require “rubygems”
require “ruport” #depends on fastercsv, but should not load it at
this point
ruport will be loaded here regardless of rubygems.
Ruport::DataSet.load(“foo.csv”) #now fastercsv should be loaded
This works fine when running via ruby -Ilib over my sources, but when
I use it as a gem, it autoloads all it’s deps, which takes between 3
and 5 seconds and is really annoying.
With older versions of rubygems require time was proportional to the
number of gems installed (I don’t know if this is still true).
You probably want to run gem cleanup.
–
Eric H. - [email protected] - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant
http://trackmap.robotcoop.com
On 6/13/06, Eric H. [email protected] wrote:
them to require those libs only when they are called.
so for instance :
require “rubygems”
require “ruport” #depends on fastercsv, but should not load it at
this point
ruport will be loaded here regardless of rubygems.
That is not what I am referring to. I don’t want fastercsv to be
loaded at this point. Obviously ruport will be loaded 
Since the require “fastercsv” is within the DataSet.load method there
is no reason why fastercsv should load at this point. It does not
when I am not using gems.
With older versions of rubygems require time was proportional to the
number of gems installed (I don’t know if this is still true).
You probably want to run gem cleanup.
Will try this but these are generally on machines that only have
ruport and it’s dependencies installed :-/
On 6/13/06, Eric H. [email protected] wrote:
Since the require “fastercsv” is within the DataSet.load method there
is no reason why fastercsv should load at this point.
Yes, there is. ruport required it.
Eric, I think you are missing something here.
In ruby, if you put a require inside of a method, it is not required
until that method is called.
[sandal@harmonix ruport]$ irb --simple-prompt
def foo; require “rubygems”; end
=> nil
$LOADED_FEATURES.any? { |e| e =~ /rubygems/ }
=> false
foo
=> true
$LOADED_FEATURES.any? { |e| e =~ /rubygems/ }
=> true
require “ruport” does NOT call require “fastercsv”
Ruport::DataSet.load does
It does not when I am not using gems.
If you’re using ruport you’re using gems. ruport contains “require
‘rubygems’” somewhere inside.
yes, it contains
begin; require “rubygems”; rescue LoadError; nil; end
because Ruport is installable via setup.rb, and when I develop it, I
do not have the ruport gem installed, and sometimes have the other
dependant libs installed manually or not at all.
Do you understand what I’m trying to do now
On Jun 13, 2006, at 3:45 PM, Gregory B. wrote:
once.
ruport will be loaded here regardless of rubygems.
That is not what I am referring to. I don’t want fastercsv to be
loaded at this point. Obviously ruport will be loaded
ruport loads fastercsv.
Since the require “fastercsv” is within the DataSet.load method there
is no reason why fastercsv should load at this point.
Yes, there is. ruport required it.
It does not when I am not using gems.
If you’re using ruport you’re using gems. ruport contains “require
‘rubygems’” somewhere inside.
–
Eric H. - [email protected] - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant
http://trackmap.robotcoop.com
On 6/13/06, Eric H. [email protected] wrote:
Ruport::DataSet.load does
It may also, but it is not the only place.
You are not working against trunk.
[sandal@harmonix ruport]$ irb -Ilib --simple-prompt -rruport
$LOADED_FEATURES.any? { |r| r =~ /fastercsv/ }
=> false
[[1,2],[3,4]].to_ds(%w[a b]).as(:csv)
=> “a,b\n1,2\n3,4\n”
$LOADED_FEATURES.any? { |r| r =~ /fastercsv/ }
=> true
I appreciate that you are trying to help, but you really aren’t
answering my question.
On Jun 13, 2006, at 4:18 PM, Gregory B. wrote:
until that method is called.
require “ruport” does NOT call require “fastercsv”
$ cat x.rb
deps = %w[
/usr/local/lib/ruby/gems/1.8/gems/ruport-0.4.5/lib
/usr/local/lib/ruby/gems/1.8/gems/pdf-writer-1.1.3/lib
/usr/local/lib/ruby/gems/1.8/gems/transaction-simple-1.3.0/lib
/usr/local/lib/ruby/gems/1.8/gems/color-tools-1.3.0/lib
]
deps.each { |d| $:.unshift d } # add paths manually outside rubygems
$" << ‘rubygems.rb’ # pretend we already required rubygems
require ‘ruport’ # raises an error, fastercsv not found
$ ruby x.rb
/usr/local/lib/ruby/gems/1.8/gems/ruport-0.4.5/lib/ruport/format/
plugin.rb:48:in `require’: no such file to load – fastercsv (LoadError)
from /usr/local/lib/ruby/gems/1.8/gems/ruport-0.4.5/lib/
ruport/format/plugin.rb:48
from /usr/local/lib/ruby/gems/1.8/gems/ruport-0.4.5/lib/
ruport/format.rb:73
from /usr/local/lib/ruby/gems/1.8/gems/ruport-0.4.5/lib/
ruport/format.rb:72
from /usr/local/lib/ruby/gems/1.8/gems/ruport-0.4.5/lib/
ruport.rb:60
from /usr/local/lib/ruby/gems/1.8/gems/ruport-0.4.5/lib/
ruport.rb:59
from x.rb:12
$ cat -n /usr/local/lib/ruby/gems/1.8/gems/ruport0.4.5/lib/ruport/
format/plugin.rb | grep -A3 -B3 48
45
46
47 class CSVPlugin < Format::Plugin
48 require “fastercsv”
49
50 format_field_names do
51 FasterCSV.generate { |csv| csv << data.fields }
$ cat -n /usr/local/lib/ruby/gems/1.8/gems/ruport-0.4.5/lib/ruport.rb
| grep -A3 -B3 60
57
58
59 %w[config report format query data_row data_set].each { |lib|
60 require “ruport/#{lib}”
61 }
Ruport::DataSet.load does
It may also, but it is not the only place.
–
Eric H. - [email protected] - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant
http://trackmap.robotcoop.com
On 6/13/06, Eric H. [email protected] wrote:
You are not working against trunk.
Then you’ll need to provide code I can work with to discover your
real problem. Downloading an entire project without gems is a heavy
burden.
svn checkout svn://rubyforge.org//var/svn/ruport
Not exactly a heavy burden.
Note that Rubygems doesn’t do that much to require, it is roughly this:
- Call Kernel#require
- If we got a LoadError, go looking for a gem with that file
- If we found a gem, add its libraries to $LOAD_PATH
- Call Kernel#require
So I don’t believe rubygems is at fault here.
[sandal@harmonix pkg]$ irb --simple-prompt -rubygems -rruport
$LOADED_FEATURES.length
=> 166
[sandal@harmonix pkg]$ irb --simple-prompt -I…/lib -rruport
$LOADED_FEATURES.length
=> 80
(both against trunk)
Rubygems doesn’t do any auto-require unless you have set an
autorequire in your gemspec and use require_gem (IIRC, old versions
of rubygems would require everything in autorequire even when using
require).
no auto_require in my spec. No require_gem
[sandal@harmonix ruport]$ gem -v
0.8.11
On Jun 13, 2006, at 4:52 PM, Gregory B. wrote:
On 6/13/06, Eric H. [email protected] wrote:
Ruport::DataSet.load does
It may also, but it is not the only place.
You are not working against trunk.
Then you’ll need to provide code I can work with to discover your
real problem. Downloading an entire project without gems is a heavy
burden.
Note that Rubygems doesn’t do that much to require, it is roughly this:
- Call Kernel#require
- If we got a LoadError, go looking for a gem with that file
- If we found a gem, add its libraries to $LOAD_PATH
- Call Kernel#require
So I don’t believe rubygems is at fault here.
[sandal@harmonix ruport]$ irb -Ilib --simple-prompt -rruport
$LOADED_FEATURES.any? { |r| r =~ /fastercsv/ }
=> false
[[1,2],[3,4]].to_ds(%w[a b]).as(:csv)
=> “a,b\n1,2\n3,4\n”
$LOADED_FEATURES.any? { |r| r =~ /fastercsv/ }
=> true
I appreciate that you are trying to help, but you really aren’t
answering my question.
Rubygems doesn’t do any auto-require unless you have set an
autorequire in your gemspec and use require_gem (IIRC, old versions
of rubygems would require everything in autorequire even when using
require).
–
Eric H. - [email protected] - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant
http://trackmap.robotcoop.com
On 6/13/06, Eric H. [email protected] wrote:
I don’t see fastercsv being required when rubygems is involved.
Okay, maybe i’m not being super clear. The problem is it’s loading
all dependencies.
I’m curious though, how is this command running at all:
ruby -e “require ‘ruport’; $LOADED_FEATURES.each { |f| puts f }” >
norubygems
It does not look like you’ve included the lib dir?
e.g -Ilib
or are you running this within lib?
I imagine that what is happening if you are running it within lib, is
that both commands are pulling ruport from the svn code directly, not
from a gem build and installed.
On Jun 13, 2006, at 5:39 PM, Gregory B. wrote:
Not exactly a heavy burden.
I don’t see fastercsv being required when rubygems is involved.
$ svn info .
Path: .
URL: svn://rubyforge.org/var/svn/ruport/trunk/ruport/lib
Repository Root: svn://rubyforge.org/var/svn/ruport
Repository UUID: 0132c854-6f07-0410-9f12-e6e9c8fde30c
Revision: 300
Node Kind: directory
Schedule: normal
Last Changed Author: binarywaterfall
Last Changed Rev: 300
Last Changed Date: 2006-06-13 17:04:33 -0700 (Tue, 13 Jun 2006)
$ ruby -rrubygems -e “require ‘ruport’; $LOADED_FEATURES.each { |f|
puts f }” > rubygems
$ ruby -e “require ‘ruport’; $LOADED_FEATURES.each { |f| puts f }” >
norubygems
$ diff -u norubygems rubygems
— norubygems 2006-06-13 18:23:55.000000000 -0700
+++ rubygems 2006-06-13 18:23:48.000000000 -0700
@@ -1,4 +1,3 @@
-ruport.rb
rubygems.rb
rbconfig.rb
rubygems/rubygems_version.rb
@@ -25,6 +24,7 @@
fcntl.bundle
openssl/x509.rb
rubygems/custom_require.rb
+ruport.rb
ruport/config.rb
singleton.rb
ostruct.rb
–
Eric H. - [email protected] - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant
http://trackmap.robotcoop.com
On 6/13/06, Eric H. [email protected] wrote:
I don’t see fastercsv being required when rubygems is involved.
$ ruby -rrubygems -e “require ‘ruport’; $LOADED_FEATURES.each { |f|
puts f }” > rubygems
$ ruby -e “require ‘ruport’; $LOADED_FEATURES.each { |f| puts f }” >
norubygems
I get this when I use either -Ilib or call from within lib, because
RubyGems will check the loadpath first, afaik.
but try this:
[sandal@harmonix lib]$ cd …
[sandal@harmonix ruport]$ ruby -rrubygems -e “require ‘ruport’;
$LOADED_FEATURES.each { |f|
puts f }” > rubygems
[sandal@harmonix ruport]$ diff -u rubygems lib/norubygems
— rubygems 2006-06-13 21:56:44.000000000 -0400
+++ lib/norubygems 2006-06-13 21:55:13.000000000 -0400
@@ -1,3 +1,4 @@
+ruport.rb
rubygems.rb
rbconfig.rb
rubygems/rubygems_version.rb
@@ -22,96 +23,6 @@
fcntl.so
openssl/x509.rb
rubygems/custom_require.rb
-faster_csv.rb
-English.rb
-enumerator.so
-date.rb
-stringio.so
-redcloth.rb
-color.rb
-color/rgb.rb
-color/rgb-colors.rb
-color/cmyk.rb
-color/grayscale.rb
-color/hsl.rb
-color/yiq.rb
-color/rgb/metallic.rb
-transaction/simple.rb
-pdf/writer.rb
-thread.rb
-open-uri.rb
-uri.rb
-uri/common.rb
-uri/generic.rb
-uri/ftp.rb
-uri/http.rb
-uri/https.rb
-uri/ldap.rb
-uri/mailto.rb
-pdf/math.rb
-pdf/writer/lang.rb
-pdf/writer/lang/en.rb
-zlib.so
-pdf/writer/arc4.rb
-digest/md5.so
-digest.so
-pdf/writer/fontmetrics.rb
-pdf/writer/object.rb
-pdf/writer/object/action.rb
-pdf/writer/object/annotation.rb
-pdf/writer/object/catalog.rb
-pdf/writer/object/contents.rb
-pdf/writer/object/destination.rb
-pdf/writer/object/encryption.rb
-pdf/writer/object/font.rb
-pdf/writer/object/fontdescriptor.rb
-pdf/writer/object/fontencoding.rb
-pdf/writer/object/image.rb
-pdf/writer/oreader.rb
-pdf/writer/object/info.rb
-pdf/writer/object/outlines.rb
-pdf/writer/object/outline.rb
-pdf/writer/object/page.rb
-pdf/writer/object/pages.rb
-pdf/writer/object/procset.rb
-pdf/writer/object/viewerpreferences.rb
-pdf/writer/ohash.rb
-pdf/writer/strokestyle.rb
-pdf/writer/graphics.rb
-pdf/writer/graphics/imageinfo.rb
-pdf/writer/state.rb
-rexml/document.rb
-rexml/element.rb
-rexml/parent.rb
-rexml/child.rb
-rexml/node.rb
-rexml/parseexception.rb
-rexml/namespace.rb
-rexml/xmltokens.rb
-rexml/attribute.rb
-rexml/text.rb
-rexml/entity.rb
-rexml/source.rb
-rexml/encoding.rb
-rexml/doctype.rb
-rexml/attlistdecl.rb
-rexml/cdata.rb
-rexml/xpath.rb
-rexml/functions.rb
-rexml/xpath_parser.rb
-rexml/syncenumerator.rb
-rexml/parsers/xpathparser.rb
-rexml/xmldecl.rb
-rexml/comment.rb
-rexml/instruction.rb
-rexml/rexml.rb
-rexml/output.rb
-rexml/parsers/baseparser.rb
-rexml/parsers/streamparser.rb
-rexml/parsers/treeparser.rb
-rexml/validation/validationexception.rb
-rexml/encodings/UTF-8.rb
-ruport.rb
ruport/config.rb
singleton.rb
ostruct.rb
@@ -119,6 +30,7 @@
erb.rb
strscan.so
yaml.rb
+stringio.so
yaml/error.rb
yaml/syck.rb
syck.so
@@ -128,6 +40,7 @@
yaml/stream.rb
yaml/constants.rb
yaml/rubytypes.rb
+date.rb
yaml/types.rb
logger.rb
monitor.rb
@@ -138,6 +51,8 @@
net/protocol.rb
socket.so
timeout.rb
+digest/md5.so
+digest.so
ruport/format.rb
ruport/format/builder.rb
ruport/format/open_node.rb
On 6/13/06, Eric H. [email protected] wrote:
So I don’t believe rubygems is at fault here.
I decided to make a simple example gem to illustrate this. Here is
the Rakefile with simple spec
require “rake/rdoctask”
require “rake/testtask”
require “rake/gempackagetask”
require “rubygems”
spec = Gem::Specification.new do |spec|
spec.name = “dummie”
spec.version = “0.0.1”
spec.platform = Gem::Platform::RUBY
spec.summary = “A dummy lib”
spec.require_path = “lib”
spec.files = “lib/dumb.rb”
spec.add_dependency(‘ruport’, ‘>= 0.1.0’)
end
Rake::GemPackageTask.new(spec) do |pkg|
pkg.need_zip = true
pkg.need_tar = true
end
lib/dumb.rb actually is just an empty file. There are no requires.
If you unpack the gem once installing it, you can see this.
Now take a look at this.
[sandal@harmonix ~]$ irb -rubygems --simple-prompt
require “dumb”
=> true
$LOADED_FEATURES.include? “faster_csv.rb”
=> true
Any reason why this should happen?
Because this sounds like a bug, here is some system info.
[sandal@harmonix ~]$ ruby -v
ruby 1.8.4 (2005-12-24) [i686-linux]
[sandal@harmonix ~]$ gem -v
0.8.11
[sandal@harmonix ~]$ rake --version
rake, version 0.7.1
[sandal@harmonix ~]$ uname -a
Linux harmonix 2.6.16-ARCH #1 SMP PREEMPT Wed May 31 06:41:53 UTC 2006
i686 Mobile AMD Sempron™ Processor 3000+ AuthenticAMD GNU/Linux
On Jun 13, 2006, at 6:54 PM, Gregory B. wrote:
On 6/13/06, Eric H. [email protected] wrote:
I don’t see fastercsv being required when rubygems is involved.
Okay, maybe i’m not being super clear. The problem is it’s loading
all dependencies.
Well, yes, I was saying I don’t see that behavior.
I’m curious though, how is this command running at all:
ruby -e “require ‘ruport’; $LOADED_FEATURES.each { |f| puts f }” >
norubygems
It does not look like you’ve included the lib dir?
e.g -Ilib
or are you running this within lib?
running within lib.
I imagine that what is happening if you are running it within lib, is
that both commands are pulling ruport from the svn code directly, not
from a gem build and installed.
I tried building a gem from SVN and got the same behavior, no
difference between rubygems and rubygems2. (This wouldn’t make a
difference, Rubygems doesn’t load anything extra.)
–
Eric H. - [email protected] - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant
http://trackmap.robotcoop.com
Gregory B. wrote:
On 6/13/06, Eric H. [email protected] wrote:
I don’t see fastercsv being required when rubygems is involved.
I think the difference here is that Eric is probably running with a very
recent development version of RubyGems and Gregory is running the last
stable release.
I setup two gems, a and b, and had a depend upon b. Both libraries
announce themselves while loading:
– Using 0.8.11 -----------
$ gem -v
0.8.11
$ irb
irb(main):001:0> require ‘a’
Loading B
Loading A
=> true
irb(main):002:0>
– Using 0.8.99 -----------
$ gem -v
0.8.99
$ irb
irb(main):001:0> require ‘a’
Loading A
=> true
irb(main):002:0>
Version 0.9.0 should be out within the week, barring unexpected issues.
If you want the try the release candidate for 0.9.0 (which is version
0.8.99), do:
gem update --system --source http://onestepback.org/betagems
If you do try it, please report any issues to me or to the Rubygems
developer mailing list (on RubyForge).
Thanks.
– Jim W.
A quick followup…
Jim W. wrote:
– Using 0.8.11 -----------
$ gem -v
0.8.11
$ irb
irb(main):001:0> require ‘a’
Loading B
Loading A
=> true
irb(main):002:0>
Note that you only get this behavior when the b package specifies an
autorequire file (which b does in this case, and which the fastercsv
package does as well). When RubyGems activates b, it does any
autorequires specified by b.
This is why:
(1) I really believe autorequire is evil. It isn’t the only subtle bug
it has caused.
(2) The soon to be released RubyGems removes a lot of the autorequire
behavior.
– Jim W.
On Jun 13, 2006, at 7:23 PM, Gregory B. wrote:
require “rubygems”
spec = Gem::Specification.new do |spec|
spec.name = “dummie”
spec.version = “0.0.1”
spec.platform = Gem::Platform::RUBY
spec.summary = “A dummy lib”
spec.require_path = “lib”
spec.files = “lib/dumb.rb”
spec.add_dependency(‘ruport’, ‘>= 0.1.0’)
What happens if you remove this line?
Now take a look at this.
[sandal@harmonix ~]$ irb -rubygems --simple-prompt
require “dumb”
=> true
$LOADED_FEATURES.include? “faster_csv.rb”
=> true
Any reason why this should happen?
Because this sounds like a bug, here is some system info.
Do you have the RUBYOPT shell variable set? Do you have
a .gemrc? .irbrc?
[sandal@harmonix ~]$ ruby -v
ruby 1.8.4 (2005-12-24) [i686-linux]
[sandal@harmonix ~]$ gem -v
0.8.11
[sandal@harmonix ~]$ rake --version
rake, version 0.7.1
[sandal@harmonix ~]$ uname -a
Linux harmonix 2.6.16-ARCH #1 SMP PREEMPT Wed May 31 06:41:53 UTC 2006
i686 Mobile AMD Sempron™ Processor 3000+ AuthenticAMD GNU/Linux
–
Eric H. - [email protected] - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant
http://trackmap.robotcoop.com
On 6/13/06, Jim W. [email protected] wrote:
Gregory B. wrote:
On 6/13/06, Eric H. [email protected] wrote:
I don’t see fastercsv being required when rubygems is involved.
I think the difference here is that Eric is probably running with a very
recent development version of RubyGems and Gregory is running the last
stable release.
Thank you jim.
This patch seems to resolve the problem, also:
— rubygems.rb 2006-06-13 22:48:48.000000000 -0400
+++ rubygems-fix.rb 2006-06-13 22:49:22.000000000 -0400
@@ -164,7 +164,7 @@
# Load dependent gems first
spec.dependencies.each do |dep_gem|
On 6/13/06, Jim W. [email protected] wrote:
Version 0.9.0 should be out within the week, barring unexpected issues.
If you want the try the release candidate for 0.9.0 (which is version
0.8.99), do:
gem update --system --source http://onestepback.org/betagems
Yup, that did it!
[sandal@harmonix pkg]$ irb -rubygems --simple-prompt
require “ruport”
=> true
$LOADED_FEATURES.length
=> 83
$LOADED_FEATURES.include? “faster_csv.rb”
=> false
[[1,2]].to_ds(%w[a b]).as(:csv)
=> “a,b\n1,2\n”
$LOADED_FEATURES.include? “faster_csv.rb”
=> true
quit
[sandal@harmonix pkg]$ gem -v
0.8.99
If you do try it, please report any issues to me or to the Rubygems
developer mailing list (on RubyForge).
I’m subbed to the list, I would have asked there first, but I didn’t
suspect a bug at first, just a problem with my Rakefile maybe.
Thanks Eric and Jim!
On 6/13/06, Eric H. [email protected] wrote:
Already solved this, but for clarity, I’ll answer these questions too.
spec.add_dependency(‘ruport’, ‘>= 0.1.0’)
What happens if you remove this line?
Well, then ruport and it’s deps would disappear
Do you have the RUBYOPT shell variable set? Do you have
a .gemrc? .irbrc?
[sandal@harmonix ~]$ echo $RUBY_OPT
[sandal@harmonix ~]$ cat ~/.gemrc
cat: /home/sandal/.gemrc: No such file or directory
[sandal@harmonix ~]$ cat ~/.irbrc
cat: /home/sandal/.irbrc: No such file or directory
This machine is pretty clean. I haven’t tweaked my install much
because it’s my primary dev box and I like to assume as little as
possible about other peoples configs.