hii ,
I uploaded 1 GB of file to filesystem,but now when i trying to
download this 1 GB of file ,i am getting failed to allocate memory.
am running mongrel as server in development mode.
NoMemoryError (failed to allocate memory):
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/cache.rb:60:in
<<' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/cache.rb:60:inexpand_cache_key’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/response.rb:118:in
etag=' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/response.rb:148:inhandle_conditional_get!’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/response.rb:138:in
prepare!' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/base.rb:532:insend_response’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/base.rb:526:in
process_without_filters' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/filters.rb:606:inprocess_without_session_ma
gement_support’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/session_management.rb:134:in
process' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/base.rb:392:inprocess’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/dispatcher.rb:183:in
handle_request' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/dispatcher.rb:110:indispatch_unlocked’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/dispatcher.rb:123:in
dispatch' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/dispatcher.rb:122:insynchronize’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/dispatcher.rb:122:in
dispatch' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/dispatcher.rb:132:indispatch_cgi’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/dispatcher.rb:39:in
dispatch' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/bin/../lib/mongrel/rails.rb:76:inprocess’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/bin/…/lib/mongrel/rails.rb:74:in
synchronize' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/bin/../lib/mongrel/rails.rb:74:inprocess’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:159:in
process_client' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:158:ineach’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:158:in
process_client' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:285:inrun’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:285:in
initialize' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:285:innew’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:285:in
run' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:268:ininitialize’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:268:in
new' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:268:inrun’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel/configurator.rb:282:in
run' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel/configurator.rb:281:ineach’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel/configurator.rb:281:in
run' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/bin/mongrel_rails:128:inrun’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel/command.rb:212:in
run' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/bin/mongrel_rails:281 C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:142:inload_without_new_cons
nt_marking’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:142:in
load' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:521:innew_constants_in’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:142:in
load' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/rails-2.2.2/lib/commands/servers/mongrel.rb:64 C:/InstantRails-2.0-win/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:ingem_original_require’
C:/InstantRails-2.0-win/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in
require' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:153:inrequire’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:521:in
new_constants_in' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:153:inrequire’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/rails-2.2.2/lib/commands/server.rb:49
C:/InstantRails-2.0-win/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in
gem_original_require' C:/InstantRails-2.0-win/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:inrequire’
script/server:3
ndered
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/templates/rescues/_trace
(250.0ms)
On Mon, Sep 13, 2010 at 7:33 AM, Amit T. [email protected]
wrote:
hii ,
I uploaded 1 GB of file to filesystem,but now when i trying to
download this 1 GB of file ,i am getting failed to allocate memory.
am running mongrel as server in development mode.
NoMemoryError (failed to allocate memory):
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/cache.rb:60:in
<<' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/cache.rb:60:in expand_cache_key’
This has nothing to do with Mongrel.
From the backtrace it seems you’re trying to cache all the file
contents in memory.
What is the code you’re using to serve the file? Mongrel is quite
capable to serve any file from public folder without any Rails code
interfering it.
Luis L.
AREA 17
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
Luis L. wrote:
On Mon, Sep 13, 2010 at 7:33 AM, Amit T. [email protected]
wrote:
hii ,
? ? ?I uploaded 1 GB of file to filesystem,but now when i trying to
download this 1 GB of file ,i am getting failed to allocate memory.
am running mongrel as server in development mode.
NoMemoryError (failed to allocate memory):
?C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/cache.rb:60:in
<<' ?C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/cache.rb:60:in expand_cache_key’
This has nothing to do with Mongrel.
From the backtrace it seems you’re trying to cache all the file
contents in memory.
What is the code you’re using to serve the file? Mongrel is quite
capable to serve any file from public folder without any Rails code
interfering it.
Luis L.
AREA 17
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupé²¹
Luis ,this is how am downloading my files
send_data(@data,:filename => @containerformat.name+extension,
:disposition => ‘attachment’)
but i am getting error failed to allocate memory for large files ,for
small files it’s working fine
and one thing more i observed while uploading large files , whenever i
upload data more than 2gb from IE OR from mozila ,both restarts but for
google chorme its working fine???
On Tue, Sep 14, 2010 at 2:49 AM, Amit T. [email protected]
wrote:
Luis ,this is how am downloading my files
send_data(@data,:filename => @containerformat.name+extension,
:disposition => ‘attachment’)
but i am getting error failed to allocate memory for large files ,for
small files it’s working fine
Because send_data will try to load the whole file in memory.
http://apidock.com/rails/ActionController/Streaming/send_data
Since the file is not dynamically generated, serve it directly, don’t
put it behind Rails.
and one thing more i observed while uploading large files , whenever i
upload data more than 2gb from IE OR from mozila ,both restarts but for
google chorme its working fine???
Neither IE or Mozilla do chunked uploads. Chrome does and mongrels
properly streams the chunked upload to a temporary file.
Luis L.
AREA 17
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
On Tue, Sep 14, 2010 at 9:21 AM, Amit T. [email protected]
wrote:
thanks luis
but could you please explain how i serve it directly
Place the file inside public, provide the URL to the file as a link,
your browser will start download once the file is served.
send_data is not recommended for big file serving.
and for my
application i can’nt use IE because as you said it don’t do chunked
upload.
Can’t help you on that, please research about it.
Luis L.
AREA 17
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
Luis L. wrote:
On Tue, Sep 14, 2010 at 9:21 AM, Amit T. [email protected]
wrote:
thanks luis
but could you please explain how i serve ?it directly
Place the file inside public, provide the URL to the file as a link,
your browser will start download once the file is served.
send_data is not recommended for big file serving.
and for my
application i can’nt use ?IE because as you said it don’t ?do chunked
upload.
Can’t help you on that, please research about it.
Luis L.
AREA 17
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupé²¹
sorry luis but i didn’t get you
i uploaded my file in filesystem than how can i place file in public
folder
could you please give some example??
Luis L. wrote:
On Tue, Sep 14, 2010 at 2:49 AM, Amit T. [email protected]
wrote:
Luis ,this is how am downloading my files
send_data(@data,:filename => @containerformat.name+extension,
? ? ? ?:disposition => ‘attachment’)
but i am getting error failed to allocate memory for large files ,for
small files it’s working fine
Because send_data will try to load the whole file in memory.
send_data (ActionController::Streaming) - APIdock
Since the file is not dynamically generated, serve it directly, don’t
put it behind Rails.
and one thing more i observed while uploading large files , whenever i
upload data more than 2gb from IE OR from mozila ,both restarts but for
google chorme its working fine???
Neither IE or Mozilla do chunked uploads. Chrome does and mongrels
properly streams the chunked upload to a temporary file.
Luis L.
AREA 17
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupé²¹
thanks luis
but could you please explain how i serve it directly and for my
application i can’nt use IE because as you said it don’t do chunked
upload.
thanks again
On Tue, Sep 14, 2010 at 9:39 AM, Amit T. [email protected]
wrote:
sorry luis but i didn’t get you
i uploaded my file in filesystem than how can i place file in public
folder
could you please give some example??
Are you using a plugin to manage your uploads? How are you saving the
file?
Take the temporary file and use File.move or FileUtils.mv, wonder what
are you doing right now for that.
Please use google to search, there are plenty of file uploading
tutorials about Ruby on Rails on the internet covering “do from
scratch” or using gem/plugins.
Luis L.
AREA 17
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
Luis L. wrote:
On Tue, Sep 14, 2010 at 9:39 AM, Amit T. [email protected]
wrote:
sorry luis but i didn’t get you
i uploaded my file in filesystem than how ?can ?i place file in public
folder
could you please give some example??
Are you using a plugin to manage your uploads? How are you saving the
file?
Take the temporary file and use File.move or FileUtils.mv, wonder what
are you doing right now for that.
Please use google to search, there are plenty of file uploading
tutorials about Ruby on Rails on the internet covering “do from
scratch” or using gem/plugins.
Luis L.
AREA 17
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupé²¹
this is how am uploading
def upload(params)
path = get_path(params)
File.open(path, "wb") do |f|
while buff= params[:location].read(4096)
f.write(buff)
end
end
end
def get_path(params)
name = params[:location].original_filename
directory = “//192.168.247.77/Smruti/streams”
path = File.join(directory, name)
return path
end
def copyFile(uploaded_file)
@uploaded_file = uploaded_file
directory = “//192.168.147.17/Smruti/streams”
#url_dir = “data”
url_dir = “//192.168.147.17/Smruti/streams”
name = File.basename(@uploaded_file)
path = File.join(directory, name)
if (!FileTest.exist?(path))
FileUtils.copy @uploaded_file, path
else
return “duplicate”
end
url = File.join(url_dir, name)
return url
end
end
and after copying it filesystem
Luis L. wrote:
On Tue, Sep 14, 2010 at 9:50 AM, Amit T. [email protected]
wrote:
? ? ? ?end
end
That is copying the file every 4k for you, you can simply File.move
form the temporay filename to the final name in your remote location.
It is expected you dispose the temporary file when you’re done with it.
Please see how proper file uploading and processing is done with Ruby on
Rails.
Luis L.
AREA 17
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupé²¹
Thanks luis
you said right ,i have to check how uploading takes place…
could you suggest some documnet ??
On Tue, Sep 14, 2010 at 9:50 AM, Amit T. [email protected]
wrote:
end
end
That is copying the file every 4k for you, you can simply File.move
form the temporay filename to the final name in your remote location.
It is expected you dispose the temporary file when you’re done with it.
Please see how proper file uploading and processing is done with Ruby on
Rails.
Luis L.
AREA 17
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
Luis L. wrote:
On Tue, Sep 14, 2010 at 10:18 AM, Amit T. [email protected]
wrote:
Thanks luis
you said right ,i have to check how uploading takes place…
could you suggest some documnet ??
Google is your friend, “Rails File upload”
Rails File Uploading 101
And others.
If the files are located in a remote server that is not in public
directory, then you will not have alternative but use something like
apache to serve map the files to the remote disk and serve them.
But please, avoid send_data at any cost, and even the file
reading/writing you’re doing, because with the size you’re working,
you’re just slowing your application with IO operations.
Luis L.
AREA 17
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupé²¹
luis ,but do n’t have option other than send_data and one thing i can do
is while downloading large files i can do it in chunks but i only
download 4096 byte ,this is my code
File.open(@containerformat.location,“rb”)do
|f| @data = f.read(4096) end
ext = File.extname(@containerformat.streamName)
send_data(@data,:filename => name+extension,
:disposition => 'attachment')
end
what happening here…
On Tue, Sep 14, 2010 at 11:14 AM, Amit T. [email protected]
wrote:
luis ,but do n’t have option other than send_data and one thing i can do
is while downloading large files i can do it in chunks but i only
download 4096 byte ,this is my code
please read about streaming data without loading all the file contents.
You will need to either use rack directly or create a mongrel handler
to serve these files outside rails.
Rails can only deliver one full request meaning it will load the full
file before completing.
–
Luis L.
AREA 17
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
On Tue, Sep 14, 2010 at 10:18 AM, Amit T. [email protected]
wrote:
Thanks luis
you said right ,i have to check how uploading takes place…
could you suggest some documnet ??
Google is your friend, “Rails File upload”
http://www.rubyinside.com/rails-file-uploading-101-406.html
And others.
If the files are located in a remote server that is not in public
directory, then you will not have alternative but use something like
apache to serve map the files to the remote disk and serve them.
But please, avoid send_data at any cost, and even the file
reading/writing you’re doing, because with the size you’re working,
you’re just slowing your application with IO operations.
Luis L.
AREA 17
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
Sorry i think am taking lot of your time
Luis L. wrote:
On Tue, Sep 14, 2010 at 11:14 AM, Amit T. [email protected]
wrote:
luis ,but do n’t have option other than send_data and one thing i can do
is while downloading large files i can do it in chunks but i only
download 4096 byte ,this is my code
please read about streaming data without loading all the file contents.
You will need to either use rack directly or create a mongrel handler
to serve these files outside rails.
Rails can only deliver one full request meaning it will load the full
file before completing.
–
Luis L.
AREA 17
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupé²¹
File.open(@containerformat.location,"rb") do |f|
pos=f.tell
while @buffer = f.read(4096)
ext = File.extname(@containerformat.streamName)
if ext == ''
extension = File.extname(@containerformat.location)
#send_to_web_app(pos, buffer)
send_data(@buffer,:filename => @containerformat.name+extension,
:disposition => 'attachment')
else
send_data(@buffer,:filename => @containerformat.streamName,
:disposition => 'attachment')
pos=f.tell
end
end
now am doing it like above and trying to download 284mb of file
but only 211 bytes of file being downloaded…
why??
Luis L. wrote:
please read about streaming data without loading all the file contents.
You will need to either use rack directly or create a mongrel handler
to serve these files outside rails.
Rails can only deliver one full request meaning it will load the full
file before completing.
Notice that now you can use send_file which is now (Rails 3) implemented
through a rack middleware and includes X-sendfile headers.
Daniel Cadenas wrote:
Luis L. wrote:
please read about streaming data without loading all the file contents.
You will need to either use rack directly or create a mongrel handler
to serve these files outside rails.
Rails can only deliver one full request meaning it will load the full
file before completing.
Notice that now you can use send_file which is now (Rails 3) implemented
through a rack middleware and includes X-sendfile headers.
yaa Daniel i already imlemented send_file method and now i downloaded 7
gb of file without fail …
The only problem am facing is how do i delete
mongrel temporary after upload large files…