Download via Net::Http blows up the file size

addis_a · July 30, 2014, 1:25pm

Hi,

I am trying to download a file over Http using the Net::Http module. The
file I am trying to download is
http://public.dhe.ibm.com/ibmdl/export/pub/software/data/db2/drivers/odbc_cli/linuxx64_odbc_cli.tar.gz
which is of size 25M.

When tried to download via Ruby using the code below, I see that in Ruby
-2.1.2p95 the size is blown up to approx 85M. Now when trying to untar
this archive using the zlib utils [ Gem::TarReader.new(
Zlib::GzipReader.open(archive) ) do |tar| ] it fails complaining it is
not in gzip format.

But when I try the same code in Ruby-1.9.3p429 it works fine and I am
able to see a file of 25M downloaded and untarred successfully.

Any idea what is going wrong? Am I missing something?

Code to download the file:

def downloadCLIPackage(destination)
uri =
URI.parse(“http://public.dhe.ibm.com/ibmdl/export/pub/software/data/db2/drivers/odbc_cli/linuxx64_odbc_cli.tar.gz”)
filename = “/home/praveend/clidriver.tar.gz”

http_object = Net::HTTP.new(uri.host, uri.port)
http_object.use_ssl = true if uri.scheme == ‘https’
response = Net::HTTP.get_response(uri)
f = open(filename, ‘wb’)
f.write(response.body)
f.close()

filename
end

Thanks

Praveen

praveend · July 30, 2014, 6:53pm

This is happening because Net::HTTP is automatically decompressing the
data when it sees that the data is of type gzip.

github.com

ruby/ruby/blob/448c87008e1bc66a93b8f21eedea060f281874a4/lib/net/http/response.rb#L254


      
          #
          # If a Content-Range header is present a plain socket is yielded as the
          # bytes in the range may not be a complete deflate block.
          
          
def inflater # :nodoc:
            return yield @socket unless Net::HTTP::HAVE_ZLIB
            return yield @socket unless @decode_content
            return yield @socket if self['content-range']
          
          
  case self['content-encoding']
            when 'deflate', 'gzip', 'x-gzip' then
              self.delete 'content-encoding'
          
          
    inflate_body_io = Inflater.new(@socket)
          
          
    begin
                yield inflate_body_io
              ensure
                e = $!
                begin
                  inflate_body_io.finish

Thanks to @yorickpeterse on ruby irc channel who helped on this.

But still this doesn’t help me getting it working end to end. Any one
knows how can I save this untarred content [it contains directories and
I want to preserve the tree structure] onto disk and make use of it?

Thanks

Praveen

praveend · July 31, 2014, 5:42am

From:
http://www.ruby-doc.org/stdlib-2.1.1/libdoc/net/http/rdoc/Net/HTTP.html#class-Net::HTTP-label-Compression

=====
Compression

Net::HTTP automatically adds Accept-Encoding for compression of response
bodies and automatically decompresses gzip and deflate responses unless
a Range header was sent.

Compression can be disabled through the Accept-Encoding: identity
header.

For some background on ‘identity’:

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

So set a header in your request, e.g.

uri = URI(http://public.dhe.ibm.com/ibmdl/export/pub/soft…)

headers = {
‘Accept-Encoding’ => ‘identity’,
}
req = Net::HTTP::Get.new(uri.request_uri, headers)

http = Net::HTTP.new(uri.host, uri.port)
resp = http.request(req)

Then, you can unzip the file yourself.

praveend · July 31, 2014, 3:12pm

Perfect!!

Thank you 7stud. This works like a charm.

Regards

Praveen

praveend · July 30, 2014, 6:56pm

Also, now that this needs to work for both Ruby-1.9.3 and Ruby-2.1.2 any
idea how can I determine if the data is inflated or not in the code and
accordingly take up next action?

I should be reading headers (?) as I see the code deletes
content-encoding. If this is correct…any pointers on how can I
achieve this?

Thanks

Praveen

Download via Net::Http blows up the file size

Compression can be disabled through the Accept-Encoding: identity header.

Compression can be disabled through the Accept-Encoding: identity
header.