Rubyzip - `dup': can't dup NilClass (TypeError)

lukich · June 8, 2010, 7:14am

Hello. I’ve trying to figure out rubyzip. Here’s the code I had:

require ‘rubygems’
require ‘zip/zip’

zf = Zip::ZipFile.open(‘616910.zip’)

However, when I run it, it throws an error

/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
dup': can't dup NilClass (TypeError) from /Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:indup’
from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
map' from /Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:indup’
from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1399:in
initialize' from /Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:innew’
from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in
`open’
from zip.rb:5

I’m pretty sure the zip file is there and is not empty. What might be
causing it? I googled the problem, but couldn’t find a definitive
answer.

Thank you,
Luka

lukich · June 8, 2010, 10:11am

Luka S. wrote:

require ‘rubygems’
require ‘zip/zip’

zf = Zip::ZipFile.open(‘616910.zip’)

However, when I run it, it throws an error

/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`dup’: can’t dup NilClass (TypeError)

Works fine for me with rubyzip-0.9.1 + “ruby 1.8.6 (2007-09-24
patchlevel 111) [i486-linux]”, and I just upgraded to rubyzip-0.9.4 with
the same results.

What platform are you on?

The offending code is here:

# deep clone
def dup
  newZipEntrySet = ZipEntrySet.new(@entrySet.values.map { |e| e.dup

})
end

which suggests to me that the zipfile is corrupt or an unsupported
format, since @entrySet must contain a {value=>nil} pair.

Try modifying this code (line 657):

def ZipEntry.read_c_dir_entry(io)  #:nodoc:all
  entry = new(io.path)
  entry.read_c_dir_entry(io)
  return entry
rescue ZipError
  return nil
end

For example, comment out the rescue ZipError // return nil pair.

It seems that this error handling is bad. Either an exception should be
raised here, or the bad entry should be skipped (not saved as a nil
value in @entrySet which causes the dup error you saw)

Regards,

Brian.

lukich · June 8, 2010, 6:41pm

I’ll give it a try, thank you!

lukich · June 8, 2010, 7:30pm

Luka S. wrote:

I’ll give it a try, thank you!

Just to be clear: I’d expect the program to crash still, but this time
at an earlier point which will give a much more useful error about what
went wrong when parsing the zip directory entry.

lukich · August 27, 2010, 10:17am

The 1.57 zipfile on this page fails for me:
http://www.vim.org/scripts/script.php?script_id=2441

The Unix unzip command unzips it just fine.

Here’s the direct link to the failing zip:
http://www.vim.org/scripts/download_script.php?src_id=11978

To test it in irb:

require ‘open-uri’
require ‘zip/zipfilesystem’
open(‘Error : vim online’,
‘rb’) { |f| Zip::ZipFile.open(f) }

this dies with:

TypeError: can’t dup NilClass
from rubyzip-0.9.4/lib/zip/zip.rb:1163:in dup' from rubyzip-0.9.4/lib/zip/zip.rb:1163:in block in dup’
from rubyzip-0.9.4/lib/zip/zip.rb:1163:in `map’
…etc

the zipfiles before 1.57 are fine. For instance:

require ‘open-uri’
require ‘zip/zipfilesystem’
open(‘http://www.vim.org/scripts/download_script.php?src_id=11524’,
‘rb’) { |f| Zip::ZipFile.open(f) }

works just fine.

The nil entry is actually being added by read_central_directory_entries
around line 1250 (I added the raise):

@entrySet = ZipEntrySet.new
@size.times {
    @entrySet << ZipEntry.read_c_dir_entry(io) || raise("nil

entry!")
}

For some reason, read_c_dir_entry is returning nil. I haven’t tried to
figure out why since I’m not familiar with the internals of a zipfile.

lukich · August 27, 2010, 10:43am

Scott B. wrote:

require ‘open-uri’
require ‘zip/zipfilesystem’
open(‘Error : vim online’,
‘rb’) { |f| Zip::ZipFile.open(f) }

For me (with ruby 1.8.7 and rubyzip-0.9.4) that dies with “cannot
convert Tempfile to String”, but if I download the zip locally and then
do Zip::ZipFile.open(“ert.zip”) then I get the same error as you.

For some reason, read_c_dir_entry is returning nil. I haven’t tried to
figure out why since I’m not familiar with the internals of a zipfile.

I suggest you apply the following patch:

— rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:34:19.673351372 +0100
@@ -658,8 +658,8 @@
entry = new(io.path)
entry.read_c_dir_entry(io)
return entry

rescue ZipError
```
 return nil
```

#rescue ZipError
return nil

end

def file_stat(path) # :nodoc:

Then you get a more helpful error:

/var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:645:in
read_c_dir_entry': unknown file type 00 (Zip::ZipInternalError) from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:659:in read_c_dir_entry’
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1255:in
read_central_directory_entries' from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1254:in times’
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1254:in
read_central_directory_entries' from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1261:in read_from_stream’
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in
initialize' from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in open’
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in
initialize' from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in new’
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in
`open’

To be honest, I have no idea why rubyzip is catching these errors and
returning nil, instead of letting them propagate upwards. All it does is
make a more obscure error later on (“cannot dup nil”)

With another patch:

— rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:38:07.475854345 +0100
@@ -642,7 +642,7 @@
when 012
@ftype = :symlink
else

     raise ZipInternalError, "unknown file type #{'0%o' %

(@externalFileAttributes >> 28)}"

     raise ZipInternalError, "unknown file type #{'0%o' %

(@externalFileAttributes >> 28)} for entry #{@name.inspect}"
end
else
if name_is_directory?

you can see that the affected entry is .DS_Store. That unpacks as a
regular file from unix unzip.

I don’t know why this entry happens to have a type of 0, but you can
allow it like this:

— rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:41:14.853352187 +0100
@@ -637,12 +637,12 @@
case (@externalFileAttributes >> 28)
when 04
@ftype = :directory

```
   when 010
```

   when 010, 00
     @ftype = :file
   when 012
     @ftype = :symlink

Anyway, since you have a good test case for this, I suggest you post it
to the rubyzip mailing list or tracker, if there is one.

Regards,

Brian.

lukich · September 10, 2010, 10:14pm

Thomas Sondergaard wrote:

The encoding of the external file attributes appears to be illegal. To
verify this with another tool try running “zipinfo -v” on the zip-file
and notice that it prints a ‘?’ as the file type in the external file
attributes.

Very strange. My zipinfo -v doesn’t show anything amiss. I’m using:

ZipInfo 3.00 of 20 April 2009, by Greg Roelofs and the Info-ZIP
group.

The problem should also be reported to the author of the tool generating
the zip archives.

Whatever tool is generating them, it’s already in widespread use.
For instance, most zipfiles in the following packages are valid, but
rubyzip can’t handle them.

http://www.vim.org/scripts/script.php?script_id=3114
http://www.vim.org/scripts/script.php?script_id=3123
http://www.vim.org/scripts/script.php?script_id=3148
http://www.vim.org/scripts/script.php?script_id=3150
http://www.vim.org/scripts/script.php?script_id=3169

Thanks Thomas. For now I’ve worked around it by shelling out to the
unzip command. If I have time I’ll try your patch next week.

- Scott

lukich · September 10, 2010, 10:16pm

Scott B. wrote:

Whatever tool is generating them, it’s already in widespread use.

That might not actually be true… Most failing zipfiles were upped by
the same author. I’ll ask Peter Odding what he used to zip them.

- Scott

lukich · September 10, 2010, 9:54pm

Brian C. wrote:

I don’t know why this entry happens to have a type of 0, but you can
allow it like this:

— rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:41:14.853352187 +0100
@@ -637,12 +637,12 @@
case (@externalFileAttributes >> 28)
when 04
@ftype = :directory
   when 010
   when 010, 00
     @ftype = :file
   when 012
     @ftype = :symlink

The encoding of the external file attributes appears to be illegal. To
verify this with another tool try running “zipinfo -v” on the zip-file
and notice that it prints a ‘?’ as the file type in the external file
attributes.

rubyzip should not throw if the value is unknown - it should simply
translate it to :unknown.

My familiarity with both ruby and rubyzip are rusting, but perhaps
something like this:

diff -u -r1.47 zip.rb
— lib/zip/zip.rb 14 May 2010 20:19:08 -0000 1.47
+++ lib/zip/zip.rb 10 Sep 2010 19:51:32 -0000
@@ -643,7 +643,7 @@
when 012
@ftype = :symlink
else

     raise ZipInternalError, "unknown file type #{'0%o' %

(@externalFileAttributes >> 28)}"

     @ftype = :unknown
   end
 else
   if name_is_directory?

@@ -709,11 +709,11 @@
when :symlink
ft = 012
@unix_perms ||= 0755

```
   else
```

     raise ZipInternalError, "unknown file type #{self.inspect}"
   end

   @externalFileAttributes = (ft << 12 | (@unix_perms & 07777)) <<

16

```
   if (!ft.nil?)
```

     @externalFileAttributes = (ft << 12 | (@unix_perms & 07777))

<< 16

```
   end
 end

 io <<
```

The problem should also be reported to the author of the tool generating
the zip archives.

Regards,

Thomas

lukich · September 11, 2010, 12:25pm

Hi Scott,

Whatever tool is generating them, it’s already in widespread use.

That might not actually be true… Most failing zipfiles were upped by
the same author. I’ll ask Peter Odding what he used to zip them.

I publish my Vim plug-ins with a Python script that uses the zipfile [1]
module to generate ZIP files. All of the archives uploaded in the last
few months were generated on Ubuntu 9.10 (Karmic) with Python 2.6,
although I just yesterday upgraded my laptop to Ubuntu 10.4 (Lucid) so I
can’t check the exact version. The ZIP file generation comes down to the
following code:

from zipfile import ZipFile, ZIP_DEFLATED
archive = ZipFile(‘easytags.zip’, ‘w’, ZIP_DEFLATED)
for filename in [‘plugin/easytags.vim’, ‘autoload/easytags.vim’, …]:
# This is where “git cat-file” is used, see below.
with open(filename) as handle:
archive.writestr(filename, handle.read())
archive.close()

One peculiarity about the Python code is that it uses ZipFile.writestr()
[2] instead of ZipFile.write() [3] so that it will only package
committed versions of my scripts (using “git cat-file -p
HEAD:some/file”).

Let me know if you need any more information. If it helps I might be
able to resurrect my old environment in a virtual machine, but I haven’t
tried yet.

Peter Odding

[1] zipfile — Work with ZIP archives — Python 3.11.4 documentation
[2] zipfile — Work with ZIP archives — Python 3.11.4 documentation
[3] zipfile — Work with ZIP archives — Python 3.11.4 documentation

lukich · September 11, 2010, 2:13pm

I’ve filed the following issue against the python ZipFile module:

http://bugs.python.org/issue9835