OutOfMemory reading large GZipped file

Ruby JRuby
1

Hi all!

I am working on a log analysis tool, and I have added reading og .gz log
files. When I scan very large files, I run out of heap space. Here is
the part of the code reading the file. Will this try to keep the whole
file in memory? Is there a way to avoid this?

begin
file = Zlib::GzipReader.open(filename)
file.each_line do |line|
…
end
ensure
file.close if file
end

–
With kind regards
Uwe K.
Kubosch Consulting
[email protected]

2

Sorry for the spam, but it seems I have got a corrupted log file
containing lots of binary data. each_line will try to put this in one
line that is larger than available heap space. Problem found :slight_smile:

Now, how to remove binary data at start of file?

On May 15, 2010, at 10:30 AM, Uwe K. wrote:

file.close if file

–
With kind regards
Uwe K.
Kubosch Consulting
[email protected]

3

On May 15, 2010, at 11:10 AM, Uwe K. wrote:

Sorry for the spam, but it seems I have got a corrupted log file containing lots of binary data. each_line will try to put this in one line that is larger than available heap space. Problem found :slight_smile:

Now, how to remove binary data at start of file?

Found the answer here shell-scripting - Best way to skip first line? | DaniWeb

tail -n+2

–
With kind regards
Uwe K.
Kubosch Consulting
[email protected]

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email