[Open] multipart/form-data for net/http

dubstep · June 20, 2011, 7:28am

Issue #4142 has been updated by Hiroshi NAKAMURA.

Status changed from Closed to Open

Naruse-san, would you please add an explanation of this feature to NEWS
file?

Feature #4142: multipart/form-data for net/http

Author: Yui NARUSE
Status: Open
Priority: Normal
Assignee:
Category: lib
Target version:

=begin
multipart/form-data 対応を net/http に入れませんか。
追加される API は Net::HTTPRequest#set_form になります。

akr さんからは multipart/form-data 用のデータを出力する API 案も示唆されたのですが、
chunked encoding を考慮に入れるとうまくまとまらなかったので見送っています。

diff --git a/lib/net/http.rb b/lib/net/http.rb
index 4d475b1…2751f77 100644
— a/lib/net/http.rb
+++ b/lib/net/http.rb
@@ -22,6 +22,7 @@
require ‘net/protocol’
autoload :OpenSSL, ‘openssl’
require ‘uri’
+autoload :SecureRandom, ‘securerandom’

module Net #:nodoc:

@@ -1772,7 +1773,8 @@ module Net #:nodoc:
alias content_type= set_content_type

  # Set header fields and a body from HTML form data.

+params+ should be a Hash containing HTML form data.

+params+ should be an Array of Arrays or
a Hash containing HTML form data.

Optional argument +sep+ means data record separator.

Values are URL encoded as necessary and the content-type is set

to
@@ -1792,6 +1794,48 @@ module Net #:nodoc:

  alias form_data= set_form_data

Set a HTML form data set.
+params+ is the form data set; it is an Array of Arrays or a

Hash

+enctype is the type to encode the form data set.
It is application/x-www-form-urlencoded or multipart/form-data.
+formpot+ is an optional hash to specify the detail.
boundary:: the boundary of the multipart message
charset:: the charset of the message. All names and the values

of

non-file fields are encoded as the charset.
Each item of params is an array and contains following items:
+name+:: the name of the field
+value+:: the value of the field, it should be a String or a

File

+opt+:: an optional hash to specify additional information
Each item is a file field or a normal field.
If +value+ is a File object or the +opt+ have a filename key,
the item is treated as a file field.
If Transfer-Encoding is set as chunked, this send the request in
chunked encoding. Because chunked encoding is HTTP/1.1 feature,
you must confirm the server to support HTTP/1.1 before sending

it.

Example:
http.set_form([[“q”, “ruby”], [“lang”, “en”]])
See also RFC 2388, RFC 2616, HTML 4.01, and HTML5
def set_form(params, enctype=‘application/x-www-form-urlencoded’,
formopt={})
```
 @body_data = params
```
```
 @body = nil
```
```
 @body_stream = nil
```
```
 @form_option = formopt
```
```
 case enctype
```

 when /\Aapplication\/x-www-form-urlencoded\z/i,

```
   /\Amultipart\/form-data\z/i
```
```
   self.content_type = enctype
```
```
 else
```

   raise ArgumentError, "invalid enctype: #{enctype}"

```
 end
```
end
Set the Authorization: header for “Basic” authorization.

def basic_auth(account, password)
@header[‘authorization’] = [basic_encode(account, password)]
@@ -1849,6 +1893,7 @@ module Net #:nodoc:
self[‘User-Agent’] ||= ‘Ruby’
@body = nil
@body_stream = nil
```
 @body_data = nil
```
end

attr_reader :method
@@ -1876,6 +1921,7 @@ module Net #:nodoc:
def body=(str)
@body = str
@body_stream = nil
```
 @body_data = nil
 str
```
end

@@ -1884,6 +1930,7 @@ module Net #:nodoc:
def body_stream=(input)
@body = nil
@body_stream = input

```
 @body_data = nil
 input
```
end

@@ -1901,6 +1948,8 @@ module Net #:nodoc:
send_request_with_body sock, ver, path, @body
elsif @body_stream
send_request_with_body_stream sock, ver, path, @body_stream

```
 elsif @body_data
```

   send_request_with_body_data sock, ver, path, @body_data
 else
   write_header sock, ver, path
 end

@@ -1935,6 +1984,92 @@ module Net #:nodoc:
end
end

def send_request_with_body_data(sock, ver, path, params)

 if /\Amultipart\/form-data\z/i !~ self.content_type

   self.content_type = 'application/x-www-form-urlencoded'

   return send_request_with_body(sock, ver, path,

URI.encode_www_form(params))

```
 end
```
```
 opt = @form_option.dup
```

 opt[:boundary] ||= SecureRandom.urlsafe_base64(40)

 self.set_content_type(self.content_type, boundary:

opt[:boundary])

```
 if chunked?
```
```
   write_header sock, ver, path
```

   encode_multipart_form_data(sock, params, opt)

```
 else
```
```
   require 'tempfile'
```
```
   file = Tempfile.new('multipart')
```

   encode_multipart_form_data(file, params, opt)

```
   file.rewind
```
```
   self.content_length = file.size
```
```
   write_header sock, ver, path
```
```
   IO.copy_stream(file, sock)
```
```
 end
```
end
def encode_multipart_form_data(out, params, opt)
```
 charset = opt[:charset]
```
```
 boundary = opt[:boundary]
```

 boundary ||= SecureRandom.urlsafe_base64(40)

```
 chunked_p = chunked?
```
```
 buf = ''
```
```
 params.each do |key, value, h={}|
```
```
   key = quote_string(key, charset)
```
```
   filename =
```

     h.key?(:filename) ? h[:filename] :

     value.respond_to?(:to_path) ? File.basename(value.to_path) :

```
     nil
```
```
   buf << "--#{boundary}\r\n"
```
```
   if filename
```

     filename = quote_string(filename, charset)

     type = h[:content_type] || 'application/octet-stream'

     buf << "Content-Disposition: form-data; " \

       "name=\"#{key}\"; filename=\"#{filename}\"\r\n" \

```
       "Content-Type: #{type}\r\n\r\n"
```

     if !out.respond_to?(:write) || !value.respond_to?(:read)

       # if +out+ is not an IO or +value+ is not an IO

       buf << (value.respond_to?(:read) ? value.read : value)

     elsif value.respond_to?(:size) && chunked_p

       # if +out+ is an IO and +value+ is a File, use

IO.copy_stream

       flush_buffer(out, buf, chunked_p)

       out << "%x\r\n" % value.size if chunked_p

```
       IO.copy_stream(value, out)
```
```
       out << "\r\n" if chunked_p
```
```
     else
```

       # +out+ is an IO, and +value+ is not a File but an IO

       flush_buffer(out, buf, chunked_p)

       1 while flush_buffer(out, value.read(4096), chunked_p)

```
     end
```
```
   else
```
```
     # non-file field:
```

     #   HTML5 says, "The parts of the generated

multipart/form-data

     #   resource that correspond to non-file fields must not

have a

     #   Content-Type header specified."

     buf << "Content-Disposition: form-data;

name="#{key}"\r\n\r\n"

     buf << (value.respond_to?(:read) ? value.read : value)

```
   end
```
```
   buf << "\r\n"
```
```
 end
```
```
 buf << "--#{boundary}--\r\n"
```
```
 flush_buffer(out, buf, chunked_p)
```
```
 out << "0\r\n\r\n" if chunked_p
```
end
def quote_string(str, charset)
```
 str = str.encode(charset, 
```

fallback:->(c){‘&#%d;’%c.encode(“UTF-8”).ord}) if charset

```
 str = str.gsub(/[\\"]/, '\\\\\&')
```
end
def flush_buffer(out, buf, chunked_p)
```
 return unless buf
```

 out << "%x\r\n"%buf.bytesize if chunked_p

```
 out << buf
```
```
 out << "\r\n" if chunked_p
```
```
 buf.clear
```
end
def supply_default_content_type
return if content_type()
warn ‘net/http: warning: Content-Type did not set; using
application/x-www-form-urlencoded’ if $VERBOSE
diff --git a/lib/net/protocol.rb b/lib/net/protocol.rb
index 2a6cfb4…a3ffa71 100644
— a/lib/net/protocol.rb
+++ b/lib/net/protocol.rb
@@ -168,6 +168,8 @@ module Net # :nodoc:
}
end
alias << write
def writeline(str)
writing {
write0 str + “\r\n”
diff --git a/test/net/http/test_http.rb b/test/net/http/test_http.rb
index 76280ad…12c03a4 100644
— a/test/net/http/test_http.rb
+++ b/test/net/http/test_http.rb
@@ -303,6 +303,102 @@ module TestNetHTTP_version_1_2_methods
assert_equal data.size, res.body.size
assert_equal data, res.body
end
def test_set_form
require ‘tempfile’
file = Tempfile.new(‘ruby-test’)
file << “\u{30c7}\u{30fc}\u{30bf}”
data = [
```
 ['name', 'Gonbei Nanashi'],
```
```
 ['name', 
```

“\u{540d}\u{7121}\u{3057}\u{306e}\u{6a29}\u{5175}\u{885b}”],

 ['s"i\o', StringIO.new("\u{3042 3044 4e9c 925b}")],

```
 ["file", file, filename: "ruby-test"]
```
]
expected = <<“EOM”.gsub(/\n/, “\r\n”)
±-
+Content-Disposition: form-data; name=“name”

+Gonbei Nanashi
±-
+Content-Disposition: form-data; name=“name”
+
+\xE5\x90\x8D\xE7\x84\xA1\xE3\x81\x97\xE3\x81\xAE\xE6\xA8\xA9\xE5\x85\xB5\xE8\xA1\x9B
±-
+Content-Disposition: form-data; name="s\“i\\o”
+
+\xE3\x81\x82\xE3\x81\x84\xE4\xBA\x9C\xE9\x89\x9B
±-
+Content-Disposition: form-data; name=“file”; filename=“ruby-test”
+Content-Type: application/octet-stream
+
+\xE3\x83\x87\xE3\x83\xBC\xE3\x82\xBF
±-–
+EOM

start {|http|
```
 _test_set_form_urlencoded(http, 
```

data.reject{|k,v|!v.is_a?(String)})

 _test_set_form_multipart(http, false, data, expected)

 _test_set_form_multipart(http, true, data, expected)

}
end
def _test_set_form_urlencoded(http, data)
req = Net::HTTP::Post.new(‘/’)
req.set_form(data)
res = http.request req
assert_equal
“name=Gonbei+Nanashi&name=%E5%90%8D%E7%84%A1%E3%81%97%E3%81%AE%E6%A8%A9%E5%85%B5%E8%A1%9B”,
res.body
end
def _test_set_form_multipart(http, chunked_p, data, expected)
data.each{|k,v|v.rewind rescue nil}
req = Net::HTTP::Post.new(‘/’)
req.set_form(data, ‘multipart/form-data’)
req[‘Transfer-Encoding’] = ‘chunked’ if chunked_p
res = http.request req
body = res.body
assert_match(/\A–(?\S+)/, body)
/\A–(?\S+)/ =~ body
expected = expected.gsub(//, boundary)
assert_equal(expected, body)
end
def test_set_form_with_file
require ‘tempfile’
file = Tempfile.new(‘ruby-test’)
file << $test_net_http_data
filename = File.basename(file.to_path)
data = [[‘file’, file]]
expected = <<“EOM”.gsub(/\n/, “\r\n”)
±-
+Content-Disposition: form-data; name=“file”; filename=“”
+Content-Type: application/octet-stream

+
±-–
+EOM

expected.sub!(//, filename)
expected.sub!(//, $test_net_http_data)
start {|http|
```
 data.each{|k,v|v.rewind rescue nil}
```
```
 req = Net::HTTP::Post.new('/')
```

 req.set_form(data, 'multipart/form-data')

```
 res = http.request req
```
```
 body = res.body
```
```
 header, _ = body.split(/\r\n\r\n/, 2)
```

 assert_match(/\A--(?<boundary>\S+)/, body)

```
 /\A--(?<boundary>\S+)/ =~ body
```

 expected = expected.gsub(/<boundary>/, boundary)

 assert_match(/^--(?<boundary>\S+)\r\n/, header)

```
 assert_match(
```

   /^Content-Disposition: form-data; name="file";

filename=“#{filename}”\r\n/,

```
   header)
```
```
 assert_equal(expected, body)
```
```
 data.each{|k,v|v.rewind rescue nil}
```
```
 req['Transfer-Encoding'] = 'chunked'
```
```
 res = http.request req
```
```
 #assert_equal(expected, res.body)
```
}
end
end

class TestNetHTTP_version_1_1 < Test::Unit::TestCase
=end

[Open] multipart/form-data for net/http

Naruse-san, would you please add an explanation of this feature to NEWS file?

+params+ should be a Hash containing HTML form data.

+params+ should be an Array of Arrays or

a Hash containing HTML form data.

Optional argument +sep+ means data record separator.

Values are URL encoded as necessary and the content-type is set

Set a HTML form data set.

+params+ is the form data set; it is an Array of Arrays or a

+enctype is the type to encode the form data set.

It is application/x-www-form-urlencoded or multipart/form-data.

+formpot+ is an optional hash to specify the detail.

boundary:: the boundary of the multipart message

charset:: the charset of the message. All names and the values

non-file fields are encoded as the charset.

Each item of params is an array and contains following items:

+name+:: the name of the field

+value+:: the value of the field, it should be a String or a

+opt+:: an optional hash to specify additional information

Each item is a file field or a normal field.

If +value+ is a File object or the +opt+ have a filename key,

the item is treated as a file field.

If Transfer-Encoding is set as chunked, this send the request in

chunked encoding. Because chunked encoding is HTTP/1.1 feature,

you must confirm the server to support HTTP/1.1 before sending

Example:

http.set_form([[“q”, “ruby”], [“lang”, “en”]])

See also RFC 2388, RFC 2616, HTML 4.01, and HTML5

Set the Authorization: header for “Basic” authorization.

Naruse-san, would you please add an explanation of this feature to NEWS
file?