Ruby Forum NGINX > HTTP PUT for file uploads

Posted by mike (Guest)
on 01.08.2008 01:58
(Received via mailing list)
It appears that nginx has built-in support for the PUT method.

My company needs a file upload solution that will support large file
uploads (2GB limit is optional - if we have to tell them less than 2GB
that's fine) that will keep re-trying the upload until it is done. We
have slow geo users and then just flat out large files to deal with
even from fast connections.

There's a variety of Java-based PUT uploaders.

So far, we haven't found any Flash ones (we'd love to NOT use Java) -
but there is a way to do it apparently, we just can't find anyone
who's done it yet.

Anyway, from the nginx side, I assume I would need to have some
timeouts set pretty long. Would anyone here have any ideas about that?

I'm assuming that we should keep the connection open as long as there
is some activity and maybe timeout after a minute or two... the
client-side applet should have the logic to continue retrying and
since it is PUT, the PHP script will accept the data and use fseek()
on the file to resume at the offset supplied (the client will have to
give us that info)

See the examples here:
http://www.radinks.com/upload/examples/ - look at the "Handlers that
support resume" section.

Anyone have any thoughts? I think I also need to tweak PHP settings
too possibly as well, for max execution time and such.
Posted by Phillip B Oldham (Guest)
on 01.08.2008 14:22
Attachment: phill.vcf (273 Bytes)
(Received via mailing list)
mike wrote:
> My company needs a file upload solution that will support large file
> uploads (2GB limit is optional - if we have to tell them less than 2GB
> that's fine) that will keep re-trying the upload until it is done. We
> have slow geo users and then just flat out large files to deal with
> even from fast connections.
>   
Don't want to sound off, but isn't that what FTP was designed for?
> There's a variety of Java-based PUT uploaders.
>
> So far, we haven't found any Flash ones (we'd love to NOT use Java) -
> but there is a way to do it apparently, we just can't find anyone
> who's done it yet.
>   
You may have to test this, but I'd assume the maint-stream browsers
support PUT via the XMLHTTPRequest javascript object.

--

*Phillip B Oldham*
The Activity People
phill@theactivitypeople.co.uk <mailto:phill@theactivitypeople.co.uk>

------------------------------------------------------------------------

*Policies*

This e-mail and its attachments are intended for the above named
recipient(s) only and may be confidential. If they have come to you in
error, please reply to this e-mail and highlight the error. No action
should be taken regarding content, nor must you copy or show them to 
anyone.

This e-mail has been created in the knowledge that Internet e-mail is
not a 100% secure communications medium, and we have taken steps to
ensure that this e-mail and attachments are free from any virus. We must
advise that in keeping with good computing practice the recipient should
ensure they are completely virus free, and that you understand and
observe the lack of security when e-mailing us.
Posted by Michael (Guest)
on 01.08.2008 16:13
(Received via mailing list)
On Fri, Aug 01, 2008 at 13:10:46, Phillip B Oldham said...

> Don't want to sound off, but isn't that what FTP was designed for?

FTP is such an awful protocol to deal with regarding firewalls, I'd much 
rather
go the way the original poster is and use HTTP PUT for insecure stuff, 
and SFTP
for secure stuff.

But 2GB?  That's huge... a lot for an end-user to just sit and watch 
without
clicking reload or something.

> You may have to test this, but I'd assume the maint-stream browsers support 
> PUT via the XMLHTTPRequest javascript object.

I was just doing some of this myself yesterday - doesn't seem like 
there's a
way to do file uploads with AJAX, but you can easily make a form with a 
target
iframe... but this is really offtopic for the nginx list.
Posted by Phillip B Oldham (Guest)
on 01.08.2008 16:24
Attachment: phill.vcf (262 Bytes)
(Received via mailing list)
Michael wrote:
> But 2GB?  That's huge... a lot for an end-user to just sit and watch without
> clicking reload or something.
>   
Exactly - 2GB is going to be hell to upload. At least with FTP it can
use multiple ports for data transfer. And with a web interface I'd
expect users to get annoyed with clicking "back" accidentally or having
their email-client open a link using the same window. With FTP you can
"background" the process.

As for firewalls; just use an active connection rather than passive. Job
done. You'll loose the benefit of the different ports, though.
>   
>> You may have to test this, but I'd assume the maint-stream browsers support 
>> PUT via the XMLHTTPRequest javascript object.
>>     
>
> I was just doing some of this myself yesterday - doesn't seem like there's a
> way to do file uploads with AJAX, but you can easily make a form with a target
> iframe... but this is really offtopic for the nginx list.
>
>   

--

*Phillip B Oldham*
The Activity People
phill@theactivitypeople.co.uk <mailto:phill@theactivitypeople.co.uk>

------------------------------------------------------------------------

*Policies*

This e-mail and its attachments are intended for the above named
recipient(s) only and may be confidential. If they have come to you in
error, please reply to this e-mail and highlight the error. No action
should be taken regarding content, nor must you copy or show them to 
anyone.

This e-mail has been created in the knowledge that Internet e-mail is
not a 100% secure communications medium, and we have taken steps to
ensure that this e-mail and attachments are free from any virus. We must
advise that in keeping with good computing practice the recipient should
ensure they are completely virus free, and that you understand and
observe the lack of security when e-mailing us.
Posted by Michael (Guest)
on 01.08.2008 17:00
(Received via mailing list)
On Fri, Aug 01, 2008 at 15:17:20, Phillip B Oldham said...

> Exactly - 2GB is going to be hell to upload. At least with FTP it can use 
> multiple ports for data transfer. And with a web interface I'd expect users 
> to get annoyed with clicking "back" accidentally or having their 
> email-client open a link using the same window. With FTP you can 
> "background" the process.

Yeah, users would do something like that... I wonder how youtube deals 
with it?
Those uploads take a loooong time.. do users get frustrated and abort 
the
transfer?

Handling the upload in a web server or backend application wouldn't be 
hard at
all though, it's mostly just handling the user that's the problem.

> As for firewalls; just use an active connection rather than passive. Job 
> done. You'll loose the benefit of the different ports, though.

There's more to it than that... but again, not nginx related.
Posted by mike503 @ruby-forum (mike503)
on 02.08.2008 13:15
FYI: this thread did not seem to get sent to the nginx mailing list for 
some reason, oddly enough.




Michael wrote:
> On Fri, Aug 01, 2008 at 13:10:46, Phillip B Oldham said...
> 
>> Don't want to sound off, but isn't that what FTP was designed for?

I always tell people - FTP was designed for this. But this is a) not a 
solution for our stakeholders and end-users and b) a common thing coming 
up nowadays on many sites. you can't just give out FTP accounts and hope 
to have some smart system that understands when files are done, how to 
integrate them into the system, etc.

> FTP is such an awful protocol to deal with regarding firewalls, I'd much 
> rather
> go the way the original poster is and use HTTP PUT for insecure stuff, 
> and SFTP
> for secure stuff.

Exactly. and I believe since PUT uses HTTP, it can be sent over a secure 
channel as well just the same.

> But 2GB?  That's huge... a lot for an end-user to just sit and watch 
> without
> clicking reload or something.

That's why an intelligent uploader would handle the re-trying/etc. until 
it was complete. Machines don't get bored :)

> 
>> You may have to test this, but I'd assume the maint-stream browsers support 
>> PUT via the XMLHTTPRequest javascript object.
> 
> I was just doing some of this myself yesterday - doesn't seem like 
> there's a
> way to do file uploads with AJAX, but you can easily make a form with a 
> target
> iframe... but this is really offtopic for the nginx list.

I don't believe it can either. I am fine with it not supporting that as 
well.

Also if anyone has any pointers on how to get nginx to accept PUTs.. I 
was trying to proof of concept this quick with an HTML form + simple PHP 
script:

<form method="put" action="work.php">
 File: <input type="file" />
 <input type="submit" value="Submit" />
</form>

Shouldn't that work? It keeps issuing a GET:

192.168.1.2 - - [02/Aug/2008:04:08:07 -0700] GET /put/work.php? HTTP/1.1 
"200" 23 "http://192.168.1.3/put/" "Mozilla/4.0 (compatible; MSIE 6.0; 
Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 
3.0.04506.30; .NET CLR 3.0.04506.648)" "-"

Seems a bit odd. I've done this in nginx config too:

dav_methods  PUT;
create_full_put_path on;
dav_access all:r;

(nginx is compiled with the dav module)

(I figured I'd at least see a PUT request with a failure)
Posted by Michael (Guest)
on 02.08.2008 16:01
(Received via mailing list)
On Sat, Aug 02, 2008 at 13:15:27, Mike Mike said...

> <form method="put" action="work.php">
>  File: <input type="file" />
>  <input type="submit" value="Submit" />
> </form>

I've had that work with POST, though I know that's not what you're 
aiming for.

> Shouldn't that work? It keeps issuing a GET:

Just tried it myself, it does send GET rather than PUT.  Weird.

> dav_methods  PUT;
> create_full_put_path on;
> dav_access all:r;

Do you mean to have 'rw' there?
Posted by mike (Guest)
on 02.08.2008 20:18
(Received via mailing list)
On 8/2/08, Michael <nginx@thismetalsky.org> wrote:

> I've had that work with POST, though I know that's not what you're aiming for.

Yeah obviously not standard post multipart/enc-type stuff

> Just tried it myself, it does send GET rather than PUT.  Weird.

This is my issue :) Hoping someone can help say "duh stupid - you missed 
this"

> > dav_methods  PUT;
> > create_full_put_path on;
> > dav_access all:r;
>
> Do you mean to have 'rw' there?

No; at first I was thinking it would try to write and give me access
denied, or perhaps pass the data to the PHP script and not just
clobber/overwrite the script. But at least I'd see which activity it
chose first.
Posted by mike (Guest)
on 02.08.2008 20:21
(Received via mailing list)
On 8/2/08, Michael <nginx@thismetalsky.org> wrote:

Just read another email I had posted to the PHP list.

"> I can appreciate why one might imagine otherwise, but XHTML 1.x forms 
only
> support GET and POST. GET and POST are the only allowed values for the
> "method" attribute."

Sounds like it's a browser/HTML thing. So I need to use curl/another
tool to test this. Oh well :)
Posted by mike (Guest)
on 02.08.2008 20:52
(Received via mailing list)
On 8/2/08, Michael <nginx@thismetalsky.org> wrote:

> Do you mean to have 'rw' there?

Nope :)

Got it to work:

nginx access log:

192.168.1.3 - - [02/Aug/2008:11:39:53 -0700] PUT /put/work.php
HTTP/1.1 "200" 5 "-" "-" "-"

PHP curl script to test (ran from shell):

$file = 'china.txt';
$length = filesize($file);
$ch = curl_init();
$url = "http://192.168.1.3/put/work.php";
$fh = fopen($file, 'r');
curl_setopt($ch, CURLOPT_INFILE, $fh);
curl_setopt($ch, CURLOPT_INFILESIZE, $length);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_PUT, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_exec($ch);
curl_close($ch);

i believe you can use a @ or something for the input to a curl script
(instead of opening a filehandle) and it will consider that a file on
the filesystem, but i was just looking for a quick test script. it
will actually be an in-browser applet uploading this, i just wanted to
pick apart the receiving end and test it can be done.


PHP work.php script to receive (VERY crappy code, just trying
different things out):

$fname = '/tmp/uploadtest.txt';
$data = '';
$putdata = fopen('php://input', 'r');
while(!feof($putdata)) {
    $data .= fread($putdata,1024);
}
fclose($putdata);
file_put_contents($fname, $data);

exit();
Posted by Michael (Guest)
on 02.08.2008 22:07
(Received via mailing list)
On Sat, Aug 02, 2008 at 11:13:15, mike said...

> Sounds like it's a browser/HTML thing. So I need to use curl/another
> tool to test this. Oh well :)

Ah darn, that's prety limiting.  Nice that we know nginx can do this 
though :)
Posted by mike (Guest)
on 02.08.2008 22:23
(Received via mailing list)
On 8/2/08, Michael <nginx@thismetalsky.org> wrote:
> On Sat, Aug 02, 2008 at 11:13:15, mike said...
>
> > Sounds like it's a browser/HTML thing. So I need to use curl/another
> > tool to test this. Oh well :)
>
> Ah darn, that's prety limiting.  Nice that we know nginx can do this though :)

well a browser doesn't have the functionality we're looking for anyway:
- multiple file upload
- progress bar
- retransmission on timeout, failure, etc.

i just wanted to see if i could get the PHP server side piece of it as
clean and working as possible since we're still not sure what our
applet-based client will be yet.
Posted by Phillip B Oldham (Guest)
on 04.08.2008 10:41
Attachment: phill.vcf (262 Bytes)
(Received via mailing list)
Michael wrote:
> Just tried it myself, it does send GET rather than PUT.  Weird.
>   
PUT support for forms is down to the browser. IE doesn't support it, but
I'm pretty sure FF/Safari/Opera do.
--

*Phillip B Oldham*
The Activity People
phill@theactivitypeople.co.uk <mailto:phill@theactivitypeople.co.uk>

------------------------------------------------------------------------

*Policies*

This e-mail and its attachments are intended for the above named
recipient(s) only and may be confidential. If they have come to you in
error, please reply to this e-mail and highlight the error. No action
should be taken regarding content, nor must you copy or show them to 
anyone.

This e-mail has been created in the knowledge that Internet e-mail is
not a 100% secure communications medium, and we have taken steps to
ensure that this e-mail and attachments are free from any virus. We must
advise that in keeping with good computing practice the recipient should
ensure they are completely virus free, and that you understand and
observe the lack of security when e-mailing us.
Posted by mike (Guest)
on 04.08.2008 10:44
(Received via mailing list)
On 8/4/08, Phillip B Oldham <phill@theactivitypeople.co.uk> wrote:

> PUT support for forms is down to the browser. IE doesn't support it, but I'm
> pretty sure FF/Safari/Opera do.

Actually, I got my answer from the PHP list. Someone looked it up and
it's not in W3C spec for XHTML. Only GET/POST are. People want to put
PUT/DELETE in, but it's not supported.

This was just to make a simple test script for myself anyway. I wound
up just using curl via PHP to PUT the file to the work.php script to
mess around with it.
Posted by Michael (Guest)
on 04.08.2008 14:04
(Received via mailing list)
On Mon, Aug 04, 2008 at 09:30:33, Phillip B Oldham said...

>> Just tried it myself, it does send GET rather than PUT.  Weird.
>>   
> PUT support for forms is down to the browser. IE doesn't support it, but 
> I'm pretty sure FF/Safari/Opera do.

I tested with Firefox 3.

I'm sure you could use AJAX to do it (something like prototype.js), but 
in a
plain form, it doesn't seem to work.

Also, it's not real useful if the one of the most popular browsers 
doesn't
support it, yeah?
Posted by mike (Guest)
on 04.08.2008 19:03
(Received via mailing list)
On 8/4/08, Michael <nginx@thismetalsky.org> wrote:

> I tested with Firefox 3.

They went all Microsoft then and started implementing things prior to
them being W3C :P

> I'm sure you could use AJAX to do it (something like prototype.js), but in a
> plain form, it doesn't seem to work.

AJAX would be neat for this but again, can't do it in just a plain old 
browser.

> Also, it's not real useful if the one of the most popular browsers doesn't
> support it, yeah?

Correct. and it doesn't have the bells and whistles previously 
mentioned:

- multiple file upload
- progress bar
- retransmission on timeout, failure, etc.

It would be great if even single files could be re-transmitted, etc. I
am sure I could get buyoff from people if we had a fully non-applet
required method for this. But we don't right now. It still requires
Java, Flash, or some other thick client crap.
Posted by Brian Moran (Guest)
on 06.09.2008 08:46
(Received via mailing list)
mike <mike503@...> writes:


> There's a variety of Java-based PUT uploaders.
> 
> So far, we haven't found any Flash ones (we'd love to NOT use Java) -
> but there is a way to do it apparently, we just can't find anyone
> who's done it yet.

Check out SWFUPLOAD -- swfupload.org - flash based (with some 
javascript);
relatively robust; out of the box it's not persistent, but you can get 
around
that with re-submitting failed uploads using some Javascript-fu.

You might want to check jupload.sf.net, which *is* java based, however 
it
handles HTTP or FTP upload methods, can do chunk-of-a-file-at-a-time, 
and
probably could be made as robust as you'd like it to be.
Posted by mike (Guest)
on 06.09.2008 08:55
(Received via mailing list)
Thanks...

I've put together what I think is a decent solution below:
http://michaelshadle.com/2008/08/26/large-file-uploads-over-http-the-final-solution-i-think/

that jupload might be worthwhile as it sounds like it already has the
splitting idea but i haven't looked into it. this should allow for
anyone to support any file sizes (within filesystem limits) and if
something gets off the ground i am quite sure my company would sponsor
someone to write an nginx module to offload PHP-based handling of the
file uploading...
Posted by Valery Kholodkov (Guest)
on 06.09.2008 14:16
(Received via mailing list)
mike wrote:
> I've put together what I think is a decent solution below:
> http://michaelshadle.com/2008/08/26/large-file-uploads-over-http-the-final-solution-i-think/

1) multipart/form-data does not bloat the size of the file, it doesn't
encode anything. rfc 1867 doesn't explicitly say that there any encoding
should be applied;
2) the ideal solution is to have byte ranges instead of Segment ID,
since concatenation of parts is not a _scalable operation_. With byte
ranges the server will be able to put the part into the proper place in
the file, while leaving other parts empty. I.e. if I have two parts
with byte ranges 0-19999/80000 and 40000-59999/80000, I can lseek to the
second part and start writing two parts simultaneously:

|<-- part1 0-19999/80000 -->|<-- zeroes 000000 -->|<-- part2 
40000-59999/80000 -->|
Posted by Kon Wilms (Guest)
on 06.09.2008 17:43
(Received via mailing list)
On Fri, Sep 5, 2008 at 11:48 PM, mike <mike503@gmail.com> wrote:
> Thanks...
>
> I've put together what I think is a decent solution below:
> http://michaelshadle.com/2008/08/26/large-file-uploads-over-http-the-final-solution-i-think/

Interesting. This is exactly what we have been doing to get files
transmitted over satellite for many years ;-). Ofcourse the problem
there is one of no ACKs or NACKs during the transmission -- they must
be batched up and sent via backchannel after the transmission has
ended, for retransmit. If this is something that interests you I would
suggest looking at protocols like NORM.

Cheers
Kon
Posted by mike (Guest)
on 06.09.2008 23:22
(Received via mailing list)
On Sat, Sep 6, 2008 at 4:45 AM, Valery Kholodkov
<valery+nginxen@grid.net.ru> wrote:

> 1) multipart/form-data does not bloat the size of the file, it doesn't
> encode anything. rfc 1867 doesn't explicitly say that there any encoding
> should be applied;

that's weird. i thought someone said it bloats it. i will have to
update that next time i post about this.

> 2) the ideal solution is to have byte ranges instead of Segment ID, since
> concatenation of parts is not a _scalable operation_. With byte ranges the
> server will be able to put the part into the proper place in the file, while
> leaving other parts empty. I.e. if I have two parts
> with byte ranges 0-19999/80000 and 40000-59999/80000, I can lseek to the
> second part and start writing two parts simultaneously:
>
> |<-- part1 0-19999/80000 -->|<-- zeroes 000000 -->|<-- part2
> 40000-59999/80000 -->|

The reason I decided segments is due to being able to transfer
multiple segments in parallel, and I don't know enough about server
side code and shared filesystems to know if it would work properly
over NFS or something else. I am thinking of a solution that has no
"it may not work" type of restrictions.

If only one file can be sent at a time, then I was thinking PHP (since
this was a first attempt into it, and I only know PHP) can seek to the
specific byte range; however, being able to split up the file into
segments and send them at will allows for multiple segments to be
uploaded for the same file and does not have any NFS/locking risks.

If you want to code an extension that does this cleaner and uses
byteranges that will be safe over a network filesystem like NFS that
works for me :) I only know PHP, and have assumptions based on how
other things do it.

A 2 gig file at 128k chunks (segments) would wind up being 2000 * 8 =
16000 chunks. Thats a lot of files. I started thinking of making
"superchunks" which would be groupings of 100 chunks or something, so
after 100 chunks (in a row) were successful it would glue those
together, and reduce the number of files...

If this idea at least sounds viable, I think I could scrap together a
decent amount of cash from my side business and my company to fund
this. It would have to support operations safely over NFS, CIFS, or
single servers (so a local /tmp file wouldn't work for NFS, since the
requests could be sent to any of the webservers, so it would have to
be on a shared directory, which should be user configurable)

I suppose client-side isn't too hard to seek throughout a file as it
doesn't have to worry about odd locking issues and writing. It would
be great if the server end could be created in a way that it could be
come an "unofficial" standard on how to upload large files or with
unreliable or slow connections.

It would also take care of progress bar stuff as it could give
feedback when chunks get completed back to the client, and the client
knows how fast it is sending data... so during a chunk it would be
relying on it's own internal transfer stats, and it would be able to
confirm up to byte 70000 is completed on the server for example...

Also, there's an issue of garbage collection. A job would have to
clean out the [shared] temp directory after a while - I thought
something like 4 days would be nice [user configurable is best],
because a 2 gig file could take a long time and people might have to
resume it over the course of a couple days, but any longer we'd have
to assume it's an orphan file that won't be resumed again.

If you're interested in this, I would love to have someone as
experienced as you - who has already dealt with handling file uploads
and created nginx modules! Let me know, feel free to contact me off
list. We might have to work out some more specifics, and you might
want to know how much $$ - I'd have to ask at work what they would pay
for it, but I'd pledge $500 out of my own pocket. I don't believe any
other webserver or anything out there has anything like this (besides
maybe some thick client apps with specific servers that only handle
file uploads...)

Ideally, this would be something that other people could create
modules for Apache, etc. as well and it could be adopted by browsers
directly and alleviate the need for thick Java/Flash/etc apps. If it's
done in a "standard" enough way to be re-creatable on the client and
the server...

I'd love to hear any thoughts, opinions, get any code going, etc. I've
actually got a PHP version of the server component that I think is
actually functional (with minimal amounts of code, surprisingly) but
don't have a client to test it with yet. Was going to create a PHP
client to test it too.
Posted by mike (Guest)
on 06.09.2008 23:22
(Received via mailing list)
On Sat, Sep 6, 2008 at 8:35 AM, Kon Wilms <konfoo@gmail.com> wrote:

> Interesting. This is exactly what we have been doing to get files
> transmitted over satellite for many years ;-). Ofcourse the problem
> there is one of no ACKs or NACKs during the transmission -- they must
> be batched up and sent via backchannel after the transmission has
> ended, for retransmit. If this is something that interests you I would
> suggest looking at protocols like NORM.

Well, I'd want to speak standard HTTP, just like anything else, and I
think it would be easy. Doesn't have to get into low-level TCP, as the
chunks should be done in such a way that it's like any normal HTTP
request. This will alleviate firewall issues, max upload size issues
(both webserver and PHP/scripting level limits), etc.

Just want to make sure the client side is memory efficient and doesn't
load the entire file into memory to seek byte ranges, for example. Not
sure if that can be done or not, or if it's language specific, etc.

But it is a good idea, no? I've been struggling to find any products
for a long time, and nothing is out there that doesn't require ugly
applets and server configuration changes and/or separate servers etc.
Posted by mike (Guest)
on 08.09.2008 23:22
(Received via mailing list)
On Sat, Sep 6, 2008 at 4:45 AM, Valery Kholodkov
<valery+nginxen@grid.net.ru> wrote:

> |<-- part1 0-19999/80000 -->|<-- zeroes 000000 -->|<-- part2
> 40000-59999/80000 -->|

Does anyone know what would be easiest to add in to nginx? I assume
most would require external libraries and I would like to keep nginx's
footprint low and not require an excessive amount of external
libraries.

The idea here would be the client encodes using one of these, and the
server will decode, to ensure safe transfer of the file (please let me
know Valery if you think this is totally not needed, I expect every
segment to be checksummed anyway, so maybe it doesn't need to be)

uuencode/uudecode
base64 encoding/decoding
yenc (www.yenc.org)
? any other encoding methods ?
Posted by Grzegorz Nosek (gnosek)
on 09.09.2008 09:52
(Received via mailing list)
On pon, wrz 08, 2008 at 02:12:30 -0700, mike wrote:
> uuencode/uudecode
> base64 encoding/decoding
> yenc (www.yenc.org)
> ? any other encoding methods ?

UU/base64 are quite similar (both are rather simple and expand the
encoded text by 33%), yenc I haven't heard of before. Do you *need* to
encode the chunks in any way? I'd probably send them raw with some
checksum (CRC32, MD4, MD5, SHA-something, depending on the application)
in headers.

BTW, if you keep the chunk size as a multiple of page size (usually 4K,
though you may encounter 8K and maybe more), you should be able to
mmap() chunks of the file for minimum overhead.

Best regards,
 Grzegorz Nosek
Posted by mike (Guest)
on 09.09.2008 10:34
(Received via mailing list)
On Tue, Sep 9, 2008 at 12:43 AM, Grzegorz Nosek
<grzegorz.nosek@gmail.com> wrote:

> UU/base64 are quite similar (both are rather simple and expand the
> encoded text by 33%), yenc I haven't heard of before. Do you *need* to
> encode the chunks in any way? I'd probably send them raw with some
> checksum (CRC32, MD4, MD5, SHA-something, depending on the application)
> in headers.
>
> BTW, if you keep the chunk size as a multiple of page size (usually 4K,
> though you may encounter 8K and maybe more), you should be able to
> mmap() chunks of the file for minimum overhead.

Thanks for the pointers - Valery would be implementing this for nginx
(unless someone else did) but he's already got quite a good amount of
experience and maybe I can talk him into it :)

After thinking about it more, it probably doesn't need to be encoded,
as long as it is checksummed. I suppose headers can be used for that.

Also it would be neat if the server advertised it's advanced upload
capability, or something so that the client side applet (and possibly
built-in to webservers some day) would know "hey, if I upload to this
host, I can use the intelligent transfer system"
Posted by Phillip B Oldham (Guest)
on 09.09.2008 10:45
Attachment: phill.vcf (273 Bytes)
(Received via mailing list)
mike wrote:
> Also it would be neat if the server advertised it's advanced upload
> capability, or something so that the client side applet (and possibly
> built-in to webservers some day) would know "hey, if I upload to this
> host, I can use the intelligent transfer system"
>   
And if you provide enough documentation on it, someone will probably
write a firefox plug-in to take advantage of the system with the default
<input type="file" /> control. I might even take a whack at it if I've
got the time.
--

*Phillip B Oldham*
The Activity People
phill@theactivitypeople.co.uk <mailto:phill@theactivitypeople.co.uk>

------------------------------------------------------------------------

*Policies*

This e-mail and its attachments are intended for the above named
recipient(s) only and may be confidential. If they have come to you in
error, please reply to this e-mail and highlight the error. No action
should be taken regarding content, nor must you copy or show them to 
anyone.

This e-mail has been created in the knowledge that Internet e-mail is
not a 100% secure communications medium, and we have taken steps to
ensure that this e-mail and attachments are free from any virus. We must
advise that in keeping with good computing practice the recipient should
ensure they are completely virus free, and that you understand and
observe the lack of security when e-mailing us.
Posted by mike (Guest)
on 09.09.2008 11:35
(Received via mailing list)
That would be the plan. First gotta get a working implementation done
and the kinks in the "protocol" but you can bet I will push the idea/
protocol/whatever to every browser and web server to get more adoption.

On Sep 9, 2008, at 1:37 AM, Phillip B Oldham 
<phill@theactivitypeople.co.uk
Posted by Kon Wilms (Guest)
on 09.09.2008 18:18
(Received via mailing list)
The only way to do this would be to put some ammunition behind it - an
IETF draft.

Cheers
Kon
Posted by mike (Guest)
on 09.09.2008 21:48
(Received via mailing list)
On Tue, Sep 9, 2008 at 8:32 AM, Kon Wilms <konfoo@gmail.com> wrote:
> The only way to do this would be to put some ammunition behind it - an
> IETF draft.
>
> Cheers
> Kon

Of course.

Although I think it might be good to have some real-world traction and
examples. I also don't have time to try to rally official support, I
need it done sooner rather than later. Once it's working then i might
try to bug people about getting something added to a spec for more
robust HTTP uploading (and hopefully adopt our key learnings /
"protocol")