Nginx git repository

[gnosek]

Hello all,

I have finally came around to publishing my git repository. You can
access it at http://git.localdomain.pl/

It currently isn’t cloneable (I haven’t set up a git-daemon, only
gitweb) – I hope to enable it soon, if time permits.

You can find there all the patches (admittedly trivial) that we’re
currently using. The main areas of interest are:

• CGI support (from nginx’ side it’s only a few rewrite flags for
  workable PATH_INFO support, i.e. -F, -E and -X)
• security in a shared environment
• nested locations – this is currently WIP.

Notes:

1. I’m only tracking the stable branch (0.5.x)
2. All the commits in `vanilla’ branch are (of course) not mine. I
   didn’t bother to put the real author in there because the repo started
   as an internal one, but if anybody finds this misleading or otherwise
   unappropriate, please let me know and I’ll change it.
3. All my code is based on my (limited) understanding of nginx
   internals and comes without any warranty. That being said, we’re using
   it in production without problems. If you encounter any issues while
   running our patched version, I’d be glad to hear about it.
4. The CGI+security patches have a performance impact due to many stat()
   calls.
5. The nested location support is grossly incomplete.

Best regards,
Grzegorz N.

[gnosek]

Grzegorz N. ha scritto:

Hello all,

I have finally came around to publishing my git repository. You can
access it at http://git.localdomain.pl/

I think that it can be a good idea to provide a git and Mercurial
repositories for the stable and development branches of nginx, so
developers can more easily work on nginx.

As an example, Mercurial offer the MQ extension, that integrates the
features of quilt (used for Linux kernel development), so you can keep
patches well separed from nginx core.

[…]

Regards Manlio P.

[gnosek]

2007/10/5, Manlio P. [email protected]:

I think that it can be a good idea to provide a git and Mercurial
repositories for the stable and development branches of nginx, so
developers can more easily work on nginx.

Well, the stable branch is in my repo and I may just as well keep the
development version there for the community at large but I won’t
probably use the dev. version for the time being.

If you want (and accept my repo as a pseudo-official one till a really
official one appears), I have absolutely no problem with pulling stuff
from you.

As an example, Mercurial offer the MQ extension, that integrates the
features of quilt (used for Linux kernel development), so you can keep
patches well separed from nginx core.

I see you are into Mercurial and frankly I have never used it (git
stole my heart ;)) but even raw git without quilt etc. can serve
pretty well for tracking different patch sets – just set up lots of
branches and merge every now and then for example.

In my repo, the vanilla branch contains nothing but nginx releases
(plus a single hotfix from the mailing list) so it should be a pretty
good base for custom patches. I update it with every nginx release but
frankly, I’d love a more detailed history.

Igor, is there a possibility of publishing your repo? ISTR you’re
using Subversion – if so, I’d need no commit access, just the ability
to pull changesets and apply them to my tree.

Manlio, do you have some documentation for your mod_wsgi? I’m pretty
interested in it and would like to try it out. Could you provide some
example configuration (both from nginx’ side and the app’s side)? Is
the module safe enough for general consumption? I cannot really afford
bringing the web server down in flames but if the module is stable
(even if incomplete), I’d give it a shot.

Best regards,
Grzegorz N.

[gnosek]

2007/10/5, Manlio P. [email protected]:

If you want (and accept my repo as a pseudo-official one till a really
official one appears), I have absolutely no problem with pulling stuff
from you.

Thanks, but I’m using Mercurial (and I can setup a Mercurial Nginx
unofficial repository on my server).

Do you know of some gateway interface between git and hg? It might prove
useful.

http://hg.mperillo.ath.cx/nginx/mod_wsgi/file/tip/examples/
Thanks a lot, will look into these. Did you try running bigger things
a’la Django under mod_wsgi?

At every commit I try to do as many tests as I can, and I execute a
stress test using Apache Bench with many request and high concurrency
levels.

That’s pretty reassuring Thanks.

The module is still in development and not yet stabilized.
Tests are always really appreciated, but I can suggest you to wait until
I will implement the full WSGI 1.0 spec, if yuo want start to use it.

OK, I think I’ll give it a try anyway.

Best regards,
Grzegorz N.

[gnosek]

Grzegorz N. ha scritto:

I don’t know, but there should be an easy way to keep in sync a
Mercurial repository with a Git repository, since there is a Mercurial
clone of the whole kernel repository.

Thanks a lot, will look into these. Did you try running bigger things
a’la Django under mod_wsgi?

Not yet.

However you should know that running a “big” application like Django can
blocks the nginx server, especially when using a client server database
over a slow connection.

In the future I will try to add threading support to support “legacy”
applications.

I have written mod_wsgi for best support fast and asynchronous
applications.

OK, I think I’ll give it a try anyway.

I’ll wait for a review, thanks.

Do not forget that the current revision of mod_wsgi implements the WSGI
2.0 draft and not the current WSGI 1.0.

Best regards,
Grzegorz N.

Regards Manlio P.

[gnosek]

El Fri, 5 Oct 2007 19:02:18 +0200
“Grzegorz N.” [email protected] escribió:

Do you know of some gateway interface between git and hg? It might
prove useful.

I have used Tailor (http://wiki.darcs.net/index.html/Tailor)
occasionally to convert Subversion repositories to Bazaar branches and
it worked well for me. Theoretically, both Git and Mercurial are
supported, too. You may want to give it a try.

Thanks a lot, will look into these. Did you try running bigger things
a’la Django under mod_wsgi?

I tried to run Trac 0.10.4, MoinMoin 1.5.8 and the Bazaar 0.90 smart
server, without luck. As both are WSGI 1.0 applications, I did the
quick-and-dirt trick of wrapping them so they are exposed to Nginx
as WSGI 2.0 ones. I need to further investigate, but I suppose they
yielding more than a single result (which currently mod_wsgi does not
support, AFAIK).

The module is still in development and not yet stabilized.
Tests are always really appreciated, but I can suggest you to wait
until I will implement the full WSGI 1.0 spec, if yuo want start to
use it.

Just for the sake of completeness: I’ve built and executed the
included tests successfully in the following configurations:

• Apple iBook G3 (PowerPC 750FX), with Gentoo Linux.
• Dell Dimension C521 (AMD Athlon X2), with Gentoo linux.

I am using Nginx 0.6.13 here, changing usage of ngx_http_discard_body
to ngx_http_discard_request_body as I have said before. Next time I
will try to build it in a fresh NetBSD installation I want to do in an
old Pentium II machine.

Best regards,

[gnosek]

Grzegorz N. ha scritto:

2007/10/5, Manlio P. [email protected]:

I think that it can be a good idea to provide a git and Mercurial
repositories for the stable and development branches of nginx, so
developers can more easily work on nginx.

Well, the stable branch is in my repo and I may just as well keep the
development version there for the community at large but I won’t
probably use the dev. version for the time being.

I too use only the stable branch.

If you want (and accept my repo as a pseudo-official one till a really
official one appears), I have absolutely no problem with pulling stuff
from you.

Thanks, but I’m using Mercurial (and I can setup a Mercurial Nginx
unofficial repository on my server).

[…]

Manlio, do you have some documentation for your mod_wsgi?

For the documentation:
(very incomplete)
http://wiki.codemongers.com/NginxNgxWSGIModule

For the examples:
http://hg.mperillo.ath.cx/nginx/mod_wsgi/file/tip/examples/

I’m pretty
interested in it and would like to try it out. Could you provide some
example configuration (both from nginx’ side and the app’s side)? Is
the module safe enough for general consumption?

At every commit I try to do as many tests as I can, and I execute a
stress test using Apache Bench with many request and high concurrency
levels.

I cannot really afford
bringing the web server down in flames but if the module is stable
(even if incomplete), I’d give it a shot.

The module is still in development and not yet stabilized.
Tests are always really appreciated, but I can suggest you to wait until
I will implement the full WSGI 1.0 spec, if yuo want start to use it.

Best regards,
Grzegorz N.

Regards Manlio P.

[gnosek]

Adrian P. de Castro ha scritto:

support, AFAIK).

The current revision now fully supports WSGI 2.0, but does not yet
support asynchronous writes (I’m adding support right now).

Can you send me the logs?
The log level should be “info”;

[…]

Thanks and regards Manlio P.

[gnosek]

On Fri, Oct 05, 2007 at 02:35:23PM +0200, Grzegorz N. wrote:

• nested locations – this is currently WIP.

What exatly this means ? nginx already has support of nested locations,
though incomplete.

[gnosek]

On Fri, Oct 05, 2007 at 06:11:00PM +0200, Grzegorz N. wrote:

In my repo, the vanilla branch contains nothing but nginx releases
(plus a single hotfix from the mailing list) so it should be a pretty
good base for custom patches. I update it with every nginx release but
frankly, I’d love a more detailed history.

Igor, is there a possibility of publishing your repo? ISTR you’re
using Subversion – if so, I’d need no commit access, just the ability
to pull changesets and apply them to my tree.

I can place weekly svn repo dump. However, I’m using svn since 0.4.0 and
all versions before are in tarballs. I plan to import old tarballs
in repo, but all revison numbers will shift. Actually year ago
I had made automated procedure that imports all tarballs from 0.1.0 to
0.3.x,
but I want to import pre-0.1.0 tarballs too.

[gnosek]

2007/10/5, Igor S. [email protected]:

On Fri, Oct 05, 2007 at 02:35:23PM +0200, Grzegorz N. wrote:

• nested locations – this is currently WIP.

What exatly this means ? nginx already has support of nested locations,
though incomplete.

I’m not reinventing nested locations – I have noticed that nginx
supports them partially and I want them to be supported completely
(note that I have replaced my previous code with something hopefully
less braindead).

The patch set currently consists of the top 5 commits.

The first two lay some foundations, namely they store cf and cf->ctx
inside core module’s loc_conf for further modules’ reference. This
allows the merge stage to access other modules’ configuration (removes
the limitation commented in ngx_http_config.h). This is required e.g.
for the fastcgi module which needs to set clcf->handler.

It also converts the script engine from overriding the configuration
to merging it at run time (for if {} blocks). Merging it at
configuration parsing/merging time would be nicer (no request-time
overhead) but would require more intrusive changes to the rewrite
module. Maybe soon.

The other three build upon this foundation to support configuration
inheritance of fastcgi, proxy and rewrite modules.

The net result is that a simple configuration of:

location / {
location /secret {
auth_basic “Secret”;
auth_basic_user_file /etc/nginx/htpasswd;
}
if (!-f $request_filename) {
fastcgi_pass (somewhere);
}
}

works as expected (all nonexistent files are passed to fastcgi, even
under /secret, and also all URLs under /secret require authentication,
even dynamic ones).

Also, please consider applying to your tree:

http://git.localdomain.pl/?p=nginx.git;a=commitdiff;h=0ec0c91de07399f4c2b8f43cd88e0fa459a22a79
(Don’t add double WWW-Authenticate headers)

http://git.localdomain.pl/?p=nginx.git;a=commitdiff;h=9cac24c18d4eb4a2946f9722bb021087d660a42f
(Remember original error code when an error occurs while serving error
pages)

The first one is a simple bugfix I have reported some time ago. The
other one makes error handling more intuitive when error pages are
missing, i.e. if the error page cannot be retrieved, report the
original error which caused the page to be looked up in the first
place, instead of the error while retrieving the error page (yes, this
sounds convoluted). For example:

error_page 403 /nonexistent.html;
deny all;

would serve the 404 error (as that’s the last one encountered). With
this patch applied, it properly returns 403.

Best regards,
Grzegorz N.

[gnosek]

Adrian P. ha scritto:

Hello,

First of all, sorry for the late reply, I have been quite busy lately.

No problem.
I have just released the new version of mod_wsgi, and it took me a lot
of time!

Now it should be easy to add support for WSGI 1.0, I hope to release the
next version on the next weekend.

[…]

Can you send me the logs?
The log level should be “info”;

I will re-run the tests with log level set to “info”, but I will first
update to the latest revision of the mod_wsgi module.

Ok, thanks.

def __init__(self):
    context = Context()
    results = application(environ, context)
    return context.response, context.headers, results
return wrapper

This seems ok.

It is not fully WSGI compliant, however, since a WSGI application can
call start_response inside the returned generator, but this should not
be a problem since common WSGI applications just returns a list as
result.

I will do some tests.

def returnone(application):
“”“Makes a 2.0 app return only one element as result”""
def wrapper(environ):
response, headers, results = application(environ)
return response, headers, “”.join(results)

Here you should do “”.join(list(results)), since results can be a
generator.

However now returnone is no more necessary.

Cheers,

Regards and thanks for the support
Manlio P.

[gnosek]

Manlio P. ha scritto:

Now it should be easy to add support for WSGI 1.0, I hope to release the

I am using the
following Python snippet to adapt WSGI 1.0 apps to 2.0, they are a set
of decorators for WSGI applications, I hope they are ok although I have
not read the WSGI standard in depth:

Here is the version of wsgi1to2 that I have used:
class Context(object):
def init(self):
self.response = None
self.headers = ()
self.buf = []

 def __call__(self, response, headers):
     self.response = response
     self.headers  = headers

     return self.write

 def write(self, buf):
     from warnings import warn

     warn('write callable is deprecated', DeprecationWarning)
     self.buf.append(buf)

def wsgi1to2(application):
“”“Converts a 1.0 app into a 2.0 app
“””

 import sys

 # Some applications require argv
 sys.argv = ['wsgi application']

 def wrapper(environ):
     context = Context()
     results = application(environ, context)

     if context.buf:
         buf = context.buf + list(results)
         return context.response, context.headers, ''.join(buf)

     return context.response, context.headers, results

 return wrapper

I have managed to run the Mercurial web application.
It is very slow on mod_wsgi, since it uses the write callable and is CPU
bound.

With Apache Bench (n = 1000, c = 50), I get:

• 27 requests per seconds with the builtin server
• 20 requests per seconds with mod_wsgi (and 2 worker processes)

However with mod_wsgi my system load is above 2, with the builtin server
it is over 6.

Regards Manlio P.

[gnosek]

Manlio P. ha scritto:

Now it should be easy to add support for WSGI 1.0, I hope to release the
next version on the next weekend.

Adrian, if you are still interested, the latest changeset of mod_wsgi
now finally supports WSGI 1.0.

There is a limitation, however: when the write callable cannot be
entirely send the passed-in string to the client, it raises an IOError
exception.

I have tested the implementation with Mercurial, and it works (and now
it is more efficient then the builtin server!).

One more thing: you should not enable the wsgi_enable_subinterpreters
directive, since the subinterpreters don’t have argv in the sys module
and Mercurial uses it.

[…]

Regards Manlio P.

[gnosek]

Hello,

First of all, sorry for the late reply, I have been quite busy lately.

On Fri, 05 Oct 2007 19:43:40 +0200
Manlio P. [email protected] wrote:

yielding more than a single result (which currently mod_wsgi does
not support, AFAIK).

The current revision now fully supports WSGI 2.0, but does not yet
support asynchronous writes (I’m adding support right now).

Can you send me the logs?
The log level should be “info”;

I will re-run the tests with log level set to “info”, but I will first
update to the latest revision of the mod_wsgi module. I am using the
following Python snippet to adapt WSGI 1.0 apps to 2.0, they are a set
of decorators for WSGI applications, I hope they are ok although I have
not read the WSGI standard in depth:

---- ----

class Context(object):
slots = (“response”, “headers”)

def __init__(self):
    self.response = None
    self.headers  = ()
def __call__(self, response, headers):
    self.response = response
    self.headers  = headers

def wsgi1to2(application):
“”“Converts a 1.0 app into a 2.0 app”“”
def wrapper(environ):
context = Context()
results = application(environ, context)
return context.response, context.headers, results
return wrapper

def returnone(application):
“”“Makes a 2.0 app return only one element as result”“”
def wrapper(environ):
response, headers, results = application(environ)
return response, headers, “”.join(results)
return wrapper

def fornginx(application):
“”“Applies the two above decorators, which is needed for
a 1.0 application to work inside Nginx.
“””
return returnone(wsgi1to2(application))

---- ----

Cheers,

[gnosek]

Grzegorz N. wrote:

Hello all,

I have finally came around to publishing my git repository. You can
access it at http://git.localdomain.pl/

It currently isn’t cloneable (I haven’t set up a git-daemon, only
gitweb) – I hope to enable it soon, if time permits.

Hi Grzegorz,

It would be great if you could setup a copy of your repo that you push
your changes to at GitHub.com. There are several repositories there
that are trying to track your project but none of them are up to date (I
guess they are doing it manually).

If you could push your changes there, in addition to your main repo that
drives your gitweb, it would be awesome since others can then simply
fork off of your repo and track you changes or contribute to you more
easily.

I can give you an invite to GitHub if you need one.

Thanks,

Glenn

[gnosek]

Grzegorz N. wrote:

Oh, I’m that famous already? My repo is cloneable btw so you don’t
need to apply commit diffs by hand.

Sure! Lots of people are tracking progress of the fair balancer. The
cool people of course.

What is the git URL for cloning? I couldn’t figure it out from your
gitweb site and I tried a bunch of variations. You have previously said
that it was gitweb viewing only as you had not setup a git daemon. Did
that change?

If you could push your changes there, in addition to your main repo that
drives your gitweb, it would be awesome since others can then simply
fork off of your repo and track you changes or contribute to you more
easily.

No problem at all. However, I’ll probably drop the fair balancer from my
nginx tree and distribute it as a separate repo, as soon as I manage to
do it without losing all the commit history (and find the time of
course).

Either way, you can have multiple repositories on GitHub and they are
free for OpenSource projects.

I can give you an invite to GitHub if you need one.

That means I probably need one.

I sent one to root [at] localdomain dot P L. GitHub is turning out to
be a great way to collaborate on code.

I look forward to seeing your repo up there.

Glenn

PS - I am running my site using your fair proxy and several ‘Thin’
servers in front of my Rails app with Nginx fair proxying across several
UNIX domain sockets that thin is listening on (instead of TCP). Working
like a charm so far.

[gnosek]

Hi,

On Thu, Mar 20, 2008 at 06:46:16AM +0100, Glenn Rempe wrote:

Hi Grzegorz,

It would be great if you could setup a copy of your repo that you push
your changes to at GitHub.com. There are several repositories there
that are trying to track your project but none of them are up to date (I
guess they are doing it manually).

Oh, I’m that famous already? My repo is cloneable btw so you don’t
need to apply commit diffs by hand.

If you could push your changes there, in addition to your main repo that
drives your gitweb, it would be awesome since others can then simply
fork off of your repo and track you changes or contribute to you more
easily.

No problem at all. However, I’ll probably drop the fair balancer from my
nginx tree and distribute it as a separate repo, as soon as I manage to
do it without losing all the commit history (and find the time of
course).

I can give you an invite to GitHub if you need one.

That means I probably need one.

Best regards,
Grzegorz N.

[gnosek]

On Thu, Mar 20, 2008 at 05:52:57PM +0100, Glenn Rempe wrote:

Grzegorz N. wrote:

Oh, I’m that famous already? My repo is cloneable btw so you don’t
need to apply commit diffs by hand.

Sure! Lots of people are tracking progress of the fair balancer. The
cool people of course.

Heh

What is the git URL for cloning? I couldn’t figure it out from your
gitweb site and I tried a bunch of variations. You have previously said
that it was gitweb viewing only as you had not setup a git daemon. Did
that change?

Yeah, I didn’t know anybody actually cared so I didn’t publish the URL.

You can grab it at git://git.localdomain.pl/nginx.git

No problem at all. However, I’ll probably drop the fair balancer from my
nginx tree and distribute it as a separate repo, as soon as I manage to
do it without losing all the commit history (and find the time of
course).

Either way, you can have multiple repositories on GitHub and they are
free for OpenSource projects.

Great, but I’d like to split the repos before half the world clone it.

I sent one to root [at] localdomain dot P L. GitHub is turning out to
be a great way to collaborate on code.

Looks like the invite bounced on some mail filter Could you please
resend it?

I look forward to seeing your repo up there.

Glenn

PS - I am running my site using your fair proxy and several ‘Thin’
servers in front of my Rails app with Nginx fair proxying across several
UNIX domain sockets that thin is listening on (instead of TCP). Working
like a charm so far.

Great to hear that Just make sure you always have the latest
snapshot as there are still bugs lurking and I’m taking them out one by
one

Best regards,
Grzegorz N.

[gnosek]

On Thu, Mar 20, 2008 at 06:57:11PM +0100, Glenn Rempe wrote:

Grzegorz N. wrote:
I have not tried this myself, but this may help:

Yaakov Nemoy: How to split a repository in git

Thanks a lot, bookmarked

Thats exactly why I want it all available in Git! I am deploying
your Nginx Git repo to my remote server using Capistrano. I then do a
git pull, compile, install on the remote server from within that Git
repo. Makes it easy to update my remote servers with any updates.

Wow. A brave soul you are BTW this looks interesting, would you care
to provide a brief howto?

Best regards,
Grzegorz N.