http://wiki.nginx.org/Redmine

Hello,

webpage Redmine | NGINX has some security problems:

  1. All redmine config files are available for anybody in internet,
    for example: https://redmine.example.com/config/database.yml
    contains in plain text login and password for database connection.

  2. wiki.nginx.org use nginx/1.5.12 with known security vulnerabilities

  3. Unsafe variable $http_host was used instead of safe one $host

===================================================================

Content of page Redmine | NGINX for now:

[…]

This is very nearly a drop in configuration. The only thing you should
need to change will be the root location, upstream servers, and the
server name.

upstream redmine {
server 127.0.0.1:8000;
server 127.0.0.1:8001;
server 127.0.0.1:8002;
}

server {
server_name redmine.DOMAIN.TLD;
root /var/www/redmine;

     location / {
             try_files $uri @ruby;
     }

     location @ruby {
             proxy_set_header  X-Real-IP  $remote_addr;
             proxy_set_header  X-Forwarded-For

$proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_read_timeout 300;
proxy_pass http://redmine;
}
}

[…]

===================================================================


Best regards,
Gena

On Sun, Mar 08, 2015 at 04:58:05PM +0200, Gena M. wrote:

Hi there,

webpage Redmine | NGINX has some security problems:

  1. All redmine config files are available for anybody in internet,
    for example: https://redmine.example.com/config/database.yml
    contains in plain text login and password for database connection.

I don’t think that one is an nginx problem.

From reading the redmine docs, it looks like the contents of the “root”
directive directory should be whatever is in the distributed redmine
public/ directory; not the entire installation including configuration.

And if /var/www/redmine does just have the public/ contents and the
upstream servers reveal secret information, that would be their problem
and not nginx’s, I think.

  1. wiki.nginx.org use nginx/1.5.12 with known security vulnerabilities

  2. Unsafe variable $http_host was used instead of safe one $host

I’m not sure how $http_host is less safe than $host. It is proxy_pass’ed
to the “real” redmine server as the Host header. That server must be
able to handle it safely anyway, no?

f

Francis D. [email protected]

On 08.03.2015 22:50, Francis D. wrote:

webpage Redmine | NGINX has some security problems:

  1. All redmine config files are available for anybody in internet,
    for example: https://redmine.example.com/config/database.yml
    contains in plain text login and password for database connection.

I don’t think that one is an nginx problem.

Yes, this is not nginx problem. This is nginx configuration problem,
which provided at wiki.nginx.org as “drop in configuration” for redmine.

From reading the redmine docs, it looks like the contents of the “root”
directive directory should be whatever is in the distributed redmine
public/ directory; not the entire installation including configuration.

I am talk about configuration recommended
at webpage Redmine | NGINX
not about “reading the redmine docs”.

And if /var/www/redmine does just have the public/ contents and the
upstream servers reveal secret information, that would be their problem
and not nginx’s, I think.

root /var/www/redmine;
try_files $uri @ruby;

Request https://redmine.example.com/config/database.yml will be
processed by nginx, because file /var/www/redmine/config/database.yml
exists. For details - see manual about try_files directive in nginx.

  1. Unsafe variable $http_host was used instead of safe one $host

I’m not sure how $http_host is less safe than $host. It is proxy_pass’ed
to the “real” redmine server as the Host header. That server must be
able to handle it safely anyway, no?

Such configuration allow to spoof nginx built-in server selection rules.
because nginx will use server name from request line, but will provide
to upstream completely different server name, from Host request header.

So, $host must be used always with proxy_pass instead of $http_host.


Best regards,
Gena

On 09.03.2015 16:48, Edho A. wrote:

From reading the redmine docs, it looks like the contents of the “root”
directive directory should be whatever is in the distributed redmine
public/ directory; not the entire installation including configuration.

It’s a public wiki, not some official documentation. If there’s error
you can just go ahead and change it.

And it will be silent fixing of security vulnerability in nginx
configuration recommended for redmine, so all previous redmine
instances, configured by this manual will be vulnerable.

I prefer to report about this vulnerability in nginx mail list,
so all people who configure redmine by this recommended manual
can fix this security vulnerability in their own redmine installs.

===============================================================

Also, I can’t fix security vulnerabilities in nginx/1.5.12
used at site http://wiki.nginx.org/ and can’t contact with
Cliff W. by e-mail [email protected] and other e-mails.


Best regards,
Gena

Hi Gena,

I’m happy to have you update the wiki now that you’ve reported your
concerns.

Do you have an account on the wiki? If not, please request one and let
me know via email at [email protected] and we’ll get you set up with
privileges to edit the page.

Sarah

On Mon, Mar 9, 2015 at 11:44 PM, Gena M. [email protected] wrote:

Yes, this is not nginx problem. This is nginx configuration problem,
which provided at wiki.nginx.org as “drop in configuration” for redmine.

From reading the redmine docs, it looks like the contents of the “root”
directive directory should be whatever is in the distributed redmine
public/ directory; not the entire installation including configuration.

It’s a public wiki, not some official documentation. If there’s error
you can just go ahead and change it.

On Mon, Mar 09, 2015 at 04:44:05PM +0200, Gena M. wrote:

On 08.03.2015 22:50, Francis D. wrote:

Hi there,

webpage Redmine | NGINX has some security problems:

  1. All redmine config files are available for anybody in internet,
    for example: https://redmine.example.com/config/database.yml
    contains in plain text login and password for database connection.

I don’t think that one is an nginx problem.

Yes, this is not nginx problem. This is nginx configuration problem,
which provided at wiki.nginx.org as “drop in configuration” for redmine.

I think that you are incorrect in your understanding of it as an nginx
configuration problem, and as a drop-in configuration.

From reading the redmine docs, it looks like the contents of the “root”
directive directory should be whatever is in the distributed redmine
public/ directory; not the entire installation including configuration.

I am talk about configuration recommended
at webpage Redmine | NGINX
not about “reading the redmine docs”.

But the user must have followed some documentation to install redmine in
the first place; and if they unthinkingly install it into
/var/www/redmine
they are probably doing something wrong before nginx gets involved.

I see instructions to install to /opt/redmine, and to /var/lib/redmine,
and to /usr/share/redmine, and in each case they say to do something
like

ln -s /usr/share/redmine/public /var/www/redmine

to have only the web-accessible content below /var/www/redmine.

If the user really wants to install to /var/www/redmine, then they must
modify the “root” directive (to be /var/www/redmine/public), as the
words on the wiki page already say.

I do not see this as an nginx-related security problem.

And if /var/www/redmine does just have the public/ contents and the
upstream servers reveal secret information, that would be their problem
and not nginx’s, I think.

root /var/www/redmine;
try_files $uri @ruby;

Request https://redmine.example.com/config/database.yml will be
processed by nginx, because file /var/www/redmine/config/database.yml
exists. For details - see manual about try_files directive in nginx.

The file /var/www/redmine/config/database.yml should not exist.

If the file /var/www/redmine/config/database.yml does exist and the
above nginx configuration is used, then the user will find that no part
of their redmine web-related installation will work, because all of the
images and stylesheets and javascripts are inaccessible.

Correspondingly, if the user has installed only web content below
/var/www, then using a different “root” directive will cause that
installation not to work.

  1. Unsafe variable $http_host was used instead of safe one $host

I’m not sure how $http_host is less safe than $host. It is proxy_pass’ed
to the “real” redmine server as the Host header. That server must be
able to handle it safely anyway, no?

Such configuration allow to spoof nginx built-in server selection rules.
because nginx will use server name from request line, but will provide
to upstream completely different server name, from Host request header.

It is true that $http_host is completely controlled by the client, and
$host is mostly controlled by the client. It is true that they can have
different values. I do not see that the difference is a security issue
in this case.

So, $host must be used always with proxy_pass instead of $http_host.

If the upstream server would do anything security-relevant with the
Host:
header that it gets from nginx, it would do exactly the same with the
Host: header that it would get from the client directly, no?

Also: I suspect that $http_host was there because if you run nginx
on not-port-80, using $host will probably lose that information. The
server{} has no “listen”, so it will use port 80 or 8080 depending on
the invoking user.

The config that was there (probably) works under some circumstances, and
fails under some others. It’s fine to change it to another configuration
which works under some different circumstances, but you should probably
be aware that you are failing under different circumstances too.

As as “example” config, it should be understood that it does not cover
all circumstances. And the newer example config may be more suitable
for more installations – I’ll let someone else count them, if they
care. But I don’t see how security is involved in the change.

Cheers,

f

Francis D. [email protected]

On 09.03.2015 19:25, Francis D. wrote:

It is true that $http_host is completely controlled by the client, and
$host is mostly controlled by the client. It is true that they can have
different values. I do not see that the difference is a security issue
in this case.

server {
listen 443 ssl;
server_name private.example.com;
location / {
auth_basic “closed site”;
auth_basic_user_file conf/htpasswd;
proxy_set_header Host $http_host;
proxy_pass http://backend;
}
}

server {
listen 443 ssl;
server_name public.example.com;
location / {
proxy_set_header Host $http_host;
proxy_pass http://backend;
}
}

in such configuration anybody can bypass nginx auth_basic restriction
and access content from private.example.com without any login/password:

GET https://public.example.com/top-secret.pdf HTTP/1.1
Host: private.example.com

nginx will use host name public.example.com for server selection,
and process request in second server, but send to backend
“Host: private.example.com” and relative URI in request line.

and backend will process such request as request to private.example.com

because backend server see only relative uri in request line,
and will use host name from Host: request header in this case.

=======================================================================

for proxy_pass such bug can be fixed just by using always
$host instead of $http_host in proxy_set_header Host directive.

for fastcgi_pass such bug can be fixed only by using two nginx
servers - first for frontend, and second for backend,
because nginx send to fastcgi value of $http_host

bug cause:

fastcgi spec was created when only HTTP/1.0 exists
and don’t know about absoluteURI in request line -
such feature was added in HTTP/1.1, after FastCGI spec.

So, $host must be used always with proxy_pass instead of $http_host.

If the upstream server would do anything security-relevant with the Host:
header that it gets from nginx, it would do exactly the same with the
Host: header that it would get from the client directly, no?

No.

  1. RFC 7230 - Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing

  2. Module ngx_http_core_module

$host
in this order of precedence: host name from the request line, or
host name from the “Host” request header field, or the server name
matching a request


Best regards,
Gena

On 09.03.2015 19:25, Francis D. wrote:

they are probably doing something wrong before nginx gets involved.
redmine documentation:
RedmineInstall - Redmine
don’t forbid users to install redmine into /var/www/redmine

even more, redmine documentation:

RECOMMENDS to install redmine into /var/www/redmine
see: “Configure /var/www/redmine/config/database.yml”

see: “Configure /var/www/redmine/config/database.yml”

also, FHS Filesystem Hierarchy Standard
don’t say what /var/www/… must contain only “static” files.

I see instructions to install to /opt/redmine, and to /var/lib/redmine,
and to /usr/share/redmine, and in each case they say to do something like

ln -s /usr/share/redmine/public /var/www/redmine

to have only the web-accessible content below /var/www/redmine.

I don’t see such instructions at the Redmine | NGINX

If the user really wants to install to /var/www/redmine, then they must
modify the “root” directive (to be /var/www/redmine/public), as the
words on the wiki page already say.

I modify root directive,
but change /var/www/redmine to /home/www/redmine
and all works fine, but with vulnerability.

User must guess than they must change from “root /var/www/redmine;”
to “root /var/www/redmine/public;” to fix this unobvious vulnerability?

I do not see this as an nginx-related security problem.

As I already say, this is not nginx-related security problem,
it was by default vulnerable configuration as wiki recommendation.

The file /var/www/redmine/config/database.yml should not exist.

it MUST exists, because redmine install
instructions suppose that such file exists:
“Configure /var/www/redmine/config/database.yml”

If the file /var/www/redmine/config/database.yml does exist and the
above nginx configuration is used, then the user will find that no part
of their redmine web-related installation will work, because all of the
images and stylesheets and javascripts are inaccessible.

No, if nginx frontend can’t process such non-existend files,
it just silently proxy request to backend and backend process
such request without any problems.

So, this security vulnerability will be invisible for users.

Try it youself, if you don’t believe me.

Correspondingly, if the user has installed only web content below
/var/www, then using a different “root” directive will cause that
installation not to work.

redmine documentation at redmine site recomments install
entire redmine into /var/www/redmine directory, not only public content.


Best regards,
Gena

On 10.03.2015 0:50, Francis D. wrote:

even more, redmine documentation:
HowTo install Redmine on CentOS 5 - Redmine
RECOMMENDS to install redmine into /var/www/redmine
see: “Configure /var/www/redmine/config/database.yml”

Yes, that url shows redmine installed to /var/www/redmine.

In that case, the nginx “root” should be /var/www/redmine/public.

Redmine | NGINX now fixed and provides correct info.

Perhaps it will be useful for someone to note that the “root” directive
value in nginx must be the root directory of the redmine web content,
which is the “public” directory of the redmine distribution. That appears
not to have been clear on the nginx wiki page.

Current nginx redmine config example at wiki is more safe, because
even in case of “ln -s /var/lib/redmine/public /var/www/redmine”
and “root /var/www/redmine/public;” in the nginx config

  • all should work fine, without any security vulnerabilities.

And Debian users probably easy can guess what
they should replace “root /var/www/redmine/public;”
with “root /var/www/redmine;” because /var/www/redmine
is symlink to /var/lib/redmine/public in their install.

redmine documentation at redmine site recomments install
entire redmine into /var/www/redmine directory, not only public content.

The redmine installation instructions are something that the redmine
people might be interested in making more consistent.

May be this is Debian-way, make symlinks to only “static” files
inside /var/www ? And all other services in Debian configured
in the same way? So redmine can’t break Debian packaging rules?

And all other UNIX-like OS and distros do not have such requirements
about creating such useless and potentially dangerous symlinks?
Dangerous, - if nginx configured with “disable_symlinks on;”


Best regards,
Gena

You specifically configured nginx to pass the Host header ($http_host)
to
the backend, thus the backend has only this piece of information
available…
If you specified $host to be passed over, you would not have this flaw
in
your configuration.
nginx does exactly what you configured. By default there is no such
header
set for the backend.

Moreover using a Host header as a ‘security feature’ is… strange at
the
very least.

The difference between the host (machine) and the Host header is common,
and should not be relied on for security. Have you ever played with
curl?
If you use auth_basic for security, you should follow the same process
when
dealing with the backend. The $remote_user variable allows you to check
which user has been authenticated. I would pass that to the backend. No
user = no authentication.

The only ‘security advisory’ I see here is to teach some basic course
about
security to your sysadmin.

B. R.

On Mon, Mar 09, 2015 at 08:24:43PM +0200, Gena M. wrote:

On 09.03.2015 19:25, Francis D. wrote:

Hi there,

But the user must have followed some documentation to install redmine in
the first place; and if they unthinkingly install it into /var/www/redmine
they are probably doing something wrong before nginx gets involved.

redmine documentation:
RedmineInstall - Redmine
don’t forbid users to install redmine into /var/www/redmine

Yes, redmine can be installed anywhere on the filesystem.

even more, redmine documentation:
HowTo install Redmine on CentOS 5 - Redmine
RECOMMENDS to install redmine into /var/www/redmine
see: “Configure /var/www/redmine/config/database.yml”

Yes, that url shows redmine installed to /var/www/redmine.

In that case, the nginx “root” should be /var/www/redmine/public.

The url

includes the line “ln -s /usr/share/redmine/public /var/www/redmine”.

The url

includes the line “ln -s /var/lib/redmine/public /var/www/redmine”.

The url

includes the lines “cd /var/www; ln -s /opt/redmine/public redmine”

In those cases the nginx “root” should be /var/www/redmine.

The nginx configuration must match the redmine installation that was
done.

There does not appear to be consistent documentation on where redmine
is expected to be installed.

From an nginx perspective, change “root” to match where the web
documents
are. That’s what the wiki page said, and that’s what the wiki page says,
so it’s all good.

I see instructions to install to /opt/redmine, and to /var/lib/redmine,
and to /usr/share/redmine, and in each case they say to do something like

ln -s /usr/share/redmine/public /var/www/redmine

to have only the web-accessible content below /var/www/redmine.

I don’t see such instructions at the Redmine | NGINX

Correct.

It looks like the nginx wiki page assumed one type of redmine install,
without documenting exactly what type of install it assumed. That
appears
to still be the case, so that’s all good too.

User must guess than they must change from “root /var/www/redmine;”
to “root /var/www/redmine/public;” to fix this unobvious vulnerability?

Perhaps it will be useful for someone to note that the “root” directive
value in nginx must be the root directory of the redmine web content,
which is the “public” directory of the redmine distribution. That
appears
not to have been clear on the nginx wiki page.

Request https://redmine.example.com/config/database.yml will be
processed by nginx, because file /var/www/redmine/config/database.yml
exists. For details - see manual about try_files directive in nginx.

The file /var/www/redmine/config/database.yml should not exist.

it MUST exists, because redmine install
instructions suppose that such file exists:
“Configure /var/www/redmine/config/database.yml”

There is more than one set of redmine install instructions. The one you
followed wanted a different “root” directive than what was on the nginx
wiki. Now it wants the “root” directive that is on the nginx wiki.

If the file /var/www/redmine/config/database.yml does exist and the
above nginx configuration is used, then the user will find that no part
of their redmine web-related installation will work, because all of the
images and stylesheets and javascripts are inaccessible.

No, if nginx frontend can’t process such non-existend files,
it just silently proxy request to backend and backend process
such request without any problems.

That, I was not aware of. Thank you for correcting me.

I guess it must happen because the back-end web server has been
configured
with the “right” DocumentRoot or whatever equivalent it uses. (Or
maybe it is hardcoded to always look in “public” so does not need extra
configuration.)

Correspondingly, if the user has installed only web content below
/var/www, then using a different “root” directive will cause that
installation not to work.

redmine documentation at redmine site recomments install
entire redmine into /var/www/redmine directory, not only public content.

The redmine installation instructions are something that the redmine
people might be interested in making more consistent.

In nginx, everything on the filesystem below your “root” value is
potentially accessible to the world. So it is prudent to set it to a
directory that only contains public information. The nginx wiki page
shows that now.

All the best,

f

Francis D. [email protected]

On Mon, Mar 09, 2015 at 08:56:28PM +0200, Gena M. wrote:

On 09.03.2015 19:25, Francis D. wrote:

Hi there,

thank you for the explanation.

It is true that $http_host is completely controlled by the client, and
$host is mostly controlled by the client. It is true that they can have
different values. I do not see that the difference is a security issue
in this case.

and access content from private.example.com without any login/password:
You are correct.

It is possible to construct a scenario where the difference between
$http_host and $host matters from a security perspective.

It is also possible to construct a scenario where the difference between
$http_host and $host matters from a correctness perspective.

The “common” redmine configuration appears to be one of the latter,
if nginx is not run on port 80 (such as, by using the original example
config and running nginx as non-root).

If nginx runs on port 80, then everything probably Just Works.

I suspect that if the “proxy_set_header Host” is omitted entirely, then
the default “proxy_redirect” probably means that everything Just Works,
whatever port nginx listens on. And the current example config does just
that, so it’s all good.

for fastcgi_pass such bug can be fixed only by using two nginx
servers - first for frontend, and second for backend,
because nginx send to fastcgi value of $http_host

I think it may be possible for the fastcgi application to avoid that
confusion – nginx sends the http headers plus whatever is configured to
be sent. So, for creation of “redirect” urls back to the client,
HTTP_HOST may be sensible to use; but for anything where the nginx
server{} chosen matters, sending $server_name or $host from nginx in a
well-known fastcgi_param is probably a better option.

So, $host must be used always with proxy_pass instead of $http_host.

If the upstream server would do anything security-relevant with the Host:
header that it gets from nginx, it would do exactly the same with the
Host: header that it would get from the client directly, no?

No.

You are correct, thanks.

I had forgotten about the absolute form of request to nginx, which
becomes origin form when nginx speaks to upstream.

nginx will see the full request and choose the server{} block to use,
and
set $host and $server_name and $http_host appropriately. If upstream
cares
about the difference between them, then nginx must be configured to send
the correct one (or none at all).

nginx will send no more than one in the Host: header, while the client
can send different names in Host: and in the request line.

Cheers,

f

Francis D. [email protected]

On 10.03.2015 23:09, Francis D. wrote:

and access content from private.example.com without any login/password:

You are correct.

It is possible to construct a scenario where the difference between
$http_host and $host matters from a security perspective.

It is also possible to construct a scenario where the difference between
$http_host and $host matters from a correctness perspective.

I don’t know situations where system administrator need use

proxy_set_header Host $http_host;

instead of

proxy_set_header Host $host;

because only $host variable contains correct host name,
and $http_host may be spoofed and contains anything other.

So, for me I create simple rule of nginx configuration:
always use $host in directive proxy_set_header Host
and never use $http_host there.

The “common” redmine configuration appears to be one of the latter,
if nginx is not run on port 80 (such as, by using the original example
config and running nginx as non-root).

If nginx runs on port 80, then everything probably Just Works.

I suspect that if the “proxy_set_header Host” is omitted entirely, then
the default “proxy_redirect” probably means that everything Just Works,
whatever port nginx listens on. And the current example config does just
that, so it’s all good.

In redmine settings: https://redmine.example.com/settings
you can select “Host name and path” to use and protocol, HTTP or HTTPS.

for fastcgi_pass such bug can be fixed only by using two nginx
servers - first for frontend, and second for backend,
because nginx send to fastcgi value of $http_host

I think it may be possible for the fastcgi application to avoid that
confusion – nginx sends the http headers plus whatever is configured to
be sent. So, for creation of “redirect” urls back to the client,
HTTP_HOST may be sensible to use; but for anything where the nginx
server{} chosen matters, sending $server_name or $host from nginx in a
well-known fastcgi_param is probably a better option.

Yes, I am talking about protecting of FastCGI application,
from system administrator point of view, - some application,
open source or closed source, and I don’t know how reliable code
of this application and want to make it safe and protected from
such spoofed $http_host in HTTP_HOST variable, thanks to FastCGI spec.


Best regards,
Gena