Dear developers,
I have experienced SEGFAULTs on ARM using fastcgi and discovered it
compiles with “NGX_HAVE_ATOMIC_OPS 0” on ‘other’ architectures than
x86, amd64, sparc and the such defined in src/os/unix/ngx_atomic.h.
Therefore I’d like to contribute the patch linked below [1], which
introduces configure option “–with-libatomic” and which makes nginx
use atomic operations of that library on these ‘other’ architectures.
For more information on the library please see [2]. (Indeed, this does
not result in additional runtime dependencies and the atomic ops
compile to less code than those of newer GCC versions [3].)
–with-libatomic compiles on ARM, does no SEGFAULT, and yields higher
requests per second than with the otherwise used lock file.
The option is disabled by default, and even if enabled still used as
last resort. I hope that after a brief review (I have little
experience with configure scripts) you can integrate it in the next
release.
–
W-Mark K.
[1]
https://svn.hurrikane.de/all/ossdl/www-servers/nginx/files/nginx-0.8.16-libatomic.patch
[2] http://www.hpl.hp.com/research/linux/atomic_ops/
CVS Info for project bdwgc
[3] Atomic Builtins - Using the GNU Compiler Collection (GCC)
for gcc 4.1.0 and later
On Fri, Sep 25, 2009 at 09:46:33PM +0200, W-Mark K. wrote:
not result in additional runtime dependencies and the atomic ops
W-Mark K.
http://mark.ossdl.de/
[1] https://svn.hurrikane.de/all/ossdl/www-servers/nginx/files/nginx-0.8.16-libatomic.patch
[2] http://www.hpl.hp.com/research/linux/atomic_ops/
CVS Info for project bdwgc
[3] Atomic Builtins - Using the GNU Compiler Collection (GCC)
for gcc 4.1.0 and later
Thank you for the patch and information about gcc 4.1.
I’m going to add gcc builtins, since they are slighty lesser than
my code at least on x86.
I’m not sure should libatomic be added, since gcc 4.1+ is common
compiler
these days.
Could you show backtrace of the segfault ? I could not reproduce it
on x86 with disabled atomic ops.
2009/9/29 Igor S. [email protected]:
Thank you for the patch and information about gcc 4.1.
I’m going to add gcc builtins, since they are slighty lesser than
my code at least on x86.
I’m not sure should libatomic be added, since gcc 4.1+ is common compiler
these days.
Could you show backtrace of the segfault ? I could not reproduce it
on x86 with disabled atomic ops.
The segfaults happened on ARM architecture, not x86.
Eliminating lock_file by introducing atomic ops the only code which
got not called after that were the emulations at the bottom of
ngx_atomic.h.
GCC builtins work for me, too, but they make nginx be linked against a
specific libgcc (that of the particular GCC version) and yield more
overhead (2kb libatomic_ops w/o any link vs. 13kb GCC builtins).
Finally libatomic_ops can be compiled by other means (such as MSVC,
ICC, SUNC for example) therefore I prefer them for portability.
What do you think about the compromise using GCC builtins if
–with-libatomic is not set?
Only architectures not covered by the #ifs would be affected and you
could remove lock_file code entirely in future.
On Tue, Sep 29, 2009 at 12:26:43PM +0200, W-Mark K. wrote:
got not called after that were the emulations at the bottom of
ngx_atomic.h.
I understand that segfaults was happened on ARM. However, I tried
to reproduce them on x86 with disabled atomic operations.
GCC builtins work for me, too, but they make nginx be linked against a
specific libgcc (that of the particular GCC version) and yield more
overhead (2kb libatomic_ops w/o any link vs. 13kb GCC builtins).
Finally libatomic_ops can be compiled by other means (such as MSVC,
ICC, SUNC for example) therefore I prefer them for portability.
What do you think about the compromise using GCC builtins if
–with-libatomic is not set?
Only architectures not covered by the #ifs would be affected and you
could remove lock_file code entirely in future.
On i386/amd64 gcc builtins are just inlines and do not require libgcc.
As to libatomic, I’m not sure now. I prefer to use complex libraries
such as OpenSSL, zlib, etc.
2009/10/8 Igor S. [email protected]:
such as OpenSSL, zlib, etc.
On x86/amd64 the shipped assembler code works fine, and my libatomic
patch was intended for ‘other’ architectures.
Despite of the latter I believe reinventing the wheel (as did php-fpm)
is no good engineering - and you need to maintain that code yourself
rather than have the ‘community’ do that for you (and other projects).
Unfortunately I don’t know the “complex” libraries good enough to tell
whether they contain atomic ops nor whether they export it.
Igor, if it was my project I would have deleted the asm part
completely and used GCC atomics (the same goes for php-fpm) where
available, and libatomic (or any better replacement, I am not dogmatic
here) else. As far as I can tell the latter are inlined, too.
In the meantime I’ve created modified Gentoo ebuilds [1] (for all
archs) and binaries [2] (for ARM) - with the option for libatomic.
Find it attached as the link mentioned previously does not work
anymore because I replaced subversion by GIT.
–
W-Mark K.
[1] http://git.ossdl.de/?p=ossdl-overlay.git;a=summary
[2] http://binhost.ossdl.de/armv5tel-softfloat-linux-gnueabi/
(set this as binhost in Gentoo and the packages will be pulled
automatically)
Hello!
On Fri, Oct 09, 2009 at 11:57:24AM +0200, W-Mark K. wrote:
As to libatomic, I’m not sure now. I prefer to use complex libraries
such as OpenSSL, zlib, etc.
On x86/amd64 the shipped assembler code works fine, and my libatomic
patch was intended for ‘other’ architectures.
Despite of the latter I believe reinventing the wheel (as did php-fpm)
is no good engineering - and you need to maintain that code yourself
rather than have the ‘community’ do that for you (and other projects).
Unfortunately I don’t know the “complex” libraries good enough to tell
whether they contain atomic ops nor whether they export it.
The idea is that for simple and rarely changing things like atomic
ops it’s better to have them in-place instead of in external
library. As external library may change, broke, disappear and so
on, as a result requiring much more attention then simple in-place
code.
Note well: I believe the actual problem isn’t atomic ops that
isn’t here. By design nginx should work without atomic ops. The
real problem is SIGSEGV you see without them. It should be fixed,
but unfortunately we have no hardware to reproduce the problem.
Maxim D.
On Fri, Oct 09, 2009 at 11:57:24AM +0200, W-Mark K. wrote:
As to libatomic, I’m not sure now. I prefer to use complex libraries
–
W-Mark K.
http://mark.ossdl.de/
[1] http://git.ossdl.de/?p=ossdl-overlay.git;a=summary
[2] http://binhost.ossdl.de/armv5tel-softfloat-linux-gnueabi/
(set this as binhost in Gentoo and the packages will be pulled
automatically)
I agree that reinventing wheel is not good engineering and I prefer
to use libraries, however, in reason. I want to minimize external
dependances for general usage as much as possible. nginx may depend on
OpenSSL, libXSLT/libXML, perl, libGD, GeoIP, but usually it requires
just zlib and PCRE. And you can build nginx even without these
libraries.
I’m not sure that community maintenance is always better.
PCRE and zlib support is certainly very good, however, I can say
this about any library.
As to gcc 4.1 builtin/libatomic only solution, now nginx can be built
by gcc 2.8 and 2.95, and I do not think that installing libatomic
on FreeBSD 4 (gcc 2.95) is necessary thing to run nginx.
Besides, for example, MacOSX/Darwin has atomic(3) and barrier(3)
operations
optimized at run-time for UP and SMP (they does to use lock on UP).
I will add libatomic as last resort (gcc4.1 built-ins, my atomic code,
libatomic) and will add building procedures since it seems Linux
distributions are the only platform that has libatomic package.
BTW, could you please show coredump backtrace of segfault without atomic
ops ? I still want to fix the bug to allow nginx run even on platforms
without atomic ops at all.
Hello!
On Tue, Sep 29, 2009 at 12:26:43PM +0200, W-Mark K. wrote:
got not called after that were the emulations at the bottom of
ngx_atomic.h.
Just for the record: I’ve tried to reproduce the problem on
QEMU-emulated ARM but failed. Everything works fine for me.
Could you please provide some more details? Hardware, OS,
nginx -V output, config and request pattern which triggers fault,
backtrace, etc…
Maxim D.
Hello!
I will send you the debug informations asap, but unfortunately no
sooner than by the end of this week.
Let me open a separate thread for this, then, with more details.
From what I remember, lockfile was on /dev/shm, vanilla configuration
files. Nginx was compiled with zlib and flv, only the former was
actually used and later on even none of them. Segfaults began as soon
as concurrent requests occured (e.g. by ab -c 5, one to two workers,
epoll), even to static files. The logfile was flooded by “worker…
segfault” for every (unhandled due to the fault) request. (Linux
2.6.29-rc5, 2.6.30; deadlock scheduler)