Documentating myself on proper benchmarking, I ran into the following
page:
Their conclusion is that their product is the best of all. Well, ‘of
course’ one might say… ;o)
What surprised me most that they claim to use less resources AND perform
better. That particularly strikes me because usually ot favor one side,
you
take blows on the other one.
To me, the problem of such tests is that they are a mix of
realistic/unrealistic behaviors, the first being invoked to justify
useful
conclusions, the latter to make a specific environment so that features
from the Web server (as opposed to other components of the
infrastructure)
are tested.
They are arrogant enough to claim theirs is bigger and paranoid enough
to
call almost every other benchmark biased or coming from haste/FUD
campaigns. That is only OK if they are as pure as the driven snow…
I need expert eyes of yours to determine to which end those claims are
grounded.
Why is your wrk test tool built in such way in pre-establishes TCP?
Why is nginx pre-allocating resources so its memory footprint is large
when connections are pre-established? I thought nginx event-based system
was allocating resources on-the-fly, as G-WAN seems to be doing it. (cf.
‘The (long) story of Nginx’s “wrk”’ section)
Why is wrk (in G-WAN’s opinion) ‘too slow under 10,000 simultaneous
connections’? (cf. ‘The (long) story of Nginx’s “wrk”’ section)
And of course, you should ask what the relevance of delivering a single,
100byte file a huge number of times actually has in real life, or where
useful things - like comparison of server side languages behind various
web servers - are tested.
And yes, any web servers latency should be approximately zero for
testing delivery of a static file locally ( also pointless ).
returning empty_gif, rather than nop.gif is also exercising internal
code, rather than just delivering a file, so the processing is not
comparable.
saving open file metadata may also help, but the test is so skewed and
unrealistic it’s not really worth the effort.
That’s just my view as a SysAdmin
On Sun, 2015-02-08 at 00:52 +0100, B.R. wrote:
What surprised me most that they claim to use less resources
thought nginx event-based system was allocating resources