Consider making the runs longer

[tomas-langer]

Some frameworks/languages have runtime optimisations (such as Java with JIT), which trigger after some time.
As far as I could find out, these benchmarks do not warmp-up the servers (or the warmup is very short), and the data is collected from three 5 second runs.

Is there a plan to have longer runs once you switch from "alpha", or is this the current intent?

If so, I would like to ask to consider changing this, as most HTTP servers run over longer periods of time, and any runtime optimisation is relevant. This also penalises servers that have any kind of JIT equivalent.

I understand this would make the whole suite run really long, if we were to switch from seconds to minutes.

[MDA2AV]

This is a long living issue we have yes, there are plans to change the duration, new tests are 3x 20 seconds already and the endgame goal is 2 minutes (in some tests) which to be honest, is not perfect either, 15 mins, 30 mins would be a lot better but also unrealistic when there are hundreads of frameworks to run on a single server machine.

Currently the test duration is still very short because these tests are not final either, we are testing the waters and collecting statistics on each run for cpu usage, tail latency, throughput variance etc. While the 5 second duration sounds very thin, it is somehow surprisingly stable across many frameworks even when we compare with "private" longer runs we do with 30 sec and 1 min duration. Our greatest issue are the tail latencies, p99, p99.9 and p99.99 swing hard, specially when garbage collected languages are tested, p99.9 needs at least a 2 minutes run to become somewhat stable.

When it comes to throughput, 15-30 seconds is already very close to a 3-5 min run on every tested framework, from C to Java, C# or even Python. On engines, 5 seconds is actually enough, these are very stable and optimized engines have very low allocations.

So answering you, yes, we plan on increasing test duration to 20-30 seconds and 1-2 min depending on the test. This will have some consequences though, there will be a trim on the number of tests and frameworks.
On the test variety we will likely drop unscored tests and merge existing ones (like baseline and limited-con for example), on top of this we will move away from "saturation" or "all out" tests and focus on pinned throughput at medium to low loads, which are the typical cases a server operates on, instead of measuring top throughput, shift to measure CPU efficiency and tail latencies. This latter change will also help on the electricity bill.. We will at least keep on all out test, likely baseline.

On the framework trimming, so far we accepted any framework, a lot due to the fact that it also helps us covering more runtimes and technologies, useful to understand the limits on our hardware and tests. This is likely to shift, frameworks with no maintainers will be gradually dropped.

[MDA2AV]

For example idle or close to idle CPU efficiency turns out to be something a lot of people overlook, yet likely to be way more valuable than a "top speed" they will never hit.

https://www.http-arena.com/#p=latency-10k&conns=1024&sort=cputot:-1

we can see here that some frameworks that can hit well over 1 million requests per second.. somehow need 11-13% total cpu when serving 10k, at the light of such metric, I would definetly prefer a framework with a ceiling of 200k throughput that consumes 5% at 10k

[tomas-langer]

I am getting unstable results on these runs - the output varies by up to 40% due to initial GC and JIT, and due to cutoff/teardown behavior (i.e. h2load).

[MDA2AV]

In the "best of 3" the first 1 or 2 rounds are the warmup.. I know, not amazing.

Between tests the dockers are dropped and restarted, this will force a warmup for each test, one possibility is to not drop the dockers between tests and have a big warmup sequence at the start of the benchmark. This would add noise between tests, maybe not a good option.

Ill try to introduce longer duration with warmup rounds this week and see how it goes.