Entries without database tests seem to be massively penalized

[uNetworkingAB]

If you look at for instance Drogon, it has very high baseline score but scores almost bottom in composite score. It does not have database entries (which have nothing to do with the web server and is not part of a web server benchmark).

Seems like all entries who do not have database tests automatically lose 100 points off the composite and thus cannot compete on web serving against other web servers?

Makes the whole HTTP benchmark useless as many entries will not participate in database tests (they are not part of web server benchmarks and have no correlation to the web server).

[uNetworkingAB]

Clear rationale and common sense explanation:

A database benchmark has literally no correlation with the performance of a web server. They are orthogonal. You are only benchmarking the database usage and the database connector library, none of which are related to HTTP, the HTTP server or what the name "Http-Arena" infers.

Only in very rare cases, such as a massively underperforming web server will the web server have any kind of correlation with the database score (by being the limiter). This has, practically, never happened.

Empirical evidence for the common sense

If you look at TechEmpower database tests, you can see a clear pattern: All the winning entries ("the frameworks") use PostgreSQL and all the losing entries use MySQL. In other words, the TechEmpower database testing proves that the "framework" has literally no impact in the ordering. It all segments, sorts, only by the database being used. This is as clear of empirical evidence you can get.

Contradiction with other arbitrary rules

Also, the need for a database entry contradicts with your other arbitrary rules such as "you can only do what the "framework" supports out of the box". Okay but many web servers do not even have a built-in database connector, so by sheer contradiction, this makes no sense.

[MDA2AV]

I agree that the database factor is orthogonal and we can make those tests unscored but then we need to fill a gap, asynchronous workload, many servers are purely synchronous, meaning that if you do any I/O like httpclient or database access they block the thread until it completes, these servers will score very high in baseline and simple synchronous tests but totally collapse with any asynchronous workload, so any db, redis, httpclient etc.. which is the typical use cases for these frameworks.

So my question is, what kind of test can test asynchronous capabilities without adding this ¨noise¨ that a database access adds?

One clear example is https://www.http-arena.com/#sort=rps:-1&type=engine see minima-sync entry, this framework cannot process async at all and will have less than 10k rps in async-db tests, yet on baseline it matches tokio.. even though it is a fully c# managed entry vs one of the most optimized rust engines there is.

So minima-sync passes all validations, is very simple, scores ultra high yet is absolutely useless to build any REST service, async-db exposes that

[uNetworkingAB]

You can catch that even with just a simple timer, like the request has to wait 1 second before it responds. There should be no way to cheat that, since a timer will only fire some ms after the timeout happened. But you can also probably catch that behavior by just opening lots of connections at the same time and see how poorly the single-request-per-thread handle those.

If you establish many connections (thuosands) and have each response be 1 second delayed, that should be heavily in favor of modern event driven solutions and ruin threaded servers.

[uNetworkingAB]

Maybe you even want something like a latency test:

/delay/:seconds

The request has to delay that many seconds (needs to be dynamically parsed)

Then you open a shit ton of connections and send random delays and look at the latencies compared to their expected times

[MDA2AV]

Ok that actually sounds promising, I had a test before that did something similar, a 1 second delay at 64k connections but many frameworks could deliver the 64k rps and if I pushed past to higher connections some would start collapsing, crashes, non repetitive results where it would swing 40-50% between 2 consecutive runs. A lower adaptive delay may fix this, the timers are not very precise in some languages like python but it should average out with thousands of requests.

Ill test tonight whether we can lower delay enough to actually saturate some frameworks, worst case we lower available cores for this test.

[uNetworkingAB]

You can normalize their performance by the CPU they consume. So even if a shitty server can manage, it will do so at higher cost. That's basically the same thing as limiting the CPU cores, though.

[MDA2AV]

Measuring CPU has been tricky, some frameworks delegate work to threadpool always using the ¨next available core" while others use thread per core model, the latter case tends to show lower CPU usage when working on a non saturated state, this value is quite misleading as is typically not a good measure as the threadpool model tries to use as many CPU cores it has available, showing higher CPU usage even though in reality it isnt always the case, we would need a much better tool to check CPU usage than the current docker stats.

[MDA2AV]

Maybe also a test where throughput is fixed and score is based on CPU efficiency could be interesting too. Most of the times servers are close to idle