goodput higher than throughput

[miniHive]

We have repeatedly observed goodput higher than throughput (for TPC-H and YCSB on PostgreSQL);

{
"Start timestamp (milliseconds)":1739569996066,
"Current Timestamp (milliseconds)":1739572194220,
"Elapsed Time (nanoseconds)":2198000068342,
"DBMS Type":"POSTGRES",
"DBMS Version":"PostgreSQL 17.3 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit",
"Benchmark Type":"tpch",
"Final State":"EXIT",
"Measured Requests":22,
"isolation":"TRANSACTION_SERIALIZABLE",
"scalefactor":"1",
"terminals":"1",
"Latency Distribution":{
"95th Percentile Latency (microseconds)":8171213,
"Maximum Latency (microseconds)":1046028244,
"Median Latency (microseconds)":1145302,
"Minimum Latency (microseconds)":47069,
"25th Percentile Latency (microseconds)":381294,
"90th Percentile Latency (microseconds)":5461144,
"99th Percentile Latency (microseconds)":1046028244,
"75th Percentile Latency (microseconds)":2895255,
"Average Latency (microseconds)":49201791
},
"Throughput (requests/second)":0.010009098869862678,
"Goodput (requests/second)":0.020018197739725355
}

This seems counter-intuitive.

[bpkroth]

Thanks. I've seen that before too and haven't looked into it yet. Will try and add it to the queue. Also happy to take PRs in case you get to it before me.

[dwslim]

I have observed this before too. I read the implementation and noticed that the calculation of goodput includes the requests processed during the warmup phase.

@miniHive You may want to check if that is the case? A simple check involves just multiplying throughput by the duration of (WARMUP + MEASURE) phases, and comparing that with the goodput and see if they roughly correspond.

I was confused but thought it was somehow intended.

[zyuiop]

I have observed this as well on TPC-C and TPC-H.

Interestingly, for TPC-H, the goodput is PRECISELY twice the throughput. This also corresponds to a weird problem in the histograms printout at the end of the benches: each query appears as if it had run four times, even though the TPC-H config only runs each query twice.

I have not configured any warmup for my TPC-H runs, but maybe benchbase still runs one under the hood for each query?

edit: ok, ThreadBench.java:146: for serial runs, there is a COLD_RUN step before the actual benchmark