6
$\begingroup$

I am trying to calculate whether the difference between the two benchmarks is statically different or not.

The input is req/sec of a HTTP Server and I'm using scipy.stats.ttest_ind to calculate the p-value.

A1 = [
  4670, 4646, 4612, 4618, 4646,
  4609, 4623, 4629, 4566, 4628,
  4582, 4636, 4621, 4574, 4624,
  4563, 4651, 4642, 4586, 4621,
  4606, 4628, 4575, 4631, 4646,
  4600, 4594, 4661, 4568, 4611
]

B1 = [
  4630, 4655, 4652, 4633, 4637,
  4661, 4625, 4680, 4647, 4639,
  4633, 4661, 4638, 4621, 4630,
  4682, 4703, 4665, 4652, 4648,
  4673, 4651, 4669, 4646, 4612,
  4654, 4651, 4619, 4637, 4620
]
st.ttest_ind(A1, B1)
# Ttest_indResult(statistic=-4.855056212284194, pvalue=9.47100493260572e-06)

Why the value is 9.47100493260572e-06? I was expecting to see a value bigger than 0.05 because the input is pretty similar and the means are relatively close, too: 4615 vs 4647

Am I missing something?

Improve this question
$\endgroup$
2
  • $\begingroup$ The question omits details about the domain and the problem at hand, so it's hard to consider whether a t-test is a great choice. It might be interesting to read about equivalence testing. $\endgroup$
    – dipetkov
    Commented Nov 6, 2022 at 15:57
  • 1
    $\begingroup$ Change the origin of your units to 4600 and look again at the data. Plotting the distributions of the two groups would be a good idea. $\endgroup$
    – whuber ♦
    Commented Apr 4, 2023 at 17:32

3 Answers 3

Reset to default
15
$\begingroup$

The t-test does not care about the magnitudes of your values. The t-test concerns itself with their variance. You are correct that your numbers look to be roughly aligned. However, the distributions appear to be rather tightly clustered, meaning low enough variance for the difference in means to be statistically significant.

What you’re allowed to do is ignore the statistical significance and decide that the means are close enough together, based on your knowledge of the process under study, that you accept this difference. This gets into practical significance, as opposed to statistical significance.

Improve this answer
$\endgroup$
9
  • $\begingroup$ Thanks, that was enlightening. However, trusting the mean between results, specifically on HTTP Benchmarks, is not reliable considering the fact of outliers. I am trying to use the student t-test approach to compare performance optimizations between two branches. Is there a better approach to use? Thanks in advance $\endgroup$
    – Rafael
    Commented Nov 6, 2022 at 1:03
  • 5
    $\begingroup$ @RafaelGonzaga Then why are you hypothesis testing the means? You might consider posting a new question where you explain your data and goals in more detail, as the specifics of those really are not pertinent to the question asked here. $\endgroup$
    – Dave
    Commented Nov 6, 2022 at 1:05
  • $\begingroup$ You right. Thanks! $\endgroup$
    – Rafael
    Commented Nov 6, 2022 at 1:17
  • $\begingroup$ It is really correct to say that "the t-test concerns itself with (...) variance"? $\endgroup$
    – dipetkov
    Commented Nov 6, 2022 at 15:41
  • 2
    $\begingroup$ My point was that the test doesn’t care about raw magnitudes, just how spread out the groups are from their means. I’ll think about how to edit in an improved phrasing, bearing in mind that the OP found the original post to be helpful. $\endgroup$
    – Dave
    Commented Nov 6, 2022 at 15:49
15
$\begingroup$

This is more of a comment or extension to the answer by @Dave. You should always plot your data, and could have included such a plot in your question. Below is plots that help to show the difference between the groups. I use R for the plots, at the end I give the code used.

enter image description here

This is simply a boxplot with the individual points overplotted. Alternatively, we can show histograms:

enter image description here

The R code used is:

A1 <- c(
  4670, 4646, 4612, 4618, 4646,
  4609, 4623, 4629, 4566, 4628,
  4582, 4636, 4621, 4574, 4624,
  4563, 4651, 4642, 4586, 4621,
  4606, 4628, 4575, 4631, 4646,
  4600, 4594, 4661, 4568, 4611
)

B1 <- c(
  4630, 4655, 4652, 4633, 4637,
  4661, 4625, 4680, 4647, 4639,
  4633, 4661, 4638, 4621, 4630,
  4682, 4703, 4665, 4652, 4648,
  4673, 4651, 4669, 4646, 4612,
  4654, 4651, 4619, 4637, 4620
)
library(ggplot2)
library(hrbrthemes)

df <- data.frame( req = c(A1, B1),
                  benchmark=c(rep("A", length(A1)), rep("B", length(B1)))
)

ggplot(df, aes(benchmark, req, color=benchmark)) + geom_boxplot() + 
  geom_point()

### Side-by-side histograms

ggplot(df,aes(req, ..density.., fill=benchmark)) + 
  geom_histogram(color="#e9ecef",alpha=0.4, bins=10, position="identity") + 
  theme_ipsum() + scale_fill_manual(values=c("#69b3a2", "#404080"))
Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the explanation of the graphs. I came from a different area, therefore, some foundations are not very clear to me. It's not very clear to me to understand when should I use the t-test approach, should I create a new question? $\endgroup$
    – Rafael
    Commented Nov 6, 2022 at 22:11
  • 2
    $\begingroup$ @Rafael The focus of the suggest is not shall I use a t-test?. Instead, it is a recommendation to visualize the data very early (as a tool complementary to e.g., a t-test) by an affordable plot. This could be: to recognize if a distribution might be normal, or skewed, or tailed, plot a histogram. For a relationship between dependent and independent variable perhaps linear, or exponential, or different, plot a scatter plot (Anscombe's quartet). And then engage with the computation(s), tests, modelization, plots for the final report. $\endgroup$
    – Buttonwood
    Commented Nov 7, 2022 at 6:46
  • $\begingroup$ @Rafael Looking at this plot, the mean different between groups is very clear, and outliers are not an issue. So your test result is fine, what's wrong was your expectation what the result should be. $\endgroup$
    – Christian Hennig
    Commented Apr 5, 2023 at 11:08
1
$\begingroup$

I strongly support the stance of @kjetil b halvorsen that visualization is the first priority. This would be a comment on his answer except that I have a different graph to show.

The spirit of plotting all the data, plus a summary, is excellent. Kjetil's graph in practice raises two comments on details.

  1. The plot doesn't show means. As it happens, the means are close to the medians, but you are not always so lucky.

  2. A dot plot or strip plot in a single line will not be easy to work with if there is much over=plotting of similar or identical values. Same comment: As it happens, that is not a major problem here, but you are not always so lucky.

This plot follows the spirit of Emanuel Parzen's suggestion that a quantile plot together with a box gives a good picture of the data.

The boxes conventionally show medians and quartiles. The quantile plos show the points in order with a tacit horizontal scale of rank or plotting position. So, outliers, gaps, tied values and other fine structure are all evident if they exist. The longer horizontal lines show the means. (Incidentally, each mean is also the area under the quantile function expressed as a continuous curve and as a function of cumulative probability.)

enter image description here

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.