Suppose, Accuracies are:

Algorithm 1: 87%

Algorithm 2: 86%

Algorithm 3: 88%

Paired t-test value between Algorithm 1 and Algorithm 3 is p = 0.44

Paired t-test value between Algorithm 2 and Algorithm 3 is p = 0.073

Paired t-test value between Algorithm 1 and Algorithm 2 is p = 0.013

I used scipy.stats.ttest_rel for calculating the paired t-test.

My question: When

i) Algorithm 2 is significantly worse than Algorithm 1

ii) Algorithm 1 is not statistically different from Algorithm 3.

How is the following thing possible?: Algorithm 2 is not significantly worse than Algorithm 3.

Can anyone please explain why is this happening?

  • 1
    $\begingroup$ What you are overlooking is that statistical significance depends on variability as well on effect size, i.e., the effect size needs to be large relative to variability to achieve significance. In your case, the results from Algorithm 3 are presumably more variable than those from Algorithm 1 or 2, so that no comparisons involving Algorithm 3 are significant. Algorithms 1 and 2 are more consistent in terms of accuracy, so that the 1 vs 2 comparison is significant even though the average difference in accuracy is small. $\endgroup$ – Gordon Smyth Oct 3 '19 at 6:07
  • 2
    $\begingroup$ Put another way, Algorithm 3 has the best average accuracy so far, but it is so variable that you can't be entirely confident whether it will be the best or the worst in the long run. By contrast, Algorithm 1 is almost always slightly better than Algorithm 2, so the evidence is already sufficient to rank 1 above 2. $\endgroup$ – Gordon Smyth Oct 3 '19 at 6:22
  • $\begingroup$ One way to think about this is that a statistical test doesn't tell you if two things are actually "different" or the "same". But instead the degree of evidence you have for a hypothesis. And your conclusion is relative to the standards for evidence you have set up (here, alpha = 0.05). For example, consider if you changed your alpha value to 0.01 or to 0.1. In each case your conclusions would be quite different. But the underlying "different"ness of the populations being investigated didn't change. $\endgroup$ – Sal Mangiafico Oct 3 '19 at 8:50

Here are fake data simulated to have roughly the means you report. Notice that, 'not significantly different' is not the same as 'equal'.

We have approximate means as follows:

  A2        A1        A3
 85.9      87.3      87.6

A1 and A2 differ with P-value .001; A2 and A3 differ with P-value only .026; and A1 and A3 do not differ (P-value .71).

As @GordonSmyth has commented, this pattern is partly due to the high variability of A3.

Additionally, in the case of my fake data, the sample size of only $n = 30$ is not sufficient to guarantee that the same mean and standard deviations I used in the simulation will always give this same pattern of differences. If you use R, you can try changing seeds to see that somewhat different patterns can occur at random. You don't say what $n$ you have, but if it small relative to the variances, you might see a different pattern upon repeating your experiment.

Here is R code for the simulated dataset:

e = rnorm(30, 0, 1)
x1 = rnorm(30,87, 2) + e
x2 = rnorm(30,86, 1) + e
x3 = rnorm(30,88, 4) + e
mean(x1); mean(x2); mean(x3)
[1] 87.30957
[1] 85.97732
[1] 87.57734

[1] 0.001443453
[1] 0.7081056
[1] 0.02586592

Correlations amon the three vectors x3,x2,x3 are relatively small. Real paired data often have larger correlations. To increase correlation for the fake data: increase the SD of e in the code above.

          x1        x2        x3
x1 1.0000000 0.1848602 0.1912384
x2 0.1848602 1.0000000 0.2327309
x3 0.1912384 0.2327309 1.0000000

pairs(cbind(x1,x2,x3), pch=19)

enter image description here

Notes: (1) The third parameter of rnorm is the population standard deviation (not variance). (2) To see the complete printout of the paired t tests, remove the suffix $p.val at the end of each t.test statement.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.