Suppose that I have two systems $A$ and $B$ whose output are $O_A$ and $O_B$, respectively. The statistic of interest that I use to decide which system is better is $s(O_A)$ and $s(O_B)$, respectively. This statistic is larger when a system is better.

I found that $s(O_A) > O(O_B)$, therefore I concluded that the system $A$ is better than the system $B$. However the question at hand is whether such difference is due to a systematic difference, as opposed to sheer dumb luck.

So I asked my self the question: "What is the probability of observing $s(O_A) > s(O_B)$ if the differences between $A$ and $B$ were non-systematic?". This question is also equivalent to the question: "What is the probability of observing differences $\ge |s(O_A) - s(O_B)|$ under the null hypothesis?".

To answer that question, I used approximate randomization and it gave me a $p$ value. My questions are centered around how to interpret such $p$ values.


  • Q1: When to say that differences between $A$ and $B$ are significant.
  • Q2: When to say that difference between $A$ and $B$ are insignificant.
  • Q3: When to say that available data is not enough to conclude anything at all.

Examples for further clarification:

For example, usually when $p \le 0.05$, it is concluded that differences between $A$ and $B$ are significant. This is probably a classical use case of $p$ values.

But what if $p \approx 0.1$? Should we state that the difference between $A$ and $B$ is insignificant? Or should we refrain from stating anything and conclude that the data is simply too small to allow for any healthy conclusion?

Similarly, what if $p \approx 0.999$? Should we finally be able to state that the differences between $A$ and $B$ are insignificant?


1 Answer 1


First of all, you may want to reconsider your hypothesis test "What is the probability of observing differences $≥|s(O_A)−s(O_B)|$ under the null hypothesis?". Since you're using the absolute value of the difference you're just testing the hypothesis that the mean of $s(O_A)$ is different to the mean of $s(O_B)$. You may want to test that the mean of $s(O_A)$ is greater than the mean of $s(O_B)$. Simply showing that the means are different does not indicate that the system with the greater sample mean has the greater true mean.

For convenience for the rest of the answer let the mean of $s(O_A)$ be $\mu_A$ and likewise for $\mu_B$. Also I'll assume that you're testing the hypothesis that $\mu_A=\mu_B$

Suppose you've chosen $\alpha=0.05$ as your significance level.

In hypothesis testing we only ever reject the hypothesis (find a significant difference) or fail to reject it (cannot find a significant difference). It is not possible to prove that the means are the same, even if the sample means are very similar it is always possible that they are very slightly different and we haven't been able to measure that difference with the limited sample size.

If you get $p=0.1$ then you have failed to reject the hypothesis that the means are equal, you can either do another experiment with a larger sample size to try to detect a significant difference or you can accept that it's not known if $\mu_A=\mu_B$ or $\mu_A \neq \mu_B$.

If you get $p=0.99$ this tells you that the difference between sample means was likely under the hypothesis that $\mu_A=\mu_B$, but it could also be likely to observe that difference between the sample means under many other hypotheses.

In other words, your experimental results could be likely in cases where there's no difference and in cases where there's a big difference between the means. So although a low p-value tells you that the means are different, a high $p$ value doesn't tell you that the means are the same.

  • $\begingroup$ No, you can't just increase your sample size. You can do another experiment with perhaps an increased sample size but that's not the same thing. Perhaps that's what you meant. If so, please edit the answer. $\endgroup$
    – John
    Sep 21, 2016 at 0:26
  • $\begingroup$ Could you please rephrase the last paragraph? I am completely lost there. $\endgroup$
    – caveman
    Sep 21, 2016 at 0:30
  • $\begingroup$ @caveman I tried to break it down a bit more, I'm not too sure how I could reword it. I hope it's clear enough now. $\endgroup$
    – Hugh
    Sep 21, 2016 at 11:08
  • $\begingroup$ So is there any way to show that the means are the same? $\endgroup$
    – caveman
    Sep 22, 2016 at 16:58
  • 1
    $\begingroup$ @Caveman that's correct. After you collect thousands of data points you will be able to rule out the possibility that $S_1$ comes from the distribution $N(\mu,\sigma^2)$ and $S_2$ comes from the distribution $N(\mu+0.01,\sigma^2)$ but you wont rule out the possibility that $S_1$ comes from the distribution $N(\mu,\sigma^2)$ and $S_2$ comes from the distribution $N(\mu+0.0001,\sigma^2)$ As you increase your sample size you will detect smaller differences but it is always possible for the difference between means to be small enough that you wont be able to disprove that that difference exists. $\endgroup$
    – Hugh
    Sep 24, 2016 at 9:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.