How to interpret the $p$ value?

Question

Suppose that I have two systems $A$ and $B$ whose output are $O_A$ and $O_B$, respectively. The statistic of interest that I use to decide which system is better is $s(O_A)$ and $s(O_B)$, respectively. This statistic is larger when a system is better.

I found that $s(O_A) > O(O_B)$, therefore I concluded that the system $A$ is better than the system $B$. However the question at hand is whether such difference is due to a systematic difference, as opposed to sheer dumb luck.

So I asked my self the question: "What is the probability of observing $s(O_A) > s(O_B)$ if the differences between $A$ and $B$ were non-systematic?". This question is also equivalent to the question: "What is the probability of observing differences $\ge |s(O_A) - s(O_B)|$ under the null hypothesis?".

To answer that question, I used approximate randomization and it gave me a $p$ value. My questions are centered around how to interpret such $p$ values.

Questions:

• Q1: When to say that differences between $A$ and $B$ are significant.
• Q2: When to say that difference between $A$ and $B$ are insignificant.
• Q3: When to say that available data is not enough to conclude anything at all.

Examples for further clarification:

For example, usually when $p \le 0.05$, it is concluded that differences between $A$ and $B$ are significant. This is probably a classical use case of $p$ values.

But what if $p \approx 0.1$? Should we state that the difference between $A$ and $B$ is insignificant? Or should we refrain from stating anything and conclude that the data is simply too small to allow for any healthy conclusion?

Similarly, what if $p \approx 0.999$? Should we finally be able to state that the differences between $A$ and $B$ are insignificant?

Answer 1

First of all, you may want to reconsider your hypothesis test "What is the probability of observing differences $≥|s(O_A)−s(O_B)|$ under the null hypothesis?". Since you're using the absolute value of the difference you're just testing the hypothesis that the mean of $s(O_A)$ is different to the mean of $s(O_B)$. You may want to test that the mean of $s(O_A)$ is greater than the mean of $s(O_B)$. Simply showing that the means are different does not indicate that the system with the greater sample mean has the greater true mean.

For convenience for the rest of the answer let the mean of $s(O_A)$ be $\mu_A$ and likewise for $\mu_B$. Also I'll assume that you're testing the hypothesis that $\mu_A=\mu_B$

Suppose you've chosen $\alpha=0.05$ as your significance level.

In hypothesis testing we only ever reject the hypothesis (find a significant difference) or fail to reject it (cannot find a significant difference). It is not possible to prove that the means are the same, even if the sample means are very similar it is always possible that they are very slightly different and we haven't been able to measure that difference with the limited sample size.

If you get $p=0.1$ then you have failed to reject the hypothesis that the means are equal, you can either do another experiment with a larger sample size to try to detect a significant difference or you can accept that it's not known if $\mu_A=\mu_B$ or $\mu_A \neq \mu_B$.

If you get $p=0.99$ this tells you that the difference between sample means was likely under the hypothesis that $\mu_A=\mu_B$, but it could also be likely to observe that difference between the sample means under many other hypotheses.

In other words, your experimental results could be likely in cases where there's no difference and in cases where there's a big difference between the means. So although a low p-value tells you that the means are different, a high $p$ value doesn't tell you that the means are the same.