Suppose that I have two systems $A$ and $B$ whose output are $O_A$ and $O_B$, respectively. The statistic of interest that I use to decide which system is better is $s(O_A)$ and $s(O_B)$, respectively. This statistic is larger when a system is better.
I found that $s(O_A) > O(O_B)$, therefore I concluded that the system $A$ is better than the system $B$. However the question at hand is whether such difference is due to a systematic difference, as opposed to sheer dumb luck.
So I asked my self the question: "What is the probability of observing $s(O_A) > s(O_B)$ if the differences between $A$ and $B$ were non-systematic?". This question is also equivalent to the question: "What is the probability of observing differences $\ge |s(O_A) - s(O_B)|$ under the null hypothesis?".
To answer that question, I used approximate randomization and it gave me a $p$ value. My questions are centered around how to interpret such $p$ values.
- Q1: When to say that differences between $A$ and $B$ are significant.
- Q2: When to say that difference between $A$ and $B$ are insignificant.
- Q3: When to say that available data is not enough to conclude anything at all.
Examples for further clarification:
For example, usually when $p \le 0.05$, it is concluded that differences between $A$ and $B$ are significant. This is probably a classical use case of $p$ values.
But what if $p \approx 0.1$? Should we state that the difference between $A$ and $B$ is insignificant? Or should we refrain from stating anything and conclude that the data is simply too small to allow for any healthy conclusion?
Similarly, what if $p \approx 0.999$? Should we finally be able to state that the differences between $A$ and $B$ are insignificant?