5
$\begingroup$

I have been recently doing two-sample analysis and have a little trouble interpreting the resulting $p$-value. The $p$-value was 0.12, thus at the risk rate of 5% I cannot say the samples are different.

However, does $p=0.12$ still mean I can say there is a detectable difference between samples, just not as obvious?

My current intuition is that until $p<0.5$ the interpretation that the datasets are different is more plausible that they are the same, yet of course at e.g 0.4 the risk of that claim is very high and thus maybe not useful.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ +1. Very good and interesting question, and I don't think it's been really answered yet (even though you marked one answer as accepted). $\endgroup$
    – amoeba
    Commented Mar 20, 2016 at 15:40

3 Answers 3

Reset to default
3
$\begingroup$

The wikipedia for p-value should give you a precise definition. Essentially, it's the probability how your null hypothesis would be inconsistent to your data.

With a p-value of 0.12, you would need a large significance level to reject your hypothesis.

  • You can think like there is some difference between the samples, but not as inconsistent as you might have thought. If you have no or very little difference, your p-value should be close to 1.

  • Do you mean p < 0.05? 0.5 is simply too big.

  • You should never think your groups are different or same, because they will never be the same. If they were the same, you wouldn't have to do a statistical test at all. You should ask yourself, is a p-value of 0.12 enough to convince you that two groups are statistically different enough such that you can call them significant? The answer is no, we usually reject the hypothesis when the p-value is less than 0.05.

Your results indicate your groups are not statistically significant unless you want a significance level 12%.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for clarification. I am fully aware this example needs higher significance level, I will stick to very strict and precise interpretation. $\endgroup$
    – sdgaw erzswer
    Commented Feb 28, 2016 at 18:46
  • 3
    $\begingroup$ Regarding your second bullet: I think the OP meant 0.5 (and not 0.05) with the idea being that 0.5=1/2 is half-way between 0 and 1. $\endgroup$
    – amoeba
    Commented Mar 20, 2016 at 15:37
  • 3
    $\begingroup$ The key phrase "the probability how your null hypothesis would be inconsistent to your data" is ungrammatical and ambiguous--perhaps it got corrupted? It looks all too easy to misinterpret it and come away with exactly the wrong idea of what a $p$-value is. $\endgroup$
    – whuber ♦
    Commented Mar 20, 2016 at 16:23
2
$\begingroup$

If you want an interpretation that corresponds to the posterior probability of the alternative hypothesis being true, then you would likely want a Bayesian analysis. A frequentist analysis (as you had conducted) may or may not have the Bayesian properties one "intuitively desires" (such as allowing such an interpretation in the absence of prior information). Often a p < 0.5 will offer some (often extremely weak) support for the alternative, but that is not necessarily so.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Good point about Bayesian analysis, however p-values can fail to correspond to Bayesian posterior probabilities even for the low p-values; one can have $p<0.05$ but $P(H_0|\mathrm{data})>0.5$ -- see Lindley's paradox. Currently your answer can be interpreted as if such disagreements can arise only for large p-values. $\endgroup$
    – amoeba
    Commented Mar 20, 2016 at 15:31
  • $\begingroup$ So if I understand correctly, doing the Bayesian alternative of e.g., t-test is better option? $\endgroup$
    – sdgaw erzswer
    Commented Nov 6, 2017 at 7:36
  • $\begingroup$ A Bayesian analysis (with informative priors that reflect the knowledge on the question under analysis) is the one, where you could interpret results in the sense of what parameter values are more probable. A frequentist analysis simply cannot be interpreted that way, except when - in the absence of any relevant prior information - the two approaches give the same answer (e.g. often the case for t-test analogues). $\endgroup$
    – Björn
    Commented Nov 6, 2017 at 12:12
1
$\begingroup$

When you achieve a certain significance level, whatever it is 0.001, 0.01, 0.4, you are saying that either (a) there is something real going on or (b) something unusual has happened. So in your case either there is a real difference or something that would happen roughly once in every six attempts by chance has happened. How unusual do you want to be? Most people want more unusual than that.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ This is a common kind of interpretation. But nothing makes (a) and (b) mutually exclusive! $\endgroup$
    – Nick Cox
    Commented Mar 21, 2016 at 0:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.