8
$\begingroup$

I am a little confused about what p-values mean under Fisher's signficance testing & Neyman-Pearson's hypothesis testing.

Fisher uses p-values as a continuous measure of evidence against a null hypothesis? So a p-value of 0.06 would indicate that there is no difference and the null hypothesis is true?

However, does this mean the same thing under Neyman-Pearson. I know that you have to pre-set alpha values for type 1 errors but does this affect p? Does a p-value greater than alpha indicate that there is >5% chance of a type one error occurring?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
12
$\begingroup$

Fisher uses p-values as a continuous measure of evidence against a null hypothesis?

Perhaps. What convinces you of this?

So a p-value of 0.06 would indicate that there is no difference and the null hypothesis is true?

Not at all. How did you go from 'continuous measure of evidence against' to 'there is no difference'?

In particular, Fisher would not make the mistake of thinking that failure to reject makes $H_0$ actually true.

Does a p-value greater than alpha indicate that there is >5% chance of a type one error occuring

No, for two reasons.

(i) if $p>\alpha$ you won't reject, so you can't commit a type I error at all

(ii) You don't even have an $\alpha$ probability of making a type I error, since the type I error rate is a conditional probability, and in real situations, the joint probability is close to zero (that is, point null hypotheses are almost never exactly true; you can only make a type I error when they are exactly true).

[ ... I suppose that I'm arguably acting more as a Bayesian there]

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ +1 Minor quibble: many null hypotheses are indeed exactly true, especially in one-sided tests and in other situations with composite nulls. $\endgroup$
    – whuber ♦
    Commented Dec 5, 2013 at 23:36
  • $\begingroup$ @whuber Yes, I should have specified that I meant point nulls. I will edit. $\endgroup$
    – Glen_b
    Commented Dec 6, 2013 at 4:35
  • $\begingroup$ Minor quibble for a minor quibble answer: point null can also be exactly true (due to laws of physics). Say, anything I do on Earth can only begin to affect the Sun in 8 minutes from now. The effect of me clapping hands right now on the position of Sun spots as recorded by the Sun telescope at that same moment as I am clapping is exactly zero. $\endgroup$
    – akhmed
    Commented Apr 23, 2018 at 8:23
  • $\begingroup$ Even with physics you must be careful; what our best models say does change over time. An exact equality under Newtonian physics turns out in particular situations not to quite work (the null in a Newtonian model of the situation is not exactly true and eventually we can identify situations where we can spot it). Einstein comes along and we then get agreement to as much accuracy as we can muster - but we must remember that was once true for Newton. Or we have a conservation law that we can eventually see doesn't quite hold and eventually we refine it to a broader law that does seem to hold. $\endgroup$
    – Glen_b
    Commented Apr 24, 2018 at 0:57
  • $\begingroup$ As a result even in physics we must at all times seriously consider the possibility that our point null -- while it looks good now -- may in fact be false, even though it must be an excellent approximation in many situations $\endgroup$
    – Glen_b
    Commented Apr 24, 2018 at 0:58
8
$\begingroup$

The issue here is that you need to be clearer on the definitions of these terms, and what those definitions imply.

Taking the p-value as a continuous measure of evidence against the null hypothesis means that there is no 'bright line' between "no difference" and "difference". As a result of this, $p=.04$ is essentially identical to $p=.06$, $p=.06$ is essentially identical to $p=.08$, and, moreover, $p=.04$ is still pretty similar to $p=.08$, in terms of the amount of evidence against the null hypothesis.

If you follow the Neyman-Pearson paradigm correctly, you would not reject the null hypothesis when $p>\alpha$. Thus, a type I error is not possible. Remember that a type I error is defined as rejecting the null hypothesis when it's true. Since you're not rejecting the null hypothesis, this can't apply.

It may help you to read my answer to a related question: When to use Fisher and Neyman-Pearson framework?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.