3
$\begingroup$

I've seen the following equivalent statements about p-values:

  • "It is the probability of wrongly rejecting the null hypothesis if it is in fact true." [1]

  • "The P value or calculated probability is the estimated probability of rejecting the null hypothesis (H0) of a study question when that hypothesis is true." [2]

Some people I know insist these are wrong, and say these are equivalent to type I error. But the type I error is in fact determined by the significance level. Are these right or wrong?

[1] http://www.stats.gla.ac.uk/steps/glossary/hypothesis_testing.html#pvalue

[2] http://www.statsdirect.com/help/default.htm#basics/p_values.htm

$\endgroup$
  • $\begingroup$ This previous discussion might help. $\endgroup$ – Julian Schuessler Mar 6 '14 at 20:42
4
$\begingroup$

There are two p-values of interest. The critical p-value, also known as the $\alpha$-level or significance level, is decided and fixed before the study / analysis is performed. This critical p-value is in fact the probability of Type I error. Your examples are talking about the critical p-value, and I agree with those who are saying they are incorrect or at least misleading, because in common usage "the p-value" refers to the next type of p-value.

The observed p-value, or calculated p-value, is calculated as part of a statistical test. The observed p-value is the probability under the null hypothesis of observing data that leads to a test statistic as or more extreme than the test statistic actually observed in the experiment. The definition of "extreme" varies depending on your test, e.g. one-sided versus two-sided. The common usage of the phrase "the p-value" usually refers to the observed p-value.

We reject the null hypothesis if the observed p-value is less than or equal to the critical p-value. This is because, under the null hypothesis, the observed p-value is uniformly distributed on $[0,1]$. Thus the probability that we reject the null when in fact the null is true is equal to the critical p-value.

Edit: The full section in [1] is talking about the observed p-values, but the one line you quoted only makes sense when talking about the critical p-value. If you were to force it to be about observed p-values and be correct, it would be something like "It is the probability of a new, identical study wrongly rejecting the null hypothesis if you set your alpha-level equal to it." Strictly speaking, once you've done your analysis, the probability of wrongly rejecting the null hypothesis is either 0 or 1, depending on whether or not the null hypothesis is true and whether or not you rejected it!

$\endgroup$
  • $\begingroup$ Thanks. But 1 seems to make the statement I quoted about (observed) p-values, not significance levels. Hence the confusion. $\endgroup$ – John Tyson Mar 6 '14 at 20:51
  • $\begingroup$ I've updated my answer to address this. $\endgroup$ – P Schnell Mar 6 '14 at 21:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.