I understand that NHST assumes the null hypothesis is true, and that we can't infer from a low-value that a null hypothesis is necessarily false. However, is it fair to say that the p-value tells us nothing about the conditional probability $Pr(H_0 = T|T_{obs})?$

If it is fair to say that p-values tell us nothing about the probability null hypotheses are true, why does anyone use them?

  • 1
    $\begingroup$ Sounds like what you want is Bayesian inference, not frequentist inference :) $\endgroup$ Mar 18 '14 at 2:59
  • $\begingroup$ I think most probably I would! But I'm actually asking just to double-check the accuracy of a comment that was made to me in conversation; my question doesn't relate directly to any research I'm undertaking. $\endgroup$ Mar 18 '14 at 3:02
  • 1
    $\begingroup$ I know. I'll let others give you better answers, but I'd say yes to your question, p-values tell you nothing about the null hypothesis--p-values give you $Pr(\text{Data}|H_0)$, while what you would like is $Pr(H_0|\text{Data})$. As I hinted to in my previous comment, Bayesian statistics can give you the latter. $\endgroup$ Mar 18 '14 at 3:06
  • 2
    $\begingroup$ I would say no to your question. For example, if P(data|H0) is zero (the observed data could not have occurred under H0) then P(H0|data) is also zero. $\endgroup$
    – Flounderer
    Mar 18 '14 at 4:22

This is almost certainly a repeat question, but it remains a good one. It is also a question that is likely to elicit contradictory answers, some of which might be misleading.

The "NHST" in your question is not a 'real' thing, but is a hybrid of the frequentist hypothesis test and the Fisherian significance test. The hybrid is very confusing as, according to Gigerennzer, it is a mishmash. The distinction between the two approaches is made clear in a paper of mine that is intended to be accessible to non-statistically sophisticated readers: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3419900/

The short answer to your question is that the result of a Neyman-Pearsonian hypothesis test is not a type of evidence and so it is not able to support inferences about the truth or falsehood of hypotheses. Within a strictly frequentist paradigm there is no such thing as evidence because, as Neyman and Pearson said quite clearly in their original paper about frequentist hypothesis tests, their approach relates to long-run error rates and not to how data reflect on the truth or otherwise of a particular hypothesis:

We are inclined to think that as far as a particular hypothesis is concerned, no test based upon the theory of probability can by itself provide any valuable evidence of the truth or falsehood of that hypothesis.

But we may look at the purpose of tests from another view-point. Without hoping to know whether each separate hypothesis is true or false, we may search for rules to govern our behaviour with regard to them, in following which we insure that, in the long run of experience, we shall not be too often wrong.

However, the p-values in your question do not have to be seen as Neyman-Pearson things that served as criteria for dichotomous accept or reject decisions. When interpreted as a continuous scale they serve as an index of the strength of evidence against the null hypothesis. Not a measure, but an index. As a number they directly they quantify the probability of observing data as discrepant as those observed if the null hypothesised parameter value is true, but that is not often exactly what is wanted by an experimenter, and it is not a useable numerical value of the strength of evidence.

Statistics doesn't have a clear agreed and unambiguous conception of evidence. However, the statistical object that has, to my mind, the best claim to being a quantification of evidence is the likelihood function. P-values serve as one-to-one index to members of the relevant family of likelihood functions and so in that sense they are indices to the evidence functions. I've written about that too, but it has so far been rejected by several journals, although its arguments have not been refuted and its evidence not contested. I've put a version on ArXive: http://arxiv.org/abs/1311.0081

If you want to be able to evaluate the evidence for and against various hypothesised parameter values, including the null hypothesis, then the likelihood function that the p-value points to is your friend. The strength of evidence in favour of one hypothesis over another is measured by the ratio of the likelihoods at those two parameter values.

If you want to change your opinion on the merits of various hypotheses, then you need to use a Bayesian approach, as Patrick Coulombe points out in his brief comments. In such an approach the likelihood function has to be multiplied by your prior probability distribution and scaled to yield the Bayesian posterior probability distribution. The likelihood function is the same as the one that is pointed to by the p-value and so, while it is true that the probability of the hypo[thesis can only be calculated by a Bayesian approach, it is not true to think that the p-value has nothing to say about it. A very small p-value says that the data speak strongly against the null hypothesis, whereas a large p-value says that the data do not contradict the null hypothesis. That is more than nothing, even while it is not exactly what you might want.

  • 1
    $\begingroup$ (+1) Michael, I have enjoyed reading your papers and answers here and elsewhere on CV. However, I am still struggling to understand what is so bad about F/N-P "hybrid"/"mishmash". Let's say I report p as either <0.05, <0.01, or <0.001, and it it's >0.05 declare it not significant. This is, quoting from your "Bad statistical practice..." paper, "a hybrid approach that neither controls error rates nor allows assessment of the strength of evidence" -- why? Error rate is controlled at 0.05. Strength of evidence is assessed, albeit in a suboptimal, quantized way. What is the big deal? $\endgroup$
    – amoeba
    Aug 18 '14 at 9:57
  • $\begingroup$ (cont.) In addition, could you maybe point me to some texts that argue against such "hybrid" in more detail? I read Gigerenzer 1993 and Cohen 1994, but am still at a loss. This looks relevant Halpin&Stan 2006), but I do not have access. Let me know if my question is reasonable, but comment space is too small to provide an answer; then I could post it as a separate question on CV. I have not done it so far, because I am afraid I am missing something obvious. $\endgroup$
    – amoeba
    Aug 18 '14 at 10:02
  • $\begingroup$ @amoeba The difficulty you are having is probably due to not fully recognising that the error rate of the procedure relates to the procedure itself, not to the experiment in which you used the procedure. Thus, if you would have claimed significance for any observed P<0.05, then 0.05 is the error rate even if the observed P-value is less than some smaller value. $\endgroup$ Aug 20 '14 at 1:45
  • $\begingroup$ The problem with 'suboptimal' evaluation of evidence is that it is, well, suboptimal! $\endgroup$ Aug 20 '14 at 1:47
  • $\begingroup$ Thanks for the reply, @Michael! I think I understand these both points, but I still cannot see why you (and others) claim that "a hybrid approach neither controls error rates nor allows assessment of the strength of evidence". I think it does both! Maybe assessment of the strength of evidence is done "suboptimally", but it is still done. And the error rate is controlled at 0.05, as you say (independent of the observed p-value, yes). I don't see anything "inconsistent" or "mishmashy" here. $\endgroup$
    – amoeba
    Aug 20 '14 at 9:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.