11
$\begingroup$

I came across this image in a blog post here.

Someone in a mosh pit experiencing a EUREKA! moment

I was disappointed that reading the statement did not elicit the same facial expression for me as it did for this guy.

So, what is meant by the statement that the null hypothesis is how frequentists express an uninformative prior? Is it really true?


Edit: I'm hoping someone can offer a charitable interpretation that makes the statement true, even in some loose sense.

$\endgroup$
  • 2
    $\begingroup$ I don't think it is accurate. First, can someone write down the Likelihood of a T-Test? Then we can start talking about analogies. Well, and if you can't... we'll that picture does not make sense. $\endgroup$ – joint_p May 2 '13 at 22:11
9
$\begingroup$

The null hypothesis isn't equivalent to a Bayesian uninformative prior for the simple reason that Bayesians can also use null hypotheses and perform hypothesis tests using Bayes' factors. If they were equivalent, Bayesians wouldn't use null hypotheses.

However, both frequentist and Bayesian hypothesis testing incorporate an element of self-skepticism, in that we are required to show that there is some evidence that our alternative hypothesis is in some way a more plausible explanation for the observations than random chance. Frequentists do this by having a significance level, Bayesians do this by having a scale of interpretation for the Bayes factor, such that we wouldn't strongly promulgate a hypothesis unless the Bayes factor over the null hypothesis were sufficiently high.

Now the reason why frequentist hypothesis tests are counter-intuitive is because a frequentist cannot assign a non-trivial probability to the truth of a hypothesis, which sadly is generally what we actually want. The closest they can get to this is to compute the p-value (the likelihood of the observations under H0) and then draw a subjective conclusion from this as to whether H0 or H1 are plausible. The Bayesian can assign a probability to the truth of a hypothesis, and so can work out the ratio of these probabilities to provide an indication of their relative plausibilities, or at least of how the observations change the ratio of these probabilities (which is what a Bayes factor does).

In my opinion it is a bad idea to try to draw too close a parallel between frequentist and Bayesian hypothesis testing methods as they are fundamentally different and answer fundamentally different questions. Treating them as if they were equivalent encourages a Bayesian interpretation of the frequentist test (e.g. the p-value fallacy) which is potentially dangerous (for example climate skeptics often assume that a lack of a statistically significant trend in global mean surface temperature means that there has been no warming - which is not at all correct).

$\endgroup$
9
$\begingroup$

The reason you don't have the same epiphanic look on your face as that guy is I think that . . . the statement isn't true.

A null hypothesis is the hypothesis that any difference between the control and experimental conditions is due to chance.

An uninformative prior is meant to state that you have prior data on a question, but that it doesn't tell you anything about what to expect this next time round. A Bayesian is likely to maintain that there's information in any prior, even the uniform distribution.

So the null hypothesis says that there's no difference between control and experimental; an uninformative prior on the other hand may or may not be possible, and if it did would indicate nothing about the difference between control and experimental (which is different from indicating that any difference is due to chance).

Perhaps I am lacking in my understanding of uninformative priors, though. I look forward to other answers.

$\endgroup$
  • 2
    $\begingroup$ I would only add that noninformative priors are more about the attitude of the researcher than any especially interesting properties of the distribution itself. This is the attitude that Gelman argues for in Bayesian Data Analysis, although I can't seem to find the page number. $\endgroup$ – Sycorax Apr 24 '13 at 19:27
  • 7
    $\begingroup$ The null hypothesis is not always the same. The null hypothesis is just an alternative "boring" hypothesis that you compare to your "interesting" hypothesis, to see whether the data supports one over the other. Actually, "no difference" is actually a bad null hypothesis, since you know a-priori it's false. Better is "the difference is below some threshold of me caring". $\endgroup$ – Stumpy Joe Pete Apr 24 '13 at 20:28
  • $\begingroup$ Thanks for the answer @Krysta, and I basically had the same thoughts to the statement, but perhaps there's a sense in which the statement is kind of true? $\endgroup$ – jerad Apr 24 '13 at 21:07
  • $\begingroup$ My best guess is that the null hypothesis is the starting point for frequentists, or the empty set of hypotheses?; perhaps this writer thinks that the uninformative prior is the starting point for Bayesians, but a regular informative prior is a better analog if that's what they meant. Null hypothesis & uninformative prior do sorta have conceptual similarities--they're both sort of about assuming no information/effect. But that's pretty vague! $\endgroup$ – Krysta Apr 24 '13 at 21:11
  • $\begingroup$ "A Bayesian is likely to maintain that there's information in any prior". But, a Jeffreys prior is uninformative. $\endgroup$ – Neil G Apr 30 '13 at 16:35
4
$\begingroup$

See this Wikipedia article:

For the case of a single parameter and data that can be summarized in a single sufficient statistic, it can be shown that the credible interval and the confidence interval will coincide if the unknown parameter is a location parameter (...) with a prior that is a uniform flat distribution (...) and also if the unknown parameter is a scale parameter (...) with a Jeffreys' prior.

In fact, the reference points to Jaynes:

Jaynes, E.T. (1976), Confidence Intervals vs Bayesian Intervals.

In page 185 we can find:

If case (I) arises (and it does more often than realized), the Bayesian and orthodox tests are going to lead us to exactly the same results and the same conclusion, with a verbal disagreement as whether we should use 'probability' or 'significance' to describe them.

So, in fact there are similar cases, but I wouldn't say the statement in the image is truth if you are, for example, using a Cauchy distribution as likelihood...

$\endgroup$
4
$\begingroup$

I'm the one that created the graphic, though as noted in the accompanying post it's not originally my insight. Let me provide some context for how it came up and do my best to explain how I understand it. The realization occurred during a discussion with a student who had mostly learned the Bayesian approach to inference up to that point. He was having a hard time understanding the whole hypothesis testing paradigm, and I was doing my best to explain this decidedly confusing approach (if you consider “difference” to be a negative - as in not equal to - then the standard null hypothesis approach is a triple negative: the researchers’ goal is to show that there is not no difference). In general, and as stated in another response, the researchers usually expect some difference to exist; what they really hope to find is convincing evidence to “reject” the null. To be unbiased, though, they begin by essentially feigning ignorance, as in, “Well, maybe this drug has zero effect on people.” Then they proceed to demonstrate through data collection and analysis (if they can), that this null hypothesis, given the data, was a bad assumption.

To a Bayesian, this must seem like a convoluted starting point. Why not just begin by announcing your prior beliefs directly, and be clear about what you are (and aren't) assuming by encoding it in a prior? A key point here is that a uniform prior is not the same as an uninformative prior. If I toss a coin 1000 times and get 500 heads, my new prior assigns equal (uniform) weight to both heads and tails, but its distribution curve is very steep. I am encoding additional information that is highly informative! A true uninformative prior (taken to the limit) would carry no weight at all. It means, in effect, starting from scratch and, to use a frequentist expression, let the data speak for itself. The observation made by "Clarence" was that the frequentist way to encode this lack of info is with the null hypothesis. It’s not exactly the same as an uninformative prior; it's the frequentist approach to expressing maximal ignorance in an honest way, one that doesn't presume what you wish to prove.

$\endgroup$
  • 2
    $\begingroup$ The frequentist null hypothesis does not express maximal ignorance, it starts of assuming that the null hypothesis is true and we should only proceed with the alternative hypothesis if the observations are sufficiently unlikely under H0. It might be argued that null hypothesis testing encodes some prior, but it is a decidedly informative one. In my opinion attempting to interpret frequentist hypothesis testing in Bayesian terms is misguided and a recipe for error; they are not answers to the same question. $\endgroup$ – Dikran Marsupial May 3 '13 at 6:14
  • $\begingroup$ @Dikran Marsupial this is to some extent an endless debate, but from a frequentest perspective I see no way to view the null as "decidedly informative". If this were the case, then failing to reject the null would be viewed as proof of the null (since we "already" have information about the null). IMO all approaches to inference are attempting to answer the same interrelated questions: "How should the data be interpreted?" and "how strong is the case?" $\endgroup$ – Matt Asher May 3 '13 at 12:56
  • 1
    $\begingroup$ The null itself isn't informative or uninformative, but conventional frequentist hypothesis testing is inherently (and quite rightly) biased towards the H0 (unless you also perform a power analysis). This bias can be compared to a prior, but it would be an informative one. It simply isn't meaningful to compare priors and hypotheses, they serve different purposes in the analysis; note Bayesian also use null hypotheses in hypothesis testing (see my answer to the question) where it serves the same purpose as in frequentist hypothesis testing. $\endgroup$ – Dikran Marsupial May 3 '13 at 14:45
  • 2
    $\begingroup$ To be clear, using the drug example, we don't start by feigning ignorance "Well, maybe this drug has zero effect on people.", we start by assuming the null hypothesis is correct "The drug has zero effect and it is up to the drug company to establish that it does have an effect by showing that the results cannot be adequately explained by random chance". The self-skepticism that this approach provides is why the "null ritual", despite its many faults, is still of practical value in science. $\endgroup$ – Dikran Marsupial May 3 '13 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.