51
$\begingroup$

In the article "Discussion: Should Ecologists Become Bayesians?" Brian Dennis gives a surprisingly balanced and positive view of Bayesian statistics when his aim seems to be to warn people about it. However, in one paragraph, without any citations or justifications, he says:

Bayesians, you see, are not allowed to look at their residuals. It violates the likelihood principle to judge an outcome by how extreme it is under a model. To a Bayesian, there are no bad models, just bad beliefs.

Why would a Bayesian not be allowed to look at the residuals? What would be the appropriate citation for this (i.e. who is he quoting)?

Dennis, B.
Discussion: Should Ecologists Become Bayesians?
Ecological Applications, Ecological Society of America, 1996, 6, 1095-1103

Improve this question
$\endgroup$
  • 6
    $\begingroup$ If that argument worked, frequentists couldn't use the likelihood principle either - for the same reason. $\endgroup$ – Glen_b Feb 6 '14 at 9:32
  • 3
    $\begingroup$ @Glen: A frequentist truly beholden to the LP (the weak version, equivalent to the Sufficiency Principle - the strong version is simply incompatible with the frequentist approach) would have to shun model checking. The ones who just admire it are glad when they can use it for the job of estimating the parameters of a specified model & still have more-or-less independent ancillaries - the residuals - left over for model checking any old how. $\endgroup$ – Scortchi - Reinstate Monica♦ Feb 6 '14 at 11:21
  • 1
    $\begingroup$ Even when the frequentist do ML estimation he still violates the LP because he considers the sampling distribution of the MLE to find a confidence interval for his estimate. $\endgroup$ – Zen Feb 6 '14 at 14:24
  • 2
    $\begingroup$ @Zen: He doesn't violate the weak LP as long as the confidence interval depends on the data only through the likelihood function. But he may sooner or later violate the strong LP by making a different confidence interval based on the same likelihood function from a different experiment with a different sampling space. $\endgroup$ – Scortchi - Reinstate Monica♦ Feb 6 '14 at 15:27
  • 2
    $\begingroup$ @BayesianNewbie you might find the work of Dempster interesting. See for instance the following passage in Logicist Statistics I. Models and Modeling: "Unfortunately, a subclass of Bayesian subjectivists courts ridicule by ignoring the intrusions of objective evidence into model construction, and have long raised the ire of objectivists who accuse them of using prior distribution to approximate mental states of uncertainty that are plainly poorly constructed" $\endgroup$ – Sextus Empiricus Jun 9 '20 at 14:06
19
$\begingroup$

Of course Bayesians can look at the residuals! And of course there are bad models in Bayesian analysis. Maybe a few Bayesians in the 70's supported views like that (and I doubt that), but you will hardly find any Bayesian supporting this view these days.

I didn't read the text, but Bayesians use things like Bayes factors to compare models. Actually, a Bayesian can even compute the probability of a model being true and pick the model which is more likely to be true. Or a Bayesian can average across models, to achieve a better model. Or can use posterior predictive checks. There are a lot of options to check a model and each one may favor one approach or another, but to say that there are no bad models in Bayesian analysis is non-sense.

So, at most, it would be more appropriate to say that in some extreme versions of Bayesianism (extreme versions that almost no one uses in applied settings, by the way) you're not allowed to check your model. But than you could say that in some extreme versions of frequentism you're not allowed to use observational data as well. But why waste time discussing these silly things, when we can discuss if and when, in an applied setting, we should use Bayesian or frequentist methods or whatever? That's what's important, in my humble opinion.

Update: The OP asked for a reference of someone advocating the extreme version of Bayes. Since I never read any extreme version of Bayes, I can't provide this reference. But I'd guess that Savage may be such a reference. I never read anything written by him, so I may be wrong.

ps.: Think about the problem of the "well-calibrated Bayesian" (Dawid (1982), JASA, 77, 379). A coherent subjectivist Bayesian forecaster can't be uncalibrated, and so wouldn't review his model/forecasts despite any overwhelming evidence that he's uncalibrated. But I don't think anyone in practice can claim to be that coherent. Thus, model review is important.

ps2.: I like this paper by Efron as well. The full reference is: Efron, Bradley (2005). "Bayesians, frequentists, and scientists." Journal of the American Statistical Association 100(469).

Improve this answer
$\endgroup$
  • 5
    $\begingroup$ I also assumed that the prohibition was never taken seriously in practice, so I was surprised to read this from Gelman:- " I certainly don’t want to return to the circa-1990 status quo in Bayesian statistics, in which it was considered virtually illegal to check your model’s fit to data." $\endgroup$ – Scortchi - Reinstate Monica♦ Feb 6 '14 at 12:36
  • 1
    $\begingroup$ I don't know how was Bayesian statistics in the nineties. But it's hard to believe that in applied settings Bayesians didn't check their models. Maybe they checked, but didn't tell! $\endgroup$ – Manoel Galdino Feb 6 '14 at 12:53
  • 2
    $\begingroup$ You're probably right: illegal$\neq$unusual. Perhaps it was more common back then, at least among academics, to defend Bayesian methods in principle rather than pointing to evident successes in application, & therefore any non-conformity to (real or supposed) principles would get swept under the rug. $\endgroup$ – Scortchi - Reinstate Monica♦ Feb 6 '14 at 13:02
  • $\begingroup$ I definitely agree that this is not a major issue, I was just curious if anybody had published on this. Have you ever read anybody advocating these "extreme versions of Bayesianism"? $\endgroup$ – Mankka Feb 6 '14 at 13:30
37
$\begingroup$

They can look but not touch. After all, the residuals are the part of the data that don't carry any information about model parameters, and their prior expresses all uncertainty about those—they can't change their prior based on what they see in the data.

For example, suppose you're fitting a Gaussian model, but notice far too much kurtosis in the residuals. Perhaps your prior hypothesis should have been a t-distribution with non-zero probability over low degrees of freedom, but it wasn't—it was effectively a t-distribution with zero probability everywhere except on infinite degrees of freedom. Nothing in the likelihood can result in non-zero probabilities over regions of the posterior density where the prior density is zero. So the notion of continually updating priors based on likelihoods from data doesn't work when the original prior is mis-specified.

Of course if you Google "Bayesian model checking", you'll see this is a parody of actual Bayesian practice; still, it does represent something of a difficulty for Logic of Science-type arguments for the superiority of Bayesianism on philosophical grounds. Andrew Gelman's blog is interesting on this topic.

Improve this answer
$\endgroup$
  • $\begingroup$ Do you have any references on this "difficulty for Logic of Science"? $\endgroup$ – Mankka Feb 6 '14 at 13:32
  • 8
    $\begingroup$ I was referring to Jaynes, Probability Theory: The Logic of Science, in which the repeated use of Bayes' theorem to update probability distributions as new data comes in is claimed to be a paradigm for the growth of scientific knowledge. I'm sure he deals with the problem of a prior that is too narrow, but I can't remember how, or how satisfactorily. And I'm going to change "general superiority" to "superiority on philosophical grounds", as that seems to better convey what I meant. $\endgroup$ – Scortchi - Reinstate Monica♦ Feb 6 '14 at 13:54
  • $\begingroup$ This example of Bayesian prior usage was applied to reduce the occasional (2%) occurrence of unphysical outcomes. This lack of physicality has been attributed to a non-physical instant mixing (of drug in the body) and was corrected by assuming zero initial mixing using a better model. It seems better to adapt the model to the problem than to fudge the answers to conform to preconceptions. (+1) $\endgroup$ – Carl Nov 1 '18 at 21:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.