Residuals and errors are related but not exchangeable. In Wikipedia I read:

In statistics and optimization, statistical errors and residuals are two closely related and easily confused measures of the deviation of an observed value of an element of a statistical sample from its "theoretical value". The error of an observed value is the deviation of the observed value from the (unobservable) true function value, while the residual of an observed value is the difference between the observed value and the estimated function value.

I would agree with this - but only if we look at it from a frequentist perspective. That is, we have to know the one true population value in order to be able to talk about statistical errors. Now, as far as I understand, in the Bayesian framework there are no such things as true values.

What does that mean for the use of the term "error"? In principle one could probably still talk about "errors" and assume that they, themselves have another underlying distribution but somehow it seems not correct to me to use the above definition in a framework without true values?

Any thoughts on this?


1 Answer 1


in the Bayesian framework there are no such things as true values.

Although this is sort of true because basically everything is a random variable in the bayesian paradigm, I think it's incorrect to disallow the notion of a "true distribution" (i.e. true sampling function) in the bayesian paradigm. If this is the case, error could be understood as the distance between the prediction and the MAP estimate of the value under its true (unknown) distribution, which the Bayesian is attempting to converge upon in the posterior. If we're making point estimates (otherwise I'm not sure the notion of "residuals" has meaning, either) then this definition seems suitable and analogous to the frequentist use of the term.

EDIT: Not necessarily "MAP" estimate, I guess. Really what I mean is whatever estimation procedure is being applied to the posterior, use that estimation procedure on the "true distribution" and let the difference between these estimates be your error.

  • 3
    $\begingroup$ I agree there is a true distribution in the Bayesian framework. In the mathematical treatment of Bayesian statistics we consider the parameter as a random variable distributed according to the prior distribution, but the practical interpretation is different: there is a true value of the parameter, and the prior distribution is a distribution describing our beliefs about it. $\endgroup$ Jul 15, 2013 at 8:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.