3
$\begingroup$

I'm reading up on Bayesian techniques for Linear models and time series. While the texts are great at teaching the theory I would like to get a better handle on the pro's/con's of Bayesian analysis vs their frequentist equivalents.

I can't find an article or even articles that do this. I really want it as a fairly high level rather than getting caught up in the technical detail right away. Ideally highlighting where and in what situations Bayesian is better and vice versa.

Most just hint that the frequestist approaches also rely on prior information but are less honest about this, but it didn't go into much detail.

$\endgroup$
  • 1
    $\begingroup$ Your last sentence is a statement that many people would regard as incorrect, whether it is your summary or someone else's. Seems to me that time series analysis is finding more value for models based on updating in time in a way that makes any such debate irrelevant as well as wearisome. New information lets you change your mind, either way. $\endgroup$ – Nick Cox May 22 '13 at 14:32
  • 1
    $\begingroup$ You might want to review the questions that already exist on this site under the tag bayesian. Your question is a bit loose, but I suspect much of what you want to know is already there among several extant questions. Reading that material & thinking about it might let you focus your question more, & get info that's more helpful to you. $\endgroup$ – gung - Reinstate Monica May 22 '13 at 14:46
  • $\begingroup$ Will do thanks Gung. Did not mean to be wearisome, just trying to get a handle on the differences between the two approaches. Seems most texts for the beginner prefer to concentrate on theory rather than the philosophy of the approach and its relative merits/demerits vs frequentist approaches. $\endgroup$ – Baz May 22 '13 at 15:24
  • 1
    $\begingroup$ Sorry about any ambiguity: it's the never-ending debate out there that is wearisome, not your question. $\endgroup$ – Nick Cox May 22 '13 at 16:57
2
$\begingroup$

Bayesian and Frequentist statistics rest upon a different understanding of what a probability distribution is. To a Bayesian, probability reflects degree of belief. The purpose of an experiment (or data collection) is to update one's beliefs about the parameter values of the system. And since our beliefs concern parameters (unknown) rather than data (which we see), a probability distribution is attached to the parameters. Confusingly, a pdf is also attached to the data.

To a frequentist, a probability distribution reflects the frequency with which events occur. A theoretical distribution is (in theory), the (measure theoretic) limit of the observed frequency distributions. The observed frequency distribution must, in turn, reflect the sampling scheme used to collect the data, since what is at issue is the probability of given items showing up in the sample you select.

But you know all this.

I did my doctorate at CMU, when Morrie Degroot and Jay Kadane were there. It was a big Bayesian shop in those days, and perhaps still is. At no time did anyone talk about the cost-benefits of Bayesian vs frequentist analyses. It was a simple question of right vs wrong. No one at CMU would have thought of using a Bayesian method for time series (say), but not in everything else. This may account for the absence of papers on this subject.

In general, Bayesian methods cost more than frequentist ones, since the calculations are more complex and laborious. That used to matter. Now it doesn't, which may explain the proliferation of Bayesian methods.

There is also a mental cost involved with Bayesian methods, since you have to justify to yourself why, in an exploratory situation where you know nothing and you have an apparently symmetric and Normal-like sample, the sample average is not an acceptable estimate of the mean. And don't get me started on improper priors. Improper priors mean that you can only justify the sample mean by using mathematical techniques only a graduate student in analysis could understand. And yet ordinary people use sample averages all the time. You can justify OLS and the mean without reference to probability at all (see linear algebra and basic calc). Why not do so?

Frequentists do not "rely on prior information". Frequentists, like everyone else, have assumptions and build these into their models. Frequentists do not apply probability distributions to those parameters of the system that are of scientific interest. I put it this way to get around latent variable models and hiercharchical or empirical Bayes models.

$\endgroup$
  • $\begingroup$ +1, this is one of the best (& least partisan) descriptions I've seen. $\endgroup$ – gung - Reinstate Monica May 22 '13 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.