35
$\begingroup$

I am looking at some lecture slides on a data science course which can be found here:

https://github.com/cs109/2015/blob/master/Lectures/01-Introduction.pdf

I, unfortunately, cannot see the video for this lecture and at one point on the slide, the presenter has the following text:

Some Key Principles

Think like a Bayesian, check like a Frequentist (reconciliation)

Does anyone know what that actually means? I have a feeling there is a good insight about these two schools of thought to be gathered from this.

Improve this question
$\endgroup$
  • 2
    $\begingroup$ Think it may be to do with model checking: see Why is a Bayesian not allowed to look at the residuals?. $\endgroup$ – Scortchi - Reinstate Monica♦ Aug 16 '16 at 14:58
  • $\begingroup$ @Scortchi From what I gather, does this not have to do with separation of training, validation and test datasets in a way or maybe a Bayesian is not allowed to adjust the priors even during the training phase of the model (to use a ML term here). However, I am still confused as to what it means by check like a frequentist... $\endgroup$ – Luca Aug 16 '16 at 15:04
  • 1
    $\begingroup$ A "proper" Bayesian never adjusts their priors, but only updates them according to new information using Bayes' Theorem. But I'm only guessing what this "key principle" might be on about. $\endgroup$ – Scortchi - Reinstate Monica♦ Aug 16 '16 at 15:13
  • 4
    $\begingroup$ I was not able to load the link. My guess is that they mean even if you use Bayesian methods, you should care about Frequentist operating characteristics: if you're generating 95% credible intervals that are extremely tight, but in practice cover the true parameter of interest 20% of the time, should you be concerned? An overly rigid Bayesian might say "no" (but very few Bayesians of such rigidity actually exist). $\endgroup$ – Cliff AB Aug 16 '16 at 15:24
  • 3
    $\begingroup$ Looking forwards into the future slides, they are endorsing Empirical Bayes. This can be seen on the following set of slides $\endgroup$ – Cliff AB Aug 16 '16 at 22:39
32
$\begingroup$

The main difference between the Bayesian and frequentist schools of statistics arises due to a difference in interpretation of probability. A Bayesian probability is a statement about personal belief that an event will (or has) occurred. A frequentist probability is a statement about the proportion of similar events that occur in the limit as the number of those events increases.

For me, to "think like a Bayesian" means to update your personal belief as new information arises and to "check [or worry] like a frequentist" means to be concerned with performance of statistical procedures aggregated across the times those procedures are used, e.g. what is the coverage of credible intervals, what is the Type I/II error rates, etc.

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ Thank you for your answer. Concise and effective even for layman like me! $\endgroup$ – Luca Aug 16 '16 at 16:05
  • 2
    $\begingroup$ Is it not possible to check or worry like a Bayesian by investigating the influence of priors or using a non-informative one? Is this only applicable to sequential analyses? There's been a lot of work about where Bayesian and Frequentist statistics intersect with sequential analyses, "belief updating" is not essential, and seqeuential statistics can be made rigorous in the frequentist setting. $\endgroup$ – AdamO Aug 16 '16 at 16:33
  • 1
    $\begingroup$ Yes it is possible to worry like a Bayesian, e.g. investigating the influence of your prior. No, my answer is not only applicable to sequential analyses, i.e. the new information could arise all at once. $\endgroup$ – jaradniemi Aug 17 '16 at 11:13
15
$\begingroup$

Bayesian statistics summarize beliefs whereas frequentist statistics summarize evidence. The Bayesians view probability as a degree of belief. This inclusive and generative type of reasoning is useful for formulating hypotheses. For instance, Bayesians may be able to arbitrarily assign some probability to the notion that the moon is made of green cheese, regardless of whether astronauts have actually been able to travel there to verify this. This hypothesis is perhaps supported by the idea that, from afar, the moon looks like green cheese. Frequentists cannot singularly conceive of a hypothesis that is more than a strawman, nor can they say evidence favors one hypothesis over another. Even maximum likelihood only generates a statistic which is "most consistent with what was observed". Formally, Bayesian statistics allows us to think outside the box and propose defensible ideas from data. But this is strictly hypothesis generating in nature.

Frequentist statistics are best applied to confirm hypotheses. When an experiment is conducted well, frequentist statistics provide an "independent observer" or "empirical" context to the findings by eschewing priors. This is consistent with the Karl Popper philosophy of science. The point of evidence is not to promulgate a certain idea. Plenty of evidence is consistent with incorrect hypotheses. Evidence can merely falsify beliefs.

The influence of priors is generally regarded as a bias in statistical reasoning. As you know, we can make up any great number of reasons for why things happen. Psychologically, many people believe that our observer bias is the result of priors in our brain that keep us from truly weighting what we see. "Hope clouds observation" as the Reverend Mother said in Dune. Popper made this idea rigorous.

This had great historical importance in some of the greatest scientific experiments of our time. For instance, John Snow meticulously collected evidence for the Cholera epidemic and concluded astutely that Cholera is not caused by moral deprivation, and pointed out that the evidence was highly consistent with sewage contamination: note he did not conclude this, Snow's findings predated the discovery of bacteria, and there was no mechanistic or etiologic understanding. A similar discourse is found in Origin of Species. We didn't actually know whether the moon was made of green cheese until astronauts actually landed on the surface and collected samples. At that point, Bayesian posteriors have assigned very, very low probability to any other possibility, and Frequentists at best can say that the samples are highly inconsistent with anything except moon dust.

In summary, Bayesian statistics are amenable to hypothesis generating and frequentist statistics are amenable to hypothesis confirmation. Ensuring that data are collected independently in these endeavors is one of the greatest challenges modern statisticians face.

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ Thanks for the answer. What did you mean when you say Plenty of evidence is consistent with incorrect hypotheses. $\endgroup$ – Luca Aug 16 '16 at 16:44
  • 2
    $\begingroup$ @Luca A common statistical example might be found in confounding. For instance, I might say, "Smoking gives adolescents better lung function". I could go further to rationalize this by saying that smoking is a stimulant which encourages better physical activity, healthier appetite, and encourages healthy socialization. If I collected data, they would indeed show that adolescents who smoke have better lung function. The associative conclusion is correct, but the causal one is false. The relation is confounded by age, since older children are more likely to smoke. $\endgroup$ – AdamO Aug 16 '16 at 17:38
  • $\begingroup$ Thank you! I have learnt a lot from this very well-written answer. $\endgroup$ – Luca Aug 17 '16 at 7:23
5
$\begingroup$

Per Cliff AB's comment to the OP, it sounds like they are heading towards an Empirical Bayesian philosophy. There are three main Bayesian schools of thought, and Empirical Bayes estimates priors from data, often with frequentist methods. That doesn't conform exactly to the quote (which implies Bayes up front, frequentist-like concerns afterwards), but we shouldn't overlook Cliff AB's excellent comment.

Also, there was, and may still be, a school of Bayesian thought that you don't have to check anything after a Bayesian procedure. More modern thought would use posterior predictive checks, and perhaps that kind of double-check-your-answers approach is what the quote is referring to.

Also, frequentist philosophy is concerned with procedures rather than inferences from data. So perhaps that is also a clue to the quote's meaning.

Improve this answer
$\endgroup$
  • $\begingroup$ I think you referred to my first comment, and my second comment was that after closer inspection, you are correct that they are very specifically referring to Empirical Bayes. I was actually disappointed that the quote was simply just an endorsement of Empirical Bayes rather than a more general call to consider the advantages of both schools of thought. Oh well. $\endgroup$ – Cliff AB Aug 20 '16 at 18:53
2
$\begingroup$

In the context of this data science class, my interpretation of "check like a frequentist" is that you evaluate the performance of your prediction function or decision function on held-out validation data. The advice to "think like a Bayesian" expresses the opinion that a prediction function derived from a Bayesian approach will generally give good results.

Improve this answer
$\endgroup$
  • $\begingroup$ (playing Devil's advocate:) Why should Bayesian approach give "good results" and frequentist not? $\endgroup$ – Tim♦ Aug 16 '16 at 21:53
  • $\begingroup$ Bayesian methods are prescriptive about the approach. Frequentist statistics can be seen as part of decision theory, and it gives a framework to evaluate any decision function (whether based on Bayesian or some frequentist principle). Certain methods, such as maximum likelihood methods, are often used in a frequentist context because they have good frequentist properties (e.g. asymptotically they do they right thing, and they get there faster than most other methods). A Bayesian method could certainly be used by a frequentist, but they would have different reasons for using it. $\endgroup$ – DavidR Aug 16 '16 at 22:04
  • $\begingroup$ Bayesian methods have also lots in common with decision theory. I also do not think that Bayesian methods can be used in frequentist context (how would you imagine using priors in frequentist context?) - it is rather the other way around: many frequentist methods have Bayesian interpretations. I do not think there is point in discussing this, what I'm saying that your statements a little bit oversimplify things. $\endgroup$ – Tim♦ Aug 16 '16 at 22:11
  • $\begingroup$ One can prove plenty of nice frequentist properties about Bayesian approaches, so in that sense, doing something Bayesian is pretty safe, as long as you have enough data. $\endgroup$ – DavidR Aug 16 '16 at 22:11
  • 1
    $\begingroup$ Suppose I want to estimate the probability p of heads in a coin flip. As a Bayesian, I'd start with a prior on the probability p, I'd observe some data, and then I'd get a posterior on p. We need to come up with a point estimate of p, and I choose to use the mean of my posterior distribution as my point estimate. All told, this describes a method to go from data to a point estimate. This method can be evaluated in a frequentist way: e.g. is it biased? consistent? asymptotically efficient? The fact that a prior was involved should not, per se, concern the frequentist. $\endgroup$ – DavidR Aug 16 '16 at 22:37
1
$\begingroup$

It sounds like "think like a Bayesian, check like a frequentist" refers to one's approach in statistical design and analysis. As I understand it, Bayesian thinking involves some belief about prior situations (experimentally or statistically), let's say for example that the mean reading scores for 4th-graders is 80 words per minute, and that some intervention might increase this to 90 words per minute. These are beliefs based on prior studies and hypotheses. Frequentist thinking extrapolates the findings (of the intervention) to obtain confidence intervals or other statistics that are based on the theoretical and practical frequency or probability of these results happening again (i.e., how "frequently"). For example the post-intervention reading score might be 91 words per minute with a 95% confidence interval of 85 to 97 words per minute and an associated p-value (probability value) of this being different from the pre-intervention score. So 95% of the time, the new reading scores would be between 85 and 97 words per minute after the intervention. Therefore "think like a Bayesian"---i.e., theorize, hypothesize, look at previous evidence, and "check like a frequentist"---i.e., how frequently would these experimental results occur, and how likely are they to be due to chance rather than the intervention.

Improve this answer
$\endgroup$
  • 2
    $\begingroup$ Your last sentence - the "check like a frequentist" part - has really nothing to do with frequentist setting: Bayesian estimate would also tell you "how frequently" we expect something to occur, or "how likely" it is... $\endgroup$ – Tim♦ Aug 16 '16 at 21:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.