Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now
100

I will focus this answer on the specific question of what are the alternatives to $p$-values. There are 21 discussion papers published along with the ASA statement (as Supplemental Materials): by Naomi Altman, Douglas Altman, Daniel J. Benjamin, Yoav Benjamini, Jim Berger, Don Berry, John Carlin, George Cobb, Andrew Gelman, Steve Goodman, Sander Greenland, ...


66

Stats is not Math First, I steal @whuber's words from a comment in Stats is not maths? (applied in a different context, so I'm stealing words, not citing): If you were to replace "statistics" by "chemistry," "economics," "engineering," or any other field that employs mathematics (such as home economics), it appears none of your argument would change. ...


58

I actually mildly disagree with the premise. Everyone is a Bayesian, if they really do have a probability distribution handed to them as a prior. The trouble comes about when they don't, and I think there's still a pretty good-sized divide on that topic. Having said that, though, I do agree that more and more people are less inclined to fight holy wars ...


58

I'm going to take your questions in order: The question is, Who are the Bayesians today? Anybody who does Bayesian data analysis and self-identifies as "Bayesian". Just like a programmer is someone who programs and self-identifies as a "programmer". A slight difference is that for historical reasons Bayesian has ideological connotations, because of the ...


54

Here's five reasons why frequentists methods may be preferred: Faster. Given that Bayesian statistics often give nearly identical answers to frequentist answers (and when they don't, it's not 100% clear that Bayesian is always the way to go), the fact that frequentist statistics can be obtained often several orders of magnitude faster is a strong argument. ...


44

The main issue is that the first experiment (Sun gone nova) is not repeatable, which makes it highly unsuitable for frequentist methodology that interprets probability as estimate of how frequent an event is giving that we can repeat the experiment many times. In contrast, bayesian probability is interpreted as our degree of belief giving all available prior ...


44

Answer to question 1: This occurs because the $p$-value becomes arbitrarily small as the sample size increases in frequentist tests for difference (i.e. tests with a null hypothesis of no difference/some form of equality) when a true difference exactly equal to zero, as opposed to arbitraily close to zero, is not realistic (see Nick Stauner's comment to the ...


44

I will consider both Matloff's points: With large samples, significance tests pounce on tiny, unimportant departures from the null hypothesis. The logic here is that if somebody reports highly significant $p=0.0001$, then from this number alone we cannot say if the effect is large and important or irrelevantly tiny (as can happen with large $n$). I find ...


35

This is a difficult question to answer. The number of people who truly do both is still very limited. Hard core Bayesians despise the users of mainstream statistics for their use of $p$-values, a nonsensical, internally inconsistent statistic for Bayesians; and the mainstream statisticians just do not know Bayesian methods well enough to comment on them. In ...


35

They can look but not touch. After all, the residuals are the part of the data that don't carry any information about model parameters, and their prior expresses all uncertainty about those—they can't change their prior based on what they see in the data. For example, suppose you're fitting a Gaussian model, but notice far too much kurtosis in the ...


35

Some existing answers talk about statistical inference and some about interpretation of probability, and none clearly makes the distinction. The main purpose of this answer is to make this distinction. The word "frequentism" (and "frequentist") can refer to TWO DIFFERENT THINGS: One is the question about what is the definition or the interpretation of "...


32

The main difference between the Bayesian and frequentist schools of statistics arises due to a difference in interpretation of probability. A Bayesian probability is a statement about personal belief that an event will (or has) occurred. A frequentist probability is a statement about the proportion of similar events that occur in the limit as the number of ...


32

The mathematical basis for the Bayesian vs frequentist debate is very simple. In Bayesian statistics the unknown parameter is treated as a random variable; in frequentist statistics it is treated as a fixed element. Since a random variable is a much more complicated mathematical object than a simple element of the set, the mathematical difference is quite ...


29

I take great offense at the following two ideas: With large samples, significance tests pounce on tiny, unimportant departures from the null hypothesis. Almost no null hypotheses are true in the real world, so performing a significance test on them is absurd and bizarre. It is such a strawman argument about p-values. The very foundational problem ...


27

In the frequentist approach, it is asserted that the only sense in which probabilities have meaning is as the limiting value of the number of successes in a sequence of trials, i.e. as $$p = \lim_{n\to\infty} \frac{k}{n}$$ where $k$ is the number of successes and $n$ is the number of trials. In particular, it doesn't make any sense to associate a ...


27

As far as I can see the frequentist bit is reasonable this far: Let $H_0$ be the hypothesis that the sun has not exploded and $H_1$ be the hypothesis that it has. The p-value is thus the probability of observing the result (the machine saying "yes") under $H_0$. Assuming that the machine correctly detects the presence of absence of neutrinos, then if the ...


27

Here is my two cents. I think that at some point, many applied scientists stated the following "theorem": Theorem 1: $p\text{-value}<0.05\Leftrightarrow \text{my hypothesis is true}.$ and most of the bad practices come from here. The $p$-value and scientific induction I used to work with people using statistics without really understanding it and ...


26

In plain english, I would say that Bayesian and Frequentist reasoning are distinguished by two different ways of answering the question: What is probability? Most differences will essentially boil down to how each answers this question, for it basically defines the domain of valid applications of the theory. Now you can't really give either answer in ...


25

Why does this result seem "wrong?" A Bayesian would say that the result seems counter-intuitive because we have "prior" beliefs about when the sun will explode, and the evidence provided by this machine isn't enough to wash out those beliefs (mostly because of it's uncertainty due to the coin flipping). But a frequentist is able to make such an assessment, ...


25

Bayesians are people who define probabilities as a numerical representation of the plausibility of some proposition. Frequentists are people who define probabilities as representing long run frequencies. If you are only happy with one or other of these definitions then you are either a Bayesian or a frequentist. If you are happy with either, and use the ...


25

I don't like philosophy, but I do like math, and I want to work exclusively within the framework of Kolmogorov's axioms. How exactly would you apply Kolmogorov's axioms alone without any interpretation? How would you interpret probability? What would you say to someone who asked you "What does your estimate of probability $0.5$ mean?" Would you say that ...


25

Both Bayesian statistics and frequentist statistics are based on probability theory, but I'd say that the former relies more heavily on the theory from the start. On the other hand, surely the concept of a credible interval is more intuitive than that of a confidence interval, once the student has a good understanding of the concept of probability. So, ...


24

He's referring, rather clumsily, to the well known fact that frequentist analysis doesn't model the state of our knowledge about an unknown parameter with a probability distribution, so having calculated a (say 95%) confidence interval (say 1.2 to 3.4) for a population parameter (say the mean of a Gaussian distribution) from some data you can't then go ahead ...


24

The only reasons I continue to use $P$-values are More software is available for frequentist methods than Bayesian methods. Currently, some Bayesian analyses take a long time to run. Bayesian methods require more thinking and more time investment. I don't mind the thinking part but time is often short so we take shortcuts. The bootstrap is a highly ...


24

There is no difference in the definition - in both cases, the likelihood function is any function of the parameter that is proportional to the sampling density. Strictly speaking we do not require that the likelihood be equal to the sampling density; it needs only be proportional, which allows removal of multiplicative parts that do not depend on the ...


23

A few concrete advantages of frequentist statistics: There are often closed-form solutions to frequentist problems whereas you would need a conjugate prior to have a closed form solution in the Bayesian analogue. This is useful for a number of reasons - one of which is computation time. A reason that'll, hopefully, eventually go away: laymen are taught ...


22

Bayesian and frequentist ask different questions. Bayesian asks what parameter values are credible, given the observed data. Frequentist asks about the probability of imaginary simulated data if some hypothetical parameter values were true. Frequentist decisions are motivated by controlling errors, Bayesian decisions are motivated by uncertainty in model ...


21

In reality, I think much of the philosophy surrounding the issue is just grandstanding. That's not to dismiss the debate, but it is a word of caution. Sometimes, practical matters take priority - I'll give an example below. Also, you could just as easily argue that there are more than two approaches: Neyman-Pearson ('frequentist') Likelihood-based ...


21

Typically you'd carry out inference conditional on the actual sample size $n$, because it's ancillary to the parameters of interest; i.e. it contains no information about their true values, only affecting the precision with which you can measure them. Cox (1958), "Some Problems Connected with Statistical Inference", Ann. Math. Statist. 29, 2 is usually cited ...


Only top voted, non community-wiki answers of a minimum length are eligible