234

Let me first explain what a conjugate prior is. I will then explain the Bayesian analyses using your specific example. Bayesian statistics involve the following steps: Define the prior distribution that incorporates your subjective beliefs about a parameter (in your example the parameter of interest is the proportion of left-handers). The prior can be "...


107

Well by distributing the $P(H)$ term, we obtain $$ P(H|X) = \frac{P(X|H)P(H)}{P(X)} P(C) + P(H) [1 - P(C)], $$ which we can interpret as the Law of Total Probability applied to the event $C =$ "you are using Bayesian statistics correctly." So if you are using Bayesian statistics correctly, then you recover Bayes' law (the left fraction above) and if you aren'...


100

I will focus this answer on the specific question of what are the alternatives to $p$-values. There are 21 discussion papers published along with the ASA statement (as Supplemental Materials): by Naomi Altman, Douglas Altman, Daniel J. Benjamin, Yoav Benjamini, Jim Berger, Don Berry, John Carlin, George Cobb, Andrew Gelman, Steve Goodman, Sander Greenland, ...


93

[Warning: as a card-carrying member of the Objective Bayes Section of ISBA, my views are not representative of all Bayesian statisticians!, quite the opposite...] In summary, there is no such thing as a prior with "truly no information". Indeed, the "uninformative" prior is sadly a misnomer. Any prior distribution contains some specification that is akin ...


92

Simple linear regression model $$ y_i = \alpha + \beta x_i + \varepsilon $$ can be written in terms of probabilistic model behind it $$ \mu_i = \alpha + \beta x_i \\ y_i \sim \mathcal{N}(\mu_i, \sigma) $$ i.e. dependent variable $Y$ follows normal distribution parametrized by mean $\mu_i$, that is a linear function of $X$ parametrized by $\alpha,\beta$, ...


79

John Kruschke released a book in mid 2011 called Doing Bayesian Data Analysis: A Tutorial with R and BUGS. (A second edition was released in Nov 2014: Doing Bayesian Data Analysis, Second Edition: A Tutorial with R, JAGS, and Stan.) It is truly introductory. If you want to walk from frequentist stats into Bayes though, especially with multilevel modelling, I ...


68

It is a very broad question and my answer here only begins to scratch the surface a bit. I will use the Bayes's rule to explain the concepts. Let’s assume that a set of probability distribution parameters, $\theta$, best explains the dataset $D$. We may wish to estimate the parameters $\theta$ with the help of the Bayes’ Rule: $$p(\theta|D)=\frac{p(D|\...


66

Stats is not Math First, I steal @whuber's words from a comment in Stats is not maths? (applied in a different context, so I'm stealing words, not citing): If you were to replace "statistics" by "chemistry," "economics," "engineering," or any other field that employs mathematics (such as home economics), it appears none of your argument would change. ...


65

The Dirichlet distribution is a multivariate probability distribution that describes $k\ge2$ variables $X_1,\dots,X_k$, such that each $x_i \in (0,1)$ and $\sum_{i=1}^N x_i = 1$, that is parametrized by a vector of positive-valued parameters $\boldsymbol{\alpha} = (\alpha_1,\dots,\alpha_k)$. The parameters do not have to be integers, they only need to be ...


58

I actually mildly disagree with the premise. Everyone is a Bayesian, if they really do have a probability distribution handed to them as a prior. The trouble comes about when they don't, and I think there's still a pretty good-sized divide on that topic. Having said that, though, I do agree that more and more people are less inclined to fight holy wars ...


58

The Dirichlet distribution is a conjugate prior for the multinomial distribution. This means that if the prior distribution of the multinomial parameters is Dirichlet then the posterior distribution is also a Dirichlet distribution (with parameters different from those of the prior). The benefit of this is that (a) the posterior distribution is easy to ...


58

I'm going to take your questions in order: The question is, Who are the Bayesians today? Anybody who does Bayesian data analysis and self-identifies as "Bayesian". Just like a programmer is someone who programs and self-identifies as a "programmer". A slight difference is that for historical reasons Bayesian has ideological connotations, because of the ...


54

Here's five reasons why frequentists methods may be preferred: Faster. Given that Bayesian statistics often give nearly identical answers to frequentist answers (and when they don't, it's not 100% clear that Bayesian is always the way to go), the fact that frequentist statistics can be obtained often several orders of magnitude faster is a strong argument. ...


45

This is a good question, that seems to pop up a lot: link 1, link 2. The paper Bayesian Estimation Superseeds the T-Test that Cam.Davidson.Pilon pointed out is an excellent resource on this subject. It is also very recent, published in 2012, which I think in part is due to the current interest in the area. I will try to summarize a mathematical explanation ...


44

The main issue is that the first experiment (Sun gone nova) is not repeatable, which makes it highly unsuitable for frequentist methodology that interprets probability as estimate of how frequent an event is giving that we can repeat the experiment many times. In contrast, bayesian probability is interpreted as our degree of belief giving all available prior ...


44

I will consider both Matloff's points: With large samples, significance tests pounce on tiny, unimportant departures from the null hypothesis. The logic here is that if somebody reports highly significant $p=0.0001$, then from this number alone we cannot say if the effect is large and important or irrelevantly tiny (as can happen with large $n$). I find ...


43

The general term Naive Bayes refers the the strong independence assumptions in the model, rather than the particular distribution of each feature. A Naive Bayes model assumes that each of the features it uses are conditionally independent of one another given some class. More formally, if I want to calculate the probability of observing features $f_1$ ...


43

This has been discussed in my paper (published only on the internet) "On an Example of Larry Wasserman" [1] and in a blog exchange between me, Wasserman, Robins, and some other commenters on Wasserman's blog: [2] The short answer is that Wasserman (and Robins) generate paradoxes by suggesting that priors in high dimensional spaces "must" have ...


43

I do not see much appeal in this example, esp. as a potential criticism of Bayesians and likelihood-wallahs.... The constant $c$ is known, being equal to $$ 1\big/ \int_\mathcal{X} g(x) \text{d}x $$ If $c$ is the only "unknown" in the picture, given a sample $x_1,\ldots,x_n$, then there is no statistical issue about the problem and I do not agree that there ...


41

Suppose that you and a friend are analyzing the same set of data using a normal model. You adopt the usual parameterization of the normal model using the mean and the variance as parameters, but your friend prefers to parameterize the normal model with the coefficient of variation and the precision as parameters (which is perfectly "legal"). If both of you ...


41

IMHO "yes"! Here is one of my favorite quotes by Greenland (2006: 767): It is often said (incorrectly) that ‘parameters are treated as fixed by the frequentist but as random by the Bayesian’. For frequentists and Bayesians alike, the value of a parameter may have been fixed from the start or may have been generated from a physically random ...


41

I believe that you and your colleague are correct. Statistics.com has the correct line of thinking, but makes a simple mistake. Out of the 90 "OK" claims, we expect 20% of them to be incorrectly classified as fraud, not 80%. 20% of 90 is 18, leading to 9 correctly identified claims and 18 incorrect claims, with a ratio of 1/3, exactly what Bayes' rule yields....


40

I'm going to give you an answer. Four drawbacks actually. Note that none of these are actually objections that should drive one all the way to frequentist analysis, but there are cons to going with a Bayesian framework: Choice of prior. This is the usual carping for a reason, though in my case it's not the usual "priors are subjective!" but that coming up ...


40

I agree that the example is weird. I meant it to be more of a puzzle really. (The example is actually due to Ed George.) It does raise the question of what it means for something to be "known". Christian says that $c$ is known. But, at least from the purely subjective probability point of view, you don't know it just because it can in principle be known. (...


40

Because, assuming normal errors is effectively the same as assuming that large errors do not occur! The normal distribution has so light tails, that errors outside $\pm 3$ standard deviations have very low probability, errors outside of $\pm 6$ standard deviations are effectively impossible. In practice, that assumption is seldom true. When analyzing ...


37

Even though variational autoencoders (VAEs) are easy to implement and train, explaining them is not simple at all, because they blend concepts from Deep Learning and Variational Bayes, and the Deep Learning and Probabilistic Modeling communities use different terms for the same concepts. Thus when explaining VAEs you risk either concentrating on the ...


36

It is not that easy. Information in your data overwhelms prior information not only your sample size is large, but when your data provides enough information to overwhelm the prior information. Uninformative priors get easily persuaded by data, while strongly informative ones may be more resistant. In extreme case, with ill-defined priors, your data may not ...


35

This is a difficult question to answer. The number of people who truly do both is still very limited. Hard core Bayesians despise the users of mainstream statistics for their use of $p$-values, a nonsensical, internally inconsistent statistic for Bayesians; and the mainstream statisticians just do not know Bayesian methods well enough to comment on them. In ...


35

They can look but not touch. After all, the residuals are the part of the data that don't carry any information about model parameters, and their prior expresses all uncertainty about those—they can't change their prior based on what they see in the data. For example, suppose you're fitting a Gaussian model, but notice far too much kurtosis in the ...


35

Posterior predictive checks are, in simple words, "simulating replicated data under the fitted model and then comparing these to the observed data" (Gelman and Hill, 2007, p. 158). So, you use posterior predictive to "look for systematic discrepancies between real and simulated data" (Gelman et al. 2004, p. 169). The argument about "using the data twice" is ...


Only top voted, non community-wiki answers of a minimum length are eligible