Hot answers tagged

233

Let me first explain what a conjugate prior is. I will then explain the Bayesian analyses using your specific example. Bayesian statistics involve the following steps: Define the prior distribution that incorporates your subjective beliefs about a parameter (in your example the parameter of interest is the proportion of left-handers). The prior can be "...


93

[Warning: as a card-carrying member of the Objective Bayes Section of ISBA, my views are not representative of all Bayesian statisticians!, quite the opposite...] In summary, there is no such thing as a prior with "truly no information". Indeed, the "uninformative" prior is sadly a misnomer. Any prior distribution contains some specification that is akin ...


46

The answer depends on whether you are assuming the symmetric or asymmetric dirichlet distribution (or, more technically, whether the base measure is uniform). Unless something else is specified, most implementations of LDA assume the distribution is symmetric. For the symmetric distribution, a high alpha-value means that each document is likely to contain a ...


41

Suppose that you and a friend are analyzing the same set of data using a normal model. You adopt the usual parameterization of the normal model using the mean and the variance as parameters, but your friend prefers to parameterize the normal model with the coefficient of variation and the precision as parameters (which is perfectly "legal"). If both of you ...


35

It is not that easy. Information in your data overwhelms prior information not only your sample size is large, but when your data provides enough information to overwhelm the prior information. Uninformative priors get easily persuaded by data, while strongly informative ones may be more resistant. In extreme case, with ill-defined priors, your data may not ...


34

A half-Cauchy is one of the symmetric halves of the Cauchy distribution (if unspecified, it is the right half that's intended): Since the area of the right half of a Cauchy is $\frac12$ the density must then be doubled. Hence the 2 in your pdf (though it's missing a $\frac{1}{\pi}$ as whuber noted in comments). The half-Cauchy has many properties; some are ...


31

First of all, there is no such a thing as uninformative prior. Below you can see posterior distributions resulting from five different "uninformative" priors (described below the plot) given different data. As you can clearly see, the choice of "uninformative" priors affected the posterior distribution, especially in cases where the data itself did not ...


30

Let me complete Zen's answer. I don't very like the notion of "representing ignorance". The important thing is not the Jeffreys prior but the Jeffreys posterior. This posterior aims to reflect as best as possible the information about the parameters brought by the data. The invariance property is naturally required for the two following points. Consider for ...


29

The Jeffreys prior coincides with the Bernardo reference prior for one-dimensional parameter space (and "regular" models). Roughly speaking, this is the prior for which the Kullback-Leibler divergence between the prior and the posterior is maximal. This quantity represents the amount of information brought by the data. This is why the prior is considered to ...


29

For simplicity let's just consider a single observation of a variable $Y$ such that $$Y|\mu, \sigma^2 \sim N(\mu, \sigma^2),$$ $\mu \sim \mbox{Laplace}(\lambda)$ and the improper prior $f(\sigma) \propto \mathbb{1}_{\sigma>0}$. Then the joint density of $Y, \mu, \sigma^2$ is proportional to $$ f(Y, \mu, \sigma^2 | \lambda) \propto \frac{1}{\sigma}\...


27

Yes. The posterior distribution for a parameter $\theta$, given a data set ${\bf X}$ can be written as $$ p(\theta | {\bf X}) \propto \underbrace{p({\bf X} | \theta)}_{{\rm likelihood}} \cdot \underbrace{p(\theta)}_{{\rm prior}} $$ or, as is more commonly displayed on the log scale, $$ \log( p(\theta | {\bf X}) ) = c + L(\theta;{\bf X}) + \log(p(\theta)...


24

Two reasons one may go with a Bayesian approach even if you're using highly non-informative priors: Convergence problems. There are some distributions (binomial, negative binomial and generalized gamma are the ones I'm most familiar with) that have convergence issues a non-trivial amount of the time. You can use a "Bayesian" framework - and particular ...


24

This is obvious by inspection of the quantity the LASSO is optimizing. Take the prior for $\beta_i$ to be independent Laplace with mean zero and some scale $\tau$. So $p(\beta|\tau) \propto e^{-\frac{1}{2\tau} \sum_i|\beta_i|}$. The model for the data is the usual regression assumption $y \stackrel{\text{iid}}{\sim}N(X\beta,\sigma^2)$. $f(\mathbf{y}|\...


21

Notice that: \begin{equation} \frac{\alpha\cdot\beta}{(\alpha+\beta)^2}=(\frac{\alpha}{\alpha+\beta})\cdot(1-\frac{\alpha}{\alpha+\beta}) \end{equation} This means the variance can therefore be expressed in terms of the mean as \begin{equation} \sigma^2=\frac{\mu\cdot(1-\mu)}{\alpha+\beta+1} \\ \end{equation} If you want a mean of $.27$ and a standard ...


21

The relation of Laplace distribution prior with median (or L1 norm) was found by Laplace himself, who found that using such prior you estimate median rather than mean as with Normal distribution (see Stingler, 1986 or Wikipedia). This means that regression with Laplace errors distribution estimates the median (like e.g. quantile regression), while Normal ...


21

Many frequentist confidence intervals (CIs) are based on the likelihood function. If the prior distribution is truly non-informative, then the a Bayesian posterior has essentially the same information as the likelihood function. Consequently, in practice, a Bayesian probability interval (or credible interval) may be very similar numerically to a frequentist ...


20

Here is an attempt to illustrate the last paragraph in Macro's excellent (+1) answer. It shows two priors for the parameter $p$ in the ${\rm Binomial}(n,p)$ distribution. For a few different $n$, the posterior distributions are shown when $x=n/2$ has been observed. As $n$ grows, both posteriors become more and more concentrated around $1/2$. For $n=2$ the ...


19

You should definitely invest some time in learning the bases of Bayesian statistics and MCMC methods from textbooks or on-line courses. The title and the wording of the question seem to indicate some confusion between the prior modelling [which pertains to the statistical model] and the MCMC implementation [which pertains to the computational resolution]. ...


18

From (an earlier version of) the Stan reference manual: Not specifying a prior is equivalent to specifying a uniform prior. A uniform prior is only proper if the parameter is bounded[...] Improper priors are also allowed in Stan programs; they arise from unconstrained parameters without sampling statements. In some cases, an improper prior may ...


18

Generally, informative priors are typically viewed as your information about parameters (or hypotheses) before seeing the data. So any data-based prior is violating the likelihood principle since evidence from the sample is coming through the likelihood function and the prior.


17

In frequentist decision theory, there exist complete class results that characterise admissible procedures as Bayes procedures or as limits of Bayes procedures. For instance, Stein necessary and sufficient condition (Stein. 1955; Farrell, 1968b) states that, under the following assumptions the sampling density $f(x|\theta)$ is continuous in $\theta$ and ...


16

An appealing property of formal noninformative priors is the "frequentist-matching property" : it means that a posterior 95%-credibility interval is also (at least, approximately) a 95%-confidence interval in the frequentist sense. This property holds for Bernardo's reference prior although the fundations of these noninformative priors are not oriented ...


16

Although the results are going to be very similar, their interpretations differ. Confidence intervals imply the notion of repeating an experiment many times and being able to capture the true parameter 95% of times. But you cannot say you have a 95% chance of capturing it. Credible intervals (Bayesian), on the other hand, allow you to say that there is a ...


16

Improper priors are $\sigma$-finite non-negative measures $\text{d}\pi$ on the parameter space $\Theta$ such that$$\int_\Theta \text{d}\pi(\theta) = +\infty$$As such they generalise the notion of a prior distribution, which is a probability distribution on the parameter space $\Theta$ such that$$\int_\Theta \text{d}\pi(\theta) =1$$They are useful in several ...


16

First of all, there is no such a thing as "uninformative priors" (we rather talk about "weakly informative" priors), each prior brings some kind of assumptions into your model. On another hand, the more information does your data provides, the less influential your prior becomes. But taking this aside, with "uninformative" priors the point estimates from ...


15

To give a more narrow response than the excellent ones that have already been posted, and focus on the advantage in interpretation - the Bayesian interpretation of a, e.g., "95% credible interval" is that the probability that the true parameter value lies within the interval equals 95%. One of the two common frequentist interpretations of a, e.g., "95% ...


15

Simply put, a flat/non-informative prior is used when one has little/no knowledge about the data and hence it has the least effect on outcomes of your analysis (i.e. posterior inference). Conjugate distributions are those whose prior and posterior distributions are the same, and the prior is called the conjugate prior. It is favoured for its algebraic ...


15

As stated in comment, the prior distribution represents prior beliefs about the distribution of the parameters. When prior beliefs are actually available, you can: convert them in terms of moments (e.g. mean and variance) to fit a common distribution to these moments (e.g. Gaussian if your parameter lies to the real line, Gamma if it lies to $R^+$). use ...


Only top voted, non community-wiki answers of a minimum length are eligible