25

The debate about non-informative priors has been going on for ages, at least since the end of the 19th century with criticism by Bertrand and de Morgan about the lack of invariance of Laplace's uniform priors (the same criticism reported by Stéphane Laurent in the above comments). This lack of invariance sounded like a death stroke for the Bayesian approach ...


18

Your computation seems to be verifying that, when we have a particular prior distribution $p(\theta)$ the following two procedures Compute the posterior $p_{\theta \mid D}(\theta \mid D)$ Transform the aforementioned posterior into the other parametrization to obtain $p_{\psi \mid D}(\psi \mid D)$ and Transform the prior $p_\theta(\theta)$ into the other ...


15

Lets have $\phi = g(\theta)$, where $g$ is a monotone function of $\theta$ and let $h$ be the inverse of $g$, so that $\theta = h(\phi)$. We can obtain Jeffrey's prior distribution $p_{J}(\phi)$ in two ways: Start with the Binomial model (1) \begin{equation} \label{original} p(y | \theta) = \binom{n}{y} \theta^{y} (1-\theta)^{n-y} \end{equation} ...


11

As indicated in this paper by Yang and Berger (1999) that provides a list of Jeffreys priors, the Jeffreys prior associated with the Beta distribution is the determinant of a $2\times 2$ matrix that involves the polygamma function. Nothing close to a standard distribution.


8

When transforming a uniform distribution on $\log(\sigma)$ to a distribution on $\sigma$ you need to take into account the Jacobian of the transformation. This corresponds, as you correctly intuited, to $1/\sigma$. Writing this a little more clearly, let $X=\log(\sigma)$ and the transformation we're after is $T(X)=\sigma=e^{X}=Y$, which has inverse ...


7

You can define a proper or improper prior in the Stan language using the increment_log_prob() function, which will add its input to the accumulated log-posterior value that is used in the Metropolis step to decide whether to accept or reject a proposal for the parameters. In your example, the model block would need to include the new line ...


6

I think the discrepancy is explained by whether the authors consider the density over $\sigma$ or the density over $\sigma^2$. Supporting this interpretation, the exact thing that Kass and Wassermann write is $$ \pi(\mu, \sigma) = 1 / \sigma^2, $$ while Yang and Berger write $$ \pi(\mu, \sigma^2) = 1 / \sigma^4. $$


6

The short answer to your question is that it is impossible to simulate from $p(\sigma)=1/\sigma$ as this is not a probability density but a measure with infinite mass. Since you also mention Jeffreys and importance in the same sentence, it may however be that you are actually asking about simulating a posterior associated with $p(\sigma)=1/\sigma$ as the ...


5

Warning! It is impossible to compute a posterior probability when an improper prior is used on one or both models since the answer depends on an arbitrary constant $c$. As illustrated by the final value $$\pi(H_0|x) = \frac{1}{1 + c \sqrt{2\pi}e^{x^2/2}}$$


5

The Wikipedia page that you provided does not really use the term "variance-stabilizing transformation". The term "variance-stabilizing transformation" is generally used to indicate transformations that make the variance of the random variable a constant. Although in the Bernoulli case, this is what is happening with the transformation, that is not exactly ...


5

The uniform prior does not move the mode. The mode of the posterior is equal to the mle in this case. The comparison is made between the posterior expectation, or mean value, and the mle. From a pragmatic perspective, the argument is a bit silly, as the two values differ by an order of $\frac {2}{N_1+N_0 } $ anyway. So the only time they are different is ...


4

No, a Gamma(0,0) is not equivalent to the Jeffreys prior of the Poisson and Exponential rates (it is not even well defined). By Gammao(0,0) people usually mean a $Gamma(\epsilon,\epsilon)$ with $\epsilon\approx 0$. Its use became popular since the people from WINBUGS claimed that it "resembles" the shape of the Jeffreys prior for the variance parameters in ...


4

A partial answer to this is found in Gelman et al., Bayesian Data Analysis, 3rd ed. Jeffreys' principle can be extended to multiparameter models, but the results are more controversial. Simpler approaches based on assuming independent noninformative prior distributions for the components of the vector parameter $\theta$ can give different results than are ...


4

It is difficult to perceive where you get stuck: $$\begin{align*}p(b|x)&\propto \dfrac{b^a}{\Gamma (a)}x^{a-1}e^{-xb}\cdot\dfrac{\sqrt{a}}{b} \\ &\propto b^{a-1} e^{-xb} \\ &\propto \dfrac{x^a\,b^{a-1}}{\Gamma(a)}\,e^{-xb}\end{align*}$$ which shows the posterior is a Gamma $\mathcal{G}(a,x)$ distribution.


4

Here is a frivolous example that may have some intuitive value. In US Major League Baseball each team plays 162 games per season. Suppose a team is equally likely to win or lose each of its games. What proportion of the time will such a team have more wins than losses? (In order to have symmetry, if a team's wins and losses are tied at any point, we say it ...


3

If $X\sim \frac{1}{\theta}f(x/\theta)$, with $\theta$ a scale parameter, like the standard deviation, then $$Y=\log\{X\}\sim f\circ\exp\{y-\log\{\theta\}\}\,\exp\{y-\log\{\theta\}$$enjoys a location distribution with location $\xi=\log\{\theta\}$. Since Fisher's information on $\xi$ is constant, Jeffreys's prior on $\xi$ is uniform, $$\pi^J(\xi)=c$$ and $$\...


3

The Jeffreys prior is invariant under reparametrization. For that reason, many Bayesians consider it to be a “non-informative prior”. (Hartigan showed that there is a whole space of such priors $J^\alpha H^\beta$ for $\alpha + \beta=1$ where $J$ is Jeffreys' prior and $H$ is Hartigan's asymptotically locally invariant prior. — Invariant Prior Distributions)...


3

$\frac{1}{\sigma^3}$ is the Jeffreys prior. However in practice $\frac{1}{\sigma^2}$ is quite often used cause it leads to a relatively simple posterior, the "intuition" of this prior is that it corresponds with a flat prior on $\log(\sigma)$.


3

Taking your example and adjusting it slightly to $\pi(\mu,\sigma^2)\propto\frac{1}{\sigma^2}$ similar to Wikipedia's example: an argument that this prior is non-informative is that it is location-invariant and scale-invariant (uniform on the logarithmic scale), for example with properties that it leads to equal likelihoods for all possible values of the ...


3

The information brought by $n$ iid observations is $n$ times the information brought by one observation. They both lead to the same Jeffreys prior. As a side remark, note the typo in the quote where "the second inequality" should be "the second equality".


2

The existing answers already well answer the original question. As a physicist, I would just like to add to this discussion a dimensionality argument. If you consider $\mu$ and $\sigma^2$ to describe a distribution of a random variable in a real 1D space and measured in meters, they have the dimensions $[\mu] \sim m$ and $[\sigma^2] \sim m^2$. To have a ...


2

I will assume your model is a uniform distribution on the interval $(0, \theta)$. So let $X_1, \dotsc, X_n$ iid with that distribution, with $\theta>0$. Then the likelihood function can be written $$ L(\theta) = \theta^{-n} \cdot \mathbb{1}(\theta \ge T) $$ where $T=\max_{i=1}^n X_i$. The first idea is the Jeffrey' prior, and your statement of that ...


2

It comes as \begin{align*} \mathbb{E}[\| Y - X\hat{\beta_{\alpha}}\|^2]&= \mathbb{E}[\| Y - X\beta\|^2] \overbrace{+}^{\text{Pythagore's}} \mathbb{E}[\| X\beta - X\hat{\beta_{\alpha}}\|^2]\\ &= \underbrace{C}_\text{independent of $\alpha$} + \mathbb{E}[(\beta - \hat{\beta_{\alpha}})'X'X (\beta - \hat{\beta_{\alpha}})]\\ &= C + \mathbb{E}[(\beta - ...


2

If $p \sim \text{Beta}(a,b)$, then $T = p/(1-p)$ has a Pearson Type VI distribution, also known as a beta-prime, inverted beta, or beta distribution of the second kind, so I suppose you could call the distribution of $\log T$ a log beta-prime distribution. N. L. Johnson in "Systems of frequency curves generated by methods of translation", Biometrika, 36, in ...


2

What is optimal? There is no general and generic "optimality" result for the Jeffreys prior. It all depends on the purpose of the statistical analysis and of the loss function adopted to evaluate and compare procedures. Otherwise, $\pi(\theta,\sigma)\propto \dfrac{1}{\sigma}$ cannot be compared with $\pi(\theta,\sigma)\propto \dfrac{1}{\sigma}^2$. As I wrote ...


2

Hint 1: find the square root of the determinant of the Fisher information matrix, and you have your answer. The determinant of a diagonal matrix is the product of the diagonal entries. Hint 2: decide whether the variance or the standard deviation is your scale parameter, and stick with that. If you choose the variance, you're taking derivatives with respect ...


2

A simple example is when data are binomial. If you place a uniform prior on $p$ from 0 to 1, then the posterior distribution of $p$ is $Beta(x+1,n-x+1)$, so we can see that a flat prior on p is actually not objective in the sense that it is equivalent to assuming an additional success and an additional failure before collecting data, which pushes the ...


1

In general the posterior is the product of the prior and the likelihood. The subscript J is only to indicate that in the particular example the Jeffrey's prior is used. Consequently the $p_{J}(\theta|\ldots)$ and $p_{J}(\sigma^2|\ldots)$ are the full conditional for the parameters $\theta,\sigma^2$ which are obtained from the full posterior under the ...


1

Regarding your edit, that's not right. You also need the product rule: \begin{align*} \frac{d^2\log p(y | \phi)}{d\phi^2} &= \frac{d}{d\phi} \left( \frac{d \log p(y|\theta(\phi))}{d \theta} \frac{d\theta}{d\phi} \right) \tag{chain rule}\\ &= \left(\frac{d^2 \log p(y|\theta(\phi))}{d \theta d\phi}\right)\left( \frac{d\theta}{d\phi}\right) + \left(\...


Only top voted, non community-wiki answers of a minimum length are eligible