# How to choose a importance density for Jeffreys prior?

I want to draw Bayesian inference via importance sampling and I do not come up with a good idea of an importance density for $$p(\sigma)\sim\frac{1}{\sigma}.$$ Is there a way to sample from this distribution directly? I am not sure whether $$\frac{1}{z}, z\sim\mathcal{U}[0,1]$$ will give me the desired property.

Update: I implemented the proposed importance function and computed the weights $$w_t=p(\sigma_t|\mathbf{y})\Big/\left\{\frac{1}{2a}\mathbb{I}_{[0,a]}(\sigma_t)+\frac{a}{2}\mathbb{I}_{[a,\infty)}(\sigma_t)\sigma_t^{-2}\right\}$$ for a sample of $m=1.000.000$ draws. Hereby I computed the log likelihoods $\omega_{log}$, substracted the maximum $\omega_{log}$ and transformed $\omega=\exp(\omega_{log}-\omega_\max)$. What can I learn from the fact, that roughly $0.98 %$ percent of the weights are $0$? I suppose that in this case the choice of the importance sample is not sensitive, as the sample size does not really matter but only a couple of observations drive the whole result?

• Well, I want to implement Importance sampling for a model that uses the Jeffreys prior. Therefore I am searching for a way to come up with a suitable importance density $q(\sigma)$ in order to compute posterior integrals $\int h(\sigma)p(\sigma|Y)d\sigma=\int h(\sigma)\frac{p(\sigma|Y)}{q(\sigma)}q(\sigma)d\sigma$. Commented Nov 23, 2015 at 10:57

The short answer to your question is that it is impossible to simulate from $p(\sigma)=1/\sigma$ as this is not a probability density but a measure with infinite mass.

Since you also mention Jeffreys and importance in the same sentence, it may however be that you are actually asking about simulating a posterior associated with $p(\sigma)=1/\sigma$ as the Jeffreys prior. In this case, a fat tailed importance sampler would do. For instance, if the posterior accumulates mass around $a$, e.g., if $a$ is roughly the mode. you could use $$\frac{1}{2}\mathcal{U}[0,a]+\frac{1}{2}1\big/\mathcal{U}[0,1/a]$$ as an importance function. This means

1. Simulate a sample $\sigma_1,\ldots,\sigma_M$ from this distribution;
2. Weight the above simulations by $$w_t=p(\sigma_t|\mathbf{y})\Big/\left\{\frac{1}{2a}\mathbb{I}_{[0,a]}(\sigma_t)+\frac{a}{2}\mathbb{I}_{[a,\infty)}(\sigma_t)\sigma_t^{-2}\right\}$$

since the density of an inverse uniform is $$f(z)=a\mathbb{I}_{(0,1/a)}(1/z)\frac{\text{d}u}{\text{d}z}=a\mathbb{I}_{z>a}\frac{1}{z^2}$$

Remember, always check that the resulting importance weights have finite variance!

• I just realized that I do not know what to use as $q(\sigma)$ under this importance density...can you give a comment on that? Commented Nov 23, 2015 at 12:37
• Is there a theoretical framework recommending this idea? I get the point with the fat tails, however, I would never come up with this distribution. Is it somewhat standard in importance sampling literature? Commented Nov 24, 2015 at 10:00
• By the way, is the weighting function really correct? Why does the second term include a \$\sigma^{-2} and the first one not? Commented Nov 24, 2015 at 10:08
• @muffin1974: No to all questions:1. this is a default proposal if you cannot come up with anything else; 2. it has no particular theoretical property that I know of; 3. it is not standard. I made this proposal to illustrate importance sampling not to suggest a universal solution. Commented Nov 24, 2015 at 10:10

Is it really impossible to sample from $$p(\sigma) = 1/\sigma$$?

Consider: $$p(\sigma) = 1/ \sigma$$ (the Jeffreys prior on scale parameter), and $$(y \mid \sigma) \sim N(0, \sigma^2)$$, then the posterior can be derived with usual Bayesian analysis: \begin{align*} p(\sigma \mid y) &\stackrel{\sigma}{\propto} p(\sigma, y) = p(y \mid \sigma) p(\sigma)\\ &= \frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{y^2}{2\sigma^2}\right) \cdot \frac{1}{\sigma}\\ \end{align*} To match the last expression with that of an inverse-Gamma distribution, reparameterize in terms of $$\sigma^2$$: $$p(\sigma^2) = \left.p(\sigma) \middle/ \left|\text{det}\left(\frac{d(\sigma^2)}{d\sigma}\right)\right| \right. = p(\sigma) / (2\sigma)$$. Thus \begin{align*} p(\sigma^2 \mid y) & = p(\sigma \mid y) / (2\sigma) \\ &\stackrel{\sigma}{\propto}\frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{y^2}{2\sigma^2}\right) \cdot \frac{1}{\sigma} / (2\sigma) \\ &\stackrel{\sigma}{\propto} \frac{1}{(\sigma^2)^{3/2}}\exp\left(-\frac{y^2}{2\sigma^2}\right)\\ (\sigma^2 \mid y) &\sim \text{IG}\left(\alpha=\frac{1}{2}, \beta = \frac{y^2}{2}\right) \end{align*}

Now that we've got both $$(\sigma^2 \mid y)$$ (inverse Gamma) and $$(y \mid \sigma^2)$$ (Gaussian), a Gibbs sampler is can be done from the initial value, of say, $$\sigma_\text{init} = 1$$, and alternate between $$y$$ and $$\sigma$$.

Or, is this what Hobart & Casella was about?