I want to draw Bayesian inference via importance sampling and I do not come up with a good idea of an importance density for $$p(\sigma)\sim\frac{1}{\sigma}.$$ Is there a way to sample from this distribution directly? I am not sure whether $$\frac{1}{z}, z\sim\mathcal{U}[0,1]$$ will give me the desired property.

Update: I implemented the proposed importance function and computed the weights $$w_t=p(\sigma_t|\mathbf{y})\Big/\left\{\frac{1}{2a}\mathbb{I}_{[0,a]}(\sigma_t)+\frac{a}{2}\mathbb{I}_{[a,\infty)}(\sigma_t)\sigma_t^{-2}\right\}$$ for a sample of $m=1.000.000$ draws. Hereby I computed the log likelihoods $\omega_{log}$, substracted the maximum $\omega_{log}$ and transformed $\omega=\exp(\omega_{log}-\omega_\max)$. What can I learn from the fact, that roughly $0.98 %$ percent of the weights are $0$? I suppose that in this case the choice of the importance sample is not sensitive, as the sample size does not really matter but only a couple of observations drive the whole result?

  • $\begingroup$ Well, I want to implement Importance sampling for a model that uses the Jeffreys prior. Therefore I am searching for a way to come up with a suitable importance density $q(\sigma)$ in order to compute posterior integrals $\int h(\sigma)p(\sigma|Y)d\sigma=\int h(\sigma)\frac{p(\sigma|Y)}{q(\sigma)}q(\sigma)d\sigma$. $\endgroup$ Commented Nov 23, 2015 at 10:57

2 Answers 2


The short answer to your question is that it is impossible to simulate from $p(\sigma)=1/\sigma$ as this is not a probability density but a measure with infinite mass.

Since you also mention Jeffreys and importance in the same sentence, it may however be that you are actually asking about simulating a posterior associated with $p(\sigma)=1/\sigma$ as the Jeffreys prior. In this case, a fat tailed importance sampler would do. For instance, if the posterior accumulates mass around $a$, e.g., if $a$ is roughly the mode. you could use $$\frac{1}{2}\mathcal{U}[0,a]+\frac{1}{2}1\big/\mathcal{U}[0,1/a]$$ as an importance function. This means

  1. Simulate a sample $\sigma_1,\ldots,\sigma_M$ from this distribution;
  2. Weight the above simulations by $$w_t=p(\sigma_t|\mathbf{y})\Big/\left\{\frac{1}{2a}\mathbb{I}_{[0,a]}(\sigma_t)+\frac{a}{2}\mathbb{I}_{[a,\infty)}(\sigma_t)\sigma_t^{-2}\right\}$$

since the density of an inverse uniform is $$ f(z)=a\mathbb{I}_{(0,1/a)}(1/z)\frac{\text{d}u}{\text{d}z}=a\mathbb{I}_{z>a}\frac{1}{z^2} $$

Remember, always check that the resulting importance weights have finite variance!

  • $\begingroup$ I just realized that I do not know what to use as $q(\sigma)$ under this importance density...can you give a comment on that? $\endgroup$ Commented Nov 23, 2015 at 12:37
  • $\begingroup$ Is there a theoretical framework recommending this idea? I get the point with the fat tails, however, I would never come up with this distribution. Is it somewhat standard in importance sampling literature? $\endgroup$ Commented Nov 24, 2015 at 10:00
  • $\begingroup$ By the way, is the weighting function really correct? Why does the second term include a $\sigma^{-2} and the first one not? $\endgroup$ Commented Nov 24, 2015 at 10:08
  • 1
    $\begingroup$ @muffin1974: No to all questions:1. this is a default proposal if you cannot come up with anything else; 2. it has no particular theoretical property that I know of; 3. it is not standard. I made this proposal to illustrate importance sampling not to suggest a universal solution. $\endgroup$
    – Xi'an
    Commented Nov 24, 2015 at 10:10

Is it really impossible to sample from $p(\sigma) = 1/\sigma$?

Consider: $p(\sigma) = 1/ \sigma$ (the Jeffreys prior on scale parameter), and $(y \mid \sigma) \sim N(0, \sigma^2)$, then the posterior can be derived with usual Bayesian analysis: \begin{align*} p(\sigma \mid y) &\stackrel{\sigma}{\propto} p(\sigma, y) = p(y \mid \sigma) p(\sigma)\\ &= \frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{y^2}{2\sigma^2}\right) \cdot \frac{1}{\sigma}\\ \end{align*} To match the last expression with that of an inverse-Gamma distribution, reparameterize in terms of $\sigma^2$: $p(\sigma^2) = \left.p(\sigma) \middle/ \left|\text{det}\left(\frac{d(\sigma^2)}{d\sigma}\right)\right| \right. = p(\sigma) / (2\sigma)$. Thus \begin{align*} p(\sigma^2 \mid y) & = p(\sigma \mid y) / (2\sigma) \\ &\stackrel{\sigma}{\propto}\frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{y^2}{2\sigma^2}\right) \cdot \frac{1}{\sigma} / (2\sigma) \\ &\stackrel{\sigma}{\propto} \frac{1}{(\sigma^2)^{3/2}}\exp\left(-\frac{y^2}{2\sigma^2}\right)\\ (\sigma^2 \mid y) &\sim \text{IG}\left(\alpha=\frac{1}{2}, \beta = \frac{y^2}{2}\right) \end{align*}

Now that we've got both $(\sigma^2 \mid y)$ (inverse Gamma) and $(y \mid \sigma^2)$ (Gaussian), a Gibbs sampler is can be done from the initial value, of say, $\sigma_\text{init} = 1$, and alternate between $y$ and $\sigma$.

Or, is this what Hobart & Casella was about?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.