3
$\begingroup$

I have a model with several parameters. I apply Bayesian inference with a uniform prior for all of the parameters. After the process is finished, I realize that I need one of those parameters $x$ parametrized as its logarithm:

$$y = log(x/a)$$

where $a$ is a constant. As far as I understand, a uniform prior in $x$ is a very much non-uniform prior in $y$.

Can I transform the posterior I got using a uniform prior in $x$ into a posterior with a uniform prior in $y$?

$\endgroup$

2 Answers 2

1
$\begingroup$

Parameters are usually written in Greek letters, so I'll write $\eta = \log \frac{\theta}{c}$ instead. Denoting the likelihood by $L(\theta\mid X)$, the posterior distribution you sampled from is $$ \pi(\theta\mid X) \propto L(\theta\mid X)\pi(\theta) \propto L(\theta\mid X), $$ since $\pi(\theta) \propto 1$. If you apply $\eta(\theta) = \log \frac{\theta}{c}$ to the posterior sample $\{\theta^{(1)},\ldots,\theta^{(S)}\}$ directly, it is equivalent to sampling from the following posterior: $$ \pi^*(\eta\mid X) \propto L(\eta\mid X) \mathcal{J}(\eta\to\theta), $$ where $\mathcal{J}(\eta \to \theta)$ is the Jacobian $\left|\frac{d\theta}{d\eta} \right| = ce^{\eta}$.

My understanding is that you would like to transform the posterior sample $\{\theta^{(1)},\ldots, \theta^{(S)}\}$ so that it represents a sample from $\pi(\eta\mid X) \propto L(\eta\mid X)$, which lacks the Jacobian term in $\pi^*(\eta\mid X)$. In general, prior swapping is not easy, and you would do well to start over with the correct prior distribution.

However, if we give up equal weights for every posterior sample point, we can correct the weights so that the weighted sample represents the desired distribution, an idea often seen in importance-sampling literature. To that end, first transform $\eta^{(s)} = \log \frac{\theta^{(s)}}{c}$ for $s=1,\ldots,S$, and attach a weight to each sample point: $\{(\eta^{(1)},w^{(1)}),\ldots, (\eta^{(S)},w^{(S)})\}$. @Peter Pang's answer is correct in that the weights are the ratio of the correct posterior to the original: $w^{(s)} = \frac{L(\eta\mid X)}{L(\eta\mid X)\mathcal{J}(\eta\to\theta)} = \frac{1}{\mathcal{J}(\eta \to \theta)}=\frac{1}{c\exp(\eta^{(s)})} = \frac{1}{\theta^{(s)}}$. Subsequent analysis should be carefuly tuned following the rules of importance sampling. See Neiswanger et el. for more information.

$\endgroup$
1
$\begingroup$

First, for a uniform prior on $x$, the corresponding prior on $y$ is given by $$ p(y) = p(x) \left\vert\frac{dy}{dx}\right\vert^{-1} \propto x. $$

Using this Jacobian, you can re-weight the posterior to a flat prior on $y$.

If you are using python and pandas for the analysis, you may find the following code useful.

y = np.log(samples['x'] / a)
J = samples['x'] # Jacobian
weights = 1 / J
weights /= np.sum(weights) # normalization
plt.hist(samples['y'], weights=weights, bins=50, deisnty=True)

The resulting histogram would be the posterior on $y$ given a uniform prior on $y$. Or you can simply resample based on the weight. But this method is not perfect as it could suffer from a low effective sample size $n_{\rm eff}$, which is given by $$ n_{\rm eff} = \frac{(\sum_{i=1}^n w_i)^2}{\sum_{i=1}^n w_i^2}, $$ where $w_i = 1 / x_i$ for your case. A re-analysis with a uniform prior on $y$ is always a better way.

You may find an example plot as follows enter image description here

This plot nicely shows both the method works and also its shortcomings.

  1. The resulting distribution is uniform for relatively higher $y$
  2. For lower $y$, because there are no samples with such a low value to begin with, the reweighting does not help.
$\endgroup$
3
  • $\begingroup$ I did not downvote and I'd like to know the reason for the downvote. Is the answer incorrect? $\endgroup$
    – Gabriel
    Commented Mar 14, 2022 at 18:08
  • $\begingroup$ Maybe you can try with samples uniform in $x$ and resample with the weighting I presented and see if the distribution on $y$ is uniform. $\endgroup$
    – Peter Pang
    Commented Mar 14, 2022 at 18:15
  • $\begingroup$ @DaeyoungLim OP is asking a method for transforming a posterior with one prior (uniform in $x$) to another one (uniform in $y$) without redoing the analysis. That's why I suggested the method I described, which reweight the posterior from one prior to another. $\endgroup$
    – Peter Pang
    Commented Mar 14, 2022 at 21:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.