1
$\begingroup$

Assume $\mathbf{X}$ has a pdf that depends on the two parameters $\mu, \sigma$ given by $h(\mathbf{x}|\mu, \sigma)$. Traditionally, Bayes' theorem allows the computation of the posterior pdf $g(\mu, \sigma|\mathbf{x})$ via $$ g(\mu, \sigma|\mathbf{x}) \propto h(\mathbf{x}|\mu, \sigma)\cdot g(\mu, \sigma). $$ Suppose, however, that I wish to reparameterize the pdf of $\mathbf{X}$ via $h(\mathbf{x}|f_1(\mu, \sigma), f_2(\mu, \sigma))$ for some functions $f_1$ and $f_2$ of the original parameters. I would still like to specify my prior in terms of the original parameters, like $$ g(\mu, \sigma|\mathbf{x}) \propto h(\mathbf{x}|f_1(\mu, \sigma), f_2(\mu, \sigma))\cdot g(\mu, \sigma), $$ but I'm not sure this is permissible. Would I have to compute the pdf $\tilde{g}$ of $f_1(\mu, \sigma), f_2(\mu, \sigma)$ and then use $$ g(\mu, \sigma|\mathbf{x}) \propto h(\mathbf{x}|f_1(\mu, \sigma), f_2(\mu, \sigma))\cdot \tilde{g}(f_1(\mu, \sigma), f_2(\mu, \sigma)) $$ instead?

Example: Suppose each $X_i \sim \mathcal{N}\left( \left(\mu - \frac{\sigma^2}{2}\right)\Delta t, \sigma^2 \Delta t \right)$ and is independent for $i=1,\dots,N$. Here, $\Delta t$ is just some fixed constant. In the notation above, $f_1(\mu,\sigma) = \left(\mu - \frac{\sigma^2}{2}\right)\Delta t$ and $f_2(\mu,\sigma) = \sigma^2 \Delta t$ (so $f_2$ is not dependent on $\mu$ in this case). I'd like to specify the marginal pdfs of $\mu$ and $\sigma$ and maximize the posterior pdf $g(\mu,\sigma|\mathbf{x})$.

$\endgroup$
3
  • $\begingroup$ Do you mean you are interested in the posterior distribution of $(f_1,f_2)$ ? Otherwise I don't see why you want to change the parameterization of the pdf (may be it could be clearer with an example). $\endgroup$ Jul 13 '15 at 20:36
  • $\begingroup$ @StéphaneLaurent In my context, I'm interested in maximizing the posterior joint pdf of $\mu$ and $\sigma$. Example added. $\endgroup$
    – bcf
    Jul 13 '15 at 20:45
  • $\begingroup$ if you can solve $f_1$ and $f_2$ so that you have explicit form for $\mu$ and $\sigma^2$ then you can substitute them in likelihood. For priors you are free to choose pdfs (if you have less knowledge about them). Finally you can reverse estimation based on $f_1$ and $f_2$ an get estimation of them $\endgroup$
    – TPArrow
    Jul 13 '15 at 21:20
1
$\begingroup$

To answer the more broad question, $$ g(\mu, \sigma|\mathbf{x}) \propto h(\mathbf{x}|f_1(\mu, \sigma), f_2(\mu, \sigma))\cdot g(\mu, \sigma) $$ would be correct and in most instances the functional notation is implied so we write $$ g(\mu, \sigma|\mathbf{x}) \propto h(\mathbf{x}|\mu, \sigma)\cdot g(\mu, \sigma) $$ regardless. Using your notation,

$$ g(f_1(\mu, \sigma), f_2(\mu, \sigma)|\mathbf{x}) \propto h(\mathbf{x}|f_1(\mu, \sigma), f_2(\mu, \sigma))\cdot \tilde{g}(f_1(\mu, \sigma), f_2(\mu, \sigma)) $$

would also be acceptable, and may be preferred if you can invert $f_1$ and $f_2$ to simulate $g(\mu|\sigma,\mathbf{X})$ and $g(\sigma|\mu,\mathbf{X})$ from $g(f_1(\mu, \sigma), f_2(\mu, \sigma)|\mathbf{x})$ which I do below using your example.

In practice, the parameters of the likelihood are almost always expressed as functions of multiple other parameters relevant to the model being analysis. For example the likelihood of a non-linear regression model $y_i = f(x_i,\boldsymbol{\beta})+\varepsilon_i$ may be written as $\prod_{i=1}^n h(y_i|x_i,\boldsymbol{\beta},\sigma^2)$ where $h(y_i|x_i,\boldsymbol{\beta},\sigma^2) \equiv N(f(x_i,\boldsymbol{\beta}),\sigma^2)$.

It is certainly true that $f_1$ and $f_2$ make life harder because it is more challenging if not impossible to find conjugate priors. However, we can usually use Monte Carlo techniques to simulate the posterior distribution anyway for these cases.

Take your example for instance. This looks like a continually compacted return of a stock where that stock price follows geometric brownian motion, in which case $\Delta t$ is not a random variable but is in fact known. So without loss of generality, let $\Delta t=1$.

One option would be to use substitution by letting $\mu_* = \bigg(\mu-\frac{\sigma^2}{2} \bigg)$ so that $X_i \sim N(\mu_*,\sigma^2)$. Choose the appropriate conjugate priors for $\mu_*$ and $\sigma^2$ (i.e. normal-gamma) so that the posterior distribution of the parameters is known and has a closed form. Then you can simply simulate $\mu=\mu_*+\frac{\sigma^2}{2}$ from that posterior distribution to obtain a Monte Carlo sample from the posterior distribution of $\mu$.

An alternative approach would be to specify a normal prior for $\mu$ so that the conditional posterior $g(\mu|X,\sigma^2)$ is conjugate normal. You can then use a Gibbs sampling type approach where $g(\sigma^2|X,\mu)$ would have to be drawn using a Metropolis-Hastings step.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.