0
$\begingroup$

I have two physical models $f(\theta)$ and $g(\theta)$ (not probability distributions) parameterized on the same set of parameters $\theta$. I also have data $y$ with measurement noise $\epsilon$ which can be used to obtain posterior estimates $p(\theta | y)$ through one of the models: \begin{align} y &= f(\theta) + \epsilon \\ \text{where}\ \epsilon &\sim \text{Normal}(0, \sigma) \end{align} so a Gaussian likelihood function can be defined: \begin{equation} p(y | \theta) \sim \exp\left(\frac{-(y - f(\theta))^2}{2\sigma^2} \right) \end{equation}

Questions:

  1. Most of what I've read seems to be concerned with constructing posterior predictive distributions for new $y$ (denoting as $y_{\text{new}}$), as in $p(y_{\text{new}}|y) = \int p(y_{\text{new}}|\theta) p(\theta|y)d\theta$. But don't I just want the parameter-weighted distribution $f(\theta)p(\theta|y)$ without the measurement noise $\epsilon$ for the best estimate of $f(\theta)$?
  2. Letting $z=g(\theta)$, is it necessary to specify a probability density function $p(z|\theta)$ so that new predictions of $z$ are defined as the marginal distribution $p(z|y) = \int p(z|\theta) p(\theta|y)d\theta$ (e.g., Robert, The Bayesian Choice, 2007)? Can I not just construct a probability distribution for the second physical model from $g(\theta)p(\theta|y)$? For instance using MCMC I could construct the probability distribution from the sequence $\{g(\theta_1), g(\theta_2), \ldots, g(\theta_t)\}$.
$\endgroup$
  • $\begingroup$ When I started reading the question, I had the impression that $\theta$ was a fixed (but unknown) parameter, but at the end of the question $\theta$ is given an index that makes it seem more like a stochastic process. Did you really mean something like $y_t = f(x_t,\theta) + \varepsilon_t$ and $z_t = g(w_t,\theta) + e_t$? $\endgroup$ – mef Aug 12 at 14:08
  • $\begingroup$ Sorry for the confusion - the indices are meant to signify realizations generated by MCMC. $\endgroup$ – hatmatrix Aug 12 at 21:42
1
$\begingroup$
  1. Most of what I've read seems to be concerned with constructing posterior predictive distributions for new $y$ (denoting as $y_{\text{new}}$), as in $p(y_{\text{new}}|y) = \int p(y_{\text{new}}|\theta) p(\theta|y)d\theta$. But don't I just want the parameter-weighted distribution $f(\theta)p(\theta|y)$ without the measurement noise $\epsilon$ for the best estimate of $f(\theta)$?

Expected value from the posterior distribution is "best" only in terms of minimizing squared error (for details, check The Bayesian Choice, already mentioned by you), but you could want instead look at other statistics, for example median of the posterior distribution (minimize absolute loss), or mode (maximum a posteriori estimate), etc. If you have estimate of the posterior distribution, you can easily estimate any of those quantities.

  1. Letting $z=g(\theta)$, is it necessary to specify a probability density function $p(z|\theta)$ so that new predictions of $z$ are defined as the marginal distribution $p(z|y) = \int p(z|\theta) p(\theta|y)d\theta$ (e.g., Robert, The Bayesian Choice, 2007)? Can I not just construct a probability distribution for the second physical model from $g(\theta)p(\theta|y)$? For instance using MCMC I could construct the probability distribution from the sequence $\{g(\theta_1), g(\theta_2), \ldots, g(\theta_t)\}$.

By the law of unconscious statistician

$$ E[g(x)] = \int g(x)\,p(x)\, dx $$

If you have samples from the posterior distribution of $\tilde\theta_1,\tilde\theta_2,\dots,\tilde\theta_n \sim p(\theta|y)$, then to obtain the samples of their transformations, you just need to transform the samples; to get estimate of the expected value of the transformation of $\theta$, just calculate the empirical mean of the transformed MCMC samples.

$\endgroup$
  • $\begingroup$ I didn't mean to imply that the "mean" was best - only that the posterior mode, mean, median etc. for $f$ could be calculated from the distribution $f(\theta)p(\theta|y)$ instead of invoking the likelihood function $p(y|\theta)$. But great to have confirmation on LOTUS. $\endgroup$ – hatmatrix Aug 12 at 21:45
  • 1
    $\begingroup$ @hatmatrix if you have way of estimating $p(\theta|y)$ without using Bayes theorem, then yes, you don't need likelihood, but usually don't have other way. $\endgroup$ – Tim Aug 12 at 22:53
  • $\begingroup$ For prediction, to reproduce $y$-like data I need the likelihood, but if I'm actually interested in $f$ or $g$ I don't need to use the likelihood again after using it for estimation of $p(\theta|y)$. $\endgroup$ – hatmatrix Aug 13 at 9:48
  • 1
    $\begingroup$ @hatmatrix yes, if you already know the posterior, you don't need the likelihood, since you needed the likelihood only to estimate the posterior. $\endgroup$ – Tim Aug 13 at 10:33
  • $\begingroup$ Thanks for the clarification - I think most textbooks seem focused on reproducing $y$ so they show the posterior distribution for prediction shown in the form above, but I just wanted a sanity check that the posterior parameter distribution can be used to derive distributions for other models (using for instance theorem of the unconscious statistician). $\endgroup$ – hatmatrix Aug 13 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.