3
$\begingroup$

I was reading https://people.stat.sc.edu/Hitchcock/stat535slidesday18.pdf and I came across with:

enter image description here

It gives us an way to get a distribution for a new observation called for $x_{new}$ given the past sample values $\mathbf{x}$ but $x_{new}$ independs from $\mathbf{x}$ so $P(x_{new}|\mathbf{x})=P(x_{new})$ and:

enter image description here

So here are my questions:

1-Why does $P(x_{new}|\mathbf{x})=\int_{\Theta}P(x_{new}|\theta,\mathbf{x})P(\theta|\mathbf{x})d\theta$ hold?

2-If so then Why is use it worth since I'm able to evaluate $P(x_{new}|\mathbf{x})$ by the second figure?

$\endgroup$

1 Answer 1

2
$\begingroup$

$p(x_{new}|x)$ is the posterior predictive distribution. It can be computed as an integral over the parameter $\theta$ as stated in your question, in fact this is just an application of marginal probability density functions (https://en.wikipedia.org/wiki/Marginal_distribution): $p(x_{new}|x)=\int_\theta p(x_{new},\theta|x) d\theta$.

$p(x_{new})$ is the prior predictive distribution. $p(x_{new}|x)$ is never equal to $p(x_{new})$ except in trivial cases, because $x_{new}$ is not independent of $x$. When the slides mention independence of $x$ and $x_{new}$, they actually mean independence conditional on $\theta$ i.e. the only 'connection' between $x$ and $x_{new}$ is via the parameter $\theta$.

Conditional independence (https://en.wikipedia.org/wiki/Conditional_independence): in this context it means that $p(x_{new}|x,\theta)=p(x_{new}|\theta)$. This is a standard assumption that is made in Bayesian modelling. It's particularly clear that this is reasonable if, say, the data are (conditionally) independent Bernoulli trials with parameter $p$. If we know $p$, then knowing the outcome of any number of Bernoulli trials ($x$) won't tell us anything more about the outcome of future trials ($x_{new}$) (because they're conditionally independent). If we don't know $p$, then $x$ can tell us a lot about $x_{new}$ (because they're not independent).

If conditional independence doesn't hold, we can still proceed but the computations might be more complex. This might happen if, for example, $x$ comprises observation of a continuous time stochastic process over a time interval $[0,T]$ and $x_{new}$ is observations over the future interval $(T,T+1]$.

$\endgroup$
5
  • $\begingroup$ What do you mean by 'independence conditional on θ' and why** is that true or even reasonable? $\endgroup$ Commented Nov 18, 2021 at 23:39
  • $\begingroup$ I made some edits... $\endgroup$ Commented Nov 18, 2021 at 23:42
  • $\begingroup$ it sounds like $x$ is an useless information if we know $\theta$ and it pretty makes sense.There is just a problem remaining, I didn't stated $P(x_{new}|x)=\int_{\theta} p(x_{new},\theta|x)d\theta$ I do $P(x_{new}|x)=\int_{\theta} p(x_{new}|\theta ,x)p(\theta|x)d\theta$ and your link just tell us about two random variables and its marginalization not 3 as we are in. (It actually tell us a little bit in the end's page but it doesnt use conditional regarding.) $\endgroup$ Commented Nov 19, 2021 at 0:01
  • $\begingroup$ The two integrands in your comment are equal. This is because $P(A,B|C)=P(A|B,C)P(B|C)$ $\endgroup$ Commented Nov 19, 2021 at 0:12
  • $\begingroup$ great, thank you $\endgroup$ Commented Nov 19, 2021 at 0:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.