Notation predictive posterior distribution and $x^*$, $y^*$

Question

I often see the posterior predictive distribution in ML defined as follows:

$$p(y^* \mid x^*, X, Y) = \int p(y^* \mid x^*, \omega)p(\omega, X, Y) d\omega$$

where $\omega$ are all parameters, $x^*$ is a new input point and $X, Y$ is the training dataset.

What confuses me is the lower case $y^*$ and $x^*$, because I am not sure whether it is a random variable and where it comes from.

Without knowing a lot about Bayesian statistics, I would first define the posterior $P(W \mid X, Y)$ (with $W$ being the parameters). Then use the law of total probability to obtain

$$P(Y \mid X) = \int P(Y \mid X, W)P(W)dW$$

Next, when I get a new point $x^*$, I would set $P(Y= y^* \mid X = x^*)$. Are $y^*$ and $x^*$ as random variables necessary?

Answer 1

In the notation of the posterior predictive distribution $$p(y^* \mid x^*, X, Y) = \int p(y^* \mid x^*, \omega)p(\omega, X, Y)\, \text{d}\omega\tag{1}$$ in the question,

0. the posterior density on the parameter vector $\omega$ should be denoted $p(\omega|X,Y)$;
1. $p(\cdot|x^*,x,y)$ is a density function $$p(\cdot|x^*,x,y)\,:\ \mathcal Y \longmapsto \mathbb R^*_+\tag{2}$$ indexed by the triplet $(x^*,x,y)$; it is the conditional density function of a random variable, possibly denoted $Y^*$, given $X^*=x$ and the learning sample $(X,Y)$;
2. $y^*$ is the dummy argument of the density, it could be equally, written $z$, $\zeta$, $\Upsilon$, or even ygrec, as well; $y^*$ is preferred for the analogy with the components of the learning set $Y$ but it is not a random variable by default, even though the function $p$ in (2) could be applied to a random variable;
3. $\omega$ is the vector of parameters indexing the conditional sampling probability density $p(\cdot \mid x^*, \omega)$; while $\omega$ is a random variable in the Bayesian framework, with prior density $q(\omega|X)$ say, it is not denoted by a capital letter, like $\Omega$ or $W$. The reason is that (i) this could prove confusing, since $\Omega$ [capital Greek letter] is also traditionally the parameter space and the underlying Borel space of measurable sets, while (ii) Bayesian inference returns the posterior distribution and computes summaries of that distribution, like posterior moments or quantiles. Writing$$\int p(y^* \mid x^*, W)p(W, X, Y)\, \text{d}\tag{3}W$$is unusual if formally correct because the notation $W$ indicates a random variable but the integral is computed wrt a dummy variable, rarely denoted by a capital letter (and (3) is not a random quantity);
4. $(X,Y)$ is the training dataset and the capitals are intended for vectors and matrices rather than for random variables, and furthermore $Y$ is actually a realisation of a random vector, thus not a random variable (and starting from the model, everything is conditional on $X$);
5. the expression $P(Y=y^*|X=x^*)$ is incorrect because $Y$ and $y^*$ (as well as $X$ and $x^*$) are objects of different dimensions. For instance $Y$ is made of $n$ replications $y_i$'s, of the same dimension as $y^*$. Furthermore, if $Y^*$ is a continuous variable, $$P(Y^*=y^*|X^*=x^*,X,Y)=0$$ In learning terms, $(X,Y)$ is the learning set, $X=(x_1,\ldots,x_n)$ being the predictors and $Y=(y_1,\ldots,y_n)$ the outcomes.

Answer 2

What you defined in $P(Y|X)=\int P(Y|X,\omega)P(\omega)d\omega$ is the prior predictive function, which is generally used to check if the prior distribution for $\omega$ is reasonable. Notice that the distribution of $\omega$ that appears in this formula is the prior $P(\omega)$, not the posterior $P(\omega|X,Y)$.

Now, we can define $x^*, y^*$ as new random variables such that:

$$y^*|\omega,x^*\sim Y_i|\omega,X_i$$

That is, the new data will follow the same conditional distribution as the training data. Why do we create a new random variable if it has the same distribution as $Y$? Because, as good Bayesians, we want to condition on $Y$. It wouldn't make much sense to condition $Y$ on itself, as it would result in a singular distribution. Also, $Y$ is usually taken to be a vector containing all the outputs of the training dataset.

Now, the random variable $y^*$ depends only on $x^*$ and $\omega$, and $\omega$ depends on $X$ and $Y$. Using this (in)dependence structure and the total probability law, we have:

$$\begin{align} P(y^*|x^*,X,Y)&=\int P(y^*,\omega|x^*,X,Y)d\omega\\ &=\int P(y^*|x^*,\omega)P(\omega|X,Y) \end{align}$$

Which is the posterior predictive distribution.