Expected loss function for regression: a Bayesian perspective

I am reading Bishop's PRML. Section 1.5.5 is about the expected loss function $$L$$ for regression, which is defined as:

$$E [L] = \int \int L( y(x),t ) \ p(x, t) \ dx dt$$

where $$x$$ is the input and $$t$$ is the target.
I don't understand this formulation. What is the intuition of the double integral over $$x$$ and $$t$$ and the expectation of a multivariate function over a continuous domain?

• Think about the double integral as double sum for a probability mass function and then it’s pretty intuitive that a similar kind of reasoning should extend to its continous counterpart. Apr 19, 2020 at 22:03

The loss function for a given sample $$X$$ with target value $$T$$ is defined as $$L(y(x),t)$$, where $$y$$ denotes the target estimation function. Here, both $$X$$ and $$T$$ are random variables, and one may wonder the expected loss given the regressor, i.e. $$\mathbb E[L(y(X),T)]$$ which can be calculated using the law of the unconscious statistician:

$$\mathbb E[L(y(X),T)] = \int_{\mathcal X}\int_{\mathcal T} L(y(x),t)p_{X,T}(x,t)dtdx$$

The intuition under joint integration is that $$X$$ and $$T$$ has dependence (o/w you wouldn't be able to predict $$T$$ from $$X$$) and any expected value expression consisting of these RVs, in general, must use the joint distribution. Being it in the continuous domain is specific to setup, apparently $$X$$ and $$T$$ are not assumed to be discrete RVs.

• Thanks. I don't understand why X and T are considered RVs. Aren't they usually given? Apr 20, 2020 at 16:43
• What you're given are observations which are sampled from $p_{X,T}(x,t)$. Apr 20, 2020 at 16:44
• @maurock is it still unclear? Apr 26, 2020 at 18:46

In his book, Bishop starts of with easy examples involving a few random variables and subsequently becomes more and more general as the chapters progess, until, finally, all quantities become random variables.

In the context at hand, both quantities, the regressors/features $$x$$ and the target $$t$$ are stochastic. Therefore, there exists uncertainty in both, the values of the regressors/features and those of the target given the regressors. This becomes all the more apparent, if the joint probability distribution $$p(x,t)$$ is decomposed into $$p(x,t) = p(t|x)p(x)$$ using Bayes' rule.

Clearly, the goal of learning is to minimize the expected error stemming from both sources, the uncertainty of the features and the target alike.

Classical regression only considers modeling the $$p(t|x)$$ part and assumes the $$x$$s are given (or equivalently that p(x) is given by an empirical distribution consisting of a sum of dirac-delta distributions). In a more general setup however one may head for a generative model for the $$x$$s as well, which involves the modeling of $$p(x)$$. Therefore, the formula in question is general enough to allow for both types of modeling.

• Thanks, it is indeed the first time I see the loss function expressed this way. Usually Xs are given, but now I understand why this was done. Is considering X as a random variable necessary to be able to generalize over the whole input space? Apr 19, 2020 at 21:06
• Not sure if I got the question. When you take a look at the context of section 1.5.5 Bishop uses the expression for the expected loss to derive a very general result, namely the form any optimal regression function $y(x)$, mapping features $x$ to target $y$, should assume, leading to eq. (1.89). To obtain that result in turn he needs to take $p(x)$ into account as witnessed likewise by eq. (1.89) Apr 19, 2020 at 21:41