Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I am reading Bishop's PRML. Section 1.5.5 is about the expected loss function $L$ for regression, which is defined as:
$E [L] = \int \int L( y(x),t ) \ p(x, t) \ dx dt $
where $x$ is the input and $t$ is the target.
I don't understand this formulation. What is the intuition of the double integral over $x$ and $t$ and the expectation of a multivariate function over a continuous domain?
The loss function for a given sample $X$ with target value $T$ is defined as $L(y(x),t)$, where $y$ denotes the target estimation function. Here, both $X$ and $T$ are random variables, and one may wonder the expected loss given the regressor, i.e. $\mathbb E[L(y(X),T)]$ which can be calculated using the law of the unconscious statistician:
$$\mathbb E[L(y(X),T)] = \int_{\mathcal X}\int_{\mathcal T} L(y(x),t)p_{X,T}(x,t)dtdx$$
The intuition under joint integration is that $X$ and $T$ has dependence (o/w you wouldn't be able to predict $T$ from $X$) and any expected value expression consisting of these RVs, in general, must use the joint distribution. Being it in the continuous domain is specific to setup, apparently $X$ and $T$ are not assumed to be discrete RVs.
In his book, Bishop starts of with easy examples involving a few random variables and subsequently becomes more and more general as the chapters progess, until, finally, all quantities become random variables.
In the context at hand, both quantities, the regressors/features $x$ and the target $t$ are stochastic. Therefore, there exists uncertainty in both, the values of the regressors/features and those of the target given the regressors. This becomes all the more apparent, if the joint probability distribution $p(x,t)$ is decomposed into $$p(x,t) = p(t|x)p(x)$$ using Bayes' rule.
Clearly, the goal of learning is to minimize the expected error stemming from both sources, the uncertainty of the features and the target alike.
Classical regression only considers modeling the $p(t|x)$ part and assumes the $x$s are given (or equivalently that p(x) is given by an empirical distribution consisting of a sum of dirac-delta distributions). In a more general setup however one may head for a generative model for the $x$s as well, which involves the modeling of $p(x)$. Therefore, the formula in question is general enough to allow for both types of modeling.
