# Mathematical Motivation of Splitting Into Training and Testing Sets

In Learning from Data course taught by Caltech Professor Yaser Abu-Mostafa the following notation is used to describe the in sample and out of sample errors.

$$E_{in}[h]=\dfrac{1}{N}\sum_{n=1}^Ne(h(x_n),f(x_n))$$

where $$h(x_n)$$ is our selected model, and $$f(x_n)$$ is the target function.

$$E_{out}[h]=E_{X}[e(h(x_n),f(x_n))]$$

It's clear that in sample error is only an empirical estimate of the out of sample error.

My question is the following, in order to evaluate our model we use the testing accuracy instead of the training accuracy. Assuming that the samples are i.i.d, is there a mathematical reason using testing accuracy is a better evaluation of the out of sample performance of the model than the training accuracy?

I'm aware of the notion of overfitting. However, I'm interested in a mathematical description which leads us to the motivation of separating training and testing sets.

The very obvious reason that first comes to my mind is the fact that we actually minimize $$E_{in}$$ in training process, via explicit solving or iteratively, like gradient descent approaches. For example, for linear regression, we find the coefficients that minimizes the MSE, by explicitly taking the derivative and equating it to zero for solving, i.e. $$\partial E_{in} / \partial w=0$$. Finding such $$w$$ does guarantee the minimization of $$E_{in}$$, but not $$E_{out}$$. And, in the cases where our final error metric (e.g. 1-accuracy) is different than the training loss function, we normally assume that our loss function is a good representative of the final error metric.