0
$\begingroup$

In Learning from Data course taught by Caltech Professor Yaser Abu-Mostafa the following notation is used to describe the in sample and out of sample errors.

$E_{in}[h]=\dfrac{1}{N}\sum_{n=1}^Ne(h(x_n),f(x_n))$

where $h(x_n)$ is our selected model, and $f(x_n)$ is the target function.

$E_{out}[h]=E_{X}[e(h(x_n),f(x_n))]$

It's clear that in sample error is only an empirical estimate of the out of sample error.

My question is the following, in order to evaluate our model we use the testing accuracy instead of the training accuracy. Assuming that the samples are i.i.d, is there a mathematical reason using testing accuracy is a better evaluation of the out of sample performance of the model than the training accuracy?

I'm aware of the notion of overfitting. However, I'm interested in a mathematical description which leads us to the motivation of separating training and testing sets.

$\endgroup$
1
$\begingroup$

The very obvious reason that first comes to my mind is the fact that we actually minimize $E_{in}$ in training process, via explicit solving or iteratively, like gradient descent approaches. For example, for linear regression, we find the coefficients that minimizes the MSE, by explicitly taking the derivative and equating it to zero for solving, i.e. $\partial E_{in} / \partial w=0$. Finding such $w$ does guarantee the minimization of $E_{in}$, but not $E_{out}$. And, in the cases where our final error metric (e.g. 1-accuracy) is different than the training loss function, we normally assume that our loss function is a good representative of the final error metric.

$\endgroup$
  • $\begingroup$ Thank you for your answer. $\endgroup$ – dikey y May 17 '19 at 16:03
  • $\begingroup$ @dikeyy you’re welcome; if you think the answer is ok for you, can you please accept and/or upvote it? $\endgroup$ – gunes May 17 '19 at 16:30
  • $\begingroup$ Although, the answer is helpful I'm looking for a more comprehensive one. A follow up question could be if the samples in training are representative of the population why there should be a difference between E_{in} and E_{out}? $\endgroup$ – dikey y Nov 11 '19 at 19:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.