While doing hyperparameter tuning to find appropriate k, we use validation error to find k. Then what is the use of training error here?
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
When doing cross-validation you care about validation error, not the train error. With $k$-NN, you would get perfect train error with $k=1$ i.e. returning the same sample as your prediction. Train error may be useful to judge overfitting, when the difference between train and validation error is relatively big.
$\begingroup$
$\endgroup$
The training error is what is being minimized internally with the optimization algorithm, so it is not useless. The optimization algorithm in itself is unaware of the validation error. The model selection algorithm is what uses the validation error. Together (optimization + model selection) they make up the learning algorithm.