4
$\begingroup$

I was reading about learning curve and in a page, this curve is shown:

enter image description here


But I think something is wrong with it. If an estimator tunes it parameters on validation set, then validation error should be lower than training error. Because we have tuned estimator's hyper parameters to achieve the best result on validation error.
Why is training error lower than validation error in this figure?

Improve this question
$\endgroup$
6
  • 4
    $\begingroup$ I think you have it backwards. The parameters are tuned based on the training data. $\endgroup$ – dsaxton Jul 16 '15 at 19:32
  • $\begingroup$ @dsaxton seems to have identified the issue. Here is an article that helps describe the terms more clearly. $\endgroup$ – Jason Sanchez Jul 16 '15 at 19:37
  • $\begingroup$ @dsaxton you mean he has trained on training set and optimized hyper parameters by testing the model on validation set and named it "training error" ? $\endgroup$ – Mohammad Jul 16 '15 at 19:43
  • $\begingroup$ @Mohammad. The picture looks fine. The lower the curve, the smaller the error. $\endgroup$ – Michael M Jul 16 '15 at 20:32
  • $\begingroup$ @Mohammad Even though certain "meta" parameters are being optimized on the cross validation data, the model is still ultimately fit so as to minimize the error in the training data. For a given set of meta parameters, this will always tend to give smaller training error rates. The optimizing of those meta parameters just means that for a different set of such parameters both error rates would likely have been higher (with the training error still being the lesser). $\endgroup$ – dsaxton Jul 16 '15 at 20:33
1
$\begingroup$

Training error tends to be lower than cross validation error. This is an intuitive explanation, ignoring the random effects: In cross validation, you divide the train set T into two parts T1 and T2, train on T1 and test on T2. You tune the parameter to minimize the error on T2, but the validation error on T2 tends to be higher than the train error on T1 : $$er(T1)<e(T2)$$ because you train the model on T1 and have more opportunity to fit the model on it. On the other hand, er(T1)~er(T) as you train the model with the same tuned parameter on T1 and T. All together $$er(T2)>er(T)$$ which is what you also see in the diagram.

Improve this answer
$\endgroup$
1
$\begingroup$

In a very simple case, you have one training set and one test set. You train on the training set and then you test on the test set. As you have trained on the training set, the network has already seen the data and the optimization method was optimized for this data. Hence the error on the test set should be higher.

If you have hyper parameters which you want to optimize, you will split your training set into a training and a development set. You train for different choices on the training set, see the error on the development set and at the end, when you think everything is fine, you test on the test set. The test set should NEVER be used for optimization.

n-fold cross-validation makes better use of your data. You make $n$ disjunct sets $b_1, \dots, b_n$ of data. One bin $b_i$ is your test set and the rest $T_i = b_1 \cup \dots \cup b_n \setminus b_i$ is your training data. You train on $T_i$ and test on $b_i$ for all $i \in 1, \dots, n$. Then you average the error. You should not use this for hyper parameter optimization as this will add knowledge of the test set to your system. Hyper parameter optimization should only be done on the training set.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.