# Why should validation error be higher than training error?

I was reading about learning curve and in a page, this curve is shown:

But I think something is wrong with it. If an estimator tunes it parameters on validation set, then validation error should be lower than training error. Because we have tuned estimator's hyper parameters to achieve the best result on validation error.
Why is training error lower than validation error in this figure?

• I think you have it backwards. The parameters are tuned based on the training data. – dsaxton Jul 16 '15 at 19:32
• @dsaxton seems to have identified the issue. Here is an article that helps describe the terms more clearly. – Jason Sanchez Jul 16 '15 at 19:37
• @dsaxton you mean he has trained on training set and optimized hyper parameters by testing the model on validation set and named it "training error" ? – Mohammad Jul 16 '15 at 19:43
• @Mohammad. The picture looks fine. The lower the curve, the smaller the error. – Michael M Jul 16 '15 at 20:32
• @Mohammad Even though certain "meta" parameters are being optimized on the cross validation data, the model is still ultimately fit so as to minimize the error in the training data. For a given set of meta parameters, this will always tend to give smaller training error rates. The optimizing of those meta parameters just means that for a different set of such parameters both error rates would likely have been higher (with the training error still being the lesser). – dsaxton Jul 16 '15 at 20:33

Training error tends to be lower than cross validation error. This is an intuitive explanation, ignoring the random effects: In cross validation, you divide the train set T into two parts T1 and T2, train on T1 and test on T2. You tune the parameter to minimize the error on T2, but the validation error on T2 tends to be higher than the train error on T1 : $$er(T1)<e(T2)$$ because you train the model on T1 and have more opportunity to fit the model on it. On the other hand, er(T1)~er(T) as you train the model with the same tuned parameter on T1 and T. All together $$er(T2)>er(T)$$ which is what you also see in the diagram.
n-fold cross-validation makes better use of your data. You make $n$ disjunct sets $b_1, \dots, b_n$ of data. One bin $b_i$ is your test set and the rest $T_i = b_1 \cup \dots \cup b_n \setminus b_i$ is your training data. You train on $T_i$ and test on $b_i$ for all $i \in 1, \dots, n$. Then you average the error. You should not use this for hyper parameter optimization as this will add knowledge of the test set to your system. Hyper parameter optimization should only be done on the training set.