When training a multinomial classifier (7 different classes) with different feature sets, I am noticing that the learning curve error always peaks around the number of training samples that are equal to the number of features used in training. I am using k-fold cross validation with k=10 for generating the learning curve.

In the example below, I am using around 500 features for training. I am using a Gaussian Discriminant Analysis model for learning. If I change the number of features, the peak follows.

Is this expected? If so, what is the fundamental reason behind such a behavior?

enter image description here

  • $\begingroup$ What was the size of the test set? Did you vary that along with training data size? $\endgroup$ Sep 11 '16 at 5:49
  • $\begingroup$ The test set was set by the k-fold cross validation. In my case, since k=10 it would have been 10% of the full training set for each fold. $\endgroup$
    – Daniel V
    Sep 11 '16 at 6:07

You are probably overfitting until the number of samples is significantly higher than the number of features. As the number of samples grow, so does the model precision. And the low measured error with number of samples << number of features is probably just the lack of precision in measuring it via cross-validation. Around n samples ~ n features these two effects balance each other out.

  • $\begingroup$ This sounds reasonable but why do the effects balance out precisely at the number of features used? What is special about that number? $\endgroup$
    – Daniel V
    Sep 11 '16 at 6:09
  • $\begingroup$ Each additional sample starts to add to training instead of encouraging the overfitting. $\endgroup$
    – Diego
    Sep 11 '16 at 7:35
  • $\begingroup$ Yes. However, it's not clear to me still why the balance between overfitting and and training happens at the exact number of features. Does it mean that the dimension of the data is in the entire feature set space? Or are you saying that the entire data set is overfit by the model? I'm not an expert at stats so this is not obvious to me. Any further clarification would be nice $\endgroup$
    – Daniel V
    Sep 12 '16 at 21:10
  • 1
    $\begingroup$ Why do you use cross validation on the train test of you have plenty more test samples? I think that Deigo has a point. You use 10 fold cross validation on dataset smaller than 500, which means that you are verifying on few dozens. The idea behind cross validation is testing on samples from the same distribution that are new to the classifier. Since you have such samples, use them and check the results. Diego is also right regarding the overfitting. $\endgroup$
    – DaL
    Sep 13 '16 at 6:25
  • 1
    $\begingroup$ Yes, I meant that when you measure the accuracy on a small set, the measure accuracy is less reliable. $\endgroup$
    – DaL
    Sep 18 '16 at 5:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.