I plotted the learning curve using the false-positive rate (FPR) as the scoring function. The function was of course adjusted to reflect the lower FPR as the higher score. The resulting learning curve looks something like this: enter image description here

What do I make of this curve? More samples is bad? But less samples would also mean over-fitting, isn't it?

To address one of the comments, here is a little more context. I am using a Random-Forest Classifier (so not a linear model). Also, the learning curve uses the StratifiedKFold cross-validator with shuffling enabled. The problem that I am solving is classifying static documents into one of two classes and therefore features are based on the properties extracted out of the documents. I have about 20 numerical features.

For completeness, here's a similar plot with accuracy score (default) as the scoring function.

enter image description here

  • 1
    $\begingroup$ Have you also plotted the true positive rate? It seems like maybe initially the classifier is outputting only positives, and then gradually starts learning to output some negatives, some of which are incorrect. $\endgroup$ – Dougal Jun 6 '17 at 18:13
  • $\begingroup$ @Dougal I did plot the accuracy score, not the true-positive-rate, and it seemed like an expected plot. But not with false positive rate. $\endgroup$ – sandyp Jun 6 '17 at 18:16

But less samples would also mean over-fitting, isn't it?

Assuming your training/validation/test splits are shuffled and random, this is not necessarily true. Fewer samples can mean lower bias, but higher variance. Your model must generalize solutions based on sparse information, rather than becoming overly opinionated based on spurious patterns in the input space.

You didnt mention what kind of data you have or what type of problem you are trying to solve, but it is entirely possible that your model overfits after 5,000 samples, in which case, you should stop training after it has seen those 5,000 samples.

I would imagine it is possible that your training data has a distribution of features/classes that is not optimal, or that you are using the wrong model to solve this problem. Maybe you are using a linear model to fit non-linear data, maybe all of positive classes are in the first 5,000 training samples, etc.

  • $\begingroup$ Just updated the question in that I am using the RF classifier. Non-optimal features would be one guess but how can we work around with this set of features. I do not agree with your comment about over-fitting after 5K samples as we observe the training score drops after 5K samples. Also, stopping training after a fixed number of samples does not sound right. More samples should help generalize the model better? $\endgroup$ – sandyp Jun 6 '17 at 18:05
  • $\begingroup$ When your accuracy goes down, it is possible that your model is overfitting. Also, look at my edit. Is it possible you havent shuffled your training data? Maybe the first 5k samples are all the same class $\endgroup$ – redress Jun 6 '17 at 18:07
  • $\begingroup$ Sorry, forgot to mention that I have shuffled my data as well. Also, note that the score is not based on accuracy, rather, the false positive rate. $\endgroup$ – sandyp Jun 6 '17 at 18:08
  • $\begingroup$ Accuracy is a function of precision, which is itself a calculation of the false positive rate $\endgroup$ – redress Jun 6 '17 at 18:09
  • $\begingroup$ You have yet to mention what kind of data you are using and the type of problem you are attempting to solve $\endgroup$ – redress Jun 6 '17 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.