I plotted the learning curve using the false-positive rate (FPR) as the scoring function. The function was of course adjusted to reflect the lower FPR as the higher score. The resulting learning curve looks something like this:
What do I make of this curve? More samples is bad? But less samples would also mean over-fitting, isn't it?
To address one of the comments, here is a little more context. I am using a Random-Forest Classifier (so not a linear model). Also, the learning curve uses the StratifiedKFold cross-validator with shuffling enabled. The problem that I am solving is classifying static documents into one of two classes and therefore features are based on the properties extracted out of the documents. I have about 20 numerical features.
For completeness, here's a similar plot with accuracy score (default) as the scoring function.