I have a balanced training set; the test set follows the original distribution and is imbalanced. I fit a number of models to the training set, using 5-fold cross-validation. I perform this for 5 seeds, get the prediction of each run (per seed), average the predictions and round it up. I then get the train and test accuracy from here. I do this for different train sets of different sizes, but keep the test set the same every time (n_test=100000). I'm sure the train and test sets don't overlap.

So the seeds don't turn out to be much different from each other; the train and test accuracy are also fairly close. However, though the train and test accuracy increases with training size at first (until n_train=10000), at some point, it starts to dip for some models (SVM-RBF and KNN) (n=13000). This is a binary classification problem by the way.

Is this even possible, or does this mean I have a bug somewhere?

EDIT: I've added a plot of the train accuracy for various models with train size. Apologies for the quality...

enter image description here

  • $\begingroup$ You mentioned imbalance, describe in more detail. $\endgroup$ Dec 14, 2018 at 7:46
  • $\begingroup$ Original dataset has a 60-40 imbalance, with a total dataset size of ~250000 corresponding to individuals of difference demographics. The train-test split was stratified based on these demographics (age and gender categories); further, the train set was further subset by (a) making sure n_0_train = n_1_train = n_train/2 and (b) the demographic distribution of n_0_train and n_1_train are the same as n_0_test and n_1_test (stratified sampling). $\endgroup$
    – irene
    Dec 14, 2018 at 10:54
  • $\begingroup$ Can you check what the accuracies look like throughout time? To answer your question, yes, accuracy can decrease over time if you are overfitting, which is suggestive of your parameters, why are you using so many iterations? $\endgroup$ Dec 14, 2018 at 13:27
  • $\begingroup$ What do you mean "throughout time"? I use sklearn's built-in functions for the SVM and it gives me just the final results. Do you mean at every step of gradient descent? $\endgroup$
    – irene
    Dec 24, 2018 at 3:45
  • $\begingroup$ Let's back up. So you keep the test set fixed for all the models ($10^5$ observations) and you built many models using different number of observations for the training set (from somewhere below $10^4$ up to and to more than $1.3\cdot 10^4$ observations)? And your accuracy first increases and then decreases after using $1.3 \cdot 10^4$ observations for the training set for some models? Is you training set always balanced in all these models? Are the observations chosen at random for each training set or are you sequentially adding observations based on some ordering? $\endgroup$ Dec 24, 2018 at 13:56


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.