I have a balanced training set; the test set follows the original distribution and is imbalanced. I fit a number of models to the training set, using 5-fold cross-validation. I perform this for 5 seeds, get the prediction of each run (per seed), average the predictions and round it up. I then get the train and test accuracy from here. I do this for different train sets of different sizes, but keep the test set the same every time (n_test=100000). I'm sure the train and test sets don't overlap.
So the seeds don't turn out to be much different from each other; the train and test accuracy are also fairly close. However, though the train and test accuracy increases with training size at first (until n_train=10000), at some point, it starts to dip for some models (SVM-RBF and KNN) (n=13000). This is a binary classification problem by the way.
Is this even possible, or does this mean I have a bug somewhere?
EDIT: I've added a plot of the train accuracy for various models with train size. Apologies for the quality...