I have 20 observations labeled for each class. The number of classes is 5, so I have a total of 100 observations. I want to classify one class vs. other classes (one vs. all).
For this, I first labeled '1' for one of five classes (= 20 observations), and '2' for the other classes (= 80 observations). After that, I spiltted the whole data into training data set and test data set for each class ('1' or '2'). Since I used 10-fold cross-validation method, 20 observations for class '1' were separated into 18 and 2 for training data set and test data set, respectively (62 training data and 18 test data for class '2'). I trained a LDA classifier for '1' or '2' labeled observations using training data set, and estimated classification accuracy using test data set.
My question is that isn't there any bias problem when using different number of observations to train a classifier. In my case, the difference is quite big (20 vs 80). I am worrying about this issue. If so, please tell me a proper solution (e.g., a specific classifier) and reference paper to overcome this problem.