How does size of test set affect the performance of a model?

Question

My data set is divided into 80:20 train and test...i have performed 10 fold cross validation on the train data set and tested the 20 % dataset on each iteration ( so that test set is not touched while training). Finally i get the scores by averaging the scores in each iterations.
I have been trying classification on a 7 class problem. I have 8 sensors generated data .(8features). Every time the classifier misclassifies the last class. I tried decreasing the number of classes, still the last class got miss classified.
Finally i started decreasing the test set to increase training. I got good results (90%accuracy) when test data is only 8%.

Is any other way around or any scope of increasing scores without further decreasing the size of the test set? following are the snipets of the two cases

Answer 1

Generally speaking, with more training data, the model will learn the underlying distribution of the real data better. Since a larger training set in your case improves the performance of on training and test set, you should get more data if you can. The performance on the test set might not be that reliable if your test set is small. In other words, your performance might be different if you change to another test set. That is one of the reasons that you perform cross validation. I suggest you also take a look at the CV accuracy.

You should also take a look at the class distribution in your training and test set. If there are only a few data points for the last class, the classifier will not able to learn well. You can upsample the minority class to make the classifier work better.

Answer 2

You have a pretty small dataset from the start. You need to consider 2 considerations:

1. You want to maximize the training set since it increases the model's chance to learn a real result that generalizes.
2. You want to maximize the test set since the bigger it is, the better it represents the model's ability to generalize. for example, in the extreme case of 1 sample in the test set, it obviously doesn't represent the model's performance out of sample.

I personally don't like going below 20% in small datasets, but it depends on the problem domain and your experience with the domain and data.

I would check how the data is split into train and test sets. because what might happen is that the train set is from a different distribution than the test set.

For example let's assume your sensors measure temperature, if you split by time, and your 80% are data from January-October, and the test is from November-December, then class 8 might occur in the train set when the temperature is above 20 degrees, but in the test set it's only below 20 degrees. Now that you add Nov. to the train set by decreasing the test set to 8%, you shift the training distribution to include samples where class 8 occurs below 20 degrees, thereby improving your accuracy on the test data.

So my guess is there's some kind of shift in the features representing the problematic class between test and train. You can look into data drift for further information.