1
$\begingroup$

My data set is divided into 80:20 train and test...i have performed 10 fold cross validation on the train data set and tested the 20 % dataset on each iteration ( so that test set is not touched while training). Finally i get the scores by averaging the scores in each iterations.
I have been trying classification on a 7 class problem. I have 8 sensors generated data .(8features). Every time the classifier misclassifies the last class. I tried decreasing the number of classes, still the last class got miss classified.
Finally i started decreasing the test set to increase training. I got good results (90%accuracy) when test data is only 8%.

Is any other way around or any scope of increasing scores without further decreasing the size of the test set? following are the snipets of the two casesenter image description here

$\endgroup$
4
  • $\begingroup$ How does the distribution of your classes look? Have you examined the class distributions in the entire sample and then in such smaller subsamples? $\endgroup$
    – deemel
    Oct 27, 2019 at 11:26
  • $\begingroup$ The classes are distributed almost equally, plus-minus one or two raws between the subsamples $\endgroup$
    – Sudipt
    Oct 27, 2019 at 12:38
  • $\begingroup$ How is the accuracy in both cases per class? It could of course be that due to a smaller training partition resulting from a bigger test partition, the fitted model generalizes worse than with a bigger training partition, but this depends strongly on the problem and the sample size we're talking about. On the other hand, it might as well be that your smaller test sets merely present a more 'accomodating' sample for your model. This is something we cannot really answer without speculation. You could investigate both angles and report what you find, maybe then the picture gets clearer. $\endgroup$
    – deemel
    Oct 27, 2019 at 13:40
  • $\begingroup$ i have added a confusion matrix and some scores as an image. Regarding your view on training and test partitions, i will check and surely report. thanks $\endgroup$
    – Sudipt
    Oct 27, 2019 at 13:58

2 Answers 2

1
$\begingroup$

Generally speaking, with more training data, the model will learn the underlying distribution of the real data better. Since a larger training set in your case improves the performance of on training and test set, you should get more data if you can. The performance on the test set might not be that reliable if your test set is small. In other words, your performance might be different if you change to another test set. That is one of the reasons that you perform cross validation. I suggest you also take a look at the CV accuracy.

You should also take a look at the class distribution in your training and test set. If there are only a few data points for the last class, the classifier will not able to learn well. You can upsample the minority class to make the classifier work better.

$\endgroup$
1
$\begingroup$

You have a pretty small dataset from the start. You need to consider 2 considerations:

  1. You want to maximize the training set since it increases the model's chance to learn a real result that generalizes.
  2. You want to maximize the test set since the bigger it is, the better it represents the model's ability to generalize. for example, in the extreme case of 1 sample in the test set, it obviously doesn't represent the model's performance out of sample.

I personally don't like going below 20% in small datasets, but it depends on the problem domain and your experience with the domain and data.

I would check how the data is split into train and test sets. because what might happen is that the train set is from a different distribution than the test set.

For example let's assume your sensors measure temperature, if you split by time, and your 80% are data from January-October, and the test is from November-December, then class 8 might occur in the train set when the temperature is above 20 degrees, but in the test set it's only below 20 degrees. Now that you add Nov. to the train set by decreasing the test set to 8%, you shift the training distribution to include samples where class 8 occurs below 20 degrees, thereby improving your accuracy on the test data.

So my guess is there's some kind of shift in the features representing the problematic class between test and train. You can look into data drift for further information.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.