3
$\begingroup$

Suppose that one partitions the data to training/validation/test sets for further application of some classification algorithm, and it happens that training set doesn't contain all class labels that were present in the complete dataset, i.e. if say some records with label "x" appear only in validation set and not in the training.

Is it the valid partitioning? The above can have many consequences like confusion matrix would be no longer square, also during the algorithm we may evaluate an error and this would be affected by unseen labels in training set.

The second question is following: is it common for partitioning algorithms to take care about above issue and partition the data in the way that training set has all existing labels?

Thank you

$\endgroup$
  • $\begingroup$ It suggests you have a small set of cases with that level or feature. Some statisticians advise doing derivation on the full sample and validation on bootstraps neo-samples. $\endgroup$ – DWin Dec 7 '13 at 2:14
  • $\begingroup$ If you do not have samples with a target label in training data set, for sure your model will not be able to predict it. Any variant of stratified sampling should be better than nothing $\endgroup$ – rapaio Apr 16 '14 at 9:25
  • $\begingroup$ As others have pointed out, if you can control the partitioning then stratified sampling the way to go as the classifier needs to see the data from all the classes. But in case you cannot but expect to get data later which includes the "new" class then you can consider some sophisticated methods developed for that purpose, for example this paper researchgate.net/profile/Robi_Polikar/publication/… $\endgroup$ – DataD'oh Jul 29 '17 at 9:02
1
$\begingroup$

If you consider stratified sampling, I think something similar could be done here, assuming your class is not so under represented that it does not even have 3 examples (one for training, one for testing and one for cross-validation).

Using a method like stratified sampling, you would make certain that each class is represented by randomly selecting instances of that class for each data set.

If you are running into this problem, you also might question whether you have enough data to train your algorithm well. Sometimes the correct answer is to get more data, assuming that is possible.

$\endgroup$
  • $\begingroup$ Why does one need to hold a separate copy for cross validation? As far as I know cross validation is done using training data, it is a waste to save separate copy only for this purpose. $\endgroup$ – rapaio Apr 16 '14 at 9:22
1
$\begingroup$

Pragmatically speaking, if you have a choice over how you partition the data (i.e. there are no time constraints or such), I would say you would be better off shuffling your dataset so as to distribute the classes more or less evenly over all partitions.

If this is not an option, I would simply drop any classes that you don't have any training data on: if the model has never seen a class, it cannot by any reasonable means make a valid prediction over this class. This is more of a data quality issue than a modelling issue.

As to your second question, I have seen it being done in the literature for unbalanced datasets where some class is heavily underrepresented (causing the same problem of not having information in some cuts of your dataset). Uniformly redistributing instances per class over the full dataset being the solution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.