Class labels in data partitions Suppose that one partitions the data to training/validation/test sets for further application of some classification algorithm, and it happens that training set doesn't contain all class labels that were present in the complete dataset, i.e. if say some records with label "x" appear only in validation set and not in the training.
Is it the valid partitioning? The above can have many consequences like confusion matrix would be no longer square, also during the algorithm we may evaluate an error and this would be affected by unseen labels in training set.
The second question is following: is it common for partitioning algorithms to take care about above issue and partition the data in the way that training set has all existing labels?
Thank you
 A: Pragmatically speaking, if you have a choice over how you partition the data (i.e. there are no time constraints or such), I would say you would be better off shuffling your dataset so as to distribute the classes more or less evenly over all partitions.
If this is not an option, I would simply drop any classes that you don't have any training data on: if the model has never seen a class, it cannot by any reasonable means make a valid prediction over this class. This is more of a data quality issue than a modelling issue.
As to your second question, I have seen it being done in the literature for unbalanced datasets where some class is heavily underrepresented (causing the same problem of not having information in some cuts of your dataset). Uniformly redistributing instances per class over the full dataset being the solution. 
A: If you consider stratified sampling, I think something similar could be done here, assuming your class is not so under represented that it does not even have 3 examples (one for training, one for testing and one for cross-validation).
Using a method like stratified sampling, you would make certain that each class is represented by randomly selecting instances of that class for each data set.
If you are running into this problem, you also might question whether you have enough data to train your algorithm well. Sometimes the correct answer is to get more data, assuming that is possible.
