In a book I'm reading I've come across the following quote:
Accuracy on the test set is a good performance measure only when there is a relatively uniform distribution of target categories in the training set.
If the real world categories are distributed 10% A and 90% B, then should the training categories also be 10% A and 90% B, or should they be 50% A and 50% B?
