Training data is imbalanced - but should my validation set also be? I have labelled data composed of 10000 positive examples, and 50000 negative examples, giving a total of 60000 examples. Obviously this data is imbalanced. 
Now let us say I want to create my validation set, and I want to use 10% of my data to do so. My question is as follows:
Should I make sure that my validation set is ALSO imbalanced, (as a nod to the true distribution of the training set), or should I make sure that my validation set is balanced? So for example, should my validation set be made from:


*

*10% positive example + 10% negative, giving 1000+ and 5000- examples. (This validation set reflects the original data imbalance). 

*Or should the validation set be made from say, 10% positive, giving 1000+, and (10/5 = 2%) negatives, also giving 1000- examples? 


(Same question for the test set). 
There seem to be plenty of methods on how to train with imbalanced data, but no where can I see to find best practices on whether or not my validation set should ALSO reflect the original imbalance, or not. Finally, I am NOT doing cross-validation, I will be using a single validation set, and a neural network. 
Thanks!
 A: Using a naturally unbalanced training and test data I ran into a scenario where my model appeared to be improving over time but was actually just memorizing the minority class samples in the training set and learning to always predict the majority class for everything else.
I diagnosed this problem by balancing my test data and observing that the accuracy and loss for the test data got worse with more epochs (which was not the case with the unbalanced test data).
Another way to diagnose issues like this would be to use metrics like sensitivity, specificity, or accuracy for a single class instead of looking at the overall accuracy/loss. This blog post goes into more detail on this and gives a sample implementation of these metrics in Keras http://www.deepideas.net/unbalanced-classes-machine-learning/
I think the right solution depends on the real world application of your model and how important the accuracy of minority classes is. E.g. if you are training something like imagenet and notice that it has low accuracy for it's "sea slug" class that is probably ok. But if you were training a model to predict if someone has cancer then the accuracy of the minority class would be very important.
A: It depends on what you're optimizing for: what is your target distribution, over which you would like to minimize the expected loss? If it is the same as the training distribution, then the validation data should follow the same distribution. If it is a different distribution (e.g. you would like your algorithm to perform well on balanced data), you actually want to minimize the expected loss over that distribution, and your validation set should correspondingly be sampled according to that distribution. This situtation is called "covariate shift", and there is a whole branch of research on what's called "covariate shift adaption," i.e., dealing with this problem. There is a book by Sugiyama / Kawanabe on the subject from 2012, called "Machine Learning in Non-Stationary Environments".
A: The point of the validation set is to select the epoch/iteration where the neural network is most likely to perform the best on the test set. Subsequently, it is preferable that the distribution of classes in the validation set reflects the distribution of classes in the test set, so that performance metrics on the validation set are a good approximation of the performance metrics on the test set.  In other words, the validation set should reflect the original data imbalance.
