Learning model from skewed distribution Say I have regression problem which is to predict score between [0,1] and data is really skewed.
And train data set and test data set, # of regression answer label 0 is 10% and 1 is 90% (so I know prior distribution of 0s and 1s in both test and train dataset)
But what I really want to focus is model to predict 0 well. But in training data there is so many 1's so deep regression model tend to fit to train on 1, so even if answer is 0, it outputs 0.5.
Is it okay to train model to see 0 more (over-sampling) and will it solve the problem? (I'm expecting possible increased total loss but less error on label 0)
I'm using MLP model trained with SGD adam optimizer.
Since I know the prior distribution and real goal is to predict 0 with less error, I'm thinking of each batch to see 0 value examples with increased possibility. Could there be another smarter way?
 A: The problem with oversampling from a smaller part of the dataset is potential overfitting: most of the data the network is going to see is from a small dataset. I say potential, because in some cases it really works the best (see for instance this discussion: How to improve F1 score with skewed classes?), it's just you should be aware of this.
An alternative way of dealing with highly skewed binary data is to use weighted cross-entropy that assigns bigger loss to the rare class error. For instance, in tensorflow it can be done with tf.nn.weighted_cross_entropy_with_logits:

This is like sigmoid_cross_entropy_with_logits() except that pos_weight, allows one to trade off recall and precision by up- or down-weighting the cost of a positive error relative to a negative error.

This is in many cases a better approach because the network learns from a bigger sample and is paying more attention to the minor class at the same time. But you should try both approaches or the mix.
I would also suggest using F1 score for final performance evaluation, because raw accuracy can be pretty misleading.
