2
$\begingroup$

Say I have regression problem which is to predict score between [0,1] and data is really skewed.

And train data set and test data set, # of regression answer label 0 is 10% and 1 is 90% (so I know prior distribution of 0s and 1s in both test and train dataset)

But what I really want to focus is model to predict 0 well. But in training data there is so many 1's so deep regression model tend to fit to train on 1, so even if answer is 0, it outputs 0.5.

Is it okay to train model to see 0 more (over-sampling) and will it solve the problem? (I'm expecting possible increased total loss but less error on label 0)

I'm using MLP model trained with SGD adam optimizer.

Since I know the prior distribution and real goal is to predict 0 with less error, I'm thinking of each batch to see 0 value examples with increased possibility. Could there be another smarter way?

Improve this question
$\endgroup$
3
  • $\begingroup$ oversampling will bring the sampling bias of its own. your problem is a typical setup in many kind of analysis at banks, e.g. default and prepay analysis of loans, where default or prepay rates are in single digit percentages. $\endgroup$
    – Aksakal
    Commented Feb 6, 2018 at 14:47
  • $\begingroup$ @Aksakal I'm thinking of binning examples of model can see [0,0.1] 10%, [0.1,0.2] 10% and so on. can you give some solution to these kind of problem? to tell you more about my data, ratio increases exponentially as value go up. so label [0,0.1] takes 1%, [0.1,0.2] takes 1.5% .... [0.9,1] takes 17% and so on. so its not that discrete as I mentioned on question, but it's really skewed though $\endgroup$
    – amityaffliction
    Commented Feb 8, 2018 at 4:55
  • $\begingroup$ I think you mean stratified sampling. Look up the term in internet. You'll sample more 0 than 1s, but then you must apply correct weights to get sensible results. $\endgroup$
    – Aksakal
    Commented Feb 8, 2018 at 5:50

1 Answer 1

Reset to default
3
$\begingroup$

The problem with oversampling from a smaller part of the dataset is potential overfitting: most of the data the network is going to see is from a small dataset. I say potential, because in some cases it really works the best (see for instance this discussion: How to improve F1 score with skewed classes?), it's just you should be aware of this.

An alternative way of dealing with highly skewed binary data is to use weighted cross-entropy that assigns bigger loss to the rare class error. For instance, in tensorflow it can be done with tf.nn.weighted_cross_entropy_with_logits:

This is like sigmoid_cross_entropy_with_logits() except that pos_weight, allows one to trade off recall and precision by up- or down-weighting the cost of a positive error relative to a negative error.

This is in many cases a better approach because the network learns from a bigger sample and is paying more attention to the minor class at the same time. But you should try both approaches or the mix.

I would also suggest using F1 score for final performance evaluation, because raw accuracy can be pretty misleading.

Improve this answer
$\endgroup$
5
  • $\begingroup$ My statement was that raw accuracy can be misleading. Do you disagree with that? $\endgroup$
    – Maxim
    Commented Feb 6, 2018 at 18:30
  • $\begingroup$ I didn't mean $F_1$ score alone will fix the issue, just that it's better than accuracy. If you mean that $F_1$ is not a good option too, I'll be interested to know why $\endgroup$
    – Maxim
    Commented Feb 6, 2018 at 19:15
  • 1
    $\begingroup$ +1, I had a minute to think about this, and recall my previous comments. $\endgroup$
    – Aksakal
    Commented Feb 7, 2018 at 18:48
  • $\begingroup$ I think accuracy alone ( MSE loss in regression case) can be inaccurate measure if distribution is so skewed. because by just guessing skewed label, it will gain good performance. so performance metric has to be redefined in my case. just focus on values between 0~0.5 which is what I want my model to predict $\endgroup$
    – amityaffliction
    Commented Feb 8, 2018 at 3:46
  • $\begingroup$ You certainly cannot use accuracy here. You must use some combination of precision and recall, such as $F_\beta$ measure, where $\beta$ is how much you value recall more than precision. $\endgroup$
    – Aksakal
    Commented Feb 8, 2018 at 5:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.