This is my first post, so let me know if I break any rules. I'm trying to learn about neural networks and I have implemented some toy examples; now I'm trying with a real data set.

My data set has about 70000 points $(x_i, y_i)$ where $x_i$ has about $30$ dimensions and $y_i$ is binary variable. I'm trying to train a feedforward multilayer NN with ReLu nonlinearities and softmax loss via stochastic gradient descent.

The problem I'm running into is that overall only ~1% of training cases has $y_i=1$, so my batches need to be pretty big to get any positive samples.

My question is generally what kind of things should I be paying attention to or tweak in order to deal with this kind of improbable event data.


  • 1
    $\begingroup$ Artificially increase the probability of drawing data points having $y_i = 1$? For instance, see en.wikipedia.org/wiki/Importance_sampling . I'm just throwing this out there for your consideration, not saying ti is a good thing for you to do in this situation. $\endgroup$ – Mark L. Stone Jul 31 '16 at 0:22

The problem you are facing is called class imbalance. Here are some strategies to mitigate the issue:

  • Oversampling: duplicating samples the label of which is the minority class (or equivalently increase the probability of drawing them when creating the minibatches). I believe this is the most common strategy.
  • Undersampling: duplicating samples the label of which is the majority class.
  • Changing the cost function to more heavily penalized mistakes done one the minority samples
  • Depending on your dataset, you could slightly perturb some of your minority samples to create new minority samples. For example, this is often done with images, which are easy to perturb (rotation, translation, etc.). It's often performed in some field such as computer vision.

You might also want to have a look at anomaly detection.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.