1
$\begingroup$

I'm new to machine learning, and want to train a regression model based on 1000000 labelled samples (with, say, r features and 1 target).

On a histogram, I see that the targets approximately follow a normal distribution around 0, with standard deviation of about 0.75.

When I train the model, it tends to spit out predictions that are very close to zero. Specifically, almost all of the predictions are in the order of 0.01 (for example, I mostly see predictions like 0.0205 or -0.005).

What I suspect is going on: since so many targets are close to 0 (the mean is 0 and the standard deviation is small), it's a bit like in classification when we have an imbalanced dataset (too many instances of one class), so the model tends to just spit out predictions falling into in that class. Likewise, here, a lot of the targets are close to 0, so the model is outputting predictions near to 0 a lot of the time.

I don't think the problem has to do with scaling the features (I've tried that but the problem persists). I've tried different models (namely LSTM, linear regression, neural networks) and encounter the same problem.

Is my suspicion correct that this is an example of data imbalance, and if so, what can I do to overcome the problem?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Can you add some plots to illustrate? $\endgroup$
    – mkt
    Commented Sep 24, 2019 at 14:00
  • 1
    $\begingroup$ It's hard to understand what you mean, at least in part because the predictions depend on the covariance between your predictor and target. $\endgroup$
    – mkt
    Commented Sep 24, 2019 at 14:01
  • $\begingroup$ Learning curves (training vs cross-validation error for varying sample sizes) and plots of regularization strength vs errors/coefficient values etc. would be helpful. $\endgroup$
    – mzunhammer
    Commented Sep 24, 2019 at 14:50
  • $\begingroup$ If you define unbalancedness as having low entropy, any normal distribution is the opposite of being unbalanced. It has maximum entropy among a broad family of distributions with fixed standard deviation. Your model probably just has low predictive performance, leading to similar predictions. $\endgroup$
    – Michael M
    Commented Sep 24, 2019 at 17:16

1 Answer 1

Reset to default
2
$\begingroup$

No, this isn't imbalanced data and it isn't a classification problem.

You say that the target variable has a mean of 0, so it's natural that your regression will predict with a mean of 0. Why are so many of the predicted values too close to 0? Because your model just isn't that good. That is, your features aren't strongly related to your target.

What can you do? Well, you can conclude that these targets don't work well on this data set.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.