To describe my problem. I'm predicting the price of an item depending on some text and other features in an ad. The training data contains a bunch of cheap items, some medium price items and few expensive items.

I log1p the prices but even then it's no surprise that when the real price of an item is low my model gives good results. However when the item is expensive the prediction is very bad.

Is it possible to balance my dataset in a way? I tried oversampling with imblearn but even tiny amounts causes lots of overfit due to the text data I think.

  • 1
    $\begingroup$ your post seems to be focusing on the point estimate from the regression have you also considered the variance or standard deviation of the point estimates? I suspect the predictions on the expensive items have a high variance which is exactly what you would expect and want. $\endgroup$ Jan 6, 2018 at 14:17
  • 1
    $\begingroup$ Hard to answer a question about data without seeing the data. Include it, please. $\endgroup$
    – Carl
    Jan 6, 2018 at 16:16
  • $\begingroup$ Possible duplicate of When is unbalanced data really a problem in Machine Learning? $\endgroup$ Dec 13, 2018 at 23:56

1 Answer 1


Your model is kind of behaving how it should. It knows that values are likely to be at the low end of the range of observed values. Can you fault the model for making predictions that reflect this fact? More mathematically, the prior probability, which often comes up when discussing "classification" models but which also makes sense for regression models, of high values is low, and the posterior probability reflects this. Unless you have a strong signal from your features that such a value is likely, Bayes' theorem gives a low posterior probability to that region, as the posterior probability is dragged down by the low prior probability.

(Since Bayes' theorem is a mathematical theorem and not an opinion about how statistical modeling should be performed, this applies to a frequentist regression; you do not have to use Bayesian modeling for this rationale to apply.)

Most likely, you simply lack the features that predict the extreme events. You either do not have a predictive variable or you have not figured out the characteristics of your existing variables that can allow you to make such predictions (such as interactions and/or polynomial terms as two possibilities). Unfortunately, it is not a given that you will be able to reliably predict such events, and part of your job as a statistician or data scientist (analyst, machine learning engineer, etc) is to produce models that reflect this limitation of your data. Sure, your boss wants perfect performance and the ability to catch those values, but there is a phrase about who wants ice water, too. Just because someone wants something does not entitle them to it.


Being explicit about how Bayes’ theorem plays a role here, consider that you, at some level level, want to estimate something like $P(Y>\ell\vert X=x)$, where $\ell$ is some threshold for being considered a “large” value, and $x$ is your feature vector.

$$ P(Y>\ell\vert X=x)=\dfrac{ P(X=x\vert Y>\ell)P(Y>\ell) }{ P(X=x) } $$

Since $P(Y>\ell)$ is small (there are few large values), the rest of the fraction needs to give a gigantic value to give a high posterior probability. If you do not have the features screaming out that the outcome will be large, this will not occur, hence my suspicion that you either lack a variable that is highly predictive of such a result or that you have not used your existing variables in the right way.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.