# Sampling for Imbalanced Data in Regression

There have been good questions on handling imbalanced data in the classification context, but I am wondering what people do to sample for regression.

Say the problem domain is very sensitive to the sign but only somewhat sensitive to the magnitude of the target. However the magnitude is important enough that the model should be regression (continuous target) not classification (positive vs. negative classes). And say in this problem domain that any set of training data will have 10x more negative than positive targets.

In this scenario, I might oversample the positive-target examples to match the count of negative-target examples, and then train a model to differentiate the two cases. Obviously the training approach does badly on imbalanced data, so I need to do sampling of some sort. What would be a decent way to "undo" this oversampling when making predictions? Perhaps translating by the (negative) mean or median of the target of the natural training data?

Imbalance is not necessarily a problem, but how you get there can be. It is unsound to base your sampling strategy on the target variable. Because this variable incorporates the randomness in your regression model, if you sample based on this you will have big problems doing any kind of inference. I doubt it is possible to "undo" those problems.

You can legitimately over- or under-sample based on the predictor variables. In this case, provided you carefully check that the model assumptions seem valid (eg homoscedasticity one that springs to mind as important in this situation, if you have an "ordinary" regression with the usuals assumptions), I don't think you need to undo the oversampling when predicting. Your case would now be similar to an analyst who has designed an experiment explicitly to have a balanced range of the predictor variables.

Edit - addition - expansion on why it is bad to sample based on Y

In fitting the standard regression model $y=Xb+e$ the $e$ is expected to be normally distributed, have a mean of zero, and be independent and identically distributed. If you choose your sample based on the value of the y (which includes a contribution of $e$ as well as of $Xb$) the e will no longer have a mean of zero or be identically distributed. For example, low values of y which might include very low values of e might be less likely to be selected. This ruins any inference based on the usual means of fitting such models. Corrections can be made similar to those made in econometrics for fitting truncated models, but they are a pain and require additional assumptions, and should only be employed whenm there is no alternative.

Consider the extreme illustration below. If you truncate your data at an arbitrary value for the response variable, you introduce very significant biases. If you truncate it for an explanatory variable, there is not necessarily a problem. You see that the green line, based on a subset chosen because of their predictor values, is very close to the true fitted line; this cannot be said of the blue line, based only on the blue points.

This extends to the less severe case of under or oversampling (because truncation can be seen as undersampling taken to its logical extreme).

# generate data
x <- rnorm(100)
y <- 3 + 2*x + rnorm(100)

# demonstrate
plot(x,y, bty="l")
abline(v=0, col="grey70")
abline(h=4, col="grey70")
abline(3,2, col=1)
abline(lm(y~x), col=2)
abline(lm(y[x>0] ~ x[x>0]), col=3)
abline(lm(y[y>4] ~ x[y>4]), col=4)
points(x[y>4], y[y>4], pch=19, col=4)
points(x[x>0], y[x>0], pch=1, cex=1.5, col=3)
legend(-2.5,8, legend=c("True line", "Fitted - all data", "Fitted - subset based on x",
"Fitted - subset based on y"), lty=1, col=1:4, bty="n")

• Thanks for the answer, Peter. Would you please elaborate on what you mean by "Because this variable incorporates the randomness in your regression model"? The target is an observable in the environment, so do you means measurement error? – someben Jun 10 '12 at 1:56
• Here is a paper from an NYU prof named Foster Provost on the issue: pages.stern.nyu.edu/~fprovost/Papers/skew.PDF In my case, I am doing regression with imbalanced data and not classification. Hence my question. – someben Jun 10 '12 at 2:17
• @someben - I've elaborated and added an example. It is well described in the regression literature that you cannot sample based on the dependent variable. This should apply to other models too. A sample that is "unbalanced" is a different sort of thing and is not a problem; unless you have deliberately created it by an unjustifiable sampling strategy. It's not the balance or lack of it that is the problem, but how you get your data. – Peter Ellis Jun 10 '12 at 2:57
• @someben, no I don't think it makes any difference. The issue is more fundamental than that. – Peter Ellis Jun 10 '12 at 4:17
• Nice example! Your graph reminds me of a paper by Richard Berk (1983) on Sample selection bias. Also to note, you can "undo" those problems if you explicitly know the sample selection mechanism, and there are a series of econometric models built around that notion (such as the tobit model, or the work of James Heckman). – Andy W Jun 10 '12 at 13:24

I think what might help you given your problem is Synthetic Minority Over-Sampling Technique for Regression (SMOTER). There is some research on this topic. However, it remains less explored than its classification counterpart, as you have likely encountered.

I might suggest the paper cited below (and the conference it was presented at http://proceedings.mlr.press/v74/) depending on how interested you are in understanding it from a research perspective. I really appreciated the introduction of Gaussian noise in generating the synthetic observations.

If you're more interested in a practical solution, the first author has an R implementation available on her Github page. https://github.com/paobranco/SMOGN-LIDTA17

If Python is more of your persuasion, I recently distributed an entirely Pythonic implementation of the SMOGN algorithm that is now available and currently being unit tested. https://github.com/nickkunz/smogn

I hope this helped!

Branco, P., Torgo, L., Ribeiro, R. (2017). "SMOGN: A Pre-Processing Approach for Imbalanced Regression". Proceedings of Machine Learning Research, 74:36-50.http://proceedings.mlr.press/v74/branco17a/branco17a.pdf.

There is actually a field of research that deals particularly with this question and has developed practically feasible solutions. It is called Covariate Shift Adaptation and has been popularized by a series of highly cited papers by Sugiyama et al., starting around 2007 (I believe). There is also a whole book devoted to this subject by Sugiyama / Kawanabe from 2012, called "Machine Learning in Non-Stationary Environments".

I will try to give a very brief summary of the main idea. Suppose your training data are drawn from a distribution $$p_{\text{train}}(x)$$, but you would like the model to perform well on data drawn from another distribution $$p_{\text{target}}(x)$$. This is what's called "covariate shift". Then, instead of minimizing the expected loss over the training distribution

$$\theta^* = \arg \min_\theta E[\ell(x, \theta)]_{p_{\text{train}}} = \arg \min_\theta \frac{1}{N}\sum_{i=1}^N \ell(x_i, \theta)$$

as one would usually do, one minimizes the expected loss over the target distribution:

$$\theta^* = \arg \min_\theta E[\ell(x, \theta)]_{p_{\text{target}}} \\ = \arg \min_\theta E\left[\frac{p_{\text{target}}(x)}{p_{\text{train}}(x)}\ell(x, \theta)\right]_{p_{\text{train}}} \\ = \arg \min_\theta \frac{1}{N}\sum_{i=1}^N \underbrace{\frac{p_{\text{target}}(x_i)}{p_{\text{train}}(x_i)}}_{=w_i} \ell(x_i, \theta)$$

In practice, this amounts to simply weighting individual samples by their importance $$w_i$$. The key to practically implementing this is an efficient method for estimating the importance, which is generally nontrivial. This is one of the main topic of papers on this subject, and many methods can be found in the literature (keyword "Direct importance estimation").

So, finally getting back to the original question, this method consists not of resampling or creating artificial samples, but of simply weighting the existing samples in an appropriate manner.

Interestingly, none of the papers I cited below cites this branch of research. Possibly, because the authors are/were unaware of it?

This is not an attempt at providing a practical solution to your problem, but I just did a bit of research on dealing with imbalanced datasets in regression problems and wanted to share my results:

It is a question of whether you are doing causal analysis or prediction. Read http://m.statisticalhorizons.com/?ref=http%3A%2F%2Ft.co%2F54MWZrVAyX&url=http%3A%2F%2Fstatisticalhorizons.com%2Fprediction-vs-causation-in-regression-analysis&width=640

Resampling on the target variable for training for the purposes of prediction works as long as one tests on an non-resampled hold out sample. The final performance chart must be based solely on the hold out. For most accuracy in the determination of the predictability of the model, cross validation techniques should be employed.

You "undo" by the final analysis of the regression model and on the imbalanced data set.

• link isn't working now – Pake Jul 30 at 19:23

first of all, 1:10 ration is not bad at all. there are simple way of undoing sampling-

1) for classification problem, If you have sub-sampled any negative class by 10. the resulting probability is 10 times more what is should be. you can simple divide resulting probability by 10.(known as model re calibration)

2) Facebook also sub-samples(for click prediction in logistic regression) and do a negative down sampling. recalibartion is done by simple formula p/(p+(1-p)/w); where p is prediction in downsampling,n w is negative down sampling rate.

• I don't think this it's that simple, Arpit. Many nonlinear algos don't see enough instances of undersampled class and become skewed towards oversampled one, and due to their nonlinearity you won't have means to fix that. – Anatoly Alekseev Aug 22 '19 at 23:09

A recent technique proposed for dealing with imbalanced distribution in regression is the Weighted Relevance-based Combination Strategy (WERCS). Branco et al. (2019) https://doi.org/10.1016/j.neucom.2018.11.100.

The author has reproducible R implementation of the method at GitHub here: https://github.com/paobranco/Pre-processingApproachesImbalanceRegression.

The authors note this WERCS method does not involve the generation of synthetic samples and yet show it often outperforms popular methods such as random over/under-sampling, Gaussian Noise, and SMOTER.