1
$\begingroup$

I'm trying to create a regression model using data that are heavily skewed, almost like a half-normal distribution with a mode at zero. The lower scores are over-represented due to sampling technique - my outcome of interest (the x-axis in the histogram, the y-axis in the scatter plot) is only present in the minority of this population. I'm trying to use this model to predict outcomes in a different population which will not be skewed in the same manner. Ideally it would be great if I could drop some of the low scores in order to normalize the distribution somehow, but I can't find a validated approach to undersampling a continuous distribution. It would also be nice to be able to transform the data somehow, but that just doesn't seem to make sense because the low scores will still be overrepresented and would still drive the model.

Here's a histogram: enter image description here

So the zeros (and other low values) end up driving my model (see below). In the real world, I think that the relationship is only relevant when y>1.

enter image description here

Just for fun, I tried dropping all of the data with y<1, and it worked beautifully well for predicting outcomes in two independent validation datasets. Of course, I can't just drop half the data without some sort of strong statistical justification. If I don't drop any data, I can still predict outcomes in one of my validation datasets, but it doesn't quite work in the other one (p=0.1). I tried using various transforms (log, beta, log-beta, rank, square root, and Box-Cox), and they all give me the same issue - the histogram looks less skewed after transformation, but the effect is still driven by the over-represented low values.

Any suggestions for how I could approach this?

Some side notes that may or may not be relevant:

*I also have a handful of covariates which have a small impact on the results, but I didn't include them in the above info because I'm not sure if that's relevant here.

*I'll be using MATLAB to repeat the analysis across thousands of predictors and then use a permutation test (thousands of permutations) to confirm significance, so it would be nice if any proposed solution could be implemented in MATLAB. If that's not possible, that's OK - I can figure out a way to switch back and forth.

EDIT: I understand that I could undersample the over-represented part of the distribution. The problem is that my outcome is continuous, so I can't just "undersample the major variable" without arbitrarily choosing a cutoff between the "major" and "minor" variables. This would introduce bias, since I'm choosing a cutoff after looking at the data. I was hoping I could find a validated algorithm that automatically chooses which parts of the distribution to undersample.

$\endgroup$
0
$\begingroup$

Hi so if rephrase the question, its How to do modelling on an unbalanced data right.

There are multiple approchrs to these kind of problem.

  1. Under sample the Major variable to drop down total distribution inline with lowest
  2. Oversample lowest distribution to match it in line with highest available destribution.
  3. Synthetically induce data

But if you are dealing with bussiness sensitive data. I would never recommend inducing artifial data with either of the sampling method. The data we recived is untampered, the skewed destribution is a part of it, and you are suppose to analyse that of the data.

So how to proceed in these kind of situvations are

  1. Confirm the skewness with client, ask for his opinion like should we keep this skewness, wether thats the natural observed pattern or shall i induce artifical data to balance the data.

  2. Random sample the data, run two different algos one is sensitive towards outliers and skness and one not. Present both results.

  3. For fun you are always drop skeweed data. and have fun.

May the code be with you.

$\endgroup$
  • $\begingroup$ Thanks for the response! I'd definitely be happy to undersample the data, but the issue is that undersampling would require me to arbitrarily choose a boundary between the "major variable" and "minor variable." The variables lie on a continuous distribution, so I can't find an objective way to pick a cutoff between those two variables. This isn't for a business case, it's for medical research. Is there an algorithm that would help me undersample the distribution without having to pick an arbitrary cutoff point? $\endgroup$ – Shan Oct 22 '19 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.