3
$\begingroup$

It seems that because values are missing from a specific range of my target variable, my model performs poorly when predicting samples that are actually in that range. My target variable is income and my predictors are years_of_school and years_at_current_company so my model looks something like:

income ~ years_of_school + years_at_current_company

My training dataset has very few samples where income < $5000. Why such values are not in my training set is not clear to me, but they are not there and are not recoverable.

It turns out my model performs very poorly on new data that contains incomes below 5k. In fact it almost never predicts a value below 5k (which makes sense since I'm using a random forest). I tried a linear model and it performed worse than the random forest.

Is there a way to effectively deal with this problem? Would it help to oversample the few <5k samples that do exist?

$\endgroup$
  • 2
    $\begingroup$ This sounds more like truncation than the standard conception of missing data. I added the tag. You might read our wiki (linked). If you have some <5k's from your new data, you could add them to the training set. It's not the kind of statistics I do, but you could try some method of combining multiple models RF & linear for beyond the bounds. $\endgroup$ – gung - Reinstate Monica Aug 19 '19 at 20:14
  • $\begingroup$ @gung thanks for adding the tag, I'll be sure to check it out $\endgroup$ – George Aug 20 '19 at 3:46
4
$\begingroup$

This is not really a missing data problem. Your data is not missing, you simply believe that the proportions in your training data do not accurately reflect the population that they were drawn from.
You say

It turns out my model performs very poorly on new data that contains incomes below 5k. In fact it almost never predicts a value below 5k (which makes sense since I'm using a random forest).

Which I find a little confusing, if it is new data how do you know what the value for your target data is? Do you mean that on test data with values under 5k?

I would suggest that if your belief that your training data has less records under 5k than truly exist in the population is correct, than oversampling could be a method to solve this. However I would oversample prior to splitting your data (as opposed to the conventional oversampling only the training data) since your test data won't accuractely reflect the population otherwise.

$\endgroup$
  • $\begingroup$ Thanks for the response @astel. Clearly put/correct that the proportions of my training data don't accurately reflect the population. As for the 'new' values of my target data being under 5k, it's slightly more complicated. In reality, my data contains lagged variables that are sensitive to look ahead bias. Because of this I can't CV my data using a shuffled split, but instead need to use a Time Series split(bit.ly/31O2Ufx). The data gets more balanced/realistic as we get closer to the present day, but the early samples are almost completely missing the <5k samples. $\endgroup$ – George Aug 20 '19 at 3:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.