How do to deal with data missing NOT at random?

Question

It seems that because values are missing from a specific range of my target variable, my model performs poorly when predicting samples that are actually in that range. My target variable is income and my predictors are years_of_school and years_at_current_company so my model looks something like:

income ~ years_of_school + years_at_current_company

My training dataset has very few samples where income < $5000. Why such values are not in my training set is not clear to me, but they are not there and are not recoverable.

It turns out my model performs very poorly on new data that contains incomes below 5k. In fact it almost never predicts a value below 5k (which makes sense since I'm using a random forest). I tried a linear model and it performed worse than the random forest.

Is there a way to effectively deal with this problem? Would it help to oversample the few <5k samples that do exist?

Answer 1

This is not really a missing data problem. Your data is not missing, you simply believe that the proportions in your training data do not accurately reflect the population that they were drawn from.
You say

It turns out my model performs very poorly on new data that contains incomes below 5k. In fact it almost never predicts a value below 5k (which makes sense since I'm using a random forest).

Which I find a little confusing, if it is new data how do you know what the value for your target data is? Do you mean that on test data with values under 5k?

I would suggest that if your belief that your training data has less records under 5k than truly exist in the population is correct, than oversampling could be a method to solve this. However I would oversample prior to splitting your data (as opposed to the conventional oversampling only the training data) since your test data won't accuractely reflect the population otherwise.