Missing data when actually predicting - Additional model legit?

Question

This is a theoretical question, but I have already stumbled upon this issue a few times:

My learning data are not complete, but I manage to handle the missing values. Now it's time for actually predicting (is there a name for a dataset where you are actually predicting?), and one pretty important predictor is missing very often, let's say in 50% of the observations. It is missing so often that I'm not feeling confident to somehow impute it at all.

Anyway, if i alter my model and leave that predictor out, model evaluation states that I still have an acceptable fit (also when only using training data, that came without the said predictor as well).

So for some observations I can use the model with all predictors but if that value is missing, I still have that fallback option.

Is it legitimate to combine the predictions of the two models and to calculate the fit of this combination? Or is there some reason not to do that? Are there any additional points to keep in mind when doing so?

EDIT 1 for clarification and in response to the first answer:

If you know the values of the response variable for the dataset you're trying to >predict for, then it could be called the 'test' or 'validation' dataset (because >you're testing how well your model fits to new data/validating whether or not it >works)._

If it is your response variable that is missing, then you can't use that data in >a test dataset!_

No, these are new observations with actually unknown response variables, not the training set. I trained and tested my model before, now I got fresh data, and i have to predict their unknown response. Is there a term for such a set? let's call it the 'prediction set'.

In these prediction sets, one important explanatory variable is missing very often. (It's missing because I get part of the data from another source and they don't list this variable)

I'd recommend that you test the fit of both models on the data for which you >have all the explanatory variables, and compare the fit on that data.

That's exactly what I did. The model with the additional predictor had a better fit.

So, that's why planned to use the model with the additional predictor in cases where this predictor also exists in the 'predicting-set'. The outcome for observations from the 'predicting-set' that miss this explanatory variable, will be estimated by the smaller model.

So back to my original question: Is this legitimate or for some reasons frowned upon? Because I actually did not see people using slightly different models based on the availability of predictors and later on join these results again. In my case, this way leads to better results then using the small model for all observations and ignoring the only partly missing predictor. Of course imputing the predictor would be an option too, but imo it's missing far too often for that.

I don't see why a pragmatic approach should be frowned upon: it will make it >harder to explain what you did, but to ignore a dataset just because it didn't >have one variable is wasteful/biasing your analysis in another way.

Thanks for the Feedback. Glad to hear a second opinion about this 'new' approach

With your validation dataset (note, much better to use a separate dataset from >the training dataset), you could compare your fit if you remove the value for >the important variable for 50% of the data points (at random), then: A. Impute >the missing variable and use model 1 (do this multiple times with random >samples) B. Use model 2 for the variables with missing data and model 1 for the >others. Do this multiple times with different random samples for the data points >with missing values. This would give you some evidence to justify your approach.

This sounds like a great way to actually compare the method 'model 1 + Imputations' against the method 'model 1 + model 2'

Answer 1

If you know the values of the response variable for the dataset you're trying to predict for, then it could be called the 'test' or 'validation' dataset (because you're testing how well your model fits to new data/validating whether or not it works).

If it is your response variable that is missing, then you can't use that data in a test dataset!

If it's an explanatory variable that's missing, then you still need to be careful - there could be something systematically different about the datapoints where the explanatory variable is missing (would be sensible to investigate whether this is the case), so you might get a worse or better fit on those particular points.

I'd recommend that you test the fit of both models on the data for which you have all the explanatory variables, and compare the fit on that data. Note that if you have insufficient test data that has all the explanatory variables to do this with, you may need to consider reassigning some training data to be test data (and retrain your model), assuming you have enough training data to sacrifice some in this way. If not, you may need more data to proceed (you can look into bootstrapping, but ultimately there's no replacement for genuine replication).

Also report, separately, the fit of the second model on your data that had missing values.

If the fit is as good with the second model as the first, the second model may be preferable for actual practical use, because it's likely to be applicable to a greater proportion of future cases where you need to use it for prediction (assuming your test dataset is representative), and it's less confusing to just have one model to consider.

If the fit with the second model is worse (on the test dataset with all the explanatory variables), then it's still pragmatic to use that if you do have missing data, but you'll need to factor in that you have greater uncertainty in your predictions when using that model.

Edit: Additional comments in response to updated information.

If you are using a model to predict for new data, a model that requires data you don't have is not of much use! So I think in these circumstances it would be entirely reasonable to use a second model to predict for those datapoints, and your better model for the data points where you have the data for it. However, I think if you do this, you should report the greater uncertainty in the values predicted using model 2, and also be careful to take this into account if you are then doing anything further with those predicted values, such as using them as an input to yet another model.

I don't see why a pragmatic approach should be frowned upon: it will make it harder to explain what you did, but to ignore a dataset just because it didn't have one variable is wasteful/biasing your analysis in another way. On the other hand, if model 2 is so uncertain as to not be very useful, then you might want to consider not using that data.

With your validation dataset (note, much better to use a separate dataset from the training dataset), you could compare your fit if you remove the value for the important variable for 50% of the data points (at random), then: A. Impute the missing variable and use model 1. or (to compare with A.): B. Use model 2 for the variables with missing data and model 1 for the others. Do this multiple times with different random samples for the data points with missing values. This would give you some evidence to justify your approach.