This is a theoretical question, but I have already stumbled upon this issue a few times:
My learning data are not complete, but I manage to handle the missing values. Now it's time for actually predicting (is there a name for a dataset where you are actually predicting?), and one pretty important predictor is missing very often, let's say in 50% of the observations. It is missing so often that I'm not feeling confident to somehow impute it at all.
Anyway, if i alter my model and leave that predictor out, model evaluation states that I still have an acceptable fit (also when only using training data, that came without the said predictor as well).
So for some observations I can use the model with all predictors but if that value is missing, I still have that fallback option.
Is it legitimate to combine the predictions of the two models and to calculate the fit of this combination? Or is there some reason not to do that? Are there any additional points to keep in mind when doing so?
EDIT 1 for clarification and in response to the first answer:
If you know the values of the response variable for the dataset you're trying to >predict for, then it could be called the 'test' or 'validation' dataset (because >you're testing how well your model fits to new data/validating whether or not it >works)._
If it is your response variable that is missing, then you can't use that data in >a test dataset!_
No, these are new observations with actually unknown response variables, not the training set. I trained and tested my model before, now I got fresh data, and i have to predict their unknown response. Is there a term for such a set? let's call it the 'prediction set'.
In these prediction sets, one important explanatory variable is missing very often. (It's missing because I get part of the data from another source and they don't list this variable)
I'd recommend that you test the fit of both models on the data for which you >have all the explanatory variables, and compare the fit on that data.
That's exactly what I did. The model with the additional predictor had a better fit.
So, that's why planned to use the model with the additional predictor in cases where this predictor also exists in the 'predicting-set'. The outcome for observations from the 'predicting-set' that miss this explanatory variable, will be estimated by the smaller model.
So back to my original question: Is this legitimate or for some reasons frowned upon? Because I actually did not see people using slightly different models based on the availability of predictors and later on join these results again. In my case, this way leads to better results then using the small model for all observations and ignoring the only partly missing predictor. Of course imputing the predictor would be an option too, but imo it's missing far too often for that.
I don't see why a pragmatic approach should be frowned upon: it will make it >harder to explain what you did, but to ignore a dataset just because it didn't >have one variable is wasteful/biasing your analysis in another way.
Thanks for the Feedback. Glad to hear a second opinion about this 'new' approach
With your validation dataset (note, much better to use a separate dataset from >the training dataset), you could compare your fit if you remove the value for >the important variable for 50% of the data points (at random), then: A. Impute >the missing variable and use model 1 (do this multiple times with random >samples) B. Use model 2 for the variables with missing data and model 1 for the >others. Do this multiple times with different random samples for the data points >with missing values. This would give you some evidence to justify your approach.
This sounds like a great way to actually compare the method 'model 1 + Imputations' against the method 'model 1 + model 2'