Impute Nulls in Train but remove Nulls in Test I have a continuous variable, call it A, with missing data. In my model, I have created a dummy variable, call it B, where 1 indicates missing in A and 0 otherwise. I impute missing values in A with the mean and include both variables A and B in the model. The test set also has missing values for A. My question: does it make sense to remove the missing values in the test set?
Update: This is regression problem. The response variable follows a gamma distribution.
 A: What would you do at prediction time? Say your model is deployed and it gets data with missing values (is it possible?), what do you do? Can you fall back to something else instead of making predictions? If you need to make a prediction, then you would either need to fill in the missing values or have a model using only the features that are guaranteed to have no missing values--that's the model you need to test. Your testing strategy should reflect how you are going to actually use your model. Your testing data should be similar to the actual data the model would see in the wild. Otherwise, you risk that the performance you measure is not the one you care about.
A: It is not quite clear, which type of model you are working with (regression or classification). I would try to describe possible solutions depending on data and model.
You have (surprisingly) two alternatives:

*

*Drop missing values both from train and test to keep this 2
subsamples consistent. I personally would have done that only if
missing values represent very small % of observations and if
observations with NaNs are not important (consider classification
problem with imbalanced classes, where underrepresented class
contains many NaNs, in such case we can't simply drop NaNs).

*Keep nulls in train and test and encode missing values with separate variable or fill them with mean/median/some sample from
non-null data with random noise/... I don't know much about your
problem, but if I were you, I would keep NaNs in test because:

*

*You have already fitted model on sample with NaNs

*If NaNs will appear in new data used for prediction with fitted model, your model should be ready for that)

*Generally, it is better when train and test are sampled from the same distribution (it gives a chance that model fitted on sample can work normally on test). Thus, if train contains nulls, it could be rational to keep them in test.



