Is excluding cases with missing data fine for a predictive (not explanatory/descriptive) model?

Suppose I have some observations $(x_i,y_i)$ from some population—y is binary and x is a real number or vector of real numbers.

x    y
1    1
1    0
nan  1
nan  0
...
4    1


I would like to build a model to predict $y|x$

Assume all cases have $y$ and that I remove all, for example, 50% of the cases that have missing $x$.

We then have a predictive model $m$ for $y|\{x, \text{$x$not missing}\}$. This could be extremely useful. For example, suppose that we are trying to predict some disease; any time someone presents with $x$ not missing, we can use our model. It's too bad that we cannot say anything for those with $x$ missing, but we have generally improved the world for the subpopulation for which $x$ is collected.

If, however, we remove cases with missing $x$, and we then (1) decide to use $m$ on cases with $x$ missing or (2) make some statement about the whole population based on our estimated coefficient, this is clearly not correct. For (1), we would be using the model on a population different than the one for which it was trained. For (2), we would be ignoring the bias we might have introduced by removing cases with missing $x$.

I think it is for the second reason especially that removing missing data gets a bad rap. However, using $m$ as originally described if you are honest about the model not applying to cases with missing x seems to be a good idea (although not the $best$ idea, which would be to impute and have $m'$ for the whole population), and not to be kind of incorrect like (1) or (2). In this way, removing missing data does not introduce 'bias' it introduces constraints on the usability of the model?

• Think about whether the fact that the data is missing is significant or independent from the true (unreported) value of $x$. The fact that the data is missing might be an informative feature. Also: is the distribution of $y$ over those vectors with missing $x$ equal to the distribution of $y$ over vectors with other values of $x$, or over all vectors with non-missing values of $x$? – Open Season Feb 25 '18 at 13:32
• The informative feature part is reasonable, but not including an informative feature is not a bad thing, right? Suppose the distribution of $y$ over vectors with missing $x$ is completely different from the distribution of $y$ over vectors without missing $x$, eg, suppose the conditional distribution for missing $x$ is normal with variance 10 and the conditional distribution for present $x$ is normal with variance 1 – user0 Feb 25 '18 at 14:30
• @user86895 just to clarify some points: (1) do I understand correctly that $y$ is assumed binary, specifically an element of $\{0,1\}$? (2) is $x$ assumed to "live" in $\mathbb{N}$, $\mathbb{Z}$, $\mathbb{R}$ or something else? – Jim Mar 4 '18 at 16:47
• @Jim (1) yes (2) let X be a real number or vector of real numbers – user0 Mar 4 '18 at 18:21

The answer is - it depends.

The issue with missing data and leaving it out of your model completely is that it might affect the representativeness of your sampled data.

The kind of deletion you are referring to in your question is known as listwise deletion. This is where you exclude an observation completely because for at least one of your observed variables you do not have any data. Leaving out this data point might introduce bias into your model.

I will illustrate with an example: One common way this occurs is in surveys and is known as a non-response bias. Participants might not answer certain sensitive questions such as "Are you HIV positive?" etc. So in your model if you leave out all the unanswered questions you might think that there is a low incidence of HIV in your sample but this might simply be wrong.

In essence dealing with missing data comes down to understanding why the data point is missing at all: Is it random or is it systematic?

There are ways to find out if the missing data is systematic or not and the approach is similar to @David Dale's suggestion. You introduce new missing variables that indicate wether a data point is missing or not. We can then compare the mean likelihood of y for the 1’s and 0’s (missing and non-missing). If there is a significant difference in means, we have evidence that the data is not missing at random. In other words, there’s a pattern to the missingness. This is the first type of missing data pattern called missing not at random (MNAR).

If there is no significant difference in means, between our primary variable and y then we have evidence that the data is missing at random (MAR).

And finally, if there is no signifact difference in means between all our variables (primary and not) and the y then we have evidence that the data is missing completely at random (MCAR). This is the most desirable situation to be in.

If your data is MNAR then you will affect your model by listwise deletion of those values.

• Thank you for the HIV example. It seems however in that case that you are inferring the incidence of HIV, not predicting. Do you have an example where prediction is in play? Consider that HIV status is now a predictor variable for Y. When deploying the predictive model, we would likely see the same rate of non-response .. so, if the training data is like the testing data, aren’t we fine? – user0 Mar 1 '18 at 17:54

It Depends

Whether excluding cases with missing data is fine or not depends on a few factors. Whatever choice is made requires justification, so there is a bit or work to do with the data. The more ancillary information available to you, the better you can make a choice. Testing the randomness of the missing values is essential to guide your steps. This is because things become much more complex once you start to look at biased missingness.

I'll use an example of an arbitrary disease that requires some blood test and a questionaire.

Are values

Missing Completely At Random (MCAR): no pattern at all to the missingness. This may be that one subject is missing because the person got caught in traffic and didn't have time to complete all the tests, another because they faint at the sight of needles, another because the doctor's scrawl was completely uninterpretable. The reasons are unnconnected to each other and to whether the person has the disease or not.

Missing At Random (MAR). Has a relationship with what you have observed, but is not due to the underlying value. For example a questionnaire answer may be more likely to be missing in old subjects who have worse memory, but the likelihood of forgetting is not related what the answer would have been. Age may be a risk factor in the disease, so the forgetting is confounded with disease risk even through it is not caused by the disease.

Missing Not At Random (MNAR) there is a pattern to which variables are missing, the likelihood of being missing is biased by the actual value of the variable. For example, males may not wish to report impotence and choose to not report it. To diagnose this often needs additional data as you don’t know what the value should be. See https://www.theanalysisfactor.com/missing-data-mechanism/ which explains these and provides some advice on how to deal with the different types.

Possible Solutions

The five options I am aware off (1st 2 you have already discussed) are:

1. Ignore missing data in model. When you come to predict new samples that have missing data it will return and invalid result due to the missing value. You are right that this approach will limit your ability to make generalisations about the entire population unless you can prove that the missing data is MCAR. The model will fail when missing data is present, so you are not going to produce false predictions. Failure is useful information and not to be ignored. Mechanisms to handle the failure need to be appropriately designed. This will have theoretically have a neglible effect if data is truely MCAR as the missingness is unbiased. With MNAR there is definite bias in the missingness and this would be passed onto the model that ignores missingness. With MAR there is a risk that although the missingness is not due to the disease, it may be linked to factors that are confounded with disease and so may still bias your model.

2. Impute missing data into model. As a couple of commenters have pointed out the main value of imputation is maintaining statistical robustness of the dataset. It doesn't help the individual receiving a diagnosis if you replace their true value with (for a simple case) the mean of the population. Its aim is to preserve the overall statistical properties of the dataset for the general population. Imputation works best in a multivariate dataset where the relationship with other variables can help improve the imputation (specific patterns in variables T, U, V, W may be associated with more probable values of X) as these allow tailored estmation of the individual's value. If mechanisms causing missingness (MAR or MNAR) can be identified these should be used to guide the imputation. Validation of the model should include missing data to ensure the imputation and model is handled well in independent samples. As @JWH2006 points out the extent of missingness will also affect your choice of tool - the lower the proportion of observations the riskier imputation becomes, requiring more powerful methods.

3. If your variable types allow it, you can recode missing data to a new numeric level in X that does not otherwise arise. This implicitly handles the impact of missingness and will capture an element of any bias that exists. This should mitigate against missingness bias in all three categories.
4. A mashup of 2 and 3 alongside imputation you can create a Xmissing variable (e.g. 0 observed, 1 missing) that will be given its own model coefficient and confidence intervals to explicitly handle the missingness. Again, this should mitigate against bias in the missingness.
5. If there is the possibility of revisiting the data source (even for a partial sample of the missing and complete samples) and reacquiring it then this may be useful to better understand the mechanism of missingness to make a more robust procedure. Above I suggested some approaches can mitigate against bias, but they will never be guaranteed to eliminate its effect. If there is any way to go back and fill in the blanks it will improve your understanding of how the missingness arose and therefore give clues on how to handle it best.

Remember

It also matters if the data is truly missing as opposed to something like not-applicable, not-detected or a saturation event. Not detected events are usually recoded to half the detection limit. Saturation events usually set to the saturation threshold.

• (+1) Really nice discussion. It's worth pointing out that imputation does not confer much practical advantage over complete case analysis in most settings, except for boosting the power and effective sample size slightly. – AdamO Mar 1 '18 at 15:42
• Thank you--still confused. Can you give a scenario where you are bulding a predictive model and data are mcar and what you might do, mar and what you might do, and mnar and what you might do, and for each case explain why simply removing cases with missing x is not preferable? – user0 Mar 1 '18 at 20:40
• The above is a very nice explanation. I will also add that another way to deal with missing data is to mean code missing values. This preserves power compared to listwise deletion. That being said, if you are facing a large portion of your data having missing values, its probably time to bring out the big guns: multiple imputation and maximum likelihood estimation. – JWH2006 Mar 1 '18 at 20:45
• Thanks for the suggestions @AdamO, user86895 and JWH2006. I think they strengthen the answer quite a bit. If its still unclear which solutions work with which type of missingness I'll try and put something clearer together. – ReneBt Mar 2 '18 at 15:53
• @user86895 MCAR/MAR doesn't matter: multiply impute or weight the data or exclude to complete cases if you have adequate power. NMAR means your data are critically flawed and no fancy schmancy methods can save you from that. – AdamO Mar 2 '18 at 15:57

The answer is "Yes". If you both train and apply your model only on the samples with non-missing $x$, then it is totally OK. Your single heroic assumption here is that the process that makes $x$ be missing is the same for the training and test data, which is quite reasonable.

If you want to train and apply your model on any data (both with and without missing values), there is a preprocessing scheme with wich popular families of models (linear, neural network, decision trees and their ensembles) can work adequately. You impute missing $x$ with any value you want (e.g. 0), and at the same time you create a binary marker of missingnes for each column with potential missing values. For example, the table

x1    x2
0     1
2     nan
3     1
nan   0
nan   nan


turns into

x1    x2    x1m   x2m
0     1     0     0
2     0     0     1
3     1     0     0
0     0     1     0
0     0     1     1


In this case e.g. a linear model would find values for coefficients to x1m and x2m which give the best prediction, even if the values are missing not-at-random.

• Can you explain the assumption that the process that causes x to be missing is the same in the training and test data? – user0 Mar 1 '18 at 20:31
• @user86895 sure - I'll give an example of its violation. Imagine you are trying to improve a credit scoring model by including data from the Facebook profiles of your clients (e.g. number of followers). Then you parse the whole Facebook and augment your $X$ by the Facebook variables for the people you managed to match (by name, age, geography etc). Thus, in the training dataset "missing" means "could not match". When you apply your model, however, you explicitly ask users to provide a link to their Facebook account. Thus, in the "test" time "missing" means "the user didn't give the URL". – David Dale Mar 1 '18 at 20:39
• Gotcha, thanks. So this assumption is very unlikely to be violated when a single dataset is split into test and training. Does this relate to MCAR, MAR, MNAR categories in the other answers? – user0 Mar 1 '18 at 20:45
• It has no direct relation: even in the MCAR case the pattern of missingness might be different in the training and test data (e.g. 10% vs 50% of missing data), and in MAR and MNAR cases as well. – David Dale Mar 1 '18 at 20:49