10
$\begingroup$

I wonder if someone could provide some insight into if an why imputation for missing data is better than simply building different models for cases with missing data. Especially in the case of [generalized] linear models (I can perhaps see in non-linear cases things are different)

Suppose we have the basic linear model:

$ Y = \beta_1 X_1 + \beta_2 X_2 + \beta_3 X_3 + \epsilon$

But our data set contains some records with $X_3$ missing. In the prediction data set where the model will be used there will also be cases of missing $X_3$. There seem to be two ways to proceed:

Multiple models

We could split the data into $X_3$ and non-$X_3$ cases and build a separate model for each. If we suppose that $X_3$ is closely related to $X_2$ then the missing data model can overweight $X_2$ to get the best two-predictor prediction. Also if the missing data cases are slightly different (due to the missing data mechanism) then it can incorporate that difference. On the down side, the two models are fitting on only a portion of the data each, and aren't "helping" each other out, so the fit might be poor on limited datasets.

Imputation

Regression multiple imputation would first fill in $X_3$ by building a model based on $X_1$ and $X_2$ and then randomly sampling to maintain the noise in the imputed data. Since this is again two models, will this not just end up being the same as the multiple model method above? If it is able to outperform - where does the gain come from? Is it just that the fit for $X_1$ is done on the entire set?

EDIT:

While Steffan's answer so far explains that fitting the complete case model on imputed data will outperform fitting on complete data, and it seems obvious the reverse is true, there is still some misunderstanding about the missing data forecasting.

If I have the above model, even fitted perfectly, it will in general be a terrible forecasting model if I just put zero in when predicting. Imagine, for example, that $X_2 = X_3+\eta$ then $X_2$ is completely useless ($\beta_2 = 0$) when $X_3$ is present, but would still be useful in the absence of $X_3$.

The key question I don't understand is: is it better to build two models, one using $(X_1, X_2)$ and one using $(X_1, X_2, X_3)$, or is it better to build a single (full) model and use imputation on the forecast datasets - or are these the same thing?

Bringing in Steffan's answer, it would appear that it is better to build the complete case model on an imputed training set, and conversely it is probably best to build the missing data model on the full data set with $X_3$ discarded. Is this second step any different from using an imputation model in the forecasting data?

$\endgroup$
4
$\begingroup$

I think the key here is understanding the missing data mechanism; or at least ruling some out. Building seperate models is akin to treating missing and non-missing groups as random samples. If missingness on X3 is related to X1 or X2 or some other unobserved variable, then your estimates will likely be biased in each model. Why not use multiple imputation on the development data set and use the combined coefficients on a multiply imputed prediction set? Average across the predictions and you should be good.

$\endgroup$
  • $\begingroup$ But if missingness is related to X1 or X2 then surely it is good to have two separate models - since they will be incorporating that information. That is to say, when in future I get a missing X3 I will know to be biased in the correct direction. $\endgroup$ – Corone Mar 1 '13 at 22:35
3
$\begingroup$

I assume that you are interested in obtaining unbiased estimates of the regression coefficients. The analysis of the complete cases yields unbiased estimates of your regression coefficients provided that the probability that X3 is missing does not depend on Y. This holds even if the missingness probability depends on X1 or X2, and for any type of regression analysis.

Of course, the estimates may be inefficient if the proportion of complete cases is small. In that case you could use multiple imputation of X3 given X2, X1 and Y to increase precision. See White and Carlin (2010) Stat Med for details.

$\endgroup$
  • $\begingroup$ Ah, so is imputation all about getting the coefficients right? The coefficients themselves are of no interest to me - I just want to maximise my predictive power on new data (which may also have missingness) $\endgroup$ – Corone Mar 2 '13 at 15:32
  • 1
    $\begingroup$ That's fine. To achieve maximal predictive power you would also want precise and unbiased estimates of the model coefficients. $\endgroup$ – Stef van Buuren Mar 2 '13 at 15:49
  • $\begingroup$ If I only use the complete cases, then I can't use that model for prediction when I have missing data, because the coefficients will generally be incorrect (for example if there is correlation between X2 and X3). I must therefore either impute X3 when making the prediction or build a second model in just X1 & X2. The question is if this results in different predictions and which is better? $\endgroup$ – Corone Mar 2 '13 at 17:58
  • $\begingroup$ Ah, I think I understand one point you are making: if I fit the model for complete cases prediction using imputation then that will improve the complete case forecast, vs fitting it with just the compete cases. The remaining question is what is best for the incomplete cases? $\endgroup$ – Corone Mar 2 '13 at 18:07
  • $\begingroup$ Suppose that beta_1 = beta_2 = 0 and beta_3 = 1. Using just X1 and X2 will predict a constant, whereas prediction using X3 will explain some of the variance of Y, and hence result in a lower the residual error. Thus, the imputed version produces better predictions. $\endgroup$ – Stef van Buuren Mar 3 '13 at 0:36
0
$\begingroup$

One study out of Harvard suggests multiple imputation with five forecasts of the missing data (here is refererence, http://m.circoutcomes.ahajournals.org/content/3/1/98.full ). Even then, I do recall comments that imputation models may still not produce cover intervals for the model parameters that do not include the true underlying values!

With that in mind, it appears best to use five simple naive models for the missing value (assuming not missing at random in the current discussion) that produce a good spread of values, so that cover intervals may, at least, contain the true parameters.

My experience in Sampling theory is that much resources are often spent in subsampling the non-response population which, at times, appears to be very different from the response population. As such, I would recommend a similar exercise in missing value regression at least once in the particular area of application. The relationships unrecovered in such an exploration of the missing data can be of historical value in constructing better missing data forecast models for the future.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.