Imputation of missing parameter value by regressing the other parameters

Question

I have a question on imputation of missing values. A support vector machine does not work with missing values, and in practice i thus hence na.omit the rows / cases with missing values on any of the parameters (e.g. X1 to X10). From a theoretical perspective, can one improve the overall classification performance if one imputes the missing values for the values of one missing parameter (lets say X1) through predicting the missing values of this parameter based on e.g. a multiple regression, SVM on the remaining parameters (e.g. X2-X10)? Asking the question, because the performance in practice is often disappointing, and hence wondering whether there is a theoretical reason why my approach per above might be nonsense / not useful.

In a similar vein - as a decision tree does work with missing values, and hence should use the full information of the existing dataset, am i right to assume that in this case from a theoretical perspective there is definitely no value-add / benefit in imputing missing values by using the other input parameters?

Answer 1

Other than the issue of what mechanism caused the missing data in the first place (missing at random, missing not at random, missing completely at random) the problem with using imputation by regression is that it does not allow for uncertainty in the imputed values - and hence standard errors will be too small.

You could extend the idea by allowing uncertainty by adding "noise" to the predictions, say based on the residual variance of the regression and doing this several times to obtain a distribution of plausible values. You could further extend the idea by allowing for uncertainty in the regression parameters too - eg. using a Bayesian regression model.

What you then have is several completed datasets, each one containing different plausible values for those that were missing. This is the basis for "multiple imputation" using chained equations, and this is essentially how particular methods in the MICE package are implemented. Models are then run on each set of completed data and the results pooled according to well-established rules (Rubin's rules).

A further issue to consider is that there should be a correspondence between the imputation and the model of interest (this is known as "congeniality" in the multiple imputation literature).

There is a huge literature on this. A great starting point is the classic book by Little and Rubin:

Statistical Analysis with Missing Data, 2nd Edition (Wiley Series in Probability and Statistics) 2002

Answer 2

Imputing missing values by regression is based on the implicit assumption that the true value (the one that is missing) is indeed dependent on the remaining variables in the way used for imputation. If this is in fact the case, this way of imputation will be good, whereas if what you get is rather different from the true values, this way of imputation will be rather misleading. The problem with missing value imputation in general is that it is not observable how good this assumption actually is, because the missing values are missing. So theoretically this question cannot be answered either way (same holds for decision trees). Whether this is good or not in principle depends on something you cannot observe (although of course you can use cross-validation etc. to check whether it works). You can also check whether the estimated regression is any good for the non-missing values, although even if it is, it may still be different for the true values of the missing entries, unless you have them "missing at random" (which is typically not known and not observable either).