I have been reading about regression models for missing data imputation and I'm quite confused regarding the following: if I can perfectly predict the value of feature f2 using feature f1, why would I use f2? If both were real, would this mean that they are highly correlated, even if in a non-linear fashion? As far as I know, this class of imputation methods tries to predict a feature using another set of features.


To give some technical/theoretical background, in section 3.2.1 of the book "Flexible Imputation of Missing Data":

For univariate $Y$ we write lowercase $y$ for $Y$ . Any predictors in the imputation model are collected in $X$. Symbol $X_{obs}$ indicates the subset of $n_1$ rows of $X$ for which $y$ is observed, and $X_{mis}$ is the complementing subset of n 0 rows of $X$ for which $y$ is missing. The vector containing the $n_1$ observed data in $y$ is denoted by $y_{obs}$ , and the vector of $n_0$ imputed values in $y$ is indicated by $\dot{y}$. This section reviews four different ways of creating imputations under the normal linear model. The four methods are:

  1. Predict. $\dot{y} = \hat{\beta_{0}} + X_{mis} \hat{\beta_{1}}$ , where $\hat{\beta_{0}}$ and $\hat{\beta_{1}}$ are least squares estimates calculated from the observed data. Section 1.3.4 named this regression imputation. In mice this method is available as "norm.predict".

  2. Predict + noise. $\dot{y} = \hat{\beta_{0}} + X_{mis} \hat{\beta_{1}} + \dot{\epsilon}$, where $\dot{\epsilon}$ is randomly drawn from the normal distribution as $\dot{\epsilon} \sim N(0, \hat{\sigma}^2)$. Section 1.3.5 named this stochastic regression imputation. In mice this method is available as "norm.nob".

  3. Bayesian multiple imputation. $\dot{y} = \dot{\beta_{0}} + X_{mis} \dot{\beta_{1}} + \dot{\epsilon}$, where $\dot{\epsilon} \sim N(0, \dot{\sigma}^2)$ and $\dot\beta_{0}$ , $\dot\beta_{1}$ and $\dot\sigma$ are random draws from their posterior distribution, given the data. Section 3.1.3 named this “predict + noise + parameters uncertainty.” The method is available as "norm".

  4. Bootstrap multiple imputation. $\dot{y} = \dot{\beta_{0}} + X_{mis} \dot{\beta_{1}} + \dot{\epsilon}$, and where $\dot{\epsilon} \sim N(0, \dot{\sigma}^2)$ and $\dot\beta_{0}$ , $\dot\beta_{1}$ and $\dot\sigma$ are the least squares estimates calculated from a bootstrap sample taken from the observed data. This is an alternative way to implement “predict + noise + parameters uncertainty.” The method is available as "norm.boot".


1 Answer 1


I believe that the core idea here is that you are filling in the missing bits, not deriving the entire value. If you are working with something more strictly decisive like a decision trees, it prevents the "missing" values from becoming a class of their own. If you can PERFECTLY predict f2 from only f1 in all cases then sure it might not be a valuable feature since it is too correlated, but if you are missing values from each on different samples then together you can produce a complete picture.

However, if you don't do the gap fill then a decision tree or a neuron applied to the value would get a completely different output / y_hat and you are at risk of weakening a good correlation between the features and the result, especially if it is a representative example on a minority class.

  • 2
    $\begingroup$ Thanks for the answer, Justin! I know that method is used to fill the dataset but it doesn't seem to make any sense to predict missing values of a feature $f_{2}$ with $f_{1}$ if both of them are independent. I must be missing something... $\endgroup$
    – echo66
    Feb 10, 2018 at 23:15
  • $\begingroup$ If f2 and f1 are completely independent then you are correct that it would be useless to predict f2 using f1. However, from the background you provided, the predictors (X) are multivariate and the idea is that y is somewhat correlated to the variables of X as a whole, which makes imputation possibly better than leaving the blanks as an outlier or discarding them. $\endgroup$ Feb 23, 2018 at 18:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.