Missing data which simply cannot exist

Question

I have read 80% of missing data in a single variable and understand the approach for dealing with missing data which simply cannot exist for 1 variable.

I am trying to generalise this up to 2 or more variables, where different subsets of the sample will have different sets of variables in which data can exist. For simplicity, I started with 2 variables, of which there are four subsets of the population:

1. The subset where both variables can exist
2. The subset where the first variable can exist
3. The subset where the second variable can exist
4. The subset where neither variable can exist

My first thought was to simply create:

a) one dummy variable for whether variable 1 is missing or not missing

b) one dummy variable for whether variable 2 is missing or not missing

On paper, this seems to make sense: I obtain four different intercepts for the four subsets. However, I am worried about what happens when I impute values of 0 for the missing values for the 2 variables. More specifically, it is going to adjust the correlation between the 2 variables and therefore adjust the parameter estimates.

Is there a way to overcome this or wiil I simply need to run separate regressions for scenarios involving 2 or more variables where missing data simply cannot exist?

Answer 1

For sources, I'd suggest Missing Data Problem in Machine Learning, by Benjamin M. Merlin (Thesis), or Statistical Analysis with Missing Data, by Roderick and Rubin. What you are explaining in your answer is similar to the augmented model.

There are multiple approaches to deal with missing data, and as a quick look into it, the easiest ones to implement are

• Multiple Mean Imputation: This works if the data is Missing At Random, i.e., the fact that the data is missing as no impact on the dependent variable, outside the fact that you do not know the value of the variable. The idea is to assign the mean of the variable to missing values, and add a random error to avoid having a big spike in the distribution. Do this multiple times to get different training sets, and average the models. Another technique that works in this context is matrix reconstruction.
• Reduced Models: If the data is Not Missing At Random, you get into troubles with mean imputation. The idea of reduced models is to use multiple models for the different patterns of training data. In your case, you would train four models, $[x_1,x_2],[x_1,?],[?,x_2],[?,?]$. Note that when training $[?,x_2]$ you can use data that is valid from the $[x_1,x_2]$ pattern. The question of if you should will come from testing the resulting models. If you do not include other pattern and the dependent variable is dependent on the pattern, not including other data will improve the result. If the independent variable are missing based on their value, but the dependent variable does not depend on the "missingness", including more data should improve the results.
• Augmented Models: This is the model you finally choose to implement: Set unknown values to $0$ and complement the model with $[0,1]$ variable to represent missingness of features. This works for some models, such as linear regression, where $0$ has a special meaning. This would probably work less well using trees or SVMs, but should still work using NeuralNets. This provides a way to compensate the intercept for different pattern types. This should work well in your case, but when a lot of features have missing values, and there are a lot of patterns with similarities, two patterns may need compensation in different ways, and some additional work may be required to properly identify patterns that are similar.

Answer 2

So I sat down and worked this through. I thought I would share the answer - I struggled to find a source which worked this through and the couple of upvotes suggest interest in the solution.

Firstly, lets set up a scenario:

1 dependent variable: Y

2 variables: X1 & X2

X1 & X2 are correlated

Both have a subset of observations where the data cannot exist (going to call this 'missing' for simplicity)

This is such that there exists 2 or more observations which X1 & X2 both cannot exist.

There are four combinations:

X1 & X2 are both avaliable

X1 is missing, X2 isn't

X2 is missing, X1 isn't

X1 and X2 are both missing

We could run four separate regressions to take each combination in turn. This would give us valid estimates. Not including observations where they have a 'missing' observation is not bad in any capacity, as the observations simply do not hold any information.

However, we want to run all four within one regression. To do this, we need to be aware of two things:

1. We need to be able to generate four different intercepts depending on the four combinations above.
2. Changing 'missings' to 0s will adjust the coefficient estimates of X1 and X2. Therefore, we need to be able to regulate the coefficient estimates so they are correct for situations where at least one of X1 and X2 are present.

How?

To achieve both of the points above, 3 dummy variables (denoted by D1, D2 and D3 respectively) need to be used to avoid incorrect coefficients:

1. D1 takes the value 1 if X1 is missing AND X2 is not (0 otherwise)
2. D2 takes the value 1 if X2 is missing AND X1 is not (0 otherwise)
3. D3 takes the value 1 if X1 and X2 are both NOT missing (0 otherwise)

We then need to interact D1 with X2 & D2 with X1. In total we have 5 additional independent variables, alongside the intercept, X1 and X2.

D1, D2 and D3 regulate the intercept depending on whether

a) X1 and X2 are not missing,

b) X1 and X2 are missing,

c) X1 is missing,

d) X2 is missing

The interactions, D1*X2 and D2*X1 regulate the coefficients for X1 and X2 such that:

a) If X1 is missing AND X2 is not, D1*X2 regulates the coefficient on X2, akin to running a regression of Y on just the intercept and X2

b) If X2 is missing AND X1 is not, D2*X1 regulates the coefficient on X1, akin to running a regression of Y on just the intercept and X1.

The inclusion of the 5 additional variables allow you to achieve the coefficient and intercept estimates for all four combinations within one regression.

Scaling the approach up to 3 or more variables

As the number of variables with missing values increase, the number of additional variables needed also increases. In the scenario of 3 variables, for example, assuming all six combinations of missing data can exist for 2 or more observations (so, for example, four observation are missing X1 and X2 but have X3 present; six observations are missing X2 and X3 but have X1 present; five observations are missing X1, X2 and X3) a total of 11 additional variables are needed to create the coefficient and intercept estimates.

It is easy to see that the approach gets more unweildy in scenarios of 4+ variables.