I have a dataset with both continuous, categorical, and ordinal variables (roughly 1k of them), over 100 Million rows, and a lot of missing values. I am aware of various methods for imputation, but I'm worried these will either be overly biased (mean or mode imputation, etc.) or unfeasible given the size of the data (MI, other modeling imputations).

I am planning on using LASSO (logistic regression with an L1 penalty) to predict a binary response variable.

I have an idea that I would love to hear people's thoughts on: just add a one-hot encoded variable for each variable with any missing values that would indicate a missing value for that variable.

For the categorical variables, I will be one-hot encoding them anyway, so this seems reasonable. For the continuous and ordinal variables, I would replace the missing values with zero, and then put a 1 in corresponding one-hot "missing" indicator variable.

To be more clear, this:

contvar1  contvar2
45        3
100       NA
73        5
NA        7
62        NA
NA        4
94        6

would become this:

contvar1  contvar1_NA  contvar2  contvar2_NA
45        0            3         0
100       0            0         1
73        0            5         0
0         1            7         0
62        0            0         1
0         1            4         0
94        0            6         0

The idea is that, for instance in row 4 above, the coefficient for contvar1 would be multiplied by 0 and therefore not affect the prediction, but then a separate coefficient for contvar1_NA would be learned and that would offset the prediction by the appropriate amount.

This intuitively makes sense to me, but I can't find much written about an approach like this (other than this post, which calls it "often useful") so I wanted to see what others thought. It feels to me sort of analogous to what XGBoost does (some discussion here) of learning the appropriate split for a missing value, just like it would for any other value of that variable.

As a sidenote, I'm not 100% tied to using LASSO, but the size of my data makes it not ideal to use something like XGBoost because of both training and prediction time. LASSO is much much faster, especially for predictions (which I have to do a lot of).

Any thoughts are much-appreciated.

  • $\begingroup$ Adding an indicator variable for just 1 observation ("one-hot" encoding) is essentially the same as dropping that observation. It leads to fitting just a specific mean for that variable, so its residual will be zero, and it will not influence the fitting of the other parameters. So, just dropping obs with missing will be simpler! and having the same effect. $\endgroup$ – kjetil b halvorsen Apr 19 '18 at 17:35
  • 1
    $\begingroup$ @kjetil sorry, I wasn't clear. I mean to do this for all observations in a given column that are missing. I have updated the example above to make this more clear. $\endgroup$ – seth127 May 2 '18 at 20:49

Ok, I have found an answer to this question from talking to someone I know (a novel approach), so I figured I should share in case someone else stumbles onto this.

My methodology makes sense, if the data has been normalized and/or standardized so that the mean is 0. However, if it has not (like in my example data above) then, in addition to adding the extra indicator variable, you need to do mean- or mode-imputation instead of replacing the value with 0. This is the appropriate way to "neutralize" the effect the given coefficient for that observation. That was the fundamental flaw in my thought process.

It still makes sense to add the extra indicator column, because this will (plausibly) account for any effect on the response that is caused by the data missing not-at-random.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.