1
$\begingroup$

Last time I posted a question in Stackoverflow how to fill nans based on frequency.

I got some comments about whether it is a good idea or not. So I am seeking some suggestions if this is actually a good idea or not here in Cross Validated.

My Post in stackoverflow:

Let's say I am doing a binary classification and I have a feature called "sex" with some missing values.

Is it better to fill nans with mode or fill nans with frequency?

import numpy as np
import pandas as pd


df = pd.DataFrame({'sex': [1,1,1,1,0,0,np.nan,np.nan,np.nan]})
df['sex_fillna'] = df['sex'].fillna(df.sex.mode()[0])
print(df)
   sex  sex_fillna_mode   sex_fillna_freq
0  1.0         1.0        1.0
1  1.0         1.0        1.0
2  1.0         1.0        1.0
3  1.0         1.0        1.0
4  0.0         0.0        0.0
5  0.0         0.0        0.0
6  NaN         1.0        1.0
7  NaN         1.0        1.0
8  NaN         1.0        0.0  

# Here, we have 4 males and 2 females. 
# We have 3 missing values.
# I have filled 2 missing values with 1.0
# and one missing value with 0.0.
# Is this a better idea?

Which option is better feature sex_fillna_mode or sex_fillna_freq?

Update

The missing value imputation depends on how the missing values are generated in the first place such as missing at random, systematic no response to certain questions and so on. Here, I am unknown with the data generation process and working with the data as provided. I am trying to build a classification model, but one of the features "sex" has some nans and I was exploring ways to impute the missing values.

Usually, for a given column, we fill NaNs with only one value (eg. mean or median or "Unknown" etc). But my question here is "is there anything harm filling missing values with more than one values for a single column?"

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ The answer is likely "neither," but it must depend on (a) why you are performing this imputation exercise and (b) the reasons these missing values occur. Could you edit your post to include this information? $\endgroup$ – whuber♦ Aug 24 '20 at 17:42
  • 1
    $\begingroup$ It depends on the information I asked for. Otherwise, your question reads something like "which makes a better material for footballs, styrofoam or asphalt shingles?" That would be a tough one to answer, wouldn't it? $\endgroup$ – whuber♦ Aug 24 '20 at 17:50
  • $\begingroup$ @whuber I have edited the question to make it problem specific. My aplogies for vague description. $\endgroup$ – astro123 Aug 24 '20 at 18:01
  • 2
    $\begingroup$ One concern is that whatever method you select from these generic solutions might be arbitrary, thereby potentially casting doubt on the accuracy and reliability of everything else you do afterwards. Even if you can justify your selected procedure, it would be good to know how it stacks up against alternative methods in light of achieving your ultimate objectives. $\endgroup$ – whuber♦ Aug 24 '20 at 18:02
  • $\begingroup$ In particular, is there some reason why you are limiting yourself to a single imputation? It's generally best practice to do multiple imputations so that the uncertainty introduced by imputation can be taken into account. See Stef van Buuren's book for an introduction to why this is important and to the well-established procedures for doing such imputations. $\endgroup$ – EdM Aug 24 '20 at 21:09
1
$\begingroup$

is there anything harm filling missing values with more than one values for a single column?

No. In fact, you should not just fill in missing values with potentially different values within a single column, you should do so repeatedly to get multiple versions of imputed data sets. That's the correct way to avoid bias or incorrect standard errors when dealing with missing data.

Section 1.3 of Stef van Buuren's book examines several approaches for filling in missing data, discussing the conditions under which they can produce unbiased results and give correct standard errors when you only generate one data set with imputations. Except under specific circumstances they have substantial problems.

The best approach is to use a probabilistic estimation of the missing values, based on the other information you have about the cases, and repeat the process several times to get multiple imputed data sets. With sex having missing values, you could use methods based on logistic regression to combine information available from the relationships of the other predictors and of the outcome values with sex. You then do the analysis separately on each of the imputed data sets, and combine the results across the imputations. In that way you correctly incorporate the uncertainty that comes from both the modeling itself and the imputation process. See the book for details, and the mice package in R for one implementation.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.