Last time I posted a question in Stackoverflow how to fill nans based on frequency.
I got some comments about whether it is a good idea or not. So I am seeking some suggestions if this is actually a good idea or not here in Cross Validated.
My Post in stackoverflow:
Let's say I am doing a binary classification and I have a feature called "sex" with some missing values.
Is it better to fill nans with mode or fill nans with frequency?
import numpy as np
import pandas as pd
df = pd.DataFrame({'sex': [1,1,1,1,0,0,np.nan,np.nan,np.nan]})
df['sex_fillna'] = df['sex'].fillna(df.sex.mode()[0])
print(df)
sex sex_fillna_mode sex_fillna_freq
0 1.0 1.0 1.0
1 1.0 1.0 1.0
2 1.0 1.0 1.0
3 1.0 1.0 1.0
4 0.0 0.0 0.0
5 0.0 0.0 0.0
6 NaN 1.0 1.0
7 NaN 1.0 1.0
8 NaN 1.0 0.0
# Here, we have 4 males and 2 females.
# We have 3 missing values.
# I have filled 2 missing values with 1.0
# and one missing value with 0.0.
# Is this a better idea?
Which option is better feature sex_fillna_mode
or sex_fillna_freq
?
Update
The missing value imputation depends on how the missing values are generated in the first place such as missing at random, systematic no response to certain questions and so on. Here, I am unknown with the data generation process and working with the data as provided. I am trying to build a classification model, but one of the features "sex" has some nans and I was exploring ways to impute the missing values.
Usually, for a given column, we fill NaNs with only one value (eg. mean or median or "Unknown" etc). But my question here is "is there anything harm filling missing values with more than one values for a single column?"