3
$\begingroup$

Is there way to apply median imputation method on the column Income_category based on the Education_level column. I want to be able to guess the Income_category for NA's using the Educational level column, but the problem is Income_category column contains symbols and other values which prevent me from applying median imputation on it. Any help to solve this problem would be really helpful.

Sample code:

med_uned_inc<-median(ref_data[ref_data$Education_Level == 'Uneducated','Income_Category'],na.rm = TRUE)

Sample data:

 Customer_Age Gender Dependent_count Education_Level Marital_Status Income_Category Card_Category
1            45      M               3     High School        Married     $60K - $80K          Blue
2            49      F               5        Graduate         Single  Less than $40K          Blue
3            51      M               3        Graduate        Married    $80K - $120K          Blue
5            40      M               3      Uneducated        Married     $60K - $80K          Blue
6            44      M               2        Graduate        Married     $40K - $60K          Blue
7            51      M               4            <NA>        Married         $120K +          Gold
9            37      M               3      Uneducated         Single     $60K - $80K          Blue
10           48      M               2        Graduate         Single    $80K - $120K          Blue
12           65      M               1            <NA>        Married     $40K - $60K          Blue
13           56      M               1         College         Single    $80K - $120K          Blue
15           57      F               2        Graduate        Married  Less than $40K          Blue
17           48      M               4   Post-Graduate         Single    $80K - $120K          Blue
18           41      M               3            <NA>        Married    $80K - $120K          Blue
19           61      M               1     High School        Married     $40K - $60K          Blue
20           45      F               2        Graduate        Married            <NA>          Blue
21           47      M               1       Doctorate       Divorced     $60K - $80K          Blue
22           62      F               0        Graduate        Married  Less than $40K          Blue
23           41      M               3     High School        Married     $40K - $60K          Blue
24           47      F               4            <NA>         Single  Less than $40K          Blue
25           54      M               2            <NA>        Married    $80K - $120K          Blue 
$\endgroup$
4
  • $\begingroup$ Can you show the expected output. The values in 'Income_Category' are in a range $\endgroup$
    – akrun
    Commented Feb 10, 2021 at 18:27
  • 1
    $\begingroup$ It makes no sense to "guess" categorical data of this kind using median imputation methods. You should use multiple imputation or consult a local statistician. $\endgroup$
    – jay.sf
    Commented Feb 10, 2021 at 18:52
  • $\begingroup$ @akrun I was wondering if there is any method that would help me fill in the NA's in the income_group and Education level columns. I thought you could use median imputation method but since income group is a range it's not possible. $\endgroup$
    – Gokul
    Commented Feb 10, 2021 at 19:03
  • $\begingroup$ @jay.sf Is there way to fill in the missing NA values ? $\endgroup$
    – Gokul
    Commented Feb 10, 2021 at 19:06

1 Answer 1

4
$\begingroup$

Any kind of single value imputation (eg median or mean imputation) is usually a very bad idea and can lead to severe biases

One approach to this problem is to run a regression model, where the variable containing the NA values is the response, and the other data is used to "predict" the missing values. If there are more than one variable that has missing values then this will be done as many times as there are variables with missing values. So, if we do this procedure once we will obtain a complete dataset. The problem with this is that it does not account for uncertainty in the missing values that were imputed. It's as though we were certain that the values we imputed were the real values. Obviously this is wrong and will also lead to bias (though usually less bias than just doing single value imputation). A better approach is therefore to introduce some uncertainty/randomness, and then repeat the whole process so that we have several imputed datasets, each having slightly different imputed values. The final analysis model can then be run on each imputed dataset, and the results pooled to account for variation between and within imputations.

This is the essence of multiple imputation and the extent to which this will give less biased and more accurate results depends on the reason(s) for the missingness in the first place. I would suggest reading up about missing data problems in general and multiple imputation in particular.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.