2
$\begingroup$

I am developing a Survey Analysis Application that works with Categorical Data and looking for a way to treat missing data fairly without losing the original data variance.

Thus, I came up with a way to deal with these values, that is, for every missing value, we make a list of non-missing values in the same column, randomly select a value from that list by using np.random.choice, and use it to fill in the missing value. The randomization by itself is weighted. If a value has a greater frequency, it would have a higher chance of being filled.

The problem is, I have not seen anyone done it this way before so I am not sure whether it would maintain the data variance and the fairness of the survey. Any feedback, suggestions, or flaws that you find in this approach would be greatly appreciated.

$\endgroup$

1 Answer 1

1
$\begingroup$

This seems like a simple form of single imputation. You could improve it by:

  • First considering whether these values are missing (completely) at random or not. If they are not, imputation alone will bias the results and you should look into ways to model missing not at random.
  • Instead of filling the missing values once, making many copies of your original data, where the missing data are filled in randomly. This is known as multiple imputation.
  • Taking into account not only the relative frequencies of categories, but also the relationships with other variables. One way to do this is with multiple imputation by chained equations (MICE). In R, the package mice can do this,$^{[1]}$ perhaps there is an implementation in Python as well.

$[1]$: Buuren, Stef. Flexible imputation of missing data. Boca Raton, FL: CRC Press, 2018. Print.

$\endgroup$
1
  • $\begingroup$ I really like the idea of doing multiple imputations, as well as the chained equations that we can use to link variables together. Thank you very much for your suggestion. $\endgroup$
    – My Name
    May 28, 2021 at 2:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.