2
$\begingroup$

My supervisor suggests that I impute the missing data in my dataset through various methods (namely complete cases, k-nearest neighbors, last observation carried forward and multiple imputation with predictive mean matching) and compare the derived estimates to choose the most appropriate one. Does it make sense to do so if I'm not doing a simulation study for method comparison or is it an overkill? Is it more common to just go for an algorithm with an established performance such as MICE?

(Chances are, he just wants me to practice but I wonder what a common way of doing it is when the focus is on the actual research question)

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ What do those abbreviations mean (please description include in the body). Then, is MICE a method or a name of a program? $\endgroup$
    – ttnphns
    Commented Jun 28, 2022 at 12:34
  • $\begingroup$ @ttnphns, thanks for the suggestion, edited the body. I meant MICE as a method (Multiple Imputation by Chained Equations) $\endgroup$
    – olke
    Commented Jun 29, 2022 at 11:02

1 Answer 1

Reset to default
4
$\begingroup$

In an ideal world, the method of handling missing data would be specified as part of the design of the experiment. We could imagine a shady researcher who attempts every known method for missing data adjustment and then reports the one which gives the largest treatment effect. This would be a form of p-hacking and lead to incorrect estimates, confidence intervals, and hypothesis tests. Pre-specifying the method to use avoids this danger, and thinking about potential problems in advance is good practice.

On the other hand, one often does not know in advance the exact properties of the missing data, whether MCAR, MAR, or NMAR are more plausible, the amount of missing data, any strange patterns of missingness, and so on. That makes choosing a method in advance difficult. The assumptions of the chosen method will always need to be checked after data has been collected, and violations of those assumptions may lead us to change methods.

Another issue is sensitivity analysis. We would like that the final results don't depend strongly on the choice of method we used. The simplest way to assess this is to try different methods and see if the results differ. If all methods give similar results then we can report that other methods were considered and found to be similar. If the results are different then choosing the best one to use is difficult. The best approach would probably be to report all of the methods, but that makes it difficult to interpret the results, especially difficult for anyone who isn't familiar with the different methods.

Personally, I would recommend comparing the complete case analysis with one alternate method at a minimum, and varying the parameters used in that alternate method. If there isn't too much missing data and the results are similar it's probably okay to stop there. If there are any notable differences between those two methods or if the amount of missing data is larger, say >10% missing or a certain group is missing more, then it's probably worth examining in more detail. In general I wouldn't be concerned if a research tried many different methods, as long as that's reported clearly.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for such an elaborate answer! As a very fresh PhD student, I had no idea it may be considered shady. Could you please elaborate on why it's the case? I was under the impression that finding the most favourable imputation method (i.e. the one that yields the most accurate imputed values) is desirable (as opposed to p-hacking). $\endgroup$
    – olke
    Commented Jun 29, 2022 at 11:07
  • $\begingroup$ That's the "good" definition of "most favourable"; there's another definition - "gives me the best-looking study results" - that is akin to p-hacking. $\endgroup$
    – jbowman
    Commented Jun 29, 2022 at 13:52
  • $\begingroup$ @jbowman 's second definition is what I was thinking of, I've edited to clarify. Trying different imputation methods to find the most accurate may lead to overfitting the data, but it's probably balanced out by the more accurate imputation values. Hard to say. Train-Test splitting or Cross Validation can be used as well, but get computationally prohibitive. $\endgroup$
    – David L Thiessen
    Commented Jun 29, 2022 at 15:48
  • $\begingroup$ Ah now I see, you meant choosing the imputation method that yields the values most 'desirable' for the study outcome. Makes perfect sense. Thanks for the answers you both, very helpful! $\endgroup$
    – olke
    Commented Jun 29, 2022 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.