Does it make sense to compare different imputation techniques? My supervisor suggests that I impute the missing data in my dataset through various methods (namely complete cases, k-nearest neighbors, last observation carried forward and multiple imputation with predictive mean matching) and compare the derived estimates to choose the most appropriate one. Does it make sense to do so if I'm not doing a simulation study for method comparison or is it an overkill? Is it more common to just go for an algorithm with an established performance such as MICE?
(Chances are, he just wants me to practice but I wonder what a common way of doing it is when the focus is on the actual research question)
 A: In an ideal world, the method of handling missing data would be specified as part of the design of the experiment. We could imagine a shady researcher who attempts every known method for missing data adjustment and then reports the one which gives the largest treatment effect. This would be a form of p-hacking and lead to incorrect estimates, confidence intervals, and hypothesis tests. Pre-specifying the method to use avoids this danger, and thinking about potential problems in advance is good practice.
On the other hand, one often does not know in advance the exact properties of the missing data, whether MCAR, MAR, or NMAR are more plausible, the amount of missing data, any strange patterns of missingness, and so on. That makes choosing a method in advance difficult. The assumptions of the chosen method will always need to be checked after data has been collected, and violations of those assumptions may lead us to change methods.
Another issue is sensitivity analysis. We would like that the final results don't depend strongly on the choice of method we used. The simplest way to assess this is to try different methods and see if the results differ. If all methods give similar results then we can report that other methods were considered and found to be similar. If the results are different then choosing the best one to use is difficult. The best approach would probably be to report all of the methods, but that makes it difficult to interpret the results, especially difficult for anyone who isn't familiar with the different methods.
Personally, I would recommend comparing the complete case analysis with one alternate method at a minimum, and varying the parameters used in that alternate method. If there isn't too much missing data and the results are similar it's probably okay to stop there. If there are any notable differences between those two methods or if the amount of missing data is larger, say >10% missing or a certain group is missing more, then it's probably worth examining in more detail. In general I wouldn't be concerned if a research tried many different methods, as long as that's reported clearly.
