2
$\begingroup$

I have a model generated using an imputed data set with imputation accuracy of 75%.

If the model using imputed data has an accuracy of 80%

What would be the community consensus to report the accuracy of such models with imputed data?

I assume the 80% accuracy has an error-rate of 25% because of the imputation error? Is there any consensus to report such results by taking into consideration of the imputation accuracy and model accuracy? Any methods available to estimate the consensus errors of the imputation and predictive model?

A bit of background: I work in healthcare data science; data is often noisy, missing or unavailable to compute using a complete matrix. Many of us use imputation methods to augment this; however for various projects that I worked non-imputed models perform better in terms of both accuracy and model quality. Also, clinical intuition suggests that a missing value indicates the real-world scenario where a test is specialized that imputing the test values using statistical estimates may ultimately lead to unwarranted clinical implications.

$\endgroup$
  • 1
    $\begingroup$ Good question. It seems there are 3 different types of literature related to your question: 1st literature considers the impact of different strategies wrt missing values in regression, 2nd literature (e.g., Rubin and Little or Paul Allison) addresses issues related to the accuracy of the imputation process itself, 3rd literature addresses issues related to the accuracy of predictive models. I'm not aware of literature that addresses the specific issue of incorporating imputation uncertainty when evaluating the accuracy of predictive models but other CVrs may have more information>> ctd $\endgroup$ – Mike Hunter Feb 27 '18 at 16:55
  • 1
    $\begingroup$ >>ctd. The applied simplifying assumption is that once the imputations are finalized any subsequent steps (e.g., predictive modeling) make deterministic assumptions about their results, i.e., uncertainty wrt the imputation process is ignored. One possible heuristic is to do a 'sensitivity analysis' (en.wikipedia.org/wiki/Sensitivity_analysis) which would explicitly incorporate uncertainty associated with the imputations. $\endgroup$ – Mike Hunter Feb 27 '18 at 16:56
  • $\begingroup$ 80% accuracy -> 25% error-rate? $\endgroup$ – AdamO Feb 27 '18 at 19:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.