Report classifier accuracy with missing data

Question

Suppose we are reading blood pressure. Some of the readings are corrupt and unusable. We then train a binary classifer to detect high blood pressure. What does the theory of experimental design say about reporting classifier accuracy when some of the data is unusable by our model?

Answer 1

You say:

Some of the readings are corrupt and unusable.

Do you know why they became corrupted? In missing data problems, it is important to understand the "missingness mechanism". There are three main types of missing data mechanisms:

• Missing Completely at Random (MCAR): If the missingness is due to random chance and unrelated to any variables in your dataset.

• Missing at Random (MAR): If the missingness is related to other observed variables in your dataset.

• Missing Not at Random (MNAR): If the missingness is related to unobserved factors or the value of the variable itself.

For MCAR or MAR, using multiple imputation can lead to unbiased estimates. MI fills in the missing data with plausible values based on the relationships in your dataset, allowing you to use the entire dataset for model training and evaluation. It creates multiple imputed datasets on which the model is used, and the results are then pooled.

However, if the data is MNAR, the situation is more complicated, and imputation methods may not be sufficient to correct for bias.

What does the theory of experimental design say about reporting classifier accuracy when some of the data is unusable by our model?

I am no expert in experimental design, but as far as I know it doesn't have a great deal to say on the matter. The theory of missing data on the other hand, does. The seminal paper by Rubin (1976) is an excellent starting place to learn about this. One implementation of Rubin's ideas is Van Buren & Groothuis-Oudshoorn (2011): Multivariate Imputation By Chained Equations (MICE)

Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581-592.

Van Buuren, S., & Groothuis-Oudshoorn, K. (2011). mice: Multivariate imputation by chained equations in R. Journal of statistical software, 45, 1-67.