Thank you for joining me on this semi-theoretical journey. Here we will discuss how to account for "predictable" bias in your data.

Let's say we have a test composed on many subtests. A composite score is generated through successive aggregation and weighting of sub-tests.

Let's talk about sub-test A.

You can get a total score of 100 on sub-test A, and each item on the test has a different possible max score. Now you're missing (at-random) some answer for sub-test, rather than try to impute these scores, you can drop the missing items from the subtest and rescale.

How are we rescaling? Let's say you drop q10 and the MAX score you can get on this sub-test in 90. Multiply 90 by the NEW percent correct on this test to scale it back up. Now recalculate your composite score the same way as usual.

Now calculate the difference between your newly calculated/rescaled score and the original score in a sample of subjects who have COMPLETE sub-test sections and figure out how your data will be biased.

Here we are, when you use the method outlined above, we will on average skew the data by ~0.5 points if we drop the missing sub-items. Since I can measure this bias, would it be reasonable to then add back this bias as a constant in a linear regression?

enter image description here


It sounds like your approach assumes that the missing values are missing completely at random (MCAR). That's a very strong assumption, one that wouldn't hold if the probability of missingness to one answer is a function of the responses to other answers on the tests.

Multiple imputation allows you to analyze data efficiently under a much weaker assumption of missing at random (MAR). MAR holds if the probability of missingness is related to data that you have but not to data that you don't have.

See Stef van Buuren's online book for extensive discussion. Chapter 1 of the book explains the different types of missingness and the disadvantages of other approaches, including those that assume MCAR.

  • $\begingroup$ The data is missing because the item wasn't live scored on paper and we don't have the video files needed to score it retroactively. So we know exactly how and why the data is missing, is imputation still appropriate? $\endgroup$
    – myfatson
    Jan 14 at 21:56
  • $\begingroup$ @myfatson imputation is certainly appropriate. One might argue that your scenario could represent true MCAR. But why force yourself into that assumption when another solution is available that doesn't depend on it? Look at van Buuren's book, linked in the answer. $\endgroup$
    – EdM
    Jan 14 at 22:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.