I have a question about missing values. We used 3 versions of a questionnaire were the possible answers were numeric (0-10) with 0 = no pain and 10 = worst possible pain. We had 4 pain questions worst, least, average and pain now and only the first version of the questionnaire asked all 4 pain questions. I scored the data as 999 for truly missing values and 888 for missing values when the question was not asked. I want to create a 'total pain score'. I summing the scores of average and pain now. I divided the result by 2 if there are two valid values (e.g. 0 and 4 =2). If one sore was 888 treated this like a dummy variable and did not delete the paired value and did not divide by 2 (e.g. 888 and 2 =2). Does this sound like the appropriate way to deal with my missing values? -If I use pair wise deletion I lose 70% of my data.
1 Answer
So you have a sample of responses for 3 parallel versions of questionnaire, where some of the questions between versions are the same and you have one version where all the questions are present, correct?
For me, this sounds like an test equating problem. Consider that while you want the tests to be parallel, you do not know if people answered all the questions the same way. So there could be different response patterns for question 1 and question 4 and if so, then the same raw sum of scores on two tests consisting of different questions would mean different things. In this case you could employ test equating, that is, a statistical procedure that lets you to transform raw scores of one questionnaire into the scale of another, so that both tests share the same scale and are equivalent.
The simple example is linear equating, where if you want co transform raw score of questionnaire $X$ into scale of $Y$ you use mean and standard deviation:
$$Lin_Y(x) = \frac{\sigma_Y}{\sigma_X}x + \left( \mu_Y - \frac{\sigma_Y}{\sigma_X}\mu_X \right) $$
there is also more advanced method that use cumulative distribution functions - the equipercentile equating method:
$$Equi_Y(x) = F^{-1}_Y \left[ F_X(x) \right]$$
However, the second method needs special data preparation beforehand i.e. continuization of the scores. Of course, there are also other methods as well.
Wile the questionnaires were answered by different populations, you could use the questions that are common between versions of questionnaires as anchor tests in equivalent groups design.
Check books on equating by Kolen and Brennan and von Davier at al for more informations.
-
$\begingroup$ The challenge with evaluating consistency in this way is two-fold: basically the type I and type II error cannot be characterized. As we know in survey design, the period effect can be enormous: those response to the first wave of a survey are different from those who response to later waves/invitations. A significant finding may indicate a shift in the type of responding sample rather than a questionnaire effect. Similarly, a null finding may coincide with a questionnaire effect balanced with the sample characteristics. $\endgroup$– AdamOCommented Jan 29, 2018 at 15:25
-
$\begingroup$ The expression you use for linear equating can result in the shift in the range of support of the distribution of Y compared to X. How can we match the mean without changing the range of support. Say, X is defined on an interval [a, b] and we want to transform its sample (or empirical distribution) to have the same support but different mean - how can we achieve this? And what about variance? And how do we transform only one of them, ie transform mean keeping variance the same, or transforming variance keeping mean the same. Thanks $\endgroup$ Commented Apr 7, 2019 at 16:29
-
$\begingroup$ @Confounded equipercentile equating (mentioned above) seems to be what you're asking. $\endgroup$– TimCommented Apr 7, 2019 at 17:20
-
$\begingroup$ Thank you for your reply @Tim. Unfortunately, I only have information on a few target moments, so can't use equi-percentile approach. $\endgroup$ Commented Apr 7, 2019 at 19:08