2
$\begingroup$

My data are in SPSS. I teach in a nursing program.

During their junior year, 68 students participated in a simulation scenario for heart failure. They were by rated by observers (0/1, did not do/did do) on 102 items, thus a student's score could range from 0-102 (actual range was 26-86 with a mean of 57.41 and SD = 15.49). Higher scores indicated better performance.

Their senior year, these same 68 students participated in a pneumonia scenario. Once again, they were rated by observers but the rating scale had 91 items. Actual range of scores was 0-91 with a mean of 52.48 and SD = 14.6.

I want to compare student performance from jr. to sr. year to see if it improved. Note that the mean score sr. year is lower but that may be because the rating scale had 11 fewer items.

Since the scales have a different number of items, I am thinking I need to standardize the scores. Thus I had SPSS calculate z-scores for the total score for the pneumonia and heart failure rating scales. Where I am stuck is what to do next? Calculating a paired samples t-test using the z-score is not working, I am assuming because the scores are standardized to have a mean of 0 and SD of 1.

I am sure there is a simple solution but I am stumped. After looking at this all afternoon I can't come up with a solution. Therefore, thanks in advance to anyone who can give me an answer/suggestion of what to do. Thanks!

$\endgroup$
  • $\begingroup$ We might be able to rescue these data instead of consigning them to the statistical junk pile. Is there substantial overlap among the items in the two ratings? If so, perhaps those could be used to calibrate the two tests. (Or, more simply, you could confine your comparisons to common items in the two ratings.) Calibration would require making (and testing) some assumptions and introduce some uncertainty, but it might enable you to draw some kind of valid conclusions. The chief difficulty is dealing with the confounding: all differences could be attributed to the change in scenario. $\endgroup$ – whuber Mar 27 '13 at 23:01
2
$\begingroup$

$Z$-scores are definitely NOT the answer. You would need to make too many assumptions about the meaning on standard deviations. You have both time and a change in the disease being simulated so it's not clear on what you mean by 'improved. But to your original question you would need to have a clear and meaningful way to calibrate the scales to each other. The simplest approach is to divide by the number of items but this would need a lot of though. Overall your design is confounded and interpretation will be difficult

$\endgroup$
  • $\begingroup$ Thank you, Frank. The idea with "better" is that they would receive more 1 ratings (do things as opposed to forgetting to do things). Their performance would improve because they have had an additional year of clinical, learned more, etc., plus be more familiar with participating in a simulation. $\endgroup$ – Leslie Nicoll Mar 27 '13 at 21:51
  • $\begingroup$ Also, divide what by the number of items? $\endgroup$ – Leslie Nicoll Mar 27 '13 at 21:52
  • $\begingroup$ A pneumonia scenario is different from a HF scenario, so it seems to me you are test both time and scenario. I was thinking of summing all item scores and normalizing by number of items. An improvement would be to normalize by summing by all the maximum possible scores per item. This is not without danger though. $\endgroup$ – Frank Harrell Mar 27 '13 at 22:40
1
$\begingroup$

Adding to @Frank 's excellent answer:

If you are willing to assume that all the items are equally "difficult" in some abstract way, then you could compare percentage scores (thus the number of "corrects" divided by the number of items). But this is a big assumption.

I don't think there is any way to really tell if there is improvement in any real sense, given your design. You would need to have both juniors and seniors take both tests (this gets rid of some of the confounding). You would also want to adjust for the fact that it is the same students taking the two tests, thus, the data are not independent.

Making sure two tests are equally difficult, though, is a huge task. In psychometrics it is known as test equating. It's difficult to do even when the tests are on the same subject (a guy who was in the same PhD program that I was in did test equating full-time for a while).

$\endgroup$
  • $\begingroup$ Just to clarify, the students aren't taking a test, they are being observed and rated if they do/don't do things during the scenario (ie, verify a pt's identity by looking at the armband). We went through a rater validation process so we know they know what they are doing. And yes, the results are not independent, thus a paired samples t-test to compare the students from jr. to sr. year. There are subscales in the rating scale (pt. safety, assessment, etc) but right now I am just focusing on the total score. Thanks again for your help. LN $\endgroup$ – Leslie Nicoll Mar 27 '13 at 22:25
  • 1
    $\begingroup$ Your clarification is fine, but it doesn't change anything in what I wrote. "Rating" is a form of test, I think (although not a traditional paper and pencil test). $\endgroup$ – Peter Flom Mar 27 '13 at 22:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.