How do I analyze a z-score to determine a difference? My data are in SPSS. I teach in a nursing program.
During their junior year, 68 students participated in a simulation scenario for heart failure. They were by rated by observers (0/1, did not do/did do) on 102 items, thus a student's score could range from 0-102 (actual range was 26-86 with a mean of 57.41 and SD = 15.49). Higher scores indicated better performance.
Their senior year, these same 68 students participated in a pneumonia scenario. Once again, they were rated by observers but the rating scale had 91 items. Actual range of scores was 0-91 with a mean of 52.48 and SD = 14.6.
I want to compare student performance from jr. to sr. year to see if it improved. Note that the mean score sr. year is lower but that may be because the rating scale had 11 fewer items.
Since the scales have a different number of items, I am thinking I need to standardize the scores. Thus I had SPSS calculate z-scores for the total score for the pneumonia and heart failure rating scales. Where I am stuck is what to do next? Calculating a paired samples t-test using the z-score is not working, I am assuming because the scores are standardized to have a mean of 0 and SD of 1. 
I am sure there is a simple solution but I am stumped. After looking at this all afternoon I can't come up with a solution. Therefore, thanks in advance to anyone who can give me an answer/suggestion of what to do. Thanks!
 A: $Z$-scores are definitely NOT the answer.  You would need to make too many assumptions about the meaning on standard deviations.  You have both time and a change in the disease being simulated so it's not clear on what you mean by 'improved.  But to your original question you would need to have a clear and meaningful way to calibrate the scales to each other.  The simplest approach is to divide by the number of items but this would need a lot of though.  Overall your design is confounded and interpretation will be difficult
A: Adding to @Frank 's excellent answer:
If you are willing to assume that all the items are equally "difficult" in some abstract way, then you could compare percentage scores (thus the number of "corrects" divided by the number of items). But this is a big assumption.
I don't think there is any way to really tell if there is improvement in any real sense, given your design. You would need to have both juniors and seniors take both tests (this gets rid of some of the confounding). You would also want to adjust for the fact that it is the same students taking the two tests, thus, the data are not independent. 
Making sure two tests are equally difficult, though, is a huge task. In psychometrics it is known as test equating. It's difficult to do even when the tests are on the same subject (a guy who was in the same PhD program that I was in did test equating full-time for a while). 
