Paired-samples t-test or a test between two dependent mean proportions

Question

Suppose after an initial pre-test and a treatment, a group of 30 participants are presented with 10 sentences written in a foreign language, each containing a grammatical error.

To measure participants' performance, we examine what proportion of the 10 errors has been identified on the pre-test, and on the post-test by each participant.

Now, we have a set of 30 proportions for the pre-test, and 30 corresponding proportions for the post-test.

Question

Should one now do a "Paired-Samples t-test" between the proportions on the pre-test and the proportions on the "post-test" OR ALTERNATIVELY one should do a "significance test on the difference between the two (from pre-test, and from post-test) dependent mean proportions"?

Answer 1

I think that a paired-test should certainly be used for making comparison before and after for the same individual. However, you should consider McNemar's test for comparing proportions instead. It wouldn't be right to use paired t-test as the test is used for comparing mean and it assumes the variable is about normally distributed which may not be the case for proportions. On the other hand, McNemar's test assumes a $\chi^2$ distribution. Also, the method of estimation for variance component is also different.

Answer 2

I would take a different approach. You have a "before" score and an "after" score for each subject. You can find the distribution of score changes by subtracting the "before" score from the "after" score. If you find a significant improvement of 2.2 sentences (significantly different from zero), you can phrase that in terms of percent improvement: 2.2/30 = 7.33 percentage points average increase.

This relies on the fact that each subject evaluates ten sentences. If some evaluated ten sentences and some evaluated nine (or if there were more sentences evaluated after the training), this would not work.

If you decide to examine the proportions directly, then your setup calls for paired testing. The first test that you described is not a paired test but a two-sample test. The regular two-sample t-test of means or two-sample Kolmogorov-Smirnov test of general distribution equality would apply in that setting. In your example, it would mean that students $X_1,\dots,X_{30}$ are not treated, but thirty separate students $Y_1,\dots,Y_{30}$ are treated before they evaluate the sentences for errors. Since you measure $X_1,\dots,X_{30}$ before the treatment and then again after the treatment, you have dependence between the two groups.