0
$\begingroup$

Suppose after an initial pre-test and a treatment, a group of 30 participants are presented with 10 sentences written in a foreign language, each containing a grammatical error.

To measure participants' performance, we examine what proportion of the 10 errors has been identified on the pre-test, and on the post-test by each participant.

Now, we have a set of 30 proportions for the pre-test, and 30 corresponding proportions for the post-test.

Question

Should one now do a "Paired-Samples t-test" between the proportions on the pre-test and the proportions on the "post-test" OR ALTERNATIVELY one should do a "significance test on the difference between the two (from pre-test, and from post-test) dependent mean proportions"?

$\endgroup$
  • $\begingroup$ What is "significance test on the difference between the two (from pre-test, and from post-test) dependent mean proportions"? link? $\endgroup$ – user158565 Jun 13 '17 at 1:38
  • $\begingroup$ @a_statistician, you mean paired samples t-test is ok to be performed here? $\endgroup$ – rnorouzian Jun 13 '17 at 3:13
  • $\begingroup$ I mean I cannot understand that sentence and want to have a reference to learn it. $\endgroup$ – user158565 Jun 23 '17 at 15:22
0
$\begingroup$

I think that a paired-test should certainly be used for making comparison before and after for the same individual. However, you should consider McNemar's test for comparing proportions instead. It wouldn't be right to use paired t-test as the test is used for comparing mean and it assumes the variable is about normally distributed which may not be the case for proportions. On the other hand, McNemar's test assumes a $\chi^2$ distribution. Also, the method of estimation for variance component is also different.

$\endgroup$
0
$\begingroup$

I would take a different approach. You have a "before" score and an "after" score for each subject. You can find the distribution of score changes by subtracting the "before" score from the "after" score. If you find a significant improvement of 2.2 sentences (significantly different from zero), you can phrase that in terms of percent improvement: 2.2/30 = 7.33 percentage points average increase.

This relies on the fact that each subject evaluates ten sentences. If some evaluated ten sentences and some evaluated nine (or if there were more sentences evaluated after the training), this would not work.

If you decide to examine the proportions directly, then your setup calls for paired testing. The first test that you described is not a paired test but a two-sample test. The regular two-sample t-test of means or two-sample Kolmogorov-Smirnov test of general distribution equality would apply in that setting. In your example, it would mean that students $X_1,\dots,X_{30}$ are not treated, but thirty separate students $Y_1,\dots,Y_{30}$ are treated before they evaluate the sentences for errors. Since you measure $X_1,\dots,X_{30}$ before the treatment and then again after the treatment, you have dependence between the two groups.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.