I have a response dataset for twenty-five questions, which measures three grammar topics. The participants are two different English proficiency groups of intermediate and advanced. So, I want to compare their correct number of answers within each of the three grammar topics. For example, is there any significant difference between the number of correct answers of INT and ADV groups in the questions testing tenses? As can be noticed, I want to compare these two groups' means of correct answers only within these conditions, not across conditions (which would correspond to MANOVA, I guess, in that case).

For purposes of example, the following are the participants' total number of correct answers out of 10 in each condition. The number 5 in the first horizantal line is the total correct number of first intermediate participant.


INT = c(5,6,4,2,1,......); ADV = c(9,7,6,8,8,......)


INT = c(2,3,2,1,4.......); ADV = c(9,9,8,7,5.......)

So, how could I examine their performance to each other in every condition (grammar topic in my case)? Normally, I conducted an analysis using a t-test with three dependent variables (as the number of groups' correct answers in each of the three topics), but I heard that multiple t-tests can yield Type-I error.

  • $\begingroup$ How about an analysis of variance with two factors (level of proficiency and grammar topic)? This assumes that the response variable (number of correct answers) is treated as a continuous variable. You may want to consider post-hoc tests (akin to t-tests, corrected for multiple comparisons) to compare within-condition results. You will probably find a lot of threads on this site using such keywords. $\endgroup$ – chl Jan 21 at 19:43
  • $\begingroup$ Insufficient information. How many subjects? What kind of data? How are grades computed? Posting actual data in format such as INT = c(21, 32, 15, ... (horizontal list, comma separated) might make it easy to show an appropriate test. Just from what you say, absolutely not appropriate to speculate on appropriate test. $\endgroup$ – BruceET Jan 21 at 19:44
  • $\begingroup$ @BruceET Oh sorry. I reformatted the question content using the notation you stated. 25 participants in each proficiency group, so categorical independent variable. Grades are just the total number of correct answers out 10 (interval). $\endgroup$ – Süleyman Yaman Jan 21 at 21:36

If you got scores such as those below, I'd use a Welch two-sample t test to see if one group tends to get higher scores than the other. (Vectors p give my speculative relative proportions for each score.)

INT = sample(0:10, 25, rep=T, 
             p=c(1,2,3, 3,3,3, 2,1,1, 1,1))
ADV = sample(0:10, 25, rep=T,
             p=c(1,1,1, 2,2,2, 4,4,4, 4,5))

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    1.0     3.0     5.0     5.2     6.0    10.0 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    2.0     5.0     8.0     7.2    10.0    10.0 
boxplot(INT,ADV, col="skyblue2", notch=T)

enter image description here

In my fake data the ADV scores seem somewhat higher than the INT scores. Nonoverlapping notches in sides of boxes of two boxplots suggest a significant difference..

It seems OK to use a Welch two-sample t test for these data. With its P-value near 1.5% you can say the scores for the two groups are statistically significant at the 5% level.


        Welch Two Sample t-test

data:  INT and ADV
t = -2.5065, df = 47.894, p-value = 0.01564
alternative hypothesis: 
   true difference in means is not equal to 0
95 percent confidence interval:
 -3.6044055 -0.3955945
sample estimates:
mean of x mean of y 
      5.2       7.2 

Sums of scores correct out of 10 should be approximately normal; doing a Welch two-sample t test would be OK even if standard deviations for the two groups are noticeably different.

Note: Some statisticians might argue that a two-sample Wilcoxon rank sum test would be better, but you will typically have too many tied scores with only ten questions for a Wilcoxon test to give a reliable P-value.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.