# The validity of using paired sample t-test to compare results from two different test methods

My research involved two types of contextualized tests to examine two types of knowledge of collocations. The first test is a multiple-choice (with 3 options) and it was used to examine receptive knowledge of collocations. The second test is gap-filling and it was used to examine productive knowledge of collocations. Both tests had been done by the same sample size at the same time, but the items in the multiple-choice test are different from the items in the gap-filling test. One of the research questions is to find whether there is a significant difference between receptive knowledge of collocations and productive knowledge of collocations. I normalized the data for both tests and I used a paired-sample t-test to answer this question.

I am wondering whether what I have done by using a paired t-test is correct.

I'm not entirely sure if my reading of your situation is correct, but it appears you are taking one groups mean scores for multiple choice questions and comparing them to another groups mean scores. If that is the case, then yes, that is an appropriate way of handling the analysis. Simulating your example with this dataset:

set.seed(123)
df <- data.frame(test.group = as.factor(rbinom(n=1000,
size=1,
prob = .5)),
scores = rnorm(n=1000,
mean=100,
sd=15))


We get this sort of data, test group 0 can be considered the control and test group 1 can be considered the comparison group, and the scores are their total scores for the test:

  test.group    scores
1          0  90.97161
2          1  85.09452
3          0 115.40178
4          1 111.26592
5          1  77.36250
6          0  98.57279


We can then run a t-test:

t.test(df$$scores ~ df$$test.group)


Which gives us this:

    Welch Two Sample t-test

data:  df$$scores by df$$test.group
t = -0.052732, df = 997.98, p-value = 0.958
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
-1.915064  1.814835
sample estimates:
mean in group 0 mean in group 1
100.1543        100.2044


Now if there is some other important part I'm missing, please clarify and I can correct what I have said here.

• Thank you Shawn Hemelstrand for the quick answer. To clarify my question, I intended to compare the results of two tests for one group. My research has only one group and they did the two tests (multiple-choice and gap-filling). My research question is to find the significant difference between the study group mean scores on multiple-choice test and gap-filling test. As I said, I normalized the data to rescale them and I used paired sample t-test. Is normalizing the data and using paired t-test to answer this question valid? Sep 13, 2022 at 1:05
• Yes. So if you have one group getting tested on two tests, it is functionally equivalent to saying it is two groups (Gap Fill Group vs MC Group) getting tested on the differences between their means. This is all a t-test does...looking at two means and seeing if they have a difference that is statistically significant. Sep 13, 2022 at 1:09