The validity of using paired sample t-test to compare results from two different test methods

Question

My research involved two types of contextualized tests to examine two types of knowledge of collocations. The first test is a multiple-choice (with 3 options) and it was used to examine receptive knowledge of collocations. The second test is gap-filling and it was used to examine productive knowledge of collocations. Both tests had been done by the same sample size at the same time, but the items in the multiple-choice test are different from the items in the gap-filling test. One of the research questions is to find whether there is a significant difference between receptive knowledge of collocations and productive knowledge of collocations. I normalized the data for both tests and I used a paired-sample t-test to answer this question.

I am wondering whether what I have done by using a paired t-test is correct.

Answer 1

I'm not entirely sure if my reading of your situation is correct, but it appears you are taking one groups mean scores for multiple choice questions and comparing them to another groups mean scores. If that is the case, then yes, that is an appropriate way of handling the analysis. Simulating your example with this dataset:

set.seed(123)
df <- data.frame(test.group = as.factor(rbinom(n=1000,
                                     size=1,
                                     prob = .5)),
                 scores = rnorm(n=1000,
                                mean=100,
                                sd=15))
head(df)

We get this sort of data, test group 0 can be considered the control and test group 1 can be considered the comparison group, and the scores are their total scores for the test:

  test.group    scores
1          0  90.97161
2          1  85.09452
3          0 115.40178
4          1 111.26592
5          1  77.36250
6          0  98.57279

We can then run a t-test:

t.test(df$scores ~ df$test.group)

Which gives us this:

    Welch Two Sample t-test

data:  df$scores by df$test.group
t = -0.052732, df = 997.98, p-value = 0.958
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
 -1.915064  1.814835
sample estimates:
mean in group 0 mean in group 1 
       100.1543        100.2044 

Now if there is some other important part I'm missing, please clarify and I can correct what I have said here.