Paired test or two-sample t-test?

Question

I am looking at whether there is a difference in the extent of emotional tone in student essays on two exams - one at T1 and another at T2. Some of the students who took the exam at T1 also took exam at T2. But there are students unique to each exam as well.

In order to examine the hypothesis that the mean emotional tone in essays at T2 is greater than mean emotional tone at T1, should I be doing a paired t-test or two-sample t-test?

Or do you suggest that I separate students who took both exams and do a paired t-test on this sample and a two-sample t-test for test takers who only took the exam at T1 or T2?

Answer 1

Your plan seems OK. But you have to understand that the paired test for subjects who took both tests will be more likely to show a difference in emotional tone, if such a difference exists. (The two-sample test for two groups of independently selected subjects will have lower power.) The following example, with data suitably simulated in R, illustrate.

Paired scores. Suppose we have 50 subjects who took both tests. They average around 100 on the first test, and there is an average increase of several points in 'emotional tone' for each student. Because data are paired, we are able to look mainly at the increase in emotional tone without being distracted by the variability of test scores due to differences among the 50 subjects. Data might look somewhat like the data simulated in R below.

set.seed(2020)
x1 = rnorm(50, 100, 15)
et = rnorm(50, 4, 2)
x2 = .98*x1 + et + rnorm(50, 0, 1)
d = x2 - x1
summary(d);  sd(d)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
-2.5581  0.4485  1.6450  2.0571  3.6346  8.5226 
[1] 2.442555  # SD of differences

Due to pairing, there is a positive correlation between first and second test scores, illustrated in the plot below. The $40$ points above the line (through the origin with unit slope) represent students with higher scores on the second exam, mainly due to the emotional tone effect. A paired t test (that is, a one-sample test on differences in scores) shows a highly significant effect (P-value very nearly $0)$.

cor(x1,x2)
[1] 0.9892561
plot(x1,x2,pch=20)
 abline(a=0,b=1,col="green")

t.test(d)

        One Sample t-test

data:  d
t = 5.9553, df = 49, p-value = 2.742e-07
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 1.362981 2.751314
sample estimates:
mean of x 
 2.057147 

Two independent samples of subjects. Suppose that 50 randomly chosen subjects took the first test and a different 50 randomly chosen subjects anticipated to score several points higher in emotional tone took the second test.

We have two separate samples, and so inevitable variability in exam-taking ability among subjects (modeled here by $\sigma = 15)$ will be apparent when we compare scores on the first and second tests.

set.seed(420)
y1 = rnorm(50, 100, 15)
y2 = rnorm(50, 104, 15)
summary(y1); sd(y1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  55.77   90.20   98.28   98.50  108.76  128.80 
[1] 15.25291
summary(y2); sd(y2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  72.71   88.47  105.48  100.88  111.72  127.97 
[1] 14.01788

A stripchart plots the two samples; group means are shown as red Xs. We are looking through a heavy 'fog' of variability, trying to discern the difference between $\mu_x = 100$ and $\mu_y = 104.$

y = c(y1, y2);  g = rep(1:2, each=50)
stripchart(y ~ g, ylim=c(0.5,2.5), pch="|")

Because we have two independent samples with no inherent order relationship between them, it is not meaningful to find a sample correlation. Various randomly selected 'pairings' might give correlations anywhere between $\pm 0.95,$ about half of them between $\pm 0.1.]$

A Welch 2-sample t test shows no significance.

t.test(y2,y1)

           Welch Two Sample t-test

data:  y2 and y1
t = 0.81253, df = 97.31, p-value = 0.4185
alternative hypothesis: 
  true difference in means is not equal to 0
95 percent confidence interval:
 -3.433908  8.194846
...