1
$\begingroup$

I would like to check with CrossValidated community whether I interpret the assumptions that are needed to be tested for paired t-test correctly. My questions primarily arise from working with a grouped data:

My set up: I measured the Self-Efficacy and Anxiety of students in Math, Physics and Biology classes (so for each subject each student get 2 scores: one score to measure Self-efficacy and the second one as a measure of Anxiety). Self Efficacy and Anxiety was measured at the beginning and at the end of the course. I want to see whether there was a significant difference in Self Efficacy and Anxiety scores at teh beginning and end of the course for each subject (So to be clear, I am not comparing data between the subjects or I am not comparing Anxiety vs Self Efficacy scores: I am just interested to see it within each subject).

so for instance: for Physics:

  • is there a significant difference in the Self-efficacy score at the beginning and end of the course?
  • is there a significant difference in the Anxiety score at the beginning and end of the course?

and the same questions for the rest of the subjects.

So after reading about the topic, I found the following assumptions that have to be made:

  1. Assumption 1: The dependent variable must be continuous (interval/ratio).
  2. Assumption 2: The two groups are paired
  3. Assumption 3: No significant outliers in the difference between the two related groups
  4. Assumption 4: The difference between the two related treatment groups should be normally distributed.

I have the following questions:

Assumption 1: the scores were obtained using the Likert scale questionary, which is not a truly continuous scale. I wonder if you can suggest any references that discuss the issue of whether it is acceptable to treat the scale ( i have a 7 point Likert scale) as a continuous variable.

Assumption 3: this one is a bit tricky: I get different opinions on whether to remove the extreme variable from the data set or not. I personally would like to keep them in, but I am not exactly sure how to test that keeping the outliers in how will affect the paired t-test results. In addition, do I treat each Subject and each category (anxiety and Self -Efficacy) as a separate group (i.e., I will have 3subject x 2 score types = 6 groups), so that I will look for outliers in each group separately, or do I just use 3 groups ( ie 3 subjects).

Assumption 4: the same question for testing normality do I do 6 tests for each group or not.

$\endgroup$

2 Answers 2

0
$\begingroup$

Your four rules are specifically for a paired t test. If you feel comfortable pretending that Likert-7 data are numerical rather than ordinal, then you should feel comfortable taking differences in Likert-7 Before and After scores as numerical. You must know that this assumption is somewhat controversial. One question to ask yourself before checking this box, is whether you believe a difference between 2 and 3 on the seven point scale is of the same importance as a difference between 6 and 7 (and similar comparisons).

I also would be very reluctant to prune responses that seem to be outliers. Especially reluctant for differences between two Likert-7 scores which can hardly get really far positive or negative. You are wondering if people change, so you shouldn't be alarmed if some change noticeably more than others.

If the number of subjects is small and the sample distribution of differences is very far from normal in shape, it seems especially risky to pretend that the data are normal. Because a paired t test is just a one-sample t test on differences, I would judge normality in terms of differences rather then in terms of before and after Likert scores.

In order to give some examples of tests, I will use R to generate $n = 75$ differences d.

set.seed(403)
x1 = sample(1:7, 75, rep=T, p = c(1,1,2, 2,2,3,1))
x2 = sample(1:7, 75, rep=T, p = c(1,2,2, 3,3,1,1)) 
d = x2 - x1

summary(d)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
-5.0000 -2.0000 -1.0000 -0.6267  1.0000  5.0000 

boxplot(d, horizontal=T, col="skyblue2")

enter image description here

I don't happen to have any outliers, but there is moderate right skewness. Even so, a Shapiro-Wild test (narrowly) fails to reject these differences as having come from a normal population.

shapiro.test(d)

        Shapiro-Wilk normality test

data:  d
W = 0.96904, p-value = 0.06298

A one-sample t test on the differences strongly rejects the null hypothesis $H_0: \mu_d = 0$ in favor of $H_a: \mu_d \ne 0$ with P-value about $0.02 < 0.05 = 5\%.$

t.test(d)

        One Sample t-test

data:  d
t = -2.3838, df = 74, p-value = 0.0197
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 -1.1504726 -0.1028607
sample estimates:
  mean of x 
 -0.6266667 

Of course, I have no idea whether your real data are much like my simulated data. So, I will go looking for trouble that is not really evident in my data. If you want to stay more with the spirit of ordinal categorical data and to avoid an assumption that differences are normally distributed, you could use a paired nonparametric Wilcoxon signed rank test, which finds a significant difference in location (medians) before vs. after the course.

wilcox.test(x1, x2, paired=T)

        Wilcoxon signed rank test with continuity correction

data:  x1 and x2
V = 1394, p-value = 0.01693
alternative hypothesis: true location shift is not equal to 0

Another possibility is to use the paired t statistic as 'metric' in a nonparametric permutation test. We believe that this 'metric' is a meaningful way to measure differences before vs. after, but we do not assume data are normal or that the "t statistic" has any relationship with a Student t distribution.

A permutation test: According to the earlier t test, we know that the observed value t.obs of the t statistic is $-2.3838.$ Then we randomly permute the signs of the differences and find a value t.perm of the sign-permuted data. By doing this many times we can get an idea of the permutation distribution of the t statistic, and then check to see where t.obs within that distribution.

t.obs = t.test(d)$stat
set.seed(2021)
B = 10000;  t.prm = numeric(B)
for (i in 1:B) {
 d.prm = d * sample(c(-1,1), 75, rep=T)
 t.prm[i] = t.test(d.prm)$stat 
}
mean(abs(t.prm) >= abs(t.obs))
[1] 0.024    # P-value of permutation test

hdr = "Simulated Permutation Distn of t Statistic"
hist(t.prm, prob=T, col="skyblue2", main=hdr)
 abline(v = c(t.obs,-t.obs), lwd=2, lty="dotted", col="red")

The (approximate) P-value of the two-tailed permutation test is the sum of the areas outside the vertical dotted lines in both tails of the permutation distribution. For my fake dat the P-value is not much different from the P-value of the paired t test, but for your data there may be more of a difference.

For the permutation test we are assuming the possibility that subjects might change up or down on the Likert scale during the course, and that the data were random samples from the specified testing procedures. We make no assumption about normality, but the computations to get the permuted t statistic at each iteration assume that the differences can be regarded as numeric to the extent that the required means and standard deviations can be computed. Other metrics involving less arithmetic might be used in place of the t statistic.

enter image description here

Note: There many possible choices for the 'metric' in a permutation test. We might have chosen the mean of the differences, with about the same result. (But for inherently discrete data, such as yours, it would not be a good idea to take the median of the differences because there are too few possible median values to make a useful permutation distribution.)

a.obs = mean(d)
set.seed(1234)
B = 10000;  a.prm = numeric(B)
for (i in 1:B) {
 a.prm[i] = mean(d * sample(c(-1,1), 75, rep=T))  
}
mean(abs(a.prm) >= abs(a.obs))
[1] 0.0193  # P-val
length(unique(d))
[1] 11      # highly discrete
length(unique(a.prm))
[1] 83      # enough
$\endgroup$
3
  • 1
    $\begingroup$ Were the design a parallel group design, the proportional odds model (a generalization of the Wilcoxon-Mann-Whitney two-sample rank sum test) would have been an ideal choice. What do we know about the use of a random effects proportional odds model for the paired data situation? $\endgroup$ Apr 4, 2021 at 13:18
  • $\begingroup$ @FrankHarrell. Thanks. Not aware of that test. Illustration or link welcome $\endgroup$
    – BruceET
    Apr 4, 2021 at 18:30
  • $\begingroup$ I have this worked out at hbiostat.org/bbr/nonpar#sec-nonpar-pairmodel where you'll see 2 methods of getting accurate p-values while handling things much more generally than with a Wilcoxon test: robust covariance correction for within-person correlation and a random effects model. To get accurate results the example shows that you have to pay attention to the numerical integration method chosen to handle the random effects. $\endgroup$ Dec 29, 2023 at 9:24
0
$\begingroup$

You have two research questions, and conducting two t-tests will increase your risk of a Type I error - that you've made a mistake when rejecting a null hypothesis. You must consider this risk and either adjust the p-values of the t-tests (e.g., Bonferroni adjustment) or conduct a test that incorporates two outcomes (e.g., multivariate ANOVA - "multivariate" meaning multiple outcomes).

Since your data is ordinal, I would investigate multivariate non-parametric tests instead of t-tests and ANOVAs. I don't think your data meets the assumptions of these two models: Are the measures normally distributed (t-test)? Would the residuals (estimation / prediction errors) be normally distributed (ANOVA)? I think your data would violate these two assumptions.

I would think a chi-squared analysis of each outcome with an adjustment on the p-value for multiple tests would be your best choice, unless you can find a more complex multivariate non-parametric model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.