How do I interpret the results of a paired t-test?

Question

I'm new to R and statistical data analysis in general.

Here is what I am trying to do:

I have data from patients before and after a therapy. I want to compare the means of the two measures to see if there is a significant difference to see if the therapy had any effect.

First I checked whether the data is normal distributed by doing a shapiro-wilk test:

differences <- data$SymptomsBefore - data$SymptomsAfter       #calculate the differences
shapiro.test(differences)                                     #do the test 

Output:

Shapiro-Wilk normality test

data:  differences
W = 0.92445, p-value = 0.2878

--> Since the p-value is bigger than 0.05, I can assume that the data is normal distributed.

So now I can do a paired samples t-test. Since I don't know if the effect is positive or negative I choose the two-tailed option.

t.test(data$SymptomsBefore, data$SymptomsAfter, paired = TRUE, alternative = "two")

Here is the result:

    Paired t-test

data:  data$SymptomsBefore and data$SymptomsAfter

t = -2.8939, df = 12, p-value = 0.01348

alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -1.5506404 -0.2185903

sample estimates:
mean of the differences 
             -0.8846154 

So, if I interpret the results correctly, this means there was a significant difference in the means (because the p-value was less than 0.05), meaning the therapy had an effect.
Also, on average the symptoms were -0.8846154 lower than before the therapy.

In a paper I would report that like this: "The results indicate that the therapy resulted in an decrease in Symptoms, t(12) = -2.8939, p = .01348."

Is my interpretation of the R-output correct? Am I missing something? Is there something I still need to check that I didn't think of?

Thank you for your help.

Answer 1

I will consider your Question along with several of the accumulated Comments:

Are data normal? A Shapiro-Wilk test on a sample of size 13 can give you some idea whether the population from which the sample was chosen was uniform.

Below we look at 100,000 simulated samples of size 13, from each of standard uniform, exponential, and normal populations.

• As is to be expected, the Shapiro-Wilk test at the 5% level did reject about 5% of truly normal samples as not consistent with normal data.
• However, only about 10% of samples from a uniform population (mainly symmetrical and almost never with outliers) were rejected as not normal, and
• About 59% of samples from an exponential population (highly skewed and typically with outliers) were rejected as not normal.

The S-W test is one of the best tests of normality, but hardly definitive for samples as small as yours.

set.seed(2020)
pv.u = replicate(10^5, shapiro.test(runif(13))$p.val )
mean(pv.u <= .05)
[1] 0.10436

pv.e = replicate(10^5, shapiro.test(rexp(13))$p.val )
mean(pv.e <= .05)
[1} 0.59062


pv.n = replicate(10^5, shapiro.test(rnorm(13))$p.val )
mean(pv.n <= .05)
[1] 0.04858

Paired t test or Wilcoxon signed rank test? Moreover, it is not the individual observations that need to be nearly normal in order to get reliable results from a paired t test. It is the sample mean that needs to be nearly normal. If a sample of about a dozen differences is roughly symmetrical and without extreme outliers, then one can usually rely on a paired t test to give useful results.

I have only your results from a paired t test. Without your data, I can't look to see for sure what results you would have gotten from a Wilcoxon signed rank test on the differences. Especially with such a small sample, the Wilcoxon test is less powerful (likely to reject when $H_0$ untrue) than a t test.

Working backward from your printout it, I deduce that you must have had $\bar D \approx 0.88$ with standard deviations $S_D \approx 1.1.$ Symmetrical data centered at 0.88 and with SD 1.1 would lead the Wilcoxon test to reject with probability about 73%. So using a Wilcoxon test might not have been a bad choice. Without seeing your actual data, I can only guess.

pv.wilcox = replicate(10^5, wilcox.test(rnorm(13, .88, 1.1))$p.val)
mean(pv.wilcox <= .05)
[1] 0.73003

Your summary. In your summary you can only claim that patients reported lower symptom scores after the period of treatment than before. The improvement might have been either from the treatment or the passage of time. I think it is reasonable to give the mean of difference, the SD of differences, and the number of differences in treatment scores along with the P-value.

Whether this is for a local tech report or to be submitted to a journal for publication, an editor may ask you for a slightly different summary, perhaps including Cohen's D (which can be obtained from sample size, mean and SD). If you feel that a decrease of about 1 (0.88) on your symptom scale amounts to an important improvement for patients, then somewhere in your report you need to explain your scale and say why the average observed difference is of practical importance. If you have no control group, you must say so explicitly.

Note: If you have any data on decrease in symptoms for untreated subjects, it would be useful to mention that, commenting on similarities and differences (demographic and seriousness of condition) of those patients from your treated ones.