Two sample t test with equal variances hypothesis

Question

Given a trial to test a new drug. There are 20 volunteers with some disease. The volunteers are split into two groups (10 volunteers in each) randomly: the treatment group and the control group. Volunteers in the treatment group receive the new drug, volunteers in the control group receive placebo (pills that looks like a drug but do not have active substance). You conclude that new drug is effective if people who take the drug will recover faster (on average) than people in the control group. If your drug is effective, you will invest in its production, otherwise you will look for another drug. Assume that you obtained the following data (disease duration in days)

c​ontrol group	treatment group
6	7
7	6
7	6
5	5
7	5
8	6
8	7
7	5
7	5
7	8

Would you invest into production of this drug?

For the above problem formulation, I am using 2 sample unpaired t test with equal variance and the following hypothesis:

$$H_0: \mu_{X} = \mu_{Y}$$ $$H_1: \mu_{X} > \mu_{Y}$$

where the population parameters $\mu_{X}$ and $\mu_{Y}$ representing average recovery duration for control and treatment group. Thus when the significance level is 0.05, the null hypothesis can be rejected in favor of alternate hypothesis after getting the t-statistic and p value according to the test.

Could you please confirm if my approach to the above problem formulation is correct? Also I have taken equal variances as sample sizes are same and are independent and the sample standard deviation is approximately same and the alternative hypothesis as upper one tailed.

Answer 1

I have put your data into R and looked at numerical and graphical summaries.

ctrl = c(6,7,7, 5,7,8,8, 7,7,7)
trtm = c(7,6,6, 5,5,6,7, 5,5,8)

summary(ctrl); sd(ctrl)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    5.0     7.0     7.0     6.9     7.0     8.0 
[1] 0.875595  # sample SD
summary(trtm); sd(trtm)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   5.00    5.00    6.00    6.00    6.75    8.00 
[1] 1.054093

stripchart(list(ctrl,trtm), meth="stack", ylim=c(.5,2.7), pch=20)

First impressions are the Treatment counts may tend to be smaller than Control counts. However, the average improvement is a bit less than one day for a malady that seems to last about a week without treatment. So, at the levels of drug administered in this small study the effect is not huge even if it is real.

Also, I am always skeptical when equal variances are assumed without a good prior rationale. Unless, you are somehow sure ahead of time that the treatment can do no harm, I would do a two-sided test.

A two sided Welch two-sample t test is significant at the 10% level, but not at the 5% level. So I would say the data suggest the Treatment may be effective. These are very small groups for a clinical trial, so I am not surprised that there is only weak evidence.

t.test(ctrl, trtm)

        Welch Two Sample t-test

data:  ctrl and trtm
t = 2.0769, df = 17.414, p-value = 0.05291
alternative hypothesis: 
 true difference in means is not equal to 0
95 percent confidence interval:
 -0.01260024  1.81260024
sample estimates:
mean of x mean of y 
      6.9       6.0 

Your background information says, "If your drug is effective, you will invest in its production, otherwise you will look for another drug. Assume that you obtained the following data (disease duration in days)." So it seems this is a preliminary feasibility study to see if the drug may have promise.

As you say, if you do a one-sided, pooled t-test, you can get a P-value below 5%.

t.test(ctrl, trtm, var.eq=T, alt = "g")

        Two Sample t-test

data:  ctrl and trtm
t = 2.0769, df = 18, p-value = 0.0262
alternative hypothesis: 
 true difference in means is greater than 0
95 percent confidence interval:
 0.1485724       Inf
sample estimates:
mean of x mean of y 
      6.9       6.0 

Just from the information in your Question, I am skeptical both about doing a one-sided test and about pooling. So this 'significance' barely at the 5% level seems to me more like P-hacking than like good statistical practice.

---

Finally, I will mention that if one were planning in advance to have an 90% or 90% probability of detecting an effect of about one day with standard deviations about one day, a 'power and sample size' procedure (this one from Minitab) shows that one should use about 20 subjects in each group instead of about 10. So the lack of clear cut significance should hardly have been a surprise. (Of course, power and sample size information depends on knowing effect size and standard deviations in advance. But in my experience trials of such a small size as this one are quite rare.)

Power and Sample Size 

2-Sample t Test

Testing mean 1 = mean 2 (versus ≠)
Calculating power for mean 1 = mean 2 + difference
α = 0.05  Assumed standard deviation = 1

            Sample  Target
Difference    Size   Power  Actual Power
         1      17     0.8      0.807037
         1      23     0.9      0.912498

The sample size is for each group.

Addendum per Comments: Six tests with P-values. Best practice is to decide before you see the data which one test meets assumptions and can tell you what you want to know. [Running multiple tests and 'cherry picking' the one with the smallest P-value is not appropriate.]

t.test(ctrl, trtm)$p.val
[1] 0.05290571   # Welch two-sided (AS ABOVE)
t.test(ctrl, trtm, alt="gr")$p.val
[1] 0.02645285   # Welch one-sided (P-val half) 
t.test(ctrl, trtm, alt="less")$p.val
[1] 0.9735471    # Welch one-sided (wrong side)

t.test(ctrl, trtm, var.eq=T)$p.val
[1] 0.05240201   # Pooled two-sided (slightly smaller than W)
t.test(ctrl, trtm, var.eq=T, alt="gr")$p.val
[1] 0.026201     # Pooled one-sided (P-val half, AS ABOVE)
t.test(ctrl, trtm, var.eq=T, alt="less")$p.val
[1] 0.973799     # Pooled one-sided (wrong side)