What are the problems if in a AB test the control group is not big enough?

Question

I've been asked to analyze some data but I'm quite new in statistic. I want to compare the means of a continuous parameter in two independent population A and B: A can be seen as the treatment group whereas B can be seen as the control group. I have two samples, for population A and for population B, whose dimensions are 79 and 15 respectively. I would like to do a hypothesis test with null hypothesis $H_0: \mu_A=\mu_B$ (I haven't seen the data yet, but I expect to reject $H_0$). Obviously I will use the sample mean $\overline{X}_A$ as estimator for $\mu_A$ and $\overline{X}_B$ as estimator for $\mu_B$, $H_0$ can be written as $H_0: \mu_A-\mu_B=0$, so the estimator I have to use for $\mu_A-\mu_B$ is $\overline{X}_A-\overline{X}_B$

1. The variance of $\overline{X}_A$ and of $\overline{X}_B$ are unkown, so I have to use the sample standard deviations $s_A$ and $s_B$, hence I extimate the standard deviation of $\overline{X}_A-\overline{X}_B$ with $\sqrt{s_A^2+s_B^2}$. I use a t-test with $$t=\frac{\overline{X}_A-\overline{X}_B}{\sqrt{s_A^2+s_B^2}}.$$ The question is: why do I use a t-statistic? Is it correct? I read that, when the standard deviation is unkown, the t-statistic is appropriate provided that $\overline{X}_A-\overline{X}_B$ is normally distributed... But I have no idea if $\overline{X}_A-\overline{X}_B$ is normally distributed.

2. Is it ok the proportion between the dimensions of sample A and B? We have 79 vs 15 so 84% vs 16%. What are the problems I can run into if the proportion is not appropriate?

3. For example, given that using a t-test as above is correct, if I fix the probability of a I type error of 5%, can I be sure that it is a real 5% or is it bigger? What about the power of the test? I expect to reject $H_0$, so I hope my test is powerful enough in order to do that.

4. If this partition (84|16) is not good, is increasing the dimension of the control group the only way to make a more significant statistic?

Any suggestion or good reference will be more than welcome.

Answer 1

The basic assumptions of the two-sample t-test are that the distributions of the populations are approximately normal and that their variances are the same. There is no requirement that the sizes of the groups be the same, however if the variances are different, it is only possible to proceed if the sample sizes are roughly equal, which is not the case here.

So, in your case, provided that the variances of the two groups are approximately equal, and both samples are approximately normally distributed, you can perform a t-test.

To test for the equality of variance you could use Levine’s test (or an F test)

To test for normality formally you could use the Shapiro-Wilk test (or the Anderson-Darling test or the Kolmogorov-Smirnov test), but first it is a good idea to assess normality informally using a QQ plot and histogram. It is OK that the groups are of unequal size – provided that the variances are approximately equal, although the test would certainly have more power if the smaller group was bigger - in fact, power is maximised when the group sizes are the same.

You ask “why do I use a t-statistic” ? I’m not sure how to answer that, other than saying “because you are doing a t-test and the sampling distribution of the test statistic follows a t-distribution with the relevant degrees of freedom, under the null hypothesis.”

If your data are obviously not normally distributed (and can’t be transformed) then you can use a non-parametric two-sample test such as the Mann-Whitney U test (Wilcoxon two-sample test) If the variances are obviously unequal (but the data are plausibly normal) you can consider other tests such as the Welch-Satterthwaite T test