3
$\begingroup$

I have 2 samples, one with sample size of 30,000 customers and the other with 150,000. I have to perform a 2 sample t test(on conversion rates of the 2 groups). My question is, will t test in this case be biased towards the smaller sample? If yes, what is the correct approach to perform a test?

$\endgroup$
  • 8
    $\begingroup$ Samples of that size will almost certainly result in statistically significant findings, but the differences may not be of any practical significance. See here for another discussion about this: stats.stackexchange.com/questions/4075/…. What are the actual goals of your analysis too? $\endgroup$ – StatsStudent Apr 22 at 19:24
  • $\begingroup$ the test was to determine which list is better in conversion for emails.1 list was from a prediction model(30,000) and the other, the current list(150,000). We had set up an initial test frame but previous conversion rates(0.05%) and power analysis yielding huge sample sizes for significance, we decided to disregard the framework(our model could not have produced huge sample without lowering the accuracy). Hence, we decided to send the emails to both the lists and compute the results after. We have the conversions now and are trying to establish whether or not the difference is significant $\endgroup$ – Shivam Tiwari Apr 23 at 17:53
  • $\begingroup$ Are the 30,000 predicted a selection of the predictive most likely to respond from the larger list of 150,000? Can there be any overlap? $\endgroup$ – StatsStudent Apr 23 at 18:15
  • $\begingroup$ there were overlaps, we had removed them from the current list of 150,000(so that a customer didn't receive the same email twice). But while computing conversions we did include the overlap in both the lists(for fair comparison). Please note as the test was to compare conversion rates of lists; same email was sent to both the lists $\endgroup$ – Shivam Tiwari Apr 23 at 18:20
10
$\begingroup$

I can hardly imagine any worthwhile effect size that requires such a large sample size to be decently powered. There's no "bias" of having unequal sample sizes$^1$. The only disadvantage is that the power of the test tends to be somewhat limited by the smaller group. For even very small effects, 30,000 observations may confer quite a powerful test.

$^1$ except if you inappropriately use the "equal variance" assumption, in which case the "pooled variance" estimate is more heavily weighted toward the larger group (not toward the smaller as you suggested).

$\endgroup$
8
$\begingroup$

Maybe a couple of examples will help to illustrate some of the issues.

Suppose the two populations are $X \sim \mathsf{Norm}(\mu = 500, \sigma =30)$ and $Y \sim \mathsf{Norm}(\mu = 501, \sigma = 20.)$

If both sample sizes are $150,000,$ then there is sufficient power to detect the small difference in means.

set.seed(422)
x = rnorm(150000, 500, 30)
y = rnorm(150000, 501, 20)
t.test(x, y)

        Welch Two Sample t-test

data:  x and y
t = -10.983, df = 261530, p-value < 2.2e-16
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -1.2042715 -0.8395487
sample estimates:
mean of x mean of y 
 499.9804  501.0023 

If we use only the first 30,000 values in the first sample, results are very nearly the same for most practical purposes.

t.test(x[1:30000], y)

        Welch Two Sample t-test

data:  x[1:30000] and y
t = -6.3728, df = 35463, p-value = 1.879e-10
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -1.5126269 -0.8010336
sample estimates:
mean of x mean of y 
 499.8455  501.0023 

Here is a boxplot of the data used in the second t test (the wider box indicates a larger sample):

enter image description here

Issues of minimal concern:

  • Even though labeled as 'Welch t tests', sample sizes are sufficiently large that these are essentially t tests. Unless the data are very far from normal, we would still detect the small difference in means.

  • The power of the test is heavily dependent on the smaller sample size. But power is not a concern here.

Issues warranting attention:

  • With such large samples in the real world (not the simulation world), one is entitled to wonder whether data are truly simple random samples from their respective populations. Could smaller, more carefully collected samples provide better information?

  • Although we did not do a formal test to confirm that variances differ, it seems clear from the boxplot that they do. In the Welch test, it is OK for variances to differ. But would different variances have important practical implications?

  • Although the null hypothesis that the two population means are equal is soundly rejected with minuscule P-values, it is important to realize that "statistically significant" differences (by whatever definition) are not necessarily differences of practical importance or interest. For what purpose are you taking the effort of check whether means are different? And what do the results of the t test actually contribute to that purpose?

$\endgroup$
  • 1
    $\begingroup$ the test was to determine which list is better in conversion for emails.1 list was from a prediction model(30,000) and the other, the current list(150,000). We had set up an initial test frame but previous conversion rates(0.05%) and power analysis yielding huge sample sizes for significance, we decided to disregard the framework(our model could not have produced huge sample without lowering the accuracy). Hence, we decided to send the emails to both the lists and compute the results after. We have the conversions now and are trying to establish whether or not the difference is significant $\endgroup$ – Shivam Tiwari Apr 23 at 14:35
0
$\begingroup$

I agree with the most that was said so far but I do not completely agree with the satement from @AdamO that "There's no "bias" of having unequal sample sizes".

Unfortunately, we don't know what the purpose of your study is. But let's assume you are interested in gender differences in regard to salary. We know that in population there should be about 50% male and 50% female and hence if you had drawn random samples and if there was MAR (missing at random) we would expect both groups having approximately same sample sizes. To put it differently, if the ratio between sample sizes of both groups is very different than their ratio in the population this can indicate that either the samples are not random (what could cause a bias) or that the missings are not random (what could cause a bias, too). Talking about the gender example I would be surprised if someone would report such big differences in samples sizes (for example: Did more women refuse to answer questions about their salary? Are the women who responded to the question representative or did only those with a high salary answer the question? And so on... Non-random missings would obviously cause a bias here and make the results misleading).

Thus the question that I would ask myself is why the groups have unequal sample sizes. If there is an reasonable answer like "There are more people without heart failure than people with heart failure" the data might be alright. But if you would expect equal sample sizes based on what you know about the groups in the population there might be some bias because the samples/ the missings seem to be not random.

EDIT

Because of the comment from AdamO who says that I use the term "bais" completely wrong I want to add some sources that support my usage of this term and use it in the same way. To sum up, my answer is about a sample bias and bias because of nonrandom missings.

$\endgroup$
  • 1
    $\begingroup$ You seem to be conflating the idea of "bias" and "inefficient design". One is the property of a statistic (in this case the mean difference), the other is a property of a test. The mean difference is never biased no matter how imbalanced the sample. But the power of the test can suffer. $\endgroup$ – AdamO Apr 25 at 15:45
  • 1
    $\begingroup$ @AdamO: I find your comment quite harsh and want to show you some definition of my "fundamentally incorrect understanding of a term". Maybe you simply never learnt this meaning of that term? See here: en.m.wikipedia.org/wiki/Selection_bias $\endgroup$ – stats.and.r Apr 25 at 16:34
  • $\begingroup$ I think you are assuming the OP intended a balanced design. But neither did the OP describe that he/she wanted to obtain a balanced sample, nor is it common or expected in practice. If 150,000 people were invited to participate in a survey but only 30,000 responded, methods have to be used to account for non-response, regardless of whether the sample was balanced or not. There are methods for that (which you don't describe) and you didn't quite articulate that part correctly. So your answer is not correct in general. $\endgroup$ – AdamO Apr 29 at 14:21
  • 1
    $\begingroup$ @AdamO: "I think you are assuming the OP intended a balanced design". I don't and because "we don't know what the purpose of his/her study is" I described a situation where unequal sample size indicates a bias. And in fact my description doesn't assume balanced design either. Please read carefully. $\endgroup$ – stats.and.r Apr 29 at 16:21
  • 1
    $\begingroup$ @AdamO: "If 150,000 people were invited to participate in a survey but only 30,000 responded, methods have to be used to account for non-response". You misunderstand his design. He/she writes that there are 2 samples not that there was one survey with 150,000 people and 30,000 of them responded. That is a big difference. To use your situation: if they sent 150,000 surveys per group and in one all of them answered and in the other condition only 30,000 answered one can not assume that missings where random what can bias the results. $\endgroup$ – stats.and.r Apr 29 at 16:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.