Consider the distribution shown in the below histogram:
I have computed a Welch's t-test for a difference in means between these two groups, as well as a Kruskall-Wallis test to see whether these two groups come from the same distribution. Both test statistics were statistically significant at p < 0.001. Looking at the histogram below, these conclusions (rejections of the null) seem reasonable.
However, the number of observations in group A is about 1500, whereas the number of observations in B is about 400. I specifically chose Welch's t-test because it makes no assumption about the number of samples in each group. While I want to go with the results of that test, I can't help but view the distribution below and think, "Well, group A had many more opportunities to have higher
Days between T and E," in the sense that, perhaps if I had more observations for B, the two distributions would begin to look the same (i.e. both would still be positively skewed, but group B would "bulk up" on the skewed side to approximately match group A).
Is the concern valid that group B really might come from the same distribution at A, if B had more samples, given that there's significant statistical evidence against what I just said? I can't obtain more observations for group B.
The other part of me thinks the test statistics are right: the relative proportion of observations with high
Days between T and E for group B is much less than that of group A. That is, group A, on average, does tend to have a higher median/mean
Days between T and E, and a different variance.
So I undersampled from group B to create a new dataset with an equal number of observations from both groups and created the same type of histogram for these "new" data:
The new Welch's t-stat was even more significant than the first. So this undersampling to make the sample sizes equal seems to corroborate the idea that the number of days between targeting and engagement for these two groups is, in fact, different.