5
$\begingroup$

Consider the distribution shown in the below histogram:

enter image description here

I have computed a Welch's t-test for a difference in means between these two groups, as well as a Kruskall-Wallis test to see whether these two groups come from the same distribution. Both test statistics were statistically significant at p < 0.001. Looking at the histogram below, these conclusions (rejections of the null) seem reasonable.

However, the number of observations in group A is about 1500, whereas the number of observations in B is about 400. I specifically chose Welch's t-test because it makes no assumption about the number of samples in each group. While I want to go with the results of that test, I can't help but view the distribution below and think, "Well, group A had many more opportunities to have higher Days between T and E," in the sense that, perhaps if I had more observations for B, the two distributions would begin to look the same (i.e. both would still be positively skewed, but group B would "bulk up" on the skewed side to approximately match group A).

Is the concern valid that group B really might come from the same distribution at A, if B had more samples, given that there's significant statistical evidence against what I just said? I can't obtain more observations for group B.

The other part of me thinks the test statistics are right: the relative proportion of observations with high Days between T and E for group B is much less than that of group A. That is, group A, on average, does tend to have a higher median/mean Days between T and E, and a different variance.

Edit:

So I undersampled from group B to create a new dataset with an equal number of observations from both groups and created the same type of histogram for these "new" data:

blah

The new Welch's t-stat was even more significant than the first. So this undersampling to make the sample sizes equal seems to corroborate the idea that the number of days between targeting and engagement for these two groups is, in fact, different.

$\endgroup$
2
$\begingroup$

The answer to your bolded question is no. (And you don't need to upsample or downsample anything.)

If A and B are both random samples from their respective populations the sample cdf will converge to the population cdf; that doesn't creep up the axis with sample size.

If A and B are both random samples from their respective populations, taking a larger sample from B wouldn't tend to move the distribution along the line (except as random variation allows there could be a little movement in either direction); as you sample more you just get a more precise estimate of the shape of B's distribution. The whole thing would be higher, not just the upper part.

You can adjust for A's larger count by scaling your histograms to have area 1. That will make the picture of the shapes more constant as sample sizes might change (in vanilla R, hist with freq=FALSE does that).

If the distribution of B didn't have finite mean, then its possible that larger samples could possibly tend to look more extreme than smaller ones, if you compare something like the sample mean (rather than the overall distribution), but then your t-test wouldn't be valid either. This would require a very heavy upper tail though.


Scaling the heights to make them (roughly) close to comparable on the left side (below about 80), we have:

Histogram of both data sets with larger one scaled down

-- and now we can see that although they're not so much different (in percentage terms) on the left side, A's right tail (above 80-odd) is still much higher.

It's possible to do this rescaling "by eye", without having to physically do it. Given the large samples, this means that it's immediately obvious by looking at your first plot that A tends to be bigger than B.

$\endgroup$
  • $\begingroup$ +1 "taking a larger sample from B wouldn't tend to move the distribution along the line", in other words, the mean is an unbiased and consistent estimator $\endgroup$ – Rose Hartman Mar 2 '17 at 6:05
  • 2
    $\begingroup$ @Rose when it exists that's true; I mentioned the cdf in part because it's unbiased and its convergence is more general (the population cdf always exists and the sample cdf converges to it almost surely), while the mean need not be any finite value (e.g consider the Cauchy or a Pareto with $\alpha\leq 1$) in the case of the Cauchy for example there isn't a mean to converge to. $\endgroup$ – Glen_b Mar 2 '17 at 7:11
  • 2
    $\begingroup$ Yes, exactly. I just wanted to throw in the alternate vocab words to help folks connect the dots in case they found your description familiar but weren't sure if it was the same phenomenon. $\endgroup$ – Rose Hartman Mar 2 '17 at 7:23
  • $\begingroup$ That's my dilemma, though. For B, if the whole thing were higher (i.e. if there were more observations in each of those buckets), my eyes see that B distribution "rising" up to look very similar to that of A. They already look fairly symmetric, IMO, in the first graph, outside of A's being "higher" due to its larger number of observations. $\endgroup$ – blacksite Mar 2 '17 at 14:16
  • 1
    $\begingroup$ No, A clearly has a much heavier upper tail than B. I'll make some additions explaining why that's clear from the first plot. This is something that would be obvious if you scale to unit area (as suggested above in place of up-sampling B or down-sampling A) $\endgroup$ – Glen_b Mar 2 '17 at 14:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.