# t-test in large, unequal sample size + additional info such as effect size

I have 2 samples, named "1" and "3":

• (1) n = 1282, mean = 14.77, sd = 8.27
• (3) n = 5360, mean = 15.88, sd = 8.55

Their distribution look:

ggplot(creu, aes(x=creu$V7, fill=creu$V1)) +
+   geom_histogram(binwidth=3) + theme(legend.position="none")


I want to show the histogram in a research paper to suggest a possible tendency in group 1 to have lower values than group 3.

Data isn't completely independent so my plan was to show the histogram with a t.test:

t.test(V7 ~ V1, data=creu)

Welch Two Sample t-test

data:  V7 by V1
t = -4.3086, df = 1986.994, p-value = 1.723e-05
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-1.6240990 -0.6080744
sample estimates:
mean in group 1 mean in group 3
14.77288        15.88897


I'm aware that significance is highly influenced by the sample size, but considering that this is not the core of the research, just an hypothesis, I don't want to put too much weight on it, simply complement the plot with some statistical data.

• given this conditions (unequal sample size, but very large dataset, not independent and their representation). Is this test the best option?
• Should I add additional information such as effect size (e.g. cohen's d)?

EDIT: Why I said they aren't independent? I took 82 models based on 82 datasets and test each model on the remaining 81 datasets (all but the one used to train it). I expect to see a bad performance in those tests, but for ~19% of them (n=1282) I see a good performance. I compute a "similarity distance" between each pair (dataset-of-the-model & dataset-of-the-test) to hypothesize that maybe those that obtain a good performance are closer, and that's why they can predict reasonable good. I say they aren't independent because maybe model X based on dataset X performed well on dataset A and D but bad on dataset B, C and E. So somehow model X is "behind" both groups.

• I was surprised to see you propose using a histogram and t-test to deal that data that "isn't completely independent." Neither of these procedures will reveal or deal correctly with a lack of independence. Could you therefore explain the sense in which the data are not independent?
– whuber
Jul 27, 2015 at 15:51
• @whuber I tried to explain it below. Any suggestion is really welcome. Thanks Jul 27, 2015 at 16:03