help interpreting t-test and p-value - or help me finding a good statistical test

Question

I have two samples

background

and

input

these are the histograms:

sample sizes are well above the 1000s, so the sample variance of the mean is pretty much low.

I'm trying to have a statistic that tells me that these two distributions are the same. I'm trying with t-test and Mann-whitney U but both have VERY low pvalues (10^-100 and lower).

I understand that this is because the sample sizes are large and the sample mean can be considered separated even if very near (because of very small sample variance), but here lies the core of my problem.

I don't have a background in statistics (physics major) so forgive me if I use wrong terms, correct me please.

Answer 1

Firstly, whether you want them to or not, the two distributions do look different. You may wish to consider why you want to find a test statistic that gives you the answer you want, rather than the answer that reflects the data you have!

The two-sample Kolmogorov-Smirnov test (https://en.wikipedia.org/wiki/Kolmogorov-Smirnov_test) is more appropriate than either the t or Mann-Whitney test to identify general differences in distributions. As you have identified, a large sample size will give you power to detect a small difference in these two distributions and I fully expect the KS test will also be highly significant.

Although they are different, their shapes look very similar. It's as if they were sampled from the same underlying distribution, except that some weighting has been applied so that 'input' is progressively more likely to draw from the higher end than the lower end. 'input' is therefore more likely to have higher values than 'background' - this is exactly what the Mann-Whitney test is detecting.

If you want to make the argument that they appear to be derived from the same distribution except for a re-weighting, then as far as I'm aware there's no out-the-box test for this.