Compare two samples?

Similar questions have been asked but have not managed to get a conclusion from them.

I am comparing two sets of samples, where ratios have been obtained for several analytes per sample. So the values are restricted to be in the [0,1] interval. One set of samples has been used as reference to set the limits for each analyte. In fact, the analytes are generally restricted to be in a range included within this interval (say between 0 and 0.27, or between 0.34 and 0.63, depending on analyte).

I show an example where I have plotted the density plots for both sets of samples (reference in green) and also the QQplot. You can see that the possible ratios for this analyte are restricted to be less than 0.17. The limit was empirically set by the green distribution which is the reference. Using R commands and the sm package to graphically compare distributions

length((X[][,8])
 332
length((X[][,8])
 210
sm.density.compare(c(X[][, 8], X[][, 8]),
group=c(rep(1, length(X[][, 8])),
rep(2, length(X[][,8]))), model="equal", band=FALSE) Distributions seem normal but a normality test says otherwise:

shapiro.test(X[][, 8])

Shapiro-Wilk normality test

data:  X[][, 8]
W = 0.9693, p-value = 1.748e-06

shapiro.test(X[][, 8])

Shapiro-Wilk normality test

data:  X[][, 8]
W = 0.9782, p-value = 0.002443

From the density plot, visually I would say that the two distributions are not different, since they are both within my limits (between 0 and 0.17) and nearly overlapping. In fact the red distribution seems to follow pretty well the reference distribution, which is what I expected.

But if I test this formally, I get that there is a significant shift in the distributions.

t.test(X[][,8],X[][,8]) # regardless of non-normality
# since sample size is rather large

Welch Two Sample t-test

data:  X[][, 8] and X[][,8]
t = 4.7662, df = 470.761, p-value = 2.506e-06
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
0.003906504 0.009387357
sample estimates:
mean of x  mean of y
0.06604217 0.05939524

wilcox.test(X[][,8],X[][,8])

Wilcoxon rank sum test with continuity correction

data:  X[][, 8] and X[][,8]
W = 44455.5, p-value = 6.545e-08
alternative hypothesis: true location shift is not equal to 0

ks.test(X[][,8],X[][,8])

Two-sample Kolmogorov-Smirnov test

data:  X[][, 8] and X[][,8]
D = 0.2584, p-value = 6.951e-08
alternative hypothesis: two-sided

I also used the method proposed by Harrell-Davis in case comparing quantiles would show that in fact they are similar, but also got statistical differences (see http://www.r-bloggers.com/comparing-all-quantiles-of-two-distributions-simultaneously/).

qcomhd(X[][, 8], X[][, 8], q = seq(.1, .9, by=.1))
q  n1  n2      est.1      est.2 est.1_minus_est.2        ci.low      ci.up      p_crit p.value signif
1 0.1 332 210 0.04670665 0.04048838       0.006218272  0.0026910365 0.00928384 0.016666667   0.001    YES
2 0.2 332 210 0.05333198 0.04579936       0.007532620  0.0036450303 0.01091243 0.012500000   0.000    YES
3 0.3 332 210 0.05827195 0.05148210       0.006789846  0.0038079921 0.01072252 0.010000000   0.000    YES
4 0.4 332 210 0.06297303 0.05443425       0.008538777  0.0057801177 0.01134807 0.008333333   0.000    YES
5 0.5 332 210 0.06700634 0.05719850       0.009807839  0.0071635533 0.01236572 0.007142857   0.000    YES
6 0.6 332 210 0.07026849 0.06059111       0.009677378  0.0062359534 0.01242078 0.006250000   0.000    YES
7 0.7 332 210 0.07382629 0.06592428       0.007902005  0.0037463285 0.01219320 0.005555556   0.000    YES
8 0.8 332 210 0.07889714 0.07283984       0.006057305  0.0013907057 0.01080278 0.025000000   0.009    YES
9 0.9 332 210 0.08528227 0.08124071       0.004041560 -0.0004749558 0.00909416 0.050000000   0.079     NO

It seems that I am kind of looking for a method that would say what I want to say, but maybe I am biasing my analysis. Is there a reasonable justification (maybe statistically sound) that I can say to back my idea of both samples being comparable?

Thanks ever so much for your contributions. I did believe that I was going the wrong way. It seems indeed that could be an issue of not asking the right question. Effect sizes can be of great help in this context, asking "how difference are they?". Given that the distributions are not normal, I should probably not use differences in means and move to something like Cliff's delta or non-parametric approaches. Could another question to answer be "how probable is that the second set of data comes from the same population as the first -reference- sample?" Would that be an issue of goodnes of fit test? But I think I am trying to address that with the Kolmogorov-Smirnoff test, aren't I?

• Let's simplify discussion down to just looking at means (though much of it carries over to other things). If the question of interest is "are they reasonably similar", then a hypothesis test answers entirely the wrong question, and at sufficiently large sample size can be relied on to always reject the null, no matter how trivial the differences. Instead, you need to think more about what the actual question of interest is and work on answering that (I expect it will have something to do with effect sizes - such as an estimate of the difference in mean, or whatever else is of interest). Apr 23 '14 at 9:54
• Which is to say "how different are they?" is an interesting question, and probably much nearer to your actual problem. Apr 23 '14 at 9:57