Similar questions have been asked but have not managed to get a conclusion from them.

I am comparing two sets of samples, where ratios have been obtained for several analytes per sample. So the values are restricted to be in the [0,1] interval. One set of samples has been used as reference to set the limits for each analyte. In fact, the analytes are generally restricted to be in a range included within this interval (say between 0 and 0.27, or between 0.34 and 0.63, depending on analyte).

I show an example where I have plotted the density plots for both sets of samples (reference in green) and also the QQplot. You can see that the possible ratios for this analyte are restricted to be less than 0.17. The limit was empirically set by the green distribution which is the reference. Using R commands and the sm package to graphically compare distributions

    [1] 332
    [1] 210
    sm.density.compare(c(X[[44]][, 8], X[[45]][, 8]), 
         group=c(rep(1, length(X[[44]][, 8])), 
         rep(2, length(X[[45]][,8]))), model="equal", band=FALSE)

Density plots. Green is reference sample

Distributions seem normal but a normality test says otherwise:

    shapiro.test(X[[44]][, 8])

            Shapiro-Wilk normality test

    data:  X[[44]][, 8] 
    W = 0.9693, p-value = 1.748e-06

    shapiro.test(X[[45]][, 8])

           Shapiro-Wilk normality test

    data:  X[[45]][, 8] 
    W = 0.9782, p-value = 0.002443

From the density plot, visually I would say that the two distributions are not different, since they are both within my limits (between 0 and 0.17) and nearly overlapping. In fact the red distribution seems to follow pretty well the reference distribution, which is what I expected.

But if I test this formally, I get that there is a significant shift in the distributions.

    t.test(X[[44]][,8],X[[45]][,8]) # regardless of non-normality 
                                  # since sample size is rather large

            Welch Two Sample t-test

    data:  X[[44]][, 8] and X[[45]][,8] 
    t = 4.7662, df = 470.761, p-value = 2.506e-06
    alternative hypothesis: true difference in means is not equal to 0 
    95 percent confidence interval:
     0.003906504 0.009387357 
    sample estimates:
     mean of x  mean of y 
    0.06604217 0.05939524


            Wilcoxon rank sum test with continuity correction

    data:  X[[44]][, 8] and X[[45]][,8] 
    W = 44455.5, p-value = 6.545e-08
    alternative hypothesis: true location shift is not equal to 0


            Two-sample Kolmogorov-Smirnov test

    data:  X[[44]][, 8] and X[[45]][,8] 
    D = 0.2584, p-value = 6.951e-08
    alternative hypothesis: two-sided

I also used the method proposed by Harrell-Davis in case comparing quantiles would show that in fact they are similar, but also got statistical differences (see http://www.r-bloggers.com/comparing-all-quantiles-of-two-distributions-simultaneously/).

    qcomhd(X[[44]][, 8], X[[45]][, 8], q = seq(.1, .9, by=.1))
    q  n1  n2      est.1      est.2 est.1_minus_est.2        ci.low      ci.up      p_crit p.value signif
    1 0.1 332 210 0.04670665 0.04048838       0.006218272  0.0026910365 0.00928384 0.016666667   0.001    YES
    2 0.2 332 210 0.05333198 0.04579936       0.007532620  0.0036450303 0.01091243 0.012500000   0.000    YES
    3 0.3 332 210 0.05827195 0.05148210       0.006789846  0.0038079921 0.01072252 0.010000000   0.000    YES
    4 0.4 332 210 0.06297303 0.05443425       0.008538777  0.0057801177 0.01134807 0.008333333   0.000    YES
    5 0.5 332 210 0.06700634 0.05719850       0.009807839  0.0071635533 0.01236572 0.007142857   0.000    YES
    6 0.6 332 210 0.07026849 0.06059111       0.009677378  0.0062359534 0.01242078 0.006250000   0.000    YES
    7 0.7 332 210 0.07382629 0.06592428       0.007902005  0.0037463285 0.01219320 0.005555556   0.000    YES
    8 0.8 332 210 0.07889714 0.07283984       0.006057305  0.0013907057 0.01080278 0.025000000   0.009    YES
    9 0.9 332 210 0.08528227 0.08124071       0.004041560 -0.0004749558 0.00909416 0.050000000   0.079     NO

It seems that I am kind of looking for a method that would say what I want to say, but maybe I am biasing my analysis. Is there a reasonable justification (maybe statistically sound) that I can say to back my idea of both samples being comparable?

Thanks ever so much for your contributions. I did believe that I was going the wrong way. It seems indeed that could be an issue of not asking the right question. Effect sizes can be of great help in this context, asking "how difference are they?". Given that the distributions are not normal, I should probably not use differences in means and move to something like Cliff's delta or non-parametric approaches. Could another question to answer be "how probable is that the second set of data comes from the same population as the first -reference- sample?" Would that be an issue of goodnes of fit test? But I think I am trying to address that with the Kolmogorov-Smirnoff test, aren't I?

  • 2
    $\begingroup$ Let's simplify discussion down to just looking at means (though much of it carries over to other things). If the question of interest is "are they reasonably similar", then a hypothesis test answers entirely the wrong question, and at sufficiently large sample size can be relied on to always reject the null, no matter how trivial the differences. Instead, you need to think more about what the actual question of interest is and work on answering that (I expect it will have something to do with effect sizes - such as an estimate of the difference in mean, or whatever else is of interest). $\endgroup$
    – Glen_b
    Apr 23 '14 at 9:54
  • $\begingroup$ Which is to say "how different are they?" is an interesting question, and probably much nearer to your actual problem. $\endgroup$
    – Glen_b
    Apr 23 '14 at 9:57
  • $\begingroup$ You can regain control over your question, David, by merging your accounts: please visit stats.stackexchange.com/help/merging-accounts. $\endgroup$
    – whuber
    Apr 23 '14 at 15:42
  • $\begingroup$ I think I have managed to merge my accounts, thanks whuber. Though I now seem to have a different account that the initially used for posting. My apologies for that. Regarding the matter above, you both (Glen_b and JupiterM104) mention difference in means to check effect size, but is it advisable to use mean values even when the tests say the samples are not normal? $\endgroup$
    – user44388
    Apr 25 '14 at 10:37

It certainly sounds initially like you should be careful not to bias your analysis, because you provide an example (the green and red distributions) where two distributions are clearly not the same, otherwise they would overlap exactly, and then seem to want to prove they are the same.

How different they are/in what way they differ though is another question, and you may well conclude that while they are statistically 'significantly' different, this difference is not important for your system.

You mention the sample size is large, and essentially with a large enough sample size standard tests like a t-test will eventually show a 'significant' difference no matter how small. Therefore, given the two distributions certainly look like they have different means, it may be that you just have enough power in your test, because of the large sample size, to detect this. What is more important though is to look at the difference between the sample means and decide based on your system whether this is important or not. If not then you may well be able to just treat the two samples as the same, but this will depend on system specific knowledge, and I would always be careful doing this if you have a large sample size and there is a consistent, statistically detectable difference.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.