I am currently assessing whether or not a location-shift can be assumed in non-parametric comparisons to be able to formulate the rejection of the null hypothesis in other terms than the probabilistic index. To do so, I center my data and execute a pairwise two-sample Kolmogorov-Smirnov test on each pair. The data is in some cases unbalanced and not-normally distributed.

Currently I am using this function (in R):

pairwise.ks.test<-function (x, g, p.adjust.method = p.adjust.methods, alternative="two.sided",centered=T,...) 
    p.adjust.method <- match.arg(p.adjust.method)
    DNAME <- paste(deparse(substitute(x)), "and", deparse(substitute(g)))
    g <- factor(g)
    METHOD <- if (centered) 
    "Pairwise KS test on centered data"
    else "Paiwise KS test "
    compare.levels <- function(i, j) {
        xi <- x[as.integer(g) == i]
        xj <- x[as.integer(g) == j]
        ks.test(xi, xj, alternative=alternative, ...)$p.value
        xi <- x[as.integer(g) == i]-mean(x[as.integer(g) == i],na.rm=T)
        xj <- x[as.integer(g) == j]-mean(x[as.integer(g) == j],na.rm=T)
        ks.test(xi, xj, alternative=alternative, ...)$p.value

     PVAL <- pairwise.table(compare.levels.centered, levels(g), p.adjust.method)
else PVAL <- pairwise.table(compare.levels, levels(g), p.adjust.method)
ans <- list(method = METHOD, data.name = DNAME, p.value = PVAL, 
    p.adjust.method = p.adjust.method)
class(ans) <- "pairwise.htest"

And at the moment I apply it on my list of datasets without p-value correction:


My set has ties and therefore an exact p-value cannot be calculated (hence exact=F). As I only want to assess a possible location shift for each pairwise comparison to be able to formulate rejection of $H_0$ in terms of medians or means in Holm-corrected pairwise Wilcoxon-rank sum tests, should I also apply a (Holm?) p-value correction to assess multiple location shifts?

  • $\begingroup$ I'm not sure I understand - doesn't centering remove the very effect you're testing for? $\endgroup$ – Glen_b Dec 11 '13 at 0:24
  • $\begingroup$ @Glen_b I center the data to see if there is a location shift: after centering I can use a two-sample KS test to see if data CDFs differ significantly. If they don't that means that there was a location shift (e.g by subtracting a location parameter the CDF's are equal). $\endgroup$ – FM Kerckhof Dec 11 '13 at 11:28
  • $\begingroup$ "If they don't that means that there was a location shift" -- no, it doesn't necessarily mean that. Unless there's something I missed, your logic is faulty in several ways at once. $\endgroup$ – Glen_b Dec 11 '13 at 14:38
  • $\begingroup$ I'm sorry, @Glen_b but I do not see my faulty logic here. $H_0$ is that there is a location shift. i.e. $H0$: $CDF_1$=$CDF_2$ - $\Delta$. If after substracting a location parameter (either median or mean, doesn't really matter under $H_0$) under $H_0$ the CDFs should be the same. Hence they will not be significantly different. Hence you cannot reject $H_0$ in favor of $H_1$. Hence we withold equality of centered CDFs. Hence the location shift assumption holds. Or am I missing something here? $\endgroup$ – FM Kerckhof Dec 12 '13 at 16:17
  • $\begingroup$ Here's a few problems to get you started: (1) failure to reject doesn't mean $H_0$ (and the other assumptions) are true. That is, if you assume a location shift, subtract an estimate of one, and fail to reject a goodness of fit test, it doesn't lead to the conclusion that it was a location shift, it may be only that you lacked power to pick up any other difference. (2) The KS is based on a completely specified distribution. It doesn't have the desired properties when you do what you did. Your nominal p-values don't take account of what you did to the data; they're less likely to reject. $\endgroup$ – Glen_b Dec 13 '13 at 3:14

It looks like a "general" multiple testing problem to me, since you’re assessing location shift for each of the pairs of datasets? It may lead to a bigger than alpha family-wise false rejection rate and thus a smaller than alpha familywise chance to incorrectly make an interpretation on the mean/median instead of the probabilistic index, which is of course no problem (PI interpretation is of course okay for location shift models also) but may not be what you want to achieve here.

On the other hand, if you choose to apply a multiple testing correction, there may (will) be a tradeoff with the power and thus correcting for the multiple testing may lead you to falsely make interpretations on the mean/median while you should actually be making them on the probabilistic index... I think the easy and safe thing to do here is not apply a multiple testing correction.

  • $\begingroup$ Thanks, @Matt.Vn this was exactly my reasoning. I don't want to loose too much power whilst still being able to express rejection of the $H_0$ in terms of location parameters as the PI is a challenging thing for people to deal with. $\endgroup$ – FM Kerckhof Dec 11 '13 at 11:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.