Is a p-value correction necessary for assessing pairwise location shifts on centered data?

Question

I am currently assessing whether or not a location-shift can be assumed in non-parametric comparisons to be able to formulate the rejection of the null hypothesis in other terms than the probabilistic index. To do so, I center my data and execute a pairwise two-sample Kolmogorov-Smirnov test on each pair. The data is in some cases unbalanced and not-normally distributed.

Currently I am using this function (in R):

pairwise.ks.test<-function (x, g, p.adjust.method = p.adjust.methods, alternative="two.sided",centered=T,...) 
{
    p.adjust.method <- match.arg(p.adjust.method)
    DNAME <- paste(deparse(substitute(x)), "and", deparse(substitute(g)))
    g <- factor(g)
    METHOD <- if (centered) 
    "Pairwise KS test on centered data"
    else "Paiwise KS test "
    compare.levels <- function(i, j) {
        xi <- x[as.integer(g) == i]
        xj <- x[as.integer(g) == j]
        ks.test(xi, xj, alternative=alternative, ...)$p.value
        }
        compare.levels.centered<-function(i,j)
        {
        xi <- x[as.integer(g) == i]-mean(x[as.integer(g) == i],na.rm=T)
        xj <- x[as.integer(g) == j]-mean(x[as.integer(g) == j],na.rm=T)
        ks.test(xi, xj, alternative=alternative, ...)$p.value
    }

if(centered)
     PVAL <- pairwise.table(compare.levels.centered, levels(g), p.adjust.method)
else PVAL <- pairwise.table(compare.levels, levels(g), p.adjust.method)
ans <- list(method = METHOD, data.name = DNAME, p.value = PVAL, 
    p.adjust.method = p.adjust.method)
class(ans) <- "pairwise.htest"
ans
}

And at the moment I apply it on my list of datasets without p-value correction:

lapply(datalist,function(x)pairwise.ks.test(x$value,x$trt,p.adjust.method="none",alternative="two.sided",centered=T,exact=F))

My set has ties and therefore an exact p-value cannot be calculated (hence exact=F). As I only want to assess a possible location shift for each pairwise comparison to be able to formulate rejection of $H_0$ in terms of medians or means in Holm-corrected pairwise Wilcoxon-rank sum tests, should I also apply a (Holm?) p-value correction to assess multiple location shifts?

Answer 1

It looks like a "general" multiple testing problem to me, since you’re assessing location shift for each of the pairs of datasets? It may lead to a bigger than alpha family-wise false rejection rate and thus a smaller than alpha familywise chance to incorrectly make an interpretation on the mean/median instead of the probabilistic index, which is of course no problem (PI interpretation is of course okay for location shift models also) but may not be what you want to achieve here.

On the other hand, if you choose to apply a multiple testing correction, there may (will) be a tradeoff with the power and thus correcting for the multiple testing may lead you to falsely make interpretations on the mean/median while you should actually be making them on the probabilistic index... I think the easy and safe thing to do here is not apply a multiple testing correction.