Suppose I have $k$ samples of different sizes, each of a different univariate variable. I want to test the significance of the hypothesis that the population of $k_0$ has a mean greater than all of the other means.

I know of one-way ANOVA, but my understanding is that it only checks for significant differences across the groups.

Is there a statistical test that establishes significance for which mean is greatest? Feel free to give Bayesian or Frequentist solutions (preferably both).

Here is my example:

I have to compare the heights of 5 different people; however, my tape measure operator has a very unsteady hand, so there is significant variance on measurements. I can measure as many times as I like. Obviously I do not know beforehand which person is tallest, but I would like to give a confidence that a particular person is tallest.

Would I perform 5 one-versus-all two-sample $t$-tests, and quintuple their $p$-values?

What would be the Bayesian approach?

  • $\begingroup$ Can you show you how I would engage with a bootstrap approach? I'm much more of a computer-simulation guy than a algebraic-manipulation guy. $\endgroup$ Jun 29, 2015 at 18:00
  • $\begingroup$ @EngrStudent You might on each bootstrap compute all 5 sample means, and then tally the frequencies with which any particular sample has the greatest of all 5 sample means. The $p$-value is then how often one group's mean is not the greatest. (edit: this is not correct, I believe) $\endgroup$ Jun 29, 2015 at 18:36
  • $\begingroup$ significance is a frequentist concept, so you won't find a Bayesian version. $\endgroup$ Jun 29, 2015 at 18:40
  • $\begingroup$ @MarcClaesen What about $\text{P}\left(\text{H}_0\right)$? $\endgroup$ Jun 29, 2015 at 18:41
  • 2
    $\begingroup$ Just to be sure : Is $k_o$ picked before the experiment ? Have you thought about crafting a permutation test ? $\endgroup$
    – brumar
    Jun 29, 2015 at 19:32

2 Answers 2


So I'm thinking about bootstrapping.


  • There are 200 measurements for each height
  • the true height values are: 1.6, 1.7, 1.8, 1.9 and x
  • the standard deviation of the variation in height from true is 10% of the true
  • "x" is 0.5 standard deviations OUTSIDE the range of the others


a. the value of "x"
b. a measure of outlier-ness of "x" from its "buddies"


part a.

the standard deviation of the given values is:

x <- 1.9 + 0.5* sd(c(1.6, 1.7, 1.8, 1.9))

which yields $$ x = 1.96455 $$

part b.


mu_true <- c(1.6, 1.7, 1.8, 1.9, 1.96455)
sig_scale <- 0.1
maxloops <- 200

y <- matrix(nrow=maxloops,ncol=5)

#main loop
for (i in 1:maxloops){

    y[i,1] <- rnorm(1,mean=mu_true[1],sd=sig_scale*mu_true[1])
    y[i,2] <- rnorm(1,mean=mu_true[2],sd=sig_scale*mu_true[2])
    y[i,3] <- rnorm(1,mean=mu_true[3],sd=sig_scale*mu_true[3])
    y[i,4] <- rnorm(1,mean=mu_true[4],sd=sig_scale*mu_true[4])
    y[i,5] <- rnorm(1,mean=mu_true[5],sd=sig_scale*mu_true[5])




title(main="Compare measurements",xlab="person index",ylab="measurement")

This gives a decent picture of variation across the various groups. variation on variability plot

This reminds me of the comparison of means platform in JMP. (link)

Inspection can show that groups 4 and 5 are very similar while group 1 and 5 are moderately different. The mean of group 5 is outside the shoulders of groups 1, 2, and 3. If I sampled and computed the mean of group 1 for 300 iterations then I would estimate the range of variation in the mean as 0.0648.

So I try this for the remainder, using the following code:

mu_true <- c(1.6, 1.7, 1.8, 1.9, 1.96455)
sig_scale <- 0.1
maxloops <- 300

y <- matrix(nrow=maxloops,ncol=5)

#main loop
for (i in 1:maxloops){

    temp1 <- rnorm(200,mean=mu_true[1],sd=sig_scale*mu_true[1])
    y[i,1] <- mean(temp1)

    temp2 <- rnorm(200,mean=mu_true[2],sd=sig_scale*mu_true[2])
    y[i,2] <- mean(temp2)

    temp3 <- rnorm(200,mean=mu_true[3],sd=sig_scale*mu_true[3])
    y[i,3] <- mean(temp3)

    temp4 <- rnorm(200,mean=mu_true[4],sd=sig_scale*mu_true[4])
    y[i,4] <- mean(temp4)

    temp5 <- rnorm(200,mean=mu_true[5],sd=sig_scale*mu_true[5])
    y[i,5] <- mean(temp5)

eY <- colMeans(y)

sY1 <- sd(y[,1])
sY2 <- sd(y[,2])
sY3 <- sd(y[,3])
sY4 <- sd(y[,4])
sY5 <- sd(y[,5])

ucl <- numeric()
ucl[1] <- eY[1]+3*sY1
ucl[2] <- eY[2]+3*sY2
ucl[3] <- eY[3]+3*sY3
ucl[4] <- eY[4]+3*sY4
ucl[5] <- eY[5]+3*sY5

lcl <- numeric()
lcl[1] <- eY[1]-3*sY1
lcl[2] <- eY[2]-3*sY2
lcl[3] <- eY[3]-3*sY3
lcl[4] <- eY[4]-3*sY4
lcl[5] <- eY[5]-3*sY5


My results were:

1.636037 1.735657 1.840171 1.937747 2.005664
1.599576 1.700506 1.800816 1.899996 1.963631
1.563115 1.665356 1.761460 1.862244 1.921598

To me this says that as long as the mean is between 1.56 and 1.63, then it should be group 1. As long as the mean is between 1.66 and 1.73 then it should be group 2. And so on. The problem arises in that the upper limit of group 4 is above the lower limit of group 5. There is overlap. There is a small but nonzero chance they might be from the same distribution - given the sample size, the mean, and the variation.

I think Tukey's method can also apply.

  • 1
    $\begingroup$ If I were more clever I would have set box-plot shoulders to the 3-$\sigma$ limit for the range of the mean. $\endgroup$ Jun 30, 2015 at 0:09

Would I perform 5 one-versus-all two-sample t-tests, and quintuple their p-values?

Intuitively, I would think that it's less powerful than a permutation test. I hope you will receive others answer to this question as I would like to know what the alternatives are.

My suggestion is to compute how unlikely it is that the group with the biggest mean is that far or farther from the group with the second biggest mean. To measure how unlikely it is, you do a random permutation mixing which individual is in which group. At each permutation you look if the difference between the two biggest mean is bigger or equal to the two biggest mean you currently have. If it's a very rare event, you prove your point that the apparent gap between the biggest mean and the others cannot be caused by chance.

In more algorithmic way :

1-Compute $d=mean(k_0)-max(mean(k_1),mean(k_2)...mean(k_n))$
2-Permute the label indicating which points belongs to which group.
For each of these permuted dataset :
3-Compute $d'$ with taking care to pick as $k_{0}$ the group which has the biggest mean (because you mentioned that $k_{0}$ is selected after the experiment).

4- Count the ratio of times when $d'\geqslant d$, this is your p-value.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.