I have MLE curve parameter estimates for 3 populations (2 per population), and am looking for a clever way to compare them. At the moment, I am non-parametrically bootstrapping my datasets in order to create n datasets, which I then compare using a standard MANOVA procedure. I have a number of issues with this, one of them being that the variances of my bootstrapped parameters are heteroskedastic and non normally distributed.

Ideally, I was thinking that I could compare the bootstrapped confidence intervals around my parameters, i.e. show that the bootstrapped confidence intervals around each of my popuation parameters "overlap".

I was looking at bivariate comparisons, but many of these rely on the fact that it is a MV gaussian distribution, which according to multivariate shapiro tests, I do not have.

To wrap things up, my questions come down to this:

  1. What would people see as the most sound procedure?
  2. I had a look through this post Comparing points in a bivariate space, however, am curious about the second answer and if anybody has any suggestions as to where I could find information on non normal mulitvariate analysis.

Thanks for your help.


1 Answer 1


Bootstrapping is naturally a way of estimating the parameters under the alternative, not under the null. As such it does not immediately lend itself to hypothesis testing. In order to perform hypothesis testing using resampling procedures you must devise a way to resample under the null. Presumably the null hypothesis you wish to reject is that the three populations are not all the same. A simple permutation test will suffice: Instead of resampling the data, permute the group labels.

The idea is basically that if the null is true, then everyone is from the same population and you expect to get similar statistics no matter how you form your 3 groups. So about B = 10,000 times or so, put all of your data in one big pot and then divide them into 3 new groups. Then, with your new permuted data, which has the same distribution as your observed data if and only if the null is true, you want to calculate a statistic that allows you to compare this null distribution to the observed distribution. I am not immediately familiar with what a good statistic would be. Off the top of my head I would consider $$\hat{S} = \|\hat{\theta}_1-\hat{\theta}_2\|^2 + \|\hat{\theta}_2-\hat{\theta}_3\|^2+\|\hat{\theta}_3-\hat{\theta}_1\|^2$$ where $\hat{\theta}_j = (\hat{\alpha}_j, \hat{\beta}_j), j=1,2,3.$ We would expect $\hat{S}$ to be small if the null is true, and large if it isn't. For each relabeled data set you get a new $\hat{S}^*$. Altogether your $B$ bootstrapped $\hat{S}^*$ form the null distribution of $\hat{S}$. Then you just look to see how extreme the observed $\hat{S}$ is in comparison to the null distribution. If it is more extreme than 95% of the bootstrap samples, you reject, that is,
$$p = \frac{1}{B}\sum_{b=1}^B I(\hat{S}^*>\hat{S}) $$ and you reject if $p<.05$.

The problem with this statistic is that it might not have the correct size, and it might not be very powerful for this hypothesis. I'm sorry I don't know of a better test statistic off the top of my head--perhaps you can adapt your MANOVA idea (I don't really know what a MANOVA is) to be under the null.

If you reject you may proceed to pairwise comparisons in an analogous manner.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.