shuffle my data to investigate differences between 3 groups

Question

I am sure this already exists but I just don't know the terminology to look for.

I have three sets of 10 measurements. Each set corresponds to a different geographic region. So in total I have 30 measurements of my variable, and I have the factor "region" with 3 levels (west region, middle region, east region).

Let's say I do a simple ANOVA and I get differences between the 3 regions. But I want to play a little with the possibility of this differences being "by chance". Or, in another scenario, let's say I can't use ANOVA for some reason (eg. strongly inhomogeneous variances) and I use a non-parametric test and I don't find differences

I want to know if it's possible to do the following (or if the idea is appropriate):

If there is really no difference between the 3 regions, then I can assume that any test (eg ANOVA or a non-parametric equivalent) will find approximate the same results even if I randomly mix all data once and again. So I thought I could simulate this, using my own data but just in different grouping. for example: 1- take all the 30 values from my own measurements 2- shuffle them into 3 groups, ie. randomly choose 10 values and assign them to a randomly chosen group; repeat with the next 10 data and then you have again 3 groups of 10 measurements. 3- run the test (eg. ANOVA)

Now I go back to 1, and repeat this eg 1000 times, and see if there is a convergence towards a "stable" pattern. If there is, then there are actually no differences. If the convergence deviates a lot from the results I found with my "real" dataset, then I may think there are actually differences between the 3 regions.

Is my reasoning correct/sound? I know there is something like this, I just don't remember the name.. I thought it was related to permutations but I'm not sure...

Answer 1

If there is really no difference between the 3 regions, then I can assume that any test (eg ANOVA or a non-parametric equivalent) will find approximate the same results even if I randomly mix all data once and again.

This is the central insight that underlies resampling methods, such as permutation tests / randomization tests.

e.g. see Wikipedia, for example here

The basic idea of a permutation test (let's take a one way ANOVA-like situation) is that if the null is true, the group labels are arbitrary - you don't change the distributions by shuffling them.

So if you look at all possible arrangements of the group labels and compute some test statistic of interest, you obtain what's called the permutation distribution of the test statistic. You can then see if your particular sample (which will be one of the possible permutations - or more accurately, possible combinations) is unusually far 'in the tails' of that null distribution (giving a p-values).

Many of the common nonparametric rank-based tests are actually permutation-tests carried out on the ranks (which is a practical way of doing permutation tests without computers, which are otherwise very tedious unless you have very small sample sizes).

When the sample sizes are large, an option is to sample (with replacement) from the permutation distribution, typically because there are too many combinations to evaluate them all. Generally this is achieved by randomly permuting the labels rather than systematically re-arranging them to cover every possibility. The test statistic is then computed for each such arrangement. The sample value of the statistic is then compared with the distribution (it is normally included as part of the distribution for computing the p-value, and counts in the values 'at least as extreme' as itself). Some authors call this sampled permutation test a randomization test (though other authors reserve that term for a somewhat different notion also connected to permutation tests).

What you described was pretty close to this randomly sampled permutation test (randomization test).

I advise trying such a randomization test, not least for its ability to expand your horizons in terms of the standard tools you have available for tackling problems. The procedure is distribution-free (conditional on the sample) - it requires fewer assumptions while still allowing you to use either familiar statistics or ones custom-designed to your circumstances (e.g. you could slot in a more robust measure of location).

In practice I'd advise more than 1000 resamples for a randomization test. Consider a test with a p-value near 5%. The standard error of an estimated p-value for a sample size of 1000 will be nearly 0.007; when the true p-value is just on one side of 5%, nearly 15% of the time you'll see a value more than 1% on the wrong side (more than 6% or less than 4% when it should be the other side). I usually regard 10000 as toward the low end of what I do unless I just want a rough idea of the ballpark of the p-value. If I was doing a formal test, I'd want to pin it down a bit better. I often do 100,000 and sometimes a million or more - at least for the simpler tests.

If you search here on permutation tests or randomization tests you should find a number of relevant questions and answers and even some examples.