I am sure this already exists but I just don't know the terminology to look for.
I have three sets of 10 measurements. Each set corresponds to a different geographic region. So in total I have 30 measurements of my variable, and I have the factor "region" with 3 levels (west region, middle region, east region).
Let's say I do a simple ANOVA and I get differences between the 3 regions. But I want to play a little with the possibility of this differences being "by chance". Or, in another scenario, let's say I can't use ANOVA for some reason (eg. strongly inhomogeneous variances) and I use a non-parametric test and I don't find differences
I want to know if it's possible to do the following (or if the idea is appropriate):
If there is really no difference between the 3 regions, then I can assume that any test (eg ANOVA or a non-parametric equivalent) will find approximate the same results even if I randomly mix all data once and again. So I thought I could simulate this, using my own data but just in different grouping. for example: 1- take all the 30 values from my own measurements 2- shuffle them into 3 groups, ie. randomly choose 10 values and assign them to a randomly chosen group; repeat with the next 10 data and then you have again 3 groups of 10 measurements. 3- run the test (eg. ANOVA)
Now I go back to 1, and repeat this eg 1000 times, and see if there is a convergence towards a "stable" pattern. If there is, then there are actually no differences. If the convergence deviates a lot from the results I found with my "real" dataset, then I may think there are actually differences between the 3 regions.
Is my reasoning correct/sound? I know there is something like this, I just don't remember the name.. I thought it was related to permutations but I'm not sure...