Permutation testing: Which variable should be shuffled?

Question

When conducting a permutation test, which variable should be shuffled?

Let's say I have two variables X and Y, and my test statistic is the correlation coefficient between the two variables. To test if the correlation is significant, I shuffle one of the two variables many times (e.g. 10000 permutations) and record how often the resulting correlation coefficient exceeds the original one. The proportion of exceeding correlation coefficients gives me a p-value that I can use for hypothesis testing.

So far so good, but when are there any restrictions on which variable I should shuffle? I guess it does not matter if I shuffle variable X or Y in the case of simple correlation coefficients. However, if I use the same procedure testing a multiple linear regression that has two independent variables, do I shuffle those or can I simply shuffle the dependent variable? If I need to shuffle the independent variables, do I need to shuffle them independent of each other (i.e. the observations of the two independent variables are now on different rows of the corresponding data-vectors)?

More specifically, this question originated from a problem where I correlate a variable that I derived from some raw data with another measured variable. Computing the derived variable from the raw data is expensive and therefore, I would rather shuffle the measured variable instead of the raw data. The alternative would be to shuffle the derived variable but I assume this would destroy statistical properties induced by processing the raw data (e.g. spatial filtering). I'd appreciate your thoughts on this.

Answer 1

With respect to a simple correlation between a single X and a single Y, you could shuffle either. That said, the p-value isn't "how often the resulting correlation coefficient exceeds the original one" unless you're interested in a one-tailed test. To be conservative, you'd double that.

In a multiple regression context, the standard tests of the variables assumes you are controlling for the other variables in the model. (There are other ways of testing variables, though.) To get a permutation test that corresponds to that, you'd leave the other X variables and the Y variable alone, and you'd just shuffle the X variable you are interested in testing.

I'm not sure I quite follow your situation / question at the end. If I had some raw data that I turned into a computed variable, C, that I entered into a multiple regression model while controlling for other X variables, I would conduct a permutation test of C by shuffling it alone. I don't see any point in shuffling the raw data and recomputing C each time first.