1
$\begingroup$

When conducting a permutation test, which variable should be shuffled?

Let's say I have two variables X and Y, and my test statistic is the correlation coefficient between the two variables. To test if the correlation is significant, I shuffle one of the two variables many times (e.g. 10000 permutations) and record how often the resulting correlation coefficient exceeds the original one. The proportion of exceeding correlation coefficients gives me a p-value that I can use for hypothesis testing.

So far so good, but when are there any restrictions on which variable I should shuffle? I guess it does not matter if I shuffle variable X or Y in the case of simple correlation coefficients. However, if I use the same procedure testing a multiple linear regression that has two independent variables, do I shuffle those or can I simply shuffle the dependent variable? If I need to shuffle the independent variables, do I need to shuffle them independent of each other (i.e. the observations of the two independent variables are now on different rows of the corresponding data-vectors)?

More specifically, this question originated from a problem where I correlate a variable that I derived from some raw data with another measured variable. Computing the derived variable from the raw data is expensive and therefore, I would rather shuffle the measured variable instead of the raw data. The alternative would be to shuffle the derived variable but I assume this would destroy statistical properties induced by processing the raw data (e.g. spatial filtering). I'd appreciate your thoughts on this.

$\endgroup$
3
  • 1
    $\begingroup$ Please see: stats.stackexchange.com/questions/292597/… $\endgroup$
    – Todd D
    May 19 at 20:47
  • 1
    $\begingroup$ "However, if I use the same procedure testing a multiple linear regression that has two independent variables," ... this is only going to work if you're testing the null of both coefficients =0; more generally you won't have exact exchangeability (you could shuffle residuals but they're only asymptotically exchangeable, with some conditions). Note that in regression you condition on the x's. $\endgroup$
    – Glen_b
    May 20 at 1:19
  • $\begingroup$ @Glen_b and ToddD, I picked up your comments in reply to gung's Reinstate Monica answer. $\endgroup$
    – avaruus
    May 20 at 7:44

1 Answer 1

1
$\begingroup$

With respect to a simple correlation between a single X and a single Y, you could shuffle either. That said, the p-value isn't "how often the resulting correlation coefficient exceeds the original one" unless you're interested in a one-tailed test. To be conservative, you'd double that.

In a multiple regression context, the standard tests of the variables assumes you are controlling for the other variables in the model. (There are other ways of testing variables, though.) To get a permutation test that corresponds to that, you'd leave the other X variables and the Y variable alone, and you'd just shuffle the X variable you are interested in testing.

I'm not sure I quite follow your situation / question at the end. If I had some raw data that I turned into a computed variable, C, that I entered into a multiple regression model while controlling for other X variables, I would conduct a permutation test of C by shuffling it alone. I don't see any point in shuffling the raw data and recomputing C each time first.

$\endgroup$
1
  • $\begingroup$ First part, is clear. The difference between shuffling the dependent variable or one of the independent variables is that in the first case I test if all coefficients=0 (as pointed out by Glen_b and in the question linked to by Todd D), while in the second case I test if a specific coefficient=0? In research articles, I often read that researchers permute the raw data and my intuition is (they never explicitly state why) that this is done to test against the statistical properties of the original data while preserving effects induced by the processing steps. Does that make sense? $\endgroup$
    – avaruus
    May 20 at 7:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.