Permutation test in multiple linear regression, which way is correct?

Question

I am fitting a linear regression model $$Y = \beta_0+ \beta_1 Z + \beta_2 X+\epsilon$$ Response $Y$ and covariate $X$ are continuous variables and $Z$ is a 0/1 dichotomous treatment group indicator, all are $N \times 1$ vectors. $\epsilon$ is a $N \times 1$ vector of normally distributed i.i.d. random errors with expectation 0 and unknown variance $\sigma^2$.

My goal is to obtain the joint distribution of 30 t-test statistics under the null hypothesis. This is to achive adjusted p-values for 30 correlated tests each corresponding to a separate response variabale $Y$. For this purpose, I want to obatin the permutation distribution of the t-test for the coefficient $\beta_1$. In permutations, which one of the following is correct:

(1) Randomly permute $Y$, and obtain the distribution of $\beta_1$ accross 1000 permutations

(2) Randomly permute $Z$, and obtain the distribution of $\beta_1$ accross 1000 permutations

(3) Let $Z(X)$ be the vector of residuals from fitting $Z$ on $X$, and let $Y(X)$ be the residuals from fitting $Y$ on $X$.
Then permute $Y(X)$ 1000 times and record the 1000 regression coefficients of $Z(X)$ when regressing $Y(X)$ on $Z(X)$.

(4) Permute $Y(X)$ 1000 times and record the 1000 regression coefficients, where $Y(X)$ is regressed on $Z$. This would be same as comparing the means of the "X-adjusted values" of $Y$ in the 2 groups.

I would think (3) is the correct way as it preserves the relationships of $X$ to $Y$ and $Z$. Thus before permutation I should clear the effect of $X$ from both $Z$ and $Y$. But (4) also does not sound completely unreasonable to me. I cannot find a good reference where this is appropriately discussed.

Answer 1

Great question. This paper compares various permutation tests for assessing the affect of a single predictor in a multivariable linear regression model:

An empirical comparison of permutation methods for tests of partial regression coefficients in a linear model

• All of the methods compared were asymptotically equivalent in most situations.
• Thus my personal preference is the approach where you simply permute the outcome, since it is the simplest.
• However, for small sample sizes or if you have outliers in the main predictor of interest, then the "Reduced model residual permutation method" from their paper may be better.

Answer 2

Inspired by the refrence provided by @jarbet, I studied the issue a bit deeper and as suggested (for example) by Anderson and Robinson(2008) Permutation Tests for Linear Models the preferred approach appears to be the Freedman-Lane method described below as method (5). Recall that in my initial post $Z$ is the variable of interest and $X$ is a nuisance variable to be adjusted for:

(5) (Freedman-Lane method)
(a) Calculate residuals $Y(X)$ (see my earlier item 3)
$\ \ \ \ \ \ \ \ \ \ $and also the corresponding fitted values $FittedY(X)$=$Y$-$Y(X)$.
(b) Permute the residuals $Y(X)$, and obtain the permuted residuals $Y^*(X)$.
(c) Regress $Y^*(X)$+$FittedY(X)$ on the original unpermuted variables $Z$ and $X$
$\ \ \ \ \ \ \ \ \ \ $and obtain coefficient for $X$ and the desired statistic (for example t-test)
(d) repeat items b,c 1000 times and obtain the permutation distribution of the statistic.

Actually this makes sense. We are shuffling only the part of $Y$ that is not related to $X$, so the effect of $X$ on $Z$ and $Y$ is not perturbed. But it is still unclear to me why this approach is better than approach (3) in my original post which also does not perturbe the effect of $X$ on $Z$ and $Y$. I assume the difference is not huge.