Adjusting p-values in multiple regression using permutation test I have run a multiple regression model using a set of 10 categorical and numerical explanatory variables. This model results in 10 coefficients and their associated p-values. I would like to report the adjusted p-values (to correct for multiple comparisons). However, I would like to use a permutation test rather than the conservative Bonferroni adjustment. In this case, what is the best approach to generate the empirical distribution under the null hypothesis using permutations?
Given that some explanatory variables are categorical and others numeric, is it correct to use the empirical distribution of maximum absolute coefficients to find the p-value threshold? Alternatively, can one use the distribution of smallest p-values instead of the coefficients?
 A: You can't do an exact permutation test for individual coefficients in multiple regression, because the necessary exchangeable quantity is unobservable.
Things you can do:

*

*An exact permutation test of the whole regression (against a null/constant model).


*An exact permutation test of a simple regression coefficient.


*An asymptotic permutation test of a regression coefficient (based on treated some form of (possibly scaled, standardized or studentized depending on where you wish to make your compromises) residual as approximately exchangeable (resampling some form of residuals without replacement). This should tend to work well in large samples.


*A bootstrap test of the coefficient. This will also be approximate, and should also tend to work well in large samples. (There are a couple of possible ways to do such a test, including replicating "3" but sampling with replacement rather than without.)
With single tests I would probably be using the usual t-statistic (or possibly its absolute value) as the quantity whose distribution I'd compare the sample value with.
With multiple tests you'd need to look at the specific combined criterion you want to control to have overall rate alpha.
