4
$\begingroup$

I have run a multiple regression model using a set of 10 categorical and numerical explanatory variables. This model results in 10 coefficients and their associated p-values. I would like to report the adjusted p-values (to correct for multiple comparisons). However, I would like to use a permutation test rather than the conservative Bonferroni adjustment. In this case, what is the best approach to generate the empirical distribution under the null hypothesis using permutations?

Given that some explanatory variables are categorical and others numeric, is it correct to use the empirical distribution of maximum absolute coefficients to find the p-value threshold? Alternatively, can one use the distribution of smallest p-values instead of the coefficients?

Improve this question
$\endgroup$
3
  • $\begingroup$ It's quite unclear why you want to adjust your pvalues. Do you have 10 different models? You want to correct based on FWER or FDR? Please add some context. $\endgroup$
    – Mangnier Loïc
    Commented Feb 24, 2023 at 0:25
  • $\begingroup$ I have 1 model with 10 explanatory variables and I want to correct based on FWER. I suspect Bonferroni correction is too conservative because of correlations between some explanatory variables so I was wondering whether I can construct an empirical distribution under the null to set the p-value threshold. $\endgroup$
    – lmb8
    Commented Feb 24, 2023 at 12:54
  • $\begingroup$ If you suspect multicollinearity among your variables, lasso regression should be the way to go. Anyways no need to correct for FWER if you only have one model and 10 variables. $\endgroup$
    – Mangnier Loïc
    Commented Feb 24, 2023 at 13:26

1 Answer 1

Reset to default
5
$\begingroup$

You can't do an exact permutation test for individual coefficients in multiple regression, because the necessary exchangeable quantity is unobservable.

Things you can do:

  1. An exact permutation test of the whole regression (against a null/constant model).

  2. An exact permutation test of a simple regression coefficient.

  3. An asymptotic permutation test of a regression coefficient (based on treated some form of (possibly scaled, standardized or studentized depending on where you wish to make your compromises) residual as approximately exchangeable (resampling some form of residuals without replacement). This should tend to work well in large samples.

  4. A bootstrap test of the coefficient. This will also be approximate, and should also tend to work well in large samples. (There are a couple of possible ways to do such a test, including replicating "3" but sampling with replacement rather than without.)

With single tests I would probably be using the usual t-statistic (or possibly its absolute value) as the quantity whose distribution I'd compare the sample value with.

With multiple tests you'd need to look at the specific combined criterion you want to control to have overall rate alpha.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I do not fully understand what do you mean with looking at the specific combined criterion for multiple test correction. If I were to do simple regression with the 10 explanatory variables, let's say 10 different SNPs, I would construct the empirical distribution using the maximum absolute coefficient in each permuted dataset. This would help me control for FWER. However, I am not sure that this is the best even for simple regression for my case since I have categorical and numerical variables (not SNPs). Do you have any reference for the permutation test on residuals? $\endgroup$
    – lmb8
    Commented Feb 24, 2023 at 15:04
  • $\begingroup$ I believe that using the maximum absolute coefficient will not strictly control the overall type I error, since coefficients are not all in the same units. $\endgroup$
    – Glen_b
    Commented Feb 24, 2023 at 18:24
  • $\begingroup$ (Though maybe I'm not thinking about it clearly right now.) $\endgroup$
    – Glen_b
    Commented Feb 26, 2023 at 4:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.